🥇Conto aplikasi anu didorong acara dumasar kana webhooks dina panyimpenan objék S3 Mail.ru Cloud Solutions

Mesin kopi Rube Goldberg

Arsitéktur anu didorong ku acara ningkatkeun efisiensi biaya sumber daya anu dianggo sabab ngan ukur dianggo dina waktos anu diperyogikeun. Aya seueur pilihan ngeunaan cara nerapkeun ieu sareng henteu nyiptakeun éntitas awan tambahan salaku aplikasi pagawe. Sareng dinten ayeuna kuring moal ngobrol ngeunaan FaaS, tapi ngeunaan webhooks. Kuring bakal nunjukkeun conto tutorial ngeunaan nanganan acara nganggo panyimpen obyék wéb.

Sababaraha kecap ngeunaan neundeun objék jeung webhooks. Panyimpen obyék ngamungkinkeun anjeun pikeun nyimpen data dina méga dina bentuk objék, tiasa diaksés via S3 atanapi API anu sanés (gumantung kana palaksanaan) ngalangkungan HTTP / HTTPS. Webhooks umumna custom HTTP callbacks. Aranjeunna biasana dipicu ku hiji acara, sapertos kode anu didorong ka gudang atanapi koméntar anu dipasang dina blog. Nalika hiji kajadian lumangsung, situs asal ngirim hiji pamundut HTTP ka URL dieusian pikeun webhook nu. Hasilna, anjeun tiasa ngadamel acara dina hiji situs memicu tindakan dina situs anu sanés (wiki). Dina kasus dimana situs sumber mangrupa gudang obyék, acara meta salaku parobahan eusina.

Conto kasus basajan nalika automation sapertos tiasa dianggo:

Nyiptakeun salinan sadaya objék dina panyimpenan awan anu sanés. Salinan kudu dijieun on laleur iraha wae file ditambahkeun atawa dirobah.
Nyieun otomatis runtuyan gambar leutik tina file grafis, nambahkeun watermarks kana poto, sarta modifikasi gambar lianna.
Bewara ngeunaan datangna dokumén anyar (contona, jasa akuntansi anu disebarkeun unggah laporan ka awan, sareng ngawaskeun kauangan nampi béwara ngeunaan laporan énggal, pariksa sareng nganalisaana).
Kasus anu rada rumit ngalibatkeun, contona, ngahasilkeun pamundut ka Kubernetes, anu nyiptakeun pod sareng wadah anu diperyogikeun, ngalebetkeun parameter tugas ka dinya, sareng saatos ngolah wadahna runtuh.

Salaku conto, urang bakal nyieun varian tugas 1, nalika parobahan dina Mail.ru Cloud Solutions (MCS) ember gudang obyék disingkronkeun dina neundeun objék AWS maké webhooks. Dina kasus nyata dimuat, karya Asynchronous kudu disadiakeun ku ngadaptar webhooks dina antrian, tapi pikeun tugas latihan urang bakal ngalakukeun palaksanaan tanpa ieu.

Skéma gawé

Protokol interaksi dijelaskeun sacara rinci dina Pituduh ka S3 webhooks on MCS. Skéma karya ngandung unsur handap:

jasa penerbitan, nu aya di sisi gudang S3 sarta publishes requests HTTP nalika webnhook ieu dipicu.
Webhook narima server, anu ngadangukeun pamundut ti jasa penerbitan HTTP sareng ngalaksanakeun tindakan anu luyu. Server tiasa ditulis dina basa naon waé; dina conto urang, urang bakal nyerat server dina Go.

A fitur husus tina palaksanaan webhooks dina S3 API nyaeta pendaptaran ti webhook narima server dina layanan penerbitan. Khususna, server panampi webhook kedah ngonfirmasi langganan kana seratan tina jasa penerbitan (dina palaksanaan webhook anu sanés, konfirmasi langganan biasana henteu diperyogikeun).

Sasuai, server panarima webhook kudu ngarojong dua operasi utama:

ngabales pamundut jasa penerbitan pikeun ngonfirmasi pendaptaran,
prosés kajadian datang.

Masang server panarima webhook

Pikeun ngajalankeun server panarima webhook, anjeun peryogi server Linux. Dina artikel ieu, sabagé conto, kami nganggo instansi virtual anu kami sebarkeun dina MCS.

Hayu urang pasang parangkat lunak anu diperyogikeun sareng luncurkeun server panampi webhook.

ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install git
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  bc dns-root-data dnsmasq-base ebtables landscape-common liblxc-common 
liblxc1 libuv1 lxcfs lxd lxd-client python3-attr python3-automat 
python3-click python3-constantly python3-hyperlink
  python3-incremental python3-pam python3-pyasn1-modules 
python3-service-identity python3-twisted python3-twisted-bin 
python3-zope.interface uidmap xdelta3
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui 
gitk gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  git
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
Need to get 3915 kB of archives.
After this operation, 32.3 MB of additional disk space will be used.
Get:1 http://MS1.clouds.archive.ubuntu.com/ubuntu bionic-updates/main 
amd64 git amd64 1:2.17.1-1ubuntu0.7 [3915 kB]
Fetched 3915 kB in 1s (5639 kB/s)
Selecting previously unselected package git.
(Reading database ... 53932 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.7_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.7) ...
Setting up git (1:2.17.1-1ubuntu0.7) ...

Klon polder sareng server panampi webhook:

ubuntu@ubuntu-basic-1-2-10gb:~$ git clone
https://github.com/RomanenkoDenys/s3-webhook.git
Cloning into 's3-webhook'...
remote: Enumerating objects: 48, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 114 (delta 20), reused 45 (delta 18), pack-reused 66
Receiving objects: 100% (114/114), 23.77 MiB | 20.25 MiB/s, done.
Resolving deltas: 100% (49/49), done.

Hayu urang ngamimitian server:

ubuntu@ubuntu-basic-1-2-10gb:~$ cd s3-webhook/
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80

Ngalanggan jasa penerbitan

Anjeun tiasa ngadaptarkeun server panarima webhook anjeun ngaliwatan API atawa panganteur web. Pikeun kesederhanaan, urang bakal ngadaptar via panganteur wéb:

Hayu urang angkat ka bagian ember di rohangan kadali.
Pindah ka ember dimana urang bakal ngonpigurasikeun webhooks teras klik dina gear:

Buka tab Webhooks teras klik Tambahkeun:

Eusian widang:

ID - nami webhook.

Kajadian - nu acara pikeun ngirimkeun. Kami parantos nyetél pangiriman sadaya kajadian anu lumangsung nalika damel sareng file (nambahkeun sareng ngahapus).

URL - webhook narima alamat server.

Awalan/akhiran saringan nyaéta saringan anu ngamungkinkeun anjeun ngahasilkeun webhook ngan ukur pikeun objék anu namina cocog sareng aturan anu tangtu. Contona, dina urutan pikeun webhook pikeun memicu ukur file kalawan extension .png, di Sufiks saringan anjeun kedah nyerat "png".

Ayeuna, ngan port 80 jeung 443 nu dirojong pikeun ngakses webhook narima server.

Hayu urang klik Tambahkeun hook sareng urang bakal ningali ieu:

Hook ditambahkeun.

Server panampi webhook nunjukkeun dina log na kamajuan prosés pendaptaran hook:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80
2020/06/15 12:01:14 [POST] incoming HTTP request from 
95.163.216.92:42530
2020/06/15 12:01:14 Got timestamp: 2020-06-15T15:01:13+03:00 TopicArn: 
mcs5259999770|myfiles-ash|s3:ObjectCreated:*,s3:ObjectRemoved:* Token: 
E2itMqAMUVVZc51pUhFWSp13DoxezvRxkUh5P7LEuk1dEe9y URL: 
http://89.208.199.220/webhook
2020/06/15 12:01:14 Generate responce signature: 
3754ce36636f80dfd606c5254d64ecb2fd8d555c27962b70b4f759f32c76b66d

Pendaptaran parantos réngsé. Dina bagian salajengna, urang bakal nyandak katingal ngadeukeutan dina algoritma operasi tina webhook narima server.

Katerangan ngeunaan webhook narima server

Dina conto urang, server ditulis dina Go. Hayu urang nempo prinsip dasar operasi na.

package main

// Generate hmac_sha256_hex
func HmacSha256hex(message string, secret string) string {
}

// Generate hmac_sha256
func HmacSha256(message string, secret string) string {
}

// Send subscription confirmation
func SubscriptionConfirmation(w http.ResponseWriter, req *http.Request, body []byte) {
}

// Send subscription confirmation
func GotRecords(w http.ResponseWriter, req *http.Request, body []byte) {
}

// Liveness probe
func Ping(w http.ResponseWriter, req *http.Request) {
    // log request
    log.Printf("[%s] incoming HTTP Ping request from %sn", req.Method, req.RemoteAddr)
    fmt.Fprintf(w, "Pongn")
}

//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {
}

func main() {

    // get command line args
    bindPort := flag.Int("port", 80, "number between 1-65535")
    bindAddr := flag.String("address", "", "ip address in dot format")
    flag.StringVar(&actionScript, "script", "", "external script to execute")
    flag.Parse()

    http.HandleFunc("/ping", Ping)
    http.HandleFunc("/webhook", Webhook)

log.Fatal(http.ListenAndServe(*bindAddr+":"+strconv.Itoa(*bindPort), nil))
}

Mertimbangkeun fungsi utama:

Ping () - jalur nu responds via URL / ping, palaksanaan pangbasajanna usik liveness.
Webhook() - jalur utama, URL/webhook handler:
- negeskeun pendaptaran dina jasa penerbitan (buka fungsi Konfirmasi Langganan),
- prosés webhooks asup (fungsi Gorecords).
Fungsi HmacSha256 sareng HmacSha256hex mangrupikeun palaksanaan algoritma enkripsi HMAC-SHA256 sareng HMAC-SHA256 kalayan kaluaran salaku senar héksadesimal pikeun ngitung tanda tangan.
utama nyaéta fungsi utama, prosés parameter garis paréntah sarta registers pawang URL.

Parameter garis paréntah ditarima ku server:

-port teh port nu server bakal ngadangukeun.
-alamat - alamat IP nu server bakal ngadangukeun.
-naskah mangrupa program éksternal anu disebut pikeun tiap hook asup.

Hayu urang tingali sababaraha fungsi:

//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {

    // Read body
    body, err := ioutil.ReadAll(req.Body)
    defer req.Body.Close()
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }

    // log request
    log.Printf("[%s] incoming HTTP request from %sn", req.Method, req.RemoteAddr)
    // check if we got subscription confirmation request
    if strings.Contains(string(body), 
""Type":"SubscriptionConfirmation"") {
        SubscriptionConfirmation(w, req, body)
    } else {
        GotRecords(w, req, body)
    }

}

Pungsi ieu nangtukeun naha pamundut pikeun ngonfirmasi pendaptaran atawa webhook geus anjog. Ieu di handap ti dokuméntasi, upami pendaptaran dikonfirmasi, struktur Json ieu ditampi dina pamundut Post:

POST http://test.com HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation
content-type: application/json

{
    "Timestamp":"2019-12-26T19:29:12+03:00",
    "Type":"SubscriptionConfirmation",
    "Message":"You have chosen to subscribe to the topic $topic. To confirm the subscription you need to response with calculated signature",
    "TopicArn":"mcs2883541269|bucketA|s3:ObjectCreated:Put",
    "SignatureVersion":1,
    "Token":«RPE5UuG94rGgBH6kHXN9FUPugFxj1hs2aUQc99btJp3E49tA»
}

Patarosan ieu kedah dijawab:

content-type: application/json

{"signature":«ea3fce4bb15c6de4fec365d36bcebbc34ccddf54616d5ca12e1972f82b6d37af»}

Dimana tanda tangan diitung salaku:

signature = hmac_sha256(url, hmac_sha256(TopicArn, 
hmac_sha256(Timestamp, Token)))

Upami webhook sumping, struktur pamundut Post sapertos kieu:

POST <url> HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation

{ "Records":
    [
        {
            "s3": {
                "object": {
                    "eTag":"aed563ecafb4bcc5654c597a421547b2",
                    "sequencer":1577453615,
                    "key":"some-file-to-bucket",
                    "size":100
                },
            "configurationId":"1",
            "bucket": {
                "name": "bucketA",
                "ownerIdentity": {
                    "principalId":"mcs2883541269"}
                },
                "s3SchemaVersion":"1.0"
            },
            "eventVersion":"1.0",
            "requestParameters":{
                "sourceIPAddress":"185.6.245.156"
            },
            "userIdentity": {
                "principalId":"2407013e-cbc1-415f-9102-16fb9bd6946b"
            },
            "eventName":"s3:ObjectCreated:Put",
            "awsRegion":"ru-msk",
            "eventSource":"aws:s3",
            "responseElements": {
                "x-amz-request-id":"VGJR5rtJ"
            }
        }
    ]
}

Sasuai, gumantung kana pamundut, Anjeun kudu ngarti kumaha carana ngolah data. Kuring milih entri salaku indikator "Type":"SubscriptionConfirmation", Kusabab éta hadir dina pamundut konfirmasi langganan jeung teu aya dina webhook nu. Dumasar kana ayana / henteuna éntri ieu dina pamundut POST, palaksanaan salajengna program mana boh kana fungsi. SubscriptionConfirmation, atawa kana fungsi GotRecords.

Kami moal nganggap fungsi Konfirmasi Langganan sacara rinci; éta dilaksanakeun dumasar kana prinsip anu dijelaskeun dina dokuméntasi. Anjeun tiasa ningali kode sumber pikeun fungsi ieu di proyék git repositories.

Fungsi GotRecords parses hiji pamundut asup jeung unggal objék Rékam nelepon hiji Aksara éksternal (anu ngaranna diliwatan dina parameter -script) kalawan parameter:

ngaran ember
konci objék
tindakan:
- salinan - lamun dina pamundut aslina EventName = ObjectCreated | PutObject | PutObjectCopy
- ngahapus - lamun dina pamundut aslina EventName = ObjectRemoved | HapusObjék

Ku kituna, lamun hook a datangna kalawan pamundut Post, sakumaha ditétélakeun luhur, sareng parameter -script=script.sh teras naskah bakal disebat kieu:

script.sh  bucketA some-file-to-bucket copy

Ieu kudu dipikaharti yén webhook narima server sanes solusi produksi lengkep, tapi conto saderhana tina palaksanaan mungkin.

Conto gawé

Hayu urang nyingkronkeun file tina ember utama di MCS ka ember cadangan di AWS. Ember utama disebut myfiles-ash, anu cadangan disebut myfiles-backup (konfigurasi LIPI di AWS saluareun ruang lingkup artikel ieu). Sasuai, nalika file disimpen dina ember utama, salinan na kedah muncul dina cadangan, sareng nalika dihapus tina anu utama, éta kedah dihapus dina cadangan.

Urang bakal tiasa dianggo sareng ember nganggo utilitas awscli, anu cocog sareng panyimpenan awan MCS sareng panyimpenan awan AWS.

ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install awscli
Reading package lists... Done
Building dependency tree
Reading state information... Done
After this operation, 34.4 MB of additional disk space will be used.
Unpacking awscli (1.14.44-1ubuntu1) ...
Setting up awscli (1.14.44-1ubuntu1) ...

Hayu urang ngonpigurasikeun aksés ka S3 MCS API:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile mcs
AWS Access Key ID [None]: hdywEPtuuJTExxxxxxxxxxxxxx
AWS Secret Access Key [None]: hDz3SgxKwXoxxxxxxxxxxxxxxxxxx
Default region name [None]:
Default output format [None]:

Hayu urang ngonpigurasikeun aksés ka API AWS S3:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile aws
AWS Access Key ID [None]: AKIAJXXXXXXXXXXXX
AWS Secret Access Key [None]: dfuerphOLQwu0CreP5Z8l5fuXXXXXXXXXXXXXXXX
Default region name [None]:
Default output format [None]:

Hayu urang pariksa aksés:

Pikeun AWS:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile aws
2020-07-06 08:44:11 myfiles-backup

Pikeun MCS, nalika ngajalankeun paréntah anjeun kedah nambihan -endpoint-url:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile mcs --endpoint-url 
https://hb.bizmrg.com
2020-02-04 06:38:05 databasebackups-0cdaaa6402d4424e9676c75a720afa85
2020-05-27 10:08:33 myfiles-ash

Diaksés.

Ayeuna hayu urang nulis naskah pikeun ngolah hook nu datang, hayu urang sebut wae s3_backup_mcs_aws.sh

#!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"

SOURCE_BUCKET=""
SOURCE_FILE=""
ACTION=""

SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"

case ${ACTION} in
    "copy")
    ${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
    ${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
    rm ${TEMP}
    ;;

    "delete")
    ${AWSCLI_AWS} rm ${TARGET}
    ;;

    *)
    echo "Usage: #!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"
SOURCE_BUCKET="${1}"
SOURCE_FILE="${2}"
ACTION="${3}"
SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"
case ${ACTION} in
"copy")
${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
rm ${TEMP}
;;
"delete")
${AWSCLI_AWS} rm ${TARGET}
;;
*)
echo "Usage: ${0} sourcebucket sourcefile copy/delete"
exit 1
;;
esac sourcebucket sourcefile copy/delete"
    exit 1
    ;;
esac

Hayu urang ngamimitian server:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80 -
script scripts/s3_backup_mcs_aws.sh

Hayu urang tingali kumaha gawéna. Ngaliwatan panganteur web MCS tambahkeun file test.txt kana ember myfiles-ash. Log konsol nunjukkeun yén pamundut dilakukeun ka server webhook:

2020/07/06 09:43:08 [POST] incoming HTTP request from 
95.163.216.92:56612
download: s3://myfiles-ash/test.txt to ../../../tmp/myfiles-ash/test.txt
upload: ../../../tmp/myfiles-ash/test.txt to 
s3://myfiles-backup/test.txt

Hayu urang parios eusi ember myfiles-backup di AWS:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls 
myfiles-backup
2020-07-06 09:43:10       1104 test.txt

Ayeuna, ngaliwatan antarmuka wéb, urang bakal ngahapus file tina ember myfiles-ash.

Log server:

2020/07/06 09:44:46 [POST] incoming HTTP request from 
95.163.216.92:58224
delete: s3://myfiles-backup/test.txt

eusi ember:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls 
myfiles-backup
ubuntu@ubuntu-basic-1-2-10gb:~$

Filena dihapus, masalahna direngsekeun.

Kacindekan jeung ToDo

Sadaya kode anu dianggo dina tulisan ieu nyaéta dina gudang kuring. Aya ogé conto naskah jeung conto cacah tanda tangan pikeun ngadaptar webhooks.

Kode ieu teu leuwih ti hiji conto kumaha anjeun tiasa nganggo S3 webhooks dina kagiatan anjeun. Sakumaha anu kuring nyarios di awal, upami anjeun ngarencanakeun ngagunakeun server sapertos kitu dina produksi, anjeun kedah sahenteuna nyerat ulang server pikeun padamelan anu teu sinkron: ngadaptarkeun webhooks anu asup dina antrian (RabbitMQ atanapi NATS), sareng ti dinya parse sareng ngolah aranjeunna. kalawan aplikasi pagawe. Upami teu kitu, nalika webhooks sumping sacara ageung, anjeun tiasa mendakan kakurangan sumber server pikeun ngarengsekeun tugas. Ayana antrian ngamungkinkeun anjeun ngadistribusikaeun pangladén sareng pagawé, ogé ngarengsekeun masalah sareng ngulang tugas upami gagal. Disarankeun ogé pikeun ngarobih logging ka anu langkung rinci sareng langkung standar.

Ayaan!

Bacaan langkung seueur ngeunaan topik:

sumber: www.habr.com

Conto aplikasi anu didorong acara dumasar kana webhooks dina panyimpenan obyék S3 Mail.ru Cloud Solutions