🥇Conto aplikasi sing didhukung acara adhedhasar webhooks ing panyimpenan obyek S3 Mail.ru Cloud Solutions

Mesin kopi Rube Goldberg

Arsitektur sing didorong acara nambah efisiensi biaya sumber daya sing digunakake amarga mung digunakake nalika dibutuhake. Ana akeh opsi babagan carane ngleksanakake iki lan ora nggawe entitas awan tambahan minangka aplikasi pekerja. Lan dina iki aku ora bakal ngomong babagan FaaS, nanging babagan webhooks. Aku bakal nuduhake conto tutorial nangani acara nggunakake webhooks panyimpenan obyek.

Sawetara tembung babagan panyimpenan obyek lan webhook. Panyimpenan obyek ngidini sampeyan nyimpen data apa wae ing awan ing bentuk obyek, bisa diakses liwat S3 utawa API liyane (gumantung implementasine) liwat HTTP / HTTPS. Webhooks umume minangka panggilan balik HTTP khusus. Biasane dipicu dening acara, kayata kode sing di-push menyang repositori utawa komentar sing dikirim ing blog. Nalika ana acara, situs asal ngirim panjalukan HTTP menyang URL sing ditemtokake kanggo webhook. Akibaté, sampeyan bisa nggawe acara ing siji situs micu tumindak ing liyane (wiki). Ing kasus ngendi situs sumber minangka panyimpenan obyek, acara tumindak minangka owah-owahan kanggo isi.

Conto kasus prasaja nalika otomatisasi kasebut bisa digunakake:

Nggawe salinan kabeh obyek ing panyimpenan maya liyane. Salinan kudu digawe kanthi cepet nalika file ditambahake utawa diganti.
Nggawe otomatis seri gambar cilik file grafis, nambahake watermark ing foto, lan modifikasi gambar liyane.
Kabar babagan tekane dokumen anyar (contone, layanan akuntansi sing disebarake ngunggah laporan menyang awan, lan ngawasi finansial nampa kabar babagan laporan anyar, mriksa lan nganalisa).
Kasus sing rada rumit kalebu, contone, nggawe panjalukan menyang Kubernetes, sing nggawe pod karo kontaner sing dibutuhake, nglewati paramèter tugas kasebut, lan sawise diproses, wadhah kasebut ambruk.

Minangka conto, kita bakal nggawe varian saka tugas 1, nalika owah-owahan ing Mail.ru Cloud Solutions (MCS) ember panyimpenan obyek disinkronake ing panyimpenan obyek AWS nggunakake webhooks. Ing kasus sing dimuat nyata, karya asinkron kudu disedhiyakake kanthi ndhaptar webhooks ing antrian, nanging kanggo tugas latihan kita bakal nindakake implementasine tanpa iki.

Skema kerja

Protokol interaksi diterangake kanthi rinci ing Pandhuan kanggo S3 webhooks ing MCS. Skema kerja ngemot unsur ing ngisor iki:

Layanan penerbitan, sing ana ing sisih panyimpenan S3 lan nerbitake panjalukan HTTP nalika webnhook dipicu.
Webhook nampa server, sing ngrungokake panjalukan saka layanan penerbitan HTTP lan nindakake tumindak sing cocog. Server bisa ditulis ing basa apa wae; ing conto kita, kita bakal nulis server ing Go.

Fitur khusus saka implementasi webhooks ing S3 API yaiku registrasi server panampa webhook ing layanan penerbitan. Utamane, server panampa webhook kudu konfirmasi langganan pesen saka layanan penerbitan (ing implementasi webhook liyane, konfirmasi langganan biasane ora dibutuhake).

Mulane, server panampa webhook kudu ndhukung rong operasi utama:

nanggapi panjalukan layanan penerbitan kanggo konfirmasi registrasi,
proses acara mlebu.

Nginstal server panampa webhook

Kanggo mbukak server panampa webhook, sampeyan butuh server Linux. Ing artikel iki, minangka conto, kita nggunakake conto virtual sing disebarake ing MCS.

Ayo nginstal piranti lunak sing dibutuhake lan bukak server panampa webhook.

ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install git
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  bc dns-root-data dnsmasq-base ebtables landscape-common liblxc-common 
liblxc1 libuv1 lxcfs lxd lxd-client python3-attr python3-automat 
python3-click python3-constantly python3-hyperlink
  python3-incremental python3-pam python3-pyasn1-modules 
python3-service-identity python3-twisted python3-twisted-bin 
python3-zope.interface uidmap xdelta3
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui 
gitk gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  git
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
Need to get 3915 kB of archives.
After this operation, 32.3 MB of additional disk space will be used.
Get:1 http://MS1.clouds.archive.ubuntu.com/ubuntu bionic-updates/main 
amd64 git amd64 1:2.17.1-1ubuntu0.7 [3915 kB]
Fetched 3915 kB in 1s (5639 kB/s)
Selecting previously unselected package git.
(Reading database ... 53932 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.7_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.7) ...
Setting up git (1:2.17.1-1ubuntu0.7) ...

Kloning folder nganggo server panampa webhook:

ubuntu@ubuntu-basic-1-2-10gb:~$ git clone
https://github.com/RomanenkoDenys/s3-webhook.git
Cloning into 's3-webhook'...
remote: Enumerating objects: 48, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 114 (delta 20), reused 45 (delta 18), pack-reused 66
Receiving objects: 100% (114/114), 23.77 MiB | 20.25 MiB/s, done.
Resolving deltas: 100% (49/49), done.

Ayo miwiti server:

ubuntu@ubuntu-basic-1-2-10gb:~$ cd s3-webhook/
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80

Langganan layanan penerbitan

Sampeyan bisa ndhaptar server panampa webhook liwat API utawa antarmuka web. Kanggo gamblang, kita bakal ndhaftar liwat antarmuka web:

Ayo menyang bagean ember ing kamar kontrol.
Pindhah menyang ember sing bakal kita atur webhooks lan klik ing gear:

Pindhah menyang tab Webhooks banjur klik Tambah:

Isi ing kolom:

ID - jeneng webhook.

Acara - acara sing bakal dikirim. Kita wis nyetel transmisi kabeh acara sing kedadeyan nalika nggarap file (nambah lan mbusak).

URL — webhook nampa alamat server.

Ater-ater/akhiran filter yaiku saringan sing ngidini sampeyan nggawe webhooks mung kanggo obyek sing jenenge cocog karo aturan tartamtu. Contone, supaya webhook mung bisa micu file kanthi ekstensi .png, ing Saringan sufiks sampeyan kudu nulis "png".

Saiki, mung port 80 lan 443 sing didhukung kanggo ngakses server panampa webhook.

Ayo klik Tambah pancing lan kita bakal weruh ing ngisor iki:

Hook ditambahake.

Server panampa webhook nuduhake ing log kemajuan proses registrasi pancing:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80
2020/06/15 12:01:14 [POST] incoming HTTP request from 
95.163.216.92:42530
2020/06/15 12:01:14 Got timestamp: 2020-06-15T15:01:13+03:00 TopicArn: 
mcs5259999770|myfiles-ash|s3:ObjectCreated:*,s3:ObjectRemoved:* Token: 
E2itMqAMUVVZc51pUhFWSp13DoxezvRxkUh5P7LEuk1dEe9y URL: 
http://89.208.199.220/webhook
2020/06/15 12:01:14 Generate responce signature: 
3754ce36636f80dfd606c5254d64ecb2fd8d555c27962b70b4f759f32c76b66d

Registrasi wis rampung. Ing bagean sabanjure, kita bakal luwih rinci babagan algoritma operasi server panampa webhook.

Katrangan saka server panampa webhook

Ing conto kita, server ditulis ing Go. Ayo katon ing prinsip dhasar saka sawijining operasi.

package main

// Generate hmac_sha256_hex
func HmacSha256hex(message string, secret string) string {
}

// Generate hmac_sha256
func HmacSha256(message string, secret string) string {
}

// Send subscription confirmation
func SubscriptionConfirmation(w http.ResponseWriter, req *http.Request, body []byte) {
}

// Send subscription confirmation
func GotRecords(w http.ResponseWriter, req *http.Request, body []byte) {
}

// Liveness probe
func Ping(w http.ResponseWriter, req *http.Request) {
    // log request
    log.Printf("[%s] incoming HTTP Ping request from %sn", req.Method, req.RemoteAddr)
    fmt.Fprintf(w, "Pongn")
}

//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {
}

func main() {

    // get command line args
    bindPort := flag.Int("port", 80, "number between 1-65535")
    bindAddr := flag.String("address", "", "ip address in dot format")
    flag.StringVar(&actionScript, "script", "", "external script to execute")
    flag.Parse()

    http.HandleFunc("/ping", Ping)
    http.HandleFunc("/webhook", Webhook)

log.Fatal(http.ListenAndServe(*bindAddr+":"+strconv.Itoa(*bindPort), nil))
}

Coba fungsi utama:

Ping () - rute sing nanggapi liwat URL / ping, implementasine paling gampang saka probe liveness.
Webhook() - rute utama, URL/webhook handler:
- konfirmasi registrasi ing layanan penerbitan (pindhah menyang fungsi Konfirmasi Langganan),
- pangolahan webhooks mlebu (fungsi Gorecords).
Fungsi HmacSha256 lan HmacSha256hex minangka implementasi saka algoritma enkripsi HMAC-SHA256 lan HMAC-SHA256 kanthi output minangka senar saka nomer heksadesimal kanggo ngitung teken.
utama minangka fungsi utama, ngolah paramèter baris perintah lan ndhaptar panangan URL.

Parameter baris perintah sing ditampa dening server:

-port punika port kang server bakal ngrungokake.
-alamat - alamat IP sing server bakal ngrungokake.
-script iku sawijining program external sing disebut kanggo saben pancing mlebu.

Ayo dipikirake sawetara fungsi:

//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {

    // Read body
    body, err := ioutil.ReadAll(req.Body)
    defer req.Body.Close()
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }

    // log request
    log.Printf("[%s] incoming HTTP request from %sn", req.Method, req.RemoteAddr)
    // check if we got subscription confirmation request
    if strings.Contains(string(body), 
""Type":"SubscriptionConfirmation"") {
        SubscriptionConfirmation(w, req, body)
    } else {
        GotRecords(w, req, body)
    }

}

Fungsi iki nemtokake manawa panjalukan kanggo konfirmasi registrasi utawa webhook wis teka. Kaya ing ngisor iki saka dokumentasi, yen registrasi dikonfirmasi, struktur Json ing ngisor iki ditampa ing panjalukan Post:

POST http://test.com HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation
content-type: application/json

{
    "Timestamp":"2019-12-26T19:29:12+03:00",
    "Type":"SubscriptionConfirmation",
    "Message":"You have chosen to subscribe to the topic $topic. To confirm the subscription you need to response with calculated signature",
    "TopicArn":"mcs2883541269|bucketA|s3:ObjectCreated:Put",
    "SignatureVersion":1,
    "Token":«RPE5UuG94rGgBH6kHXN9FUPugFxj1hs2aUQc99btJp3E49tA»
}

Pitakonan iki kudu dijawab:

content-type: application/json

{"signature":«ea3fce4bb15c6de4fec365d36bcebbc34ccddf54616d5ca12e1972f82b6d37af»}

Ing endi teken diwilang minangka:

signature = hmac_sha256(url, hmac_sha256(TopicArn, 
hmac_sha256(Timestamp, Token)))

Yen webhook teka, struktur panjalukan Post katon kaya iki:

POST <url> HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation

{ "Records":
    [
        {
            "s3": {
                "object": {
                    "eTag":"aed563ecafb4bcc5654c597a421547b2",
                    "sequencer":1577453615,
                    "key":"some-file-to-bucket",
                    "size":100
                },
            "configurationId":"1",
            "bucket": {
                "name": "bucketA",
                "ownerIdentity": {
                    "principalId":"mcs2883541269"}
                },
                "s3SchemaVersion":"1.0"
            },
            "eventVersion":"1.0",
            "requestParameters":{
                "sourceIPAddress":"185.6.245.156"
            },
            "userIdentity": {
                "principalId":"2407013e-cbc1-415f-9102-16fb9bd6946b"
            },
            "eventName":"s3:ObjectCreated:Put",
            "awsRegion":"ru-msk",
            "eventSource":"aws:s3",
            "responseElements": {
                "x-amz-request-id":"VGJR5rtJ"
            }
        }
    ]
}

Mulane, gumantung saka panyuwunan, sampeyan kudu ngerti carane ngolah data. Aku milih entri minangka indikator "Type":"SubscriptionConfirmation", amarga ana ing panjalukan konfirmasi langganan lan ora ana ing webhook. Adhedhasar anane / ora ana entri iki ing panyuwunan POST, eksekusi program luwih lanjut menyang fungsi kasebut. SubscriptionConfirmation, utawa menyang fungsi GotRecords.

Kita ora bakal nimbang fungsi Konfirmasi Langganan kanthi rinci; iki dileksanakake miturut prinsip sing ditetepake ing dokumentasi. Sampeyan bisa ndeleng kode sumber kanggo fungsi iki ing repositori project git.

Fungsi GotRecords ngurai panjalukan sing mlebu lan kanggo saben obyek Rekam nelpon skrip eksternal (sing jenenge diwarisake ing parameter -script) kanthi paramèter:

jeneng ember
tombol obyek
tumindak:
- salinan - yen ing request asli EventName = ObjectCreated | PutObject | PutObjectCopy
- mbusak - yen ing request asli EventName = ObjectRemoved | DeleteObject

Mangkono, yen pancing teka karo panjalukan Post, minangka diterangake luwih, lan parameter -script=script.sh banjur skrip bakal diarani kaya ing ngisor iki:

script.sh  bucketA some-file-to-bucket copy

Sampeyan kudu dimangerteni manawa server panampa webhook iki ora minangka solusi produksi sing lengkap, nanging minangka conto sing gampang kanggo implementasine.

Tuladha karya

Ayo nyinkronake file saka ember utama ing MCS menyang ember serep ing AWS. Ember utama diarani myfiles-ash, sing serep diarani myfiles-backup (konfigurasi ember ing AWS ngluwihi ruang lingkup artikel iki). Mulane, nalika file diselehake ing ember utama, salinan kasebut kudu katon ing serep, lan nalika dibusak saka sing utama, kudu dibusak ing serep.

Kita bakal nggarap ember nggunakake sarana awscli, sing kompatibel karo panyimpenan maya MCS lan panyimpenan maya AWS.

ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install awscli
Reading package lists... Done
Building dependency tree
Reading state information... Done
After this operation, 34.4 MB of additional disk space will be used.
Unpacking awscli (1.14.44-1ubuntu1) ...
Setting up awscli (1.14.44-1ubuntu1) ...

Ayo konfigurasi akses menyang S3 MCS API:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile mcs
AWS Access Key ID [None]: hdywEPtuuJTExxxxxxxxxxxxxx
AWS Secret Access Key [None]: hDz3SgxKwXoxxxxxxxxxxxxxxxxxx
Default region name [None]:
Default output format [None]:

Ayo konfigurasi akses menyang API AWS S3:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile aws
AWS Access Key ID [None]: AKIAJXXXXXXXXXXXX
AWS Secret Access Key [None]: dfuerphOLQwu0CreP5Z8l5fuXXXXXXXXXXXXXXXX
Default region name [None]:
Default output format [None]:

Ayo priksa akses:

Kanggo AWS:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile aws
2020-07-06 08:44:11 myfiles-backup

Kanggo MCS, nalika mbukak printah sampeyan kudu nambah -endpoint-url:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile mcs --endpoint-url 
https://hb.bizmrg.com
2020-02-04 06:38:05 databasebackups-0cdaaa6402d4424e9676c75a720afa85
2020-05-27 10:08:33 myfiles-ash

Diakses.

Saiki ayo nulis skrip kanggo ngolah pancing sing mlebu, ayo diarani s3_backup_mcs_aws.sh

#!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"

SOURCE_BUCKET=""
SOURCE_FILE=""
ACTION=""

SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"

case ${ACTION} in
    "copy")
    ${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
    ${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
    rm ${TEMP}
    ;;

    "delete")
    ${AWSCLI_AWS} rm ${TARGET}
    ;;

    *)
    echo "Usage: #!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"
SOURCE_BUCKET="${1}"
SOURCE_FILE="${2}"
ACTION="${3}"
SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"
case ${ACTION} in
"copy")
${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
rm ${TEMP}
;;
"delete")
${AWSCLI_AWS} rm ${TARGET}
;;
*)
echo "Usage: ${0} sourcebucket sourcefile copy/delete"
exit 1
;;
esac sourcebucket sourcefile copy/delete"
    exit 1
    ;;
esac

Ayo miwiti server:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80 -
script scripts/s3_backup_mcs_aws.sh

Ayo ndeleng carane iki bisa. Liwat antarmuka web MCS nambah file test.txt menyang ember myfiles-awu. Log konsol nuduhake manawa panyuwunan digawe menyang server webhook:

2020/07/06 09:43:08 [POST] incoming HTTP request from 
95.163.216.92:56612
download: s3://myfiles-ash/test.txt to ../../../tmp/myfiles-ash/test.txt
upload: ../../../tmp/myfiles-ash/test.txt to 
s3://myfiles-backup/test.txt

Ayo priksa isi ember myfiles-backup ing AWS:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls 
myfiles-backup
2020-07-06 09:43:10       1104 test.txt

Saiki, liwat antarmuka web, kita bakal mbusak file saka ember myfiles-ash.

Log server:

2020/07/06 09:44:46 [POST] incoming HTTP request from 
95.163.216.92:58224
delete: s3://myfiles-backup/test.txt

Isi ember:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls 
myfiles-backup
ubuntu@ubuntu-basic-1-2-10gb:~$

File wis dibusak, masalah wis ditanggulangi.

Kesimpulan lan ToDo

Kabeh kode sing digunakake ing artikel iki yaiku ing repositoriku. Ana uga conto skrip lan conto ngetang tandha tangan kanggo ndhaptar webhooks.

Kode iki ora luwih saka conto carane sampeyan bisa nggunakake webhooks S3 ing aktivitas. Kaya sing dakkandhakake ing wiwitan, yen sampeyan pengin nggunakake server kasebut ing produksi, sampeyan kudu paling ora nulis ulang server kanggo karya bedo: ndhaftar webhooks mlebu ing antrian (RabbitMQ utawa NATS), lan saka ing kono parse lan proses. karo aplikasi buruh. Yen ora, nalika webhooks teka kanthi akeh, sampeyan bisa nemoni kekurangan sumber daya server kanggo ngrampungake tugas. Ngarsane antrian ngidini sampeyan nyebarake server lan buruh, uga ngrampungake masalah kanthi mbaleni tugas yen gagal. Sampeyan uga dianjurake kanggo ngganti log menyang sing luwih rinci lan luwih standar.

Good luck!

Waca liyane babagan topik:

Source: www.habr.com

Conto aplikasi sing didhukung acara adhedhasar webhooks ing panyimpenan obyek S3 Mail.ru Cloud Solutions