Muenzaniso wechiitiko chinofambiswa nechiitiko chakavakirwa pawebhooks muS3 chinhu chekuchengetedza Mail.ru Cloud Solutions

Rube Goldberg coffee muchina

Zvivakwa zvinofambiswa nezviitiko zvinowedzera kudhura kwezviwanikwa zvinoshandiswa nekuti zvinoshandiswa chete panguva iyo pazvinenge zvichidikanwa. Pane zvakawanda zvingasarudzwa pamusoro pekuita izvi uye kwete kugadzira mamwe makore masangano sevashandi vekushandisa. Uye nhasi handisi kuzotaura nezveFaaS, asi nezvewebhooks. Ini ndicharatidza muenzaniso wedzidziso wekubata zviitiko uchishandisa chinhu chekuchengetedza webhooks.

Mazwi mashoma pamusoro pekuchengetedza chinhu uye webhooks. Chengetedzo yechinhu inobvumidza iwe kuchengetedza chero data mugore iri muchimiro chezvinhu, inowanikwa kuburikidza neS3 kana imwe API (zvinoenderana nekuitwa) kuburikidza neHTTP/HTTPS. Webhooks kazhinji tsika HTTP callbacks. Iwo anowanzo kukonzereswa nechiitiko, senge kodhi inosundirwa kunzvimbo inochengeterwa kana chirevo chiri kutumirwa pablog. Kana chiitiko chikaitika, saiti yekutanga inotumira chikumbiro cheHTTP kuURL yakatsanangurwa yewebhook. Nekuda kweizvozvo, iwe unogona kuita zviitiko pane imwe saiti kukonzeresa zviito pane imwe (wiki Vasai) Muchiitiko icho saiti sosi chinhu chekuchengetedza, zviitiko zvinoita sekuchinja kune zvirimo.

Mienzaniso yemakesi akareruka apo otomatiki akadaro anogona kushandiswa:

1. Kugadzira makopi ezvese zvinhu mune imwe chengetedzo yegore. Makopi anofanirwa kugadzirwa panhunzi pese kana mafaera awedzerwa kana kuchinjwa.
2. Kugadzira otomatiki kweakatevedzana zvigunwe zvemifananidzo mafaera, kuwedzera watermark kumifananidzo, uye kumwe kugadziridzwa kwemifananidzo.
3. Chiziviso pamusoro pekuuya kwemagwaro matsva (semuenzaniso, yakagoverwa accounting sevhisi inorodha mishumo kune gore, uye kuongororwa kwemari kunogamuchira zviziviso nezve mishumo mitsva, kutarisa uye kuiongorora).
4. Mhosva dzakanyanya kuoma dzinosanganisira, semuenzaniso, kugadzira chikumbiro kuna Kubernetes, iyo inogadzira podhi ine midziyo inodiwa, inopfuudza basa paramita kwairi, uye mushure mekugadzirisa inodonha mudziyo.

Semuenzaniso, tichaita mutsauko webasa 1, apo shanduko muMail.ru Cloud Solutions (MCS) chinhu chekuchengetedza bhaketi inowiriraniswa muAWS chinhu chekuchengetedza uchishandisa webhooks. Muchiitiko chaicho chakaremerwa, basa reasynchronous rinofanira kupiwa nekunyoresa webhooks mumutsetse, asi nokuda kwebasa rekudzidzisa tichaita kushandiswa pasina izvi.

Chikamu chebasa

Iyo yekudyidzana protocol inotsanangurwa zvakadzama mu Nhungamiro yeS3 webhooks paMCS. Iyo work scheme ine zvinotevera zvinhu:

• Publishing service, iyo iri padivi reS3 yekuchengetedza uye inoburitsa zvikumbiro zveHTTP kana webnhook yatanga.
• Webhook inogamuchira server, iyo inoteerera zvikumbiro kubva kuHTTP yekushambadzira sevhisi uye inoita zviito zvakakodzera. Sevha inogona kunyorwa mune chero mutauro; mumuenzaniso wedu, isu tichanyora sevha muGo.

Chinhu chakakosha pakuitwa kwewebhooks muS3 API kunyoreswa kwewebhook inogamuchira server pane yekutsikisa sevhisi. Kunyanya, iyo webhook inogamuchira sevha inofanirwa kusimbisa kunyoreswa kune mameseji kubva kune yekutsikisa sevhisi (mune mamwe mashandisirwo ewebhu webhook, kusimbiswa kwekunyoreswa kazhinji hakudiwi).

Saizvozvo, iyo webhook inogamuchira sevha inofanirwa kutsigira maviri makuru mashandiro:

• pindura kuchikumbiro chebasa rekushambadzira chekusimbisa kunyoreswa,
• gadzirisa zviitiko zvinouya.

Kuisa webhook yekugamuchira server

Kuti umhanye webhook yekugamuchira sevha, unoda Linux server. Muchinyorwa chino, semuenzaniso, tinoshandisa chiitiko chaicho chatinoisa paMCS.

Ngatiisei software inodiwa uye titange iyo webhook yekugamuchira server.

ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install git
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  bc dns-root-data dnsmasq-base ebtables landscape-common liblxc-common 
liblxc1 libuv1 lxcfs lxd lxd-client python3-attr python3-automat 
python3-click python3-constantly python3-hyperlink
  python3-incremental python3-pam python3-pyasn1-modules 
python3-service-identity python3-twisted python3-twisted-bin 
python3-zope.interface uidmap xdelta3
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui 
gitk gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  git
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
Need to get 3915 kB of archives.
After this operation, 32.3 MB of additional disk space will be used.
Get:1 http://MS1.clouds.archive.ubuntu.com/ubuntu bionic-updates/main 
amd64 git amd64 1:2.17.1-1ubuntu0.7 [3915 kB]
Fetched 3915 kB in 1s (5639 kB/s)
Selecting previously unselected package git.
(Reading database ... 53932 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.7_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.7) ...
Setting up git (1:2.17.1-1ubuntu0.7) ...

Clone iyo folda neiyo webhook yekugamuchira server:

ubuntu@ubuntu-basic-1-2-10gb:~$ git clone
https://github.com/RomanenkoDenys/s3-webhook.git
Cloning into 's3-webhook'...
remote: Enumerating objects: 48, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 114 (delta 20), reused 45 (delta 18), pack-reused 66
Receiving objects: 100% (114/114), 23.77 MiB | 20.25 MiB/s, done.
Resolving deltas: 100% (49/49), done.

Ngatitange sevha:

ubuntu@ubuntu-basic-1-2-10gb:~$ cd s3-webhook/
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80

Nyorera kushumiro yekutsikisa

Iwe unogona kunyoresa yako webhook yekugamuchira sevha kuburikidza neAPI kana webhu interface. Kuti zvive nyore, isu tichanyoresa kuburikidza newebhu interface:

1. Handei kuchikamu chemabhaketi muimba yekutonga.
2. Enda kune bhaketi ratinozogadzirisa webhooks uye tinya pane giya:

Enda kune Webhooks tebhu uye tinya Wedzera:

Zadza minda:

ID - zita rewebhook.

Chiitiko - izvo zviitiko zvekutumira. Isu takaisa kutapurirana kwezviitiko zvese zvinoitika kana uchishanda nemafaira (kuwedzera nekudzima).

URL - webhook inogamuchira server kero.

Sefa prefix / suffix sefa inobvumidza iwe kugadzira webhooks chete kune zvinhu zvine mazita anoenderana nemimwe mitemo. Semuenzaniso, kuitira kuti webhook itange mafaira chete ane .png extension, in Sefa suffix unofanira kunyora "png".

Parizvino, zviteshi 80 uye 443 chete zvinotsigirwa kuti uwane iyo webhook yekugamuchira server.

Ngatidzvanye Wedzera hoko uye tichaona zvinotevera:

Hook yakawedzerwa.

Iyo webhook inogamuchira server inoratidza mumatanda ayo kufambira mberi kweiyo hook yekunyoresa maitiro:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80
2020/06/15 12:01:14 [POST] incoming HTTP request from 
95.163.216.92:42530
2020/06/15 12:01:14 Got timestamp: 2020-06-15T15:01:13+03:00 TopicArn: 
mcs5259999770|myfiles-ash|s3:ObjectCreated:*,s3:ObjectRemoved:* Token: 
E2itMqAMUVVZc51pUhFWSp13DoxezvRxkUh5P7LEuk1dEe9y URL: 
http://89.208.199.220/webhook
2020/06/15 12:01:14 Generate responce signature: 
3754ce36636f80dfd606c5254d64ecb2fd8d555c27962b70b4f759f32c76b66d

Kunyoresa kwapera. Muchikamu chinotevera, tichanyatsotarisisa algorithm yekushanda kwewebhook yekugamuchira server.

Tsanangudzo yewebhook yekugamuchira server

Mumuenzaniso wedu, sevha yakanyorwa muGo. Ngatitarisei nheyo dzinokosha dzekushanda kwayo.

package main

// Generate hmac_sha256_hex
func HmacSha256hex(message string, secret string) string {
}

// Generate hmac_sha256
func HmacSha256(message string, secret string) string {
}

// Send subscription confirmation
func SubscriptionConfirmation(w http.ResponseWriter, req *http.Request, body []byte) {
}

// Send subscription confirmation
func GotRecords(w http.ResponseWriter, req *http.Request, body []byte) {
}

// Liveness probe
func Ping(w http.ResponseWriter, req *http.Request) {
    // log request
    log.Printf("[%s] incoming HTTP Ping request from %sn", req.Method, req.RemoteAddr)
    fmt.Fprintf(w, "Pongn")
}

//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {
}

func main() {

    // get command line args
    bindPort := flag.Int("port", 80, "number between 1-65535")
    bindAddr := flag.String("address", "", "ip address in dot format")
    flag.StringVar(&actionScript, "script", "", "external script to execute")
    flag.Parse()

    http.HandleFunc("/ping", Ping)
    http.HandleFunc("/webhook", Webhook)

log.Fatal(http.ListenAndServe(*bindAddr+":"+strconv.Itoa(*bindPort), nil))
}

Funga nezvemabasa makuru:

• Ping () - nzira inopindura kuburikidza ne URL/ping, iri nyore kuita yehupenyu probe.
• Webhook() - nzira huru, URL/webhook mubati:
  • inosimbisa kunyoreswa pane yekushambadzira sevhisi (enda kune iyo SubscriptionConfirmation basa),
  • maitiro anouya webhooks (Gorecords basa).
• Mabasa HmacSha256 uye HmacSha256hex kushandiswa kweHMAC-SHA256 uye HMAC-SHA256 encryption algorithms ine zvinobuda setambo yenhamba dzehexadecimal pakuverenga siginicha.
• chikuru ndiro basa guru, maitiro ekuraira mitsara paramita uye marejista evanobata URL.

Command line parameters inogamuchirwa neserver:

• -port ndiyo chiteshi painoteerera sevha.
• -kero - IP kero iyo sevha inoteerera.
• -script chirongwa chekunze chinodaidzwa kune imwe neimwe chirauro chinouya.

Ngatitarisei zvakanyanya mamwe emabasa:

//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {

    // Read body
    body, err := ioutil.ReadAll(req.Body)
    defer req.Body.Close()
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }

    // log request
    log.Printf("[%s] incoming HTTP request from %sn", req.Method, req.RemoteAddr)
    // check if we got subscription confirmation request
    if strings.Contains(string(body), 
""Type":"SubscriptionConfirmation"") {
        SubscriptionConfirmation(w, req, body)
    } else {
        GotRecords(w, req, body)
    }

}

Iri basa rinotarisa kana chikumbiro chekusimbisa kunyoreswa kana webhook chasvika. Sezvinotevera kubva zvinyorwa, kana kunyoreswa kwakasimbiswa, iyo inotevera Json chimiro inogamuchirwa muPost chikumbiro:

POST http://test.com HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation
content-type: application/json

{
    "Timestamp":"2019-12-26T19:29:12+03:00",
    "Type":"SubscriptionConfirmation",
    "Message":"You have chosen to subscribe to the topic $topic. To confirm the subscription you need to response with calculated signature",
    "TopicArn":"mcs2883541269|bucketA|s3:ObjectCreated:Put",
    "SignatureVersion":1,
    "Token":«RPE5UuG94rGgBH6kHXN9FUPugFxj1hs2aUQc99btJp3E49tA»
}

Uyu mubvunzo unofanirwa kupindurwa:

content-type: application/json

{"signature":«ea3fce4bb15c6de4fec365d36bcebbc34ccddf54616d5ca12e1972f82b6d37af»}

Apo siginicha inoverengwa se:

signature = hmac_sha256(url, hmac_sha256(TopicArn, 
hmac_sha256(Timestamp, Token)))

Kana webhook ikasvika, chimiro chePost chikumbiro chinotaridzika seizvi:

POST <url> HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation

{ "Records":
    [
        {
            "s3": {
                "object": {
                    "eTag":"aed563ecafb4bcc5654c597a421547b2",
                    "sequencer":1577453615,
                    "key":"some-file-to-bucket",
                    "size":100
                },
            "configurationId":"1",
            "bucket": {
                "name": "bucketA",
                "ownerIdentity": {
                    "principalId":"mcs2883541269"}
                },
                "s3SchemaVersion":"1.0"
            },
            "eventVersion":"1.0",
            "requestParameters":{
                "sourceIPAddress":"185.6.245.156"
            },
            "userIdentity": {
                "principalId":"2407013e-cbc1-415f-9102-16fb9bd6946b"
            },
            "eventName":"s3:ObjectCreated:Put",
            "awsRegion":"ru-msk",
            "eventSource":"aws:s3",
            "responseElements": {
                "x-amz-request-id":"VGJR5rtJ"
            }
        }
    ]
}

Naizvozvo, zvichienderana nechikumbiro, iwe unofanirwa kunzwisisa maitiro ekugadzirisa iyo data. Ndakasarudza kupinda sechiratidzo "Type":"SubscriptionConfirmation", sezvo iripo muchikumbiro chekusimbisa kunyorera uye haipo muwebhook. Zvichienderana nekuvapo / kusavapo kwekupinda uku muchikumbiro chePOST, kuenderera mberi kwechirongwa kunoenda kune basa. SubscriptionConfirmation, kana kuti muchiitiko GotRecords.

Isu hatizotarise iyo SubscriptionConfirmation basa zvakadzama; rinoitwa zvinoenderana nemisimboti yakarongwa mukati zvinyorwa. Iwe unogona kuona iyo source code yeiyi basa pa project git repositories.

Basa reGotRecords rinoparura chikumbiro chinouya uye kune chimwe nechimwe Rekodhi chinhu chinodaidza script yekunze (zita rayo rakapfuudzwa mu -script parameter) nematanho:

• bhaketi zita
• chinhu kiyi
• chiito:
  • kopi - kana muchikumbiro chepakutanga EventName = ObjectCreated | PutObject | PutObjectCopy
  • bvisa - kana muchikumbiro chepakutanga EventName = ObjectRemoved | DeleteObject

Saka, kana hoko ichisvika nechikumbiro chePost, sezvakatsanangurwa yepamusorosoro, uye parameter -script=script.sh ipapo script ichadaidzwa seizvi:

script.sh  bucketA some-file-to-bucket copy

Izvo zvinofanirwa kunzwisiswa kuti iyi webhook inogamuchira sevha haisi yakakwana yekugadzira mhinduro, asi muenzaniso wakareruka wekugona kuita.

Muenzaniso webasa

Ngatiwiriranei mafaera kubva kubhaketi guru muMCS kuenda kubhaketi rekuchengetedza muAWS. Bhakiti guru rinonzi myfiles-ash, iyo yekuchengetedza inonzi myfiles-backup (bucket configuration muAWS inopfuura chiyero chechinyorwa ichi). Saizvozvo, kana faira yaiswa mubhaketi guru, kopi yaro inofanirwa kuoneka mune yekuchengetera imwe, uye kana yadzimwa kubva kune iyo huru, inofanirwa kudzimwa mune backup imwe.

Tichashanda nemabhakiti tichishandisa awscli utility, iyo inoenderana nezvose MCS cloud storage uye AWS cloud storage.

ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install awscli
Reading package lists... Done
Building dependency tree
Reading state information... Done
After this operation, 34.4 MB of additional disk space will be used.
Unpacking awscli (1.14.44-1ubuntu1) ...
Setting up awscli (1.14.44-1ubuntu1) ...

Ngatigadzirisei kupinda kune S3 MCS API:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile mcs
AWS Access Key ID [None]: hdywEPtuuJTExxxxxxxxxxxxxx
AWS Secret Access Key [None]: hDz3SgxKwXoxxxxxxxxxxxxxxxxxx
Default region name [None]:
Default output format [None]:

Ngatigadzirisei kupinda kune AWS S3 API:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile aws
AWS Access Key ID [None]: AKIAJXXXXXXXXXXXX
AWS Secret Access Key [None]: dfuerphOLQwu0CreP5Z8l5fuXXXXXXXXXXXXXXXX
Default region name [None]:
Default output format [None]:

Ngatitarisei mapindiro:

KuAWS:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile aws
2020-07-06 08:44:11 myfiles-backup

Kune MCS, kana uchimhanyisa murairo waunoda kuwedzera -endpoint-url:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile mcs --endpoint-url 
https://hb.bizmrg.com
2020-02-04 06:38:05 databasebackups-0cdaaa6402d4424e9676c75a720afa85
2020-05-27 10:08:33 myfiles-ash

Accessed.

Zvino ngatinyorei script yekugadzirisa hoku inouya, ngatiidaidze s3_backup_mcs_aws.sh

#!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"

SOURCE_BUCKET=""
SOURCE_FILE=""
ACTION=""

SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"

case ${ACTION} in
    "copy")
    ${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
    ${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
    rm ${TEMP}
    ;;

    "delete")
    ${AWSCLI_AWS} rm ${TARGET}
    ;;

    *)
    echo "Usage: 
#!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"
SOURCE_BUCKET="${1}"
SOURCE_FILE="${2}"
ACTION="${3}"
SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"
case ${ACTION} in
"copy")
${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
rm ${TEMP}
;;
"delete")
${AWSCLI_AWS} rm ${TARGET}
;;
*)
echo "Usage: ${0} sourcebucket sourcefile copy/delete"
exit 1
;;
esac
 sourcebucket sourcefile copy/delete"
    exit 1
    ;;
esac

Ngatitange sevha:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80 -
script scripts/s3_backup_mcs_aws.sh

Ngationei kuti zvinoshanda sei. Kuburikidza MCS web interface wedzera test.txt faira kune myfiles-ash bucket. Maconsole matanda anoratidza kuti chikumbiro chakaitwa kune webhook server:

2020/07/06 09:43:08 [POST] incoming HTTP request from 
95.163.216.92:56612
download: s3://myfiles-ash/test.txt to ../../../tmp/myfiles-ash/test.txt
upload: ../../../tmp/myfiles-ash/test.txt to 
s3://myfiles-backup/test.txt

Ngatitarisei zviri mukati myfiles-backup bucket muAWS:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls 
myfiles-backup
2020-07-06 09:43:10       1104 test.txt

Iye zvino, kuburikidza newebhu interface, isu tichadzima faira kubva myfiles-ash bucket.

Server logs:

2020/07/06 09:44:46 [POST] incoming HTTP request from 
95.163.216.92:58224
delete: s3://myfiles-backup/test.txt

Mubhaketi zviri mukati:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls 
myfiles-backup
ubuntu@ubuntu-basic-1-2-10gb:~$

Iyo faira inobviswa, dambudziko rinogadziriswa.

Mhedziso uye ToDo

Yese kodhi inoshandiswa muchinyorwa ichi ndeye mudura rangu. Kune zvakare mienzaniso yezvinyorwa uye mienzaniso yekuverenga masiginecha ekunyoresa webhooks.

Iyi kodhi haisi chinhu chinopfuura muenzaniso wemashandisiro aungaite S3 webhooks mumabasa ako. Sezvandambotaura pakutanga, kana ukaronga kushandisa sevha yakadaro mukugadzira, unofanirwa kunyora zvakare sevha yebasa reasynchronous: kunyoresa mawebhooks anouya mumutsara (RabbitMQ kana NATS), uye kubva ipapo tsanangura uye uzvigadzirise. nezvikumbiro zvevashandi. Zvikasadaro, kana webhooks asvika zvakanyanya, unogona kusangana nekushomeka kwesevha zviwanikwa kuti upedze mabasa. Kuvapo kwemitsara kunobvumidza iwe kugovera sevha nevashandi, pamwe nekugadzirisa matambudziko nekudzokorora mabasa kana ukatadza. Zvinokurudzirwawo kushandura kutema matanda kune yakadzama uye yakawanda yakajairwa imwe.

Good luck!

Zvimwe kuverenga pamusoro penyaya:

• Kushanda neMail.ru Cloud Solutions S3 chinhu chekuchengetedza sefaira system.
• Kuchengetedzwa kwezvinhu kunoshandiswa sei kukunda makwikwi.

Source: www.habr.com