He tauira o te tono-a-takahanga i runga i nga matapae tukutuku i roto i te rokiroki ahanoa S3 Mail.ru Cloud Solutions

Miihini kawhe Rube Goldberg

Ko te hoahoanga-a-takahanga ka piki ake te pai o te utu o nga rauemi i whakamahia na te mea ka whakamahia noa i te waa e hiahiatia ana. He maha nga whiringa mo te whakatinana i tenei me te kore e hanga i etahi atu hinonga kapua hei tono kaimahi. A i tenei ra kaore au e korero mo te FaaS, engari mo nga maataki tukutuku. Ka whakaatu ahau i tetahi tauira akoranga mo te whakahaere i nga huihuinga ma te whakamahi i nga maau tukutuku rokiroki ahanoa.

He kupu torutoru mo te rokiroki ahanoa me nga matau tukutuku. Ko te rokiroki ahanoa ka taea e koe te penapena i nga raraunga i roto i te kapua i roto i te ahua o nga taonga, ka taea ma te S3 tetahi atu API ranei (i runga i te whakatinanatanga) ma te HTTP/HTTPS. Ko nga Webhooks te tikanga he waea hokinga HTTP ritenga. I te nuinga o te wa ka puta mai i tetahi huihuinga, penei i te peia o te waehere ki tetahi putunga, he korero ranei e whakairihia ana ki runga i te rangitaki. Ina puta he takahanga, ka tukuna e te pae takenga he tono HTTP ki te URL kua tohua mo te maatukutuku. Ko te mutunga, ka taea e koe te hanga takahanga ki tetahi pae ka whakaoho i nga mahi ki tetahi atu (wiki). Mēnā he rokiroki ahanoa te pae pūtake, ka huri ngā takahanga ki ōna ihirangi.

Ko nga tauira o nga keehi ngawari ka taea te whakamahi i enei mahi aunoa:

1. Te hanga kape o nga mea katoa ki tetahi atu putunga kapua. Me hanga kape i runga i te rere i nga wa ka taapirihia nga konae ka whakarerekehia ranei.
2. Te hanga aunoa i te raupapa karakōnui o nga konae whakairoiro, te taapiri tohuwai ki nga whakaahua, me etahi atu whakarereketanga ahua.
3. Ko te whakamohiotanga mo te taenga mai o nga tuhinga hou (hei tauira, ka tukuna e te ratonga kaute kaute nga purongo ki te kapua, ka whiwhi panui te aroturuki putea mo nga purongo hou, ka tirohia me te tātari).
4. Ko nga keehi uaua iti ake, hei tauira, ko te whakaputa tono ki a Kubernetes, ka hangaia he putea me nga ipu e tika ana, ka tukuna nga tawhā mahi ki a ia, ka mutu te tukatuka ka hinga te ipu.

Hei tauira, ka mahia e matou he rereke o te mahi 1, i te wa e tukutahia ai nga huringa o te peere rokiroki ahanoa Mail.ru Cloud Solutions (MCS) ki te rokiroki ahanoa AWS ma te whakamahi i nga matapae tukutuku. I roto i te keehi tino utaina, me tuku nga mahi tukutahi ma te rehita i nga webhooks i roto i te rarangi, engari mo te mahi whakangungu ka mahia e matou te whakatinanatanga me te kore tenei.

Mahere mahi

Ko te kawa taunekeneke e whakaahuahia ana i roto Aratohu mo te S3 webhooks i MCS. Kei roto i te kaupapa mahi nga waahanga e whai ake nei:

• Ratonga whakaputa, kei te taha rokiroki S3 me te whakaputa i nga tono HTTP i te wa e whakaohohia ana te webnhook.
• Tukutuku whiwhi tūmau, e whakarongo ana ki nga tono mai i te ratonga whakaputa HTTP me te mahi i nga mahi tika. Ka taea te tuhi i te tūmau i roto i tetahi reo; i roto i to maatau tauira, ka tuhia e matou te tūmau i roto i te Haere.

Ko tetahi ahuatanga motuhake o te whakatinanatanga o nga webhooks i roto i te S3 API ko te rehitatanga o te kaitoro whiwhi webhook i runga i te ratonga whakaputa. Otirā, me whakaū te tūmau whiwhi webhook i te ohaurunga ki nga karere mai i te ratonga whakaputa (i etahi atu whakatinanatanga webhook, ko te nuinga o te waa kaore e hiahiatia te whakamana ohaurunga).

Na reira, me tautoko te tūmau whiwhi webhook kia rua nga mahi matua:

• whakautu ki te tono a te ratonga whakaputa kia whakamanahia te rehitatanga,
• te tukatuka i nga kaupapa taumai.

Tāuta ana i te tūmau whiwhi webhook

Hei whakahaere i te tūmau whiwhi webhook, me hiahia koe ki te tūmau Linux. I roto i tenei tuhinga, hei tauira, ka whakamahia e matou he tauira mariko ka tukuna e matou ki runga MCS.

Tāutahia te pūmanawa e tika ana ka whakarewahia te tūmau whiwhi webhook.

ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install git
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  bc dns-root-data dnsmasq-base ebtables landscape-common liblxc-common 
liblxc1 libuv1 lxcfs lxd lxd-client python3-attr python3-automat 
python3-click python3-constantly python3-hyperlink
  python3-incremental python3-pam python3-pyasn1-modules 
python3-service-identity python3-twisted python3-twisted-bin 
python3-zope.interface uidmap xdelta3
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui 
gitk gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  git
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
Need to get 3915 kB of archives.
After this operation, 32.3 MB of additional disk space will be used.
Get:1 http://MS1.clouds.archive.ubuntu.com/ubuntu bionic-updates/main 
amd64 git amd64 1:2.17.1-1ubuntu0.7 [3915 kB]
Fetched 3915 kB in 1s (5639 kB/s)
Selecting previously unselected package git.
(Reading database ... 53932 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.7_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.7) ...
Setting up git (1:2.17.1-1ubuntu0.7) ...

Korohia te kōpaki me te tūmau whiwhi webhook:

ubuntu@ubuntu-basic-1-2-10gb:~$ git clone
https://github.com/RomanenkoDenys/s3-webhook.git
Cloning into 's3-webhook'...
remote: Enumerating objects: 48, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 114 (delta 20), reused 45 (delta 18), pack-reused 66
Receiving objects: 100% (114/114), 23.77 MiB | 20.25 MiB/s, done.
Resolving deltas: 100% (49/49), done.

Me timata te tūmau:

ubuntu@ubuntu-basic-1-2-10gb:~$ cd s3-webhook/
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80

Ohauru ki te ratonga whakaputa

Ka taea e koe te rehita i to tuumau whiwhinga tukutuku ma te API, te atanga tukutuku ranei. Mo te ngawari, ka rehita matou ma te atanga tukutuku:

1. Me haere tatou ki te wahanga peere i roto i te ruma whakahaere.
2. Haere ki te peere ka whirihorahia e matou nga matau tukutuku ka paato i te taputapu:

Haere ki te ripa Webhooks ka paato Tāpiri:

Whakakiia nga mara:

ID — te ingoa o te matukutuku.

Takahanga - ko nga huihuinga hei tuku. Kua tautuhia e matou te tuku o nga huihuinga katoa ka puta i te wa e mahi ana me nga konae (te taapiri me te whakakore).

URL — webhook whiwhi wāhitau tūmau.

Ko te tātari prefix/suffix he tātari e taea ai e koe te whakaputa i nga matau tukutuku mo nga mea e rite ana nga ingoa ki etahi ture. Hei tauira, kia taea e te webhook te whakaoho i nga konae me te toronga .png, i roto Tātari kūmuri me tuhi koe "png".

I tenei wa, ko nga tauranga 80 me te 443 anake e tautokohia ana mo te uru atu ki te tūmau whiwhi webhook.

Kia paato tatou Tāpiri matau a ka kite tatou i nga mea e whai ake nei:

Kua tapirihia te matau.

E whakaatu ana te tūmau whiwhi webhook i roto i ana rangitaki te ahunga whakamua o te tukanga rehita matau:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80
2020/06/15 12:01:14 [POST] incoming HTTP request from 
95.163.216.92:42530
2020/06/15 12:01:14 Got timestamp: 2020-06-15T15:01:13+03:00 TopicArn: 
mcs5259999770|myfiles-ash|s3:ObjectCreated:*,s3:ObjectRemoved:* Token: 
E2itMqAMUVVZc51pUhFWSp13DoxezvRxkUh5P7LEuk1dEe9y URL: 
http://89.208.199.220/webhook
2020/06/15 12:01:14 Generate responce signature: 
3754ce36636f80dfd606c5254d64ecb2fd8d555c27962b70b4f759f32c76b66d

Kua oti te rehitatanga. I roto i te waahanga e whai ake nei, ka ata titiro tatou ki te algorithm o te mahi a te kaimau whiwhi webhook.

Whakaahuatanga o te tūmau whiwhi webhook

I roto i to maatau tauira, kua tuhia te tūmau ki te Haere. Kia titiro tatou ki nga kaupapa matua o tana mahi.

package main

// Generate hmac_sha256_hex
func HmacSha256hex(message string, secret string) string {
}

// Generate hmac_sha256
func HmacSha256(message string, secret string) string {
}

// Send subscription confirmation
func SubscriptionConfirmation(w http.ResponseWriter, req *http.Request, body []byte) {
}

// Send subscription confirmation
func GotRecords(w http.ResponseWriter, req *http.Request, body []byte) {
}

// Liveness probe
func Ping(w http.ResponseWriter, req *http.Request) {
    // log request
    log.Printf("[%s] incoming HTTP Ping request from %sn", req.Method, req.RemoteAddr)
    fmt.Fprintf(w, "Pongn")
}

//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {
}

func main() {

    // get command line args
    bindPort := flag.Int("port", 80, "number between 1-65535")
    bindAddr := flag.String("address", "", "ip address in dot format")
    flag.StringVar(&actionScript, "script", "", "external script to execute")
    flag.Parse()

    http.HandleFunc("/ping", Ping)
    http.HandleFunc("/webhook", Webhook)

log.Fatal(http.ListenAndServe(*bindAddr+":"+strconv.Itoa(*bindPort), nil))
}

Whakaarohia nga mahi matua:

• Ping() - he ara e whakautu ana ma te URL/ping, te whakatinanatanga ngawari o te rangahau ora.
• Webhook() - ara matua, URL/kaiwhakahaere maau:
  • ka whakamana i te rehitatanga ki te ratonga whakaputa (haere ki te mahi Whakaaetanga Ohaurunga),
  • ka tukatuka i nga matapae tukutuku taumai (mahi Gorecords).
• Ko nga Mahi HmacSha256 me HmacSha256hex he whakatinanatanga o te HMAC-SHA256 me te HMAC-SHA256 algorithms whakamunatanga me te putanga hei aho o nga tau hautekauono mo te tatau i te waitohu.
• Ko te matua te mahi matua, ka whakahaere i nga tawhā raina whakahau me te rehita i nga kaikawe URL.

Ko nga tawhā raina whakahau e whakaaetia ana e te tūmau:

• -port ko te tauranga e whakarongo ai te tūmau.
• -wāhitau - wāhitau IP ka whakarongo te tūmau.
• -Ko te tuhinga he kaupapa o waho e kiia ana mo ia matau taumai.

Kia ata titiro tatou ki etahi o nga mahi:

//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {

    // Read body
    body, err := ioutil.ReadAll(req.Body)
    defer req.Body.Close()
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }

    // log request
    log.Printf("[%s] incoming HTTP request from %sn", req.Method, req.RemoteAddr)
    // check if we got subscription confirmation request
    if strings.Contains(string(body), 
""Type":"SubscriptionConfirmation"") {
        SubscriptionConfirmation(w, req, body)
    } else {
        GotRecords(w, req, body)
    }

}

Ma tenei mahi e whakatau mena kua tae mai he tono ki te whakau i te rehitatanga, he maau tukutuku ranei. E whai ake nei mai tuhinga, ki te whakamanahia te rehitatanga, ko te hanganga Json e whai ake nei ka whakawhiwhia ki te tono Post:

POST http://test.com HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation
content-type: application/json

{
    "Timestamp":"2019-12-26T19:29:12+03:00",
    "Type":"SubscriptionConfirmation",
    "Message":"You have chosen to subscribe to the topic $topic. To confirm the subscription you need to response with calculated signature",
    "TopicArn":"mcs2883541269|bucketA|s3:ObjectCreated:Put",
    "SignatureVersion":1,
    "Token":«RPE5UuG94rGgBH6kHXN9FUPugFxj1hs2aUQc99btJp3E49tA»
}

Me whakautu tenei patai:

content-type: application/json

{"signature":«ea3fce4bb15c6de4fec365d36bcebbc34ccddf54616d5ca12e1972f82b6d37af»}

Ko te waitohu i tatauhia ko:

signature = hmac_sha256(url, hmac_sha256(TopicArn, 
hmac_sha256(Timestamp, Token)))

Mena ka tae mai he maau tukutuku, penei te ahua o te hanga o te tono Panui:

POST <url> HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation

{ "Records":
    [
        {
            "s3": {
                "object": {
                    "eTag":"aed563ecafb4bcc5654c597a421547b2",
                    "sequencer":1577453615,
                    "key":"some-file-to-bucket",
                    "size":100
                },
            "configurationId":"1",
            "bucket": {
                "name": "bucketA",
                "ownerIdentity": {
                    "principalId":"mcs2883541269"}
                },
                "s3SchemaVersion":"1.0"
            },
            "eventVersion":"1.0",
            "requestParameters":{
                "sourceIPAddress":"185.6.245.156"
            },
            "userIdentity": {
                "principalId":"2407013e-cbc1-415f-9102-16fb9bd6946b"
            },
            "eventName":"s3:ObjectCreated:Put",
            "awsRegion":"ru-msk",
            "eventSource":"aws:s3",
            "responseElements": {
                "x-amz-request-id":"VGJR5rtJ"
            }
        }
    ]
}

Na reira, i runga i te tono, me maarama koe me pehea te tukatuka i nga raraunga. I whiriwhiria e au te urunga hei tohu "Type":"SubscriptionConfirmation", no te mea kei roto i te tono whakau ohaurunga, kaore i te waahi i te webhook. I runga i te aroaro/kore o tenei urunga i roto i te tono POST, ka haere tonu te mahi o te papatono ki te mahi SubscriptionConfirmation, ki te mahi ranei GotRecords.

E kore matou e whai whakaaro mo te mahi Whakaaetanga Ohaurunga; kua whakatinanahia i runga i nga maapono kua whakatakotoria ki roto tuhinga. Ka taea e koe te tiro i te waehere puna mo tenei mahi i kaupapa git repositories.

Ko te mahi a GotRecords ka poroporoaki i tetahi tono taumai, a mo ia ahanoa Rekoata ka karanga he tuhi waho (ko tona ingoa i tukuna ki te tawhā -script) me nga tawhā:

• ingoa peere
• kī ahanoa
• mahi:
  • kape - mena kei te tono taketake EventName = ObjectCreated | PutObject | PutObjectCopy
  • muku - mena kei te tono taketake EventName = ObjectRemoved | Mukua Ahanoa

No reira, ki te tae mai he matau me te tono Panui, pera me te korero teitei, me te tawhā -script=script.sh katahi ka kiia te tuhinga penei:

script.sh  bucketA some-file-to-bucket copy

Me mohio ko tenei tūmau whiwhi webhook ehara i te otinga whakaputa katoa, engari he tauira ngawari o te whakatinanatanga ka taea.

He tauira mahi

Me tukutahi nga konae mai i te peere matua i MCS ki te peere taapiri i AWS. Ko te peere matua e kiia ana ko myfiles-ash, ko te taapiri ko te myfiles-backup (ko te whirihoranga peere i AWS kei tua atu i te waahanga o tenei tuhinga). No reira, ka tuuhia he konae ki roto i te peere matua, me puta tana kape ki roto i te taarua, a ka mukua mai i te peere matua, me whakakorea ki te peera.

Ka mahi tahi matou me nga peere ma te whakamahi i te taputapu awscli, he hototahi ki te rokiroki kapua MCS me te rokiroki kapua AWS.

ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install awscli
Reading package lists... Done
Building dependency tree
Reading state information... Done
After this operation, 34.4 MB of additional disk space will be used.
Unpacking awscli (1.14.44-1ubuntu1) ...
Setting up awscli (1.14.44-1ubuntu1) ...

Me whirihora te uru ki te S3 MCS API:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile mcs
AWS Access Key ID [None]: hdywEPtuuJTExxxxxxxxxxxxxx
AWS Secret Access Key [None]: hDz3SgxKwXoxxxxxxxxxxxxxxxxxx
Default region name [None]:
Default output format [None]:

Me whirihora te uru ki te AWS S3 API:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile aws
AWS Access Key ID [None]: AKIAJXXXXXXXXXXXX
AWS Secret Access Key [None]: dfuerphOLQwu0CreP5Z8l5fuXXXXXXXXXXXXXXXX
Default region name [None]:
Default output format [None]:

Kia tirohia nga urunga:

Ki AWS:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile aws
2020-07-06 08:44:11 myfiles-backup

Mo te MCS, i te wa e whakahaere ana i te tono me taapiri —endpoint-url:

ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile mcs --endpoint-url 
https://hb.bizmrg.com
2020-02-04 06:38:05 databasebackups-0cdaaa6402d4424e9676c75a720afa85
2020-05-27 10:08:33 myfiles-ash

Kua uru atu.

Inaianei me tuhi tuhi tuhi mo te tukatuka i te matau taumai, me kii ko s3_backup_mcs_aws.sh

#!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"

SOURCE_BUCKET=""
SOURCE_FILE=""
ACTION=""

SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"

case ${ACTION} in
    "copy")
    ${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
    ${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
    rm ${TEMP}
    ;;

    "delete")
    ${AWSCLI_AWS} rm ${TARGET}
    ;;

    *)
    echo "Usage: 
#!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"
SOURCE_BUCKET="${1}"
SOURCE_FILE="${2}"
ACTION="${3}"
SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"
case ${ACTION} in
"copy")
${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
rm ${TEMP}
;;
"delete")
${AWSCLI_AWS} rm ${TARGET}
;;
*)
echo "Usage: ${0} sourcebucket sourcefile copy/delete"
exit 1
;;
esac
 sourcebucket sourcefile copy/delete"
    exit 1
    ;;
esac

Me timata te tūmau:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80 -
script scripts/s3_backup_mcs_aws.sh

Kia kite tatou pehea te mahi. Na roto Atanga tukutuku MCS tāpirihia te kōnae test.txt ki te peere myfiles-ash. Ko nga raarangi papatohu e whakaatu ana i tukuna he tono ki te tūmau webhook:

2020/07/06 09:43:08 [POST] incoming HTTP request from 
95.163.216.92:56612
download: s3://myfiles-ash/test.txt to ../../../tmp/myfiles-ash/test.txt
upload: ../../../tmp/myfiles-ash/test.txt to 
s3://myfiles-backup/test.txt

Kia tirohia nga ihirangi o te peere myfiles-backup i AWS:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls 
myfiles-backup
2020-07-06 09:43:10       1104 test.txt

Na, na roto i te atanga tukutuku, ka mukua e matou te konae mai i te peere myfiles-ash.

Rangitaki tūmau:

2020/07/06 09:44:46 [POST] incoming HTTP request from 
95.163.216.92:58224
delete: s3://myfiles-backup/test.txt

Nga ihirangi peere:

ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls 
myfiles-backup
ubuntu@ubuntu-basic-1-2-10gb:~$

Kua mukua te konae, kua whakatauhia te raru.

Whakamutunga me te Mahi

Ko nga waehere katoa e whakamahia ana i tenei tuhinga i roto i taku putunga. He tauira ano mo nga tuhinga tuhi me nga tauira mo te tatau i nga hainatanga mo te rehita i nga maataki tukutuku.

Ko tenei waehere he tauira noa mo te whakamahi i nga matapae tukutuku S3 i roto i o mahi. Ka rite ki taku korero i te timatanga, ki te whakamahere koe ki te whakamahi i taua tuumau i roto i te whakaputanga, me tuhi ano koe i te kaimau mo nga mahi koretake: rehitahia nga maataki tukutuku taumai i roto i te rarangi (RabbitMQ, NATS ranei), a mai i reira ka poroporoaki ka tukatukahia. me nga tono kaimahi. Ki te kore, ka tae nui mai nga webhooks, ka tupono pea koe ki te kore rawa o nga rauemi tūmau hei whakaoti i nga mahi. Ko te aroaro o nga rarangi ka taea e koe te tohatoha i te kaimau me nga kaimahi, me te whakaoti rapanga me te whakahoki ano i nga mahi mena he rahua. He mea tika ano kia huri i te rakaraka ki tetahi atu taipitopito me te paerewa ake.

Ngā manaakitanga pai!

He panui ano mo te kaupapa:

• Te mahi me te Mail.ru Cloud Solutions S3 te rokiroki ahanoa hei punaha konae.
• Me pehea te whakamahi rokiroki ahanoa ki te wikitoria i te whakataetae.

Source: will.com