Tsarin gine-ginen abubuwan da suka faru yana haɓaka ƙimar ƙimar albarkatun da ake amfani da su saboda ana amfani da su kawai a lokacin da ake buƙata. Akwai zaɓuɓɓuka da yawa akan yadda ake aiwatar da wannan kuma ba ƙirƙirar ƙarin abubuwan girgije azaman aikace-aikacen ma'aikaci ba. Kuma a yau ba zan yi magana game da FaaS ba, amma game da webhooks. Zan nuna misalin koyawa na tafiyar da al'amura ta amfani da ma'ajiyar kayan yanar gizo.
Kalmomi kaɗan game da ma'ajiyar abu da ƙugiya. Ma'ajiyar abu yana ba ka damar adana kowane bayanai a cikin gajimare a cikin nau'ikan abubuwa, samun dama ta hanyar S3 ko wani API (dangane da aiwatarwa) ta HTTP/HTTPS. Webhooks gabaɗaya al'ada ce ta kiran kiran HTTP. Yawanci abin da ya faru ke jawo su, kamar tura lambar zuwa wurin ajiya ko kuma ana buga sharhi akan bulogi. Lokacin da wani abu ya faru, asalin rukunin yanar gizon yana aika buƙatun HTTP zuwa URL da aka kayyade don ƙugiya. Sakamakon haka, kuna iya yin abubuwan da suka faru a kan rukunin yanar gizon guda ɗaya suna jawo ayyuka akan wani (). A cikin yanayin inda rukunin yanar gizon tushen abu ne na ajiya, abubuwan da ke faruwa suna aiki azaman canje-canje ga abinda ke ciki.
Misalai na lokuta masu sauƙi lokacin da za'a iya amfani da irin wannan ta atomatik:
- Ƙirƙirar kwafi na duk abubuwa a cikin wani ajiyar girgije. Dole ne a ƙirƙira kwafi akan tashi a duk lokacin da aka ƙara ko canza fayiloli.
- Ƙirƙiri ta atomatik na jerin hotuna masu hoto, ƙara alamar ruwa zuwa hotuna, da sauran gyare-gyaren hoto.
- Sanarwa game da zuwan sabbin takardu (misali, sabis ɗin lissafin rarraba rarraba yana aika rahotanni zuwa gajimare, kuma saka idanu na kuɗi yana karɓar sanarwa game da sabbin rahotanni, dubawa da tantance su).
- Abubuwan da suka fi rikitarwa sun haɗa da, alal misali, samar da buƙatun zuwa Kubernetes, wanda ke haifar da kwafsa tare da kwantena masu dacewa, ya ba da sigogin ɗawainiya zuwa gare shi, kuma bayan sarrafa kwandon ya rushe.
A matsayin misali, za mu yi wani bambance-bambancen aiki 1, lokacin da canje-canje a cikin Mail.ru Cloud Solutions (MCS) bucket ajiya abu suna aiki tare a cikin AWS kayan ajiya ta amfani da webhooks. A cikin akwati na gaske wanda aka ɗora, ya kamata a samar da aikin asynchronous ta hanyar yin rijistar webhooks a cikin jerin gwano, amma don aikin horarwa za mu yi aiwatarwa ba tare da wannan ba.
Tsarin aiki
An yi bayanin ka'idar hulɗar dalla-dalla a ciki . Tsarin aikin ya ƙunshi abubuwa masu zuwa:
- Sabis na bugawa, wanda ke gefen S3 ajiya kuma yana buga buƙatun HTTP lokacin da aka kunna webnhook.
- Webhook mai karɓar uwar garken, wanda ke sauraron buƙatun daga sabis ɗin bugawa na HTTP kuma yana aiwatar da ayyuka masu dacewa. Ana iya rubuta uwar garken a kowane harshe; a cikin misalinmu, za mu rubuta sabar a cikin Go.
Siffa ta musamman na aiwatar da ƙugiya a cikin S3 API shine rajistar sabar mai karɓar gidan yanar gizo akan sabis ɗin bugawa. Musamman ma, uwar garken gidan yanar gizon da ke karɓar ƙugiya dole ne ya tabbatar da biyan kuɗin shiga ga saƙonni daga sabis ɗin wallafe-wallafe (a cikin wasu ayyukan aiwatarwa, ba a buƙatar tabbatar da biyan kuɗi).
Saboda haka, uwar garken mai karɓar gidan yanar gizo dole ne ya goyi bayan manyan ayyuka guda biyu:
- amsa buƙatun sabis ɗin bugawa don tabbatar da rajista,
- aiwatar abubuwan da ke shigowa.
Shigar da sabar mai karɓar gidan yanar gizo
Don gudanar da sabar mai karɓar gidan yanar gizo, kuna buƙatar uwar garken Linux. A cikin wannan labarin, a matsayin misali, muna amfani da misalin kama-da-wane da muke turawa akan MCS.
Bari mu shigar da dole software da kaddamar da webhook mai karɓar uwar garken.
ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install git
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
bc dns-root-data dnsmasq-base ebtables landscape-common liblxc-common
liblxc1 libuv1 lxcfs lxd lxd-client python3-attr python3-automat
python3-click python3-constantly python3-hyperlink
python3-incremental python3-pam python3-pyasn1-modules
python3-service-identity python3-twisted python3-twisted-bin
python3-zope.interface uidmap xdelta3
Use 'sudo apt autoremove' to remove them.
Suggested packages:
git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui
gitk gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
git
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
Need to get 3915 kB of archives.
After this operation, 32.3 MB of additional disk space will be used.
Get:1 http://MS1.clouds.archive.ubuntu.com/ubuntu bionic-updates/main
amd64 git amd64 1:2.17.1-1ubuntu0.7 [3915 kB]
Fetched 3915 kB in 1s (5639 kB/s)
Selecting previously unselected package git.
(Reading database ... 53932 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.7_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.7) ...
Setting up git (1:2.17.1-1ubuntu0.7) ...Rufe babban fayil ɗin tare da sabar mai karɓar gidan yanar gizo:
ubuntu@ubuntu-basic-1-2-10gb:~$ git clone
https://github.com/RomanenkoDenys/s3-webhook.git
Cloning into 's3-webhook'...
remote: Enumerating objects: 48, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 114 (delta 20), reused 45 (delta 18), pack-reused 66
Receiving objects: 100% (114/114), 23.77 MiB | 20.25 MiB/s, done.
Resolving deltas: 100% (49/49), done.Bari mu fara uwar garken:
ubuntu@ubuntu-basic-1-2-10gb:~$ cd s3-webhook/
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80Biyan kuɗi zuwa sabis ɗin bugawa
Kuna iya yin rajistar sabar mai karɓar gidan yanar gizon ku ta hanyar API ko mu'amalar yanar gizo. Don sauƙi, za mu yi rajista ta hanyar haɗin yanar gizo:
- a cikin dakin sarrafawa.
- Je zuwa guga wanda za mu saita ƙugiya ta yanar gizo kuma danna kayan aiki:
Je zuwa shafin yanar gizo kuma danna Ƙara:
Cika filayen:
ID - sunan gidan yanar gizo.
Event - waɗanne abubuwan da ke faruwa don watsawa. Mun saita watsa duk abubuwan da suka faru lokacin aiki tare da fayiloli (ƙara da sharewa).
URL - adireshin uwar garke mai karɓar gidan yanar gizo.
Filter prefix/suffix shine tacewa wanda ke ba ku damar ƙirƙirar ƙugiya ta yanar gizo kawai don abubuwan da sunayensu suka dace da wasu dokoki. Misali, domin webhook ya kunna fayiloli kawai tare da tsawo na .png, in Tace sufi Kuna buƙatar rubuta "png".
A halin yanzu, tashoshin jiragen ruwa 80 da 443 ne kawai ake tallafawa don samun dama ga uwar garken mai karɓar gidan yanar gizo.
Mu danna Ƙara ƙugiya kuma za mu ga kamar haka:
An ƙara ƙugiya.
Sabar mai karɓar gidan yanar gizo tana nuna ci gaban aikin rajistar ƙugiya a cikin rajistan ayyukansa:
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80
2020/06/15 12:01:14 [POST] incoming HTTP request from
95.163.216.92:42530
2020/06/15 12:01:14 Got timestamp: 2020-06-15T15:01:13+03:00 TopicArn:
mcs5259999770|myfiles-ash|s3:ObjectCreated:*,s3:ObjectRemoved:* Token:
E2itMqAMUVVZc51pUhFWSp13DoxezvRxkUh5P7LEuk1dEe9y URL:
http://89.208.199.220/webhook
2020/06/15 12:01:14 Generate responce signature:
3754ce36636f80dfd606c5254d64ecb2fd8d555c27962b70b4f759f32c76b66dAn kammala rajista. A cikin sashe na gaba, za mu yi nazari sosai kan algorithm na aiki na sabar mai karɓar gidan yanar gizo.
Bayanin uwar garken mai karɓar gidan yanar gizo
A cikin misalinmu, an rubuta uwar garken a cikin Go. Bari mu dubi ainihin ka'idodin aikinsa.
package main
// Generate hmac_sha256_hex
func HmacSha256hex(message string, secret string) string {
}
// Generate hmac_sha256
func HmacSha256(message string, secret string) string {
}
// Send subscription confirmation
func SubscriptionConfirmation(w http.ResponseWriter, req *http.Request, body []byte) {
}
// Send subscription confirmation
func GotRecords(w http.ResponseWriter, req *http.Request, body []byte) {
}
// Liveness probe
func Ping(w http.ResponseWriter, req *http.Request) {
// log request
log.Printf("[%s] incoming HTTP Ping request from %sn", req.Method, req.RemoteAddr)
fmt.Fprintf(w, "Pongn")
}
//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {
}
func main() {
// get command line args
bindPort := flag.Int("port", 80, "number between 1-65535")
bindAddr := flag.String("address", "", "ip address in dot format")
flag.StringVar(&actionScript, "script", "", "external script to execute")
flag.Parse()
http.HandleFunc("/ping", Ping)
http.HandleFunc("/webhook", Webhook)
log.Fatal(http.ListenAndServe(*bindAddr+":"+strconv.Itoa(*bindPort), nil))
}Yi la'akari da manyan ayyuka:
- Ping() - hanyar da ke amsa ta hanyar URL/ping, mafi sauƙin aiwatar da binciken rayuwa.
- Webhook() - babbar hanya, URL/mai sarrafa gidan yanar gizo:
- ya tabbatar da rajista akan sabis ɗin bugawa (je zuwa aikin Tabbatar da Biyan Kuɗi),
- aiwatar da ƙugiya masu shiga yanar gizo (aikin Gorecords).
- Ayyuka HmacSha256 da HmacSha256hex aiwatarwa ne na HMAC-SHA256 da HMAC-SHA256 algorithms ɓoye tare da fitarwa azaman jeri na lambobi hexadecimal don ƙididdige sa hannu.
- babba shine babban aiki, aiwatar da sigogin layin umarni da rajistar masu sarrafa URL.
Sabar layin umarni da uwar garken ke karɓa:
- - tashar jiragen ruwa ita ce tashar da uwar garken zai saurare.
- -address - Adireshin IP wanda uwar garken zai saurare.
- -script shiri ne na waje wanda ake kira ga kowane ƙugiya mai shigowa.
Bari mu dubi wasu ayyuka:
//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {
// Read body
body, err := ioutil.ReadAll(req.Body)
defer req.Body.Close()
if err != nil {
http.Error(w, err.Error(), 500)
return
}
// log request
log.Printf("[%s] incoming HTTP request from %sn", req.Method, req.RemoteAddr)
// check if we got subscription confirmation request
if strings.Contains(string(body),
""Type":"SubscriptionConfirmation"") {
SubscriptionConfirmation(w, req, body)
} else {
GotRecords(w, req, body)
}
}Wannan aikin yana ƙayyade ko buƙatar tabbatar da rajista ko ƙugiya ta yanar gizo ta isa. Kamar haka daga , idan an tabbatar da rajista, ana karɓar tsarin Json mai zuwa a cikin buƙatun Post:
POST http://test.com HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation
content-type: application/json
{
"Timestamp":"2019-12-26T19:29:12+03:00",
"Type":"SubscriptionConfirmation",
"Message":"You have chosen to subscribe to the topic $topic. To confirm the subscription you need to response with calculated signature",
"TopicArn":"mcs2883541269|bucketA|s3:ObjectCreated:Put",
"SignatureVersion":1,
"Token":«RPE5UuG94rGgBH6kHXN9FUPugFxj1hs2aUQc99btJp3E49tA»
}Wannan tambayar tana buƙatar amsa:
content-type: application/json
{"signature":«ea3fce4bb15c6de4fec365d36bcebbc34ccddf54616d5ca12e1972f82b6d37af»}Inda aka lissafta sa hannun kamar haka:
signature = hmac_sha256(url, hmac_sha256(TopicArn,
hmac_sha256(Timestamp, Token)))Idan ƙugiya ta zo, tsarin buƙatun Post yayi kama da haka:
POST <url> HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation
{ "Records":
[
{
"s3": {
"object": {
"eTag":"aed563ecafb4bcc5654c597a421547b2",
"sequencer":1577453615,
"key":"some-file-to-bucket",
"size":100
},
"configurationId":"1",
"bucket": {
"name": "bucketA",
"ownerIdentity": {
"principalId":"mcs2883541269"}
},
"s3SchemaVersion":"1.0"
},
"eventVersion":"1.0",
"requestParameters":{
"sourceIPAddress":"185.6.245.156"
},
"userIdentity": {
"principalId":"2407013e-cbc1-415f-9102-16fb9bd6946b"
},
"eventName":"s3:ObjectCreated:Put",
"awsRegion":"ru-msk",
"eventSource":"aws:s3",
"responseElements": {
"x-amz-request-id":"VGJR5rtJ"
}
}
]
}
Dangane da haka, ya danganta da buƙatar, kuna buƙatar fahimtar yadda ake aiwatar da bayanan. Na zaɓi shigarwar azaman mai nuni "Type":"SubscriptionConfirmation", tunda yana cikin buƙatun tabbatar da biyan kuɗi kuma baya cikin saƙon gidan yanar gizo. Dangane da kasancewar / rashin wannan shigarwar a cikin buƙatun POST, ƙarin aiwatar da shirin yana tafiya ko dai zuwa aikin. SubscriptionConfirmation, ko kuma cikin aiki GotRecords.
Ba za mu yi la'akari da aikin Tabbatar da Biyan Kuɗi dalla-dalla ba; ana aiwatar da shi bisa ga ƙa'idodin da aka tsara a ciki. . Kuna iya duba lambar tushe don wannan aikin a .
Ayyukan GotRecords yana ƙaddamar da buƙatun mai shigowa kuma ga kowane abu Rikodi yana kiran rubutun waje (wanda aka sanya sunansa a cikin ma'aunin rubutun) tare da sigogi:
- sunan guga
- abu key
- aiki:
- kwafi - idan a cikin ainihin buƙatun EventName = Abun Kirkirar | PutObject | PutObjectCopy
- share - idan a cikin ainihin buƙatar EventName = Abun Cire | Share Abu
Don haka, idan ƙugiya ta zo tare da buƙatar Post, kamar yadda aka bayyana , da parameter -script=script.sh sai a kira rubutun kamar haka:
script.sh bucketA some-file-to-bucket copyYa kamata a fahimci cewa wannan uwar garken gidan yanar gizo ba shine cikakken bayani na samarwa ba, amma misali mai sauƙi na yiwuwar aiwatarwa.
Misalin aiki
Bari mu daidaita fayilolin daga babban guga a MCS zuwa guga na ajiya a AWS. Ana kiran babban guga myfiles-ash, madadin wanda ake kira myfiles-backup (tsarin guga a cikin AWS ya wuce iyakar wannan labarin). Don haka, idan an sanya fayil ɗin a cikin babban bokiti, kwafinsa ya kamata ya bayyana a cikin ajiyar, sannan idan an goge shi daga babban fayil ɗin, sai a goge shi a madadin.
Za mu yi aiki tare da buckets ta amfani da mai amfani awscli, wanda ya dace da duka ajiyar girgije na MCS da ajiyar girgije na AWS.
ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install awscli
Reading package lists... Done
Building dependency tree
Reading state information... Done
After this operation, 34.4 MB of additional disk space will be used.
Unpacking awscli (1.14.44-1ubuntu1) ...
Setting up awscli (1.14.44-1ubuntu1) ...Bari mu saita dama ga S3 MCS API:
ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile mcs
AWS Access Key ID [None]: hdywEPtuuJTExxxxxxxxxxxxxx
AWS Secret Access Key [None]: hDz3SgxKwXoxxxxxxxxxxxxxxxxxx
Default region name [None]:
Default output format [None]:Bari mu saita damar zuwa API AWS S3:
ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile aws
AWS Access Key ID [None]: AKIAJXXXXXXXXXXXX
AWS Secret Access Key [None]: dfuerphOLQwu0CreP5Z8l5fuXXXXXXXXXXXXXXXX
Default region name [None]:
Default output format [None]:Mu duba hanyoyin shiga:
Zuwa AWS:
ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile aws
2020-07-06 08:44:11 myfiles-backupDon MCS, lokacin gudanar da umarni kuna buƙatar ƙara -endpoint-url:
ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile mcs --endpoint-url
https://hb.bizmrg.com
2020-02-04 06:38:05 databasebackups-0cdaaa6402d4424e9676c75a720afa85
2020-05-27 10:08:33 myfiles-ashAn shiga
Yanzu bari mu rubuta rubutun don sarrafa ƙugiya mai shigowa, bari mu kira shi s3_backup_mcs_aws.sh
#!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"
SOURCE_BUCKET=""
SOURCE_FILE=""
ACTION=""
SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"
case ${ACTION} in
"copy")
${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
rm ${TEMP}
;;
"delete")
${AWSCLI_AWS} rm ${TARGET}
;;
*)
echo "Usage: #!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"
SOURCE_BUCKET="${1}"
SOURCE_FILE="${2}"
ACTION="${3}"
SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"
case ${ACTION} in
"copy")
${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
rm ${TEMP}
;;
"delete")
${AWSCLI_AWS} rm ${TARGET}
;;
*)
echo "Usage: ${0} sourcebucket sourcefile copy/delete"
exit 1
;;
esac
sourcebucket sourcefile copy/delete"
exit 1
;;
esacBari mu fara uwar garken:
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80 -
script scripts/s3_backup_mcs_aws.shBari mu ga yadda yake aiki. Ta hanyar ƙara fayil ɗin test.txt zuwa bukitin myfiles-ash. Rukunin na'urorin wasan bidiyo sun nuna cewa an yi buƙatu zuwa sabar gidan yanar gizo:
2020/07/06 09:43:08 [POST] incoming HTTP request from
95.163.216.92:56612
download: s3://myfiles-ash/test.txt to ../../../tmp/myfiles-ash/test.txt
upload: ../../../tmp/myfiles-ash/test.txt to
s3://myfiles-backup/test.txtBari mu bincika abubuwan da ke cikin bukitin madadin myfiles a cikin AWS:
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls
myfiles-backup
2020-07-06 09:43:10 1104 test.txtYanzu, ta hanyar haɗin yanar gizo, za mu share fayil ɗin daga bukitin myfiles-ash.
Rubutun sabar:
2020/07/06 09:44:46 [POST] incoming HTTP request from
95.163.216.92:58224
delete: s3://myfiles-backup/test.txtAbubuwan da ke cikin guga:
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls
myfiles-backup
ubuntu@ubuntu-basic-1-2-10gb:~$An share fayil ɗin, an warware matsalar.
Ƙarshe da ToDo
Duk lambar da aka yi amfani da ita a cikin wannan labarin shine . Akwai kuma misalan rubutun da misalan kirga sa hannu don yin rajistar ƙugiya ta yanar gizo.
Wannan lambar ba kome ba ce face misali na yadda zaku iya amfani da ƙugiya na S3 a cikin ayyukanku. Kamar yadda na fada a farkon, idan kuna shirin yin amfani da irin wannan uwar garke wajen samarwa, kuna buƙatar aƙalla sake rubuta uwar garken don aikin asynchronous: rajistar ƙugiya masu shigowa a cikin jerin gwano (RabbitMQ ko NATS), kuma daga can ku rarraba su kuma sarrafa su. tare da aikace-aikacen ma'aikaci. In ba haka ba, lokacin da ƙugiya na yanar gizo suka zo da yawa, za ku iya fuskantar rashin albarkatun sabar don kammala ayyuka. Kasancewar jerin gwano yana ba ku damar rarraba uwar garken da ma'aikata, da kuma magance matsaloli tare da maimaita ayyuka idan akwai gazawa. Har ila yau yana da kyau a canza gunkin zuwa mafi cikakken bayani kuma mafi daidaitacce.
Nasara!
Karin karatu kan batun:
source: www.habr.com