Mehaho e tsamaisoang ke liketsahalo e eketsa katleho ea litšenyehelo tsa lisebelisoa tse sebelisoang hobane li sebelisoa feela nakong eo li hlokahalang. Ho na le likhetho tse ngata mabapi le mokhoa oa ho kenya ts'ebetsong sena le ho se thehe mekhatlo e meng ea maru joalo ka lits'ebetso tsa basebetsi. 'Me kajeno nke ke ka bua ka FaaS, empa ka li-webhooks. Ke tla bontša mohlala oa thuto oa ho sebetsana le liketsahalo ka ho sebelisa li-webhook tsa polokelo ea lintho.
Mantsoe a seng makae mabapi le polokelo ea ntho le li-webhooks. Polokelo ea ntho e u lumella ho boloka data efe kapa efe marung ka mokhoa oa lintho, tse fumanehang ka S3 kapa API e 'ngoe (ho latela ts'ebetsong) ka HTTP/HTTPS. Li-webhook hangata ke li-callback tsa HTTP tse tloaelehileng. Hangata li hlahisoa ke ketsahalo, joalo ka khoutu e sutumelletsoang sebakeng sa polokelo kapa maikutlo a romelloang ho blog. Ha ketsahalo e etsahala, sebaka sa mantlha se romela kopo ea HTTP ho URL e boletsoeng bakeng sa webhook. Ka lebaka leo, o ka etsa liketsahalo sebakeng se seng sa marang-rang se hlahisang liketso ho se seng (). Tabeng eo sebaka sa mohloli e leng polokelo ea ntho, liketsahalo li sebetsa e le liphetoho ho litaba tsa eona.
Mehlala ea linyeoe tse bonolo ha othomathike e joalo e ka sebelisoa:
- Ho etsa likopi tsa lintho tsohle polokelong e 'ngoe ea leru. Likopi li tlameha ho etsoa hang-hang ha lifaele li eketsoa kapa li fetoloa.
- Ho iketsetsa letoto la linepe tsa lifaele tsa litšoantšo, ho eketsa li-watermark lifotong, le liphetoho tse ling tsa litšoantšo.
- Tsebiso mabapi le ho fihla ha litokomane tse ncha (mohlala, tšebeletso ea liak'haonte e phatlalalitsoeng e kenya litlaleho ho leru, 'me tlhahlobo ea lichelete e fumana litemoso mabapi le litlaleho tse ncha, ho li hlahloba le ho li sekaseka).
- Linyeoe tse thata haholoanyane li kenyelletsa, ho etsa mohlala, ho hlahisa kopo ho Kubernetes, e hlahisang pod e nang le lijana tse hlokahalang, e fetisa litekanyetso tsa mosebetsi ho eona, 'me ka mor'a hore ho sebetsanoe ho oela setshelo.
Ka mohlala, re tla etsa phapang ea mosebetsi oa 1, ha liphetoho tse ka har'a nkho ea polokelo ea lintho tsa Mail.ru Cloud Solutions (MCS) li hokahanngoa ka polokelo ea ntho ea AWS ho sebelisa li-webhooks. Tabeng ea sebele e imetsoeng, mosebetsi oa asynchronous o lokela ho fanoa ka ho ngolisa li-webhooks ka mokoloko, empa bakeng sa mosebetsi oa koetliso re tla etsa ts'ebetsong ntle le sena.
Morero oa mosebetsi
Protocol ea ho sebelisana e hlalositsoe ka botlalo ho . Lenaneo la ts'ebetso le na le lintlha tse latelang:
- Tšebeletso ea khatiso, e ka lehlakoreng la polokelo ea S3 mme e phatlalatsa likōpo tsa HTTP ha webnhook e qala.
- Seva ea ho amohela webhook, e mamelang likōpo tsa tšebeletso ea khatiso ea HTTP le ho etsa liketso tse nepahetseng. Seva e ka ngoloa ka puo efe kapa efe; mohlaleng oa rona, re tla ngola seva ho Go.
Karolo e ikhethang ea ts'ebetsong ea li-webhooks ho S3 API ke ngoliso ea seva e amohelang webhook tšebeletsong ea khatiso. Haholo-holo, seva se amohelang webhook se tlameha ho netefatsa ho ngolisoa ha melaetsa e tsoang tšebeletsong ea khatiso (ka ts'ebetsong e meng ea webhook, hangata ha ho hlokahale hore ho be le tiiso ea ho ngolisa).
Ka hona, seva se amohelang webhook se tlameha ho ts'ehetsa lits'ebetso tse peli tsa mantlha:
- araba kopo ea tšebeletso ea khatiso ea ho netefatsa ngoliso,
- sebetsa diketsahalo tse tlang.
Ho kenya seva ea ho amohela webhook
Ho tsamaisa seva e amohelang webhook, o hloka seva ea Linux. Sehloohong sena, e le mohlala, re sebelisa mohlala oo re o sebelisang ho MCS.
Ha re kenye software e hlokahalang mme re qale seva e amohelang webhook.
ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install git
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
bc dns-root-data dnsmasq-base ebtables landscape-common liblxc-common
liblxc1 libuv1 lxcfs lxd lxd-client python3-attr python3-automat
python3-click python3-constantly python3-hyperlink
python3-incremental python3-pam python3-pyasn1-modules
python3-service-identity python3-twisted python3-twisted-bin
python3-zope.interface uidmap xdelta3
Use 'sudo apt autoremove' to remove them.
Suggested packages:
git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui
gitk gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
git
0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
Need to get 3915 kB of archives.
After this operation, 32.3 MB of additional disk space will be used.
Get:1 http://MS1.clouds.archive.ubuntu.com/ubuntu bionic-updates/main
amd64 git amd64 1:2.17.1-1ubuntu0.7 [3915 kB]
Fetched 3915 kB in 1s (5639 kB/s)
Selecting previously unselected package git.
(Reading database ... 53932 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.7_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.7) ...
Setting up git (1:2.17.1-1ubuntu0.7) ...Koala foldara ka seva e amohelang webhook:
ubuntu@ubuntu-basic-1-2-10gb:~$ git clone
https://github.com/RomanenkoDenys/s3-webhook.git
Cloning into 's3-webhook'...
remote: Enumerating objects: 48, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 114 (delta 20), reused 45 (delta 18), pack-reused 66
Receiving objects: 100% (114/114), 23.77 MiB | 20.25 MiB/s, done.
Resolving deltas: 100% (49/49), done.Ha re qale seva:
ubuntu@ubuntu-basic-1-2-10gb:~$ cd s3-webhook/
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80Ingolise ho tšebeletso ea khatiso
U ka ngolisa seva ea hau e amohelang webhook ka API kapa sebopeho sa webo. Ho nolofatsa, re tla ingolisa ka sebopeho sa webo:
- ka phaposing ya taolo.
- E-ea nkhong eo re tla e hlophisa li-webhooks ebe u tobetsa gear:
Eya ho Webhooks tab ebe o tobetsa Eketsa:
Tlatsa libaka:
ID - lebitso la webhook.
Ketsahalo - ke diketsahalo life tse lokelang ho fetisa. Re behile phetiso ea liketsahalo tsohle tse etsahalang ha u sebetsa le lifaele (ho eketsa le ho hlakola).
URL — aterese ea seva e amohelang webhook.
Filter prefix / suffix ke filthara e u lumellang hore u hlahise li-webhooks feela bakeng sa lintho tseo mabitso a tsona a lumellanang le melao e itseng. Ka mohlala, e le hore webhook e qale feela lifaele ka .png extension, in Sehlongoana sa sefe o hloka ho ngola "png".
Hajoale, ke likou tsa 80 le 443 feela tse tšehetsoang bakeng sa ho fihlella seva e amohelang webhook.
Ha re tobetse Eketsa hook mme re tla bona tse latelang:
Hook e kentsoe.
Seva e amohelang webhook e bonts'a lits'ebetsong tsa eona tsoelo-pele ea ts'ebetso ea ho ngolisa hook:
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80
2020/06/15 12:01:14 [POST] incoming HTTP request from
95.163.216.92:42530
2020/06/15 12:01:14 Got timestamp: 2020-06-15T15:01:13+03:00 TopicArn:
mcs5259999770|myfiles-ash|s3:ObjectCreated:*,s3:ObjectRemoved:* Token:
E2itMqAMUVVZc51pUhFWSp13DoxezvRxkUh5P7LEuk1dEe9y URL:
http://89.208.199.220/webhook
2020/06/15 12:01:14 Generate responce signature:
3754ce36636f80dfd606c5254d64ecb2fd8d555c27962b70b4f759f32c76b66dNgoliso e phethiloe. Karolong e latelang, re tla shebisisa algorithm ea ts'ebetso ea seva e amohelang webhook.
Tlhaloso ea seva e amohelang webhook
Mohlala oa rona, seva se ngotsoe ho Go. A re shebeng melao-motheo ea ts'ebetso ea eona.
package main
// Generate hmac_sha256_hex
func HmacSha256hex(message string, secret string) string {
}
// Generate hmac_sha256
func HmacSha256(message string, secret string) string {
}
// Send subscription confirmation
func SubscriptionConfirmation(w http.ResponseWriter, req *http.Request, body []byte) {
}
// Send subscription confirmation
func GotRecords(w http.ResponseWriter, req *http.Request, body []byte) {
}
// Liveness probe
func Ping(w http.ResponseWriter, req *http.Request) {
// log request
log.Printf("[%s] incoming HTTP Ping request from %sn", req.Method, req.RemoteAddr)
fmt.Fprintf(w, "Pongn")
}
//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {
}
func main() {
// get command line args
bindPort := flag.Int("port", 80, "number between 1-65535")
bindAddr := flag.String("address", "", "ip address in dot format")
flag.StringVar(&actionScript, "script", "", "external script to execute")
flag.Parse()
http.HandleFunc("/ping", Ping)
http.HandleFunc("/webhook", Webhook)
log.Fatal(http.ListenAndServe(*bindAddr+":"+strconv.Itoa(*bindPort), nil))
}Nahana ka mesebetsi ea mantlha:
- Ping() - tsela e arabelang ka URL/ping, ts'ebetsong e bonolo ka ho fetisisa ea lipatlisiso tsa bophelo.
- Webhook() - tsela e kholo, URL/webhook handler:
- e netefatsa ngoliso ea tšebeletso ea khatiso (e-ea mosebetsing oa SubscriptionConfirmation),
- e sebetsa li-webhooks tse kenang (Gorecords function).
- Mesebetsi HmacSha256 le HmacSha256hex ke ts'ebetsong ea li-algorithms tsa encryption tsa HMAC-SHA256 le HMAC-SHA256 tse nang le tlhahiso e le letoto la linomoro tsa hexadecimal bakeng sa ho bala tekeno.
- ntho e ka sehloohong ke ts'ebetso e ka sehloohong, lits'ebetso tsa li-parameter tsa line ea litaelo le li-registering tsa li-URL.
Litekanyetso tsa mola oa taelo tse amoheloang ke seva:
- -port ke kou eo seva e tla mamela ho eona.
- -aterese - aterese ea IP eo seva e tla e mamela.
- -script ke lenaneo la kantle le bitsoang bakeng sa hook e 'ngoe le e 'ngoe e kenang.
Ha re shebisiseng tse ling tsa mesebetsi:
//Webhook
func Webhook(w http.ResponseWriter, req *http.Request) {
// Read body
body, err := ioutil.ReadAll(req.Body)
defer req.Body.Close()
if err != nil {
http.Error(w, err.Error(), 500)
return
}
// log request
log.Printf("[%s] incoming HTTP request from %sn", req.Method, req.RemoteAddr)
// check if we got subscription confirmation request
if strings.Contains(string(body),
""Type":"SubscriptionConfirmation"") {
SubscriptionConfirmation(w, req, body)
} else {
GotRecords(w, req, body)
}
}Ts'ebetso ena e etsa qeto ea hore na kopo ea ho netefatsa ngoliso kapa webhook e fihlile. Ka tsela e latelang ho tloha , haeba ngoliso e tiisitsoe, sebopeho se latelang sa Json se amoheloa kopong ea poso:
POST http://test.com HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation
content-type: application/json
{
"Timestamp":"2019-12-26T19:29:12+03:00",
"Type":"SubscriptionConfirmation",
"Message":"You have chosen to subscribe to the topic $topic. To confirm the subscription you need to response with calculated signature",
"TopicArn":"mcs2883541269|bucketA|s3:ObjectCreated:Put",
"SignatureVersion":1,
"Token":«RPE5UuG94rGgBH6kHXN9FUPugFxj1hs2aUQc99btJp3E49tA»
}Potso ena e tlameha ho arajoa:
content-type: application/json
{"signature":«ea3fce4bb15c6de4fec365d36bcebbc34ccddf54616d5ca12e1972f82b6d37af»}Moo tekeno e baloang joalo ka:
signature = hmac_sha256(url, hmac_sha256(TopicArn,
hmac_sha256(Timestamp, Token)))Haeba webhook e fihla, sebopeho sa kopo ea Post se shebahala tjena:
POST <url> HTTP/1.1
x-amz-sns-messages-type: SubscriptionConfirmation
{ "Records":
[
{
"s3": {
"object": {
"eTag":"aed563ecafb4bcc5654c597a421547b2",
"sequencer":1577453615,
"key":"some-file-to-bucket",
"size":100
},
"configurationId":"1",
"bucket": {
"name": "bucketA",
"ownerIdentity": {
"principalId":"mcs2883541269"}
},
"s3SchemaVersion":"1.0"
},
"eventVersion":"1.0",
"requestParameters":{
"sourceIPAddress":"185.6.245.156"
},
"userIdentity": {
"principalId":"2407013e-cbc1-415f-9102-16fb9bd6946b"
},
"eventName":"s3:ObjectCreated:Put",
"awsRegion":"ru-msk",
"eventSource":"aws:s3",
"responseElements": {
"x-amz-request-id":"VGJR5rtJ"
}
}
]
}
Ka hona, ho latela kopo, o hloka ho utloisisa mokhoa oa ho sebetsana le data. Ke khethile ho kena joalo ka sesupo "Type":"SubscriptionConfirmation", kaha e teng kopong ea netefatso ea peeletso mme ha e eo ho webhook. Ho ipapisitsoe le boteng / ho ba sieo ha keno ena kopong ea POST, ts'ebetso e tsoelang pele ea lenaneo e ea ts'ebetsong. SubscriptionConfirmation, kapa ho kena tšebetsong GotRecords.
Re ke ke ra nahana ka ts'ebetso ea SubscriptionConfirmation ka botlalo; e sebelisoa ho latela melaoana e boletsoeng ho . U ka sheba khoutu ea mohloli bakeng sa mosebetsi ona ho .
Mosebetsi oa GotRecords o fetisa kopo e kenang 'me bakeng sa ntho e' ngoe le e 'ngoe ea Rekoto e bitsa script e ka ntle (eo lebitso la eona le fetisitsoeng ho -script parameter) le li-parameter:
- lebitso la bakete
- senotlolo sa ntho
- ketso:
- kopitsa - haeba e le kopo ea pele EventName = ObjectCreated | Phatlalatso | PutObjectCopy
- hlakola - haeba ho kopo ea pele EventName = ObjectRemoved | TlosaObject
Kahoo, haeba hook e fihla ka kopo ea Post, joalo ka ha ho hlalositsoe , le parameter -script=script.sh joale script e tla bitsoa ka tsela e latelang:
script.sh bucketA some-file-to-bucket copyHo lokela ho utloisisoa hore seva sena se amohelang webhook ha se tharollo e felletseng ea tlhahiso, empa ke mohlala o nolofalitsoeng oa ts'ebetsong e ka khonehang.
Mohlala oa mosebetsi
Ha re hokahanyang lifaele ho tloha ka nkhong e kholo ho MCS ho ea nkhong ea bekapo ho AWS. Bakete e kholo e bitsoa myfiles-ash, "backup" e bitsoa myfiles-backup (sebopeho sa bakete ho AWS se feta boholo ba sengoloa sena). Ka hona, ha faele e behiloe ka nkhong e kholo, kopi ea eona e lokela ho hlaha ka har'a "backup", 'me ha e hlakotsoe ho e kholo, e lokela ho hlakoloa ho backup.
Re tla sebetsa ka libakete re sebelisa sesebelisoa sa awscli, se lumellanang le polokelo ea leru ea MCS le polokelo ea leru ea AWS.
ubuntu@ubuntu-basic-1-2-10gb:~$ sudo apt-get install awscli
Reading package lists... Done
Building dependency tree
Reading state information... Done
After this operation, 34.4 MB of additional disk space will be used.
Unpacking awscli (1.14.44-1ubuntu1) ...
Setting up awscli (1.14.44-1ubuntu1) ...Ha re lokise phihlello ho S3 MCS API:
ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile mcs
AWS Access Key ID [None]: hdywEPtuuJTExxxxxxxxxxxxxx
AWS Secret Access Key [None]: hDz3SgxKwXoxxxxxxxxxxxxxxxxxx
Default region name [None]:
Default output format [None]:Ha re lokise phihlello ho AWS S3 API:
ubuntu@ubuntu-basic-1-2-10gb:~$ aws configure --profile aws
AWS Access Key ID [None]: AKIAJXXXXXXXXXXXX
AWS Secret Access Key [None]: dfuerphOLQwu0CreP5Z8l5fuXXXXXXXXXXXXXXXX
Default region name [None]:
Default output format [None]:Ha re hlahlobeng libaka tsa ho kena:
Ho AWS:
ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile aws
2020-07-06 08:44:11 myfiles-backupBakeng sa MCS, ha u tsamaisa taelo eo u hlokang ho e eketsa -endpoint-url:
ubuntu@ubuntu-basic-1-2-10gb:~$ aws s3 ls --profile mcs --endpoint-url
https://hb.bizmrg.com
2020-02-04 06:38:05 databasebackups-0cdaaa6402d4424e9676c75a720afa85
2020-05-27 10:08:33 myfiles-ashE kenelletse.
Joale ha re ngoleng mongolo oa ho sebetsana le hook e kenang, ha re e bitse s3_backup_mcs_aws.sh
#!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"
SOURCE_BUCKET=""
SOURCE_FILE=""
ACTION=""
SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"
case ${ACTION} in
"copy")
${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
rm ${TEMP}
;;
"delete")
${AWSCLI_AWS} rm ${TARGET}
;;
*)
echo "Usage: #!/bin/bash
# Require aws cli
# if file added — copy it to backup bucket
# if file removed — remove it from backup bucket
# Variables
ENDPOINT_MCS="https://hb.bizmrg.com"
AWSCLI_MCS=`which aws`" --endpoint-url ${ENDPOINT_MCS} --profile mcs s3"
AWSCLI_AWS=`which aws`" --profile aws s3"
BACKUP_BUCKET="myfiles-backup"
SOURCE_BUCKET="${1}"
SOURCE_FILE="${2}"
ACTION="${3}"
SOURCE="s3://${SOURCE_BUCKET}/${SOURCE_FILE}"
TARGET="s3://${BACKUP_BUCKET}/${SOURCE_FILE}"
TEMP="/tmp/${SOURCE_BUCKET}/${SOURCE_FILE}"
case ${ACTION} in
"copy")
${AWSCLI_MCS} cp "${SOURCE}" "${TEMP}"
${AWSCLI_AWS} cp "${TEMP}" "${TARGET}"
rm ${TEMP}
;;
"delete")
${AWSCLI_AWS} rm ${TARGET}
;;
*)
echo "Usage: ${0} sourcebucket sourcefile copy/delete"
exit 1
;;
esac
sourcebucket sourcefile copy/delete"
exit 1
;;
esacHa re qale seva:
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ sudo ./s3-webhook -port 80 -
script scripts/s3_backup_mcs_aws.shA re bone hore na e sebetsa joang. Ka eketsa faele ea teko.txt ho myfiles-ash backet. Li-log tsa console li bontša hore kopo e entsoe ho seva sa webhook:
2020/07/06 09:43:08 [POST] incoming HTTP request from
95.163.216.92:56612
download: s3://myfiles-ash/test.txt to ../../../tmp/myfiles-ash/test.txt
upload: ../../../tmp/myfiles-ash/test.txt to
s3://myfiles-backup/test.txtHa re hlahlobeng litaba tsa nkho ea myfiles-backup ho AWS:
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls
myfiles-backup
2020-07-06 09:43:10 1104 test.txtHona joale, ka sebopeho sa marang-rang, re tla hlakola faele ho tswa ho myfiles-ash backet.
Li-log tsa seva:
2020/07/06 09:44:46 [POST] incoming HTTP request from
95.163.216.92:58224
delete: s3://myfiles-backup/test.txtLikahare tsa bakete:
ubuntu@ubuntu-basic-1-2-10gb:~/s3-webhook$ aws s3 --profile aws ls
myfiles-backup
ubuntu@ubuntu-basic-1-2-10gb:~$Faele e hlakotsoe, bothata bo rarollotsoe.
Qetello le ToDo
Khoutu eohle e sebelisitsoeng sehloohong sena ke . Hape ho na le mehlala ea mangolo le mehlala ea ho bala li-signature tsa ho ngolisa li-webhooks.
Khoutu ena ha se letho haese mohlala oa kamoo u ka sebelisang li-webhook tsa S3 mesebetsing ea hau. Joalokaha ke boletse qalong, haeba u rera ho sebelisa seva e joalo tlhahisong, u hloka bonyane ho ngola seva hape bakeng sa mosebetsi o tšoanang: ngodisa li-webhooks tse kenang moleng (RabbitMQ kapa NATS), 'me ho tloha moo u li hlalose ebe u li sebetsana. ka likopo tsa basebetsi. Ho seng joalo, ha li-webhooks li fihla ka bongata, u ka kopana le khaello ea lisebelisoa tsa seva ho phethela mesebetsi. Ho ba teng ha mela ho u lumella ho aba seva le basebetsi, hammoho le ho rarolla mathata ka ho pheta-pheta mesebetsi haeba ho ka ba le liphoso. Ho boetse ho eletsoa ho fetola ho rema lifate hore e be e qaqileng le e tloaelehileng haholoanyane.
Katleho!
Bala haholoanyane ka sehlooho:
Source: www.habr.com