Kulesi sihloko ngithanda ukwabelana ngolwazi lwami no-NJS, umhumushi we-JavaScript we-Nginx othuthukiswe yi-Nginx Inc, echaza amakhono ayo amakhulu esebenzisa isibonelo sangempela. I-NJS iyisethi engaphansi ye-JavaScript ekuvumela ukuthi unwebe ukusebenza kwe-Nginx. Embuzweni U-Dmitry Volntsev uphendule ngokuningiliziwe. Ngamafuphi: I-NJS iyindlela ye-nginx, futhi i-JavaScript iqhubeka kakhulu, "yendabuko" futhi ayinayo i-GC, ngokungafani ne-Lua.
Kudala…
Emsebenzini wami wokugcina, ngathola i-gitlab enamapayipi amaningi e-motley CI/CD ane-docker-compose, i-dind nezinye izinto ezijabulisayo, ezidluliselwe kuma-kaniko rails. Izithombe ebezisetshenziswa ngaphambilini ku-CI zihanjiswe ngendlela yazo yoqobo. Basebenza kahle kwaze kwaba usuku lapho i-IP yethu ye-gitlab ishintsha futhi i-CI yaphenduka ithanga. Inkinga bekuwukuthi esinye sezithombe ze-docker esibambe iqhaza ku-CI sine-git, edonsa amamojula wePython nge-ssh. Ku-ssh udinga ukhiye oyimfihlo futhi... ubusesithombeni kanye nabasingathi_abaziwayo. Futhi noma iyiphi i-CI yehlulekile ngephutha lokuqinisekisa elingukhiye ngenxa yokungafani phakathi kwe-IP yangempela naleyo ecaciswe kubasingathi_abaziwayo. Isithombe esisha sihlanganiswe ngokushesha kusuka ku-Dockfiles ekhona futhi inketho yengezwa StrictHostKeyChecking no. Kodwa ukunambitheka okubi kwasala futhi kwakukhona isifiso sokuhambisa ama-libs endaweni yokugcina ye-PyPI yangasese. Ibhonasi eyengeziwe, ngemva kokushintshela ku-PyPI yangasese, bekuyipayipi elilula nencazelo evamile ye-requirements.txt
Isinqumo senziwe, Madoda!
Sigijima yonke into emafini nase-Kubernetes, futhi ekugcineni sasifuna ukuthola isevisi encane eyayiyisitsha esingenasimo esinesitoreji sangaphandle. Hhayi-ke, njengoba sisebenzisa i-S3, kwabekwa kuqala kuyona. Futhi, uma kungenzeka, ngokufakazela ubuqiniso ku-gitlab (ungangeza ngokwakho uma kunesidingo).
Ukusesha okusheshayo kuveze imiphumela eminingana: i-s3pypi, i-pypicloud kanye nenketho ngokudala “ngesandla” kwamafayela e-html amatheniphu. Inketho yokugcina yanyamalala ngokwayo.
s3pypi: Lena i-cli yokusebenzisa ukusingathwa kwe-S3. Silayisha amafayela, sikhiqize i-html futhi siyilayishe kubhakede elifanayo. Ifanele ukusetshenziswa ekhaya.
I-pypicloud: Kubukeka sengathi iphrojekthi ethokozisayo, kepha ngemuva kokufunda imibhalo ngadumala. Naphezu kwemibhalo emihle kanye nekhono lokunwebeka ukuze ivumelane nezidingo zakho, empeleni kuvele kungasasebenzi futhi kunzima ukukumisa. Ukulungisa ikhodi ukuze ivumelane nemisebenzi yakho, ngokwezilinganiso zangaleso sikhathi, kwakuzothatha izinsuku ezingu-3-5. Isevisi idinga futhi isizindalwazi. Siyishiye uma kwenzeka singatholi lutho olunye.
Ukusesha okujule kakhudlwana kuveze imojuli ye-Nginx, ngx_aws_auth. Umphumela wokuhlolwa kwakhe kwaba i-XML ekhonjiswe esipheqululini, ebonisa okuqukethwe kwebhakede le-S3. Ukuzinikela kokugcina ngesikhathi sokuseshwa kwaba unyaka odlule. Inqolobane ibibukeka ishiywe.
Ngokuya emthonjeni nokufunda Ngabona ukuthi i-XML ingaguqulelwa ku-HTML ngokushesha futhi inikezwe i-pip. Ngemva kokuhamba kancane mayelana ne-Nginx ne-S3, ngihlangabezane nesibonelo sokuqinisekisa ku-S3 esibhalwe nge-JS ye-Nginx. Ngahlangana kanjalo no-NJS.
Ngithatha lesi sibonelo njengesisekelo, ngemva kwehora ngabona esipheqululini sami i-XML efanayo nalapho ngisebenzisa imojuli ethi ngx_aws_auth, kodwa yonke into yayisivele ibhalwe nge-JS.
Ngithande kakhulu isixazululo se-nginx. Okokuqala, imibhalo emihle nezibonelo eziningi, okwesibili, sithola zonke izinto ezinhle ze-Nginx zokusebenza ngamafayela (ngaphandle kwebhokisi), okwesithathu, noma ubani owazi ukubhala izilungiselelo ze-Nginx uzokwazi ukuthola ukuthi yini. I-Minimalism nayo iyinhlanganisela kimi, uma iqhathaniswa nePython noma i-Go (uma ibhalwe kusukela ekuqaleni), ingasaphathwa i-nexus.
TL;DR Ngemva kwezinsuku ezingu-2, inguqulo yokuhlola ye-PyPi yayisivele isetshenziswa ku-CI.
Isebenza kanjani?
Imojula ilayishwa ku-Nginx ngx_http_js_module, kufakwe esithombeni sedokhu esisemthethweni. Singenisa umbhalo wethu sisebenzisa umyalelo js_importekucushweni kwe-Nginx. Umsebenzi ubizwa ngomyalelo js_content. Umyalelo usetshenziselwa ukusetha okuguquguqukayo js_set, okuthatha njengengxabano kuphela umsebenzi ochazwe kuskripthi. Kodwa singakwazi ukwenza imibuzo engezansi ku-NJS sisebenzisa i-Nginx kuphela, hhayi noma iyiphi i-XMLHttpRequest. Ukuze wenze lokhu, indawo ehambisanayo kufanele yengezwe ekucushweni kwe-Nginx. Futhi umbhalo kufanele uchaze isicelo esingaphansi kule ndawo. Ukuze ukwazi ukufinyelela umsebenzi kusukela ku-Nginx config, igama lomsebenzi kufanele lithunyelwe kuskripthi ngokwaso export default.
nginx.conf
load_module modules/ngx_http_js_module.so;
http {
js_import imported_name from script.js;
server {
listen 8080;
...
location = /sub-query {
internal;
proxy_pass http://upstream;
}
location / {
js_content imported_name.request;
}
}script.js
function request(r) {
function call_back(resp) {
// handler's code
r.return(resp.status, resp.responseBody);
}
r.subrequest('/sub-query', { method: r.method }, call_back);
}
export default {request}Uma iceliwe kusiphequluli http://localhost:8080/ singene location /lapho isiqondiso js_content ibiza umsebenzi request kuchazwe kusikripthi sethu script.js. Ngokulandelayo, emsebenzini request i-subquery yenziwa ukuze location = /sub-query, ngendlela (esibonelweni samanje esithi GET) etholwe kumpikiswano (r), idluliswe ngokungagunci lapho lo msebenzi ubizwa. Impendulo yesicelo esingaphansi izocutshungulwa emsebenzini call_back.
Izama i-S3
Ukwenza isicelo endaweni eyimfihlo ye-S3, sidinga:
ACCESS_KEY
SECRET_KEY
S3_BUCKET
Kusukela endleleni esetshenzisiwe ye-http, idethi/isikhathi samanje, i-S3_NAME ne-URI, uhlobo oluthile lweyunithi yezinhlamvu luyakhiqizwa, olusayinwa (HMAC_SHA1) kusetshenziswa i-SECRET_KEY. Okulandelayo umugqa ofana AWS $ACCESS_KEY:$HASH, ingasetshenziswa kusihloko sokugunyazwa. Idethi/isikhathi esifanayo esisetshenziswe ukukhiqiza iyunithi yezinhlamvu esinyathelweni sangaphambilini kufanele yengezwe kunhlokweni X-amz-date. Ngekhodi ibonakala kanje:
nginx.conf
load_module modules/ngx_http_js_module.so;
http {
js_import s3 from s3.js;
js_set $s3_datetime s3.date_now;
js_set $s3_auth s3.s3_sign;
server {
listen 8080;
...
location ~* /s3-query/(?<s3_path>.*) {
internal;
proxy_set_header X-amz-date $s3_datetime;
proxy_set_header Authorization $s3_auth;
proxy_pass $s3_endpoint/$s3_path;
}
location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
js_content s3.request;
}
}s3.js(Isibonelo sokugunyazwa kwe-AWS Sign v2, sishintshelwe esimweni esihoxisiwe)
var crypt = require('crypto');
var s3_bucket = process.env.S3_BUCKET;
var s3_access_key = process.env.S3_ACCESS_KEY;
var s3_secret_key = process.env.S3_SECRET_KEY;
var _datetime = new Date().toISOString().replace(/[:-]|.d{3}/g, '');
function date_now() {
return _datetime
}
function s3_sign(r) {
var s2s = r.method + 'nnnn';
s2s += `x-amz-date:${date_now()}n`;
s2s += '/' + s3_bucket;
s2s += r.uri.endsWith('/') ? '/' : r.variables.s3_path;
return `AWS ${s3_access_key}:${crypt.createHmac('sha1', s3_secret_key).update(s2s).digest('base64')}`;
}
function request(r) {
var v = r.variables;
function call_back(resp) {
r.return(resp.status, resp.responseBody);
}
var _subrequest_uri = r.uri;
if (r.uri === '/') {
// root
_subrequest_uri = '/?delimiter=/';
} else if (v.prefix !== '' && v.postfix === '') {
// directory
var slash = v.prefix.endsWith('/') ? '' : '/';
_subrequest_uri = '/?prefix=' + v.prefix + slash;
}
r.subrequest(`/s3-query${_subrequest_uri}`, { method: r.method }, call_back);
}
export default {request, s3_sign, date_now}Incazelo encane mayelana _subrequest_uri: lokhu okuguquguqukayo okuthi, kuye ngokuthi i-uri yokuqala, kwenza isicelo ku-S3. Uma udinga ukuthola okuqukethwe "impande", udinga ukudala isicelo se-uri esibonisa i-delimiter. delimiter, ezobuyisela uhlu lwazo zonke izici ze-CommonPrefixes xml, ezihambisana nezinkomba (esimweni se-PyPI, uhlu lwawo wonke amaphakheji). Uma udinga ukuthola uhlu lokuqukethwe kumkhombandlela othize (uhlu lwazo zonke izinguqulo zephakeji), isicelo se-uri kufanele siqukathe inkambu yesiqalo enegama lenkomba (iphakheji) egcina ngo-slash /. Uma kungenjalo, ukungqubuzana kungenzeka uma ucela okuqukethwe kohla lwemibhalo, isibonelo. Kukhona uhla lwemibhalo aiohttp-sicelo kanye nezicelo ze-aiohttp futhi uma isicelo sisho /?prefix=aiohttp-request, bese impendulo izoqukatha okuqukethwe kuzo zombili izinkomba. Uma kukhona i-slash ekugcineni, /?prefix=aiohttp-request/, bese impendulo izoqukatha uhla lwemibhalo oludingekayo kuphela. Futhi uma sicela ifayela, khona-ke i-uri ewumphumela akufanele ihluke kweyokuqala.
Londoloza futhi uqale kabusha i-Nginx. Esipheqululini sifaka ikheli le-Nginx yethu, umphumela wesicelo uzoba yi-XML, isibonelo:
Uhlu lwemibhalo
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>myback-space</Name>
<Prefix></Prefix>
<Marker></Marker>
<MaxKeys>10000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>new/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>old/</Prefix>
</CommonPrefixes>
</ListBucketResult>Kusukela kuhlu lwezinkomba uzodinga kuphela izakhi CommonPrefixes.
Ngokungeza uhla lwemibhalo esiludingayo ekhelini lethu esipheqululini, sizothola okuqukethwe kwalo ngefomu le-XML:
Uhlu lwamafayela kuhla lwemibhalo
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name> myback-space</Name>
<Prefix>old/</Prefix>
<Marker></Marker>
<MaxKeys>10000</MaxKeys>
<Delimiter></Delimiter>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>old/giphy.mp4</Key>
<LastModified>2020-08-21T20:27:46.000Z</LastModified>
<ETag>"00000000000000000000000000000000-1"</ETag>
<Size>1350084</Size>
<Owner>
<ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
<DisplayName></DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>old/hsd-k8s.jpg</Key>
<LastModified>2020-08-31T16:40:01.000Z</LastModified>
<ETag>"b2d76df4aeb4493c5456366748218093"</ETag>
<Size>93183</Size>
<Owner>
<ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
<DisplayName></DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
</Contents>
</ListBucketResult>Kuhlu lwamafayela sizothatha izakhi kuphela Key.
Okusele nje ukuhlaziya i-XML ewumphumela bese uyithumela njenge-HTML, uqale ngokushintsha isihloko sohlobo Lokuqukethwe ngombhalo/html.
function request(r) {
var v = r.variables;
function call_back(resp) {
var body = resp.responseBody;
if (r.method !== 'PUT' && resp.status < 400 && v.postfix === '') {
r.headersOut['Content-Type'] = "text/html; charset=utf-8";
body = toHTML(body);
}
r.return(resp.status, body);
}
var _subrequest_uri = r.uri;
...
}
function toHTML(xml_str) {
var keysMap = {
'CommonPrefixes': 'Prefix',
'Contents': 'Key',
};
var pattern = `<k>(?<v>.*?)</k>`;
var out = [];
for(var group_key in keysMap) {
var reS;
var reGroup = new RegExp(pattern.replace(/k/g, group_key), 'g');
while(reS = reGroup.exec(xml_str)) {
var data = new RegExp(pattern.replace(/k/g, keysMap[group_key]), 'g');
var reValue = data.exec(reS);
var a_text = '';
if (group_key === 'CommonPrefixes') {
a_text = reValue.groups.v.replace(///g, '');
} else {
a_text = reValue.groups.v.split('/').slice(-1);
}
out.push(`<a href="/zu/${reValue.groups.v}">${a_text}</a>`);
}
}
return '<html><body>n' + out.join('</br>n') + 'n</html></body>'
}Izama i-PyPI
Siyahlola ukuthi akukho yini ephuka noma yikuphi kumaphakheji aziwayo ukuthi ayasebenza.
# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate
# Скачиваем рабочие пакеты.
pip download aiohttp
# Загружаем в приватную репу
for wheel in *.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done
rm -f *.whl
# Устанавливаем из приватной репы
pip install aiohttp -i http://localhost:8080Siphinda ngama-libs ethu.
# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate
pip install setuptools wheel
python setup.py bdist_wheel
for wheel in dist/*.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done
pip install our_pkg --extra-index-url http://localhost:8080Ku-CI, ukudala nokulayisha iphakheji kubukeka kanjena:
pip install setuptools wheel
python setup.py bdist_wheel
curl -sSfT dist/*.whl -u "gitlab-ci-token:${CI_JOB_TOKEN}" "https://pypi.our-domain.com/${CI_PROJECT_NAME}"Ukufakazela ubuqiniso
Ku-Gitlab kungenzeka ukusebenzisa i-JWT ukuze uqinisekise/ugunyaze izinsizakalo zangaphandle. Sisebenzisa i-auth_request isiyalezo ku-Nginx, sizoqondisa kabusha idatha yokuqinisekisa esicelweni esingaphansi esiqukethe ikholi yomsebenzi kusikripthi. Umbhalo uzokwenza esinye isicelo esingaphansi ku-url ye-Gitlab futhi uma idatha yokuqinisekisa icaciswe kahle, i-Gitlab izobuyisela ikhodi engu-200 futhi ukulayishwa/ukulanda kwephakheji kuzovunyelwa. Kungani ungasebenzisi umbuzo owodwa futhi ngokushesha uthumele idatha ku-Gitlab? Ngoba lapho-ke kuzodingeka sihlele ifayela lokucushwa le-Nginx njalo lapho senza noma yiziphi izinguquko ekugunyazweni, futhi lokhu kuwumsebenzi oyisicefe. Futhi, uma i-Kubernetes isebenzisa inqubomgomo yesistimu yefayela yempande yokufunda kuphela, lokhu kusho ukuthi kunezela ubunkimbinkimbi nakakhulu lapho kushintsha i-nginx.conf nge-configmap. Futhi kuba yinto engenakwenzeka neze ukumisa i-Nginx nge-configmap ngenkathi ngesikhathi esifanayo usebenzisa izinqubomgomo ezivimbela ukuxhumeka kwamavolumu (pvc) kanye nesistimu yefayela lempande yokufunda kuphela (lokhu kuyenzeka futhi).
Ngokusebenzisa i-NJS emaphakathi, sithola ithuba lokushintsha imingcele ecacisiwe ku-nginx config sisebenzisa okuguquguqukayo kwemvelo futhi senze ukuhlola okuthile kusikripthi (isibonelo, i-URL ecaciswe ngokungalungile).
nginx.conf
location = /auth-provider {
internal;
proxy_pass $auth_url;
}
location = /auth {
internal;
proxy_set_header Content-Length "";
proxy_pass_request_body off;
js_content auth.auth;
}
location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
auth_request /auth;
js_content s3.request;
}s3.js
var env = process.env;
var env_bool = new RegExp(/[Tt]rue|[Yy]es|[Oo]n|[TtYy]|1/);
var auth_disabled = env_bool.test(env.DISABLE_AUTH);
var gitlab_url = env.AUTH_URL;
function url() {
return `${gitlab_url}/jwt/auth?service=container_registry`
}
function auth(r) {
if (auth_disabled) {
r.return(202, '{"auth": "disabled"}');
return null
}
r.subrequest('/auth-provider',
{method: 'GET', body: ''},
function(res) {
r.return(res.status, "");
});
}
export default {auth, url}Kungenzeka ukuthi umbuzo uyaphuzwa: -Kungani ungasebenzisi amamojula enziwe ngomumo? Konke sekuvele kwenziwa lapho! Isibonelo, i-var AWS = idinga('aws-sdk') futhi asikho isidingo sokubhala “ibhayisikili” elinobuqiniso be-S3!
Asiqhubekele kobubi
Kimina, ukungakwazi ukungenisa amamojula e-JS angaphandle kube yinto engathandeki, kodwa elindelwe. Okuchazwe esibonelweni esingenhla kudinga('crypto') is futhi zidinga imisebenzi yazo kuphela. Ayikho futhi indlela yokusebenzisa kabusha ikhodi evela emibhalweni futhi kufanele uyikopishe futhi uyinamathisele kumafayela ahlukene. Ngethemba ukuthi ngolunye usuku lokhu kusebenza kuzoqaliswa.
Ukucindezela kumele kukhutshazwe kuphrojekthi yamanje ku-Nginx gzip off;
Ngoba ayikho imojula ye-gzip ku-NJS futhi akunakwenzeka ukuyixhuma; ngakho-ke, ayikho indlela yokusebenza ngedatha ecindezelwe. Yiqiniso, lokhu akukhona ngempela ukususa kuleli cala. Awukho umbhalo omningi, futhi amafayela adlulisiwe asevele ecindezelwe futhi ukucindezela okwengeziwe ngeke kubasize kakhulu. Futhi, lena akuyona isevisi elayishiwe noma ebucayi kangangokuthi kufanele uzihluphe ngokuletha okuqukethwe ngama-millisecond ambalwa ngokushesha.
Ukulungisa iphutha lombhalo kuthatha isikhathi eside futhi kungenzeka kuphela “ngamaphrinti” ku- error.log. Kuye ngesethi yolwazi lwezinga lokugawulwa kwemithi, xwayisa noma iphutha, kungenzeka ukusebenzisa izindlela ezi-3 r.log, r.warn, r.error ngokulandelanayo. Ngizama ukulungisa iphutha elithile ku-Chrome (v8) noma ithuluzi le-njs console, kodwa akuyona yonke into engabhekwa lapho. Lapho ususa iphutha ikhodi, ukuhlola okusebenzayo, umlando ubukeka kanje:
docker-compose restart nginx
curl localhost:8080/
docker-compose logs --tail 10 nginxfuthi kungaba namakhulu okulandelana okunjalo.
Ikhodi yokubhala kusetshenziswa imibuzo engezansi kanye neziguquguqukayo kuzo iphenduka isiphithiphithi esiphithene. Kwesinye isikhathi uqala ukugijima uzungeze i-IDE ehlukile windows uzama ukuthola ukulandelana kwezenzo zekhodi yakho. Akunzima, kodwa ngezinye izikhathi kuyacasula kakhulu.
Akukho ukusekelwa okugcwele kwe-ES6.
Kungase kube khona okunye ukushiyeka, kodwa angikaze ngihlangabezane nanoma yini enye. Yabelana ngolwazi uma unolwazi olubi usebenzisa i-NJS.
isiphetho
I-NJS ingumhumushi womthombo ovulekile ongasindi okuvumela ukuthi usebenzise imibhalo ehlukahlukene ye-JavaScript ku-Nginx. Phakathi nokuthuthukiswa kwayo, ukunakwa okukhulu kwakhokhelwa ekusebenzeni. Yebo, kusekuningi okushodayo, kodwa iphrojekthi ithuthukiswa ithimba elincane futhi lengeza ngenkuthalo izici ezintsha futhi lilungisa iziphazamisi. Ngithemba ukuthi ngolunye usuku i-NJS izokuvumela ukuthi uxhume amamojula angaphandle, okuzokwenza ukusebenza kwe-Nginx kucishe kungabi namkhawulo. Kodwa kukhona i-NGINX Plus futhi cishe ngeke kube khona izici!
и
/ Inkulumo ka-Dmitry Volnyev e-Saint HighLoad++ 2019
/ Inkulumo ka-Vasily Soshnikov ku-HighLoad++ 2019
Source: www.habr.com