Sehloohong sena ke rata ho arolelana phihlelo ea ka le NJS, mofetoleli oa JavaScript bakeng sa Nginx e entsoeng ke Nginx Inc, e hlalosang bokhoni ba eona bo ka sehloohong ho sebelisa mohlala oa sebele. NJS ke karoloana ea JavaScript e u lumellang ho holisa ts'ebetso ea Nginx. Ho potso
Khale koana…
Mosebetsing oa ka oa ho qetela, ke futsitse gitlab e nang le liphaephe tse ngata tsa motley CI / CD tse nang le li-docker-compose, dind le tse ling tse monate, tse ileng tsa fetisetsoa ho liporo tsa kaniko. Litšoantšo tse neng li sebelisoa pele ho CI li tsamaisitsoe ka sebopeho sa tsona sa mantlha. Ba ile ba sebetsa hantle ho fihlela letsatsing leo ka lona gitlab IP ea rona e fetohang 'me CI e fetoha mokopu. Bothata e ne e le hore e 'ngoe ea litšoantšo tsa docker tse nkileng karolo ho CI e ne e na le git, e ileng ea hula li-module tsa Python ka ssh. Bakeng sa ssh o hloka senotlolo sa poraefete le ... e ne e le setšoantšong hammoho le tse tsejoang_mahosts. 'Me CI leha e le efe e hlōlehile ka phoso ea bohlokoa ea ho netefatsa ka lebaka la ho se lumellane pakeng tsa IP ea sebele le e boletsoeng ho tsejoa_hosts. Setšoantšo se secha se ile sa bokelloa kapele ho tsoa ho li-Dockfiles tse seng li ntse li le teng mme khetho ea eketsoa StrictHostKeyChecking no
. Empa tatso e mpe e ile ea sala 'me ho ne ho e-na le takatso ea ho fetisetsa libs sebakeng sa polokelo ea poraefete ea PyPI. Bonase e eketsehileng, kamora ho fetohela ho PyPI ea poraefete, e ne e le phaephe e bonolo le tlhaloso e tloaelehileng ea litlhoko.txt
Khetho e se e entsoe, Bahlomphehi!
Re tsamaisa ntho e 'ngoe le e' ngoe marung le Kubernetes, 'me qetellong re ne re batla ho fumana tšebeletso e nyenyane e neng e le setsi se se nang naha se nang le polokelo ea ka ntle. Ho joalo, kaha re sebelisa S3, e ile ea etelletsoa pele ho eona. Mme, haeba ho khonahala, ka netefatso ho gitlab (o ka e eketsa ka bouena ha ho hlokahala).
Patlo e potlakileng e hlahisitse liphetho tse 'maloa: s3pypi, pypicloud le khetho e nang le "manual" ea ho theha lifaele tsa html bakeng sa turnips. Khetho ea ho qetela e ile ea nyamela ka boeona.
s3pypi: Ena ke cli bakeng sa ho sebelisa S3 hosting. Re kenya lifaele, re hlahisa html ebe re e kenya ka nkhong e le 'ngoe. E loketse tšebeliso ea lapeng.
pypicloud: Ho ne ho bonahala eka ke morero o thahasellisang, empa ka mor'a ho bala litokomane ke ile ka nyahama. Leha ho na le litokomane tse ntle le bokhoni ba ho hola ho lumellana le litlhoko tsa hau, ha e le hantle ho ile ha fumaneha hore ha e na thuso ebile ho thata ho e hlophisa. Ho lokisa khoutu hore e lumellane le mesebetsi ea hau, ho latela likhakanyo ka nako eo, ho ka be ho nkile matsatsi a 3-5. Tšebeletso e boetse e hloka database. Re e tlohetse haeba re sa fumane letho le leng.
Patlisiso e tebileng haholoanyane e hlahisitse mojule oa Nginx, ngx_aws_auth. Sephetho sa tlhahlobo ea hae e bile XML e bonts'itsoeng ho sebatli, e bonts'itseng litaba tsa bakete ea S3. Boitlamo ba ho qetela nakong ea patlo e bile selemo se fetileng. Sebaka sa polokelo se ne se shebahala se lahliloe.
Ka ho ea mohloling le ho bala
Ka ho nka mohlala ona e le motheo, hora hamorao ke ile ka bona sebatli sa ka XML e tšoanang le ha ke sebelisa ngx_aws_auth module, empa ntho e 'ngoe le e' ngoe e ne e se e ngotsoe ka JS.
Ke ratile tharollo ea nginx haholo. Ntlha ea pele, litokomane tse ntle le mehlala e mengata, ea bobeli, re fumana lintho tsohle tse ntle tsa Nginx bakeng sa ho sebetsa le lifaele (ka ntle ho lebokose), boraro, mang kapa mang ea tsebang ho ngola li-configs bakeng sa Nginx o tla khona ho tseba hore na ke eng. Minimalism e boetse e na le phaello ho nna, ha e bapisoa le Python kapa Go (haeba e ngotsoe ho tloha qalong), re sa bue ka nexus.
TL; DR Kamora matsatsi a 2, mofuta oa tlhahlobo oa PyPi o ne o se o sebelisitsoe ho CI.
Hona e sebetsa?
Module e kentsoe ho Nginx ngx_http_js_module
, e kenyellelitsoeng setšoantšong sa semmuso sa docker. Re kenya mongolo oa rona ka taelo js_import
ho tlhophiso ea Nginx. Mosebetsi o bitsoa ka taelo js_content
. Taelo e sebelisoa ho beha mefuta-futa js_set
, e nkang e le khang feela mosebetsi o hlalositsoeng ho mongolo. Empa re ka etsa li-subqueries ho NJS feela re sebelisa Nginx, eseng XMLHttpRequest efe kapa efe. Ho etsa sena, sebaka se lumellanang se tlameha ho eketsoa ho tlhophiso ea Nginx. Mme sengoloa se tlameha ho hlalosa kopo e nyane sebakeng sena. Ho khona ho fumana ts'ebetso ho tsoa ho config ea Nginx, lebitso la ts'ebetso le tlameha ho romelloa kantle ho script ka boeona export default
.
nginx.conf
load_module modules/ngx_http_js_module.so;
http {
js_import imported_name from script.js;
server {
listen 8080;
...
location = /sub-query {
internal;
proxy_pass http://upstream;
}
location / {
js_content imported_name.request;
}
}
script.js
function request(r) {
function call_back(resp) {
// handler's code
r.return(resp.status, resp.responseBody);
}
r.subrequest('/sub-query', { method: r.method }, call_back);
}
export default {request}
Ha o kopuoa ho sebatli http://localhost:8080/
re kena location /
eo ho eona taelo js_content
letsa tshebetso request
e hlalositsoe bukeng ea rona script.js
. Ka lehlakoreng le leng, mosebetsing request
subquery e etsoa ho location = /sub-query
, ka mokhoa (mohlaleng oa hajoale GET) o fumanoeng ho tsoa khang (r)
, e fetisitsoe ka mokhoa o hlakileng ha mosebetsi ona o bitsoa. Karabo ea kopo e nyane e tla sebetsoa tšebetsong call_back
.
Ho leka S3
Ho etsa kopo ea polokelo ea poraefete ea S3, re hloka:
ACCESS_KEY
SECRET_KEY
S3_BUCKET
Ho tsoa mokhoeng o sebelisitsoeng oa http, letsatsi/nako ea hajoale, S3_NAME le URI, ho hlahisoa mofuta o itseng oa khoele, o saenneng (HMAC_SHA1) ho sebelisoa SECRET_KEY. E latelang ke mola joalo ka AWS $ACCESS_KEY:$HASH
, e ka sebelisoa sehloohong sa tumello. Letsatsi / nako e sebelisitsoeng ho hlahisa khoele mohatong o fetileng e tlameha ho kenngoa hloohong X-amz-date
. Ka khoutu e shebahala tjena:
nginx.conf
load_module modules/ngx_http_js_module.so;
http {
js_import s3 from s3.js;
js_set $s3_datetime s3.date_now;
js_set $s3_auth s3.s3_sign;
server {
listen 8080;
...
location ~* /s3-query/(?<s3_path>.*) {
internal;
proxy_set_header X-amz-date $s3_datetime;
proxy_set_header Authorization $s3_auth;
proxy_pass $s3_endpoint/$s3_path;
}
location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
js_content s3.request;
}
}
s3.js
(Mohlala oa tumello ea AWS Sign v2, o fetotsoe ho boemo bo theohileng)
var crypt = require('crypto');
var s3_bucket = process.env.S3_BUCKET;
var s3_access_key = process.env.S3_ACCESS_KEY;
var s3_secret_key = process.env.S3_SECRET_KEY;
var _datetime = new Date().toISOString().replace(/[:-]|.d{3}/g, '');
function date_now() {
return _datetime
}
function s3_sign(r) {
var s2s = r.method + 'nnnn';
s2s += `x-amz-date:${date_now()}n`;
s2s += '/' + s3_bucket;
s2s += r.uri.endsWith('/') ? '/' : r.variables.s3_path;
return `AWS ${s3_access_key}:${crypt.createHmac('sha1', s3_secret_key).update(s2s).digest('base64')}`;
}
function request(r) {
var v = r.variables;
function call_back(resp) {
r.return(resp.status, resp.responseBody);
}
var _subrequest_uri = r.uri;
if (r.uri === '/') {
// root
_subrequest_uri = '/?delimiter=/';
} else if (v.prefix !== '' && v.postfix === '') {
// directory
var slash = v.prefix.endsWith('/') ? '' : '/';
_subrequest_uri = '/?prefix=' + v.prefix + slash;
}
r.subrequest(`/s3-query${_subrequest_uri}`, { method: r.method }, call_back);
}
export default {request, s3_sign, date_now}
Tlhaloso e nyane mabapi le _subrequest_uri
: sena ke phetoho eo, ho itšetlehile ka uri oa pele, e etsang kopo ho S3. Haeba o hloka ho fumana litaba tsa "motso", joale o hloka ho etsa kopo ea uri e bonts'ang delimiter. delimiter
, e tla khutlisa lethathamo la likarolo tsohle tsa CommonPrefixes xml, tse tsamaellanang le li-directory (tabeng ea PyPI, lethathamo la liphutheloana tsohle). Haeba o hloka ho fumana lethathamo la litaba bukeng e itseng (lenane la mefuta eohle ea liphutheloana), kopo ea uri e tlameha ho ba le sebaka sa prefix se nang le lebitso la sephutheloana (sephutheloana) se qetellang ka slash /. Ho seng joalo, ho ka etsahala ho thulana ha u kopa litaba tsa buka, mohlala. Ho na le li-directory aiohttp-kopo le aiohttp-likopo mme haeba kopo e bolela /?prefix=aiohttp-request
, joale karabo e tla ba le likahare tsa litsamaiso ka bobeli. Haeba ho na le slash qetellong, /?prefix=aiohttp-request/
, joale karabo e tla ba le bukana e hlokahalang feela. 'Me haeba re kopa faele, joale uri e hlahisoang ha ea lokela ho fapana le ea pele.
Boloka le ho qala Nginx hape. Ho sebatli re kenya aterese ea Nginx ea rona, sephetho sa kopo e tla ba XML, mohlala:
Lenane la li-directory
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>myback-space</Name>
<Prefix></Prefix>
<Marker></Marker>
<MaxKeys>10000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>new/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>old/</Prefix>
</CommonPrefixes>
</ListBucketResult>
Ho tsoa lethathamong la li-directory u tla hloka feela likarolo CommonPrefixes
.
Ka ho kenyelletsa bukana eo re e hlokang atereseng ea rona ho sebatli, re tla fumana litaba tsa eona ka mokhoa oa XML:
Lethathamo la lifaele tse bukeng
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name> myback-space</Name>
<Prefix>old/</Prefix>
<Marker></Marker>
<MaxKeys>10000</MaxKeys>
<Delimiter></Delimiter>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>old/giphy.mp4</Key>
<LastModified>2020-08-21T20:27:46.000Z</LastModified>
<ETag>"00000000000000000000000000000000-1"</ETag>
<Size>1350084</Size>
<Owner>
<ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
<DisplayName></DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>old/hsd-k8s.jpg</Key>
<LastModified>2020-08-31T16:40:01.000Z</LastModified>
<ETag>"b2d76df4aeb4493c5456366748218093"</ETag>
<Size>93183</Size>
<Owner>
<ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
<DisplayName></DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
</Contents>
</ListBucketResult>
Ho tsoa lethathamong la lifaele re tla nka likarolo feela Key
.
Ho setseng ke ho hlalosa XML e hlahang le ho e romela e le HTML, ka mor'a hore u nkele sehlooho sa Content-Type sebaka ka text/html.
function request(r) {
var v = r.variables;
function call_back(resp) {
var body = resp.responseBody;
if (r.method !== 'PUT' && resp.status < 400 && v.postfix === '') {
r.headersOut['Content-Type'] = "text/html; charset=utf-8";
body = toHTML(body);
}
r.return(resp.status, body);
}
var _subrequest_uri = r.uri;
...
}
function toHTML(xml_str) {
var keysMap = {
'CommonPrefixes': 'Prefix',
'Contents': 'Key',
};
var pattern = `<k>(?<v>.*?)</k>`;
var out = [];
for(var group_key in keysMap) {
var reS;
var reGroup = new RegExp(pattern.replace(/k/g, group_key), 'g');
while(reS = reGroup.exec(xml_str)) {
var data = new RegExp(pattern.replace(/k/g, keysMap[group_key]), 'g');
var reValue = data.exec(reS);
var a_text = '';
if (group_key === 'CommonPrefixes') {
a_text = reValue.groups.v.replace(///g, '');
} else {
a_text = reValue.groups.v.split('/').slice(-1);
}
out.push(`<a href="/st/${reValue.groups.v}">${a_text}</a>`);
}
}
return '<html><body>n' + out.join('</br>n') + 'n</html></body>'
}
Ho leka PyPI
Re hlahloba hore ha ho letho le robehang kae kapa kae liphutheloana tse tsejoang hore li sebetsa.
# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate
# Скачиваем рабочие пакеты.
pip download aiohttp
# Загружаем в приватную репу
for wheel in *.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done
rm -f *.whl
# Устанавливаем из приватной репы
pip install aiohttp -i http://localhost:8080
Re pheta ka libs tsa rona.
# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate
pip install setuptools wheel
python setup.py bdist_wheel
for wheel in dist/*.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done
pip install our_pkg --extra-index-url http://localhost:8080
Ho CI, ho theha le ho kenya sephutheloana ho shebahala tjena:
pip install setuptools wheel
python setup.py bdist_wheel
curl -sSfT dist/*.whl -u "gitlab-ci-token:${CI_JOB_TOKEN}" "https://pypi.our-domain.com/${CI_PROJECT_NAME}"
Netefatso
Ho Gitlab hoa khoneha ho sebelisa JWT bakeng sa netefatso / tumello ea lits'ebeletso tsa kantle. Re sebelisa taelo ea auth_request ho Nginx, re tla fetisetsa data ea netefatso ho subrequest e nang le mohala oa ts'ebetso ho script. Sengoloa se tla etsa kopo e 'ngoe ho url ea Gitlab mme haeba data ea netefatso e hlalositsoe ka nepo, Gitlab e tla khutlisa khoutu ea 200 mme ho kenya / ho jarolla sephutheloana ho tla lumelloa. Hobaneng u sa sebelise subquery e le 'ngoe' me hang-hang u romelle data ho Gitlab? Hobane joale re tla tlameha ho hlophisa faele ea tlhophiso ea Nginx nako le nako ha re etsa liphetoho leha e le life tsa tumello, 'me ona ke mosebetsi o boima. Hape, haeba Kubernetes e sebelisa leano la ho bala feela motso oa tsamaiso ea lifaele, joale sena se eketsa ho rarahana le ho feta ha ho nkela nginx.conf sebaka ka configmap. 'Me e ba ntho e ke keng ea khoneha ho lokisa Nginx ka configmap ha ka nako e ts'oanang u sebelisa melaoana e thibelang khokahanyo ea li-volumes (pvc) le mokhoa oa ho bala feela (sena le sona sea etsahala).
Ka ho sebelisa NJS e bohareng, re fumana monyetla oa ho fetola li-parameter tse boletsoeng ho nginx config re sebelisa mefuta-futa ea tikoloho le ho etsa licheke tse ling ho script (mohlala, URL e sa nepahalang).
nginx.conf
location = /auth-provider {
internal;
proxy_pass $auth_url;
}
location = /auth {
internal;
proxy_set_header Content-Length "";
proxy_pass_request_body off;
js_content auth.auth;
}
location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
auth_request /auth;
js_content s3.request;
}
s3.js
var env = process.env;
var env_bool = new RegExp(/[Tt]rue|[Yy]es|[Oo]n|[TtYy]|1/);
var auth_disabled = env_bool.test(env.DISABLE_AUTH);
var gitlab_url = env.AUTH_URL;
function url() {
return `${gitlab_url}/jwt/auth?service=container_registry`
}
function auth(r) {
if (auth_disabled) {
r.return(202, '{"auth": "disabled"}');
return null
}
r.subrequest('/auth-provider',
{method: 'GET', body: ''},
function(res) {
r.return(res.status, "");
});
}
export default {auth, url}
Ho ka etsahala hore ebe potso ke ho riteloa: -Hobaneng u sa sebelise li-modules tse seng li entsoe? Lintho tsohle li se li entsoe moo! Mohlala, var AWS = hloka('aws-sdk') 'me ha ho hlokahale ho ngola "baesekele" e nang le netefatso ea S3!
A re feteleng pele ho ea bokhopo
Ho 'na, ho se khone ho kenya li-module tsa JS tsa kantle ho ile ha fetoha ntho e sa thabiseng, empa e lebelletsoeng. E hlalositsoe mohlaleng o ka holimo need('crypto') is
Compression le eona e tlameha ho koaloa bakeng sa projeke ea hajoale ea Nginx gzip off;
Hobane ha ho na module ea gzip ho NJS mme ha ho khonehe ho e hokahanya; ka hona, ha ho na mokhoa oa ho sebetsa ka data e hatelitsoeng. Ke 'nete, sena ha se phoso bakeng sa nyeoe ena. Ha ho na mongolo o mongata, mme lifaele tse fetisitsoeng li se li hatelitsoe mme khatello e eketsehileng e ke ke ea ba thusa haholo. Hape, sena ha se ts'ebeletso e imetsoeng kapa e mahlonoko hoo u tlamehang ho khathatseha ka ho fana ka litaba ka li-milliseconds tse 'maloa kapele.
Ho lokisa script ho nka nako e telele mme ho khoneha feela ka "prints" ho error.log. Ho ipapisitsoe le tlhaiso-leseling ea boemo ba ho rema lifate, tlhokomeliso kapa phoso, hoa khoneha ho sebelisa mekhoa e 3 r.log, r.warn, r.error ka ho latellana. Ke leka ho lokisa mangolo a mang ho Chrome (v8) kapa sesebelisoa sa khomphutha sa njs, empa ha se tsohle tse ka hlahlojoang moo. Ha o hlakola khoutu, tlhahlobo e sebetsang, nalane e shebahala tjena:
docker-compose restart nginx
curl localhost:8080/
docker-compose logs --tail 10 nginx
mme ho ka ba le tatellano e joalo e makholo.
Khoutu ea ho ngola o sebelisa li-subqueries le mefuta-futa bakeng sa tsona e fetoha moferefere o ferekaneng. Ka linako tse ling u qala ho potlaka ho potoloha IDE lifensetere tse fapaneng u leka ho fumana tatellano ea liketso tsa khoutu ea hau. Ha ho thata, empa ka linako tse ling hoa tena haholo.
Ha ho na tšehetso e felletseng bakeng sa ES6.
Ho ka 'na ha e-ba le mefokolo e meng, empa ha ke e-s'o kopane le letho le leng. Arolelana lintlha haeba u na le phihlelo e mpe ka ho sebelisa NJS.
fihlela qeto e
NJS ke mofetoleli o bobebe oa mohloli o bulehileng o o lumellang ho kenya tšebetsong mangolo a fapaneng a JavaScript ho Nginx. Nakong ea tsoelo-pele ea eona, tlhokomelo e kholo e ile ea lefshoa mosebetsing. Ha e le hantle, ho ntse ho e-na le lintho tse ngata tse sieo, empa morero o ntse o ntlafatsoa ke sehlopha se senyenyane 'me ba ntse ba eketsa likarolo tse ncha le ho lokisa liphoso. Ke tšepa hore ka letsatsi le leng NJS e tla u lumella ho hokahanya li-module tsa ka ntle, tse tla etsa hore ts'ebetso ea Nginx e be e se nang moeli. Empa ho na le NGINX Plus mme ho ka etsahala hore ebe ha ho na likarolo!
Source: www.habr.com