Ndinapanga malo anga a PyPI ndi chilolezo ndi S3. Pa Nginx

M'nkhaniyi ndikufuna kugawana zomwe ndakumana nazo ndi NJS, womasulira JavaScript wa Nginx wopangidwa ndi Nginx Inc, kufotokoza mphamvu zake zazikulu pogwiritsa ntchito chitsanzo chenicheni. NJS ndi gawo la JavaScript lomwe limakupatsani mwayi wowonjezera magwiridwe antchito a Nginx. Ku funso chifukwa chani womasulira wekha??? Dmitry Volntsev anayankha mwatsatanetsatane. Mwachidule: NJS ndi nginx-way, ndipo JavaScript ikupita patsogolo, "yachibadwidwe" komanso yopanda GC, mosiyana ndi Lua.

Kalekale…

Pantchito yanga yomaliza, ndidalandira cholowa cha gitlab chokhala ndi mapaipi angapo a motley CI / CD okhala ndi docker-compose, dind ndi zokondweretsa zina, zomwe zidasamutsidwa ku njanji za kaniko. Zithunzi zomwe zidagwiritsidwa ntchito m'mbuyomu mu CI zidasunthidwa ngati mawonekedwe ake oyamba. Anagwira ntchito bwino mpaka tsiku limene gitlab IP yathu inasintha ndipo CI inasanduka dzungu. Vuto linali loti chimodzi mwazithunzi za docker zomwe zidatenga nawo gawo mu CI zinali ndi git, zomwe zidakoka ma module a Python kudzera pa ssh. Kwa ssh mumafunika kiyi yachinsinsi ndipo ... inali pachithunzichi pamodzi ndi odziwika_makamu. Ndipo CI iliyonse inalephera ndi cholakwika chachikulu chotsimikizira chifukwa cha kusagwirizana pakati pa IP yeniyeni ndi yomwe yatchulidwa mu known_hosts. Chithunzi chatsopano chinasonkhanitsidwa mwachangu kuchokera ku Dockfiles omwe analipo ndipo njirayo idawonjezedwa StrictHostKeyChecking no. Koma kukoma koyipa kunatsalira ndipo panali chikhumbo chosuntha ma libs kumalo osungirako a PyPI. Bhonasi yowonjezera, mutasinthira ku PyPI yachinsinsi, inali payipi yosavuta komanso kulongosola kwabwino kwa zofunikira.txt

Chisankho chapangidwa, Amuna!

Timayendetsa chirichonse mumitambo ndi Kubernetes, ndipo pamapeto pake tinkafuna kupeza ntchito yaing'ono yomwe inali chidebe chopanda malire ndi kusungirako kunja. Chabwino, popeza timagwiritsa ntchito S3, choyambirira chidaperekedwa kwa icho. Ndipo, ngati n'kotheka, ndi kutsimikizika mu gitlab (mukhoza kuwonjezera nokha ngati kuli kofunikira).

Kusaka mwachangu kunapeza zotsatira zingapo: s3pypi, pypicloud ndi njira yokhala ndi "pamanja" kupanga mafayilo a html a turnips. Njira yomaliza idazimiririka yokha.

s3pypi: Ichi ndi cli chogwiritsa ntchito kuchititsa S3. Timayika mafayilo, kupanga html ndikuyiyika ku chidebe chomwecho. Zoyenera kugwiritsidwa ntchito kunyumba.

pypicloud: Zinkawoneka ngati ntchito yosangalatsa, koma nditawerenga zolembazo ndinakhumudwa. Ngakhale zolemba zabwino komanso kuthekera kokulitsa kuti zigwirizane ndi zosowa zanu, zenizeni zidakhala zosafunikira komanso zovuta kuzikonza. Kukonza kachidindoko kuti kagwirizane ndi ntchito zanu, malinga ndi kuyerekezera panthawiyo, kukadatenga masiku 3-5. Utumiki umafunikanso database. Tinazisiya ngati sitinapeze china chilichonse.

Kufufuza mozama kunapereka gawo la Nginx, ngx_aws_auth. Zotsatira za kuyesa kwake zinali XML yowonetsedwa mu msakatuli, yomwe ikuwonetsa zomwe zili mu chidebe cha S3. Kudzipereka komaliza pa nthawi yakusaka kunali chaka chapitacho. Malo osungiramo zinthu adawoneka atasiyidwa.

Mwa kupita ku gwero ndi kuwerenga PEP-503 Ndinazindikira kuti XML ikhoza kusinthidwa kukhala HTML pa ntchentche ndikupatsidwa pip. Nditayang'ana pang'ono za Nginx ndi S3, ndidapeza chitsanzo cha kutsimikizika mu S3 cholembedwa mu JS cha Nginx. Umu ndi momwe ndinakumana ndi NJS.

Kutengera chitsanzo ichi ngati maziko, ola pambuyo pake ndinawona mu msakatuli wanga XML yofanana ndikugwiritsa ntchito ngx_aws_auth module, koma zonse zinali zitalembedwa kale mu JS.

Ndinkakonda kwambiri yankho la nginx. Choyamba, zolemba zabwino ndi zitsanzo zambiri, kachiwiri, timapeza zabwino zonse za Nginx pogwira ntchito ndi mafayilo (kunja kwa bokosi), chachitatu, aliyense amene amadziwa kulemba configs kwa Nginx adzatha kudziwa chomwe chiri. Minimalism ndiyowonjezeranso kwa ine, poyerekeza ndi Python kapena Go (ngati yalembedwa kuyambira pachiyambi), osatchulanso nexus.

TL; DR Pambuyo pa masiku a 2, kuyesa kwa PyPi kudagwiritsidwa ntchito kale ku CI.

Kodi ntchito?

Module imayikidwa mu Nginx ngx_http_js_module, ophatikizidwa mu chithunzi chovomerezeka cha docker. Timalowetsa zolemba zathu pogwiritsa ntchito malangizowo js_importku Nginx kasinthidwe. Ntchitoyi imatchedwa ndi Directive js_content. Dongosolo limagwiritsidwa ntchito kukhazikitsa zosintha js_set, zomwe zimatengera mtsutso wokha ntchito yomwe yafotokozedwa mu script. Koma titha kuchita ma subqueries mu NJS pogwiritsa ntchito Nginx, osati XMLHttpRequest iliyonse. Kuti muchite izi, malo oyenerera ayenera kuwonjezeredwa ku kasinthidwe ka Nginx. Ndipo script iyenera kufotokoza pempho laling'ono kumalo ano. Kuti muthe kupeza ntchito kuchokera ku Nginx config, dzina la ntchitoyo liyenera kutumizidwa kunja kwa script yokha export default.

nginx.conf

load_module modules/ngx_http_js_module.so;
http {
  js_import   imported_name  from script.js;

server {
  listen 8080;
  ...
  location = /sub-query {
    internal;

    proxy_pass http://upstream;
  }

  location / {
    js_content imported_name.request;
  }
}

script.js

function request(r) {
  function call_back(resp) {
    // handler's code
    r.return(resp.status, resp.responseBody);
  }

  r.subrequest('/sub-query', { method: r.method }, call_back);
}

export default {request}

Mukafunsidwa mu msakatuli http://localhost:8080/ timalowa mu location /momwe malangizowo js_content kuyitana ntchito request zafotokozedwa mu script yathu script.js. Komanso, mu ntchito request subquery imapangidwa location = /sub-query, ndi njira (muchitsanzo chamakono GET) yopezera mkangano (r), idadutsa mosabisa pamene ntchitoyi imatchedwa. Yankho la subrequest lidzakonzedwa mu ntchitoyi call_back.

Kuyesa S3

Kuti tipemphe kusungirako kwachinsinsi kwa S3, tifunika:

ACCESS_KEY

SECRET_KEY

S3_BUCKET

Kuchokera pa njira yogwiritsiridwa ntchito ya http, tsiku/nthawi yamakono, S3_NAME ndi URI, chingwe chamtundu wina chimapangidwa, chomwe chimasaina (HMAC_SHA1) pogwiritsa ntchito SECRET_KEY. Chotsatira ndi mzere ngati AWS $ACCESS_KEY:$HASH, angagwiritsidwe ntchito pamutu wovomerezeka. Tsiku / nthawi yomweyi yomwe idagwiritsidwa ntchito kupanga chingwe mu sitepe yapitayi iyenera kuwonjezeredwa pamutu X-amz-date. Mu code zikuwoneka ngati izi:

nginx.conf

load_module modules/ngx_http_js_module.so;
http {
  js_import   s3      from     s3.js;

  js_set      $s3_datetime     s3.date_now;
  js_set      $s3_auth         s3.s3_sign;

server {
  listen 8080;
  ...
  location ~* /s3-query/(?<s3_path>.*) {
    internal;

    proxy_set_header    X-amz-date     $s3_datetime;
    proxy_set_header    Authorization  $s3_auth;

    proxy_pass          $s3_endpoint/$s3_path;
  }

  location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
    js_content s3.request;
  }
}

s3.js(Chitsanzo chovomerezeka cha AWS Sign v2, chasinthidwa kukhala chotsitsidwa)

var crypt = require('crypto');

var s3_bucket = process.env.S3_BUCKET;
var s3_access_key = process.env.S3_ACCESS_KEY;
var s3_secret_key = process.env.S3_SECRET_KEY;
var _datetime = new Date().toISOString().replace(/[:-]|.d{3}/g, '');

function date_now() {
  return _datetime
}

function s3_sign(r) {
  var s2s = r.method + 'nnnn';

  s2s += `x-amz-date:${date_now()}n`;
  s2s += '/' + s3_bucket;
  s2s += r.uri.endsWith('/') ? '/' : r.variables.s3_path;

  return `AWS ${s3_access_key}:${crypt.createHmac('sha1', s3_secret_key).update(s2s).digest('base64')}`;
}

function request(r) {
  var v = r.variables;

  function call_back(resp) {
    r.return(resp.status, resp.responseBody);
  }

  var _subrequest_uri = r.uri;
  if (r.uri === '/') {
    // root
    _subrequest_uri = '/?delimiter=/';

  } else if (v.prefix !== '' && v.postfix === '') {
    // directory
    var slash = v.prefix.endsWith('/') ? '' : '/';
    _subrequest_uri = '/?prefix=' + v.prefix + slash;
  }

  r.subrequest(`/s3-query${_subrequest_uri}`, { method: r.method }, call_back);
}

export default {request, s3_sign, date_now}

Kufotokozera pang'ono za _subrequest_uri: uku ndikusintha komwe, kutengera uri woyambirira, kumapanga pempho ku S3. Ngati mukufuna kupeza zomwe zili mu "muzu", ndiye kuti muyenera kupanga pempho la uri kusonyeza delimiter. delimiter, yomwe idzabweretsanso mndandanda wazinthu zonse za CommonPrefixes xml, zogwirizana ndi maulamuliro (pankhani ya PyPI, mndandanda wa mapepala onse). Ngati mukufuna kupeza mndandanda wazomwe zili mu bukhu linalake (mndandanda wamitundu yonse ya phukusi), ndiye kuti pempho la uri liyenera kukhala ndi gawo loyambira ndi dzina lachikwatu (phukusi) lomaliza ndi slash /. Apo ayi, kugunda kumatheka popempha zomwe zili mu bukhu, mwachitsanzo. Pali akalozera aiohttp-pempho ndi aiohttp-zopempha ndipo ngati pempho likunena /?prefix=aiohttp-request, ndiye yankho lidzakhala ndi zomwe zili m'mawunivesite onse awiri. Ngati pali slash kumapeto, /?prefix=aiohttp-request/, ndiye yankho likhala ndi chikwatu chomwe chikufunika. Ndipo ngati tipempha fayilo, ndiye kuti uri wotsatira sayenera kusiyana ndi woyambayo.

Sungani ndikuyambitsanso Nginx. Mu msakatuli timalowetsa adilesi ya Nginx yathu, zotsatira za pempholi zidzakhala XML, mwachitsanzo:

Mndandanda wamakalata

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>myback-space</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>10000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
  <CommonPrefixes>
    <Prefix>new/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>old/</Prefix>
  </CommonPrefixes>
</ListBucketResult>

Kuchokera pamndandanda wamakanema mudzafunika zinthu zokha CommonPrefixes.

Powonjezera chikwatu chomwe tikufuna ku adilesi yathu mu msakatuli, tilandilanso zomwe zili mu mawonekedwe a XML:

Mndandanda wamafayilo mu ndandanda

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name> myback-space</Name>
  <Prefix>old/</Prefix>
  <Marker></Marker>
  <MaxKeys>10000</MaxKeys>
  <Delimiter></Delimiter>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>old/giphy.mp4</Key>
    <LastModified>2020-08-21T20:27:46.000Z</LastModified>
    <ETag>&#34;00000000000000000000000000000000-1&#34;</ETag>
    <Size>1350084</Size>
    <Owner>
      <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
      <DisplayName></DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <Contents>
    <Key>old/hsd-k8s.jpg</Key>
    <LastModified>2020-08-31T16:40:01.000Z</LastModified>
    <ETag>&#34;b2d76df4aeb4493c5456366748218093&#34;</ETag>
    <Size>93183</Size>
    <Owner>
      <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
      <DisplayName></DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
</ListBucketResult>

Kuchokera pamndandanda wa mafayilo tidzatenga zinthu zokha Key.

Chotsalira ndikuwunika XML yomwe idatuluka ndikuitumiza ngati HTML, mutasintha mutu wa Content-Type ndi text/html.

function request(r) {
  var v = r.variables;

  function call_back(resp) {
    var body = resp.responseBody;

    if (r.method !== 'PUT' && resp.status < 400 && v.postfix === '') {
      r.headersOut['Content-Type'] = "text/html; charset=utf-8";
      body = toHTML(body);
    }

    r.return(resp.status, body);
  }
  
  var _subrequest_uri = r.uri;
  ...
}

function toHTML(xml_str) {
  var keysMap = {
    'CommonPrefixes': 'Prefix',
    'Contents': 'Key',
  };

  var pattern = `<k>(?<v>.*?)</k>`;
  var out = [];

  for(var group_key in keysMap) {
    var reS;
    var reGroup = new RegExp(pattern.replace(/k/g, group_key), 'g');

    while(reS = reGroup.exec(xml_str)) {
      var data = new RegExp(pattern.replace(/k/g, keysMap[group_key]), 'g');
      var reValue = data.exec(reS);
      var a_text = '';

      if (group_key === 'CommonPrefixes') {
        a_text = reValue.groups.v.replace(///g, '');
      } else {
        a_text = reValue.groups.v.split('/').slice(-1);
      }

      out.push(`<a href="/ny/${reValue.groups.v}">${a_text}</a>`);
    }
  }

  return '<html><body>n' + out.join('</br>n') + 'n</html></body>'
}

Kuyesa PyPI

Timaonetsetsa kuti palibe chomwe chimasweka paliponse pamaphukusi omwe amadziwika kuti amagwira ntchito.

# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate

# Скачиваем рабочие пакеты.
pip download aiohttp

# Загружаем в приватную репу
for wheel in *.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done

rm -f *.whl

# Устанавливаем из приватной репы
pip install aiohttp -i http://localhost:8080

Timabwereza ndi libs zathu.

# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate

pip install setuptools wheel
python setup.py bdist_wheel
for wheel in dist/*.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done

pip install our_pkg --extra-index-url http://localhost:8080

Mu CI, kupanga ndi kutsitsa phukusi kumawoneka motere:

pip install setuptools wheel
python setup.py bdist_wheel

curl -sSfT dist/*.whl -u "gitlab-ci-token:${CI_JOB_TOKEN}" "https://pypi.our-domain.com/${CI_PROJECT_NAME}"

Kutsimikizika

Ku Gitlab ndizotheka kugwiritsa ntchito JWT kutsimikizira / kuvomereza ntchito zakunja. Pogwiritsa ntchito lamulo la auth_request ku Nginx, tidzalozera deta yovomerezeka ku subrequest yomwe ili ndi foni yogwira ntchito mu script. Cholembacho chidzapanganso kufunsiranso kwa url ya Gitlab ndipo ngati chidziwitso chotsimikizika chidanenedwa molondola, ndiye Gitlab idzabwezera khodi 200 ndipo kutsitsa / kutsitsa phukusi kudzaloledwa. Bwanji osagwiritsa ntchito funso limodzi ndikutumiza nthawi yomweyo ku Gitlab? Chifukwa ndiye tifunika kusintha fayilo yosinthira ya Nginx nthawi iliyonse tikasintha zilolezo, ndipo iyi ndi ntchito yotopetsa. Komanso, ngati Kubernetes amagwiritsa ntchito ndondomeko yowerengera mizu yamafayilo, ndiye kuti izi zimawonjezera zovuta kwambiri pochotsa nginx.conf kudzera pa configmap. Ndipo zimakhala zosatheka kukhazikitsa Nginx kudzera pa configmap panthawi imodzimodziyo pogwiritsa ntchito ndondomeko zoletsa kugwirizanitsa ma volumes (pvc) ndi mizu yowerengera yokha (izi zimachitikanso).

Pogwiritsa ntchito NJS yapakatikati, timapeza mwayi wosintha magawo omwe atchulidwa mu nginx config pogwiritsa ntchito zosintha za chilengedwe ndikuyang'ana pa script (mwachitsanzo, URL yotchulidwa molakwika).

nginx.conf

location = /auth-provider {
  internal;

  proxy_pass $auth_url;
}

location = /auth {
  internal;

  proxy_set_header Content-Length "";
  proxy_pass_request_body off;
  js_content auth.auth;
}

location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
  auth_request /auth;

  js_content s3.request;
}

s3.js

var env = process.env;
var env_bool = new RegExp(/[Tt]rue|[Yy]es|[Oo]n|[TtYy]|1/);
var auth_disabled  = env_bool.test(env.DISABLE_AUTH);
var gitlab_url = env.AUTH_URL;

function url() {
  return `${gitlab_url}/jwt/auth?service=container_registry`
}

function auth(r) {
  if (auth_disabled) {
    r.return(202, '{"auth": "disabled"}');
    return null
  }

  r.subrequest('/auth-provider',
                {method: 'GET', body: ''},
                function(res) {
                  r.return(res.status, "");
                });
}

export default {auth, url}

Nthawi zambiri funso ndilakuti: -Bwanji osagwiritsa ntchito ma module okonzeka? Zonse zachitika kale kumeneko! Mwachitsanzo, var AWS = amafuna('aws-sdk') ndipo palibe chifukwa cholembera "njinga" yokhala ndi kutsimikizika kwa S3!

Tiyeni tipitirire ku zoyipa

Kwa ine, kulephera kuitanitsa ma module a JS akunja kunakhala chinthu chosasangalatsa, koma choyembekezeredwa. Zofotokozedwa mu chitsanzo pamwambapa amafuna('crypto') ndi ma module owonjezera ndipo amangofuna ntchito kwa iwo okha. Palibenso njira yogwiritsiranso ntchito kachidindo kuchokera m'malemba ndipo muyenera kukopera ndikuyiyika m'mafayilo osiyanasiyana. Ndikukhulupirira kuti tsiku lina izi zidzakwaniritsidwa.

Kuponderezana kuyeneranso kuyimitsidwa pantchito yomwe ilipo mu Nginx gzip off;

Chifukwa palibe gawo la gzip mu NJS ndipo ndizosatheka kulumikiza; chifukwa chake, palibe njira yogwirira ntchito ndi data yothinikizidwa. Zowona, uku sikungochotsera pamlanduwu. Palibe zolemba zambiri, ndipo mafayilo osamutsidwa ali othinikizidwa kale ndipo kuponderezana kwina sikungawathandize kwambiri. Komanso, iyi sintchito yodzaza kapena yovuta kotero kuti muyenera kuvutitsidwa ndikupereka zomwe zili ma millisecond ochepa mwachangu.

Kuthetsa vutoli kumatenga nthawi yayitali ndipo kumatheka kudzera mu "prints" mu error.log. Kutengera zomwe zakhazikitsidwa, kuchenjeza kapena zolakwika, ndizotheka kugwiritsa ntchito njira zitatu r.log, r.warn, r.error motsatana. Ndimayesetsa kukonza zolemba zina mu Chrome (v3) kapena chida cha njs console, koma sizinthu zonse zomwe zingayang'anitsidwe pamenepo. Mukachotsa code, aka kuyesa magwiridwe antchito, mbiri imawoneka motere:

docker-compose restart nginx
curl localhost:8080/
docker-compose logs --tail 10 nginx

ndipo pakhoza kukhala mazana otsatizana oterowo.

Kulemba ma code pogwiritsa ntchito ma subqueries ndi zosinthika kwa iwo kumasintha kukhala tangle yopiringizika. Nthawi zina mumayamba kuthamangira mozungulira ma IDE osiyanasiyana windows kuyesa kudziwa momwe ma code anu amayendera. Sizovuta, koma nthawi zina zimakhala zokhumudwitsa kwambiri.

Palibe chithandizo chonse cha ES6.

Pakhoza kukhala zophophonya zina, koma sindinakumanepo ndi china chilichonse. Gawani zambiri ngati muli ndi vuto pogwiritsa ntchito NJS.

Pomaliza

NJS ndi womasulira wopepuka wotsegula yemwe amakulolani kugwiritsa ntchito zolemba zosiyanasiyana za JavaScript mu Nginx. Pa chitukuko chake, chidwi chachikulu chinaperekedwa ku ntchito. Zoonadi, pali zambiri zomwe zikusowa, koma polojekitiyi ikupangidwa ndi gulu laling'ono ndipo likuwonjezera zatsopano ndikukonza zolakwika. Ndikuyembekeza kuti tsiku lina NJS idzakulolani kuti mugwirizane ndi ma modules akunja, zomwe zidzapangitse Nginx kugwira ntchito mopanda malire. Koma pali NGINX Plus ndipo mwina sipadzakhala mawonekedwe!

Malo okhala ndi code yonse ya nkhaniyo

njs-pypi ndi thandizo la AWS Sign v4

Kufotokozera kwa malangizo a ngx_http_js_module

Malo ovomerezeka a NJS и zolemba

Zitsanzo za kugwiritsa ntchito NJS kuchokera kwa Dmitry Volintsev

njs - zolemba zaku JavaScript mu nginx / Zolankhula za Dmitry Volnyev ku Saint HighLoad++ 2019

NJS pakupanga / Zolankhula za Vasily Soshnikov ku HighLoad++ 2019

Kusaina ndi Kutsimikizira Zopempha za REST mu AWS

Source: www.habr.com