Na yi wurin ajiyar PyPI na tare da izini da S3. Na Nginx

A cikin wannan labarin zan so in raba gwaninta tare da NJS, mai fassarar JavaScript na Nginx wanda Nginx Inc ya haɓaka, yana kwatanta babban ƙarfinsa ta amfani da misali na gaske. NJS wani yanki ne na JavaScript wanda ke ba ku damar tsawaita ayyukan Nginx. Zuwa tambaya meyasa mai fassara ku??? Dmitry Volyntsev ya amsa daki-daki. A takaice: NJS ita ce hanyar nginx, kuma JavaScript ya fi ci gaba, "yan ƙasa" kuma ba tare da GC ba, sabanin Lua.

Wani lokaci mai tsawo…

A aikina na ƙarshe, na gaji gitlab tare da bututun motley CI/CD da yawa tare da docker-compose, dind da sauran abubuwan jin daɗi, waɗanda aka tura su zuwa layin kaniko. Hotunan da aka yi amfani da su a baya a cikin CI an motsa su a cikin asali. Sun yi aiki da kyau har zuwa ranar da gitlab IP ya canza kuma CI ya zama kabewa. Matsalar ita ce ɗayan hotunan docker da suka shiga cikin CI yana da git, wanda ya ja tsarin Python ta ssh. Don ssh kuna buƙatar maɓallin sirri kuma... yana cikin hoton tare da sanannun_hosts. Kuma duk wani CI ya gaza tare da kuskuren tabbatarwa mai mahimmanci saboda rashin daidaituwa tsakanin ainihin IP da wanda aka ƙayyade a cikin sanannun_hosts. An haɗa sabon hoto da sauri daga Dockfiles na yanzu kuma an ƙara zaɓi StrictHostKeyChecking no. Amma mummunan dandano ya kasance kuma akwai sha'awar motsa libs zuwa wurin ajiyar PyPI mai zaman kansa. Ƙarin kari, bayan canzawa zuwa PyPI mai zaman kansa, ya kasance mafi sauƙi bututu da bayanin al'ada na bukatun.txt

An yi zabi, Ya ku Jama'a!

Muna gudanar da komai a cikin gajimare da Kubernetes, kuma a ƙarshe muna so mu sami ƙaramin sabis wanda ya kasance akwati marar ƙasa tare da ajiyar waje. To, tunda muna amfani da S3, an ba shi fifiko. Kuma, idan zai yiwu, tare da tabbaci a gitlab (zaka iya ƙara shi da kanka idan ya cancanta).

Bincike mai sauri ya haifar da sakamako da yawa: s3pypi, pypicloud da zaɓi tare da “manual” ƙirƙirar fayilolin html don turnips. Zaɓin ƙarshe ya ɓace da kanta.

s3pypi: Wannan cli don amfani da S3 hosting. Muna loda fayilolin, samar da html kuma mu loda su cikin guga ɗaya. Ya dace da amfanin gida.

pypicloud: Ya zama kamar aiki mai ban sha'awa, amma bayan karanta takardun na ji takaici. Duk da kyawawan takardu da ikon faɗaɗa don dacewa da buƙatun ku, a zahiri ya juya ya zama mai yawa kuma yana da wahala a daidaita shi. Gyara lambar don dacewa da ayyukanku, bisa ga ƙididdiga a lokacin, zai ɗauki kwanaki 3-5. Sabis ɗin kuma yana buƙatar rumbun adana bayanai. Mun bar shi idan ba mu sami wani abu ba.

Ƙarin bincike mai zurfi ya samar da tsari don Nginx, ngx_aws_auth. Sakamakon gwajin nasa an nuna XML a cikin mashigar bincike, wanda ya nuna abubuwan da ke cikin bokitin S3. Ƙarshe na ƙarshe a lokacin binciken shine shekara guda da ta wuce. Ma'ajiyar ta duba an watsar.

Ta hanyar zuwa tushe da karatu Farashin PEP-503 Na gane cewa ana iya canza XML zuwa HTML akan tashi kuma a ba shi pip. Bayan yin ɗan ƙara ɗan ƙarawa game da Nginx da S3, na ci karo da misalin tantancewa a cikin S3 da aka rubuta a JS don Nginx. A haka na hadu da NJS.

Daukar wannan misali a matsayin tushe, sa'a guda bayan haka na ga XML iri ɗaya a cikin burauzarta kamar lokacin amfani da ngx_aws_auth module, amma an riga an rubuta komai a cikin JS.

Ina matukar son maganin nginx. Da fari dai, kyawawan takardu da misalai da yawa, na biyu, muna samun duk kyawawan abubuwan Nginx don aiki tare da fayiloli (daga cikin akwatin), na uku, duk wanda ya san yadda ake rubuta saiti don Nginx zai iya gano menene menene. Minimalism kuma ƙari ne a gare ni, idan aka kwatanta da Python ko Go (idan an rubuta shi daga karce), ba tare da ambaton haɗin gwiwa ba.

TL;DR Bayan kwanaki 2, an riga an yi amfani da sigar gwajin PyPi a cikin CI.

Yaya ta yi aiki?

Ana loda tsarin a cikin Nginx ngx_http_js_module, an haɗa cikin hoton docker na hukuma. Muna shigo da rubutun mu ta amfani da umarnin js_importzuwa tsarin Nginx. Ana kiran aikin ta hanyar umarni js_content. Ana amfani da umarnin don saita masu canji js_set, wanda ke ɗaukar azaman hujja kawai aikin da aka kwatanta a cikin rubutun. Amma za mu iya aiwatar da subqueries a cikin NJS kawai ta amfani da Nginx, ba kowane XMLHttpRequest ba. Don yin wannan, dole ne a ƙara wurin da ya dace zuwa tsarin Nginx. Kuma dole ne rubutun ya siffanta batun neman wannan wurin. Don samun damar samun damar aiki daga saitin Nginx, dole ne a fitar da sunan aikin a cikin rubutun kanta export default.

nginx.conf

load_module modules/ngx_http_js_module.so;
http {
  js_import   imported_name  from script.js;

server {
  listen 8080;
  ...
  location = /sub-query {
    internal;

    proxy_pass http://upstream;
  }

  location / {
    js_content imported_name.request;
  }
}

script.js

function request(r) {
  function call_back(resp) {
    // handler's code
    r.return(resp.status, resp.responseBody);
  }

  r.subrequest('/sub-query', { method: r.method }, call_back);
}

export default {request}

Lokacin da aka nema a cikin burauzar http://localhost:8080/ mu shiga location /a cikinsa umarnin js_content kira aiki request aka bayyana a cikin rubutun mu script.js. Bi da bi, a cikin aikin request an yi subquery zuwa location = /sub-query, tare da hanyar (a cikin misali na yanzu GET) da aka samo daga gardama (r), ya wuce a fakaice lokacin da ake kiran wannan aikin. Za a aiwatar da martanin da aka nema a cikin aikin call_back.

Gwada S3

Don yin buƙatu zuwa ma'ajiyar S3 masu zaman kansu, muna buƙatar:

ACCESS_KEY

SECRET_KEY

S3_BUCKET

Daga hanyar http da aka yi amfani da ita, kwanan wata/lokaci na yanzu, S3_NAME da URI, ana samar da wani nau'in kirtani, wanda aka sanya hannu (HMAC_SHA1) ta amfani da SECRET_KEY. Na gaba shine layi kamar AWS $ACCESS_KEY:$HASH, ana iya amfani dashi a cikin taken izini. Kwanan wata/lokaci guda ɗaya da aka yi amfani da shi don samar da kirtani a matakin da ya gabata dole ne a ƙara shi zuwa kan taken X-amz-date. A cikin code yana kama da haka:

nginx.conf

load_module modules/ngx_http_js_module.so;
http {
  js_import   s3      from     s3.js;

  js_set      $s3_datetime     s3.date_now;
  js_set      $s3_auth         s3.s3_sign;

server {
  listen 8080;
  ...
  location ~* /s3-query/(?<s3_path>.*) {
    internal;

    proxy_set_header    X-amz-date     $s3_datetime;
    proxy_set_header    Authorization  $s3_auth;

    proxy_pass          $s3_endpoint/$s3_path;
  }

  location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
    js_content s3.request;
  }
}

s3.js(AWS Alamar misali v2 izini, an canza zuwa matsayin da aka yanke)

var crypt = require('crypto');

var s3_bucket = process.env.S3_BUCKET;
var s3_access_key = process.env.S3_ACCESS_KEY;
var s3_secret_key = process.env.S3_SECRET_KEY;
var _datetime = new Date().toISOString().replace(/[:-]|.d{3}/g, '');

function date_now() {
  return _datetime
}

function s3_sign(r) {
  var s2s = r.method + 'nnnn';

  s2s += `x-amz-date:${date_now()}n`;
  s2s += '/' + s3_bucket;
  s2s += r.uri.endsWith('/') ? '/' : r.variables.s3_path;

  return `AWS ${s3_access_key}:${crypt.createHmac('sha1', s3_secret_key).update(s2s).digest('base64')}`;
}

function request(r) {
  var v = r.variables;

  function call_back(resp) {
    r.return(resp.status, resp.responseBody);
  }

  var _subrequest_uri = r.uri;
  if (r.uri === '/') {
    // root
    _subrequest_uri = '/?delimiter=/';

  } else if (v.prefix !== '' && v.postfix === '') {
    // directory
    var slash = v.prefix.endsWith('/') ? '' : '/';
    _subrequest_uri = '/?prefix=' + v.prefix + slash;
  }

  r.subrequest(`/s3-query${_subrequest_uri}`, { method: r.method }, call_back);
}

export default {request, s3_sign, date_now}

Bayani kadan game da _subrequest_uri: wannan maɓalli ne wanda, dangane da farkon uri, yana samar da buƙatu zuwa S3. Idan kuna buƙatar samun abubuwan da ke cikin "tushen", to kuna buƙatar ƙirƙirar buƙatar uri wanda ke nuna mai iyaka. delimiter, wanda zai dawo da jerin duk abubuwan CommonPrefixes xml, daidai da kundin adireshi (a cikin yanayin PyPI, jerin duk fakitin). Idan kuna buƙatar samun jerin abubuwan da ke ciki a cikin takamaiman jagorar (jerin duk nau'ikan fakitin), to buƙatar uri dole ne ta ƙunshi filin prefix tare da sunan directory (kunshin) tabbas yana ƙarewa tare da slash /. In ba haka ba, karo na iya yiwuwa lokacin neman abin da ke cikin kundin adireshi, misali. Akwai kundayen adireshi aiohttp-request da aiohttp-requests kuma idan buƙatar ta fayyace /?prefix=aiohttp-request, to, amsa zai ƙunshi abubuwan da ke cikin kundayen adireshi biyu. Idan akwai slash a karshen, /?prefix=aiohttp-request/, to, amsar za ta ƙunshi kundin da ake buƙata kawai. Kuma idan muka nemi fayil, to, sakamakon uri bai kamata ya bambanta da na asali ba.

Ajiye kuma sake kunna Nginx. A cikin browser mun shigar da adireshin Nginx namu, sakamakon buƙatar zai zama XML, misali:

Jerin kundayen adireshi

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>myback-space</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>10000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
  <CommonPrefixes>
    <Prefix>new/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>old/</Prefix>
  </CommonPrefixes>
</ListBucketResult>

Daga jerin kundayen adireshi za ku buƙaci abubuwa kawai CommonPrefixes.

Ta ƙara directory ɗin da muke buƙata zuwa adireshinmu a cikin burauzar, za mu kuma karɓi abubuwan da ke ciki a cikin sigar XML:

Jerin fayiloli a cikin kundin adireshi

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name> myback-space</Name>
  <Prefix>old/</Prefix>
  <Marker></Marker>
  <MaxKeys>10000</MaxKeys>
  <Delimiter></Delimiter>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>old/giphy.mp4</Key>
    <LastModified>2020-08-21T20:27:46.000Z</LastModified>
    <ETag>&#34;00000000000000000000000000000000-1&#34;</ETag>
    <Size>1350084</Size>
    <Owner>
      <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
      <DisplayName></DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <Contents>
    <Key>old/hsd-k8s.jpg</Key>
    <LastModified>2020-08-31T16:40:01.000Z</LastModified>
    <ETag>&#34;b2d76df4aeb4493c5456366748218093&#34;</ETag>
    <Size>93183</Size>
    <Owner>
      <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
      <DisplayName></DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
</ListBucketResult>

Daga jerin fayiloli za mu ɗauki abubuwa kawai Key.

Abin da ya rage shi ne a tantance sakamakon XML da aika shi azaman HTML, tun da farko an maye gurbin rubutun Nau'in Abun ciki da rubutu/html.

function request(r) {
  var v = r.variables;

  function call_back(resp) {
    var body = resp.responseBody;

    if (r.method !== 'PUT' && resp.status < 400 && v.postfix === '') {
      r.headersOut['Content-Type'] = "text/html; charset=utf-8";
      body = toHTML(body);
    }

    r.return(resp.status, body);
  }
  
  var _subrequest_uri = r.uri;
  ...
}

function toHTML(xml_str) {
  var keysMap = {
    'CommonPrefixes': 'Prefix',
    'Contents': 'Key',
  };

  var pattern = `<k>(?<v>.*?)</k>`;
  var out = [];

  for(var group_key in keysMap) {
    var reS;
    var reGroup = new RegExp(pattern.replace(/k/g, group_key), 'g');

    while(reS = reGroup.exec(xml_str)) {
      var data = new RegExp(pattern.replace(/k/g, keysMap[group_key]), 'g');
      var reValue = data.exec(reS);
      var a_text = '';

      if (group_key === 'CommonPrefixes') {
        a_text = reValue.groups.v.replace(///g, '');
      } else {
        a_text = reValue.groups.v.split('/').slice(-1);
      }

      out.push(`<a href="/ha/${reValue.groups.v}">${a_text}</a>`);
    }
  }

  return '<html><body>n' + out.join('</br>n') + 'n</html></body>'
}

Gwada PyPI

Muna duba cewa babu wani abu da ke karya ko'ina akan fakitin da aka sani suna aiki.

# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate

# Скачиваем рабочие пакеты.
pip download aiohttp

# Загружаем в приватную репу
for wheel in *.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done

rm -f *.whl

# Устанавливаем из приватной репы
pip install aiohttp -i http://localhost:8080

Muna maimaitawa da libyoyin mu.

# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate

pip install setuptools wheel
python setup.py bdist_wheel
for wheel in dist/*.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done

pip install our_pkg --extra-index-url http://localhost:8080

A cikin CI, ƙirƙira da loda kunshin yayi kama da haka:

pip install setuptools wheel
python setup.py bdist_wheel

curl -sSfT dist/*.whl -u "gitlab-ci-token:${CI_JOB_TOKEN}" "https://pypi.our-domain.com/${CI_PROJECT_NAME}"

Gasktawa

A cikin Gitlab yana yiwuwa a yi amfani da JWT don tabbatarwa / ba da izini na ayyukan waje. Yin amfani da umarnin auth_request a cikin Nginx, za mu tura bayanan tabbatarwa zuwa wani buƙatun da ke ɗauke da kiran aiki a cikin rubutun. Rubutun zai sake yin wani bincike ga Gitlab url kuma idan an ƙayyade bayanan tabbatarwa daidai, to Gitlab zai dawo da lambar 200 kuma za a ba da izinin saukewa/zazzagewar kunshin. Me zai hana a yi amfani da juzu'i ɗaya kuma nan da nan aika bayanan zuwa Gitlab? Domin a lokacin dole ne mu gyara fayil ɗin sanyi na Nginx duk lokacin da muka yi kowane canje-canje a cikin izini, kuma wannan aiki ne mai wahala. Hakanan, idan Kubernetes yana amfani da tsarin tushen tsarin fayil ɗin karantawa kawai, to wannan yana ƙara ƙarin rikitarwa yayin maye gurbin nginx.conf ta hanyar daidaitawa. Kuma ya zama ba zai yuwu ba don saita Nginx ta hanyar daidaitawa yayin amfani da manufofin da ke hana haɗin juzu'i (pvc) da tsarin tushen tushen karantawa kawai (wannan kuma yana faruwa).

Yin amfani da matsakaicin NJS, muna samun damar canza ƙayyadaddun sigogi a cikin saitin nginx ta amfani da masu canjin yanayi kuma muna yin wasu cak a cikin rubutun (misali URL ɗin da ba daidai ba).

nginx.conf

location = /auth-provider {
  internal;

  proxy_pass $auth_url;
}

location = /auth {
  internal;

  proxy_set_header Content-Length "";
  proxy_pass_request_body off;
  js_content auth.auth;
}

location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
  auth_request /auth;

  js_content s3.request;
}

s3.js

var env = process.env;
var env_bool = new RegExp(/[Tt]rue|[Yy]es|[Oo]n|[TtYy]|1/);
var auth_disabled  = env_bool.test(env.DISABLE_AUTH);
var gitlab_url = env.AUTH_URL;

function url() {
  return `${gitlab_url}/jwt/auth?service=container_registry`
}

function auth(r) {
  if (auth_disabled) {
    r.return(202, '{"auth": "disabled"}');
    return null
  }

  r.subrequest('/auth-provider',
                {method: 'GET', body: ''},
                function(res) {
                  r.return(res.status, "");
                });
}

export default {auth, url}

Mafi mahimmanci tambayar ita ce yin burodi: -Me ya sa ba a yi amfani da kayan aikin da aka shirya ba? An riga an yi komai a can! Misali, var AWS = buƙatar ('aws-sdk') kuma babu buƙatar rubuta "bike" tare da ingantaccen S3!

Bari mu matsa zuwa ga fursunoni

A gare ni, rashin iya shigo da kayayyaki na JS na waje ya zama mara daɗi, amma fasalin da ake tsammani. An bayyana a cikin misalin da ke sama yana buƙatar ('crypto') shine gina-in kayayyaki kuma suna buƙatar aiki kawai a gare su. Hakanan babu wata hanya ta sake amfani da lambar daga rubutun kuma dole ne ku kwafa da liƙa ta cikin fayiloli daban-daban. Ina fatan cewa wata rana za a aiwatar da wannan aikin.

Hakanan dole ne a kashe matsi don aikin na yanzu a cikin Nginx gzip off;

Domin babu gzip module a cikin NJS kuma ba shi yiwuwa a haɗa shi; sabili da haka, babu wata hanyar aiki tare da matsa lamba. Gaskiya, wannan ba ainihin ragi bane ga wannan harka. Babu rubutu da yawa, kuma fayilolin da aka canjawa wuri sun riga sun matsa kuma ƙarin matsawa ba zai taimaka musu da yawa ba. Hakanan, wannan ba irin wannan sabis ɗin da aka ɗora ba ne ko mai mahimmanci wanda dole ne ku damu tare da isar da abun ciki 'yan millise seconds cikin sauri.

Gyara rubutun yana ɗaukar lokaci mai tsawo kuma yana yiwuwa ta hanyar "bugu" a cikin kuskure.log. Dangane da saitin bayanin matakin shiga, gargadi ko kuskure, yana yiwuwa a yi amfani da hanyoyin 3 r.log, r.warn, r.error bi da bi. Ina ƙoƙarin gyara wasu rubutun a cikin Chrome (v8) ko kayan aikin njs console, amma ba duk abin da za'a iya bincika a can ba. Lokacin gyara lambar, aka gwada gwajin aiki, tarihi yayi kama da haka:

docker-compose restart nginx
curl localhost:8080/
docker-compose logs --tail 10 nginx

kuma ana iya samun ɗaruruwan irin waɗannan jeri.

Rubutun lambar ta yin amfani da ƙananan tambayoyi da masu canji a gare su yana juya zuwa tangle mai rikitarwa. Wani lokaci ka fara zagayawa tagar windows IDE daban-daban suna ƙoƙarin gano jerin ayyukan lambar ku. Ba shi da wahala, amma wani lokacin yana da ban haushi sosai.

Babu cikakken goyon baya ga ES6.

Wataƙila akwai wasu gazawa, amma ban ci karo da komai ba. Raba bayanai idan kuna da kwarewa mara kyau ta amfani da NJS.

ƙarshe

NJS shine fassarar buɗaɗɗen tushe mai sauƙi wanda ke ba ku damar aiwatar da rubutun JavaScript daban-daban a cikin Nginx. A lokacin ci gabanta, an biya kulawa sosai ga aikin. Tabbas, har yanzu akwai da yawa da ba a rasa ba, amma ƙaramin ƙungiya ne ke haɓaka aikin kuma suna ƙara sabbin abubuwa da kuma gyara kurakurai. Ina fatan cewa wata rana NJS zai ba ku damar haɗa na'urori na waje, wanda zai sa aikin Nginx kusan marar iyaka. Amma akwai NGINX Plus kuma mai yiwuwa ba za a sami fasali ba!

Wurin ajiya tare da cikakken lambar labarin

njs-pypi tare da tallafin AWS Sign v4

Bayanin umarnin ngx_http_js_module module

Ma'ajiyar NJS ta hukuma и takardun shaida

Misalai na amfani da NJS daga Dmitry Volintsev

njs - rubutun JavaScript na asali a cikin nginx / Jawabin Dmitry Volnyev a Saint HighLoad++ 2019

NJS a cikin samarwa / Jawabin Vasily Soshnikov a HighLoad++ 2019

Shiga da Tabbatar da Buƙatun REST a cikin AWS

source: www.habr.com