ProHoster > Блог > Whakahaerenga > I hanga e au taku ake putunga PyPI me te whakamana me te S3. Kei runga Nginx

I hanga e au taku ake putunga PyPI me te whakamana me te S3. Kei runga Nginx

I roto i tenei tuhinga ka hiahia ahau ki te whakapuaki i taku wheako ki a NJS, he kaiwhakamaori JavaScript mo Nginx i whakawhanakehia e Nginx Inc, e whakaatu ana i ona kaha matua ma te whakamahi i tetahi tauira pono. Ko te NJS he waahanga o te JavaScript e taea ai e koe te whakawhānui ake i te mahi a Nginx. Ki te patai he aha to koutou ake kaiwhakamaori??? I whakautu a Dmitry Volyntsev i nga korero. I roto i te poto: NJS he nginx-ara, me te JavaScript he pai ake te ahu whakamua, "maori" me te kore GC, kaore i rite ki a Lua.

I mua -…

I taku mahi whakamutunga, i riro mai i a au te gitlab me te maha o nga pipeline CI/CD motley me te docker-compose, te dind me etahi atu mea pai, i whakawhitia ki te reera kaniko. Ko nga whakaahua i whakamahia i mua i roto i te CI i nekehia i to raatau ahua taketake. I mahi tika ratou tae noa ki te ra i huri ai to tatou IP gitlab ka huri a CI hei paukena. Ko te raru ko tetahi o nga whakaahua docker i uru ki te CI he git, nana i kumea nga waahanga Python ma te ssh. Mo te ssh ka hiahia koe ki tetahi kii motuhake me ... kei roto i te ahua me nga mohiotia_hosts. Na tetahi CI i rahua me te hapa manatoko matua na te kore o te taurite i waenga i te IP tuuturu me te mea kua tohua i roto i nga mohiotanga_hosts. I whakaemihia he ahua hou mai i nga Dockfiles o naianei ka taapirihia te whiringa StrictHostKeyChecking no. Engari i mau tonu te reka kino me te hiahia ki te neke i nga libs ki tetahi putunga PyPI motuhake. He putea taapiri, i muri i te huri ki PyPI tūmataiti, he paipa ngawari ake me te whakaahuatanga noa o requirements.txt

Kua oti te whiriwhiri, e te rangatira!

Ka whakahaerehia e matou nga mea katoa i roto i nga kapua me nga Kubernetes, a, i te mutunga ka hiahia matou ki te tiki ratonga iti he ipu kore whenua me te rokiroki o waho. Ana, i te mea kei te whakamahi matou i te S3, ko te kaupapa matua. A, ki te taea, me te whakamotuhēhēnga i roto i te gitlab (ka taea e koe te taapiri i a koe mena e tika ana).

He maha nga hua i puta mai i te rapunga tere: s3pypi, pypicloud me tetahi whiringa me te hanga "a-ringa" o nga konae html mo nga keha. Ko te whiringa whakamutunga i ngaro ko ia ano.

s3pypi: He cli tenei mo te whakamahi i te manaaki S3. Ka tukuna e matou nga konae, ka whakaputa i te html ka tukuna ki te peere kotahi. He pai mo te whakamahi kaainga.

pypicloud: Ko te ahua he kaupapa whakamere, engari i muri i te panui i nga tuhinga ka pouri ahau. Ahakoa te pai o nga tuhinga me te kaha ki te whakawhänui kia rite ki o hiahia, i roto i te mooni ka puta he nui, he uaua ki te whirihora. Ko te whakatika i te waehere kia rite ki o mahi, e ai ki nga whakatau tata i tera wa, ka roa te 3-5 ra. Kei te hiahia hoki te ratonga i te papaa raraunga. I whakarerea e matou mehemea kaore i kitea e matou tetahi atu mea.

Ko te rapunga hohonu ake ka puta he kōwae mo Nginx, ngx_aws_auth. Ko te hua o tana whakamatautau ko te XML i whakaatuhia i roto i te tirotiro, i whakaatu i nga ihirangi o te peere S3. Ko te mahi whakamutunga i te wa o te rapunga he tau ki muri. Ko te whare putunga kua whakarerea.

Ma te haere ki te puna me te panui PEP-503 I mohio ahau ka taea te huri i te XML ki te HTML i runga i te rere ka hoatu ki te pip. I muri i te rapu korero mo Nginx me S3, ka kite ahau i tetahi tauira o te motuhēhēnga i roto i te S3 i tuhia ki te JS mo Nginx. Koira taku tutaki ki a NJS.

Ma te tango i tenei tauira hei kaupapa, he haora i muri mai ka kite ahau i roto i taku kaitirotiro te XML rite i te wa e whakamahi ana i te ngx_aws_auth module, engari kua tuhia nga mea katoa ki te JS.

I tino pai ahau ki te otinga nginx. Tuatahi, he pai nga tuhinga me te maha o nga tauira, tuarua, ka whiwhi tatou i nga pai katoa o Nginx mo te mahi me nga konae (mai i te pouaka), tuatoru, ko te tangata e mohio ana ki te tuhi whirihora mo Nginx ka mohio he aha. Ko te Minimalism he mea taapiri mo au, ka whakatauritea ki te Python, ki te Haere ranei (mehemea i tuhia mai i te wahanga), kaua e whakahua i te nohonga.

TL;DR I muri i nga ra e 2, kua whakamahia kee te putanga whakamatautau o PyPi ki CI.

Ka pēhea te mahi i te reira?

Ka utaina te kōwae ki Nginx ngx_http_js_module, kei roto i te ahua docker mana. Ka kawemai i a maatau tuhinga ma te whakamahi i te tohutohu js_importki te whirihoranga Nginx. Ka karangahia te mahi e tetahi tohutohu js_content. Ka whakamahia te tohutohu hei whakarite taurangi js_set, ka waiho hei tautohe ko te mahi anake i whakaahuatia i roto i te tuhinga. Engari ka taea e taatau te whakahaere i nga paatai ​​​​i roto i te NJS ma te whakamahi noa i a Nginx, ehara i te XMLHttpRequest. Ki te mahi i tenei, me taapiri te waahi e tika ana ki te whirihoranga Nginx. A me whakaatu e te tuhinga he tono tuarua ki tenei waahi. Kia taea ai e koe te uru atu ki tetahi mahi mai i te whirihora Nginx, me kawe te ingoa mahi i roto i te tuhinga ake export default.

nginx.conf

load_module modules/ngx_http_js_module.so;
http {
  js_import   imported_name  from script.js;

server {
  listen 8080;
  ...
  location = /sub-query {
    internal;

    proxy_pass http://upstream;
  }

  location / {
    js_content imported_name.request;
  }
}

script.js

function request(r) {
  function call_back(resp) {
    // handler's code
    r.return(resp.status, resp.responseBody);
  }

  r.subrequest('/sub-query', { method: r.method }, call_back);
}

export default {request}

Ina tonohia i roto i te tirotiro http://localhost:8080/ ka uru tatou ki roto location /i roto i nei te tohutohu js_content ka karanga he mahi request whakaahuatia i roto i ta maatau tuhinga script.js. I roto i te tahuri, i roto i te mahi request ka mahia he patai ki location = /sub-query, me tetahi tikanga (i te tauira o naianei GET) i whiwhi mai i te tohenga (r), ka paahi i te wa e kiia ana tenei mahi. Ka tukatukahia te whakautu tono i roto i te mahi call_back.

E ngana ana ki te S3

Hei tuku tono ki te rokiroki S3 motuhake, me:

ACCESS_KEY

SECRET_KEY

S3_BUCKET

Mai i te tikanga http i whakamahia, te ra/wa o naianei, S3_NAME me te URI, ka hangaia he momo aho, ka hainatia (HMAC_SHA1) ma te whakamahi i te SECRET_KEY. Kei muri ko te rarangi rite AWS $ACCESS_KEY:$HASH, ka taea te whakamahi i te pane whakamana. Ko te ra/wa ano i whakamahia hei whakaputa i te aho i te taahiraa o mua me taapiri ki te pane X-amz-date. I roto i te waehere he penei te ahua:

nginx.conf

load_module modules/ngx_http_js_module.so;
http {
  js_import   s3      from     s3.js;

  js_set      $s3_datetime     s3.date_now;
  js_set      $s3_auth         s3.s3_sign;

server {
  listen 8080;
  ...
  location ~* /s3-query/(?<s3_path>.*) {
    internal;

    proxy_set_header    X-amz-date     $s3_datetime;
    proxy_set_header    Authorization  $s3_auth;

    proxy_pass          $s3_endpoint/$s3_path;
  }

  location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
    js_content s3.request;
  }
}

s3.js(He tauira whakamanatanga Waitohu AWS v2, kua huri ki te mana kua mutu)

var crypt = require('crypto');

var s3_bucket = process.env.S3_BUCKET;
var s3_access_key = process.env.S3_ACCESS_KEY;
var s3_secret_key = process.env.S3_SECRET_KEY;
var _datetime = new Date().toISOString().replace(/[:-]|.d{3}/g, '');

function date_now() {
  return _datetime
}

function s3_sign(r) {
  var s2s = r.method + 'nnnn';

  s2s += `x-amz-date:${date_now()}n`;
  s2s += '/' + s3_bucket;
  s2s += r.uri.endsWith('/') ? '/' : r.variables.s3_path;

  return `AWS ${s3_access_key}:${crypt.createHmac('sha1', s3_secret_key).update(s2s).digest('base64')}`;
}

function request(r) {
  var v = r.variables;

  function call_back(resp) {
    r.return(resp.status, resp.responseBody);
  }

  var _subrequest_uri = r.uri;
  if (r.uri === '/') {
    // root
    _subrequest_uri = '/?delimiter=/';

  } else if (v.prefix !== '' && v.postfix === '') {
    // directory
    var slash = v.prefix.endsWith('/') ? '' : '/';
    _subrequest_uri = '/?prefix=' + v.prefix + slash;
  }

  r.subrequest(`/s3-query${_subrequest_uri}`, { method: r.method }, call_back);
}

export default {request, s3_sign, date_now}

He whakamarama iti mo _subrequest_uri: he taurangi tenei, i runga i te uri tuatahi, ka hanga tono ki te S3. Mena ka hiahia koe ki te tiki i nga ihirangi o te "pakiaka", me hanga e koe he tono uri e tohu ana i te kaiwehe. delimiter, ka whakahoki mai i te rarangi o nga huānga xml CommonPrefixes katoa, e rite ana ki nga raarangi whaiaronga (i te keehi o PyPI, he rarangi o nga kohinga katoa). Mena ka hiahia koe ki te tiki rarangi ihirangi kei roto i tetahi raarangi motuhake (he rarangi o nga momo momo kete katoa), katahi ka uru ki te tono uri he mara prefix me te ingoa o te whaiaronga (painga) me mutu ki te rītaha /. Ki te kore, ka taea te tukinga ina tono i nga ihirangi o te whaiaronga, hei tauira. He whaiaronga aiohttp-tono me te aiohttp-tono a mena ka tohua e te tono /?prefix=aiohttp-request, katahi ka mau i te whakautu nga ihirangi o nga raarangi e rua. Mena he rītaha kei te mutunga, /?prefix=aiohttp-request/, katahi ka mau i te whakautu ko te whaiaronga e hiahiatia ana. A, ki te tono tatou i tetahi konae, kia kaua e rereke nga uri ka puta mai i te uri taketake.

Tiakina ka timata ano a Nginx. I roto i te tirotiro ka whakauruhia e matou te wahitau o to tatou Nginx, ko te hua o te tono ko XML, hei tauira:

Rārangi o ngā whaiaronga

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>myback-space</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>10000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
  <CommonPrefixes>
    <Prefix>new/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>old/</Prefix>
  </CommonPrefixes>
</ListBucketResult>

Mai i te rarangi o nga raarangi ka hiahia koe ki nga huānga anake CommonPrefixes.

Ma te taapiri i te whaiaronga e hiahiatia ana e matou ki ta matou wahitau i roto i te tirotiro, ka whiwhi ano matou i ona ihirangi i roto i te puka XML:

Rarangi o nga konae kei roto i te whaiaronga

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name> myback-space</Name>
  <Prefix>old/</Prefix>
  <Marker></Marker>
  <MaxKeys>10000</MaxKeys>
  <Delimiter></Delimiter>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>old/giphy.mp4</Key>
    <LastModified>2020-08-21T20:27:46.000Z</LastModified>
    <ETag>&#34;00000000000000000000000000000000-1&#34;</ETag>
    <Size>1350084</Size>
    <Owner>
      <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
      <DisplayName></DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <Contents>
    <Key>old/hsd-k8s.jpg</Key>
    <LastModified>2020-08-31T16:40:01.000Z</LastModified>
    <ETag>&#34;b2d76df4aeb4493c5456366748218093&#34;</ETag>
    <Size>93183</Size>
    <Owner>
      <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>
      <DisplayName></DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
</ListBucketResult>

Mai i te rarangi o nga konae ka tangohia e matou nga huānga anake Key.

Ko nga mea e toe ana ko te poroporoaki i te putanga XML ka tukuna hei HTML, na te tuatahi i whakakapi i te pane Ihirangi-Momo ki te tuhinga/html.

function request(r) {
  var v = r.variables;

  function call_back(resp) {
    var body = resp.responseBody;

    if (r.method !== 'PUT' && resp.status < 400 && v.postfix === '') {
      r.headersOut['Content-Type'] = "text/html; charset=utf-8";
      body = toHTML(body);
    }

    r.return(resp.status, body);
  }
  
  var _subrequest_uri = r.uri;
  ...
}

function toHTML(xml_str) {
  var keysMap = {
    'CommonPrefixes': 'Prefix',
    'Contents': 'Key',
  };

  var pattern = `<k>(?<v>.*?)</k>`;
  var out = [];

  for(var group_key in keysMap) {
    var reS;
    var reGroup = new RegExp(pattern.replace(/k/g, group_key), 'g');

    while(reS = reGroup.exec(xml_str)) {
      var data = new RegExp(pattern.replace(/k/g, keysMap[group_key]), 'g');
      var reValue = data.exec(reS);
      var a_text = '';

      if (group_key === 'CommonPrefixes') {
        a_text = reValue.groups.v.replace(///g, '');
      } else {
        a_text = reValue.groups.v.split('/').slice(-1);
      }

      out.push(`<a href="/mi/${reValue.groups.v}">${a_text}</a>`);
    }
  }

  return '<html><body>n' + out.join('</br>n') + 'n</html></body>'
}

E ngana ana ki te PyPI

Ka tirohia e matou kaore he mea e pakaru i nga waahi katoa i runga i nga kete e mohiotia ana he mahi.

# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate

# Скачиваем рабочие пакеты.
pip download aiohttp

# Загружаем в приватную репу
for wheel in *.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done

rm -f *.whl

# Устанавливаем из приватной репы
pip install aiohttp -i http://localhost:8080

Ka whakahoki ano tatou me o tatou libs.

# Создаем для тестов новое окружение
python3 -m venv venv
. ./venv/bin/activate

pip install setuptools wheel
python setup.py bdist_wheel
for wheel in dist/*.whl; do curl -T $wheel http://localhost:8080/${wheel%%-*}/$wheel; done

pip install our_pkg --extra-index-url http://localhost:8080

I roto i te CI, ko te hanga me te uta i tetahi kete penei te ahua:

pip install setuptools wheel
python setup.py bdist_wheel

curl -sSfT dist/*.whl -u "gitlab-ci-token:${CI_JOB_TOKEN}" "https://pypi.our-domain.com/${CI_PROJECT_NAME}"

Whakaaturanga

I roto i te Gitlab ka taea te whakamahi i te JWT mo te whakamotuhēhētanga / whakamanatanga o nga ratonga o waho. Ma te whakamahi i te arata'i auth_request i Nginx, ka tukuna e matou nga raraunga whakamotuhēhēnga ki te tono tuarua kei roto he waea mahi i roto i te tuhinga. Ka tukuna e te tuhinga tetahi atu tono ki te url Gitlab, a, ki te tika te tautuhi i nga raraunga motuhēhēnga, ka whakahokia e Gitlab te waehere 200 ka whakaaetia te tukunga/tango ake o te kete. He aha e kore ai e whakamahi i tetahi patai ka tukuna tonu nga raraunga ki Gitlab? Na te mea ka whakatikahia e matou te konae whirihoranga Nginx i nga wa katoa ka whakarereketia e matou te whakamanatanga, he mahi tino hoha tenei. Ano, ki te whakamahi a Kubernetes i te kaupapa here punaha konae-anake, ka nui ake te uaua ka whakakapia te nginx.conf ma te mahere whirihora. A ka kore e taea te whirihora i a Nginx ma te configmap i te wa e whakamahi ana i nga kaupapa here e aukati ana i te honohono o nga pukapuka (pvc) me te punaha konae pakiaka panui-anake (ka puta ano tenei).

Ma te whakamahi i te takawaenga NJS, ka whai waahi tatou ki te whakarereke i nga tawhā kua tohua i roto i te whirihora nginx ma te whakamahi i nga taurangi taiao me te mahi i etahi arowhai i roto i te tuhinga (hei tauira, he URL kua tohua).

nginx.conf

location = /auth-provider {
  internal;

  proxy_pass $auth_url;
}

location = /auth {
  internal;

  proxy_set_header Content-Length "";
  proxy_pass_request_body off;
  js_content auth.auth;
}

location ~ "^/(?<prefix>[w-]*)[/]?(?<postfix>[w-.]*)$" {
  auth_request /auth;

  js_content s3.request;
}

s3.js

var env = process.env;
var env_bool = new RegExp(/[Tt]rue|[Yy]es|[Oo]n|[TtYy]|1/);
var auth_disabled  = env_bool.test(env.DISABLE_AUTH);
var gitlab_url = env.AUTH_URL;

function url() {
  return `${gitlab_url}/jwt/auth?service=container_registry`
}

function auth(r) {
  if (auth_disabled) {
    r.return(202, '{"auth": "disabled"}');
    return null
  }

  r.subrequest('/auth-provider',
                {method: 'GET', body: ''},
                function(res) {
                  r.return(res.status, "");
                });
}

export default {auth, url}

Ko te mea pea kei te pupuhi te patai: -He aha te kore e whakamahi i nga waahanga kua oti te hanga? Kua oti kē nga mea katoa ki reira! Hei tauira, var AWS = me rapu('aws-sdk') karekau he take ki te tuhi i te “pahikara” me te motuhēhēnga S3!

Kia neke tatou ki te kino

Ki ahau, ko te kore e kaha ki te kawemai i nga waahanga JS o waho he ahua kino, engari e tumanakohia ana. I whakaahuahia i roto i te tauira i runga ake nei me rapu('crypto') he kōwae hanga-i roto me mahi anake ma ratou. Kaore ano he huarahi hei whakamahi ano i te waehere mai i nga tuhinga tuhi me kape me te whakapiri ki nga konae rereke. Te ti'aturi nei au ka whakatinanahia tenei mahi a tetahi ra.

Me monoa ano te whakakopeketanga mo te kaupapa o naianei kei Nginx gzip off;

Na te mea karekau he kōwae gzip i roto i te NJS karekau e taea te hono atu; no reira, karekau he huarahi ki te mahi me nga raraunga kōpeke. Pono, ehara tenei i te mea he iti mo tenei keehi. Kaore he maha o nga tuhinga, a ko nga konae kua whakawhitia kua oti kee te kopaki, a, kaore e nui te awhina i a raatau. Ano hoki, ehara tenei i te ratonga utaina, tino nui ranei e raru ai koe ki te tuku i nga ihirangi mo etahi mirihakona tere ake.

Ko te patuiro i te tuhinga he roa te wa, ka taea anake ma te "ta" kei error.log. I runga i nga korero taumata takiuru kua whakaritea, whakatupato, hapa ranei, ka taea te whakamahi i nga tikanga 3 r.log, r.warn, r.error ia. Ka ngana ahau ki te patuiro i etahi tuhinga i roto i te Chrome (v8) ranei te taputapu papatohu njs, engari kaore nga mea katoa e taea te tirotiro ki reira. I te wa e tarai ana te waehere, he whakamatautau mahi, he penei te ahua o te hitori:

docker-compose restart nginx
curl localhost:8080/
docker-compose logs --tail 10 nginx

a tera pea e hia rau o aua raupapa.

Ko te tuhi i te waehere ma te whakamahi i nga paatai ​​me nga taurangi mo ratou ka huri ki te kurupae. I etahi wa ka timata koe ki te huri i nga matapihi IDE rereke e ngana ana ki te whakaaro i te raupapa o nga mahi o to waehere. Ehara i te mea uaua, engari i etahi wa ka tino hoha.

Karekau he tino tautoko mo te ES6.

Tera pea etahi atu ngoikoretanga, engari kare ano au i pa ki tetahi atu mea. Tuarihia nga korero mena he wheako kino koe ki te whakamahi i te NJS.

mutunga

Ko te NJS he kaiwhakamaori tuwhera-puna marama e taea ai e koe te whakatinana i nga momo tuhinga JavaScript i Nginx. I te wa o tona whanaketanga, i aro nui ki te mahi. Ae ra, he nui tonu te ngaro, engari kei te whakawhanakehia te kaupapa e te roopu iti, a kei te kaha te taapiri i nga ahuatanga hou me te whakatika i nga pepeke. Te ti'aturi nei au ka taea e NJS tetahi ra ki te hono atu i nga waahanga o waho, ka kore e mutu te mahi a Nginx. Engari kei reira te NGINX Plus me te mea pea karekau he ahuatanga!

Pupuri me te waehere katoa mo te tuhinga

njs-pypi me te AWS Waitohu v4 tautoko

Whakaahuatanga o nga tohutohu o te ngx_http_js_module module

Putanga NJS mana и te tuhi

He tauira mo te whakamahi NJS mai i a Dmitry Volintsev

njs - tuhi JavaScript taketake i roto i te nginx / Korero na Dmitry Volnyev i Saint HighLoad++ 2019

NJS i roto i te hanga / Korero na Vasily Soshnikov i HighLoad++ 2019

Te Waitohu me te Motuhēhēnga REST Tono i AWS

Source: will.com