Ukufuduka kusuka ku-Nginx kuya kummeleli wenxusa

Sawubona, Habr! Ngikulethela ukuhunyushwa kokuthunyelwe: Ukufuduka kusuka ku-Nginx kuya kummeleli wenxusa.

I-Envoy iyiseva ebamba iqhaza esabalaliswe esebenza kahle kakhulu (ebhalwe ku-C++) eyenzelwe izinsiza ngazinye nezinhlelo zokusebenza, futhi iyibhasi lokuxhumana kanye "nendiza yedatha yendawo yonke" eyenzelwe izakhiwo ezinkulu ze-microservice "mesh yesevisi". Lapho uyidala, izixazululo zezinkinga ezavela ngesikhathi sokuthuthukiswa kwamaseva afana ne-NGINX, i-HAProxy, izilinganisi zomthwalo wehadiwe kanye nezilinganisi zomthwalo wamafu zacatshangelwa. Inxusa lisebenza eceleni kwesicelo ngasinye futhi lifushanisa inethiwekhi ukuze linikeze ukusebenza okufanayo kungakhathaliseki inkundla. Uma yonke ithrafikhi yesevisi kwingqalasizinda igeleza nge-Mesh Yezithunywa, kuba lula ukubona ngeso lengqondo izindawo zezinkinga ezinokubonakala okungaguquki, shuna ukusebenza kukonke, futhi wengeze ukusebenza okubalulekile endaweni ethile.

Izici

  • Isakhiwo esingaphandle kwenqubo: isithunywa siyiseva ezimele, esebenza kahle kakhulu ethatha inani elincane le-RAM. Isebenza ngokuhlanganyela nanoma yiluphi ulimi lohlelo lokusebenza noma uhlaka.
  • Usekelo lwe-http/2 kanye ne-grpc: umthunywa unosekelo lwesigaba sokuqala se-http/2 kanye ne-grpc yokuxhumana okungenayo nokuphumayo. Lona ummeleli obonakalayo osuka ku-http/1.1 ukuya ku-http/2.
  • Ukulinganisela Okuthuthukisiwe Komthwalo: isithunywa sisekela izici ezithuthukisiwe zokulinganisa umthwalo ohlanganisa ukuzama futhi okuzenzakalelayo, ukunqanyulwa kweketango, ukukhawulela isilinganiso somhlaba wonke, ukwenza ithunzi kwesicelo, ukulinganisela komthwalo wendawo, njll.
  • I-Configuration Management API: isithunywa sihlinzeka nge-API eqinile yokuphatha ukulungiselelwa kwakho.
  • Ukubonwa: Ukubonwa okujulile kwethrafikhi ye-L7, ukusekelwa komdabu kokulandela okusabalalisiwe nokubonwa kwe-mongodb, i-dynamodb nezinye izinhlelo zokusebenza eziningi.

Isinyathelo 1 β€” Isibonelo se-NGINX Config

Lesi script sisebenzisa ifayela elakhiwe ngokukhethekile nginx.conf, ngokusekelwe esibonelweni esigcwele esivela ku I-NGINX Wiki. Ungabuka ukucushwa kusihleli ngokuvula nginx.conf

nginx source config

user  www www;
pid /var/run/nginx.pid;
worker_processes  2;

events {
  worker_connections   2000;
}

http {
  gzip on;
  gzip_min_length  1100;
  gzip_buffers     4 8k;
  gzip_types       text/plain;

  log_format main      '$remote_addr - $remote_user [$time_local]  '
    '"$request" $status $bytes_sent '
    '"$http_referer" "$http_user_agent" '
    '"$gzip_ratio"';

  log_format download  '$remote_addr - $remote_user [$time_local]  '
    '"$request" $status $bytes_sent '
    '"$http_referer" "$http_user_agent" '
    '"$http_range" "$sent_http_content_range"';

  upstream targetCluster {
    172.18.0.3:80;
    172.18.0.4:80;
  }

  server {
    listen        8080;
    server_name   one.example.com  www.one.example.com;

    access_log   /var/log/nginx.access_log  main;
    error_log  /var/log/nginx.error_log  info;

    location / {
      proxy_pass         http://targetCluster/;
      proxy_redirect     off;

      proxy_set_header   Host             $host;
      proxy_set_header   X-Real-IP        $remote_addr;
    }
  }
}

Ukulungiselelwa kwe-NGINX kuvame ukuba nezinto ezintathu ezibalulekile:

  1. Ilungiselela iseva ye-NGINX, ukwakheka kwelogi nokusebenza kwe-Gzip. Lokhu kuchazwa emhlabeni wonke kuzo zonke izimo.
  2. Ilungiselela i-NGINX ukuze yamukele izicelo kumsingathi eyodwa.example.com ethekwini 8080.
  3. Ukusetha indawo eqondiwe, indlela yokusingatha ithrafikhi yezingxenye ezihlukene ze-URL.

Akuwona wonke ukulungiselelwa okuzosebenza Kummeleli Wezithunywa, futhi awudingi ukulungisa ezinye izilungiselelo. Ummeleli Wezithunywa unakho izinhlobo ezine ezibalulekile, esekela ingqalasizinda eyinhloko ehlinzekwa yi-NGINX. Umongo ngu:

  • Abalaleli: Banquma ukuthi Ummeleli Wezithunywa uzamukela kanjani izicelo ezingenayo. Ummeleli Wezithunywa okwamanje usekela kuphela abalaleli abasekelwe ku-TCP. Uma uxhumano selumisiwe, ludluliselwa kusethi yezihlungi ukuze zicutshungulwe.
  • Izihlungi: Ziyingxenye yezakhiwo zamapayipi ezingacubungula idatha engenayo nephumayo. Lokhu kusebenza kufaka phakathi izihlungi ezifana ne-Gzip, ecindezela idatha ngaphambi kokuyithumela kuklayenti.
  • Amarutha: Badlulisela ithrafikhi endaweni edingekayo, echazwa njengeqoqo.
  • Amaqoqo: Bachaza indawo yokugcina yethrafikhi namapharamitha wokumisa.

Sizosebenzisa lezi zingxenye ezine ukuze sidale ukucushwa kommeleli wezithunywa ukuze kufane nokucushwa okuthile kwe-NGINX. Umgomo we-Essay ukusebenza nama-API kanye nokucushwa okuguquguqukayo. Kulesi simo, ukucushwa kwesisekelo kuzosebenzisa izilungiselelo ezimile, ezinekhodi eqinile kusuka ku-NGINX.

Isinyathelo sesi-2 - Ukucushwa kwe-NGINX

Ingxenye yokuqala nginx.conf ichaza abanye abangaphakathi be-NGINX okudingeka balungiswe.

Ukuxhumana Kwabasebenzi

Ukucushwa okungezansi kunquma inani lezinqubo zabasebenzi nokuxhumana. Lokhu kukhombisa ukuthi i-NGINX izokala kanjani ukuze ihlangabezane nesidingo.

worker_processes  2;

events {
  worker_connections   2000;
}

Ummeleli Wezithunywa ulawula ukuhamba komsebenzi nokuxhumana ngezindlela ezihlukene.

Umthunywa udala intambo yesisebenzi yochungechunge ngalunye lwehadiwe ohlelweni. Uchungechunge ngalunye lwesisebenzi lwenza iluphu yomcimbi engavimbeli enesibopho sayo

  1. Ukulalela umlaleli ngamunye
  2. Ukwamukela ukuxhumana okusha
  3. Idala isethi yezihlungi zoxhumo
  4. Cubungula yonke imisebenzi ye-I/O ngesikhathi sokuphila kokuxhumeka.

Konke ukucubungula okwengeziwe kokuxhumana kuphathwa ngokuphelele kuchungechunge lwesisebenzi, okuhlanganisa noma yikuphi ukuziphatha kokudlulisela phambili.

Ochungechungeni ngalunye lwesisebenzi ku-Envoy, kukhona indawo yokuxhumana. Ngakho-ke amachibi okuxhumana e-HTTP/2 asungula kuphela ukuxhumana okukodwa ngosokhaya ngamunye wangaphandle ngesikhathi, uma kunezintambo ezine zezisebenzi kuzoba nezixhumi ezine ze-HTTP/2 ngosokhaya ngamunye wangaphandle esimeni esizinzile. Ngokugcina yonke into kuyintambo eyodwa yesisebenzi, cishe yonke ikhodi ingabhalwa ngaphandle kokuvinjwa, njengokungathi ifakwe intambo eyodwa. Uma kwabiwa izintambo eziningi kunesidingo, lokhu kungaholela ekumoshweni kwenkumbulo, kudale inombolo enkulu yoxhumo olungenzi lutho, futhi kunciphise inani lezikhathi lapho ukuxhumeka kubuyiselwa khona echibini.

Ukuze uthole ulwazi olwengeziwe vakashela Ibhulogi yommeleli wenxusa.

Ukucushwa kwe-HTTP

Ibhulokhi yokucushwa elandelayo ye-NGINX ichaza izilungiselelo ze-HTTP ezifana:

  • Yiziphi izinhlobo ze-mime ezisekelwayo
  • Ukuphela kwesikhathi okuzenzakalelayo
  • Ukucushwa kwe-Gzip

Ungenza lezi zici ngendlela oyifisayo usebenzisa izihlungi Kummeleli Wezithunywa, esizoxoxa ngakho kamuva.

Isinyathelo sesi-3 - Ukucushwa Kweseva

Kubhulokhi yokucushwa kwe-HTTP, ukucushwa kwe-NGINX kucacisa ukulalela ku-port 8080 futhi uphendule izicelo ezingenayo zezizinda. eyodwa.example.com ΠΈ www.one.example.com.

 server {
    listen        8080;
    server_name   one.example.com  www.one.example.com;

Ngaphakathi kwe-Envoy, ilawulwa ngabalaleli.

Abalaleli bezithunywa

Isici esibaluleke kakhulu sokuqalisa nge-Envoy Proxy ukuchaza abalaleli bakho. Udinga ukudala ifayela lokumisa elichaza ukuthi ufuna ukwenza kanjani isibonelo se-Evoy.

Amazwibela angezansi azodala isilaleli esisha futhi asibophe ku-port 8080. Ukulungiselelwa kutshela Ummeleli Wezithunywa ukuthi yiziphi izimbobo okufanele ibophe kuzo ngezicelo ezingenayo.

Ummeleli Wezithunywa usebenzisa inothi ye-YAML ekuyicupheni. Ukuze uthole isingeniso salesi sihloko, bheka lapha isixhumanisi.

Copy to Editorstatic_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }

Asikho isidingo sokuchaza igama le-server, njengoba izihlungi Zommeleli Wezithunywa zizophatha lokhu.

Isinyathelo sesi-4 - Ukucushwa Kwendawo

Uma isicelo singena ku-NGINX, ibhulokhi yendawo inquma ukuthi izocutshungulwa kanjani nokuthi izohanjiswa kuphi ithrafikhi. Esiqeshini esilandelayo, yonke i-traffic eya kusayithi idluliselwa phezulu (inothi lomhumushi: umfudlana okhuphuka ngokuvamile iseva yohlelo lokusebenza) iqoqo eliqanjwe igama. targetCluster. Iqoqo elikhuphukayo lichaza amanodi okufanele acubungule isicelo. Sizoxoxa ngalokhu esinyathelweni esilandelayo.

location / {
    proxy_pass         http://targetCluster/;
    proxy_redirect     off;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
}

Ku-Envoy, Izihlungi zenza lokhu.

Izihlungi Zezithunywa

Ngokucushwa okumile, izihlungi zinquma ukuthi zicutshungulwa kanjani izicelo ezingenayo. Kulokhu sibeka izihlungi ezifanayo amagama_eseva esinyathelweni sangaphambilini. Uma izicelo ezingenayo zifika ezifana nezizinda ezithile nemizila, ithrafikhi ihanjiswa ku-cluster. Lokhu kulingana nokucushwa kwe-NGINX phansi kuya phezulu.

Copy to Editor    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
                - "one.example.com"
                - "www.one.example.com"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: targetCluster
          http_filters:
          - name: envoy.router

Имя isithunywa.http_connection_manager yisihlungi esakhelwe ngaphakathi Kummeleli Wezithunywa. Ezinye izihlungi zihlanganisa Redis, Mongo, I-TCP. Ungathola uhlu oluphelele ku imibhalo.

Ukuze uthole ulwazi olwengeziwe mayelana nezinye izinqubomgomo zokulinganisa umthwalo, vakashela Umbhalo Wezithunywa.

Isinyathelo sesi-5 - Ukumiswa kommeleli nokukhuphuka komfula

Ku-NGINX, ukucushwa komfula kuchaza isethi yamaseva aqondiwe azocubungula ithrafikhi. Kulokhu, kwabelwa amaqoqo amabili.

  upstream targetCluster {
    172.18.0.3:80;
    172.18.0.4:80;
  }

Ku-Envoy, lokhu kulawulwa amaqoqo.

Amaqoqo Ezithunywa

Okulingana nomfula kuchazwa ngokuthi amaqoqo. Kulokhu, ababungazi abazonikeza ithrafikhi bahlonziwe. Indlela abasingathi abafinyelelwa ngayo, njengokuvala isikhathi, kuchazwa njengokucushwa kweqoqo. Lokhu kuvumela ukulawula okuyimbudumbudu okwengeziwe phezu kwezici ezifana nokubambezeleka nokulinganisa komthwalo.

Copy to Editor  clusters:
  - name: targetCluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts: [
      { socket_address: { address: 172.18.0.3, port_value: 80 }},
      { socket_address: { address: 172.18.0.4, port_value: 80 }}
    ]

Uma usebenzisa ukutholwa kwesevisi STRICT_DNS Inxusa lizoqhubeka futhi ngokuqhubekayo lixazulule izinhloso ze-DNS ezishiwo. Ikheli ngalinye le-IP elibuyisiwe elivela kumphumela we-DNS lizobhekwa njengomsingathi osobala kuqoqo elikhuphuka nomfula. Lokhu kusho ukuthi uma isicelo sibuyisela amakheli amabili e-IP, isithunywa sizothatha ngokuthi kukhona abasingathi ababili kuqoqo, futhi bobabili kufanele balayishe ngokulinganayo. Uma umsingathi esuswa kumphumela, isithunywa sizothatha ngokuthi akasekho futhi sizodonsa ithrafikhi kunoma yimaphi amachibi okuxhumana akhona.

Ngeminye imininingwane bheka Amadokhumenti ommeleli wenxusa.

Isinyathelo sesi-6 - Ukufinyelela Kwelogi namaphutha

Ukucushwa kokugcina ukubhalisa. Esikhundleni sokusunduza amalogi wephutha kudiski, Ummeleli Wezithunywa uthatha indlela esekelwe emafini. Wonke amalogi ohlelo lokusebenza akhishelwa kuwo stdout ΠΈ stderr.

Uma abasebenzisi benza isicelo, amalogi okufinyelela ayakhethwa futhi akhutshazwe ngokuzenzakalela. Ukuze unike amandla amalogi okufinyelela ezicelo ze-HTTP, nika amandla ukulungiselelwa ukufinyelela_log kumphathi woxhumano lwe-HTTP. Indlela kungaba noma idivayisi ezifana stdout, noma ifayela elikudiski, kuye ngezimfuneko zakho.

Ukulungiselelwa okulandelayo kuzoqondisa kabusha wonke amalogi wokufinyelela kuwo stdout (inothi lomhumushi - i-stdout iyadingeka ukuze usebenzise isithunywa esingaphakathi kwedokha. Uma sisetshenziswa ngaphandle kwedokha, shintsha /dev/stdout ngendlela eya kufayela lokungena elivamile). Kopisha amazwibela engxenyeni yokumisa yesiphathi sokuxhuma:

Copy to Clipboardaccess_log:
- name: envoy.file_access_log
  config:
    path: "/dev/stdout"

Imiphumela kufanele ibukeke kanje:

      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          route_config:

Ngokuzenzakalelayo, i-Envoy inochungechunge lwefomethi ehlanganisa imininingwane yesicelo se-HTTP:

[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"n

Umphumela walolu chungechunge lwefomethi uthi:

[2018-11-23T04:51:00.281Z] "GET / HTTP/1.1" 200 - 0 58 4 1 "-" "curl/7.47.0" "f21ebd42-6770-4aa5-88d4-e56118165a7d" "one.example.com" "172.18.0.4:80"

Okuqukethwe okukhiphayo kungenziwa ngendlela oyifisayo ngokusetha inkambu yefomethi. Ngokwesibonelo:

access_log:
- name: envoy.file_access_log
  config:
    path: "/dev/stdout"
    format: "[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"n"

Ulayini welogi ungase futhi ukhishwe ngefomethi ye-JSON ngokusetha inkambu json_format. Isibonelo:

access_log:
- name: envoy.file_access_log
  config:
    path: "/dev/stdout"
    json_format: {"protocol": "%PROTOCOL%", "duration": "%DURATION%", "request_method": "%REQ(:METHOD)%"}

Ukuze uthole ulwazi olwengeziwe mayelana Nendlela Yokubhaliswa Kwezithunywa, vakashela

https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#config-access-log-format-dictionaries

Ukungena ngemvume akuyona ukuphela kwendlela yokuthola ukuqonda ekusebenzeni noMmeleli Wezithunywa. Inamandla okulandela umkhondo namamethrikhi akhelwe kuwo. Ungathola okwengeziwe ku ukulandelela imibhalo noma nge Iskripthi sokulandelela esisebenzisanayo.

Isinyathelo 7 - Qalisa

Manje uthuthe ukulungisa kwakho kusuka ku-NGINX kuya kummeleli Wezithunywa. Isinyathelo sokugcina ukwethula isibonelo sommeleli wezithunywa ukuze usihlole.

Qalisa njengomsebenzisi

Phezulu komugqa wokucushwa we-NGINX umsebenzisi www; icacisa ukuqalisa i-NGINX njengomsebenzisi ongenamalungelo aphansi ukuze kuthuthukiswe ukuphepha.

Ummeleli Wezithunywa uthatha indlela esekelwe emafini yokuphatha ukuthi ngubani umnikazi wenqubo. Uma sisebenzisa Ummeleli Wezithunywa ngesiqukathi, singacacisa umsebenzisi onelungelo eliphansi.

Kwethulwa Ummeleli Wezithunywa

Umyalo ongezansi uzosebenzisa Ummeleli Wezithunywa ngokusebenzisa isitsha se-Docker kumsingathi. Lo myalo unikeza isithunywa ikhono lokulalela izicelo ezingenayo kumbobo 80. Nokho, njengoba kucacisiwe ekucushweni komlaleli, Ummeleli Wezithunywa ulalela ithrafikhi engenayo ku-port 8080. Lokhu kuvumela inqubo ukuthi isebenze njengomsebenzisi ongenamalungelo aphansi.

docker run --name proxy1 -p 80:8080 --user 1000:1000 -v /root/envoy.yaml:/etc/envoy/envoy.yaml envoyproxy/envoy

Ukuhlola

Ngokusebenza kommeleli, ukuhlola manje kungenziwa futhi kusetshenzwe. Umyalo olandelayo we-cURL ukhipha isicelo ngesihloko sosokhaya esichazwe ekucushweni kommeleli.

curl -H "Host: one.example.com" localhost -i

Isicelo se-HTTP sizoholela ephutha 503. Lokhu kungenxa yokuthi uxhumo oluphezulu nomfula alusebenzi futhi alutholakali. Ngakho-ke, Ummeleli Wezithunywa awunazo izindawo ezitholakalayo zesicelo. Umyalo olandelayo uzoqala uchungechunge lwezinsizakalo ze-HTTP ezifana nokucushwa okuchazwe ku-Envoy.

docker run -d katacoda/docker-http-server; docker run -d katacoda/docker-http-server;

Ngamasevisi atholakalayo, Inxusa lingabamba ngempumelelo ithrafikhi lapho liya khona.

curl -H "Host: one.example.com" localhost -i

Kufanele ubone impendulo ekhombisa ukuthi yisiphi isiqukathi se-Docker esicubungule isicelo. Kumalogu Ommeleli Wezithunywa kufanele futhi ubone ukuphuma kwentambo yokufinyelela.

Izihloko Ezingeziwe Zempendulo ye-HTTP

Uzobona izihloko ezengeziwe ze-HTTP kuzihloko zempendulo zesicelo sangempela. Isihloko sibonisa isikhathi umsingathi okhuphuka nomfula asichithe ecubungula isicelo. Kuvezwe ngama-millisecond. Lokhu kuyasiza uma iklayenti lifuna ukunquma isikhathi sesevisi uma siqhathaniswa nokubambezeleka kwenethiwekhi.

x-envoy-upstream-service-time: 0
server: envoy

Ukulungiselelwa kokugcina

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
                - "one.example.com"
                - "www.one.example.com"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: targetCluster
          http_filters:
          - name: envoy.router
          clusters:
  - name: targetCluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts: [
      { socket_address: { address: 172.18.0.3, port_value: 80 }},
      { socket_address: { address: 172.18.0.4, port_value: 80 }}
    ]

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9090 }

Ulwazi olwengeziwe oluvela kumhumushi

Imiyalo yokufaka Ummeleli Wezithunywa ingatholakala kuwebhusayithi https://www.getenvoy.io/

Ngokuzenzakalelayo, i-rpm ayinakho ukulungiselelwa kwesevisi ye-systemd.

Engeza i-systemd service config /etc/systemd/system/envoy.service:

[Unit]
Description=Envoy Proxy
Documentation=https://www.envoyproxy.io/
After=network-online.target
Requires=envoy-auth-server.service
Wants=nginx.service

[Service]
User=root
Restart=on-failure
ExecStart=/usr/bin/envoy --config-path /etc/envoy/config.yaml
[Install]
WantedBy=multi-user.target

Udinga ukwakha uhla lwemibhalo /etc/envoy/ futhi ubeke config.yaml config lapho.

Kunengxoxo yocingo esebenzisa ummeleli wenxusa: https://t.me/envoyproxy_ru

Ummeleli Wezithunywa akakusekeli ukunikeza okuqukethwe okumile. Ngakho-ke, ubani ongavotela lesi sici: https://github.com/envoyproxy/envoy/issues/378

Abasebenzisi ababhalisiwe kuphela abangabamba iqhaza kuhlolovo. Ngena ngemvume, wamukelekile.

Ingabe lokhu okuthunyelwe kukukhuthaze ukuthi ufake futhi uhlole ummeleli wesithunywa?

  • yebo

  • akukho

Bangu-75 abasebenzisi abavotile. Abasebenzisi abangu-18 bagobile.

Source: www.habr.com

Engeza amazwana