ProHoster > Ibhulogi > Ulawulo > Ukufuduka ukusuka kwi-Nginx ukuya kwi-Proxy yomthunywa

Ukufuduka ukusuka kwi-Nginx ukuya kwi-Proxy yomthunywa

Molo, Habr! Ndizisa kwingqwalasela yakho inguqulelo yesi sithuba: Ukufuduka ukusuka kwi-Nginx ukuya kwi-Proxy yomthunywa.

Umthunywa ngumsebenzi ophezulu we-proxy server (ebhalwe kwi-C ++) eyenzelwe iinkonzo zomntu ngamnye kunye nezicelo, kwakhona ibhasi yonxibelelwano kunye "nendiza yedatha yendalo yonke" eyenzelwe i-microservice enkulu "inkonzo ye-mesh" izakhiwo. Xa uyidala, izisombululo kwiingxaki eziye zavela ngexesha lokuphuhliswa kweeseva ezifana ne-NGINX, i-HAProxy, i-hardware yokulinganisa i-hardware kunye ne-cloud load balancers zathathelwa ingqalelo. Umthunywa usebenza ecaleni kwesicelo ngasinye kwaye uthatha inethiwekhi ukubonelela ngokusebenza okuqhelekileyo kungakhathaliseki iqonga. Xa yonke i-traffic yenkonzo kwi-infrastructure ihamba nge-Mesh ye-Envoy, kuba lula ukujonga iindawo zengxaki kunye nokubonwa okungaguqukiyo, ukucula ukusebenza ngokubanzi, kunye nokongeza ukusebenza okungundoqo kwindawo ethile.

Izixhobo

  • Ulwakhiwo olungaphandle kwenkqubo: umthunywa ngumncedisi ozimeleyo, osebenza kakhulu othatha i-RAM encinci. Isebenza ngokubambisana naluphi na ulwimi lwesicelo okanye isakhelo.
  • I-http/2 kunye nenkxaso ye-grpc: umthunywa uneklasi yokuqala ye-http/2 kunye nenkxaso ye-grpc yoqhagamshelwano olungenayo noluphumayo. Lo ngummeli ocacileyo ukusuka ku-http/1.1 ukuya ku-http/2.
  • Ulungelelwaniso oluPhambili loMthwalo: umthunywa uxhasa iimpawu zolungelelaniso lomthwalo okwinqanaba eliphambili kubandakanya ukuzama kwakhona okuzenzekelayo, ukwaphulwa kwekhonkco, ukunciphisa izinga lehlabathi, ukucela isithunzi, ulungelelwaniso lomthwalo wendawo, njl.
  • Uqwalaselo loLawulo lwe-API: umthunywa ubonelela ngeAPI eyomeleleyo yokulawula uqwalaselo lwakho.
  • Ukuqwalaselwa: Ukujongwa okunzulu kwetrafikhi ye-L7, inkxaso yemveli yokuhanjiswa kunye nokubonwa kwe-mongodb, i-dynamodb kunye nezinye izicelo ezininzi.

Inyathelo 1 β€” Umzekelo NGINX Config

Le script isebenzisa ifayile eyenziwe ngokukodwa nginx.confngokusekelwe kumzekelo opheleleyo ovela NGINX Wiki. Ungajonga uqwalaselo kumhleli ngokuvula nginx.conf

nginx umthombo woqwalaselo

user  www www;
pid /var/run/nginx.pid;
worker_processes  2;

events {
  worker_connections   2000;
}

http {
  gzip on;
  gzip_min_length  1100;
  gzip_buffers     4 8k;
  gzip_types       text/plain;

  log_format main      '$remote_addr - $remote_user [$time_local]  '
    '"$request" $status $bytes_sent '
    '"$http_referer" "$http_user_agent" '
    '"$gzip_ratio"';

  log_format download  '$remote_addr - $remote_user [$time_local]  '
    '"$request" $status $bytes_sent '
    '"$http_referer" "$http_user_agent" '
    '"$http_range" "$sent_http_content_range"';

  upstream targetCluster {
    172.18.0.3:80;
    172.18.0.4:80;
  }

  server {
    listen        8080;
    server_name   one.example.com  www.one.example.com;

    access_log   /var/log/nginx.access_log  main;
    error_log  /var/log/nginx.error_log  info;

    location / {
      proxy_pass         http://targetCluster/;
      proxy_redirect     off;

      proxy_set_header   Host             $host;
      proxy_set_header   X-Real-IP        $remote_addr;
    }
  }
}

Ulungelelwaniso lwe-NGINX lunezinto ezintathu eziphambili:

  1. Ukuqwalasela iseva ye-NGINX, isakhiwo selogi kunye nokusebenza kweGzip. Oku kuchazwa kwihlabathi kuzo zonke iimeko.
  2. Ukuqwalasela i-NGINX ukwamkela izicelo kumamkeli enye.example.com kwizibuko 8080.
  3. Ukumisela indawo ekujoliswe kuyo, indlela yokusingatha i-traffic kwiindawo ezahlukeneyo ze-URL.

Ayilulo lonke uqwalaselo oluya kusebenza kuMmeli womthunywa, kwaye awudingi kuqwalasela ezinye iisetingi. Ummeli womthunywa uye iintlobo ezine eziphambili, ezixhasa iziseko ezingundoqo ezinikezelwa yi-NGINX. Undoqo ngu:

  • Abaphulaphuli: Bamisela indlela uMmeli woMthunywa azamkela ngayo izicelo ezingenayo. Ummeli womthunywa okwangoku uxhasa kuphela abaphulaphuli abasekwe kwi-TCP. Nje ukuba uxhulumaniso lusekiwe, lugqithiselwa kwiseti yezihluzi ukuze ziqwalaselwe.
  • Izihluzi: Ziyinxalenye yoyilo lwemibhobho ekwazi ukucubungula idatha engenayo nephumayo. Lo msebenzi uquka izihluzi ezifana neGzip, ecinezela idatha phambi kokuba iyithumele kumxhasi.
  • Iindlela: Bahambisa itrafikhi kwindawo efunekayo, echazwa njengeqela.
  • Amaqela: Bachaza indawo yokuphela kwetrafikhi kunye neeparamitha zoqwalaselo.

Siza kusebenzisa ezi zixhobo ezine ukwenza ulungelelwaniso lwe-Proxy ye-Envoy ukuze lihambelane noqwalaselo oluthile lwe-NGINX. Injongo yoMthunywa kukusebenza ngee-APIs kunye noqwalaselo oluguquguqukayo. Kule meko, ukucwangciswa kwesiseko kuya kusebenzisa i-static, i-hard-coded izicwangciso ezivela kwi-NGINX.

Inyathelo 2 - NGINX Configuration

Inxalenye yokuqala nginx.conf ichaza abanye abangaphakathi be-NGINX abafuna ukuqwalaselwa.

Uqhagamshelwano lwabasebenzi

Ubume obungezantsi bumisela inani leenkqubo zabasebenzi kunye noqhagamshelo. Oku kubonisa indlela i-NGINX eya kukala ngayo ukuhlangabezana neemfuno.

worker_processes  2;

events {
  worker_connections   2000;
}

Ummeli womthunywa ulawula ukuhamba komsebenzi kunye noqhagamshelwano ngeendlela ezahlukeneyo.

Umthunywa wenza intambo yomsebenzi kumsonto ngamnye wehardware kwisixokelelwano. Umsonto ngamnye wabasebenzi wenza i-loop yesiganeko esingathinteliyo enoxanduva

  1. Ukumamela umphulaphuli ngamnye
  2. Ukwamkela uqhagamshelwano olutsha
  3. Ukwenza iseti yezihluzi zoqhagamshelwano
  4. Yenza yonke imisebenzi ye-I/O ngexesha lokuphila koqhagamshelwano.

Lonke ukuqhubekeka koqhagamshelo olongezelelweyo luphathwa ngokupheleleyo kumsonto wabasebenzi, kuquka nayiphi na indlela yokuziphatha yokuthumela.

Kumsebenzi ngamnye umsonto kwi-Envoy, kukho i-pool yokudibanisa. Ngoko ke i-HTTP/2 iiphuli zokudibanisa ziseka kuphela uxhulumaniso olunye kwinginginya yangaphandle ngexesha, ukuba kukho imisonto emine yabasebenzi kuya kubakho uxhulumaniso lwe-HTTP / 2 kwi-host host yangaphandle kwisimo esizinzileyo. Ngokugcina yonke into kumsonto omnye wabasebenzi, phantse yonke ikhowudi inokubhalwa ngaphandle kokubhloka, ngokungathi inentambo enye. Ukuba iintambo ezininzi zabasebenzi zabiwe kunokuba ziyimfuneko, oku kunokukhokelela kwimemori elahlekileyo, ukudala inani elikhulu loqhagamshelo olungasebenziyo, kunye nokunciphisa inani lamaxesha okudityaniswa kubuyiswa kwakhona echibini.

Ngolwazi oluthe vetshe ndwendwela Ibhlog yeProxy yomthunywa.

Uqwalaselo lweHTTP

Ibhloko yoqwalaselo elandelayo ye-NGINX ichaza useto lweHTTP olunje:

  • Zeziphi iintlobo ze-mime ezixhaswayo
  • Ixesha elimiselweyo
  • Uqwalaselo lweGzip

Unokwenza le miba ngokusesikweni usebenzisa izihluzi kwi-Proxy yomthunywa, esiza kuyixoxa kamva.

Inyathelo lesi-3 - Ubumbeko lweseva

Kwibhloko yoqwalaselo lwe-HTTP, uqwalaselo lwe-NGINX lucacisa ukuphulaphula kwi-port 8080 kwaye uphendule izicelo ezingenayo zemimandla. enye.example.com ΠΈ www.one.example.com.

 server {
    listen        8080;
    server_name   one.example.com  www.one.example.com;

Ngaphakathi uMthunywa, ilawulwa ngabaphulaphuli.

Umthunywa wabaphulaphuli

Owona mba ubalulekileyo wokuqalisa nge-Proxy yomthunywa kukuchaza abaphulaphuli bakho. Kufuneka wenze ifayile yoqwalaselo echaza indlela ofuna ukuwuqhuba ngayo uMthunywa.

I-snippet engezantsi iyakwenza umphulaphuli omtsha kwaye ayibophe kwi-port 8080. Ubumbeko luxelela uMmeli woMmeli weliphi izibuko ekufuneka libophelele kwizicelo ezingenayo.

Ummeli womthunywa usebenzisa ubhalo lwe-YAML kuqwalaselo lwayo. Ngentshayelelo yolu phawu, jonga apha unxibelelwano.

Copy to Editorstatic_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }

Akukho mfuneko yokuchaza igama lomncedisi, kuba izihluzo zoMmeli womthunywa ziyakuyiphatha le nto.

Inyathelo lesi-4 - Uqwalaselo lweNdawo

Xa isicelo sifika kwi-NGINX, ibhloko yendawo inquma indlela yokuqhuba kunye nalapho uhambisa i-traffic. Kwesi siqwenga silandelayo, zonke iitrafikhi kwisiza zikhutshelwa phezulu (inqaku lomguquleli: umlambo udla ngokuba ngumncedisi wesicelo) iqela elibizwa ngokuba Iqela ekujoliswe kulo. Iqela elinyukayo lichaza iindawo ekufuneka ziqhubekisele phambili isicelo. Siza kuxubusha oku kwinyathelo elilandelayo.

location / {
    proxy_pass         http://targetCluster/;
    proxy_redirect     off;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
}

KuMthunywa, izihluzi zenza oku.

Izihluzo zoMthunywa

Kuqwalaselo olungatshintshiyo, izihluzi zimisela indlela yokuqhuba izicelo ezingenayo. Kule meko sibeka izihluzi ezihambelanayo Amagama_eseva kwinyathelo langaphambili. Xa izicelo ezingenayo zifika ezihambelana nemimandla ethile kunye neendlela, i-traffic ihanjiswa kwi-cluster. Oku kukulingana ne-NGINX ezantsi-up uqwalaselo.

Copy to Editor    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
                - "one.example.com"
                - "www.one.example.com"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: targetCluster
          http_filters:
          - name: envoy.router

Igama umthunywa.http_unxibelelwano_umphathi sisihluzo esakhelwe-ngaphakathi kwi-Proxy yomthunywa. Ezinye izihluzi ziquka Redis, IMongo, TCP. Ungalufumana uluhlu olupheleleyo apha amaxwebhu.

Ngolwazi oluthe vetshe malunga neminye imigaqo-nkqubo yokulinganisa umthwalo, tyelela Uxwebhu lomthunywa.

Inyathelo lesi-5 - Ummeli kunye noBumbeko oluPhezulu

Kwi-NGINX, ulungelelwaniso oluphezulu luchaza isethi yeeseva ezijoliswe kuyo eziza kuqhuba i-traffic. Kule meko, kwabelwa amaqela amabini.

  upstream targetCluster {
    172.18.0.3:80;
    172.18.0.4:80;
  }

KuMthunywa, oku kulawulwa ngamaqela.

Amaqela oMthunywa

Umsinga ongasentla uchazwa njengamaqela. Kule meko, imikhosi eya kukhonza i-traffic ichongiwe. Indlela ababuthi abafikelelwa ngayo, njengokuphuma kwexesha, ichazwa njengobumbeko lweqela. Oku kuvumela ulawulo lwegranular ngakumbi kwimiba efana ne-latency kunye nokulinganisa umthwalo.

Copy to Editor  clusters:
  - name: targetCluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts: [
      { socket_address: { address: 172.18.0.3, port_value: 80 }},
      { socket_address: { address: 172.18.0.4, port_value: 80 }}
    ]

Xa usebenzisa ukufunyanwa kwenkonzo STRICT_DNS Umthunywa uya kuqhuba kwaye ngokuqhubekayo asombulule iithagethi zeDNS ezichaziweyo. Idilesi nganye ye-IP ebuyisiweyo ukusuka kwisiphumo se-DNS iya kuthathwa njengenginginya ecacileyo kwiqela elinyukayo. Oku kuthetha ukuba ukuba isicelo sibuyisela iidilesi ezimbini ze-IP, uMthunywa uya kucinga ukuba kukho iinginginya ezimbini kwiqela, kwaye zombini kufuneka zilayishe ibhalansi. Ukuba umamkeli ususiwe kwisiphumo, uMthunywa uya kucinga ukuba akasekho kwaye uya kutsala i-traffic kuwo nawaphi na amachibi oqhagamshelwano akhoyo.

Ukuze ufumane inkcazelo engakumbi jonga Amaxwebhu ommeli womthunywa.

Inyathelo lesi-6 - UFikelelo lwelogi kunye neempazamo

Ubumbeko lokugqibela lubhaliso. Endaweni yokutyhala iilogi zempazamo kwidiski, uMmeli womthunywa uthatha indlela esekwe kwilifu. Zonke iilog zesicelo ziphuma kuyo Yima ΠΈ kwiba.

Xa abasebenzisi benza isicelo, iilogi zofikelelo zinokuzikhethela kwaye zikhubaziwe ngokungagqibekanga. Ukuvumela iilogi zofikelelo kwizicelo zeHTTP, yenza ulungelelwaniso access_log yomphathi woqhagamshelwano lweHTTP. Umendo unokuba nokuba isixhobo esinje Yima, okanye ifayile ekwidiski, kuxhomekeke kwiimfuno zakho.

Olu qwalaselo lulandelayo luyakwalathisa zonke iilog zofikelelo Yima (inqaku lomguquli-i-stdout iyafuneka ukusebenzisa umthunywa ngaphakathi kwidokhi. Ukuba isetyenziswe ngaphandle kwedokhi, buyisela i-/dev/stdout ngendlela eya kwifayile yelog eqhelekileyo). Khuphela i-snippet kwicandelo loqwalaselo lomphathi woqhagamshelwano:

Copy to Clipboardaccess_log:
- name: envoy.file_access_log
  config:
    path: "/dev/stdout"

Iziphumo kufuneka zibukeke ngolu hlobo:

      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          route_config:

Ngokungagqibekanga, uMthunywa unomtya wefomathi oquka iinkcukacha zesicelo seHTTP:

[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"n

Isiphumo solu luhlu lwamagama sithi:

[2018-11-23T04:51:00.281Z] "GET / HTTP/1.1" 200 - 0 58 4 1 "-" "curl/7.47.0" "f21ebd42-6770-4aa5-88d4-e56118165a7d" "one.example.com" "172.18.0.4:80"

Umxholo wemveliso unokwenziwa ngokwezifiso ngokucwangcisa indawo yefomati. Umzekelo:

access_log:
- name: envoy.file_access_log
  config:
    path: "/dev/stdout"
    format: "[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"n"

Umgca welog nawo unokukhutshwa kwifomathi ye-JSON ngokucwangcisa indawo json_format. Umzekelo:

access_log:
- name: envoy.file_access_log
  config:
    path: "/dev/stdout"
    json_format: {"protocol": "%PROTOCOL%", "duration": "%DURATION%", "request_method": "%REQ(:METHOD)%"}

Ngolwazi oluthe kratya malunga neNdlela yoBhaliso loMthunywa, tyelela

https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#config-access-log-format-dictionaries

Ukuloga ayisiyiyo kuphela indlela yokufumana ukuqonda ekusebenzeni noMmeli woMthunywa. Inomkhondo ophezulu wokulandela kunye neemetrics ezakhelwe kuyo. Ungafumana ngakumbi kwi umkhondo wamaxwebhu okanye Isikripthi sokulandela umkhondo.

Inyathelo 7 - Qalisa

Ngoku ufuduse ulungelelwaniso lwakho ukusuka kwi-NGINX ukuya kuMmeli womthunywa. Inyathelo lokugqibela kukuphehlelela umzekelo woMmeli woMthunywa ukuyivavanya.

Baleka njengomsebenzisi

Phezulu kumgca woqwalaselo lwe-NGINX umsebenzisi www; icacisa ukuqhuba i-NGINX njengomsebenzisi onelungelo eliphantsi lokuphucula ukhuseleko.

Ummeli womthunywa uthatha indlela esekwe kwilifu ekulawuleni ukuba ngubani ophethe inkqubo. Xa siqhuba uMmeli womthunywa kwisikhongozeli, sinokukhankanya umsebenzisi onamalungelo aphantsi.

Ukuphehlelelwa koMmeli woMmeli

Umyalelo ongezantsi uya kuqhuba uMmeli woMthunywa ngesikhongozeli seDocker kumamkeli. Lo myalelo unika uMthunywa amandla okuphulaphula izicelo ezingenayo kwi-port 80. Nangona kunjalo, njengoko kuchaziwe kuqwalaselo lomphulaphuli, uMmeli womthunywa uphulaphule i-traffic engenayo kwi-port 8080. Oku kuvumela inkqubo ukuba iqhube njengomsebenzisi ophantsi.

docker run --name proxy1 -p 80:8080 --user 1000:1000 -v /root/envoy.yaml:/etc/envoy/envoy.yaml envoyproxy/envoy

Ukuvavanywa

Ngokusebenza kwe-proxy, iimvavanyo ngoku zingenziwa kwaye ziqwalaselwe. Lo myalelo ulandelayo we-cURL ukhupha isicelo kunye nesihloko somninimzi esichazwe kuqwalaselo lommeli.

curl -H "Host: one.example.com" localhost -i

Isicelo seHTTP siya kubangela impazamo 503. Oku kungenxa yokuba imidibaniso enyukayo ayisebenzi kwaye ayifumaneki. Ke ngoko, uMmeli woMthunywa akanandawo yokuya kwisicelo. Lo myalelo ulandelayo uyakuqala uthotho lweenkonzo zeHTTP ezihambelana noqwalaselo oluchazwe kuMthunywa.

docker run -d katacoda/docker-http-server; docker run -d katacoda/docker-http-server;

Ngeenkonzo ezifumanekayo, uMthunywa angakwazi ukwenza i-traffic ngempumelelo kwindawo asiya kuyo.

curl -H "Host: one.example.com" localhost -i

Kuya kufuneka ubone impendulo ebonisa ukuba sesiphi isikhongozeli seDocker esiqhubekisele phambili isicelo. Kwiilogi zoMmeli womthunywa kufuneka kwakhona ubone imveliso yomtya wokufikelela.

Izihloko ezongezelelweyo ze-HTTP zokuphendula

Uza kubona iiheader ezongezelelweyo zeHTTP kwiheader zempendulo zeyona sicelo. Iheda ibonisa ixesha elichithwe ngumamkeli onyukayo ekuqhubeni isicelo. Ibonakaliswe ngeemilliseconds. Oku kuluncedo ukuba umxhasi ufuna ukumisela ixesha lenkonzo xa kuthelekiswa nokubaleka komsebenzi womnatha.

x-envoy-upstream-service-time: 0
server: envoy

Uqwalaselo lokugqibela

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
                - "one.example.com"
                - "www.one.example.com"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: targetCluster
          http_filters:
          - name: envoy.router
          clusters:
  - name: targetCluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts: [
      { socket_address: { address: 172.18.0.3, port_value: 80 }},
      { socket_address: { address: 172.18.0.4, port_value: 80 }}
    ]

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9090 }

Ulwazi olongezelelweyo oluvela kumguquleli

Imiyalelo yokufakela uMmeli womthunywa inokufumaneka kwiwebhusayithi https://www.getenvoy.io/

Ngokungagqibekanga, i-rpm ayinayo inkqubo yenkonzo yoqwalaselo.

Yongeza ulungiselelo lwenkonzo yenkqubo/etc/systemd/system/envoy.service:

[Unit]
Description=Envoy Proxy
Documentation=https://www.envoyproxy.io/
After=network-online.target
Requires=envoy-auth-server.service
Wants=nginx.service

[Service]
User=root
Restart=on-failure
ExecStart=/usr/bin/envoy --config-path /etc/envoy/config.yaml
[Install]
WantedBy=multi-user.target

Kufuneka wenze ulawulo /etc/envoy/ kwaye ubeke config.yaml config apho.

Kukho incoko yetelegram usebenzisa umthunywa womthunywa: https://t.me/envoyproxy_ru

Ummeli womthunywa akakuxhasi ukukhonza umxholo ongatshintshiyo. Ke ngoko, ngubani onokuvotela eli nqaku: https://github.com/envoyproxy/envoy/issues/378

Ngabasebenzisi ababhalisiweyo kuphela abanokuthatha inxaxheba kuphando. Ngena, ndiyacela.

Ngaba esi sithuba sikukhuthaze ukuba ufake kwaye uvavanye umthunywa womthunywa?

  • ewe

  • akukho

Bali-75 abasebenzisi abavotileyo. Abasebenzisi abasi-18 abakhange.

umthombo: www.habr.com

Yongeza izimvo