Ukusuka kubomi kunye noKubernetes: Umncedisi we-HTTP akazange athande abantu baseSpain

Ummeli womxhasi wethu, onesitaki sesicelo sakhe esihlala kwilifu leMicrosoft (Azure), ulungise ingxaki: kutsha nje, ezinye izicelo zabathengi abavela eYurophu zaqala ukuphela ngempazamo 400 (Isicelo esibi). Zonke izicelo zibhalwe kwi-.NET, zibekwe kwi-Kubernetes...

Esinye sezicelo yi-API, apho yonke i-traffic iza ekugqibeleni. Le traffic imanyelwa ngumncedisi weHTTP kestrel, iqwalaselwe ngumthengi we.NET kwaye isingathwe kwi pod. Ngokulungisa ingxaki, sibe nethamsanqa ngengqiqo yokuba bekukho umsebenzisi othile othe rhoqo wavelisa ingxaki. Nangona kunjalo, yonke into yayinzima ngenxa yekhonkco lezendlela:

Impazamo kwi-Ingress ijongeka ngolu hlobo:

{
   "number_fields":{
      "status":400,
      "request_time":0.001,
      "bytes_sent":465,
      "upstream_response_time":0,
      "upstream_retries":0,
      "bytes_received":2328
   },
   "stream":"stdout",
   "string_fields":{
      "ingress":"app",
      "protocol":"HTTP/1.1",
      "request_id":"f9ab8540407208a119463975afda90bc",
      "path":"/api/sign-in",
      "nginx_upstream_status":"400",
      "service":"app",
      "namespace":"production",
      "location":"/front",
      "scheme":"https",
      "method":"POST",
      "nginx_upstream_response_time":"0.000",
      "nginx_upstream_bytes_received":"120",
      "vhost":"api.app.example.com",
      "host":"api.app.example.com",
      "user":"",
      "address":"83.41.81.250",
      "nginx_upstream_addr":"10.240.0.110:80",
      "referrer":"https://api.app.example.com/auth/login?long_encrypted_header",
      "service_port":"http",
      "user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
      "time":"2019-03-06T18:29:16+00:00",
      "content_kind":"cache-headers-not-present",
      "request_query":""
   },
   "timestamp":"2019-03-06 18:29:16",
   "labels":{
      "app":"nginx",
      "pod-template-generation":"6",
      "controller-revision-hash":"1682636041"
   },
   "namespace":"kube-nginx-ingress",
   "nsec":6726612,
   "source":"kubernetes",
   "host":"k8s-node-55555-0",
   "pod_name":"nginx-v2hcb",
   "container_name":"nginx",
   "boolean_fields":{}
}

Kwangelo xesha, uKestrel wanikela:

HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0

Nokuba i-verbosity ephezulu, impazamo ye-Kestrel yayiqulathe kakhulu ulwazi oluncinci oluluncedo:

{
   "number_fields":{"ThreadId":76},
   "stream":"stdout",
   "string_fields":{
      "EventId":"{"Id"=>17, "Name"=>"ConnectionBadRequest"}",
      "SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
      "ConnectionId":"0HLL2VJSST5KV",
      "@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
      "@t":"2019-03-07T13:06:48.1449083Z",
      "@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
      "message":"Malformed request: invalid headers."
   },
   "timestamp":"2019-03-07 13:06:48",
   "labels":{
      "pod-template-hash":"2368795483",
      "service":"app"
   },
   "namespace":"production",
   "nsec":145341848,
   "source":"kubernetes",
   "host":"k8s-node-55555-1",
   "pod_name":"app-67bdcf98d7-mhktx",
   "container_name":"app",
   "boolean_fields":{}
}

Kubonakala ngathi kuphela i-tcpdump eya kunceda ukusombulula le ngxaki ... kodwa ndiya kuphinda malunga nekhonkco lendlela:

Uphando

Ngokucacileyo, kungcono ukumamela i-traffic kuloo nodi ethile, apho i-Kubernetes isebenzise i-pod: umthamo wokulahla uya kuba yinto enokuthi ifumaneke ubuncinane into ngokukhawuleza. Kwaye eneneni, xa uyiphonononga, isakhelo silandelayo saqatshelwa:

GET /back/user HTTP/1.1
Host: api.app.example.com
X-Request-ID: 27ceb14972da8c21a8f92904b3eff1e5
X-Real-IP: 83.41.81.250
X-Forwarded-For: 83.41.81.250
X-Forwarded-Host: api.app.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Original-URI: /front/back/user
X-Scheme: https
X-Original-Forwarded-For: 83.41.81.250
X-Nginx-Geo-Client-Country: Spain
X-Nginx-Geo-Client-City: M.laga
Accept-Encoding: gzip
CF-IPCountry: ES
CF-RAY: 4b345cfd1c4ac691-MAD
CF-Visitor: {"scheme":"https"}
pragma: no-cache
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
referer: https://app.example.com/auth/login
accept-language: en-US,en;q=0.9,en-GB;q=0.8,pl;q=0.7
cookie: many_encrypted_cookies; .AspNetCore.Identity.Application=something_encrypted; 
CF-Connecting-IP: 83.41.81.250
True-Client-IP: 83.41.81.250
CDN-Loop: cloudflare

HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0

Ekuhlolisisweni ngokusondeleyo kwendawo yokulahla, igama laqatshelwa M.laga. Kulula ukuqikelela ukuba asikho isixeko saseM.laga eSpain (kodwa sikhona Málaga). Ukubamba le ngcamango, sijonge i-Ingress configs, apho sabona enye ifakwe kwinyanga edlulileyo (ngesicelo somthengi) "engenabungozi" isiqwenga:

    ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Nginx-Geo-Client-Country $geoip_country_name;
      proxy_set_header X-Nginx-Geo-Client-City $geoip_city;

Emva kokuvala ukuthunyelwa kwezi zihloko, yonke into yahamba kakuhle! (Kwakhawuleza kwacaca ukuba isicelo ngokwaso asisazidingi ezi zihloko.)

Ngoku makhe sijonge ingxaki ngokubanzi. Inokuphinda iveliswe ngokulula ngaphakathi kwisicelo ngokwenza isicelo se-telnet ku localhost:80:

GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Desiree

... ibuyisela 401 Unauthorized, njengoko kulindelekile. Kwenzeka ntoni xa sisenza:

GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Désirée

?

Uya kubuya 400 Bad request - kwilog yesicelo siya kufumana impazamo esele siyiqhelile:

{
   "@t":"2019-03-31T12:59:54.3746446Z",
   "@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
   "@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
   "ConnectionId":"0HLLLR1J974L9",
   "message":"Malformed request: invalid headers.",
   "EventId":{
      "Id":17,
      "Name":"ConnectionBadRequest"
   },
   "SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
   "ThreadId":71
}

Iziphumo

Ngokukodwa iKestrel akakwazi lungisa ngokuchanekileyo iiheader zeHTTP ezinamagama achanekileyo kwi-UTF-8, equlathwe kumagama enani elikhulu ngokufanelekileyo lezixeko.

Into eyongezelelweyo kwimeko yethu kukuba umxhasi akacebi ngoku ukutshintsha ukuphunyezwa kweKestrel kwisicelo. Nangona kunjalo, imiba kwi-AspNetCore ngokwayo (No.4318, No.7707) bathi ayizukunceda lonto...

Ukushwankathela: inqaku alisekho malunga neengxaki ezithile ze-Kestrel okanye i-UTF-8 (ngo-2019?!), Kodwa malunga nenyaniso yokuba ukuqonda kunye nokufunda rhoqo Inyathelo ngalinye olithathayo ngelixa ukhangela iingxaki liya kuba neziqhamo kungekudala. Nqwenelela impumelelo!

PS

Funda nakwibhlog yethu:

• «6 inkqubo yokonwabisa bugs ekusebenzeni kweKubernetes [kunye nesisombululo sabo]";
• «Iingcebiso ze-Kubernetes kunye namaqhinga: amaphepha eempazamo zesiko kwi-NGINX Ingress";
• «Amagqabantshintshi kunye nokuthelekisa abalawuli be-Ingress be-Kubernetes";
• «Ukubeka iliso kwi-pings phakathi kwe-Kubernetes nodes - iresiphi yethu";
• «Iimeko ezi-3 ezingaqhelekanga malunga nenkqubo esezantsi yothungelwano ye-Linux».

umthombo: www.habr.com