Saka urip karo Kubernetes: Kepiye server HTTP ora milih wong Spanyol

Perwakilan klien kita, sing tumpukan aplikasi ana ing awan saka Microsoft (Azure), ngatasi masalah: bubar, sawetara panjaluk saka sawetara klien saka Eropa wiwit rampung kanthi kesalahan 400 (Panjaluk sing ala). Kabeh aplikasi ditulis ing .NET, disebarake ing Kubernetes...

Salah sawijining aplikasi yaiku API, ing ngendi kabeh lalu lintas pungkasane teka. Lalu lintas iki dirungokake dening server HTTP kestrel, diatur dening klien .NET lan dadi tuan rumah ing pod. Kanthi debugging, kita begja amarga ana pangguna tartamtu sing terus-terusan ngasilake masalah kasebut. Nanging, kabeh rumit dening rantai lalu lintas:

Kesalahan ing Ingress katon kaya iki:

{
   "number_fields":{
      "status":400,
      "request_time":0.001,
      "bytes_sent":465,
      "upstream_response_time":0,
      "upstream_retries":0,
      "bytes_received":2328
   },
   "stream":"stdout",
   "string_fields":{
      "ingress":"app",
      "protocol":"HTTP/1.1",
      "request_id":"f9ab8540407208a119463975afda90bc",
      "path":"/api/sign-in",
      "nginx_upstream_status":"400",
      "service":"app",
      "namespace":"production",
      "location":"/front",
      "scheme":"https",
      "method":"POST",
      "nginx_upstream_response_time":"0.000",
      "nginx_upstream_bytes_received":"120",
      "vhost":"api.app.example.com",
      "host":"api.app.example.com",
      "user":"",
      "address":"83.41.81.250",
      "nginx_upstream_addr":"10.240.0.110:80",
      "referrer":"https://api.app.example.com/auth/login?long_encrypted_header",
      "service_port":"http",
      "user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
      "time":"2019-03-06T18:29:16+00:00",
      "content_kind":"cache-headers-not-present",
      "request_query":""
   },
   "timestamp":"2019-03-06 18:29:16",
   "labels":{
      "app":"nginx",
      "pod-template-generation":"6",
      "controller-revision-hash":"1682636041"
   },
   "namespace":"kube-nginx-ingress",
   "nsec":6726612,
   "source":"kubernetes",
   "host":"k8s-node-55555-0",
   "pod_name":"nginx-v2hcb",
   "container_name":"nginx",
   "boolean_fields":{}
}

Ing wektu sing padha, Kestrel menehi:

HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0

Malah kanthi verbosity maksimal, kesalahan Kestrel ana banget informasi migunani sethitik:

{
   "number_fields":{"ThreadId":76},
   "stream":"stdout",
   "string_fields":{
      "EventId":"{"Id"=>17, "Name"=>"ConnectionBadRequest"}",
      "SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
      "ConnectionId":"0HLL2VJSST5KV",
      "@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
      "@t":"2019-03-07T13:06:48.1449083Z",
      "@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
      "message":"Malformed request: invalid headers."
   },
   "timestamp":"2019-03-07 13:06:48",
   "labels":{
      "pod-template-hash":"2368795483",
      "service":"app"
   },
   "namespace":"production",
   "nsec":145341848,
   "source":"kubernetes",
   "host":"k8s-node-55555-1",
   "pod_name":"app-67bdcf98d7-mhktx",
   "container_name":"app",
   "boolean_fields":{}
}

Kayane mung tcpdump sing bakal mbantu ngatasi masalah iki ... nanging aku bakal mbaleni babagan rantai lalu lintas:

Investigasi

Temenan, luwih becik ngrungokake lalu lintas ing simpul tartamtu kasebut, ing ngendi Kubernetes wis masang pod: volume mbucal bakal kaya sing bakal bisa kanggo nemokake ing paling soko cantik cepet. Lan pancen, nalika mriksa, pigura ing ngisor iki katon:

GET /back/user HTTP/1.1
Host: api.app.example.com
X-Request-ID: 27ceb14972da8c21a8f92904b3eff1e5
X-Real-IP: 83.41.81.250
X-Forwarded-For: 83.41.81.250
X-Forwarded-Host: api.app.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Original-URI: /front/back/user
X-Scheme: https
X-Original-Forwarded-For: 83.41.81.250
X-Nginx-Geo-Client-Country: Spain
X-Nginx-Geo-Client-City: M.laga
Accept-Encoding: gzip
CF-IPCountry: ES
CF-RAY: 4b345cfd1c4ac691-MAD
CF-Visitor: {"scheme":"https"}
pragma: no-cache
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
referer: https://app.example.com/auth/login
accept-language: en-US,en;q=0.9,en-GB;q=0.8,pl;q=0.7
cookie: many_encrypted_cookies; .AspNetCore.Identity.Application=something_encrypted; 
CF-Connecting-IP: 83.41.81.250
True-Client-IP: 83.41.81.250
CDN-Loop: cloudflare

HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0

Sawise dipriksa kanthi tliti babagan mbucal, tembung kasebut katon M.laga. Iku gampang kanggo guess sing ora ana kutha M.laga ing Spanyol (nanging ana Málaga). Ngrebut ide iki, kita ndeleng konfigurasi Ingress, ing ngendi kita ndeleng sing dipasang wulan kepungkur (kanthi panjaluk klien) "ora mbebayani" cuplikan:

    ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Nginx-Geo-Client-Country $geoip_country_name;
      proxy_set_header X-Nginx-Geo-Client-City $geoip_city;

Sawise mateni terusake header iki, kabeh dadi apik! (Dadi dadi cetha yen aplikasi kasebut ora mbutuhake header kasebut maneh.)

Saiki ayo ndelok masalahe luwih umum. Iku bisa gampang maleh nang aplikasi dening nggawe panjalukan telnet kanggo localhost:80:

GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Desiree

... bali 401 Unauthorized, kaya sing dikarepake. Apa sing kedadeyan yen kita nindakake:

GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Désirée

?

Bakal bali 400 Bad request - ing log aplikasi kita bakal nampa kesalahan sing wis kita kenal:

{
   "@t":"2019-03-31T12:59:54.3746446Z",
   "@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
   "@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
   "ConnectionId":"0HLLLR1J974L9",
   "message":"Malformed request: invalid headers.",
   "EventId":{
      "Id":17,
      "Name":"ConnectionBadRequest"
   },
   "SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
   "ThreadId":71
}

Hasil

Khusus Kestrel ora bisa ngolah header HTTP kanthi bener kanthi karakter sing bener ing UTF-8, sing ana ing jeneng kutha sing cukup akeh.

Faktor tambahan ing kasus kita yaiku klien saiki ora ngrancang ngganti implementasine Kestrel ing aplikasi kasebut. Nanging, masalah ing AspNetCore dhewe (No.4318, No.7707) dheweke ngomong yen iki ora bakal mbantu ...

Kanggo ngringkes: cathetan kasebut ora ana maneh babagan masalah khusus Kestrel utawa UTF-8 (ing 2019?!), nanging babagan kasunyatan sing mindfulness lan sinau konsisten Saben langkah sing sampeyan lakoni nalika nggoleki masalah cepet utawa mengko bakal metokake woh. Sugeng enjang!

PS

Waca uga ing blog kita:

• «6 bug sistem sing nyenengake ing operasi Kubernetes [lan solusine]";
• «Tip & trik Kubernetes: kaca kesalahan khusus ing NGINX Ingress";
• «Ringkesan lan perbandingan saka pengontrol Ingress kanggo Kubernetes";
• «Ngawasi ping antarane simpul Kubernetes - resep kita";
• «3 kasus sing ora biasa babagan subsistem jaringan Linux".

Source: www.habr.com