In fertsjintwurdiger fan ús kliïnt, waans applikaasjestapel leit yn 'e Microsoft-wolk (Azure), hat in probleem oanpakt: koartlyn begon guon oanfragen fan guon kliïnten út Jeropa te einigjen mei flater 400 (). Alle applikaasjes binne skreaun yn .NET, ynset yn Kubernetes...
Ien fan de tapassingen is de API, dêr't alle ferkear úteinlik troch komt. Dit ferkear wurdt harke troch de HTTP-tsjinner , ynsteld troch de .NET-kliïnt en hosted yn in pod. Mei debuggen wiene wy gelok yn 't sin dat d'r in spesifike brûker wie dy't it probleem konsekwint reprodusearre. Alles waard lykwols komplisearre troch de ferkearsketting:
De flater yn Ingress seach der sa út:
{
"number_fields":{
"status":400,
"request_time":0.001,
"bytes_sent":465,
"upstream_response_time":0,
"upstream_retries":0,
"bytes_received":2328
},
"stream":"stdout",
"string_fields":{
"ingress":"app",
"protocol":"HTTP/1.1",
"request_id":"f9ab8540407208a119463975afda90bc",
"path":"/api/sign-in",
"nginx_upstream_status":"400",
"service":"app",
"namespace":"production",
"location":"/front",
"scheme":"https",
"method":"POST",
"nginx_upstream_response_time":"0.000",
"nginx_upstream_bytes_received":"120",
"vhost":"api.app.example.com",
"host":"api.app.example.com",
"user":"",
"address":"83.41.81.250",
"nginx_upstream_addr":"10.240.0.110:80",
"referrer":"https://api.app.example.com/auth/login?long_encrypted_header",
"service_port":"http",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
"time":"2019-03-06T18:29:16+00:00",
"content_kind":"cache-headers-not-present",
"request_query":""
},
"timestamp":"2019-03-06 18:29:16",
"labels":{
"app":"nginx",
"pod-template-generation":"6",
"controller-revision-hash":"1682636041"
},
"namespace":"kube-nginx-ingress",
"nsec":6726612,
"source":"kubernetes",
"host":"k8s-node-55555-0",
"pod_name":"nginx-v2hcb",
"container_name":"nginx",
"boolean_fields":{}
}Tagelyk joech Kestrel:
HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0Sels mei maksimale verbosity befette de Kestrel-flater ekstreem bytsje brûkbere ynformaasje:
{
"number_fields":{"ThreadId":76},
"stream":"stdout",
"string_fields":{
"EventId":"{"Id"=>17, "Name"=>"ConnectionBadRequest"}",
"SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
"ConnectionId":"0HLL2VJSST5KV",
"@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
"@t":"2019-03-07T13:06:48.1449083Z",
"@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
"message":"Malformed request: invalid headers."
},
"timestamp":"2019-03-07 13:06:48",
"labels":{
"pod-template-hash":"2368795483",
"service":"app"
},
"namespace":"production",
"nsec":145341848,
"source":"kubernetes",
"host":"k8s-node-55555-1",
"pod_name":"app-67bdcf98d7-mhktx",
"container_name":"app",
"boolean_fields":{}
}It liket derop dat allinich tcpdump dit probleem sil helpe oplosse ... mar ik sil werhelje oer de ferkearsketting:
Undersyk
Fansels is it better om nei it ferkear te harkjen op dat spesifike knooppunt, dêr't Kubernetes in pod ynset hat: it folume fan 'e dump sil sa wêze dat it mooglik is om teminsten frij gau wat te finen. En yndie, by it ûndersiikjen, waard it folgjende frame opmurken:
GET /back/user HTTP/1.1
Host: api.app.example.com
X-Request-ID: 27ceb14972da8c21a8f92904b3eff1e5
X-Real-IP: 83.41.81.250
X-Forwarded-For: 83.41.81.250
X-Forwarded-Host: api.app.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Original-URI: /front/back/user
X-Scheme: https
X-Original-Forwarded-For: 83.41.81.250
X-Nginx-Geo-Client-Country: Spain
X-Nginx-Geo-Client-City: M.laga
Accept-Encoding: gzip
CF-IPCountry: ES
CF-RAY: 4b345cfd1c4ac691-MAD
CF-Visitor: {"scheme":"https"}
pragma: no-cache
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
referer: https://app.example.com/auth/login
accept-language: en-US,en;q=0.9,en-GB;q=0.8,pl;q=0.7
cookie: many_encrypted_cookies; .AspNetCore.Identity.Application=something_encrypted;
CF-Connecting-IP: 83.41.81.250
True-Client-IP: 83.41.81.250
CDN-Loop: cloudflare
HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0
By neier ynspektearjen fan de stoart waard it wurd opfallen M.laga. It is maklik te rieden dat d'r gjin M.laga-stêd is yn Spanje (mar d'r is ). Troch dit idee oan te pakken, seagen wy nei de Ingress-konfiguraasjes, wêr't wy dejinge seagen in moanne lyn ynfoege (op fersyk fan 'e kliïnt) "harmless" snippet:
ingress.kubernetes.io/configuration-snippet: |
proxy_set_header X-Nginx-Geo-Client-Country $geoip_country_name;
proxy_set_header X-Nginx-Geo-Client-City $geoip_city;Nei it útskeakeljen fan it trochstjoeren fan dizze kopteksten, waard alles goed! (It waard al gau dúdlik dat de applikaasje sels dizze kopteksten net mear nedich hie.)
Litte wy no nei it probleem sjen mear algemien. It kin maklik wurde reprodusearre binnen de applikaasje troch it meitsjen fan in telnet-fersyk oan localhost:80:
GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Desiree
... werom 401 Unauthorized, as ferwachte. Wat bart der as wy dogge:
GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Désirée?
Sil weromkomme 400 Bad request - yn it applikaasjelog krije wy in flater dy't ús al bekend is:
{
"@t":"2019-03-31T12:59:54.3746446Z",
"@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
"@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
"ConnectionId":"0HLLLR1J974L9",
"message":"Malformed request: invalid headers.",
"EventId":{
"Id":17,
"Name":"ConnectionBadRequest"
},
"SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
"ThreadId":71
}Resultaten
Spesifyk Kestrel HTTP-headers korrekt ferwurkje mei de juste karakters yn UTF-8, dy't yn 'e nammen fan in frij grut oantal stêden binne befette.
In ekstra faktor yn ús gefal is dat de klant op it stuit net fan plan is de ymplemintaasje fan Kestrel yn 'e applikaasje te feroarjen. Problemen yn AspNetCore sels (, ) se sizze dat dit net helpt ...
Gearfetsjend: de notysje giet net mear oer de spesifike problemen fan Kestrel of UTF-8 (yn 2019?!), mar oer it feit dat mindfulness en konsekwint stúdzje Elke stap dy't jo nimme by it sykjen nei problemen sil ier of letter frucht drage. Súkses!
PS
Lês ek op ús blog:
- «";
- «";
- «";
- «";
- «".
Boarne: www.habr.com