Mwakilishi wa mteja wetu, ambaye rundo la maombi yake hukaa kwenye wingu kutoka kwa Microsoft (Azure), alishughulikia tatizo: hivi majuzi, baadhi ya maombi kutoka kwa wateja wengine kutoka Uropa yalianza na hitilafu 400 () Programu zote zimeandikwa katika .NET, zimetumwa katika Kubernetes...
Moja ya programu ni API, ambayo trafiki yote hatimaye huja. Trafiki hii inasikilizwa na seva ya HTTP , iliyosanidiwa na mteja wa .NET na kupangishwa katika ganda. Kwa utatuzi, tulikuwa na bahati kwa maana kwamba kulikuwa na mtumiaji mahususi ambaye alizalisha tena tatizo mara kwa mara. Walakini, kila kitu kilikuwa ngumu na mlolongo wa trafiki:
Kosa katika Ingress ilionekana kama hii:
{
"number_fields":{
"status":400,
"request_time":0.001,
"bytes_sent":465,
"upstream_response_time":0,
"upstream_retries":0,
"bytes_received":2328
},
"stream":"stdout",
"string_fields":{
"ingress":"app",
"protocol":"HTTP/1.1",
"request_id":"f9ab8540407208a119463975afda90bc",
"path":"/api/sign-in",
"nginx_upstream_status":"400",
"service":"app",
"namespace":"production",
"location":"/front",
"scheme":"https",
"method":"POST",
"nginx_upstream_response_time":"0.000",
"nginx_upstream_bytes_received":"120",
"vhost":"api.app.example.com",
"host":"api.app.example.com",
"user":"",
"address":"83.41.81.250",
"nginx_upstream_addr":"10.240.0.110:80",
"referrer":"https://api.app.example.com/auth/login?long_encrypted_header",
"service_port":"http",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
"time":"2019-03-06T18:29:16+00:00",
"content_kind":"cache-headers-not-present",
"request_query":""
},
"timestamp":"2019-03-06 18:29:16",
"labels":{
"app":"nginx",
"pod-template-generation":"6",
"controller-revision-hash":"1682636041"
},
"namespace":"kube-nginx-ingress",
"nsec":6726612,
"source":"kubernetes",
"host":"k8s-node-55555-0",
"pod_name":"nginx-v2hcb",
"container_name":"nginx",
"boolean_fields":{}
}Wakati huo huo, Kestrel alitoa:
HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0Hata ikiwa na kitenzi cha juu zaidi, hitilafu ya Kestrel ilikuwa kubwa sana habari muhimu kidogo:
{
"number_fields":{"ThreadId":76},
"stream":"stdout",
"string_fields":{
"EventId":"{"Id"=>17, "Name"=>"ConnectionBadRequest"}",
"SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
"ConnectionId":"0HLL2VJSST5KV",
"@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
"@t":"2019-03-07T13:06:48.1449083Z",
"@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
"message":"Malformed request: invalid headers."
},
"timestamp":"2019-03-07 13:06:48",
"labels":{
"pod-template-hash":"2368795483",
"service":"app"
},
"namespace":"production",
"nsec":145341848,
"source":"kubernetes",
"host":"k8s-node-55555-1",
"pod_name":"app-67bdcf98d7-mhktx",
"container_name":"app",
"boolean_fields":{}
}Inaweza kuonekana kuwa tcpdump pekee itasaidia kutatua shida hii ... lakini nitarudia juu ya mlolongo wa trafiki:
Uchunguzi
Ni wazi, ni bora kusikiliza trafiki kwenye nodi hiyo maalum, ambapo Kubernetes imepeleka ganda: kiasi cha dampo itakuwa hivyo kwamba itawezekana kupata angalau kitu haraka sana. Na kwa kweli, wakati wa kuichunguza, sura ifuatayo iligunduliwa:
GET /back/user HTTP/1.1
Host: api.app.example.com
X-Request-ID: 27ceb14972da8c21a8f92904b3eff1e5
X-Real-IP: 83.41.81.250
X-Forwarded-For: 83.41.81.250
X-Forwarded-Host: api.app.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Original-URI: /front/back/user
X-Scheme: https
X-Original-Forwarded-For: 83.41.81.250
X-Nginx-Geo-Client-Country: Spain
X-Nginx-Geo-Client-City: M.laga
Accept-Encoding: gzip
CF-IPCountry: ES
CF-RAY: 4b345cfd1c4ac691-MAD
CF-Visitor: {"scheme":"https"}
pragma: no-cache
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
referer: https://app.example.com/auth/login
accept-language: en-US,en;q=0.9,en-GB;q=0.8,pl;q=0.7
cookie: many_encrypted_cookies; .AspNetCore.Identity.Application=something_encrypted;
CF-Connecting-IP: 83.41.81.250
True-Client-IP: 83.41.81.250
CDN-Loop: cloudflare
HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0
Baada ya kukagua kwa karibu dampo hilo, neno hilo liligunduliwa M.laga. Ni rahisi kukisia kuwa hakuna jiji la M.laga nchini Uhispania (lakini lipo ) Kuzingatia wazo hili, tuliangalia usanidi wa Ingress, ambapo tuliona ile iliyoingizwa mwezi mmoja uliopita (kwa ombi la mteja) kijisehemu "isiyo na madhara".:
ingress.kubernetes.io/configuration-snippet: |
proxy_set_header X-Nginx-Geo-Client-Country $geoip_country_name;
proxy_set_header X-Nginx-Geo-Client-City $geoip_city;Baada ya kuzima usambazaji wa vichwa hivi, kila kitu kilikuwa sawa! (Hivi karibuni ikawa wazi kuwa programu yenyewe haikuhitaji tena vichwa hivi.)
Sasa tuangalie tatizo kwa ujumla zaidi. Inaweza kunakiliwa kwa urahisi ndani ya programu kwa kufanya ombi la telnet kwa localhost:80:
GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Desiree
... inarudi 401 Unauthorized, kama ilivyotarajiwa. Nini kitatokea ikiwa tutafanya:
GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=DΓ©sirΓ©e?
Itarudi 400 Bad request - katika logi ya maombi tutapokea hitilafu ambayo tayari inajulikana kwetu:
{
"@t":"2019-03-31T12:59:54.3746446Z",
"@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
"@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
"ConnectionId":"0HLLLR1J974L9",
"message":"Malformed request: invalid headers.",
"EventId":{
"Id":17,
"Name":"ConnectionBadRequest"
},
"SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
"ThreadId":71
}Matokeo ya
Hasa Kestrel kuchakata kwa usahihi vichwa vya HTTP na herufi sahihi katika UTF-8, ambazo zimo katika majina ya idadi kubwa ya miji.
Sababu ya ziada katika kesi yetu ni kwamba mteja kwa sasa hana mpango wa kubadilisha utekelezaji wa Kestrel katika programu. Walakini, maswala katika AspNetCore yenyewe (, ) wanasema kuwa hii haitasaidia...
Kwa muhtasari: noti sio tena juu ya shida maalum za Kestrel au UTF-8 (mnamo 2019?!), lakini juu ya ukweli kwamba akili na utafiti thabiti Kila hatua unayochukua wakati wa kutafuta shida itazaa matunda mapema au baadaye. Bahati njema!
PS
Soma pia kwenye blogi yetu:
- Β«";
- Β«";
- Β«";
- Β«";
- Β«'.
Chanzo: mapenzi.com