Kubernetes bilan hayotdan: Qanday qilib HTTP serveri ispanlarga yoqmadi

Ilovalar to'plami Microsoft (Azure) bulutida joylashgan mijozimiz vakili muammoga murojaat qildi: yaqinda Evropadan ba'zi mijozlarning ba'zi so'rovlari 400 xato bilan yakunlana boshladi (Notugri surov). Barcha ilovalar .NET da yozilgan, Kubernetes da joylashtirilgan...

Ilovalardan biri API bo'lib, u orqali barcha trafik oxir-oqibat keladi. Ushbu trafik HTTP serveri tomonidan tinglanadi kerkenez, .NET mijozi tomonidan sozlangan va podda joylashtirilgan. Nosozliklarni tuzatish bilan biz muammoni doimiy ravishda takrorlaydigan aniq foydalanuvchi borligi ma'nosida omadli edik. Biroq, hamma narsa transport zanjiri bilan murakkablashdi:

Ingressdagi xato quyidagicha ko'rindi:

{
   "number_fields":{
      "status":400,
      "request_time":0.001,
      "bytes_sent":465,
      "upstream_response_time":0,
      "upstream_retries":0,
      "bytes_received":2328
   },
   "stream":"stdout",
   "string_fields":{
      "ingress":"app",
      "protocol":"HTTP/1.1",
      "request_id":"f9ab8540407208a119463975afda90bc",
      "path":"/api/sign-in",
      "nginx_upstream_status":"400",
      "service":"app",
      "namespace":"production",
      "location":"/front",
      "scheme":"https",
      "method":"POST",
      "nginx_upstream_response_time":"0.000",
      "nginx_upstream_bytes_received":"120",
      "vhost":"api.app.example.com",
      "host":"api.app.example.com",
      "user":"",
      "address":"83.41.81.250",
      "nginx_upstream_addr":"10.240.0.110:80",
      "referrer":"https://api.app.example.com/auth/login?long_encrypted_header",
      "service_port":"http",
      "user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
      "time":"2019-03-06T18:29:16+00:00",
      "content_kind":"cache-headers-not-present",
      "request_query":""
   },
   "timestamp":"2019-03-06 18:29:16",
   "labels":{
      "app":"nginx",
      "pod-template-generation":"6",
      "controller-revision-hash":"1682636041"
   },
   "namespace":"kube-nginx-ingress",
   "nsec":6726612,
   "source":"kubernetes",
   "host":"k8s-node-55555-0",
   "pod_name":"nginx-v2hcb",
   "container_name":"nginx",
   "boolean_fields":{}
}

Shu bilan birga, Kestrel shunday dedi:

HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0

Maksimal aniqlik bilan ham, Kestrel xatosi juda katta edi ozgina foydali ma'lumotlar:

{
   "number_fields":{"ThreadId":76},
   "stream":"stdout",
   "string_fields":{
      "EventId":"{"Id"=>17, "Name"=>"ConnectionBadRequest"}",
      "SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
      "ConnectionId":"0HLL2VJSST5KV",
      "@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
      "@t":"2019-03-07T13:06:48.1449083Z",
      "@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
      "message":"Malformed request: invalid headers."
   },
   "timestamp":"2019-03-07 13:06:48",
   "labels":{
      "pod-template-hash":"2368795483",
      "service":"app"
   },
   "namespace":"production",
   "nsec":145341848,
   "source":"kubernetes",
   "host":"k8s-node-55555-1",
   "pod_name":"app-67bdcf98d7-mhktx",
   "container_name":"app",
   "boolean_fields":{}
}

Faqat tcpdump bu muammoni hal qilishga yordam beradi, deb tuyuladi ... lekin men trafik zanjiri haqida takrorlayman:

Tergov

Shubhasiz, trafikni tinglash yaxshiroqdir ushbu maxsus tugun ustida, bu erda Kubernetes podni o'rnatgan: axlatxonaning hajmi shunday bo'ladiki, hech bo'lmaganda tezda biror narsani topish mumkin bo'ladi. Va haqiqatan ham, uni o'rganayotganda, quyidagi ramka e'tiborga olindi:

GET /back/user HTTP/1.1
Host: api.app.example.com
X-Request-ID: 27ceb14972da8c21a8f92904b3eff1e5
X-Real-IP: 83.41.81.250
X-Forwarded-For: 83.41.81.250
X-Forwarded-Host: api.app.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Original-URI: /front/back/user
X-Scheme: https
X-Original-Forwarded-For: 83.41.81.250
X-Nginx-Geo-Client-Country: Spain
X-Nginx-Geo-Client-City: M.laga
Accept-Encoding: gzip
CF-IPCountry: ES
CF-RAY: 4b345cfd1c4ac691-MAD
CF-Visitor: {"scheme":"https"}
pragma: no-cache
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
referer: https://app.example.com/auth/login
accept-language: en-US,en;q=0.9,en-GB;q=0.8,pl;q=0.7
cookie: many_encrypted_cookies; .AspNetCore.Identity.Application=something_encrypted; 
CF-Connecting-IP: 83.41.81.250
True-Client-IP: 83.41.81.250
CDN-Loop: cloudflare

HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0

Chiqindixonani sinchiklab ko‘rib chiqqach, bu so‘z sezildi M.laga. Ispaniyada M.laga shahri yo'qligini taxmin qilish oson (lekin bor Malaga). Ushbu g'oyadan kelib chiqib, biz Ingress konfiguratsiyasini ko'rib chiqdik, u erda biz bir oy oldin kiritilganini ko'rdik (mijozning iltimosiga binoan) "zararsiz" parcha:

    ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Nginx-Geo-Client-Country $geoip_country_name;
      proxy_set_header X-Nginx-Geo-Client-City $geoip_city;

Ushbu sarlavhalarni yo'naltirishni o'chirib qo'ygandan so'ng, hamma narsa yaxshi bo'ldi! (Tez orada dasturning o'zi bu sarlavhalarga muhtoj emasligi ma'lum bo'ldi.)

Endi muammoni ko'rib chiqaylik umumiyroq. Telnet so'rovi orqali uni ilova ichida osongina ko'paytirish mumkin localhost:80:

GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Desiree

... qaytadi 401 Unauthorized, kutilganidek. Agar shunday qilsak nima bo'ladi:

GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Désirée

?

Qaytadi 400 Bad request — ilovalar jurnalida biz allaqachon tanish bo'lgan xatoni olamiz:

{
   "@t":"2019-03-31T12:59:54.3746446Z",
   "@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
   "@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
   "ConnectionId":"0HLLLR1J974L9",
   "message":"Malformed request: invalid headers.",
   "EventId":{
      "Id":17,
      "Name":"ConnectionBadRequest"
   },
   "SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
   "ThreadId":71
}

natijalar

Xususan, Kestrel bo'lishi mumkin emas juda ko'p sonli shaharlar nomlarida mavjud bo'lgan UTF-8-dagi to'g'ri belgilar bilan HTTP sarlavhalarini to'g'ri qayta ishlash.

Bizning holatimizda qo'shimcha omil shundaki, mijoz hozirda dasturda Kestrel dasturini o'zgartirishni rejalashtirmaydi. Biroq, AspNetCore'dagi muammolar (№4318, №7707) ular bu yordam bermaydi, deyishadi ...

Xulosa qilib aytadigan bo'lsak: eslatma endi Kestrel yoki UTF-8 ning o'ziga xos muammolari haqida emas (2019 yilda?!), balki haqiqat haqida. diqqat va izchil o'rganish Muammolarni qidirishda qilgan har bir qadamingiz ertami-kechmi o'z mevasini beradi. Omad!

PS

Shuningdek, bizning blogimizda o'qing:

• «Kubernetes ishidagi 6 ta qiziqarli tizim xatolari [va ularning yechimi]";
• «Kubernetes maslahatlari va fokuslari: NGINX kirishidagi maxsus xato sahifalari";
• «Kubernetes uchun Ingress kontrollerlarini ko'rib chiqish va taqqoslash";
• «Kubernetes tugunlari orasidagi pinglarni kuzatish - bizning retseptimiz";
• «3 необычных кейса о сетевой подсистеме Linux".

Manba: www.habr.com