ืคึฟื•ืŸ ืœืขื‘ืŸ ืžื™ื˜ Kubernetes: ื•ื•ื™ ื“ื™ ื”ื˜ื˜ืคึผ ืกืขืจื•ื•ืขืจ ื”ืื˜ ื ื™ืฉื˜ ืคื™ื™ื•ื•ืขืจื“ ื“ื™ ืกืคึผืึทื ื™ืึทืจื“ืก

ืคึฟื•ืŸ ืœืขื‘ืŸ ืžื™ื˜ Kubernetes: ื•ื•ื™ ื“ื™ ื”ื˜ื˜ืคึผ ืกืขืจื•ื•ืขืจ ื”ืื˜ ื ื™ืฉื˜ ืคื™ื™ื•ื•ืขืจื“ ื“ื™ ืกืคึผืึทื ื™ืึทืจื“ืก

ื ืคืืจืฉื˜ื™ื™ืขืจ ืคื•ืŸ ืื•ื ื“ื–ืขืจ ืงืœื™ืขื ื˜, ื•ื•ืขืžืขื ืก ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ืกื˜ืึทืง ืจื™ื–ื™ื™ื“ื– ืื™ืŸ ื“ื™ ื•ื•ืึธืœืงืŸ ืคื•ืŸ ืžื™ื™ืงืจืึธืกืึธืคึฟื˜ (Azure), ื”ืึธื˜ ื’ืขืจืขื“ื˜ ืึท ืคึผืจืึธื‘ืœืขื: ืœืขืฆื˜ื ืก, ืขื˜ืœืขื›ืข ืจื™ืงื•ื•ืขืก ืคื•ืŸ ืขื˜ืœืขื›ืข ืงืœื™ื™ืึทื ืฅ ืคื•ืŸ ืื™ื™ืจืึธืคึผืข ืื ื’ืขื”ื•ื™ื‘ืŸ ืฆื• ืขื ื“ื™ืงืŸ ืžื™ื˜ ื˜ืขื•ืช 400 (ืฉืœืขื›ื˜ืข ื‘ืงืฉื”). ื›ืœ ืึทืคึผืœืึทืงื™ื™ืฉืึทื ื– ื–ืขื ืขืŸ ื’ืขืฉืจื™ื‘ืŸ ืื™ืŸ .NET, ื“ื™ืคึผืœื•ื™ื“ ืื™ืŸ Kubernetes ...

ืื™ื™ื ืขืจ ืคื•ืŸ ื“ื™ ืึทืคึผืœืึทืงื™ื™ืฉืึทื ื– ืื™ื– ื“ื™ API, ื“ื•ืจืš ื•ื•ืึธืก ืึทืœืข ืคืึทืจืงืขืจ ืœืขืกืึธืฃ ืงื•ืžื˜. ื“ืขืจ ืคืึทืจืงืขืจ ืื™ื– ืฆื•ื’ืขื”ืขืจื˜ ื“ื•ืจืš ื“ื™ ื”ื˜ื˜ืคึผ ืกืขืจื•ื•ืขืจ ืงืขืกื˜ืจืขืœ, ืงืึทื ืคื™ื’ื™ืขืจื“ ื“ื•ืจืš ื“ื™. ื ืขืฅ ืงืœื™ืขื ื˜ ืื•ืŸ ื›ืึธื•ืกื˜ื™ื“ ืื™ืŸ ืึท ืคึผืึธื“. ืžื™ื˜ ื“ื™ื‘ืึทื’ื™ื ื’, ืžื™ืจ ื–ืขื ืขืŸ ืžืึทื–ืœื“ื™ืง ืื™ืŸ ื“ืขื ื–ื™ื ืขืŸ ืึทื– ืขืก ืื™ื– ื’ืขื•ื•ืขืŸ ืึท ืกืคึผืขืฆื™ืคื™ืฉ ื‘ืึทื ื™ืฆืขืจ ื•ื•ืึธืก ืงืึทื ืกื™ืกื˜ืึทื ื˜ืœื™ ืจื™ืคึผืจืึทื“ื•ืกื˜ ื“ื™ ืคึผืจืึธื‘ืœืขื. ืึธื‘ืขืจ, ืึทืœืฅ ืื™ื– ื’ืขื•ื•ืขืŸ ืงืึธืžืคึผืœื™ืฆื™ืจื˜ ื“ื•ืจืš ื“ื™ ืคืึทืจืงืขืจ ืงื™ื™ื˜:

ืคึฟื•ืŸ ืœืขื‘ืŸ ืžื™ื˜ Kubernetes: ื•ื•ื™ ื“ื™ ื”ื˜ื˜ืคึผ ืกืขืจื•ื•ืขืจ ื”ืื˜ ื ื™ืฉื˜ ืคื™ื™ื•ื•ืขืจื“ ื“ื™ ืกืคึผืึทื ื™ืึทืจื“ืก

ื“ืขืจ ื˜ืขื•ืช ืื™ืŸ Ingress ื”ืื˜ ื’ืขืงื•ืงื˜ ื•ื•ื™ ื“ืึธืก:

{
   "number_fields":{
      "status":400,
      "request_time":0.001,
      "bytes_sent":465,
      "upstream_response_time":0,
      "upstream_retries":0,
      "bytes_received":2328
   },
   "stream":"stdout",
   "string_fields":{
      "ingress":"app",
      "protocol":"HTTP/1.1",
      "request_id":"f9ab8540407208a119463975afda90bc",
      "path":"/api/sign-in",
      "nginx_upstream_status":"400",
      "service":"app",
      "namespace":"production",
      "location":"/front",
      "scheme":"https",
      "method":"POST",
      "nginx_upstream_response_time":"0.000",
      "nginx_upstream_bytes_received":"120",
      "vhost":"api.app.example.com",
      "host":"api.app.example.com",
      "user":"",
      "address":"83.41.81.250",
      "nginx_upstream_addr":"10.240.0.110:80",
      "referrer":"https://api.app.example.com/auth/login?long_encrypted_header",
      "service_port":"http",
      "user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
      "time":"2019-03-06T18:29:16+00:00",
      "content_kind":"cache-headers-not-present",
      "request_query":""
   },
   "timestamp":"2019-03-06 18:29:16",
   "labels":{
      "app":"nginx",
      "pod-template-generation":"6",
      "controller-revision-hash":"1682636041"
   },
   "namespace":"kube-nginx-ingress",
   "nsec":6726612,
   "source":"kubernetes",
   "host":"k8s-node-55555-0",
   "pod_name":"nginx-v2hcb",
   "container_name":"nginx",
   "boolean_fields":{}
}

ื“ืขืจื‘ืฒึท ื”ืึธื˜ ืงืขืกื˜ืจืขืœ ื’ืขื’ืขื‘ืŸ:

HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0

ืืคื™ืœื• ืžื™ื˜ ืžืึทืงืกื™ืžื•ื ื•ื•ืขืจื‘ืึธื•ืกืึทื˜ื™, ื“ื™ ืงืขืกื˜ืจืขืœ ื˜ืขื•ืช ืงืึทื ื˜ื™ื™ื ื“ ื’ืึธืจ ื‘ื™ืกืœ ื ื•ืฆื™ืง ืื™ื ืคึฟืึธืจืžืึทืฆื™ืข:

{
   "number_fields":{"ThreadId":76},
   "stream":"stdout",
   "string_fields":{
      "EventId":"{"Id"=>17, "Name"=>"ConnectionBadRequest"}",
      "SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
      "ConnectionId":"0HLL2VJSST5KV",
      "@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
      "@t":"2019-03-07T13:06:48.1449083Z",
      "@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
      "message":"Malformed request: invalid headers."
   },
   "timestamp":"2019-03-07 13:06:48",
   "labels":{
      "pod-template-hash":"2368795483",
      "service":"app"
   },
   "namespace":"production",
   "nsec":145341848,
   "source":"kubernetes",
   "host":"k8s-node-55555-1",
   "pod_name":"app-67bdcf98d7-mhktx",
   "container_name":"app",
   "boolean_fields":{}
}

ืขืก ื•ื•ืึธืœื˜ ื•ื™ืกืงื•ืžืขืŸ ืึทื– ื‘ืœื•ื™ื– tcpdump ื•ื•ืขื˜ ื”ืขืœืคึฟืŸ ืกืึธืœื•ื•ืข ื“ืขื ืคึผืจืึธื‘ืœืขื ... ืึธื‘ืขืจ ืื™ืš ื•ื•ืขื˜ ืื™ื‘ืขืจื—ื–ืจืŸ ื•ื•ืขื’ืŸ ื“ื™ ืคืึทืจืงืขืจ ืงื™ื™ื˜:

ืคึฟื•ืŸ ืœืขื‘ืŸ ืžื™ื˜ Kubernetes: ื•ื•ื™ ื“ื™ ื”ื˜ื˜ืคึผ ืกืขืจื•ื•ืขืจ ื”ืื˜ ื ื™ืฉื˜ ืคื™ื™ื•ื•ืขืจื“ ื“ื™ ืกืคึผืึทื ื™ืึทืจื“ืก

ื•ื™ืกืคืึธืจืฉื•ื ื’

ื“ืึธืš, ืขืก ืื™ื– ื‘ืขืกืขืจ ืฆื• ื”ืขืจืŸ ืฆื• ืคืึทืจืงืขืจ ืื•ื™ืฃ ื“ืขื ืกืคึผืขืฆื™ืคื™ืฉ ื ืึธื“ืข, ื•ื•ื• Kubernetes ื”ืื˜ ื“ื™ืคึผืœื•ื™ื“ ืึท ืคึผืึธื“: ื“ืขืจ ื‘ืึทื ื“ ืคื•ืŸ ื“ื™ ื“ืึทืžืคึผ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืึทื–ื•ื™ ืึทื– ืขืก ื•ื•ืขื˜ ื–ื™ื™ืŸ ืžืขื’ืœืขืš ืฆื• ื’ืขืคึฟื™ื ืขืŸ ื‘ื™ื™ึท ืžื™ื ื“ืกื˜ืขืจ ืขืคึผืขืก ืฉื™ื™ืŸ ื’ืขืฉื•ื•ื™ื ื“. ืื•ืŸ ื˜ืืงืข ื•ื•ืขืŸ ืžืขืŸ ื”ืื˜ ืขืก ืื•ื ื˜ืขืจื–ื•ื›ื˜, ื”ืื˜ ืžืขืŸ ื‘ืืžืขืจืงื˜ ื“ื™ ืคืืœื’ื ื“ืข ืจืื:

GET /back/user HTTP/1.1
Host: api.app.example.com
X-Request-ID: 27ceb14972da8c21a8f92904b3eff1e5
X-Real-IP: 83.41.81.250
X-Forwarded-For: 83.41.81.250
X-Forwarded-Host: api.app.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Original-URI: /front/back/user
X-Scheme: https
X-Original-Forwarded-For: 83.41.81.250
X-Nginx-Geo-Client-Country: Spain
X-Nginx-Geo-Client-City: M.laga
Accept-Encoding: gzip
CF-IPCountry: ES
CF-RAY: 4b345cfd1c4ac691-MAD
CF-Visitor: {"scheme":"https"}
pragma: no-cache
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
referer: https://app.example.com/auth/login
accept-language: en-US,en;q=0.9,en-GB;q=0.8,pl;q=0.7
cookie: many_encrypted_cookies; .AspNetCore.Identity.Application=something_encrypted; 
CF-Connecting-IP: 83.41.81.250
True-Client-IP: 83.41.81.250
CDN-Loop: cloudflare

HTTP/1.1 400 Bad Request
Connection: close
Date: Wed, 06 Mar 2019 12:34:20 GMT
Server: Kestrel
Content-Length: 0

ื‘ืฒ ื ื ืื ื˜ืข ืจ ื“ื•ืจื›ืงื•ืง ืŸ ื“ ื™ ืฐืืจ ื˜ ื”ื ื˜ ืžืข ืŸ ื‘ืืžืขืจืง ื˜ ื“ื ืก ืฐืืจื˜ M.laga. ืขืก ืื™ื– ื’ืจื™ื ื’ ืฆื• ื˜ืจืขืคืŸ ืึทื– ืขืก ืื™ื– ืงื™ื™ืŸ M.laga ืฉื˜ืึธื˜ ืื™ืŸ ืกืคึผืึทื™ืŸ (ืึธื‘ืขืจ ืขืก ืื™ื– Mรกlaga). ืžื™ืจ ืึธื ื›ืึทืคึผืŸ ื“ืขื ื’ืขื“ืึทื ืง, ืžื™ืจ ื’ืขืงื•ืงื˜ ืื•ื™ืฃ ื“ื™ ื™ื ื’ืจืขืกืก ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทื ื–, ื•ื•ื• ืžื™ืจ ื”ืึธื‘ืŸ ื’ืขื–ืขืŸ ื“ืขื ืื™ื™ื ืขืจ ื™ื ืกืขืจื˜ืึทื“ ืึท ื—ื•ื“ืฉ ืฆื•ืจื™ืง (ืื™ืŸ ื“ืขืจ ื‘ืงืฉื” ืคื•ืŸ ื“ืขื ืงืœื™ืขื ื˜) "ืžืฉืขื“ืœืขืš" ืกื ื™ืคึผืึทื˜:

    ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Nginx-Geo-Client-Country $geoip_country_name;
      proxy_set_header X-Nginx-Geo-Client-City $geoip_city;

ื ืึธืš ื“ื™ืกื™ื™ื‘ืึทืœื™ื ื’ ื“ื™ ืคืึธืจื•ื•ืขืจื“ื™ื ื’ ืคื•ืŸ ื“ื™ ื›ืขื“ืขืจื–, ืึทืœืฅ ื’ืขื•ื•ืืจืŸ ืคื™ื™ึทืŸ! (ืขืก ืื™ื– ื‘ืึทืœื“ ื’ืขื•ื•ืืจืŸ ืงืœืึธืจ ืึทื– ื“ื™ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ื–ื™ืš ื ื™ื˜ ืžืขืจ ื“ืืจืฃ ื“ื™ ื›ืขื“ืขืจื–.)

ืื™ืฆื˜ ืœืึธืžื™ืจ ืงื•ืงืŸ ืื•ื™ืฃ ื“ื™ ืคึผืจืึธื‘ืœืขื ืžืขืจ ื‘ื›ืœืœ. ืขืก ืงืขื ืขืŸ ื–ื™ื™ืŸ ืœื™ื™ื›ื˜ ืจื™ืคึผืจืึทื“ื•ืกื˜ ื™ืŸ ื“ืขืจ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ื“ื•ืจืš ืžืึทื›ืŸ ืึท ื˜ืขืœื ืขื˜ ื‘ืขื˜ืŸ ืฆื• localhost:80:

GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Desiree

... ืงืขืจื˜ ื–ื™ืš 401 Unauthorized, ื•ื•ื™ ื“ืขืจื•ื•ืึทืจื˜. ื•ื•ืึธืก ื›ืึทืคึผืึทื ื– ืื•ื™ื‘ ืžื™ืจ ื˜ืึธืŸ:

GET /back/user HTTP/1.1
Host: api.app.example.com
cache-control: no-cache
accept: application/json, text/plain, */*
origin: https://app.example.com
Cookie: test=Dรฉsirรฉe

?

ื•ื•ืขื˜ ืฆื•ืจื™ืงืงื•ืžืขืŸ 400 Bad request - ืื™ืŸ ื“ื™ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ืงืœืึธืฅ ืžื™ืจ ื•ื•ืขืœืŸ ื‘ืึทืงื•ืžืขืŸ ืึท ื˜ืขื•ืช ื•ื•ืึธืก ืื™ื– ืฉื•ื™ืŸ ื‘ืึทืงืึทื ื˜ ืฆื• ืื•ื ื“ื–:

{
   "@t":"2019-03-31T12:59:54.3746446Z",
   "@mt":"Connection id "{ConnectionId}" bad request data: "{message}"",
   "@x":"Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequestsAsync>d__185`1.MoveNext()",
   "ConnectionId":"0HLLLR1J974L9",
   "message":"Malformed request: invalid headers.",
   "EventId":{
      "Id":17,
      "Name":"ConnectionBadRequest"
   },
   "SourceContext":"Microsoft.AspNetCore.Server.Kestrel",
   "ThreadId":71
}

ืจืขื–ื•ืœื˜ืึทื˜ืŸ ืคื•ืŸ

ืกืคึผืขืฆื™ืขืœ ืงืขืกื˜ืจืขืœ ืงืขืŸ ื ื™ืฉื˜ ืจื™ื›ื˜ื™ืง ืคึผืจืึธืฆืขืก ื”ื˜ื˜ืคึผ ื›ืขื“ืขืจื– ืžื™ื˜ ื“ื™ ืจื™ื›ื˜ื™ืง ืื•ืชื™ื•ืช ืื™ืŸ UTF-8, ื•ื•ืึธืก ื–ืขื ืขืŸ ืงืึทื ื˜ื™ื™ื ื“ ืื™ืŸ ื“ื™ ื ืขืžืขืŸ ืคื•ืŸ ืึท ื’ืึทื ืฅ ื’ืจื•ื™ืก ื ื•ืžืขืจ ืคื•ืŸ ืฉื˜ืขื˜.

ืึทืŸ ื ืึธืš ืคืึทืงื˜ืึธืจ ืื™ืŸ ืื•ื ื“ื–ืขืจ ืคืึทืœ ืื™ื– ืึทื– ื“ืขืจ ืงืœื™ืขื ื˜ ื˜ื•ื˜ ื ื™ืฉื˜ ืื™ืฆื˜ ืคึผืœืึทืŸ ืฆื• ื˜ื•ื™ืฉืŸ ื“ื™ ื™ืžืคึผืœืึทืžืขื ื˜ื™ื™ืฉืึทืŸ ืคื•ืŸ ืงืขืกื˜ืจืขืœ ืื™ืŸ ื“ื™ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ. ืึธื‘ืขืจ, ื™ืฉื•ื– ืื™ืŸ AspNetCore ื–ื™ืš (โ„– ืงืกื ื•ืžืงืก, โ„– ืงืกื ื•ืžืงืก) ื–ื™ื™ ื–ืื’ืŸ ืื– ื“ืืก ื•ื•ืขื˜ ื ื™ืฉื˜ ื”ืขืœืคืŸ...

ืฆื• ืงื™ืฆืขืจ: ื“ืขืจ ืฆืขื˜ืœ ืื™ื– ื ื™ื˜ ืžืขืจ ื•ื•ืขื’ืŸ ื“ื™ ืกืคึผืขืฆื™ืคื™ืฉ ืคึผืจืึธื‘ืœืขืžืก ืคื•ืŸ ืงืขืกื˜ืจืขืœ ืึธื“ืขืจ UTF-8 (ืื™ืŸ 2019?!), ืึธื‘ืขืจ ื•ื•ืขื’ืŸ ื“ืขื ืคืึทืงื˜ ืึทื– ืžื™ื ื“ืคื•ืœื ืขืกืก ืื•ืŸ ืงืึธื ืกื™ืกื˜ืขื ื˜ ืœืขืจื ืขืŸ ื™ืขื“ืขืจ ืฉืจื™ื˜ ืื™ืจ ื ืขืžืขืŸ ื•ื•ืขืŸ ืื™ืจ ื–ื•ื›ื˜ ืคึฟืึทืจ ืคึผืจืึธื‘ืœืขืžืก ื•ื•ืขื˜ ื’ื™ื›ืขืจ ืึธื“ืขืจ ืฉืคึผืขื˜ืขืจ ื˜ืจืึธื’ืŸ ืคืจื•ื›ื˜. ื–ืึธืœ ื–ื™ื™ึทื  ืžื™ื˜ ืžืึทื–ืœ!

ืคึผืก

ืœื™ื™ืขื ืขืŸ ืื•ื™ืš ืื•ื™ืฃ ืื•ื ื“ื–ืขืจ ื‘ืœืึธื’:

ืžืงื•ืจ: www.habr.com

ืœื™ื™ื’ืŸ ืึท ื‘ืึทืžืขืจืงื•ื ื’