A Visual Guide kune Troubleshooting Kubernetes

Cherechedza. transl.: Ichi chinyorwa chikamu chezvigadzirwa zveprojekiti zvakaburitswa munzvimbo yeruzhinji kudzidzak8s, makambani ekudzidzisa uye vatungamiri vega kuti vashande neKubernetes. Mariri, Danieri Polencic, maneja wepurojekiti, anogovera mirairo yekuona pane matanho ekutora kana matambudziko akajairwa nemaapplication ari kushanda paK8s cluster.

TL; DR: heino dhizaini inokubatsira kubvisa debug kuKubernetes:

Flowchart yekutsvaga nekugadzirisa zvikanganiso musumbu. Yekutanga (muChirungu) inowanikwa pa PDF и semufananidzo.

Kana uchiendesa application kuKubernetes, pane kazhinji zvinhu zvitatu zvaunoda kutsanangura:

• Deployment - iyi imhando yekubika yekugadzira makopi echishandiso, anonzi pods;
• sevhisi - yemukati mutoro balancer inogovera traffic pakati pemapods;
• Ingress - tsananguro yekuti traffic ichawana sei kubva kunze kwenyika kuenda kuSevhisi.

Heino pfupiso yemifananidzo inokurumidza:

1) MuKubernetes, zvikumbiro zvinogashira traffic kubva kunze kwenyika kuburikidza nezvikamu zviviri zvezviyereso zvemutoro: zvemukati nekunze.

2) Balancer yemukati inonzi Service, yekunze inonzi Ingress.

3) Deployment inogadzira mapodhi uye inoatarisa (haana kugadzirwa nemaoko).

Ngatiti iwe unoda kuendesa nyore application a la Mhoro Nyika. Iyo YAML kumisikidzwa kwayo ichaita seizvi:

apiVersion: apps/v1
kind: Deployment # <<<
metadata:
  name: my-deployment
  labels:
    track: canary
spec:
  selector:
    matchLabels:
      any-name: my-app
  template:
    metadata:
      labels:
        any-name: my-app
    spec:
      containers:
      - name: cont1
        image: learnk8s/app:1.0.0
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service # <<<
metadata:
  name: my-service
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    name: app
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress # <<<
metadata:
  name: my-ingress
spec:
  rules:
  - http:
    paths:
    - backend:
        serviceName: app
        servicePort: 80
      path: /

Tsanangudzo yacho yakarebesa uye zviri nyore kuvhiringika pamusoro pekuti izvo zvikamu zvinodyidzana sei kune mumwe nemumwe.

Somuenzaniso:

• Ndeipi yaunofanira kushandisa port 80 uye iwe unofanirwa kushandisa 8080 rini?
• Ndinofanira kugadzira chiteshi chitsva chesevhisi yega yega kuti vasapokane?
• Mazita emazita ane basa here? Vanofanira kufanana kwose kwose here?

Tisati tatarisana nekugadzirisa, ngatirangarirei kuti zvikamu zvitatu zvinodyidzana sei kune chimwe nechimwe. Ngatitange neDeployment uye Service.

Hukama pakati peDeployment uye Service

Iwe uchashamisika, asi Deployment uye Service hazvina hukama. Pane kudaro, Sevhisi inonongedza zvakananga kuPods, ichipfuura Deployment.

Nekudaro, isu tinofarira kuti maPods nemaSevhisi ane hukama kune mumwe nemumwe. Zvinhu zvitatu zvekuyeuka:

1. Selector (selector) yeSevhisi inofanira kufanana nePod label imwe chete.
2. targetPort inofanira kufanana containerPort mudziyo mukati mePod.
3. port Basa rinogona kuva chero chinhu. Masevhisi akasiyana anogona kushandisa chiteshi chimwe chete nekuti ane akasiyana IP kero.

Dhiagiramu inotevera inomiririra zvese zviri pamusoro muchimiro chemifananidzo:

1) Fungidzira kuti sevhisi inotungamira traffic kune imwe pod:

2) Paunenge uchigadzira pod, unofanirwa kutsanangura containerPort pamudziyo wega wega mumapodhi:

3) Paunenge uchigadzira sevhisi, unofanirwa kutsanangura port и targetPort. Asi ndeipi inoshandiswa kubatanidza kune mudziyo?

4) Via targetPort. Zvinofanira kuenderana containerPort.

5) Ngatiti port 3000 yakavhurika mugaba. Ipapo kukosha targetPort zvinofanira kuva zvakafanana.

Mune iyo YAML faira, mavara uye ports / targetPort inofanira kufanana:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    track: canary
spec:
  selector:
    matchLabels:
      any-name: my-app
  template:
    metadata:
     labels:  # <<<
        any-name: my-app  # <<<
   spec:
      containers:
      - name: cont1
        image: learnk8s/app:1.0.0
        ports:
       - containerPort: 8080  # <<<
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  ports:
  - port: 80
   targetPort: 8080  # <<<
 selector:  # <<<
    any-name: my-app  # <<<

Ko chitupa track: canary pamusoro peDeployment chikamu? Inofanira kufanana here?

Iyi label ndeye deployment chaiyo uye haishandiswe nesevhisi kunzira traffic. Mune mamwe mazwi, inogona kubviswa kana kupihwa kukosha kwakasiyana.

Ko kuzosarudza matchLabels?

Inofanirwa kugara ichienderana nemazita ePod, sezvo ichishandiswa neDeployment kuronda mapods.

Ngatifungei kuti magadzirisa zvakanaka. Nzira yekuvatarisa sei?

Unogona kutarisa pod label nemurairo unotevera:

kubectl get pods --show-labels

Kana, kana mapodhi ari emamwe maapplication:

kubectl get pods --selector any-name=my-app --show-labels

Kupi any-name=my-app is a label any-name: my-app.

Pane matambudziko asara here?

Unogona kubatana kune pod! Kuti uite izvi unofanirwa kushandisa murairo port-forward mu kubectl. Inokubvumira kuti ubatanidze kune sevhisi uye tarisa kubatana.

kubectl port-forward service/<service name> 3000:80

Здесь:

• service/<service name> - zita rebasa; kwatiri ndizvo my-service;
• 3000 ndiyo chiteshi chinoda kuvhurwa pakombuta;
• 80 - chiteshi chakatsanangurwa mumunda port service.

Kana iyo yekubatanidza yakasimbiswa, saka zvigadziriso ndezvechokwadi.

Kana iyo yekubatanidza ikatadza, pane dambudziko nemazita kana madoko haaenderane.

Hukama pakati peSevhisi uye Ingress

Nhanho inotevera yekupa mukana kune iyo application inosanganisira kumisikidza Ingress. Ingress inoda kuziva nzira yekuwana sevhisi, wobva watsvaga mapods uye kutungamira traffic kwavari. Ingress inowana iyo inodiwa sevhisi nemazita uye yakavhurika chiteshi.

Mutsanangudzo yeIngress uye Service maparamita maviri anofanira kuenderana:

1. servicePort muIngress inofanira kufanana neparameter port muSevhisi;
2. serviceName muIngress inofanira kufanana nemunda name muSevhisi.

Iyo inotevera dhizaini inopfupisa iyo port yekubatanidza:

1) Sezvaunotoziva, Sevhisi inoteerera kune imwe port:

2) Ingress ine parameter inonzi servicePort:

3) Iyi parameter (servicePort) inofanira kugara ichienderana port mune tsananguro yeSevhisi:

4) Kana port 80 yakatsanangurwa muSevhisi, saka zvakafanira kuti servicePort yakaenzanawo ne80:

Mukuita, iwe unofanirwa kuterera kune inotevera mitsara:

apiVersion: v1
kind: Service
metadata:
 name: my-service  # <<<
spec:
  ports:
 - port: 80  # <<<
   targetPort: 8080
  selector:
    any-name: my-app
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
    paths:
    - backend:
       serviceName: my-service  # <<<
       servicePort: 80  # <<<
     path: /

Nzira yekutarisa kana Ingress iri kushanda?

Unogona kushandisa nzira ne kubectl port-forward, asi panzvimbo yebasa raunoda kubatanidza kune Ingress controller.

Kutanga iwe unofanirwa kutsvaga zita repodhi neIngress controller:

kubectl get pods --all-namespaces
NAMESPACE   NAME                              READY STATUS
kube-system coredns-5644d7b6d9-jn7cq          1/1   Running
kube-system etcd-minikube                     1/1   Running
kube-system kube-apiserver-minikube           1/1   Running
kube-system kube-controller-manager-minikube  1/1   Running
kube-system kube-proxy-zvf2h                  1/1   Running
kube-system kube-scheduler-minikube           1/1   Running
kube-system nginx-ingress-controller-6fc5bcc  1/1   Running

Tsvaga iyo Ingress pod (inogona kunge iri mune imwe nzvimbo yezita) uye mhanyisa iwo murairo describekuti uwane nhamba dzechiteshi:

kubectl describe pod nginx-ingress-controller-6fc5bcc 
--namespace kube-system 
 | grep Ports
Ports:         80/TCP, 443/TCP, 18080/TCP

Pakupedzisira, batanidza kune pod:

kubectl port-forward nginx-ingress-controller-6fc5bcc 3000:80 --namespace kube-system

Iye zvino pese paunotumira chikumbiro chekutumira 3000 pakombuta yako, chinozoendeswa kune port 80 yepodhi ine Ingress controller. Nekuenda ku http://localhost:3000, iwe unofanirwa kuona iyo peji inogadzirwa neiyo application.

Pfupiso yezviteshi

Ngatirangarirei zvakare kuti ndeapi madoko uye mavara anofanira kuenderana:

1. Sarudzo mutsanangudzo yeSevhisi inofanirwa kuenderana nerebhu yepodhi;
2. targetPort mutsanangudzo Sevhisi inofanira kufanana containerPort mudziyo mukati mepodhi;
3. port mutsanangudzo Service inogona kuva chero chinhu. Masevhisi akasiyana anogona kushandisa chiteshi chimwe chete nekuti ane akasiyana IP kero;
4. servicePort Ingress inofanira kufanana port mutsanangudzo yeSevhisi;
5. Zita rebasa rinofanira kufanana nemunda serviceName muIngress.

Nehurombo, hazvina kukwana kuziva maitiro ekugadzirisa YAML kumisikidzwa.

Chii chinoitika kana zvinhu zvikashata?

Iyo pod inogona kusatanga kana kuti inogona kuparara.

3 Matanho Ekuongorora Matambudziko Ekushandisa muKubernetes

Usati watanga kugadzirisa dhizaini yako, unofanirwa kuve nekunzwisisa kwakanaka kwemashandiro anoita Kubernetes.

Sezvo imwe neimwe application yakatorwa muK8s ine zvinhu zvitatu, inofanirwa kugadziriswa mune imwe hurongwa, kutanga kubva pasi.

1. Kutanga iwe unofanirwa kuve nechokwadi chekuti mapodhi ari kushanda, zvino ...
2. Tarisa kana sevhisi ichipa traffic kune pods, uyezve ...
3. Tarisa kana Ingress yakagadziriswa nemazvo.

Mumiriri wekuona:

1) Iwe unofanirwa kutanga kutsvaga matambudziko kubva pasi. Chekutanga tarisa kuti mapods ane zvimiro Ready и Running:

2) Kana mapodhi akagadzirira (Ready), iwe unofanirwa kuona kana sevhisi ichigovera traffic pakati pemapods:

3) Chekupedzisira, unofanirwa kuongorora kubatana pakati pesevhisi neIngress:

1. Diagnostics yemapods

Muzviitiko zvakawanda, dambudziko rinosangana nebhokisi. Ita shuwa kuti mapods akanyorwa se Ready и Running. Unogona kutarisa izvi uchishandisa murairo:

kubectl get pods
NAME                    READY STATUS            RESTARTS  AGE
app1                    0/1   ImagePullBackOff  0         47h
app2                    0/1   Error             0         47h
app3-76f9fcd46b-xbv4k   1/1   Running           1         47h

Mune yekuraira yakabuda pamusoro, iyo yekupedzisira pod yakanyorwa se Running и Ready, zvisinei, izvi hazvisi izvo kune mamwe maviri.

Nzira yekunzwisisa chii chakakanganisika?

Pane mirairo mina inobatsira yekuongorora mapods:

1. kubectl logs <имя pod'а> inokubvumira kubvisa matanda kubva mumidziyo mune pod;
2. kubectl describe pod <имя pod'а> inokubvumira kuti uone rondedzero yezviitiko zvakabatana nepodhi;
3. kubectl get pod <имя pod'а> inokutendera iwe kuti uwane iyo YAML kumisikidzwa kwepodhi yakachengetwa muKubernetes;
4. kubectl exec -ti <имя pod'а> bash inokutendera iwe kuvhura inodyidzana yekuraira shell mune imwe yemidziyo yepodhi

Ndeipi yaunofanira kusarudza?

Chokwadi ndechekuti hapana murairo wepasi rose. Musanganiswa weizvi unofanirwa kushandiswa.

Matambudziko epod chaiwo

Kune marudzi maviri makuru ezvikanganiso zvepod: zvikanganiso zvekutanga uye zvikanganiso zvekumhanya.

Zvikanganiso zvekutanga:

• ImagePullBackoff
• ImageInspectError
• ErrImagePull
• ErrImageNeverPull
• RegistryUnavailable
• InvalidImageName

Runtime kukanganisa:

• CrashLoopBackOff
• RunContainerError
• KillContainerError
• VerifyNonRootError
• RunInitContainerError
• CreatePodSandboxError
• ConfigPodSandboxError
• KillPodSandboxError
• SetupNetworkError
• TeardownNetworkError

Zvimwe zvikanganiso zvinowanzoitika pane zvimwe. Heano mamwe ezvikanganiso zvinowanzoitika uye maitiro ekuzvigadzirisa.

ImagePullBackOff

Iko kukanganisa kunoitika apo Kubernetes asingakwanisi kuwana mufananidzo weimwe yemidziyo yepod. Heano zvikonzero zvitatu zvakajairika zveizvi:

1. Zita remufananidzo harina kururama - semuenzaniso, iwe wakakanganisa mukati maro, kana chifananidzo hachipo;
2. Mucherechedzo usipo wakatsanangurwa pamufananidzo;
3. Mufananidzo wacho unochengetwa mune yakavanzika registry uye Kubernetes haana mvumo yekuiwana.

Zvikonzero zviviri zvekutanga zviri nyore kubvisa - ingo gadzirisa zita remufananidzo uye tag. Mune iyo yekupedzisira, iwe unofanirwa kuisa zvitupa zveiyo yakavharwa registry muChakavanzika uye wedzera zvinongedzo kwairi mumapods. Mune Kubernetes zvinyorwa pane muenzaniso kuti izvi zvingaitwa sei.

Crash Loop Back Off

Kubenetes anokanda kukanganisa CrashLoopBackOff, kana mudziyo usingagoni kutanga. Izvi zvinowanzoitika kana:

1. Pane tsikidzi mune application inoidzivirira kubva pakuvhurwa;
2. Mudziyo kugadzirwa zvisirizvo;
3. Muedzo weLiveness wakakundikana kakawanda.

Iwe unofanirwa kuedza kusvika kumatanda kubva mumudziyo kuti uwane chikonzero chekutadza kwayo. Kana zvakaoma kuwana matanda nekuti mudziyo unotangazve nekukasira, unogona kushandisa unotevera kuraira:

kubectl logs <pod-name> --previous

Iyo inoratidza mameseji ekukanganisa kubva kune yakapfuura incarnation yemudziyo.

RunContainerError

Kukanganisa uku kunoitika kana mudziyo ukatadza kutanga. Inoenderana nenguva isati yatanga application. Inowanzo kukonzerwa nezvisina kururama marongero, semuenzaniso:

• kuedza kukwidza vhoriyamu isipo seConfigMap kana Zvakavanzika;
• kuedza kukwidza vhoriyamu yekuverenga chete sekuverenga-nyora.

Chikwata chakanyatsokodzera kuongorora zvikanganiso zvakadaro kubectl describe pod <pod-name>.

Mapodhi ari muPending state

Kana yangosikwa, iyo pod inoramba iri muhurumende Pending.

Sei izvi zvichiitika?

Heano zvikonzero zvinogoneka (ndiri kufunga kuti scheduler iri kushanda zvakanaka):

1. Iyo cluster haina zviwanikwa zvakakwana, senge simba rekugadzirisa uye ndangariro, kumhanya pod.
2. Chinhu chacho chinoiswa munzvimbo yakakodzera yezita ResourceQuota uye kugadzira pod kuchaita kuti zita rezita riende kupfuura quota.
3. Pod inosungirirwa kuPending PersistentVolumeClaim.

Muchiitiko ichi, zvinokurudzirwa kushandisa murairo kubectl describe uye tarisa chikamu Events:

kubectl describe pod <pod name>

Kana pane zvikanganiso zvine chekuita ne ResourceQuotas, zvinokurudzirwa kuona masumbu ematanda uchishandisa murairo

kubectl get events --sort-by=.metadata.creationTimestamp

Mapodhi haana Kugadzirira

Kana pod yakanyorwa se Running, asi haisi mumamiriro ezvinhu Ready, zvinoreva kuongorora kugadzirira kwayo (kugadzirira probe) anokundikana.

Kana izvi zvikaitika, iyo pod haina kubatana kune sevhisi uye hapana traffic inoyerera kwairi. Iko kugadzirira bvunzo kukundikana kunokonzerwa nematambudziko mukushandisa. Muchiitiko ichi, kuti uwane kukanganisa, unoda kuongorora chikamu Events mune yekuraira kubuda kubectl describe.

2. Kuongororwa kwebasa

Kana mapods akanyorwa se Running и Ready, asi hapasati paine mhinduro kubva kuchishandiso, iwe unofanirwa kutarisa marongero ebasa.

Masevhisi ane basa rekufambisa traffic kune pods zvichienderana nemazita avo. Naizvozvo, chinhu chekutanga chaunofanirwa kuita kutarisa kuti mangani mapodhi anoshanda nesevhisi. Kuti uite izvi, unogona kutarisa magumo musevhisi:

kubectl describe service <service-name> | grep Endpoints

Endpoint imhando mbiri dzefomu <IP-адрес:порт>, uye ingangoita imwe peya yakadaro inofanira kunge iripo mune zvakabuda (kureva kuti, kanenge podhi imwe inoshanda nesevhisi).

Kana chikamu Endpoins isina chinhu, sarudzo mbiri dzinogoneka:

1. hapana mapodhi ane label chaiyo (hint: tarisa kana nzvimbo yezita yakasarudzwa nemazvo);
2. Pane mhosho mumalebhu ebasa mune yekusarudzi.

Kana iwe ukaona rondedzero yemagumo asi uchiri kusakwanisa kuwana iyo application, saka angangove ane mhosva ibug mukati. targetPort mutsanangudzo yebasa.

Nzira yekutarisa kushanda kwesevhisi?

Pasinei nerudzi rwebasa, unogona kushandisa murairo kubectl port-forward kubatana nayo:

kubectl port-forward service/<service-name> 3000:80

Здесь:

• <service-name> - zita rebasa;
• 3000 ndiyo chiteshi chaunovhura pakombuta;
• 80 - chiteshi padivi rebasa.

3. Ingress diagnostics

Kana wakaverenga kusvika zvino, zvino:

• mapodhi akanyorwa se Running и Ready;
• iyo sevhisi inobudirira kugovera traffic pakati pemapods.

Nekudaro, haugone kusvika paapp.

Izvi zvinoreva kuti iyo Ingress controller inogona kunge isina kugadzirwa nemazvo. Sezvo Ingress controller iri yechitatu-bato chikamu musumbu, kune dzakasiyana nzira dzekugadzirisa zvinoenderana nerudzi rwayo.

Asi usati watanga kushandisa maturusi akakosha kugadzirisa Ingress, unogona kuita chimwe chinhu chakareruka. Ingress inoshandisa serviceName и servicePort kuti ubatanidze kushumiro. Iwe unofanirwa kutarisa kana dzakagadziriswa nemazvo. Iwe unogona kuita izvi uchishandisa murairo:

kubectl describe ingress <ingress-name>

Kana column Backend isina chinhu, pane mukana mukuru wekukanganisa kwekugadzirisa. Kana iyo yekumashure iri munzvimbo, asi iyo application haisati yasvika, saka dambudziko rinogona kunge rine chekuita ne:

• Ingress kusvikika zvigadziriso kubva paruzhinji Internet;
• cluster accessibility settings kubva paruzhinji Internet.

Iwe unogona kuona matambudziko ane zvivakwa nekubatanidza zvakananga kune Ingress pod. Kuti uite izvi, tanga watsvaga iyo Ingress Controller pod (inogona kunge iri mune imwe nzvimbo yemazita):

kubectl get pods --all-namespaces
NAMESPACE   NAME                              READY STATUS
kube-system coredns-5644d7b6d9-jn7cq          1/1   Running
kube-system etcd-minikube                     1/1   Running
kube-system kube-apiserver-minikube           1/1   Running
kube-system kube-controller-manager-minikube  1/1   Running
kube-system kube-proxy-zvf2h                  1/1   Running
kube-system kube-scheduler-minikube           1/1   Running
kube-system nginx-ingress-controller-6fc5bcc  1/1   Running

Shandisa murairo describekuseta chiteshi:

kubectl describe pod nginx-ingress-controller-6fc5bcc
--namespace kube-system 
 | grep Ports

Pakupedzisira, batanidza kune pod:

kubectl port-forward nginx-ingress-controller-6fc5bcc 3000:80 --namespace kube-system

Iye zvino zvikumbiro zvese zvekuisa 3000 pakombuta zvichaendeswa kune port 80 yepodhi.

Zvinoshanda here?

• Kana hongu, saka dambudziko riri pane zvivakwa. Izvo zvinodikanwa kuti uone chaizvo kuti traffic inofambiswa sei kune cluster.
• Kana zvisiri, saka dambudziko riri neIngress controller.

Kana iwe usingakwanise kuita kuti Ingress controller ishande, iwe unofanirwa kuigadzirisa.

Kune akawanda marudzi eIngress controllers. Inonyanya kufarirwa ndeye Nginx, HAProxy, Traefik, nezvimwe. (kuti uwane rumwe ruzivo nezve mhinduro dziripo, ona ongororo yedu - approx. transl.) Iwe unofanirwa kutarisa kune dhairekitori rekugadzirisa mune yakakodzera controller zvinyorwa. Nokuti iyo Ingress Nginx ndiye anonyanya kufarirwa Ingress controller, isu takabatanidza mamwe matipi muchinyorwa kugadzirisa matambudziko ane chekuita nazvo.

Kugadzirisa iyo Ingress Nginx controller

Iyo Ingress-nginx chirongwa chine mukuru plugin ye kubectl. Team kubectl ingress-nginx inogona kushandiswa ku:

• kuongororwa kwematanda, backends, zvitupa, nezvimwe;
• kubatana kuIngress;
• kudzidza gadziriso yazvino.

Mirairo mitatu inotevera ichakubatsira pane izvi:

• kubectl ingress-nginx lint - cheki nginx.conf;
• kubectl ingress-nginx backend - inoongorora backend (yakafanana ne kubectl describe ingress <ingress-name>);
• kubectl ingress-nginx logs - anotarisa matanda.

Ziva kuti mune dzimwe nguva ungangoda kudoma chaiyo namespace yeIngress controller uchishandisa mureza --namespace <name>.

Summary

Kugadzirisa Kubernetes kunogona kunetsa kana usingazive pekutangira. Iwe unofanirwa kugara uchiswedera kune dambudziko kubva pasi kumusoro: tanga nemapods, uye wozoenda kushumiro uye Ingress. Maitiro ekugadzirisa anotsanangurwa muchinyorwa chino anogona kushandiswa kune zvimwe zvinhu, senge:

• idle Jobs uye CronJobs;
• StatefulSets uye DaemonSets.

Ndinotaura kutenda kwangu Gergely Risko, Daniel Weibel и Charles Christyraj nokuda kwemashoko anokosha uye kuwedzera.

PS kubva kumushanduri

Verenga zvakare pablog yedu:

• «kubectl-debug plugin ye debugging muKubernetes pods";
• «6 inovaraidza system bugs mukushanda kweKubernetes [uye mhinduro yavo]";
• «Zvishandiso zvevagadziri vezvishandiso zvinomhanya paKubernetes";
• «6 nyaya dzinoshanda kubva kuSRE yedu yemazuva ese hupenyu ".

Source: www.habr.com