Typice, semper opus est ut applicationi propria copia opum praebeatur ad eius rectam stabilemque operationem. Quid autem si plures applicationes eisdem opibus currunt? Quomodo singulis applicationibus minimis opibus necessariis provideri potest? Quomodo consumptio opum limitari potest? Quomodo onus inter nodos sapienter distribui potest? Quomodo scalatio horizontalis praeberi potest cum onus applicationis crescit?
Initium facere debemus a praecipuis generibus opum in systemate — scilicet, tempore CPU et memoria RAM. In manifestis K8S, hi typi opum his unitatibus mensurantur:
- CPU - in nucleis
- Memoria RAM — in octetis
Praeterea, pro singulis opibus duo genera requisitorum statui possunt: petitiones и fines"Requests" minima requisita pro liberis nodorum opibus ad continens (et pod in genere) currendum describunt, dum "limits" limitem rigidum opibus continenti praesto statuunt.
Interest intellegere non necesse esse utrumque typum in manifesto explicite definire, sed mores sic sequentes fore:
- Si tantum limites cuiusdam opis explicite specificantur, petitiones pro illa ope valorem limitibus aequalem automatice accipiunt (hoc per invocationem functionis `description` entitatis verificare potest). Aliis verbis, operatio continentis effecte ad eundem numerum opum limitabitur quas ad currendum requirit.
- Si solae petitiones explicite pro quadam re specificantur, nullae restrictiones huic re imponuntur — id est, continens solum a propriis nodi opibus limitatur.
Etiam administrationem opum configurare licet non solum in gradu continentis specifici, sed etiam in gradu spatii nominum utens entitatibus sequentibus:
- LimitRange — describit rationem limitationis in gradu continentis/pod in ns et necessaria est ad describendos limites implicitos pro continente/pod, necnon ad impediendam creationem continentium/pod manifeste pinguium (vel vice versa), ad limitandum numerum eorum et ad determinandam possibilem differentiam in valoribus in limitibus et petitionibus.
- ResourceQuotas — describit rationem restrictionis pro omnibus receptaculis in ns in genere et typice adhibetur ad opes per ambitum delimitandas (utile cum ambitus non stricte in gradu nodi delimitati sunt)
Infra exempla manifestorum quae limites opum constituunt sunt:
In gradu continentis specifici:
containers: - name: app-nginx image: nginx resources: requests: memory: 1Gi limits: cpu: 200mHoc est, hoc in casu, ad receptaculum cum nginx incipiendum, saltem 1G memoriae RAM liberae et 0.2 CPU in nodo requirentur, dum maximum quod receptaculum consumere potest est 0.2 CPU et omnis memoria RAM praesto in nodo.
In gradu integro ns:
apiVersion: v1 kind: ResourceQuota metadata: name: nxs-test spec: hard: requests.cpu: 300m requests.memory: 1Gi limits.cpu: 700m limits.memory: 2GiId est, summa omnium receptaculorum petitionum in ns implicito non potest excedere 300m pro CPU et 1G pro RAM, et summa omnium limitum est 700m pro CPU et 2G pro RAM.
Limites impliciti pro receptaculis in ns:
apiVersion: v1 kind: LimitRange metadata: name: nxs-limit-per-container spec: limits: - type: Container defaultRequest: cpu: 100m memory: 1Gi default: cpu: 1 memory: 2Gi min: cpu: 50m memory: 500Mi max: cpu: 2 memory: 4GiHoc significat spatium nominum implicitum pro omnibus receptaculis petitionem ad 100M pro CPU et 1G pro RAM constituturum esse, cum limite 1 CPU et 2G. Est etiam limes in valoribus possibilibus petitionum/limitum pro CPU (50M < x < 2) et RAM (500M < x < 4G).
Restrictiones gradus podorum (ns):
apiVersion: v1 kind: LimitRange metadata: name: nxs-limit-pod spec: limits: - type: Pod max: cpu: 4 memory: 1GiHoc significat pro quolibet pod in ns implicito, limitem quattuor vCPU et 1G constitutum iri.
Nunc vobis dicere velim quae commoda nobis hae restrictiones statutae afferre possint.
Mechanismus librationis oneris inter nodos
Ut notum est, distributio capsularum inter nodos est responsabilitas talis componenti k8s qualis est... scheduler, quae secundum algorithmum specificum operatur. Hic algorithmus per duos gradus transit cum nodum optimum ad incipiendum eligitur:
- eliquare
- Ranging
Id est, secundum rationem descriptam, nodi initio eliguntur in quibus pod immitti potest secundum seriem... praedicata (inclusa probatione utrum nodus satis opum habeat ad pod currendum - PodFitsResources), deinde pro singulis his nodis, secundum bonorumque Puncta tribuuntur (quo plures opes liberas nodus habet, eo plura puncta ei assignantur - LeastResourceAllocation/LeastRequestedPriority/BalancedResourceAllocation) et pod in nodo cum maximo numero punctorum emittitur (si plures nodi hanc condicionem simul implent, tum unus fortuitus eligitur).
Intellegendum est, cum nodi facultates praesto aestimantur, ordinatorem in datis in "etcd" reconditis niti — id est, facultatibus requisitis/limitatis cuiusque pod in illo nodo currentis — sed non in vero usu facultatum. Haec informatio ex exitu mandati obtineri potest. kubectl describe node $NODEFor example:
# kubectl describe nodes nxs-k8s-s1
..
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
ingress-nginx nginx-ingress-controller-754b85bf44-qkt2t 0 (0%) 0 (0%) 0 (0%) 0 (0%) 233d
kube-system kube-flannel-26bl4 150m (0%) 300m (1%) 64M (0%) 500M (1%) 233d
kube-system kube-proxy-exporter-cb629 0 (0%) 0 (0%) 0 (0%) 0 (0%) 233d
kube-system kube-proxy-x9fsc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 233d
kube-system nginx-proxy-k8s-worker-s1 25m (0%) 300m (1%) 32M (0%) 512M (1%) 233d
nxs-monitoring alertmanager-main-1 100m (0%) 100m (0%) 425Mi (1%) 25Mi (0%) 233d
nxs-logging filebeat-lmsmp 100m (0%) 0 (0%) 100Mi (0%) 200Mi (0%) 233d
nxs-monitoring node-exporter-v4gdq 112m (0%) 122m (0%) 200Mi (0%) 220Mi (0%) 233d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 487m (3%) 822m (5%)
memory 15856217600 (2%) 749976320 (3%)
ephemeral-storage 0 (0%) 0 (0%)Hic videmus omnes "pods" (capsulae) in nodo specifico currentes, necnon opes a singulis pod requisitas. Et ecce quomodo acta ordinatoris (scheduler log) apparent cum pod `cronjob-cron-events-1573793820-xt6q9` curritur (haec informatio in actis ordinatoris apparebit si gradum actae 10 in argumentis mandati initialis `--v=10` constitueris):
lignum
I1115 07:57:21.637791 1 scheduling_queue.go:908] About to try and schedule pod nxs-stage/cronjob-cron-events-1573793820-xt6q9
I1115 07:57:21.637804 1 scheduler.go:453] Attempting to schedule pod: nxs-stage/cronjob-cron-events-1573793820-xt6q9
I1115 07:57:21.638285 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s5 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638300 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s6 is allowed, Node is running only 20 out of 110 Pods.
I1115 07:57:21.638322 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s3 is allowed, Node is running only 20 out of 110 Pods.
I1115 07:57:21.638322 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s4 is allowed, Node is running only 17 out of 110 Pods.
I1115 07:57:21.638334 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s10 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638365 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s12 is allowed, Node is running only 9 out of 110 Pods.
I1115 07:57:21.638334 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s11 is allowed, Node is running only 11 out of 110 Pods.
I1115 07:57:21.638385 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s1 is allowed, Node is running only 19 out of 110 Pods.
I1115 07:57:21.638402 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s2 is allowed, Node is running only 21 out of 110 Pods.
I1115 07:57:21.638383 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s9 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638335 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s8 is allowed, Node is running only 18 out of 110 Pods.
I1115 07:57:21.638408 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s13 is allowed, Node is running only 8 out of 110 Pods.
I1115 07:57:21.638478 1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s10 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638505 1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s8 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638577 1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s9 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638583 1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s7 is allowed, Node is running only 25 out of 110 Pods.
I1115 07:57:21.638932 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: BalancedResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 2343 millicores 9640186880 memory bytes, score 9
I1115 07:57:21.638946 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: LeastResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 2343 millicores 9640186880 memory bytes, score 8
I1115 07:57:21.638961 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: BalancedResourceAllocation, capacity 39900 millicores 66620170240 memory bytes, total request 4107 millicores 11307422720 memory bytes, score 9
I1115 07:57:21.638971 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: BalancedResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 5847 millicores 24333637120 memory bytes, score 7
I1115 07:57:21.638975 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: LeastResourceAllocation, capacity 39900 millicores 66620170240 memory bytes, total request 4107 millicores 11307422720 memory bytes, score 8
I1115 07:57:21.638990 1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: LeastResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 5847 millicores 24333637120 memory bytes, score 7
I1115 07:57:21.639022 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639030 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639034 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639041 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639053 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639059 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639061 1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639063 1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639073 1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639077 1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639085 1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639088 1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639103 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639109 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639114 1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639127 1 generic_scheduler.go:781] Host nxs-k8s-s10 => Score 100037
I1115 07:57:21.639150 1 generic_scheduler.go:781] Host nxs-k8s-s8 => Score 100034
I1115 07:57:21.639154 1 generic_scheduler.go:781] Host nxs-k8s-s9 => Score 100037
I1115 07:57:21.639267 1 scheduler_binder.go:269] AssumePodVolumes for pod "nxs-stage/cronjob-cron-events-1573793820-xt6q9", node "nxs-k8s-s10"
I1115 07:57:21.639286 1 scheduler_binder.go:279] AssumePodVolumes for pod "nxs-stage/cronjob-cron-events-1573793820-xt6q9", node "nxs-k8s-s10": all PVCs bound and nothing to do
I1115 07:57:21.639333 1 factory.go:733] Attempting to bind cronjob-cron-events-1573793820-xt6q9 to nxs-k8s-s10Hic videmus ordinatorem initialiter filtrare et indicem trium nodorum generare in quibus currere potest (nxs-k8s-s8, nxs-k8s-s9, nxs-k8s-s10). Deinde puncta pro singulis his nodis computat secundum plures parametros (inter quos BalancedResourceAllocation et LeastResourceAllocation) ut nodum aptissimum determinet. Tandem, "pod" in nodo cum punctis maximis ordinatur (hic, duo nodi eundem punctum 100037 habent, ergo qui temere electus est — nxs-k8s-s10 — eligitur).
conclusio,Si nodus "pods" (capsulae) sine finibus statutis currentes habet, k8s hoc tractabit quasi nullae omnino "pods" in nodo currerent, quod ad consumptionem opum attinet. Ergo, si "pod" habes cum processu opibus avido (exempli gratia, wowza) et nulli fines ei statuti sunt, hic "pod" omnes opes nodi consumere potest. Attamen, k8s hunc nodum exoneratum iudicat et eadem puncta classificationis (praesertim, quod ad opes praesto attinet) ac nodo sine "pods" currentibus tribuet, quod tandem ad inaequalem distributionem oneris inter nodos ducere potest.
Eiectio capsulae
Ut scis, cuique pod una ex tribus classibus QoS assignatur:
- garantitum — assignatur cum petitio et limes pro singulis receptaculis in pod pro memoria et CPU constituuntur, et hi valores congruere debent
- burstable — saltem unum receptaculum in "pod" petitionem et limitem habet, et petitio < limes
- optimus conatus - cum nullum receptaculum in capsula opibus limitatum est
Cum nodus inopiam opum (disci, memoriae) experitur, kubelet incipit ordinare et eicere pods utens algorithmo specifico qui prioritatem pod et classem QoS in rationem habet. Exempli gratia, si pod RAM-aliatum est, puncta secundum classem QoS hoc modo tribuuntur:
- Back Guarantee: -998
- Optimus Conatus: 1000
- Burstable: min(max(2, 1000 - (1000 * memoriaeRequestBytes) / machinaeMemoriaeCapacitasBytes), 999)
Id est, eadem prioritate, kubelet primum pods cum optima classe QoS e nodo eiciet.
conclusio,Si probabilitatem ejectionis capsulae requisitae e nodo, si opes inopia accidant, minuere vis, tum una cum prioritate, etiam curam habere debes de petitione/limite pro ea constituendo.
Mechanismus Automatice Scalabilis Horizontalis pro Capsulis Applicationis (HPA)
Cum munus sit numerum podorum (vel "pods") automatice augere et minuere secundum usum opum (systema - CPU/RAM vel usorem - rps), talis entitas k8s ut... HPA (Autoscalator Pod Horizontalis). Algorithmus est ut sequitur:
- Lectiones praesentes opis monitoratae (currentMetricValue) determinantur.
- Valores desiderati pro opibus (desiredMetricValue) determinantur, qui pro opibus systematis per petitionem specificantur.
- Numerus praesens replicationum determinatur (currentReplicas)
- Formula sequens numerum desideratum replicationum (desiredReplicas) computat.
ExemplaDesiderata = [ ExemplaPraesens * ( ValorMetricaPraesens / ValorMetricaDesideratus )]
Hoc in casu, scalatio non fiet cum coefficiens (valorMetricusCurrens / valorMetricusDesiratus) prope 1 est (tamen errorem permissum ipsi constituere possumus; per default 0.1 est).
Consideremus operationem HPA utens exemplo applicationis "app-test" (quae "Deployment" appellatur), ubi necesse est numerum replicarum secundum usum CPU mutare:
Manifestum applicationis
kind: Deployment apiVersion: apps/v1beta2 metadata: name: app-test spec: selector: matchLabels: app: app-test replicas: 2 template: metadata: labels: app: app-test spec: containers: - name: nginx image: registry.nixys.ru/generic-images/nginx imagePullPolicy: Always resources: requests: cpu: 60m ports: - name: http containerPort: 80 - name: nginx-exporter image: nginx/nginx-prometheus-exporter resources: requests: cpu: 30m ports: - name: nginx-exporter containerPort: 9113 args: - -nginx.scrape-uri - http://127.0.0.1:80/nginx-statusId est, videmus pod cum applicatione initialiter in duobus exemplaribus emitti, quorum unumquodque duo receptacula "nginx" et "nginx-exporter" continet, pro quibus utroque... petitiones pro CPU.
Manifestum HPA
apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: app-test-hpa spec: maxReplicas: 10 minReplicas: 2 scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: app-test metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 30Id est, hpa creavimus quod probationem applicationis in distributione observabit et numerum podorum cum applicatione secundum indicem CPU moderabitur (expectamus pod 30% CPU quam petit consumere debere), dum numerus replicationum in intervallo 2-10 est.
Nunc, inspiciamus quomodo HPA operatur si onus uni ex capsularum imponimus:
# kubectl top pod NAME CPU(cores) MEMORY(bytes) app-test-78559f8f44-pgs58 101m 243Mi app-test-78559f8f44-cj4jz 4m 240Mi
Itaque haec habemus:
- ValorMetricusDesiratus — secundum nostras configurationes hpa, est 30%
- Valor currens (currentMetricValue) - ad calculandum, moderator-administrator valorem medium consumptionis opum in % computat, id est, haec facit:
- Valores absolutos metricorum podorum ex servo metricorum accipit, i.e. 101m et 4m.
- Valorem absolutum medium computat, i.e. (101m + 4m) / 2 = 53m.
- Valorem absolutum pro desiderata consumptione opum obtinet (ad hoc, petitiones omnium receptaculorum summantur) 60m + 30m = 90m
- Mediam partem consumptionis CPU pro petitione pod computat, i.e. 53m / 90m * 100% = 59%
Nunc omnia habemus quae necessaria sunt ad determinandum utrum numerum replicationum mutare debeamus. Ad hoc faciendum, coefficiens computamus:
ratio = 59% / 30% = 1.96
Id est, numerus replicationum ~duplo augeri debet et ad [2 * 1.96] = 4 pervenire.
conclusioni, Ut videre potes, ut hic mechanismus operetur, condicio necessaria est praesentia petitionum pro omnibus receptaculis in pod observato.
Scalator Automaticus Gregum
Ad mitigandum impulsum negativum in systema durante impetu oneris, saepe non sufficit HPA configuratum habere. Exempli gratia, secundum configurationes moderatoris HPA, numerus replicationum duplicandus est, sed nodi non habent facultates praesto ad tot pods exsequendas (id est, nodus facultates petitas pod petitionum praebere non potest), et hi pods statum "Pendens" ingrediuntur.
Hoc in casu, si praebitor IaaS/PaaS correspondentem habet (e.g. GKE/GCE, AKS, EKS, etc.), instrumentum simile... Autoscalator NodiHoc tibi permittit numerum maximum et minimum nodorum in grege constituere et numerum nodorum praesentem automatice accommodare (per API provisoris nubis ad nodum ordinandum/delendum accedendo) cum inopia opum in grege est et "pods" (capsulae) ordinari non possunt (in statu "Pendentis" sunt).
conclusioni, Ad autoscalationem nodorum efficiendam, necesse est petitiones in receptaculis pod specificare ut K8S recte onus nodorum aestimare possit et proinde nuntiare nullas opes in grege adesse ad pod proximum incipiendum.
conclusio,
Notandum est limites opum receptaculi constituere non esse requisitum necessarium ad prosperam applicationis initium, sed tamen bonum consilium esse id facere ob causas sequentes:
- Ad accuratiorem operationem ordinatoris quod ad librationem oneris inter nodos k8s attinet,
- Ad probabilitatem eventus "eiectionis capsularum" (vel "ejectionis capsularum") minuendam
- Ut automatismus applicationis horizontalis pod (HPA) functionet
- Ad amplificationem automaticam nodorum horizontalium (Ampliationem Automaticam Gregum) in praebitoribus nubium activandam
Legunt etiam alia capitula in nostro diario:
Source: www.habr.com
