Ukusetshenziswa okuwusizo kwe-ELK. Isetha i-logstash

Isingeniso

Ngenkathi sisebenzisa olunye uhlelo, besibhekene nesidingo sokucubungula inani elikhulu lamalogi ahlukene. I-ELK ikhethwe njengethuluzi. Lesi sihloko sizoxoxa ngolwazi lwethu ekusetheni lesi sitaki.

Asizibeki umgomo wokuchaza wonke amakhono ayo, kodwa sifuna ukugxila ngokukhethekile ekuxazululeni izinkinga ezingokoqobo. Lokhu kungenxa yokuthi noma kunenani elikhulu lemibhalo nezithombe esezilungile, ziningi izingibe, okungenani sizitholile.

Sikhiphe isitaki nge-docker-compose. Ngaphezu kwalokho, sibe ne-docker-compose.yml ebhalwe kahle, eyasivumela ukuthi siphakamise isitaki cishe ngaphandle kwezinkinga. Futhi kubonakala ngathi ukunqoba kwase kusondele kakade, manje sizoyilungisa kancane ukuze ihambisane nezidingo zethu futhi yilokho.

Ngeshwa, umzamo wokumisa isistimu ukuthi yamukele futhi icubungule amalogi kusuka kuhlelo lwethu lokusebenza awuzange uphumelele ngokushesha. Ngakho-ke, sinqume ukuthi kufanelekile ukutadisha ingxenye ngayinye ngokwehlukana, bese sibuyela ekuxhumekeni kwabo.

Ngakho-ke, siqale nge-logstash.

Imvelo, ukusetshenziswa, ukusebenzisa i-Logstash esitsheni

Ukuze sisetshenziswe sisebenzisa i-docker-compose; izivivinyo ezichazwe lapha zenziwa ku-MacOS naku-Ubuntu 18.0.4.

Isithombe se-logstash esibhaliswe ku-docker-compose.yml yethu yasekuqaleni ithi docker.elastic.co/logstash/logstash:6.3.2

Sizoyisebenzisela ukuhlola.

Sibhale i-docker-compose.yml ehlukile ukuze siqalise i-logstash. Vele, bekungenzeka ukwethula isithombe kusuka kumugqa womyalo, kepha besixazulula inkinga ethile, lapho siqhuba yonke into kusuka ku-docker-compose.

Kafushane mayelana namafayela okusetha

Ngokulandelayo kusuka encazelweni, i-logstash ingasebenza noma kusiteshi esisodwa, lapho idinga ukudlula ifayela *.conf, noma iziteshi ezimbalwa, lapho idinga ukudlulisa ifayela le-pipelines.yml, yona, , izoxhuma kumafayela .conf wesiteshi ngasinye.
Sathatha indlela yesibili. Kithina kwakubonakala kuyinto evamile futhi ikhula. Ngakho-ke, sidale i-pipelines.yml, futhi senza uhla lwemibhalo lwamapayipi lapho sizobeka khona amafayela we-.conf wesiteshi ngasinye.

Ngaphakathi kwesiqukathi kukhona elinye ifayela lokumisa - logstash.yml. Asiyithinti, siyisebenzisa njengoba injalo.

Ngakho, uhlaka lwethu lwemibhalo:

Ukusetshenziswa okuwusizo kwe-ELK. Isetha i-logstash

Ukwamukela idatha yokufaka, okwamanje sicabanga ukuthi lena yi-tcp ku-port 5046, futhi ngokukhiphayo sizosebenzisa i-stdout.

Nakhu ukucushwa okulula kokwethulwa kokuqala. Ngoba umsebenzi wokuqala uwukuqalisa.

Ngakho, sinale docker-compose.yml

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      	- elk
    ports:
      	- 5046:5046
    volumes:
      	- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
	- ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Sibonani lapha?

  1. Amanethiwekhi namavolumu athathwe ku-docker-compose.yml yasekuqaleni (lapho kuqaliswe khona sonke isitaki) futhi ngicabanga ukuthi asithinti kakhulu isithombe sisonke lapha.
  2. Sakha isevisi eyodwa ye-logstash kusuka ku-docker.elastic.co/logstash/logstash:6.3.2 isithombe futhi siyiqambe ngokuthi logstash_one_channel.
  3. Sidlulisela phambili imbobo engu-5046 ngaphakathi kwesiqukathi, embobeni efanayo yangaphakathi.
  4. Senza imephu yefayela lethu lokumisa ipayipi ./config/pipelines.yml efayeleni /usr/share/logstash/config/pipelines.yml ngaphakathi kwesiqukathi, lapho i-logstash izolithatha futhi ilenze lifundeke kuphela, uma kwenzeka.
  5. Senza imephu ./config/pipelines directory, lapho sinamafayela anezilungiselelo zesiteshi, kuhlu lwemibhalo /usr/share/logstash/config/pipelines futhi silwenza lufundeke kuphela.

Ukusetshenziswa okuwusizo kwe-ELK. Isetha i-logstash

Ifayela le-Pipelines.yml

- pipeline.id: HABR
  pipeline.workers: 1
  pipeline.batch.size: 1
  path.config: "./config/pipelines/habr_pipeline.conf"

Isiteshi esisodwa esinesihlonzi se-HABR kanye nendlela eya kufayela laso lokucushwa kuchazwe lapha.

Futhi ekugcineni ifayela elithi “./config/pipelines/habr_pipeline.conf”

input {
  tcp {
    port => "5046"
   }
  }
filter {
  mutate {
    add_field => [ "habra_field", "Hello Habr" ]
    }
  }
output {
  stdout {
      
    }
  }

Masingangeni encazelweni yayo okwamanje, asizame ukuyiqhuba:

docker-compose up

Sibonani?

Isiqukathi sesiqalile. Singahlola ukusebenza kwayo:

echo '13123123123123123123123213123213' | nc localhost 5046

Futhi sibona impendulo kukhonsoli yesiqukathi:

Ukusetshenziswa okuwusizo kwe-ELK. Isetha i-logstash

Kodwa ngesikhathi esifanayo, sibona futhi:

logstash_one_channel | [2019-04-29T11:28:59,790][ERROR][logstash.licensechecker.licensereader] Ayikwazi ukubuyisa ulwazi lwelayisense kuseva yelayisensi {:message=>“Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore ::ResolutionFailure] elasticsearch", ...

logstash_one_channel | [2019-04-29T11:28:59,894][INFO ][logstash.pipeline ] Ipayipi iqale ngempumelelo {:pipeline_id=>".monitoring-logstash", :thread=>"# "}

logstash_one_channel | [2019-04-29T11:28:59,988][INFO ][logstash.agent ] Amapayipi agijima {:count=>2, :running_pipelines=>[:HABR, :".monitoring-logstash"], :non_running_pipelines=>[ ]}
logstash_one_channel | [2019-04-29T11:29:00,015][ERROR][logstash.inputs.metrics] I-X-Pack ifakwe ku-Logstash kodwa hhayi ku-Elasticsearch. Sicela ufake i-X-Pack ku-Elasticsearch ukuze usebenzise isici sokuqapha. Ezinye izici zingase zibe khona.
logstash_one_channel | [2019-04-29T11:29:00,526][INFO ][logstash.agent ] Iqale ngempumelelo i-Logstash API endpoint {:port=>9600}
logstash_one_channel | [2019-04-29T11:29:04,478][INFO ][logstash.outputs.elasticsearch] Isebenzisa ukuhlolwa kwezempilo ukuze ibone ukuthi ingabe uxhumano lwe-Elasticsearch luyasebenza yini {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,487][XWAYISA ][logstash.outputs.elasticsearch] Uzame ukuvusa ukuxhumeka kusibonelo se-ES esifile, kodwa uthole iphutha. {:url=>“umabhebhana:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][ManticoreFail::Reso] elasticsearch"}
logstash_one_channel | [2019-04-29T11:29:04,704][INFO ][logstash.licensechecker.licensereader] Isebenzisa ukuhlolwa kwezempilo ukuze ibone ukuthi ingabe uxhumano lwe-Elasticsearch luyasebenza yini {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,710][XWAYISO ][logstash.licensechecker.licensereader] Uzame ukuvusa ukuxhumeka kusibonelo se-ES esifile, kodwa uthole iphutha. {:url=>“umabhebhana:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][ManticoreFail::Reso] elasticsearch"}

Futhi i-log yethu ikhuphuka ngaso sonke isikhathi.

Lapha ngiqokomise ngokuluhlaza umlayezo ipayipi elethulwe ngempumelelo, ngokubomvu umyalezo wephutha futhi ngokuphuzi umyalezo mayelana nomzamo wokuxhumana. umabhebhana: 9200.
Lokhu kwenzeka ngoba i-logstash.conf, efakwe esithombeni, iqukethe isheke lokutholakala kwe-elasticsearch. Phela, i-logstash ithatha ukuthi isebenza njengengxenye yesitaki se-Elk, kodwa siyihlukanise.

Kungenzeka ukusebenza, kodwa akulula.

Isixazululo ukukhubaza lokhu kuhlola usebenzisa i-XPACK_MONITORING_ENABLED eguquguqukayo yemvelo.

Masenze ushintsho ku-docker-compose.yml bese siyiqhuba futhi:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Manje, konke kuhamba kahle. Isiqukathi silungele ukuhlolwa.

Singaphinda sithayiphe kukhonsoli elandelayo:

echo '13123123123123123123123213123213' | nc localhost 5046

Futhi bona:

logstash_one_channel | {
logstash_one_channel |         "message" => "13123123123123123123123213123213",
logstash_one_channel |      "@timestamp" => 2019-04-29T11:43:44.582Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |            "host" => "gateway",
logstash_one_channel |            "port" => 49418
logstash_one_channel | }

Ukusebenza ngaphakathi kwesiteshi esisodwa

Ngakho sethula. Manje ungakwazi ngempela ukuthatha isikhathi ukumisa logstash ngokwayo. Masingalithinti ifayela le-pipelines.yml okwamanje, ake sibone ukuthi yini esingayithola ngokusebenzisa isiteshi esisodwa.

Mangisho ukuthi isimiso esijwayelekile sokusebenza nefayela lokumisa isiteshi sichazwe kahle encwadini esemthethweni, lapha lapha
Uma ufuna ukufunda ngesiRashiya, sisebenzise lesi isihloko(kodwa i-syntax yombuzo indala, sidinga ukukucabangela lokhu).

Asihambe ngokulandelana ukusuka esigabeni Sokokufaka. Sesivele sawubona umsebenzi ku-TCP. Yini enye engaba mnandi lapha?

Hlola imilayezo usebenzisa ukushaya kwenhliziyo

Kukhona ithuba elithakazelisayo elinjalo lokukhiqiza imilayezo yokuhlola ezenzakalelayo.
Ukuze wenze lokhu, udinga ukunika amandla i-plugin yenhliziyo engxenyeni yokufaka.

input {
  heartbeat {
    message => "HeartBeat!"
   }
  } 

Ivule, qala ukwamukela kanye ngomzuzu

logstash_one_channel | {
logstash_one_channel |      "@timestamp" => 2019-04-29T13:52:04.567Z,
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "HeartBeat!",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "host" => "a0667e5c57ec"
logstash_one_channel | }

Uma sifuna ukwamukela kaningi, sidinga ukungeza ipharamitha yesikhawu.
Lena yindlela esizothola ngayo umlayezo njalo ngemizuzwana eyi-10.

input {
  heartbeat {
    message => "HeartBeat!"
    interval => 10
   }
  }

Ibuyisa idatha efayelini

Siphinde sanquma ukubheka imodi yefayela. Uma isebenza kahle ngefayela, mhlawumbe akekho odingekayo, okungenani ukusetshenziswa kwendawo.

Ngokusho kwencazelo, imodi yokusebenza kufanele ifane nomsila -f, i.e. ifunda imigqa emisha noma, njengenketho, ifunde lonke ifayela.

Ngakho-ke esifuna ukukuthola:

  1. Sifuna ukuthola imigqa enezelwe efayeleni elilodwa lokungena.
  2. Sifuna ukuthola idatha ebhalwe kumafayela elogi amaningana, kuyilapho sikwazi ukuhlukanisa okutholwa lapho.
  3. Sifuna ukwenza isiqiniseko sokuthi uma i-logstash iqalwa kabusha, ayitholi le datha futhi.
  4. Sifuna ukuhlola ukuthi uma i-logstash ivaliwe, futhi idatha iqhubeka ibhalwa kumafayela, lapho-ke siyiqhuba, sizothola le datha.

Ukwenza isilingo, ake sengeze omunye umugqa ku-docker-compose.yml, sivula uhla lwemibhalo esibeka kulo amafayela.

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input

Futhi ushintshe ingxenye yokokufaka kokuthi habr_pipeline.conf

input {
  file {
    path => "/usr/share/logstash/input/*.log"
   }
  }

Ake siqale:

docker-compose up

Ukwakha nokubhala amafayela welogi sizosebenzisa umyalo:


echo '1' >> logs/number1.log

{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:53.876Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Yebo, kuyasebenza!

Ngesikhathi esifanayo, siyabona ukuthi sengeze ngokuzenzakalelayo inkambu yendlela. Lokhu kusho ukuthi esikhathini esizayo, sizokwazi ukuhlunga amarekhodi ngakho.

Ake sizame futhi:

echo '2' >> logs/number1.log

{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:59.906Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "2",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Futhi manje kwelinye ifayela:

 echo '1' >> logs/number2.log

{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:29:26.061Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log"
logstash_one_channel | }

Kuhle! Ifayela lithathwe, indlela ichazwe kahle, konke kuhamba kahle.

Misa i-logstash bese uqala futhi. Ake silinde. Kuthuleke. Labo. Asisawatholi la marekhodi futhi.

Futhi manje isilingo esinesibindi kakhulu.

Faka i-logstash bese uqhuba:

echo '3' >> logs/number2.log
echo '4' >> logs/number1.log

Qalisa i-logstash futhi ubone:

logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "3",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.589Z
logstash_one_channel | }
logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "4",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.856Z
logstash_one_channel | }

Hooray! Kwacoshwa yonke into.

Kodwa kumele sikuxwayise ngalokhu okulandelayo. Uma isiqukathi se-logstash sisusiwe (i-docker stop logstash_one_channel && docker rm logstash_one_channel), akukho lutho oluzolandwa. Indawo yefayela kuze kufike lapho ifundwe khona yayigcinwe ngaphakathi kwesitsha. Uma uyiqalisa kusukela ekuqaleni, izokwamukela imigqa emisha kuphela.

Ukufunda amafayela akhona

Ake sithi sethula i-logstash okokuqala ngqa, kodwa sesivele sinawo amalogi futhi singathanda ukuwacubungula.
Uma sisebenzisa i-logstash ngesigaba sokufaka esisisebenzise ngenhla, ngeke sithole lutho. Imigqa emisha kuphela ezocutshungulwa yi-logstash.

Ukuze imigqa esuka kumafayela akhona idonswe phezulu, kufanele wengeze umugqa owengeziwe engxenyeni yokufaka:

input {
  file {
    start_position => "beginning"
    path => "/usr/share/logstash/input/*.log"
   }
  }

Ngaphezu kwalokho, kune-nuance: lokhu kuthinta kuphela amafayela amasha i-logstash engakawaboni. Kumafayela afanayo abesevele esesigabeni sokubuka se-logstash, isivele ikhumbule usayizi wawo futhi manje izothatha okufakiwe okusha kuwo.

Ake sime lapha futhi sifunde ingxenye yokufaka. Kusenezinketho eziningi, kodwa lokho kwanele kithi kwezinye izivivinyo okwamanje.

Umzila kanye Nokuguqulwa Kwedatha

Ake sizame ukuxazulula le nkinga elandelayo, ake sithi sinemiyalezo evela esiteshini esisodwa, eminye yayo inolwazi, kanti eminye imiyalezo yamaphutha. Ayahluka ngomaka. Ezinye ZINGULWAZI, ezinye YIPHUTHA.

Sidinga ukuwahlukanisa ekuphumeni. Labo. Sibhala imilayezo yolwazi kwesinye isiteshi, bese sibhala imilayezo yamaphutha kwesinye.

Ukuze wenze lokhu, suka engxenyeni yokokufaka ukuze uhlunge futhi ukhiphe.

Sisebenzisa isigaba sokuhlunga, sizohlaziya umlayezo ongenayo, sithole i-hashi (amapheya okhiye-inani) kuwo, esingavele sisebenze nawo, i.e. qaqa ngokwemibandela. Futhi esigabeni sokuphumayo, sizokhetha imilayezo futhi sithumele ngamunye esiteshini sawo.

Ukuhlaziya umlayezo nge-grok

Ukuze uhlukanise izintambo zombhalo futhi uthole isethi yezinkambu kuzo, kune-plugin ekhethekile esigabeni sokuhlunga - grok.

Ngaphandle kokuzibekela umgomo wokunikeza incazelo enemininingwane ngakho lapha (ngalokhu ngibhekisela kukho imibhalo esemthethweni), ngizonikeza isibonelo sami esilula.

Ukuze wenze lokhu, udinga ukunquma ngefomethi yezintambo zokufaka. Nginawo kanje:

1 ULWAZI umyalezo1
2 Umlayezo IPHUTHA2

Labo. Inkomba iza kuqala, bese kuba ULWAZI/IPHUTHA, bese kuba igama elithile ngaphandle kwezikhala.
Akunzima, kodwa kwanele ukuqonda isimiso sokusebenza.

Ngakho-ke, esigabeni sokuhlunga se-plugin ye-grok, kufanele sichaze iphethini yokuhlaziya izintambo zethu.

Kuzobukeka kanje:

filter {
  grok {
    match => { "message" => ["%{INT:message_id} %{LOGLEVEL:message_type} %{WORD:message_text}"] }
   }
  } 

Empeleni inkulumo evamile. Amaphethini enziwe ngomumo asetshenziswa, njenge-INT, LOGLEVEL, WORD. Incazelo yabo, kanye namanye amaphethini, ingatholakala lapha lapha

Manje, uma sidlula kulesi sihlungi, iyunithi yezinhlamvu yethu izophenduka i-hashi yezinkambu ezintathu: message_id, message_type, message_text.

Azoboniswa esigabeni sokuphumayo.

Ukudlulisela imiyalezo engxenyeni yokuphumayo usebenzisa umyalo othi if

Esigabeni sokuphumayo, njengoba sikhumbula, sasizohlukanisa imilayezo ibe imifudlana emibili. Ezinye - okuyi-iNFO, zizokhishwa kukhonsoli, futhi ngamaphutha, sizokhipha ifayela.

Siyihlukanisa kanjani le miyalezo? Isimo senkinga sesivele sisiphakamisa isixazululo - phela sesivele sinenkambu yohlobo lwe-message_type, engathatha amanani amabili kuphela: INFO kanye ne-ERROR. Kungalesi sisekelo lapho sizokwenza ukukhetha sisebenzisa isitatimende uma.

if [message_type] == "ERROR" {
        # Здесь выводим в файл
       } else
     {
      # Здесь выводим в stdout
    }

Incazelo yokusebenza nezinkambu nama-opharetha ingatholakala kulesi sigaba imanuwali esemthethweni.

Manje, mayelana nesiphetho sangempela ngokwaso.

Okukhiphayo kwekhonsoli, yonke into icacile lapha - stdout {}

Kodwa okukhiphayo efayeleni - khumbula ukuthi sisebenzisa konke lokhu esitsheni futhi ukuze ifayela esibhala kulo umphumela lifinyeleleke ngaphandle, sidinga ukuvula lolu hlu lwemibhalo ku-docker-compose.yml.

Inani:

Ingxenye yokukhiphayo yefayela lethu ibukeka kanje:


output {
  if [message_type] == "ERROR" {
    file {
          path => "/usr/share/logstash/output/test.log"
          codec => line { format => "custom format: %{message}"}
         }
    } else
     {stdout {
             }
     }
  }

Ku-docker-compose.yml sengeza enye ivolumu yokuphumayo:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input
      - ./output:/usr/share/logstash/output

Siyayethula, siyizame, futhi sibone ukuhlukaniswa kube imifudlana emibili.

Source: www.habr.com

Engeza amazwana