Pulogalamu ya ProHoster > Blog > Ulamuliro > Kugwiritsa ntchito ELK. Kupanga logstash

Kugwiritsa ntchito ELK. Kupanga logstash

Mau oyamba

Pamene tikugwiritsa ntchito dongosolo lina, tinayang'anizana ndi kufunikira kokonza zipika zambiri zosiyanasiyana. ELK idasankhidwa kukhala chida. Nkhaniyi ifotokoza zomwe takumana nazo pokhazikitsa stack iyi.

Sitikhala ndi cholinga chofotokozera mphamvu zake zonse, koma tikufuna kuyang'ana kwambiri kuthetsa mavuto. Izi ndichifukwa choti ngakhale pali zolembedwa zambiri komanso zithunzi zopangidwa kale, pali misampha yambiri, mwina tidazipeza.

Tidatumiza stack kudzera pa docker-compose. Komanso, tinali ndi docker-compose.yml yolembedwa bwino, yomwe inatilola kukweza stack pafupifupi popanda mavuto. Ndipo zinkawoneka kwa ife kuti chigonjetso chinali pafupi kale, tsopano tidzasintha pang'ono kuti tigwirizane ndi zosowa zathu ndipo ndizomwezo.

Tsoka ilo, kuyesa kukonza dongosolo kuti mulandire ndi kukonza zipika kuchokera ku pulogalamu yathu sikunapambane nthawi yomweyo. Choncho, tinaganiza kuti ndi bwino kuphunzira chigawo chilichonse padera, ndiyeno kubwerera ku malumikizidwe awo.

Kotero, tinayamba ndi logstash.

Chilengedwe, kutumiza, kuyendetsa Logstash mu chidebe

Potumiza timagwiritsa ntchito docker-compose; zoyeserera zomwe zafotokozedwa apa zidachitika pa MacOS ndi Ubuntu 18.0.4.

Chithunzi cha logstash chomwe chidalembetsedwa mu docker-compose.yml yathu yoyambirira ndi docker.elastic.co/logstash/logstash:6.3.2

Tidzagwiritsa ntchito poyesera.

Tinalemba zosiyana docker-compose.yml kuyendetsa logstash. Inde, zinali zotheka kukhazikitsa chithunzicho kuchokera pamzere wolamula, koma tinali kuthetsa vuto linalake, kumene timayendetsa chirichonse kuchokera ku docker-compose.

Mwachidule za mafayilo osinthira

Motsatira ndondomekoyi, logstash ikhoza kuyendetsedwa kapena njira imodzi, momwemo iyenera kudutsa * .conf file, kapena njira zingapo, zomwe ziyenera kudutsa pipelines.yml file, yomwe, , idzalumikizana ndi mafayilo .conf pa tchanelo chilichonse.
Tinatenga njira yachiwiri. Zinkawoneka kwa ife zambiri padziko lonse komanso scalable. Chifukwa chake, tidapanga pipelines.yml, ndikupanga chikwatu cha mapaipi momwe tidzayika mafayilo a .conf panjira iliyonse.

Mkati mwa chidebecho muli fayilo ina yosinthira - logstash.yml. Sitichikhudza, timachigwiritsa ntchito momwe chilili.

Chifukwa chake, dongosolo lathu lachikwatu:

Kugwiritsa ntchito ELK. Kupanga logstash

Kuti tilandire deta yolowera, pakadali pano tikuganiza kuti iyi ndi tcp pa port 5046, ndipo pazotulutsa tidzagwiritsa ntchito stdout.

Pano pali kasinthidwe kosavuta koyambitsa koyamba. Chifukwa ntchito yoyambira ndiyo kukhazikitsa.

Chifukwa chake, tili ndi docker-compose.yml iyi

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      	- elk
    ports:
      	- 5046:5046
    volumes:
      	- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
	- ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Kodi tikuwona chiyani apa?

  1. Maukonde ndi ma voliyumu adatengedwa kuchokera ku docker-compose.yml yoyambirira (yomwe stack yonse idakhazikitsidwa) ndipo ndikuganiza kuti sizikhudza kwambiri chithunzi chonse pano.
  2. Timapanga mautumiki amodzi a logstash kuchokera pa docker.elastic.co/logstash/logstash:6.3.2 chithunzi ndikuchitcha logstash_one_channel.
  3. Timatumiza doko 5046 mkati mwa chidebecho, kupita ku doko lamkati lomwelo.
  4. Timajambula fayilo yathu yokonza chitoliro ./config/pipelines.yml ku fayilo /usr/share/logstash/config/pipelines.yml mkati mwa chidebecho, kumene logstash idzatenga ndikuyiwerenga-yokha, pokhapokha.
  5. Timayika ./config/pipelines directory, komwe tili ndi mafayilo okhala ndi makonda, mu /usr/share/logstash/config/pipelines directory ndikupangitsanso kuti iwerenge.

Kugwiritsa ntchito ELK. Kupanga logstash

Pipelines.yml fayilo

- pipeline.id: HABR
  pipeline.workers: 1
  pipeline.batch.size: 1
  path.config: "./config/pipelines/habr_pipeline.conf"

Njira imodzi yokhala ndi chizindikiritso cha HABR ndi njira yopita ku fayilo yake yosinthira ikufotokozedwa apa.

Ndipo pamapeto pake fayilo "./config/pipelines/habr_pipeline.conf"

input {
  tcp {
    port => "5046"
   }
  }
filter {
  mutate {
    add_field => [ "habra_field", "Hello Habr" ]
    }
  }
output {
  stdout {
      
    }
  }

Tisapite kukufotokozera kwake pakadali pano, tiyeni tiyese kuyendetsa:

docker-compose up

Kodi tikuwona chiyani?

Chidebe chayamba. Tikhoza kuyang'ana ntchito yake:

echo '13123123123123123123123213123213' | nc localhost 5046

Ndipo tikuwona yankho mu chotengera chotengera:

Kugwiritsa ntchito ELK. Kupanga logstash

Koma nthawi yomweyo, tikuwonanso:

logstash_one_channel | [2019-04-29T11:28:59,790][ERROR][logstash.licensechecker.licensereader] Takanika kupeza zidziwitso za laisensi kuchokera pa seva ya laisensi {:message=>β€œElasticsearch Unreachable: [http://elasticsearch:9200/][Manticore ::ResolutionFailure] elasticsearch", ...

logstash_one_channel | [2019-04-29T11:28:59,894][INFO ][logstash.pipeline ] Pipeline inayamba bwino {:pipeline_id=>".monitoring-logstash", :thread=>"# "}

logstash_one_channel | [2019-04-29T11:28:59,988][INFO ][logstash.agent ] Mapaipi akuyenda {:count=>2, :running_pipelines=>[:HABR, :".monitoring-logstash"], :non_running_pipelines=>[ ]}
logstash_one_channel | [2019-04-29T11:29:00,015][ERROR][logstash.inputs.metrics] X-Pack yaikidwa pa Logstash koma osati pa Elasticsearch. Chonde ikani X-Pack pa Elasticsearch kuti mugwiritse ntchito zowunikira. Zina zitha kupezeka.
logstash_one_channel | [2019-04-29T11:29:00,526][INFO ][logstash.agent ] Anayambitsa bwino Logstash API endpoint {:port=>9600}
logstash_one_channel | [2019-04-29T11:29:04,478][INFO ][logstash.outputs.elasticsearch] Kuyang'ana zaumoyo kuti muwone ngati kulumikizana kwa Elasticsearch kukugwira ntchito {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,487][CHENJEZANI ][logstash.outputs.elasticsearch] Anayesa kuukitsa kulumikizidwa kwachitsanzo cha ES chakufa, koma adapeza cholakwika. {:url=>β€œelasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticoreil::Reso] elasticsearch"}
logstash_one_channel | [2019-04-29T11:29:04,704][INFO ][logstash.licensechecker.licensereader] Kuyang'ana zaumoyo kuti muwone ngati kulumikizana kwa Elasticsearch kukugwira ntchito {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,710][CHENJEZANI ][logstash.licensechecker.licensereader] Anayesa kuukitsa kulumikizidwa kwa ES yakufa, koma adapeza cholakwika. {:url=>β€œelasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticoreil::Reso] elasticsearch"}

Ndipo tsamba lathu limakula nthawi zonse.

Apa ndaunikira mobiriwira uthenga womwe payipi yakhazikitsa bwino, mofiira uthenga wolakwika komanso wachikasu uthenga wofuna kulumikizana. elasticsearch: 9200.
Izi zimachitika chifukwa logstash.conf, yomwe ili pachithunzichi, ili ndi cheke cha kupezeka kwa elasticsearch. Kupatula apo, logstash imaganiza kuti imagwira ntchito ngati gawo la Elk stack, koma tidayilekanitsa.

N'zotheka kugwira ntchito, koma sizothandiza.

Yankho ndikuyimitsa chekechi pogwiritsa ntchito XPACK_MONITORING_ENABLED zosintha zachilengedwe.

Tiyeni tisinthe ku docker-compose.yml ndikuyendetsanso:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Tsopano, chirichonse chiri bwino. Chidebecho chakonzeka kuyesa.

Titha kulembanso mu console yotsatira:

echo '13123123123123123123123213123213' | nc localhost 5046

Ndipo onani:

logstash_one_channel | {
logstash_one_channel |         "message" => "13123123123123123123123213123213",
logstash_one_channel |      "@timestamp" => 2019-04-29T11:43:44.582Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |            "host" => "gateway",
logstash_one_channel |            "port" => 49418
logstash_one_channel | }

Kugwira ntchito mkati mwa njira imodzi

Ndiye tinayambitsa. Tsopano mutha kutenga nthawi yokonza logstash yokha. Tisakhudze fayilo ya pipelines.yml pakadali pano, tiyeni tiwone zomwe tingapeze pogwira ntchito ndi njira imodzi.

Ndiyenera kunena kuti mfundo zonse zogwirira ntchito ndi fayilo ya kasinthidwe kanjira zafotokozedwa bwino m'buku lovomerezeka, apa apa
Ngati mukufuna kuwerenga mu Russian, tidagwiritsa ntchito iyi nkhani(koma mawu ofotokozera omwe alipo ndi akale, tiyenera kuganizira izi).

Tiyeni tipite motsatizana kuchokera pagawo la Input. Tawona kale ntchito pa TCP. Ndi chiyani chinanso chomwe chingasangalatse apa?

Yesani mauthenga pogwiritsa ntchito kugunda kwa mtima

Pali mwayi wosangalatsa wopanga mauthenga oyesera okha.
Kuti muchite izi, muyenera kuyatsa pulogalamu yowonjezera yamtima pagawo lolowera.

input {
  heartbeat {
    message => "HeartBeat!"
   }
  }

Yatsani, yambani kulandira kamodzi pa miniti 

logstash_one_channel | {
logstash_one_channel |      "@timestamp" => 2019-04-29T13:52:04.567Z,
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "HeartBeat!",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "host" => "a0667e5c57ec"
logstash_one_channel | }

Ngati tikufuna kulandira nthawi zambiri, tiyenera kuwonjezera parameter yapakati.
Umu ndi momwe tidzalandirira uthenga masekondi 10 aliwonse.

input {
  heartbeat {
    message => "HeartBeat!"
    interval => 10
   }
  }

Kubweza deta kuchokera mufayilo

Tinaganizanso kuyang'ana mawonekedwe a fayilo. Ngati zikuyenda bwino ndi fayilo, ndiye kuti palibe wothandizira omwe akufunika, makamaka kuti agwiritse ntchito kwanuko.

Malingana ndi kufotokozera, njira yogwiritsira ntchito iyenera kukhala yofanana ndi mchira -f, i.e. amawerenga mizere yatsopano kapena, ngati njira, amawerenga fayilo yonse.

Ndiye zomwe tikufuna kupeza:

  1. Tikufuna kulandira mizere yomwe yawonjezeredwa ku fayilo imodzi ya log.
  2. Tikufuna kulandira deta yomwe imalembedwa pamafayilo angapo a chipika, ndikutha kusiyanitsa zomwe zimalandiridwa kuchokera komwe.
  3. Tikufuna kuwonetsetsa kuti logstash ikayatsidwanso, silandilanso izi.
  4. Tikufuna kuwona kuti ngati logstash yazimitsidwa, ndipo deta ikupitiriza kulembedwa ku mafayilo, ndiye tikamayendetsa, tidzalandira deta iyi.

Kuti tichite kuyesako, tiyeni tiwonjezere mzere wina ku docker-compose.yml, ndikutsegula chikwatu chomwe timayikamo mafayilo.

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input

Ndipo sinthani gawo lolowera mu habr_pipeline.conf

input {
  file {
    path => "/usr/share/logstash/input/*.log"
   }
  }

Tiyeni tiyambe:

docker-compose up

Kuti tipange ndi kulemba mafayilo a log tidzagwiritsa ntchito lamulo:


echo '1' >> logs/number1.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:53.876Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Inde, zimagwira ntchito!

Nthawi yomweyo, tikuwona kuti tawonjezera gawo lanjira. Izi zikutanthauza kuti mtsogolomu, tidzatha kusefa zolemba ndi izo.

Tiyeni tiyesenso:

echo '2' >> logs/number1.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:59.906Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "2",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Ndipo tsopano ku fayilo ina:

 echo '1' >> logs/number2.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:29:26.061Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log"
logstash_one_channel | }

Zabwino! Fayilo idatengedwa, njira idanenedwa molondola, zonse zili bwino.

Imitsani logstash ndikuyambanso. Tiyeni tidikire. Chete. Iwo. Sitilandiranso zolemba izi.

Ndipo tsopano kuyesera molimba mtima kwambiri.

Ikani logstash ndikuchita:

echo '3' >> logs/number2.log
echo '4' >> logs/number1.log

Thamanganso logstash ndikuwona:

logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "3",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.589Z
logstash_one_channel | }
logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "4",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.856Z
logstash_one_channel | }

Uwu! Chirichonse chinatengedwa.

Koma tiyenera kukuchenjezani za izi. Ngati chidebe cha logstash chachotsedwa (docker stop logstash_one_channel && docker rm logstash_one_channel), ndiye kuti palibe chomwe chidzatengedwa. Malo a fayilo mpaka pomwe adawerengedwa adasungidwa mkati mwa chidebecho. Ngati mutayiyendetsa kuyambira pachiyambi, ingovomereza mizere yatsopano.

Kuwerenga mafayilo omwe alipo

Tiyerekeze kuti tikuyambitsa logstash koyamba, koma tili ndi zipika kale ndipo tikufuna kuzikonza.
Ngati tithamanga logstash ndi gawo lolowera lomwe tidagwiritsa ntchito pamwambapa, sitipeza kalikonse. Mizere yatsopano yokhayo idzakonzedwa ndi logstash.

Kuti mizere yochokera pamafayilo omwe alipo kale ikwezedwe, muyenera kuwonjezera mzere wowonjezera pagawo lolowetsa:

input {
  file {
    start_position => "beginning"
    path => "/usr/share/logstash/input/*.log"
   }
  }

Komanso, pali kusiyana: izi zimangokhudza mafayilo atsopano omwe logstash sanawonepo. Kwa mafayilo omwewo omwe anali kale m'munda wa logstash, adakumbukira kale kukula kwake ndipo tsopano angotenga zolemba zatsopano mwa iwo.

Tiyeni tiyime apa ndikuphunzira gawo lolowetsa. Pali zosankha zambiri, koma ndizokwanira kwa ife pazoyeserera zina pakadali pano.

Njira ndi Kusintha kwa Data

Tiyeni tiyesetse kuthetsa vuto ili, tinene kuti tili ndi mauthenga ochokera ku tchanelo chimodzi, ena ndi a chidziwitso, ndipo ena ndi mauthenga olakwika. Amasiyana ndi tag. Zina ndi INFO, zina ndi ZOPHUNZITSA.

Tiyenera kuwalekanitsa potuluka. Iwo. Timalemba mauthenga achidziwitso mu njira imodzi, ndi mauthenga olakwika mu njira ina.

Kuti muchite izi, sunthani kuchokera pagawo lolowera kuti musefa ndi zotuluka.

Pogwiritsa ntchito gawo la fyuluta, tidzafotokozera uthenga womwe ukubwera, kupeza hashi (mawotchi amtengo wapatali) kuchokera kwa iwo, omwe tingathe kugwira nawo ntchito, i.e. sukani molingana ndi mikhalidwe. Ndipo mu gawo lotulutsa, tidzasankha mauthenga ndikutumiza aliyense ku njira yake.

Kutumiza uthenga ndi grok

Kuti mudutse zingwe ndikupeza magawo angapo kuchokera kwa iwo, pali pulogalamu yowonjezera yapadera mu gawo la fyuluta - grok.

Popanda kudziikira cholinga chofotokozera mwatsatanetsatane apa (pazimene ndikulozera zolemba zovomerezeka), ndipereka chitsanzo changa chophweka.

Kuti muchite izi, muyenera kusankha mtundu wa zingwe zolowera. Ndili nawo monga chonchi:

1 INFO uthenga1
2 ZOPHUNZITSA message2

Iwo. Chizindikiritso chimabwera koyamba, kenako INFO/ERROR, kenako mawu opanda mipata.
Sizovuta, koma ndikwanira kumvetsetsa mfundo ya ntchito.

Chifukwa chake, mu gawo losefera la pulogalamu yowonjezera ya grok, tiyenera kufotokozera njira yosinthira zingwe zathu.

Zidzawoneka motere:

filter {
  grok {
    match => { "message" => ["%{INT:message_id} %{LOGLEVEL:message_type} %{WORD:message_text}"] }
   }
  }

Kwenikweni ndi mawu okhazikika. Mapangidwe okonzeka amagwiritsidwa ntchito, monga INT, LOGLEVEL, WORD. Kufotokozera kwawo, komanso machitidwe ena, angapezeke apa apa

Tsopano, kudutsa fyuluta iyi, chingwe chathu chidzasanduka hashi ya magawo atatu: message_id, message_type, message_text.

Iwo adzawonetsedwa mu linanena bungwe gawo.

Kutumiza mauthenga ku gawo lotulutsa pogwiritsa ntchito if command

Mu gawo lotulutsa, monga tikukumbukira, timagawa mauthengawo kukhala mitsinje iwiri. Zina - zomwe ndi iNFO, zidzatuluka ku console, ndipo ndi zolakwika, tidzatulutsa fayilo.

Kodi timasiyanitsa bwanji mauthengawa? Mkhalidwe wavutowu ukuwonetsa kale yankho - pambuyo pake, tili ndi gawo lodzipatulira la message_type, lomwe lingatenge zinthu ziwiri zokha: INFO ndi ERROR. Pachifukwa ichi tidzapanga chisankho pogwiritsa ntchito mawu akuti if.

if [message_type] == "ERROR" {
        # Π—Π΄Π΅ΡΡŒ Π²Ρ‹Π²ΠΎΠ΄ΠΈΠΌ Π² Ρ„Π°ΠΉΠ»
       } else
     {
      # Π—Π΄Π΅ΡΡŒ Π²Ρ‹Π²ΠΎΠ΄ΠΈΠΌ Π² stdout
    }

Kufotokozera za ntchito ndi minda ndi ogwira ntchito angapezeke mu gawo ili buku lovomerezeka.

Tsopano, za mapeto enieniwo.

Kutulutsa kwa Console, zonse ndi zomveka apa - stdout {}

Koma zotuluka ku fayilo - kumbukirani kuti tikuyendetsa zonsezi kuchokera m'chidebe ndipo kuti fayilo yomwe timalemba kuti ipezeke kuchokera kunja, tiyenera kutsegula bukhuli mu docker-compose.yml.

Chiwerengero:

Gawo lotulutsa la fayilo yathu likuwoneka motere:


output {
  if [message_type] == "ERROR" {
    file {
          path => "/usr/share/logstash/output/test.log"
          codec => line { format => "custom format: %{message}"}
         }
    } else
     {stdout {
             }
     }
  }

Mu docker-compose.yml timawonjezera voliyumu ina yotulutsa:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input
      - ./output:/usr/share/logstash/output

Timayiyambitsa, yesani, ndikuwona kugawanika kukhala mitsinje iwiri.

Source: www.habr.com

Kuwonjezera ndemanga