ClickHouse + Graphite: maitiro ekudzikisa zvakanyanya dhisiki nzvimbo yekushandisa

Kwaziwai, habr.

Kana mumwe munhu akashandisa iyo system graphite-web uye yakasangana nedambudziko rekuita kwekuchengetedza zevezeve (IO, dhisiki nzvimbo inopedzwa), ipapo mukana wekuti ClickHouse yakakandwa sechinotsiva inofanira kuenda kune imwe. Ichi chirevo chinoreva kuti chechitatu-bato kuita chatoshandiswa sedhimoni rinogamuchira metrics, semuenzaniso. carbonwriter kana go-carbon.

ClickHouse inogadzirisa matambudziko akatsanangurwa zvakanaka. Semuenzaniso, mushure mekutamisa 2TiB yedata kubva kuzevezera, inokwana mu300GiB. Ini handisi kuzogara pakuenzanisa zvakadzama; kune zvakawanda zvinyorwa pane iyi nyaya. Mukuwedzera, kusvika nguva pfupi yadarika, hazvisi zvese zvaive zvakakwana neyedu ClickHouse chengetedzo.

Matambudziko nenzvimbo yakadyiwa

Pakutanga kuona, zvinhu zvose zvinofanira kushanda zvakanaka. Kutevera zvinyorwa, gadzira gadziriso yemametrics ekuchengetedza chirongwa (kuwedzera retention), wobva wagadzira tafura zvinoenderana nekurudziro yeakasarudzwa backend yegraphite-web: carbon-clickhouse+graphite-clickhouse kana graphouse, zvichienderana nekuti ndeipi stack inoshandiswa. Uye ... nguva bhomba rinoenda.

Kuti unzwisise kuti ndeipi, iwe unofanirwa kuziva mashandisiro anoita basa uye imwe nzira yehupenyu yedata mumatafura einjini dzemhuri *.MergeTree ClickHouse (machati akatorwa kubva mharidzo Alexey Zatelepin):

• Inseted блок data. Kwatiri, aive ma metrics akasvika.
  
• Imwe neimwe block yakadaro inorongwa zvinoenderana nekiyi isati yanyorwa kune dhisiki. ORDER BYinotsanangurwa pakugadzira tafura.
• Mushure mekusarudza, кусок (part) data inonyorerwa kudhisiki.
  
• Sevha inotarisisa kumashure kuitira kuti pasave nezvimedu zvakawanda zvakadaro, uye inotangisa kumashure слияния (merge, zvino sangana).
  
  
• Iyo sevha inomira kumhanya inosanganiswa yega kana data yamira ichiyerera ichiyerera ichipinda mu партицию (partition), asi unogona kutanga maitiro nemawoko nekuraira OPTIMIZE.
• Kana paine chidimbu chimwe chete chasara muchikamu, saka haugone kumhanyisa kusanganisa uchishandisa iyo yakajairwa kuraira; iwe unofanirwa kushandisa OPTIMIZE ... FINAL

Saka, ma metrics ekutanga anosvika. Uye vanotora imwe nzvimbo. Zviitiko zvinotevera zvinogona kusiyana zvichienderana nezvikonzero zvakawanda:

• Kiyi yekuparadzanisa inogona kunge iri diki (zuva) kana yakakura kwazvo (mwedzi yakati wandei).
• Iyo yekuchengeta gadziriso inogona kukwana akati wandei akakosha data aggregation zvikumbaridzo mukati meiyo inoshanda partition (painorekodhwa metrics), kana pamwe kwete.
• Kana paine data rakawanda, saka machunks ekutanga, ayo nekuda kwekubatanidzwa kwemashure anogona kunge atove akakura (kana ukasarudza isina-yakakwana yekugovera kiyi), haizozvisanganise nemadiki madiki chunks.

Uye nguva dzose inopera zvakafanana. Iyo nzvimbo inogarwa nemametrics muClickHouse inongowedzera kana:

• usashande OPTIMIZE ... FINAL nemaoko kana
• usaise data muzvikamu zvese nguva nenguva, kuitira kuti nekukurumidza kana kuti gare gare kusanganiswa kwemashure kunotanga

Yechipiri nzira inoratidzika kunge iri nyore kuita uye, saka, haina kururama uye yakaedzwa kutanga.
Ndakanyora script yakapusa python iyo yakatumira dummy metrics yezuva rega rega kwemakore apfuura 4 uye yakamhanya cron awa yega yega.
Sezvo kushanda kwese kweClickHouse DBMS kwakavakirwa pakuti iyi sisitimu ichakurumidza kana gare gare kuita basa rese rekumashure, asi hazvizivikanwe kuti riini, handina kukwanisa kumirira nguva iyo zvidimbu zvakakura zvakatanga kutanga kubatanidzwa. zvitsva zvitsva. Zvakava pachena kuti taifanira kutsvaga nzira yekuita otomatiki optimizations.

Ruzivo muClickHouse system matafura

Ngatitarisei chimiro chetafura system.parts. Iri iruzivo rwakakwana nezve chidimbu chega chega cheese matafura pane ClickHouse server. Iine, pakati pezvimwe zvinhu, zvinotevera makoramu:

• db zita (database);
• zita retafura (table);
• zita uye chikamu ID (partition & partition_id);
• apo chidimbu chakagadzirwa (modification_time);
• diki uye zuva rakawanda muchidimbu (kugovera kunoitwa nezuva) (min_date & max_date);

Kunewo tafura system.graphite_retentions, nendima dzinonakidza dzinotevera:

• db zita (Tables.database);
• zita retafura (Tables.table);
• metric age apo iyo inotevera aggregation inofanirwa kushandiswa (age);

Saka:

1. Tine tafura yezvimedu uye tafura yemitemo yekuunganidza.
2. Isu tinosanganisa mharadzano yavo uye tinotora ese matafura * GraphiteMergeTree.
3. Isu tiri kutsvaga ese mapartitions umo:
   • zvinopfuura chidimbu chimwe
   • kana kuti nguva yasvika yekushandisa mutemo unotevera wekuunganidza, uye modification_time mukuru kudarika nguva ino.

Kutevedzera

Chikumbiro ichi

SELECT
    concat(p.database, '.', p.table) AS table,
    p.partition_id AS partition_id,
    p.partition AS partition,
    -- Самое "старое" правило, которое может быть применено для
    -- партиции, но не в будущем, см (*)
    max(g.age) AS age,
    -- Количество кусков в партиции
    countDistinct(p.name) AS parts,
    -- За самую старшую метрику в партиции принимается 00:00:00 следующего дня
    toDateTime(max(p.max_date + 1)) AS max_time,
    -- Когда партиция должна быть оптимизированна
    max_time + age AS rollup_time,
    -- Когда самый старый кусок в партиции был обновлён
    min(p.modification_time) AS modified_at
FROM system.parts AS p
INNER JOIN
(
    -- Все правила для всех таблиц *GraphiteMergeTree
    SELECT
        Tables.database AS database,
        Tables.table AS table,
        age
    FROM system.graphite_retentions
    ARRAY JOIN Tables
    GROUP BY
        database,
        table,
        age
) AS g ON
    (p.table = g.table)
    AND (p.database = g.database)
WHERE
    -- Только активные куски
    p.active
    -- (*) И только строки, где правила аггрегации уже должны быть применены
    AND ((toDateTime(p.max_date + 1) + g.age) < now())
GROUP BY
    table,
    partition
HAVING
    -- Только партиции, которые младше момента оптимизации
    (modified_at < rollup_time)
    -- Или с несколькими кусками
    OR (parts > 1)
ORDER BY
    table ASC,
    partition ASC,
    age ASC

inodzosa imwe neimwe ye *GraphiteMergeTree tafura partitions iyo kusanganisa kunofanirwa kusunungura dhisiki nzvimbo. Chinhu chasara kuita kungopfuura mavari vese nechikumbiro OPTIMIZE ... FINAL. Kuitwa kwekupedzisira kunofungawo nezve chokwadi chekuti hapana chikonzero chekubata zvikamu neanoshanda kurekodha.

Izvi ndizvo chaizvo zvinoita chirongwa graphite-ch-optimizer. Vaimbova vashandi kubva kuYandex.Market vakaidza mukugadzirwa, chigumisiro chebasa chinogona kuonekwa pasi apa.

Kana iwe ukamhanyisa chirongwa pane server neClickHouse, inongotanga kushanda mu daemon mode. Kamwe paawa chikumbiro chichaitwa, kutarisa kana zvikamu zvitsva zvakakura kupfuura mazuva matatu zvakaonekwa zvinogona kugadziriswa.

Zvirongwa zvedu zvepakarepo ndezvekupa angangoita dhibhi mapakeji, uye kana zvichibvira zvakare rpm.

Pane mhedziso

Kwemwedzi ye9+ yapfuura ndanga ndiri mukati mekambani yangu innogames akapedza nguva yakawanda achizeya pamharadzano yeClickHouse uye graphite-web. Chaive chiitiko chakanaka, chakakonzera shanduko yekukurumidza kubva kuzevezeve kuenda kuClickHouse semametrics ekuchengetedza. Ndinovimba kuti chinyorwa ichi ndechekutanga kwenhevedzano nezve zvigadziriso zvatakaita kunzvimbo dzakasiyana dzestack iyi, uye chii chichaitwa mune ramangwana.

Akawanda malita edoro uye admin mazuva akashandiswa mukugadzira chikumbiro, pamwe chete v0dhiyabhorosi, izvo ndinoda kuratidza kuonga kwangu kwaari. Uye zvakare yekuongorora chinyorwa ichi.

Peji yeprojekiti pane github

Source: www.habr.com