ClickHouse + Graphite: momwe mungachepetsere kwambiri kugwiritsa ntchito malo a disk

Moni, habr.

Ngati wina akugwiritsa ntchito ndondomekoyi graphite-webu ndipo tidakumana ndi vuto la magwiridwe antchito kunong'oneza (IO, disk space idyedwa), ndiye mwayi woti ClickHouse idaponyedwa m'malo mwake iyenera kukhala imodzi. Mawu awa akutanthauza kuti kukhazikitsidwa kwa chipani chachitatu kumagwiritsidwa ntchito kale ngati ma metric omwe amalandila daemon, mwachitsanzo. carbonwriter kapena go-carbon.

ClickHouse imathetsa mavuto omwe afotokozedwa bwino. Mwachitsanzo, mutasamutsa 2TiB ya data kuchokera kunong'onong'ono, imalowa mu 300GiB. Sindidzangokhalira kuyerekeza mwatsatanetsatane; pali zolemba zambiri pamutuwu. Kuphatikiza apo, mpaka posachedwa, sizinali zonse zomwe zinali zangwiro ndi malo athu osungira a ClickHouse.

Mavuto ndi malo ogwiritsidwa ntchito

Poyamba, zonse ziyenera kugwira ntchito bwino. Kutsatira zolemba, pangani zokonzekera zosungira ma metrics (kupitilira apo retention), kenako pangani tebulo molingana ndi malingaliro a backend osankhidwa a graphite-web: kaboni-clickhouse+graphite-clickhouse kapena graphhouse, kutengera ndi stack yomwe imagwiritsidwa ntchito. Ndipo... bomba la nthawi likuphulika.

Kuti mumvetsetse kuti ndi iti, muyenera kudziwa momwe zoyikamo zimagwirira ntchito komanso njira yopitilira yamoyo ya data pamatebulo a injini za banja *MergeTree ClickHouse (matchati otengedwa kuchokera презентации Alexey Zatelepin):

• anaika блок deta. Kwa ife, ndi ma metric omwe adafika.
  
• Chida chilichonse chotere chimasanjidwa molingana ndi kiyi isanalembedwe ku disk. ORDER BYotchulidwa popanga tebulo.
• Pambuyo kusankha, кусок (part) deta imalembedwa ku disk.
  
• Seva imayang'anira kumbuyo kuti pasakhale zidutswa zambiri zotere, ndikuyambitsa maziko слияния (merge, phatikizani apa).
  
  
• Seva imasiya kuthamanga zophatikiza zokha data ikangosiya kulowa mu партицию (partition), koma mutha kuyambitsa ntchitoyi pamanja ndi lamulo OPTIMIZE.
• Ngati pali chidutswa chimodzi chokha chomwe chatsala mu magawo, ndiye kuti simungathe kuyendetsa kuphatikiza pogwiritsa ntchito lamulo lanthawi zonse; muyenera kugwiritsa ntchito OPTIMIZE ... FINAL

Kenako, ma metrics oyamba amafika. Ndipo amatenga malo. Zochitika zotsatila zimatha kusiyanasiyana malinga ndi zinthu zambiri:

• Kiyi yogawa imatha kukhala yaying'ono (tsiku) kapena yayikulu kwambiri (miyezi ingapo).
• Kusungirako kungagwirizane ndi magawo angapo ophatikiza deta mkati mwa magawo omwe akugwira ntchito (pomwe ma metrics amajambulidwa), kapena ayi.
• Ngati pali zambiri zambiri, ndiye kuti ma chunks oyambirira, omwe chifukwa cha kugwirizanitsa maziko angakhale aakulu kale (ngati mutasankha makiyi osagawanitsa bwino), sangadziphatikize ndi timagulu tating'onoting'ono tatsopano.

Ndipo izo nthawizonse zimatha mofanana. Malo okhala ndi ma metrics ku ClickHouse amangowonjezeka ngati:

• musagwiritse ntchito OPTIMIZE ... FINAL pamanja kapena
• osayika zidziwitso m'magawo onse mosalekeza, kuti posakhalitsa kuphatikizana kwam'mbuyo kuyambike

Njira yachiwiri ikuwoneka kuti ndiyosavuta kugwiritsa ntchito ndipo, chifukwa chake, siyolondola ndipo idayesedwa poyamba.
Ndinalemba zolemba zosavuta za python zomwe zimatumiza ma metric a dummy tsiku lililonse kwa zaka 4 zapitazi ndikuthamanga cron ola lililonse.
Popeza kuti ntchito yonse ya ClickHouse DBMS imachokera ku mfundo yakuti dongosolo lino lidzachita ntchito zonse zam'mbuyo, koma sizidziwika kuti ndi liti, sindinathe kuyembekezera nthawi yomwe zidutswa zazikuluzikulu zakale zimayamba kugwirizanitsa ndi. zatsopano zazing'ono. Zinali zowonekeratu kuti tifunikira kuyang'ana njira yosinthira kukhathamiritsa kokakamizidwa.

Zambiri pamatebulo a ClickHouse system

Tiyeni tiwone momwe tebulo limapangidwira machitidwe.gawo. Ichi ndi chidziwitso chokwanira pa chidutswa chilichonse cha matebulo onse pa seva ya ClickHouse. Muli, mwa zina, magawo otsatirawa:

• db dzina (database);
• dzina la tebulo (table);
• dzina lagawo ndi ID (partition & partition_id);
• pamene chidutswacho chinalengedwa (modification_time);
• tsiku lochepera komanso lalitali pachidutswa (kugawa kumachitika masana) (min_date & max_date);

Palinso tebulo system.graphite_retentions, ndi magawo otsatirawa osangalatsa:

• db dzina (Tables.database);
• dzina la tebulo (Tables.table);
• zaka za metric pomwe kuphatikiza kotsatira kuyenera kugwiritsidwa ntchito (age);

Kotero:

1. Tili ndi tebulo la chunks ndi tebulo la malamulo ophatikiza.
2. Timaphatikiza mphambano yawo ndikupeza matebulo onse *GraphiteMergeTree.
3. Tikuyang'ana magawo onse omwe:
   • kuposa chidutswa chimodzi
   • kapena nthawi yakwana yoti agwiritse ntchito lamulo lotsatira la kuphatikizira, ndi modification_time wamkulu kuposa nthawi ino.

Реализация

Pempho ili

SELECT
    concat(p.database, '.', p.table) AS table,
    p.partition_id AS partition_id,
    p.partition AS partition,
    -- Самое "старое" правило, которое может быть применено для
    -- партиции, но не в будущем, см (*)
    max(g.age) AS age,
    -- Количество кусков в партиции
    countDistinct(p.name) AS parts,
    -- За самую старшую метрику в партиции принимается 00:00:00 следующего дня
    toDateTime(max(p.max_date + 1)) AS max_time,
    -- Когда партиция должна быть оптимизированна
    max_time + age AS rollup_time,
    -- Когда самый старый кусок в партиции был обновлён
    min(p.modification_time) AS modified_at
FROM system.parts AS p
INNER JOIN
(
    -- Все правила для всех таблиц *GraphiteMergeTree
    SELECT
        Tables.database AS database,
        Tables.table AS table,
        age
    FROM system.graphite_retentions
    ARRAY JOIN Tables
    GROUP BY
        database,
        table,
        age
) AS g ON
    (p.table = g.table)
    AND (p.database = g.database)
WHERE
    -- Только активные куски
    p.active
    -- (*) И только строки, где правила аггрегации уже должны быть применены
    AND ((toDateTime(p.max_date + 1) + g.age) < now())
GROUP BY
    table,
    partition
HAVING
    -- Только партиции, которые младше момента оптимизации
    (modified_at < rollup_time)
    -- Или с несколькими кусками
    OR (parts > 1)
ORDER BY
    table ASC,
    partition ASC,
    age ASC

imabweretsa magawo onse a tebulo la *GraphiteMergeTree omwe kuphatikiza kwawo kuyenera kumasula malo a disk. Chinthu chokha chimene chatsala ndi kuwadutsa onse ndi pempho OPTIMIZE ... FINAL. Kukhazikitsa komaliza kumaganiziranso kuti palibe chifukwa chokhudza magawo ndi kujambula kogwira.

Izi ndi zomwe polojekitiyi imachita graphite-ch-optimizer. Anzake akale ochokera ku Yandex.Market adayesa kupanga, zotsatira za ntchitoyi zikhoza kuwonetsedwa pansipa.

Ngati mutayendetsa pulogalamuyi pa seva ndi ClickHouse, idzangoyamba kugwira ntchito mu daemon mode. Kamodzi pa ola pempho lidzaperekedwa, kuwunika ngati magawo atsopano amasiku atatu awonekera omwe angathe kukonzedwa.

Mapulani athu aposachedwa ndikupereka ma phukusi osachepera a deb, ndipo ngati nkotheka komanso rpm.

M'malo mapeto

Pa miyezi 9+ yapitayi ndakhala mkati mwa kampani yanga Makhalidwe adakhala nthawi yayitali akusewera pamzere wa ClickHouse ndi graphite-web. Zinali zokumana nazo zabwino, zomwe zidapangitsa kusintha mwachangu kuchoka ku kunong'oneza kupita ku ClickHouse ngati malo osungira ma metric. Ndikukhulupirira kuti nkhaniyi ndi chiyambi cha mndandanda wazinthu zomwe tapanga m'malo osiyanasiyana a stack iyi, ndi zomwe zidzachitike mtsogolo.

Malita angapo a mowa ndi masiku a admin adagwiritsidwa ntchito popanga pempholi, limodzi ndi v0 mdierekezi, zomwe ndikufuna kumuthokoza. Komanso powunikiranso nkhaniyi.

Tsamba la polojekiti pa github

Source: www.habr.com