I-ClickHouse + Graphite: indlela yokunciphisa kakhulu ukusetshenziswa kwesikhala sediski

Sawubona, habr.

Uma othile exhaphaza uhlelo i-graphite-web futhi uhlangabezane nenkinga yokusebenza kwesitoreji kuhleba (I-IO, isikhala sediski sidliwe), khona-ke ithuba lokuthi i-ClickHouse ifakwe esikhundleni kufanele lithambekele kwelilodwa. Lesi sitatimende sibonisa ukuthi ukusetshenziswa kwenkampani yangaphandle sekuvele kusetshenziswa njengamamethrikhi athola i-daemon, isibonelo. i-carbonwriter noma hamba-carbon.

I-ClickHouse ixazulula izinkinga ezichazwe kahle. Isibonelo, ngemuva kokudlulisa u-2TiB wedatha kusuka ekunyenyezeni, angena ku-300GiB. Ngeke ngigxile ekuqhathaniseni ngemininingwane; kunenqwaba yezihloko ngalesi sihloko. Ngaphezu kwalokho, kuze kube muva nje, akuyona yonke into eyayihamba kahle ngesitoreji sethu se-ClickHouse.

Izinkinga ngendawo edliwe

Ekuboneni kokuqala, konke kufanele kusebenze kahle. Ukulandela imibhalo, dala ukulungiselelwa kwesikimu sesitoreji samamethrikhi (okwengeziwe retention), bese udala ithebula ngokuya ngesincomo se-backend ekhethiwe ye-graphite-web: i-carbon-clickhouse+i-graphite-clickhouse noma i-graphouse, kuye ngokuthi yisiphi isitaki esisetshenziswayo. Futhi... ibhomu lesikhathi liyaqhuma.

Ukuze uqonde ukuthi iyiphi, udinga ukwazi ukuthi ukufakwa kusebenza kanjani kanye nendlela yokuphila eyengeziwe yedatha kumathebula wezinjini zomndeni *I-MergeTree I-ClickHouse (amashadi athathwe ku izethulo Alexey Zatelepin):

• Kufakiwe блок idatha. Esimeni sethu, bekungamamethrikhi afikile.
  
• Ibhulokhi ngayinye enjalo ihlelwa ngokokhiye ngaphambi kokuba ibhalwe kudiski. ORDER BYkucaciswe lapho kwakhiwa ithebula.
• Ngemva kokuhlunga, кусок (part) idatha ibhalwa kudiski.
  
• Iseva iqapha ngemuva ukuze kungabikho izingcezu eziningi ezinjalo, futhi ivula ingemuva слияния (merge, hlanganisa manje).
  
  
• Iseva iyayeka ukuhlanganiswa okusebenzayo ngokwayo ngokushesha nje lapho idatha ima ngokusebenzayo ukungena ku- партицию (partition), kodwa ungaqala inqubo mathupha ngomyalo OPTIMIZE.
• Uma sekusele ucezu olulodwa kuphela ekuhlukaniseni, ngeke ukwazi ukusebenzisa ukuhlanganisa usebenzisa umyalo ojwayelekile; kufanele usebenzise OPTIMIZE ... FINAL

Ngakho-ke, amamethrikhi okuqala afika. Futhi bathatha indawo ethile. Izehlakalo ezilandelayo zingahluka kuye ngezici eziningi:

• Ukhiye wokuhlukanisa ungaba mncane kakhulu (usuku) noma ube mkhulu kakhulu (izinyanga ezimbalwa).
• Ukulungiselelwa kokugcinwa kungase kulingane nemikhawulo embalwa ebalulekile yokuhlanganisa idatha ngaphakathi kwengxenye esebenzayo (lapho amamethrikhi aqoshwa khona), noma mhlawumbe cha.
• Uma kunedatha eningi, khona-ke izingcezu zakuqala, okungenzeka ngenxa yokuhlanganiswa kwengemuva zingase zibe zinkulu (uma ukhetha ukhiye wokuhlukanisa ongafanelekile), ngeke zizihlanganise nezingcezu ezincane ezintsha.

Futhi igcina ngokufana njalo. Isikhala esihlalwa amamethrikhi ku-ClickHouse sikhuphuka kuphela uma:

• ungasebenzi OPTIMIZE ... FINAL ngesandla noma
• ungafaki idatha kuwo wonke ama-partitions ngokuqhubekayo, ukuze maduze noma kamuva ukuhlanganisa okungemuva kuqale

Indlela yesibili ibonakala ilula kakhulu ukuyisebenzisa futhi, ngakho-ke, ayilungile futhi yazanywa kuqala.
Ngabhala umbhalo we-python olula othumele amamethrikhi we-dummy nsuku zonke iminyaka engu-4 edlule futhi ngigijima i-cron njalo ngehora.
Njengoba konke ukusebenza kwe-ClickHouse DBMS kusekelwe eqinisweni lokuthi lolu hlelo luzokwenza ngokushesha noma kamuva wonke umsebenzi wangemuva, kodwa akwaziwa ukuthi nini, angikwazanga ukulinda isikhathi lapho izingcezu ezinkulu ezindala ziqala ukuhlangana nazo. ezintsha ezincane. Kwaba sobala ukuthi kwakudingeka sifune indlela yokwenza ukulungiselelwa okuphoqelelwe ngokuzenzakalelayo.

Ulwazi kumathebula esistimu ye-ClickHouse

Ake sibheke isakhiwo setafula uhlelo.izingxenye. Lolu ulwazi olubanzi mayelana nesiqephu ngasinye sawo wonke amathebula kuseva ye-ClickHouse. Iqukethe, phakathi kokunye, amakholomu alandelayo:

• db igama (database);
• igama lethebula (table);
• igama lokuhlukanisa kanye ne-ID (partition & partition_id);
• ngenkathi kwakhiwa ucezu (modification_time);
• ubuncane kanye nedethi ephezulu esiqeshini (ukuhlukaniswa kwenziwa ngosuku) (min_date & max_date);

Kukhona netafula system.graphite_retentions, ngezinkambu ezithakazelisayo ezilandelayo:

• db igama (Tables.database);
• igama lethebula (Tables.table);
• iminyaka yemethrikhi lapho kufanele kusetshenziswe ukuhlanganisa okulandelayo (age);

Ngakho:

1. Sinethebula lezinqamu kanye nethebula lemithetho yokuhlanganisa.
2. Sihlanganisa ukuhlangana kwabo futhi sithole wonke amatafula *GraphiteMergeTree.
3. Sibheka wonke ama-partitions lapho:
   • ngaphezu kwesiqephu esisodwa
   • noma isikhathi sesifikile sokusebenzisa umthetho wokuhlanganisa olandelayo, futhi modification_time mdala kunalesi sikhathi.

Ukuqaliswa

Lesi sicelo

SELECT
    concat(p.database, '.', p.table) AS table,
    p.partition_id AS partition_id,
    p.partition AS partition,
    -- Самое "старое" правило, которое может быть применено для
    -- партиции, но не в будущем, см (*)
    max(g.age) AS age,
    -- Количество кусков в партиции
    countDistinct(p.name) AS parts,
    -- За самую старшую метрику в партиции принимается 00:00:00 следующего дня
    toDateTime(max(p.max_date + 1)) AS max_time,
    -- Когда партиция должна быть оптимизированна
    max_time + age AS rollup_time,
    -- Когда самый старый кусок в партиции был обновлён
    min(p.modification_time) AS modified_at
FROM system.parts AS p
INNER JOIN
(
    -- Все правила для всех таблиц *GraphiteMergeTree
    SELECT
        Tables.database AS database,
        Tables.table AS table,
        age
    FROM system.graphite_retentions
    ARRAY JOIN Tables
    GROUP BY
        database,
        table,
        age
) AS g ON
    (p.table = g.table)
    AND (p.database = g.database)
WHERE
    -- Только активные куски
    p.active
    -- (*) И только строки, где правила аггрегации уже должны быть применены
    AND ((toDateTime(p.max_date + 1) + g.age) < now())
GROUP BY
    table,
    partition
HAVING
    -- Только партиции, которые младше момента оптимизации
    (modified_at < rollup_time)
    -- Или с несколькими кусками
    OR (parts > 1)
ORDER BY
    table ASC,
    partition ASC,
    age ASC

ibuyisela ingxenye ngayinye yetafula *GraphiteMergeTree ukuhlanganiswa kwayo okufanele kukhulule isikhala sediski. Okusele okumele ukwenze wukubhekana nazo zonke ngesicelo OPTIMIZE ... FINAL. Ukuqaliswa kokugcina kuphinde kucabangele iqiniso lokuthi asikho isidingo sokuthinta ama-partitions ngokurekhoda okusebenzayo.

Yilokhu kanye okwenziwa yiphrojekthi i-graphite-ch-optimizer. Ozakwethu bangaphambili abavela ku-Yandex.Market bazama ekukhiqizeni, umphumela womsebenzi ungabonakala ngezansi.

Uma usebenzisa uhlelo kuseva nge-ClickHouse, luzoqala ukusebenza ngemodi ye-daemon. Kanye ngehora umbuzo uzokwenziwa, kubhekwa ukuthi ingabe izingxenye ezintsha ezindala kunezinsuku ezintathu zivele zingathuthukiswa yini.

Izinhlelo zethu ezisheshayo ukuhlinzeka okungenani ngamaphakheji edeb, futhi uma kungenzeka futhi rpm.

Esikhundleni isiphetho

Ezinyangeni ezingu-9+ ezedlule bengingaphakathi kwenkampani yami I-InnoGames uchithe isikhathi esiningi ezamazama ezimpambanweni zeClickHouse kanye ne-graphite-web. Bekuyisipiliyoni esihle, esiholele ekuguqukeni okusheshayo kokuhleba kuya ku-ClickHouse njengendawo yokugcina amamethrikhi. Ngithemba ukuthi lesi sihloko siyinto yokuqala yochungechunge mayelana nokuthi iziphi intuthuko esiyenzile ezingxenyeni ezihlukahlukene zalesi sitaki, nokuthi yini ezokwenziwa esikhathini esizayo.

Kusetshenziswe amalitha amaningana kabhiya kanye nezinsuku zokuphatha ekuthuthukiseni isicelo, kanye v0 usathane, engifuna ukuzwakalisa ukubonga kwami ​​kuye. Futhi nokubukeza lesi sihloko.

Ikhasi lephrojekthi ku-github

Source: www.habr.com