Zvimiro zvemutauro weQ uye KDB+ uchishandisa muenzaniso wenguva chaiyo sevhisi

Iwe unogona kuverenga nezve iyo KDB + base, iyo Q programming mutauro, chii simba ravo uye kusasimba kwavo mune yangu yapfuura. chinyorwa uye muchidimbu munhanganyaya. Muchinyorwa chacho, tichashandisa sevhisi paQ iyo ichagadzirisa iyo inouya data rwizi uye kuverenga akasiyana ekubatanidza mabasa miniti yega yega mu "nguva chaiyo" modhi (kureva, ichave nenguva yekuverenga zvese pamberi pechikamu chinotevera chedata). Chinhu chikuru cheQ ndechekuti mutauro wevector unokubvumira kuti ushande kwete nechinhu chimwe chete, asi nehurongwa hwavo, mitsara yezvirongwa nezvimwe zvinhu zvakaoma. Mitauro yakaita seQ nehama dzayo K, J, APL ine mukurumbira nekupfupika kwayo. Kazhinji, purogiramu inotora zvikwangwani zvakawanda zvekodhi mumutauro unozivikanwa seJava inogona kunyorwa pairi mumitsara mishoma. Izvi ndizvo zvandinoda kuratidza muchinyorwa chino.

Nhanganyaya

KDB+ ndeye columnar dhatabhesi yakatarisana nehuwandu hukuru hwe data, yakarongedzerwa neimwe nzira (kunyanya nenguva). Inoshandiswa kunyanya mumasangano emari - mabhangi, mari yekudyara, makambani einishuwarenzi. Mutauro weQ ndiwo mutauro wemukati weKDB+ unokutendera kuti ushande nemazvo nedata iri. Iyo Q ideology ipfupi uye kushanda nesimba, nepo kujeka kuchipirwa. Izvi zvinoruramiswa nechokwadi chekuti mutauro wevector uchave wakaoma kunzwisisa mune chero mamiriro ezvinhu, uye kupfupika uye hupfumi hwekurekodha kunoita kuti iwe uone chikamu chikuru chechirongwa pane imwe skrini, izvo zvinozoita kuti zvive nyore kunzwisisa.

Muchinyorwa chino tinoisa chirongwa chakazara muQ uye ungangoda kuzviedza. Kuti uite izvi, iwe uchada iyo chaiyo Q. Unogona kudhawunirodha yemahara 32-bit vhezheni pane kx kambani webhusaiti - www.kx.com. Ikoko, kana iwe uchida, iwe unowana referenzi ruzivo paQ, bhuku Q Kune Vanofa uye zvinyorwa zvakasiyana-siyana pamusoro penyaya iyi.

Kugadzirwa kwedambudziko

Pane sosi inotumira tafura ine data yega 25 milliseconds. Sezvo KDB+ ichinyanya kushandiswa mune zvemari, isu tichafunga kuti iyi tafura yekutengeserana (kutengesa), iyo ine anotevera makoramu: nguva (nguva mumamilliseconds), sym (kuzivikanwa kwekambani pane stock exchange - IBM, AAPL,…), mutengo (mutengo wakatengwa nawo zvikamu), saizi (saizi yekutengeserana). Iyo 25 millisecond kupindirana ndeyekupokana, kwete kudiki uye haina kureba. Kuvapo kwayo kunoreva kuti iyo data inouya kune sevhisi yakatovharwa. Zvingave nyore kushandisa buffering padivi resevhisi, kusanganisira inochinja-chinja zvinoenderana nemutoro wazvino, asi kuti zvive nyore, isu tinozotarisa pane yakatarwa nguva.

Iyo sevhisi inofanirwa kuverenga miniti yega yega yechiratidzo chega chega chinouya kubva kune sym column seti yeaggregating mabasa - max mutengo, avg mutengo, sum size, nezvimwe. ruzivo runobatsira. Kuti zvive nyore, tichafunga kuti mabasa ose anogona kuverengwa zvishoma nezvishoma, i.e. kuti uwane hutsva hutsva, zvakakwana kuziva nhamba mbiri - yekare uye inopinda maitiro. Semuenzaniso, mafunctions max, avhareji, sum ane pfuma iyi, asi basa repakati harina.

Isu tichafunga zvakare kuti iyo inouya data rwizi inguva yakarairwa. Izvi zvichatipa mukana wekushanda chete neminiti yekupedzisira. Mukuita, zvakakwana kuti ugone kushanda neazvino uye maminetsi apfuura kana zvimwe zvigadziriso zvanonoka. Kuti zvive nyore, isu hatingatarise nyaya iyi.

Aggregation mabasa

Mabasa ekuunganidza anodiwa akanyorwa pazasi. Ndakatora akawanda azvo sezvinobvira kuti ndiwedzere mutoro pasevhisi:

• yakakwirira - mutengo mukuru - mutengo mukuru paminiti.
• yakaderera - min mutengo - shoma mutengo paminiti.
• firstPrice - mutengo wekutanga - mutengo wekutanga paminiti.
• lastPrice - yekupedzisira mutengo - yekupedzisira mutengo paminiti.
• yekutangaSize - saizi yekutanga - saizi yekutanga yekutengesa paminiti.
• lastSize - saizi yekupedzisira - saizi yekupedzisira yekutengesa muminiti.
• numTrades - count i - nhamba yekutengesa paminiti.
• vhoriyamu - saizi yehuwandu - huwandu hwekutengesa saizi paminiti.
• pvolume - mutengo wehuwandu - huwandu hwemitengo paminiti, inodiwa kune avgPrice.
• - sum turnover mutengo * saizi - yakazara vhoriyamu yekutengeserana paminiti.
• avgPrice - pvolume% numTrades - mutengo wepakati paminiti.
• avgSize - vhoriyamu% numTrades - avhareji saizi yekutengesa paminiti.
• vwap - turnover% volume - mutengo wepakati paminiti inoyerwa nehukuru hwekutengesa.
• cumVolume - sum vhoriyamu - yakaunganidzwa saizi yekutengeserana pamusoro penguva yese.

Ngatikurukurei nekukasira pfungwa imwe isiri pachena - maitiro ekutanga aya makoramu kekutanga uye paminiti imwe neimwe inotevera. Mamwe makoramu emhando yekutangaMutengo anofanirwa kutangwa kusashanda nguva yega yega; kukosha kwawo hakuna kutsanangurwa. Dzimwe mhando dzevhoriyamu dzinofanira kugara dzakaiswa ku 0. Kune zvakare makoramu anoda nzira yakasanganiswa - semuenzaniso, cumVolume inofanira kukopwa kubva paminiti yapfuura, uye yekutanga seti ku 0. Ngatisete maparameter ese aya tichishandisa data reduramazwi. mhando (inofanana nerekodhi):

// list ! list – создать словарь, 0n – float null, 0N – long null, `sym – тип символ, `sym1`sym2 – список символов
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time; // список всех вычисляемых колонок, reverse объяснен ниже

Ini ndakawedzera sym uye nguva kuduramazwi kuti zvive nyore, ikozvino initWith mutsara wakagadzirwa kubva patafura yekupedzisira yakaunganidzwa, kwainosara kuseta iyo chaiyo sym uye nguva. Unogona kuishandisa kuwedzera mitsara mitsva patafura.

Tichada aggCols kana tichigadzira basa rekuunganidza. Rondedzero yacho inofanirwa kudzokororwa nekuda kwehurongwa hunoongororwa mazwi ari muQ (kubva kurudyi kuenda kuruboshwe). Chinangwa ndechekuona kuti kuverenga kunobva kumusoro kuenda kuCumVolume, sezvo mamwe makoramu anoenderana neapfuura.

Makoramu anoda kuteedzerwa kune imwe miniti kubva kune yapfuura, iyo sym column inowedzerwa kuti zvive nyore:

rollColumns:`sym`cumVolume;

Zvino ngatipatsanurei makoramu mumapoka maererano nemagadzirirwo aanofanira kuitwa. Mhando nhatu dzinogona kusiyaniswa:

1. Accumulators (volume, turnover, ..) - tinofanira kuwedzera kukosha kunouya kune yapfuura.
2. Nechinhu chakakosha (chakakwirira, chakaderera, ..) - kukosha kwekutanga muminiti inotorwa kubva kune data inouya, mamwe ose anoverengwa achishandisa basa.
3. Zorora. Nguva dzose inoverengerwa uchishandisa basa.

Ngatitsanangurei zvinosiyana zvemakirasi aya:

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

Calculation order

Isu tichavandudza tafura yakaunganidzwa mumatanho maviri. Kuti tibudirire, tinotanga tadzikisa tafura inouya kuitira kuti pave nemutsara mumwe chete kune yega yega uye miniti. Icho chokwadi chekuti mabasa edu ese ari kuwedzera uye mubatanidzwa anovimbisa kuti mhedzisiro yeiyi danho rekuwedzera haizochinji. Unogona kuderedza tafura uchishandisa sarudza:

select high:max price, low:min price … by sym,time.minute from table

Iyi nzira ine dhizaini - iyo seti yemakoramu akaverengerwa anofanotsanangurwa. Neraki, muQ, sarudza inoitwa zvakare sebasa raunogona kutsiva zvine simba rakagadzirwa nharo:

?[table;whereClause;byClause;selectClause]

Ini handisi kuzotsanangura zvakadzama mafomati enharo; kwatiri, chete uye nekusarudza mataurirwo anenge asiri enhando uye anofanira kunge ari maduramazwi emafomu columns! mataurirwo. Nokudaro, basa rekuderera rinogona kutsanangurwa sezvinotevera:

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size"); // each это функция map в Q для одного списка
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

Kuti zvive pachena, ndakashandisa parse function, iyo inoshandura tambo ine Q kutaura kuita kukosha kunogona kupfuudzwa kune eval basa uye inodiwa mukusarudza basa. Ziva zvakare kuti preprocess inotsanangurwa sefungidziro (kureva, basa rine nharo dzakatsanangurwa zvishoma) dzesarudzo basa, nharo imwe (tafura) haipo. Kana tikashandisa preprocess patafura, tinowana tafura yakamanikidzwa.

Nhanho yechipiri ndeyekuvandudza tafura yakaunganidzwa. Ngatitangei kunyora iyo algorithm mune pseudocode:

for each sym in inputTable
  idx: row index in agg table for sym+currentTime;
  aggTable[idx;`high]: aggTable[idx;`high] | inputTable[sym;`high];
  aggTable[idx;`volume]: aggTable[idx;`volume] + inputTable[sym;`volume];
  …

MuQ, zvakajairika kushandisa mepu/kudzikisa mabasa pachinzvimbo chezvishwe. Asi sezvo Q iri mutauro wevector uye isu tinokwanisa nyore kushandisa ese maoperation kune ese zviratidzo kamwechete, zvino kune yekutanga fungidziro tinogona kuita pasina loop zvachose, tichiita maoparesheni pazviratidzo zvese kamwechete:

idx:calcIdx inputTable;
row:aggTable idx;
aggTable[idx;`high]: row[`high] | inputTable`high;
aggTable[idx;`volume]: row[`volume] + inputTable`volume;
…

Asi isu tinogona kuenda mberi, Q ine yakasarudzika uye ine simba zvakanyanya mushandisi - iyo generalized assignment operator. Iyo inokutendera iwe kuti uchinje seti yezvakakosha mune yakaoma data chimiro uchishandisa runyoro rwema indices, mabasa uye nharo. Muchiitiko chedu zvinoratidzika sezvizvi:

idx:calcIdx inputTable;
rows:aggTable idx;
// .[target;(idx0;idx1;..);function;argument] ~ target[idx 0;idx 1;…]: function[target[idx 0;idx 1;…];argument], в нашем случае функция – это присваивание
.[aggTable;(idx;aggCols);:;flip (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Sezvineiwo, kupa tafura iwe unoda runyoro rwemitsara, kwete makoramu, uye unofanirwa kushandura matrix (rondedzero yemakoramu kurongedza mitsara) uchishandisa iyo flip basa. Izvi zvinodhura patafura hombe, saka panzvimbo pacho isu tinoshandisa basa rakajairwa kune yega yega koramu zvakasiyana, tichishandisa mepu basa (rinoita senge apostrophe):

.[aggTable;;:;]'[(idx;)each aggCols; (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Isu tinoshandisa zvakare basa rekufungidzira. Ziva zvakare kuti muQ, kugadzira runyoro ibasa zvakare uye tinogona kuidaidza tichishandisa iyo yega (mepu) basa kuti uwane runyoro rwezvinyorwa.

Kuti ive nechokwadi chekuti seti yemakoramu akaverengerwa haina kugadzikiswa, isu tichagadzira kutaura kuri pamusoro zvine simba. Ngatitangei tatsanangura mashandiro ekuverenga koramu yega yega, tichishandisa mutsara uye inp zvinosiyana kureva kuunganidza uye kuisa data:

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!
 ("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");

Mamwe makoramu akakosha; kukosha kwawo kwekutanga hakufanirwe kuverengerwa nebasa racho. Tinogona kuona kuti ndiyo yekutanga nemutsara [`numTrades] column - kana iine 0, saka kukosha ndiko kwekutanga. Q ine sarudzo - ?[Boolean list;list1;list2] - iyo inosarudza kukosha kubva pane 1 kana 2 zvichienderana nemamiriro ekutanga nharo:

// high -> ?[isFirst;inp`high;row[`high]|inp`high]
// @ - тоже обобщенное присваивание для случая когда индекс неглубокий
@[`aggExpression;specialCols;{[x;y]"?[isFirst;inp`",y,";",x,"]"};string specialCols];

Apa ndakadaidza generalized assignment nebasa rangu (chiratidziro mumabrace akamonana). Inogamuchira kukosha kwezvino (nharo yekutanga) uye imwe nharo, iyo yandinopfuura mu 4th parameter.

Ngatiwedzerei vatauri vebhatiri zvakasiyana, sezvo basa rakafanana kwavari:

// volume -> row[`volume]+inp`volume
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;

Iri ibasa rakajairwa neQ zviyero, asi ini ndiri kugovera runyoro rwezvakakosha kamwechete. Pakupedzisira, ngatitangei basa guru:

// ":",/:aggExprs ~ map[{":",x};aggExpr] => ":row[`high]|inp`high" присвоим вычисленное значение переменной, потому что некоторые колонки зависят от уже вычисленных значений
// string[cols],'exprs ~ map[,;string[cols];exprs] => "high:row[`high]|inp`high" завершим создание присваивания. ,’ расшифровывается как map[concat]
// ";" sv exprs – String from Vector (sv), соединяет список строк вставляя “;” посредине
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}";

Nekutaura uku, ini ndinogadzira zvine simba basa kubva patambo ine chirevo chandapa pamusoro. Mhedzisiro ichaita seizvi:

{[aggTable;idx;inp] rows:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols ;(cumVolume:row[`cumVolume]+inp`cumVolume;… ; high:?[isFirst;inp`high;row[`high]|inp`high])]}

The column evaluation order is inverted nekuti muQ ealuation order inobva kurudyi kuenda kuruboshwe.

Iye zvino tine mabasa maviri makuru anodiwa pakuverenga, isu tinongoda kuwedzera zvishoma zvivakwa uye sevhisi yakagadzirira.

Matanho ekupedzisira

Isu tine preprocess uye updateAgg mabasa anoita basa rose. Asi zvichiri kudikanwa kuve nechokwadi chekuchinja chaiko kuburikidza nemaminitsi uye kuverenga indexes yekuunganidza. Chekutanga pane zvese, ngatitsanangure iyo init basa:

init:{
  tradeAgg:: 0#enlist[initWith]; // создаем пустую типизированную таблицу, enlist превращает словарь в таблицу, а 0# означает взять 0 элементов из нее
  currTime::00:00; // начнем с 0, :: означает, что присваивание в глобальную переменную
  currSyms::`u#`symbol$(); // `u# - превращает список в дерево, для ускорения поиска элементов
  offset::0; // индекс в tradeAgg, где начинается текущая минута 
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg; // кэш для последних значений roll колонок, таблица с ключом sym
 }

Isu tichatsanangurawo basa remupumburu, iro richachinja miniti iripo:

roll:{[tm]
  if[currTime>tm; :init[]]; // если перевалили за полночь, то просто вызовем init
  rollCache,::offset _ rollColumns#tradeAgg; // обновим кэш – взять roll колонки из aggTable, обрезать, вставить в rollCache
  offset::count tradeAgg;
  currSyms::`u#`$();
 }

Tichada basa rekuwedzera mavara matsva:

addSyms:{[syms]
  currSyms,::syms; // добавим в список известных
  // добавим в таблицу sym, time и rollColumns воспользовавшись обобщенным присваиванием.
  // Функция ^ подставляет значения по умолчанию для roll колонок, если символа нет в кэше. value flip table возвращает список колонок в таблице.
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime), (initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 }

Uye pakupedzisira, iyo upd basa (iro rechinyakare zita reiyi basa reQ masevhisi), iyo inodanwa nemutengi kuwedzera data:

upd:{[tblName;data] // tblName нам не нужно, но обычно сервис обрабатывает несколько таблиц 
  tm:exec distinct time from data:() xkey preprocess data; // preprocess & calc time
  updMinute[data] each tm; // добавим данные для каждой минуты
};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm]; // поменяем минуту, если необходимо
  data:select from data where time=tm; // фильтрация
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms]; // новые символы
  updateAgg[`tradeAgg;offset+currSyms?syms;data]; // обновим агрегированную таблицу. Функция ? ищет индекс элементов списка справа в списке слева.
 };

Ndizvo zvose. Heino kodhi yakazara yesevhisi yedu, sezvakavimbiswa, mitsetse mishoma:

initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time;
rollColumns:`sym`cumVolume;

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size");
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
@[`aggExpression;specialCols;{"?[isFirst;inp`",y,";",x,"]"};string specialCols];
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}"; / '

init:{
  tradeAgg::0#enlist[initWith];
  currTime::00:00;
  currSyms::`u#`symbol$();
  offset::0;
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg;
 };
roll:{[tm]
  if[currTime>tm; :init[]];
  rollCache,::offset _ rollColumns#tradeAgg;
  offset::count tradeAgg;
  currSyms::`u#`$();
 };
addSyms:{[syms]
  currSyms,::syms;
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime),(initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 };

upd:{[tblName;data] updMinute[data] each exec distinct time from data:() xkey preprocess data};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm];
  data:select from data where time=tm;
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms];
  updateAgg[`tradeAgg;offset+currSyms?syms;data];
 };

Kuedza

Ngationgororei kushanda kwebasa racho. Kuti tiite izvi, ngatimhanyei mune imwe nzira (isa kodhi musevhisi.q faira) uye fonera iyo init basa:

q service.q –p 5566

q)init[]

Mune imwe console, tanga yechipiri Q maitiro uye ubatanidze kune yekutanga:

h:hopen `:host:5566
h:hopen 5566 // если оба на одном хосте

Kutanga, ngatigadzire rondedzero yezviratidzo - zviuru gumi zvimedu uye tiwedzere basa rekugadzira tafura isina kurongeka. Mune yechipiri console:

syms:`IBM`AAPL`GOOG,-9997?`8
rnd:{[n;t] ([] sym:n?syms; time:t+asc n#til 25; price:n?10f; size:n?10)}

Ndakawedzera zviratidzo zvitatu zvechokwadi kune rondedzero kuti zvive nyore kuzvitsvaga mutafura. Iyo rnd basa inogadzira tafura isina kurongeka ine n mitsara, apo nguva inosiyana kubva t kusvika t + 25 milliseconds.

Iye zvino unogona kuedza kutumira data kushumiro (wedzera maawa gumi ekutanga):

{h (`upd;`trade;rnd[10000;x])} each `time$00:00 + til 60*10

Unogona kutarisa sevhisi kuti tafura yakagadziridzwa:

c 25 200
select from tradeAgg where sym=`AAPL
-20#select from tradeAgg where sym=`AAPL

Mhinduro:

sym|time|high|low|firstPrice|lastPrice|firstSize|lastSize|numTrades|volume|pvolume|turnover|avgPrice|avgSize|vwap|cumVolume
--|--|--|--|--|--------------------------------
AAPL|09:27|9.258904|9.258904|9.258904|9.258904|8|8|1|8|9.258904|74.07123|9.258904|8|9.258904|2888
AAPL|09:28|9.068162|9.068162|9.068162|9.068162|7|7|1|7|9.068162|63.47713|9.068162|7|9.068162|2895
AAPL|09:31|4.680449|0.2011121|1.620827|0.2011121|1|5|4|14|9.569556|36.84342|2.392389|3.5|2.631673|2909
AAPL|09:33|2.812535|2.812535|2.812535|2.812535|6|6|1|6|2.812535|16.87521|2.812535|6|2.812535|2915
AAPL|09:34|5.099025|5.099025|5.099025|5.099025|4|4|1|4|5.099025|20.3961|5.099025|4|5.099025|2919

Ngatiitei zvino kuyedza kuyerwa kuti tione kuti ingani data iyo sevhisi inogona kugadzirisa paminiti. Rega ndikuyeuchidze kuti tinoseta nguva yekuvandudza kusvika makumi maviri neshanu milliseconds. Saizvozvo, iyo sevhisi inofanirwa (paavhareji) kukwana mune angangoita 25 milliseconds pagadziro kuti ipe vashandisi nguva yekukumbira data. Isa zvinotevera mukuita kwechipiri:

tm:10:00:00.000
stressTest:{[n] 1 string[tm]," "; times,::h ({st:.z.T; upd[`trade;x]; .z.T-st};rnd[n;tm]); tm+:25}
start:{[n] times::(); do[4800;stressTest[n]]; -1 " "; `min`avg`med`max!(min times;avg times;med times;max times)}

4800 maminetsi maviri. Unogona kuyedza kumhanya kutanga kwe1000 mitsara yega yega 25 milliseconds:

start 1000

Mune yangu kesi, mhedzisiro yacho yakatenderedza akati wandei milliseconds pakuvandudza. Saka ini ndichakurumidza kuwedzera huwandu hwemitsara kusvika gumi,10.000:

start 10000

Mhinduro:

min| 00:00:00.004
avg| 9.191458
med| 9f
max| 00:00:00.030

Zvakare, hapana chakakosha, asi iyi mitsara yemamiriyoni makumi maviri nemana paminiti, 24 zviuru pasekondi. Kweanopfuura 400 milliseconds, iyo yekuvandudza yakadzikira chete 25 nguva, sezviri pachena apo miniti yakachinja. Ngatiwedzere ku5:

start 100000

Mhinduro:

min| 00:00:00.013
avg| 25.11083
med| 24f
max| 00:00:00.108
q)sum times
00:02:00.532

Sezvauri kuona, sevhisi haigone kurarama, asi zvakadaro inokwanisa kuramba yakanyura. Huwandu hwakadaro hwedata (240 miriyoni mitsara paminiti) yakakura kwazvo; mumamiriro ezvinhu akadaro, zvakajairika kuburitsa akati wandei ma clones (kana kunyange akawanda emakoni) esevhisi, imwe neimwe inogadzirisa chikamu chemavara chete. Zvakadaro, mhedzisiro yacho inokatyamadza kumutauro wakadudzirwa unotarisa zvakanyanya pakuchengetwa kwedata.

Mubvunzo unogona kumuka wekuti sei nguva ichikura isiri-mutsara nehukuru hwega yega update. Chikonzero ndechekuti iyo shrink basa ibasa reC, iro rinonyanya kushanda kupfuura updateAgg. Kutanga kubva kune imwe saizi yekuvandudza (yakatenderedza zviuru gumi), updateAgg inosvika padenga rayo uyezve nguva yayo yekuuraya haienderane nehukuru hwekuvandudza. Imhaka yenhanho yekutanga Q iyo sevhisi inokwanisa kugaya mavhoriyamu e data. Izvi zvinoratidza kukosha kwazvakaita kusarudza algorithm chaiyo kana uchishanda nedata hombe. Imwe pfungwa ndeyekuchengetedza kwakaringana data mundangariro. Kana iyo data isina kuchengetwa columnarly kana isina kurairwa nenguva, saka isu taizojairana nechinhu chakadai seTLB cache yekupotsa - kusavapo kwekero yepeji yekurangarira mune processor kero cache. Kutsvaga kero kunotora kanenge ka10.000 kureba kana ikasabudirira, uye kana iyo data yakapararira, inogona kudzikamisa sevhisi kakawanda.

mhedziso

Muchinyorwa chino, ndakaratidza kuti KDB + neQ dhatabhesi haina kukodzera chete kuchengetedza data hombe uye nyore kuiwana kuburikidza nekusarudza, asiwo nekugadzira dhata data masevhisi anokwanisa kugaya mazana emamiriyoni emitsara / gigabytes yedata kunyangwe mukati. imwe chete Q maitiro. Mutauro weQ pachawo unobvumira kuita kwakanyatso kupfupika uye nekuita kwealgorithms ine chekuita nekugadzirisa data nekuda kwechimiro chayo chevector, yakavakirwa-mukati SQL dialect muturikiri uye seti yakabudirira kwazvo yemabasa eraibhurari.

Ini ndichacherechedza kuti zviri pamusoro ingori chikamu chezvinogona kuitwa naQ, ine zvimwe zvakasarudzika maficha zvakare. Semuyenzaniso, yakanyanyisa kupfava IPC protocol inodzima muganho pakati pega Q maitiro uye inokutendera kuti ubatanidze mazana ematanho aya kuita network imwe chete, inogona kuwanikwa pamaseva mazhinji munzvimbo dzakasiyana dzepasi.

Source: www.habr.com