Iimpawu ze-Q kunye nolwimi lwe-KDB+ usebenzisa umzekelo wenkonzo yexesha lokwenene

Unokufunda malunga nokuba yintoni isiseko se-KDB+, ulwimi lwenkqubo ye-Q, yintoni amandla abo kunye nobuthathaka bangaphambili. nqaku nangokufutshane kwintshayelelo. Kwinqaku, siya kuphumeza inkonzo kwi-Q eya kuqhuba umjelo wedatha engenayo kwaye ibale imisebenzi eyahlukeneyo yokudibanisa yonke iminithi kwimodi "yexesha langempela" (oko kukuthi, kuya kuba nexesha lokubala yonke into phambi kwesahlulo esilandelayo sedatha). Inqaku eliphambili le-Q kukuba lulwimi lwe-vector oluvumela ukuba ungasebenzi ngezinto ezilodwa, kodwa kunye noluhlu lwazo, ii-arrays kunye nezinye izinto ezinzima. Iilwimi ezinje nge-Q kunye nezihlobo zayo K, J, APL zidume ngobufutshane bazo. Rhoqo, inkqubo ethatha izikrini ezininzi zekhowudi kulwimi oluqhelekileyo njengeJava inokubhalwa kuzo kwimigca embalwa. Yile nto endifuna ukuyibonisa kweli nqaku.

Intshayelelo

I-KDB+ yi-database ye-columnar egxininise kwixabiso elikhulu kakhulu ledatha, ecwangciswe ngendlela ethile (ingakumbi ngexesha). Isetyenziselwa ikakhulu kumaziko emali - iibhanki, iimali zotyalo-mali, iinkampani zomshuwalense. Ulwimi lwe-Q lulwimi lwangaphakathi lwe-KDB+ olukuvumela ukuba usebenze ngokufanelekileyo ngale datha. Umbono we-Q ubufutshane kunye nokusebenza kakuhle, ngelixa ukucaca kunikelwa. Oku kuqinisekiswa kukuba ulwimi lwe-vector luya kuba nzima ukuyiqonda nakweyiphi na imeko, kwaye ubufutshane kunye nobutyebi bokurekhoda kukuvumela ukuba ubone inxalenye enkulu yeprogram kwisikrini esinye, ekugqibeleni yenza kube lula ukuyiqonda.

Kweli nqaku siphumeza inkqubo ye-Q epheleleyo kwaye unokufuna ukuyizama. Ukwenza oku, uya kufuna i-Q ngokwayo. www.kx.com. Apho, ukuba unomdla, uya kufumana ulwazi lwereferensi ku-Q, incwadi Q Kubantu Abafayo kunye namanqaku ahlukeneyo ngalo mbandela.

Џџ ѕЃ ° °

Kukho umthombo othumela itafile kunye nedatha rhoqo nge-25 milliseconds. Ekubeni i-KDB+ isetyenziswa ikakhulu kwezemali, siya kucinga ukuba le yitheyibhile yentengiselwano (yorhwebo), enezintlu zilandelayo: ixesha (ixesha kwi-milliseconds), i-sym (igama lenkampani kwi-stock exchange - IBM, AAPL,…), ixabiso (ixabiso ezathengwa ngalo izabelo), ubungakanani (ubungakanani bentengiselwano). Ikhefu le-25 millisecond lingenasizathu, alikho lincinci kwaye alikho lide kakhulu. Ubukho bayo buthetha ukuba idatha iza kwinkonzo esele ikhuselwe. Kuya kuba lula ukuphumeza i-buffering kwicala lenkonzo, kubandakanywa isithinteli esiguqukayo ngokuxhomekeke kumthwalo wangoku, kodwa ukwenza lula, siya kugxila kwisithuba esisisigxina.

Inkonzo mayibale umzuzu ngamnye ngesimboli ngasinye esingenayo kwikholamu yesym iseti yeaggregating functions - max price, avg price, sum size, njl. ulwazi oluluncedo. Ukuze kube lula, siya kucinga ukuba yonke imisebenzi ingabalwa ngokunyuka, i.e. ukufumana ixabiso elitsha, kwanele ukwazi amanani amabini - amadala kunye namaxabiso angenayo. Umzekelo, imisebenzi max, i-avareji, sum inale propati, kodwa umsebenzi ophakathi awunawo.

Siya kuphinda sicinge ukuba ukuhanjiswa kwedatha engenayo ixesha liyalelwa. Oku kuya kusinika ithuba lokusebenza kuphela ngomzuzu wokugqibela. Ngokwenza umsebenzi, kwanele ukukwazi ukusebenza kunye nemizuzu yangoku kunye neyodlulileyo xa kwenzeka ukuba uhlaziyo lufike emva kwexesha. Ukuze kube lula, asiyi kuqwalasela le meko.

Imisebenzi yokudibanisa

Imisebenzi yokudibanisa efunekayo idweliswe ngezantsi. Ndithathe uninzi lwazo kangangoko ndinako ukwandisa umthwalo kwinkonzo:

• eliphezulu - ixabiso eliphezulu - ixabiso eliphezulu ngomzuzu.
• ixabiso eliphantsi - ixabiso elincinci - ixabiso elincinci ngomzuzu.
• firstPrice - ixabiso lokuqala - ixabiso lokuqala ngomzuzu.
• Ixabiso lokugqibela - ixabiso lokugqibela - ixabiso lokugqibela ngomzuzu.
• Ubungakanani bokuqala - ubungakanani bokuqala - ubungakanani bokuqala borhwebo ngomzuzu.
• Ubungakanani bokugqibela - ubukhulu bokugqibela - ubungakanani bokugqibela borhwebo ngomzuzu.
• numTrades - count i - inani lorhwebo ngomzuzu.
• umthamo - ubungakanani besamba - isixa seesayizi zorhwebo ngomzuzu.
• pvolume – ixabiso elisisixa – isixa samaxabiso ngomzuzu, esifunekayo kwi-avgPrice.
• - ixabiso lentengo yexabiso * ubungakanani - umthamo opheleleyo wentengiselwano ngomzuzu.
• avgPrice – pvolume%numTrades – ixabiso eliphakathi ngomzuzu.
• avgSize – volume%numTrades – avareji ubungakanani borhwebo ngomzuzu.
• i-vwap – i-turnover% volume – ixabiso eliphakathi ngomzuzu lilinganiswe ngobungakanani bentengiselwano.
• cumVolume – isixa somthamo-ubungakanani obuqokelelweyo bentengiselwano ngalo lonke ixesha.

Makhe sixoxe ngokukhawuleza inqaku elinye elingacacanga - indlela yokuqalisa le kholamu okokuqala kunye nomzuzu ngamnye olandelayo. Eminye imiqolo yohlobo lwexabiso lokuqala kufuneka iqalwe ukuze ingabikho xesha ngalinye; Ezinye iintlobo zevolumu kufuneka zihlale zisetelwe ku-0. Kukwakho neekholamu ezifuna indlela edibeneyo - umzekelo, i-cumVolume kufuneka ikhutshelwe kumzuzu odlulileyo, kwaye eyokuqala isete ku-0. uhlobo (lufana nerekhodi):

// list ! list – создать словарь, 0n – float null, 0N – long null, `sym – тип символ, `sym1`sym2 – список символов
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time; // список всех вычисляемых колонок, reverse объяснен ниже

Ndongeze i-sym kunye nexesha kwisichazi-magama ukuze kube lula, ngoku i-initWith ngumgca osele ulungile ukusuka kwitheyibhile edityanisiweyo yokugqibela, apho ihlala khona ukuseta i-sym kunye nexesha elichanekileyo. Ungayisebenzisa ukongeza imiqolo emitsha kwitafile.

Siza kufuna ii-aggCols xa sisenza umsebenzi wokudibanisa. Uluhlu kufuneka luguqulwe ngenxa yolandelelwano amabinzana akwi-Q avavanywa ngayo (ukusuka ekunene ukuya ekhohlo). Injongo kukuqinisekisa ukubala ukusuka phezulu ukuya kwi-cumVolume, kuba ezinye iikholomu zixhomekeke kwezo zangaphambili.

Iikholamu ezifuna ukukhutshelwa kumzuzu omtsha ukusuka kulowo wangaphambili, ikholam yesim yongezwa ukwenzela ukuba kube lula:

rollColumns:`sym`cumVolume;

Ngoku masahlule iikholamu zibe ngamaqela ngokwendlela ekufanele zihlaziywe ngayo. Iindidi ezintathu zinokwahlulwa:

1. I-Accumulators (ivolumu, i-turnover, ..) - kufuneka songeze ixabiso elingenayo kulowo wangaphambili.
2. Ngenqaku elikhethekileyo (eliphezulu, eliphantsi, ..) - ixabiso lokuqala kumzuzu lithathwa kwidatha engenayo, abanye babalwa ngokusebenzisa umsebenzi.
3. Phumla. Isoloko ibalwa kusetyenziswa umsebenzi.

Makhe sichaze iinguqu kwezi klasi:

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

Ulandelelwano lokubala

Siza kuhlaziya itheyibhile edityanisiweyo ngokwezigaba ezibini. Ukusebenza kakuhle, siqale sinciphise itafile engenayo ukuze kubekho umqolo omnye kuphela kumlinganiswa ngamnye kunye nomzuzu. Inyaniso yokuba yonke imisebenzi yethu iyenyuka kwaye idibanisa iqinisekisa ukuba umphumo weli nyathelo elongezelelweyo awuyi kutshintsha. Ungayicutha itafile usebenzisa ukukhetha:

select high:max price, low:min price … by sym,time.minute from table

Le ndlela ine-disavantage - isethi yeentsika ezibalwayo zichazwe kwangaphambili. Ngethamsanqa, kwi-Q, ukhetho lukwaphunyeziwe njengomsebenzi apho unokutshintsha iingxoxo ezenziwe ngokuguqukayo:

?[table;whereClause;byClause;selectClause]

Andizi kuyichaza ngokweenkcukacha ifomathi yeengxoxo; kwimeko yethu, kuphela kwaye ukhethe iintetho eziya kuba zingekho ngqiqweni kwaye kufuneka zibe zizichazi-magama zeekholamu zefomu! iintetho. Ke, umsebenzi wokuncipha ungachazwa ngolu hlobo lulandelayo:

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size"); // each это функция map в Q для одного списка
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

Ukucacisa, ndisebenzise umsebenzi wokwahlulahlula, ojika umtya ngentetho ye-Q ibe lixabiso elinokudluliselwa kumsebenzi wokulinganisa kwaye ofunekayo kumsebenzi okhethiweyo. Kwakhona qaphela ukuba inkqubo yangaphambili ichazwa njengoqikelelo (oko kukuthi, umsebenzi oneengxoxo ezichazwe ngokuyinxenye) womsebenzi okhethiweyo, ingxabano enye (itafile) ayikho. Ukuba sisebenzisa i-preprocess kwitafile, siya kufumana itafile ecinezelweyo.

Inqanaba lesibini kukuhlaziya itheyibhile edityanisiweyo. Masiqale sibhale i-algorithm kwipseudocode:

for each sym in inputTable
  idx: row index in agg table for sym+currentTime;
  aggTable[idx;`high]: aggTable[idx;`high] | inputTable[sym;`high];
  aggTable[idx;`volume]: aggTable[idx;`volume] + inputTable[sym;`volume];
  …

Kwi-Q, kuqhelekile ukusebenzisa imephu/ukunciphisa imisebenzi endaweni yeelophu. Kodwa ekubeni i-Q ilulwimi lwe-vector kwaye sinokusebenzisa ngokulula yonke imisebenzi kuzo zonke iisimboli ngaxeshanye, emva koko kuqikelelo lokuqala singenza ngaphandle kwe-loop konke konke, siqhuba imisebenzi kuzo zonke iisimboli ngaxeshanye:

idx:calcIdx inputTable;
row:aggTable idx;
aggTable[idx;`high]: row[`high] | inputTable`high;
aggTable[idx;`volume]: row[`volume] + inputTable`volume;
…

Kodwa singaya phambili, i-Q inomsebenzi owahlukileyo kwaye onamandla kakhulu-umsebenzi wesabelo ngokubanzi. Ikuvumela ukuba utshintshe uluhlu lwamaxabiso kulwakhiwo lwedatha oluntsonkothileyo usebenzisa uluhlu lwezalathisi, imisebenzi kunye neengxoxo. Kwimeko yethu ibonakala ngolu hlobo:

idx:calcIdx inputTable;
rows:aggTable idx;
// .[target;(idx0;idx1;..);function;argument] ~ target[idx 0;idx 1;…]: function[target[idx 0;idx 1;…];argument], в нашем случае функция – это присваивание
.[aggTable;(idx;aggCols);:;flip (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Ngelishwa, ukwabela itafile udinga uluhlu lwemiqolo, hayi iikholamu, kwaye kufuneka uguqule imatrix (uluhlu lwezintlu kuluhlu lwemiqolo) usebenzisa umsebenzi weflip. Oku kuyabiza kwitafile enkulu, ngoko endaweni yoko sisebenzisa isabelo esiqhelekileyo kwikholamu nganye ngokwahlukeneyo, sisebenzisa umsebenzi wemephu (okhangeleka ngathi sisimeli-mazwi):

.[aggTable;;:;]'[(idx;)each aggCols; (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Siphinda sisebenzise uqikelelo lomsebenzi. Kwakhona qaphela ukuba ku Q, ukwenza uluhlu kukwangumsebenzi kwaye singawubiza ngokusebenzisa (imephu) umsebenzi ngamnye ukufumana uluhlu loluhlu.

Ukuqinisekisa ukuba isethi yeentsika ezibaliweyo ayilungiswanga, siya kudala le ntetho ingentla ngamandla. Masiqale sichaze imisebenzi yokubala ikholamu nganye, sisebenzisa umqolo kunye noguquguquko lwe-inp ukubhekisa kwidatha edityanisiweyo kunye negalelo:

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!
 ("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");

Ezinye iintsika zikhethekile; ixabiso labo lokuqala akufunekanga libalwe ngumsebenzi. Singamisela ukuba yeyokuqala ngomqolo[`numTrades] ikholamu - ukuba iqulethe u-0, ngoko ixabiso lelokuqala. Q unomsebenzi okhethiweyo - ?[Uluhlu lweBoolean;uluhlu1;uluhlu2] - olukhetha ixabiso kuluhlu loku-1 okanye lwesi-2 ngokuxhomekeke kwimeko kwimpikiswano yokuqala:

// high -> ?[isFirst;inp`high;row[`high]|inp`high]
// @ - тоже обобщенное присваивание для случая когда индекс неглубокий
@[`aggExpression;specialCols;{[x;y]"?[isFirst;inp`",y,";",x,"]"};string specialCols];

Apha ndibize umsebenzi owenziwe ngokubanzi ngomsebenzi wam (inkcazo kwiibrayisi ezigobileyo). Ifumana ixabiso langoku (ingxabano yokuqala) kunye nengxabano eyongezelelweyo, endiyidlulayo kwi-parameter yesi-4.

Masiyongeze izithethi zebhetri ngokwahlukeneyo, kuba umsebenzi uyafana kubo:

// volume -> row[`volume]+inp`volume
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;

Esi sisabelo esiqhelekileyo ngokwemigangatho ye-Q, kodwa ndinikezela ngoluhlu lwamaxabiso kanye. Okokugqibela, makhe senze owona msebenzi uphambili:

// ":",/:aggExprs ~ map[{":",x};aggExpr] => ":row[`high]|inp`high" присвоим вычисленное значение переменной, потому что некоторые колонки зависят от уже вычисленных значений
// string[cols],'exprs ~ map[,;string[cols];exprs] => "high:row[`high]|inp`high" завершим создание присваивания. ,’ расшифровывается как map[concat]
// ";" sv exprs – String from Vector (sv), соединяет список строк вставляя “;” посредине
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}";

Ngolu phawu, ndidala ngokuguqukayo umsebenzi ukusuka kumtya oqulathe intetho endiyinike ngasentla. Isiphumo siya kujongeka ngolu hlobo:

{[aggTable;idx;inp] rows:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols ;(cumVolume:row[`cumVolume]+inp`cumVolume;… ; high:?[isFirst;inp`high;row[`high]|inp`high])]}

Ulandelelwano lovavanyo lwekholamu lujikisiwe kuba kuQ ulandelelwano lovavanyo lusuka ekunene ukuya ekhohlo.

Ngoku sinemisebenzi emibini ephambili eyimfuneko yokubala, sifuna nje ukongeza isiseko esincinci kwaye inkonzo ilungile.

Amanyathelo okugqibela

Sine-preprocess kunye nokuhlaziya imisebenzi yeAgg eyenza wonke umsebenzi. Kodwa kusekho imfuneko yokuqinisekisa utshintsho oluchanekileyo ngemizuzu kwaye ubale izalathisi zokuhlanganisa. Okokuqala, makhe sichaze umsebenzi we-init:

init:{
  tradeAgg:: 0#enlist[initWith]; // создаем пустую типизированную таблицу, enlist превращает словарь в таблицу, а 0# означает взять 0 элементов из нее
  currTime::00:00; // начнем с 0, :: означает, что присваивание в глобальную переменную
  currSyms::`u#`symbol$(); // `u# - превращает список в дерево, для ускорения поиска элементов
  offset::0; // индекс в tradeAgg, где начинается текущая минута 
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg; // кэш для последних значений roll колонок, таблица с ключом sym
 }

Siza kuphinda sichaze umsebenzi we-roll, oya kutshintsha umzuzu wangoku:

roll:{[tm]
  if[currTime>tm; :init[]]; // если перевалили за полночь, то просто вызовем init
  rollCache,::offset _ rollColumns#tradeAgg; // обновим кэш – взять roll колонки из aggTable, обрезать, вставить в rollCache
  offset::count tradeAgg;
  currSyms::`u#`$();
 }

Siza kufuna umsebenzi wokongeza abalinganiswa abatsha:

addSyms:{[syms]
  currSyms,::syms; // добавим в список известных
  // добавим в таблицу sym, time и rollColumns воспользовавшись обобщенным присваиванием.
  // Функция ^ подставляет значения по умолчанию для roll колонок, если символа нет в кэше. value flip table возвращает список колонок в таблице.
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime), (initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 }

Kwaye ekugqibeleni, umsebenzi wokuhlaziya (igama lemveli lalo msebenzi kwiinkonzo ze-Q), ebizwa ngokuba ngumxhasi ukongeza idatha:

upd:{[tblName;data] // tblName нам не нужно, но обычно сервис обрабатывает несколько таблиц 
  tm:exec distinct time from data:() xkey preprocess data; // preprocess & calc time
  updMinute[data] each tm; // добавим данные для каждой минуты
};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm]; // поменяем минуту, если необходимо
  data:select from data where time=tm; // фильтрация
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms]; // новые символы
  updateAgg[`tradeAgg;offset+currSyms?syms;data]; // обновим агрегированную таблицу. Функция ? ищет индекс элементов списка справа в списке слева.
 };

Kuko konke. Nantsi ikhowudi epheleleyo yenkonzo yethu, njengoko kuthembisiwe, imigca embalwa nje:

initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time;
rollColumns:`sym`cumVolume;

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size");
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
@[`aggExpression;specialCols;{"?[isFirst;inp`",y,";",x,"]"};string specialCols];
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}"; / '

init:{
  tradeAgg::0#enlist[initWith];
  currTime::00:00;
  currSyms::`u#`symbol$();
  offset::0;
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg;
 };
roll:{[tm]
  if[currTime>tm; :init[]];
  rollCache,::offset _ rollColumns#tradeAgg;
  offset::count tradeAgg;
  currSyms::`u#`$();
 };
addSyms:{[syms]
  currSyms,::syms;
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime),(initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 };

upd:{[tblName;data] updMinute[data] each exec distinct time from data:() xkey preprocess data};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm];
  data:select from data where time=tm;
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms];
  updateAgg[`tradeAgg;offset+currSyms?syms;data];
 };

Ukuvavanywa

Makhe sijonge ukusebenza kwenkonzo. Ukwenza oku, masiyiqhube kwinkqubo eyahlukileyo (beka ikhowudi kwinkonzo.q ifayile) kwaye ubize umsebenzi we-init:

q service.q –p 5566

q)init[]

Kwenye i-console, qalisa inkqubo yesibini ye-Q kwaye uqhagamshele kweyokuqala:

h:hopen `:host:5566
h:hopen 5566 // если оба на одном хосте

Okokuqala, makhe uluhlu lweesimboli - amaqhekeza angama-10000 kwaye songeze umsebenzi ukwenza itafile engacwangciswanga. Kwi-console yesibini:

syms:`IBM`AAPL`GOOG,-9997?`8
rnd:{[n;t] ([] sym:n?syms; time:t+asc n#til 25; price:n?10f; size:n?10)}

Ndongeze iisimboli ezintathu zokwenyani kuluhlu ukwenza kube lula ukuzijonga etafileni. Umsebenzi we-rnd wenza itheyibhile engacwangciswanga enemiqolo n, apho ixesha liyahluka ukusuka ku-t ukuya ku-t +25 milliseconds.

Ngoku ungazama ukuthumela idatha kwinkonzo (yongeza iiyure ezilishumi zokuqala):

{h (`upd;`trade;rnd[10000;x])} each `time$00:00 + til 60*10

Ungajonga kwinkonzo ukuba itheyibhile ihlaziywe:

c 25 200
select from tradeAgg where sym=`AAPL
-20#select from tradeAgg where sym=`AAPL

Isiphumo:

sym|time|high|low|firstPrice|lastPrice|firstSize|lastSize|numTrades|volume|pvolume|turnover|avgPrice|avgSize|vwap|cumVolume
--|--|--|--|--|--------------------------------
AAPL|09:27|9.258904|9.258904|9.258904|9.258904|8|8|1|8|9.258904|74.07123|9.258904|8|9.258904|2888
AAPL|09:28|9.068162|9.068162|9.068162|9.068162|7|7|1|7|9.068162|63.47713|9.068162|7|9.068162|2895
AAPL|09:31|4.680449|0.2011121|1.620827|0.2011121|1|5|4|14|9.569556|36.84342|2.392389|3.5|2.631673|2909
AAPL|09:33|2.812535|2.812535|2.812535|2.812535|6|6|1|6|2.812535|16.87521|2.812535|6|2.812535|2915
AAPL|09:34|5.099025|5.099025|5.099025|5.099025|4|4|1|4|5.099025|20.3961|5.099025|4|5.099025|2919

Ngoku masenze uvavanyo lomthwalo ukuze sifumanise ukuba ingakanani idatha enokwenziwa yinkonzo ngomzuzu. Mandikukhumbuze ukuba siseta isithuba sohlaziyo sibe yi-25 milliseconds. Ngokufanelekileyo, inkonzo kufuneka (ngokomndilili) ingene ubuncinane kwii-milliseconds ze-20 ngohlaziyo ngalunye ukunika abasebenzisi ixesha lokucela idatha. Ngenisa oku kulandelayo kwinkqubo yesibini:

tm:10:00:00.000
stressTest:{[n] 1 string[tm]," "; times,::h ({st:.z.T; upd[`trade;x]; .z.T-st};rnd[n;tm]); tm+:25}
start:{[n] times::(); do[4800;stressTest[n]]; -1 " "; `min`avg`med`max!(min times;avg times;med times;max times)}

4800 yimizuzu emibini. Ungazama ukubaleka kuqala kwimiqolo eyi-1000 rhoqo nge-25 milliseconds:

start 1000

Kwimeko yam, isiphumo sijikeleze i-millisecond ezimbalwa ngohlaziyo ngalunye. Ngoko ke ndiza kwandisa ngokukhawuleza inani lemiqolo ukuya kwi-10.000:

start 10000

Isiphumo:

min| 00:00:00.004
avg| 9.191458
med| 9f
max| 00:00:00.030

Kwakhona, akukho nto ikhethekileyo, kodwa le yimigca ye-24 yezigidi ngomzuzu, i-400 lamawaka ngomzuzwana. Ngaphezulu kwe-25 milliseconds, ukuhlaziywa kwanciphisa kuphela amaxesha e-5, ngokucacileyo xa umzuzu utshintshile. Masinyuke siye kwi-100.000:

start 100000

Isiphumo:

min| 00:00:00.013
avg| 25.11083
med| 24f
max| 00:00:00.108
q)sum times
00:02:00.532

Njengoko ubona, inkonzo ayinakukwazi ukumelana nayo, kodwa nangona kunjalo iyakwazi ukuhlala ihamba. Umthamo onjalo wedatha (i-240 yezigidi zeerowu ngomzuzu) mkhulu ngokugqithisileyo kwiimeko ezinjalo, kuqhelekile ukusungula ii-clones ezininzi (okanye ii-clones ezininzi) zenkonzo, nganye eyenza inxalenye yabalinganiswa kuphela. Sekunjalo, umphumo uyancomeka kulwimi olutolikwayo olujolise ikakhulu kugcino lwedatha.

Umbuzo unokuvela malunga nokuba kutheni ixesha likhula ngokungafaniyo nobungakanani bohlaziyo ngalunye. Isizathu sesokuba umsebenzi wokucutha ngenene ngumsebenzi we-C, osebenza ngakumbi kunokuhlaziyaAgg. Ukuqala kwisayizi ethile yohlaziyo (malunga ne-10.000), i-updateAgg ifikelela kwisilingi yayo kwaye ke ixesha layo lokuphumeza alixhomekeke kubungakanani bohlaziyo. Kungenxa yenyathelo lokuqala u-Q ukuba inkonzo iyakwazi ukwetyisa umthamo onjalo wedatha. Oku kugxininisa indlela ekubaluleke ngayo ukukhetha i-algorithm efanelekileyo xa usebenza ngedatha enkulu. Enye inqaku kukugcinwa okuchanekileyo kwedatha kwimemori. Ukuba idatha ayigcinwanga nge-columnar okanye ayizange ilandelwe ngexesha, ngoko siya kuqhelana nento efana ne-TLB cache miss - ukungabikho kwedilesi yekhasi lememori kwi-cache yedilesi yeprosesa. Ukukhangela idilesi kuthatha malunga namaxesha angama-30 ubude ukuba awuphumelelanga, kwaye ukuba idatha isasazekile, inokunciphisa inkonzo ngamaxesha amaninzi.

isiphelo

Kweli nqaku, ndibonise ukuba i-database ye-KDB + kunye ne-Q ayifanelekanga kuphela ukugcina idatha enkulu kwaye ifikeleleke ngokulula ngokukhetha, kodwa nokudala iinkonzo zokucwangcisa idatha ezikwazi ukwetyisa amakhulu ezigidi zemiqolo / iigigabytes zedatha nakwi. inkqubo enye Q . Ulwimi lwe-Q ngokwalo luvumela ukuphunyezwa ngokugqithisileyo nangokufanelekileyo kwe-algorithms enxulumene nokusetyenzwa kwedatha ngenxa yendalo yayo ye-vector, itoliki ye-dilect eyakhelwe-ngaphakathi ye-SQL kunye neseti ephumelele kakhulu yemisebenzi yethala leencwadi.

Ndiza kuqaphela ukuba oku kungasentla yinxalenye nje yezinto ezinokwenziwa ngu-Q, inezinye iimpawu ezizodwa ngokunjalo. Umzekelo, iprotocol ye-IPC elula kakhulu ecima umda phakathi kwenkqubo nganye ye-Q kwaye ikuvumela ukuba udibanise amakhulu ezi nkqubo zibe kuthungelwano olunye, olunokuthi lubekwe kuthotho lweeseva kwiindawo ezahlukeneyo zehlabathi.

umthombo: www.habr.com