Izici zolimi lwe-Q ne-KDB+ zisebenzisa isibonelo sesevisi yesikhathi sangempela

Ungafunda mayelana nesisekelo se-KDB+, ulimi lohlelo lwe-Q, ukuthi amandla abo kanye nobuthakathaka buyini kweyami eyedlule. isihloko futhi kafushane esethulweni. Esihlokweni, sizosebenzisa isevisi ku-Q ezocubungula ukusakazwa kwedatha engenayo futhi ibale imisebenzi ehlukahlukene yokuhlanganisa njalo ngeminithi ngemodi "yesikhathi sangempela" (okungukuthi, izoba nesikhathi sokubala yonke into ngaphambi kwengxenye elandelayo yedatha). Isici esiyinhloko se-Q ukuthi iwulimi lwe-vector oluvumela ukuthi ungasebenzisi ngezinto ezilodwa, kodwa ngama-arrays azo, ama-arrays of arrays nezinye izinto eziyinkimbinkimbi. Izilimi ezinjengo-Q nezihlobo zayo K, J, APL zidume ngobufushane bazo. Ngokuvamile, uhlelo oluthatha izikrini ezimbalwa zekhodi ngolimi olujwayelekile njenge-Java lungabhalwa kuzo ngemigqa embalwa. Yilokhu engifuna ukukubonisa kulesi sihloko.

Isingeniso

I-KDB+ isizindalwazi sekholomu esigxile enanini elikhulu kakhulu ledatha, ehlelwe ngendlela ethile (ikakhulukazi ngesikhathi). Isetshenziswa ngokuyinhloko ezikhungweni zezezimali - amabhange, izimali zokutshala izimali, izinkampani zomshuwalense. Ulimi lwe-Q ulimi lwangaphakathi lwe-KDB+ olukuvumela ukuthi usebenze ngempumelelo nale datha. I-Q ideology imfishane nokusebenza kahle, kuyilapho ukucaca kuyanikelwa. Lokhu kufakazelwa yiqiniso lokuthi ulimi lwe-vector luzoba nzima ukuluqonda kunoma yikuphi, futhi ubufushane nokunotha kokuqoshwa kukuvumela ukuthi ubone ingxenye enkulu kakhulu yohlelo esikrinini esisodwa, okwenza kube lula ukuyiqonda.

Kulesi sihloko sisebenzisa uhlelo olugcwele ku-Q futhi ungase ufune ukuluzama. Ukwenza lokhu, uzodinga u-Q wangempela. Ungalanda inguqulo yamahhala engu-32-bit kuwebhusayithi yenkampani ye-kx - www.kx.com. Lapho, uma unentshisekelo, uzothola imininingwane yereferensi ku-Q, incwadi Q Kwabantu Abafayo kanye nezihloko ezahlukahlukene ngalesi sihloko.

Ukwakheka kwenkinga

Kunomthombo othumela ithebula elinedatha njalo ngama-millisecond angama-25. Njengoba i-KDB+ isetshenziswa ngokuyinhloko kwezezimali, sizothatha ngokuthi leli itafula lokuthengiselana (ukuhweba), elinamakholomu alandelayo: isikhathi (isikhathi ngama-millisecond), i-sym (igama lenkampani esitokoleni - IBM, AAPL,…), inani (inani lapho amasheya athengwa khona), usayizi (usayizi wokwenziwe). Isikhawu esingama-millisecond angu-25 asinasizathu, asincane kakhulu futhi asiside kakhulu. Ukuba khona kwayo kusho ukuthi idatha iza kusevisi isivele ivikelwe. Kungaba lula ukusebenzisa ukugcina kumthamo ohlangothini lwesevisi, okuhlanganisa ukugcina kumthamo okuguquguqukayo kuye ngomthwalo wamanje, kodwa ukuze kube lula, sizogxila kusikhawu esinqunyiwe.

Isevisi kufanele ibale iminithi ngalinye ngophawu ngalunye olungenayo olusuka kukholomu yohlelo isethi yemisebenzi yokuhlanganisa - inani eliphezulu, inani elimaphakathi, usayizi wesamba, njll. ulwazi oluwusizo. Ukwenza kube lula, sizocabanga ukuthi yonke imisebenzi ingabalwa ngokuqhubekayo, i.e. ukuze uthole inani elisha, kwanele ukwazi izinombolo ezimbili - amanani amadala nangenayo. Isibonelo, imisebenzi ubuningi, isilinganiso, isamba sinalesi sici, kodwa umsebenzi we-median awunawo.

Sizophinde sicabange ukuthi ukusakazwa kwedatha engenayo kuhlelwe isikhathi. Lokhu kuzosinika ithuba lokusebenza ngomzuzu wokugcina kuphela. Empeleni, kwanele ukuthi ukwazi ukusebenza ngemizuzu yamanje neyangaphambilini uma kwenzeka ezinye izibuyekezo zephuzile. Ukuze kube lula, ngeke sicabangele leli cala.

Imisebenzi yokuhlanganisa

Imisebenzi edingekayo yokuhlanganisa ibalwe ngezansi. Ngithathe abaningi babo ngangokunokwenzeka ukuze ngandise umthwalo wenkonzo:

• intengo ephezulu – intengo ephezulu ngomzuzu.
• intengo ephansi – encane – inani elincane ngomzuzu.
• firstPrice - intengo yokuqala - intengo yokuqala ngomzuzu.
• lastPrice - intengo yokugcina - intengo yokugcina ngomzuzu.
• OkokuqalaSize - usayizi wokuqala - usayizi wokuqala wokuhweba ngomzuzu.
• lastSize - usayizi wokugcina - usayizi wokugcina wokuhweba ngomzuzu.
• numTrades - count i - inombolo yokuhweba ngomzuzu.
• umthamo - usayizi wesamba - isamba samasayizi okuhweba ngomzuzu.
• i-pvolume – isamba senani – isamba sezintengo ngomzuzu, esidingekayo ku-avgPrice.
• - isamba sentengo yenzuzo * usayizi - umthamo ophelele wokuthengiselana ngomzuzu.
• avgPrice – pvolume%numTrades – isilinganiso senani ngomzuzu.
• avgSize - ivolumu%numTrades - isilinganiso sosayizi wokuhweba ngomzuzu.
• vwap – turnover%volume – intengo emaphakathi ngomzuzu enesisindo ngosayizi wokwenziwe.
• cumVolume - isamba sevolumu - usayizi oqoqiwe wokuthengiselana phakathi naso sonke isikhathi.

Masixoxe ngokushesha ngephuzu elilodwa elingabonakali - ukuthi ungawaqalisa kanjani lawa makholomu okokuqala nangomzuzu ngamunye olandelayo. Amanye amakholomu ohlobo lwe-firstPrice kufanele aqaliswe ukuze angasebenzi isikhathi ngasinye; inani lawo alichazwanga. Ezinye izinhlobo zevolumu kufanele zihlale zisethelwe ku-0. Kukhona futhi amakholomu adinga indlela ehlanganisiwe - isibonelo, i-cumVolume kufanele ikopishwe ukusuka kumzuzu wangaphambilini, futhi kweyokuqala isethwe ku-0. Masimise yonke le mingcele sisebenzisa idatha yesichazamazwi. uhlobo (okufana nerekhodi):

// list ! list – создать словарь, 0n – float null, 0N – long null, `sym – тип символ, `sym1`sym2 – список символов
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time; // список всех вычисляемых колонок, reverse объяснен ниже

Ngingeze i-sym nesikhathi kusichazamazwi ukuze kube lula, manje i-initWith ingumugqa osuvele usuwenziwe osuka kuthebula elihlanganisiwe lokugcina, lapho kusala khona ukusetha i-sym nesikhathi esifanele. Ungayisebenzisa ukuze wengeze imigqa emisha etafuleni.

Sizodinga ama-aggCols lapho sidala umsebenzi wokuhlanganisa. Uhlu kufanele luguqulwe ngenxa yendlela izinkulumo eziku-Q ezihlolwa ngayo (kusuka kwesokudla kuye kwesokunxele). Umgomo uwukuqinisekisa ukuthi isibalo sisuka phezulu siye ku-cumVolume, njengoba amanye amakholomu ancike kwedlule.

Amakholomu adinga ukukopishelwa kuminithi elisha ukusuka kwedlule, ikholomu yosim yengezwa ukuze kube lula:

rollColumns:`sym`cumVolume;

Manje ake sihlukanise amakholomu ngamaqembu ngendlela okufanele abuyekezwe ngayo. Izinhlobo ezintathu zingahlukaniswa:

1. Ama-accumulators (ivolumu, inzuzo, ..) - kufanele sengeze inani elingenayo kwedlule.
2. Ngephuzu elikhethekile (eliphezulu, eliphansi, ..) - inani lokuqala ngomzuzu lithathwa kudatha engenayo, okusele kubalwa kusetshenziswa umsebenzi.
3. Phumula. Njalo kubalwa kusetshenziswa umsebenzi.

Ake sichaze okuguquguqukayo kulawa makilasi:

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

I-oda lokubala

Sizobuyekeza ithebula elihlanganisiwe ngezigaba ezimbili. Ngokusebenza kahle, siqala sinciphisa ithebula elingenayo ukuze kube nomugqa owodwa kuphela wohlamvu nomzuzu ngamunye. Iqiniso lokuthi yonke imisebenzi yethu iyakhula futhi iyahlanganisa iqinisekisa ukuthi umphumela walesi sinyathelo esengeziwe ngeke ushintshe. Unganciphisa ithebula usebenzisa ukukhetha:

select high:max price, low:min price … by sym,time.minute from table

Le ndlela inesimo esibi - isethi yamakholomu abaliwe ichazwe ngaphambilini. Ngenhlanhla, ku-Q, ukukhetha kuphinda kusetshenziswe njengomsebenzi lapho ungashintsha khona ama-agumenti adalwe ngamandla:

?[table;whereClause;byClause;selectClause]

Ngeke ngichaze ngokuningiliziwe ifomethi yama-agumenti; esimweni sethu, kuphela noma ngokukhetha izinkulumo eziyoba ezingasho lutho futhi kufanele zibe izichazamazwi zamakholomu efomu!izinkulumo. Ngakho, umsebenzi wokuncipha ungachazwa kanje:

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size"); // each это функция map в Q для одного списка
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

Ukuze kube sobala, ngisebenzise umsebenzi wokuhlaziya, oguqula iyunithi yezinhlamvu ngesisho esingu-Q sibe inani elingadluliswa kumsebenzi we-eval futhi odingekayo ekukhetheni umsebenzi. Futhi qaphela ukuthi ukucubungula kwangaphambili kuchazwa njengokuqagela (okungukuthi, umsebenzi onama-agumenti achazwe ngokwengxenye) womsebenzi okhethiwe, i-agumenti eyodwa (ithebula) ayikho. Uma sisebenzisa i-preprocess etafuleni, sizothola itafula elicindezelwe.

Isigaba sesibili sibuyekeza ithebula elihlanganisiwe. Masiqale sibhale i-algorithm ku-pseudocode:

for each sym in inputTable
  idx: row index in agg table for sym+currentTime;
  aggTable[idx;`high]: aggTable[idx;`high] | inputTable[sym;`high];
  aggTable[idx;`volume]: aggTable[idx;`volume] + inputTable[sym;`volume];
  …

Ku-Q, kuvamile ukusebenzisa imephu/ukunciphisa imisebenzi esikhundleni samaluphu. Kodwa njengoba i-Q iwulimi lwe-vector futhi singasebenzisa kalula yonke imisebenzi kuzo zonke izimpawu ngesikhathi esisodwa, lapho-ke esilinganisweni sokuqala singenza ngaphandle kweluphu nhlobo, senza imisebenzi kuzo zonke izimpawu ngesikhathi esisodwa:

idx:calcIdx inputTable;
row:aggTable idx;
aggTable[idx;`high]: row[`high] | inputTable`high;
aggTable[idx;`volume]: row[`volume] + inputTable`volume;
…

Kodwa singaqhubekela phambili, u-Q uno-opharetha ohlukile futhi onamandla ngokwedlulele - u-opharetha wesabelo ojwayelekile. Ikuvumela ukuthi uguqule isethi yamanani esakhiweni sedatha esiyinkimbinkimbi usebenzisa uhlu lwezinkomba, imisebenzi kanye nezimpikiswano. Esimweni sethu kubonakala kanje:

idx:calcIdx inputTable;
rows:aggTable idx;
// .[target;(idx0;idx1;..);function;argument] ~ target[idx 0;idx 1;…]: function[target[idx 0;idx 1;…];argument], в нашем случае функция – это присваивание
.[aggTable;(idx;aggCols);:;flip (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Ngeshwa, ukuze unikeze ithebula udinga uhlu lwemigqa, hhayi amakholomu, futhi kufanele uguqule i-matrix (uhlu lwamakholomu ohlwini lwemigqa) usebenzisa umsebenzi wokupheqa. Lokhu kuyabiza etafuleni elikhulu, ngakho esikhundleni salokho sisebenzisa isabelo esijwayelekile kukholomu ngayinye ngokwehlukana, sisebenzisa umsebenzi wemephu (obukeka njenge-apostrophe):

.[aggTable;;:;]'[(idx;)each aggCols; (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Siphinda sisebenzisa ukuqagela komsebenzi. Futhi qaphela ukuthi ku-Q, ukudala uhlu kuwumsebenzi futhi singawubiza sisebenzisa (imephu) umsebenzi ngamunye ukuthola uhlu lohlu.

Ukuqinisekisa ukuthi isethi yamakholomu abaliwe ayigxili, sizodala inkulumo engenhla ngamandla. Ake siqale sichaze imisebenzi yokubala ikholomu ngayinye, sisebenzisa okuguquguqukayo komugqa nokungenayo ukuze sibhekisele kudatha ehlanganisiwe neyokufaka:

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!
 ("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");

Amanye amakholomu akhethekile; inani lawo lokuqala akufanele libalwe ngomsebenzi. Singanquma ukuthi ingeyokuqala ngomugqa[`numTrades] ikholomu - uma iqukethe u-0, inani lingowokuqala. U-Q unomsebenzi wokukhetha - ?[Uhlu oluphusile;uhlu1;uhlu2] - olukhetha inani ohlwini 1 noma 2 kuye ngesimo esiku-agumenti yokuqala:

// high -> ?[isFirst;inp`high;row[`high]|inp`high]
// @ - тоже обобщенное присваивание для случая когда индекс неглубокий
@[`aggExpression;specialCols;{[x;y]"?[isFirst;inp`",y,";",x,"]"};string specialCols];

Lapha ngibize umsebenzi ojwayelekile ngomsebenzi wami (inkulumo kumabrace asontekile). Ithola inani lamanje (i-agumenti yokuqala) kanye ne-agumenti eyengeziwe, engiyidlulisa kupharamitha yesi-4.

Ake sengeze izipikha zebhethri ngokuhlukana, njengoba umsebenzi uyafana kuzo:

// volume -> row[`volume]+inp`volume
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;

Lesi isabelo esivamile ngamazinga we-Q, kodwa nginikeza uhlu lwamanani ngesikhathi esisodwa. Ekugcineni, masidale umsebenzi oyinhloko:

// ":",/:aggExprs ~ map[{":",x};aggExpr] => ":row[`high]|inp`high" присвоим вычисленное значение переменной, потому что некоторые колонки зависят от уже вычисленных значений
// string[cols],'exprs ~ map[,;string[cols];exprs] => "high:row[`high]|inp`high" завершим создание присваивания. ,’ расшифровывается как map[concat]
// ";" sv exprs – String from Vector (sv), соединяет список строк вставляя “;” посредине
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}";

Ngalesi sisho, ngidala umsebenzi osuka kuyunithi yezinhlamvu equkethe isisho engisinikeze ngenhla. Umphumela uzobukeka kanje:

{[aggTable;idx;inp] rows:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols ;(cumVolume:row[`cumVolume]+inp`cumVolume;… ; high:?[isFirst;inp`high;row[`high]|inp`high])]}

Uhlelo lokuhlola ikholomu luhlanekezelwe ngenxa yokuthi ku-Q uhlelo lokuhlola lusuka kwesokudla uye kwesokunxele.

Manje sinemisebenzi emibili eyinhloko edingekayo ekubaleni, sidinga nje ukwengeza ingqalasizinda encane futhi isevisi isilungile.

Izinyathelo zokugcina

Sinemisebenzi ye-preprocess kanye ne-updateAgg eyenza wonke umsebenzi. Kodwa kusadingeka ukuthi uqinisekise uguquko olulungile ngemizuzu futhi ubale izinkomba zokuhlanganisa. Okokuqala, ake sichaze umsebenzi we-init:

init:{
  tradeAgg:: 0#enlist[initWith]; // создаем пустую типизированную таблицу, enlist превращает словарь в таблицу, а 0# означает взять 0 элементов из нее
  currTime::00:00; // начнем с 0, :: означает, что присваивание в глобальную переменную
  currSyms::`u#`symbol$(); // `u# - превращает список в дерево, для ускорения поиска элементов
  offset::0; // индекс в tradeAgg, где начинается текущая минута 
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg; // кэш для последних значений roll колонок, таблица с ключом sym
 }

Sizophinde sichaze umsebenzi we-roll, ozoshintsha iminithi lamanje:

roll:{[tm]
  if[currTime>tm; :init[]]; // если перевалили за полночь, то просто вызовем init
  rollCache,::offset _ rollColumns#tradeAgg; // обновим кэш – взять roll колонки из aggTable, обрезать, вставить в rollCache
  offset::count tradeAgg;
  currSyms::`u#`$();
 }

Sizodinga umsebenzi ukwengeza izinhlamvu ezintsha:

addSyms:{[syms]
  currSyms,::syms; // добавим в список известных
  // добавим в таблицу sym, time и rollColumns воспользовавшись обобщенным присваиванием.
  // Функция ^ подставляет значения по умолчанию для roll колонок, если символа нет в кэше. value flip table возвращает список колонок в таблице.
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime), (initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 }

Futhi ekugcineni, umsebenzi wokuvuselela (igama lendabuko lalo msebenzi wezinsizakalo ze-Q), obizwa iklayenti ukwengeza idatha:

upd:{[tblName;data] // tblName нам не нужно, но обычно сервис обрабатывает несколько таблиц 
  tm:exec distinct time from data:() xkey preprocess data; // preprocess & calc time
  updMinute[data] each tm; // добавим данные для каждой минуты
};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm]; // поменяем минуту, если необходимо
  data:select from data where time=tm; // фильтрация
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms]; // новые символы
  updateAgg[`tradeAgg;offset+currSyms?syms;data]; // обновим агрегированную таблицу. Функция ? ищет индекс элементов списка справа в списке слева.
 };

Yilokho kuphela. Nansi ikhodi ephelele yenkonzo yethu, njengoba kuthenjisiwe, imigqa embalwa nje:

initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time;
rollColumns:`sym`cumVolume;

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size");
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
@[`aggExpression;specialCols;{"?[isFirst;inp`",y,";",x,"]"};string specialCols];
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}"; / '

init:{
  tradeAgg::0#enlist[initWith];
  currTime::00:00;
  currSyms::`u#`symbol$();
  offset::0;
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg;
 };
roll:{[tm]
  if[currTime>tm; :init[]];
  rollCache,::offset _ rollColumns#tradeAgg;
  offset::count tradeAgg;
  currSyms::`u#`$();
 };
addSyms:{[syms]
  currSyms,::syms;
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime),(initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 };

upd:{[tblName;data] updMinute[data] each exec distinct time from data:() xkey preprocess data};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm];
  data:select from data where time=tm;
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms];
  updateAgg[`tradeAgg;offset+currSyms?syms;data];
 };

Ukuhlola

Ake sihlole ukusebenza kwesevisi. Ukwenza lokhu, asiyiqalise ngenqubo ehlukile (faka ikhodi kufayela le-service.q) bese sishayela umsebenzi we-init:

q service.q –p 5566

q)init[]

Kwenye ikhonsoli, qala inqubo yesibili ye-Q bese uxhuma kweyokuqala:

h:hopen `:host:5566
h:hopen 5566 // если оба на одном хосте

Okokuqala, masidale uhlu lwezimpawu - izingcezu eziyi-10000 futhi sengeze umsebenzi ukuze sakhe ithebula elingahleliwe. Kwikhonsoli yesibili:

syms:`IBM`AAPL`GOOG,-9997?`8
rnd:{[n;t] ([] sym:n?syms; time:t+asc n#til 25; price:n?10f; size:n?10)}

Ngengeze izimpawu zangempela ezintathu ohlwini ukuze kube lula ukuzibheka etafuleni. Umsebenzi we-rnd udala ithebula elingahleliwe elinemigqa engu-n, lapho isikhathi sihluka ukusuka ku-t ukuya ku-t+25 millisecond.

Manje ungazama ukuthumela idatha kusevisi (engeza amahora ayishumi okuqala):

{h (`upd;`trade;rnd[10000;x])} each `time$00:00 + til 60*10

Ungahlola isevisi ukuthi ithebula libuyekeziwe:

c 25 200
select from tradeAgg where sym=`AAPL
-20#select from tradeAgg where sym=`AAPL

Umphumela:

sym|time|high|low|firstPrice|lastPrice|firstSize|lastSize|numTrades|volume|pvolume|turnover|avgPrice|avgSize|vwap|cumVolume
--|--|--|--|--|--------------------------------
AAPL|09:27|9.258904|9.258904|9.258904|9.258904|8|8|1|8|9.258904|74.07123|9.258904|8|9.258904|2888
AAPL|09:28|9.068162|9.068162|9.068162|9.068162|7|7|1|7|9.068162|63.47713|9.068162|7|9.068162|2895
AAPL|09:31|4.680449|0.2011121|1.620827|0.2011121|1|5|4|14|9.569556|36.84342|2.392389|3.5|2.631673|2909
AAPL|09:33|2.812535|2.812535|2.812535|2.812535|6|6|1|6|2.812535|16.87521|2.812535|6|2.812535|2915
AAPL|09:34|5.099025|5.099025|5.099025|5.099025|4|4|1|4|5.099025|20.3961|5.099025|4|5.099025|2919

Manje ake senze ukuhlola komthwalo ukuze sithole ukuthi ingakanani idatha isevisi engayicubungula ngomzuzu. Ake ngikukhumbuze ukuthi simisa isikhathi sokubuyekeza sibe ama-millisecond angu-25. Ngokufanelekile, isevisi kufanele (ngokwesilinganiso) ingene okungenani kuma-millisecond angu-20 isibuyekezo ngasinye ukuze inikeze abasebenzisi isikhathi sokucela idatha. Faka okulandelayo ohlelweni lwesibili:

tm:10:00:00.000
stressTest:{[n] 1 string[tm]," "; times,::h ({st:.z.T; upd[`trade;x]; .z.T-st};rnd[n;tm]); tm+:25}
start:{[n] times::(); do[4800;stressTest[n]]; -1 " "; `min`avg`med`max!(min times;avg times;med times;max times)}

4800 imizuzu emibili. Ungazama ukugijima kuqala ngemigqa engu-1000 njalo ngama-millisecond angama-25:

start 1000

Endabeni yami, umphumela uzungeze ama-millisecond ambalwa ngesibuyekezo ngasinye. Ngakho-ke ngizokwengeza ngokushesha inombolo yemigqa ibe ngu-10.000:

start 10000

Umphumela:

min| 00:00:00.004
avg| 9.191458
med| 9f
max| 00:00:00.030

Futhi, akukho okukhethekile, kodwa lokhu kuyimigqa eyizigidi ezingu-24 ngomzuzu, izinkulungwane ezingama-400 ngomzuzwana. Ngama-millisecond angaphezu kwangu-25, isibuyekezo sehlise ijubane izikhathi ezi-5 kuphela, ngokusobala lapho iminithi ishintsha. Masikhuphukele ku-100.000:

start 100000

Umphumela:

min| 00:00:00.013
avg| 25.11083
med| 24f
max| 00:00:00.108
q)sum times
00:02:00.532

Njengoba ubona, insizakalo ayikwazi ukubhekana nalokhu, kodwa noma kunjalo iyakwazi ukuhlala ihamba. Ivolumu enjalo yedatha (imigqa eyizigidi ezingama-240 ngomzuzu) inkulu kakhulu; ezimweni ezinjalo, kuvamile ukwethula ama-clones amaningana (noma inqwaba yama-clones) wesevisi, ngayinye ecubungula ingxenye yezinhlamvu kuphela. Noma kunjalo, umphumela uyahlaba umxhwele ngolimi olutolikiwe olugxile kakhulu ekugcinweni kwedatha.

Kungase kuphakame umbuzo wokuthi kungani isikhathi sikhula ngokungahambisani nosayizi wesibuyekezo ngasinye. Isizathu siwukuthi umsebenzi we-shrink empeleni uwumsebenzi we-C, osebenza kahle kakhulu kune-updateAgg. Kusukela kusayizi othile wokubuyekeza (cishe u-10.000), i-updateAgg ifinyelela ophahleni lwayo bese isikhathi sayo sokwenza singanciki kusayizi wokubuyekeza. Kungenxa yesinyathelo sokuqala u-Q ukuthi isevisi ikwazi ukugaya amavolumu anjalo edatha. Lokhu kugqamisa ukuthi kubaluleke kangakanani ukukhetha i-algorithm efanele uma usebenza nedatha enkulu. Elinye iphuzu ukugcinwa okufanele kwedatha kumemori. Uma idatha ibingagcinwanga ngokwekholomu noma ibinga-odwanga ngesikhathi, besiyobe sesijwayelana nento efana nokuphuthelwa kwenqolobane ye-TLB - ukungabikho kwekheli lekhasi lememori kunqolobane yekheli lokucubungula. Ukusesha ikheli kuthatha cishe izikhathi ezingu-30 ubude uma kungasebenzi, futhi uma idatha ihlakazeka, inganciphisa isevisi izikhathi ezimbalwa.

isiphetho

Kulesi sihloko, ngibonise ukuthi i-database ye-KDB+ ne-Q ayifanele kuphela ukugcina idatha enkulu futhi uyifinyelele kalula ngokukhetha, kodwa futhi nokudala izinsizakalo zokucubungula idatha ezikwazi ukugaya amakhulu ezigidi zemigqa/gigabytes yedatha ngisho naku- inqubo eyodwa Q eyodwa. Ulimi lwe-Q ngokwalo luvumela ukusetshenziswa okufushane kakhulu nangempumelelo kwama-algorithms ahlobene nokucutshungulwa kwedatha ngenxa yemvelo yayo ye-vector, umhumushi wolimi we-SQL owakhelwe ngaphakathi kanye nesethi ephumelele kakhulu yemisebenzi yelabhulali.

Ngizoqaphela ukuthi lokhu okungenhla kuyingxenye yalokho u-Q angakwenza, inezinye izici eziyingqayizivele futhi. Isibonelo, iphrothokholi ye-IPC elula kakhulu esula umngcele phakathi kwezinqubo ezingazodwana ze-Q futhi ikuvumela ukuthi uhlanganise amakhulu alezi zinqubo ube yinethiwekhi eyodwa, etholakala kumaseva amaningi ezingxenyeni ezihlukene zomhlaba.

Source: www.habr.com