Hiki iā ʻoe ke heluhelu e pili ana i ka waihona KDB +, ka ʻōlelo hoʻolālā Q, he aha nā ikaika a me nā nāwaliwali o kaʻu mua. a pōkole ma ka hoʻolauna. Ma ka ʻatikala, e hoʻokō mākou i kahi lawelawe ma Q e hoʻoponopono i ke kahawai ʻikepili e hiki mai ana a helu i nā hana aggregation like ʻole i kēlā me kēia minuke i ke ʻano "manawa maoli" (ʻo ia hoʻi, e loaʻa ka manawa e helu ai i nā mea āpau ma mua o ka ʻāpana o ka ʻikepili aʻe). ʻO ka hiʻohiʻona nui o Q ʻo ia ka ʻōlelo vector e hiki ai iā ʻoe ke hana ʻaʻole me nā mea hoʻokahi, akā me kā lākou arrays, arrays of arrays a me nā mea paʻakikī ʻē aʻe. Ua kaulana nā ʻōlelo e like me Q a me kona mau hoahānau K, J, APL no ko lākou pōkole. ʻO ka manawa pinepine, hiki ke kākau ʻia kahi papahana e lawe ana i kekahi mau pale o ke code ma kahi ʻōlelo maʻamau e like me Java ma luna o lākou i kekahi mau laina. ʻO kēia kaʻu makemake e hōʻike i kēia ʻatikala.
Hōʻike
ʻO KDB+ kahi waihona kolamu e kālele ana i ka nui o nā ʻikepili, kauoha ʻia ma kahi ala kikoʻī (ma ka manawa nui). Hoʻohana nui ʻia ia i nā ʻoihana kālā - nā panakō, nā kālā hoʻopukapuka, nā hui ʻinikua. ʻO ka ʻōlelo Q ka ʻōlelo kūloko o KDB+ e hiki ai iā ʻoe ke hana pono me kēia ʻikepili. ʻO ka manaʻo Q ka pōkole a me ka maikaʻi, ʻoiai ua mōhai ʻia ka mālamalama. Hoʻomaopopo ʻia kēia e ka paʻakikī o ka ʻōlelo vector i kēlā me kēia hihia, a ʻo ka pōkole a me ka waiwai o ka hoʻopaʻa ʻana e hiki ai iā ʻoe ke ʻike i kahi hapa nui o ka papahana ma ka pale hoʻokahi, ʻo ia ka mea maʻalahi ke hoʻomaopopo.
Ma kēia ʻatikala mākou e hoʻokō i kahi papahana piha i ka Q a makemake paha ʻoe e hoʻāʻo. No ka hana ʻana i kēia, pono ʻoe i ka Q maoli. Hiki iā ʻoe ke hoʻoiho i ka mana 32-bit manuahi ma ka pūnaewele ʻoihana kx - . Ma laila, inā makemake ʻoe, e ʻike ʻoe i ka ʻike kuhikuhi ma Q, ka puke a me nā ʻatikala like ʻole e pili ana i kēia kumuhana.
Ka hoʻokumu ʻana i ka pilikia
Aia kekahi kumu e hoʻouna i kahi papa me ka ʻikepili i kēlā me kēia 25 milliseconds. No ka mea, hoʻohana nui ʻia ʻo KDB+ i ke kālā, e manaʻo mākou he papaʻaina kēia o nā kālepa (kūʻai), nona nā kolamu penei: manawa (manawa i milliseconds), sym (ka inoa o ka hui ma ke kūʻai kālā - IBM, AAPL,…), kumukūʻai (ke kumu kūʻai i kūʻai ʻia ai nā ʻāpana), ka nui (ka nui o ke kālepa). ʻAʻole liʻiliʻi a lōʻihi loa ka 25 millisecond interval. ʻO kona hele ʻana ʻo ia ka hiki ʻana o ka ʻikepili i ka lawelawe i hoʻopaʻa ʻia. E maʻalahi ka hoʻokō ʻana i ka buffering ma ka ʻaoʻao lawelawe, e komo pū me ka buffering dynamic e pili ana i ka ukana o kēia manawa, akā no ka maʻalahi, e kālele mākou i kahi manawa paʻa.
Pono e helu ka lawelawe i kēlā me kēia minuke no kēlā me kēia hōʻailona e komo mai ana mai ke kolamu sym kahi hoʻonohonoho o nā hana hoʻohui - kumu kūʻai kiʻekiʻe, kumu kūʻai avg, nui nui, etc. ʻike pono. No ka maʻalahi, e manaʻo mākou e hiki ke helu ʻia nā hana āpau, ʻo ia hoʻi. no ka loaʻa ʻana o kahi waiwai hou, ua lawa ka ʻike ʻana i nā helu ʻelua - nā helu kahiko a me nā helu komo. No ka laʻana, loaʻa kēia waiwai i nā hana max, average, sum, akā ʻaʻole ka hana waena.
E noʻonoʻo nō hoʻi mākou i ka manawa i kauoha ʻia ke kahawai ʻikepili komo. Hāʻawi kēia iā mākou i ka manawa e hana wale me ka minuke hope. I ka hoʻomaʻamaʻa, ua lawa ka hiki ke hana me nā minuke o kēia manawa a me nā minuke i hala inā ua lohi kekahi mau mea hou. No ka maʻalahi, ʻaʻole mākou e noʻonoʻo i kēia hihia.
Nā hana hoʻohui
Ua helu ʻia nā hana hoʻohui pono ma lalo nei. Ua lawe au i ka nui o lākou e hoʻonui i ka ukana ma ka lawelawe:
- kiʻekiʻe - kumu kūʻai kiʻekiʻe - kumu kūʻai kiʻekiʻe i kēlā me kēia minuke.
- haʻahaʻa - kumu kūʻai min - kumu kūʻai liʻiliʻi i kēlā me kēia minuke.
- ke kumu kūʻai mua - kumu kūʻai mua - kumu kūʻai mua i kēlā me kēia minuke.
- ke kumu kūʻai hope - kumu kūʻai hope loa i kēlā me kēia minuke.
- firstSize - ka nui mua - ka nui kālepa mua i kēlā me kēia minuke.
- lastSize - ka nui hope - ka nui kālepa hope loa i hoʻokahi minuke.
- numTrades - helu i - helu o nā kālepa i kēlā me kēia minuke.
- ka nui - ka nui o ka nui - ka huina o nā nui kālepa i kēlā me kēia minuke.
- pvolume - huina kumu kūʻai - huina o nā kumukūʻai i kēlā me kēia minuke, koi ʻia no ka avgPrice.
- - huina kumukūʻai huli * nui - ka nui o nā hana i kēlā me kēia minuke.
- avgPrice - pvolume%numTrades - kumu kūʻai maʻamau i kēlā me kēia minuke.
- avgSize - volume%numTrades - ka nui kālepa maʻamau i kēlā me kēia minuke.
- vwap - huli%volume - kumu kūʻai maʻamau i kēlā me kēia minuke i kaupaona ʻia e ka nui o ke kālepa.
- cumVolume - huina nui - hōʻiliʻili nui o nā kālepa i ka manawa holoʻokoʻa.
E kūkākūkā koke kākou i hoʻokahi wahi ʻike ʻole - pehea e hoʻomaka ai i kēia mau kolamu no ka manawa mua a no kēlā me kēia minuke ma hope. Pono e hoʻomaka ʻia kekahi mau kolamu o ke ʻano kumu mua i ka null i kēlā me kēia manawa; ʻaʻole i wehewehe ʻia ko lākou waiwai. Pono e hoʻonohonoho mau ʻia nā ʻano leo ʻē aʻe i ka 0. Aia kekahi mau kolamu e koi ana i kahi ala hoʻohui - no ka laʻana, pono e kope ʻia ka cumVolume mai ka minuke mua, a no ka mea mua i hoʻonohonoho ʻia i 0. E hoʻonohonoho i kēia mau ʻāpana āpau me ka hoʻohana ʻana i ka ʻikepili puke wehewehe. ʻano (e like me kahi moʻolelo):
// list ! list – создать словарь, 0n – float null, 0N – long null, `sym – тип символ, `sym1`sym2 – список символов
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time; // список всех вычисляемых колонок, reverse объяснен ниже
Ua hoʻohui au i ka sym a me ka manawa i ka puke wehewehe'ōlelo no ka maʻalahi, ʻo initWith kahi laina i hana ʻia mai ka papa ʻaina hope loa, kahi e hoʻonoho ai i ka sym a me ka manawa kūpono. Hiki iā ʻoe ke hoʻohana iā ia e hoʻohui i nā lālani hou i kahi papaʻaina.
Pono mākou i nā aggCols i ka wā e hana ai i kahi hana hoʻohui. Pono e hoʻohuli ʻia ka papa inoa ma muli o ke ʻano o ka loiloi ʻana o nā manaʻo ma Q (mai ka ʻākau a i ka hema). ʻO ka pahuhopu e hōʻoia i ka helu ʻana mai ke kiʻekiʻe a hiki i ka cumVolume, ʻoiai ke hilinaʻi nei kekahi mau kolamu i nā kolamu ma mua.
ʻO nā kolamu e pono e kope ʻia i kahi minuke hou mai ka mea ma mua, ua hoʻohui ʻia ke kolamu sym no ka maʻalahi:
rollColumns:`sym`cumVolume;
I kēia manawa, e māhele kākou i nā kolamu i mau pūʻulu e like me ke ʻano o ka hoʻonui ʻia ʻana. Hiki ke hoʻokaʻawale ʻia nā ʻano ʻekolu:
- Accumulators (volume, turnover, ..) - pono mākou e hoʻohui i ka waiwai komo i ka mea mua.
- Me kahi kiko kūikawā (kiʻekiʻe, haʻahaʻa, ..) - lawe ʻia ka waiwai mua i ka minuke mai ka ʻikepili e hiki mai ana, ua helu ʻia ke koena me ka hoʻohana ʻana i ka hana.
- Hoomaha. E helu mau ʻia me ka hoʻohana ʻana i kahi hana.
E wehewehe i nā ʻano like ʻole no kēia mau papa:
accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;
Kauoha helu
E hōʻano hou mākou i ka papa ʻaina i ʻelua pae. No ka maikaʻi, hoʻemi mua mākou i ka papaʻaina e hiki mai ana i hoʻokahi lālani no kēlā me kēia ʻano a me nā minuke. ʻO ka ʻoiaʻiʻo ʻo kā mākou mau hana āpau he incremental a hui pū e hōʻoiaʻiʻo ʻaʻole e loli ka hopena o kēia hana hou. Hiki iā ʻoe ke hōʻemi i ka papaʻaina me ka koho:
select high:max price, low:min price … by sym,time.minute from table
He hemahema kēia ʻano hana - ua hoʻonohonoho mua ʻia ka hoʻonohonoho o nā kolamu i helu ʻia. ʻO ka mea pōmaikaʻi, ma Q, ua hoʻokō ʻia ke koho ma ke ʻano he hana kahi e hiki ai iā ʻoe ke hoʻololi i nā hoʻopaʻapaʻa dynamically i haku ʻia:
?[table;whereClause;byClause;selectClause]
ʻAʻole wau e wehewehe kikoʻī i ke ʻano o nā manaʻo; i kā mākou hihia, ʻaʻole wale nā huaʻōlelo i koho ʻia a he mau puke wehewehe ʻōlelo lākou o nā kolamu! No laila, hiki ke wehewehe ʻia ka hana hōʻemi penei:
selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size"); // each это функция map в Q для одного списка
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];
No ka akaka, ua hoʻohana au i ka hana parse, e hoʻohuli i kahi kaula me kahi huaʻōlelo Q i kahi waiwai e hiki ke hāʻawi ʻia i ka hana eval a koi ʻia i ka hana koho. E hoʻomaopopo hoʻi ua wehewehe ʻia ka preprocess ma ke ʻano he projection (ʻo ia hoʻi, kahi hana me nā manaʻo wehewehe ʻāpana) o ka hana koho, ʻaʻole naʻe kekahi pane (ka papa). Inā mākou e hoʻopili i ka preprocess i kahi papaʻaina, e loaʻa iā mākou kahi pākaukau paʻa.
ʻO ka hana ʻelua ka hoʻonui ʻana i ka papaʻaina i hōʻuluʻulu ʻia. E kākau mua i ka algorithm ma pseudocode:
for each sym in inputTable
idx: row index in agg table for sym+currentTime;
aggTable[idx;`high]: aggTable[idx;`high] | inputTable[sym;`high];
aggTable[idx;`volume]: aggTable[idx;`volume] + inputTable[sym;`volume];
…
Ma Q, he mea maʻamau ka hoʻohana ʻana i ka palapala ʻāina/hōʻemi i nā hana ma kahi o nā puka lou. Akā, no ka mea he ʻōlelo vector ʻo Q a hiki iā mākou ke hoʻohana maʻalahi i nā hana āpau i nā hōʻailona a pau i ka manawa hoʻokahi, a laila i kahi hoʻohālikelike mua hiki iā mākou ke hana me ka ʻole o ka loop, e hana i nā hana ma nā hōʻailona a pau i ka manawa hoʻokahi:
idx:calcIdx inputTable;
row:aggTable idx;
aggTable[idx;`high]: row[`high] | inputTable`high;
aggTable[idx;`volume]: row[`volume] + inputTable`volume;
…
Akā hiki iā mākou ke hele i mua, loaʻa iā Q kahi mea hoʻohana kūʻokoʻa a ikaika loa - ka mea hoʻohana maʻamau. Hiki iā ʻoe ke hoʻololi i kahi hoʻonohonoho o nā waiwai i loko o kahi hoʻolālā data paʻakikī me ka hoʻohana ʻana i kahi papa inoa o nā helu, nā hana a me nā hoʻopaʻapaʻa. I kā mākou hihia, ua like kēia:
idx:calcIdx inputTable;
rows:aggTable idx;
// .[target;(idx0;idx1;..);function;argument] ~ target[idx 0;idx 1;…]: function[target[idx 0;idx 1;…];argument], в нашем случае функция – это присваивание
.[aggTable;(idx;aggCols);:;flip (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];
ʻO ka mea pōʻino, no ka hāʻawi ʻana i kahi papaʻaina pono ʻoe i kahi papa inoa o nā lālani, ʻaʻole nā kolamu, a pono ʻoe e hoʻololi i ka matrix (papa inoa o nā kolamu i ka papa inoa o nā lālani) me ka hoʻohana ʻana i ka hana flip. He pipiʻi kēia no ka papaʻaina nui, no laila ke hoʻohana nei mākou i kahi hana maʻamau i kēlā me kēia kolamu, me ka hoʻohana ʻana i ka hana palapala (e like me ka apostrophe):
.[aggTable;;:;]'[(idx;)each aggCols; (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];
Hoʻohana hou mākou i ka projection function. E hoʻomaopopo hoʻi ma Q, ʻo ka hana ʻana i kahi papa inoa he hana a hiki iā mākou ke kāhea iā ia me ka hoʻohana ʻana i kēlā me kēia (palapala palapala) e kiʻi i kahi papa inoa o nā papa inoa.
No ka hoʻopaʻa ʻole ʻana i ka hoʻonohonoho o nā kolamu i helu ʻia, e hana mākou i ka ʻōlelo ma luna me ka ikaika. E wehewehe mua i nā hana no ka helu ʻana i kēlā me kēia kolamu, me ka hoʻohana ʻana i ka lālani a me nā ʻano hoʻololi inp e kuhikuhi i ka ʻikepili i hōʻuluʻulu ʻia a hoʻokomo:
aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!
("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
He kūikawā kekahi mau kolamu; ʻaʻole pono e helu ʻia ko lākou waiwai mua e ka hana. Hiki iā mākou ke hoʻoholo ʻo ia ka mua ma ka lālani [`numTrades] kolamu - inā loaʻa iā 0, a laila ʻo ka waiwai ka mua. Loaʻa iā Q kahi hana koho - ?[Boolean list;list1;list2] - e koho ana i kahi waiwai mai ka papa inoa 1 a i ʻole 2 ma muli o ke kūlana o ka pane mua:
// high -> ?[isFirst;inp`high;row[`high]|inp`high]
// @ - тоже обобщенное присваивание для случая когда индекс неглубокий
@[`aggExpression;specialCols;{[x;y]"?[isFirst;inp`",y,";",x,"]"};string specialCols];
Ma ʻaneʻi ua kāhea wau i kahi hana maʻamau me kaʻu hana (he ʻōlelo ma nā pale pale). Loaʻa iā ia ka waiwai o kēia manawa (ka hoʻopaʻapaʻa mua) a me kahi hoʻopaʻapaʻa hou aʻe, aʻu e hele ai ma ka ʻāpana 4th.
E hoʻohui kaʻawale i nā ʻōlelo pākahi, no ka mea ua like ka hana iā lākou:
// volume -> row[`volume]+inp`volume
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
He hana maʻamau kēia e nā kūlana Q, akā hāʻawi wau i kahi papa inoa o nā waiwai i ka manawa hoʻokahi. ʻO ka hope, e hana kākou i ka hana nui:
// ":",/:aggExprs ~ map[{":",x};aggExpr] => ":row[`high]|inp`high" присвоим вычисленное значение переменной, потому что некоторые колонки зависят от уже вычисленных значений
// string[cols],'exprs ~ map[,;string[cols];exprs] => "high:row[`high]|inp`high" завершим создание присваивания. ,’ расшифровывается как map[concat]
// ";" sv exprs – String from Vector (sv), соединяет список строк вставляя “;” посредине
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}";
Me kēia ʻōlelo, hana au i kahi hana mai kahi kaula i loaʻa ka ʻōlelo aʻu i hāʻawi ai ma luna. Penei ka hopena:
{[aggTable;idx;inp] rows:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols ;(cumVolume:row[`cumVolume]+inp`cumVolume;… ; high:?[isFirst;inp`high;row[`high]|inp`high])]}
Hoʻohuli ʻia ke kauoha loiloi kolamu no ka mea ma Q ke kauoha loiloi mai ka ʻākau a i ka hema.
I kēia manawa, ʻelua mau hana nui e pono ai no ka helu ʻana, pono mākou e hoʻohui i kahi ʻoihana liʻiliʻi a mākaukau ka lawelawe.
Nā ʻanuʻu hope
Loaʻa iā mākou nā hana preprocess a updateAgg e hana i nā hana āpau. Akā, pono e hōʻoia i ka hoʻololi kūpono ma o nā minuke a helu i nā kuhikuhi no ka hoʻohui. ʻO ka mea mua, e wehewehe i ka hana init:
init:{
tradeAgg:: 0#enlist[initWith]; // создаем пустую типизированную таблицу, enlist превращает словарь в таблицу, а 0# означает взять 0 элементов из нее
currTime::00:00; // начнем с 0, :: означает, что присваивание в глобальную переменную
currSyms::`u#`symbol$(); // `u# - превращает список в дерево, для ускорения поиска элементов
offset::0; // индекс в tradeAgg, где начинается текущая минута
rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg; // кэш для последних значений roll колонок, таблица с ключом sym
}
E wehewehe pū mākou i ka hana ʻōwili, e hoʻololi i ka minuke o kēia manawa:
roll:{[tm]
if[currTime>tm; :init[]]; // если перевалили за полночь, то просто вызовем init
rollCache,::offset _ rollColumns#tradeAgg; // обновим кэш – взять roll колонки из aggTable, обрезать, вставить в rollCache
offset::count tradeAgg;
currSyms::`u#`$();
}
Pono mākou i kahi hana e hoʻohui i nā huaʻōlelo hou:
addSyms:{[syms]
currSyms,::syms; // добавим в список известных
// добавим в таблицу sym, time и rollColumns воспользовавшись обобщенным присваиванием.
// Функция ^ подставляет значения по умолчанию для roll колонок, если символа нет в кэше. value flip table возвращает список колонок в таблице.
`tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime), (initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
}
A ʻo ka hope, ʻo ka hana upd (ka inoa kuʻuna no kēia hana no nā lawelawe Q), i kāhea ʻia e ka mea kūʻai aku e hoʻohui i ka ʻikepili:
upd:{[tblName;data] // tblName нам не нужно, но обычно сервис обрабатывает несколько таблиц
tm:exec distinct time from data:() xkey preprocess data; // preprocess & calc time
updMinute[data] each tm; // добавим данные для каждой минуты
};
updMinute:{[data;tm]
if[tm<>currTime; roll tm; currTime::tm]; // поменяем минуту, если необходимо
data:select from data where time=tm; // фильтрация
if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms]; // новые символы
updateAgg[`tradeAgg;offset+currSyms?syms;data]; // обновим агрегированную таблицу. Функция ? ищет индекс элементов списка справа в списке слева.
};
ʻo ia wale nō. Eia ka code piha o kā mākou lawelawe, e like me ka mea i ʻōlelo ʻia, he mau laina wale nō:
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time;
rollColumns:`sym`cumVolume;
accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;
selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size");
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];
aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
@[`aggExpression;specialCols;{"?[isFirst;inp`",y,";",x,"]"};string specialCols];
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}"; / '
init:{
tradeAgg::0#enlist[initWith];
currTime::00:00;
currSyms::`u#`symbol$();
offset::0;
rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg;
};
roll:{[tm]
if[currTime>tm; :init[]];
rollCache,::offset _ rollColumns#tradeAgg;
offset::count tradeAgg;
currSyms::`u#`$();
};
addSyms:{[syms]
currSyms,::syms;
`tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime),(initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
};
upd:{[tblName;data] updMinute[data] each exec distinct time from data:() xkey preprocess data};
updMinute:{[data;tm]
if[tm<>currTime; roll tm; currTime::tm];
data:select from data where time=tm;
if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms];
updateAgg[`tradeAgg;offset+currSyms?syms;data];
};
Hoʻoponopono
E nānā kākou i ka hana o ka lawelawe. No ka hana ʻana i kēia, e holo kāua i kahi kaʻina kaʻawale (e kau i ke code i ka faila service.q) a kāhea i ka hana init:
q service.q –p 5566
q)init[]
Ma kahi console ʻē aʻe, e hoʻomaka i ke kaʻina Q lua a hoʻohui i ka mea mua:
h:hopen `:host:5566
h:hopen 5566 // если оба на одном хосте
ʻO ka mea mua, e hana mākou i kahi papa inoa o nā hōʻailona - 10000 mau ʻāpana a hoʻohui i kahi hana e hana ai i kahi papaʻaina. Ma ka console lua:
syms:`IBM`AAPL`GOOG,-9997?`8
rnd:{[n;t] ([] sym:n?syms; time:t+asc n#til 25; price:n?10f; size:n?10)}
Ua hoʻohui au i ʻekolu mau hōʻailona maoli i ka papa inoa i mea e maʻalahi ai ka nānā ʻana iā lākou ma ka papa. Hoʻokumu ka hana rnd i ka papaʻaina maʻamau me nā lālani n, kahi e ʻokoʻa ai ka manawa mai t a t+25 milliseconds.
I kēia manawa hiki iā ʻoe ke hoʻouna i ka ʻikepili i ka lawelawe (hoʻohui i nā hola he ʻumi mua):
{h (`upd;`trade;rnd[10000;x])} each `time$00:00 + til 60*10
Hiki iā ʻoe ke nānā i ka lawelawe i hōʻano hou ʻia ka papaʻaina:
c 25 200
select from tradeAgg where sym=`AAPL
-20#select from tradeAgg where sym=`AAPL
Hualoaʻa:
sym|time|high|low|firstPrice|lastPrice|firstSize|lastSize|numTrades|volume|pvolume|turnover|avgPrice|avgSize|vwap|cumVolume
--|--|--|--|--|--------------------------------
AAPL|09:27|9.258904|9.258904|9.258904|9.258904|8|8|1|8|9.258904|74.07123|9.258904|8|9.258904|2888
AAPL|09:28|9.068162|9.068162|9.068162|9.068162|7|7|1|7|9.068162|63.47713|9.068162|7|9.068162|2895
AAPL|09:31|4.680449|0.2011121|1.620827|0.2011121|1|5|4|14|9.569556|36.84342|2.392389|3.5|2.631673|2909
AAPL|09:33|2.812535|2.812535|2.812535|2.812535|6|6|1|6|2.812535|16.87521|2.812535|6|2.812535|2915
AAPL|09:34|5.099025|5.099025|5.099025|5.099025|4|4|1|4|5.099025|20.3961|5.099025|4|5.099025|2919E ho'āʻo mākou i kēia manawa e ʻike i ka nui o ka ʻikepili i hiki i ka lawelawe ke hana i kēlā me kēia minuke. E hoʻomanaʻo wau iā ʻoe ua hoʻonohonoho mākou i ka manawa hoʻonui i 25 milliseconds. No laila, pono ka lawelawe (ma ka awelika) i loko o ka liʻiliʻi o 20 milliseconds no ka hoʻonui e hāʻawi i nā mea hoʻohana i ka manawa e noi ai i ka ʻikepili. E hoʻokomo i kēia ma ke kaʻina hana ʻelua:
tm:10:00:00.000
stressTest:{[n] 1 string[tm]," "; times,::h ({st:.z.T; upd[`trade;x]; .z.T-st};rnd[n;tm]); tm+:25}
start:{[n] times::(); do[4800;stressTest[n]]; -1 " "; `min`avg`med`max!(min times;avg times;med times;max times)}
ʻO 4800 ʻelua mau minuke. Hiki iā ʻoe ke hoʻāʻo e holo mua no nā lālani 1000 i kēlā me kēia 25 milliseconds:
start 1000
I koʻu hihia, aia ka hopena ma kahi o ʻelua mau milliseconds no ka hoʻonui. No laila e hoʻonui koke au i ka helu o nā lālani i 10.000:
start 10000
Hualoaʻa:
min| 00:00:00.004
avg| 9.191458
med| 9f
max| 00:00:00.030
Eia hou, ʻaʻohe mea kūikawā, akā ʻo 24 miliona mau laina i kēlā me kēia minuke, 400 tausani i kekona. Ma mua o 25 milliseconds, ua lohi ka hoʻonui ʻana i nā manawa 5 wale nō, ʻike ʻia ke loli ka minuke. E hoʻonui kākou i 100.000:
start 100000
Hualoaʻa:
min| 00:00:00.013
avg| 25.11083
med| 24f
max| 00:00:00.108
q)sum times
00:02:00.532
E like me kāu e ʻike ai, hiki ʻole i ka lawelawe ke hoʻokō, akā naʻe, hoʻokele ia e noho mau. ʻO ia ka nui o ka ʻikepili (240 miliona mau lālani i kēlā me kēia minuke) he nui loa; ma ia mau hihia, he mea maʻamau ka hoʻomaka ʻana i kekahi mau clones (a i ʻole nā kini he nui o nā clones) o ka lawelawe, ʻo kēlā me kēia mea e hana i kahi hapa o nā huaʻōlelo. Eia nō naʻe, mahalo ka hopena no kahi ʻōlelo unuhi e kālele nui ana i ka mālama ʻikepili.
Hiki ke nīnau i ke kumu e ulu ai ka manawa me ka nui o kēlā me kēia mea hou. ʻO ke kumu, ʻo ka hana shrink he hana C maoli nō ia, ʻoi aku ka maikaʻi ma mua o ka updateAgg. E hoʻomaka ana mai kahi nui hou (ma kahi o 10.000), hiki i ka updateAgg i kona kaupaku a laila ʻaʻole hilinaʻi kona manawa hoʻokō i ka nui o ka hoʻonui. Ma muli o ka hana mua Q i hiki ai i ka lawelawe ke hoʻoheheʻe i nā nui o ka ʻikepili. Hōʻike kēia i ke koʻikoʻi o ke koho ʻana i ka algorithm kūpono i ka wā e hana ai me ka ʻikepili nui. ʻO kekahi mea ʻē aʻe ʻo ka mālama pono ʻana o ka ʻikepili i ka hoʻomanaʻo. Inā ʻaʻole i mālama ʻia ka ʻikepili i ka columnarly a ʻaʻole i kauoha ʻia e ka manawa, a laila e kamaʻāina mākou i kahi mea e like me ka TLB cache miss - ka nele o kahi helu ʻaoʻao hoʻomanaʻo i ka cache address processor. ʻO ka ʻimi ʻana i kahi helu wahi he 30 mau manawa lōʻihi inā ʻaʻole i kūleʻa, a inā e hoʻopuehu ka ʻikepili, hiki iā ia ke hoʻolohi i ka lawelawe i nā manawa he nui.
hopena
Ma kēia ʻatikala, ua hōʻike wau he kūpono ka waihona KDB + a me Q ʻaʻole wale no ka mālama ʻana i ka ʻikepili nui a me ka maʻalahi o ke komo ʻana ma o ke koho ʻana, akā no ka hana ʻana i nā lawelawe hoʻoili ʻikepili i hiki ke hoʻokaʻawale i nā haneli miliona o nā lālani / gigabytes o ka ʻikepili a hiki i loko. hoʻokahi kaʻina Q hoʻokahi. ʻO ka ʻōlelo Q ponoʻī e ʻae i ka hoʻokō pōkole loa a me ka maikaʻi o nā algorithm e pili ana i ka hoʻoili ʻana i ka ʻikepili ma muli o kona ʻano vector, i kūkulu ʻia i loko o ka SQL dialect interpreter a me kahi hoʻonohonoho holomua o nā hana waihona.
E hoʻomaopopo wau ʻo ka mea i luna he ʻāpana wale nō o ka mea hiki iā Q ke hana, he mau hiʻohiʻona ʻē aʻe kekahi. No ka laʻana, kahi protocol IPC maʻalahi loa e holoi i ka palena ma waena o nā kaʻina Q hoʻokahi a hiki iā ʻoe ke hoʻohui i nā haneli o kēia mau kaʻina i loko o kahi pūnaewele hoʻokahi, hiki ke loaʻa ma nā kikowaena o nā kikowaena ma nā wahi like ʻole o ka honua.
Source: www.habr.com