U ka bala ka hore na motheo oa KDB +, puo ea lenaneo la Q ke eng, hore na matla le mefokolo ea bona ke efe nakong e fetileng. le ka bokhutšoane selelekeleng. Sehloohong sena, re tla kenya ts'ebetsong ts'ebeletso ho Q e tla sebetsana le phallo ea data e kenang le ho bala mesebetsi e fapa-fapaneng ea ho kopanya motsotso o mong le o mong ka mokhoa oa "nako ea sebele" (ke hore, e tla ba le nako ea ho bala ntho e 'ngoe le e' ngoe pele ho karolo e latelang ea data). Ntho e ka sehloohong ea Q ke hore ke puo ea vector e u lumellang hore u se ke ua sebetsa ka lintho tse le 'ngoe, empa ka lihlopha tsa tsona, lihlopha tsa lihlopha le lintho tse ling tse rarahaneng. Lipuo tse kang Q le beng ka eona K, J, APL li tumme ka bokhutšoanyane ba tsona. Hangata, lenaneo le nkang li-skrini tse 'maloa tsa khoutu ka puo e tloaelehileng joaloka Java le ka ngoloa ho tsona ka mela e seng mekae. Sena ke seo ke batlang ho se bontša sehloohong sena.
Selelekela
KDB+ ke sebaka sa polokelo ea litaba se shebaneng le lintlha tse ngata haholo, tse hlophisitsoeng ka tsela e itseng (haholo-holo ka nako). E sebelisoa haholo-holo mekhatlong ea lichelete - libanka, lichelete tsa lichelete, lik'hamphani tsa inshorense. Puo ea Q ke puo ea ka hare ea KDB+ e u lumellang hore u sebetse ka katleho le data ena. Khopolo ea Q e bokhutšoanyane ebile e sebetsa hantle, ha ho hlaka ho etsoa sehlabelo. Sena se lokafatsoa ke taba ea hore puo ea vector e tla ba thata ho e utloisisa ho sa tsotellehe boemo leha e le bofe, 'me bokhutšoanyane le bokhabane ba ho rekota bo u lumella ho bona karolo e kholoanyane ea lenaneo skrineng se le seng, seo qetellong se etsang hore ho be bonolo ho se utloisisa.
Sehloohong sena re kenya tšebetsong lenaneo le felletseng ho Q mme o kanna oa batla ho le leka. Ho etsa sena, o tla hloka Q ea sebele. O ka khoasolla mofuta oa mahala oa 32-bit webosaeteng ea kx ea kx - . Ha u le moo, haeba u thahasella, u tla fumana lintlha tsa litšupiso ho Q, buka le lihlooho tse fapaneng tse buang ka taba ena.
Mokhoa oa bothata
Ho na le mohloli o romellang tafole e nang le data ka mor'a 25 milliseconds. Kaha KDB+ e sebelisoa haholo-holo licheleteng, re tla nka hore lena ke tafole ea litšebelisano (trades), e nang le likholomo tse latelang: nako (nako ka milliseconds), sym (lebitso la k'hamphani ho stock exchange - IBM, AAPL,…), theko (theko eo likarolo li rekiloeng ka eona), boholo (boholo ba thekiso). Nako ea nako ea 25 millisecond ke ea boikhethelo, ha e nyane haholo ebile ha e telele haholo. Ho ba teng ha eona ho bolela hore data e tla ts'ebeletso e seng e ntse e buffer. Ho ka ba bonolo ho kenya ts'ebetso ea "buffering" lehlakoreng la ts'ebeletso, ho kenyelletsa le buffering e matla ho latela mojaro oa hajoale, empa molemong oa ho nolofatsa, re tla shebana le nako e tsitsitseng.
Ts'ebeletso e tlameha ho bala motsotso o mong le o mong bakeng sa lets'oao le leng le le leng le kenang ho tsoa kholumong ea sym sete ea likarolo tse kopanyang - theko e phahameng, theko e mahareng, boholo ba kakaretso, jj. tlhahisoleseding e molemo. Bakeng sa ho nolofatsa, re tla nka hore mesebetsi eohle e ka baloa ka ho eketseha, i.e. ho fumana boleng bo bocha, ho lekane ho tseba linomoro tse peli - ea khale le ea boleng bo tlang. Mohlala, boholo ba mesebetsi, kakaretso, kakaretso e na le thepa ena, empa tšebetso ea bohareng ha e eo.
Hape re tla nka hore molaetsa oa data o kenang o laetsoe ke nako. Sena se tla re fa monyetla oa ho sebetsa feela ka motsotso oa ho qetela. Ha e le hantle, ho lekane ho khona ho sebetsa le metsotso ea morao-rao le e fetileng haeba lintlafatso tse ling li ka lieha. Bakeng sa ho nolofatsa, re ke ke ra nahana ka nyeoe ena.
Mesebetsi ea ho kopanya
Mesebetsi e hlokahalang ea ho kopanya e thathamisitsoe ka tlase. Ke nkile tse ngata kamoo ho ka khonehang ho eketsa mojaro oa ts'ebeletso:
- phahameng - theko e phahameng - theko e phahameng ka motsotso.
- theko e tlase - theko e tlase - theko e tlase ka motsotso.
- firstPrice - theko ea pele - theko ea pele ka motsotso.
- lastPrice - theko ea ho qetela - theko ea ho qetela ka motsotso.
- firstSize - boholo ba pele - boholo ba khoebo ea pele ka motsotso.
- lastSize - boholo ba ho qetela - boholo ba khoebo ea ho qetela ka motsotso.
- numTrades - count i - palo ea khoebo ka motsotso.
- bophahamo ba modumo - kakaretso ea boholo - kakaretso ea boholo ba khoebo ka motsotso.
- pvolume - kakaretso ea theko - kakaretso ea litheko ka motsotso, e hlokahalang bakeng sa avgPrice.
- - theko ea kakaretso ea kakaretso * boholo - kakaretso ea palo ea litšebelisano ka motsotso.
- avgPrice - pvolume%numTrades - theko e tloaelehileng ka motsotso.
- avgSize - volume%numTrades - karolelano ea boholo ba khoebo ka motsotso.
- vwap - turnover% volume - karolelano ea theko ka motsotso e lekantsoeng ka boholo ba transaction.
- cumVolume - palo ea kakaretso - boholo bo bokelletsoeng ba litšebelisano ka nako eohle.
Ha re bue hang hang ka ntlha e le 'ngoe e sa totobalang - mokhoa oa ho qala likholomo tsena ka lekhetlo la pele le motsotso o mong le o mong o latelang. Likholomo tse ling tsa mofuta oa firstPrice li tlameha ho qalisoa hore li be lefeela nako le nako; boleng ba tsona ha bo hlalosoe. Mefuta e meng ea molumo e tlameha ho lula e behiloe ho 0. Ho boetse ho na le litšiea tse hlokang mokhoa o kopantsoeng - mohlala, cumVolume e tlameha ho kopitsoa ho tloha motsotsong o fetileng, 'me bakeng sa oa pele o behiloe ho 0. Ha re behe liparamente tsena kaofela re sebelisa data ea bukantswe. mofuta (o tšoana le rekoto):
// list ! list – создать словарь, 0n – float null, 0N – long null, `sym – тип символ, `sym1`sym2 – список символов
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time; // список всех вычисляемых колонок, reverse объяснен ниже
Ke kentse sym le nako ho dikishinari bakeng sa boiketlo, hona joale initWith ke mohala o lokiselitsoeng ho tloha tafoleng ea ho qetela e kopantsoeng, moo e setseng ho beha sym le nako e nepahetseng. U ka e sebelisa ho kenya mela e mecha tafoleng.
Re tla hloka li-aggCols ha re theha ts'ebetso ea ho kopanya. Lethathamo le tlameha ho khelosoa ka lebaka la tatellano eo lipoleloana ho Q li hlahlojoang (ho tloha ho le letona ho ea ho le letšehali). Sepheo ke ho netefatsa hore lipalo li tloha holimo ho ea ho cumVolume, kaha likholomo tse ling li itšetlehile ka tse fetileng.
Likholomo tse hlokang ho kopitsoa motsotsong o mocha ho tloha ho o fetileng, kholumo ea sym e eketsoa molemong oa:
rollColumns:`sym`cumVolume;
Joale ha re aroleng litšiea ka lihlopha ho latela hore na li lokela ho nchafatsoa joang. Mefuta e meraro e ka khetholloa:
- Li-accumulators (volume, turnover, ..) - re tlameha ho eketsa boleng bo kenang ho e fetileng.
- Ka ntlha e khethehileng (e phahameng, e tlaase, ..) - boleng ba pele ka motsotso bo nkiloe ho data e kenang, tse ling kaofela li baloa ho sebelisoa mosebetsi.
- Phomola. Ka linako tsohle e baloa ho sebelisoa ts'ebetso.
Ha re hlalose mefuta e fapaneng ea lihlopha tsena:
accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;
Taelo ea lipalo
Re tla nchafatsa tafole e kopaneng ka mekhahlelo e 'meli. Bakeng sa katleho, re qala ka ho fokotsa tafole e kenang e le hore ho be le mola o le mong bakeng sa sebapali le motsotso ka mong. Taba ea hore mesebetsi eohle ea rona e ea eketseha ebile e kopane e tiisa hore sephetho sa mohato ona oa tlatsetso se ke ke sa fetoha. U ka fokotsa tafole ka ho khetha:
select high:max price, low:min price … by sym,time.minute from table
Mokhoa ona o na le bothata - sete sa litšiea tse baloang se hlalositsoe esale pele. Ka lehlohonolo, ho Q, khetha e boetse e kengoa ts'ebetsong e le ts'ebetso moo o ka kenyang likhang tse entsoeng ka matla:
?[table;whereClause;byClause;selectClause]
Nke ke ka hlalosa ka botlalo sebopeho sa likhang; molemong oa rona, ke feela ka ho khetha lipolelo tse tla ba tse sa reng letho 'me e lokela ho ba lidikishinari tsa likholomo tsa foromo!lipolelo. Kahoo, mosebetsi oa ho fokotseha o ka hlalosoa ka tsela e latelang:
selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size"); // each это функция map в Q для одного списка
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];
Bakeng sa ho hlaka, ke sebelisitse parse function, e fetolang khoele e nang le polelo ea Q hore e be boleng bo ka fetisetsoang ho eval function le e hlokahalang ho khetho ea mosebetsi. Hape hlokomela hore preprocess e hlalosoa e le projeke (ke hore, ts'ebetso e nang le likhang tse sa hlalosoang ka mokhoa o itseng) ea mosebetsi o khethiloeng, khang e le 'ngoe (tafole) ha e eo. Haeba re sebelisa preprocess ho tafole, re tla fumana tafole petelitsoe.
Mohato oa bobeli ke ho ntlafatsa tafole e kopaneng. Ha re qaleng ho ngola algorithm ka pseudocode:
for each sym in inputTable
idx: row index in agg table for sym+currentTime;
aggTable[idx;`high]: aggTable[idx;`high] | inputTable[sym;`high];
aggTable[idx;`volume]: aggTable[idx;`volume] + inputTable[sym;`volume];
…
Ho Q, ho tloaelehile ho sebelisa 'mapa/fokotsa mesebetsi ho e-na le li-loops. Empa kaha Q ke puo ea vector mme re ka sebelisa lits'ebetso tsohle habonolo ho matšoao ohle ka nako e le 'ngoe, joale ka tekanyo ea pele re ka e etsa ntle le loop ho hang, re etsa ts'ebetso ho matšoao ohle hang-hang:
idx:calcIdx inputTable;
row:aggTable idx;
aggTable[idx;`high]: row[`high] | inputTable`high;
aggTable[idx;`volume]: row[`volume] + inputTable`volume;
…
Empa re ka fetela pele, Q e na le opareitara e ikhethang le e matla haholo - mosebetsi o akaretsang oa kabelo. E u lumella ho fetola sete ea boleng ka har'a sebopeho se rarahaneng sa data u sebelisa lethathamo la li-indices, mesebetsi le likhang. Tabeng ea rona, e shebahala tjena:
idx:calcIdx inputTable;
rows:aggTable idx;
// .[target;(idx0;idx1;..);function;argument] ~ target[idx 0;idx 1;…]: function[target[idx 0;idx 1;…];argument], в нашем случае функция – это присваивание
.[aggTable;(idx;aggCols);:;flip (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];
Ka bomalimabe, ho abela tafole u hloka lethathamo la mela, eseng likholomo, 'me u tlameha ho fetolela matrix (lenane la litšiea ho lethathamo la mela) u sebelisa ts'ebetso ea flip. Sena se bitsa chelete e ngata bakeng sa tafole e kholo, kahoo re sebelisa kabelo e akaretsang kholomong ka 'ngoe ka thoko, re sebelisa ts'ebetso ea 'mapa (e shebahalang joalo ka apostrophe):
.[aggTable;;:;]'[(idx;)each aggCols; (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];
Re boetse re sebelisa projeke ea tšebetso. Hape hlokomela hore ho Q, ho theha lenane ke ts'ebetso mme re ka e bitsa ka ho sebelisa (mapa) ts'ebetso ho fumana lethathamo la manane.
Ho etsa bonnete ba hore sete sa litšiea tse baloang ha sea ts'oaroa, re tla theha polelo e kaholimo ka matla. Ha re qale ka ho hlalosa mesebetsi ho bala kholomo ka 'ngoe, re sebelisa li-rolo le li-inp tse fapaneng ho bua ka data e kopaneng le e kentsoeng:
aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!
("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
Likholomo tse ling li ikhethile; boleng ba tsona ba pele ha boa lokela ho baloa ho latela tšebetso. Re ka tseba hore ke ea pele ka mola [`numTrades] kholomo - haeba e na le 0, joale boleng ke ba pele. Q e na le mosebetsi o khethiloeng - ?[Boolean list;list1;list2] - e khethang boleng ho tsoa lethathamong la 1 kapa la 2 ho latela maemo a khang ea pele:
// high -> ?[isFirst;inp`high;row[`high]|inp`high]
// @ - тоже обобщенное присваивание для случая когда индекс неглубокий
@[`aggExpression;specialCols;{[x;y]"?[isFirst;inp`",y,";",x,"]"};string specialCols];
Mona ke bitsitse kabelo e akaretsang ka ts'ebetso ea ka (polelo e ka har'a lithapo tse kobehileng). E amohela boleng ba hona joale (khang ea pele) le khang e eketsehileng, eo ke e fetisang ka parameter ea 4.
Ha re kenye libui tsa betri ka thoko, hobane ts'ebetso e ts'oana ho tsona:
// volume -> row[`volume]+inp`volume
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
Ena ke kabelo e tloaelehileng ho latela maemo a Q, empa ke fana ka lethathamo la litekanyetso hang-hang. Qetellong, ha re theheng mosebetsi oa mantlha:
// ":",/:aggExprs ~ map[{":",x};aggExpr] => ":row[`high]|inp`high" присвоим вычисленное значение переменной, потому что некоторые колонки зависят от уже вычисленных значений
// string[cols],'exprs ~ map[,;string[cols];exprs] => "high:row[`high]|inp`high" завершим создание присваивания. ,’ расшифровывается как map[concat]
// ";" sv exprs – String from Vector (sv), соединяет список строк вставляя “;” посредине
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}";
Ka polelo ena, ke theha ts'ebetso ho tsoa khoeleng e nang le polelo eo ke faneng ka eona ka holimo. Sephetho se tla shebahala tjena:
{[aggTable;idx;inp] rows:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols ;(cumVolume:row[`cumVolume]+inp`cumVolume;… ; high:?[isFirst;inp`high;row[`high]|inp`high])]}
Taelo ea tlhahlobo ea kholomo e fetotsoe hobane ho Q tatellano ea tlhahlobo e tloha ho le letona ho ea ho le letšehali.
Hona joale re na le mesebetsi e 'meli e meholo e hlokahalang bakeng sa lipalo, re hloka feela ho eketsa lisebelisoa tse nyane mme tšebeletso e se e loketse.
Mehato ea ho qetela
Re na le mesebetsi ea preprocess le updateAgg e etsang mosebetsi oohle. Empa ho ntse ho hlokahala ho netefatsa phetoho e nepahetseng ka metsotso le ho bala li-index bakeng sa ho kopanya. Pele ho tsohle, a re hlaloseng ts'ebetso ea init:
init:{
tradeAgg:: 0#enlist[initWith]; // создаем пустую типизированную таблицу, enlist превращает словарь в таблицу, а 0# означает взять 0 элементов из нее
currTime::00:00; // начнем с 0, :: означает, что присваивание в глобальную переменную
currSyms::`u#`symbol$(); // `u# - превращает список в дерево, для ускорения поиска элементов
offset::0; // индекс в tradeAgg, где начинается текущая минута
rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg; // кэш для последних значений roll колонок, таблица с ключом sym
}
Re tla boela re hlalose ts'ebetso ea roll, e tla fetola motsotso oa hona joale:
roll:{[tm]
if[currTime>tm; :init[]]; // если перевалили за полночь, то просто вызовем init
rollCache,::offset _ rollColumns#tradeAgg; // обновим кэш – взять roll колонки из aggTable, обрезать, вставить в rollCache
offset::count tradeAgg;
currSyms::`u#`$();
}
Re tla hloka tšebetso ho eketsa litlhaku tse ncha:
addSyms:{[syms]
currSyms,::syms; // добавим в список известных
// добавим в таблицу sym, time и rollColumns воспользовавшись обобщенным присваиванием.
// Функция ^ подставляет значения по умолчанию для roll колонок, если символа нет в кэше. value flip table возвращает список колонок в таблице.
`tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime), (initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
}
Qetellong, ts'ebetso ea upd (lebitso la setso bakeng sa ts'ebetso ena bakeng sa lits'ebeletso tsa Q), e bitsoang ke moreki ho eketsa data:
upd:{[tblName;data] // tblName нам не нужно, но обычно сервис обрабатывает несколько таблиц
tm:exec distinct time from data:() xkey preprocess data; // preprocess & calc time
updMinute[data] each tm; // добавим данные для каждой минуты
};
updMinute:{[data;tm]
if[tm<>currTime; roll tm; currTime::tm]; // поменяем минуту, если необходимо
data:select from data where time=tm; // фильтрация
if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms]; // новые символы
updateAgg[`tradeAgg;offset+currSyms?syms;data]; // обновим агрегированную таблицу. Функция ? ищет индекс элементов списка справа в списке слева.
};
Ke phetho. Khoutu e felletseng ea ts'ebeletso ea rona ke ena, joalo ka ha e tšepisitsoe, mela e 'maloa feela:
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time;
rollColumns:`sym`cumVolume;
accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;
selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size");
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];
aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
@[`aggExpression;specialCols;{"?[isFirst;inp`",y,";",x,"]"};string specialCols];
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}"; / '
init:{
tradeAgg::0#enlist[initWith];
currTime::00:00;
currSyms::`u#`symbol$();
offset::0;
rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg;
};
roll:{[tm]
if[currTime>tm; :init[]];
rollCache,::offset _ rollColumns#tradeAgg;
offset::count tradeAgg;
currSyms::`u#`$();
};
addSyms:{[syms]
currSyms,::syms;
`tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime),(initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
};
upd:{[tblName;data] updMinute[data] each exec distinct time from data:() xkey preprocess data};
updMinute:{[data;tm]
if[tm<>currTime; roll tm; currTime::tm];
data:select from data where time=tm;
if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms];
updateAgg[`tradeAgg;offset+currSyms?syms;data];
};
Teko
A re hlahlobeng tshebetso ya tshebeletso. Ho etsa sena, ha re e tsamaise ka mokhoa o ikhethileng (kenya khoutu ho file.q file) ebe u letsetsa init function:
q service.q –p 5566
q)init[]
Khomong e 'ngoe, qala ts'ebetso ea bobeli ea Q ebe u hokela ea pele:
h:hopen `:host:5566
h:hopen 5566 // если оба на одном хосте
Taba ea pele, ha re theheng lethathamo la matšoao - likotoana tse 10000 ebe re eketsa ts'ebetso ho theha tafole e sa reroang. Ho console ea bobeli:
syms:`IBM`AAPL`GOOG,-9997?`8
rnd:{[n;t] ([] sym:n?syms; time:t+asc n#til 25; price:n?10f; size:n?10)}
Ke kentse matšoao a mararo a sebele lethathamong ho etsa hore ho be bonolo ho li batla tafoleng. Mosebetsi oa rnd o etsa tafole e sa tloaelehang e nang le mela e n, moo nako e fapaneng ho tloha ho t ho ea ho t +25 milliseconds.
Joale o ka leka ho romella data ts'ebeletso (eketsa lihora tse leshome tsa pele):
{h (`upd;`trade;rnd[10000;x])} each `time$00:00 + til 60*10
U ka sheba ts'ebeletso hore tafole e ntlafalitsoe:
c 25 200
select from tradeAgg where sym=`AAPL
-20#select from tradeAgg where sym=`AAPL
Sephetho:
sym|time|high|low|firstPrice|lastPrice|firstSize|lastSize|numTrades|volume|pvolume|turnover|avgPrice|avgSize|vwap|cumVolume
--|--|--|--|--|--------------------------------
AAPL|09:27|9.258904|9.258904|9.258904|9.258904|8|8|1|8|9.258904|74.07123|9.258904|8|9.258904|2888
AAPL|09:28|9.068162|9.068162|9.068162|9.068162|7|7|1|7|9.068162|63.47713|9.068162|7|9.068162|2895
AAPL|09:31|4.680449|0.2011121|1.620827|0.2011121|1|5|4|14|9.569556|36.84342|2.392389|3.5|2.631673|2909
AAPL|09:33|2.812535|2.812535|2.812535|2.812535|6|6|1|6|2.812535|16.87521|2.812535|6|2.812535|2915
AAPL|09:34|5.099025|5.099025|5.099025|5.099025|4|4|1|4|5.099025|20.3961|5.099025|4|5.099025|2919Ha re etseng tlhahlobo ea mojaro ho fumana hore na tšebeletso e ka sebetsana le data e kae ka motsotso. E re ke u hopotse hore re beha nako ea ntlafatso ho 25 milliseconds. Ka hona, ts'ebeletso e tlameha (ka karolelano) ho lekana bonyane li-milliseconds tse 20 ka ntlafatso ho fa basebelisi nako ea ho kopa data. Kenya tse latelang ts'ebetsong ea bobeli:
tm:10:00:00.000
stressTest:{[n] 1 string[tm]," "; times,::h ({st:.z.T; upd[`trade;x]; .z.T-st};rnd[n;tm]); tm+:25}
start:{[n] times::(); do[4800;stressTest[n]]; -1 " "; `min`avg`med`max!(min times;avg times;med times;max times)}
4800 ke metsotso e 'meli. U ka leka ho matha pele bakeng sa mela e 1000 ka metsotsoana e meng le e meng e 25:
start 1000
Tabeng ea ka, sephetho se haufi le li-milliseconds tse 'maloa ka ntlafatso. Kahoo hang-hang ke tla eketsa palo ea mela ho 10.000:
start 10000
Sephetho:
min| 00:00:00.004
avg| 9.191458
med| 9f
max| 00:00:00.030
Hape, ha ho letho le ikhethang, empa sena ke mela e limilione tse 24 ka motsotso, likete tse 400 ka motsotsoana. Bakeng sa li-milliseconds tse fetang 25, ntlafatso e ile ea fokotseha ka makhetlo a 5 feela, ho hlakile ha motsotso o fetoha. Ha re eketsehe ho 100.000:
start 100000
Sephetho:
min| 00:00:00.013
avg| 25.11083
med| 24f
max| 00:00:00.108
q)sum times
00:02:00.532
Joalokaha u bona, ts'ebeletso ha e khone ho sebetsana ka katleho le eona, empa leha ho le joalo e khona ho lula e phaphametse. Palo e joalo ea data ( mela e limilione tse 240 ka motsotso) e kholo haholo; maemong a joalo, ho tloaelehile ho hlahisa li-clones tse 'maloa (kapa esita le li-clone tse ngata) tsa ts'ebeletso, e' ngoe le e 'ngoe e sebetsana le karolo ea batho bao ho buuoang ka bona. Leha ho le joalo, sephetho se khahla bakeng sa puo e tolokiloeng e shebaneng haholo le polokelo ea data.
Potso e ka hlaha mabapi le hore na ke hobane'ng ha nako e ntse e hōla ka tsela e sa tloaelehang ka boholo ba ntlafatso ka 'ngoe. Lebaka ke hore ts'ebetso ea shrink ha e le hantle ke ts'ebetso ea C, e sebetsang hantle ho feta updateAgg. Ho qala ka boholo bo itseng ba ntlafatso (e ka bang 10.000), updateAgg e fihla siling ea eona ebe nako ea eona ea ts'ebetso ha e itšetlehe ka boholo ba ntlafatso. Ke ka lebaka la mohato oa pele oa Q hore ts'ebeletso e khona ho cheka palo e joalo ea data. Sena se totobatsa hore na ho bohlokoa hakae ho khetha algorithm e nepahetseng ha u sebetsa le data e kholo. Ntlha e 'ngoe ke polokelo e nepahetseng ea data mohopolong. Haeba data e ne e sa bolokoa ka mokhoa o hlophisitsoeng kapa e sa laeloa ka nako, re ne re tla tloaelana le ntho e kang "cache ea TLB miss" - ho ba sieo ha aterese ea leqephe la memori ho cache ea aterese ea processor. Ho batla aterese ho nka nako e telele ka makhetlo a 30 haeba ho sa atlehe, 'me haeba data e hasane, e ka fokotsa tšebeletso ka makhetlo a mangata.
fihlela qeto e
Sehloohong sena, ke bonts'itse hore database ea KDB + le Q ha e loketse feela ho boloka data e kholo le ho e fumana habonolo ka khetho, empa hape le bakeng sa ho theha lits'ebeletso tsa ts'ebetso ea data tse khonang ho cheka mela/gigabytes ea limilione tse makholo tsa data esita le ho ts'ebetso e le 'ngoe ea Q. Puo ea Q ka boeona e lumella ts'ebetsong e khuts'oane haholo le e sebetsang ea li-algorithms tse amanang le ts'ebetso ea data ka lebaka la tlhaho ea eona ea vector, mofetoleli oa puo ea SQL e hahiloeng ka har'a SQL le sete e atlehileng haholo ea mesebetsi ea laeborari.
Ke tla hlokomela hore tse ka holimo ke karolo feela ea seo Q e ka se etsang, e na le likarolo tse ling tse ikhethang. Mohlala, protocol e bonolo haholo ea IPC e hlakolang moeli lipakeng tsa lits'ebetso tsa Q mme e o lumella ho kopanya makholo a lits'ebetso tsena ho netweke e le 'ngoe, e ka fumanehang ho li-server tse ngata libakeng tse fapaneng tsa lefats'e.
Source: www.habr.com