Siffofin Q da KDB+ ta amfani da misalin sabis na lokaci-lokaci

Kuna iya karanta game da menene tushen KDB+, yaren shirye-shiryen Q, menene ƙarfi da raunin su a cikin na baya labarin kuma a takaice a gabatarwa. A cikin labarin, za mu aiwatar da sabis akan Q wanda zai aiwatar da rafi mai shigowa da ƙididdige ayyukan tarawa daban-daban a kowane minti a cikin yanayin "ainihin lokaci" (watau zai sami lokaci don ƙididdige komai kafin ɓangaren bayanan na gaba). Babban fasalin Q shine yaren vector wanda ke ba ku damar aiki ba tare da abubuwa guda ɗaya ba, amma tare da tsararrun su, tsararrun tsararru da sauran abubuwa masu rikitarwa. Harsuna irin su Q da danginsa K, J, APL sun shahara saboda gajarta. Sau da yawa, shirin da ke ɗaukar hotuna da yawa na lamba a cikin harshen da aka saba kamar Java ana iya rubuta su a cikin ƴan layika. Wannan shine abin da nake so in nuna a cikin wannan labarin.

Gabatarwar

KDB+ shi ne tushen bayanai na columnar da aka mayar da hankali kan adadi mai yawa na bayanai, wanda aka yi oda ta takamammen hanya (musamman ta lokaci). Ana amfani da shi da farko a cikin cibiyoyin kuɗi - bankuna, kudaden zuba jari, kamfanonin inshora. Harshen Q shine yaren cikin gida na KDB+ wanda ke ba ku damar yin aiki tare da wannan bayanan yadda ya kamata. Q akidar gajarta ce da inganci, yayin da ake sadaukar da bayyananne. Wannan ya tabbatar da gaskiyar cewa harshen vector zai yi wuyar fahimta a kowane hali, kuma gajeriyar rakodin da wadatar rikodin yana ba ku damar ganin babban ɓangaren shirin akan allo ɗaya, wanda a ƙarshe ya sa sauƙin fahimta.

A cikin wannan labarin muna aiwatar da cikakken shiri a cikin Q kuma kuna iya gwada shi. Don yin wannan, kuna buƙatar ainihin Q. Kuna iya saukar da sigar 32-bit kyauta akan gidan yanar gizon kamfanin kx - www.kx.com. A can, idan kuna sha'awar, za ku sami bayanan tunani akan Q, littafin Q Ga Matattu da labarai daban-daban akan wannan batu.

Tsara matsalar

Akwai tushen da ke aika tebur tare da bayanai kowane miliyon 25. Tun da KDB + da aka yi amfani da farko a cikin kudi, za mu ɗauka cewa wannan shi ne tebur na ma'amaloli (cinikai), wanda yana da wadannan ginshikan: lokaci (lokaci a cikin milliseconds), sym (nadi kamfani a kan stock musayar -). IBM, AAPL,…), farashin (farashin da aka sayi hannun jari), girman (girman ciniki). Tazarar millisecond 25 na sabani ne, ba ƙanƙanta ba kuma ba tsayi da yawa ba. Kasancewar sa yana nufin cewa bayanan sun zo kan sabis ɗin da aka riga an ɓoye su. Zai zama mai sauƙi don aiwatar da buffer a gefen sabis, gami da buffering mai ƙarfi dangane da nauyin da ake ciki yanzu, amma don sauƙi, za mu mai da hankali kan tsayayyen tazara.

Dole ne sabis ɗin ya ƙidaya kowane minti ɗaya don kowace alama mai shigowa daga ginshiƙin alamar saitin ayyukan tarawa - max farashin, matsakaicin farashin, jimlar girman, da sauransu. bayanai masu amfani. Don sauƙi, za mu ɗauka cewa duk ayyuka za a iya ƙididdige su da yawa, watau. don samun sabon ƙima, ya isa ya san lambobi biyu - tsoho da ƙimar mai shigowa. Misali, madaidaicin ayyuka, matsakaita, jimla suna da wannan kadara, amma aikin tsaka-tsaki baya.

Za mu kuma ɗauka cewa rafin bayanai masu shigowa lokaci ne da aka yi oda. Wannan zai ba mu damar yin aiki tare da minti na ƙarshe kawai. A aikace, ya isa ya iya yin aiki tare da na yanzu da mintuna na baya idan wasu sabuntawa sun makara. Don sauƙi, ba za mu yi la'akari da wannan batu ba.

Ayyukan tarawa

Ayyukan tarawa da ake buƙata an jera su a ƙasa. Na ɗauki yawancinsu gwargwadon yiwuwa don ƙara nauyi akan sabis:

• high – max price – matsakaicin farashin a minti daya.
• low -min farashin - mafi ƙarancin farashi a minti daya.
• firstPrice - farashin farko - farashin farko a minti daya.
• lastPrice - farashin karshe - farashin karshe a minti daya.
• Girman farko - girman farko - girman ciniki na farko a minti daya.
• Girman ƙarshe - Girman ƙarshe - Girman ciniki na ƙarshe a cikin minti ɗaya.
• numTrades - ƙidaya i - adadin cinikai a minti daya.
• girma – jimlar girman – jimlar girman ciniki a minti daya.
• pvolume - jimlar farashin - jimlar farashin a minti daya, ana buƙata don ƙimar ƙimar.
• - jimlar farashin juyawa * girman - jimlar yawan ma'amaloli a minti daya.
• avgPrice - pvolume%numTrades - matsakaicin farashi a minti daya.
• avgSize – girma%numTrades – matsakaicin girman ciniki a minti daya.
• vwap - juzu'i% girma - matsakaicin farashi a minti daya wanda aka yi nauyi ta girman ciniki.
• cumVolume – jimlar girma – tara girman ma’amaloli a tsawon lokaci.

Nan da nan mu tattauna batu ɗaya wanda ba a bayyane yake ba - yadda ake fara waɗannan ginshiƙan a karon farko da kowane minti na gaba. Wasu ginshiƙai na nau'in Farko na farko dole ne a fara su su soke kowane lokaci; ƙimar su ba ta fayyace ba. Dole ne a saita sauran nau'ikan girma koyaushe zuwa 0. Hakanan akwai ginshiƙai waɗanda ke buƙatar hanyar haɗin gwiwa - alal misali, cumVolume dole ne a kwafi daga mintin da ya gabata, kuma na farko saita saita zuwa 0. Bari mu saita duk waɗannan sigogi ta amfani da bayanan ƙamus. nau'in (mai kama da rikodin):

// list ! list – создать словарь, 0n – float null, 0N – long null, `sym – тип символ, `sym1`sym2 – список символов
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time; // список всех вычисляемых колонок, reverse объяснен ниже

Na ƙara sym da lokaci zuwa ƙamus don dacewa, yanzu initWith shine layin da aka shirya daga tebur ɗin ƙarshe na ƙarshe, inda ya rage don saita madaidaicin alamar da lokaci. Kuna iya amfani da shi don ƙara sabbin layuka zuwa tebur.

Za mu buƙaci aggCols lokacin ƙirƙirar aikin tarawa. Dole ne a jujjuya lissafin saboda tsarin da aka tantance maganganu a cikin Q (daga dama zuwa hagu). Manufar ita ce tabbatar da lissafin ya tashi daga sama zuwa girma, tun da wasu ginshiƙai sun dogara da waɗanda suka gabata.

ginshiƙan da ke buƙatar kwafi zuwa sabon minti daga wanda ya gabata, ana ƙara ginshiƙin alamar don dacewa:

rollColumns:`sym`cumVolume;

Yanzu bari mu raba ginshiƙai zuwa rukuni gwargwadon yadda ya kamata a sabunta su. Ana iya bambanta iri uku:

1. Masu tarawa (ƙararar, juyawa, ..) - dole ne mu ƙara ƙimar mai shigowa zuwa na baya.
2. Tare da ma'ana ta musamman (high, low, ..) - ƙimar farko a cikin minti daya ana ɗaukar shi daga bayanan mai shigowa, sauran ana ƙididdige su ta amfani da aikin.
3. Huta Kullum ana lissafta ta amfani da aiki.

Bari mu ayyana masu canji na waɗannan azuzuwan:

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

Odar lissafi

Za mu sabunta teburin da aka tara a matakai biyu. Don dacewa, da farko muna rage tebur mai shigowa domin akwai layi ɗaya kawai ga kowane hali da minti. Kasancewar duk ayyukanmu na karuwa ne da haɗin kai yana ba da tabbacin cewa sakamakon wannan ƙarin matakin ba zai canza ba. Kuna iya rage teburin ta amfani da zaɓi:

select high:max price, low:min price … by sym,time.minute from table

Wannan hanya tana da hasara - saitin ginshiƙan ƙididdiga an riga an ƙayyade shi. Abin farin ciki, a cikin Q, ana kuma aiwatar da zaɓi azaman aiki inda zaku iya musanyawa ƙirƙira gardama:

?[table;whereClause;byClause;selectClause]

Ba zan bayyana dalla-dalla tsarin muhawarar ba; a cikin yanayinmu, kawai ta hanyar zaɓin maganganu ba za su zama marasa mahimmanci ba kuma ya kamata su zama ƙamus na ginshiƙan nau'ikan! Don haka, ana iya ayyana aikin raguwa kamar haka:

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size"); // each это функция map в Q для одного списка
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

Don tsabta, na yi amfani da aikin ɓarna, wanda ke juya kirtani tare da magana Q zuwa ƙimar da za a iya wuce zuwa aikin eval kuma wanda ake buƙata a cikin zaɓin aikin. Har ila yau lura cewa an ayyana preprocess azaman tsinkaya (watau, aiki tare da takamaiman takamaiman dalilai) na zaɓin aikin, hujja ɗaya (tebur) ya ɓace. Idan muka yi amfani da preprocess zuwa tebur, za mu sami tebur da aka matsa.

Mataki na biyu shine sabunta teburin da aka tara. Bari mu fara rubuta algorithm a pseudocode:

for each sym in inputTable
  idx: row index in agg table for sym+currentTime;
  aggTable[idx;`high]: aggTable[idx;`high] | inputTable[sym;`high];
  aggTable[idx;`volume]: aggTable[idx;`volume] + inputTable[sym;`volume];
  …

A Q, ya zama ruwan dare don amfani da taswira/rage ayyuka maimakon madaukai. Amma tunda Q shine harshen vector kuma zamu iya amfani da dukkan ayyuka cikin sauƙi zuwa dukkan alamomin lokaci ɗaya, sannan zuwa ƙimar farko za mu iya yi ba tare da madauki kwata-kwata ba, muna aiwatar da ayyuka akan duk alamomin lokaci ɗaya:

idx:calcIdx inputTable;
row:aggTable idx;
aggTable[idx;`high]: row[`high] | inputTable`high;
aggTable[idx;`volume]: row[`volume] + inputTable`volume;
…

Amma za mu iya ci gaba, Q yana da ma'aikaci na musamman kuma mai ƙarfi - ma'aikacin ɗawainiya gabaɗaya. Yana ba ku damar canza saitin dabi'u a cikin hadadden tsarin bayanai ta amfani da jerin fihirisa, ayyuka da muhawara. A wajenmu yana kama da haka:

idx:calcIdx inputTable;
rows:aggTable idx;
// .[target;(idx0;idx1;..);function;argument] ~ target[idx 0;idx 1;…]: function[target[idx 0;idx 1;…];argument], в нашем случае функция – это присваивание
.[aggTable;(idx;aggCols);:;flip (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Abin takaici, don sanya wa tebur kuna buƙatar jerin layuka, ba ginshiƙai ba, kuma dole ne ku canza matrix (jerin ginshiƙai zuwa jerin layuka) ta amfani da aikin juyewa. Wannan yana da tsada ga babban teburi, don haka a maimakon haka muna amfani da aikin gama gari ga kowane ginshiƙi daban, ta amfani da aikin taswira (wanda yayi kama da ridda):

.[aggTable;;:;]'[(idx;)each aggCols; (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Muna sake amfani da tsinkayar aiki. Hakanan lura cewa a cikin Q, ƙirƙirar jeri shima aiki ne kuma zamu iya kiransa ta amfani da kowane aikin (taswirar) don samun jerin jeri.

Don tabbatar da cewa ba a daidaita saitin ginshiƙan ƙididdiga ba, za mu ƙirƙiri maganganun da ke sama da ƙarfi. Bari mu fara ayyana ayyuka don ƙididdige kowane ginshiƙi, ta yin amfani da jeri da mabambantan inp don komawa zuwa ga haɗaɗɗun bayanan shigar da bayanai:

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!
 ("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");

Wasu ginshiƙai na musamman ne; bai kamata a ƙididdige ƙimar su ta farko ta aikin ba. Zamu iya ƙayyade cewa shine farkon ta layin [`numTrades] shafi - idan ya ƙunshi 0, to ƙimar ita ce ta farko. Q yana da zaɓin aiki -?[Lissafin Boolean; list1; list2] - wanda ke zaɓar ƙima daga lissafin 1 ko 2 dangane da yanayin da ke cikin hujja ta farko:

// high -> ?[isFirst;inp`high;row[`high]|inp`high]
// @ - тоже обобщенное присваивание для случая когда индекс неглубокий
@[`aggExpression;specialCols;{[x;y]"?[isFirst;inp`",y,";",x,"]"};string specialCols];

Anan na kira aiki na gaba ɗaya tare da aikina (magana a cikin takalmin gyaran kafa). Yana karɓar ƙimar halin yanzu (hujja ta farko) da ƙarin hujja, wanda na wuce a cikin siga na 4th.

Bari mu ƙara masu magana da baturi daban, tunda aikin ɗaya ne gare su:

// volume -> row[`volume]+inp`volume
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;

Wannan aiki ne na al'ada ta ma'aunin Q, amma ina ba da jerin ƙima a lokaci ɗaya. A ƙarshe, bari mu ƙirƙiri babban aikin:

// ":",/:aggExprs ~ map[{":",x};aggExpr] => ":row[`high]|inp`high" присвоим вычисленное значение переменной, потому что некоторые колонки зависят от уже вычисленных значений
// string[cols],'exprs ~ map[,;string[cols];exprs] => "high:row[`high]|inp`high" завершим создание присваивания. ,’ расшифровывается как map[concat]
// ";" sv exprs – String from Vector (sv), соединяет список строк вставляя “;” посредине
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}";

Da wannan furci, na ƙirƙiri aiki a hankali daga igiyar da ke ɗauke da kalmar da na bayar a sama. Sakamakon zai yi kama da haka:

{[aggTable;idx;inp] rows:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols ;(cumVolume:row[`cumVolume]+inp`cumVolume;… ; high:?[isFirst;inp`high;row[`high]|inp`high])]}

Ana jujjuya odar kimanta shafi saboda a cikin Q odar kimantawa daga dama zuwa hagu ne.

Yanzu muna da manyan ayyuka guda biyu da suka wajaba don ƙididdigewa, kawai muna buƙatar ƙara ƙaramin kayan aiki kuma sabis ɗin yana shirye.

Matakai na ƙarshe

Muna da preprocess da sabunta ayyukanAgg waɗanda ke yin duk aikin. Amma har yanzu yana da mahimmanci don tabbatar da madaidaiciyar canji ta hanyar mintuna da ƙididdige fihirisa don tarawa. Da farko, bari mu ayyana aikin init:

init:{
  tradeAgg:: 0#enlist[initWith]; // создаем пустую типизированную таблицу, enlist превращает словарь в таблицу, а 0# означает взять 0 элементов из нее
  currTime::00:00; // начнем с 0, :: означает, что присваивание в глобальную переменную
  currSyms::`u#`symbol$(); // `u# - превращает список в дерево, для ускорения поиска элементов
  offset::0; // индекс в tradeAgg, где начинается текущая минута 
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg; // кэш для последних значений roll колонок, таблица с ключом sym
 }

Za mu kuma ayyana aikin nadi, wanda zai canza minti na yanzu:

roll:{[tm]
  if[currTime>tm; :init[]]; // если перевалили за полночь, то просто вызовем init
  rollCache,::offset _ rollColumns#tradeAgg; // обновим кэш – взять roll колонки из aggTable, обрезать, вставить в rollCache
  offset::count tradeAgg;
  currSyms::`u#`$();
 }

Za mu buƙaci aiki don ƙara sabbin haruffa:

addSyms:{[syms]
  currSyms,::syms; // добавим в список известных
  // добавим в таблицу sym, time и rollColumns воспользовавшись обобщенным присваиванием.
  // Функция ^ подставляет значения по умолчанию для roll колонок, если символа нет в кэше. value flip table возвращает список колонок в таблице.
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime), (initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 }

Kuma a ƙarshe, aikin haɓakawa (sunan gargajiya na wannan aikin don ayyukan Q), wanda abokin ciniki ke kira don ƙara bayanai:

upd:{[tblName;data] // tblName нам не нужно, но обычно сервис обрабатывает несколько таблиц 
  tm:exec distinct time from data:() xkey preprocess data; // preprocess & calc time
  updMinute[data] each tm; // добавим данные для каждой минуты
};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm]; // поменяем минуту, если необходимо
  data:select from data where time=tm; // фильтрация
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms]; // новые символы
  updateAgg[`tradeAgg;offset+currSyms?syms;data]; // обновим агрегированную таблицу. Функция ? ищет индекс элементов списка справа в списке слева.
 };

Shi ke nan. Ga cikakken lambar sabis ɗinmu, kamar yadda aka yi alkawari, kaɗan kaɗan:

initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time;
rollColumns:`sym`cumVolume;

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size");
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
@[`aggExpression;specialCols;{"?[isFirst;inp`",y,";",x,"]"};string specialCols];
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}"; / '

init:{
  tradeAgg::0#enlist[initWith];
  currTime::00:00;
  currSyms::`u#`symbol$();
  offset::0;
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg;
 };
roll:{[tm]
  if[currTime>tm; :init[]];
  rollCache,::offset _ rollColumns#tradeAgg;
  offset::count tradeAgg;
  currSyms::`u#`$();
 };
addSyms:{[syms]
  currSyms,::syms;
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime),(initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 };

upd:{[tblName;data] updMinute[data] each exec distinct time from data:() xkey preprocess data};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm];
  data:select from data where time=tm;
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms];
  updateAgg[`tradeAgg;offset+currSyms?syms;data];
 };

Gwaji

Bari mu duba aikin sabis ɗin. Don yin wannan, bari mu gudanar da shi a cikin wani tsari daban (sanya lambar a cikin fayil ɗin service.q) kuma mu kira aikin init:

q service.q –p 5566

q)init[]

A cikin wani na'ura wasan bidiyo, fara aikin Q na biyu kuma haɗa zuwa na farko:

h:hopen `:host:5566
h:hopen 5566 // если оба на одном хосте

Da farko, bari mu ƙirƙiri jerin alamomin - guda 10000 kuma mu ƙara aiki don ƙirƙirar tebur bazuwar. A cikin na'ura mai kwakwalwa ta biyu:

syms:`IBM`AAPL`GOOG,-9997?`8
rnd:{[n;t] ([] sym:n?syms; time:t+asc n#til 25; price:n?10f; size:n?10)}

Na ƙara alamomin gaske guda uku zuwa jerin don sauƙaƙe neman su a cikin tebur. Aikin rnd yana ƙirƙirar tebur bazuwar tare da n layuka, inda lokaci ya bambanta daga t zuwa t +25 millise seconds.

Yanzu zaku iya gwada aika bayanai zuwa sabis ɗin (ƙara awanni goma na farko):

{h (`upd;`trade;rnd[10000;x])} each `time$00:00 + til 60*10

Kuna iya duba sabis ɗin cewa an sabunta teburin:

c 25 200
select from tradeAgg where sym=`AAPL
-20#select from tradeAgg where sym=`AAPL

Sakamako:

sym|time|high|low|firstPrice|lastPrice|firstSize|lastSize|numTrades|volume|pvolume|turnover|avgPrice|avgSize|vwap|cumVolume
--|--|--|--|--|--------------------------------
AAPL|09:27|9.258904|9.258904|9.258904|9.258904|8|8|1|8|9.258904|74.07123|9.258904|8|9.258904|2888
AAPL|09:28|9.068162|9.068162|9.068162|9.068162|7|7|1|7|9.068162|63.47713|9.068162|7|9.068162|2895
AAPL|09:31|4.680449|0.2011121|1.620827|0.2011121|1|5|4|14|9.569556|36.84342|2.392389|3.5|2.631673|2909
AAPL|09:33|2.812535|2.812535|2.812535|2.812535|6|6|1|6|2.812535|16.87521|2.812535|6|2.812535|2915
AAPL|09:34|5.099025|5.099025|5.099025|5.099025|4|4|1|4|5.099025|20.3961|5.099025|4|5.099025|2919

Bari yanzu mu aiwatar da gwajin lodi don gano adadin bayanan da sabis ɗin zai iya aiwatarwa a cikin minti daya. Bari in tunatar da ku cewa mun saita tazarar sabuntawa zuwa miliyon 25. Saboda haka, dole ne sabis ɗin (a matsakaita) ya dace da aƙalla mil 20 a kowace sabuntawa don baiwa masu amfani lokaci don neman bayanai. Shigar da waɗannan a cikin tsari na biyu:

tm:10:00:00.000
stressTest:{[n] 1 string[tm]," "; times,::h ({st:.z.T; upd[`trade;x]; .z.T-st};rnd[n;tm]); tm+:25}
start:{[n] times::(); do[4800;stressTest[n]]; -1 " "; `min`avg`med`max!(min times;avg times;med times;max times)}

4800 minti biyu ne. Kuna iya gwada gudu na farko don layuka 1000 kowane miliyon 25:

start 1000

A cikin yanayina, sakamakon yana kusa da daƙiƙa biyu a kowace sabuntawa. Don haka nan da nan zan ƙara adadin layuka zuwa 10.000:

start 10000

Sakamako:

min| 00:00:00.004
avg| 9.191458
med| 9f
max| 00:00:00.030

Bugu da ƙari, babu wani abu na musamman, amma wannan shine layin 24 miliyan a minti daya, 400 dubu a cikin dakika. Fiye da miliyon 25, sabuntawar ya ragu sau 5 kawai, a fili lokacin da minti ya canza. Mu kara zuwa 100.000:

start 100000

Sakamako:

min| 00:00:00.013
avg| 25.11083
med| 24f
max| 00:00:00.108
q)sum times
00:02:00.532

Kamar yadda kuke gani, da ƙyar sabis ɗin ba zai iya jurewa ba, amma duk da haka yana gudanar da zama a cikin ruwa. Irin wannan juzu'in bayanai (layukan miliyan 240 a cikin minti daya) yana da girma sosai; a irin waɗannan lokuta, ya zama ruwan dare don ƙaddamar da clones da yawa (ko ma da yawa na clones) na sabis ɗin, waɗanda kowannensu ke aiwatar da ɓangaren haruffa kawai. Duk da haka, sakamakon yana da ban sha'awa ga harshen da aka fassara wanda ya fi mayar da hankali kan ajiyar bayanai.

Tambayar na iya tasowa game da dalilin da yasa lokaci ke girma ba daidai ba tare da girman kowane sabuntawa. Dalilin shi ne cewa aikin raguwa shine ainihin aikin C, wanda ya fi dacewa fiye da updateAgg. Fara daga takamaiman girman ɗaukakawa (kusan 10.000), updateAgg ya kai rufin sa sannan lokacin aiwatarwarsa bai dogara da girman sabuntawa ba. Saboda matakin farko na Q cewa sabis ɗin ya sami damar narkar da irin wannan kundin bayanai. Wannan yana nuna yadda yake da mahimmanci don zaɓar madaidaicin algorithm lokacin aiki tare da manyan bayanai. Wani batu shine daidaitaccen ajiyar bayanai a ƙwaƙwalwar ajiya. Idan ba a adana bayanan ba a kan layi ko kuma ba a ba da umarnin lokaci ba, to za mu saba da irin wannan abu kamar kuskuren cache TLB - rashin adireshin shafin ƙwaƙwalwar ajiya a cikin cache adireshin mai sarrafawa. Neman adireshi yana ɗaukar tsawon kusan sau 30 idan bai yi nasara ba, kuma idan bayanan sun warwatse, zai iya rage hidimar sau da yawa.

ƙarshe

A cikin wannan labarin, na nuna cewa KDB+ da Q database sun dace ba kawai don adana manyan bayanai da kuma samun sauƙin shiga ta hanyar zaɓi ba, har ma don ƙirƙirar ayyukan sarrafa bayanai waɗanda ke da ikon narkar da daruruwan miliyoyin layuka / gigabytes na bayanai har ma a ciki. tsari Q guda daya. Harshen Q da kansa yana ba da izinin aiwatar da ƙayyadaddun ƙayyadaddun ƙayyadaddun algorithms masu alaƙa da sarrafa bayanai saboda yanayin yanayin sa, ginanniyar fassarar yaren SQL da ingantaccen tsarin ayyukan laburare.

Zan lura cewa abin da ke sama wani ɓangare ne kawai na abin da Q zai iya yi, yana da wasu siffofi na musamman kuma. Misali, ƙa'idar IPC mai sauƙi mai sauƙi wacce ke share iyaka tsakanin matakan Q guda ɗaya kuma tana ba ku damar haɗa ɗaruruwan waɗannan hanyoyin zuwa cibiyar sadarwa guda ɗaya, wacce za ta iya kasancewa akan sabar da yawa a sassa daban-daban na duniya.

source: www.habr.com