Cov yam ntxwv ntawm Q thiab KDB + cov lus siv piv txwv ntawm kev pabcuam tiag tiag

Koj tuaj yeem nyeem txog dab tsi KDB + lub hauv paus, Q programming lus yog dab tsi, lawv lub zog thiab qhov tsis zoo yog dab tsi hauv kuv yav dhau los Tshooj thiab luv luv hauv kev taw qhia. Hauv tsab xov xwm, peb yuav siv qhov kev pabcuam ntawm Q uas yuav ua cov ntaub ntawv xa tuaj thiab suav ntau qhov sib sau ua haujlwm txhua feeb hauv "lub sijhawm tiag tiag" hom (piv txwv li, nws yuav muaj sijhawm los xam txhua yam ua ntej cov ntaub ntawv txuas ntxiv). Lub ntsiab feature ntawm Q yog tias nws yog ib hom lus vector uas tso cai rau koj ua haujlwm tsis yog nrog ib qho khoom, tab sis nrog lawv cov arrays, arrays ntawm arrays thiab lwm yam khoom nyuaj. Cov lus xws li Q thiab nws cov txheeb ze K, J, APL yog nto moo rau lawv brevity. Feem ntau, ib qho kev pab cuam uas siv ntau lub vijtsam ntawm cov lej hauv cov lus paub zoo li Java tuaj yeem sau rau ntawm lawv ob peb kab. Nov yog qhov kuv xav qhia hauv kab lus no.

Taw qhia

KDB + yog columnar database tsom rau cov ntaub ntawv loj heev, xaj hauv ib txoj kev tshwj xeeb (feem ntau los ntawm lub sijhawm). Nws yog siv feem ntau hauv tuam txhab nyiaj txiag - tsev txhab nyiaj, peev nyiaj, tuam txhab tuav pov hwm. Cov lus Q yog cov lus sab hauv ntawm KDB + uas tso cai rau koj ua haujlwm zoo nrog cov ntaub ntawv no. Q lub tswv yim yog brevity thiab efficiency, thaum clarity yog txi. Qhov no yog qhov tsim nyog los ntawm qhov tseeb tias cov lus vector yuav nyuaj rau kev nkag siab nyob rau hauv txhua rooj plaub, thiab qhov brevity thiab nplua nuj ntawm cov ntaub ntawv tso cai rau koj pom ntau qhov loj ntawm qhov kev pab cuam ntawm ib qho screen, uas thaum kawg ua rau nws yooj yim rau kev nkag siab.

Hauv kab lus no peb siv qhov kev pabcuam puv ntoob hauv Q thiab koj yuav xav sim nws. Ua li no, koj yuav xav tau qhov tseeb Q. Koj tuaj yeem rub tawm dawb 32-ntsis version ntawm kx tuam txhab lub vev xaib - www.kx.com ua. Nyob ntawd, yog tias koj txaus siab, koj yuav pom cov ntaub ntawv siv ntawm Q, phau ntawv Q rau Mortals thiab ntau yam kab lus ntawm lub ncauj lus no.

Nqe lus ntawm qhov teeb meem

Muaj ib qhov chaw xa ib lub rooj nrog cov ntaub ntawv txhua 25 milliseconds. Txij li thaum KDB + yog siv feem ntau hauv kev nyiaj txiag, peb yuav xav tias qhov no yog lub rooj ntawm kev lag luam (kev lag luam), uas muaj cov kab hauv qab no: lub sij hawm (lub sij hawm hauv milliseconds), sym (lub tuam txhab npe ntawm kev sib pauv khoom lag luam - IBM, AAPL,…), tus nqi (tus nqi ntawm cov shares tau yuav), qhov loj (qhov loj ntawm qhov kev sib pauv). Lub sijhawm 25 millisecond yog arbitrary, tsis dhau me me thiab tsis ntev dhau. Nws lub xub ntiag txhais tau hais tias cov ntaub ntawv tuaj rau qhov kev pabcuam twb tau txais kev pabcuam. Nws yuav yooj yim rau kev siv buffering ntawm cov kev pabcuam sab, suav nrog dynamic buffering nyob ntawm qhov kev thauj khoom tam sim no, tab sis rau kev yooj yim, peb yuav tsom mus rau lub sijhawm ruaj khov.

Cov kev pabcuam yuav tsum suav txhua feeb rau txhua lub cim nkag los ntawm kab ke sym ib txheej ntawm kev sib sau ua haujlwm - max price, avg price, sum size, etc. cov ntaub ntawv tseem ceeb. Rau kev yooj yim, peb yuav xav tias tag nrho cov haujlwm tuaj yeem suav tau nce ntxiv, piv txwv li. kom tau txais tus nqi tshiab, nws yog txaus kom paub ob tus lej - qhov qub thiab cov nqi tuaj. Piv txwv li, cov dej num max, nruab nrab, sum muaj cov cuab yeej no, tab sis qhov nruab nrab muaj nuj nqi tsis.

Peb tseem yuav xav tias cov ntaub ntawv xa tuaj yog lub sijhawm xaj. Qhov no yuav muab sijhawm rau peb ua haujlwm nrog rau feeb kawg nkaus xwb. Hauv kev xyaum, nws txaus los ua haujlwm nrog cov feeb tam sim no thiab yav dhau los yog tias qee qhov kev hloov tshiab lig. Rau qhov yooj yim, peb yuav tsis xav txog qhov no.

Aggregation muaj nuj nqi

Cov haujlwm sib sau ua ke yuav tsum tau teev tseg hauv qab no. Kuv coj lawv ntau npaum li qhov ua tau kom nce lub nra ntawm qhov kev pabcuam:

• siab – max nqi – siab tshaj tus nqi ib feeb.
• qis – min nqi – yam tsawg kawg nkaus tus nqi ib feeb.
• firstPrice – thawj nqe – thawj nqe ib feeb.
• LastPrice - tus nqi kawg - tus nqi kawg ib feeb.
• firstSize – thawj qhov loj – thawj pauv loj ib feeb.
• LastSize - qhov loj kawg - qhov kawg pauv loj hauv ib feeb.
• numTrades - suav i - tus lej ntawm kev lag luam ib feeb.
• ntim – sum size – sum of trade sizes per feeb.
• pvolume – sum price – sum of price per feeb, xav tau rau avgPrice.
• – sum turnover price * size – tag nrho cov ntim ntawm muas ib feeb.
• avgPrice – pvolume%numTrades – nruab nrab nqi ib feeb.
• avgSize – ntim%numTrades – nruab nrab pauv loj ib feeb.
• vwap - turnover% ntim - tus nqi nruab nrab ntawm ib feeb uas hnyav los ntawm kev hloov pauv loj.
• cumVolume – sum volume – accumulated sizes of transactions tag nrho lub sij hawm.

Cia peb tham tam sim ntawd ib qho tsis pom tseeb - yuav ua li cas pib cov kab ntawv no thawj zaug thiab rau txhua feeb tom ntej. Qee kab ntawm FirstPrice hom yuav tsum tau pib rau null txhua lub sijhawm; lawv tus nqi yog undefined. Lwm hom ntim yuav tsum tau teem caij rau 0. Kuj tseem muaj cov kab uas yuav tsum muaj kev sib koom ua ke - piv txwv li, cumVolume yuav tsum tau theej los ntawm cov feeb dhau los, thiab rau thawj zaug teeb tsa rau 0. Cia peb teeb tag nrho cov kev txwv no siv cov ntaub ntawv txhais lus. hom (analogous rau cov ntaub ntawv):

// list ! list – создать словарь, 0n – float null, 0N – long null, `sym – тип символ, `sym1`sym2 – список символов
initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time; // список всех вычисляемых колонок, reverse объяснен ниже

Kuv ntxiv sym thiab sijhawm rau phau ntawv txhais lus kom yooj yim, tam sim no initWith yog ib txoj kab npaj ua los ntawm cov lus sib sau zaum kawg, qhov twg nws tseem yuav teeb tsa qhov tseeb sym thiab sijhawm. Koj tuaj yeem siv nws los ntxiv kab tshiab rau lub rooj.

Peb yuav xav tau aggCols thaum tsim kom muaj kev sib sau ua ke. Daim ntawv teev npe yuav tsum tau hloov pauv vim qhov kev txiav txim uas cov lus qhia hauv Q raug ntsuas (los ntawm sab xis mus rau sab laug). Lub hom phiaj yog los xyuas kom meej qhov kev suav mus los ntawm qhov siab mus rau cumVolume, vim qee cov kab nyob ntawm cov yav dhau los.

Cov kab uas yuav tsum tau theej rau ib feeb tshiab los ntawm yav dhau los, kab ke sym ntxiv rau kev yooj yim:

rollColumns:`sym`cumVolume;

Tam sim no cia peb faib cov kab rau hauv pawg raws li lawv yuav tsum hloov kho li cas. Peb hom tuaj yeem paub qhov txawv:

1. Accumulators (ntim, hloov pauv, ..) - peb yuav tsum ntxiv tus nqi nkag mus rau yav dhau los.
2. Nrog rau qhov tshwj xeeb taw tes (siab, qis, ..) - thawj tus nqi hauv feeb yog muab los ntawm cov ntaub ntawv tuaj, tus so yog suav nrog kev ua haujlwm.
3. So. Ib txwm suav nrog kev ua haujlwm.

Cia peb txhais cov variables rau cov chav kawm no:

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

xam kev txiav txim

Peb yuav hloov kho lub rooj sib sau ua ke hauv ob theem. Rau kev ua tau zoo, peb ua ntej shrink lub rooj tuaj kom muaj tsuas yog ib kab rau txhua tus cim thiab feeb. Qhov tseeb tias tag nrho peb cov dej num yog incremental thiab associative guarantees tias qhov tshwm sim ntawm cov kauj ruam ntxiv no yuav tsis hloov. Koj tuaj yeem shrink lub rooj siv xaiv:

select high:max price, low:min price … by sym,time.minute from table

Txoj kev no muaj qhov tsis zoo - cov txheej txheem suav tau ua ntej. Hmoov zoo, hauv Q, xaiv kuj yog siv los ua ib txoj haujlwm uas koj tuaj yeem hloov cov lus sib cav tsim tawm:

?[table;whereClause;byClause;selectClause]

Kuv yuav tsis piav qhia meej txog hom kev sib cav; nyob rau hauv peb cov ntaub ntawv, tsuas yog los ntawm thiab xaiv cov kab lus yuav yog nontrivial thiab lawv yuav tsum yog dictionaries ntawm daim ntawv kab lus! Yog li, lub shrinking muaj nuj nqi tuaj yeem txhais raws li hauv qab no:

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size"); // each это функция map в Q для одного списка
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

Txhawm rau kom meej meej, kuv siv qhov kev ua haujlwm parse, uas hloov txoj hlua nrog Q qhia rau hauv tus nqi uas tuaj yeem dhau mus rau qhov kev ua haujlwm eval thiab uas yuav tsum tau ua hauv cov haujlwm xaiv. Tsis tas li ntawd nco ntsoov tias preprocess yog txhais raws li qhov projection (piv txwv li, ib qho kev ua haujlwm nrog qee qhov kev sib cav) ntawm kev xaiv muaj nuj nqi, ib qho kev sib cav (lub rooj) ploj lawm. Yog hais tias peb thov preprocess rau ib lub rooj, peb yuav tau txais ib tug compressed rooj.

Qhov thib ob theem yog hloov kho lub rooj sib sau ua ke. Cia peb xub sau cov algorithm hauv pseudocode:

for each sym in inputTable
  idx: row index in agg table for sym+currentTime;
  aggTable[idx;`high]: aggTable[idx;`high] | inputTable[sym;`high];
  aggTable[idx;`volume]: aggTable[idx;`volume] + inputTable[sym;`volume];
  …

Hauv Q, nws yog ib qho uas siv daim ntawv qhia / txo cov haujlwm tsis yog lub voj voog. Tab sis txij li Q yog ib hom lus vector thiab peb tuaj yeem siv tau yooj yim siv tag nrho cov haujlwm rau txhua lub cim ib zaug, tom qab ntawd mus rau qhov kwv yees thawj zaug peb tuaj yeem ua yam tsis muaj lub voj voog txhua, ua haujlwm ntawm txhua lub cim ib zaug:

idx:calcIdx inputTable;
row:aggTable idx;
aggTable[idx;`high]: row[`high] | inputTable`high;
aggTable[idx;`volume]: row[`volume] + inputTable`volume;
…

Tab sis peb tuaj yeem mus ntxiv, Q muaj tus neeg ua haujlwm tshwj xeeb thiab muaj zog heev - tus neeg ua haujlwm ua haujlwm dav dav. Nws tso cai rau koj los hloov cov txheej txheem ntawm qhov tseem ceeb hauv cov ntaub ntawv nyuaj uas siv cov npe ntawm cov ntsuas, cov haujlwm thiab cov lus sib cav. Hauv peb qhov xwm txheej nws zoo li no:

idx:calcIdx inputTable;
rows:aggTable idx;
// .[target;(idx0;idx1;..);function;argument] ~ target[idx 0;idx 1;…]: function[target[idx 0;idx 1;…];argument], в нашем случае функция – это присваивание
.[aggTable;(idx;aggCols);:;flip (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Hmoov tsis zoo, txhawm rau muab rau lub rooj koj xav tau ib daim ntawv teev cov kab, tsis yog kab, thiab koj yuav tsum tau hloov cov matrix (cov npe ntawm kab rau cov npe kab) siv lub flip muaj nuj nqi. Qhov no yog kim heev rau lub rooj loj, yog li ntawd peb siv ib qho kev qhia dav dav rau txhua kab sib cais, siv daim ntawv qhia ua haujlwm (uas zoo li apostrophe):

.[aggTable;;:;]'[(idx;)each aggCols; (row[`high] | inputTable`high;row[`volume] + inputTable`volume;…)];

Peb rov siv ua haujlwm projection. Tsis tas li ntawd nco ntsoov tias hauv Q, tsim ib daim ntawv teev npe kuj yog ib txoj haujlwm thiab peb tuaj yeem hu nws siv txhua qhov haujlwm (daim ntawv qhia) kom tau txais cov npe ntawm cov npe.

Txhawm rau kom ntseeg tau tias cov txheej txheem suav tsis raug kho, peb yuav tsim cov lus qhia saum toj no dynamically. Cia peb ua ntej txhais cov haujlwm los suav txhua kab, siv kab thiab inp sib txawv los xa mus rau cov ntaub ntawv sib sau ua ke thiab nkag:

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!
 ("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");

Qee kab ntawv tshwj xeeb; lawv thawj tus nqi yuav tsum tsis txhob suav los ntawm kev ua haujlwm. Peb tuaj yeem txiav txim siab tias nws yog thawj zaug ntawm kab [`numTrades] kem - yog tias nws muaj 0, ces tus nqi yog thawj zaug. Q muaj ib txoj haujlwm xaiv - ?[Boolean list;list1;list2] - uas xaiv tus nqi ntawm daim ntawv teev npe 1 lossis 2 nyob ntawm qhov xwm txheej hauv thawj qhov kev sib cav:

// high -> ?[isFirst;inp`high;row[`high]|inp`high]
// @ - тоже обобщенное присваивание для случая когда индекс неглубокий
@[`aggExpression;specialCols;{[x;y]"?[isFirst;inp`",y,";",x,"]"};string specialCols];

Ntawm no kuv hu ua txoj haujlwm dav dav nrog kuv txoj haujlwm (kev qhia hauv curly braces). Nws tau txais tus nqi tam sim no (qhov kev sib cav thawj zaug) thiab ib qho kev sib cav ntxiv, uas kuv dhau ntawm qhov ntsuas 4.

Cia peb ntxiv cov roj teeb hais lus sib cais, vim tias txoj haujlwm yog tib yam rau lawv:

// volume -> row[`volume]+inp`volume
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;

Qhov no yog ib txwm ua haujlwm los ntawm Q cov qauv, tab sis kuv tab tom muab cov npe ntawm cov txiaj ntsig ib zaug. Thaum kawg, cia peb tsim lub luag haujlwm tseem ceeb:

// ":",/:aggExprs ~ map[{":",x};aggExpr] => ":row[`high]|inp`high" присвоим вычисленное значение переменной, потому что некоторые колонки зависят от уже вычисленных значений
// string[cols],'exprs ~ map[,;string[cols];exprs] => "high:row[`high]|inp`high" завершим создание присваивания. ,’ расшифровывается как map[concat]
// ";" sv exprs – String from Vector (sv), соединяет список строк вставляя “;” посредине
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}";

Nrog rau qhov kev qhia no, kuv dynamically tsim ib tug muaj nuj nqi los ntawm ib txoj hlua uas muaj cov lus uas kuv tau muab rau saum toj no. Cov txiaj ntsig yuav zoo li no:

{[aggTable;idx;inp] rows:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols ;(cumVolume:row[`cumVolume]+inp`cumVolume;… ; high:?[isFirst;inp`high;row[`high]|inp`high])]}

Daim ntawv ntsuam xyuas kab ntawv yog inverted vim nyob rau hauv Q qhov kev ntsuam xyuas yog los ntawm sab xis mus rau sab laug.

Tam sim no peb muaj ob lub luag haujlwm tseem ceeb tsim nyog rau kev suav, peb tsuas yog xav tau ntxiv me ntsis kev tsim kho thiab cov kev pabcuam tau npaj txhij.

Cov kauj ruam kawg

Peb muaj preprocess thiab updateAgg functions uas ua tag nrho cov hauj lwm. Tab sis nws tseem tsim nyog los xyuas kom meej qhov kev hloov pauv ntawm cov feeb thiab suav cov indexes rau kev sib sau. Ua ntej tshaj plaws, cia peb txhais cov haujlwm init:

init:{
  tradeAgg:: 0#enlist[initWith]; // создаем пустую типизированную таблицу, enlist превращает словарь в таблицу, а 0# означает взять 0 элементов из нее
  currTime::00:00; // начнем с 0, :: означает, что присваивание в глобальную переменную
  currSyms::`u#`symbol$(); // `u# - превращает список в дерево, для ускорения поиска элементов
  offset::0; // индекс в tradeAgg, где начинается текущая минута 
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg; // кэш для последних значений roll колонок, таблица с ключом sym
 }

Peb kuj tseem yuav txhais cov haujlwm yob, uas yuav hloov lub feeb tam sim no:

roll:{[tm]
  if[currTime>tm; :init[]]; // если перевалили за полночь, то просто вызовем init
  rollCache,::offset _ rollColumns#tradeAgg; // обновим кэш – взять roll колонки из aggTable, обрезать, вставить в rollCache
  offset::count tradeAgg;
  currSyms::`u#`$();
 }

Peb yuav xav tau lub luag haujlwm ntxiv cov cim tshiab:

addSyms:{[syms]
  currSyms,::syms; // добавим в список известных
  // добавим в таблицу sym, time и rollColumns воспользовавшись обобщенным присваиванием.
  // Функция ^ подставляет значения по умолчанию для roll колонок, если символа нет в кэше. value flip table возвращает список колонок в таблице.
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime), (initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 }

Thiab thaum kawg, qhov kev ua haujlwm upd (lub npe ib txwm siv rau cov haujlwm no rau Q cov kev pabcuam), uas yog hu los ntawm tus neeg siv khoom ntxiv cov ntaub ntawv:

upd:{[tblName;data] // tblName нам не нужно, но обычно сервис обрабатывает несколько таблиц 
  tm:exec distinct time from data:() xkey preprocess data; // preprocess & calc time
  updMinute[data] each tm; // добавим данные для каждой минуты
};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm]; // поменяем минуту, если необходимо
  data:select from data where time=tm; // фильтрация
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms]; // новые символы
  updateAgg[`tradeAgg;offset+currSyms?syms;data]; // обновим агрегированную таблицу. Функция ? ищет индекс элементов списка справа в списке слева.
 };

Yog tag nrho. Nov yog txoj cai tiav ntawm peb cov kev pabcuam, raws li tau cog lus tseg, tsuas yog ob peb kab:

initWith:`sym`time`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover`avgPrice`avgSize`vwap`cumVolume!(`;00:00;0n;0n;0n;0n;0N;0N;0;0;0.0;0.0;0n;0n;0n;0);
aggCols:reverse key[initWith] except `sym`time;
rollColumns:`sym`cumVolume;

accumulatorCols:`numTrades`volume`pvolume`turnover;
specialCols:`high`low`firstPrice`firstSize;

selExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`numTrades`volume`pvolume`turnover!parse each ("max price";"min price";"first price";"last price";"first size";"last size";"count i";"sum size";"sum price";"sum price*size");
preprocess:?[;();`sym`time!`sym`time.minute;selExpression];

aggExpression:`high`low`firstPrice`lastPrice`firstSize`lastSize`avgPrice`avgSize`vwap`cumVolume!("row[`high]|inp`high";"row[`low]&inp`low";"row`firstPrice";"inp`lastPrice";"row`firstSize";"inp`lastSize";"pvolume%numTrades";"volume%numTrades";"turnover%volume";"row[`cumVolume]+inp`volume");
@[`aggExpression;specialCols;{"?[isFirst;inp`",y,";",x,"]"};string specialCols];
aggExpression[accumulatorCols]:{"row[`",x,"]+inp`",x } each string accumulatorCols;
updateAgg:value "{[aggTable;idx;inp] row:aggTable idx; isFirst_0=row`numTrades; .[aggTable;;:;]'[(idx;)each aggCols;(",(";"sv string[aggCols],'":",/:aggExpression aggCols),")]}"; / '

init:{
  tradeAgg::0#enlist[initWith];
  currTime::00:00;
  currSyms::`u#`symbol$();
  offset::0;
  rollCache:: `sym xkey update `u#sym from rollColumns#tradeAgg;
 };
roll:{[tm]
  if[currTime>tm; :init[]];
  rollCache,::offset _ rollColumns#tradeAgg;
  offset::count tradeAgg;
  currSyms::`u#`$();
 };
addSyms:{[syms]
  currSyms,::syms;
  `tradeAgg upsert @[count[syms]#enlist initWith;`sym`time,cols rc;:;(syms;currTime),(initWith cols rc)^value flip rc:rollCache ([] sym: syms)];
 };

upd:{[tblName;data] updMinute[data] each exec distinct time from data:() xkey preprocess data};
updMinute:{[data;tm]
  if[tm<>currTime; roll tm; currTime::tm];
  data:select from data where time=tm;
  if[count msyms:syms where not (syms:data`sym)in currSyms; addSyms msyms];
  updateAgg[`tradeAgg;offset+currSyms?syms;data];
 };

Kev Ntsuam Xyuas

Cia peb xyuas qhov ua tau zoo ntawm kev pabcuam. Txhawm rau ua qhov no, cia peb khiav nws hauv cov txheej txheem cais (muab cov lej tso rau hauv cov ntaub ntawv service.q) thiab hu rau init muaj nuj nqi:

q service.q –p 5566

q)init[]

Hauv lwm lub console, pib qhov txheej txheem Q thib ob thiab txuas mus rau thawj:

h:hopen `:host:5566
h:hopen 5566 // если оба на одном хосте

Ua ntej, cia peb tsim ib daim ntawv teev cov cim - 10000 daim thiab ntxiv ib txoj haujlwm los tsim ib lub rooj random. Hauv qhov thib ob console:

syms:`IBM`AAPL`GOOG,-9997?`8
rnd:{[n;t] ([] sym:n?syms; time:t+asc n#til 25; price:n?10f; size:n?10)}

Kuv ntxiv peb lub cim tiag rau hauv daim ntawv kom nws yooj yim dua los nrhiav lawv hauv lub rooj. rnd muaj nuj nqi tsim ib lub rooj random nrog n kab, qhov twg lub sij hawm txawv ntawm t mus rau t + 25 milliseconds.

Tam sim no koj tuaj yeem sim xa cov ntaub ntawv mus rau qhov kev pabcuam (ntxiv rau thawj kaum teev):

{h (`upd;`trade;rnd[10000;x])} each `time$00:00 + til 60*10

Koj tuaj yeem tshawb xyuas hauv qhov kev pabcuam uas lub rooj tau hloov kho:

c 25 200
select from tradeAgg where sym=`AAPL
-20#select from tradeAgg where sym=`AAPL

Tshwm sim:

sym|time|high|low|firstPrice|lastPrice|firstSize|lastSize|numTrades|volume|pvolume|turnover|avgPrice|avgSize|vwap|cumVolume
--|--|--|--|--|--------------------------------
AAPL|09:27|9.258904|9.258904|9.258904|9.258904|8|8|1|8|9.258904|74.07123|9.258904|8|9.258904|2888
AAPL|09:28|9.068162|9.068162|9.068162|9.068162|7|7|1|7|9.068162|63.47713|9.068162|7|9.068162|2895
AAPL|09:31|4.680449|0.2011121|1.620827|0.2011121|1|5|4|14|9.569556|36.84342|2.392389|3.5|2.631673|2909
AAPL|09:33|2.812535|2.812535|2.812535|2.812535|6|6|1|6|2.812535|16.87521|2.812535|6|2.812535|2915
AAPL|09:34|5.099025|5.099025|5.099025|5.099025|4|4|1|4|5.099025|20.3961|5.099025|4|5.099025|2919

Tam sim no cia peb ua cov kev sim thauj khoom kom paub seb cov ntaub ntawv ntau npaum li cas cov kev pabcuam tuaj yeem ua haujlwm ib feeb. Cia kuv ceeb toom koj tias peb teem caij hloov tshiab rau 25 milliseconds. Raws li, qhov kev pabcuam yuav tsum (qhov nruab nrab) haum rau tsawg kawg 20 milliseconds ib qhov hloov tshiab kom muab sijhawm rau cov neeg siv los thov cov ntaub ntawv. Nkag mus rau cov hauv qab no hauv tus txheej txheem thib ob:

tm:10:00:00.000
stressTest:{[n] 1 string[tm]," "; times,::h ({st:.z.T; upd[`trade;x]; .z.T-st};rnd[n;tm]); tm+:25}
start:{[n] times::(); do[4800;stressTest[n]]; -1 " "; `min`avg`med`max!(min times;avg times;med times;max times)}

4800 yog ob feeb. Koj tuaj yeem sim khiav ua ntej rau 1000 kab txhua 25 milliseconds:

start 1000

Hauv kuv qhov xwm txheej, qhov tshwm sim yog nyob ib puag ncig ob peb milliseconds ib qhov hloov tshiab. Yog li kuv mam li nce cov kab mus rau 10.000 tam sim:

start 10000

Tshwm sim:

min| 00:00:00.004
avg| 9.191458
med| 9f
max| 00:00:00.030

Ib zaug ntxiv, tsis muaj dab tsi tshwj xeeb, tab sis qhov no yog 24 lab kab ib feeb, 400 txhiab ib ob. Rau ntau tshaj 25 milliseconds, qhov hloov tshiab qeeb tsuas yog 5 zaug, pom meej thaum lub feeb hloov. Cia peb nce mus rau 100.000:

start 100000

Tshwm sim:

min| 00:00:00.013
avg| 25.11083
med| 24f
max| 00:00:00.108
q)sum times
00:02:00.532

Raws li koj tuaj yeem pom, qhov kev pabcuam tsis tshua muaj peev xwm tiv taus, tab sis txawm li cas los xij nws tswj kom nyob twj ywm. Xws li qhov ntim ntawm cov ntaub ntawv (240 lab kab hauv ib feeb) yog qhov loj heev; nyob rau hauv cov ntaub ntawv no, nws yog ib qho kev tso tawm ntau lub clones (lossis txawm tias ntau ntau clones) ntawm cov kev pabcuam, txhua tus txheej txheem tsuas yog ib feem ntawm cov cim. Txawm li cas los xij, qhov tshwm sim yog qhov zoo rau cov lus txhais uas tsom ntsoov rau kev khaws cov ntaub ntawv.

Cov lus nug yuav tshwm sim vim li cas lub sij hawm loj hlob tsis-linearly nrog qhov loj ntawm txhua qhov hloov tshiab. Yog vim li cas yog tias lub ntsws muaj nuj nqi yog qhov ua haujlwm C, uas yog qhov ua tau zoo dua li updateAgg. Pib los ntawm qee qhov hloov tshiab loj (ib puag ncig 10.000), updateAgg ncav cuag nws lub qab nthab thiab tom qab ntawd nws lub sijhawm ua tiav tsis nyob ntawm qhov hloov tshiab loj. Nws yog vim cov kauj ruam ua ntej Q uas cov kev pabcuam muaj peev xwm zom tau cov ntaub ntawv ntim. Qhov no qhia txog qhov tseem ceeb npaum li cas los xaiv txoj cai algorithm thaum ua haujlwm nrog cov ntaub ntawv loj. Lwm lub ntsiab lus yog qhov tseeb khaws cov ntaub ntawv hauv nco. Yog tias cov ntaub ntawv tsis tau khaws cia hauv columnarly lossis tsis raug xaj los ntawm lub sijhawm, ces peb yuav paub txog qhov zoo li TLB cache nco - qhov tsis muaj lub cim xeeb nplooj ntawv chaw nyob hauv processor chaw nyob cache. Kev tshawb nrhiav qhov chaw nyob yuav siv sijhawm ntev dua 30 lub sijhawm yog tias tsis ua tiav, thiab yog tias cov ntaub ntawv tau tawg, nws tuaj yeem ua rau qeeb ntawm kev pabcuam ntau zaus.

xaus

Hauv tsab xov xwm no, kuv tau pom tias KDB + thiab Q cov ntaub ntawv tsim nyog tsis yog tsuas yog khaws cov ntaub ntawv loj thiab yooj yim nkag mus rau nws los ntawm kev xaiv, tab sis kuj tseem tsim cov kev pabcuam ua cov ntaub ntawv uas muaj peev xwm zom tau ntau pua lab kab / gigabytes ntawm cov ntaub ntawv txawm nyob hauv ib qho txheej txheem Q. Cov lus Q nws tus kheej tso cai rau kev ua kom zoo thiab ua tau zoo ntawm kev siv cov algorithms ntsig txog kev ua cov ntaub ntawv vim nws cov vector xwm, built-in SQL tus neeg txhais lus thiab cov txheej txheem zoo heev ntawm cov tsev qiv ntawv.

Kuv yuav nco ntsoov tias cov saum toj no tsuas yog ib feem ntawm qhov Q tuaj yeem ua tau, nws muaj lwm yam tshwj xeeb thiab. Piv txwv li, tus txheej txheem IPC yooj yim heev uas tshem tawm ciam teb ntawm cov txheej txheem Q tus neeg thiab tso cai rau koj los ua ke ntau pua ntawm cov txheej txheem no rau hauv ib lub network, uas tuaj yeem nyob ntawm ntau lub servers hauv ntau qhov chaw ntawm lub ntiaj teb.

Tau qhov twg los: www.hab.com