Nhanganyaya kune Functional Dependencies

Muchinyorwa chino tichataura nezve kutsamira kwekushanda mumadhatabhesi - zvaari, paanoshandiswa uye ndeapi maalgorithms aripo kuti awawane.

Isu tichafunga nezve mashandiro anoenderana nemamiriro ehukama dhatabhesi. Kuti uzviise zvakanyanya, mune akadaro dhatabhesi ruzivo rwakachengetwa mumhando yematafura. Tevere, isu tinoshandisa fungidziro yekufungidzira isingachinjike mune yakasimba relational theory: isu tichadaidza iyo tafura pachayo hukama, makoramu - hunhu (yadzo seti - yehukama schema), uye seti yemitsara kukosha pane subset yehunhu. - a tuple.

Somuenzaniso, mutafura iri pamusoro, (Benson, M, M nhengo) ituple yehunhu (Murwere, Pauro, Chiremba).
Zvikuru zviri pamutemo, izvi zvakanyorwa sezvinotevera: [Murwere, Gender, Chiremba] = (Benson, M, M nhengo).
Iye zvino isu tinogona kuunza iyo pfungwa yekushanda kutsamira (FD):

Tsanangudzo 1. Hukama R hunogutsa mutemo wemubatanidzwa X → Y (uko X, Y ⊆ R) kana uye chete kana kune chero matuples , ∈ R inobata: kana [X] = [X], ipapo [Y] = [Y]. Muchiitiko ichi, tinoti X (iyo inotemerwa, kana kutsanangura seti yehunhu) inoshanda inosarudza Y (iyo inotsamira seti).

Mune mamwe mazwi, kuvapo kwemutemo wemubatanidzwa X → Y zvinoreva kuti kana tiine matuples maviri mukati R uye dzinoenderana muhunhu X, ipapo vachapindirana mune hunhu Y.
Uye zvino, muhurongwa. Ngatitarisei hunhu Mwoyo murefu и Zvepabonde izvo zvatinoda kuona kuti pane zvinotsamira pakati pavo here kana kuti kwete. Kune seti yehunhu hwakadaro, zvinotevera zvinotsamira zvinogona kuvapo:

1. Murwere → Gender
2. Gender → Mwoyo murefu

Sezvatsanangurwa pamusoro, kuitira kuti kutsamira kwekutanga kubate, imwe neimwe yakasarudzika kukosha kwekoramu Mwoyo murefu kukosha kwechikamu chimwe chete kunofanira kuenderana Zvepabonde. Uye kune muenzaniso tafura izvi ndizvo chaizvo. Nekudaro, izvi hazvishande zvakapesana, ndiko kuti, kutsamira kwechipiri hakuna kugutsikana, uye hunhu. Zvepabonde haisi determinant for Mwoyo murefu. Saizvozvowo, kana tikatora kutsamira Chiremba → Mwoyo murefu, unogona kuona kuti yakatyorwa, kubva kukosha Robin hunhu uhu hune zvirevo zvakasiyana siyana - Ellis naGraham.

Saka, kutsamira kwekushanda kunoita kuti zvikwanise kuona hukama huripo pakati pemaseti ezvimiro zvetafura. Kubva pano zvichienda mberi isu tichatarisa zvinonyanya kufadza kubatana, kana kuti zvakadaro X → Yzvavari:

• isiri-diki, ndiko kuti, rutivi rworudyi rwekutsamira haisi chikamu chekuruboshwe (Y ̸⊆ X);
• zvishoma, ndiko kuti, hapana kutsamira kwakadaro Z → Y, icho Z ⊂ X.

Izvo zvinotarisirwa kusvika panguva ino zvaive zvakasimba, kureva kuti, hazvina kupa chero kukanganisa patafura, asi kuwedzera kwavari, kunewo izvo zvinobvumira kusawirirana pakati pehutsika hwema tuples. Kutsamira kwakadaro kunoiswa mukirasi yakaparadzana, inonzi approximate, uye inobvumirwa kutyorwa kune imwe nhamba yematuples. Iyi mari inodzorwa neiyo yakanyanya kukanganisa chiratidzo emax. Semuenzaniso, chiyero chekukanganisa = 0.01 inogona kureva kuti kutsamira kunogona kutyorwa ne1% yematuples aripo pane yakatariswa seti yehunhu. Ndiko kuti, kune 1000 zvinyorwa, huwandu hwegumi tuples hunogona kutyora Federal Law. Isu tichafunga nezve metric yakasiyana zvishoma, zvichibva pane maviri akasiyana maitiro eiyo tuples ari kuenzaniswa. Zvekupindwa muropa X → Y pamafungiro r inotaridzwa seizvi:

Ngativerengei kukanganisa kwe Chiremba → Mwoyo murefu kubva pamuenzaniso uri pamusoro. Tine matuples maviri ane hunhu hwakasiyana pahunhu Mwoyo murefu, asi pindirana Chiremba: [Chiremba, Murwere] = (Robin, Ellis) uye [Chiremba, Murwere] = (Robin, Graham) Tichitevera tsananguro yekukanganisa, isu tinofanirwa kufunga nezve ese anopokana maviri maviri, zvinoreva kuti pachave nevaviri vavo: (, ) uye kukanganisa kwayo (, ) Ngatiitsive mune fomula uye titore:

Zvino ngatiedzei kupindura mubvunzo: "Sei zvose nokuda?" Muchokwadi, mitemo yemubatanidzwa yakasiyana. Rudzi rwekutanga ndeizvo zvinotsamira zvinotemerwa nemutungamiriri padanho rekugadzira dhatabhesi. Ivo vanowanzova vashoma muhuwandu, vakasimba, uye iyo huru yekushandisa ndeye data normalization uye yehukama schema dhizaini.

Rudzi rwechipiri nderekutsamira, iyo inomiririra "yakavanzika" data uye hukama hwaimbozivikanwa pakati pehunhu. Ndiko kuti, kutsamira kwakadaro hakuna kufungidzirwa panguva yekugadzira uye ivo vanowanikwa kune iripo data seti, kuitira kuti gare gare, zvichibva pamitemo yakawanda yakaonekwa yemubatanidzwa, chero mhedziso inogona kutorwa nezve ruzivo rwakachengetwa. Ndizvo chaizvo zvinotsamira izvi zvatinoshanda nazvo. Izvo zvinobatwa nemunda wese wekuchera data neakasiyana siyana ekutsvaga matekiniki uye algorithms akavakirwa pahwaro hwavo. Ngationei kuti zvakawanikwa zvinoshanda sei kutsamira (chaiyo kana fungidziro) mune chero data inogona kubatsira.

Nhasi, imwe yemashandisirwo makuru ekutsamira ndeyekuchenesa data. Zvinosanganisira kugadzira maitiro ekuona "data rakasviba" wobva wagadzirisa. Mienzaniso yakakurumbira ye "data rakasviba" inodzokororwa, zvikanganiso zve data kana typos, kushaya kukosha, data yechinyakare, dzimwe nzvimbo, nezvimwe.

Muenzaniso wekukanganisa kwedata:

Muenzaniso wezvakapetwa mune data:

Semuenzaniso, isu tine tafura uye seti yemitemo yemubatanidzwa inofanirwa kuitwa. Kucheneswa kwedata mune iyi kesi kunosanganisira kushandura iyo data kuti iyo Federal Laws ive yakarurama. Muchiitiko ichi, nhamba yekugadziridza inofanira kunge iri shoma (iyi nzira ine algorithms yayo, yatisingazotarise pane ino chinyorwa). Pazasi pane muenzaniso weiyo data shanduko. Kuruboshwe ndiko hukama hwepakutanga, umo, zviri pachena, maFL anodiwa haasati asangana (muenzaniso wekuputsa kweimwe yeFLs inoratidzirwa mutsvuku). Kurudyi ndiko hukama hwakagadziridzwa, nemaseru akasvibira anoratidza maitiro akachinja. Mushure mekuita uku, kutsamira kwaidiwa kwakatanga kuchengetedzwa.

Imwe yakakurumbira application ndeye dhizaini dhizaini. Pano zvakakodzera kuyeuka zvakajairika mafomu uye normalization. Normalization inzira yekuunza hukama mukuenderana neimwe seti yezvinodiwa, imwe neimwe inotsanangurwa neyakajairwa fomu nenzira yayo. Hatisi kuzotsanangura zvinodiwa zveakasiyana siyana mafomu (izvi zvinoitwa mune chero bhuku pane dhatabhesi kosi yevanotanga), asi isu tinongoona chete kuti mumwe nemumwe wavo anoshandisa pfungwa yekushanda kutsamira munzira yayo. Mushure mezvose, maFLs ndiwo masikirwo ekutendeseka zvipingamupinyi zvinotariswa pakugadzira dhatabhesi (mumamiriro ezvinhu ebasa iri, maFLs dzimwe nguva anodaidzwa kuti superkeys).

Ngatitarisei mashandisiro avo emafomu mana akajairwa pamufananidzo uri pazasi. Rangarira kuti Boyce-Codd yakajairika fomu yakanyanya kuomarara kupfuura yechitatu fomu, asi yakaderera pane yechina. Hatisi kufunga nezvekupedzisira ikozvino, sezvo kuumbwa kwayo kunoda kunzwisisa kwezvakasiyana-siyana zvinotsamira, izvo zvisingafadzi kwatiri munyaya ino.

Imwe nharaunda umo vanotsamira vakawana mashandisiro avo kudzikisa humiro hwenzvimbo yenzvimbo mumabasa akadai sekuvaka isina ruzivo Bayes classifier, kuona akakosha maficha, uye kudzokorora modhi yekudzoreredza. Muzvinyorwa zvepakutanga, basa iri rinonzi kugadzwa kwezvisina basa uye zvine chekuita nemamiriro ezvinhu [5, 6], uye inogadziriswa nekushandiswa kwekushandisa kwedatabase concepts. Nekuuya kwemabasa akadaro, tinogona kutaura kuti nhasi kune kudiwa kwemhinduro dzinotibvumira kuti tibatanidze dhetabhesi, analytics uye kushandiswa kwematambudziko ekugadzirisa pamusoro apa mune chimwe chinhu [7, 8, 9].

Kune akawanda algorithms (zvese azvino uye asiri emazuva ano) ekutsvaga mitemo yemubatanidzwa mune data set.Matanho akadaro anogona kukamurwa kuita mapoka matatu:

• Algorithms uchishandisa traversal yealgebraic lattices (Lattice traversal algorithms)
• Algorithms yakavakirwa pakutsvaga kweakabvumirana kukosha (Musiyano- uye kubvumirana-seti algorithms)
• Algorithms yakavakirwa pakuenzanisa paviri (Dependency induction algorithms)

Tsanangudzo pfupi yemhando yega yega algorithm inoratidzwa mutafura iri pazasi:


Unogona kuverenga zvakawanda nezve kupatsanurwa uku [4]. Pazasi pane mienzaniso yealgorithms yemhando yega yega:

Parizvino, maalgorithms matsva ari kuoneka anosanganisa nzira dzinoverengeka dzekutsvaga mashandiro anoshanda. Mienzaniso yealgorithms yakadai iPyro [2] neHyFD [3]. Kuongororwa kwebasa ravo kunotarisirwa munyaya dzinotevera dzeiyi nhevedzano. Muchinyorwa chino tichaongorora chete iwo ekutanga pfungwa uye lemma inodiwa kuti tinzwisise kutsamira nzira dzekuona.

Ngatitange nei nyore - musiyano- uye kubvumirana-seti, inoshandiswa mumhando yechipiri yealgorithms. Difference-set is a set of tuples dzisina hunhu hwakafanana, ukuwo agree-set, pane kudaro, matuples ane hunhu hwakafanana. Zvakakosha kuziva kuti munyaya iyi tiri kutarisa chete rutivi rworuboshwe rwekutsamira.

Imwe pfungwa yakakosha yakasangana pamusoro apa ndeye algebraic lattice. Sezvo akawanda algorithms emazuva ano anoshanda pane iyi pfungwa, isu tinofanirwa kuve neruzivo rwekuti chii.

Kuti uzivise pfungwa yelatisi, zvinodikanwa kutsanangura seti yakarongedzerwa zvishoma (kana chikamu chakarongedzerwa seti, yakapfupikiswa seti).

Tsanangudzo 2. Seti S inonzi yakarongedzerwa zvishoma nehukama hwebhinari ⩽ kana kune ese a, b, c ∈ S zvinotevera zvivakwa zvagutsikana:

1. Reflexivity, kureva, a ⩽ a
2. Antisymmetry, kureva, kana a ⩽ b na b ⩽ a, ipapo a = b
3. Transitivity, kureva kuti ⩽ b na b ⩽ c zvinotevera kuti a ⩽ c

Hukama hwakadaro hunonzi (loose) partial order relation, uye iyo seti pachayo inodaidzwa kuti chikamu chakarairwa seti. Chinyorwa chepamutemo: ⟨S, ⩽⟩.

Semuenzaniso wakapfava weseti yakarongedzerwa zvishoma, tinogona kutora seti yenhamba dzese dzechisikigo N neyakajairwa kurongeka hukama ⩽. Zviri nyore kuona kuti ese anodiwa axioms anogutsikana.

Muenzaniso unoreva zvakawanda. Funga nezveseti yeese maseti {1, 2, 3}, akarairwa nekubatanidza hukama ⊆. Chokwadi, hukama uhwu hunogutsa mamiriro ese ekurongeka, saka ⟨P ({1, 2, 3}), ⊆⟩ iri seti yakarongedzerwa zvishoma. Mufananidzo uri pazasi unoratidza chimiro cheseti iyi: kana chimwe chinhu chinogona kusvikwa nemiseve kune chimwe chinhu, ivo vari muhukama hwehurongwa.

Tichada mamwe maviri akareruka tsananguro kubva kumunda wemasvomhu - supremum uye infimum.

Tsanangudzo 3. Rega ⟨S, ⩽⟩ ive seti yakarongedzerwa zvishoma, A ⊆ S. Mucheto wepamusoro weA chinhu u ∈ S zvekuti ∀x ∈ S: x ⩽ u. Rega U ive seti yezvose zvekumusoro miganhu yeS. Kana paine chinhu chidiki muU, chinodaidzwa kuti supremum uye chinodudzwa sup A.

Mafungiro emupendero wakaderera chaiwo anounzwa zvakafanana.

Tsanangudzo 4. Rega ⟨S, ⩽⟩ ive yakarongeka zvishoma, A ⊆ S. Infimum yeA chinhu l ∈ S zvekuti ∀x ∈ S: l ⩽ x. Rega L ive seti yezvese zviganho zvepasi zveS. Kana paine chinhu chikuru muL, chinodaidzwa kunzi infimum uye chinodudzwa seinf A.

Funga semuenzaniso iyo iri pamusoro yakarongedzerwa zvishoma seti ⟨P ({1, 2, 3}), ⊆⟩ uye tsvaga supremum uye infimum mairi:

Iye zvino isu tinokwanisa kugadzira tsananguro yealgebraic lattice.

Tsanangudzo 5. Rega ⟨P, ⩽⟩ ive yakarongedzerwa zvishoma zvekuti chero maviri-echimwe chinhu chikamu chidiki chine chepamusoro nechepazasi. Ipapo P inodaidzwa kuti algebraic lattice. Panyaya iyi, sup{x, y} inonyorwa se x ∨ y, uye inf {x, y} sa x ∧ y.

Ngatitarisei kuti muenzaniso wedu wekushanda ⟨P ({1, 2, 3}), ⊆⟩ iretisi. Chokwadi, chero a, b ∈ P ({1, 2, 3}), a∨b = a∪b, uye a∧b = a∩b. Semuenzaniso, funga nezveseti {1, 2} uye {1, 3} uye uwane infimum yavo uye supremum. Kana tikaapesanisa, tinowana iyo seti {1}, inova iyo infimum. Isu tinowana iyo supremum nekuisanganisa - {1, 2, 3}.

Mune maalgorithms ekuona matambudziko emuviri, nzvimbo yekutsvaga inowanzomiririrwa nenzira yelatisi, uko seti dzechimwe chinhu (verenga yekutanga nhanho yereti yekutsvaga, uko kuruboshwe rwekutsamira kunosanganisira chimwe hunhu) inomiririra hunhu hwega hwega. yehukama hwepakutanga.
Kutanga, isu tinotarisa zvinoenderana nefomu ∅ → Single hunhu. Danho iri rinokutendera kuti uone kuti ndeapi hunhu ari makiyi ekutanga (kune hunhu hwakadaro hapana zvinomisikidza, uye saka rutivi rwekuruboshwe haruna chinhu). Kupfuurirazve, maalgorithms akadaro anokwira kumusoro achitevedza lattice. Zvakakosha kuziva kuti haisi iyo latisi yose inogona kufambiswa, ndiko kuti, kana iyo yaidiwa yakakura saizi yekuruboshwe yakapfuudzwa kune iyo yekupinza, iyo algorithm haizoendi kupfuura nhanho ine saizi iyoyo.

Mufananidzo uri pasi apa unoratidza kuti algebraic lattice inogona kushandiswa sei mudambudziko rekutsvaga FZ. Pano muganhu wega wega (X, XY) inomiririra kutsamira X → Y. Semuenzaniso, isu takapfuura nhanho yekutanga uye tinoziva kuti kupindwa muropa kunochengetwa A → B (isu ticharatidza izvi segirini yekubatanidza pakati pema vertices A и B) Izvi zvinoreva kuti kuenderera mberi, kana isu tichikwira kumusoro nelatisi, isu tinogona kusatarisa kutsamira A, C → B, nokuti inenge isisina zvishoma. Saizvozvo, isu hatingazvitarise kana kutsamira kwaiitwa C → B.

Uye zvakare, semutemo, ese emazuva ano maalgorithms ekutsvaga mitemo yemubatanidzwa anoshandisa chimiro chedata senge kupatsanurwa (mune yekutanga sosi - yakabviswa partition [1]). Tsanangudzo yepamutemo yechikamu ndeiyi inotevera:

Tsanangudzo 6. Rega X ⊆ R ive seti yehunhu hwehukama r. Cluster seti yema indices e tuples mu r ane kukosha kwakafanana kwa X, kureva, c(t) = {i|ti[X] = t[X]}. A partition seti yemasumbu, kusasanganisa masumbu ehurefu hweyuniti:

Nemashoko akapfava, chidimbu chechimiro X seti yezvinyorwa, apo rondedzero yega yega ine nhamba dzemitsara dzine maitiro akafanana e X. Muzvinyorwa zvemazuva ano, chimiro chinomiririra zvikamu chinonzi position list index (PLI). Unit-kureba masumbu haabatanidzwe nekuda kwePLI compression zvinangwa nekuti iwo masumbu ane chete rekodhi nhamba ine yakasarudzika kukosha iyo inogara iri nyore kuziva.

Ngatitarisei muenzaniso. Ngatidzokere kutafura imwe chete nevarwere uye tivake zvikamu zvema column Mwoyo murefu и Zvepabonde (koramu itsva yaonekwa kuruboshwe, umo nhamba dzemutsara wetafura dzakanyorwa):

Uyezve, maererano netsanangudzo, chikamu chekoramu Mwoyo murefu ichave isina chinhu, sezvo zvikwata zvimwechete zvisingabatanidzwe kubva pakukamura.

Partitions inogona kuwanikwa nemaitiro akati wandei. Uye kune nzira mbiri dzekuita izvi: nekupinda netafura, kuvaka chikamu uchishandisa ese anodiwa hunhu kamwechete, kana kuivaka uchishandisa mashandiro ekupindirana kwezvikamu uchishandisa subset yehunhu. Federal law search algorithms shandisa sarudzo yechipiri.

Nemashoko akareruka, kuti, semuenzaniso, kuwana chidimbu nemakoramu ABC, unogona kutora zvikamu zve AC и B (kana chero imwe seti ye disjoint subsets) uye pindirana iwo nemumwe. Kushanda kwemharadzano yezvikamu zviviri zvinosarudza masumbu ehurefu hukuru anowanikwa kune ese ari maviri.

Ngatitarisei muenzaniso:

Muchiitiko chekutanga, takagamuchira chikamu chisina chinhu. Kana iwe ukanyatsotarisisa patafura, saka zvechokwadi, hapana akafanana maitiro ehuviri hunhu. Kana tikagadzirisa zvishoma tafura (nyaya iri kurudyi), isu tichatowana isina-isina mharadzano. Uyezve, mitsara 1 uye 2 inonyatso ine maitiro akafanana ehunhu Zvepabonde и Chiremba.

Tevere, isu tichada iyo pfungwa yakadai sehukuru hwekuparadzanisa. Formally:

Zvichitaurwa zviri nyore, saizi yekuparadzanisa ndiyo nhamba yemasumbu anosanganisirwa muchikamu (rangarira kuti masumbu mamwechete haana kuisirwa muchikamu!):

Zvino isu tinokwanisa kutsanangura imwe yemakiyi lemmas, ayo akapihwa zvikamu zvinotitendera kuona kuti kutsamira kunobatwa here kana kuti kwete:

Lemma 1. Kutsamira A, B → C kunobata kana uye chete kana

Zvinoenderana neremma, kuona kana kutsamira kunobata, matanho mana anofanira kuitwa:

1. Verenga chikamu cheruboshwe rwekutsamira
2. Verenga chikamu cherudyi rwekutsamira
3. Verenga chigadzirwa chekutanga uye chechipiri danho
4. Enzanisa saizi yezvikamu zvakawanikwa munhanho yekutanga neyechitatu

Pazasi pane muenzaniso wekutarisa kana kutsamira kunobata zvinoenderana neiyi lemma:

Muchinyorwa chino, takaongorora pfungwa dzakadai sekushanda kutsamira, fungidziro yekushanda, takatarisa paanoshandiswa, pamwe neapi maalgorithms ekutsvaga mabasa emuviri aripo. Isu takaongororawo zvakadzama pfungwa dzakakosha asi dzakakosha dzinoshandiswa zvakanyanya mualgorithms yemazuva ano yekutsvaga mitemo yemubatanidzwa.

References:

1. Huhtala Y. et al. TANE: Iyo inoshanda algorithm yekuwana inoshanda uye yakaringana kutsamira // Iyo komputa jenari. – 1999. – T. 42. – No. 2 - pp. 100-111.
2. Kruse S., Naumann F. Kuwanikwa kwakanyatsoita kweanenge anovimba // Maitiro eVLDB Endowment. – 2018. – T. 11. – Kwete. 7 - peji 759-772.
3. Papenbrock T., Naumann F. Nzira yakasanganiswa yekuwana kutsamira kwekushanda // Proceedings of the 2016 International Conference on Management of Data. – ACM, 2016. – p. 821-833.
4. Papenbrock T. et al. Kushanda kutsamira kuwanikwa: Ongororo yekuyedza yealgorithms manomwe // Proceedings yeVLDB Endowment. – 2015. – T. 8. – No. 10 - pp. 1082-1093.
5. Kumar A. et al. Kujoinha kana kusajoinha?: Kufunga kaviri nezvekujoinha kusati kwasarudzwa // Proceedings ye2016 International Conference on Management of Data. – ACM, 2016. – pp. 19-34.
6. Abo Khamis M. et al. Mu-database yekudzidza ine sparse tensor // Proceedings ye37th ACM SIGMOD-SIGACT-SIGAI Symposium pamusoro peMisimboti yeDatabase Systems. – ACM, 2018. – pp. 325-340.
7. Hellerstein JM et al. Iyo MADlib analytics raibhurari: kana MAD hunyanzvi, iyo SQL // Proceedings yeVLDB Endowment. – 2012. – T. 5. – No. 12 - mapeji 1700-1711.
8. Qin C., Rusu F. Kufungidzira kufungidzira kwe terascale yakagoverwa gradient descent optimization // Proceedings of the Fourth Workshop on Data analytics in the Cloud. – ACM, 2015. – P. 1.
9. Meng X. nevamwe. Mllib: Kudzidza kwemuchina mune apache spark // Iyo Chinyorwa cheMuchina Kudzidza Tsvagiridzo. – 2016. – T. 17. – No. 1 - mapeji 1235-1241.

Vanyori vezvinyorwa: Anastasia Birillo, muongorori pa JetBrains Research, CS centre mudzidzi и Nikita Bobrov, muongorori pa JetBrains Research

Source: www.habr.com