🥇Ukugcinwa kwedatha okuqhubekayo kunye nee-API zefayile Linux

Ngelixa ndiphanda ukuzinza kokugcinwa kwedatha kwiinkqubo zamafu, ndagqiba ekubeni ndizivavanye ukuze ndiqinisekise ukuba ndiyaziqonda izinto ezisisiseko. I iqalwe ngokufunda iinkcukacha zeNVMe ukuze uqonde ukuba zeziphi iziqinisekiso malunga nokugcinwa kwedatha ezinzileyo (oko kukuthi, iqinisekisa ukuba idatha iya kufumaneka emva kokusilela kwenkqubo) kusinika iidiski ze-NMVe. Ndenze ezi zigqibo eziphambili zilandelayo: idatha kufuneka ithathelwe ingqalelo yonakele ukususela kumzuzu umyalelo wokubhala idatha unikezelwa kude kube ngumzuzu obhaliweyo kwindawo yokugcina. Nangona kunjalo, uninzi lweenkqubo ngolonwabo zisebenzisa iifowuni zenkqubo ukurekhoda idatha.

Kule nqaku, ndihlola iindlela zokugcina idatha eziqhubekayo ezibonelelwa zii-API zefayile. LinuxKubonakala ngathi yonke into apha kufuneka ibe lula: inkqubo ibiza umyalelo write(), kwaye emva kokuba lo myalelo ugqityiwe, idatha iya kugcinwa ngokukhuselekileyo kwidiski. Kodwa write() ikhuphela kuphela idatha yesicelo kwi-kernel cache ebekwe kwi-RAM. Ukuze unyanzelise inkqubo ukuba ibhale idatha kwidiski, kufuneka usebenzise ezinye iindlela ezongezelelweyo.

Ngokubanzi, le nkcazelo yingqokelela yamanqaku anxulumene noko ndikufundileyo ngesihloko esinomdla kum. Ukuba sithetha ngokufutshane kakhulu malunga neyona nto ibalulekileyo, kuvela ukuba ukulungelelanisa ukugcinwa kwedatha ezinzileyo kufuneka usebenzise umyalelo fdatasync() okanye uvule iifayile ngeflegi O_DSYNC. Ukuba unomdla wokufunda ngakumbi malunga nokuba kwenzeka ntoni kwidatha kwindlela yayo ukusuka kwikhowudi ukuya kwidiski, jonga oku inqaku.

Iimpawu zokusebenzisa bhala () umsebenzi

Umnxeba wenkqubo write() ichazwe kumgangatho IEEE POSIX njengenzame yokubhala idatha kwisichazi sefayile. Emva kokugqiba ngempumelelo write() Imisebenzi yokufunda idatha kufuneka ibuyisele kanye iibytes ebezibhalwe ngaphambili, ukwenza oku nokuba idatha ifunyenwe kwezinye iinkqubo okanye imisonto (bonani icandelo elifanelekileyo lomgangatho wePOSIX). kuyinto, kwicandelo malunga nendlela imisonto esebenzisana ngayo nemisebenzi yefayile eqhelekileyo, kukho inqaku elithi ukuba imisonto emibini nganye ibiza le misebenzi, ngoko umnxeba ngamnye kufuneka ubone nokuba zonke iziphumo ezichongiweyo zomnye umnxeba, okanye akukho nanye. iziphumo. Oku kukhokelela kwisigqibo sokuba yonke imisebenzi yefayile ye-I/O kufuneka ibambe isitshixo kwisixhobo abasebenza kuso.

Ngaba oku kuthetha ukuba utyando write() ngaba yiatom? Ukusuka kumbono wobugcisa, ewe. Ukufundwa kwedatha kufuneka kubuyise konke okanye akukho nto ibibhalwe ngayo write(). Kodwa utyando write(), ngokomgangatho, akuyomfuneko ukuba iphele ngokubhala yonke into ebicelwe ukuba ibhalwe. Uvumelekile ukuba abhale kuphela inxalenye yedatha. Umzekelo, sinokuba nemisonto emibini edibanisa i-1024 bytes nganye kwifayile echazwe yinkcazelo yefayile efanayo. Ukusuka kwimbono yomgangatho, umphumo owamkelekileyo uya kuba xa umsebenzi ngamnye wokubhala unokongeza i-byte enye kuphela kwifayile. Le misebenzi iya kuhlala i-athomu, kodwa emva kokuba igqityiwe, idatha abayibhale kwifayile iya kuxutywa. Apha Ingxoxo enika umdla kakhulu kwesi sihloko sokuPhuphuma kweSitaki.

fsync () kunye fdatasync () imisebenzi

Eyona ndlela ilula yokugungxula idatha kwidiski kukufowunela umsebenzi fsync(). Lo msebenzi ucela inkqubo yokusebenza ukuba idlulise zonke iibhloko ezilungisiweyo ukusuka kwi-cache ukuya kwidiski. Oku kubandakanya yonke imethadatha yefayile (ixesha lokufikelela, ixesha lokuguqulwa kwefayile, njalo njalo). Ndiyakholelwa ukuba le metadata ayifane idingeke, ke ukuba uyazi ukuba ayibalulekanga kuwe, ungasebenzisa umsebenzi. fdatasync(). Ku Nceda kwi fdatasync() Kuthiwa ngexesha lokusebenza kwalo msebenzi, isixa esinjalo semetadata sigcinwa kwidiski "eyimfuneko ukuze kuqhutywe ngokuchanekileyo imisebenzi elandelayo yokufunda idatha." Kwaye le yeyona nto kanye uninzi lwezicelo ezikhathalelayo.

Enye ingxaki enokuthi ivele apha kukuba ezi ndlela aziqinisekisi ukuba ifayile iya kufumaneka emva kokusilela okunokwenzeka. Ngokukodwa, xa udala ifayile entsha, kufuneka ufowunele fsync() kuluhlu oluqulathe yona. Ngaphandle koko, emva kokusilela, kunokuvela ukuba le fayile ayikho. Isizathu soku kukuba kwi UNIX, ngenxa yosebenziso lwamakhonkco anzima, ifayile ingabakho kubalawuli abaninzi. Ngoko ke, xa ufowuna fsync() akukho ndlela yokuba ifayile yazi ukuba loluphi ulawulo lwedatha ekufuneka lugungxulwe kwidisk (apha Unokufunda ngakumbi malunga noku). Kubonakala ngathi inkqubo yefayile ye-ext4 iyakwazi ukwenza ngokuzenzekelayo faka isicelo fsync() kubalawuli abaqulathe iifayile ezihambelanayo, kodwa oku kusenokungabi njalo kwezinye iindlela zefayile.

Le ndlela inokusetyenziswa ngokwahlukileyo kwiindlela ezahlukeneyo zefayile. ndidla ngoku blktrace Ukufunda malunga nokuba yeyiphi imisebenzi yediski esetyenziswayo kwiinkqubo zefayile ze ext4 kunye ne XFS. Zombini zikhupha imiyalelo yokubhala rhoqo kwidisk kuzo zombini iziqulatho zefayile kunye nejenali yenkqubo yefayile, gungxula i-cache, kwaye uphume ngokwenza iFUA (Force Unit Access, ukubhala idatha ngokuthe ngqo kwidisk, ngokudlula i-cache) bhala kwijenali. Mhlawumbi benza oku ukuze baqinisekise ukuba intengiselwano yenzekile. Kwiidrive ezingaxhasi i-FUA, oku kubangela ukugungxulwa kwecache kabini. Imifuniselo yam yabonisa ukuba fdatasync() ngokukhawuleza kancinci fsync(). Uncedo blktrace ibonisa ukuba fdatasync() ihlala ibhala idatha encinci kwidiski (kwi-ext4 fsync() ubhala 20 KB, kwaye fdatasync() - 16 KB). Kwakhona, ndiye ndafumanisa ukuba i-XFS ikhawuleza kancinci kune-ext4. Kwaye apha ngoncedo blktrace ikwazile ukufumanisa ukuba fdatasync() igungxula idatha encinci kwidisk (4 KB kwi XFS).

Iimeko ezingaqondakaliyo ezivelayo xa usebenzisa i-fsync()

Ndiyakwazi ukucinga ngeemeko ezintathu ezingaqondakaliyo malunga fsync()endidibene nayo xa ndisenza.

Ityala lokuqala elinjalo lenzeka ngo-2008. Emva koko ujongano lweFirefox 3 luye lwaba ngumkhenkce ukuba inani elikhulu leefayile zibhalelwe kwidisk. Ingxaki yayikukuba ukuphunyezwa kwe-interface kusetyenziswa i-database ye-SQLite ukugcina ulwazi malunga nesimo sayo. Emva kwenguqu nganye eyenzekayo kwi-interface, umsebenzi wabizwa fsync(), eyanika iziqinisekiso ezilungileyo zokugcina idatha ezinzileyo. Kwinkqubo yefayile ye-ext3 esetyenzisiweyo, umsebenzi fsync() ulahle onke amaphepha "amdaka" kwisixokelelwano kwidisk, kwaye hayi nje lawo ayenxulumene nefayile ehambelanayo. Oku kuthetha ukuba ukucofa iqhosha kwiFirefox kunokubangela iimegabytes zedatha ukuba ibhalwe kwidisk magnetic, enokuthatha imizuzwana emininzi. Isisombululo kwingxaki, ngokokude ndiqonde oku imathiriyeli yayikukudlulisela umsebenzi kunye nesiseko sedatha kwimisebenzi yangasemva engahambelaniyo. Oku kuthetha ukuba ngaphambili iFirefox iphumeze iimfuno ezingqongqo zogcino kunokuba ibifuneka ngokwenene, kwaye iimpawu zenkqubo yefayile ye-ext3 ziye zayenza mandundu le ngxaki.

Ingxaki yesibini yenzeka ngo-2009. Emva koko, emva kokuwa kwenkqubo, abasebenzisi benkqubo entsha yefayile ye-ext4 baye bajongana nenyaniso yokuba iifayile ezininzi ezisandul 'ukwenziwa zazinobude be-zero, kodwa oku akuzange kwenzeke ngenkqubo yefayile ye-ext3 endala. Kumhlathi odlulileyo, ndithethe malunga nendlela i-ext3 egungxula ngayo idatha eninzi kwidiski, eyanciphisa izinto kakhulu. fsync(). Ukuphucula imeko, kwi-ext4 kuphela loo maphepha amdaka ahambelana nefayile ethile agungxulwa kwidiski. Kwaye idatha evela kwezinye iifayile ihlala kwinkumbulo ixesha elide kakhulu kune ext3. Oku kwenziwa ukuphucula ukusebenza (ngokungagqibekanga, idatha ihlala kwesi simo kangangemizuzwana engama-30, ungaqwalasela oku usebenzisa ezimdaka_expire_centisecs; apha Ungafumana imathiriyeli eyongezelelweyo malunga noku). Oku kuthetha ukuba isixa esikhulu sedatha singalahleka ngokungenakubuyiseka emva kokusilela. Isisombululo kule ngxaki kukusebenzisa fsync() kwizicelo ezifuna ukuqinisekisa ukugcinwa kwedatha okuzinzileyo kunye nokukhusela kangangoko kunokwenzeka kwimiphumo yokungaphumeleli. Umsebenzi fsync() isebenza kakuhle kakhulu xa usebenzisa i-ext4 kunaxa usebenzisa i-ext3. Ukungalungi kwale ndlela kukuba ukusetyenziswa kwayo, njengangaphambili, kunciphisa ukuphunyezwa kweminye imisebenzi, njengokufakela iinkqubo. Bona iinkcukacha malunga noku apha и apha.

Ingxaki yesithathu malunga fsync(), yaqala ngo-2018. Emva koko, ngaphakathi kwesakhelo seprojekthi yePostgreSQL, kwafunyaniswa ukuba ukuba umsebenzi fsync() idibana nempazamo, iphawula amaphepha "amdaka" njenge "clean". Ngenxa yoko, le minxeba ilandelayo fsync() Ayenzi nto ngamaphepha anjalo. Ngenxa yoku, amaphepha alungisiweyo agcinwa kwinkumbulo kwaye akaze abhalwe kwidiski. Le yintlekele yokwenene, ekubeni isicelo siya kucinga ukuba idatha ethile ibhaliwe kwidiski, kodwa ngokwenene ayiyi kuba. Ukusilela okunjalo fsync() zinqabile, isicelo kwiimeko ezinjalo asinakwenza phantse nto ukulwa ingxaki. Kwezi ntsuku, xa oku kusenzeka, iPostgreSQL kunye nezinye izicelo ziyawa. kuyinto, kwizinto eziphathekayo "Ngaba ii-Applications ziyakwazi ukuBuyisa kwiiFayile ze-fsync?", Le ngxaki iphononongwa ngokubanzi. Okwangoku esona sisombululo kule ngxaki kukusebenzisa i-I/O eNgqo neflegi O_SYNC okanye ngendwe O_DSYNC. Ngale ndlela, inkqubo iya kuxela iimpazamo ezinokuthi zenzeke ngexesha lomsebenzi othile wokubhala, kodwa le ndlela ifuna ukuba isicelo silawule izithinteli ngokwazo. Funda ngakumbi ngale nto apha и apha.

Ukuvula iifayile kusetyenziswa iiflegi ze-O_SYNC kunye ne-O_DSYNC

Masibuyele kwingxoxo yeendlela Linux, ukuqinisekisa ukugcinwa kwedatha okuzinzileyo. Ngokukodwa, sithetha ngokusetyenziswa kweflegi O_SYNC okanye iflegi O_DSYNC xa uvula iifayile usebenzisa inkqubo yokufowuna vula (). Ngale ndlela, umsebenzi ngamnye wokubhala idatha wenziwa ngokungathi emva komyalelo ngamnye write() inkqubo inikwa imiyalelo ngokufanelekileyo fsync() и fdatasync(). Ku Iimpawu zePOSIX oku kubizwa ngokuba yi "Ngcaciso yeFayile ye-I/O eDityanisiweyo" kunye "nokuQeda uMfezeko weDatha". Inzuzo ephambili yale ndlela kukuba ukuqinisekisa ukuthembeka kwedatha, kufuneka wenze umnxeba omnye wenkqubo, kunokuba ezimbini (umzekelo - write() и fdatasync()). Ukungalungi okuphambili kwale ndlela kukuba zonke zibhala usebenzisa i-descriptor yefayile ehambelanayo iya kulungelelaniswa, enokunciphisa ukukwazi ukucwangcisa ikhowudi yesicelo.

Ukusebenzisa i-I/O eNgqo kunye ne-O_DIRECT iflegi

Umnxeba wenkqubo open() ixhasa iflegi O_DIRECT, eyenzelwe ukugqitha i-cache yenkqubo yokusebenza ukwenza imisebenzi ye-I/O ngokunxibelelana ngqo nediski. Oku, kwiimeko ezininzi, kuthetha ukuba imiyalelo yokubhala ekhutshwe yiprogram iya kuguqulelwa ngokuthe ngqo kwimiyalelo ejoliswe ekusebenzeni ngediski. Kodwa, ngokubanzi, le sixhobo asiyondawo yemisebenzi fsync() okanye fdatasync(). Inyaniso kukuba idiski ngokwayo inako defer okanye cache imiyalelo ehambelanayo yokubhala idatha. Kwaye, ukwenza izinto zibe mbi ngakumbi, kwezinye iimeko ezikhethekileyo imisebenzi ye-I / O eyenziwa xa kusetyenziswa iflegi O_DIRECT, usasazo kwimisebenzi yemveli ekhuselweyo. Eyona ndlela ilula yokusombulula le ngxaki kukusebenzisa iflegi ukuvula iifayile O_DSYNC, okuya kuthetha ukuba umsebenzi ngamnye wokubhala uya kulandelwa yifowuni fdatasync().

Kuye kwavela ukuba inkqubo yefayile ye-XFS isandula ukongeza "indlela ekhawulezayo" ye O_DIRECT|O_DSYNC-ukurekhoda idatha. Ukuba ibhloko iphinda ibhalwe kusetyenziswa O_DIRECT|O_DSYNC, ngoko XFS, endaweni yokugungxula indawo efihlakeleyo, izakwenza i FUA bhala umyalelo ukuba isixhobo siyayixhasa. Ndiyiqinisekisile le nto ngokusebenzisa usetyenziso blktrace kwinkqubo Linux 5.4 /Ubuntu 20.04. Le ndlela ifanele isebenze ngakumbi, kuba ibhala idatha encinci kwidiski kwaye isebenzisa umsebenzi omnye endaweni yezimbini (ukubhala nokucoca i-cache). Ndifumene ikhonkco eliya ku isiqaqa I-kernel ye-2018, esebenzisa le ndlela. Kukho ingxoxo apho malunga nokusebenzisa olu lungiselelo kwezinye iinkqubo zefayile, kodwa ngokokwazi kwam, i-XFS kuphela kwendlela yefayile exhasa oku ukuza kuthi ga ngoku.

sync_file_range () umsebenzi

В Linux kukho umnxeba wenkqubo sync_file_uluhlu (), ekuvumela ukuba ugungxule kuphela inxalenye yefayile kwidiski, kuneyo yonke ifayile. Le mnxeba iqalisa ukugungxulwa kwedatha engahambelaniyo kwaye ayilindi ukuba igqitywe. Kodwa kwisatifikethi sync_file_range() iqela kuthiwa "liyingozi kakhulu". Akukhuthazwa ukuyisebenzisa. Iimpawu kunye neengozi sync_file_range() ichazwe kakuhle kakhulu oku impahla. Ngokukodwa, le fowuni ibonakala isebenzisa i-RocksDB ukulawula xa i-kernel iphosa idatha engcolileyo kwidiski. Kodwa kwangaxeshanye, ukuqinisekisa ukugcinwa kwedatha okuzinzileyo, kuyasetyenziswa kwakhona fdatasync(). Ku ikhowudi I-RocksDB inezimvo ezinomdla kwesi sihloko. Umzekelo, kubonakala ngathi ifowuni sync_file_range() Xa usebenzisa i-ZFS, ayigungxuli idatha kwidiski. Amava andixelela ukuba ikhowudi engafane isetyenziswe kusenokwenzeka ukuba iqulathe iibugs. Ke ngoko, ndingacebisa ngokuchasene nokusebenzisa le ndlela yokufowuna ngaphandle kokuba kuyimfuneko.

Iifowuni zeSistim ezinceda ukuqinisekisa ukuzingisa kwedatha

Ndiye ndafikelela kwisigqibo sokuba kukho iindlela ezintathu ezinokuthi zisetyenziswe ukwenza imisebenzi ye-I / O eqinisekisa ukuzingisa kwedatha. Zonke zifuna umnxeba womsebenzi fsync() kulawulo apho ifayile yenziwe khona. Ezi ziindlela:

Ukufowunela umsebenzi fdatasync() okanye fsync() emva komsebenzi write() (Kungcono ukusebenzisa fdatasync()).
Ukusebenza ngesichazi sefayile sivulwe ngeflegi O_DSYNC okanye O_SYNC (ngcono - ngeflegi O_DSYNC).
Ukusetyenziswa komyalelo pwritev2() ngeflegi RWF_DSYNC okanye RWF_SYNC (ngokukhethekileyo ngeflegi RWF_DSYNC).

Amanqaku oMsebenzi

Andikhange ndilinganise ngononophelo ukusebenza kweendlela ezahlukeneyo endizivavanyileyo. Umahluko endiwuqapheleyo kwisantya somsebenzi wabo mncinane kakhulu. Oku kuthetha ukuba ndinokuba ndiphosakele, kwaye phantsi kweemeko ezahlukeneyo into enye inokuvelisa iziphumo ezahlukeneyo. Okokuqala, ndiza kuthetha ngento echaphazela ukusebenza ngakumbi, kwaye emva koko yintoni echaphazela ukusebenza kancinci.

Idatha yefayile yokubhala ngaphezulu ikhawuleza kunokufaka idatha kwifayile (inzuzo yokusebenza ingaba yi-2-100%). Ukufaka idatha kwifayile kufuna utshintsho olongezelelweyo kwimethadatha yefayile, nasemva komnxeba wenkqubo fallocate(), kodwa ubukhulu besi siphumo bunokwahluka. Ndincoma, ekusebenzeni kakuhle, ukubiza fallocate() ukwaba kwangaphambili indawo efunekayo. Emva koko esi sithuba kufuneka sigcwaliswe ngokucacileyo ngo-zero kwaye sibizwe fsync(). Oku kuya kuqinisekisa ukuba iibhloko ezingqinelanayo kwisistim yeefayili ziphawulwa njenge-"allocated" endaweni yokuba "zingabiwanga". Oku kunika kancinci (malunga ne-2%) ukuphuculwa komsebenzi. Ukongeza, ezinye iidiski zinokuba nofikelelo olucothayo lokuqala kwibhloko kunezinye. Oku kuthetha ukuba ukuzalisa indawo nge-zero kunokukhokelela ekuphuculeni okuphawulekayo (malunga ne-100%) ekusebenzeni. Ngokukodwa, oku kunokwenzeka ngeediski AWS EBS (Le yidatha engekho semthethweni, andikwazanga ukuyiqinisekisa). Okufanayo kuya kwindawo yokugcina IDiski eZingisayo ye-GCP (kwaye olu sele lulwazi olusemthethweni, olungqinisiswe luvavanyo). Nezinye iingcali zenze okufanayo ukujonga, ezinxulumene neediski ezahlukeneyo.
Iifowuni ezincinci zenkqubo, iphezulu ukusebenza (inzuzo ingaba malunga ne-5%). Kubonakala ngathi lucelomngeni open() ngeflegi O_DSYNC okanye ukufowuna pwritev2() ngeflegi RWF_SYNC ngokukhawuleza kunomnxeba fdatasync(). Ndiyakrokrela ukuba ingongoma apha kukuba le ndlela idlala indima kwinto yokuba iifowuni ezimbalwa zenkqubo kufuneka zenziwe ukusombulula ingxaki efanayo (umnxeba omnye endaweni yezimbini). Kodwa umahluko ekusebenzeni kuncinci kakhulu, ngoko unokungawuhoyi ngokupheleleyo kwaye usebenzise into kwisicelo engayi kuyenza nzima ingqiqo yayo.

Ukuba unomdla kwisihloko sokugcinwa kwedatha ezinzileyo, nazi izinto eziluncedo:

I/O Iindlela zokufikelela — amagqabantshintshi eziseko zeendlela zegalelo/imveliso.
Ukuqinisekisa ukuba idatha ifikelela kwidiski - ibali malunga nokuba kwenzekani kwidatha endleleni ukusuka kwisicelo ukuya kwidiski.
Kufuneka ufsync nini uvimba weefayili - impendulo kumbuzo wokuba usetyenziswe nini fsync() kwiincwadi zeefayile. Ngamafutshane, kuvela ukuba oku kufanele kwenziwe xa kudalwa ifayile entsha, kwaye isizathu sale ngcebiso kukuba Linux Kusenokuba kukho amakhonkco amaninzi kwifayile enye.
Iseva ye-SQL ivuliwe Linux: Abasebenzi bangaphakathi be-FUA — nantsi inkcazo yendlela yokugcina idatha engapheliyo esetyenziswa ngayo kwi-SQL Server eqongeni LinuxKukho uthelekiso olunomdla phakathi kweefowuni zenkqubo apha. Windows и LinuxNdiqinisekile ukuba kungenxa yale nto ndifunde ngayo malunga nokulungiswa kwe-FUA ye-XFS.

Ngaba ulahlekelwe idatha ocinga ukuba igcinwe ngokukhuselekileyo kwidiski?

umthombo: www.habr.com