Replication ha se bekapo. Kapa eseng joalo? Mona ke kamoo re sebelisitseng replication e botsoa bakeng sa ho hlaphoheloa ka ho hlakola likhutšoane ka phoso.
GitLab e ikarabella bakeng sa mosebetsi - mohlala o moholo ka ho fetisisa oa GitLab ka tlhaho. E na le basebelisi ba limilione tse 3 le merero e ka bang limilione tse 7, ke e 'ngoe ea libaka tse kholo tse bulehileng tsa SaaS tse nang le meaho e inehetseng. Ntle le sistimi ea database ea PostgreSQL, lisebelisoa tsa GitLab.com li ke ke tsa ea hole, le seo re sa se etseng bakeng sa mamello ea liphoso haeba ho ka ba le liphoso ha o ka lahleheloa ke data. Ha ho na monyetla oa hore koluoa e joalo e etsahale, empa re itokisitse hantle le ho boloka mekhoa e fapaneng ea ho boloka le ho pheta-pheta.
Replication ha se sesebelisoa sa hau sa polokelo ea polokelo (). Empa joale re tla bona mokhoa oa ho khutlisa kapele data e hlakotsoeng ka phoso u sebelisa katiso e botsoa: on mosebelisi bakeng sa morero le ho lahleheloa ke likhokahano le likopo tse kopantseng le mesebetsi.
Ka replica e liehileng, re ile ra fumana data ka lihora tse 1,5 feela. Bonang kamoo ho neng ho le kateng.
Ho hlaphoheloa ha nako ka nako ka PostgreSQL
PostgreSQL e na le ts'ebetso e hahelletsoeng e khutlisetsang boemo ba database sebakeng se itseng ka nako. E bitsoa (PITR) mme e sebelisa mekhoa e ts'oanang e bolokang setšoantšo sa morao-rao: ho qala ka setšoantšo se ka tšeptjoang sa sehlopha sohle sa database (backup ea motheo), re sebelisa letoto la liphetoho tsa boemo ho fihlela nako e itseng.
Ho sebelisa karolo ena bakeng sa bekapo e batang, re lula re etsa bekapo ea "database" ebe re e boloka sebakeng sa polokelo (litlaleho tsa GitLab li lula ho ). Re boetse re beha leihlo liphetoho tsa boemo ba database ka ho boloka tlaleho ea pele-pele (, WAL). 'Me ka sena sohle, re ka etsa PITR bakeng sa ho hlaphoheloa ha koluoa: re qala ka setšoantšo se nkiloeng pele ho phoso' me re sebelise liphetoho ho tloha ho WAL archive ho fihlela ho oa.
Ho lieha ho ikatisa ke eng?
Ho lieha ho pheta-pheta ke ts'ebeliso ea liphetoho tse tsoang ho WAL ka tieho. Ke hore, transaction e etsahetse ka hora X, empa e tla hlaha setšoantšong ka tieho d ka hora X + d.
Ho na le mekhoa e 'meli ea ho theha replica ea database ea 'mele ho PostgreSQL: ho khutlisa li-archive le ho pheta-pheta phetiso. , ha e le hantle e sebetsa joaloka PITR, empa e sa khaotse: re lula re ntša liphetoho ho tswa ho WAL archive le ho li sebelisa ho replica. A e lata WAL molapo ka kotloloho ho tsoa ho moamoheli oa database o holimo. Re khetha ho khutlisa ho tsoa polokelong - e bonolo ho e laola ebile e na le ts'ebetso e tloaelehileng, e sa saletseng morao ho sehlopha sa tlhahiso.
Mokhoa oa ho theha backup e liehang ho hlaphoheloa
e hlalositsoeng faeleng recovery.conf. Mohlala:
standby_mode = 'on'
restore_command = '/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-fetch -p 4 "%f" "%p"'
recovery_min_apply_delay = '8h'
recovery_target_timeline = 'latest'Ka litlhophiso tsena, re hlophisitse replica e liehileng e nang le ho khutlisa li-archive. E sebelisoa mona ho ntša likarolo tsa WAL (restore_command) ho tsoa polokelong, 'me liphetoho li tla sebelisoa kamora lihora tse robeli (recovery_min_apply_delay). Replica e tla shebella liphetoho tsa tatellano ea nako sebakeng sa polokelo, joalo ka ka lebaka la ho se sebetse hantle ha sehlopha (recovery_target_timeline).
С recovery_min_apply_delay o ka theha phetisetso ea latency, empa ho na le maraba a 'maloa a amanang le li-slots tsa ho ikatisa, maikutlo a chesang a spare, joalo-joalo. Sebaka sa polokelo ea WAL sea li qoba.
Parameter recovery_min_apply_delay e hlahile feela ho PostgreSQL 9.3. Liphetolelong tse fetileng, ho lieha ho pheta-pheta ho hloka motsoako oa (pg_xlog_replay_pause(), pg_xlog_replay_resume()) kapa u tšoare likarolo tsa WAL sebakeng sa polokelo nakong ea tieho.
PostgreSQL e e etsa joang?
Hoa thahasellisa ho bona hore na PostgreSQL e sebelisa ts'ebetso ea botsoa joang. Ha re shebeng . E bitsoa ho tloha bakeng sa ho kena ka 'ngoe ho tsoa ho WAL.
static bool
recoveryApplyDelay(XLogReaderState *record)
{
uint8 xact_info;
TimestampTz xtime;
long secs;
int microsecs;
/* nothing to do if no delay configured */
if (recovery_min_apply_delay <= 0)
return false;
/* no delay is applied on a database not yet consistent */
if (!reachedConsistency)
return false;
/*
* Is it a COMMIT record?
*
* We deliberately choose not to delay aborts since they have no effect on
* MVCC. We already allow replay of records that don't have a timestamp,
* so there is already opportunity for issues caused by early conflicts on
* standbys.
*/
if (XLogRecGetRmid(record) != RM_XACT_ID)
return false;
xact_info = XLogRecGetInfo(record) & XLOG_XACT_OPMASK;
if (xact_info != XLOG_XACT_COMMIT &&
xact_info != XLOG_XACT_COMMIT_PREPARED)
return false;
if (!getRecordTimestamp(record, &xtime))
return false;
recoveryDelayUntilTime =
TimestampTzPlusMilliseconds(xtime, recovery_min_apply_delay);
/*
* Exit without arming the latch if it's already past time to apply this
* record
*/
TimestampDifference(GetCurrentTimestamp(), recoveryDelayUntilTime,
&secs, µsecs);
if (secs <= 0 && microsecs <= 0)
return false;
while (true)
{
// Shortened:
// Use WaitLatch until we reached recoveryDelayUntilTime
// and then
break;
}
return true;
}Taba ea bohlokoa ke hore tieho e ipapisitse le nako ea 'mele e tlalehiloeng setempeng sa nako sa boitlamo ba transaction (xtime). Joalokaha u ka bona, ho lieha ho sebetsa feela ho boitlamo mme ha ho ame litlaleho tse ling - liphetoho tsohle li sebelisoa ka ho toba, 'me boitlamo bo lieha, kahoo re tla bona liphetoho feela ka mor'a ho lieha ho lokiselitsoeng.
Mokhoa oa ho sebelisa replica ea botsoa bakeng sa ho hlaphoheloa ha data
Ha re re re na le sehlopha sa polokelo ea litaba tlhahisong le setšoantšo se nang le tieho ea lihora tse robeli. Ha re boneng mokhoa oa ho khutlisa data ka mohlala .
Ha re hlokomela bothata, re bakeng sa replica e liehile:
SELECT pg_xlog_replay_pause();Ka khefu, re ne re se na kotsi ea hore setšoantšo seo se ka pheta kopo eo DELETE. Ntho ea bohlokoa haeba u hloka nako ea ho tseba tsohle.
Taba ea bohlokoa ke hore setšoantšo se liehileng se tlameha ho fihla pele ho kopo DELETE. Re ne re batla re tseba nako ea 'mele ea ho tlosoa. Re tlositse recovery_min_apply_delay le ho eketsa recovery_target_time в recovery.conf. Kahoo replica e fihla motsotsong o nepahetseng ntle le tieho:
recovery_target_time = '2018-10-12 09:25:00+00'Ka li-timestamps, ho molemo ho fokotsa ho feteletseng e le hore u se ke ua fetoa. Ke 'nete, ha ho fokotseha ho hoholo, re lahleheloa ke lintlha tse ngata. Hape, haeba re tlola kopo DELETE, ntho e 'ngoe le e' ngoe e tla hlakoloa hape 'me u tla tlameha ho qala bocha (kapa u nke bekapo e batang bakeng sa PITR).
Re qalile hape ketsahalo ea Postgres e liehileng mme likarolo tsa WAL li phetoa ho fihlela nako e behiloeng. O ka bona tsoelopele mothating ona ka ho botsa:
SELECT
-- current location in WAL
pg_last_xlog_replay_location(),
-- current transaction timestamp (state of the replica)
pg_last_xact_replay_timestamp(),
-- current physical time
now(),
-- the amount of time still to be applied until recovery_target_time has been reached
'2018-10-12 09:25:00+00'::timestamptz - pg_last_xact_replay_timestamp() as delay;Haeba setempe sa nako se sa fetohe, ho khutlisa ho felile. O ka etsa ts'ebetso ka mokhoa o ikhethileng ho koala, ho khothaletsa, kapa ho emisa ketsahalo ka mor'a ho leka hape (e emisa ka ho sa feleng).
Database e fihlile mmuso pele ho kopo eo e fosahetseng. Hona joale, ka mohlala, u ka romela data ka ntle. Re rometse data ea li-remote label le likhokahano tsohle tsa litaba le ho kopanya likopo le ho li fetisetsa polokelong ea tlhahiso. Haeba litahlehelo li le kholo, u ka khothaletsa setšoantšo feela 'me u se sebelise e le sona se ka sehloohong. Empa joale liphetoho tsohle li tla lahleha ka mor'a motsotso oo re ileng ra hlaphoheloa ho oona.
Ho molemo ho sebelisa li-ID tsa transaction ho fapana le litempe tsa linako. Ho bohlokoa ho hatisa li-ID tsena, mohlala, bakeng sa lipolelo tsa DDL (joalo ka DROP TABLE), ka ho sebelisa log_statements = 'ddl'. Haeba re ne re e-na le ID ea transaction, re ne re tla e nka recovery_target_xid mme o tsamaisitse tsohle ho theosa le thekiso pele ho kopo DELETE.
Ho khutlela mosebetsing ho bonolo haholo: tlosa liphetoho tsohle ho recovery.conf ebe o qala li-postgres hape. Haufinyane leseli le tla boela le be le tieho ea lihora tse robeli, 'me re itokiselitse mathata a tlang.
Melemo ea ho Pholosa
Ka replica e liehileng, sebakeng sa polokelo e batang, ha ua tlameha ho qeta lihora tse ngata u khutlisetsa setšoantšo sohle se bolokiloeng. Mohlala, re hloka lihora tse hlano ho fumana backup e felletseng ea 2 TB. 'Me joale u ntse u tlameha ho sebelisa WAL eohle ea letsatsi le letsatsi ho khutlela sebakeng seo u se batlang (maemong a mabe ka ho fetesisa).
Replica e liehileng e molemo ho feta ho boloka ho bata ka litsela tse peli:
- Ha ho hlokahale hore u fumane li-backups kaofela ho tsoa polokelong ea litaba.
- Ho na le fensetere e tsitsitseng ea lihora tse robeli ea likarolo tsa WAL tse lokelang ho phetoa.
Hape, re lula re lekola ho bona hore na WAL e ka ba PITRed, 'me re tla hlokomela kapele bobolu kapa mathata a mang ka polokelo ea WAL ka ho beha leihlo morao oa replica e liehileng.
Mohlala ona, ho re nkile metsotso ea 50 ho tsosolosa, ke hore, lebelo e ne e le 110 GB ea data ea WAL ka hora (sebaka sa polokelo ea litaba se ne se ntse se le teng. ). Ka kakaretso, re rarolotse bothata mme ra khutlisa data ka lihora tse 1,5.
Kakaretso: moo setšoantšo se liehileng se leng molemo (le moo se seng teng)
Sebelisa katiso e liehileng joalo ka thuso ea pele haeba u lahleheloa ke data ka phoso 'me u hlokomela koluoa ena nakong ea tieho e lokiselitsoeng.
Empa hopola: ho pheta-pheta ha se bekapo.
Ho boloka le ho pheta-pheta ho na le merero e fapaneng. Backup e batang e tla ba molemo haeba u entse ka phoso DELETE kapa DROP TABLE. Re etsa bekapo ho tsoa polokelong e batang ebe re khutlisetsa boemo bo fetileng ba tafole kapa database kaofela. Empa ka nako e tšoanang kopo DROP TABLE e batla e hlahisoe hang hang ho likopi tsohle tsa sehlopha se sebetsang, kahoo ho pheta-pheta khafetsa ho ke ke ha boloka mona. Replication ka boeona e boloka database e fumaneha ha li-server ka bomong li hirisoa 'me li aba mojaro.
Esita le ka replica e liehang, ka linako tse ling re hlile re hloka bekapo e batang sebakeng se sireletsehileng, haeba ka tšohanyetso ho na le ho hloleha ha setsi sa data, tšenyo e patiloeng, kapa liketsahalo tse ling tseo u sa li boneng hanghang. Mona ho tsoa ho replication e le 'ngoe ha ho na moelelo.
mantsoe. Tsoela pele ha joale re sireletsa feela khahlanong le tahlehelo ea data boemong ba sistimi 'me ha re khutlisetse data boemong ba mosebelisi.
Source: www.habr.com