DBA: muaj peev xwm npaj synchronizations thiab imports

Rau kev ua haujlwm nyuaj ntawm cov ntaub ntawv loj (txawv Cov txheej txheem ETL: imports, hloov dua siab tshiab thiab synchronization nrog ib qho chaw sab nraud) feem ntau xav tau ib ntus "nco" thiab ua sai sai ib yam dab tsi voluminous.

Ib txoj hauj lwm ntawm hom no feem ntau yog ib yam zoo li no: "Tam sim no accounting department unloaded los ntawm tus neeg muas zaub bank qhov kawg tau txais cov nyiaj them poob haujlwm, koj yuav tsum tau xa lawv sai sai rau lub vev xaib thiab txuas lawv rau koj tus as-qhauj "

Tab sis thaum lub ntim ntawm "ib yam dab tsi" no pib ntsuas ntau pua megabytes, thiab cov kev pabcuam yuav tsum txuas ntxiv ua haujlwm nrog cov ntaub ntawv 24x7, ntau yam teebmeem tshwm sim uas yuav ua rau koj lub neej puas.
DBA: muaj peev xwm npaj synchronizations thiab imports
Txhawm rau nrog lawv hauv PostgreSQL (thiab tsis yog hauv nws), koj tuaj yeem siv qee qhov kev ua kom zoo tshaj plaws uas yuav tso cai rau koj ua txhua yam sai dua thiab siv cov peev txheej tsawg.

1. Lub nkoj nyob qhov twg?

Ua ntej, cia peb txiav txim siab qhov twg peb tuaj yeem xa cov ntaub ntawv uas peb xav tau "kev ua haujlwm."

1.1. Temporary TABLE (Temporary TABLE)

Hauv txoj cai, rau PostgreSQL cov lus ib ntus yog tib yam li lwm yam. Yog li ntawd, superstitions nyiam "Txhua yam muaj tsuas yog khaws cia hauv lub cim xeeb, thiab nws tuaj yeem xaus". Tab sis kuj tseem muaj ntau qhov sib txawv tseem ceeb.

Koj tus kheej "namespace" rau txhua qhov kev sib txuas rau cov ntaub ntawv

Yog tias ob qho kev sib txuas sim txuas rau tib lub sijhawm CREATE TABLE x, ces ib tug yuav twv yuav raug hu tau kev tsis sib xws database khoom.

Tab sis yog tias ob leeg sim ua kom tiav CREATE TEMPORARY TABLE x, ces ob leeg yuav ua nws ib txwm, thiab txhua leej txhua tus yuav tau txais koj daim ntawv rooj. Thiab yuav tsis muaj dab tsi zoo sib xws ntawm lawv.

"Self-destruct" thaum disconnecting

Thaum qhov kev sib txuas raug kaw, tag nrho cov rooj zaum ib ntus raug rho tawm, yog li manually DROP TABLE x tsis muaj point tsuas yog...

Yog tias koj ua haujlwm dhau pgbouncer hauv kev hloov pauv, tom qab ntawd cov ntaub ntawv txuas ntxiv ntseeg tias qhov kev sib txuas no tseem ua haujlwm, thiab hauv nws lub rooj ib ntus tseem muaj.

Yog li ntawd, sim tsim nws dua, los ntawm kev sib txuas sib txawv rau pgbouncer, yuav ua rau muaj qhov yuam kev. Tab sis qhov no tuaj yeem cuam tshuam los ntawm kev siv CREATE TEMPORARY TABLE IF NOT EXISTS x.

Muaj tseeb, nws zoo dua tsis ua qhov no, vim tias tom qab ntawd koj tuaj yeem "dav ntawd" pom muaj cov ntaub ntawv seem ntawm "tus tswv yav dhau los". Hloov chaw, nws yog qhov zoo dua los nyeem phau ntawv qhia thiab pom tias thaum tsim lub rooj nws muaj peev xwm ntxiv ON COMMIT DROP - uas yog, thaum kev sib pauv ua tiav, lub rooj yuav raug rho tawm.

Tsis-replication

Vim tias lawv tsuas yog muaj kev sib txuas tshwj xeeb, cov ntxhuav ib ntus tsis rov ua dua. Tab sis qhov no tshem tawm qhov xav tau rau ob chav kaw cov ntaub ntawv hauv heap + WAL, yog li INSERT/UPDATE/DELETE rau hauv nws sai dua.

Tab sis txij li lub rooj ib ntus tseem yog "yuav luag zoo tib yam" lub rooj, nws tsis tuaj yeem tsim los ntawm ib qho kev hloov pauv. Tsawg kawg rau tam sim no, txawm hais tias cov ntaub ntawv sib txuas tau nthuav dav mus ntev.

1.2. UNLOGGED TAB

Tab sis koj yuav tsum ua li cas, piv txwv li, yog tias koj muaj qee yam txheej txheem ETL uas tsis tuaj yeem ua tiav hauv ib qho kev sib pauv, tab sis koj tseem muaj pgbouncer hauv kev hloov pauv? ..

Los yog cov ntaub ntawv ntws loj heev li ntawd Tsis muaj bandwidth txaus ntawm ib qho kev sib txuas los ntawm ib lub database (nyeem, ib txheej txheem rau CPU)?..

Los yog qee qhov kev ua haujlwm tab tom mus asynchronously hauv kev sib txawv?..

Tsuas muaj ib qho kev xaiv ntawm no - ib ntus tsim ib lub rooj tsis yog ib ntus. Pun, yog. Ntawd yog:

  • tsim "kuv tus kheej" cov lus nrog ntau lub npe random thiaj li tsis cuam tshuam nrog leej twg
  • extract: sau lawv nrog cov ntaub ntawv los ntawm lwm qhov chaw
  • Txia: hloov dua siab tshiab, sau rau hauv qhov tseem ceeb txuas teb
  • load: nchuav cov ntaub ntawv npaj rau hauv cov rooj sib tham
  • deleted "kuv" rooj

Thiab tam sim no - ya nyob rau hauv cov tshuaj pleev. Qhov tseeb, tag nrho cov ntawv sau hauv PostgreSQL tshwm sim ob zaug - thawj hauv WAL, ces mus rau hauv lub rooj / index lub cev. Tag nrho cov no yog ua los txhawb ACID thiab kho cov ntaub ntawv pom ntawm COMMIT'ntxawm thiab ROLLBACK'null muas.

Tab sis peb tsis xav tau qhov no! Peb muaj tag nrho cov txheej txheem Txawm hais tias nws tau ua tiav tiav lossis tsis yog.. Nws tsis muaj teeb meem ntau npaum li cas kev lag luam nruab nrab yuav muaj - peb tsis txaus siab rau "txuas ntxiv cov txheej txheem los ntawm nruab nrab," tshwj xeeb tshaj yog thaum nws tsis meej tias nws nyob qhov twg.

Txhawm rau ua qhov no, PostgreSQL cov neeg tsim khoom, rov qab rau hauv version 9.1, qhia txog qhov zoo li UNLOGGED cov rooj:

Nrog rau qhov kev qhia no, lub rooj yog tsim raws li unlogged. Cov ntaub ntawv sau rau cov rooj uas tsis tau sau tseg tsis mus dhau ntawm cov ntawv sau ua ntej (saib Tshooj 29), ua rau cov ntxhuav zoo li no. ua hauj lwm sai dua li niaj zaus. Txawm li cas los xij, lawv tsis tiv thaiv kev ua tsis tiav; nyob rau hauv cov ntaub ntawv ntawm server tsis ua hauj lwm los yog thaum muaj xwm ceev kaw, ib lub rooj unlogged txiav txiav. Tsis tas li ntawd, cov ntsiab lus ntawm lub rooj unlogged tsis replicated rau qhev servers. Tej indexes tsim nyob rau hauv ib lub rooj unlogged yeej yuav unlogged.

Hais luv luv, nws yuav nrawm dua, tab sis yog tias lub database server "poob", nws yuav tsis kaj siab. Tab sis ntau npaum li cas qhov no tshwm sim, thiab koj cov txheej txheem ETL puas paub yuav kho qhov no kom raug "los ntawm nruab nrab" tom qab "rov ua dua" cov ntaub ntawv? ...

Yog tias tsis yog, thiab cov ntaub ntawv saum toj no zoo ib yam li koj li, siv UNLOGGEDtab sis yeej tsis tsis txhob ua kom tus cwj pwm no ntawm cov ntxhuav tiag, cov ntaub ntawv los ntawm uas yog hlub rau koj.

1.3. RAU COMMIT { DELETE ROWS | DROP}

Qhov kev tsim no tso cai rau koj qhia tus cwj pwm tsis siv neeg thaum kev sib pauv ua tiav thaum tsim lub rooj.

rau ON COMMIT DROP Kuv twb sau saum toj no, nws generates DROP TABLE, tab sis nrog ON COMMIT DELETE ROWS qhov teeb meem yog nthuav ntau - nws yog generated ntawm no TRUNCATE TABLE.

Txij li tag nrho cov kev tsim kho vaj tse rau khaws cia cov lus piav qhia ntawm lub rooj ib ntus yog tib yam li cov lus tsis tu ncua, ces Tsis tu ncua kev tsim thiab tshem tawm ntawm cov ntxhuav ib ntus ua rau "so" loj ntawm cov rooj kaw lus pg_class, pg_attribute, pg_attrdef, pg_depend,…

Tam sim no xav txog tias koj muaj ib tus neeg ua haujlwm ntawm kev sib txuas ncaj qha rau cov ntaub ntawv, uas qhib qhov kev hloov pauv tshiab txhua ob, tsim, sau, txheej txheem thiab tshem tawm lub rooj ib ntus ... Yuav muaj ntau tshaj ntawm cov khib nyiab khaws cia hauv cov rooj kaw lus, thiab qhov no yuav ua rau muaj brakes ntxiv rau txhua qhov haujlwm.

Feem ntau, tsis txhob ua qhov no! Hauv qhov no nws yog qhov ua tau zoo dua CREATE TEMPORARY TABLE x ... ON COMMIT DELETE ROWS coj nws tawm ntawm lub voj voog kev lag luam - tom qab ntawd los ntawm qhov pib ntawm txhua qhov kev hloov pauv tshiab cov ntxhuav twb dhau lawm yuav muaj (txuag hu CREATE), mas yuav npliag, ua tsaug rau TRUNCATE (peb kuj tau khaws cia nws hu) thaum ua tiav qhov kev hloov pauv dhau los.

1.4. ZOO SIAB...SIM...

Kuv tau hais thaum pib tias ib qho ntawm cov kev siv ib ntus rau cov ntxhuav yog ntau hom kev ntshuam - thiab tus tsim tawm nkees nkees luam tawm cov npe ntawm thaj chaw ntawm lub hom phiaj lub rooj rau hauv kev tshaj tawm ntawm nws ib ntus ...

Tab sis kev tub nkeeg yog lub cav ntawm kev vam meej! Yog vim li cas tsim ib lub rooj tshiab "raws li tus qauv" nws tuaj yeem ua tau yooj yim dua:

CREATE TEMPORARY TABLE import_table(
  LIKE target_table
);

Txij li thaum koj tuaj yeem tsim ntau cov ntaub ntawv rau hauv lub rooj no, kev tshawb nrhiav los ntawm nws yuav tsis nrawm. Tab sis muaj ib txwm daws rau qhov no - indexes! Thiab, yog, lub rooj ib ntus kuj tuaj yeem muaj qhov ntsuas.

Txij li thaum, feem ntau, qhov yuav tsum tau indexes coincide nrog lub indexes ntawm lub hom phiaj lub rooj, koj tsuas sau tau LIKE target_table INCLUDING INDEXES.

Yog koj tseem xav tau DEFAULT-values ​​​​(piv txwv li, txhawm rau sau cov txiaj ntsig tseem ceeb), koj tuaj yeem siv LIKE target_table INCLUDING DEFAULTS. Los yog yooj yim - LIKE target_table INCLUDING ALL - Cov ntawv luam theej tawm, kev ntsuas, kev txwv, ...

Tab sis ntawm no koj yuav tsum nkag siab tias yog tias koj tsim import cov lus tam sim nrog indexes, ces cov ntaub ntawv yuav siv sij hawm ntev mus thauj khoomdua li yog tias koj xub sau txhua yam, thiab tsuas yog tom qab ntawd dov cov indexes - saib seb nws ua li cas ua piv txwv pg_dub.

Hauv ntej, RTFM!

2. Yuav sau li cas?

Cia kuv hais - siv nws COPY- flow es tsis txhob "pob" INSERT, acceleration ntawm lub sij hawm. Koj tuaj yeem ncaj qha los ntawm cov ntaub ntawv ua ntej tsim.

3. Yuav ua li cas?

Yog li ntawd, cia peb intro saib ib yam dab tsi zoo li no:

  • koj muaj ib lub rooj nrog cov neeg siv cov ntaub ntawv khaws cia rau hauv koj lub database 1M cov ntaub ntawv
  • txhua hnub tus neeg siv khoom xa tuaj rau koj tag nrho "duab"
  • los ntawm kev paub koj paub tias qee lub sijhawm tsis ntau tshaj 10K cov ntaub ntawv raug hloov

Ib qho piv txwv classic ntawm qhov xwm txheej no yog KLADR puag - muaj ntau qhov chaw nyob hauv tag nrho, tab sis nyob rau hauv txhua lub lim tiam upload muaj kev hloov pauv tsawg heev (renaming of settlements, combining streets, looks of new houses) txawm nyob rau hauv lub teb chaws.

3.1. Tag nrho synchronization algorithm

Rau qhov yooj yim, cia peb hais tias koj tsis tas yuav hloov kho cov ntaub ntawv - tsuas yog nqa lub rooj rau hauv daim ntawv xav tau, uas yog:

  • tshem tawm txhua yam uas tsis muaj lawm
  • tshiab txhua yam uas twb muaj lawm thiab yuav tsum tau hloov kho
  • ntxig rau txhua yam uas tseem tsis tau tshwm sim

Vim li cas thiaj yuav tsum tau ua raws li qhov kev txiav txim no? Vim qhov no yog li cas lub rooj loj yuav loj hlob tsawg (nco MVCC!).

DELETE NTAWM dst

Tsis yog, tau kawg koj tuaj yeem tau txais los ntawm tsuas yog ob txoj haujlwm:

  • tshem tawm (DELETE) txhua yam nyob rau hauv general
  • ntxig rau tag nrho los ntawm cov duab tshiab

Tab sis tib lub sijhawm, ua tsaug rau MVCC, Qhov loj ntawm lub rooj yuav nce raws nraim ob zaug! Tau txais +1M dluab ntawm cov ntaub ntawv nyob rau hauv lub rooj vim 10K hloov tshiab yog li-yog li redundancy ...

TRUNCATE dst

Ib tus neeg tsim khoom paub ntau paub paub tias tag nrho cov ntsiav tshuaj tuaj yeem ntxuav tau pheej yig heev:

  • ntshiab (TRUNCATE) tag nrho lub rooj
  • ntxig rau tag nrho los ntawm cov duab tshiab

Txoj kev yog siv tau, tej zaum kuj siv tau, tab sis muaj ib qho teeb meem... Peb yuav tau ntxiv 1M cov ntaub ntawv rau lub sijhawm ntev, yog li peb tsis tuaj yeem tso lub rooj khoob rau txhua lub sijhawm no (raws li yuav tshwm sim yam tsis tau qhwv hauv ib qho kev sib pauv).

Qhov no txhais tau tias:

  • peb tab tom pib kev lag luam ntev
  • TRUNCATE imposes Access Exclusive- thaiv
  • peb ua qhov ntxig rau lub sijhawm ntev, thiab txhua tus neeg nyob rau lub sijhawm no txawm tsis tau SELECT

Ib yam dab tsi mus tsis zoo...

ALTER TABLE… RENAME… / DROP TABLE…

Lwm txoj hauv kev yog sau txhua yam rau hauv ib lub rooj tshiab cais, thiab tom qab ntawd tsuas yog hloov npe rau qhov qub. Ob peb yam me me tsis zoo:

  • tseem thiab Access Exclusive, txawm tias lub sijhawm tsawg dua
  • tag nrho cov lus nug cov phiaj xwm / kev txheeb cais rau cov lus no tau pib dua, yuav tsum khiav ANALYZE
  • tag nrho cov yuam sij txawv teb chaws raug tawg (FK) rau lub rooj

Muaj ib qho WIP thaj los ntawm Simon Riggs uas tau hais kom ua ALTER-ib qho kev ua haujlwm los hloov lub rooj lub cev ntawm cov ntaub ntawv qib, tsis tas kov cov txheeb cais thiab FK, tab sis tsis tau sau pawg.

DELETE, UPDATE, INSERT

Yog li, peb txiav txim siab rau qhov kev xaiv tsis thaiv ntawm peb txoj haujlwm. Yuav luag peb... Yuav ua li cas qhov no zoo tshaj plaws?

-- всС Π΄Π΅Π»Π°Π΅ΠΌ Π² Ρ€Π°ΠΌΠΊΠ°Ρ… Ρ‚Ρ€Π°Π½Π·Π°ΠΊΡ†ΠΈΠΈ, Ρ‡Ρ‚ΠΎΠ±Ρ‹ Π½ΠΈΠΊΡ‚ΠΎ Π½Π΅ Π²ΠΈΠ΄Π΅Π» "ΠΏΡ€ΠΎΠΌΠ΅ΠΆΡƒΡ‚ΠΎΡ‡Π½Ρ‹Ρ…" состояний
BEGIN;

-- создаСм Π²Ρ€Π΅ΠΌΠ΅Π½Π½ΡƒΡŽ Ρ‚Π°Π±Π»ΠΈΡ†Ρƒ с ΠΈΠΌΠΏΠΎΡ€Ρ‚ΠΈΡ€ΡƒΠ΅ΠΌΡ‹ΠΌΠΈ Π΄Π°Π½Π½Ρ‹ΠΌΠΈ
CREATE TEMPORARY TABLE tmp(
  LIKE dst INCLUDING INDEXES -- ΠΏΠΎ ΠΎΠ±Ρ€Π°Π·Ρƒ ΠΈ подобию, вмСстС с индСксами
) ON COMMIT DROP; -- Π·Π° Ρ€Π°ΠΌΠΊΠ°ΠΌΠΈ Ρ‚Ρ€Π°Π½Π·Π°ΠΊΡ†ΠΈΠΈ ΠΎΠ½Π° Π½Π°ΠΌ Π½Π΅ Π½ΡƒΠΆΠ½Π°

-- быстро-быстро Π²Π»ΠΈΠ²Π°Π΅ΠΌ Π½ΠΎΠ²Ρ‹ΠΉ ΠΎΠ±Ρ€Π°Π· Ρ‡Π΅Ρ€Π΅Π· COPY
COPY tmp FROM STDIN;
-- ...
-- .

-- удаляСм ΠΎΡ‚ΡΡƒΡ‚ΡΡ‚Π²ΡƒΡŽΡ‰ΠΈΠ΅
DELETE FROM
  dst D
USING
  dst X
LEFT JOIN
  tmp Y
    USING(pk1, pk2) -- поля ΠΏΠ΅Ρ€Π²ΠΈΡ‡Π½ΠΎΠ³ΠΎ ΠΊΠ»ΡŽΡ‡Π°
WHERE
  (D.pk1, D.pk2) = (X.pk1, X.pk2) AND
  Y IS NOT DISTINCT FROM NULL; -- "Π°Π½Ρ‚ΠΈΠ΄ΠΆΠΎΠΉΠ½"

-- обновляСм ΠΎΡΡ‚Π°Π²ΡˆΠΈΠ΅ΡΡ
UPDATE
  dst D
SET
  (f1, f2, f3) = (T.f1, T.f2, T.f3)
FROM
  tmp T
WHERE
  (D.pk1, D.pk2) = (T.pk1, T.pk2) AND
  (D.f1, D.f2, D.f3) IS DISTINCT FROM (T.f1, T.f2, T.f3); -- Π½Π΅Π·Π°Ρ‡Π΅ΠΌ ΠΎΠ±Π½ΠΎΠ²Π»ΡΡ‚ΡŒ ΡΠΎΠ²ΠΏΠ°Π΄Π°ΡŽΡ‰ΠΈΠ΅

-- вставляСм ΠΎΡ‚ΡΡƒΡ‚ΡΡ‚Π²ΡƒΡŽΡ‰ΠΈΠ΅
INSERT INTO
  dst
SELECT
  T.*
FROM
  tmp T
LEFT JOIN
  dst D
    USING(pk1, pk2)
WHERE
  D IS NOT DISTINCT FROM NULL;

COMMIT;

3.2. Ntshuam tom qab ua tiav

Hauv tib KLADR, tag nrho cov ntaub ntawv hloov pauv yuav tsum tau ua ntxiv los ntawm kev ua tiav tom qab - normalized, cov ntsiab lus tseem ceeb, thiab txo qis rau cov qauv tsim nyog. Tab sis koj paub li cas - dab tsi raws nraim hloovtsis nyuaj rau synchronization code, hom phiaj tsis kov nws txhua?

Yog tias koj tus txheej txheem tsuas yog sau ntawv nkag tau thaum lub sijhawm synchronization, koj tuaj yeem siv lub cim uas yuav sau tag nrho cov kev hloov pauv rau peb:

-- Ρ†Π΅Π»Π΅Π²Ρ‹Π΅ Ρ‚Π°Π±Π»ΠΈΡ†Ρ‹
CREATE TABLE kladr(...);
CREATE TABLE kladr_house(...);

-- Ρ‚Π°Π±Π»ΠΈΡ†Ρ‹ с историСй ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΠΉ
CREATE TABLE kladr$log(
  ro kladr, -- Ρ‚ΡƒΡ‚ Π»Π΅ΠΆΠ°Ρ‚ Ρ†Π΅Π»Ρ‹Π΅ ΠΎΠ±Ρ€Π°Π·Ρ‹ записСй старой/Π½ΠΎΠ²ΠΎΠΉ
  rn kladr
);

CREATE TABLE kladr_house$log(
  ro kladr_house,
  rn kladr_house
);

-- общая функция логирования ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΠΉ
CREATE OR REPLACE FUNCTION diff$log() RETURNS trigger AS $$
DECLARE
  dst varchar = TG_TABLE_NAME || '$log';
  stmt text = '';
BEGIN
  -- провСряСм Π½Π΅ΠΎΠ±Ρ…ΠΎΠ΄ΠΈΠΌΠΎΡΡ‚ΡŒ логгирования ΠΏΡ€ΠΈ ΠΎΠ±Π½ΠΎΠ²Π»Π΅Π½ΠΈΠΈ записи
  IF TG_OP = 'UPDATE' THEN
    IF NEW IS NOT DISTINCT FROM OLD THEN
      RETURN NEW;
    END IF;
  END IF;
  -- создаСм запись лога
  stmt = 'INSERT INTO ' || dst::text || '(ro,rn)VALUES(';
  CASE TG_OP
    WHEN 'INSERT' THEN
      EXECUTE stmt || 'NULL,$1)' USING NEW;
    WHEN 'UPDATE' THEN
      EXECUTE stmt || '$1,$2)' USING OLD, NEW;
    WHEN 'DELETE' THEN
      EXECUTE stmt || '$1,NULL)' USING OLD;
  END CASE;
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

Tam sim no peb tuaj yeem siv cov txiaj ntsig ua ntej pib synchronization (lossis pab lawv ntawm ALTER TABLE ... ENABLE TRIGGER ...):

CREATE TRIGGER log
  AFTER INSERT OR UPDATE OR DELETE
  ON kladr
    FOR EACH ROW
      EXECUTE PROCEDURE diff$log();

CREATE TRIGGER log
  AFTER INSERT OR UPDATE OR DELETE
  ON kladr_house
    FOR EACH ROW
      EXECUTE PROCEDURE diff$log();

Thiab tom qab ntawd peb maj mam rho tawm tag nrho cov kev hloov pauv uas peb xav tau los ntawm cov rooj log thiab khiav lawv los ntawm cov neeg ua haujlwm ntxiv.

3.3. Importing Linked Sets

Saum toj no peb suav tias yog cov ntaub ntawv thaum cov ntaub ntawv tsim ntawm qhov chaw thiab qhov chaw nyob yog tib yam. Tab sis yuav ua li cas yog tias qhov upload los ntawm ib qho chaw sab nraud muaj cov qauv sib txawv ntawm cov qauv cia hauv peb cov ntaub ntawv?

Cia peb ua piv txwv qhov khaws cia ntawm cov neeg siv khoom thiab lawv cov nyiaj, qhov kev xaiv "ntau-rau-ib" classic:

CREATE TABLE client(
  client_id
    serial
      PRIMARY KEY
, inn
    varchar
      UNIQUE
, name
    varchar
);

CREATE TABLE invoice(
  invoice_id
    serial
      PRIMARY KEY
, client_id
    integer
      REFERENCES client(client_id)
, number
    varchar
, dt
    date
, sum
    numeric(32,2)
);

Tab sis qhov rub tawm los ntawm qhov chaw sab nraud tuaj rau peb hauv daim ntawv ntawm "tag nrho hauv ib qho":

CREATE TEMPORARY TABLE invoice_import(
  client_inn
    varchar
, client_name
    varchar
, invoice_number
    varchar
, invoice_dt
    date
, invoice_sum
    numeric(32,2)
);

Obviously, cov neeg siv cov ntaub ntawv tuaj yeem muab luam tawm hauv qhov version no, thiab cov ntaub ntawv tseem ceeb yog "tus account":

0123456789;Вася;A-01;2020-03-16;1000.00
9876543210;ΠŸΠ΅Ρ‚Ρ;A-02;2020-03-16;666.00
0123456789;Вася;B-03;2020-03-16;9999.00

Rau cov qauv, peb tsuas yog ntxig peb cov ntaub ntawv xeem, tab sis nco ntsoov - COPY ntau npaum!

INSERT INTO invoice_import
VALUES
  ('0123456789', 'Вася', 'A-01', '2020-03-16', 1000.00)
, ('9876543210', 'ΠŸΠ΅Ρ‚Ρ', 'A-02', '2020-03-16', 666.00)
, ('0123456789', 'Вася', 'B-03', '2020-03-16', 9999.00);

Ua ntej, cia peb hais txog cov "txiav" uas peb "qhov tseeb" xa mus. Nyob rau hauv peb cov ntaub ntawv, invoices xa mus rau cov neeg muas zaub:

CREATE TEMPORARY TABLE client_import AS
SELECT DISTINCT ON(client_inn)
-- ΠΌΠΎΠΆΠ½ΠΎ просто SELECT DISTINCT, Ссли Π΄Π°Π½Π½Ρ‹Π΅ Π·Π°Π²Π΅Π΄ΠΎΠΌΠΎ Π½Π΅ΠΏΡ€ΠΎΡ‚ΠΈΠ²ΠΎΡ€Π΅Ρ‡ΠΈΠ²Ρ‹
  client_inn inn
, client_name "name"
FROM
  invoice_import;

Txhawm rau kom koom nrog cov nyiaj tau raug nrog cov neeg siv khoom ID, peb yuav tsum xub nrhiav kom paub lossis tsim cov cim no. Cia peb ntxiv cov teb hauv qab lawv:

ALTER TABLE invoice_import ADD COLUMN client_id integer;
ALTER TABLE client_import ADD COLUMN client_id integer;

Cia peb siv cov txheej txheem synchronization cov lus piav qhia saum toj no nrog kev hloov kho me me - peb yuav tsis hloov kho lossis tshem tawm ib yam dab tsi hauv lub rooj sib tham, vim tias peb tuaj yeem siv cov neeg siv khoom "append-tsuas":

-- проставляСм Π² Ρ‚Π°Π±Π»ΠΈΡ†Π΅ ΠΈΠΌΠΏΠΎΡ€Ρ‚Π° ID ΡƒΠΆΠ΅ ΡΡƒΡ‰Π΅ΡΡ‚Π²ΡƒΡŽΡ‰ΠΈΡ… записСй
UPDATE
  client_import T
SET
  client_id = D.client_id
FROM
  client D
WHERE
  T.inn = D.inn; -- unique key

-- вставляСм ΠΎΡ‚ΡΡƒΡ‚ΡΡ‚Π²ΠΎΠ²Π°Π²ΡˆΠΈΠ΅ записи ΠΈ проставляСм ΠΈΡ… ID
WITH ins AS (
  INSERT INTO client(
    inn
  , name
  )
  SELECT
    inn
  , name
  FROM
    client_import
  WHERE
    client_id IS NULL -- Ссли ID Π½Π΅ проставился
  RETURNING *
)
UPDATE
  client_import T
SET
  client_id = D.client_id
FROM
  ins D
WHERE
  T.inn = D.inn; -- unique key

-- проставляСм ID ΠΊΠ»ΠΈΠ΅Π½Ρ‚ΠΎΠ² Ρƒ записСй счСтов
UPDATE
  invoice_import T
SET
  client_id = D.client_id
FROM
  client_import D
WHERE
  T.client_inn = D.inn; -- ΠΏΡ€ΠΈΠΊΠ»Π°Π΄Π½ΠΎΠΉ ΠΊΠ»ΡŽΡ‡

Qhov tseeb, txhua yam nyob hauv invoice_import Tam sim no peb muaj qhov chaw tiv tauj sau rau hauv client_id, nrog rau qhov uas peb yuav ntxig rau cov invoice.

Tau qhov twg los: www.hab.com

Ntxiv ib saib