Kuyambira 1999, kuti athandize ofesi yakumbuyo, banki yathu yagwiritsa ntchito njira yophatikizira mabanki BISKVIT pa nsanja ya Progress OpenEdge, yomwe imagwiritsidwa ntchito kwambiri padziko lonse lapansi, kuphatikiza pazachuma. Kuchita kwa DBMS iyi kumakupatsani mwayi wowerengera mpaka ma miliyoni miliyoni kapena kupitilira apo pamphindi imodzi mu database imodzi (DB). Ntchito zathu za Progress OpenEdge pafupifupi ma depositi miliyoni 1,5 miliyoni ndi ma contract pafupifupi 22,2 miliyoni azinthu zomwe zimagwira ntchito (ngongole zamagalimoto ndi nyumba zanyumba), ndipo imayang'aniranso malo onse okhala ndi owongolera (Banki Yapakati) ndi SWIFT.
Pogwiritsa ntchito Progress OpenEdge, tidayang'anizana ndi kufunikira kopangitsa kuti igwire ntchito ndi Oracle DBMS. Poyambirira, mtolo uwu unali wolepheretsa zomangamanga zathu - mpaka tidayika ndi kukonza Pro2 CDC - chinthu cha Progress chomwe chimakulolani kutumiza deta kuchokera ku Progress DBMS kupita ku Oracle DBMS mwachindunji, pa intaneti. Mu positi iyi tikuuzani mwatsatanetsatane, ndi zovuta zonse, momwe mungapangire mabwenzi bwino pakati pa OpenEdge ndi Oracle.
Momwe zidachitikira: kukweza deta ku QCD kudzera pagawo lafayilo
Choyamba, mfundo zina za zomangamanga zathu. Chiwerengero cha ogwiritsa ntchito Nawonso achichepere ndi pafupifupi 15 zikwi. Kuchuluka kwa nkhokwe zonse zopanga, kuphatikiza zofananira ndi zoyimira, ndi 600 TB, database yayikulu kwambiri ndi 16,5 TB. Panthawi imodzimodziyo, zolembazo zikuwonjezeredwa nthawi zonse: m'chaka chatha chokha, pafupifupi 120 TB ya deta yopindulitsa yawonjezedwa. Dongosololi limayendetsedwa ndi ma seva 150 akutsogolo pa nsanja ya x86. Zosungirako zimasungidwa pa ma seva 21 a IBM.
Machitidwe akutsogolo, machitidwe osiyanasiyana amabanki ndi mabanki akuphatikizidwa ndi OpenEdge Progress (BISCUIT IBS) kudzera pa basi ya Sonic ESB. Kukweza deta ku QCD kumachitika kudzera pakusinthana kwamafayilo. Mpaka nthawi ina, yankholi linali ndi mavuto awiri akuluakulu nthawi imodzi - kuchepa kwa kukweza zidziwitso mu malo osungiramo deta (CDW) komanso nthawi yayitali yogwirizanitsa deta (kuyanjanitsa) ndi machitidwe ena.
Choncho, tinayamba kufunafuna chida chomwe chingafulumizitse njirazi. Yankho la mavuto onsewa linali latsopano Progress OpenEdge product - Pro2 CDC (Change Data Capture). Kotero, tiyeni tiyambe.
Ikani Progress OpenEdge ndi Pro2Oracle
Kuthamangitsa Pro2 Oracle pa kompyuta ya Windows ya woyang'anira, ndikokwanira kukhazikitsa Progress OpenEdge Developer Kit Classroom Edition, yomwe ingakhale. kwaulere. Maupangiri okhazikika a OpenEdge:
DLC: C:ProgressOpenEdge
WRK: C:OpenEdgeWRK
Njira za ETL zimafuna laisensi ya Progress OpenEdge version 11.7+ - yomwe ndi OE DataServer ya Oracle ndi 4GL Development System. Malayisensi awa akuphatikizidwa ndi Pro2. Kuti mugwire ntchito yonse ya DataServer ya Oracle yokhala ndi database yakutali ya Oracle, Full Oracle Client imayikidwa.
Pa seva ya Oracle muyenera kukhazikitsa Oracle Database 12+, pangani nkhokwe yopanda kanthu ndikuwonjezera wogwiritsa ntchito (tiyeni timuyitane. cdc).
Kuti muyike Pro2Oracle, tsitsani kugawa kwaposachedwa kuchokera pamalo otsitsa. Tsegulani zosungidwa mu chikwatu C:Pro2 (Kukonza Pro2 pa Unix, kugawa komweko kumagwiritsidwa ntchito ndipo mfundo zofananira zimagwiranso ntchito).
Kupanga database ya cdc yobwereza
Replication database cdc (repl) Pro2 imagwiritsidwa ntchito kusunga zidziwitso zosintha, kuphatikiza mapu obwereza, mayina a nkhokwe zotsatiridwa ndi matebulo awo. Lilinso ndi mzere wobwerezabwereza, wophatikizira zolemba zokhudzana ndi kuti mzere wa tebulo muzosungirako zasintha. Deta yochokera pamzere wobwereza imagwiritsidwa ntchito ndi njira za ETL kuzindikira mizere yomwe ikuyenera kukopera ku Oracle kuchokera pankhokwe.
Tikupanga nkhokwe yapadera ya cdc.
Njira yopangira database
- Pa seva ya database timapanga chikwatu cha database ya cdc - mwachitsanzo, pa seva /database/cdc/.
- Pangani dummy ya database ya cdc: tumizani $DLC/empty cdc
- Yambitsani kuthandizira kwamafayilo akulu: proutil cdc -C EnableLargeFiles
- Timakonzekera script kuti tiyambe nkhokwe ya cdc. Magawo oyambira ayenera kukhala ofanana ndi magawo oyambira a database yofananizidwa.
- Timayamba database ya cdc.
- Lumikizani ku database ya cdc ndikukweza Pro2 schema kuchokera pafayilo cdc.df, yomwe ikuphatikizidwa ndi Pro2.
- Timapanga otsatirawa mu nkhokwe ya cdc:
pro2adm - yolumikizana ndi gulu la oyang'anira Pro2;
pro2etl - polumikiza njira za ETL (ReplBatch);
pro2cdc - polumikiza njira za CDC (CDCBatch);
Kutsegula OpenEdge Change Data Capture
Tsopano tiyeni tiyatse makina a CDC okha, mothandizidwa ndi zomwe deta idzafotokozedwe kumalo owonjezera aukadaulo. Pa database iliyonse ya Progress OpenEdge, muyenera kuwonjezera malo osungiramo omwe gwero la data lidzabwerezedwa, ndikuyambitsa makinawo pogwiritsa ntchito lamulo. zochitika.
Chitsanzo cha ndondomeko ya bisquit database
- Kukopera kuchokera ku catalog C: Pro2db fayilo cdcadd.st ku chikwatu cha database ya bisquit.
- Timafotokozera mu cdcadd.st kukula kokhazikika kwa zigawo "ReplCDCArea" ΠΈ "ReplCDCArea_IDX". Mutha kuwonjezera malo atsopano osungira pa intaneti: prostrct addonline bisquit cdcadd.st
- Yambitsani OpenEdge CDC:
proutil bisquit -C enablecdc area "ReplCDCArea" indexarea "ReplCDCArea_IDX" - Otsatira otsatirawa ayenera kupangidwa muzosungirako kuti adziwe njira zomwe zikuyenda:
a. pro2adm - yolumikizira kuchokera pagulu la oyang'anira Pro2.
b. pro2etl - polumikiza njira za ETL (ReplBatch).
c. pro2cdc - yolumikiza njira za CDC (CDCBatch).
Kupanga Schema Holder ya DataServer ya Oracle
Kenako, tifunika kupanga database ya Schema Holder pa seva pomwe deta yochokera ku Progress DBMS idzasinthidwanso ku Oracle DBMS. DataServer Schema Holder ndi nkhokwe yopanda kanthu ya Progress OpenEdge yopanda ogwiritsa ntchito kapena chidziwitso cha pulogalamu, yomwe ili ndi mapu olumikizana pakati pa matebulo oyambira ndi matebulo akunja a Oracle.
Schema Holder database ya Progress OpenEdge DataServer ya Oracle ya Pro2 iyenera kupezeka pa seva ya ETL; imapangidwa padera pa nthambi iliyonse.
Momwe mungapangire Schema Holder
- Tsegulani kugawa kwa Pro2 mu bukhu /pro2
- Pangani ndikupita ku chikwatu /pro2/dbsh
- Pangani database ya Schema Holder pogwiritsa ntchito lamulo koperani $DLC/bisquitsh opanda kanthu
- Kuchita kutembenuka bisquits mu kabisidwe kofunikira - mwachitsanzo, mu UTF-8 ngati nkhokwe za Oracle zili ndi encoding ya UTF-8: proutil bisquitsh -C convchar convert UTF-8
- Pambuyo popanga database yopanda kanthu bisquits gwirizanitsani kwa izo mumayendedwe a munthu mmodzi: pa bisquits
- Tiyeni tipite ku Data Dictionary: Zida -> Dictionary Data -> DataServer -> ORACLE Utilities -> Pangani DataServer Schema
- Yambitsani Schema Holder
- Kukhazikitsa oracle DataServer broker:
a. Yambitsani AdminServer.
proadsv -kuyamba
b. Kuyamba kwa Oracle DataServer broker
oraman -name orabroker1 -kuyamba
Kukhazikitsa gulu loyang'anira ndi dongosolo lobwerezabwereza
Pogwiritsa ntchito gulu loyang'anira la Pro2, magawo a Pro2 amakonzedwa, kuphatikizapo kukhazikitsa ndondomeko yobwerezabwereza ndi kupanga njira za ETL (Processor Library), mapulogalamu oyambirira a synchronization (Bulk-Copy processor), zoyambitsa kubwereza ndi ndondomeko za OpenEdge CDC. Palinso zida zoyambira zowunikira ndikuwongolera njira za ETL ndi CDC. Choyamba, timayika mafayilo a parameter.
Momwe mungasinthire mafayilo a parameter
- Pitani ku catalog C:Pro2bpreplScripts
- Tsegulani fayilo kuti musinthe replProc.pf
- Onjezani magawo olumikizira ku database ya cdc:
# Replication Database
-db cdc -ld repl -H <main database hostname> -S <database broker port cdc>
-U pro2admin -P <password> - onjezani ku replProc.pf magawo olumikizira kuti mupeze ma database ndi Schema Holder mu mawonekedwe a mafayilo a parameter. Dzina la fayilo ya magawo liyenera kufanana ndi dzina la database yomwe ikulumikizidwa.
# Lumikizani kuzinthu zonse zofananizidwa ndi BISQUIT
-pf bpreplscriptsbisquit.pf - onjezani ku replProc.pf magawo olumikizirana ndi Schema Holder.
#Target Pro DB Schema Holder
-db bisquitsh -ld bisquitsh
-H <ETL ndondomeko host dzina>
-S <biskuitsh broker port>
-db bisquitsql
-ld bisquitsql
-dt ORACLE
-S 5162 -H <Oracle broker hostname>
-DataService orabroker1 - Sungani fayilo ya parameters replProc.pf
- Chotsatira, muyenera kupanga ndi kutsegula kuti musinthe mafayilo a parameter pa database iliyonse yolumikizidwa mu bukhulo C:Pro2bpreplScripts: bisquit.pf. Fayilo iliyonse ya pf imakhala ndi magawo olumikizirana ndi nkhokwe yofananira, mwachitsanzo:
-db bisquit -ld bisquit -H <hostname> -S <broker port>
-U pro2admin -P <password>
Kuti musinthe njira zazifupi za Windows, muyenera kupita ku chikwatu C:Pro2bpreplScripts ndikusintha njira yachidule ya "Pro2 - Administration". Kuti muchite izi, tsegulani katundu wachidule ndi mzere Yambirani onetsani chikwatu chokhazikitsa Pro2. Ntchito yofananira iyenera kuchitidwa panjira zazifupi za "Pro2 - Editor" ndi "RunBulkLoader".
Kukonzekera kwa Pro2 Administration: Kutsegula Kukonzekera Koyamba
Tiyeni tiyambitse console.
Pitani ku "DB Map".
Kuti mulumikizane ndi nkhokwe mu Pro2 - Administration, pitani ku tabu Mapu a DB. Onjezani mapu a nkhokwe zoyambira - Schema Holder - Oracle.
Pitani ku tabu sanjira. Zolembedwa Source Database Mwachikhazikitso, nkhokwe yoyamba yolumikizidwa imasankhidwa. Kumanja kwa mndandanda payenera kukhala zolembedwa Ma Database Onse Olumikizidwa - nkhokwe zosankhidwa zimalumikizidwa. Pansi kumanzere muyenera kuwona mndandanda wa matebulo opita patsogolo kuchokera ku bisquit. Kumanja pali mndandanda wa matebulo ochokera ku database ya Oracle.
Kupanga schemas SQL ndi nkhokwe mu Oracle
Kuti mupange mapu obwerezabwereza, muyenera kupanga kaye SQL schema mu Oracle. Mu Pro2 Administration timachita chinthu cha menyu Zida -> Pangani Code -> Target Schema, ndiye mu bokosi la zokambirana Sankhani Nawonso achichepere sankhani nkhokwe imodzi kapena zingapo ndikusunthira kumanja.
Dinani Chabwino ndikusankha chikwatu kuti musunge ma SQL schemas.
Kenako timapanga maziko. Izi zitha kuchitika, mwachitsanzo, kudzera Wolemba Oracle SQL. Kuti tichite izi, timalumikizana ndi database ya Oracle ndikuyika schema yowonjezera matebulo. Pambuyo posintha mapangidwe a matebulo a Oracle, muyenera kusintha ma SQL schemas mu Schema Holder.
Kutsitsa kukamalizidwa bwino, tulukani munkhokwe ya bisquitsh ndikutsegula gulu loyang'anira Pro2. Matebulo ochokera ku database ya Oracle akuyenera kuwonekera pa Mapu tabu kumanja.
Kujambula mapu
Kuti mupange mapu obwereza, mu gulu loyang'anira Pro2, pitani ku Mapu tabu ndikusankha nkhokwe. Dinani pa Matebulo a Mapu, sankhani Sankhani Zosintha kumanzere kwa matebulo omwe akuyenera kufotokozedwanso mu Oracle, asunthire kumanja ndikutsimikizira zomwe zasankhidwa. Mapu apangidwa okha pamatebulo osankhidwa. Timabwereza ntchitoyi kuti tipange mapu obwereza a nkhokwe zina.
Kupanga Library ya Pro2 Replication processor ndi ma Bulk-Copy processor Programs
Laibulale ya Replication processor idapangidwira njira zobwerezabwereza (ETLs) zomwe zimayendetsa mzere wobwereza wa Pro2 ndikukankhira zosintha ku database ya Oracle. Mapulogalamu a laibulale ya replication processor amasungidwa m'ndandanda pambuyo pa mibadwo bprepl/repl_proc (PROC_DIRECTORY parameter). Kuti mupange laibulale yobwerezabwereza, pitani ku Zida -> Pangani Khodi -> Laibulale ya Purosesa. M'badwo ukamaliza, mapulogalamuwa adzawonekera m'ndandanda bprepl/repl_proc.
Mapologalamu a Bulk Load Processor amagwiritsidwa ntchito kugwirizanitsa nkhokwe za Progress ndi malo osungiramo zinthu zakale a Oracle potengera chilankhulo cha Progress ABL (4GL). Kuti mupange, pitani ku menyu Zida -> Pangani Code -> Bulk-Copy processor. Mu bokosi la zokambirana la Select Database, sankhani nkhokwe zoyambira, zisunthire kumanja kwa zenera ndikudina OK. M'badwo ukamaliza, mapulogalamuwa adzawonekera m'ndandanda bpreplrepl_mproc.
Kukhazikitsa njira zobwereza mu Pro2
Kugawa matebulo m'maseti omwe amatumizidwa ndi ulusi wobwereza wosiyana kumapangitsa kuti Pro2 Oracle igwire bwino ntchito. Mwachikhazikitso, malumikizidwe onse opangidwa mu mapu obwerezabwereza a matebulo atsopano obwereza amagwirizanitsidwa ndi nambala ya ulusi 1. Ndibwino kuti tilekanitse matebulo mu ulusi wosiyana.
Zambiri zokhudzana ndi momwe ulusi wobwerezabwereza zimawonekera pazithunzi za Pro2 Administration mu tabu ya Monitor mu gawo la Replication Status. Kufotokozera mwatsatanetsatane zamtengo wapatali kumatha kupezeka muzolemba za Pro2 (cholembera C: Pro2Docs).
Pangani ndi kuyambitsa ndondomeko za CDC
Ndondomeko ndi malamulo a injini ya OpenEdge CDC kuyang'anira kusintha kwa matebulo. Panthawi yolemba, Pro2 imangogwirizira mfundo za CDC zomwe zili ndi mulingo 0, ndiye kuti, zowona zimawunikidwa zolemba zosintha.
Kuti mupange mfundo za CDC, pagawo loyang'anira, pitani ku Mapu, sankhani nkhokwe ndikudina batani Onjezani/Chotsani Mfundo. Pazenera la Sankhani Zosintha lomwe limatsegulidwa, sankhani kumanzere ndikusunthira kumanja matebulo omwe muyenera kupanga kapena kufufuta mfundo za CDC.
Kuti muyambitse, tsegulaninso tabu ya Mapu, sankhani nkhokwe ndikudina batani (Mu)Yambitsani Ndondomeko. Sankhani ndikupita kumanja kwa tebulo ndondomeko zomwe zikuyenera kutsegulidwa, dinani OK. Zitatha izi amazilemba zobiriwira. Pogwiritsa ntchito (Mu)Yambitsani Ndondomeko Mukhozanso kuyimitsa ndondomeko za CDC. Zochita zonse zimachitika pa intaneti.
Pambuyo poyambitsa ndondomeko ya CDC, zolemba zokhudzana ndi zosinthidwa zimasungidwa kumalo osungira "ReplCDCArea" malinga ndi database ya gwero. Zolemba izi zidzakonzedwa mwa njira yapadera CDCBatch, zomwe zimachokera pa izo zidzapanga zolemba mumzere wobwereza wa Pro2 mu database cdc (repl).
Chifukwa chake, tili ndi mizere iwiri yobwerezabwereza. Gawo loyamba ndi CDCBatch: kuchokera ku nkhokwe, zomwe zimayambira zimapita ku database yapakatikati ya CDC. Gawo lachiwiri ndi pomwe deta imasamutsidwa kuchokera ku database ya CDC kupita ku Oracle. Ichi ndi mbali ya zomangamanga zamakono ndi mankhwala omwewo - mpaka pano omangawo sanathe kukhazikitsa kubwereza kwachindunji.
Kulunzanitsa koyambirira
Pambuyo poyambitsa makina a CDC ndikukhazikitsa seva yobwerezabwereza ya Pro2, tifunika kuyambitsa kulunzanitsa koyambirira. Lamulo loyamba la synchronization:
/pro2/bprepl/Script/replLoad.sh bisquit table-name
Kulunzanitsa koyamba kumalizidwa, njira zobwerezabwereza zitha kuyambika.
Chiyambi cha njira zobwerezabwereza
Kuti muyambe njira zobwerezabwereza muyenera kuyendetsa script replbatch.sh. Musanayambe, onetsetsani kuti pali zolemba za replbatch pa ulusi wonse - replbatch1, replbatch2, etc. Ngati zonse zili m'malo, tsegulani mzere wolamula (mwachitsanzo, proenv), kupita ku chikwatu /bprepl/scripts ndikuyamba script. Mu gulu loyang'anira, timayang'ana kuti ndondomeko yofananayo yalandira RUNNING status.
Zotsatira
Pambuyo pokhazikitsa, tidafulumizitsa kwambiri kuyika kwa zidziwitso kumalo osungiramo zinthu zamakampani. Zambiri zimalowa mu Oracle pa intaneti. Palibe chifukwa chotaya nthawi ndikufunsa mafunso kwanthawi yayitali kuti musonkhanitse deta kuchokera kumachitidwe osiyanasiyana. Kuonjezera apo, mu njira iyi njira yobwerezabwereza imatha kukakamiza deta, yomwe imakhalanso ndi zotsatira zabwino pa liwiro. Tsopano kuyanjanitsa tsiku ndi tsiku kwa dongosolo la BISKVIT ndi machitidwe ena kunayamba kutenga mphindi 15-20 m'malo mwa maola 2-2,5, ndipo kuyanjanitsa kwathunthu kunatenga maola angapo m'malo mwa masiku awiri.
Source: www.habr.com