ዛሬ ለሀብር አንባቢዎች በPostgreSQL DBMS ውስጥ ከጠረጴዛ ጥገኝነት ጋር ለመስራት በ Python የተጻፈ መገልገያ ማቅረብ እፈልጋለሁ።
የመገልገያ ኤፒአይ ቀላል እና ሶስት ዘዴዎችን ያቀፈ ነው፡
- የማህደር_ጠረጴዛ - ከተገለጹት ዋና ቁልፎች ጋር ተደጋጋሚ መዝገብ ቤት / መሰረዝ
- የጠረጴዛ_ማጣቀሻዎችን አግኝ - ለሠንጠረዡ ጥገኝነቶችን ይፈልጉ (በተጠቀሰው አንድ የተጠቀሰውን እና እሱን በመጥቀስ ሰንጠረዦችን ያሳያል)
- የረድፎች_ማጣቀሻዎች_አግኙ - በተፈለገው ሰንጠረዥ ውስጥ የተገለጹ ረድፎችን የሚያመለክቱ ሌሎች ሰንጠረዦችን ረድፎችን ይፈልጉ
prehistory
ስሜ Oleg Borzov እባላለሁ፣ በዶምክሊክ ውስጥ ለሞርጌጅ አስተዳዳሪዎች በ CRM ቡድን ውስጥ ገንቢ ነኝ።
የ CRM ስርዓታችን ዋና የመረጃ ቋት በኩባንያው ውስጥ ካለው የድምፅ መጠን አንፃር ትልቁ ነው። እሱ በጣም ጥንታዊ ከሚባሉት ውስጥ አንዱ ነው-በፕሮጀክቱ መጀመሪያ ላይ ታየ ፣ ዛፎቹ ትልቅ ሲሆኑ ፣ Domclick ጅምር ነበር ፣ እና በፋሽኑ ፒቲን ያልተመሳሰለ ማዕቀፍ ላይ ካለው ማይክሮ ሰርቪስ ፋንታ አንድ ትልቅ ፒኤችፒ ሞኖሊት ነበር።
ከፒኤችፒ ወደ ፓይዘን የተደረገው ሽግግር በጣም ረጅም ነበር እናም የሁለቱም ስርዓቶች በአንድ ጊዜ ድጋፍ ያስፈልገዋል, ይህም የውሂብ ጎታውን ንድፍ ነካ.
በውጤቱም ፣ ብዙ ቁጥር ያላቸው በጣም የተገናኙ እና ለተለያዩ መጠይቆች ብዛት ያላቸው ኢንዴክሶች ያሉበት ግዙፍ ጠረጴዛዎች ያለው የውሂብ ጎታ አለን። ይህ ሁሉ በመረጃ ቋቱ አፈፃፀም ላይ አሉታዊ ተጽዕኖ ያሳድራል-በትልልቅ ጠረጴዛዎች እና በመካከላቸው ባሉ ግንኙነቶች ብዛት ምክንያት የጥያቄዎች ውስብስብነት በየጊዜው እያደገ ነው ፣ በተለይም በጣም ለተጫኑ ጠረጴዛዎች በጣም አስፈላጊ ነው።
በመረጃ ቋቱ ላይ ያለውን ጫና ለመቀነስ አሮጌ መዝገቦችን በየቀኑ እጅግ በጣም ብዙ እና ከተጫኑ ጠረጴዛዎች ወደ ማህደር (ለምሳሌ ከ task
в task_archive
).
ይህ ተግባር በጠረጴዛዎች መካከል ባሉ በርካታ ግንኙነቶች የተወሳሰበ ነው-ረድፎችን ከ ያስተላልፉ task
в task_archive
በቂ አይደለም ፣ ከዚያ በፊት ሁሉንም በማጣቀስ ተመሳሳይ ነገር ማድረግ ያስፈልግዎታል task
ጠረጴዛዎች.
በምሳሌ አሳይሻለሁ።
ከጠረጴዛ ላይ መዝገቦችን መሰረዝ አለብን እንበል Flights
. Postgres ልክ እንደዚህ እንድናደርግ አይፈቅድልንም፡ በመጀመሪያ ከሁሉም የማጣቀሻ ሰንጠረዦች መዝገቦችን መሰረዝ አለብን፣ እና ማንም ወደማይጠቅሳቸው ጠረጴዛዎች አዘውትረን።
በእኛ ምሳሌ ላይ Flights
ያመለክታል Ticket_flights
እና በላዩ ላይ - Boarding_passes
.
ስለዚህ, በዚህ ቅደም ተከተል መሰረዝ አለብዎት:
- የረድፎችን ዋና ቁልፎች (PK) እሴቶችን ያግኙ
Ticket_flights
ውስጥ የሚወገዱትን መስመሮች የሚያመለክቱFlights
. - PK ረድፎችን ያግኙ
Boarding_passes
የሚያመለክተውTicket_flights
. - በሰንጠረዡ ውስጥ ካለው ንጥል 2 ረድፎችን በፒኬ ይሰርዙ
Boarding_passes
. - መስመሮችን በፒኬ ከንጥል 1 ይሰርዙ
Ticket_flights
. - መስመሮችን ከ በማስወገድ ላይ
Flights
.
ውጤቱ PgGraph የሚባል መገልገያ ነበር፣ እሱም ክፍት ምንጭ ለማድረግ ወሰንን።
እንዴት መጠቀም እንደሚቻል
መገልገያው ሁለት የአጠቃቀም ሁነታዎችን ይደግፋል፡-
- የትእዛዝ መስመር ጥሪ (
pggraph …
). - በ Python ኮድ (ክፍል
PgGraphApi
).
ጭነት እና ውቅር
በመጀመሪያ መገልገያውን ከ Pypi ማከማቻ ውስጥ መጫን ያስፈልግዎታል:
pip3 install pggraph
ከዚያ የ config.ini ፋይልን በውሂብ ጎታ ውቅር እና በማህደር ስክሪፕት በሃገር ውስጥ ማሽን ላይ ይፍጠሩ፡
[db]
host = localhost
port = 5432
user = postgres
password = postgres
dbname = postgres
schema = public ; Необязательный параметр, указано значение по умолчанию
[archive] ; Данный раздел заполнять необязательно, ниже указаны значения по умолчанию
is_debug = false
chunk_size = 1000
max_depth = 20
to_archive = true
archive_suffix = 'archive'
ከኮንሶል ያሂዱ
መለኪያዎች
$ pggraph -h
usage: pggraph action [-h] --table TABLE [--ids IDS] [--config_path CONFIG_PATH]
positional arguments:
action required action: archive_table, get_table_references, get_rows_references
optional arguments:
-h, --help show this help message and exit
--table TABLE table name
--ids IDS primary key ids, separated by comma, e.g. 1,2,3
--config_path CONFIG_PATH path to config.ini
--log_path LOG_PATH path to log dir
--log_level LOG_LEVEL log level (debug, info, error)
የአቀማመጥ ክርክሮች፡-
action
- አስፈላጊ እርምጃ;archive_table
,get_table_references
ወይምget_rows_references
.
የተሰየሙ ክርክሮች፡-
--config_path
- ወደ ውቅር ፋይል መንገድ;--table
- አንድ ድርጊት ለማከናወን የሚፈልጉት ሰንጠረዥ;--ids
- በነጠላ ሰረዝ የተለየ የመታወቂያ ዝርዝር ፣ ለምሳሌ ፣1,2,3
(አማራጭ መለኪያ);--log_path
- ለመዝገቦች ወደ አቃፊው የሚወስደው መንገድ (አማራጭ ግቤት ፣ በነባሪ - የቤት አቃፊ);--log_level
- የመግቢያ ደረጃ (አማራጭ መለኪያ, ነባሪ - INFO).
የትእዛዝ ምሳሌዎች
ሰንጠረዥ በማህደር ማስቀመጥ
የመገልገያው ዋና ተግባር የውሂብ ማህደር ነው, ማለትም. ረድፎችን ከዋናው ጠረጴዛ ወደ ማህደሩ ጠረጴዛ ማንቀሳቀስ (ለምሳሌ ከጠረጴዛው መጽሐፍት в የመጽሐፍት_መዝገብ).
ያለ መዝገብ ቤት መሰረዝም ይደገፋል፡ ለዚህም በ config.ini ውስጥ መለኪያውን ማዘጋጀት ያስፈልግዎታል ወደ_ማህደር = ሐሰት).
አስፈላጊ መለኪያዎች - config_path፣ ሠንጠረዥ እና መታወቂያዎች.
ከተጀመረ በኋላ መዝገቦች በተደጋጋሚ ይሰረዛሉ ids
በሰንጠረ table ውስጥ table
እና በሁሉም ጠረጴዛዎች ላይ በማጣቀስ.
$ pggraph archive_table --config_path config.hw.local.ini --table flights --ids 1,2,3
2020-06-20 19:27:44 INFO: flights - START
2020-06-20 19:27:44 INFO: flights - start archive_recursive 3 rows (depth=0)
2020-06-20 19:27:44 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: ticket_flights - start archive_recursive 3 rows (depth=1)
2020-06-20 19:27:44 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: boarding_passes - start archive_recursive 3 rows (depth=2)
2020-06-20 19:27:44 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: boarding_passes - archive_by_ids 3 rows by ticket_no, flight_id
2020-06-20 19:27:44 INFO: boarding_passes - start archive_recursive 3 rows (depth=2)
2020-06-20 19:27:44 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: boarding_passes - archive_by_ids 3 rows by ticket_no, flight_id
2020-06-20 19:27:44 INFO: boarding_passes - start archive_recursive 3 rows (depth=2)
2020-06-20 19:27:44 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: boarding_passes - archive_by_ids 3 rows by ticket_no, flight_id
2020-06-20 19:27:44 INFO: boarding_passes - start archive_recursive 3 rows (depth=2)
2020-06-20 19:27:44 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: boarding_passes - archive_by_ids 3 rows by ticket_no, flight_id
2020-06-20 19:27:44 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: ticket_flights - archive_by_ids 3 rows by ticket_no, flight_id
2020-06-20 19:27:44 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 19:27:44 INFO: flights - archive_by_ids 3 rows by id
2020-06-20 19:27:44 INFO: flights - END
ለተጠቀሰው ሰንጠረዥ ጥገኞችን ይፈልጉ
የተጠቀሰው ሰንጠረዥ ጥገኞችን ለማግኘት ተግባር table
. አስፈላጊ መለኪያዎች - config_path
и table
.
ከተጀመረ በኋላ መዝገበ ቃላት በማያ ገጹ ላይ ይታያል፡-
in_refs
- ይህንን የሚያመለክት የሰንጠረዦች መዝገበ ቃላት, ቁልፉ የጠረጴዛው ስም ሲሆን, እሴቱ የውጭ ቁልፍ ነገሮች ዝርዝር ነው (pk_main
- በዋናው ሠንጠረዥ ውስጥ ዋና ቁልፍ;pk_ref
በማጣቀሻ ሰንጠረዥ ውስጥ ዋናው ቁልፍ ነው ፣fk_ref
- የምንጭ ሰንጠረዥ የውጭ ቁልፍ የሆነው የአምዱ ስም;out_refs
- የተሰጠው አንድ የሚያመለክተው የሰንጠረዦች መዝገበ ቃላት.
$ pggraph get_table_references --config_path config.hw.local.ini --table flights
{'in_refs': {'ticket_flights': [ForeignKey(pk_main='flight_id', pk_ref='ticket_no, flight_id', fk_ref='flight_id')]},
'out_refs': {'aircrafts': [ForeignKey(pk_main='aircraft_code', pk_ref='flight_id', fk_ref='aircraft_code')],
'airports': [ForeignKey(pk_main='airport_code', pk_ref='flight_id', fk_ref='arrival_airport'),
ForeignKey(pk_main='airport_code', pk_ref='flight_id', fk_ref='departure_airport')]}}
ከተጠቀሰው ዋና ቁልፍ ጋር የረድፎች ማጣቀሻዎችን ማግኘት
በሌሎች ሰንጠረዦች ውስጥ ረድፎችን ለመፈለግ ተግባር በውጭ ቁልፍ በኩል ረድፎችን የሚያመለክቱ ids
ጠረጴዛዎች table
. አስፈላጊ መለኪያዎች - config_path
, table
и ids
.
ከተጀመረ በኋላ፣ የሚከተለው መዋቅር ያለው መዝገበ ቃላት በማያ ገጹ ላይ ይታያል፡-
{
pk_id_1: {
reffering_table_name_1: {
foreign_key_1: [
{row_pk_1: value, row_pk_2: value},
...
],
...
},
...
},
pk_id_2: {...},
...
}
ምሳሌ ይደውሉ፡
$ pggraph get_rows_references --config_path config.hw.local.ini --table flights --ids 1,2,3
{1: {'ticket_flights': {'flight_id': [{'flight_id': 1,
'ticket_no': '0005432816945'},
{'flight_id': 1,
'ticket_no': '0005432816941'}]}},
2: {'ticket_flights': {'flight_id': [{'flight_id': 2,
'ticket_no': '0005433101832'},
{'flight_id': 2,
'ticket_no': '0005433101864'},
{'flight_id': 2,
'ticket_no': '0005432919715'}]}},
3: {'ticket_flights': {'flight_id': [{'flight_id': 3,
'ticket_no': '0005432817560'},
{'flight_id': 3,
'ticket_no': '0005432817568'},
{'flight_id': 3,
'ticket_no': '0005432817559'}]}}}
በኮድ ውስጥ መጠቀም
በኮንሶል ውስጥ ከመሮጥ በተጨማሪ, ቤተ-መጽሐፍት በ Python ኮድ ውስጥ መጠቀም ይቻላል. ከዚህ በታች የ iPython መስተጋብራዊ አካባቢን የመጥራት ምሳሌዎች አሉ።
ሰንጠረዥ በማህደር ማስቀመጥ
>>> from pg_graph.main import setup_logging
>>> setup_logging(log_level='DEBUG')
>>> from pg_graph.api import PgGraphApi
>>> api = PgGraphApi('config.hw.local.ini')
>>> api.archive_table('flights', [4,5])
2020-06-20 23:12:08 INFO: flights - START
2020-06-20 23:12:08 INFO: flights - start archive_recursive 2 rows (depth=0)
2020-06-20 23:12:08 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 DEBUG: ticket_flights - ForeignKey(pk_main='flight_id', pk_ref='flight_id, ticket_no', fk_ref='flight_id')
2020-06-20 23:12:08 DEBUG: SQL('SELECT flight_id, ticket_no FROM bookings.ticket_flights WHERE (flight_id) IN (%s, %s)')
2020-06-20 23:12:08 INFO: ticket_flights - start archive_recursive 30 rows (depth=1)
2020-06-20 23:12:08 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 DEBUG: boarding_passes - ForeignKey(pk_main='flight_id, ticket_no', pk_ref='flight_id, ticket_no', fk_ref='flight_id, ticket_no')
2020-06-20 23:12:08 INFO: boarding_passes - archive_by_fk 30 rows by ForeignKey(pk_main='flight_id, ticket_no', pk_ref='flight_id, ticket_no', fk_ref='flight_id, ticket_no')
2020-06-20 23:12:08 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.boarding_passes_archive (LIKE bookings.boarding_passes)')
2020-06-20 23:12:08 DEBUG: DELETE FROM boarding_passes by FK flight_id, ticket_no - 30 rows
2020-06-20 23:12:08 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 INFO: ticket_flights - archive_by_ids 30 rows by flight_id, ticket_no
2020-06-20 23:12:08 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.ticket_flights_archive (LIKE bookings.ticket_flights)')
2020-06-20 23:12:08 DEBUG: DELETE FROM ticket_flights by flight_id, ticket_no - 30 rows
2020-06-20 23:12:08 DEBUG: INSERT INTO ticket_flights_archive - 30 rows
2020-06-20 23:12:08 INFO: ticket_flights - start archive_recursive 30 rows (depth=1)
2020-06-20 23:12:08 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 DEBUG: boarding_passes - ForeignKey(pk_main='flight_id, ticket_no', pk_ref='flight_id, ticket_no', fk_ref='flight_id, ticket_no')
2020-06-20 23:12:08 INFO: boarding_passes - archive_by_fk 30 rows by ForeignKey(pk_main='flight_id, ticket_no', pk_ref='flight_id, ticket_no', fk_ref='flight_id, ticket_no')
2020-06-20 23:12:08 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.boarding_passes_archive (LIKE bookings.boarding_passes)')
2020-06-20 23:12:08 DEBUG: DELETE FROM boarding_passes by FK flight_id, ticket_no - 30 rows
2020-06-20 23:12:08 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 INFO: ticket_flights - archive_by_ids 30 rows by flight_id, ticket_no
2020-06-20 23:12:08 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.ticket_flights_archive (LIKE bookings.ticket_flights)')
2020-06-20 23:12:08 DEBUG: DELETE FROM ticket_flights by flight_id, ticket_no - 30 rows
2020-06-20 23:12:08 DEBUG: INSERT INTO ticket_flights_archive - 30 rows
2020-06-20 23:12:08 INFO: ticket_flights - start archive_recursive 30 rows (depth=1)
2020-06-20 23:12:08 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 DEBUG: boarding_passes - ForeignKey(pk_main='flight_id, ticket_no', pk_ref='flight_id, ticket_no', fk_ref='flight_id, ticket_no')
2020-06-20 23:12:08 INFO: boarding_passes - archive_by_fk 30 rows by ForeignKey(pk_main='flight_id, ticket_no', pk_ref='flight_id, ticket_no', fk_ref='flight_id, ticket_no')
2020-06-20 23:12:08 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.boarding_passes_archive (LIKE bookings.boarding_passes)')
2020-06-20 23:12:08 DEBUG: DELETE FROM boarding_passes by FK flight_id, ticket_no - 30 rows
2020-06-20 23:12:08 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 INFO: ticket_flights - archive_by_ids 30 rows by flight_id, ticket_no
2020-06-20 23:12:08 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.ticket_flights_archive (LIKE bookings.ticket_flights)')
2020-06-20 23:12:08 DEBUG: DELETE FROM ticket_flights by flight_id, ticket_no - 30 rows
2020-06-20 23:12:08 DEBUG: INSERT INTO ticket_flights_archive - 30 rows
2020-06-20 23:12:08 INFO: ticket_flights - start archive_recursive 3 rows (depth=1)
2020-06-20 23:12:08 INFO: START ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 DEBUG: boarding_passes - ForeignKey(pk_main='flight_id, ticket_no', pk_ref='flight_id, ticket_no', fk_ref='flight_id, ticket_no')
2020-06-20 23:12:08 INFO: boarding_passes - archive_by_fk 3 rows by ForeignKey(pk_main='flight_id, ticket_no', pk_ref='flight_id, ticket_no', fk_ref='flight_id, ticket_no')
2020-06-20 23:12:08 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.boarding_passes_archive (LIKE bookings.boarding_passes)')
2020-06-20 23:12:08 DEBUG: DELETE FROM boarding_passes by FK flight_id, ticket_no - 3 rows
2020-06-20 23:12:08 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 INFO: ticket_flights - archive_by_ids 3 rows by flight_id, ticket_no
2020-06-20 23:12:08 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.ticket_flights_archive (LIKE bookings.ticket_flights)')
2020-06-20 23:12:08 DEBUG: DELETE FROM ticket_flights by flight_id, ticket_no - 3 rows
2020-06-20 23:12:08 DEBUG: INSERT INTO ticket_flights_archive - 3 rows
2020-06-20 23:12:08 INFO: END ARCHIVE REFERRING TABLES
2020-06-20 23:12:08 INFO: flights - archive_by_ids 2 rows by flight_id
2020-06-20 23:12:09 DEBUG: SQL('CREATE TABLE IF NOT EXISTS bookings.flights_archive (LIKE bookings.flights)')
2020-06-20 23:12:09 DEBUG: DELETE FROM flights by flight_id - 2 rows
2020-06-20 23:12:09 DEBUG: INSERT INTO flights_archive - 2 rows
2020-06-20 23:12:09 INFO: flights - END
ለተጠቀሰው ሰንጠረዥ ጥገኞችን ይፈልጉ
>>> from pg_graph.api import PgGraphApi
>>> from pprint import pprint
>>> api = PgGraphApi('config.hw.local.ini')
>>> res = api.get_table_references('flights')
>>> pprint(res)
{'in_refs': {'ticket_flights': [ForeignKey(pk_main='flight_id', pk_ref='flight_id, ticket_no', fk_ref='flight_id')]},
'out_refs': {'aircrafts': [ForeignKey(pk_main='aircraft_code', pk_ref='flight_id', fk_ref='aircraft_code')],
'airports': [ForeignKey(pk_main='airport_code', pk_ref='flight_id', fk_ref='arrival_airport'),
ForeignKey(pk_main='airport_code', pk_ref='flight_id', fk_ref='departure_airport')]}}
ከተጠቀሰው ዋና ቁልፍ ጋር የረድፎች ማጣቀሻዎችን ማግኘት
>>> from pg_graph.api import PgGraphApi
>>> from pprint import pprint
>>> api = PgGraphApi('config.hw.local.ini')
>>> rows = api.get_rows_references('flights', [1,2,3])
>>> pprint(rows)
{1: {'ticket_flights': {'flight_id': [{'flight_id': 1,
'ticket_no': '0005432816945'},
{'flight_id': 1,
'ticket_no': '0005432816941'}]}},
2: {'ticket_flights': {'flight_id': [{'flight_id': 2,
'ticket_no': '0005433101832'},
{'flight_id': 2,
'ticket_no': '0005433101864'},
{'flight_id': 2,
'ticket_no': '0005432919715'}]}},
3: {'ticket_flights': {'flight_id': [{'flight_id': 3,
'ticket_no': '0005432817560'},
{'flight_id': 3,
'ticket_no': '0005432817568'},
{'flight_id': 3,
'ticket_no': '0005432817559'}]}}}
የቤተ መፃህፍቱ ምንጭ ኮድ በ ላይ ይገኛል።
አስተያየቶችን ፣ አስተያየቶችን እና ምክሮችን ለመስጠት ደስተኛ ነኝ።
እዚህ እና በማከማቻው ውስጥ በተቻለ መጠን ጥያቄዎችን ለመመለስ እሞክራለሁ.
ምንጭ: hab.com