Happy Party λ˜λŠ” PostgreSQL10의 νŒŒν‹°μ…”λ‹μ— λŒ€ν•œ μΆ”μ–΅ λͺ‡ 쀄

μ„œλ¬Έ λ˜λŠ” 단면화 아이디어가 μ–΄λ–»κ²Œ νƒ„μƒν–ˆλŠ”μ§€

μ΄μ•ΌκΈ°λŠ” μ—¬κΈ°μ„œ μ‹œμž‘λ©λ‹ˆλ‹€: λͺ¨λ“  것이 μ–΄λ–»κ²Œ μ‹œμž‘λ˜μ—ˆλŠ”μ§€ κΈ°μ–΅ν•˜μ‹­λ‹ˆκΉŒ? λͺ¨λ“  것이 μ²˜μŒμ΄μ—ˆκ³  또 λ‹€μ‹œμ˜€μŠ΅λ‹ˆλ‹€. κ·Έ λ‹Ήμ‹œ 쿼리 μ΅œμ ν™”λ₯Ό μœ„ν•œ 거의 λͺ¨λ“  λ¦¬μ†ŒμŠ€κ°€ μ†Œμ§„λœ ν›„ λ‹€μŒ 질문이 μƒκ²ΌμŠ΅λ‹ˆλ‹€. λ‹€μŒμ€ λ¬΄μ—‡μž…λ‹ˆκΉŒ? 이것이 νŒŒν‹°μ…”λ‹ 아이디어가 λ‚˜μ˜¨ λ°©λ²•μž…λ‹ˆλ‹€.

Happy Party λ˜λŠ” PostgreSQL10의 νŒŒν‹°μ…”λ‹μ— λŒ€ν•œ μΆ”μ–΅ λͺ‡ 쀄

μ„œμ •μ  인 νƒˆμ„  :
λ°”λ‘œ 'κ·Έ μˆœκ°„'μ΄λ‹ˆκΉŒμš”. μ•Œκ³  λ³΄λ‹ˆ 아직 κ°œλ°œλ˜μ§€ μ•Šμ€ μ΅œμ ν™” μ˜ˆλΉ„κ°€ μžˆμ—ˆμŠ΅λ‹ˆλ‹€.. κ³ λ§ˆμ›Œ. asmm 그리고 ν•˜λΈŒλ£¨!

κ·Έλ ‡λ‹€λ©΄ 고객을 ν–‰λ³΅ν•˜κ²Œ λ§Œλ“œλŠ” λ™μ‹œμ— μžμ‹ μ˜ κΈ°μˆ λ„ ν–₯μƒμ‹œν‚¬ 수 μžˆλŠ” 방법은 λ¬΄μ—‡μΌκΉŒμš”?

λͺ¨λ“  것을 μ΅œλŒ€ν•œ λ‹¨μˆœν™”ν•˜κΈ° μœ„ν•΄, λ°μ΄ν„°λ² μ΄μŠ€ μ„±λŠ₯을 근본적으둜 ν–₯μƒμ‹œν‚€λŠ” 방법은 두 κ°€μ§€λΏμž…λ‹ˆλ‹€.
1) ν™•μž₯ 경둜 - λ¦¬μ†ŒμŠ€λ₯Ό 늘리고 ꡬ성을 λ³€κ²½ν•©λ‹ˆλ‹€.
2) 집쀑 경둜 - 쿼리 μ΅œμ ν™”

λ°˜λ³΅ν•©λ‹ˆλ‹€. κ·Έ λ‹Ήμ‹œ 속도 ν–₯상 μš”μ²­μ—μ„œ 무엇을 λ³€κ²½ν•΄μ•Όν• μ§€ 더 이상 λͺ…ν™•ν•˜μ§€ μ•Šμ•˜κΈ° λ•Œλ¬Έμ— κ²½λ‘œκ°€ μ„ νƒλ˜μ—ˆμŠ΅λ‹ˆλ‹€. ν…Œμ΄λΈ” λ””μžμΈ λ³€κ²½.

λ”°λΌμ„œ μ£Όμš” 질문이 μƒκΉλ‹ˆλ‹€. μš°λ¦¬λŠ” 무엇을, μ–΄λ–»κ²Œ λ³€ν™”μ‹œν‚¬ 것인가?

초기 쑰건

첫째, λ‹€μŒκ³Ό 같은 ERDκ°€ μžˆμŠ΅λ‹ˆλ‹€(쑰건적으둜 λ‹¨μˆœν™”λœ λ°©μ‹μœΌλ‘œ ν‘œμ‹œλ¨).
Happy Party λ˜λŠ” PostgreSQL10의 νŒŒν‹°μ…”λ‹μ— λŒ€ν•œ μΆ”μ–΅ λͺ‡ 쀄
μ£Όμš” κΈ°λŠ₯ :

  1. λ‹€λŒ€λ‹€ 관계
  2. ν…Œμ΄λΈ”μ— 이미 잠재적인 νŒŒν‹°μ…˜ ν‚€κ°€ μžˆμŠ΅λ‹ˆλ‹€.

μ›λž˜ μš”μ²­:

SELECT
            p."PARAMETER_ID" as  parameter_id,
            pc."PC_NAME" AS pc_name,
            pc."CUSTOMER_PARTNUMBER" AS customer_partnumber,
            w."LASERMARK" AS lasermark,
            w."LOTID" AS lotid,
            w."REPORTED_VALUE" AS reported_value,
            w."LOWER_SPEC_LIMIT" AS lower_spec_limit,
            w."UPPER_SPEC_LIMIT" AS upper_spec_limit,
            p."TYPE_CALCUL" AS type_calcul,
            s."SHIPMENT_NAME" AS shipment_name,
            s."SHIPMENT_DATE" AS shipment_date,
            extract(year from s."SHIPMENT_DATE") AS year,
            extract(month from s."SHIPMENT_DATE") as month,
            s."REPORT_NAME" AS report_name,
            p."SPARAM_NAME" AS SPARAM_name,
            p."CUSTOMERPARAM_NAME" AS customerparam_name
        FROM data w INNER JOIN shipment s ON s."SHIPMENT_ID" = w."SHIPMENT_ID"
             INNER JOIN parameters p ON p."PARAMETER_ID" = w."PARAMETER_ID"
             INNER JOIN shipment_pc sp ON s."SHIPMENT_ID" = sp."SHIPMENT_ID"
             INNER JOIN pc pc ON pc."PC_ID" = sp."PC_ID"
             INNER JOIN ( SELECT w2."LASERMARK" , MAX(s2."SHIPMENT_DATE") AS "SHIPMENT_DATE"
                          FROM shipment s2 INNER JOIN data w2 ON s2."SHIPMENT_ID" = w2."SHIPMENT_ID" 
                          GROUP BY w2."LASERMARK"
                         ) md ON md."SHIPMENT_DATE" = s."SHIPMENT_DATE" AND md."LASERMARK" = w."LASERMARK"
        WHERE 
             s."SHIPMENT_DATE" >= '2018-07-01' AND s."SHIPMENT_DATE" <= '2018-09-30' ;

ν…ŒμŠ€νŠΈ λ°μ΄ν„°λ² μ΄μŠ€ μ‹€ν–‰ κ²°κ³Ό:
λΉ„μš© : 502 997.55
μ‹€ν–‰ μ‹œκ°„: 505 초.

μš°λ¦¬λŠ” 무엇을 λ΄…λ‹ˆκΉŒ? μ‹œκ°„ 쑰각을 기반으둜 ν•˜λŠ” 일반 μš”μ²­μž…λ‹ˆλ‹€.
κ°€μž₯ κ°„λ‹¨ν•œ 논리적 가정을 λ§Œλ“€μ–΄ λ΄…μ‹œλ‹€: μ‹œκ°„ 쑰각의 μƒ˜ν”Œμ΄ μžˆλ‹€λ©΄ 그것이 μš°λ¦¬μ—κ²Œ 도움이 λ κΉŒμš”? λ§žμŠ΅λ‹ˆλ‹€ - νŒŒν‹°μ…”λ‹.

무엇을 μ„Ήμ…˜μœΌλ‘œ ν• κΉŒμš”?

μ–Έλœ» 보기에 선택은 λΆ„λͺ…ν•©λ‹ˆλ‹€. "SHIPMENT_DATE" ν‚€λ₯Ό μ‚¬μš©ν•˜μ—¬ "배솑" ν…Œμ΄λΈ”μ„ μ„ μ–Έμ μœΌλ‘œ λΆ„ν• ν•˜λŠ” κ²ƒμž…λ‹ˆλ‹€(λ„ˆλ¬΄ μ•žμ„œ λ‚˜κ°„ 것 - κ²°κ΅­ μƒμ‚°μ—μ„œ μ•½κ°„ 잘λͺ»λœ κ²ƒμœΌλ‘œ 판λͺ…λ˜μ—ˆμŠ΅λ‹ˆλ‹€.).

λΆ„ν• ν•˜λŠ” 방법?

이 μ§ˆλ¬Έλ„ λ³„λ‘œ μ–΄λ ΅μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. λ‹€ν–‰νžˆ PostgreSQL 10μ—λŠ” 이제 휴먼 νŒŒν‹°μ…”λ‹ λ©”μ»€λ‹ˆμ¦˜μ΄ μžˆμŠ΅λ‹ˆλ‹€.
κ·Έλž˜μ„œ :

  1. μ†ŒμŠ€ ν…Œμ΄λΈ”μ˜ 덀프 μ €μž₯ - pg_dump μ†ŒμŠ€_ν…Œμ΄λΈ”
  2. 원본 ν…Œμ΄λΈ” μ‚­μ œ - λ“œλ‘­ ν…Œμ΄λΈ” source_table
  3. λ²”μœ„ νŒŒν‹°μ…”λ‹μ„ μ‚¬μš©ν•˜μ—¬ μƒμœ„ ν…Œμ΄λΈ” 생성 - ν…Œμ΄λΈ” source_table 생성
  4. μ„Ήμ…˜ λ§Œλ“€κΈ° - ν…Œμ΄λΈ” source_table 생성, 인덱슀 생성
  5. 1λ‹¨κ³„μ—μ„œ μƒμ„±λœ 덀프λ₯Ό κ°€μ Έμ˜΅λ‹ˆλ‹€. pg_restore

νŒŒν‹°μ…”λ‹μ„ μœ„ν•œ 슀크립트

λ‹¨μˆœμ„±κ³Ό νŽΈμ˜μ„±μ„ μœ„ν•΄ 2,3,4단계λ₯Ό ν•˜λ‚˜μ˜ 슀크립트둜 κ²°ν•©ν–ˆμŠ΅λ‹ˆλ‹€.

κ·Έλž˜μ„œ :
μ†ŒμŠ€ ν…Œμ΄λΈ”μ˜ 덀프 μ €μž₯

pg_dump postgres --file=/dump/shipment.dmp --format=c --table=shipment --verbose > /dump/shipment.log 2>&1

μ†ŒμŠ€ ν…Œμ΄λΈ” μ‚­μ œ + λ²”μœ„ νŒŒν‹°μ…”λ‹μ„ μ‚¬μš©ν•˜μ—¬ μƒμœ„ ν…Œμ΄λΈ” 생성 + νŒŒν‹°μ…˜ 생성

--create_partition_shipment.sql
do language plpgsql $$
declare 
rec_shipment_date RECORD ;
partition_name varchar;
index_name varchar;
current_year varchar ;
current_month varchar ;
begin_year varchar ;
begin_month varchar ;
next_year varchar ;
next_month varchar ;
first_flag boolean ;
i integer ;
begin
  RAISE NOTICE 'CREATE TEMPORARY TABLE FOR SHIPMENT_DATE';
  CREATE TEMP TABLE tmp_shipment_date as select distinct "SHIPMENT_DATE" from shipment order by "SHIPMENT_DATE" ;

  RAISE NOTICE 'DROP TABLE shipment';
  drop table shipment cascade ;
  
  CREATE TABLE public.shipment
  (
    "SHIPMENT_ID" integer NOT NULL DEFAULT nextval('shipment_shipment_id_seq'::regclass),
    "SHIPMENT_NAME" character varying(30) COLLATE pg_catalog."default",
    "SHIPMENT_DATE" timestamp without time zone,
    "REPORT_NAME" character varying(40) COLLATE pg_catalog."default"
  )
  PARTITION BY RANGE ("SHIPMENT_DATE")
  WITH (
      OIDS = FALSE
  )
  TABLESPACE pg_default;

  RAISE NOTICE 'CREATE PARTITIONS FOR TABLE shipment';

  current_year:='0';
  current_month:='0';

  begin_year := '0' ;
  begin_month := '0'  ;
  next_year := '0' ;
  next_month := '0'  ;

  FOR rec_shipment_date IN SELECT * FROM tmp_shipment_date LOOP
      
      RAISE NOTICE 'SHIPMENT_DATE=%',rec_shipment_date."SHIPMENT_DATE";
      
      current_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE");
      current_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE") ; 

      IF to_number(current_month,'99') < 10 THEN
        current_month := '0'||current_month ; 
      END IF ;

      --Init borders
      IF   begin_year = '0' THEN
       first_flag := true ; --first time flag
       begin_year := current_year ;
       begin_month := current_month ;   
   
        IF current_month = '12' THEN
          next_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 year') ;
        ELSE
          next_year := current_year ;
        END IF;
     
       next_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 month') ;

      END IF;

      -- Check current date into borders NOT for First time
      IF to_date( current_year||'.'||current_month, 'YYYY.MM') >= to_date( begin_year||'.'||begin_month, 'YYYY.MM') AND 
         to_date( current_year||'.'||current_month, 'YYYY.MM') < to_date( next_year||'.'||next_month, 'YYYY.MM') AND 
         NOT first_flag 
      THEN
         CONTINUE ; 
      ELSE
       --NEW borders only for second and after time 
       begin_year := current_year ;
       begin_month := current_month ;   
   
        IF current_month = '12' THEN
          next_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 year') ;
        ELSE
          next_year := current_year ;
        END IF;
     
       next_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 month') ;

      END IF;      

      partition_name := 'shipment_shipment_date_'||begin_year||'-'||begin_month||'-01-'|| next_year||'-'||next_month||'-01'  ;
 
     EXECUTE format('CREATE TABLE ' || quote_ident(partition_name) || ' PARTITION OF shipment FOR VALUES FROM ( %L ) TO ( %L )  ' , current_year||'-'||current_month||'-01' , next_year||'-'||next_month||'-01'  ) ; 

      index_name := partition_name||'_shipment_id_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID") TABLESPACE pg_default ' ) ; 

      --Drop first time flag
      first_flag := false ;
   
  END LOOP;

end
$$;

덀프 κ°€μ Έμ˜€κΈ°

pg_restore -d postgres --data-only --format=c --table=shipment --verbose  shipment.dmp > /tmp/data_dump/shipment_restore.log 2>&1

νŒŒν‹°μ…”λ‹ κ²°κ³Ό 확인

κ·Έ κ²°κ³Ό μš°λ¦¬λŠ” 무엇을 κ°–κ²Œ λ˜μ—ˆμŠ΅λ‹ˆκΉŒ? μ‹€ν–‰ κ³„νšμ˜ 전체 ν…μŠ€νŠΈλŠ” λ°©λŒ€ν•˜κ³  μ§€λ£¨ν•˜λ―€λ‘œ μ΅œμ’… 숫자둜 μ œν•œν•˜λŠ” 것이 κ°€λŠ₯ν•©λ‹ˆλ‹€.

ν–ˆλ‹€

λΉ„μš© : 502 997.55
μ‹€ν–‰ μ‹œκ°„: 505 초.

κ·Έκ²ƒμ€λ˜μ—ˆλ‹€

λΉ„μš© : 77 872.36
μ‹€ν–‰ μ‹œκ°„: 79 초.

κ½€ 쒋은 κ²°κ³Όμž…λ‹ˆλ‹€. λΉ„μš© 및 μ‹€ν–‰ μ‹œκ°„μ΄ λ‹¨μΆ•λ©λ‹ˆλ‹€. λ”°λΌμ„œ νŒŒν‹°μ…”λ‹μ„ μ‚¬μš©ν•˜λ©΄ μ˜ˆμƒν•œ 효과λ₯Ό 얻을 수 있으며 일반적으둜 λ†€λž„ 일이 μ•„λ‹™λ‹ˆλ‹€.

고객을 ν–‰λ³΅ν•˜κ²Œ ν•˜λΌ

ν…ŒμŠ€νŠΈ κ²°κ³ΌλŠ” κ²€ν† λ₯Ό μœ„ν•΄ κ³ κ°μ—κ²Œ μ œκ³΅λ˜μ—ˆμŠ΅λ‹ˆλ‹€. 그리고 이λ₯Ό κ²€ν† ν•œ ν›„ λ‹€μ†Œ μ˜ˆμƒμΉ˜ λͺ»ν•œ 결둠을 λ‚΄λ ΈμŠ΅λ‹ˆλ‹€. β€œμ’‹μŠ΅λ‹ˆλ‹€. β€œλ°μ΄ν„°β€ ν…Œμ΄λΈ”μ„ λΆ„ν• ν•˜μ„Έμš”.”

예, ν•˜μ§€λ§Œ μ™„μ „νžˆ λ‹€λ₯Έ "배솑" ν…Œμ΄λΈ”μ„ μ‘°μ‚¬ν–ˆμŠ΅λ‹ˆλ‹€. "데이터" ν…Œμ΄λΈ”μ—λŠ” "SHIPMENT_DATE" ν•„λ“œκ°€ μ—†μŠ΅λ‹ˆλ‹€.

문제 μ—†μŠ΅λ‹ˆλ‹€. μΆ”κ°€ν•˜κ³  λ³€κ²½ν•˜μ„Έμš”. κ°€μž₯ μ€‘μš”ν•œ 것은 고객이 결과에 λ§Œμ‘±ν•œλ‹€λŠ” 것이며 κ΅¬ν˜„ μ„ΈλΆ€ 사항은 νŠΉλ³„νžˆ μ€‘μš”ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.

메인 ν…Œμ΄λΈ” "데이터" νŒŒν‹°μ…”λ‹

일반적으둜 νŠΉλ³„ν•œ 어렀움은 λ°œμƒν•˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€. λ¬Όλ‘  νŒŒν‹°μ…”λ‹ μ•Œκ³ λ¦¬μ¦˜μ€ λ‹€μ†Œ λ³€κ²½λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

'데이터' ν…Œμ΄λΈ”μ— 'SHIPMENT_DATA' μ—΄ μΆ”κ°€

psql -h хост -U Π±Π°Π·Π° -d ΡŽΠ·Π΅Ρ€
=> ALTER TABLE data ADD COLUMN "SHIPMENT_DATE" timestamp without time zone ;

"data" ν…Œμ΄λΈ”μ˜ "SHIPMENT_DATA" μ—΄μ˜ 값을 "shipment" ν…Œμ΄λΈ”μ˜ λ™μΌν•œ μ΄λ¦„μ˜ μ—΄ κ°’μœΌλ‘œ μ±„μ›λ‹ˆλ‹€.

-----------------------------
--update_data.sql
--updating for altered table "data" to values of "shipment_data" from the table "shipment"
--version 1.0
do language plpgsql $$
declare 
rec_shipment_data RECORD ;
shipment_date timestamp without time zone ; 
row_count integer ;
total_rows integer ;
begin

  select count(*) into total_rows from shipment ; 
  RAISE NOTICE 'Total %',total_rows;
  row_count:= 0 ;

  FOR rec_shipment_data IN SELECT * FROM shipment LOOP

   update data set "SHIPMENT_DATE" = rec_shipment_data."SHIPMENT_DATE" where "SHIPMENT_ID" = rec_shipment_data."SHIPMENT_ID";
   
   row_count:=  row_count +1 ;
   RAISE NOTICE 'row count = % , from %',row_count,total_rows;
  END LOOP;

end
$$;

"데이터" ν…Œμ΄λΈ” 덀프 μ €μž₯

pg_dump postgres --file=/dump/data.dmp --format=c --table=data --verbose > /dump/data.log 2>&1</source

λΆ„ν• λœ ν…Œμ΄λΈ” 'data'λ₯Ό λ‹€μ‹œ μƒμ„±ν•©λ‹ˆλ‹€.

--create_partition_data.sql
--create partitions for the table "wafer data" by range column "shipment_data" with one month duration
--version 1.0
do language plpgsql $$
declare 
rec_shipment_date RECORD ;
partition_name varchar;
index_name varchar;
current_year varchar ;
current_month varchar ;
begin_year varchar ;
begin_month varchar ;
next_year varchar ;
next_month varchar ;
first_flag boolean ;
i integer ;

begin

  RAISE NOTICE 'CREATE TEMPORARY TABLE FOR SHIPMENT_DATE';
  CREATE TEMP TABLE tmp_shipment_date as select distinct "SHIPMENT_DATE" from shipment order by "SHIPMENT_DATE" ;


  RAISE NOTICE 'DROP TABLE data';
  drop table data cascade ;


  RAISE NOTICE 'CREATE PARTITIONED TABLE data';
  
  CREATE TABLE public.data
  (
    "RUN_ID" integer,
    "LASERMARK" character varying(20) COLLATE pg_catalog."default" NOT NULL,
    "LOTID" character varying(80) COLLATE pg_catalog."default",
    "SHIPMENT_ID" integer NOT NULL,
    "PARAMETER_ID" integer NOT NULL,
    "INTERNAL_VALUE" character varying(75) COLLATE pg_catalog."default",
    "REPORTED_VALUE" character varying(75) COLLATE pg_catalog."default",
    "LOWER_SPEC_LIMIT" numeric,
    "UPPER_SPEC_LIMIT" numeric , 
    "SHIPMENT_DATE" timestamp without time zone
  )
  PARTITION BY RANGE ("SHIPMENT_DATE")
  WITH (
    OIDS = FALSE
  )
  TABLESPACE pg_default ;


  RAISE NOTICE 'CREATE PARTITIONS FOR TABLE data';

  current_year:='0';
  current_month:='0';

  begin_year := '0' ;
  begin_month := '0'  ;
  next_year := '0' ;
  next_month := '0'  ;
  i := 1;

  FOR rec_shipment_date IN SELECT * FROM tmp_shipment_date LOOP
      
      RAISE NOTICE 'SHIPMENT_DATE=%',rec_shipment_date."SHIPMENT_DATE";
      
      current_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE");
      current_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE") ; 

      --Init borders
      IF   begin_year = '0' THEN
       RAISE NOTICE '***Init borders';
       first_flag := true ; --first time flag
       begin_year := current_year ;
       begin_month := current_month ;   
   
        IF current_month = '12' THEN
          next_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 year') ;
        ELSE
          next_year := current_year ;
        END IF;
     
       next_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 month') ;

      END IF;

--      RAISE NOTICE 'current_year=% , current_month=% ',current_year,current_month;
--      RAISE NOTICE 'begin_year=% , begin_month=% ',begin_year,begin_month;
--      RAISE NOTICE 'next_year=% , next_month=% ',next_year,next_month;

      -- Check current date into borders NOT for First time

      RAISE NOTICE 'Current data = %',to_char( to_date( current_year||'.'||current_month, 'YYYY.MM'), 'YYYY.MM');
      RAISE NOTICE 'Begin data = %',to_char( to_date( begin_year||'.'||begin_month, 'YYYY.MM'), 'YYYY.MM');
      RAISE NOTICE 'Next data = %',to_char( to_date( next_year||'.'||next_month, 'YYYY.MM'), 'YYYY.MM');

      IF to_date( current_year||'.'||current_month, 'YYYY.MM') >= to_date( begin_year||'.'||begin_month, 'YYYY.MM') AND 
         to_date( current_year||'.'||current_month, 'YYYY.MM') < to_date( next_year||'.'||next_month, 'YYYY.MM') AND 
         NOT first_flag 
      THEN
         RAISE NOTICE '***CONTINUE';
         CONTINUE ; 
      ELSE
       --NEW borders only for second and after time 
       RAISE NOTICE '***NEW BORDERS';
       begin_year := current_year ;
       begin_month := current_month ;   
   
        IF current_month = '12' THEN
          next_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 year') ;
        ELSE
          next_year := current_year ;
        END IF;
     
       next_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 month') ;


      END IF;      

      IF to_number(current_month,'99') < 10 THEN
        current_month := '0'||current_month ; 
      END IF ;

      IF to_number(begin_month,'99') < 10 THEN
        begin_month := '0'||begin_month ; 
      END IF ;

      IF to_number(next_month,'99') < 10 THEN
        next_month := '0'||next_month ; 
      END IF ;

      RAISE NOTICE 'current_year=% , current_month=% ',current_year,current_month;
      RAISE NOTICE 'begin_year=% , begin_month=% ',begin_year,begin_month;
      RAISE NOTICE 'next_year=% , next_month=% ',next_year,next_month;

      partition_name := 'data_'||begin_year||begin_month||'01_'||next_year||next_month||'01'  ;

      RAISE NOTICE 'PARTITION NUMBER % , TABLE NAME =%',i , partition_name;
      
      EXECUTE format('CREATE TABLE ' || quote_ident(partition_name) || ' PARTITION OF data FOR VALUES FROM ( %L ) TO ( %L )  ' , begin_year||'-'||begin_month||'-01' , next_year||'-'||next_month||'-01'  ) ; 

      index_name := partition_name||'_shipment_id_parameter_id_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID", "PARAMETER_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_lasermark_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("LASERMARK" COLLATE pg_catalog."default") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_shipment_id_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_parameter_id_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("PARAMETER_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_shipment_date_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_DATE") TABLESPACE pg_default ' ) ; 

      --Drop first time flag
      first_flag := false ;

  END LOOP;
end
$$;

3λ‹¨κ³„μ—μ„œ μƒμ„±λœ 덀프λ₯Ό λ‘œλ“œν•©λ‹ˆλ‹€.

pg_restore -h хост -ΡŽΠ·Π΅Ρ€ -d Π±Π°Π·Π° --data-only --format=c --table=data --verbose  data.dmp > data_restore.log 2>&1

였래된 데이터에 λŒ€ν•΄ λ³„λ„μ˜ μ„Ήμ…˜ λ§Œλ“€κΈ°

---------------------------------------------------
--create_partition_for_old_dates.sql
--create partitions for keeping old dates 
--version 1.0
do language plpgsql $$
declare 
rec_shipment_date RECORD ;
partition_name varchar;
index_name varchar;

begin

      SELECT min("SHIPMENT_DATE") AS min_date INTO rec_shipment_date from data ;

      RAISE NOTICE 'Old date is %',rec_shipment_date.min_date ;

      partition_name := 'data_old_dates'  ;

      RAISE NOTICE 'PARTITION NAME IS %',partition_name;

      EXECUTE format('CREATE TABLE ' || quote_ident(partition_name) || ' PARTITION OF data FOR VALUES FROM ( %L ) TO ( %L )  ' , '1900-01-01' , 
              to_char( rec_shipment_date.min_date,'YYYY')||'-'||to_char(rec_shipment_date.min_date,'MM')||'-01'  ) ; 

      index_name := partition_name||'_shipment_id_parameter_id_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID", "PARAMETER_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_lasermark_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("LASERMARK" COLLATE pg_catalog."default") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_shipment_id_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_parameter_id_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("PARAMETER_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_shipment_date_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_DATE") TABLESPACE pg_default ' ) ; 

end
$$;

μ΅œμ’… κ²°κ³Ό:

ν–ˆλ‹€
λΉ„μš© : 502 997.55
μ‹€ν–‰ μ‹œκ°„: 505 초.

κ·Έκ²ƒμ€λ˜μ—ˆλ‹€
λΉ„μš© : 68 533.70
μ‹€ν–‰ μ‹œκ°„: 69 초

κ°€μΉ˜ 있고, κ½€ κ°€μΉ˜κ°€ μžˆμŠ΅λ‹ˆλ‹€. 그리고 κ·Έ κ³Όμ •μ—μ„œ μš°λ¦¬λŠ” PostgreSQL 10의 νŒŒν‹°μ…”λ‹ λ©”μ»€λ‹ˆμ¦˜μ„ μ–΄λŠ 정도 λ§ˆμŠ€ν„°ν•  수 μžˆμ—ˆμŠ΅λ‹ˆλ‹€. μ΄λŠ” ν›Œλ₯­ν•œ κ²°κ³Όμž…λ‹ˆλ‹€.

μ„œμ •μ  침랡

더 μž˜ν•  수 μžˆλ‚˜μš”? 예, κ°€λŠ₯ν•©λ‹ˆλ‹€!이λ₯Ό μœ„ν•΄μ„œλŠ” MATERIALIZED VIEWλ₯Ό μ‚¬μš©ν•΄μ•Ό ν•©λ‹ˆλ‹€.
κ΅¬μ²΄ν™”λœ λ·° 생성 LASERMARK_VIEW

CREATE MATERIALIZED VIEW LASERMARK_VIEW 
AS
SELECT w."LASERMARK" , MAX(s."SHIPMENT_DATE") AS "SHIPMENT_DATE"
FROM shipment s INNER JOIN data w ON s."SHIPMENT_ID" = w."SHIPMENT_ID" 
GROUP BY w."LASERMARK" ;

CREATE INDEX lasermark_vw_shipment_date_ind on lasermark_view USING btree ("SHIPMENT_DATE") TABLESPACE pg_default;
analyze lasermark_view ;

λ‹€μ‹œ ν•œ 번 μš”μ²­μ„ λ‹€μ‹œ μž‘μ„±ν•©λ‹ˆλ‹€.
κ΅¬μ²΄ν™”λœ λ·°λ₯Ό μ‚¬μš©ν•œ 쿼리

SELECT
            p."PARAMETER_ID" as  parameter_id,
            pc."PC_NAME" AS pc_name,
            pc."CUSTOMER_PARTNUMBER" AS customer_partnumber,
            w."LASERMARK" AS lasermark,
            w."LOTID" AS lotid,
            w."REPORTED_VALUE" AS reported_value,
            w."LOWER_SPEC_LIMIT" AS lower_spec_limit,
            w."UPPER_SPEC_LIMIT" AS upper_spec_limit,
            p."TYPE_CALCUL" AS type_calcul,
            s."SHIPMENT_NAME" AS shipment_name,
            s."SHIPMENT_DATE" AS shipment_date,
            extract(year from s."SHIPMENT_DATE") AS year,
            extract(month from s."SHIPMENT_DATE") as month,
            s."REPORT_NAME" AS report_name,
            p."STC_NAME" AS STC_name,
            p."CUSTOMERPARAM_NAME" AS customerparam_name
        FROM data w INNER JOIN shipment s ON s."SHIPMENT_ID" = w."SHIPMENT_ID"
             INNER JOIN parameters p ON p."PARAMETER_ID" = w."PARAMETER_ID"
             INNER JOIN shipment_pc sp ON s."SHIPMENT_ID" = sp."SHIPMENT_ID"
             INNER JOIN pc pc ON pc."PC_ID" = sp."PC_ID"
             INNER JOIN LASERMARK_VIEW md ON md."SHIPMENT_DATE" = s."SHIPMENT_DATE" AND md."LASERMARK" = w."LASERMARK"
        WHERE 
              s."SHIPMENT_DATE" >= '2018-07-01' AND s."SHIPMENT_DATE" <= '2018-09-30';

그리고 μš°λ¦¬λŠ” 또 λ‹€λ₯Έ κ²°κ³Όλ₯Ό μ–»μŠ΅λ‹ˆλ‹€:
ν–ˆλ‹€
λΉ„μš© : 502 997.55
μ‹€ν–‰ μ‹œκ°„: 505 초

κ·Έκ²ƒμ€λ˜μ—ˆλ‹€
λΉ„μš© : 42 481.16
μ‹€ν–‰ μ‹œκ°„: 43 초.

λ¬Όλ‘  κ·ΈλŸ¬ν•œ μœ λ§ν•œ κ²°κ³ΌλŠ” κΈ°λ§Œμ μ΄μ§€λ§Œ μ•„μ΄λ””μ–΄λŠ” μƒˆλ‘œμ›Œμ§ˆ ν•„μš”κ°€ μžˆμŠ΅λ‹ˆλ‹€. λ”°λΌμ„œ 데이터λ₯Ό μˆ˜μ‹ ν•˜λŠ” 데 κ±Έλ¦¬λŠ” 총 μ‹œκ°„μ€ λ³„λ‘œ 도움이 λ˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. κ·ΈλŸ¬λ‚˜ μ‹€ν—˜μ μœΌλ‘œλŠ” 맀우 ν₯λ―Έλ‘­μŠ΅λ‹ˆλ‹€.

μ‹€μ œλ‘œ 결과적으둜 λ‹€μ‹œ ν•œ 번 κ°μ‚¬λ“œλ¦½λ‹ˆλ‹€. asmm 그리고 ν•˜λΈŒλ£¨!- 쿼리가 λ”μš± ν–₯상될 수 μžˆμŠ΅λ‹ˆλ‹€.

사후

κ·Έλž˜μ„œ 고객은 λ§Œμ‘±ν•©λ‹ˆλ‹€. 그리고 ν•„μš”ν•œ 상황을 ν™œμš©ν•˜μ„Έμš”.

μƒˆ μž‘μ—…: μ‹¬ν™”ν•˜κ³  ν™•μž₯ν•˜κΈ° μœ„ν•΄ 무엇을 생각해 λ‚Ό 수 μžˆμŠ΅λ‹ˆκΉŒ?

그리고 κΈ°μ–΅λ‚˜λŠ” 것은, μš°λ¦¬λŠ” PostgreSQL λ°μ΄ν„°λ² μ΄μŠ€λ₯Ό λͺ¨λ‹ˆν„°λ§ν•˜μ§€ μ•ŠλŠ”λ‹€λŠ” κ²ƒμž…λ‹ˆλ‹€.

μ‹€μ œλ‘œ AWSμ—λŠ” Cloud Watch ν˜•νƒœμ˜ λͺ¨λ‹ˆν„°λ§μ΄ μ—¬μ „νžˆ μžˆμŠ΅λ‹ˆλ‹€. ν•˜μ§€λ§Œ DBAμ—κ²Œ μ΄λŸ¬ν•œ λͺ¨λ‹ˆν„°λ§μ˜ 이점은 λ¬΄μ—‡μž…λ‹ˆκΉŒ? 일반적으둜 거의 μ—†μŠ΅λ‹ˆλ‹€.

μžμ‹ μ—κ²Œ μœ μš©ν•˜κ³  ν₯미둜운 일을 ν•  수 μžˆλŠ” κΈ°νšŒκ°€ μžˆλ‹€λ©΄, 이 기회λ₯Ό ν™œμš©ν•˜μ§€ μ•Šμ„ 수 μ—†μŠ΅λ‹ˆλ‹€...
Mbo

Happy Party λ˜λŠ” PostgreSQL10의 νŒŒν‹°μ…”λ‹μ— λŒ€ν•œ μΆ”μ–΅ λͺ‡ 쀄

이것이 μš°λ¦¬κ°€ κ°€μž₯ ν₯미둜운 뢀뢄에 λ„λ‹¬ν•˜λŠ” λ°©λ²•μž…λ‹ˆλ‹€:

3λ…„ 2018μ›” XNUMX일.
PostgreSQL 쿼리 μ„±λŠ₯을 λͺ¨λ‹ˆν„°λ§ν•˜κΈ° μœ„ν•΄ μ‚¬μš© κ°€λŠ₯ν•œ κΈ°λŠ₯에 λŒ€ν•œ 연ꡬλ₯Ό μ‹œμž‘ν•˜κΈ°λ‘œ κ²°μ •ν•©λ‹ˆλ‹€.

κ·ΈλŸ¬λ‚˜ 그것은 μ™„μ „νžˆ λ‹€λ₯Έ μ΄μ•ΌκΈ°μž…λ‹ˆλ‹€.

계속…

좜처 : habr.com

μ½”λ©˜νŠΈλ₯Ό μΆ”κ°€