Happy Party ๋˜๋Š” PostgreSQL10์˜ ํŒŒํ‹ฐ์…”๋‹์— ๋Œ€ํ•œ ์ถ”์–ต ๋ช‡ ์ค„

์„œ๋ฌธ ๋˜๋Š” ๋‹จ๋ฉดํ™” ์•„์ด๋””์–ด๊ฐ€ ์–ด๋–ป๊ฒŒ ํƒ„์ƒํ–ˆ๋Š”์ง€

์ด์•ผ๊ธฐ๋Š” ์—ฌ๊ธฐ์„œ ์‹œ์ž‘๋ฉ๋‹ˆ๋‹ค: ๋ชจ๋“  ๊ฒƒ์ด ์–ด๋–ป๊ฒŒ ์‹œ์ž‘๋˜์—ˆ๋Š”์ง€ ๊ธฐ์–ตํ•˜์‹ญ๋‹ˆ๊นŒ? ๋ชจ๋“  ๊ฒƒ์ด ์ฒ˜์Œ์ด์—ˆ๊ณ  ๋˜ ๋‹ค์‹œ์˜€์Šต๋‹ˆ๋‹ค. ๊ทธ ๋‹น์‹œ ์ฟผ๋ฆฌ ์ตœ์ ํ™”๋ฅผ ์œ„ํ•œ ๊ฑฐ์˜ ๋ชจ๋“  ๋ฆฌ์†Œ์Šค๊ฐ€ ์†Œ์ง„๋œ ํ›„ ๋‹ค์Œ ์งˆ๋ฌธ์ด ์ƒ๊ฒผ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ์€ ๋ฌด์—‡์ž…๋‹ˆ๊นŒ? ์ด๊ฒƒ์ด ํŒŒํ‹ฐ์…”๋‹ ์•„์ด๋””์–ด๊ฐ€ ๋‚˜์˜จ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค.

Happy Party ๋˜๋Š” PostgreSQL10์˜ ํŒŒํ‹ฐ์…”๋‹์— ๋Œ€ํ•œ ์ถ”์–ต ๋ช‡ ์ค„

์„œ์ •์  ์ธ ํƒˆ์„  :
๋ฐ”๋กœ '๊ทธ ์ˆœ๊ฐ„'์ด๋‹ˆ๊นŒ์š”. ์•Œ๊ณ  ๋ณด๋‹ˆ ์•„์ง ๊ฐœ๋ฐœ๋˜์ง€ ์•Š์€ ์ตœ์ ํ™” ์˜ˆ๋น„๊ฐ€ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.. ๊ณ ๋งˆ์›Œ. asmm ๊ทธ๋ฆฌ๊ณ  ํ•˜๋ธŒ๋ฃจ!

๊ทธ๋ ‡๋‹ค๋ฉด ๊ณ ๊ฐ์„ ํ–‰๋ณตํ•˜๊ฒŒ ๋งŒ๋“œ๋Š” ๋™์‹œ์— ์ž์‹ ์˜ ๊ธฐ์ˆ ๋„ ํ–ฅ์ƒ์‹œํ‚ฌ ์ˆ˜ ์žˆ๋Š” ๋ฐฉ๋ฒ•์€ ๋ฌด์—‡์ผ๊นŒ์š”?

๋ชจ๋“  ๊ฒƒ์„ ์ตœ๋Œ€ํ•œ ๋‹จ์ˆœํ™”ํ•˜๊ธฐ ์œ„ํ•ด, ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ์„ฑ๋Šฅ์„ ๊ทผ๋ณธ์ ์œผ๋กœ ํ–ฅ์ƒ์‹œํ‚ค๋Š” ๋ฐฉ๋ฒ•์€ ๋‘ ๊ฐ€์ง€๋ฟ์ž…๋‹ˆ๋‹ค.
1) ํ™•์žฅ ๊ฒฝ๋กœ - ๋ฆฌ์†Œ์Šค๋ฅผ ๋Š˜๋ฆฌ๊ณ  ๊ตฌ์„ฑ์„ ๋ณ€๊ฒฝํ•ฉ๋‹ˆ๋‹ค.
2) ์ง‘์ค‘ ๊ฒฝ๋กœ - ์ฟผ๋ฆฌ ์ตœ์ ํ™”

๋ฐ˜๋ณตํ•ฉ๋‹ˆ๋‹ค. ๊ทธ ๋‹น์‹œ ์†๋„ ํ–ฅ์ƒ ์š”์ฒญ์—์„œ ๋ฌด์—‡์„ ๋ณ€๊ฒฝํ•ด์•ผํ• ์ง€ ๋” ์ด์ƒ ๋ช…ํ™•ํ•˜์ง€ ์•Š์•˜๊ธฐ ๋•Œ๋ฌธ์— ๊ฒฝ๋กœ๊ฐ€ ์„ ํƒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ํ…Œ์ด๋ธ” ๋””์ž์ธ ๋ณ€๊ฒฝ.

๋”ฐ๋ผ์„œ ์ฃผ์š” ์งˆ๋ฌธ์ด ์ƒ๊น๋‹ˆ๋‹ค. ์šฐ๋ฆฌ๋Š” ๋ฌด์—‡์„, ์–ด๋–ป๊ฒŒ ๋ณ€ํ™”์‹œํ‚ฌ ๊ฒƒ์ธ๊ฐ€?

์ดˆ๊ธฐ ์กฐ๊ฑด

์ฒซ์งธ, ๋‹ค์Œ๊ณผ ๊ฐ™์€ ERD๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค(์กฐ๊ฑด์ ์œผ๋กœ ๋‹จ์ˆœํ™”๋œ ๋ฐฉ์‹์œผ๋กœ ํ‘œ์‹œ๋จ).
Happy Party ๋˜๋Š” PostgreSQL10์˜ ํŒŒํ‹ฐ์…”๋‹์— ๋Œ€ํ•œ ์ถ”์–ต ๋ช‡ ์ค„
์ฃผ์š” ๊ธฐ๋Šฅ :

  1. ๋‹ค๋Œ€๋‹ค ๊ด€๊ณ„
  2. ํ…Œ์ด๋ธ”์— ์ด๋ฏธ ์ž ์žฌ์ ์ธ ํŒŒํ‹ฐ์…˜ ํ‚ค๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

์›๋ž˜ ์š”์ฒญ:

SELECT
            p."PARAMETER_ID" as  parameter_id,
            pc."PC_NAME" AS pc_name,
            pc."CUSTOMER_PARTNUMBER" AS customer_partnumber,
            w."LASERMARK" AS lasermark,
            w."LOTID" AS lotid,
            w."REPORTED_VALUE" AS reported_value,
            w."LOWER_SPEC_LIMIT" AS lower_spec_limit,
            w."UPPER_SPEC_LIMIT" AS upper_spec_limit,
            p."TYPE_CALCUL" AS type_calcul,
            s."SHIPMENT_NAME" AS shipment_name,
            s."SHIPMENT_DATE" AS shipment_date,
            extract(year from s."SHIPMENT_DATE") AS year,
            extract(month from s."SHIPMENT_DATE") as month,
            s."REPORT_NAME" AS report_name,
            p."SPARAM_NAME" AS SPARAM_name,
            p."CUSTOMERPARAM_NAME" AS customerparam_name
        FROM data w INNER JOIN shipment s ON s."SHIPMENT_ID" = w."SHIPMENT_ID"
             INNER JOIN parameters p ON p."PARAMETER_ID" = w."PARAMETER_ID"
             INNER JOIN shipment_pc sp ON s."SHIPMENT_ID" = sp."SHIPMENT_ID"
             INNER JOIN pc pc ON pc."PC_ID" = sp."PC_ID"
             INNER JOIN ( SELECT w2."LASERMARK" , MAX(s2."SHIPMENT_DATE") AS "SHIPMENT_DATE"
                          FROM shipment s2 INNER JOIN data w2 ON s2."SHIPMENT_ID" = w2."SHIPMENT_ID" 
                          GROUP BY w2."LASERMARK"
                         ) md ON md."SHIPMENT_DATE" = s."SHIPMENT_DATE" AND md."LASERMARK" = w."LASERMARK"
        WHERE 
             s."SHIPMENT_DATE" >= '2018-07-01' AND s."SHIPMENT_DATE" <= '2018-09-30' ;

ํ…Œ์ŠคํŠธ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ์‹คํ–‰ ๊ฒฐ๊ณผ:
๋น„์šฉ : 502 997.55
์‹คํ–‰ ์‹œ๊ฐ„: 505 ์ดˆ.

์šฐ๋ฆฌ๋Š” ๋ฌด์—‡์„ ๋ด…๋‹ˆ๊นŒ? ์‹œ๊ฐ„ ์กฐ๊ฐ์„ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•˜๋Š” ์ผ๋ฐ˜ ์š”์ฒญ์ž…๋‹ˆ๋‹ค.
๊ฐ€์žฅ ๊ฐ„๋‹จํ•œ ๋…ผ๋ฆฌ์  ๊ฐ€์ •์„ ๋งŒ๋“ค์–ด ๋ด…์‹œ๋‹ค: ์‹œ๊ฐ„ ์กฐ๊ฐ์˜ ์ƒ˜ํ”Œ์ด ์žˆ๋‹ค๋ฉด ๊ทธ๊ฒƒ์ด ์šฐ๋ฆฌ์—๊ฒŒ ๋„์›€์ด ๋ ๊นŒ์š”? ๋งž์Šต๋‹ˆ๋‹ค - ํŒŒํ‹ฐ์…”๋‹.

๋ฌด์—‡์„ ์„น์…˜์œผ๋กœ ํ• ๊นŒ์š”?

์–ธ๋œป ๋ณด๊ธฐ์— ์„ ํƒ์€ ๋ถ„๋ช…ํ•ฉ๋‹ˆ๋‹ค. "SHIPMENT_DATE" ํ‚ค๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ "๋ฐฐ์†ก" ํ…Œ์ด๋ธ”์„ ์„ ์–ธ์ ์œผ๋กœ ๋ถ„ํ• ํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค(๋„ˆ๋ฌด ์•ž์„œ ๋‚˜๊ฐ„ ๊ฒƒ - ๊ฒฐ๊ตญ ์ƒ์‚ฐ์—์„œ ์•ฝ๊ฐ„ ์ž˜๋ชป๋œ ๊ฒƒ์œผ๋กœ ํŒ๋ช…๋˜์—ˆ์Šต๋‹ˆ๋‹ค.).

๋ถ„ํ• ํ•˜๋Š” ๋ฐฉ๋ฒ•?

์ด ์งˆ๋ฌธ๋„ ๋ณ„๋กœ ์–ด๋ ต์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋‹คํ–‰ํžˆ PostgreSQL 10์—๋Š” ์ด์ œ ํœด๋จผ ํŒŒํ‹ฐ์…”๋‹ ๋ฉ”์ปค๋‹ˆ์ฆ˜์ด ์žˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋ž˜์„œ :

  1. ์†Œ์Šค ํ…Œ์ด๋ธ”์˜ ๋คํ”„ ์ €์žฅ - pg_dump ์†Œ์Šค_ํ…Œ์ด๋ธ”
  2. ์›๋ณธ ํ…Œ์ด๋ธ” ์‚ญ์ œ - ๋“œ๋กญ ํ…Œ์ด๋ธ” source_table
  3. ๋ฒ”์œ„ ํŒŒํ‹ฐ์…”๋‹์„ ์‚ฌ์šฉํ•˜์—ฌ ์ƒ์œ„ ํ…Œ์ด๋ธ” ์ƒ์„ฑ - ํ…Œ์ด๋ธ” source_table ์ƒ์„ฑ
  4. ์„น์…˜ ๋งŒ๋“ค๊ธฐ - ํ…Œ์ด๋ธ” source_table ์ƒ์„ฑ, ์ธ๋ฑ์Šค ์ƒ์„ฑ
  5. 1๋‹จ๊ณ„์—์„œ ์ƒ์„ฑ๋œ ๋คํ”„๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค. pg_restore

ํŒŒํ‹ฐ์…”๋‹์„ ์œ„ํ•œ ์Šคํฌ๋ฆฝํŠธ

๋‹จ์ˆœ์„ฑ๊ณผ ํŽธ์˜์„ฑ์„ ์œ„ํ•ด 2,3,4๋‹จ๊ณ„๋ฅผ ํ•˜๋‚˜์˜ ์Šคํฌ๋ฆฝํŠธ๋กœ ๊ฒฐํ•ฉํ–ˆ์Šต๋‹ˆ๋‹ค.

๊ทธ๋ž˜์„œ :
์†Œ์Šค ํ…Œ์ด๋ธ”์˜ ๋คํ”„ ์ €์žฅ

pg_dump postgres --file=/dump/shipment.dmp --format=c --table=shipment --verbose > /dump/shipment.log 2>&1

์†Œ์Šค ํ…Œ์ด๋ธ” ์‚ญ์ œ + ๋ฒ”์œ„ ํŒŒํ‹ฐ์…”๋‹์„ ์‚ฌ์šฉํ•˜์—ฌ ์ƒ์œ„ ํ…Œ์ด๋ธ” ์ƒ์„ฑ + ํŒŒํ‹ฐ์…˜ ์ƒ์„ฑ

--create_partition_shipment.sql
do language plpgsql $$
declare 
rec_shipment_date RECORD ;
partition_name varchar;
index_name varchar;
current_year varchar ;
current_month varchar ;
begin_year varchar ;
begin_month varchar ;
next_year varchar ;
next_month varchar ;
first_flag boolean ;
i integer ;
begin
  RAISE NOTICE 'CREATE TEMPORARY TABLE FOR SHIPMENT_DATE';
  CREATE TEMP TABLE tmp_shipment_date as select distinct "SHIPMENT_DATE" from shipment order by "SHIPMENT_DATE" ;

  RAISE NOTICE 'DROP TABLE shipment';
  drop table shipment cascade ;
  
  CREATE TABLE public.shipment
  (
    "SHIPMENT_ID" integer NOT NULL DEFAULT nextval('shipment_shipment_id_seq'::regclass),
    "SHIPMENT_NAME" character varying(30) COLLATE pg_catalog."default",
    "SHIPMENT_DATE" timestamp without time zone,
    "REPORT_NAME" character varying(40) COLLATE pg_catalog."default"
  )
  PARTITION BY RANGE ("SHIPMENT_DATE")
  WITH (
      OIDS = FALSE
  )
  TABLESPACE pg_default;

  RAISE NOTICE 'CREATE PARTITIONS FOR TABLE shipment';

  current_year:='0';
  current_month:='0';

  begin_year := '0' ;
  begin_month := '0'  ;
  next_year := '0' ;
  next_month := '0'  ;

  FOR rec_shipment_date IN SELECT * FROM tmp_shipment_date LOOP
      
      RAISE NOTICE 'SHIPMENT_DATE=%',rec_shipment_date."SHIPMENT_DATE";
      
      current_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE");
      current_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE") ; 

      IF to_number(current_month,'99') < 10 THEN
        current_month := '0'||current_month ; 
      END IF ;

      --Init borders
      IF   begin_year = '0' THEN
       first_flag := true ; --first time flag
       begin_year := current_year ;
       begin_month := current_month ;   
   
        IF current_month = '12' THEN
          next_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 year') ;
        ELSE
          next_year := current_year ;
        END IF;
     
       next_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 month') ;

      END IF;

      -- Check current date into borders NOT for First time
      IF to_date( current_year||'.'||current_month, 'YYYY.MM') >= to_date( begin_year||'.'||begin_month, 'YYYY.MM') AND 
         to_date( current_year||'.'||current_month, 'YYYY.MM') < to_date( next_year||'.'||next_month, 'YYYY.MM') AND 
         NOT first_flag 
      THEN
         CONTINUE ; 
      ELSE
       --NEW borders only for second and after time 
       begin_year := current_year ;
       begin_month := current_month ;   
   
        IF current_month = '12' THEN
          next_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 year') ;
        ELSE
          next_year := current_year ;
        END IF;
     
       next_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 month') ;

      END IF;      

      partition_name := 'shipment_shipment_date_'||begin_year||'-'||begin_month||'-01-'|| next_year||'-'||next_month||'-01'  ;
 
     EXECUTE format('CREATE TABLE ' || quote_ident(partition_name) || ' PARTITION OF shipment FOR VALUES FROM ( %L ) TO ( %L )  ' , current_year||'-'||current_month||'-01' , next_year||'-'||next_month||'-01'  ) ; 

      index_name := partition_name||'_shipment_id_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID") TABLESPACE pg_default ' ) ; 

      --Drop first time flag
      first_flag := false ;
   
  END LOOP;

end
$$;

๋คํ”„ ๊ฐ€์ ธ์˜ค๊ธฐ

pg_restore -d postgres --data-only --format=c --table=shipment --verbose  shipment.dmp > /tmp/data_dump/shipment_restore.log 2>&1

ํŒŒํ‹ฐ์…”๋‹ ๊ฒฐ๊ณผ ํ™•์ธ

๊ทธ ๊ฒฐ๊ณผ ์šฐ๋ฆฌ๋Š” ๋ฌด์—‡์„ ๊ฐ–๊ฒŒ ๋˜์—ˆ์Šต๋‹ˆ๊นŒ? ์‹คํ–‰ ๊ณ„ํš์˜ ์ „์ฒด ํ…์ŠคํŠธ๋Š” ๋ฐฉ๋Œ€ํ•˜๊ณ  ์ง€๋ฃจํ•˜๋ฏ€๋กœ ์ตœ์ข… ์ˆซ์ž๋กœ ์ œํ•œํ•˜๋Š” ๊ฒƒ์ด ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.

ํ–ˆ๋‹ค

๋น„์šฉ : 502 997.55
์‹คํ–‰ ์‹œ๊ฐ„: 505 ์ดˆ.

๊ทธ๊ฒƒ์€๋˜์—ˆ๋‹ค

๋น„์šฉ : 77 872.36
์‹คํ–‰ ์‹œ๊ฐ„: 79 ์ดˆ.

๊ฝค ์ข‹์€ ๊ฒฐ๊ณผ์ž…๋‹ˆ๋‹ค. ๋น„์šฉ ๋ฐ ์‹คํ–‰ ์‹œ๊ฐ„์ด ๋‹จ์ถ•๋ฉ๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ํŒŒํ‹ฐ์…”๋‹์„ ์‚ฌ์šฉํ•˜๋ฉด ์˜ˆ์ƒํ•œ ํšจ๊ณผ๋ฅผ ์–ป์„ ์ˆ˜ ์žˆ์œผ๋ฉฐ ์ผ๋ฐ˜์ ์œผ๋กœ ๋†€๋ž„ ์ผ์ด ์•„๋‹™๋‹ˆ๋‹ค.

๊ณ ๊ฐ์„ ํ–‰๋ณตํ•˜๊ฒŒ ํ•˜๋ผ

ํ…Œ์ŠคํŠธ ๊ฒฐ๊ณผ๋Š” ๊ฒ€ํ† ๋ฅผ ์œ„ํ•ด ๊ณ ๊ฐ์—๊ฒŒ ์ œ๊ณต๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ์ด๋ฅผ ๊ฒ€ํ† ํ•œ ํ›„ ๋‹ค์†Œ ์˜ˆ์ƒ์น˜ ๋ชปํ•œ ๊ฒฐ๋ก ์„ ๋‚ด๋ ธ์Šต๋‹ˆ๋‹ค. โ€œ์ข‹์Šต๋‹ˆ๋‹ค. โ€œ๋ฐ์ดํ„ฐโ€ ํ…Œ์ด๋ธ”์„ ๋ถ„ํ• ํ•˜์„ธ์š”.โ€

์˜ˆ, ํ•˜์ง€๋งŒ ์™„์ „ํžˆ ๋‹ค๋ฅธ "๋ฐฐ์†ก" ํ…Œ์ด๋ธ”์„ ์กฐ์‚ฌํ–ˆ์Šต๋‹ˆ๋‹ค. "๋ฐ์ดํ„ฐ" ํ…Œ์ด๋ธ”์—๋Š” "SHIPMENT_DATE" ํ•„๋“œ๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.

๋ฌธ์ œ ์—†์Šต๋‹ˆ๋‹ค. ์ถ”๊ฐ€ํ•˜๊ณ  ๋ณ€๊ฒฝํ•˜์„ธ์š”. ๊ฐ€์žฅ ์ค‘์š”ํ•œ ๊ฒƒ์€ ๊ณ ๊ฐ์ด ๊ฒฐ๊ณผ์— ๋งŒ์กฑํ•œ๋‹ค๋Š” ๊ฒƒ์ด๋ฉฐ ๊ตฌํ˜„ ์„ธ๋ถ€ ์‚ฌํ•ญ์€ ํŠน๋ณ„ํžˆ ์ค‘์š”ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

๋ฉ”์ธ ํ…Œ์ด๋ธ” "๋ฐ์ดํ„ฐ" ํŒŒํ‹ฐ์…”๋‹

์ผ๋ฐ˜์ ์œผ๋กœ ํŠน๋ณ„ํ•œ ์–ด๋ ค์›€์€ ๋ฐœ์ƒํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค. ๋ฌผ๋ก  ํŒŒํ‹ฐ์…”๋‹ ์•Œ๊ณ ๋ฆฌ์ฆ˜์€ ๋‹ค์†Œ ๋ณ€๊ฒฝ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

'๋ฐ์ดํ„ฐ' ํ…Œ์ด๋ธ”์— 'SHIPMENT_DATA' ์—ด ์ถ”๊ฐ€

psql -h ั…ะพัั‚ -U ะฑะฐะทะฐ -d ัŽะทะตั€
=> ALTER TABLE data ADD COLUMN "SHIPMENT_DATE" timestamp without time zone ;

"data" ํ…Œ์ด๋ธ”์˜ "SHIPMENT_DATA" ์—ด์˜ ๊ฐ’์„ "shipment" ํ…Œ์ด๋ธ”์˜ ๋™์ผํ•œ ์ด๋ฆ„์˜ ์—ด ๊ฐ’์œผ๋กœ ์ฑ„์›๋‹ˆ๋‹ค.

-----------------------------
--update_data.sql
--updating for altered table "data" to values of "shipment_data" from the table "shipment"
--version 1.0
do language plpgsql $$
declare 
rec_shipment_data RECORD ;
shipment_date timestamp without time zone ; 
row_count integer ;
total_rows integer ;
begin

  select count(*) into total_rows from shipment ; 
  RAISE NOTICE 'Total %',total_rows;
  row_count:= 0 ;

  FOR rec_shipment_data IN SELECT * FROM shipment LOOP

   update data set "SHIPMENT_DATE" = rec_shipment_data."SHIPMENT_DATE" where "SHIPMENT_ID" = rec_shipment_data."SHIPMENT_ID";
   
   row_count:=  row_count +1 ;
   RAISE NOTICE 'row count = % , from %',row_count,total_rows;
  END LOOP;

end
$$;

"๋ฐ์ดํ„ฐ" ํ…Œ์ด๋ธ” ๋คํ”„ ์ €์žฅ

pg_dump postgres --file=/dump/data.dmp --format=c --table=data --verbose > /dump/data.log 2>&1</source

๋ถ„ํ• ๋œ ํ…Œ์ด๋ธ” 'data'๋ฅผ ๋‹ค์‹œ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค.

--create_partition_data.sql
--create partitions for the table "wafer data" by range column "shipment_data" with one month duration
--version 1.0
do language plpgsql $$
declare 
rec_shipment_date RECORD ;
partition_name varchar;
index_name varchar;
current_year varchar ;
current_month varchar ;
begin_year varchar ;
begin_month varchar ;
next_year varchar ;
next_month varchar ;
first_flag boolean ;
i integer ;

begin

  RAISE NOTICE 'CREATE TEMPORARY TABLE FOR SHIPMENT_DATE';
  CREATE TEMP TABLE tmp_shipment_date as select distinct "SHIPMENT_DATE" from shipment order by "SHIPMENT_DATE" ;


  RAISE NOTICE 'DROP TABLE data';
  drop table data cascade ;


  RAISE NOTICE 'CREATE PARTITIONED TABLE data';
  
  CREATE TABLE public.data
  (
    "RUN_ID" integer,
    "LASERMARK" character varying(20) COLLATE pg_catalog."default" NOT NULL,
    "LOTID" character varying(80) COLLATE pg_catalog."default",
    "SHIPMENT_ID" integer NOT NULL,
    "PARAMETER_ID" integer NOT NULL,
    "INTERNAL_VALUE" character varying(75) COLLATE pg_catalog."default",
    "REPORTED_VALUE" character varying(75) COLLATE pg_catalog."default",
    "LOWER_SPEC_LIMIT" numeric,
    "UPPER_SPEC_LIMIT" numeric , 
    "SHIPMENT_DATE" timestamp without time zone
  )
  PARTITION BY RANGE ("SHIPMENT_DATE")
  WITH (
    OIDS = FALSE
  )
  TABLESPACE pg_default ;


  RAISE NOTICE 'CREATE PARTITIONS FOR TABLE data';

  current_year:='0';
  current_month:='0';

  begin_year := '0' ;
  begin_month := '0'  ;
  next_year := '0' ;
  next_month := '0'  ;
  i := 1;

  FOR rec_shipment_date IN SELECT * FROM tmp_shipment_date LOOP
      
      RAISE NOTICE 'SHIPMENT_DATE=%',rec_shipment_date."SHIPMENT_DATE";
      
      current_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE");
      current_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE") ; 

      --Init borders
      IF   begin_year = '0' THEN
       RAISE NOTICE '***Init borders';
       first_flag := true ; --first time flag
       begin_year := current_year ;
       begin_month := current_month ;   
   
        IF current_month = '12' THEN
          next_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 year') ;
        ELSE
          next_year := current_year ;
        END IF;
     
       next_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 month') ;

      END IF;

--      RAISE NOTICE 'current_year=% , current_month=% ',current_year,current_month;
--      RAISE NOTICE 'begin_year=% , begin_month=% ',begin_year,begin_month;
--      RAISE NOTICE 'next_year=% , next_month=% ',next_year,next_month;

      -- Check current date into borders NOT for First time

      RAISE NOTICE 'Current data = %',to_char( to_date( current_year||'.'||current_month, 'YYYY.MM'), 'YYYY.MM');
      RAISE NOTICE 'Begin data = %',to_char( to_date( begin_year||'.'||begin_month, 'YYYY.MM'), 'YYYY.MM');
      RAISE NOTICE 'Next data = %',to_char( to_date( next_year||'.'||next_month, 'YYYY.MM'), 'YYYY.MM');

      IF to_date( current_year||'.'||current_month, 'YYYY.MM') >= to_date( begin_year||'.'||begin_month, 'YYYY.MM') AND 
         to_date( current_year||'.'||current_month, 'YYYY.MM') < to_date( next_year||'.'||next_month, 'YYYY.MM') AND 
         NOT first_flag 
      THEN
         RAISE NOTICE '***CONTINUE';
         CONTINUE ; 
      ELSE
       --NEW borders only for second and after time 
       RAISE NOTICE '***NEW BORDERS';
       begin_year := current_year ;
       begin_month := current_month ;   
   
        IF current_month = '12' THEN
          next_year := date_part('year' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 year') ;
        ELSE
          next_year := current_year ;
        END IF;
     
       next_month := date_part('month' ,rec_shipment_date."SHIPMENT_DATE" + interval '1 month') ;


      END IF;      

      IF to_number(current_month,'99') < 10 THEN
        current_month := '0'||current_month ; 
      END IF ;

      IF to_number(begin_month,'99') < 10 THEN
        begin_month := '0'||begin_month ; 
      END IF ;

      IF to_number(next_month,'99') < 10 THEN
        next_month := '0'||next_month ; 
      END IF ;

      RAISE NOTICE 'current_year=% , current_month=% ',current_year,current_month;
      RAISE NOTICE 'begin_year=% , begin_month=% ',begin_year,begin_month;
      RAISE NOTICE 'next_year=% , next_month=% ',next_year,next_month;

      partition_name := 'data_'||begin_year||begin_month||'01_'||next_year||next_month||'01'  ;

      RAISE NOTICE 'PARTITION NUMBER % , TABLE NAME =%',i , partition_name;
      
      EXECUTE format('CREATE TABLE ' || quote_ident(partition_name) || ' PARTITION OF data FOR VALUES FROM ( %L ) TO ( %L )  ' , begin_year||'-'||begin_month||'-01' , next_year||'-'||next_month||'-01'  ) ; 

      index_name := partition_name||'_shipment_id_parameter_id_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID", "PARAMETER_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_lasermark_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("LASERMARK" COLLATE pg_catalog."default") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_shipment_id_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_parameter_id_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("PARAMETER_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_shipment_date_idx';
      RAISE NOTICE 'INDEX NAME =%',index_name;
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_DATE") TABLESPACE pg_default ' ) ; 

      --Drop first time flag
      first_flag := false ;

  END LOOP;
end
$$;

3๋‹จ๊ณ„์—์„œ ์ƒ์„ฑ๋œ ๋คํ”„๋ฅผ ๋กœ๋“œํ•ฉ๋‹ˆ๋‹ค.

pg_restore -h ั…ะพัั‚ -ัŽะทะตั€ -d ะฑะฐะทะฐ --data-only --format=c --table=data --verbose  data.dmp > data_restore.log 2>&1

์˜ค๋ž˜๋œ ๋ฐ์ดํ„ฐ์— ๋Œ€ํ•ด ๋ณ„๋„์˜ ์„น์…˜ ๋งŒ๋“ค๊ธฐ

---------------------------------------------------
--create_partition_for_old_dates.sql
--create partitions for keeping old dates 
--version 1.0
do language plpgsql $$
declare 
rec_shipment_date RECORD ;
partition_name varchar;
index_name varchar;

begin

      SELECT min("SHIPMENT_DATE") AS min_date INTO rec_shipment_date from data ;

      RAISE NOTICE 'Old date is %',rec_shipment_date.min_date ;

      partition_name := 'data_old_dates'  ;

      RAISE NOTICE 'PARTITION NAME IS %',partition_name;

      EXECUTE format('CREATE TABLE ' || quote_ident(partition_name) || ' PARTITION OF data FOR VALUES FROM ( %L ) TO ( %L )  ' , '1900-01-01' , 
              to_char( rec_shipment_date.min_date,'YYYY')||'-'||to_char(rec_shipment_date.min_date,'MM')||'-01'  ) ; 

      index_name := partition_name||'_shipment_id_parameter_id_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID", "PARAMETER_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_lasermark_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("LASERMARK" COLLATE pg_catalog."default") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_shipment_id_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_parameter_id_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("PARAMETER_ID") TABLESPACE pg_default ' ) ; 

      index_name := partition_name||'_shipment_date_idx';
      EXECUTE format('CREATE INDEX ' || quote_ident(index_name) || ' ON '|| quote_ident(partition_name) ||' USING btree ("SHIPMENT_DATE") TABLESPACE pg_default ' ) ; 

end
$$;

์ตœ์ข… ๊ฒฐ๊ณผ:

ํ–ˆ๋‹ค
๋น„์šฉ : 502 997.55
์‹คํ–‰ ์‹œ๊ฐ„: 505 ์ดˆ.

๊ทธ๊ฒƒ์€๋˜์—ˆ๋‹ค
๋น„์šฉ : 68 533.70
์‹คํ–‰ ์‹œ๊ฐ„: 69 ์ดˆ

๊ฐ€์น˜ ์žˆ๊ณ , ๊ฝค ๊ฐ€์น˜๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ๊ทธ ๊ณผ์ •์—์„œ ์šฐ๋ฆฌ๋Š” PostgreSQL 10์˜ ํŒŒํ‹ฐ์…”๋‹ ๋ฉ”์ปค๋‹ˆ์ฆ˜์„ ์–ด๋Š ์ •๋„ ๋งˆ์Šคํ„ฐํ•  ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. ์ด๋Š” ํ›Œ๋ฅญํ•œ ๊ฒฐ๊ณผ์ž…๋‹ˆ๋‹ค.

์„œ์ •์  ์นจ๋žต

๋” ์ž˜ํ•  ์ˆ˜ ์žˆ๋‚˜์š”? ์˜ˆ, ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค!์ด๋ฅผ ์œ„ํ•ด์„œ๋Š” MATERIALIZED VIEW๋ฅผ ์‚ฌ์šฉํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
๊ตฌ์ฒดํ™”๋œ ๋ทฐ ์ƒ์„ฑ LASERMARK_VIEW

CREATE MATERIALIZED VIEW LASERMARK_VIEW 
AS
SELECT w."LASERMARK" , MAX(s."SHIPMENT_DATE") AS "SHIPMENT_DATE"
FROM shipment s INNER JOIN data w ON s."SHIPMENT_ID" = w."SHIPMENT_ID" 
GROUP BY w."LASERMARK" ;

CREATE INDEX lasermark_vw_shipment_date_ind on lasermark_view USING btree ("SHIPMENT_DATE") TABLESPACE pg_default;
analyze lasermark_view ;

๋‹ค์‹œ ํ•œ ๋ฒˆ ์š”์ฒญ์„ ๋‹ค์‹œ ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค.
๊ตฌ์ฒดํ™”๋œ ๋ทฐ๋ฅผ ์‚ฌ์šฉํ•œ ์ฟผ๋ฆฌ

SELECT
            p."PARAMETER_ID" as  parameter_id,
            pc."PC_NAME" AS pc_name,
            pc."CUSTOMER_PARTNUMBER" AS customer_partnumber,
            w."LASERMARK" AS lasermark,
            w."LOTID" AS lotid,
            w."REPORTED_VALUE" AS reported_value,
            w."LOWER_SPEC_LIMIT" AS lower_spec_limit,
            w."UPPER_SPEC_LIMIT" AS upper_spec_limit,
            p."TYPE_CALCUL" AS type_calcul,
            s."SHIPMENT_NAME" AS shipment_name,
            s."SHIPMENT_DATE" AS shipment_date,
            extract(year from s."SHIPMENT_DATE") AS year,
            extract(month from s."SHIPMENT_DATE") as month,
            s."REPORT_NAME" AS report_name,
            p."STC_NAME" AS STC_name,
            p."CUSTOMERPARAM_NAME" AS customerparam_name
        FROM data w INNER JOIN shipment s ON s."SHIPMENT_ID" = w."SHIPMENT_ID"
             INNER JOIN parameters p ON p."PARAMETER_ID" = w."PARAMETER_ID"
             INNER JOIN shipment_pc sp ON s."SHIPMENT_ID" = sp."SHIPMENT_ID"
             INNER JOIN pc pc ON pc."PC_ID" = sp."PC_ID"
             INNER JOIN LASERMARK_VIEW md ON md."SHIPMENT_DATE" = s."SHIPMENT_DATE" AND md."LASERMARK" = w."LASERMARK"
        WHERE 
              s."SHIPMENT_DATE" >= '2018-07-01' AND s."SHIPMENT_DATE" <= '2018-09-30';

๊ทธ๋ฆฌ๊ณ  ์šฐ๋ฆฌ๋Š” ๋˜ ๋‹ค๋ฅธ ๊ฒฐ๊ณผ๋ฅผ ์–ป์Šต๋‹ˆ๋‹ค:
ํ–ˆ๋‹ค
๋น„์šฉ : 502 997.55
์‹คํ–‰ ์‹œ๊ฐ„: 505 ์ดˆ

๊ทธ๊ฒƒ์€๋˜์—ˆ๋‹ค
๋น„์šฉ : 42 481.16
์‹คํ–‰ ์‹œ๊ฐ„: 43 ์ดˆ.

๋ฌผ๋ก  ๊ทธ๋Ÿฌํ•œ ์œ ๋งํ•œ ๊ฒฐ๊ณผ๋Š” ๊ธฐ๋งŒ์ ์ด์ง€๋งŒ ์•„์ด๋””์–ด๋Š” ์ƒˆ๋กœ์›Œ์งˆ ํ•„์š”๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ๋ฐ์ดํ„ฐ๋ฅผ ์ˆ˜์‹ ํ•˜๋Š” ๋ฐ ๊ฑธ๋ฆฌ๋Š” ์ด ์‹œ๊ฐ„์€ ๋ณ„๋กœ ๋„์›€์ด ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ ์‹คํ—˜์ ์œผ๋กœ๋Š” ๋งค์šฐ ํฅ๋ฏธ๋กญ์Šต๋‹ˆ๋‹ค.

์‹ค์ œ๋กœ ๊ฒฐ๊ณผ์ ์œผ๋กœ ๋‹ค์‹œ ํ•œ ๋ฒˆ ๊ฐ์‚ฌ๋“œ๋ฆฝ๋‹ˆ๋‹ค. asmm ๊ทธ๋ฆฌ๊ณ  ํ•˜๋ธŒ๋ฃจ!- ์ฟผ๋ฆฌ๊ฐ€ ๋”์šฑ ํ–ฅ์ƒ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์‚ฌํ›„

๊ทธ๋ž˜์„œ ๊ณ ๊ฐ์€ ๋งŒ์กฑํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ํ•„์š”ํ•œ ์ƒํ™ฉ์„ ํ™œ์šฉํ•˜์„ธ์š”.

์ƒˆ ์ž‘์—…: ์‹ฌํ™”ํ•˜๊ณ  ํ™•์žฅํ•˜๊ธฐ ์œ„ํ•ด ๋ฌด์—‡์„ ์ƒ๊ฐํ•ด ๋‚ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

๊ทธ๋ฆฌ๊ณ  ๊ธฐ์–ต๋‚˜๋Š” ๊ฒƒ์€, ์šฐ๋ฆฌ๋Š” PostgreSQL ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค๋ฅผ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜์ง€ ์•Š๋Š”๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.

์‹ค์ œ๋กœ AWS์—๋Š” Cloud Watch ํ˜•ํƒœ์˜ ๋ชจ๋‹ˆํ„ฐ๋ง์ด ์—ฌ์ „ํžˆ ์žˆ์Šต๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ DBA์—๊ฒŒ ์ด๋Ÿฌํ•œ ๋ชจ๋‹ˆํ„ฐ๋ง์˜ ์ด์ ์€ ๋ฌด์—‡์ž…๋‹ˆ๊นŒ? ์ผ๋ฐ˜์ ์œผ๋กœ ๊ฑฐ์˜ ์—†์Šต๋‹ˆ๋‹ค.

์ž์‹ ์—๊ฒŒ ์œ ์šฉํ•˜๊ณ  ํฅ๋ฏธ๋กœ์šด ์ผ์„ ํ•  ์ˆ˜ ์žˆ๋Š” ๊ธฐํšŒ๊ฐ€ ์žˆ๋‹ค๋ฉด, ์ด ๊ธฐํšŒ๋ฅผ ํ™œ์šฉํ•˜์ง€ ์•Š์„ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค...
Mbo

Happy Party ๋˜๋Š” PostgreSQL10์˜ ํŒŒํ‹ฐ์…”๋‹์— ๋Œ€ํ•œ ์ถ”์–ต ๋ช‡ ์ค„

์ด๊ฒƒ์ด ์šฐ๋ฆฌ๊ฐ€ ๊ฐ€์žฅ ํฅ๋ฏธ๋กœ์šด ๋ถ€๋ถ„์— ๋„๋‹ฌํ•˜๋Š” ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค:

3๋…„ 2018์›” XNUMX์ผ.
PostgreSQL ์ฟผ๋ฆฌ ์„ฑ๋Šฅ์„ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๊ธฐ ์œ„ํ•ด ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ๊ธฐ๋Šฅ์— ๋Œ€ํ•œ ์—ฐ๊ตฌ๋ฅผ ์‹œ์ž‘ํ•˜๊ธฐ๋กœ ๊ฒฐ์ •ํ•ฉ๋‹ˆ๋‹ค.

๊ทธ๋Ÿฌ๋‚˜ ๊ทธ๊ฒƒ์€ ์™„์ „ํžˆ ๋‹ค๋ฅธ ์ด์•ผ๊ธฐ์ž…๋‹ˆ๋‹ค.

๊ณ„์†โ€ฆ

์ถœ์ฒ˜ : habr.com

์ฝ”๋ฉ˜ํŠธ๋ฅผ ์ถ”๊ฐ€