áááŻáˇáááŻááş tetrisology ááᯠáĄáááşá¸áááş áĄááŻáśá¸ááťááŤá
áĄáá
áşáĄááŹáĄáŹá¸ááŻáśá¸áááş áĄááąáŹááşá¸ááᯠááąáˇááťáąáŹáˇááąááąáááşá
á
áŹáááŻááşáááŻááťáŹá¸

ááźáżááŹáááŻááŻáśááąáŹáşááźááşá¸
AWS cloud ááž áááşáážá PostgreSQL log file ááᯠlocal áááŻáˇ ááŻáśáážááş download ááŻááşáááş áááŻáĄááşááŤáááşá Linux hostá áĄááťáááşáá˛áˇáááźáąá¸áᎠáááŻááşááąáááˇáş áĄáááşá¸áááş áážáąáŹááˇáşáážáąá¸áážáŻ áážááááşáááŻáˇ ááźáąáŹááááşá
áážááşáááşá¸áááŻááşáĄááşááááşááąáŤááşá¸ááŻááşááŹááááş 5 áááá
áşááźá
áşáááşá
AWS áá˝ááş áážááşáááşá¸áááŻááşááᯠááŹááŽáááŻááşá¸ áážááˇáşááąáááşá
áĄááŻáśá¸ááźáŻááąáŹááááááŹááťáŹá¸
áážááşáááşá¸áááŻááşááᯠhost áááŻáˇ áĄááşááŻááşááŻááşáááşá AWS API "ááŻááąáŤáşáááŻááąáŹ bash script áááŻáĄááŻáśá¸ááźáŻáááş"á
options áááŻ:
- --db-instance-identifier- AWS áážá instance áĄáááşá
- --log-file-name- áááşáážáááŻááşááŻááşááŹá¸ááąáŹ áážááşáááşá¸áááŻááşáĄáááş
- --max-item- command á output áá˝ááş ááźááşááąá¸áááˇáş á áŻá áŻááąáŤááşá¸ áá ášá ááşá¸áĄááąáĄáá˝ááşáááąáŤááşá¸ááŻááşááŻááşááŹá¸ááąáŹáááŻááşá áĄááŻáśá¸áĄáá˝ááşáĄá áŹá¸á
- --starting-token- á áááşááźááşá¸ áááŻáááş áááŻáááş
á¤ááá ášá áááşáá˝ááşá áážááşáááşá¸ááťáŹá¸ááᯠááąáŤááşá¸ááŻááşááŻááşáááş áĄááŻááşá áĽáşá áĽáşáá˝ááş ááąáŤáşááŹáááşá
ááŻááşáá˛áˇá áááŻá¸áááŻá¸áážááşá¸áážááşá¸ - áĄááŻááşááťáááşáĄáá˝ááşá¸ áááşáááşá¸áážááˇáş áĄááťááŻá¸ááťááŻá¸áĄáá˝ááş á
áááşáááşá
áŹá¸á
áᏠáĄááŻááşáá
áşááŻá
ááźáżááŹááᯠááŻááşáááŻá¸ááŻááşá
ááşáĄá ááźáąáážááşá¸ááźáŽá¸ááŹá¸áááŻáˇ áá°ááááşá áááŻáˇááąáŹáş Google áááş áĄááźááşááźáąáážááşá¸áááşá¸ááťáŹá¸ááᯠáĄááźáśááźáŻááŹá¸ááźááşá¸ ááážááááˇáşáĄááźááş áááŻáááŻáááşáážááŻááşá¸á
á˝áŹ áážáŹáá˝áąáááŻááąáŹááášááááşá¸ ááážáááŤá áááşáááŻáˇáááşáááŻá
áąááŹáá° ááąáˇááťááˇáşáááşá¸ááąáŹááşá¸áá
áşááŻááźá
áşáááşá
ááŻááşáááşá¸ááŹáááşááᯠáááŹá¸áááşááźá áşá áąááźááşá¸á
ááąáŹááşááŻáśá¸ áážááşáááşá¸áááŻááşáááş ááźáąáŹááşá¸áá˛áááŻááşááąáŹ áĄáážááşááťááşá¸ááťáŹá¸ áĄá
áŻáśááźá
áşáááşá ááááşáá
áşáĄáá áážááşáááşá¸áááŻááşááᯠá¤áá˛áˇáááŻáˇ áááŻááşá
áŹá¸ááźáŻáááŻááşáááş-

áá
áşááŻááŻááᯠááááááąááźáŽááŹá¸á "tetris" áááŻáᏠááŹáá˛á ááŽáážáŹ ááŹáá˛á
ááąáŹááşáááŻááşááᯠááááşáá
áşááźááˇáş áááşáááˇáşáĄááŤáá˝ááş ááźá
áşááąáŤáşááŹáááŻááşááąáŹ áá˝áąá¸ááťááşá
ááŹááťáŹá¸ááᯠáááŻááşá
áŹá¸ááźáŻááŤá (áááŻá¸áážááşá¸á
áąáááşáĄáá˝ááşá á¤ááá
ášá
áá˝ááşá áááŻááşá¸ááťáŹá¸ááᯠáá°ááŽááąáŹáĄáážááşáážáááŤá
áą)á ááŻáśáážááş tetris ááááşá¸ááááşá¸ááťáŹá¸
1) áááŻááşáá
áşááŻááŻáśá¸ááᯠááąáŤááşá¸ááŻááşááŻááşááźáŽá¸ áĄááźáŽá¸áááşááŤáááşá áĄááŻáśá¸áĄáá˝ááşáĄá
áŹá¸áááş ááąáŹááşááŻáśá¸áááŻááşáĄáá˝ááşáĄá
áŹá¸áááş áááŻááźáŽá¸áááş-

2) áááŻááşáá˝ááş áááşáááşáážááááşá áĄááŻáśá¸áĄáá˝ááşáĄá
áŹá¸áááş ááąáŹááşááŻáśá¸áááŻááşáĄáá˝ááşáĄá
áŹá¸áááş ááąá¸áááşáááş-

3) áááŻááşáááş ááááşáááŻááşá áááşáá˝ážááşáážáŻááźá
áşááźáŽá¸ áĄáááşáĄá
ááşáážááááşá áĄááŻáśá¸áĄáá˝ááşáĄá
áŹá¸áááş ááťááşááąáŹááşááŻáśá¸áááŻááşá áĄáá˝ááşáĄá
áŹá¸áááş áááşá¸áááş-

4) áááŻááşáááş ááááşáááŻááşá áĄáááşááźá
áşááźáŽá¸ ááąáŹááşááŻáśá¸ááźá
áşáááşá áĄááŻáśá¸áĄáá˝ááşáĄá
áŹá¸áááş ááťááşááąáŹááşááŻáśá¸áááŻááşááĄáá˝ááşáĄá
áŹá¸áááş áááŻááźáŽá¸áááş-

ááŹáááşáážáŹ ááąáŹááˇáşáážááşá
ááŻááśáá
áşááŻááᯠá
áŻá
ááşá¸áááş áááŻáˇáááŻááş áĄáááˇáşáĄáá
áşáá
áşááŻáá˝ááş Tetris áá
áŹá¸áááşááźá
áşáááşá

ááźáżááŹááᯠááźáąáážááşá¸ááŹáá˝ááş ááźá áşááąáŤáşááŹááąáŹ ááźáżááŹááťáŹá¸
1) áĄáááŻááşá¸ á áááŻááşá¸á፠ááźááŻá¸áá áşááťáąáŹááşá¸ááᯠááąáŹáşááŤá

ááąááŻááťáĄáŹá¸ááźááˇáş áĄáá°á¸ááźáżááŹááťáŹá¸ ááážááá˛áˇááŤá áááŚá¸ ááááŻááááşá¸áááşá¸áááşáááşá¸ááž á
áśááŹáááşáá
áşááŻá
áĄááąáŹááşá¸ááŻáśá¸áááşááąáŹááşáážáŻáĄáá˝ááşáĄá áŹá¸
ááŤááąáááˇáş ááŤá áááşá¸áááşá¸ áááŻá
áááşáááşá
áŹá¸áááŻáˇááąáŹááşá¸áááşá
ááśáááąáŹááşá¸á
á˝áŹááźááˇáşá start chunk label ááźáŽá¸ááąáŹááş offset áááŻááŻáśá¸áááş áááşá¸áááşá¸ááážáááŤ-
áááşááááźáŽá¸áááşáážááˇáşáĄááťáž áá˝áąá¸ááťááşá áᏠ--starting-token ááᯠpaginating á áááşáááˇáşááąááŹááᯠáááşáážááşáááş áĄááŻáśá¸ááźáŻáááşá á¤áá˝áąá¸ááťááşáážáŻáááş ááąáŹááşáááŻáááşá áŹááźáąáŹááşá¸ááážáąáˇáá˝ááş áĄáąáŹáˇááşáááşáááşáááŻá¸áá áşááŻááᯠáááˇáşáááşááźááŻá¸á áŹá¸ááŤáá áá˝áąá¸ááťááşáá˝ááˇáşááᯠáĄáąáŹáˇááşáááşáĄááźá áş áááˇáşáá˝ááşá¸á ááşá¸á áŹá¸áááşáááŻááşááŤá
ááŤááźáąáŹááˇáş áĄáááŻááşá¸áá˝áąááᯠáĄáááŻááşá¸áááŻááşáááşáááŤáááşá
áĄáááŻááşá¸ááźáŽá¸ááťáŹá¸ááźááˇáş áááşááŤáá áááşáážáŻáážáŻ áĄááąáĄáá˝ááş áĄáááşá¸áááşáᏠáážáááááˇáşáááşá áááŻáˇááąáŹáş áááŹááážáŹ áĄááťáŹá¸ááŻáśá¸ ááźá
áşááááˇáşáááşá
áĄáááşá áááşáááş áĄáááŻááşá¸áááşááťáŹá¸ááźááˇáş áááşááŤáá áááˇáşááťááşáááşáĄááąáážááˇáşá áááşáážáŻáážáŻáĄááąáĄáá˝ááşáááş áĄááťáŹá¸ááŻáśá¸ááźá
áşááááˇáşáááşá áááŻáˇááąáŹáş áááŹááážáŹ áĄáááşá¸áááşááŹáážáááááˇáşáááşá
áááŻáˇááźáąáŹááˇáşá ááŹááşááźáąáŹááááşáááŻáˇáážáŻááᯠááťážáąáŹáˇááťáááşáážááˇáş ááźáąáážááşá¸ááťááşá áĄááŻáśá¸á
áŻáś áážááážáŻáĄáá˝ááşá ááśáááąáŹááşá¸á
á˝áŹáá˛á ááťááŻááşá¸ááąáŹááşáážááˇáş áááşáááşáááşáá°ááąáŹ ááźáąáážááşá¸áááşá¸áá
áşááťááŻá¸ááᯠááťá˝ááşáŻááşáážáŹáá˝áąáá˛áˇáááŤáááşá
áĽáááŹáĄááąááźááˇáşá áĄáá˝ááşáááŻá¸áážááşá¸ááąáŹááŹá¸áážááşá¸ 2 ááŻáá˝ááş áážááşáááşá¸áááŻááşáá áşááŻááᯠááąáŤááşá¸ááŻááşááŻááşááźááşá¸ááŻááşáááşá¸á ááşááᯠááŻáśá¸áááşááźááˇáşááźááŤá ááŻáˇá ááźá áşáááşáážá áşááŻá ááŻáśá¸áá˝ááş áááşáážáŻáážáŻáĄááąáĄáá˝ááşáááş áĄáááŻááşá¸áĄáá˝ááşáĄá áŹá¸ááąáŤáşáá°áááşáááşá
1) ááąá¸áááşááąáŹáĄáááŻááşá¸ááťáŹá¸áá˝ááşáááˇáşááŤá

2) ááźáŽá¸ááŹá¸ááąáŹáĄá
áááşáĄáááŻááşá¸ááťáŹá¸áááŻáááşááŤ

ááŻáśáážááşáĄáááŻááşá¸á áĄááąáŹááşá¸ááŻáśá¸ááźáąáážááşá¸ááťááşáááş áĄáááşáá˝ááşááźá
áşáááşá.
áĄáááŻááşá¸áĄáá˝ááşáĄá
áŹá¸áááş áĄáááşá¸áááşááťážááŹááźá
áşááąáŹáşáááşá¸ á
áŹáááşááźááşá¸ááŻááşáááşá¸á
ááşáá˝ááş áááşáážáŻáážáŻáĄááąáĄáá˝ááşááᯠááťážáąáŹáˇááťáááşáĄáá˝ááş áĄáá˝ááşáĄá
áŹá¸ááᯠáááŻá¸ááźážááˇáşáááŻááşáááşá
áážááşááŹá¸áááˇáşáááşá áááşáážáŻáááˇáşáĄáááŻááşá¸á áĄááąáŹááşá¸ááŻáśá¸áĄáá˝ááşáĄá áŹá¸ááᯠáá˝áąá¸ááťááşááźááşá¸áááźáżááŹááᯠááŻáśá¸ááááźáąáážááşá¸áááąá¸ááźáąáŹááşá¸áážááˇáş áááŻáááŻááąá¸áááşááąáŹááąáˇááŹáážáŻáážááˇáş áá˝á˛ááźááşá¸á áááşááźáŹáááş áááŻáĄááşááŤáááşá áááşá¸áááşá¸ááźáŹáááşá
áĄááąáŹááşáĄáááşááąáŹáşáážáŻá áĄáá˝áąáá˝áąááąáŹáşááźááťááş
áááşááąáŹááşáážáŻá áŹá¸áá˝á˛ááťáŹá¸ áĄááŻáśá¸ááźáŻáááşá
CREATE TABLE endpoint
(
id SERIAL ,
host text
);
TABLE database
(
id SERIAL ,
âŚ
last_aws_log_time text ,
last_aws_nexttoken text ,
aws_max_item_size integer
);
last_aws_log_time â вŃĐľĐźĐľĐ˝Đ˝Đ°Ń ĐźĐľŃка пОŃНодногО СагŃŃМоннОгО НОг-ŃаКНа в ŃĐžŃПаŃĐľ YYYY-MM-DD-HH24.
last_aws_nexttoken â ŃокŃŃĐžĐ˛Đ°Ń ĐźĐľŃка пОŃНодноК СагŃŃМоннОК пОŃŃии.
aws_max_item_size- ŃПпиŃиŃĐľŃкиП ĐżŃŃоП, пОдОйŃаннŃĐš наŃаНŃĐ˝ŃĐš ŃĐ°ĐˇĐźĐľŃ ĐżĐžŃŃии.
ááŹááşáá˝ážááşá¸á á áŹááŹá¸áĄááźááˇáşáĄá áŻáś
download_aws_piece.sh
#!/bin/bash
#########################################################
# download_aws_piece.sh
# downloan piece of log from AWS
# version HABR
let min_item_size=1024
let max_item_size=1048576
let growth_factor=3
let growth_counter=1
let growth_counter_max=3
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh:''STARTED'
AWS_LOG_TIME=$1
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh:AWS_LOG_TIME='$AWS_LOG_TIME
database_id=$2
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh:database_id='$database_id
RESULT_FILE=$3
endpoint=`psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE_DATABASE -A -t -c "select e.host from endpoint e join database d on e.id = d.endpoint_id where d.id = $database_id "`
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh:endpoint='$endpoint
db_instance=`echo $endpoint | awk -F"." '{print toupper($1)}'`
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh:db_instance='$db_instance
LOG_FILE=$RESULT_FILE'.tmp_log'
TMP_FILE=$LOG_FILE'.tmp'
TMP_MIDDLE=$LOG_FILE'.tmp_mid'
TMP_MIDDLE2=$LOG_FILE'.tmp_mid2'
current_aws_log_time=`psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -A -t -c "select last_aws_log_time from database where id = $database_id "`
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh:current_aws_log_time='$current_aws_log_time
if [[ $current_aws_log_time != $AWS_LOG_TIME ]];
then
is_new_log='1'
if ! psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -v ON_ERROR_STOP=1 -A -t -q -c "update database set last_aws_log_time = '$AWS_LOG_TIME' where id = $database_id "
then
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: FATAL_ERROR - update database set last_aws_log_time .'
exit 1
fi
else
is_new_log='0'
fi
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh:is_new_log='$is_new_log
let last_aws_max_item_size=`psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -A -t -c "select aws_max_item_size from database where id = $database_id "`
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: last_aws_max_item_size='$last_aws_max_item_size
let count=1
if [[ $is_new_log == '1' ]];
then
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: START DOWNLOADING OF NEW AWS LOG'
if ! aws rds download-db-log-file-portion
--max-items $last_aws_max_item_size
--region REGION
--db-instance-identifier $db_instance
--log-file-name error/postgresql.log.$AWS_LOG_TIME > $LOG_FILE
then
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: FATAL_ERROR - Could not get log from AWS .'
exit 2
fi
else
next_token=`psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -v ON_ERROR_STOP=1 -A -t -c "select last_aws_nexttoken from database where id = $database_id "`
if [[ $next_token == '' ]];
then
next_token='0'
fi
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: CONTINUE DOWNLOADING OF AWS LOG'
if ! aws rds download-db-log-file-portion
--max-items $last_aws_max_item_size
--starting-token $next_token
--region REGION
--db-instance-identifier $db_instance
--log-file-name error/postgresql.log.$AWS_LOG_TIME > $LOG_FILE
then
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: FATAL_ERROR - Could not get log from AWS .'
exit 3
fi
line_count=`cat $LOG_FILE | wc -l`
let lines=$line_count-1
tail -$lines $LOG_FILE > $TMP_MIDDLE
mv -f $TMP_MIDDLE $LOG_FILE
fi
next_token_str=`cat $LOG_FILE | grep NEXTTOKEN`
next_token=`echo $next_token_str | awk -F" " '{ print $2}' `
grep -v NEXTTOKEN $LOG_FILE > $TMP_FILE
if [[ $next_token == '' ]];
then
cp $TMP_FILE $RESULT_FILE
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: NEXTTOKEN NOT FOUND - FINISH '
rm $LOG_FILE
rm $TMP_FILE
rm $TMP_MIDDLE
rm $TMP_MIDDLE2
exit 0
else
psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -v ON_ERROR_STOP=1 -A -t -q -c "update database set last_aws_nexttoken = '$next_token' where id = $database_id "
fi
first_str=`tail -1 $TMP_FILE`
line_count=`cat $TMP_FILE | wc -l`
let lines=$line_count-1
head -$lines $TMP_FILE > $RESULT_FILE
###############################################
# MAIN CIRCLE
let count=2
while [[ $next_token != '' ]];
do
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: count='$count
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: START DOWNLOADING OF AWS LOG'
if ! aws rds download-db-log-file-portion
--max-items $last_aws_max_item_size
--starting-token $next_token
--region REGION
--db-instance-identifier $db_instance
--log-file-name error/postgresql.log.$AWS_LOG_TIME > $LOG_FILE
then
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: FATAL_ERROR - Could not get log from AWS .'
exit 4
fi
next_token_str=`cat $LOG_FILE | grep NEXTTOKEN`
next_token=`echo $next_token_str | awk -F" " '{ print $2}' `
TMP_FILE=$LOG_FILE'.tmp'
grep -v NEXTTOKEN $LOG_FILE > $TMP_FILE
last_str=`head -1 $TMP_FILE`
if [[ $next_token == '' ]];
then
concat_str=$first_str$last_str
echo $concat_str >> $RESULT_FILE
line_count=`cat $TMP_FILE | wc -l`
let lines=$line_count-1
tail -$lines $TMP_FILE >> $RESULT_FILE
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: NEXTTOKEN NOT FOUND - FINISH '
rm $LOG_FILE
rm $TMP_FILE
rm $TMP_MIDDLE
rm $TMP_MIDDLE2
exit 0
fi
if [[ $next_token != '' ]];
then
let growth_counter=$growth_counter+1
if [[ $growth_counter -gt $growth_counter_max ]];
then
let last_aws_max_item_size=$last_aws_max_item_size*$growth_factor
let growth_counter=1
fi
if [[ $last_aws_max_item_size -gt $max_item_size ]];
then
let last_aws_max_item_size=$max_item_size
fi
psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -A -t -q -c "update database set last_aws_nexttoken = '$next_token' where id = $database_id "
concat_str=$first_str$last_str
echo $concat_str >> $RESULT_FILE
line_count=`cat $TMP_FILE | wc -l`
let lines=$line_count-1
#############################
#Get middle of file
head -$lines $TMP_FILE > $TMP_MIDDLE
line_count=`cat $TMP_MIDDLE | wc -l`
let lines=$line_count-1
tail -$lines $TMP_MIDDLE > $TMP_MIDDLE2
cat $TMP_MIDDLE2 >> $RESULT_FILE
first_str=`tail -1 $TMP_FILE`
fi
let count=$count+1
done
#
#################################################################
exit 0
áĄááťááŻáˇááąáŹ áážááşá¸áááşá¸ááťááşááťáŹá¸áážááˇáş ááŹááşáá˝ážááşá¸áĄáááŻááşá¸áĄá ááťáŹá¸-
Script áááˇáşáá˝ááşá¸áážáŻ áááˇáşáááşááťááşááťáŹá¸
- YYYY-MM-DD-HH24 ááąáŹáşáááş- AWS_LOG_TIME= áážááşáááşá¸áááŻááşáĄáááşá áĄááťáááşááśááááş$1
- ááąááŹááąáˇá áş ID- database_id=$2
- á áŻááąáŹááşá¸ááŹá¸ááąáŹ áážááşáááşá¸áááŻááşáĄáááş- RESULT_FILE=$3
ááąáŹááşááŻáśá¸ áĄááşááŻááşááŻááşááŹá¸ááąáŹ áážááşáááşá¸áááŻááşá áĄááťáááşááśááááşááᯠááá°ááŤ-
current_aws_log_time=`psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -A -t -c "select last_aws_log_time from database where id = $database_id "`ááąáŹááşááŻáśá¸áááşááŹá¸ááąáŹ áážááşáááşá¸áááŻááşá áĄááťáááşááśááááşáááş áááˇáşáá˝ááşá¸áážáŻ áááˇáşáááşááťááşáážááˇáş ááááŻááşááŽááŤáá áážááşáááşá¸áááŻááşáĄáá áşááᯠáááşááąá¸áááş-
if [[ $current_aws_log_time != $AWS_LOG_TIME ]];
then
is_new_log='1'
if ! psql -h ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -v ON_ERROR_STOP=1 -A -t -c "update database set last_aws_log_time = '$AWS_LOG_TIME' where id = $database_id "
then
echo '***download_aws_piece.sh -FATAL_ERROR - update database set last_aws_log_time .'
exit 1
fi
else
is_new_log='0'
fi
áááşááŹá¸ááąáŹáááŻááşááž ááąáŹááşáááŻáááşáĄáá˝ážááşá¸ááááşáááŻá¸ááᯠááťá˝ááşáŻááşáááŻáˇááážááááş-
next_token_str=`cat $LOG_FILE | grep NEXTTOKEN`
next_token=`echo $next_token_str | awk -F" " '{ print $2}' `
ááąáŤááşá¸ááŻááşááĄááŻáśá¸áááááááşááášáááŹáááş nexttoken ááááŹáááşáááŻá¸ááźá áşáááşá
áĄáááŻááşá¸áá
áşááŻáá˝ááşá áááşá¸áá
áşááťážáąáŹááşáá˝ááşá áááŻááşááĄáááŻááşá¸ááťáŹá¸ááᯠááąáá˝ááşááŹá áááŻááşá¸ááťáŹá¸ááąáŤááşá¸á
ááşáᏠáĄáááŻááşá¸ááĄáá˝ááşáĄá
áŹá¸ááᯠáááŻá¸á
áąáááş-
Main Loop
# MAIN CIRCLE
let count=2
while [[ $next_token != '' ]];
do
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: count='$count
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: START DOWNLOADING OF AWS LOG'
if ! aws rds download-db-log-file-portion
--max-items $last_aws_max_item_size
--starting-token $next_token
--region REGION
--db-instance-identifier $db_instance
--log-file-name error/postgresql.log.$AWS_LOG_TIME > $LOG_FILE
then
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: FATAL_ERROR - Could not get log from AWS .'
exit 4
fi
next_token_str=`cat $LOG_FILE | grep NEXTTOKEN`
next_token=`echo $next_token_str | awk -F" " '{ print $2}' `
TMP_FILE=$LOG_FILE'.tmp'
grep -v NEXTTOKEN $LOG_FILE > $TMP_FILE
last_str=`head -1 $TMP_FILE`
if [[ $next_token == '' ]];
then
concat_str=$first_str$last_str
echo $concat_str >> $RESULT_FILE
line_count=`cat $TMP_FILE | wc -l`
let lines=$line_count-1
tail -$lines $TMP_FILE >> $RESULT_FILE
echo $(date +%Y%m%d%H%M)': download_aws_piece.sh: NEXTTOKEN NOT FOUND - FINISH '
rm $LOG_FILE
rm $TMP_FILE
rm $TMP_MIDDLE
rm $TMP_MIDDLE2
exit 0
fi
if [[ $next_token != '' ]];
then
let growth_counter=$growth_counter+1
if [[ $growth_counter -gt $growth_counter_max ]];
then
let last_aws_max_item_size=$last_aws_max_item_size*$growth_factor
let growth_counter=1
fi
if [[ $last_aws_max_item_size -gt $max_item_size ]];
then
let last_aws_max_item_size=$max_item_size
fi
psql -h MONITOR_ENDPOINT.rds.amazonaws.com -U USER -d MONITOR_DATABASE -A -t -q -c "update database set last_aws_nexttoken = '$next_token' where id = $database_id "
concat_str=$first_str$last_str
echo $concat_str >> $RESULT_FILE
line_count=`cat $TMP_FILE | wc -l`
let lines=$line_count-1
#############################
#Get middle of file
head -$lines $TMP_FILE > $TMP_MIDDLE
line_count=`cat $TMP_MIDDLE | wc -l`
let lines=$line_count-1
tail -$lines $TMP_MIDDLE > $TMP_MIDDLE2
cat $TMP_MIDDLE2 >> $RESULT_FILE
first_str=`tail -1 $TMP_FILE`
fi
let count=$count+1
done
ááąáŹááşáá áşááŻáááŹáá˛?
áááŻáˇááźáąáŹááˇáşá ááááĄáááşáĄáááşáĄááŻááş - "cloud áážáážááşáááşá¸áááŻááşáááŻááąáŤááşá¸ááŻááşááŻááşááŤ" áááŻááźáąáážááşá¸áááşá ááąáŤááşá¸ááŻááşááŻááşááŹá¸áá˛áˇ áážááşáááşá¸áá˛áˇ ááŹááŻááşáááá˛á
áŚá¸á
á˝áŹ áááşáááş áážááşáááşá¸áááŻááşááᯠáá˝á˛ááźááşá¸á
áááşááźáŹááźáŽá¸ áááşá¸ááž áĄáážááşááááş ááąáŹááşá¸áááŻáážáŻááťáŹá¸ááᯠááŻááşáá°áááş áááŻáĄááşáááşá
áĄááŻááşá ááááşááááşááŤáá°á¸á áĄáááŻá¸áážááşá¸ááŻáśá¸ bash-script áááş ááąáŹááşá¸áá˝ááşááŤáááşá
upload_log_query.sh
#!/bin/bash
#########################################################
# upload_log_query.sh
# Upload table table from dowloaded aws file
# version HABR
###########################################################
echo 'TIMESTAMP:'$(date +%c)' Upload log_query table '
source_file=$1
echo 'source_file='$source_file
database_id=$2
echo 'database_id='$database_id
beginer=' '
first_line='1'
let "line_count=0"
sql_line=' '
sql_flag=' '
space=' '
cat $source_file | while read line
do
line="$space$line"
if [[ $first_line == "1" ]]; then
beginer=`echo $line | awk -F" " '{ print $1}' `
first_line='0'
fi
current_beginer=`echo $line | awk -F" " '{ print $1}' `
if [[ $current_beginer == $beginer ]]; then
if [[ $sql_flag == '1' ]]; then
sql_flag='0'
log_date=`echo $sql_line | awk -F" " '{ print $1}' `
log_time=`echo $sql_line | awk -F" " '{ print $2}' `
duration=`echo $sql_line | awk -F" " '{ print $5}' `
#replace ' to ''
sql_modline=`echo "$sql_line" | sed 's/'''/''''''/g'`
sql_line=' '
################
#PROCESSING OF THE SQL-SELECT IS HERE
if ! psql -h ENDPOINT.rds.amazonaws.com -U USER -d DATABASE -v ON_ERROR_STOP=1 -A -t -c "select log_query('$ip_port',$database_id , '$log_date' , '$log_time' , '$duration' , '$sql_modline' )"
then
echo 'FATAL_ERROR - log_query '
exit 1
fi
################
fi #if [[ $sql_flag == '1' ]]; then
let "line_count=line_count+1"
check=`echo $line | awk -F" " '{ print $8}' `
check_sql=${check^^}
#echo 'check_sql='$check_sql
if [[ $check_sql == 'SELECT' ]]; then
sql_flag='1'
sql_line="$sql_line$line"
ip_port=`echo $sql_line | awk -F":" '{ print $4}' `
fi
else
if [[ $sql_flag == '1' ]]; then
sql_line="$sql_line$line"
fi
fi #if [[ $current_beginer == $beginer ]]; then
done
áááŻáááşáááş log file áážááŻááşááŻááşááŹá¸ááąáŹááąá¸áá˝ááşá¸áážááˇáşáĄááŻááşááŻááşáááŻááşááŤááźáŽá
ááźáŽá¸ááąáŹáˇ áĄááŻáśá¸áááşáááŻááşááźáąááťáŹá¸á á˝áŹáážáááŤáááşá
áá˝á˛ááźááşá¸á áááşááźáŹááŹá¸ááąáŹ ááąá¸áá˝ááşá¸ááťáŹá¸ááᯠáá áşááąááŹááŹáá˝ááş ááááşá¸áááşá¸ááŹá¸áááŤáááşá áááşá¸áĄáá˝ááşá áááşááąáŹááşáážáŻáááŹá¸ááᯠáĄááŻáśá¸ááźáŻáááşá log_query
CREATE TABLE log_query
(
id SERIAL ,
queryid bigint ,
query_md5hash text not null ,
database_id integer not null ,
timepoint timestamp without time zone not null,
duration double precision not null ,
query text not null ,
explained_plan text[],
plan_md5hash text ,
explained_plan_wo_costs text[],
plan_hash_value text ,
baseline_id integer ,
ip text ,
port text
);
ALTER TABLE log_query ADD PRIMARY KEY (id);
ALTER TABLE log_query ADD CONSTRAINT queryid_timepoint_unique_key UNIQUE (queryid, timepoint );
ALTER TABLE log_query ADD CONSTRAINT query_md5hash_timepoint_unique_key UNIQUE (query_md5hash, timepoint );
CREATE INDEX log_query_timepoint_idx ON log_query (timepoint);
CREATE INDEX log_query_queryid_idx ON log_query (queryid);
ALTER TABLE log_query ADD CONSTRAINT database_id_fk FOREIGN KEY (database_id) REFERENCES database (id) ON DELETE CASCADE ;
áá˝á˛ááźááşá¸á
áááşááźáŹááŹá¸ááąáŹ ááąáŹááşá¸áááŻááťááşááᯠááŻááşááąáŹááşááąááŤáááşá plpgsql ááŻááşááąáŹááşááťááş "log_query"á
log_query.sql
--log_query.sql
--verison HABR
CREATE OR REPLACE FUNCTION log_query( ip_port text ,log_database_id integer , log_date text , log_time text , duration text , sql_line text ) RETURNS boolean AS $$
DECLARE
result boolean ;
log_timepoint timestamp without time zone ;
log_duration double precision ;
pos integer ;
log_query text ;
activity_string text ;
log_md5hash text ;
log_explain_plan text[] ;
log_planhash text ;
log_plan_wo_costs text[] ;
database_rec record ;
pg_stat_query text ;
test_log_query text ;
log_query_rec record;
found_flag boolean;
pg_stat_history_rec record ;
port_start integer ;
port_end integer ;
client_ip text ;
client_port text ;
log_queryid bigint ;
log_query_text text ;
pg_stat_query_text text ;
BEGIN
result = TRUE ;
RAISE NOTICE '***log_query';
port_start = position('(' in ip_port);
port_end = position(')' in ip_port);
client_ip = substring( ip_port from 1 for port_start-1 );
client_port = substring( ip_port from port_start+1 for port_end-port_start-1 );
SELECT e.host , d.name , d.owner_pwd
INTO database_rec
FROM database d JOIN endpoint e ON e.id = d.endpoint_id
WHERE d.id = log_database_id ;
log_timepoint = to_timestamp(log_date||' '||log_time,'YYYY-MM-DD HH24-MI-SS');
log_duration = duration:: double precision;
pos = position ('SELECT' in UPPER(sql_line) );
log_query = substring( sql_line from pos for LENGTH(sql_line));
log_query = regexp_replace(log_query,' +',' ','g');
log_query = regexp_replace(log_query,';+','','g');
log_query = trim(trailing ' ' from log_query);
log_md5hash = md5( log_query::text );
--Explain execution plan--
EXECUTE 'SELECT dblink_connect(''LINK1'',''host='||database_rec.host||' dbname='||database_rec.name||' user=DATABASE password='||database_rec.owner_pwd||' '')';
log_explain_plan = ARRAY ( SELECT * FROM dblink('LINK1', 'EXPLAIN '||log_query ) AS t (plan text) );
log_plan_wo_costs = ARRAY ( SELECT * FROM dblink('LINK1', 'EXPLAIN ( COSTS FALSE ) '||log_query ) AS t (plan text) );
PERFORM dblink_disconnect('LINK1');
--------------------------
BEGIN
INSERT INTO log_query
(
query_md5hash ,
database_id ,
timepoint ,
duration ,
query ,
explained_plan ,
plan_md5hash ,
explained_plan_wo_costs ,
plan_hash_value ,
ip ,
port
)
VALUES
(
log_md5hash ,
log_database_id ,
log_timepoint ,
log_duration ,
log_query ,
log_explain_plan ,
md5(log_explain_plan::text) ,
log_plan_wo_costs ,
md5(log_plan_wo_costs::text),
client_ip ,
client_port
);
activity_string = 'New query has logged '||
' database_id = '|| log_database_id ||
' query_md5hash='||log_md5hash||
' , timepoint = '||to_char(log_timepoint,'YYYYMMDD HH24:MI:SS');
RAISE NOTICE '%',activity_string;
PERFORM pg_log( log_database_id , 'log_query' , activity_string);
EXCEPTION
WHEN unique_violation THEN
RAISE NOTICE '*** unique_violation *** query already has logged';
END;
SELECT queryid
INTO log_queryid
FROM log_query
WHERE query_md5hash = log_md5hash AND
timepoint = log_timepoint;
IF log_queryid IS NOT NULL
THEN
RAISE NOTICE 'log_query with query_md5hash = % and timepoint = % has already has a QUERYID = %',log_md5hash,log_timepoint , log_queryid ;
RETURN result;
END IF;
------------------------------------------------
RAISE NOTICE 'Update queryid';
SELECT *
INTO log_query_rec
FROM log_query
WHERE query_md5hash = log_md5hash AND timepoint = log_timepoint ;
log_query_rec.query=regexp_replace(log_query_rec.query,';+','','g');
FOR pg_stat_history_rec IN
SELECT
queryid ,
query
FROM
pg_stat_db_queries
WHERE
database_id = log_database_id AND
queryid is not null
LOOP
pg_stat_query = pg_stat_history_rec.query ;
pg_stat_query=regexp_replace(pg_stat_query,'n+',' ','g');
pg_stat_query=regexp_replace(pg_stat_query,'t+',' ','g');
pg_stat_query=regexp_replace(pg_stat_query,' +',' ','g');
pg_stat_query=regexp_replace(pg_stat_query,'$.','%','g');
log_query_text = trim(trailing ' ' from log_query_rec.query);
pg_stat_query_text = pg_stat_query;
--SELECT log_query_rec.query like pg_stat_query INTO found_flag ;
IF (log_query_text LIKE pg_stat_query_text) THEN
found_flag = TRUE ;
ELSE
found_flag = FALSE ;
END IF;
IF found_flag THEN
UPDATE log_query SET queryid = pg_stat_history_rec.queryid WHERE query_md5hash = log_md5hash AND timepoint = log_timepoint ;
activity_string = ' updated queryid = '||pg_stat_history_rec.queryid||
' for log_query with id = '||log_query_rec.id
;
RAISE NOTICE '%',activity_string;
EXIT ;
END IF ;
END LOOP ;
RETURN result ;
END
$$ LANGUAGE plpgsql;
ááŻááşááąáŹááşáááˇáşáĄááŤá áááşááąáŹááşáážáŻáááŹá¸ááᯠáĄááŻáśá¸ááźáŻáááşá pg_stat_db_queriesáááŹá¸ááž áááşáážáááąá¸ááźááşá¸ááťááşááťáŹá¸á ááťážááşáá áşááźááşááŻáśááŤáážáááąáŹ A pg_stat_history (áááŹá¸áĄááŻáśá¸ááźáŻáážáŻááᯠá¤ááąááŹáá˝ááş ááąáŹáşááźááŹá¸ááŤáááşá )
TABLE pg_stat_db_queries
(
database_id integer,
queryid bigint ,
query text ,
max_time double precision
);
TABLE pg_stat_history
(
âŚ
database_id integer ,
âŚ
queryid bigint ,
âŚ
max_time double precision ,
âŚ
);
ááŻááşááąáŹááşááťááşáááş áááˇáşáĄáŹá¸ áážááşáááşá¸áááŻááşáá áşááŻááž ááąáŹááşá¸áááŻááťááşááťáŹá¸ááᯠááŻááşááąáŹááşáááşáĄáá˝ááş áĄááŻáśá¸áááşááąáŹáĄááşášááŤáááşááťáŹá¸á á˝áŹááᯠáĄááąáŹááşáĄáááşááąáŹáşáááş áá˝ááˇáşááźáŻáááşá áááŻáááŻáááşáážáŹ-
áĄáá˝ááˇáşáĄáááşá¸ááśááŤááş 1 - ááąá¸ááźááşá¸áážáŻááŻááşááąáŹááşáážáŻáážááşáááşá¸
á
á˝ááşá¸ááąáŹááşáááşááźá
áşáááşáá
áşááŻá
áááşáááş áĄáá˝ááşáĄááŻáśá¸áááşáááşá ááááŚá¸á
á˝áŹ ááááŻááşá¸ááᯠáááĄáąáŹááşááŻááşá፠- áážáąáŹááˇáşáážáąá¸áážáŻá áááşáĄááťáááşá á
ááŹáá˛á
áááŻáˇááąáŹááş ááášááááşááťáŹá¸áĄáááŻááşá¸ ááźááşááĄááźáąáŹááşá¸áááŹá¸ááťáŹá¸ááᯠáážáŹáá˝áąááŤá ááąááŹááąáˇá
áş load áááş ááááááŹáᏠáááŻá¸ááŹááźáŽá¸ ááááťááąáŹ ááąáŹááşá¸áááŻááťááşáááş áááşá¸áážááˇáş ááŹáážááááŻááşááąá
log_query áááŹá¸áááŻáˇ áááşááąáŹááşáážáŻáĄáá
áşáááˇáşááŤá
port_start = position('(' in ip_port);
port_end = position(')' in ip_port);
client_ip = substring( ip_port from 1 for port_start-1 );
client_port = substring( ip_port from port_start+1 for port_end-port_start-1 );
SELECT e.host , d.name , d.owner_pwd
INTO database_rec
FROM database d JOIN endpoint e ON e.id = d.endpoint_id
WHERE d.id = log_database_id ;
log_timepoint = to_timestamp(log_date||' '||log_time,'YYYY-MM-DD HH24-MI-SS');
log_duration = to_number(duration,'99999999999999999999D9999999999');
pos = position ('SELECT' in UPPER(sql_line) );
log_query = substring( sql_line from pos for LENGTH(sql_line));
log_query = regexp_replace(log_query,' +',' ','g');
log_query = regexp_replace(log_query,';+','','g');
log_query = trim(trailing ' ' from log_query);
RAISE NOTICE 'log_query=%',log_query ;
log_md5hash = md5( log_query::text );
--Explain execution plan--
EXECUTE 'SELECT dblink_connect(''LINK1'',''host='||database_rec.host||' dbname='||database_rec.name||' user=DATABASE password='||database_rec.owner_pwd||' '')';
log_explain_plan = ARRAY ( SELECT * FROM dblink('LINK1', 'EXPLAIN '||log_query ) AS t (plan text) );
log_plan_wo_costs = ARRAY ( SELECT * FROM dblink('LINK1', 'EXPLAIN ( COSTS FALSE ) '||log_query ) AS t (plan text) );
PERFORM dblink_disconnect('LINK1');
--------------------------
BEGIN
INSERT INTO log_query
(
query_md5hash ,
database_id ,
timepoint ,
duration ,
query ,
explained_plan ,
plan_md5hash ,
explained_plan_wo_costs ,
plan_hash_value ,
ip ,
port
)
VALUES
(
log_md5hash ,
log_database_id ,
log_timepoint ,
log_duration ,
log_query ,
log_explain_plan ,
md5(log_explain_plan::text) ,
log_plan_wo_costs ,
md5(log_plan_wo_costs::text),
client_ip ,
client_port
);
áĄááşášááŤáááş ááśááŤááş 2 - Query Execution Plans ááᯠááááşá¸áááşá¸ááŤá
á¤áĄááťáááşáá˝ááş áááˇáşáá˝ááş-áážááşá¸áááşá¸ááťááş-áážááşááťááşááťáŹá¸ áá˝ááşááąáŤáşááŹáááŻááşáááş-"ááŤááąáááˇáş autoexplain áážáááźáŽá¸ááŹá¸ááŤáâ . ááŻááşááŤáááşá ááŤááąáááˇáş execution plan ááᯠáá°ááŽáá˛áˇ log file áážáŹ ááááşá¸áááşá¸ááźáŽá¸ ááąáŹááşáááşáá˝á˛ááźááşá¸á áááşááźáŹáááŻáˇáĄáá˝ááş ááááşá¸áááşá¸áááŻáˇáĄáá˝ááşá log file ááᯠáá˝á˛ááźááşá¸á áááşááźáŹáááŻáˇ áááŻáĄááşáááş ááŹáĄááťááşáá˛?
áááŻáˇááąáŹáşá ááŤáááŻáĄááşáááş-
ááááŚá¸á
á˝áŹ- á
áąáŹááˇáşááźááˇáşááąá¸ááąááŹááąáˇá
áşááááşááąáŹááşáážáŻáááŹá¸áá˝ááş áĄááąáŹááşáĄáááşááąáŹáşáážáŻáĄá
áŽáĄá
ááşááᯠááááşá¸áááşá¸ááŤá
ááŻááááĄááťááş- á
áŻáśá
ááşá¸ááąá¸ááźááşá¸áážáŻ áĄááąáŹááşáĄáááşááąáŹáşáážáŻáĄá
áŽáĄá
ááş ááźáąáŹááşá¸áá˛áá˝áŹá¸ááźáąáŹááşá¸ ááťááşááťááşá¸ááááźááşáááŻááşá
áąáááşáĄáá˝ááş áĄááąáŹááşáĄáááşááąáŹáşáážáŻáĄá
áŽáĄá
ááşááťáŹá¸ááᯠáá
áşááŻáážááˇáşáá
áşáᯠáážááŻááşá¸áážááşáááŻááşá
áąáááşá
áááşáááşáážááşáážááş ááŻááşááąáŹááşáážáŻááąáŹááşááťáŹá¸ááŤáážáááąáŹ ááąáŹááşá¸áááŻáážáŻáá
áşáᯠááááŻááşááŤáááşá EXPLAIN ááᯠáĄááŻáśá¸ááźáŻá áááşá¸á áĄááąáŹááşáĄáááşááąáŹáşáážáŻ áĄá
áŽáĄá
ááşááᯠááá°ááźááşá¸áážááˇáş ááááşá¸áááşá¸ááźááşá¸áááş áĄááźáąááśáĄááŻááşááźá
áşáááşá
áááŻáˇáĄááźááşá EXPLAIN (COSTS FALSE) áĄááŻáśá¸áĄáážáŻááşá¸ááᯠáĄááŻáśá¸ááźáŻá áĄá
áŽáĄá
ááşá hash áááşáááŻá¸ááᯠááá°áááş áĄááŻáśá¸ááźáŻáááˇáş áĄá
áŽáĄá
ááşá áá°ááąáŹááşááᯠáááşááááŻááşáááşá áááşá¸áááş áĄááąáŹááşáĄáááşááąáŹáşáážáŻáĄá
áŽáĄá
ááşá ááąáŹááşáááşáá˝á˛ááááŻááşá¸ááźáąáŹááşá¸ááᯠáá˝á˛ááźááşá¸á
áááşááźáŹááŹáá˝ááş áĄááąáŹááşáĄáá°ááźá
áşá
áąáááˇáş áĄá
áŽáĄá
ááşá hash áááşáááŻá¸ááᯠááá°áááŻááşáááşá
áĄááąáŹááşáĄáááşááąáŹáşáážáŻ áĄá
áŽáĄá
ááş ááá°ááŹááᯠááá°ááŤá
--Explain execution plan--
EXECUTE 'SELECT dblink_connect(''LINK1'',''host='||database_rec.host||' dbname='||database_rec.name||' user=DATABASE password='||database_rec.owner_pwd||' '')';
log_explain_plan = ARRAY ( SELECT * FROM dblink('LINK1', 'EXPLAIN '||log_query ) AS t (plan text) );
log_plan_wo_costs = ARRAY ( SELECT * FROM dblink('LINK1', 'EXPLAIN ( COSTS FALSE ) '||log_query ) AS t (plan text) );
PERFORM dblink_disconnect('LINK1');
áĄáá˝ááˇáşáĄáááşá¸ #3 - á áąáŹááˇáşááźááˇáşááąáˇááŹáááşáĄáá˝ááş Query Log áááŻáĄááŻáśá¸ááźáŻááźááşá¸á
á
á˝ááşá¸ááąáŹááşáááşáááşááá
áşááťáŹá¸ááᯠááąáŹááşá¸áááŻááťááşá
áŹááŹá¸áĄáá˝ááş áááŻááşááąáŹáşáááşá¸ áááşá¸á ID áĄáá˝ááşá á
á˝ááşá¸ááąáŹááşáááşáááşááá
áşááťáŹá¸ááᯠááźááşáááşáááşáážááşááŹá¸áááˇáş ááąáŹááşá¸áááŻááťááşááťáŹá¸áážááˇáş áážááşáááşá¸áááŻááşááž ááąáŹááşá¸áááŻááťááşááťáŹá¸ááᯠááąáŤááşá¸á
ááşáááş áááŻáĄááşáááşá
ááąáŹááşá¸ááźáŽá áĄáááşá¸ááŻáśá¸ á
á˝ááşá¸ááąáŹááşáááş ááźá
áşáááşáá
áşáᯠááąáŤáşááąáŤááşáááş áĄááťáááşáĄáááĄááťáážááááşá
áááŻáˇááźáąáŹááˇáş ááąáŹááşá¸áááŻááťááş ID áá
áşááŻáĄáá˝ááş á
á˝ááşá¸ááąáŹááşáááş ááźá
áşáááşáá
áşáᯠááźá
áşááąáŤáşáááˇáşáĄááŤá ááááťááąáŹ áááˇáşáááşááąáŹááşáááşáááŻá¸ááťáŹá¸áážááˇáş ááąáŹááşá¸áááŻááťááşá ááááťááąáŹ ááŻááşááąáŹááşááťáááşáážááˇáş ááźáŹááťáááşáááŻáˇááŤáážááááˇáş ááąáŹááşá¸áááŻááťááşááᯠáááŻá¸ááŹá¸áááááˇáşáááşá ááźááşáá˝ááşá¸áááŻáᏠáĄááŻáśá¸ááźáŻá ááąá¸ááŹá¸ááąáŹ áĄááťááşáĄáááşááᯠááá°ááŤá pg_stat_statements - ááŹá¸ááźá
áşááŹá¸áááşá
query á queryid áááŻáážáŹááźáŽá¸ log_query áááŹá¸áážá entry ááᯠupdate ááŻááşááŤá
SELECT *
INTO log_query_rec
FROM log_query
WHERE query_md5hash = log_md5hash AND timepoint = log_timepoint ;
log_query_rec.query=regexp_replace(log_query_rec.query,';+','','g');
FOR pg_stat_history_rec IN
SELECT
queryid ,
query
FROM
pg_stat_db_queries
WHERE
database_id = log_database_id AND
queryid is not null
LOOP
pg_stat_query = pg_stat_history_rec.query ;
pg_stat_query=regexp_replace(pg_stat_query,'n+',' ','g');
pg_stat_query=regexp_replace(pg_stat_query,'t+',' ','g');
pg_stat_query=regexp_replace(pg_stat_query,' +',' ','g');
pg_stat_query=regexp_replace(pg_stat_query,'$.','%','g');
log_query_text = trim(trailing ' ' from log_query_rec.query);
pg_stat_query_text = pg_stat_query;
--SELECT log_query_rec.query like pg_stat_query INTO found_flag ;
IF (log_query_text LIKE pg_stat_query_text) THEN
found_flag = TRUE ;
ELSE
found_flag = FALSE ;
END IF;
IF found_flag THEN
UPDATE log_query SET queryid = pg_stat_history_rec.queryid WHERE query_md5hash = log_md5hash AND timepoint = log_timepoint ;
activity_string = ' updated queryid = '||pg_stat_history_rec.queryid||
' for log_query with id = '||log_query_rec.id
;
RAISE NOTICE '%',activity_string;
EXIT ;
END IF ;
END LOOP ;
afterword
áááŻáˇááźáąáŹááˇáş ááąáŹáşááźááŹá¸ááąáŹ áááşá¸áááşá¸áááş áááşá¸á áĄááŻáśá¸ááťáááŽááąá¸áážááşá¸áĄáá˝ááş áá˝áąáˇáážáááááşá ááąáŤáşááąáŤááşááŹááąáŹ ááąá¸ááźááşá¸áážáŻ ááŻááşááąáŹááşááťááşááťáŹá¸ááᯠááźáąáážááşá¸áááˇáşáĄá፠áá˝á˛ááźááşá¸á áááşááźáŹáážáŻáĄáá˝ááş áĄááťááşáĄáááşááťáŹá¸ áááŻáááŻááážáá áąáááşááźá áşáááşá
ááťá˝ááşáŻááşááááŻááşááąá¸áááŻááşááŹáĄááźááşáĄáááąáŹáşáááşá¸ááąáŹááşá¸á ááąáŤááşá¸ááŻááşááŻááşááŹá¸ááąáŹáĄáááŻááşá¸ááĄáá˝ááşáĄá áŹá¸áááŻáá˝áąá¸ááťááşááźááşá¸áážááˇáşááźáąáŹááşá¸áá˛ááźááşá¸áĄáá˝ááş algorithm áá˝ááşáĄááŻááşááŻááşáááş áááŻáĄááşááąááąá¸áááşá ááąááŻááťáĄáŹá¸ááźááˇáş ááźáżááŹááᯠáááźáąáážááşá¸áááŻááşááąá¸áá°á¸á á áááşáááşá áŹá¸áááŻáˇááąáŹááşá¸ááŤááááˇáşáááşá
ááŤâááąáááˇáşâ ááŤáᏠááŻáśá¸áááźáŹá¸ááŹá¸áá˛áˇ ááŹááşâáááşá¸ááŤáá˛...
source: www.habr.com
