ืืื ืืืืจ!
ืืขืฆืืข ืืึทืจืืกื, Kaggle ืืึธืืกืืื ืึท ืคืึทืจืืขืกื ืฆื ืงืืึทืกืืคืืฆืืจื ืืึทื ื-ืฆืืขื ืืืืืขืจ, Quick Draw Doodle Recognition, ืืื ืืืึธืก, ืฆืืืืฉื ืื ืืขืจืข, ืึท ืงืึธืืขืงืืืื ืคืื R-Scients ืืขื ืืืขื ืึธื ืืืื: , ะธ . ืืืจ ืืืขืื ื ืืฉื ืืึทืฉืจืืึทืื ืื ืคืึทืจืืขืกื ืืื ืืขืืึทื; ืืึธืก ืืื ืฉืืื ืืืจืืืขืงืึธืื ืืื .
ืืึธืก ืืึธื ืืึธื ืขืก ื ืืฉื ืืืืกืืขืึทืจืืขื ืืื ืืขืืึทืึพืคืึทืจืืขืจื, ืึธืืขืจ ืึท ืกื ืืืขืจืืคืืืข ืืขืจืคืึทืจืื ื ืืึธื ืืขื ืืขืงืจืึธืื, ืืขืจืคืืจ ืืืื ืืื ืืขืจืฆืืืื ืื ืงืืืื ืืืขืื ืึท ืฆืึธื ืคืื ืื ืืื ืืขืจืขืกืึทื ืืกืืข ืืื ื ืืฆืืงืกืืข ืืืื ืืืืฃ ืงืึทืื ืืื ืืื ืืขืจ ืืืึธืืขืืืงืขืจ ืึทืจืืขื. ืฆืืืืฉื ืื ืืืกืงืืืืจืืข ืืขืืขืก: ืฉืืืขืจ ืืขืื ืึธื OpenCV, JSON ืคึผืึทืจืกืื ื (ืื ืืืืฉืคืืื ืื ืืขืจืืืื ืื ืื ืึทืืจืืืฉืึทื ืคืื C ++ ืงืึธื ืืื ืกืงืจืืคึผืก ืึธืืขืจ ืคึผืึทืงืึทืืืฉืึทื ืืื ืจ ื ืืฆื Rcpp), ืคึผืึทืจืึทืืขืืขืจืืืึทืืืึธื ืคืื ืกืงืจืืคึผืก ืืื ืืึธืงืขืจืืืึทืืืึธื ืคืื ืื ืืขืฆื ืืืืืื ื. ืื ืงืึธื ืคืื ืืขืจ ืึธื ืืึธื ืืื ืึท ืคืึธืจืขื ืคึผืึทืกืืง ืคึฟืึทืจ ืืืจืืคืืจืื ื ืืื ืื ืืืฆื ืืื .
ืืื ืืึทืื:
1. ืขืคืคืืกืืขื ืืื ืืึทืกืข ืืึทืื ืคืื ืงืกืื ืืื ืื MonetDB ืืึทืืึทืืืืก
ืื ืืึทืื ืืื ืืขื ืคืึทืจืืขืกื ืืขื ืขื ื ืืฉื ืฆืืืขืฉืืขืื ืืื ืื ืคืึธืจืขื ืคืื ืคืึทืจืืืง ืืืืืขืจ, ืึธืืขืจ ืืื ืื ืคืึธืจืขื ืคืื 340 ืงืกืื ืืขืงืขืก (ืืืื ืืขืงืข ืคึฟืึทืจ ืืขืืขืจ ืงืืึทืก) ืืื JSONs ืืื ืคืื ื ืงืึธืืึธืจืืึทื ืึทืฅ. ืืืจื ืงืึทื ืขืงืืื ื ืื ืคืื ืงืื ืืื ืฉืืจืืช, ืืืจ ืืึทืงืืืขื ืึท ืืขืฆื ืืืื ืืขืกืื 256x256 ืืืืืฆืขืื. ืืืื ืคึฟืึทืจ ืืขืืขืจ ืจืขืงืึธืจื ืขืก ืืื ืึท ืคืืจืืข ืืืึธืก ืื ืืืงืืืฅ ืฆื ืื ืืืื ืืื ืจืืืืืง ืืขืจืงืขื ื ืืืจื ืื ืงืืึทืกืกืืคืืขืจ ืืขื ืืฆื ืืื ืืขืจ ืฆืืื ืืืขื ืื ืืึทืืึทืกืขื ืืื ืืขืืืืื, ืึท ืฆืืืื-ืืืชืืืช ืงืึธื ืคืื ืื ืืืื ื ืคืื ืืืืื ืึธืจื ืคืื ืืขืจ ืืืืจ ืคืื ืื ืืืื, ืึท ืืื ืฆืืง ืืืืขื ืืืืขื, ืึท ืฆืืื ืกืืึทืืคึผ. ืืื ืึท ืงืืึทืก ื ืึธืืขื ืืืึธืก ืืืืึทืื ืื ืืขืงืข ื ืึธืืขื. ื ืกืืืคึผืืึทืคืืื ืืืขืจืกืืข ืคืื โโืืขืจ ืึธืจืืืื ืขื ืืึทืื ืืืืื 7.4 ืืืืืืืื ืืื ืื ืึทืจืงืืืื ืืื ืืขืขืจืขื 20 ืืืืืืืื ื ืึธื ืึทื ืคึผืึทืงืื ื, ืื ืคืื ืืึทืื ื ืึธื ืึทื ืคึผืึทืงืื ื ื ืขืื 240 ืืืืืืืื. ืื ืึธืจืืึทื ืืืืขืจื ืื ืฉืืจื ืึทื ืืืืืข ืืืขืจืกืืขืก ืจืืคึผืจืึทืืืกื ืื ืืขืืืข ืืจืึทืืืื ืืก, ืืืึทืืฉ ืื ืคืื ืืืขืจืกืืข ืืื ืืืขืจืืง. ืืื ืงืืื ืคืึทื, ืกืืึธืจืื ื 50 ืืืืืึธื ืืืืืขืจ ืืื ืืจืึทืคืืง ืืขืงืขืก ืึธืืขืจ ืืื ืื ืคืึธืจืขื ืคืื ืขืจืืื ืืื ืืืืื ืืขืจืขืื ื ืืื ืึทื ืคึผืจืึทืคืืืึทืืึทื, ืืื ืืืจ ืืึทืฉืืึธืกื ืฆื ืฆืื ืืืคืืืกื ืึทืืข ืงืกืื ืืขืงืขืก ืคืื ืื ืึทืจืงืืืื. train_simplified.zip ืืื ืื ืืึทืืึทืืืืก ืืื ืกืึทืืกืึทืงืืืึทื ื ืืืจ ืืืืืขืจ ืคืื ืื ืคืืจืืื ืื ืืจืืืก "ืืืืฃ ืื ืคืืืขื" ืคึฟืึทืจ ืืขืืขืจ ืคึผืขืงื.
ื ืืขืืื ื-ืคึผืจืึธืืืขื ืกืืกืืขื ืืื ืืืืกืืขืจืืืืืื ืืื ืื DBMS ืืึธื ืขืืื, ื ืืืืื ืึท ืืืคึผืืึทืืขื ืืืืฉืึทื ืคึฟืึทืจ R ืืื ืึท ืคึผืขืงื . ืืขืจ ืคึผืขืงื ืืืื ืึทื ืขืืืขืืื ืืืขืจืกืืข ืคืื โโโโืื ืืึทืืึทืืืืก ืกืขืจืืืขืจ ืืื ืึทืืึทืื ืืืจ ืฆื ืงืืืึทืื ืื ืกืขืจืืืขืจ ืืืืื ืคึฟืื ืึท R ืกืขืกืืข ืืื ืึทืจืืขื ืืื ืืื ืืึธืจื. ืงืจืืืืืื ื ืึท ืืึทืืึทืืืืก ืืื ืงืึทื ืขืงืืื ื ืฆื ืขืก ืืขื ืขื ืืขืืื ืืื ืืืื ืืึทืคึฟืขื:
con <- DBI::dbConnect(drv = MonetDBLite::MonetDBLite(), Sys.getenv("DBDIR"))ืืืจ ืืึทืจืคึฟื ืฆื ืฉืึทืคึฟื ืฆืืืื ืืืฉื: ืืืื ืขืจ ืคึฟืึทืจ ืึทืืข ืืึทืื, ืื ืื ืืขืจืข ืคึฟืึทืจ ืกืขืจืืืืก ืืื ืคึฟืึธืจืืึทืฆืืข ืืืขืื ืืึทืื ืืึธืืืื ืืขืงืขืก (ื ืืฆืืง ืืืื ืขืคึผืขืก ืืืื ืคืึทืืฉ ืืื ืืขืจ ืคึผืจืึธืฆืขืก ืืืื ืืืื ืจืืืืื ื ืึธื ืืึทืื ืืึธืืืื ื ืขืืืขืืข ืืขืงืขืก):
ืฉืืคื ืืึทืืืขืก
if (!DBI::dbExistsTable(con, "doodles")) {
DBI::dbCreateTable(
con = con,
name = "doodles",
fields = c(
"countrycode" = "char(2)",
"drawing" = "text",
"key_id" = "bigint",
"recognized" = "bool",
"timestamp" = "timestamp",
"word" = "text"
)
)
}
if (!DBI::dbExistsTable(con, "upload_log")) {
DBI::dbCreateTable(
con = con,
name = "upload_log",
fields = c(
"id" = "serial",
"file_name" = "text UNIQUE",
"uploaded" = "bool DEFAULT false"
)
)
}ืื ืคืึทืกืืึทืกื ืืืขื ืฆื ืืึธืื ืืึทืื ืืื ืื ืืึทืืึทืืืืก ืืื ืฆื ืืืืื ื ืึธืืืึทืื ืงืกืื ืืขืงืขืก ื ืืฆื SQL - ืืึทืคึฟืขื COPY OFFSET 2 INTO tablename FROM path USING DELIMITERS ',','n','"' NULL AS '' BEST EFFORTืืื tablename - ืืืฉ ื ืึธืืขื ืืื path - ืืขืจ ืืจื ืฆื ืืขืจ ืืขืงืข. ืืฉืขืช ืืจืืขืื ืืื ืืขื ืึทืจืงืืืื, ืขืก ืืื ืืขืืืขื ืืืกืงืึทืืืขืจื ืึทื ืื ืืขืืืื-ืืื ืืืคึผืืึทืืขื ืืืืฉืึทื unzip ืืื R ืืื ื ืืฉื ืึทืจืืขืื ืจืืืืืง ืืื ืึท ื ืืืขืจ ืคืื ืืขืงืขืก ืคึฟืื ืืขืจ ืึทืจืงืืืื, ืึทืืื ืืืจ ืืขืืืืื ื ืื ืกืืกืืขื unzip (ื ืืฆื ืืขื ืคึผืึทืจืึทืืขืืขืจ getOption("unzip")).
ืคืื ืงืฆืืข ืคึฟืึทืจ ืฉืจืืืื ืฆื ืื ืืึทืืึทืืืืก
#' @title ะะทะฒะปะตัะตะฝะธะต ะธ ะทะฐะณััะทะบะฐ ัะฐะนะปะพะฒ
#'
#' @description
#' ะะทะฒะปะตัะตะฝะธะต CSV-ัะฐะนะปะพะฒ ะธะท ZIP-ะฐัั
ะธะฒะฐ ะธ ะทะฐะณััะทะบะฐ ะธั
ะฒ ะฑะฐะทั ะดะฐะฝะฝัั
#'
#' @param con ะะฑัะตะบั ะฟะพะดะบะปััะตะฝะธั ะบ ะฑะฐะทะต ะดะฐะฝะฝัั
(ะบะปะฐัั `MonetDBEmbeddedConnection`).
#' @param tablename ะะฐะทะฒะฐะฝะธะต ัะฐะฑะปะธัั ะฒ ะฑะฐะทะต ะดะฐะฝะฝัั
.
#' @oaram zipfile ะััั ะบ ZIP-ะฐัั
ะธะฒั.
#' @oaram filename ะะผั ัะฐะนะปะฐ ะฒะฝััะธ ZIP-ะฐัั
ะธะฒะฐ.
#' @param preprocess ะคัะฝะบัะธั ะฟัะตะดะพะฑัะฐะฑะพัะบะธ, ะบะพัะพัะฐั ะฑัะดะตั ะฟัะธะผะตะฝะตะฝะฐ ะธะทะฒะปะตััะฝะฝะพะผั ัะฐะนะปั.
#' ะะพะปะถะฝะฐ ะฟัะธะฝะธะผะฐัั ะพะดะธะฝ ะฐัะณัะผะตะฝั `data` (ะพะฑัะตะบั `data.table`).
#'
#' @return `TRUE`.
#'
upload_file <- function(con, tablename, zipfile, filename, preprocess = NULL) {
# ะัะพะฒะตัะบะฐ ะฐัะณัะผะตะฝัะพะฒ
checkmate::assert_class(con, "MonetDBEmbeddedConnection")
checkmate::assert_string(tablename)
checkmate::assert_string(filename)
checkmate::assert_true(DBI::dbExistsTable(con, tablename))
checkmate::assert_file_exists(zipfile, access = "r", extension = "zip")
checkmate::assert_function(preprocess, args = c("data"), null.ok = TRUE)
# ะะทะฒะปะตัะตะฝะธะต ัะฐะนะปะฐ
path <- file.path(tempdir(), filename)
unzip(zipfile, files = filename, exdir = tempdir(),
junkpaths = TRUE, unzip = getOption("unzip"))
on.exit(unlink(file.path(path)))
# ะัะธะผะตะฝัะตะผ ััะฝะบัะธั ะฟัะตะดะพะฑัะฐะฑะพัะบะธ
if (!is.null(preprocess)) {
.data <- data.table::fread(file = path)
.data <- preprocess(data = .data)
data.table::fwrite(x = .data, file = path, append = FALSE)
rm(.data)
}
# ะะฐะฟัะพั ะบ ะะ ะฝะฐ ะธะผะฟะพัั CSV
sql <- sprintf(
"COPY OFFSET 2 INTO %s FROM '%s' USING DELIMITERS ',','n','"' NULL AS '' BEST EFFORT",
tablename, path
)
# ะัะฟะพะปะฝะตะฝะธะต ะทะฐะฟัะพัะฐ ะบ ะะ
DBI::dbExecute(con, sql)
# ะะพะฑะฐะฒะปะตะฝะธะต ะทะฐะฟะธัะธ ะพะฑ ััะฟะตัะฝะพะน ะทะฐะณััะทะบะต ะฒ ัะปัะถะตะฑะฝัั ัะฐะฑะปะธัั
DBI::dbExecute(con, sprintf("INSERT INTO upload_log(file_name, uploaded) VALUES('%s', true)",
filename))
return(invisible(TRUE))
}ืืืื ืืืจ ืืึทืจืคึฟื ืฆื ืืืขืจืืึทืื ืื ืืืฉ ืืืืืขืจ ืืืจ ืฉืจืืึทืื ืขืก ืฆื ืื ืืึทืืึทืืืืก, ืขืก ืืื ืืขื ืื ืฆื ืคืึธืจื ืื ืึทืจืืืืขื ื preprocess ืคืื ืงืฆืืข ืืืึธืก ืืืขื ืืืขืจืืึทืื ืื ืืึทืื.
ืงืึธื ืคึฟืึทืจ ืกืึทืงืืืขื ืืฉืึทื ืืึธืืืื ื ืืึทืื ืืื ืื ืืึทืืึทืืืืก:
ืฉืจืืืื ืืึทืื ืฆื ืื ืืึทืืึทืืืืก
# ะกะฟะธัะพะบ ัะฐะนะปะพะฒ ะดะปั ะทะฐะฟะธัะธ
files <- unzip(zipfile, list = TRUE)$Name
# ะกะฟะธัะพะบ ะธัะบะปััะตะฝะธะน, ะตัะปะธ ัะฐััั ัะฐะนะปะพะฒ ัะถะต ะฑัะปะฐ ะทะฐะณััะถะตะฝะฐ
to_skip <- DBI::dbGetQuery(con, "SELECT file_name FROM upload_log")[[1L]]
files <- setdiff(files, to_skip)
if (length(files) > 0L) {
# ะะฐะฟััะบะฐะตะผ ัะฐะนะผะตั
tictoc::tic()
# ะัะพะณัะตัั ะฑะฐั
pb <- txtProgressBar(min = 0L, max = length(files), style = 3)
for (i in seq_along(files)) {
upload_file(con = con, tablename = "doodles",
zipfile = zipfile, filename = files[i])
setTxtProgressBar(pb, i)
}
close(pb)
# ะััะฐะฝะฐะฒะปะธะฒะฐะตะผ ัะฐะนะผะตั
tictoc::toc()
}
# 526.141 sec elapsed - ะบะพะฟะธัะพะฒะฐะฝะธะต SSD->SSD
# 558.879 sec elapsed - ะบะพะฟะธัะพะฒะฐะฝะธะต USB->SSDืื ืืึธืืืื ื ืฆืืื ืคืื ืืึทืื ืงืขื ืืืึทืื ืืืคึผืขื ืืื ื ืืืืฃ ืื ืืืืงืืึทื ืงืขืจืึทืงืืขืจืืกืืืงืก ืคืื ืื ืคืึธืจ ืืขื ืืฆื. ืืื ืืื ืืืขืจ ืคืึทื, ืืืืขื ืขื ืืื ืฉืจืืืื ืืื ืืืื SSD ืึธืืขืจ ืคึฟืื ืึท ืืืืฅ ืคืึธืจ (ืืงืืจ ืืขืงืข) ืฆื ืึท SSD (DB) ื ืขืื ืืืืื ืืงืขืจ ืืื 10 ืืื ืื.
ืขืก ื ืขืื ื ืึธื ืึท ืืืกื ืกืขืงืื ืืขืก ืฆื ืฉืึทืคึฟื ืึท ืืืึทื ืืื ืึท ืื ืืึทืืืฉืขืจ ืงืืึทืก ืคืืจืืข ืืื ืึทื ืืื ืืขืงืก ืืืึทื (ORDERED INDEX) ืืื ืฉืืจื ื ืืืขืจื ืืืจื ืืืึธืก ืึทืืืขืจืืืืืฉืึทื ื ืืืขื ืืืื ืกืึทืืคึผืึทืื ืืืขื ืงืจืืืืืื ื ืืึทืืฉืึทื:
ืงืจืืืืืื ื ื ืึธื ืฉืคืืืื ืืื ืืื ืืขืงืก
message("Generate lables")
invisible(DBI::dbExecute(con, "ALTER TABLE doodles ADD label_int int"))
invisible(DBI::dbExecute(con, "UPDATE doodles SET label_int = dense_rank() OVER (ORDER BY word) - 1"))
message("Generate row numbers")
invisible(DBI::dbExecute(con, "ALTER TABLE doodles ADD id serial"))
invisible(DBI::dbExecute(con, "CREATE ORDERED INDEX doodles_id_ord_idx ON doodles(id)"))ืฆื ืกืึธืืืืข ืื ืคึผืจืึธืืืขื ืคืื ืงืจืืืืืื ื ืึท ืคึผืขืงื ืืืืฃ ืื ืคืืืขื, ืืืจ ืืึทืจืคึฟื ืฆื ืืขืจืืจืืืื ืื ืืึทืงืกืืืื ืืืืงืืึทื ืคืื ืขืงืกืืจืึทืงื ืจืึทื ืืึธื ืจืึธืื ืคืื ืื ืืืฉ doodles. ืคึฟืึทืจ ืืขื ืืืจ ืืขืืืืื ื 3 ืืจืืงืก. ืืขืจ ืขืจืฉืืขืจ ืืื ืืขืืืขื ืฆื ืจืขืืืฆืืจื ืื ืืืืขื ืฉืึทื ืึทืืืื ืคืื ืืขื ืืืคึผ ืืืึธืก ืกืืึธืจื ืื ืึธืืกืขืจืืืึทืฆืืข ืฉืืึทื. ืืื ืืขืจ ืึธืจืืืื ืขื ืืึทืื ืฉืืขืื, ืืขืจ ืืืคึผ ืคืืจืืื ืื ืฆื ืงืจืึธื ืื ืฉืืึทื ืืื bigint, ืึธืืขืจ ืื ื ืืืขืจ ืคืื ืึทืืืขืจืืืืืฉืึทื ื ืืืื ืขืก ืืขืืืขื ืฆื ืคึผืึทืกืืง ืืืืขืจ ืืืืขื ืืืคืืฆืืจื, ืืืืึทื ืฆื ืื ืึธืจืืึทื ืึทื ื ืืืขืจ, ืืื ืืขื ืืืคึผ int. ืืขืจ ืืืื ืืื ืคืื ืคืึทืกืืขืจ ืืื ืืขื ืคืึทื. ืื ืฆืืืืืืข ืงืื ืฅ ืืื ืืขืืืขื ืฆื ื ืืฆื ORDERED INDEX โ ืื ืจ ืืฒื ืข ื ืืขืงืืืข ื ืฆ ื ืืข ื ืืืฉืืืก , ืขืืคืืจืืฉ , ืืืจืืืขืืื ื ืื ืข ืคืืจืื ืืฐืืจืืืขื ืข . ืื ืืจืื ืืื ืฆื ื ืืฆื ืคึผืึทืจืึทืืขืืขืจืืืื ืงืืืืจืื. ืื ืขืกืึทื ืก ืคืื ืืขื ืืืคึฟื ืืื ืฆื ืืืกืคืืจื ืืขื ืืึทืคึฟืขื ืึทืืึธื PREPARE ืืื ืกืึทืืกืึทืงืืืึทื ื ื ืืฆื ืคืื ืึท ืฆืืืขืืจืืื ืืืืกืืจืืง ืืืขื ืงืจืืืืืื ื ืึท ืืื ืื ืคืื ืงืืืืจืื ืคืื ืืขืจ ืืขืืืืงืขืจ ืืืคึผ, ืึธืืขืจ ืืื ืคืึทืงื ืขืก ืืื ืึท ืืืึทืืข ืืื ืคืึทืจืืืืึทื ืืื ืึท ืคึผืฉืื ืืืื ืขืจ SELECT ืขืก ืืื ืืขืืืขื ืืื ืื ืงืืื ืคืื ืกืืึทืืืกืืืฉ ืืขืืช.
ืืขืจ ืคึผืจืึธืฆืขืก ืคืื ืืคึผืืึธืึทืืื ื ืืึทืื ืงืึทื ืกืืื ื ืื ืืขืจ ืืื 450 ืืขืืืืืืื ืคืื ืืึทืจืึทื. ืืึธืก ืืื, ืืขืจ ืืืกืงืจืืืื ืฆืืืึทื ื ืึทืืึทืื ืืืจ ืฆื ืืึทื ืืึทืืึทืกืขืฅ ืืืืืื ื ืืขื ืก ืคืื ืืืืืืืื ืืืืฃ ืึผืืขื ืงืืื ืืืืืฉืขื ืืึทืื ืืืึทืจื, ืึทืจืืึทื ืืขืจืขืื ื ืขืืืขืืข ืืืื-ืืจืขื ืืขืืืืกืขืก, ืืืึธืก ืืื ืฉืืื ืงืื.
ืึทืืข ืืืึธืก ืืืืืื ืืื ืฆื ืืขืกืื ืื ืืืืงืืึทื ืคืื ืจืืืจืืืืื ื (ืจืึทื ืืึธื) ืืึทืื ืืื ืึธืคึผืฉืึทืฆื ืื ืกืงืืืืื ื ืืืขื ืืืกืืขืจืื ื ืืึทืืฉืึทื ืคืื ืคืึทืจืฉืืืขื ืข ืกืืืขืก:
ืืึทืืึทืืึทืกืข ืืขื ืืฉืืึทืจืง
library(ggplot2)
set.seed(0)
# ะะพะดะบะปััะตะฝะธะต ะบ ะฑะฐะทะต ะดะฐะฝะฝัั
con <- DBI::dbConnect(MonetDBLite::MonetDBLite(), Sys.getenv("DBDIR"))
# ะคัะฝะบัะธั ะดะปั ะฟะพะดะณะพัะพะฒะบะธ ะทะฐะฟัะพัะฐ ะฝะฐ ััะพัะพะฝะต ัะตัะฒะตัะฐ
prep_sql <- function(batch_size) {
sql <- sprintf("PREPARE SELECT id FROM doodles WHERE id IN (%s)",
paste(rep("?", batch_size), collapse = ","))
res <- DBI::dbSendQuery(con, sql)
return(res)
}
# ะคัะฝะบัะธั ะดะปั ะธะทะฒะปะตัะตะฝะธั ะดะฐะฝะฝัั
fetch_data <- function(rs, batch_size) {
ids <- sample(seq_len(n), batch_size)
res <- DBI::dbFetch(DBI::dbBind(rs, as.list(ids)))
return(res)
}
# ะัะพะฒะตะดะตะฝะธะต ะทะฐะผะตัะฐ
res_bench <- bench::press(
batch_size = 2^(4:10),
{
rs <- prep_sql(batch_size)
bench::mark(
fetch_data(rs, batch_size),
min_iterations = 50L
)
}
)
# ะะฐัะฐะผะตััั ะฑะตะฝัะผะฐัะบะฐ
cols <- c("batch_size", "min", "median", "max", "itr/sec", "total_time", "n_itr")
res_bench[, cols]
# batch_size min median max `itr/sec` total_time n_itr
# <dbl> <bch:tm> <bch:tm> <bch:tm> <dbl> <bch:tm> <int>
# 1 16 23.6ms 54.02ms 93.43ms 18.8 2.6s 49
# 2 32 38ms 84.83ms 151.55ms 11.4 4.29s 49
# 3 64 63.3ms 175.54ms 248.94ms 5.85 8.54s 50
# 4 128 83.2ms 341.52ms 496.24ms 3.00 16.69s 50
# 5 256 232.8ms 653.21ms 847.44ms 1.58 31.66s 50
# 6 512 784.6ms 1.41s 1.98s 0.740 1.1m 49
# 7 1024 681.7ms 2.72s 4.06s 0.377 2.16m 49
ggplot(res_bench, aes(x = factor(batch_size), y = median, group = 1)) +
geom_point() +
geom_line() +
ylab("median time, s") +
theme_minimal()
DBI::dbDisconnect(con, shutdown = TRUE)
2. ืคึผืจืืคึผืขืจืื ื ืืึทืืฉืึทื
ืืขืจ ืืื ืฆืขืจ ืคึผืขืงื ืฆืืืจืืืืื ื ืคึผืจืึธืฆืขืก ืืืฉืืืื ืคืื ืื ืคืืืืขื ืืข ืกืืขืคึผืก:
- ืคึผืึทืจืกืื ื ืขืืืขืืข JSONs ืืื ืืืขืงืืึธืจืก ืคืื ืกืืจืื ืืก ืืื ืงืึธืืึธืจืืึทื ืึทืฅ ืคืื ืคืื ืงืื.
- ืฆืืืืขื ืื ื ืืื ื ืฉืืจืืช ืืืืืจื ืืืืฃ ืื ืงืึธืืึธืจืืึทื ืึทืฅ ืคืื ืืืืืื ืืืืฃ ืึท ืืืื ืคืื ืื ืคืืจืืื ืื ืืจืืืก (ืืืฉื, 256 ร 256 ืึธืืขืจ 128 ร 128).
- ืงืึทื ืืืขืจืืื ื ืื ืจืืืึทืืืื ื ืืืืืขืจ ืืื ืึท ืืขื ืกืึธืจ.
ืืื ืึท ืืืื ืคืื ืื ืคืึทืจืืขืกื ืฆืืืืฉื ืคึผืืืืึธื ืงืขืจื ืึทืื, ืื ืคึผืจืึธืืืขื ืืื ืืขืืืขื ืกืึทืืืื ืืคึฟืจื ื ืืฆื OpenCV. ืืืื ืขืจ ืคืื ืื ืกืืืคึผืืึทืกื ืืื ืืขืจืกื ืงืืึธืจ ืืื ืืขืจ ืืึธื ืึทื ืึทืืึธืืืขืก ืืื ืจ ืืืึธืื ืงืืงื ืืื ืืึธืก:
ืืืคึผืืึทืืขื ืื ื JSON ืฆื Tensor ืงืึทื ืืืขืจืืฉืึทื ืืื R
r_process_json_str <- function(json, line.width = 3,
color = TRUE, scale = 1) {
# ะะฐััะธะฝะณ JSON
coords <- jsonlite::fromJSON(json, simplifyMatrix = FALSE)
tmp <- tempfile()
# ะฃะดะฐะปัะตะผ ะฒัะตะผะตะฝะฝัะน ัะฐะนะป ะฟะพ ะทะฐะฒะตััะตะฝะธั ััะฝะบัะธะธ
on.exit(unlink(tmp))
png(filename = tmp, width = 256 * scale, height = 256 * scale, pointsize = 1)
# ะัััะพะน ะณัะฐัะธะบ
plot.new()
# ะ ะฐะทะผะตั ะพะบะฝะฐ ะณัะฐัะธะบะฐ
plot.window(xlim = c(256 * scale, 0), ylim = c(256 * scale, 0))
# ะฆะฒะตัะฐ ะปะธะฝะธะน
cols <- if (color) rainbow(length(coords)) else "#000000"
for (i in seq_along(coords)) {
lines(x = coords[[i]][[1]] * scale, y = coords[[i]][[2]] * scale,
col = cols[i], lwd = line.width)
}
dev.off()
# ะัะตะพะฑัะฐะทะพะฒะฐะฝะธะต ะธะทะพะฑัะฐะถะตะฝะธั ะฒ 3-ั
ะผะตัะฝัะน ะผะฐััะธะฒ
res <- png::readPNG(tmp)
return(res)
}
r_process_json_vector <- function(x, ...) {
res <- lapply(x, r_process_json_str, ...)
# ะะฑัะตะดะธะฝะตะฝะธะต 3-ั
ะผะตัะฝัั
ะผะฐััะธะฒะพะฒ ะบะฐััะธะฝะพะบ ะฒ 4-ั
ะผะตัะฝัะน ะฒ ัะตะฝะทะพั
res <- do.call(abind::abind, c(res, along = 0))
return(res)
}ืฆืืืืขื ืื ื ืืื ืืืจืืืขืงืึธืื ืืื ื ืึธืจืืึทื R ืืืฉืืจืื ืืื ืืขืจืืืขืืืขื ืืื ืึท ืฆืืืืืืืืืืืข PNG ืกืืึธืจื ืืื ืืึทืจืึทื (ืืื ืืื ืืงืก, ืฆืืืืืืืืืืืข R ืืืจืขืงืืขืจืื ืืขื ืขื ืืืื ืืื ืื ืืืขืืืืืึทืืขืจ /tmp, ืืึธืื ืืขื ืืื ืืึทืจืึทื). ืืขืจ ืืขืงืข ืืื ืืขืืึธืื ืืืืขื ืขื ืืื ืึท ืืจืื-ืืืืขื ืฉืึทื ืึทื ืืขื ืืข ืืื ื ืืืขืจื ืจืืื ืืืฉืื ื ืคืื 0 ืฆื 1. ืืึธืก ืืื ืืืืืืืง ืืืืึทื ืึท ืืขืจ ืงืึทื ืืืขื ืฉืึทื ืึทื ืืืคึผ ืืืึธืื ืืืื ืืืืขื ืขื ืืื ืึท ืจืื ืืขื ืืข ืืื ืืขืงืก ืงืึธืืืจ ืงืึธืืื.
ืืื ืก ืคึผืจืืืืจื ืื ืจืขืืืืืึทื:
zip_file <- file.path("data", "train_simplified.zip")
csv_file <- "cat.csv"
unzip(zip_file, files = csv_file, exdir = tempdir(),
junkpaths = TRUE, unzip = getOption("unzip"))
tmp_data <- data.table::fread(file.path(tempdir(), csv_file), sep = ",",
select = "drawing", nrows = 10000)
arr <- r_process_json_str(tmp_data[4, drawing])
dim(arr)
# [1] 256 256 3
plot(magick::image_read(arr))
ืืขืจ ืคึผืขืงื ืืื ืืืขื ืืืื ืืขืฉืืคื ืืื ืืืื:
res <- r_process_json_vector(tmp_data[1:4, drawing], scale = 0.5)
str(res)
# num [1:4, 1:128, 1:128, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
# - attr(*, "dimnames")=List of 4
# ..$ : NULL
# ..$ : NULL
# ..$ : NULL
# ..$ : NULLืื ืืืคึผืืึทืืขื ืืืืฉืึทื ืืื ืืขืืืขื ืกืืืึธืคึผืืืืึทื ืคึฟืึทืจ ืืื ืื, ืืืืึทื ืื ืคืึธืจืืืจืื ื ืคืื ืืจืืืก ืืึทืืฉืึทื ื ืขืื ืึท ืึธืจื ืืืขื ืืึทื ื ืฆืืื, ืืื ืืืจ ืืึทืฉืืึธืกื ืฆื ื ืืฆื ืื ืืขืจืคืึทืจืื ื ืคืื ืืื ืืืขืจ ืืืจืื ืืื ืึท ืฉืืึทืจืง ืืืืืืึธืืขืง. OpenCV. ืืื ืืขืจ ืฆืืื ืขืก ืืื ืืขืืืขื ืงืืื ืคืึทืจืืืง ืคึผืขืงื ืคึฟืึทืจ ืจ (ืขืก ืืื ืืึธืจื ืื ืืืฆื), ืึทืืื ืึท ืืื ืืืึทื ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืื ืคืืจืืื ืื ืคืึทื ืืงืฉืึทื ืึทืืืื ืืื ืืขืืืขื ืืขืฉืจืืื ืืื C ++ ืืื ืื ืึทืืจืืืฉืึทื ืืื ืจ ืงืึธื ื ืืฆื Rcpp.
ืฆื ืกืึธืืืืข ืืขื ืคึผืจืึธืืืขื, ืื ืคืืืืขื ืืข ืคึผืึทืงืึทืืืฉืึทื ืืื ืืืืืจืขืจืื ืืขื ืขื ืืขื ืืฆื:
-
OpenCV ืคึฟืึทืจ ืืจืืขืื ืืื ืืืืืขืจ ืืื ืฆืืืืขื ืื ื ืฉืืจืืช. ืืขืืืืื ื ืคืึทืจ-ืืื ืกืืึทืืืจื ืกืืกืืขื ืืืืืจืขืจืื ืืื ืืขืืขืจ ืืขืงืขืก, ืืื ืืขืืื ื ืืื ืืื ืึทืืืฉ ืคึฟืึทืจืืื ืืื ื.
-
xtensor ืคึฟืึทืจ ืืจืืขืื ืืื ืืืืืืืืืขื ืกืืึธื ืึทื ืขืจืืื ืืื ืืขื ืกืขืจ. ืืืจ ืืขืืืืื ื ืืขืืขืจ ืืขืงืขืก ืึทืจืืึทื ืืขืจืขืื ื ืืื ืื R ืคึผืขืงื ืืื ืื ืืขืืืข ื ืึธืืขื. ืื ืืืืืืึธืืขืง ืึทืืึทืื ืืืจ ืฆื ืึทืจืืขืื ืืื ืืืืืืืืืขื ืกืืึธื ืึทื ืขืจืืื, ืืืืืข ืืื ืจืืืขืจื ืืืืคึผื ืืื ืืืึทื ืืืืคึผื ืกืืจ.
-
ndjson ืคึฟืึทืจ ืคึผืึทืจืกืื ื JSON. ืื ืืืืืืึธืืขืง ืืื ืืขื ืืฆื ืืื xtensor ืืืืืึธืืึทืืืฉ ืืืื ืขืก ืืื ืคืึธืจืฉืืขืื ืืื ืื ืคึผืจืืืขืงื.
-
RcppThread ืคึฟืึทืจ ืึธืจืืึทื ืืืืื ื ืืึทืืื-ืืจืขืืื ืคึผืจืึทืกืขืกืื ื ืคืื ืึท ืืืขืงืืึธืจ ืคึฟืื JSON. ืืขืืืืื ื ืื ืืขืืขืจ ืืขืงืขืก ืฆืืืขืฉืืขืื ืืืจื ืืขื ืคึผืขืงื. ืคึฟืื ืืขืจ ืคืึธืืงืก RcppParallel ืืขืจ ืคึผืขืงื, ืฆืืืืฉื ืื ืืขืจืข ืืืื, ืืื ืึท ืืขืืืื-ืืื ืฉืืืืฃ ืืืขืจืจืืึทืกื ืืขืงืึทื ืืืึทื.
ืขืก ืืึธื ืืืึทื ืื ืืขืืืืื ืึทื xtensor ืขืก ืืื ืืขืืืขื ืึท ืืึทืืกืขื ื: ืืื ืึทืืืฉืึทื ืฆื ืืขื ืคืึทืงื ืึทื ืขืก ืืื ืึท ืืจืืื ืคืึทื ืืงืฉืึทื ืึทืืืื ืืื ืืืื ืคืึธืจืฉืืขืืื ื, ืื ืืขืืืขืืึธืคึผืขืจืก ืืขื ืขื ืืขืืืขื ืืึทื ืฅ ืึธืคึผืจืืคืืง ืืื ืืขืขื ืืคืขืจื ืคึฟืจืืื ืืืืื ืืื ืืื ืืขืืึทื. ืืื ืืืืขืจ ืืืืฃ, ืขืก ืืื ืืขืืืขื ืืขืืืขื ืฆื ืื ืกืืจืืืขื ื ืืจืึทื ืกืคืขืจืืืืฉืึทื ื ืคืื OpenCV ืืึทืืจืืฅ ืืื ืงืกืืขื ืกืึธืจ ืืขื ืกืขืจ, ืืื ืืขืืื ื ืืื ืึท ืืืขื ืฆื ืคืึทืจืืื ืื 3-ืืืืขื ืฉืึทื ืึทื ืืืื ืืขื ืกืขืจ ืืื ืึท 4-ืืืืขื ืฉืึทื ืึทื ืืขื ืกืขืจ ืคืื ืื ืจืืืืืง ืืืกืืขืกืืื ื (ืื ืคึผืขืงื ืืื).
ืืึทืืขืจืืึทืืก ืคึฟืึทืจ ืืขืจื ืขื Rcpp, Xtensor ืืื RcppThread
ืฆื ืฆืื ืืืคื ืขืืขื ืืขืงืขืก ืืืึธืก ื ืืฆื ืกืืกืืขื ืืขืงืขืก ืืื ืืื ืึทืืืฉ ืคึฟืึทืจืืื ืืื ื ืืื ืืืืืจืขืจืื ืืื ืกืืึทืืืจื ืืืืฃ ืื ืกืืกืืขื, ืืืจ ืืขืืืืื ื ืื ืคึผืืืืื ืืขืงืึทื ืืืึทื ืืืคึผืืึทืืขื ืึทื ืืื ืืขื ืคึผืขืงื Rcpp. ืฆื ืืืืืึธืืึทืืืฉ ืืขืคึฟืื ืขื ืคึผืึทืืก ืืื ืคืืึทืืก, ืืืจ ืืขืืืืื ื ืึท ืคืึธืืงืก ืืื ืืงืก ื ืืฆื pkg-config.
ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืื Rcpp ืคึผืืืืื ืคึฟืึทืจ ื ืืฆื ืื OpenCV ืืืืืืึธืืขืง
Rcpp::registerPlugin("opencv", function() {
# ะะพะทะผะพะถะฝัะต ะฝะฐะทะฒะฐะฝะธั ะฟะฐะบะตัะฐ
pkg_config_name <- c("opencv", "opencv4")
# ะะธะฝะฐัะฝัะน ัะฐะนะป ััะธะปะธัั pkg-config
pkg_config_bin <- Sys.which("pkg-config")
# ะัะพะฒัะตะบะฐ ะฝะฐะปะธัะธั ััะธะปะธัั ะฒ ัะธััะตะผะต
checkmate::assert_file_exists(pkg_config_bin, access = "x")
# ะัะพะฒะตัะบะฐ ะฝะฐะปะธัะธั ัะฐะนะปะฐ ะฝะฐัััะพะตะบ OpenCV ะดะปั pkg-config
check <- sapply(pkg_config_name,
function(pkg) system(paste(pkg_config_bin, pkg)))
if (all(check != 0)) {
stop("OpenCV config for the pkg-config not found", call. = FALSE)
}
pkg_config_name <- pkg_config_name[check == 0]
list(env = list(
PKG_CXXFLAGS = system(paste(pkg_config_bin, "--cflags", pkg_config_name),
intern = TRUE),
PKG_LIBS = system(paste(pkg_config_bin, "--libs", pkg_config_name),
intern = TRUE)
))
})ืืื ืึท ืจืขืืืืืึทื ืคืื ืื ืคึผืืืืื ืึธืคึผืขืจืึทืฆืืข, ืื ืคืืืืขื ืืข ืืืึทืืืขืก ืืืขื ืืืื ืกืึทืืกืืึทืืืืึทื ืืขืฉืึทืก ืื ืืึทืืืื ื ืคึผืจืึธืฆืขืก:
Rcpp:::.plugins$opencv()$env
# $PKG_CXXFLAGS
# [1] "-I/usr/include/opencv"
#
# $PKG_LIBS
# [1] "-lopencv_shape -lopencv_stitching -lopencv_superres -lopencv_videostab -lopencv_aruco -lopencv_bgsegm -lopencv_bioinspired -lopencv_ccalib -lopencv_datasets -lopencv_dpm -lopencv_face -lopencv_freetype -lopencv_fuzzy -lopencv_hdf -lopencv_line_descriptor -lopencv_optflow -lopencv_video -lopencv_plot -lopencv_reg -lopencv_saliency -lopencv_stereo -lopencv_structured_light -lopencv_phase_unwrapping -lopencv_rgbd -lopencv_viz -lopencv_surface_matching -lopencv_text -lopencv_ximgproc -lopencv_calib3d -lopencv_features2d -lopencv_flann -lopencv_xobjdetect -lopencv_objdetect -lopencv_ml -lopencv_xphoto -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_photo -lopencv_imgproc -lopencv_core"ืื ืืืคึผืืึทืืขื ืืืืฉืึทื ืงืึธื ืคึฟืึทืจ ืคึผืึทืจืกืื ื JSON ืืื ืืืฉืขื ืขืจืืืืื ื ืึท ืคึผืขืงื ืคึฟืึทืจ ืืจืึทื ืกืืืกืืข ืฆื ืื ืืึธืืขื ืืื ืืขืืขืื ืืื ืืขืจ ืื ืกืคึผืืืืขืจ. ืขืจืฉืืขืจ, ืืืืื ืึท ืืืืข ืคึผืจืืืขืงื ืืืขืืืืืึทืืขืจ ืฆื ืืืื ืคึฟืึทืจ ืืขืืขืจ ืืขืงืขืก (ื ืืื ืคึฟืึทืจ ndjson):
Sys.setenv("PKG_CXXFLAGS" = paste0("-I", normalizePath(file.path("src"))))ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื JSON ืฆื ืืขื ืกืึธืจ ืงืึทื ืืืขืจืืฉืึทื ืืื C ++
// [[Rcpp::plugins(cpp14)]]
// [[Rcpp::plugins(opencv)]]
// [[Rcpp::depends(xtensor)]]
// [[Rcpp::depends(RcppThread)]]
#include <xtensor/xjson.hpp>
#include <xtensor/xadapt.hpp>
#include <xtensor/xview.hpp>
#include <xtensor-r/rtensor.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <Rcpp.h>
#include <RcppThread.h>
// ะกะธะฝะพะฝะธะผั ะดะปั ัะธะฟะพะฒ
using RcppThread::parallelFor;
using json = nlohmann::json;
using points = xt::xtensor<double,2>; // ะะทะฒะปะตััะฝะฝัะต ะธะท JSON ะบะพะพัะดะธะฝะฐัั ัะพัะตะบ
using strokes = std::vector<points>; // ะะทะฒะปะตััะฝะฝัะต ะธะท JSON ะบะพะพัะดะธะฝะฐัั ัะพัะตะบ
using xtensor3d = xt::xtensor<double, 3>; // ะขะตะฝะทะพั ะดะปั ั
ัะฐะฝะตะฝะธั ะผะฐััะธัั ะธะทะพะพะฑัะฐะถะตะฝะธั
using xtensor4d = xt::xtensor<double, 4>; // ะขะตะฝะทะพั ะดะปั ั
ัะฐะฝะตะฝะธั ะผะฝะพะถะตััะฒะฐ ะธะทะพะฑัะฐะถะตะฝะธะน
using rtensor3d = xt::rtensor<double, 3>; // ะะฑัััะบะฐ ะดะปั ัะบัะฟะพััะฐ ะฒ R
using rtensor4d = xt::rtensor<double, 4>; // ะะฑัััะบะฐ ะดะปั ัะบัะฟะพััะฐ ะฒ R
// ะกัะฐัะธัะตัะบะธะต ะบะพะฝััะฐะฝัั
// ะ ะฐะทะผะตั ะธะทะพะฑัะฐะถะตะฝะธั ะฒ ะฟะธะบัะตะปัั
const static int SIZE = 256;
// ะขะธะฟ ะปะธะฝะธะธ
// ะกะผ. https://en.wikipedia.org/wiki/Pixel_connectivity#2-dimensional
const static int LINE_TYPE = cv::LINE_4;
// ะขะพะปัะธะฝะฐ ะปะธะฝะธะธ ะฒ ะฟะธะบัะตะปัั
const static int LINE_WIDTH = 3;
// ะะปะณะพัะธัะผ ัะตัะฐะนะทะฐ
// https://docs.opencv.org/3.1.0/da/d54/group__imgproc__transform.html#ga5bb5a1fea74ea38e1a5445ca803ff121
const static int RESIZE_TYPE = cv::INTER_LINEAR;
// ะจะฐะฑะปะพะฝ ะดะปั ะบะพะฝะฒะตััะธัะพะฒะฐะฝะธั OpenCV-ะผะฐััะธัั ะฒ ัะตะฝะทะพั
template <typename T, int NCH, typename XT=xt::xtensor<T,3,xt::layout_type::column_major>>
XT to_xt(const cv::Mat_<cv::Vec<T, NCH>>& src) {
// ะ ะฐะทะผะตัะฝะพััั ัะตะปะตะฒะพะณะพ ัะตะฝะทะพัะฐ
std::vector<int> shape = {src.rows, src.cols, NCH};
// ะะฑัะตะต ะบะพะปะธัะตััะฒะพ ัะปะตะผะตะฝัะพะฒ ะฒ ะผะฐััะธะฒะต
size_t size = src.total() * NCH;
// ะัะตะพะฑัะฐะทะพะฒะฐะฝะธะต cv::Mat ะฒ xt::xtensor
XT res = xt::adapt((T*) src.data, size, xt::no_ownership(), shape);
return res;
}
// ะัะตะพะฑัะฐะทะพะฒะฐะฝะธะต JSON ะฒ ัะฟะธัะพะบ ะบะพะพัะดะธะฝะฐั ัะพัะตะบ
strokes parse_json(const std::string& x) {
auto j = json::parse(x);
// ะ ะตะทัะปััะฐั ะฟะฐััะธะฝะณะฐ ะดะพะปะถะตะฝ ะฑััั ะผะฐััะธะฒะพะผ
if (!j.is_array()) {
throw std::runtime_error("'x' must be JSON array.");
}
strokes res;
res.reserve(j.size());
for (const auto& a: j) {
// ะะฐะถะดัะน ัะปะตะผะตะฝั ะผะฐััะธะฒะฐ ะดะพะปะถะตะฝ ะฑััั 2-ะผะตัะฝัะผ ะผะฐััะธะฒะพะผ
if (!a.is_array() || a.size() != 2) {
throw std::runtime_error("'x' must include only 2d arrays.");
}
// ะะทะฒะปะตัะตะฝะธะต ะฒะตะบัะพัะฐ ัะพัะตะบ
auto p = a.get<points>();
res.push_back(p);
}
return res;
}
// ะััะธัะพะฒะบะฐ ะปะธะฝะธะน
// ะฆะฒะตัะฐ HSV
cv::Mat ocv_draw_lines(const strokes& x, bool color = true) {
// ะัั
ะพะดะฝัะน ัะธะฟ ะผะฐััะธัั
auto stype = color ? CV_8UC3 : CV_8UC1;
// ะัะพะณะพะฒัะน ัะธะฟ ะผะฐััะธัั
auto dtype = color ? CV_32FC3 : CV_32FC1;
auto bg = color ? cv::Scalar(0, 0, 255) : cv::Scalar(255);
auto col = color ? cv::Scalar(0, 255, 220) : cv::Scalar(0);
cv::Mat img = cv::Mat(SIZE, SIZE, stype, bg);
// ะะพะปะธัะตััะฒะพ ะปะธะฝะธะน
size_t n = x.size();
for (const auto& s: x) {
// ะะพะปะธัะตััะฒะพ ัะพัะตะบ ะฒ ะปะธะฝะธะธ
size_t n_points = s.shape()[1];
for (size_t i = 0; i < n_points - 1; ++i) {
// ะขะพัะบะฐ ะฝะฐัะฐะปะฐ ัััะธั
ะฐ
cv::Point from(s(0, i), s(1, i));
// ะขะพัะบะฐ ะพะบะพะฝัะฐะฝะธั ัััะธั
ะฐ
cv::Point to(s(0, i + 1), s(1, i + 1));
// ะััะธัะพะฒะบะฐ ะปะธะฝะธะธ
cv::line(img, from, to, col, LINE_WIDTH, LINE_TYPE);
}
if (color) {
// ะะตะฝัะตะผ ัะฒะตั ะปะธะฝะธะธ
col[0] += 180 / n;
}
}
if (color) {
// ะะตะฝัะตะผ ัะฒะตัะพะฒะพะต ะฟัะตะดััะฐะฒะปะตะฝะธะต ะฝะฐ RGB
cv::cvtColor(img, img, cv::COLOR_HSV2RGB);
}
// ะะตะฝัะตะผ ัะพัะผะฐั ะฟัะตะดััะฐะฒะปะตะฝะธั ะฝะฐ float32 ั ะดะธะฐะฟะฐะทะพะฝะพะผ [0, 1]
img.convertTo(img, dtype, 1 / 255.0);
return img;
}
// ะะฑัะฐะฑะพัะบะฐ JSON ะธ ะฟะพะปััะตะฝะธะต ัะตะฝะทะพัะฐ ั ะดะฐะฝะฝัะผะธ ะธะทะพะฑัะฐะถะตะฝะธั
xtensor3d process(const std::string& x, double scale = 1.0, bool color = true) {
auto p = parse_json(x);
auto img = ocv_draw_lines(p, color);
if (scale != 1) {
cv::Mat out;
cv::resize(img, out, cv::Size(), scale, scale, RESIZE_TYPE);
cv::swap(img, out);
out.release();
}
xtensor3d arr = color ? to_xt<double,3>(img) : to_xt<double,1>(img);
return arr;
}
// [[Rcpp::export]]
rtensor3d cpp_process_json_str(const std::string& x,
double scale = 1.0,
bool color = true) {
xtensor3d res = process(x, scale, color);
return res;
}
// [[Rcpp::export]]
rtensor4d cpp_process_json_vector(const std::vector<std::string>& x,
double scale = 1.0,
bool color = false) {
size_t n = x.size();
size_t dim = floor(SIZE * scale);
size_t channels = color ? 3 : 1;
xtensor4d res({n, dim, dim, channels});
parallelFor(0, n, [&x, &res, scale, color](int i) {
xtensor3d tmp = process(x[i], scale, color);
auto view = xt::view(res, i, xt::all(), xt::all(), xt::all());
view = tmp;
});
return res;
}ืืขื ืงืึธื ืืึธื ืืืื ืืขืฉืืขืื ืืื ืืขืจ ืืขืงืข src/cv_xt.cpp ืืื ืฆืื ืืืคื ืขืืขื ืืื ืื ืืึทืคึฟืขื Rcpp::sourceCpp(file = "src/cv_xt.cpp", env = .GlobalEnv); ืืืื ืคืืจืืื ืื ืคึฟืึทืจ ืึทืจืืขื nlohmann/json.hpp ืคืื . ืืขืจ ืงืึธื ืืื ืฆืขืืืืื ืืื ืขืืืขืืข ืคืึทื ืืงืฉืึทื ื:
-
to_xt- ืึท ืืขืืคึผืืึทืืข ืคืื ืงืฆืืข ืคึฟืึทืจ ืืจืึทื ืกืคืึธืจืืื ื ืึท ืืืื ืืึทืืจืืฅ (cv::Mat) ืฆื ื ืืขื ืกืืจxt::xtensor; -
parse_json- ืื ืคึฟืื ืงืฆืืข ืคึผืึทืจืกืขืก ืึท JSON ืฉืืจืืงื, ืขืงืกืืจืึทืงืฅ ืื ืงืึธืืึธืจืืึทื ืึทืฅ ืคืื ืคืื ืงืื, ืคึผืึทืงืื ื ืืื ืืื ืึท ืืืขืงืืึธืจ; -
ocv_draw_lines- ืคืื ืื ืจืืืึทืืืื ื ืืืขืงืืึธืจ ืคืื ืืืืืื, ืืจืึธื ืืึทืืื-ืืื ื ืฉืืจืืช; -
process- ืงืึทืืืืื ื ืื ืืืืื ืคืึทื ืืงืฉืึทื ื ืืื ืืืื ืืืกืืฃ ืื ืคืืืืงืืื ืฆื ืืืึธื ืื ืจืืืึทืืืื ื ืืืื; -
cpp_process_json_str- ืจืึทืคึผืขืจ ืืืืขืจ ืื ืคึฟืื ืงืฆืืขprocess, ืืืึธืก ืขืงืกืคึผืึธืจืฅ ืื ืจืขืืืืืึทื ืฆื ืึท ืจ-ืึธืืืืฉืขืงื (ืืืืืืืืืขื ืกืืึธื ืึทื ืืขื ืืข); -
cpp_process_json_vector- ืจืึทืคึผืขืจ ืืืืขืจ ืื ืคึฟืื ืงืฆืืขcpp_process_json_str, ืืืึธืก ืึทืืึทืื ืืืจ ืฆื ืคึผืจืึธืฆืขืก ืึท ืฉืืจืืงื ืืืขืงืืึธืจ ืืื ืืืืื-ืืจืขืืื ืืึธืืข.
ืฆื ืฆืืขื ืืึทืืื-ืืื ื ืฉืืจืืช, ืื HSV ืงืึธืืืจ ืืึธืืขื ืืื ืืขื ืืฆื, ื ืืืืขืืื ืืขื ืืืจื ืงืึทื ืืืขืจืืฉืึทื ืฆื RGB. ืืื ืก ืคึผืจืืืืจื ืื ืจืขืืืืืึทื:
arr <- cpp_process_json_str(tmp_data[4, drawing])
dim(arr)
# [1] 256 256 3
plot(magick::image_read(arr))
ืคืึทืจืืืืึทื ืคืื ืื ืืืืงืืึทื ืคืื ืืืคึผืืึทืืึทื ืฅ ืืื ืจ ืืื C ++
res_bench <- bench::mark(
r_process_json_str(tmp_data[4, drawing], scale = 0.5),
cpp_process_json_str(tmp_data[4, drawing], scale = 0.5),
check = FALSE,
min_iterations = 100
)
# ะะฐัะฐะผะตััั ะฑะตะฝัะผะฐัะบะฐ
cols <- c("expression", "min", "median", "max", "itr/sec", "total_time", "n_itr")
res_bench[, cols]
# expression min median max `itr/sec` total_time n_itr
# <chr> <bch:tm> <bch:tm> <bch:tm> <dbl> <bch:tm> <int>
# 1 r_process_json_str 3.49ms 3.55ms 4.47ms 273. 490ms 134
# 2 cpp_process_json_str 1.94ms 2.02ms 5.32ms 489. 497ms 243
library(ggplot2)
# ะัะพะฒะตะดะตะฝะธะต ะทะฐะผะตัะฐ
res_bench <- bench::press(
batch_size = 2^(4:10),
{
.data <- tmp_data[sample(seq_len(.N), batch_size), drawing]
bench::mark(
r_process_json_vector(.data, scale = 0.5),
cpp_process_json_vector(.data, scale = 0.5),
min_iterations = 50,
check = FALSE
)
}
)
res_bench[, cols]
# expression batch_size min median max `itr/sec` total_time n_itr
# <chr> <dbl> <bch:tm> <bch:tm> <bch:tm> <dbl> <bch:tm> <int>
# 1 r 16 50.61ms 53.34ms 54.82ms 19.1 471.13ms 9
# 2 cpp 16 4.46ms 5.39ms 7.78ms 192. 474.09ms 91
# 3 r 32 105.7ms 109.74ms 212.26ms 7.69 6.5s 50
# 4 cpp 32 7.76ms 10.97ms 15.23ms 95.6 522.78ms 50
# 5 r 64 211.41ms 226.18ms 332.65ms 3.85 12.99s 50
# 6 cpp 64 25.09ms 27.34ms 32.04ms 36.0 1.39s 50
# 7 r 128 534.5ms 627.92ms 659.08ms 1.61 31.03s 50
# 8 cpp 128 56.37ms 58.46ms 66.03ms 16.9 2.95s 50
# 9 r 256 1.15s 1.18s 1.29s 0.851 58.78s 50
# 10 cpp 256 114.97ms 117.39ms 130.09ms 8.45 5.92s 50
# 11 r 512 2.09s 2.15s 2.32s 0.463 1.8m 50
# 12 cpp 512 230.81ms 235.6ms 261.99ms 4.18 11.97s 50
# 13 r 1024 4s 4.22s 4.4s 0.238 3.5m 50
# 14 cpp 1024 410.48ms 431.43ms 462.44ms 2.33 21.45s 50
ggplot(res_bench, aes(x = factor(batch_size), y = median,
group = expression, color = expression)) +
geom_point() +
geom_line() +
ylab("median time, s") +
theme_minimal() +
scale_color_discrete(name = "", labels = c("cpp", "r")) +
theme(legend.position = "bottom")
ืืื ืืืจ ืงืขื ืขื ืืขื, ืื ืืืืงืืึทื ืคืึทืจืืจืขืกืขืจื ืืื ืืขืืืขื ืืืืขืจ ืืึทืืืึทืืืง, ืืื ืขืก ืืื ื ืื ืืขืืืขื ืฆื ืืึทืคึผื ืืื ืืื C ++ ืงืึธื ืืืจื ืคึผืึทืจืึทืืขืืืืืื ื ืจ ืงืึธื.
3. ืืืขืจืืืืขืจื ืคึฟืึทืจ ืึทื ืืึธืืืื ื ืืึทืืฉืึทื ืคืื ืื ืืึทืืึทืืืืก
R ืืื ืึท ืืขืืื ื-ืืืืขืจืืื ืฉืขื ืคึฟืึทืจ ืคึผืจืึทืกืขืกืื ื ืืึทืื ืืืึธืก ืคึผืึทืกืืง ืืื ืืึทืจืึทื, ืืฉืขืช Python ืืื ืืขืจ ืงืขืจืึทืงืืขืจืืืื ืืืจื ืืืขืจืึทืืืื ืืึทืื ืคึผืจืึทืกืขืกืื ื, ืืืึธืก ืึทืืึทืื ืืืจ ืฆื ืืืืื ืืื ืืขืืืืื ืืืขื ืื ืกืืจืืืขื ื ืืืืก-ืคืื-ืืึทืจืฅ ืืฉืืื ืืช (ืืฉืืื ืืช ืืื ืคืื ืืจืืืกื ืืืง ืืึผืจืื). ื ืงืืึทืกืืฉ ืืื ืืึทืืืึทืืืง ืืืืฉืคึผืื ืคึฟืึทืจ ืืื ืื ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืื ืืืกืงืจืืืื ืคึผืจืึธืืืขื ืืื ืืืฃ ื ืขืืจืึทื ื ืขืืืืึธืจืงืก ืืจืืื ื ืืืจื ืื ืืจืึทืืืขื ื ืึทืจืึธืคึผืืึทื ื ืืืคึฟื ืืื ืึทืคึผืจืึทืงืกืึทืืืืฉืึทื ืคืื ืื ืืจืึทืืืขื ื ืืื ืืขืืขืจ ืฉืจืื ื ืืฆื ืึท ืงืืืื ืืืง ืคืื ืึทืืืขืจืืืืืฉืึทื ื ืึธืืขืจ ืืื ื ืคึผืขืงื.
ืืืฃ ืืขืจื ืขื ืคืจืึทืืขืืืึธืจืงืก ืืขืฉืจืืื ืืื ืคึผืืืืึธื ืืึธืื ืกืคึผืขืฆืืขื ืงืืืกื ืืืึธืก ืื ืกืืจืืืขื ื ืืืขืจืืืืขืจื ืืืืืจื ืืืืฃ ืืึทืื: ืืืฉื, ืืืืืขืจ ืืื ืคืึธืืืขืจืก, ืืืื ืขืจื ืคึฟืึธืจืืึทืืืจืื ืืขื, ืืื"ื ื. ืืืจ ืงืขื ืขื ื ืืฆื ืคืึทืจืืืง ืึธืคึผืฆืืขืก ืึธืืขืจ ืฉืจืืึทืื ืืืื ืืืืืขื ืข ืคึฟืึทืจ ืกืคึผืขืฆืืคืืฉ ืืึทืกืงืก. ืืื R, ืืืจ ืงืขื ืขื ื ืืฆื ืึทืืข ืื ืคึฟืขืึดืงืืืื ืคืื ืื ืคึผืืืืึธื ืืืืืืึธืืขืง ืงืขืจืึทืก ืืื ืืืึทื ืคืึทืจืฉืืื ืืึทืงืขื ืื ื ืืฆื ืื ืคึผืขืงื ืืื ืื ืืขืืืข ื ืึธืืขื, ืืืึธืก ืืื ืงืขืจ ืึทืจืืขื ืืืืฃ ืฉืคึผืืฅ ืคืื ืื ืคึผืขืงื ืจืขืืืงืืืึทืืข. ืืขืจ ืืขืฆืืขืจ ืคึฟืึทืจืืื ื ืึท ืืึทืืื ืืขืจื ืืึทื ืื ืึทืจืืืงื; ืขืก ื ืื ืืืืื ืึทืืึทืื ืืืจ ืฆื ืืืืคื Python ืงืึธื ืคึฟืื R, ืึธืืขืจ ืืืื ืึทืืึทืื ืืืจ ืฆื ืึทืจืืืขืจืคืืจื ืึทืืืืฉืขืงืฅ ืฆืืืืฉื R ืืื Python ืกืขืฉืึทื ื, ืืืืืึธืืึทืืืฉ ืืืจืืคืืจื ืึทืืข ืื ื ืืืืืง ืืืคึผ ืงืึทื ืืืขืจืืฉืึทื ื.
ืืืจ ืืึทืงืืืขื ืืึทืคืจืืึทืขื ืคืื ืื ื ืืื ืฆื ืงืจืึธื ืึทืืข ืื ืืึทืื ืืื ืืึทืจืึทื ืืืจื ื ืืฆื MonetDBLite, ืึทืืข ืื "ื ืขืืจืึทื ื ืขืฅ" ืึทืจืืขื ืืืขื ืืืื ืืืจืืืขืงืึธืื ืืืจื ืืขืจ ืึธืจืืืื ืขื ืงืึธื ืืื ืคึผืืืืึธื, ืืืจ ื ืึธืจ ืืึธืื ืฆื ืฉืจืืึทืื ืึทื ืืืขืจืึทืืึธืจ ืืืืขืจ ืื ืืึทืื, ืืืืึทื ืขืก ืืื ืืึธืจื ืืฉื ืืจืืื. ืคึฟืึทืจ ืึทืืึท ืึท ืกืืืืึทืฆืืข ืืื ืจ ืึธืืขืจ ืคึผืืืืึธื. ืขืก ืืขื ืขื ืืืืกืืงืื ืืืืื ืฆืืืื ืจืขืงืืืืจืขืืขื ืฅ ืคึฟืึทืจ ืขืก: ืขืก ืืืื ืฆืืจืืงืงืืืขื ืืึทืืฉืึทื ืืื ืึท ืกืึธืฃ ืฉืืืืฃ ืืื ืจืึทืืขืืืขื ืืืึทื ืฉืืึทื ืฆืืืืฉื ืืืขืจืืืฉืึทื ื (ืื ืืขืฆืืข ืืื ืจ ืืื ืืืคึผืืึทืืขื ืึทื ืืื ืื ืกืืืคึผืืึทืกื ืืืขื ื ืืฆื ืงืืึธืืืฉืขืจื). ืืื ืึทืืขืจ, ืขืก ืืื ืืขืืืขื ืคืืจืืื ืื ืฆื ืืคืืจืืฉ ืืขืจ R ืขืจืืื ืืื ื ืึทืืคึผื ืขืจืืื ืืื ืื ืืืขืจืึทืืึธืจ, ืึธืืขืจ ืื ืงืจืึทื ื ืืืขืจืกืืข ืคืื โโโโืืขื ืคึผืขืงื ืงืขืจืึทืก ืืื ืขืก ืืื.
ืืขืจ ืืืขืจืึทืืึธืจ ืคึฟืึทืจ ืืจืืื ืื ื ืืื ืืืึทืืึทืืืืฉืึทื ืืึทืื ืืื ืืขืืืขื ืืื ืืืื:
ืืืขืจืึทืืึธืจ ืคึฟืึทืจ ืืจืืื ืื ื ืืื ืืืึทืืึทืืืืฉืึทื ืืึทืื
train_generator <- function(db_connection = con,
samples_index,
num_classes = 340,
batch_size = 32,
scale = 1,
color = FALSE,
imagenet_preproc = FALSE) {
# ะัะพะฒะตัะบะฐ ะฐัะณัะผะตะฝัะพะฒ
checkmate::assert_class(con, "DBIConnection")
checkmate::assert_integerish(samples_index)
checkmate::assert_count(num_classes)
checkmate::assert_count(batch_size)
checkmate::assert_number(scale, lower = 0.001, upper = 5)
checkmate::assert_flag(color)
checkmate::assert_flag(imagenet_preproc)
# ะะตัะตะผะตัะธะฒะฐะตะผ, ััะพะฑั ะฑัะฐัั ะธ ัะดะฐะปััั ะธัะฟะพะปัะทะพะฒะฐะฝะฝัะต ะธะฝะดะตะบัั ะฑะฐััะตะน ะฟะพ ะฟะพััะดะบั
dt <- data.table::data.table(id = sample(samples_index))
# ะัะพััะฐะฒะปัะตะผ ะฝะพะผะตัะฐ ะฑะฐััะตะน
dt[, batch := (.I - 1L) %/% batch_size + 1L]
# ะััะฐะฒะปัะตะผ ัะพะปัะบะพ ะฟะพะปะฝัะต ะฑะฐััะธ ะธ ะธะฝะดะตะบัะธััะตะผ
dt <- dt[, if (.N == batch_size) .SD, keyby = batch]
# ะฃััะฐะฝะฐะฒะปะธะฒะฐะตะผ ัััััะธะบ
i <- 1
# ะะพะปะธัะตััะฒะพ ะฑะฐััะตะน
max_i <- dt[, max(batch)]
# ะะพะดะณะพัะพะฒะบะฐ ะฒััะฐะถะตะฝะธั ะดะปั ะฒัะณััะทะบะธ
sql <- sprintf(
"PREPARE SELECT drawing, label_int FROM doodles WHERE id IN (%s)",
paste(rep("?", batch_size), collapse = ",")
)
res <- DBI::dbSendQuery(con, sql)
# ะะฝะฐะปะพะณ keras::to_categorical
to_categorical <- function(x, num) {
n <- length(x)
m <- numeric(n * num)
m[x * n + seq_len(n)] <- 1
dim(m) <- c(n, num)
return(m)
}
# ะะฐะผัะบะฐะฝะธะต
function() {
# ะะฐัะธะฝะฐะตะผ ะฝะพะฒัั ัะฟะพั
ั
if (i > max_i) {
dt[, id := sample(id)]
data.table::setkey(dt, batch)
# ะกะฑัะฐััะฒะฐะตะผ ัััััะธะบ
i <<- 1
max_i <<- dt[, max(batch)]
}
# ID ะดะปั ะฒัะณััะทะบะธ ะดะฐะฝะฝัั
batch_ind <- dt[batch == i, id]
# ะัะณััะทะบะฐ ะดะฐะฝะฝัั
batch <- DBI::dbFetch(DBI::dbBind(res, as.list(batch_ind)), n = -1)
# ะฃะฒะตะปะธัะธะฒะฐะตะผ ัััััะธะบ
i <<- i + 1
# ะะฐััะธะฝะณ JSON ะธ ะฟะพะดะณะพัะพะฒะบะฐ ะผะฐััะธะฒะฐ
batch_x <- cpp_process_json_vector(batch$drawing, scale = scale, color = color)
if (imagenet_preproc) {
# ะจะบะฐะปะธัะพะฒะฐะฝะธะต c ะธะฝัะตัะฒะฐะปะฐ [0, 1] ะฝะฐ ะธะฝัะตัะฒะฐะป [-1, 1]
batch_x <- (batch_x - 0.5) * 2
}
batch_y <- to_categorical(batch$label_int, num_classes)
result <- list(batch_x, batch_y)
return(result)
}
}ืื ืคึฟืื ืงืฆืืข ื ืขืื ืืื ืึทืจืืึทื ืฉืจืืึทื ืึท ืืืึทืืขืืืืืง ืืื ืึท ืคึฟืึทืจืืื ืืื ื ืฆื ืื ืืึทืืึทืืืืก, ืื ื ืืืขืจ ืคืื ืฉืืจืืช ืืขื ืืฆื, ืื ื ืืืขืจ ืคืื ืงืืืกื, ืคึผืขืงื ืืจืืืก, ืืืึธื (scale = 1 ืงืึธืจืึทืกืคึผืึทื ืื ืฆื ืจืขื ืืขืจืื ื ืืืืืขืจ ืคืื 256x256 ืืืืืฆืขืื, scale = 0.5 - 128x128 ืืืืืฆืขืื), ืงืึธืืืจ ืืจืืื (color = FALSE ืกืคึผืขืฆืืคืืฆืืจื ืจืขื ืืขืจืื ื ืืื ืืจืืืกืงืึทืืข ืืืขื ืืขืืืืื ื color = TRUE ืืขืืขืจ ืืึทื ืืื ืฆืืขื ืืื ืึท ื ืืึทืข ืงืึธืืืจ) ืืื ืึท ืคึผืจืขืคึผืจืึธืกืขืกืกืื ื ืืจืืื ืคึฟืึทืจ ื ืขืืืืึธืจืงืก ืคืึทืจ-ืืจืืื ื ืืืืฃ ืืืึทืืืฉื ืขื. ืื ืืขืฆืืข ืืื ืืืจืฃ ืฆื ืืืึธื ืคึผืืงืกืขื ืืืึทืืืขืก ืคืื ืื ืืขืืึทืืขื [0, 1] ืฆื ืื ืืขืืึทืืขื [-1, 1], ืืืึธืก ืืื ืืขื ืืฆื ืืืขื ืืจืืื ืื ื ืื ืกืึทืคึผืืืื. ืงืขืจืึทืก ืืึธืืขืืก.
ืื ืคืื ืืจืืืกื ืืืง ืคึฟืื ืงืฆืืข ืึผืืื ืึทืจืืืืขื ื ืืืคึผ ืงืึธื ืืจืึธืืืจืื ื, ืึท ืืืฉ data.table ืืื ืจืึทื ืืึทืืื ืืขืืืฉื ืฉืืจื ื ืืืขืจื ืคืื samples_index ืืื ืคึผืขืงื ื ืืืขืจื, ืืึธืืืึทื ืง ืืื ืืึทืงืกืืืื ื ืืืขืจ ืคืื ืืึทืืฉืึทื, ืืื ืืขืืื ื ืืื ืึท ืกืงื ืืืืกืืจืืง ืคึฟืึทืจ ืึทื ืืึธืืืื ื ืืึทืื ืคืื ืื ืืึทืืึทืืืืก. ืืื ืืขืจืฆื, ืืืจ ืืืคืืื ื ืึท ืฉื ืขื ืึทื ืึทืืึธื ืคืื ืื ืคืื ืงืฆืืข ืื keras::to_categorical(). ืืืจ ืืขืืืืื ื ืึผืืขื ืึทืืข ืื ืืึทืื ืคึฟืึทืจ ืืจืืื ืื ื, ืืขืืืื ืึท ืืึทืื ืคึผืจืึธืฆืขื ื ืคึฟืึทืจ ืืืึทืืึทืืืืฉืึทื, ืึทืืื ืื ืขืคึผืึธืก ืืจืืืก ืืื ืืืืืืขื ืืืจื ืื ืคึผืึทืจืึทืืขืืขืจ steps_per_epoch ืืืขื ืืขืจืืคื keras::fit_generator(), ืืื ืื ืฆืืฉืืึทื ื if (i > max_i) ื ืึธืจ ืืขืืจืืขื ืคึฟืึทืจ ืื ืืืึทืืึทืืืืฉืึทื ืืืขืจืึทืืึธืจ.
ืืื ืื ืื ืขืจืืขื ืคึฟืื ืงืฆืืข, ืจืืืขืจื ืื ืืขืงืกืื ืืขื ืขื ืจืืืจืืืื ืคึฟืึทืจ ืื ืืืืึทืืขืจ ืคึผืขืงื, ืจืขืงืึธืจืืก ืืขื ืขื ืึทื ืืึธืืืึทื ืคืื ืื ืืึทืืึทืืืืก ืืื ืื ืคึผืขืงื ืืึธืืืึทื ืง ืื ืงืจืืกืื ื, JSON ืคึผืึทืจืกืื ื (ืคืื ืงืฆืืข). cpp_process_json_vector(), ืืขืฉืจืืื ืืื C ++) ืืื ืงืจืืืืืื ื ืขืจืืื ืงืึธืจืึทืกืคึผืึทื ืืื ื ืฆื ืืืืืขืจ. ืืขืจื ืึธื ืืืื ืขืจ-ืืืืก ืืืขืงืืึธืจืก ืืื ืงืืึทืก ืืึทืืขืืก ืืขื ืขื ืืืฉืืคื, ืขืจืืื ืืื ืคึผืืงืกืขื ืืืึทืืืขืก ืืื ืืึทืืขืืก ืืขื ืขื ืงืึทืืืืื ื ืืื ืึท ืจืฉืืื, ืืืึธืก ืืื ืื ืฆืืจืืงืงืืืขื ืืืขืจื. ืฆื ืคืึทืจืืืืขืจื ืื ืึทืจืืขื, ืืืจ ืืขืืืืื ื ืื ืฉืึทืคืื ื ืคืื ืื ืืขืงืกืื ืืื ืืืฉื data.table ืืื ืืึธืืืคืืงืึทืืืึธื ืืืจื ืื ืืื ืง - ืึธื ืื ืคึผืขืงื "ืืฉืืคึผืก" ืืึทืืึท.ืืึทืืืข ืขืก ืืื ืืึทื ืฅ ืฉืืืขืจ ืฆื ืืืึทืืืฉืึทื ืืจืืขืื ืืคืขืงืืืืืื ืืื ืงืืื ืืึทืืืืืืง ืกืืืข ืคืื โโืืึทืื ืืื ืจ.
ืื ืจืขืืืืืึทืื ืคืื ืืืืงืืึทื ืืขืืฉืขืจืืึทื ืฅ ืืืืฃ ืึท Core i5 ืืึทืคึผืืึทืคึผ ืืขื ืขื ืืื ืืืื:
ืืืขืจืึทืืึธืจ ืืขื ืืฉืืึทืจืง
library(Rcpp)
library(keras)
library(ggplot2)
source("utils/rcpp.R")
source("utils/keras_iterator.R")
con <- DBI::dbConnect(drv = MonetDBLite::MonetDBLite(), Sys.getenv("DBDIR"))
ind <- seq_len(DBI::dbGetQuery(con, "SELECT count(*) FROM doodles")[[1L]])
num_classes <- DBI::dbGetQuery(con, "SELECT max(label_int) + 1 FROM doodles")[[1L]]
# ะะฝะดะตะบัั ะดะปั ะพะฑััะฐััะตะน ะฒัะฑะพัะบะธ
train_ind <- sample(ind, floor(length(ind) * 0.995))
# ะะฝะดะตะบัั ะดะปั ะฟัะพะฒะตัะพัะฝะพะน ะฒัะฑะพัะบะธ
val_ind <- ind[-train_ind]
rm(ind)
# ะะพัััะธัะธะตะฝั ะผะฐัััะฐะฑะฐ
scale <- 0.5
# ะัะพะฒะตะดะตะฝะธะต ะทะฐะผะตัะฐ
res_bench <- bench::press(
batch_size = 2^(4:10),
{
it1 <- train_generator(
db_connection = con,
samples_index = train_ind,
num_classes = num_classes,
batch_size = batch_size,
scale = scale
)
bench::mark(
it1(),
min_iterations = 50L
)
}
)
# ะะฐัะฐะผะตััั ะฑะตะฝัะผะฐัะบะฐ
cols <- c("batch_size", "min", "median", "max", "itr/sec", "total_time", "n_itr")
res_bench[, cols]
# batch_size min median max `itr/sec` total_time n_itr
# <dbl> <bch:tm> <bch:tm> <bch:tm> <dbl> <bch:tm> <int>
# 1 16 25ms 64.36ms 92.2ms 15.9 3.09s 49
# 2 32 48.4ms 118.13ms 197.24ms 8.17 5.88s 48
# 3 64 69.3ms 117.93ms 181.14ms 8.57 5.83s 50
# 4 128 157.2ms 240.74ms 503.87ms 3.85 12.71s 49
# 5 256 359.3ms 613.52ms 988.73ms 1.54 30.5s 47
# 6 512 884.7ms 1.53s 2.07s 0.674 1.11m 45
# 7 1024 2.7s 3.83s 5.47s 0.261 2.81m 44
ggplot(res_bench, aes(x = factor(batch_size), y = median, group = 1)) +
geom_point() +
geom_line() +
ylab("median time, s") +
theme_minimal()
DBI::dbDisconnect(con, shutdown = TRUE)
ืืืื ืืืจ ืืึธืื ืึท ืืขื ืื ืกืืืข ืคืื โโืืึทืจืึทื, ืืืจ ืงืขื ืขื ืขืืขืก ืคืึทืจืืืืขืจื ืื ืึธืคึผืขืจืึทืฆืืข ืคืื โโืื ืืึทืืึทืืืืก ืืืจื ืืจืึทื ืกืคืขืจืื ื ืขืก ืฆื ืืขืจ ืืขืืืืงืขืจ ืืึทืจืึทื (32 ืืืืืืืื ืืื ืืขื ืื ืคึฟืึทืจ ืืื ืืืขืจ ืึทืจืืขื). ืืื ืืื ืืงืก, ืื ืฆืขืืืืืื ื ืืื ืืึธืื ืืขื ืืืจื ืคืขืืืงืืึทื /dev/shm, ืึทืงืืึทืคึผืืืื ื ืึทืจืืืฃ ืฆื ืืึทืื ืคืื ืื ืืึทืจืึทื ืงืึทืคึผืึทืฆืืืขื. ืืืจ ืงืขื ืขื ืืืืืคึผืื ืงื ืืขืจ ืืืจื ืขืืืืื ื /etc/fstabืฆื ืืึทืงืืืขื ืึท ืจืขืงืึธืจื ืืื tmpfs /dev/shm tmpfs defaults,size=25g 0 0. ืืืื ืืืืขืจ ืฆื ืจืขืืึธืึธื ืืื ืงืึธื ืืจืึธืืืจื ืื ืจืขืืืืืึทื ืืืจื ืืืืคื ืื ืืึทืคึฟืขื df -h.
ืืขืจ ืืืขืจืึทืืึธืจ ืคึฟืึทืจ ืคึผืจืึธืืข ืืึทืื ืงืืงื ืคืื ืกืืืคึผืืขืจ, ืืื ื ืื ืคึผืจืึธืืข ืืึทืืึทืกืขื ืคืืฅ ืืขืืึทืืจืข ืืื ืืึทืจืึทื:
ืืืขืจืึทืืึธืจ ืคึฟืึทืจ ืคึผืจืืืืจื ืืึทืื
test_generator <- function(dt,
batch_size = 32,
scale = 1,
color = FALSE,
imagenet_preproc = FALSE) {
# ะัะพะฒะตัะบะฐ ะฐัะณัะผะตะฝัะพะฒ
checkmate::assert_data_table(dt)
checkmate::assert_count(batch_size)
checkmate::assert_number(scale, lower = 0.001, upper = 5)
checkmate::assert_flag(color)
checkmate::assert_flag(imagenet_preproc)
# ะัะพััะฐะฒะปัะตะผ ะฝะพะผะตัะฐ ะฑะฐััะตะน
dt[, batch := (.I - 1L) %/% batch_size + 1L]
data.table::setkey(dt, batch)
i <- 1
max_i <- dt[, max(batch)]
# ะะฐะผัะบะฐะฝะธะต
function() {
batch_x <- cpp_process_json_vector(dt[batch == i, drawing],
scale = scale, color = color)
if (imagenet_preproc) {
# ะจะบะฐะปะธัะพะฒะฐะฝะธะต c ะธะฝัะตัะฒะฐะปะฐ [0, 1] ะฝะฐ ะธะฝัะตัะฒะฐะป [-1, 1]
batch_x <- (batch_x - 0.5) * 2
}
result <- list(batch_x)
i <<- i + 1
return(result)
}
}4. ืกืขืืขืงืฆืืข ืคืื โโืืึธืืขื ืึทืจืงืึทืืขืงืืฉืขืจ
ืืขืจ ืขืจืฉืืขืจ ืึทืจืงืึทืืขืงืืฉืขืจ ืืขื ืืฆื ืืื ืืขืืืขื , ืื ืคึฟืขืึดืงืืืื ืคืื ืืืึธืก ืืขื ืขื ืืืกืงืึทืกื ืืื ืึธื ืืึธื. ืขืก ืืื ืึทืจืืึทื ืืขืจืขืื ื ืืื ื ืึธืจืืึทื ืงืขืจืึทืก ืืื, ืืขืจืคึฟืึทืจ, ืืื ืื ืืืฆื ืืื ืืขื ืคึผืขืงื ืืื ืื ืืขืืืข ื ืึธืืขื ืคึฟืึทืจ ืจ. ืึธืืขืจ ืืืขื ืืืจ ืคึผืจืึผืืื ืฆื ื ืืฆื ืขืก ืืื ืืืื-ืงืึทื ืึทื ืืืืืขืจ, ืึท ืืึธืื ืข ืืึทื ืืื ืืจืืืก: ืืขืจ ืึทืจืืึทื ืฉืจืืึทื ืืขื ืกืขืจ ืืืื ืฉืืขื ืืืง ืืึธืื ืื ืืืกืืขืกืืื ื (batch, height, width, 3), ืืืึธืก ืืื, ืื ื ืืืขืจ ืคืื ืืฉืึทื ืึทืื ืงืขื ืขื ื ืื ืืืื ืืขืืืื. ืขืก ืืื ื ืื ืึทืืึท ืืึทืืจืขื ืขืฆืื ื ืืื Python, ืึทืืื ืืืจ ืจืึทืฉื ืืื ืืขืฉืจืืื ืืื ืืืขืจ ืืืืืขื ืข ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืืขื ืึทืจืงืึทืืขืงืืฉืขืจ, ื ืึธื ืืขืจ ืึธืจืืืื ืขื ืึทืจืืืงื (ืึธื ืื ืืจืึธืคึผืึทืื ืืื ืื ืงืขืจืึทืก ืืืขืจืกืืข):
Mobilenet v1 ืึทืจืงืึทืืขืงืืฉืขืจ
library(keras)
top_3_categorical_accuracy <- custom_metric(
name = "top_3_categorical_accuracy",
metric_fn = function(y_true, y_pred) {
metric_top_k_categorical_accuracy(y_true, y_pred, k = 3)
}
)
layer_sep_conv_bn <- function(object,
filters,
alpha = 1,
depth_multiplier = 1,
strides = c(2, 2)) {
# NB! depth_multiplier != resolution multiplier
# https://github.com/keras-team/keras/issues/10349
layer_depthwise_conv_2d(
object = object,
kernel_size = c(3, 3),
strides = strides,
padding = "same",
depth_multiplier = depth_multiplier
) %>%
layer_batch_normalization() %>%
layer_activation_relu() %>%
layer_conv_2d(
filters = filters * alpha,
kernel_size = c(1, 1),
strides = c(1, 1)
) %>%
layer_batch_normalization() %>%
layer_activation_relu()
}
get_mobilenet_v1 <- function(input_shape = c(224, 224, 1),
num_classes = 340,
alpha = 1,
depth_multiplier = 1,
optimizer = optimizer_adam(lr = 0.002),
loss = "categorical_crossentropy",
metrics = c("categorical_crossentropy",
top_3_categorical_accuracy)) {
inputs <- layer_input(shape = input_shape)
outputs <- inputs %>%
layer_conv_2d(filters = 32, kernel_size = c(3, 3), strides = c(2, 2), padding = "same") %>%
layer_batch_normalization() %>%
layer_activation_relu() %>%
layer_sep_conv_bn(filters = 64, strides = c(1, 1)) %>%
layer_sep_conv_bn(filters = 128, strides = c(2, 2)) %>%
layer_sep_conv_bn(filters = 128, strides = c(1, 1)) %>%
layer_sep_conv_bn(filters = 256, strides = c(2, 2)) %>%
layer_sep_conv_bn(filters = 256, strides = c(1, 1)) %>%
layer_sep_conv_bn(filters = 512, strides = c(2, 2)) %>%
layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
layer_sep_conv_bn(filters = 1024, strides = c(2, 2)) %>%
layer_sep_conv_bn(filters = 1024, strides = c(1, 1)) %>%
layer_global_average_pooling_2d() %>%
layer_dense(units = num_classes) %>%
layer_activation_softmax()
model <- keras_model(
inputs = inputs,
outputs = outputs
)
model %>% compile(
optimizer = optimizer,
loss = loss,
metrics = metrics
)
return(model)
}ืื ืืืกืึทืืืืึทื ืืืืืฉืื ืคืื ืืขื ืฆืืืึทื ื ืืขื ืขื ืงืืึธืจ ืืื ืืขืจ ืืึธื. ืืื ืืืืื ืฆื ืคึผืจืืืืจื ืึท ืคึผืืึทืฅ ืคืื ืืึธืืขืืก, ืึธืืขืจ ืืืืฃ ืื ืคืึทืจืงืขืจื, ืืื ืืึธื ื ืืฉื ืืืขืื ืฆื ืจืืจืืื ืืขืืขืจ ืึทืจืงืึทืืขืงืืฉืขืจ ืืึทื ืืืึทืื. ืืืจ ืืขื ืขื ืืืื ืืืคึผืจืืืืื ืคืื ืื ืืขืืขืื ืืืื ืฆื ื ืืฆื ืื ืืืืืฅ ืคืื ืืึธืืขืืก ืคึผืจื-ืืจืืื ื ืืืืฃ ืืืึทืืืฉื ืขื. ืืื ืืขืืืืื ืืืขื, ืืขืืขืจื ื ืื ืืึทืงืืืืขื ืืืืฉืึทื ืืขืืึธืืคึฟื. ืคึฟืื ืงืฆืืข get_config() ืึทืืึทืื ืืืจ ืฆื ืืึทืงืืืขื ืึท ืืึทืฉืจืืึทืืื ื ืคืื ืื ืืึธืืขื ืืื ืึท ืคืึธืจืขื ืคึผืึทืกืืง ืคึฟืึทืจ ืขืืืืื ื (base_model_conf$layers - ืึท ืจืขืืืืขืจ ืจ ืจืฉืืื), ืืื ืื ืคึฟืื ืงืฆืืข from_config() ืคึผืขืจืคืึธืจืื ืื ืคืึทืจืงืขืจื ืงืึทื ืืืขืจืืฉืึทื ืฆื ืึท ืืึธืืขื ืืืืคืขืฅ:
base_model_conf <- get_config(base_model)
base_model_conf$layers[[1]]$config$batch_input_shape[[4]] <- 1L
base_model <- from_config(base_model_conf)ืืืฆื ืขืก ืืื ื ืืฉื ืฉืืืขืจ ืฆื ืฉืจืืึทืื ืึท ืื ืืืืขืจืกืึทื ืคึฟืื ืงืฆืืข ืฆื ืืึทืงืืืขื ืงืืื ืคืื ืื ืกืึทืคึผืืืื ืงืขืจืึทืก ืืึธืืขืืก ืืื ืึธืืขืจ ืึธื ืืืืืฅ ืืจืืื ื ืืืืฃ ืืืืื ืขื:
ืคืื ืงืฆืืข ืคึฟืึทืจ ืืึธืืืื ื ืคืึทืจืืืง ืึทืจืงืึทืืขืงืืฉืขืจื
get_model <- function(name = "mobilenet_v2",
input_shape = NULL,
weights = "imagenet",
pooling = "avg",
num_classes = NULL,
optimizer = keras::optimizer_adam(lr = 0.002),
loss = "categorical_crossentropy",
metrics = NULL,
color = TRUE,
compile = FALSE) {
# ะัะพะฒะตัะบะฐ ะฐัะณัะผะตะฝัะพะฒ
checkmate::assert_string(name)
checkmate::assert_integerish(input_shape, lower = 1, upper = 256, len = 3)
checkmate::assert_count(num_classes)
checkmate::assert_flag(color)
checkmate::assert_flag(compile)
# ะะพะปััะฐะตะผ ะพะฑัะตะบั ะธะท ะฟะฐะบะตัะฐ keras
model_fun <- get0(paste0("application_", name), envir = asNamespace("keras"))
# ะัะพะฒะตัะบะฐ ะฝะฐะปะธัะธั ะพะฑัะตะบัะฐ ะฒ ะฟะฐะบะตัะต
if (is.null(model_fun)) {
stop("Model ", shQuote(name), " not found.", call. = FALSE)
}
base_model <- model_fun(
input_shape = input_shape,
include_top = FALSE,
weights = weights,
pooling = pooling
)
# ะัะปะธ ะธะทะพะฑัะฐะถะตะฝะธะต ะฝะต ัะฒะตัะฝะพะต, ะผะตะฝัะตะผ ัะฐะทะผะตัะฝะพััั ะฒั
ะพะดะฐ
if (!color) {
base_model_conf <- keras::get_config(base_model)
base_model_conf$layers[[1]]$config$batch_input_shape[[4]] <- 1L
base_model <- keras::from_config(base_model_conf)
}
predictions <- keras::get_layer(base_model, "global_average_pooling2d_1")$output
predictions <- keras::layer_dense(predictions, units = num_classes, activation = "softmax")
model <- keras::keras_model(
inputs = base_model$input,
outputs = predictions
)
if (compile) {
keras::compile(
object = model,
optimizer = optimizer,
loss = loss,
metrics = metrics
)
}
return(model)
}ืืืขื ื ืืฆื ืืืื-ืงืึทื ืึทื ืืืืืขืจ, ืงืืื ืคึผืจืืืจืืื ื ืืืืืฅ ืืขื ืขื ืืขื ืืฆื. ืืึธืก ืงืขื ืืืื ืคืึทืจืคืขืกืืืงื: ื ืืฆื ืื ืคึฟืื ืงืฆืืข get_weights() ืืึทืงืืืขื ืื ืืึธืืขื ืืืืืฅ ืืื ืื ืคืึธืจืขื ืคืื ืึท ืจืฉืืื ืคืื R ืขืจืืื, ืืืืฉื ืื ืืืกืืขืกืืื ื ืคืื ืืขืจ ืขืจืฉืืขืจ ืขืืขืืขื ื ืคืื ืืขืจ ืจืฉืืื (ืืืจื ื ืขืืขื ืืืื ืงืึธืืืจ ืงืึทื ืึทื ืึธืืขืจ ืึทืืืจืืืืฉืื ื ืึทืืข ืืจืื), ืืื ืืึทื ืืึธืื ืื ืืืืืฅ ืฆืืจืืง ืืื ืื ืืึธืืขื ืืื ืื ืคึฟืื ืงืฆืืข. set_weights(). ืืืจ ืงืืื ืืึธื ืฆืืืขืืขืื ืืขื ืคืึทื ืืงืฉืึทื ืึทืืืื, ืืืืึทื ืืื ืืขื ืืื ืข ืขืก ืืื ืฉืืื ืงืืึธืจ ืึทื ืขืก ืืื ืืขืจ ืคึผืจืึธืืืงืืืื ืฆื ืึทืจืืขืื ืืื ืงืึธืืืจ ืืืืืขืจ.
ืืืจ ืืึธืื ืืืจืืืขืงืึธืื ืจืืึฟ ืคืื ืื ืืงืกืคึผืขืจืึทืืึทื ืฅ ืืื ืืึธืืืื ืขื ืืืขืจืกืืขืก 1 ืืื 2, ืืื ืืขืืื ื ืืื ืจืขืกื ืขื34. ืืขืจ ืืึธืืขืจื ืึทืจืงืึทืืขืงืืฉืขืจื ืึทืืึท ืืื SE-ResNeXt ืืึธืื ืืืจืืืขืงืึธืื ืืื ืืื ืืขื ืคืึทืจืืขืกื. ืฆืื ืืึทืืืืขืจื, ืืืจ ืืึธืื ื ืืฉื ืืึธืื ืคืึทืจืืืง ืืืคึผืืึทืืึทื ืฅ ืฆื ืืื ืืืขืจ ืืึทืืืึทืืืงืื ื, ืืื ืืืจ ืืึธืื ื ืืฉื ืืขืฉืจืืื ืืื ืืืขืจ ืืืืืขื ืข (ืึธืืขืจ ืืืจ ืืืขืื ืืืฉืืืื ืฉืจืืึทืื).
5. ืคึผืึทืจืึทืืขืืขืจืืืึทืืืึธื ืคืื ืกืงืจืืคึผืก
ืคึฟืึทืจ ืงืึทื ืืืื ืืึทื ืก, ืึทืืข ืงืึธื ืคึฟืึทืจ ืกืืึทืจืืื ื ืืจืืื ืื ื ืืื ืืืืืื ื ืืื ืึท ืืืื ืฉืจืืคื, ืคึผืึทืจืึทืืขืืขืจืืืื ื ืืฆื ืืื ืืืื:
doc <- '
Usage:
train_nn.R --help
train_nn.R --list-models
train_nn.R [options]
Options:
-h --help Show this message.
-l --list-models List available models.
-m --model=<model> Neural network model name [default: mobilenet_v2].
-b --batch-size=<size> Batch size [default: 32].
-s --scale-factor=<ratio> Scale factor [default: 0.5].
-c --color Use color lines [default: FALSE].
-d --db-dir=<path> Path to database directory [default: Sys.getenv("db_dir")].
-r --validate-ratio=<ratio> Validate sample ratio [default: 0.995].
-n --n-gpu=<number> Number of GPUs [default: 1].
'
args <- docopt::docopt(doc)ืคึผืขืงื ืืึธืงืึธืคึผื ืจืขืคึผืจืึทืืขื ืฅ ืื ืืืคึผืืึทืืขื ืืืืฉืึทื ืคึฟืึทืจ ืจ. ืืื ืืืึทื ืืืืฃ, ืกืงืจืืคึผืก ืืขื ืขื ืืึธื ืืฉื ืืื ืคึผืฉืื ืงืึทืืึทื ืื ืืื Rscript bin/train_nn.R -m resnet50 -c -d /home/andrey/doodle_db ืึธืืขืจ ./bin/train_nn.R -m resnet50 -c -d /home/andrey/doodle_db, ืืืื ืืขืงืข train_nn.R ืืื ืขืงืกืขืงืืืึทืืืข (ืืขื ืืึทืคึฟืขื ืืืขื ืึธื ืืืืื ืืจืืื ืื ื ืื ืืึธืืขื resnet50 ืืืืฃ ืืจืื-ืงืึธืืืจ ืืืืืขืจ ืืื 128x128 ืืืืืฆืขืื, ืื ืืึทืืึทืืืืก ืืืื ืืืื ืืืื ืืื ืืขืจ ืืขืงืข /home/andrey/doodle_db). ืืืจ ืงืขื ืขื ืืืืื ืืขืจื ืขื ืืืืงืืึทื, ืึธืคึผืืืืืืขืจ ืืืคึผ ืืื ืงืืื ืื ืืขืจืข ืงืืกืืึธืืืืึทืืืข ืคึผืึทืจืึทืืขืืขืจืก ืฆื ืืขืจ ืจืฉืืื. ืืื ืืขื ืคึผืจืึธืฆืขืก ืคืื ืฆืืืจืืืืื ื ืคืื ืื ืืืกืืึทืืข, ืขืก ืคืืจืงืขืจื ืืืืก ืึทื ืื ืึทืจืงืึทืืขืงืืฉืขืจ mobilenet_v2 ืคืื ืื ืืืฆืืืงืข ืืืขืจืกืืข ืงืขืจืึทืก ืืื ืจ ื ืืฆื ืจืขืื ืฆื ืขื ืืขืจืื ืืขื ืืืึธืก ืืขื ืขื ื ืืฉื ืืขื ืืืขื ืืื ืืฉืืื ืื R ืคึผืขืงื, ืืืจ ืืืืจืื ืคึฟืึทืจ ืืื ืฆื ืคืึทืจืจืืืื ืขืก.
ืืขืจ ืฆืืืึทื ื ืืึธื ืืขืืืื ืขืก ืืขืืืขื ืฆื ืืืืืืืืง ืคืึทืจืืืืขืจื ืืงืกืคึผืขืจืึทืืึทื ืฅ ืืื ืคืึทืจืฉืืืขื ืข ืืึธืืขืืก ืงืึทืืคึผืขืจื ืืื ืื ืืขืจ ืืจืืืืฆืืื ืขืื ืงืึทืืขืจ ืคืื ืกืงืจืืคึผืก ืืื RStudio (ืืืจ ืืึธื ืืขื ืคึผืขืงื ืืื ืึท ืืขืืืขื ืื ืืขืจ ืืจืืจื ). ืึธืืขืจ ืืขืจ ืืืืคึผื ืืืึทืืข ืืื ืื ืคืืืืงืืื ืฆื ืืืืื ืคืืจื ืื ืงืึทืืขืจ ืคืื ืกืงืจืืคึผืก ืืื ืืึธืงืงืขืจ ืึธืืขืจ ืคืฉืื ืืืืฃ ืื ืกืขืจืืืขืจ, ืึธื ืื ืกืืึธืืื ื RStudio ืคึฟืึทืจ ืืขื.
6. ืืึธืงืขืจืืืึทืืืึธื ืคืื ืกืงืจืืคึผืก
ืืืจ ืืขืืืืื ื ืืึธืงืงืขืจ ืฆื ืขื ืฉืืจ ืคึผืึธืจืืึทืืืืืื ืคืื ืื ืกืืืืืืข ืคึฟืึทืจ ืืจืืื ืื ื ืืึธืืขืืก ืฆืืืืฉื ืืึทื ืฉืึทืคึฟื ืืืืืืืืขืจ ืืื ืคึฟืึทืจ ืืื ืืืคึผืืืืืึทื ื ืืื ืื ืืืึธืืงื. ืืืจ ืงืขื ืขื ืึธื ืืืืื ืืึทืงืึทื ื ืืื ืืขื ืืขืฆืืึทื, ืืืึธืก ืืื ืืขืคืืขืจืขื ืืืืขืืืืื ืืืขื ืคึฟืึทืจ ืึท ืจ ืคึผืจืึธืืจืึทืืืกื ืกืขืจืืข ืคืื โโืืืืกืืืืขืก ืึธืืขืจ .
ืืึธืงืขืจ ืึทืืึทืื ืืืจ ืฆื ืฉืึทืคึฟื ืืืื ืืืืืขื ืข ืืืืืขืจ ืคึฟืื ืงืจืึทืฆื ืืื ื ืืฆื ืื ืืขืจืข ืืืืืขืจ ืืื ืึท ืืงืขืจ ืคึฟืึทืจ ืงืจืืืืืื ื ืืืื ืืืืืขื ืข. ืืืขื ืึทื ืึทืืืืืื ื ืื ืื ืืืฆื ืึธืคึผืฆืืขืก, ืืืจ ืืขืงืืืขื ืฆื ืื ืืกืงื ื ืึทื ืื ืกืืึธืืื ื NVIDIA, CUDA + cuDNN ืืจืืืืขืจืก ืืื ืคึผืืืืึธื ืืืืืจืขืจืื ืืื ืึท ืคืขืจืื ืืืึทืืืืึทื ืึทืก ืืืื ืคืื ืื ืืืื, ืืื ืืืจ ืืึทืฉืืึธืกื ืฆื ื ืขืืขื ืื ืืึทืึทืืืขืจ ืืืื ืืื ืึท ืืงืขืจ. tensorflow/tensorflow:1.12.0-gpu, ืึทืืื ื ืื ื ืืืืืง R ืคึผืึทืงืึทืืืฉืึทื ืืึธืจื.
ืื ืืขืฆื ืืึธืงืงืขืจ ืืขืงืข ืืขืงืืงื ืืื ืืึธืก:
dockerfile
FROM tensorflow/tensorflow:1.12.0-gpu
MAINTAINER Artem Klevtsov <[email protected]>
SHELL ["/bin/bash", "-c"]
ARG LOCALE="en_US.UTF-8"
ARG APT_PKG="libopencv-dev r-base r-base-dev littler"
ARG R_BIN_PKG="futile.logger checkmate data.table rcpp rapidjsonr dbi keras jsonlite curl digest remotes"
ARG R_SRC_PKG="xtensor RcppThread docopt MonetDBLite"
ARG PY_PIP_PKG="keras"
ARG DIRS="/db /app /app/data /app/models /app/logs"
RUN source /etc/os-release &&
echo "deb https://cloud.r-project.org/bin/linux/ubuntu ${UBUNTU_CODENAME}-cran35/" > /etc/apt/sources.list.d/cran35.list &&
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 &&
add-apt-repository -y ppa:marutter/c2d4u3.5 &&
add-apt-repository -y ppa:timsc/opencv-3.4 &&
apt-get update &&
apt-get install -y locales &&
locale-gen ${LOCALE} &&
apt-get install -y --no-install-recommends ${APT_PKG} &&
ln -s /usr/lib/R/site-library/littler/examples/install.r /usr/local/bin/install.r &&
ln -s /usr/lib/R/site-library/littler/examples/install2.r /usr/local/bin/install2.r &&
ln -s /usr/lib/R/site-library/littler/examples/installGithub.r /usr/local/bin/installGithub.r &&
echo 'options(Ncpus = parallel::detectCores())' >> /etc/R/Rprofile.site &&
echo 'options(repos = c(CRAN = "https://cloud.r-project.org"))' >> /etc/R/Rprofile.site &&
apt-get install -y $(printf "r-cran-%s " ${R_BIN_PKG}) &&
install.r ${R_SRC_PKG} &&
pip install ${PY_PIP_PKG} &&
mkdir -p ${DIRS} &&
chmod 777 ${DIRS} &&
rm -rf /tmp/downloaded_packages/ /tmp/*.rds &&
rm -rf /var/lib/apt/lists/*
COPY utils /app/utils
COPY src /app/src
COPY tests /app/tests
COPY bin/*.R /app/
ENV DBDIR="/db"
ENV CUDA_HOME="/usr/local/cuda"
ENV PATH="/app:${PATH}"
WORKDIR /app
VOLUME /db
VOLUME /app
CMD bash
ืคึฟืึทืจ ืงืึทื ืืืื ืืึทื ืก, ืื ืคึผืึทืงืึทืืืฉืึทื ืืขื ืืฆื ืืขื ืขื ืฉืืขืื ืืื ืืืขืจืืึทืืึทืื; ืืขืจ ืคืึทืจื ืขื ืคืื ืื ืืขืฉืจืืื ืกืงืจืืคึผืก ืืขื ืขื ืงืึทืคึผืื ืื ืื ืงืึทื ืืืื ืขืจื ืืขืฉืึทืก ืคึฟืึทืจืืึทืืืื ื. ืืืจ ืืืื ืคืืจืขื ืืขืจื ืื ืืึทืคึฟืขื ืฉืึธื ืฆื /bin/bash ืคึฟืึทืจ ืื ืคืื ื ืืฆื ืคืื ืืื ืืึทืื /etc/os-release. ืืขื ืึทืืืืืืื ืื ื ืืื ืฆื ืกืคึผืขืฆืืคืืฆืืจื ืื ืึทืก ืืืขืจืกืืข ืืื ืื ืงืึธื.
ืืื ืืขืจืฆื, ืึท ืงืืืื ืืึทืฉ ืฉืจืืคื ืืื ืืขืฉืจืืื ืึทื ืึทืืึทืื ืืืจ ืฆื ืงืึทืืขืจ ืึท ืงืึทื ืืืื ืขืจ ืืื ืคืึทืจืฉืืื ืงืึทืืึทื ืื. ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืึธืก ืงืขื ืืืื ืกืงืจืืคึผืก ืคึฟืึทืจ ืืจืืื ืื ื ื ืขืืจืึทื ื ืขืืืืึธืจืงืก ืืืึธืก ืืขื ืขื ืคืจืืขืจ ืืขืฉืืขืื ืืื ืืขื ืงืึทื ืืืื ืขืจ, ืึธืืขืจ ืึท ืืึทืคึฟืขื ืฉืึธื ืคึฟืึทืจ ืืืืึทืืื ื ืืื ืืึธื ืืืึธืจืื ื ืื ืึธืคึผืขืจืึทืฆืืข ืคืื โโโโืืขื ืงืึทื ืืืื ืขืจ:
ืฉืจืืคื ืฆื ืงืึทืืขืจ ืืขื ืงืึทื ืืืื ืขืจ
#!/bin/sh
DBDIR=${PWD}/db
LOGSDIR=${PWD}/logs
MODELDIR=${PWD}/models
DATADIR=${PWD}/data
ARGS="--runtime=nvidia --rm -v ${DBDIR}:/db -v ${LOGSDIR}:/app/logs -v ${MODELDIR}:/app/models -v ${DATADIR}:/app/data"
if [ -z "$1" ]; then
CMD="Rscript /app/train_nn.R"
elif [ "$1" = "bash" ]; then
ARGS="${ARGS} -ti"
else
CMD="Rscript /app/train_nn.R $@"
fi
docker run ${ARGS} doodles-tf ${CMD}ืืืื ืืขื ืืึทืฉ ืฉืจืืคื ืืื ืืืืคื ืึธื ืคึผืึทืจืึทืืขืืขืจืก, ืื ืฉืจืืคื ืืืขื ืืืื ืืขืจืืคึฟื ืืื ืืขื ืงืึทื ืืืื ืขืจ train_nn.R ืืื ืคืขืืืงืืึทื ืืืึทืืืขืก; ืืืื ืืขืจ ืขืจืฉืืขืจ ืคึผืึธืกืืืืึธื ืึทื ืึทืจืืืืขื ื ืืื "ืืึทืฉ", ืืขืจ ืงืึทื ืืืื ืขืจ ืืืขื ืึธื ืืืืื ืื ืืขืจืึทืงืืืืืื ืืื ืึท ืืึทืคึฟืขื ืฉืึธื. ืืื ืึทืืข ืื ืืขืจืข ืงืึทืกืขืก, ืื ืืืึทืืืขืก ืคืื ืคึผืึธืกืืืืึธื ืึทื ืึทืจืืืืขื ืื ืืขื ืขื ืกืึทืืกืืึทืืืืึทื: CMD="Rscript /app/train_nn.R $@".
ืขืก ืืื ืืืื ืฆื ืืืืขืจืงื ืึทื ืื ืืืืจืขืงืืขืจืื ืืื ืืงืืจ ืืึทืื ืืื ืืึทืืึทืืืืก, ืืื ืืขืืื ื ืืื ืื ืืืขืืืืืึทืืขืจ ืคึฟืึทืจ ืฉืคึผืึธืจื ืืจืืื ื ืืึธืืขืืก, ืืขื ืขื ืืึธืื ืืขื ืืื ืืขื ืงืึทื ืืืื ืขืจ ืคึฟืื ืืขืจ ืืึทืืขืืึธืก ืกืืกืืขื, ืืืึธืก ืึทืืึทืื ืืืจ ืฆื ืึทืงืกืขืก ืื ืจืขืืืืืึทืื ืคืื ืื ืกืงืจืืคึผืก ืึธื ืืื ืืืืืง ืืึทื ืืคึผืืึทืืืืฉืึทื ื.
7. ื ืืฆื ืงืืืคื ืืคึผืืก ืืืืฃ Google ืงืืึธืื
ืืืื ืขืจ ืคืื ืื ืคึฟืขืึดืงืืืื ืคืื ืื ืคืึทืจืืขืกื ืืื ืืขืืืขื ืืืืขืจ ืืืืืืืง ืืึทืื (ืืขื ืืขื ืืืื ืืืื, ืืึทืจืึธืื ืคึฟืื @Leigh.plt ืคึฟืื ODS ืกืืึทืง). ืืจืืืก ืืึทืืฉืึทื ืืขืืคื ืงืึทืืืึทื ืืขื, ืืื ื ืึธื ืืงืกืคึผืขืจืึทืืึทื ืฅ ืืืืฃ ืึท ืคึผืืกื ืืื 1 ืืคึผื, ืืืจ ืืึทืฉืืึธืกื ืฆื ืืขื ืืจืืื ืื ื ืืึธืืขืืก ืืืืฃ ืขืืืขืืข ืืคึผืืก ืืื ืื ืืืึธืืงื. ืืขืืืืื ื GoogleCloud () ืจืขืื ืฆื ืืขืจ ืืจืืืก ืกืขืืขืงืฆืืข ืคืื โโืื ืืืฆื ืงืึทื ืคืืืืขืจืืืฉืึทื ื, ืืืืึทื ืคึผืจืืื ืืื $ 300 ืืึธื ืืก. ืคึฟืื ืืจืื, ืืื ืึธืจืืขืจื ืึท 4xV100 ืืืึทืฉืคึผืื ืืื ืึท SSD ืืื ืึท ืืึธื ืคืื ืืึทืจืึทื, ืืื ืืึธืก ืืื ืืขืืืขื ืึท ืืจืืืก ืืจืืึทื. ืึทืืึท ืึท ืืึทืฉืื ืขืกื ืืขืื ืืขืฉืืืื ื; ืืืจ ืงืขื ืขื ืืืื ืฆืขืืจืืื ืขืงืกืคึผืขืจืืืขื ืืื ื ืึธื ืึท ืคึผืจืึธืืืขื ืจืขืจื - ืืื ืืข. ืคึฟืึทืจ ืืืืืื ืืงืจืืื ืฆืืืขืงื, ืขืก ืืื ืืขืกืขืจ ืฆื ื ืขืืขื ืื ืง80. ืึธืืขืจ ืื ืืจืืืก ืกืืืข ืคืื โโืืึทืจืึทื ืืื ืืขืืืขื ื ืืฆืืง - ืื ืืืึธืืงื ืกืกื ืืื ื ืืฉื ืืืคึผืึธื ืืจื ืืื ืืืึทื ืคืึธืจืฉืืขืืื ื, ืึทืืื ืื ืืึทืืึทืืืืก ืืื ืืจืึทื ืกืคืขืจื ืฆื dev/shm.
ืคืื ืืจืขืกืืข ืืื ืืขืจืขืก ืืื ืื ืงืึธื ืคืจืึทืืืขื ื ืคืึทืจืึทื ืืืืึธืจืืืขื ืคึฟืึทืจ ื ืืฆื ืงืืืคื ืืคึผืืก. ืขืจืฉืืขืจ, ืืขืจ ืืึธืืขื ืืื ืืืฉืืคื ืืืืฃ ืื ืงืคึผื ื ืืฆื ืึท ืงืึธื ืืขืงืกื ืคืึทืจืืืึทืืืขืจ, ืคึผืื ืงื ืืื ืืื ืคึผืืืืึธื:
with(tensorflow::tf$device("/cpu:0"), {
model_cpu <- get_model(
name = model_name,
input_shape = input_shape,
weights = weights,
metrics =(top_3_categorical_accuracy,
compile = FALSE
)
})ืืขืจื ืึธื ืื ืึทื ืงืึธืืคึผืืืขื (ืืึธืก ืืื ืืืืืืืง) ืืึธืืขื ืืื ืงืึทืคึผืื ืฆื ืึท ืืขืืขืื ื ืืืขืจ ืคืื ืื ืืืฆื ืืคึผืืก, ืืื ืืืืื ื ืึธื ืืขื ืขืก ืืื ืงืึทืืคึผืืืื:
model <- keras::multi_gpu_model(model_cpu, gpus = n_gpu)
keras::compile(
object = model,
optimizer = keras::optimizer_adam(lr = 0.0004),
loss = "categorical_crossentropy",
metrics = c(top_3_categorical_accuracy)
)ืืขืจ ืงืืึทืกืืฉ ืืขืื ืืง ืคืื ืืึทื ืงืึทืื ืึทืืข ืืืึทืขืจืก ืึทืืืฅ ืื ืืขืฆืืข, ืืจืืื ืื ื ืื ืืขืฆืืข ืฉืืืืข, ืื ืคืจืขืืื ื ืืื ืจืืืจืืื ืื ื ืื ืืื ืฆืข ืืึธืืขื ืคึฟืึทืจ ืขืืืขืืข ืืคึผืืก ืงืขื ื ืืฉื ืืืื ืืืคึผืืึทืืขื ืึทื.
ืืจืึทืื ืื ื ืืื ืืึธื ืืืึธืจืขื ืึธื ื ืืฆื. tensorboard, ืืืืืืขื ืืื ืฆื ืจืขืงืึธืจืืืจื ืืึธืืก ืืื ืฉืคึผืึธืจื ืืึธืืขืืก ืืื ืื ืคืึธืจืืึทืืืื ื ืขืืขื ื ืึธื ืืขืืขืจ ืขืคึผืึธืก:
ืงืึทืืืืึทืงืงืก
# ะจะฐะฑะปะพะฝ ะธะผะตะฝะธ ัะฐะนะปะฐ ะปะพะณะฐ
log_file_tmpl <- file.path("logs", sprintf(
"%s_%d_%dch_%s.csv",
model_name,
dim_size,
channels,
format(Sys.time(), "%Y%m%d%H%M%OS")
))
# ะจะฐะฑะปะพะฝ ะธะผะตะฝะธ ัะฐะนะปะฐ ะผะพะดะตะปะธ
model_file_tmpl <- file.path("models", sprintf(
"%s_%d_%dch_{epoch:02d}_{val_loss:.2f}.h5",
model_name,
dim_size,
channels
))
callbacks_list <- list(
keras::callback_csv_logger(
filename = log_file_tmpl
),
keras::callback_early_stopping(
monitor = "val_loss",
min_delta = 1e-4,
patience = 8,
verbose = 1,
mode = "min"
),
keras::callback_reduce_lr_on_plateau(
monitor = "val_loss",
factor = 0.5, # ัะผะตะฝััะฐะตะผ lr ะฒ 2 ัะฐะทะฐ
patience = 4,
verbose = 1,
min_delta = 1e-4,
mode = "min"
),
keras::callback_model_checkpoint(
filepath = model_file_tmpl,
monitor = "val_loss",
save_best_only = FALSE,
save_weights_only = FALSE,
mode = "min"
)
)8. ืึทื ืฉืืึธื ืึท ืืกืงื ื
ื ื ืืืขืจ ืคืื ืคืจืืืืขืืขื ืืืึธืก ืืืจ ืืึธืื ืืขืคึผืืึธื ืืขืจื ืืขื ืขื ื ืึธื ื ืืฉื ืืึทืงืืืขื:
- ะฒ ืงืขืจืึทืก ืขืก ืืื ืงืืื ืคืึทืจืืืง ืคืื ืงืฆืืข ืคึฟืึทืจ ืืืืืึธืืึทืืืฉ ืืืื ืคึฟืึทืจ ืื ืึธืคึผืืืืึทื ืืขืจื ืขื ืงืืจืก (ืึทื ืึทืืึธื
lr_finderืืื ืืืืืืืืขืง fast.ai); ืืื ืขืืืขืืข ืื, ืขืก ืืื ืืขืืืขื ืฆื ืึทืจืืืขืจืคืืจื ืืจืื-ืคึผืึทืจืืื ืืืคึผืืึทืืึทื ืฅ ืฆื R, ืืืฉื, ; - ืืื ืึท ืงืึทื ืกืึทืงืืืึทื ืก ืคืื ืื ืคืจืืขืจืืืงืข ืคืื ื, ืขืก ืืื ื ืื ืืขืืืขื ืฆื ืืืืกืงืืืึทืื ืื ืจืืืืืง ืืจืืื ืื ื ืืืืงืืึทื ืืืขื ื ืืฆื ืขืืืขืืข ืืคึผืืก;
- ืขืก ืืื ืึท ืคืขืื ืคืื ืืึธืืขืจื ื ืขืืจืึทื ื ืขืฅ ืึทืจืงืึทืืขืงืืฉืขืจื, ืกืคึผืขืฆืืขื ืื ืคืึทืจ-ืืจืืื ื ืืืืฃ ืืืึทืืืฉื ืขื;
- ืงืืื ืืืื ืฆืืงื ืคึผืึธืืืืืง ืืื ืืืกืงืจืืืื ืึทืืืืืข ืืขืจื ืขื ืจืืืฅ (ืงืึธืกืื ืข ืึทื ืืืื ื ืืื ืืขืืืขื ืืื ืืื ืืืขืจ ืืงืฉื , ืืึทื ืงืขื ).
ืืืึธืก ื ืืฆืืง ืืื ืื ืืขื ืขื ืืขืืขืจื ื ืคืื ืืขื ืคืึทืจืืขืกื:
- ืืืืฃ ืืขืคืืขืจืขื ื ืืืขืจืืง-ืืึทืื ืืึทืื ืืืึทืจื, ืืืจ ืงืขื ืขื ืึทืจืืขืื ืืื ืืืึทืืืฉ (ืงืืืคื ืื ืืจืืืก ืคืื ืืึทืจืึทื) ืืืึทืืืืื ืคืื ืืึทืื ืึธื ืืืืืืืง. ืคึผืืึทืกืืืง ืืขืงื ืืึทืืึท.ืืึทืืืข ืกืึทืืืขืก ืืืงืึธืจื ืจืขืื ืฆื ืืขืจ ืืึธืืืคืืงืึทืืืึธื ืคืื ืืืฉื ืืื ืคึผืืึทืฅ, ืืืึธืก ืืืกืืืืื ืงืึทืคึผืืื ื ืืื, ืืื ืืืขื ืืขืืืืื ื ืจืืืืืง, ืื ืงืืืคึผืึทืืืืึทืืื ืึผืืขื ืฉืืขื ืืืง ืืึทืืืืึทืื ืื ืืขืืกืื ืืืืงืืึทื ืฆืืืืฉื ืึทืืข ืื ืืืฉืืจืื ืืืึธืก ืืขื ืขื ืืืงืื ื ืฆื ืืื ืื ืคึฟืึทืจ ืกืงืจืืคึผืืื ื ืฉืคึผืจืึทืื. ืฉืคึผืึธืจื ืืึทืื ืืื ืึท ืืึทืืึทืืืืก ืึทืืึทืื ืืืจ, ืืื ืคืืืข ืงืึทืกืขืก, ื ืืฉื ืฆื ืืจืึทืืื ืืืขืื ืื ื ืืืืืง ืฆื ืงืืืขืืฉื ืื ืืื ืฆืข ืืึทืืึทืืืืก ืืื ืืึทืจืึทื.
- ืคึผืึทืืขืืขื ืคืึทื ืืงืฉืึทื ื ืืื ืจ ืงืขื ืขื ืืืื ืจืืคึผืืืืกื ืืื ืฉื ืขื ืึธื ืขืก ืืื C ++ ื ืืฆื ืืขื ืคึผืขืงื Rcpp. ืืืื ืืื ืืขืจืฆื ืฆื ื ืืฆื RcppThread ืึธืืขืจ RcppParallel, ืืืจ ืืึทืงืืืขื ืงืจืืึทื-ืคึผืืึทืืคืึธืจืืข ืืึทืืื-ืืจืขืืื ืืืคึผืืึทืืึทื ืฅ, ืึทืืื ืขืก ืืื ื ืื ืืึทืจืคึฟื ืฆื ืคึผืึทืจืึทืืขืืืืืจื ืื ืงืึธื ืืืืฃ ืื ืจ ืืืจืื.
- ืคึผืขืงื Rcpp ืงืขื ืขื ืืืื ืืขืืืืื ื ืึธื ืขืจื ืกื ืืืืกื ืคืื C ++, ืื ืคืืจืืื ืื ืืื ืืืื ืืื ืึทืืืืืื ื . ืืขืืขืจ ืืขืงืขืก ืคึฟืึทืจ ืึท ื ืืืขืจ ืคืื ืงืื C-ืืืืจืขืจืื ืืื xtensor ืื ืืืฆื ืืืืฃ CRAN, ืืึธืก ืืืืกื, ืึทื ืื ืคืจืึทืกืืจืึทืงืืฉืขืจ ืืื ืืขืฉืืคื ืคึฟืึทืจ ืื ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืคึผืจืึทืืืฉืขืงืก ืืืึธืก ืืืกืฉืืืืขื ืคืึทืจืืืง ืืืื-ืคืึธืจืฉืืขืืื ื C ++ ืงืึธื ืืื R. ื ืึธื ืงืึทื ืืืื ืืึทื ืก ืืื ืกืื ืืึทืงืก ืืืืืืืืื ื ืืื ืึท ืกืืึทืืืง C ++ ืงืึธื ืึทื ืึทืืืืขืจ ืืื RStudio.
- ืืึธืงืึธืคึผื ืึทืืึทืื ืืืจ ืฆื ืืืืคื ืืื-ืงืึทื ืืืื ื ืกืงืจืืคึผืก ืืื ืคึผืึทืจืึทืืขืืขืจืก. ืืึธืก ืืื ืืึทืงืืืขื ืคึฟืึทืจ ื ืืฆื ืืืืฃ ืึท ืืืืึทื ืกืขืจืืืขืจ, ืื ืงื. ืืื ืืขืจ ืืึธืงืงืขืจ. ืืื RStudio, ืขืก ืืื ืืืืึทืงืืืขื ืฆื ืืืจืืคืืจื ืคืืืข ืฉืขื ืคืื ืืงืกืคึผืขืจืึทืืึทื ืฅ ืืื ืืจืืื ืื ื ื ืขืืจืึทื ื ืขืืืืึธืจืงืก, ืืื ืื ืกืืึธืืื ื ืื IDE ืืืืฃ ืื ืกืขืจืืืขืจ ืืื ืืื ื ืื ืฉืืขื ืืืง ืืขืจืขืืืคืืจืืืงื.
- ืืึธืงืขืจ ืื ืฉืืจื ืงืึธื ืคึผืึธืจืืึทืืืืืื ืืื ืจืขืคึผืจืึธืืืกืืืืืืื ืคืื ืจืขืืืืืึทืื ืฆืืืืฉื ืืขืืืขืืึธืคึผืขืจืก ืืื ืคืึทืจืฉืืืขื ืข ืืืขืจืกืืขืก ืคืื ืื ืึทืก ืืื ืืืืืจืขืจืื, ืืื ืืขืืื ื ืืื ืื ืคืื ืืืจืืคืืจืื ื ืืืืฃ ืกืขืจืืืขืจืก. ืืืจ ืงืขื ืขื ืงืึทืืขืจ ืื ืืื ืฆืข ืืจืืื ืื ื ืจืขืจื - ืืื ืืข ืืื ืืืืื ืืืื ืืึทืคึฟืขื.
- Google ืงืืึธืื ืืื ืึท ืืืืืฉืขื-ืคืจืืึทื ืืืขื ืืืขื ืฆื ืขืงืกืคึผืขืจืืืขื ื ืืืืฃ ืืืึทืขืจ ืืึทืื ืืืึทืจื, ืึธืืขืจ ืืืจ ืืึทืจืคึฟื ืฆื ืงืืืึทืื ืงืึทื ืคืืืืขืจืืืฉืึทื ื ืงืขืจืคืึทืื.
- ืืขืกืื ืื ืืืืงืืึทื ืคืื ืืืื ืงืึธื ืคืจืึทืืืึทื ืฅ ืืื ืืืืขืจ ื ืืฆืืง, ืกืคึผืขืฆืืขื ืืืขื ืงืึทืืืืื ืื ื ืจ ืืื C ++, ืืื ืืื ืืขื ืคึผืขืงื ืืึทื ืง - ืืืื ืืืืขืจ ืืจืื ื.
ืงืืืืขืืืืง, ืื ืืขืจืคืึทืจืื ื ืืื ืืขืืืขื ืืืืขืจ ืจืืืืึธืจืืื ื ืืื ืืืจ ืคืึธืจืืขืฆื ืฆื ืึทืจืืขืื ืฆื ืกืึธืืืืข ืขืืืขืืข ืคืื โโโโืื ืืฉืื.
ืืงืืจ: www.habr.com