🥇Quick Draw Doodle Recognition: yuav ua li cas phooj ywg nrog R, C ++ thiab neural networks

Hlo Habr!

Lub caij nplooj zeeg kawg, Kaggle tau tuav kev sib tw los faib cov duab kos duab, Quick Draw Doodle Recognition, uas, ntawm lwm tus, ib pab pawg ntawm R-scientists tau koom nrog: Artem Klevtsova, Philippa Manager и Andrey Ogurtsov. Peb yuav tsis piav qhia txog kev sib tw hauv kev nthuav dav; uas twb tau ua tiav hauv kev tshaj tawm tsis ntev los no.

Lub sijhawm no nws tsis ua haujlwm nrog kev ua liaj ua teb medal, tab sis ntau qhov kev paub tseem ceeb tau txais, yog li kuv xav qhia rau cov zej zog txog ntau yam tseem ceeb tshaj plaws thiab muaj txiaj ntsig ntawm Kagle thiab hauv kev ua haujlwm txhua hnub. Ntawm cov ncauj lus tau tham txog: lub neej nyuaj yam tsis muaj OpenCV, JSON parsing (cov piv txwv no tshuaj xyuas kev koom ua ke ntawm C ++ code rau hauv cov ntawv sau lossis pob hauv R siv Rcpp), parameterization ntawm scripts thiab dockerization ntawm qhov kawg daws. Tag nrho cov cai los ntawm cov lus hauv ib daim ntawv tsim nyog rau kev ua tiav yog muaj nyob rau hauv chaw khaws cia.

Txheem:

Ua haujlwm zoo thauj cov ntaub ntawv los ntawm CSV rau hauv MonetDB
Npaj batch
Iterators rau unloading batch los ntawm lub database
Xaiv ib tug qauv Architecture
Cov ntawv parameterization
Dockerization ntawm scripts
Siv ntau GPUs ntawm Google Huab
Es tsis txhob ib tug xaus

1. Ua haujlwm zoo thauj cov ntaub ntawv los ntawm CSV rau hauv MonetDB database

Cov ntaub ntawv nyob rau hauv qhov kev sib tw no yog muab tsis yog nyob rau hauv daim ntawv ntawm cov duab npaj txhij, tab sis nyob rau hauv daim ntawv ntawm 340 CSV cov ntaub ntawv (ib cov ntaub ntawv rau txhua chav kawm) muaj JSONs nrog point coordinates. Los ntawm kev txuas cov ntsiab lus no nrog cov kab, peb tau txais cov duab kawg uas ntsuas 256x256 pixels. Tsis tas li ntawd rau txhua cov ntaub ntawv muaj ib daim ntawv lo qhia seb daim duab puas raug lees paub los ntawm tus neeg faib khoom siv thaum lub sijhawm sau cov ntaub ntawv, ob tsab ntawv code ntawm lub tebchaws nyob ntawm tus sau daim duab, tus cim tshwj xeeb, lub sijhawm sau. thiab ib lub npe chav kawm uas phim cov ntaub ntawv npe. Ib qho yooj yim version ntawm tus thawj cov ntaub ntawv hnyav 7.4 GB nyob rau hauv lub archive thiab kwv yees li 20 GB tom qab unpacking, tag nrho cov ntaub ntawv tom qab unpacking yuav siv sij hawm txog 240 GB. Cov koom haum tau xyuas kom meej tias ob qho tib si rov tsim cov duab kos tib yam, txhais tau tias tag nrho cov version yog redundant. Txawm li cas los xij, khaws 50 lab dluab hauv cov duab nraaj lossis hauv daim ntawv arrays tam sim ntawd suav tias tsis muaj txiaj ntsig, thiab peb tau txiav txim siab muab tag nrho cov ntaub ntawv CSV los ntawm cov ntaub ntawv khaws tseg. train_simplified.zip mus rau hauv cov ntaub ntawv nrog rau tiam tom ntej ntawm cov duab ntawm qhov yuav tsum tau loj "ntawm ya" rau txhua batch.

Ib qho kev pov thawj zoo tau raug xaiv los ua DBMS MonetDB, uas yog ib qho kev siv rau R raws li ib pob MonetDBLite. Cov pob muaj xws li ib qho embedded version ntawm tus neeg rau zaub mov database thiab tso cai rau koj tuaj tos tus neeg rau zaub mov ncaj qha los ntawm kev sib tham R thiab ua haujlwm nrog nws nyob ntawd. Tsim ib tug database thiab txuas mus rau nws yog ua nrog ib tug hais kom ua:

con <- DBI::dbConnect(drv = MonetDBLite::MonetDBLite(), Sys.getenv("DBDIR"))

Peb yuav tsum tsim ob lub rooj: ib qho rau tag nrho cov ntaub ntawv, lwm yam rau cov ntaub ntawv kev pab cuam txog cov ntaub ntawv downloaded (pab tau yog tias muaj ib yam dab tsi mus tsis ncaj ncees lawm thiab cov txheej txheem yuav tsum tau rov pib dua tom qab rub tawm ob peb cov ntaub ntawv):

Tsim cov rooj

if (!DBI::dbExistsTable(con, "doodles")) {
  DBI::dbCreateTable(
    con = con,
    name = "doodles",
    fields = c(
      "countrycode" = "char(2)",
      "drawing" = "text",
      "key_id" = "bigint",
      "recognized" = "bool",
      "timestamp" = "timestamp",
      "word" = "text"
    )
  )
}

if (!DBI::dbExistsTable(con, "upload_log")) {
  DBI::dbCreateTable(
    con = con,
    name = "upload_log",
    fields = c(
      "id" = "serial",
      "file_name" = "text UNIQUE",
      "uploaded" = "bool DEFAULT false"
    )
  )
}

Txoj kev ceev tshaj plaws los thauj cov ntaub ntawv rau hauv cov ntaub ntawv yog ncaj qha luam CSV cov ntaub ntawv siv SQL - hais kom ua COPY OFFSET 2 INTO tablename FROM path USING DELIMITERS ',','n','"' NULL AS '' BEST EFFORTqhov twg tablename - lub npe rooj thiab path - txoj kev mus rau cov ntaub ntawv. Thaum ua haujlwm nrog cov ntaub ntawv khaws tseg, nws tau pom tias qhov kev siv ua tiav unzip nyob rau hauv R tsis ua hauj lwm kom raug nrog ib tug xov tooj ntawm cov ntaub ntawv los ntawm lub archive, yog li peb siv lub system unzip (siv parameter getOption("unzip")).

Muaj nuj nqi rau kev sau ntawv rau lub database

#' @title Извлечение и загрузка файлов
#'
#' @description
#' Извлечение CSV-файлов из ZIP-архива и загрузка их в базу данных
#'
#' @param con Объект подключения к базе данных (класс `MonetDBEmbeddedConnection`).
#' @param tablename Название таблицы в базе данных.
#' @oaram zipfile Путь к ZIP-архиву.
#' @oaram filename Имя файла внури ZIP-архива.
#' @param preprocess Функция предобработки, которая будет применена извлечённому файлу.
#'   Должна принимать один аргумент `data` (объект `data.table`).
#'
#' @return `TRUE`.
#'
upload_file <- function(con, tablename, zipfile, filename, preprocess = NULL) {
  # Проверка аргументов
  checkmate::assert_class(con, "MonetDBEmbeddedConnection")
  checkmate::assert_string(tablename)
  checkmate::assert_string(filename)
  checkmate::assert_true(DBI::dbExistsTable(con, tablename))
  checkmate::assert_file_exists(zipfile, access = "r", extension = "zip")
  checkmate::assert_function(preprocess, args = c("data"), null.ok = TRUE)

  # Извлечение файла
  path <- file.path(tempdir(), filename)
  unzip(zipfile, files = filename, exdir = tempdir(), 
        junkpaths = TRUE, unzip = getOption("unzip"))
  on.exit(unlink(file.path(path)))

  # Применяем функция предобработки
  if (!is.null(preprocess)) {
    .data <- data.table::fread(file = path)
    .data <- preprocess(data = .data)
    data.table::fwrite(x = .data, file = path, append = FALSE)
    rm(.data)
  }

  # Запрос к БД на импорт CSV
  sql <- sprintf(
    "COPY OFFSET 2 INTO %s FROM '%s' USING DELIMITERS ',','n','"' NULL AS '' BEST EFFORT",
    tablename, path
  )
  # Выполнение запроса к БД
  DBI::dbExecute(con, sql)

  # Добавление записи об успешной загрузке в служебную таблицу
  DBI::dbExecute(con, sprintf("INSERT INTO upload_log(file_name, uploaded) VALUES('%s', true)",
                              filename))

  return(invisible(TRUE))
}

Yog tias koj xav tau hloov lub rooj ua ntej sau rau hauv cov ntaub ntawv, nws yog txaus kom dhau ntawm qhov kev sib cav preprocess muaj nuj nqi uas yuav hloov cov ntaub ntawv.

Code rau sequentially loading cov ntaub ntawv rau hauv lub database:

Sau cov ntaub ntawv rau lub database

# Список файлов для записи
files <- unzip(zipfile, list = TRUE)$Name

# Список исключений, если часть файлов уже была загружена
to_skip <- DBI::dbGetQuery(con, "SELECT file_name FROM upload_log")[[1L]]
files <- setdiff(files, to_skip)

if (length(files) > 0L) {
  # Запускаем таймер
  tictoc::tic()
  # Прогресс бар
  pb <- txtProgressBar(min = 0L, max = length(files), style = 3)
  for (i in seq_along(files)) {
    upload_file(con = con, tablename = "doodles", 
                zipfile = zipfile, filename = files[i])
    setTxtProgressBar(pb, i)
  }
  close(pb)
  # Останавливаем таймер
  tictoc::toc()
}

# 526.141 sec elapsed - копирование SSD->SSD
# 558.879 sec elapsed - копирование USB->SSD

Cov ntaub ntawv thauj khoom lub sij hawm yuav txawv nyob ntawm seb tus yam ntxwv ceev ntawm tus tsav siv. Hauv peb qhov xwm txheej, kev nyeem ntawv thiab sau ntawv hauv ib qho SSD lossis los ntawm lub flash drive (qhov chaw ntaub ntawv) mus rau SSD (DB) yuav siv sijhawm tsawg dua 10 feeb.

Nws yuav siv sij hawm li ob peb feeb ntxiv los tsim ib kem nrog cov ntawv sau npe hauv chav kawm thiab ib kab ntawv ntsuas (ORDERED INDEX) nrog kab zauv los ntawm qhov kev soj ntsuam yuav raug coj mus kuaj thaum tsim batch:

Tsim cov kab ntawv ntxiv thiab Index

message("Generate lables")
invisible(DBI::dbExecute(con, "ALTER TABLE doodles ADD label_int int"))
invisible(DBI::dbExecute(con, "UPDATE doodles SET label_int = dense_rank() OVER (ORDER BY word) - 1"))

message("Generate row numbers")
invisible(DBI::dbExecute(con, "ALTER TABLE doodles ADD id serial"))
invisible(DBI::dbExecute(con, "CREATE ORDERED INDEX doodles_id_ord_idx ON doodles(id)"))

Txhawm rau daws qhov teeb meem ntawm kev tsim cov batch ntawm ya, peb xav tau kom ua tiav qhov siab tshaj plaws ntawm kev rho tawm random kab los ntawm lub rooj. doodles. Rau qhov no peb siv 3 tricks. Thawj yog los txo qhov dimensionality ntawm hom uas khaws cov kev soj ntsuam ID. Nyob rau hauv thawj cov ntaub ntawv teev, hom yuav tsum tau khaws tus ID yog bigint, tab sis tus naj npawb ntawm cov kev soj ntsuam ua rau nws ua tau kom haum lawv cov cim, sib npaug ntawm tus lej ordinal, rau hauv hom int. Kev tshawb nrhiav tau nrawm dua hauv qhov no. Qhov thib ob ua kom yuam kev yog siv ORDERED INDEX - Peb tuaj rau qhov kev txiav txim siab no, tau dhau los ntawm txhua yam muaj kev xaiv. Qhov thib peb yog siv cov lus nug parameterized. Lub ntsiab ntawm txoj kev yog ua kom tiav cov lus txib ib zaug PREPARE nrog rau tom qab siv cov lus qhia npaj thaum tsim ib pawg ntawm cov lus nug ntawm tib hom, tab sis qhov tseeb muaj qhov zoo dua piv nrog ib qho yooj yim. SELECT muab tso rau hauv qhov ntau ntawm kev ua yuam kev.

Cov txheej txheem ntawm upload cov ntaub ntawv siv tsis pub ntau tshaj 450 MB ntawm RAM. Ntawd yog, txoj hauv kev tau piav qhia tso cai rau koj txav cov ntaub ntawv hnyav kaum tawm gigabytes ntawm yuav luag txhua yam khoom siv nyiaj txiag, suav nrog qee cov khoom siv ib leeg, uas zoo nkauj heev.

Txhua yam uas tseem tshuav yog ntsuas qhov ceev ntawm kev khaws cov ntaub ntawv (random) thiab ntsuas qhov ntsuas thaum ntsuas qhov sib txawv ntawm qhov sib txawv:

Database benchmark

library(ggplot2)

set.seed(0)
# Подключение к базе данных
con <- DBI::dbConnect(MonetDBLite::MonetDBLite(), Sys.getenv("DBDIR"))

# Функция для подготовки запроса на стороне сервера
prep_sql <- function(batch_size) {
  sql <- sprintf("PREPARE SELECT id FROM doodles WHERE id IN (%s)",
                 paste(rep("?", batch_size), collapse = ","))
  res <- DBI::dbSendQuery(con, sql)
  return(res)
}

# Функция для извлечения данных
fetch_data <- function(rs, batch_size) {
  ids <- sample(seq_len(n), batch_size)
  res <- DBI::dbFetch(DBI::dbBind(rs, as.list(ids)))
  return(res)
}

# Проведение замера
res_bench <- bench::press(
  batch_size = 2^(4:10),
  {
    rs <- prep_sql(batch_size)
    bench::mark(
      fetch_data(rs, batch_size),
      min_iterations = 50L
    )
  }
)
# Параметры бенчмарка
cols <- c("batch_size", "min", "median", "max", "itr/sec", "total_time", "n_itr")
res_bench[, cols]

#   batch_size      min   median      max `itr/sec` total_time n_itr
#        <dbl> <bch:tm> <bch:tm> <bch:tm>     <dbl>   <bch:tm> <int>
# 1         16   23.6ms  54.02ms  93.43ms     18.8        2.6s    49
# 2         32     38ms  84.83ms 151.55ms     11.4       4.29s    49
# 3         64   63.3ms 175.54ms 248.94ms     5.85       8.54s    50
# 4        128   83.2ms 341.52ms 496.24ms     3.00      16.69s    50
# 5        256  232.8ms 653.21ms 847.44ms     1.58      31.66s    50
# 6        512  784.6ms    1.41s    1.98s     0.740       1.1m    49
# 7       1024  681.7ms    2.72s    4.06s     0.377      2.16m    49

ggplot(res_bench, aes(x = factor(batch_size), y = median, group = 1)) +
  geom_point() +
  geom_line() +
  ylab("median time, s") +
  theme_minimal()

DBI::dbDisconnect(con, shutdown = TRUE)

2. Npaj batch

Tag nrho cov txheej txheem npaj batch muaj cov kauj ruam hauv qab no:

Parsing ob peb JSONs uas muaj vectors ntawm cov hlua nrog kev sib koom ua ke ntawm cov ntsiab lus.
Kos cov kab xim raws li kev sib koom ua ke ntawm cov ntsiab lus ntawm cov duab ntawm qhov xav tau loj (piv txwv li, 256 × 256 lossis 128 × 128).
Hloov cov duab tshwm sim rau hauv lub tensor.

Raws li ib feem ntawm kev sib tw ntawm Python kernels, qhov teeb meem tau daws feem ntau siv OpenCV. Ib qho yooj yim thiab pom tseeb tshaj analogues hauv R yuav zoo li no:

Ua raws li JSON rau Tensor Hloov hauv R

r_process_json_str <- function(json, line.width = 3, 
                               color = TRUE, scale = 1) {
  # Парсинг JSON
  coords <- jsonlite::fromJSON(json, simplifyMatrix = FALSE)
  tmp <- tempfile()
  # Удаляем временный файл по завершению функции
  on.exit(unlink(tmp))
  png(filename = tmp, width = 256 * scale, height = 256 * scale, pointsize = 1)
  # Пустой график
  plot.new()
  # Размер окна графика
  plot.window(xlim = c(256 * scale, 0), ylim = c(256 * scale, 0))
  # Цвета линий
  cols <- if (color) rainbow(length(coords)) else "#000000"
  for (i in seq_along(coords)) {
    lines(x = coords[[i]][[1]] * scale, y = coords[[i]][[2]] * scale, 
          col = cols[i], lwd = line.width)
  }
  dev.off()
  # Преобразование изображения в 3-х мерный массив
  res <- png::readPNG(tmp)
  return(res)
}

r_process_json_vector <- function(x, ...) {
  res <- lapply(x, r_process_json_str, ...)
  # Объединение 3-х мерных массивов картинок в 4-х мерный в тензор
  res <- do.call(abind::abind, c(res, along = 0))
  return(res)
}

Kev kos duab yog ua tiav siv tus qauv R cov cuab yeej thiab khaws cia rau ib ntus PNG khaws cia hauv RAM (ntawm Linux, ib ntus R cov npe nyob hauv phau ntawv teev npe /tmp, mounted hauv RAM). Cov ntaub ntawv no yog tom qab ntawd nyeem ua peb-dimensional array nrog cov lej ntawm 0 txog 1. Qhov no yog qhov tseem ceeb vim tias BMP ntau dua yuav raug nyeem rau hauv cov khoom nyoos nrog cov lej hex xim.

Cia peb sim qhov tshwm sim:

zip_file <- file.path("data", "train_simplified.zip")
csv_file <- "cat.csv"
unzip(zip_file, files = csv_file, exdir = tempdir(), 
      junkpaths = TRUE, unzip = getOption("unzip"))
tmp_data <- data.table::fread(file.path(tempdir(), csv_file), sep = ",", 
                              select = "drawing", nrows = 10000)
arr <- r_process_json_str(tmp_data[4, drawing])
dim(arr)
# [1] 256 256   3
plot(magick::image_read(arr))

Lub batch nws tus kheej yuav raug tsim raws li hauv qab no:

res <- r_process_json_vector(tmp_data[1:4, drawing], scale = 0.5)
str(res)
 # num [1:4, 1:128, 1:128, 1:3] 1 1 1 1 1 1 1 1 1 1 ...
 # - attr(*, "dimnames")=List of 4
 #  ..$ : NULL
 #  ..$ : NULL
 #  ..$ : NULL
 #  ..$ : NULL

Qhov kev siv no zoo li tsis zoo rau peb, txij li kev tsim cov khoom loj loj yuav siv sijhawm ntev, thiab peb tau txiav txim siab coj kom zoo dua ntawm peb cov npoj yaig los ntawm kev siv lub tsev qiv ntawv muaj zog. OpenCV. Thaum lub sijhawm ntawd tsis muaj pob khoom npaj rau R (tsis muaj tam sim no), yog li kev siv tsawg kawg ntawm cov haujlwm yuav tsum tau sau hauv C ++ nrog kev koom ua ke rau R code siv Rcpp.

Txhawm rau daws qhov teeb meem, cov pob khoom thiab cov tsev qiv ntawv hauv qab no tau siv:

OpenCV rau kev ua haujlwm nrog cov duab thiab kos kab. Siv pre-installed system qiv thiab header cov ntaub ntawv, nrog rau kev sib txuas dynamic.
xtensor rau kev ua hauj lwm nrog multidimensional arrays thiab tensors. Peb siv cov ntaub ntawv header suav nrog R pob ntawm tib lub npe. Lub tsev qiv ntawv tso cai rau koj los ua haujlwm nrog ntau qhov arrays, ob qho tib si hauv kab loj thiab kab loj.
ndjson rau parsing JSON. Lub tsev qiv ntawv no yog siv hauv xtensor cia li yog tias nws nyob hauv qhov project.
RcppThread rau kev teeb tsa ntau txoj xov ua haujlwm ntawm vector los ntawm JSON. Siv cov ntaub ntawv header muab los ntawm pob ntawv no. Los ntawm nrov dua RcppParallel Lub pob, nrog rau lwm yam, muaj ib lub voj voog ua haujlwm cuam tshuam.

Nws yog tsim nyog sau cia tias xtensor muab los ua ib tug godsend: ntxiv rau qhov tseeb tias nws muaj kev ua haujlwm dav thiab ua haujlwm siab, nws cov neeg tsim khoom tau ua kom teb tau zoo thiab teb cov lus nug tam sim thiab nthuav dav. Nrog lawv cov kev pab, nws muaj peev xwm los siv cov kev hloov pauv ntawm OpenCV matrices rau hauv xtensor tensors, nrog rau txoj hauv kev los ua ke 3-dimensional duab tensors rau hauv 4-dimensional tensor ntawm qhov tseeb dimension (batch nws tus kheej).

Cov ntaub ntawv rau kev kawm Rcpp, xtensor thiab RcppThread

https://thecoatlessprofessor.com/programming/unofficial-rcpp-api-documentation

https://docs.opencv.org/4.0.1/d7/dbd/group__imgproc.html

https://xtensor.readthedocs.io/en/latest/

https://xtensor.readthedocs.io/en/latest/file_loading.html#loading-json-data-into-xtensor

https://cran.r-project.org/web/packages/RcppThread/vignettes/RcppThread-vignette.pdf

Txhawm rau sau cov ntaub ntawv uas siv cov ntaub ntawv kaw lus thiab kev sib txuas txuas nrog cov tsev qiv ntawv tau teeb tsa hauv lub kaw lus, peb siv lub plugin ua haujlwm hauv pob. Rcpp. Txhawm rau nrhiav txoj hauv kev thiab tus chij, peb tau siv cov khoom siv Linux nrov pkg-teeb tsa.

Kev nqis tes ua ntawm Rcpp plugin rau kev siv lub tsev qiv ntawv OpenCV

Rcpp::registerPlugin("opencv", function() {
  # Возможные названия пакета
  pkg_config_name <- c("opencv", "opencv4")
  # Бинарный файл утилиты pkg-config
  pkg_config_bin <- Sys.which("pkg-config")
  # Проврека наличия утилиты в системе
  checkmate::assert_file_exists(pkg_config_bin, access = "x")
  # Проверка наличия файла настроек OpenCV для pkg-config
  check <- sapply(pkg_config_name, 
                  function(pkg) system(paste(pkg_config_bin, pkg)))
  if (all(check != 0)) {
    stop("OpenCV config for the pkg-config not found", call. = FALSE)
  }

  pkg_config_name <- pkg_config_name[check == 0]
  list(env = list(
    PKG_CXXFLAGS = system(paste(pkg_config_bin, "--cflags", pkg_config_name), 
                          intern = TRUE),
    PKG_LIBS = system(paste(pkg_config_bin, "--libs", pkg_config_name), 
                      intern = TRUE)
  ))
})

Raws li qhov txiaj ntsig ntawm kev ua haujlwm ntawm plugin, cov txiaj ntsig hauv qab no yuav raug hloov pauv thaum lub sijhawm muab tso ua ke:

Rcpp:::.plugins$opencv()$env

# $PKG_CXXFLAGS
# [1] "-I/usr/include/opencv"
#
# $PKG_LIBS
# [1] "-lopencv_shape -lopencv_stitching -lopencv_superres -lopencv_videostab -lopencv_aruco -lopencv_bgsegm -lopencv_bioinspired -lopencv_ccalib -lopencv_datasets -lopencv_dpm -lopencv_face -lopencv_freetype -lopencv_fuzzy -lopencv_hdf -lopencv_line_descriptor -lopencv_optflow -lopencv_video -lopencv_plot -lopencv_reg -lopencv_saliency -lopencv_stereo -lopencv_structured_light -lopencv_phase_unwrapping -lopencv_rgbd -lopencv_viz -lopencv_surface_matching -lopencv_text -lopencv_ximgproc -lopencv_calib3d -lopencv_features2d -lopencv_flann -lopencv_xobjdetect -lopencv_objdetect -lopencv_ml -lopencv_xphoto -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_photo -lopencv_imgproc -lopencv_core"

Txoj cai siv rau parsing JSON thiab tsim ib batch rau kis mus rau tus qauv yog muab nyob rau hauv lub spoiler. Ua ntej, ntxiv ib qhov project directory hauv zos los nrhiav cov ntaub ntawv header (xav tau rau ndjson):

Sys.setenv("PKG_CXXFLAGS" = paste0("-I", normalizePath(file.path("src"))))

Kev siv ntawm JSON rau tensor conversion hauv C ++

// [[Rcpp::plugins(cpp14)]]
// [[Rcpp::plugins(opencv)]]
// [[Rcpp::depends(xtensor)]]
// [[Rcpp::depends(RcppThread)]]

#include <xtensor/xjson.hpp>
#include <xtensor/xadapt.hpp>
#include <xtensor/xview.hpp>
#include <xtensor-r/rtensor.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <Rcpp.h>
#include <RcppThread.h>

// Синонимы для типов
using RcppThread::parallelFor;
using json = nlohmann::json;
using points = xt::xtensor<double,2>;     // Извлечённые из JSON координаты точек
using strokes = std::vector<points>;      // Извлечённые из JSON координаты точек
using xtensor3d = xt::xtensor<double, 3>; // Тензор для хранения матрицы изоображения
using xtensor4d = xt::xtensor<double, 4>; // Тензор для хранения множества изображений
using rtensor3d = xt::rtensor<double, 3>; // Обёртка для экспорта в R
using rtensor4d = xt::rtensor<double, 4>; // Обёртка для экспорта в R

// Статические константы
// Размер изображения в пикселях
const static int SIZE = 256;
// Тип линии
// См. https://en.wikipedia.org/wiki/Pixel_connectivity#2-dimensional
const static int LINE_TYPE = cv::LINE_4;
// Толщина линии в пикселях
const static int LINE_WIDTH = 3;
// Алгоритм ресайза
// https://docs.opencv.org/3.1.0/da/d54/group__imgproc__transform.html#ga5bb5a1fea74ea38e1a5445ca803ff121
const static int RESIZE_TYPE = cv::INTER_LINEAR;

// Шаблон для конвертирования OpenCV-матрицы в тензор
template <typename T, int NCH, typename XT=xt::xtensor<T,3,xt::layout_type::column_major>>
XT to_xt(const cv::Mat_<cv::Vec<T, NCH>>& src) {
  // Размерность целевого тензора
  std::vector<int> shape = {src.rows, src.cols, NCH};
  // Общее количество элементов в массиве
  size_t size = src.total() * NCH;
  // Преобразование cv::Mat в xt::xtensor
  XT res = xt::adapt((T*) src.data, size, xt::no_ownership(), shape);
  return res;
}

// Преобразование JSON в список координат точек
strokes parse_json(const std::string& x) {
  auto j = json::parse(x);
  // Результат парсинга должен быть массивом
  if (!j.is_array()) {
    throw std::runtime_error("'x' must be JSON array.");
  }
  strokes res;
  res.reserve(j.size());
  for (const auto& a: j) {
    // Каждый элемент массива должен быть 2-мерным массивом
    if (!a.is_array() || a.size() != 2) {
      throw std::runtime_error("'x' must include only 2d arrays.");
    }
    // Извлечение вектора точек
    auto p = a.get<points>();
    res.push_back(p);
  }
  return res;
}

// Отрисовка линий
// Цвета HSV
cv::Mat ocv_draw_lines(const strokes& x, bool color = true) {
  // Исходный тип матрицы
  auto stype = color ? CV_8UC3 : CV_8UC1;
  // Итоговый тип матрицы
  auto dtype = color ? CV_32FC3 : CV_32FC1;
  auto bg = color ? cv::Scalar(0, 0, 255) : cv::Scalar(255);
  auto col = color ? cv::Scalar(0, 255, 220) : cv::Scalar(0);
  cv::Mat img = cv::Mat(SIZE, SIZE, stype, bg);
  // Количество линий
  size_t n = x.size();
  for (const auto& s: x) {
    // Количество точек в линии
    size_t n_points = s.shape()[1];
    for (size_t i = 0; i < n_points - 1; ++i) {
      // Точка начала штриха
      cv::Point from(s(0, i), s(1, i));
      // Точка окончания штриха
      cv::Point to(s(0, i + 1), s(1, i + 1));
      // Отрисовка линии
      cv::line(img, from, to, col, LINE_WIDTH, LINE_TYPE);
    }
    if (color) {
      // Меняем цвет линии
      col[0] += 180 / n;
    }
  }
  if (color) {
    // Меняем цветовое представление на RGB
    cv::cvtColor(img, img, cv::COLOR_HSV2RGB);
  }
  // Меняем формат представления на float32 с диапазоном [0, 1]
  img.convertTo(img, dtype, 1 / 255.0);
  return img;
}

// Обработка JSON и получение тензора с данными изображения
xtensor3d process(const std::string& x, double scale = 1.0, bool color = true) {
  auto p = parse_json(x);
  auto img = ocv_draw_lines(p, color);
  if (scale != 1) {
    cv::Mat out;
    cv::resize(img, out, cv::Size(), scale, scale, RESIZE_TYPE);
    cv::swap(img, out);
    out.release();
  }
  xtensor3d arr = color ? to_xt<double,3>(img) : to_xt<double,1>(img);
  return arr;
}

// [[Rcpp::export]]
rtensor3d cpp_process_json_str(const std::string& x, 
                               double scale = 1.0, 
                               bool color = true) {
  xtensor3d res = process(x, scale, color);
  return res;
}

// [[Rcpp::export]]
rtensor4d cpp_process_json_vector(const std::vector<std::string>& x, 
                                  double scale = 1.0, 
                                  bool color = false) {
  size_t n = x.size();
  size_t dim = floor(SIZE * scale);
  size_t channels = color ? 3 : 1;
  xtensor4d res({n, dim, dim, channels});
  parallelFor(0, n, [&x, &res, scale, color](int i) {
    xtensor3d tmp = process(x[i], scale, color);
    auto view = xt::view(res, i, xt::all(), xt::all(), xt::all());
    view = tmp;
  });
  return res;
}

Cov cai no yuav tsum muab tso rau hauv cov ntaub ntawv src/cv_xt.cpp thiab suav nrog cov lus txib Rcpp::sourceCpp(file = "src/cv_xt.cpp", env = .GlobalEnv); kuj yuav tsum tau ua haujlwm nlohmann/json.hpp los ntawm chaw cia khoom. Cov cai tau muab faib ua ntau yam haujlwm:

to_xt - templated muaj nuj nqi rau transforming ib tug duab matrix (cv::Mat) rau tensor xt::xtensor;
parse_json - cov haujlwm parses ib txoj hlua JSON, rho tawm cov kev sib koom ua ke ntawm cov ntsiab lus, ntim lawv rau hauv vector;
ocv_draw_lines - los ntawm cov txiaj ntsig vector ntawm cov ntsiab lus, kos cov kab ntau xim;
process - sib txuas cov haujlwm saum toj no thiab tseem ntxiv lub peev xwm los ntsuas cov duab tshwm sim;
cpp_process_json_str - wrapper tshaj qhov ua haujlwm process, uas exports qhov tshwm sim rau ib qho R-khoom (multidimensional array);
cpp_process_json_vector - wrapper tshaj qhov ua haujlwm cpp_process_json_str, uas tso cai rau koj ua cov hlua vector hauv ntau txoj xov.

Txhawm rau kos cov kab ntau xim, HSV xim qauv tau siv, ua raws li kev hloov pauv mus rau RGB. Cia peb sim qhov tshwm sim:

arr <- cpp_process_json_str(tmp_data[4, drawing])
dim(arr)
# [1] 256 256   3
plot(magick::image_read(arr))

Kev sib piv ntawm qhov ceev ntawm kev siv hauv R thiab C ++

res_bench <- bench::mark(
  r_process_json_str(tmp_data[4, drawing], scale = 0.5),
  cpp_process_json_str(tmp_data[4, drawing], scale = 0.5),
  check = FALSE,
  min_iterations = 100
)
# Параметры бенчмарка
cols <- c("expression", "min", "median", "max", "itr/sec", "total_time", "n_itr")
res_bench[, cols]

#   expression                min     median       max `itr/sec` total_time  n_itr
#   <chr>                <bch:tm>   <bch:tm>  <bch:tm>     <dbl>   <bch:tm>  <int>
# 1 r_process_json_str     3.49ms     3.55ms    4.47ms      273.      490ms    134
# 2 cpp_process_json_str   1.94ms     2.02ms    5.32ms      489.      497ms    243

library(ggplot2)
# Проведение замера
res_bench <- bench::press(
  batch_size = 2^(4:10),
  {
    .data <- tmp_data[sample(seq_len(.N), batch_size), drawing]
    bench::mark(
      r_process_json_vector(.data, scale = 0.5),
      cpp_process_json_vector(.data,  scale = 0.5),
      min_iterations = 50,
      check = FALSE
    )
  }
)

res_bench[, cols]

#    expression   batch_size      min   median      max `itr/sec` total_time n_itr
#    <chr>             <dbl> <bch:tm> <bch:tm> <bch:tm>     <dbl>   <bch:tm> <int>
#  1 r                   16   50.61ms  53.34ms  54.82ms    19.1     471.13ms     9
#  2 cpp                 16    4.46ms   5.39ms   7.78ms   192.      474.09ms    91
#  3 r                   32   105.7ms 109.74ms 212.26ms     7.69        6.5s    50
#  4 cpp                 32    7.76ms  10.97ms  15.23ms    95.6     522.78ms    50
#  5 r                   64  211.41ms 226.18ms 332.65ms     3.85      12.99s    50
#  6 cpp                 64   25.09ms  27.34ms  32.04ms    36.0        1.39s    50
#  7 r                  128   534.5ms 627.92ms 659.08ms     1.61      31.03s    50
#  8 cpp                128   56.37ms  58.46ms  66.03ms    16.9        2.95s    50
#  9 r                  256     1.15s    1.18s    1.29s     0.851     58.78s    50
# 10 cpp                256  114.97ms 117.39ms 130.09ms     8.45       5.92s    50
# 11 r                  512     2.09s    2.15s    2.32s     0.463       1.8m    50
# 12 cpp                512  230.81ms  235.6ms 261.99ms     4.18      11.97s    50
# 13 r                 1024        4s    4.22s     4.4s     0.238       3.5m    50
# 14 cpp               1024  410.48ms 431.43ms 462.44ms     2.33      21.45s    50

ggplot(res_bench, aes(x = factor(batch_size), y = median, 
                      group =  expression, color = expression)) +
  geom_point() +
  geom_line() +
  ylab("median time, s") +
  theme_minimal() +
  scale_color_discrete(name = "", labels = c("cpp", "r")) +
  theme(legend.position = "bottom")

Raws li koj tuaj yeem pom, qhov nce nrawm tau dhau los ua qhov tseem ceeb, thiab nws tsis tuaj yeem ntes nrog C ++ code los ntawm kev sib piv R code.

3. Iterators rau unloading batch los ntawm lub database

R muaj lub koob npe zoo tsim nyog rau kev ua cov ntaub ntawv uas haum rau hauv RAM, thaum Python yog qhov muaj txiaj ntsig ntau dua los ntawm kev ua cov ntaub ntawv rov ua dua, tso cai rau koj kom yooj yim thiab ua raws li cov kev suav ua tawm (xws li siv lub cim xeeb sab nraud). Ib qho piv txwv classic thiab tseem ceeb rau peb hauv cov ntsiab lus ntawm qhov teeb meem tau piav qhia yog qhov sib sib zog nqus neural tes hauj lwm uas tau kawm los ntawm gradient qhovntsej thiaj tsis mob nrog kev kwv yees ntawm gradient ntawm txhua kauj ruam siv ib feem me me ntawm kev soj ntsuam, lossis mini-batch.

Cov kev kawm sib sib zog nqus sau hauv Python muaj cov chav kawm tshwj xeeb uas siv iterators raws li cov ntaub ntawv: rooj, duab hauv folders, binary formats, thiab lwm yam. Koj tuaj yeem siv cov kev xaiv npaj txhij lossis sau koj tus kheej rau cov haujlwm tshwj xeeb. Hauv R peb tuaj yeem ua kom zoo dua ntawm tag nrho cov yam ntxwv ntawm lub tsev qiv ntawv Python keras nrog nws ntau backends siv lub pob ntawm tib lub npe, uas nyob rau hauv lem ua hauj lwm nyob rau sab saum toj ntawm lub pob reticulate. Qhov kawg tsim nyog ib tsab xov xwm ntev ntev; nws tsis tsuas yog tso cai rau koj khiav Python code los ntawm R, tab sis kuj tso cai rau koj hloov khoom nruab nrab ntawm R thiab Python zaug, ua tiav txhua hom kev hloov pauv tsim nyog.

Peb tau tshem ntawm qhov yuav tsum tau khaws tag nrho cov ntaub ntawv hauv RAM los ntawm kev siv MonetDBLite, tag nrho cov haujlwm "neural network" yuav ua los ntawm cov cai qub hauv Python, peb tsuas yog yuav tsum tau sau ib qho iteration dua cov ntaub ntawv, txij li tsis muaj dab tsi npaj txhij. rau qhov xwm txheej zoo li no hauv R lossis Python. Muaj qhov tseem ceeb tsuas yog ob qhov yuav tsum tau ua rau nws: nws yuav tsum rov qab cov khoom siv hauv lub voj voog tsis kawg thiab txuag nws lub xeev ntawm kev rov ua dua (qhov tom kawg hauv R tau ua raws li qhov yooj yim tshaj plaws siv kaw). Yav dhau los, nws yuav tsum tau ua kom meej meej hloov R arrays rau hauv numpy arrays hauv iterator, tab sis tam sim no version ntawm pob keras nws nws tus kheej.

Tus iterator rau kev cob qhia thiab validation cov ntaub ntawv muab tau raws li nram no:

Iterator rau kev cob qhia thiab cov ntaub ntawv siv tau

train_generator <- function(db_connection = con,
                            samples_index,
                            num_classes = 340,
                            batch_size = 32,
                            scale = 1,
                            color = FALSE,
                            imagenet_preproc = FALSE) {
  # Проверка аргументов
  checkmate::assert_class(con, "DBIConnection")
  checkmate::assert_integerish(samples_index)
  checkmate::assert_count(num_classes)
  checkmate::assert_count(batch_size)
  checkmate::assert_number(scale, lower = 0.001, upper = 5)
  checkmate::assert_flag(color)
  checkmate::assert_flag(imagenet_preproc)

  # Перемешиваем, чтобы брать и удалять использованные индексы батчей по порядку
  dt <- data.table::data.table(id = sample(samples_index))
  # Проставляем номера батчей
  dt[, batch := (.I - 1L) %/% batch_size + 1L]
  # Оставляем только полные батчи и индексируем
  dt <- dt[, if (.N == batch_size) .SD, keyby = batch]
  # Устанавливаем счётчик
  i <- 1
  # Количество батчей
  max_i <- dt[, max(batch)]

  # Подготовка выражения для выгрузки
  sql <- sprintf(
    "PREPARE SELECT drawing, label_int FROM doodles WHERE id IN (%s)",
    paste(rep("?", batch_size), collapse = ",")
  )
  res <- DBI::dbSendQuery(con, sql)

  # Аналог keras::to_categorical
  to_categorical <- function(x, num) {
    n <- length(x)
    m <- numeric(n * num)
    m[x * n + seq_len(n)] <- 1
    dim(m) <- c(n, num)
    return(m)
  }

  # Замыкание
  function() {
    # Начинаем новую эпоху
    if (i > max_i) {
      dt[, id := sample(id)]
      data.table::setkey(dt, batch)
      # Сбрасываем счётчик
      i <<- 1
      max_i <<- dt[, max(batch)]
    }

    # ID для выгрузки данных
    batch_ind <- dt[batch == i, id]
    # Выгрузка данных
    batch <- DBI::dbFetch(DBI::dbBind(res, as.list(batch_ind)), n = -1)

    # Увеличиваем счётчик
    i <<- i + 1

    # Парсинг JSON и подготовка массива
    batch_x <- cpp_process_json_vector(batch$drawing, scale = scale, color = color)
    if (imagenet_preproc) {
      # Шкалирование c интервала [0, 1] на интервал [-1, 1]
      batch_x <- (batch_x - 0.5) * 2
    }

    batch_y <- to_categorical(batch$label_int, num_classes)
    result <- list(batch_x, batch_y)
    return(result)
  }
}

Kev ua haujlwm yuav siv sij hawm raws li kev nkag mus rau qhov sib txawv nrog kev sib txuas rau cov ntaub ntawv, cov lej ntawm cov kab siv, tus naj npawb ntawm cov chav kawm, batch loj, nplai (scale = 1 sib raug rau rendering dluab ntawm 256x256 pixels, scale = 0.5 - 128x128 pixels), xim qhia (color = FALSE qhia txog qhov ua rau greyscale thaum siv color = TRUE txhua tus mob stroke yog kos nyob rau hauv ib tug tshiab xim) thiab ib tug preprocessing qhia rau tes hauj lwm ua ntej-kawm ntawm imagenet. Cov yav tas yog xav tau nyob rau hauv thiaj li yuav scale pixel qhov tseem ceeb ntawm lub ncua sij hawm [0, 1] mus rau lub ncua sij hawm [-1, 1], uas yog siv thaum cob qhia cov khoom. keras qauv.

Lub luag haujlwm sab nraud muaj kev sib cav hom kev kuaj xyuas, lub rooj data.table nrog randomly mix kab zauv los ntawm samples_index thiab batch naj npawb, txee thiab ntau tshaj ntawm batch, nrog rau ib qho kev qhia SQL rau unloading cov ntaub ntawv los ntawm lub database. Tsis tas li ntawd, peb tau txiav txim siab ceev analogue ntawm kev ua haujlwm sab hauv keras::to_categorical(). Peb siv yuav luag tag nrho cov ntaub ntawv rau kev cob qhia, tawm hauv ib nrab ib feem pua ntawm kev siv tau, yog li lub sijhawm me me raug txwv los ntawm qhov ntsuas. steps_per_epoch thaum hu keras::fit_generator(), thiab tus mob if (i > max_i) tsuas yog ua haujlwm rau validation iterator.

Hauv kev ua haujlwm sab hauv, kab indexes tau muab rov qab rau tom ntej batch, cov ntaub ntawv raug tshem tawm los ntawm cov ntaub ntawv nrog rau cov khoom txee nce, JSON parsing (ua haujlwm cpp_process_json_vector(), sau hauv C ++) thiab tsim cov arrays sib xws rau cov duab. Tom qab ntawd ib-kub vectors nrog cov ntawv sau hauv chav kawm yog tsim, arrays nrog pixel qhov tseem ceeb thiab cov ntawv sau ua ke rau hauv ib daim ntawv teev npe, uas yog tus nqi rov qab. Txhawm rau kom ua haujlwm nrawm, peb siv qhov tsim cov indexes hauv cov ntxhuav data.table thiab hloov kho ntawm qhov txuas - tsis muaj pob "chips" cov ntaub ntawv Nws yog qhov nyuaj heev los xav txog kev ua haujlwm zoo nrog ib qho tseem ceeb ntawm cov ntaub ntawv hauv R.

Cov txiaj ntsig ntawm kev ntsuas ceev ntawm Core i5 laptop yog raws li hauv qab no:

Iterator ntsuas

library(Rcpp)
library(keras)
library(ggplot2)

source("utils/rcpp.R")
source("utils/keras_iterator.R")

con <- DBI::dbConnect(drv = MonetDBLite::MonetDBLite(), Sys.getenv("DBDIR"))

ind <- seq_len(DBI::dbGetQuery(con, "SELECT count(*) FROM doodles")[[1L]])
num_classes <- DBI::dbGetQuery(con, "SELECT max(label_int) + 1 FROM doodles")[[1L]]

# Индексы для обучающей выборки
train_ind <- sample(ind, floor(length(ind) * 0.995))
# Индексы для проверочной выборки
val_ind <- ind[-train_ind]
rm(ind)
# Коэффициент масштаба
scale <- 0.5

# Проведение замера
res_bench <- bench::press(
  batch_size = 2^(4:10),
  {
    it1 <- train_generator(
      db_connection = con,
      samples_index = train_ind,
      num_classes = num_classes,
      batch_size = batch_size,
      scale = scale
    )
    bench::mark(
      it1(),
      min_iterations = 50L
    )
  }
)
# Параметры бенчмарка
cols <- c("batch_size", "min", "median", "max", "itr/sec", "total_time", "n_itr")
res_bench[, cols]

#   batch_size      min   median      max `itr/sec` total_time n_itr
#        <dbl> <bch:tm> <bch:tm> <bch:tm>     <dbl>   <bch:tm> <int>
# 1         16     25ms  64.36ms   92.2ms     15.9       3.09s    49
# 2         32   48.4ms 118.13ms 197.24ms     8.17       5.88s    48
# 3         64   69.3ms 117.93ms 181.14ms     8.57       5.83s    50
# 4        128  157.2ms 240.74ms 503.87ms     3.85      12.71s    49
# 5        256  359.3ms 613.52ms 988.73ms     1.54       30.5s    47
# 6        512  884.7ms    1.53s    2.07s     0.674      1.11m    45
# 7       1024     2.7s    3.83s    5.47s     0.261      2.81m    44

ggplot(res_bench, aes(x = factor(batch_size), y = median, group = 1)) +
    geom_point() +
    geom_line() +
    ylab("median time, s") +
    theme_minimal()

DBI::dbDisconnect(con, shutdown = TRUE)

Yog tias koj muaj RAM txaus, koj tuaj yeem ua kom nrawm nrawm ntawm kev ua haujlwm ntawm cov ntaub ntawv los ntawm kev xa mus rau tib lub RAM no (32 GB yog txaus rau peb txoj haujlwm). Hauv Linux, qhov muab faib yog mounted los ntawm lub neej ntawd /dev/shm, nyob mus txog ib nrab ntawm lub peev xwm RAM. Koj tuaj yeem qhia ntau ntxiv los ntawm kev kho /etc/fstabkom tau ib daim ntawv zoo li tmpfs /dev/shm tmpfs defaults,size=25g 0 0. Nco ntsoov reboot thiab xyuas qhov tshwm sim los ntawm kev khiav cov lus txib df -h.

Lub iterator rau kev xeem cov ntaub ntawv zoo li yooj yim npaum li cas, txij li thaum lub xeem dataset haum nkaus rau hauv RAM:

Iterator rau cov ntaub ntawv xeem

test_generator <- function(dt,
                           batch_size = 32,
                           scale = 1,
                           color = FALSE,
                           imagenet_preproc = FALSE) {

  # Проверка аргументов
  checkmate::assert_data_table(dt)
  checkmate::assert_count(batch_size)
  checkmate::assert_number(scale, lower = 0.001, upper = 5)
  checkmate::assert_flag(color)
  checkmate::assert_flag(imagenet_preproc)

  # Проставляем номера батчей
  dt[, batch := (.I - 1L) %/% batch_size + 1L]
  data.table::setkey(dt, batch)
  i <- 1
  max_i <- dt[, max(batch)]

  # Замыкание
  function() {
    batch_x <- cpp_process_json_vector(dt[batch == i, drawing], 
                                       scale = scale, color = color)
    if (imagenet_preproc) {
      # Шкалирование c интервала [0, 1] на интервал [-1, 1]
      batch_x <- (batch_x - 0.5) * 2
    }
    result <- list(batch_x)
    i <<- i + 1
    return(result)
  }
}

4. Kev xaiv cov qauv tsim qauv

Thawj architecture siv yog mobilenet v1, cov yam ntxwv ntawm uas tau tham hauv qhov no lus. Nws suav nrog raws li tus qauv keras thiab, raws li, muaj nyob rau hauv lub pob ntawm tib lub npe rau R. Tab sis thaum sim siv nws nrog ib leeg-channel dluab, ib tug coj txawv txawv tshwm sim: lub input tensor yuav tsum yeej ib txwm muaj lub dimension. (batch, height, width, 3), uas yog, tus naj npawb ntawm cov channel hloov tsis tau. Tsis muaj kev txwv zoo li no hauv Python, yog li peb maj nrawm thiab sau peb tus kheej kev siv ntawm cov qauv no, ua raws li tsab xov xwm qub (tsis muaj qhov tso tawm uas nyob hauv keras version):

Mobilenet v1 architecture

library(keras)

top_3_categorical_accuracy <- custom_metric(
    name = "top_3_categorical_accuracy",
    metric_fn = function(y_true, y_pred) {
         metric_top_k_categorical_accuracy(y_true, y_pred, k = 3)
    }
)

layer_sep_conv_bn <- function(object, 
                              filters,
                              alpha = 1,
                              depth_multiplier = 1,
                              strides = c(2, 2)) {

  # NB! depth_multiplier !=  resolution multiplier
  # https://github.com/keras-team/keras/issues/10349

  layer_depthwise_conv_2d(
    object = object,
    kernel_size = c(3, 3), 
    strides = strides,
    padding = "same",
    depth_multiplier = depth_multiplier
  ) %>%
  layer_batch_normalization() %>% 
  layer_activation_relu() %>%
  layer_conv_2d(
    filters = filters * alpha,
    kernel_size = c(1, 1), 
    strides = c(1, 1)
  ) %>%
  layer_batch_normalization() %>% 
  layer_activation_relu() 
}

get_mobilenet_v1 <- function(input_shape = c(224, 224, 1),
                             num_classes = 340,
                             alpha = 1,
                             depth_multiplier = 1,
                             optimizer = optimizer_adam(lr = 0.002),
                             loss = "categorical_crossentropy",
                             metrics = c("categorical_crossentropy",
                                         top_3_categorical_accuracy)) {

  inputs <- layer_input(shape = input_shape)

  outputs <- inputs %>%
    layer_conv_2d(filters = 32, kernel_size = c(3, 3), strides = c(2, 2), padding = "same") %>%
    layer_batch_normalization() %>% 
    layer_activation_relu() %>%
    layer_sep_conv_bn(filters = 64, strides = c(1, 1)) %>%
    layer_sep_conv_bn(filters = 128, strides = c(2, 2)) %>%
    layer_sep_conv_bn(filters = 128, strides = c(1, 1)) %>%
    layer_sep_conv_bn(filters = 256, strides = c(2, 2)) %>%
    layer_sep_conv_bn(filters = 256, strides = c(1, 1)) %>%
    layer_sep_conv_bn(filters = 512, strides = c(2, 2)) %>%
    layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
    layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
    layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
    layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
    layer_sep_conv_bn(filters = 512, strides = c(1, 1)) %>%
    layer_sep_conv_bn(filters = 1024, strides = c(2, 2)) %>%
    layer_sep_conv_bn(filters = 1024, strides = c(1, 1)) %>%
    layer_global_average_pooling_2d() %>%
    layer_dense(units = num_classes) %>%
    layer_activation_softmax()

    model <- keras_model(
      inputs = inputs,
      outputs = outputs
    )

    model %>% compile(
      optimizer = optimizer,
      loss = loss,
      metrics = metrics
    )

    return(model)
}

Qhov tsis zoo ntawm txoj kev no yog pom tseeb. Kuv xav sim ntau tus qauv, tab sis ntawm qhov tsis sib xws, kuv tsis xav rov sau dua txhua tus qauv siv manually. Peb kuj tau txais lub sijhawm los siv qhov hnyav ntawm cov qauv ua ntej kev cob qhia ntawm imagenet. Raws li ib txwm muaj, kev kawm cov ntaub ntawv tau pab. Muaj nuj nqi get_config() tso cai rau koj kom tau txais cov lus piav qhia ntawm tus qauv hauv daim ntawv tsim nyog rau kev kho (base_model_conf$layers - cov npe R tsis tu ncua), thiab ua haujlwm from_config() ua qhov rov qab hloov dua siab tshiab rau ib qho qauv khoom:

base_model_conf <- get_config(base_model)
base_model_conf$layers[[1]]$config$batch_input_shape[[4]] <- 1L
base_model <- from_config(base_model_conf)

Tam sim no nws tsis yog qhov nyuaj los sau cov haujlwm thoob ntiaj teb kom tau txais ib qho ntawm cov khoom siv keras cov qauv nrog lossis tsis muaj qhov hnyav tau kawm ntawm imagenet:

Muaj nuj nqi rau thauj khoom npaj ua vaj tse

get_model <- function(name = "mobilenet_v2",
                      input_shape = NULL,
                      weights = "imagenet",
                      pooling = "avg",
                      num_classes = NULL,
                      optimizer = keras::optimizer_adam(lr = 0.002),
                      loss = "categorical_crossentropy",
                      metrics = NULL,
                      color = TRUE,
                      compile = FALSE) {
  # Проверка аргументов
  checkmate::assert_string(name)
  checkmate::assert_integerish(input_shape, lower = 1, upper = 256, len = 3)
  checkmate::assert_count(num_classes)
  checkmate::assert_flag(color)
  checkmate::assert_flag(compile)

  # Получаем объект из пакета keras
  model_fun <- get0(paste0("application_", name), envir = asNamespace("keras"))
  # Проверка наличия объекта в пакете
  if (is.null(model_fun)) {
    stop("Model ", shQuote(name), " not found.", call. = FALSE)
  }

  base_model <- model_fun(
    input_shape = input_shape,
    include_top = FALSE,
    weights = weights,
    pooling = pooling
  )

  # Если изображение не цветное, меняем размерность входа
  if (!color) {
    base_model_conf <- keras::get_config(base_model)
    base_model_conf$layers[[1]]$config$batch_input_shape[[4]] <- 1L
    base_model <- keras::from_config(base_model_conf)
  }

  predictions <- keras::get_layer(base_model, "global_average_pooling2d_1")$output
  predictions <- keras::layer_dense(predictions, units = num_classes, activation = "softmax")
  model <- keras::keras_model(
    inputs = base_model$input,
    outputs = predictions
  )

  if (compile) {
    keras::compile(
      object = model,
      optimizer = optimizer,
      loss = loss,
      metrics = metrics
    )
  }

  return(model)
}

Thaum siv cov duab ib-channel, tsis muaj qhov hnyav ua ntej siv. Qhov no tuaj yeem kho tau: siv cov haujlwm get_weights() tau cov qauv luj nyob rau hauv daim ntawv ntawm ib daim ntawv teev R arrays, hloov lub dimension ntawm thawj lub caij ntawm daim ntawv teev npe no (los ntawm kev noj ib tug xim channel los yog nruab nrab tag nrho peb), thiab ces thauj cov luj rov qab mus rau hauv tus qauv nrog lub functionality. set_weights(). Peb yeej tsis tau ntxiv qhov kev ua haujlwm no, vim tias nyob rau theem no nws twb paub meej tias nws muaj txiaj ntsig ntau dua los ua haujlwm nrog cov duab xim.

Peb tau ua tiav feem ntau ntawm cov kev sim siv mobilenet versions 1 thiab 2, nrog rau resnet34. Ntau cov qauv niaj hnub xws li SE-ResNeXt ua tau zoo hauv qhov kev sib tw no. Hmoov tsis zoo, peb tsis muaj kev npaj ua tiav ntawm peb qhov pov tseg, thiab peb tsis tau sau peb tus kheej (tab sis peb yuav twv yuav raug sau).

5. Parameterization ntawm scripts

Rau kev yooj yim, tag nrho cov cai rau kev pib kawm tau tsim los ua ib tsab ntawv, parameterized siv docopt ua raws li nram no:

doc <- '
Usage:
  train_nn.R --help
  train_nn.R --list-models
  train_nn.R [options]

Options:
  -h --help                   Show this message.
  -l --list-models            List available models.
  -m --model=<model>          Neural network model name [default: mobilenet_v2].
  -b --batch-size=<size>      Batch size [default: 32].
  -s --scale-factor=<ratio>   Scale factor [default: 0.5].
  -c --color                  Use color lines [default: FALSE].
  -d --db-dir=<path>          Path to database directory [default: Sys.getenv("db_dir")].
  -r --validate-ratio=<ratio> Validate sample ratio [default: 0.995].
  -n --n-gpu=<number>         Number of GPUs [default: 1].
'
args <- docopt::docopt(doc)

Pob docopt ua sawv cev rau kev siv http://docopt.org/ rau R. Nrog nws cov kev pab, cov ntawv sau tau pib nrog cov lus txib yooj yim xws li Rscript bin/train_nn.R -m resnet50 -c -d /home/andrey/doodle_db los yog ./bin/train_nn.R -m resnet50 -c -d /home/andrey/doodle_db, yog file train_nn.R yog executable (qhov lus txib no yuav pib kawm tus qauv resnet50 ntawm peb-xim dluab ntsuas 128x128 pixels, cov ntaub ntawv yuav tsum nyob hauv daim nplaub tshev /home/andrey/doodle_db). Koj tuaj yeem ntxiv kev kawm nrawm, hom optimizer, thiab lwm yam kev hloov kho tsis tau rau hauv daim ntawv. Nyob rau hauv tus txheej txheem ntawm kev npaj cov ntawv tshaj tawm, nws muab tawm hais tias lub architecture mobilenet_v2 los ntawm qhov tam sim no version keras hauv R siv tsis tau vim qhov kev hloov pauv tsis tau coj mus rau hauv tus account hauv R pob, peb tos kom lawv kho nws.

Txoj hauv kev no ua rau nws muaj peev xwm ua kom nrawm nrawm nrog cov qauv sib txawv piv rau kev tshaj tawm cov ntawv tshaj tawm hauv RStudio (peb nco ntsoov lub pob yog lwm txoj hauv kev. tfruns ua). Tab sis lub ntsiab kom zoo dua yog lub peev xwm los tswj tau yooj yim tshaj tawm cov ntawv sau hauv Docker lossis tsuas yog ntawm tus neeg rau zaub mov, tsis tas yuav txhim kho RStudio rau qhov no.

6. Dockerization ntawm cov ntawv sau

Peb siv Docker los xyuas kom meej qhov kev txav mus los ntawm ib puag ncig rau kev cob qhia cov qauv ntawm cov neeg koom tes thiab rau kev xa mus sai hauv huab. Koj tuaj yeem pib paub txog cov cuab yeej no, uas yog qhov txawv txav rau R programmer, nrog qhov no series ntawm kev tshaj tawm los yog video kawm.

Docker tso cai rau koj los tsim koj tus kheej cov duab los ntawm kos thiab siv lwm cov duab ua lub hauv paus tsim koj tus kheej. Thaum txheeb xyuas cov kev xaiv muaj, peb tuaj txog qhov xaus tias kev txhim kho NVIDIA, CUDA + cuDNN tsav tsheb thiab cov tsev qiv ntawv Python yog ib feem ntawm cov duab zoo nkauj, thiab peb txiav txim siab coj cov duab los ua lub hauv paus. tensorflow/tensorflow:1.12.0-gpu, ntxiv cov pob khoom tsim nyog R muaj.

Cov ntaub ntawv docker zaum kawg zoo li no:

Dockerfile

FROM tensorflow/tensorflow:1.12.0-gpu

MAINTAINER Artem Klevtsov <[email protected]>

SHELL ["/bin/bash", "-c"]

ARG LOCALE="en_US.UTF-8"
ARG APT_PKG="libopencv-dev r-base r-base-dev littler"
ARG R_BIN_PKG="futile.logger checkmate data.table rcpp rapidjsonr dbi keras jsonlite curl digest remotes"
ARG R_SRC_PKG="xtensor RcppThread docopt MonetDBLite"
ARG PY_PIP_PKG="keras"
ARG DIRS="/db /app /app/data /app/models /app/logs"

RUN source /etc/os-release && 
    echo "deb https://cloud.r-project.org/bin/linux/ubuntu ${UBUNTU_CODENAME}-cran35/" > /etc/apt/sources.list.d/cran35.list && 
    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 && 
    add-apt-repository -y ppa:marutter/c2d4u3.5 && 
    add-apt-repository -y ppa:timsc/opencv-3.4 && 
    apt-get update && 
    apt-get install -y locales && 
    locale-gen ${LOCALE} && 
    apt-get install -y --no-install-recommends ${APT_PKG} && 
    ln -s /usr/lib/R/site-library/littler/examples/install.r /usr/local/bin/install.r && 
    ln -s /usr/lib/R/site-library/littler/examples/install2.r /usr/local/bin/install2.r && 
    ln -s /usr/lib/R/site-library/littler/examples/installGithub.r /usr/local/bin/installGithub.r && 
    echo 'options(Ncpus = parallel::detectCores())' >> /etc/R/Rprofile.site && 
    echo 'options(repos = c(CRAN = "https://cloud.r-project.org"))' >> /etc/R/Rprofile.site && 
    apt-get install -y $(printf "r-cran-%s " ${R_BIN_PKG}) && 
    install.r ${R_SRC_PKG} && 
    pip install ${PY_PIP_PKG} && 
    mkdir -p ${DIRS} && 
    chmod 777 ${DIRS} && 
    rm -rf /tmp/downloaded_packages/ /tmp/*.rds && 
    rm -rf /var/lib/apt/lists/*

COPY utils /app/utils
COPY src /app/src
COPY tests /app/tests
COPY bin/*.R /app/

ENV DBDIR="/db"
ENV CUDA_HOME="/usr/local/cuda"
ENV PATH="/app:${PATH}"

WORKDIR /app

VOLUME /db
VOLUME /app

CMD bash

Rau kev yooj yim, cov pob khoom siv tau muab tso rau hauv qhov sib txawv; feem ntau ntawm cov ntawv sau tau muab luam tawm hauv cov thawv ntim thaum sib dhos. Peb kuj tau hloov lub plhaub hais kom ua /bin/bash kom yooj yim ntawm kev siv cov ntsiab lus /etc/os-release. Qhov no zam qhov yuav tsum tau qhia meej OS version hauv cov cai.

Tsis tas li ntawd, ib tsab ntawv me me tau sau tseg uas tso cai rau koj tso lub thawv nrog ntau cov lus txib. Piv txwv li, cov no tuaj yeem yog cov ntawv sau rau kev cob qhia neural tes hauj lwm uas yav tas los muab tso rau hauv lub thawv, lossis lub plhaub hais kom debugging thiab saib xyuas kev ua haujlwm ntawm lub thawv:

Tsab ntawv los tso lub thawv

#!/bin/sh

DBDIR=${PWD}/db
LOGSDIR=${PWD}/logs
MODELDIR=${PWD}/models
DATADIR=${PWD}/data
ARGS="--runtime=nvidia --rm -v ${DBDIR}:/db -v ${LOGSDIR}:/app/logs -v ${MODELDIR}:/app/models -v ${DATADIR}:/app/data"

if [ -z "$1" ]; then
    CMD="Rscript /app/train_nn.R"
elif [ "$1" = "bash" ]; then
    ARGS="${ARGS} -ti"
else
    CMD="Rscript /app/train_nn.R $@"
fi

docker run ${ARGS} doodles-tf ${CMD}

Yog tias tsab ntawv bash no khiav yam tsis muaj qhov txwv, tsab ntawv yuav raug hu rau hauv lub thawv train_nn.R nrog tus nqi pib; yog tias thawj qhov kev sib cav yog "bash", ces lub thawv yuav pib sib tham nrog lub plhaub hais kom ua. Hauv txhua qhov xwm txheej, cov txiaj ntsig ntawm qhov kev sib cav yog hloov pauv: CMD="Rscript /app/train_nn.R $@".

Nws yog ib qho tsim nyog sau cia tias cov ntawv teev npe nrog cov ntaub ntawv los ntawm cov ntaub ntawv thiab cov ntaub ntawv khaws tseg, nrog rau cov ntawv teev npe rau kev txuag cov qauv kev cob qhia, tau muab tso rau hauv lub thawv los ntawm tus tswv system, uas tso cai rau koj nkag mus rau cov txiaj ntsig ntawm cov ntawv sau yam tsis muaj kev cuam tshuam tsis tsim nyog.

7. Siv ntau GPUs ntawm Google Huab

Ib qho ntawm cov yam ntxwv ntawm kev sib tw yog cov ntaub ntawv nrov heev (saib daim duab npe, qiv los ntawm @Leigh.plt los ntawm ODS slack). Cov khoom loj pab tiv thaiv qhov no, thiab tom qab kev sim ntawm PC nrog 1 GPU, peb tau txiav txim siab los ua tus qauv kev cob qhia ntawm ntau GPUs hauv huab. Siv GoogleCloud (cov lus qhia zoo rau cov hauv paus) vim muaj kev xaiv loj ntawm kev teeb tsa, tus nqi tsim nyog thiab $ 300 ntxiv. Tawm ntawm kev ntshaw, kuv tau xaj 4xV100 piv txwv nrog SSD thiab ib tuj ntawm RAM, thiab qhov ntawd yog qhov yuam kev loj. Xws li lub tshuab noj nyiaj sai sai; koj tuaj yeem mus tsoo sim yam tsis muaj cov kav dej pov thawj. Rau kev kawm, nws yog qhov zoo dua los coj K80. Tab sis qhov loj ntawm RAM tuaj yeem ua ke - huab SSD tsis txaus siab rau nws qhov kev ua tau zoo, yog li cov ntaub ntawv tau pauv mus rau dev/shm.

Qhov kev txaus siab loj tshaj plaws yog cov lej cim lub luag haujlwm rau kev siv ntau GPUs. Ua ntej, tus qauv tsim los ntawm CPU siv tus saib xyuas cov ntsiab lus, ib yam li hauv Python:

with(tensorflow::tf$device("/cpu:0"), {
  model_cpu <- get_model(
    name = model_name,
    input_shape = input_shape,
    weights = weights,
    metrics =(top_3_categorical_accuracy,
    compile = FALSE
  )
})

Tom qab ntawd tus qauv uas tsis tau sau ua ke (qhov no yog qhov tseem ceeb) tau theej rau ib tus lej ntawm GPUs uas muaj, thiab tsuas yog tom qab ntawd nws tau muab tso ua ke:

model <- keras::multi_gpu_model(model_cpu, gpus = n_gpu)
keras::compile(
  object = model,
  optimizer = keras::optimizer_adam(lr = 0.0004),
  loss = "categorical_crossentropy",
  metrics = c(top_3_categorical_accuracy)
)

Cov txheej txheem classic ntawm khov tag nrho cov khaubncaws sab nraud povtseg tsuas yog qhov kawg, kev cob qhia cov txheej txheem kawg, tsis khov thiab rov ua kom tag nrho cov qauv rau ob peb GPUs tsis tuaj yeem siv tau.

Kev cob qhia raug saib xyuas yam tsis siv. tensorboard, txwv peb tus kheej rau sau cov cav thiab txuag cov qauv nrog cov ntaub ntawv npe tom qab txhua lub sijhawm:

Hu rov qab

# Шаблон имени файла лога
log_file_tmpl <- file.path("logs", sprintf(
  "%s_%d_%dch_%s.csv",
  model_name,
  dim_size,
  channels,
  format(Sys.time(), "%Y%m%d%H%M%OS")
))
# Шаблон имени файла модели
model_file_tmpl <- file.path("models", sprintf(
  "%s_%d_%dch_{epoch:02d}_{val_loss:.2f}.h5",
  model_name,
  dim_size,
  channels
))

callbacks_list <- list(
  keras::callback_csv_logger(
    filename = log_file_tmpl
  ),
  keras::callback_early_stopping(
    monitor = "val_loss",
    min_delta = 1e-4,
    patience = 8,
    verbose = 1,
    mode = "min"
  ),
  keras::callback_reduce_lr_on_plateau(
    monitor = "val_loss",
    factor = 0.5, # уменьшаем lr в 2 раза
    patience = 4,
    verbose = 1,
    min_delta = 1e-4,
    mode = "min"
  ),
  keras::callback_model_checkpoint(
    filepath = model_file_tmpl,
    monitor = "val_loss",
    save_best_only = FALSE,
    save_weights_only = FALSE,
    mode = "min"
  )
)

8. Hloov qhov xaus

Ntau yam teeb meem uas peb tau ntsib tseem tsis tau kov yeej:

в keras tsis muaj kev npaj ua haujlwm rau kev tshawb nrhiav qhov kev kawm zoo tshaj plaws (analogue lr_finder hauv tsev qiv ntawv fast.ai); Nrog rau qee qhov kev siv zog, nws muaj peev xwm xa cov kev siv thib peb rau R, piv txwv li, qhov no;
Raws li qhov tshwm sim ntawm cov ntsiab lus dhau los, nws tsis tuaj yeem xaiv qhov kev cob qhia kom raug thaum siv ob peb GPUs;
tsis muaj cov niaj hnub neural network architectures, tshwj xeeb tshaj yog cov uas tau kawm ua ntej ntawm imagenet;
tsis muaj ib lub voj voog txoj cai thiab kev ntxub ntxaug kev kawm (cosine annealing yog ntawm peb qhov kev thov ua raws, Ua tsaug skydan ua).

Dab tsi muaj txiaj ntsig tau kawm los ntawm kev sib tw no:

Ntawm cov cuab yeej siv hluav taws xob tsawg, koj tuaj yeem ua haujlwm nrog kev tsim nyog (ntau zaus ntawm RAM) ntim cov ntaub ntawv yam tsis muaj mob. Lub hnab yas cov ntaub ntawv txuag lub cim xeeb vim qhov hloov kho ntawm cov ntxhuav, uas tsis txhob luam lawv, thiab thaum siv kom raug, nws lub peev xwm yuav luag ib txwm ua kom pom qhov ceev tshaj plaws ntawm txhua yam cuab yeej paub rau peb rau cov lus sau. Txuag cov ntaub ntawv nyob rau hauv ib tug database tso cai rau koj, nyob rau hauv ntau rooj plaub, tsis txhob xav txog tag nrho cov yuav tsum tau nyem tag nrho cov dataset rau RAM.
Kev ua haujlwm qeeb hauv R tuaj yeem hloov nrog ceev ceev hauv C ++ siv pob Rcpp. Yog ntxiv rau siv RcppThread los yog RcppParallel, peb tau txais cross-platform multi-threaded siv, yog li tsis tas yuav tsum tau ua tib zoo sib piv cov cai ntawm qib R.
Pob Rcpp tuaj yeem siv yam tsis muaj kev paub loj ntawm C ++, qhov tsawg kawg nkaus yuav tsum tau piav qhia no. Header cov ntaub ntawv rau ib tug xov tooj ntawm txias C-libraries nyiam xtensor muaj nyob rau ntawm CRAN, uas yog, ib qho kev tsim kho vaj tse tau tsim rau kev siv cov haujlwm uas sib koom ua ke npaj ua haujlwm siab C ++ code rau hauv R. Ntxiv kev yooj yim yog syntax highlighting thiab zoo li qub C ++ code analyzer hauv RStudio.
docopt ua tso cai rau koj los khiav cov ntawv sau tus kheej nrog cov kev txwv. Qhov no yog qhov yooj yim rau kev siv ntawm cov chaw taws teeb server, incl. hauv qab docker. Hauv RStudio, nws tsis yooj yim los ua ntau teev ntawm kev sim nrog kev cob qhia neural tes hauj lwm, thiab txhim kho IDE ntawm tus neeg rau zaub mov nws tus kheej tsis yog ib txwm tsim nyog.
Docker xyuas kom meej cov code portability thiab reproducibility ntawm cov kev tshwm sim ntawm developers nrog txawv versions ntawm lub OS thiab cov tsev qiv ntawv, raws li zoo raws li yooj yim ntawm kev tua ntawm servers. Koj tuaj yeem tso tag nrho cov raj xa dej nrog ib qho lus txib xwb.
Google Cloud yog ib txoj hauv kev siv nyiaj txiag rau kev sim ntawm cov khoom kim heev, tab sis koj yuav tsum xaiv cov kev teeb tsa kom zoo.
Kev ntsuas qhov ceev ntawm tus kheej cov lej tawg yog qhov tseem ceeb, tshwj xeeb tshaj yog thaum sib txuas R thiab C ++, thiab nrog pob raug tshem tawm - kuj yooj yim heev.

Zuag qhia tag nrho qhov kev paub no tau txais txiaj ntsig zoo heev thiab peb txuas ntxiv ua haujlwm los daws qee qhov teeb meem uas tau hais tseg.

Tau qhov twg los: www.hab.com

Quick Draw Doodle Recognition: yuav ua li cas phooj ywg nrog R, C ++ thiab neural networks