🥇Ngleksanakake perintah docker pull lan docker push tanpa klien docker liwat panjalukan HTTP

Kita duwe 2 tas suket, 75 tablet mescaline lingkungan unix, gudang docker lan tugas ngleksanakake docker pull lan docker push printah tanpa klien docker.

UPD:
Pitakonan: Apa kabeh iki kanggo?
Wangsulan: Uji coba produk (ora nggunakake bash, skrip diwenehake kanggo tujuan pendidikan). Diputusake ora nggunakake klien docker kanggo nyuda lapisan tambahan (ing watesan sing cukup) lan, mula, niru beban sing luwih dhuwur. Akibaté, kabeh tundha sistem klien Docker dibusak. Kita nampa beban relatif resik langsung ing prodhuk.
Artikel kasebut nggunakake alat versi GNU.

Pisanan, ayo ngerteni apa sing ditindakake perintah kasebut.

Dadi kanggo apa docker pull digunakake? miturut dokumentasi:

"Narik gambar utawa repositori saka pendaptaran".

Ing kana kita uga nemokake link menyang ngerti gambar, wadhah, lan driver panyimpenan.

Saka kene kita bisa ngerti yen gambar docker minangka kumpulan lapisan tartamtu sing ngemot informasi babagan owah-owahan paling anyar ing gambar kasebut, sing jelas sing kita butuhake. Sabanjure kita katon ing registri API.

Iku ngandika ing ngisor iki:

"Gambar" minangka kombinasi saka manifest JSON lan file lapisan individu. Proses narik gambar > pusat kanggo njupuk rong komponen kasebut.

Dadi langkah pisanan miturut dokumentasi yaiku "Narik Manifes Gambar".

Mesthi, kita ora bakal njupuk, nanging kita butuh data saka iku. Ing ngisor iki minangka conto panyuwunan: GET /v2/{name}/manifests/{reference}

"Jeneng lan parameter referensi ngenali gambar lan dibutuhake. Referensi bisa uga kalebu tag utawa digest."

Repositori docker kita disebarake sacara lokal, ayo nyoba nglakokake panjaluk kasebut:

curl -s -X GET "http://localhost:8081/link/to/docker/registry/v2/centos-11-10/manifests/1.1.1" -H "header_if_needed"

Nanggepi, kita nampa json saka kang saiki kita mung kasengsem ing lifelines, utawa rodo hash sing. Sawise nampa, kita bisa ngliwati saben lan nglakokake panjaluk ing ngisor iki: "GET /v2/{name}/blobs/{digest}"

"Akses menyang lapisan bakal gated kanthi jeneng repositori nanging diidentifikasi kanthi unik ing registri kanthi digest."

Digest ing kasus iki yaiku hash sing ditampa.

Ayo jajal

curl -s -X GET "http://localhost:8081/link/to/docker/registry/v2/centos-11-10/blobs/sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f" -H "header_if_needed" --output firstLayer

Ayo ndeleng file apa sing pungkasane ditampa minangka garis urip pisanan.

file firstLayer

sing. ril iku arsip tar, unpacking mau ing urutan cocok kita bakal njaluk isi gambar.

Ayo nulis skrip bash cilik supaya kabeh iki bisa otomatis

#!/bin/bash -eu

downloadDir=$1
# url as http://localhost:8081/link/to/docker/registry
url=$2
imageName=$3
tag=$4

# array of layers
layers=($(curl -s -X GET "$url/v2/$imageName/manifests/$tag" | grep -oP '(?<=blobSum" : ").+(?=")'))

# download each layer from array
for layer in "${layers[@]}"; do
    echo "Downloading ${layer}"
    curl -v -X GET "$url/v2/$imageName/blobs/$layer" --output "$downloadDir/$layer.tar"
done

# find all layers, untar them and remove source .tar files
cd "$downloadDir" && find . -name "sha256:*" -exec tar xvf {} ;
rm sha256:*.tar
exit 0

Saiki kita bisa mbukak kanthi paramèter sing dikarepake lan entuk isi gambar sing dibutuhake

./script.sh dirName “http://localhost:8081/link/to/docker/registry” myAwesomeImage 1.0

Part 2 - docker push

Iki bakal dadi luwih rumit.

Ayo dadi miwiti maneh karo dokumentasi. Dadi, kita kudu ndownload saben pimpinan, ngumpulake manifest sing cocog lan download uga. Iku misale jek prasaja.

Sawise sinau dokumentasi, kita bisa mbagi proses download dadi sawetara langkah:

Inisialisasi proses - "POST /v2/{repoName}/blobs/uploads/"
Ngunggah garis urip (kita bakal nggunakake unggahan monolitik, yaiku ngirim saben garis urip kabeh) - "PUT /v2/{repoName}/blobs/uploads/{uuid}?digest={digest}
Dawane isi: {ukuran lapisan}
Tipe-Konten: aplikasi/oktet-stream
Data Binary Lapisan".
Loading manifest - "PUT /v2/{repoName}/manifests/{referensi}".

Nanging dokumentasi ora kejawab siji langkah, tanpa sing ora bakal bisa. Kanggo loading monolithic, uga kanggo sebagean (chunked), sadurunge mbukak ril, sampeyan kudu nindakake panjalukan PATCH:

"PATCH /v2/{repoName}/blobs/uploads/{uuid}
Panjang Isi: {ukuran potongan}
Tipe-Konten: aplikasi/oktet-stream
{Layer Chunk Binary Data}".

Yen ora, sampeyan ora bakal bisa ngluwihi titik pisanan, amarga ... Tinimbang kode respon samesthine 202, sampeyan bakal nampa 4xx.

Saiki algoritma katon kaya:

Wiwitan
Patch rel
Loading handrail
Loading manifest
TCTerms 2 lan 3, mungguh, bakal bola kaping pirang-pirang minangka nomer baris kudu dimuat.

Kaping pisanan, kita butuh gambar apa wae. Aku bakal nggunakake archlinux: latest

docker pull archlinux

Saiki ayo simpen sacara lokal kanggo analisis luwih lanjut

docker save c24fe13d37b9 -o savedArch

Unpack arsip asil menyang direktori saiki

tar xvf savedArch

Kaya sing sampeyan ngerteni, saben garis urip ana ing folder sing kapisah. Saiki ayo ndeleng struktur manifes sing ditampa

cat manifest.json | json_pp

Ora akeh. Ayo ndeleng manifest apa sing dibutuhake kanggo dimuat, miturut dokumentasi.

Temenan, manifesto ana ora cocog karo kita, supaya kita bakal nggawe dhewe karo blackjack lan courtesans, lifelines lan configs.

Kita bakal tansah duwe paling siji file config lan Uploaded saka lifelines. Skema versi 2 (saiki nalika nulis), mediaType ora bakal diganti:

echo ‘{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": config_size,
      "digest": "config_hash"
   },
   "layers": [
      ’ > manifest.json

Sawise nggawe manifest dhasar, sampeyan kudu ngisi data sing bener. Kanggo nindakake iki, kita nggunakake template json saka obyek rel:

{
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": ${layersSizes[$i]},
         "digest": "sha256:${layersNames[$i]}"
      },

Kita bakal nambah menyang manifest kanggo saben ril.

Sabanjure, kita kudu ngerteni ukuran file konfigurasi lan ngganti stubs ing manifest karo data nyata

sed -i "s/config_size/$configSize/g; s/config_hash/$configName/g" $manifestFile

Saiki sampeyan bisa miwiti proses download lan nyimpen dhewe uuid, sing kudu ngiringi kabeh panjalukan sakteruse.

Skrip lengkap katon kaya iki:

#!/bin/bash -eux

imageDir=$1
# url as http://localhost:8081/link/to/docker/registry
url=$2
repoName=$3
tag=$4
manifestFile=$(readlink -f ${imageDir}/manifestCopy)
configFile=$(readlink -f $(find $imageDir -name "*.json" ! -name "manifest.json"))

# calc layers sha 256 sum, rename them accordingly, and add info about each to manifest file
function prepareLayersForUpload() {
  info_file=$imageDir/info
  # lets calculate layers sha256 and use it as layers names further
  layersNames=($(find $imageDir -name "layer.tar" -exec shasum -a 256 {} ; | cut -d" " -f1))

  # rename layers according to shasums. !!!Set required amount of fields for cut command!!!
  # this part definitely can be done easier but i didn't found another way, sry
  find $imageDir -name "layer.tar" -exec bash -c 'mv {} "$(echo {} | cut -d"/" -f1,2)/$(shasum -a 256 {} | cut -d" " -f1)"' ;

  layersSizes=($(find $imageDir -name "*.tar" -exec ls -l {} ; | awk '{print $5}'))

  for i in "${!layersNames[@]}"; do
    echo "{
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": ${layersSizes[$i]},
         "digest": "sha256:${layersNames[$i]}"
      }," >> $manifestFile
  done
  # remove last ','
  truncate -s-2 $manifestFile
  # add closing brakets to keep json consistent
  printf "nt]n}" >> $manifestFile
}

# calc config sha 256 sum and add info about it to manifest
function setConfigProps() {
  configSize=$(ls -l $configFile | awk '{print $5}')
  configName=$(basename $configFile | cut -d"." -f1)

  sed -i "s/config_size/$configSize/g; s/config_hash/$configName/g" $manifestFile
}

#prepare manifest file
prepareLayersForUpload
setConfigProps
cat $manifestFile

# initiate upload and get uuid
uuid=$(curl -s -X POST -I "$url/v2/$repoName/blobs/uploads/" | grep -oP "(?<=Docker-Upload-Uuid: ).+")

# patch layers
# in data-binary we're getting absolute path to layer file
for l in "${!layersNames[@]}"; do
  pathToLayer=$(find $imageDir -name ${layersNames[$l]} -exec readlink -f {} ;)
    curl -v -X PATCH "$url/v2/$repoName/blobs/uploads/$uuid" 
  -H "Content-Length: ${layersSizes[$i]}" 
  -H "Content-Type: application/octet-stream" 
  --data-binary "@$pathToLayer"

# put layer
  curl -v -X PUT "$url/v2/$repoName/blobs/uploads/$uuid?digest=sha256:${layersNames[$i]}" 
  -H 'Content-Type: application/octet-stream' 
  -H "Content-Length: ${layersSizes[$i]}" 
  --data-binary "@$pathToLayer"
done

# patch and put config after all layers
curl -v -X PATCH "$url/v2/$repoName/blobs/uploads/$uuid" 
  -H "Content-Length: $configSize" 
  -H "Content-Type: application/octet-stream" 
  --data-binary "@$configFile"

  curl -v -X PUT "$url/v2/$repoName/blobs/uploads/$uuid?digest=sha256:$configName" 
  -H 'Content-Type: application/octet-stream' 
  -H "Content-Length: $configSize" 
  --data-binary "@$configFile"

# put manifest
curl -v -X PUT "$url/v2/$repoName/manifests/$tag" 
  -H 'Content-Type: application/vnd.docker.distribution.manifest.v2+json' 
  --data-binary "@$manifestFile"

exit 0

kita bisa nggunakake script siap-digawe:

./uploadImage.sh "~/path/to/saved/image" "http://localhost:8081/link/to/docker/registry" myRepoName 1.0

UPD:
Apa sing kita entuk minangka asil?
Kaping pisanan, data nyata kanggo analisis, amarga tes ditindakake ing blazemeter lan data panjaluk klien docker ora informatif, ora kaya panjaluk HTTP murni.

Kapindho, transisi kasebut ngidini kita nambah jumlah pangguna virtual kanggo upload docker kira-kira 150% lan entuk wektu nanggepi rata-rata 20-25% luwih cepet. Kanggo download docker, kita bisa nambah jumlah pangguna kanthi 500%, dene wektu nanggepi rata-rata mudhun udakara 60%.

Matur nuwun kanggo manungsa waé.

Source: www.habr.com

Ngleksanakake docker pull lan docker push printah tanpa klien docker nggunakake panjalukan HTTP

Part 2 - docker push