Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

Loaʻa iā mākou 2 mau ʻeke mauʻu, 75 mescaline papa unix environment, kahi waihona docker a me ka hana o ka hoʻokō ʻana i nā kauoha huki docker a me nā docker push me ka ʻole o ka mea kūʻai aku docker.

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

UPD:
Nīnau: No ke aha kēia mau mea?
Pane: Hoʻouka i ka hoʻāʻo ʻana o ka huahana (ʻAʻole hoʻohana i ka bash, hāʻawi ʻia nā palapala no nā kumu hoʻonaʻauao). Ua hoʻoholo ʻia ʻaʻole e hoʻohana i ka mea kūʻai aku docker e hōʻemi i nā papa hou (i loko o nā palena kūpono) a, no laila, e hoʻohālike i kahi ukana kiʻekiʻe. ʻO ka hopena, ua hoʻoneʻe ʻia nā lohi āpau o ka mea kūʻai aku Docker. Ua loaʻa iā mākou kahi ukana maʻemaʻe ma luna o ka huahana.
Ua hoʻohana ka ʻatikala i nā mana GNU o nā mea hana.

ʻO ka mua, e noʻonoʻo kākou i ka hana a kēia mau kauoha.

No laila, he aha ka mea i hoʻohana ʻia ai ka docker pull? Wahi a palapala:

"E huki i kahi kiʻi a i ʻole kahi waihona mai kahi papa inoa".

Ma laila mākou e ʻike ai i kahi loulou i hoʻomaopopo i nā kiʻi, nā ipu, a me nā mea hoʻokele mālama.

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

Mai ʻaneʻi hiki iā mākou ke hoʻomaopopo i kahi kiʻi docker kahi hoʻonohonoho o kekahi mau papa i loaʻa ka ʻike e pili ana i nā loli hou loa o ke kiʻi, ʻo ia ka mea a mākou e pono ai. Ma hope aʻe e nānā mākou API kakau.

ʻŌlelo ia penei:

"ʻO kahi" kiʻi "he hui pū ʻana o kahi JSON hōʻike a me nā faila papa hoʻokahi. ʻO ke kaʻina hana o ka huki ʻana i kahi > kiʻi i waena o ke kiʻi ʻana i kēia mau ʻāpana ʻelua."

No laila ʻo ka hana mua e like me ka palapala "Huki i kahi Hōʻike Kiʻi".

ʻOiaʻiʻo, ʻaʻole mākou e pana, akā pono mākou i ka ʻikepili mai ia mea. Eia kekahi laʻana noi. GET /v2/{name}/manifests/{reference}

"ʻO ka inoa a me ka ʻāpana kuhikuhi e ʻike i ke kiʻi a koi ʻia. Hiki ke hoʻokomo ʻia ka hōʻailona a i ʻole ka digest."

Hoʻokomo ʻia kā mākou waihona docker ma ka ʻāina, e hoʻāʻo e hoʻokō i ka noi:

curl -s -X GET "http://localhost:8081/link/to/docker/registry/v2/centos-11-10/manifests/1.1.1" -H "header_if_needed"

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

I ka pane ʻana, loaʻa iā mākou ka json kahi a mākou e makemake wale nei i nā laina ola, a i ʻole kā lākou hashes. I ka loaʻa ʻana iā lākou, hiki iā mākou ke hele i kēlā me kēia a hoʻokō i kēia noi: "GET /v2/{name}/blobs/{digest}"

"E hoʻopaʻa ʻia ke komo ʻana i kahi papa e ka inoa o ka waihona akā ua ʻike ʻia ʻo ia ma ka papa inoa e ka digest."

ʻO ka digest i kēia hihia ka hash i loaʻa iā mākou.

Ke hoʻāʻo nei

curl -s -X GET "http://localhost:8081/link/to/docker/registry/v2/centos-11-10/blobs/sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f" -H "header_if_needed" --output firstLayer

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

E ʻike kākou i ke ʻano o ka faila i loaʻa iā mākou ma ke ʻano he laina ola mua.

file firstLayer

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

ka poe. ʻO nā kaʻa he mau waihona tar, e wehe ana iā lākou ma ke ʻano kūpono e loaʻa iā mākou nā mea o ke kiʻi.

E kākau i kahi palapala bash liʻiliʻi i hiki ke hoʻohana ʻia kēia mau mea āpau

#!/bin/bash -eu

downloadDir=$1
# url as http://localhost:8081/link/to/docker/registry
url=$2
imageName=$3
tag=$4

# array of layers
layers=($(curl -s -X GET "$url/v2/$imageName/manifests/$tag" | grep -oP '(?<=blobSum" : ").+(?=")'))

# download each layer from array
for layer in "${layers[@]}"; do
    echo "Downloading ${layer}"
    curl -v -X GET "$url/v2/$imageName/blobs/$layer" --output "$downloadDir/$layer.tar"
done

# find all layers, untar them and remove source .tar files
cd "$downloadDir" && find . -name "sha256:*" -exec tar xvf {} ;
rm sha256:*.tar
exit 0

I kēia manawa hiki iā mākou ke holo me nā ʻāpana i makemake ʻia a loaʻa nā ʻike o ke kiʻi i makemake ʻia

./script.sh dirName “http://localhost:8081/link/to/docker/registry” myAwesomeImage 1.0

Mahele 2 - pahu pahu

E paʻakikī iki kēia.

E hoʻomaka hou kākou me palapala. No laila pono mākou e hoʻoiho i kēlā me kēia alakaʻi, e hōʻiliʻili i ka hōʻike kūpono a hoʻoiho pū kekahi. He mea maʻalahi.

Ma hope o ke aʻo ʻana i ka palapala, hiki iā mākou ke hoʻokaʻawale i ke kaʻina hoʻoiho i kekahi mau ʻanuʻu:

  • Kaʻina hana hoʻomaka - "POST /v2/{repoName}/blobs/uploads/"
  • Ke hoʻouka nei i kahi laina ola (e hoʻohana mākou i kahi hoʻouka monolithic, ʻo ia hoʻi. hoʻouna mākou i kēlā me kēia laina ola i kona holoʻokoʻa) - "PUT /v2/{repoName}/blobs/uploads/{uuid}?digest={digest}
    Maʻiʻo-Loihi: {ka nui o ka papa}
    ʻAno maʻiʻo: noi/octet-stream
    Layer Binary Data".
  • Ke hoʻouka nei i ka hōʻike - "PUT /v2/{repoName}/manifests/{reference}".

Akā nalo ka palapala i kahi ʻanuʻu, me ka ʻole o ia mea e hana ʻole. No ka hoʻouka monolithic, a me ka hapa (chunked), ma mua o ka hoʻouka ʻana i ke kaʻaahi, pono ʻoe e hana i kahi noi PATCH:

"PATCH /v2/{repoName}/blobs/uploads/{uuid}
Maʻiʻo-Loihi: {ka nui o ka ʻāpana}
ʻAno maʻiʻo: noi/octet-stream
{Layer Chunk Binary Data}".

A i ʻole, ʻaʻole hiki iā ʻoe ke neʻe ma mua o ka helu mua, no ka mea... Ma kahi o ka helu pane i manaʻo ʻia 202, e loaʻa iā ʻoe ka 4xx.

I kēia manawa ua like ka algorithm:

  • Hoʻomaka
  • Patch rail
  • Ke hoʻouka nei i ka lima lima
  • Ke hoʻouka nei i ka hōʻike
    E hana hou ʻia nā helu 2 a me 3 e like me ka nui o nā laina e hoʻouka ʻia.

ʻO ka mea mua, pono mākou i kekahi kiʻi. E hoʻohana wau i archlinux:latest

docker pull archlinux

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

I kēia manawa, e mālama kākou ma ka ʻāina no ka nānā hou ʻana

docker save c24fe13d37b9 -o savedArch

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

Wehe i ka waihona i loaʻa i loko o ka papa kuhikuhi o kēia manawa

tar xvf savedArch

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

E like me kāu e ʻike ai, aia kēlā me kēia lifeline i kahi waihona ʻokoʻa. E nānā kākou i ke ʻano o ka hōʻike i loaʻa iā mākou

cat manifest.json | json_pp

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

ʻAʻole nui. E ʻike kākou i ka manifest e pono ai e hoʻouka, e like me palapala.

Ke hoʻokō nei i nā kauoha docker pull a docker push me ka ʻole o kahi mea kūʻai docker e hoʻohana ana i nā noi HTTP

ʻIke loa, ʻaʻole kūpono ka manifesto i kēia manawa, no laila e hana mākou iā mākou iho me ka blackjack a me nā courtesans, nā laina ola a me nā configs.

E loaʻa mau iā mākou ma kahi o hoʻokahi faila config a me kahi ʻano o nā laina ola. ʻO ka mana o ka papahana 2 (i kēia manawa i ka wā e kākau ai), e waiho ʻia ka mediaType me ka loli ʻole:

echo ‘{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": config_size,
      "digest": "config_hash"
   },
   "layers": [
      ’ > manifest.json

Ma hope o ka hana ʻana i ka hōʻike kumu, pono ʻoe e hoʻopiha iā ia me ka ʻikepili kūpono. No ka hana ʻana i kēia, hoʻohana mākou i ka template json o ka mea rail:

{
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": ${layersSizes[$i]},
         "digest": "sha256:${layersNames[$i]}"
      },

E hoʻohui mākou i ka hōʻike no kēlā me kēia kaʻa.

A laila, pono mākou e ʻike i ka nui o ka faila config a hoʻololi i nā stubs i ka hōʻike me ka ʻikepili maoli

sed -i "s/config_size/$configSize/g; s/config_hash/$configName/g" $manifestFile

I kēia manawa hiki iā ʻoe ke hoʻomaka i ke kaʻina hoʻoiho a mālama iā ʻoe iho i uuid, pono e hele pū me nā noi a pau.

ʻO ka palapala holoʻokoʻa e like me kēia:

#!/bin/bash -eux

imageDir=$1
# url as http://localhost:8081/link/to/docker/registry
url=$2
repoName=$3
tag=$4
manifestFile=$(readlink -f ${imageDir}/manifestCopy)
configFile=$(readlink -f $(find $imageDir -name "*.json" ! -name "manifest.json"))

# calc layers sha 256 sum, rename them accordingly, and add info about each to manifest file
function prepareLayersForUpload() {
  info_file=$imageDir/info
  # lets calculate layers sha256 and use it as layers names further
  layersNames=($(find $imageDir -name "layer.tar" -exec shasum -a 256 {} ; | cut -d" " -f1))

  # rename layers according to shasums. !!!Set required amount of fields for cut command!!!
  # this part definitely can be done easier but i didn't found another way, sry
  find $imageDir -name "layer.tar" -exec bash -c 'mv {} "$(echo {} | cut -d"/" -f1,2)/$(shasum -a 256 {} | cut -d" " -f1)"' ;

  layersSizes=($(find $imageDir -name "*.tar" -exec ls -l {} ; | awk '{print $5}'))

  for i in "${!layersNames[@]}"; do
    echo "{
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": ${layersSizes[$i]},
         "digest": "sha256:${layersNames[$i]}"
      }," >> $manifestFile
  done
  # remove last ','
  truncate -s-2 $manifestFile
  # add closing brakets to keep json consistent
  printf "nt]n}" >> $manifestFile
}

# calc config sha 256 sum and add info about it to manifest
function setConfigProps() {
  configSize=$(ls -l $configFile | awk '{print $5}')
  configName=$(basename $configFile | cut -d"." -f1)

  sed -i "s/config_size/$configSize/g; s/config_hash/$configName/g" $manifestFile
}

#prepare manifest file
prepareLayersForUpload
setConfigProps
cat $manifestFile

# initiate upload and get uuid
uuid=$(curl -s -X POST -I "$url/v2/$repoName/blobs/uploads/" | grep -oP "(?<=Docker-Upload-Uuid: ).+")

# patch layers
# in data-binary we're getting absolute path to layer file
for l in "${!layersNames[@]}"; do
  pathToLayer=$(find $imageDir -name ${layersNames[$l]} -exec readlink -f {} ;)
    curl -v -X PATCH "$url/v2/$repoName/blobs/uploads/$uuid" 
  -H "Content-Length: ${layersSizes[$i]}" 
  -H "Content-Type: application/octet-stream" 
  --data-binary "@$pathToLayer"

# put layer
  curl -v -X PUT "$url/v2/$repoName/blobs/uploads/$uuid?digest=sha256:${layersNames[$i]}" 
  -H 'Content-Type: application/octet-stream' 
  -H "Content-Length: ${layersSizes[$i]}" 
  --data-binary "@$pathToLayer"
done

# patch and put config after all layers
curl -v -X PATCH "$url/v2/$repoName/blobs/uploads/$uuid" 
  -H "Content-Length: $configSize" 
  -H "Content-Type: application/octet-stream" 
  --data-binary "@$configFile"

  curl -v -X PUT "$url/v2/$repoName/blobs/uploads/$uuid?digest=sha256:$configName" 
  -H 'Content-Type: application/octet-stream' 
  -H "Content-Length: $configSize" 
  --data-binary "@$configFile"

# put manifest
curl -v -X PUT "$url/v2/$repoName/manifests/$tag" 
  -H 'Content-Type: application/vnd.docker.distribution.manifest.v2+json' 
  --data-binary "@$manifestFile"

exit 0

hiki iā mākou ke hoʻohana i kahi palapala i hoʻomākaukau ʻia:

./uploadImage.sh "~/path/to/saved/image" "http://localhost:8081/link/to/docker/registry" myRepoName 1.0

UPD:
He aha ka mea i loaʻa iā mākou?
ʻO ka mea mua, ʻo ka ʻikepili maoli no ka nānā ʻana, ʻoiai ke holo nei nā hoʻāʻo i ka blazemeter a ʻaʻole ʻike nui ka ʻikepili i nā noi o ka mea kūʻai aku docker, ʻaʻole like me nā noi HTTP maʻemaʻe.

ʻO ka lua, ua ʻae ka hoʻololi iā mākou e hoʻonui i ka helu o nā mea hoʻohana virtual no ka hoʻouka docker ma kahi o 150% a loaʻa ka manawa pane avg 20-25% wikiwiki. No ka hoʻoiho docker, hiki iā mākou ke hoʻonui i ka helu o nā mea hoʻohana e 500%, ʻoiai ua emi ka manawa pane avg ma kahi o 60%.

Mahalo oe no kou noonoo.

Source: www.habr.com

Pākuʻi i ka manaʻo hoʻopuka