[{"_1":2,"_72":-5,"_73":-5},"loaderData",{"_3":4,"_12":13},"routes/_layout",{"_5":6},"pages",[7],{"_8":9,"_10":11},"slug","about","title","About me","routes/blog/index",{"_14":15,"_48":49,"_50":49,"_51":45,"_52":53,"_70":71},"posts",[16],{"_8":17,"_18":19,"_46":47},"script-for-mass-cloning-docker-images-into-self-hosted-registry","meta",{"_10":20,"_21":22,"_23":24,"_25":26,"_40":41,"_42":43},"Script for mass cloning docker images into self-hosted registry","description","A small shell script that copies many Docker images, with their tags, from Docker Hub into a self-hosted registry. It also has a regex filter to keep only the tags you want.","date","2025-01-05","keywords",[27,28,29,30,31,32,33,34,35,36,37,38,39],"docker","registry","images","cloning","self-hosted","private registry","docker hub","containers","automation","script","bulk","sync","infrastructure","author","Denys Bashkatov","tags",[44,45],"Docker","Homelab","html","

I run a private registry on my homelab. Pulling, retagging, and pushing images by hand every time I wanted a fresh copy was annoying. This script reads a list of images, gets their tags from Docker Hub, and copies them into the registry. Put it in a cronjob and the registry stays up to date on its own.

Create a working directory

mkdir -p ~/docker-sync && cd ~/docker-sync && touch ./images.txt && touch ./download.sh && chmod +x ./download.sh

Define the list of images we need (images.txt)

nginx\nnode\nrabbitmq\ntraefik\ndrakkan/sftpgo\ngogs/gogs\n

Write a script that reads the file and fetches available tags for each image (download.sh)

input="./images.txt"\nwhile IFS= read -r image\ndo\n  set -f\n  imageParts=($(echo "$image" | tr '/' '\\n'))\n  imagePartsLength=${#imageParts[@]}\n  if [[ imagePartsLength -lt 2 ]]; then\n    namespace="library"\n    imgName="${image}"\n  else\n    namespace="${imageParts[0]}"\n    imgName="${imageParts[1]}"\n  fi\n  tags="$(wget -q -O - "https://hub.docker.com/v2/namespaces/$namespace/repositories/$imgName/tags?page_size=25" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$')"\n  # Here we will add logic for images download and upload\ndone < "$input"

Images without a namespace, like nginx, are stored under library on Docker Hub, so the script uses library when there is no slash in the name. The page_size=25 only gets the 25 newest tags. Increase it if you need more, but the registry will grow fast.

Add logic for downloading images from Docker Hub and uploading them to our registry (finalized script)

input="./images.txt"\nwhile IFS= read -r image\ndo\n  set -f\n  imageParts=($(echo "$image" | tr '/' '\\n'))\n  imagePartsLength=${#imageParts[@]}\n  if [[ imagePartsLength -lt 2 ]]; then\n    namespace="library"\n    imgName="${image}"\n  else\n    namespace="${imageParts[0]}"\n    imgName="${imageParts[1]}"\n  fi\n  tags="$(wget -q -O - "https://hub.docker.com/v2/namespaces/$namespace/repositories/$imgName/tags?page_size=25" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$')"\n  while IFS= read -r line; do\n    if [[ "$line" =~ ^[0-9]{1,2}\\.[0-9]{1,2} ]]; then\n      tagname="$image:$line"\n      desttagname="$1/$imgName:$line"\n      docker pull $tagname &> /dev/null\n      docker tag $tagname $desttagname\n      docker push $desttagname &> /dev/null\n      echo "pushed $desttagname"\n    fi\n  done <<< "$tags"\ndone < "$input"

The regex ^[0-9]{1,2}\\.[0-9]{1,2} is the tag filter. It keeps version tags such as 1.27 or 18.4 and skips latest, alpine, and the others. Replace it with whatever pattern matches the tags you want. The pulls and pushes are sent to /dev/null to keep the output clean. If a push fails, remove that redirect so you can see why. This happened to me once: the script quietly skipped half a registry because the disk was full, and the errors were hidden.

Run the script

The first argument is your registry address. The script uses it as the destination prefix when retagging.

cd ~/docker-sync\n./download.sh reg.intra:5000

For a cronjob, use the full path and pass the same registry argument. If your upstream pulls are rate-limited, run it during off-peak hours.

\n","page",2,"totalPages","tag","allTags",[54,55,56,57,44,58,45,59,60,61,62,63,64,65,66,67,68,69],"Architecture","CSS","Database","Databases","Finance","JavaScript","Linux","Networking","Node.js","Performance","PostgreSQL","Project","Proxmox","React","TypeScript","Ubuntu","total",7,"actionData","errors"]

Tech Notes

Tech tips and new discoveries