Script for mass cloning docker images into self-hosted registry

I run a private registry on my homelab. Pulling, retagging, and pushing images by hand every time I wanted a fresh copy was annoying. This script reads a list of images, gets their tags from Docker Hub, and copies them into the registry. Put it in a cronjob and the registry stays up to date on its own.

Create a working directory

mkdir -p ~/docker-sync && cd ~/docker-sync && touch ./images.txt && touch ./download.sh && chmod +x ./download.sh

Define the list of images we need (images.txt)

nginx
node
rabbitmq
traefik
drakkan/sftpgo
gogs/gogs

Write a script that reads the file and fetches available tags for each image (download.sh)

input="./images.txt"
while IFS= read -r image
do
  set -f
  imageParts=($(echo "$image" | tr '/' '\n'))
  imagePartsLength=${#imageParts[@]}
  if [[ imagePartsLength -lt 2 ]]; then
    namespace="library"
    imgName="${image}"
  else
    namespace="${imageParts[0]}"
    imgName="${imageParts[1]}"
  fi
  tags="$(wget -q -O - "https://hub.docker.com/v2/namespaces/$namespace/repositories/$imgName/tags?page_size=25" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$')"
  # Here we will add logic for images download and upload
done < "$input"

Images without a namespace, like nginx, are stored under library on Docker Hub, so the script uses library when there is no slash in the name. The page_size=25 only gets the 25 newest tags. Increase it if you need more, but the registry will grow fast.

Add logic for downloading images from Docker Hub and uploading them to our registry (finalized script)

input="./images.txt"
while IFS= read -r image
do
  set -f
  imageParts=($(echo "$image" | tr '/' '\n'))
  imagePartsLength=${#imageParts[@]}
  if [[ imagePartsLength -lt 2 ]]; then
    namespace="library"
    imgName="${image}"
  else
    namespace="${imageParts[0]}"
    imgName="${imageParts[1]}"
  fi
  tags="$(wget -q -O - "https://hub.docker.com/v2/namespaces/$namespace/repositories/$imgName/tags?page_size=25" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$')"
  while IFS= read -r line; do
    if [[ "$line" =~ ^[0-9]{1,2}\.[0-9]{1,2} ]]; then
      tagname="$image:$line"
      desttagname="$1/$imgName:$line"
      docker pull $tagname &> /dev/null
      docker tag $tagname $desttagname
      docker push $desttagname &> /dev/null
      echo "pushed $desttagname"
    fi
  done <<< "$tags"
done < "$input"

The regex ^[0-9]{1,2}\.[0-9]{1,2} is the tag filter. It keeps version tags such as 1.27 or 18.4 and skips latest, alpine, and the others. Replace it with whatever pattern matches the tags you want. The pulls and pushes are sent to /dev/null to keep the output clean. If a push fails, remove that redirect so you can see why. This happened to me once: the script quietly skipped half a registry because the disk was full, and the errors were hidden.

Run the script

The first argument is your registry address. The script uses it as the destination prefix when retagging.

cd ~/docker-sync
./download.sh reg.intra:5000