Tech Notes

Script for mass cloning docker images into self-hosted registry

This article describes how to clone a large number of docker images into a private registry. All tags for each image in your list will be cloned. The ability to filter tags by a regular expression will be added.

#Docker#Homelab

Our implementation will help you if you want to have a private docker registry with a large number of cloned images from the original docker hub. In addition, our solution will allow you to automatically transfer all the tags of the images you need to your personal registry. In addition, you can add the script that we will describe in this article to the cronjob, and thus you can update all the images you need according to the schedule.

Create a working directory
mkdir -p ~/docker-sync && cd ~/docker-sync && touch ./images.txt && touch ./download.sh && chmod +x ./download.sh
Describe the format of the file with the list of images that you need (images.txt)
nginx
node
rabbitmq
traefik
drakkan/sftpgo
gogs/gogs
Now let's write a script that will read the file and download a list of available tags for each image (download.sh)
input="./images.txt"
while IFS= read -r image
do
  set -f
  imageParts=($(echo "$image" | tr '/' '\n'))
  imagePartsLength=${#imageParts[@]}
  if [[ imagePartsLength -lt 2 ]]; then
    namespace="library"
    imgName="${image}"
  else
    namespace="${imageParts[0]}"
    imgName="${imageParts[1]}"
  fi
  tags="$(wget -q -O - "https://hub.docker.com/v2/namespaces/$namespace/repositories/$imgName/tags?page_size=25" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$')"
  # Here we will add logic for images download and upload
done < "$input"
Now let's add logic for downloading images from Docker Hub and uploading them to our registry (finalized script)
input="./images.txt"
while IFS= read -r image
do
  set -f
  imageParts=($(echo "$image" | tr '/' '\n'))
  imagePartsLength=${#imageParts[@]}
  if [[ imagePartsLength -lt 2 ]]; then
    namespace="library"
    imgName="${image}"
  else
    namespace="${imageParts[0]}"
    imgName="${imageParts[1]}"
  fi
  tags="$(wget -q -O - "https://hub.docker.com/v2/namespaces/$namespace/repositories/$imgName/tags?page_size=25" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$')"
  while IFS= read -r line; do
    if [[ "$line" =~ ^[0-9]{1,2}\.[0-9]{1,2} ]]; then
      tagname="$image:$line"
      desttagname="$1/$imgName:$line"
      docker pull $tagname &> /dev/null
      docker tag $tagname $desttagname
      docker push $desttagname &> /dev/null
      echo "pushed $desttagname"
    fi
  done <<< "$tags"
done < "$input"
That's it. Now you can run the resulting script in this way:
cd ~/docker-sync
./download.sh reg.intra:5000