当前位置: 首页 > DevOps > 在离线环境下部署Talos Linux图形化管理工具Omni

在离线环境下部署Talos Linux图形化管理工具Omni

DevOps 0条评论 2024-10-8 658 views

概述:

之前已经介绍Talos Linux,虽然我比较喜欢命令行,但对于管理便捷性还是非常差的,同时有很多小伙伴也关心在内网如何使用,本期我讲带大家使用Talos Linux的Web管理平台-Omni和离线环境搭建。


相关工具:

Omni DocsOmni官方文档,包含AirGap环境搭建指导和On-Prem部署指导。

Image Factory本地TalOS构建平台,支持构建裸金属、各种云平台,Omni会集成本地的Image-Factory。

Talos Linux ExtensionsTalos Linux的扩展image,例如:iSCSI支持、虚拟机Tools等。

Talos Linux Image Factory在线Image构建平台,可自定义系统扩展(vmtools-guest-agent、Nvidia container toolkit、qemu-guest-agent等)。 Talos Docs官方文档,包含AirGap环境搭建指导。

Skopeo用于上传image到本地仓库。

Cosign用于给本地仓库的image签名(因为Image-Factory必须对签名进行验证,所以会单独对本地image进行签名,以保证Image-Factory可以工作)

Kubelogin用于oidc登录,omni平台生成的kubeconfig需要通过oidc登录认证。


环境需求:

为验证整体架构和部署,整体采用生产架构,如果只是为了体验Talos可以通过Docker运行。

  1. 一台可以上网Linux虚拟机,安装Docker、jq环境,用于获取离线环境所需的所有images;
  2. 一台内网Linux虚拟机,安装Docker环境,用于运行离线镜像仓库-Registry、Omni、Keycloak、Image-Factory;
  3. 1个虚拟化/物理机环境,用于验证通过Omni部署Talos Linux Kubernetes(单节点);
  4. 一个DHCP的网络,用于Talos Linux首次启动获取IP地址,初始化完成后不再需要。当然也可以使用静态IP地址,为了简便,这里使用HDCP;

注意:如果需要多个版本的Talos Linux和Kubernetes,离线的镜像会比较大,请提前规划好磁盘空间。

在线Linux获取image脚本

先构建一个在线拉去image的脚本,其会拉去所有所需的image到2个压缩包和2个list文件,这里采用skopeo工具进行拉去,

cat pull-image-by-skopeo.sh

#!/bin/bash

# 安装skopeo工具
wget -O skopeo-linux-amd64 https://github.com/lework/skopeo-binary/releases/download/v1.16.1/skopeo-linux-amd64
chmod +x skopeo-linux-amd64
cp skopeo-linux-amd64 /usr/local/bin
export PATH=$PATH:/usr/local/bin
# 定义需要下载的 Talos 版本(可自行调整)
TALOS_VERSIONS=(
#  "v1.7.0"
#  "v1.7.1"
#  "v1.7.2"
#  "v1.7.3"
  "v1.7.4"
  "v1.7.5"
  "v1.7.6"
  "v1.7.7"
  "v1.8.0"
)

# 定义 Kubernetes 大版本列表,这里会下载所有的版本,如果不需要所有版本,请调整get_minor_versions()函数中per_page的页数。
KUBERNETES_VERSIONS=(
#  "v1.27"
#  "v1.28"
#  "v1.29"
  "v1.30"
  "v1.31"
)

# 定义 Kubernetes 相关组件的动态镜像列表(根据版本号动态生成)
DYNAMIC_IMAGES=(
  "registry.k8s.io/kube-apiserver"
  "registry.k8s.io/kube-controller-manager"
  "registry.k8s.io/kube-scheduler"
  "registry.k8s.io/kube-proxy"
  "ghcr.io/siderolabs/kubelet"
)

# 定义额外需要拉取的静态镜像列表,这些Image用于Docker启动环境,不用上传到私有仓库中。
STATIC_IMAGES=(
  "postgres:16"
  "quay.io/keycloak/keycloak:25.0"
  "quay.io/minio/minio:RELEASE.2024-10-02T17-50-41Z"
  "ghcr.io/siderolabs/omni:latest"
  "registry:2"
  "ghcr.io/siderolabs/image-factory:v0.5.0"
)

# 定义文件及目录,动态Image采用tar.gz压缩(需要先解压缩,再执行上传脚本),静态Image采用tar压缩(可以直接Load到本地)
DYNAMIC_IMAGE_LIST_FILE="dynamic_images-list.txt"
STATIC_IMAGE_LIST_FILE="static_images-list.txt"
DYNAMIC_IMAGE_TAR_GZ="dynamic_images.tar.gz"
STATIC_IMAGE_TAR="static_images.tar"

# 定义 GitHub API 的 URL 来获取 Kubernetes Releases
GITHUB_API_URL="https://api.github.com/repos/kubernetes/kubernetes/releases"

# 清空旧的镜像列表文件
> $STATIC_IMAGE_LIST_FILE
> $DYNAMIC_IMAGE_LIST_FILE

# 1. 下载特定版本的 `talosctl`,判断是否存在,如果存在就跳过
download_talosctl() {
  local version=$1
  local filename="talosctl-${version}"
  local url="https://github.com/siderolabs/talos/releases/download/${version}/talosctl-linux-amd64"

  if [ -f "$filename" ]; then
    echo "talosctl version ${version} already exists locally, skipping download."
  else
    echo "Downloading talosctl version: ${version} from ${url}..."
    wget -q $url -O "$filename"
    chmod +x "$filename"
  fi
}

# 2. 使用特定版本的 `talosctl` 来获取默认镜像清单
fetch_talos_images() {
  local version=$1
  local filename="talosctl-${version}"
  echo "Fetching images for Talos version: ${version} using talosctl..."
  ./$filename image default | grep -oP '^\S+:\S+' >> $DYNAMIC_IMAGE_LIST_FILE
}

# 3. 从 factory.talos.dev 获取扩展组件和 overlays 镜像
fetch_factory_images() {
  local version=$1
  local api_base="https://factory.talos.dev/version/${version}"

  # 3.1 获取扩展镜像
  echo "Fetching official extensions for version: ${version}..."
  curl -s "${api_base}/extensions/official" | jq -r '.[].ref' >> $DYNAMIC_IMAGE_LIST_FILE

  # 3.2 获取 overlays 镜像
  echo "Fetching official overlays for version: ${version}..."
  curl -s "${api_base}/overlays/official" | jq -r '.[].ref' >> $DYNAMIC_IMAGE_LIST_FILE
}

# 4. 使用 GitHub API 获取所有小版本号
get_minor_versions() {
  local major_version=$1
  local api_url="https://api.github.com/repos/kubernetes/kubernetes/releases?per_page=100"
  curl -s $api_url | grep -Po "\"tag_name\":\s*\"$major_version\.\d+\"" | grep -Po "$major_version\.\d+" | sort -V
}

# 6. 根据 Talos 版本定义版本特定镜像
add_version_specific_images() {
  local version=$1

  VERSION_SPECIFIC_IMAGES+=(
    "ghcr.io/siderolabs/extensions:${version}"
    "ghcr.io/siderolabs/installer:${version}"
    "ghcr.io/siderolabs/imager:${version}"
  )
}
# 7. 生成镜像列表文件
echo "Generating image lists..."

# 7.1 收集所有 Talos 相关的镜像
for version in "${TALOS_VERSIONS[@]}"; do
  download_talosctl $version
  fetch_talos_images $version
  fetch_factory_images $version
  add_version_specific_images $version
done

# 7.2 收集所有 Kubernetes 组件的镜像
for major_version in "${KUBERNETES_VERSIONS[@]}"; do
  MINOR_VERSIONS=$(get_minor_versions "$major_version")
  echo "$MINOR_VERSIONS" | while read -r minor_version; do
    for IMAGE in "${DYNAMIC_IMAGES[@]}"; do
      IMAGE_WITH_TAG="${IMAGE}:${minor_version}"
      echo "$IMAGE_WITH_TAG" >> $DYNAMIC_IMAGE_LIST_FILE
    done
  done
done

# 7.3 收集静态镜像
for IMAGE in "${STATIC_IMAGES[@]}"; do
  echo "$IMAGE" >> $STATIC_IMAGE_LIST_FILE
done

# 7.4. 收集所有版本特定镜像
for IMAGE in "${VERSION_SPECIFIC_IMAGES[@]}"; do
  echo "$IMAGE" >> $DYNAMIC_IMAGE_LIST_FILE
done

# 8. 去重并生成最终的镜像列表
echo "Removing duplicate images..."
sort -u $STATIC_IMAGE_LIST_FILE -o $STATIC_IMAGE_LIST_FILE
sort -u $DYNAMIC_IMAGE_LIST_FILE -o $DYNAMIC_IMAGE_LIST_FILE

# 9. 下载静态镜像
echo "Downloading static images using docker pull..."
while read -r image; do
  echo "Pulling static image: $image"
  #docker pull "$image"
done < $STATIC_IMAGE_LIST_FILE

# 10. 下载动态镜像
echo "Downloading dynamic images using skopeo..."
mkdir -p dynamic_images
while read -r image; do
  echo "Copying dynamic image: $image"
  skopeo copy --retry-times 3 --all --preserve-digests "docker://${image}" "dir:dynamic_images/$(echo "$image" | tr '/:' '_')"
done < $DYNAMIC_IMAGE_LIST_FILE

# 11. 保存所有静态镜像到 tar 文件
echo "Saving static images to ${STATIC_IMAGE_TAR}..."
docker save -o $STATIC_IMAGE_TAR $(cat $STATIC_IMAGE_LIST_FILE)

# 12. 保存所有动态镜像到 tar 文件
echo "Saving dynamic images to ${DYNAMIC_IMAGE_TAR}..."
tar -czvf $DYNAMIC_IMAGE_TAR_GZ dynamic_images/

# 最终结果提示
echo "All static images have been saved to ${STATIC_IMAGE_TAR}."
echo "All dynamic images have been saved to ${DYNAMIC_IMAGE_TAR}."
echo "Static image list has been saved to ${STATIC_IMAGE_LIST_FILE}."
echo "Dynamic image list has been saved to ${DYNAMIC_IMAGE_LIST_FILE}."

在线Linux获取管理工具脚本

#!/bin/bash

# 创建目录用于保存工具
mkdir -p tools
cd tools

# 下载工具
wget -O skopeo-linux-amd64 https://github.com/lework/skopeo-binary/releases/download/v1.16.1/skopeo-linux-amd64
wget -O cosign-linux-amd64 https://github.com/sigstore/cosign/releases/download/v2.4.0/cosign-linux-amd64
wget -O kubelogin_linux_amd64.zip https://github.com/int128/kubelogin/releases/download/v1.30.1/kubelogin_linux_amd64.zip
wget -O talosctl-linux-amd64 https://github.com/siderolabs/talos/releases/download/v1.8.0/talosctl-linux-amd64
wget -O omnictl-linux-amd64 https://github.com/siderolabs/omni/releases/download/v0.42.3/omnictl-linux-amd64
wget -O kubectl https://dl.k8s.io/release/v1.31.1/bin/linux/amd64/kubectl
wget -O jq https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64
CFSSL_VERSION=1.6.5
wget -q -O cfssl https://github.com/cloudflare/cfssl/releases/download/v${CFSSL_VERSION}/cfssl_${CFSSL_VERSION}_linux_amd64
wget -q -O cfssljson https://github.com/cloudflare/cfssl/releases/download/v${CFSSL_VERSION}/cfssljson_${CFSSL_VERSION}_linux_amd64

# 解压 `kubelogin`
unzip kubelogin_linux_amd64.zip

# 修改权限
chmod +x skopeo-linux-amd64 cosign-linux-amd64 kubelogin talosctl-linux-amd64 omnictl-linux-amd64 kubectl

# 删除无关的压缩包
rm -f kubelogin_linux_amd64.zip

# 打包所有工具为 tar.gz
cd ..
tar -czvf tools_package.tar.gz tools/

echo "Tools has package to: tools_package.tar.gz"

离线Linux配置—启动本地镜像仓库

先加载所有image到本地,再解压缩动态image压缩包

sudo mkdir -p /usr/local/omni
chown -R $USER /usr/local/omni
cd /usr/local/omni

#upload  to /usr/local/omni
#dynamic_images-list.txt
#static_images-list.txt
#dynamic_images.tar.gz
#static_images.tar
#tools_package.tar.gz

docker load -i static_images.tar
tar -zxvf dynamic_images.tar.gz
查看是否包含Image的文件夹
ls dynamic_images

启动本地镜像仓库和打开防火墙端口。

mkdir -p /usr/local/omni/registrydata
docker run -d -p 5000:5000 \
--restart always \
--name registry-airgapped \
-v /usr/local/omni/registrydata:/var/lib/registry \
-e "REGISTRY_STORAGE_DELETE_ENABLED=true" \
registry:2

sudo firewall-cmd --permanent --add-port=5000/tcp
sudo firewall-cmd --reload

离线Linux Tools安装脚本

构建install_tools.sh脚本,用于安装相关工具。

#!/bin/bash

# Check if tools_package.tar.gz exists
if [ ! -f "tools_package.tar.gz" ]; then
  echo "tools_package.tar.gz not found. Please place the package in the current directory."
  exit 1
fi

# Extract the tools package
tar -xzvf tools_package.tar.gz

# Enter the tools directory
cd tools

# Check if /usr/local/bin is in $PATH
if ! echo "$PATH" | grep -q "/usr/local/bin"; then
  echo "/usr/local/bin is not in the PATH. Adding it now..."
  # Add to /etc/profile
  echo 'export PATH=$PATH:/usr/local/bin' | sudo tee -a /etc/profile
  # Update the current shell's PATH
  export PATH=$PATH:/usr/local/bin
  echo "Successfully added /usr/local/bin to the PATH."
else
  echo "/usr/local/bin is already in the PATH."
fi

# Install tools to /usr/local/bin
sudo cp skopeo-linux-amd64 /usr/local/bin/skopeo
sudo cp cosign-linux-amd64 /usr/local/bin/cosign
sudo cp kubelogin /usr/local/bin/kubectl-oidc_login
sudo cp talosctl-linux-amd64 /usr/local/bin/talosctl
sudo cp omnictl-linux-amd64 /usr/local/bin/omnictl
sudo cp kubectl /usr/local/bin/kubectl
sudo cp jq /usr/local/bin/jq
sudo cp cfssl /usr/local/bin/cfssl
sudo cp cfssljson /usr/local/bin/cfssljson

# Set executable permissions
sudo chmod +x /usr/local/bin/skopeo
sudo chmod +x /usr/local/bin/cosign
sudo chmod +x /usr/local/bin/kubectl-oidc_login
sudo chmod +x /usr/local/bin/talosctl
sudo chmod +x /usr/local/bin/omnictl
sudo chmod +x /usr/local/bin/kubectl
sudo chmod +x /usr/local/bin/jq
sudo chmod +x /usr/local/bin/cfssl
sudo chmod +x /usr/local/bin/cfssljson

# Add command completion to /etc/profile
echo "Adding command completions to /etc/profile..."
echo 'source <(talosctl completion bash)' | sudo tee -a /etc/profile
echo 'source <(omnictl completion bash)' | sudo tee -a /etc/profile
echo 'source <(kubectl completion bash)' | sudo tee -a /etc/profile

# Apply the changes immediately
source /etc/profile

echo "All tools have been successfully installed to /usr/local/bin, and command completions have been configured."

离线Linux配置—安装工具

./install_tools.sh

离线Linux配置—生成omni和keycloak所需的SSL证书

我们通过CFSSL工具来生成证书,keycloak和omni使用相同的证书,脚本如下:

cat > generate_omni_certs.sh <<EOOF
#!/bin/bash
mkdir -p /usr/local/omni/certs
cd /usr/local/omni/certs
# 参数定义
OMNI_IP="10.40.43.112"
CA_CERT_FILE="omni-ca.pem"
CA_KEY_FILE="omni-ca-key.pem"

# 创建CA配置
cat > omni-ca-csr.json <<EOF
{
  "CN": "Omni CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Changchun",
      "O": "DevOps",
      "OU": "OMNI",
      "ST": "Jilin"
    }
  ],
  "ca": {
    "expiry": "438000h"
  }
}
EOF

# 检查 CA 证书和密钥是否存在
if [[ -f "$CA_CERT_FILE" && -f "$CA_KEY_FILE" ]]; then
  echo "CA 证书和密钥已存在,跳过创建步骤。"
else
  echo "CA 证书和密钥不存在,正在创建..."
  # 创建 CA 证书和密钥的命令
  cfssl gencert -initca omni-ca-csr.json | cfssljson -bare omni-ca
  echo "CA 证书和密钥创建完成。"
fi

# 生成 Omni证书配置
cat > config.json <<EOF
{
"signing": {
  "default": {
  "usages": ["digital signature","key encipherment","signing","server auth"],
  "expiry": "87600h"
   }
}
}
EOF
#

cat > omni-csr.json <<EOF
{
  "CN": "$OMNI_IP",
  "hosts": [
      "127.0.0.1",
      "$OMNI_IP"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C":  "CN",
      "L":  "Changchun",
      "O":  "DevOps",
      "OU": "OMNI",
      "ST": "Jilin"
    }
  ]
}
EOF

# 生成 Omni 证书
cfssl gencert -ca=omni-ca.pem -ca-key=omni-ca-key.pem -config=config.json omni-csr.json | cfssljson -bare omni
cat omni.pem omni-ca.pem > omni-chain.pem
EOOF

生成私有CA和Omni证书

chmod +x generate_omni_certs.sh
./generate_omni_certs.sh

离线Linux配置—生成omni和image-factory所需的key

准备生成相关key的脚本,包括用于ETCD加密的GPG和Cosign的私钥和公钥。

# cat generate-gpg-cert.sh

#!/bin/bash

# 定义参数
EMAIL="[email protected]"
KEY_NAME="Omni (Used for etcd data encryption) <${EMAIL}>"
OUTPUT_DIR="/usr/local/omni"
GPG_ASC_FILE="${OUTPUT_DIR}/omni.asc"

# 检查并创建输出目录
mkdir -p "$OUTPUT_DIR"
cd $OUTPUT_DIR

# 使用 gpg --with-colons --fingerprint 检查是否存在相同 uid 的 GPG 密钥,并获取指纹
FPR=$(gpg --with-colons --fingerprint "$EMAIL" | grep -E "^fpr" | head -n 1 | cut -d: -f10)

if [ -n "$FPR" ]; then
  echo "Found an existing GPG key for ${EMAIL} with fingerprint: ${FPR}"
else
  echo "No existing GPG key found for ${EMAIL}. Generating a new GPG key for ${KEY_NAME}..."

  # 使用 --batch 模式生成 GPG 主密钥,并设置参数无交互生成
  gpg --batch --gen-key <<EOF
%no-protection
Key-Type: RSA
Key-Length: 4096
Subkey-Type: RSA
Subkey-Length: 4096
Name-Real: Omni
Name-Comment: Used for etcd data encryption
Name-Email: ${EMAIL}
Expire-Date: 0
%commit
EOF

  # 再次检查是否生成了相同的 GPG 密钥,并获取指纹
  FPR=$(gpg --with-colons --fingerprint "$EMAIL" | grep -E "^fpr" | head -n 1 | cut -d: -f10)

  if [ -z "$FPR" ]; then
    echo "Error: Failed to generate the GPG key or retrieve its fingerprint."
    echo "Checking the generated GPG keys for troubleshooting:"
    gpg --list-secret-keys --fingerprint
    exit 1
  fi
fi

# 导出 GPG 密钥到 omni.asc
if [ -f "${GPG_ASC_FILE}" ]; then
  echo "GPG key file ${GPG_ASC_FILE} already exists. Skipping export."
else
  echo "Exporting GPG key to ${GPG_ASC_FILE}..."
  gpg --export-secret-key --armor "${FPR}" > "${GPG_ASC_FILE}"

  if [ -f "${GPG_ASC_FILE}" ]; then
    echo "Successfully generated and exported GPG key to ${GPG_ASC_FILE}"
  else
    echo "Error: Failed to export GPG key."
    exit 1
  fi
fi
# image factory
openssl ecparam -name prime256v1 -genkey -noout -out cache-signing-key.key

echo "All keys have been successfully generated and exported!"

生成key

chmod +x generate-gpg-cert.sh
./generate-gpg-cert.sh

查看key

ls /usr/local/omni/omni.asc /usr/local/omni/cache-signing-key.key

为Cosign准备证书

这次我们采用上一步创建的omni-ca来签发证书,并在签发后使用Cosign导入成Cosign使用的key和pub。

cd /usr/local/omni/certs
# 构建profiles文件
echo "Building cosign certificate"
cat > cosign-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "cosign": {
        "usages": ["signing", "digital signing"],
        "expiry": "87600h",
        "copy_extensions": true
      }
    }
  }
}
EOF
# 配置证书所需变量
#extensions参考:https://github.com/sigstore/fulcio/blob/main/docs/oid-info.md
# OIDC_ISSUER用于验证时的--certificate-oidc-issuer or --certificate-oidc-issuer-regexp参数;
OIDC_ISSUER=http://localhost
OIDC_ISSUER_BASE64=$(echo -n $OIDC_ISSUER | base64 -w0)
#CERTIFICATE_IDENTITY用于验证时的--certificate-identity or --certificate-identity-regexp参数;
[email protected]
# 构建CSR文件
cat cosign-csr.json 
{
  "CN": "Cosign Certificate",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "C":  "CN",
      "L":  "Changchun",
      "O":  "DevOps",
      "OU": "OMNI",
      "ST": "Jilin"
    }
  ],
  "hosts": [
    "${CERTIFICATE_IDENTITY}"
  ],
  "extensions": [
    {
        "id": [1,3,6,1,4,1,57264,1,1],
        "value": "${OIDC_ISSUER_BASE64}"
    },
    {
        "id": [1,3,6,1,4,1,57264,1,8],
        "value": "${OIDC_ISSUER_BASE64}"
    }
  ]
}

# 签发Cosign证书
cfssl gencert \
-ca omni-ca.pem \
-ca-key omni-ca-key.pem  \
-config cosign-config.json   \
-profile cosign cosign-csr.json \
|cfssljson -bare cosign
# 转换成Cosign使用的key和pub
cosign import-key-pair --yes --key cosign-key.pem --output-key-prefix import-cosign
# 查看cosign的key和pub
ls import-cosign.key import-cosign.pub

Cosign测试签名

cosign sign --key certs/import-cosign.key --cert certs/cosign.pem --cert-chain certs/omni-ca.pem --tlog-upload=false 10.40.43.112:5000/siderolabs/installer:v1.8.0

Cosign测试验证签名 这里指定根证书等参数后,可以验证通过。

$ cosign verify --ca-roots=certs/omni-ca.pem --certificate-identity='[email protected]' --certificate-oidc-issuer='http://localhost' --insecure-ignore-sct --insecure-ignore-tlog  10.40.43.112:5000/siderolabs/installer:v1.8.0
WARNING: Skipping tlog verification is an insecure practice that lacks of transparency and auditability verification for the signature.

Verification for 10.40.43.112:5000/siderolabs/installer:v1.8.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The code-signing certificate was verified using trusted certificate authority certificates

[{"critical":{"identity":{"docker-reference":"10.40.43.112:5000/siderolabs/installer"},"image":{"docker-manifest-digest":"sha256:8f034be82a4f4518ad7f1cb7cc730fcb4a5b160b708a3f7b1530ca168100af32"},"type":"cosign container image signature"},"optional":{"1.3.6.1.4.1.57264.1.1":"http://localhost","Issuer":"http://localhost","Subject":"[email protected]"}}]

为SecureBoot准备证书和PCR密钥

如果希望Image-Factory支持构建SecureBoot的ISO,那么我们需要生成SecureBoot所需证书和私钥,这里直接使用Talosctl工具来生成。

cd /usr/local/omni
talosctl gen secureboot uki --common-name "SecureBoot Key" -output secureboot_keys
talosctl gen secureboot pcr --output secureboot_keys
ls secureboot_keys/
# pcr-signing-key.pem  uki-signing-cert.der  uki-signing-cert.pem  uki-signing-key.pem

创建Skopeo的配置文件

cat > /etc/containers/policy.json <<EOF
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ]
}
EOF

离线Linux上传Image到本地镜像仓库脚本

构建 push-image-by-skopeo.sh 脚本,用于上传Image到仓库,并对Image进行签名。

#!/bin/bash

# 定义私有仓库地址
REGISTRY_URL="10.40.43.112:5000"

# Set Cosign password to empty (to skip password prompt)
export COSIGN_PASSWORD=""

# Cosign key file路径
CERTS_DIR="./certs"
COSIGN_KEY="${CERTS_DIR}/import-cosign.key"
COSIGN_CERT="${CERTS_DIR}/cosign.pem"
COSIGN_CERT_CHAIN="${CERTS_DIR}/omni-ca.pem"

# 定义文件和目录路径
DYNAMIC_IMAGE_LIST_FILE="dynamic_images-list.txt"  # 只处理 image:tag 格式
IMAGE_DIR="dynamic_images"
UPLOADED_IMAGES_FILE="uploaded_images.txt"
UPLOAD_LOG="upload.log"
SIGN_LOG="sign.log"

# 清空现有文件
> "${UPLOADED_IMAGES_FILE}"
> "${UPLOAD_LOG}"
> "${SIGN_LOG}"

# Function: 去除字符串前后的空格
trim() {
  echo "$1" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
}

# Function: 使用 skopeo 上传镜像
upload_image_with_skopeo() {
  local image_tag
  image_tag=$(trim "$1")  # 去除空格
  local local_image_path="$2"  # 本地镜像目录路径
  local registry_url="$3"      # 目标仓库地址

  # 提取镜像的 repo 名称(不包含前缀)
  local repo_name="${image_tag#*/}"
  local target_tag="${registry_url}/${repo_name}"          # 目标 tag 格式

  echo "Pushing image from ${local_image_path} to ${target_tag}..." | tee -a "${UPLOAD_LOG}"

  # 使用 skopeo copy 将目录中的镜像推送到私有仓库,只上传 tag
  skopeo copy --all --preserve-digests --dest-tls-verify=false \
    "dir:${local_image_path}" \
    "docker://${target_tag}" &>> "${UPLOAD_LOG}"

  if [[ $? -eq 0 ]]; then
    echo "Successfully pushed image ${target_tag}." | tee -a "${UPLOAD_LOG}"
    echo "${target_tag}" >> "${UPLOADED_IMAGES_FILE}"
  else
    echo "Failed to push image ${target_tag}." | tee -a "${UPLOAD_LOG}"
  fi
}

# 1. 上传动态镜像:读取 dynamic_images-list.txt,并上传镜像目录中的文件
echo "Uploading dynamic images..." | tee -a "${UPLOAD_LOG}"
while IFS= read -r image_tag; do
  image_tag=$(trim "${image_tag}")  # 提取 image:tag 格式并去除空格
  # 根据 image_tag 提取 registry 前缀,并去掉 registry 前缀
  echo $image_tag
  # 根据 image_tag 提取本地镜像目录路径(确保目录结构匹配)
  image_safe_name=$(echo "${image_tag}" | tr '/:' '_')  # 替换字符以构建目录名(如 "pause_3.9")
  echo $image_safe_name
  # 根据 image_tag 提取本地镜像目录路径(确保目录结构匹配)
  local_image_path="${IMAGE_DIR}/${image_safe_name}"   # 本地镜像目录路径
  echo $local_image_path
  if [[ -d "${local_image_path}" ]]; then
    # 上传动态镜像
    upload_image_with_skopeo "$image_tag" "$local_image_path" "$REGISTRY_URL"
  else
    echo "Local image directory not found for ${image_tag}, skipping..." | tee -a "${UPLOAD_LOG}"
  fi
done < "${DYNAMIC_IMAGE_LIST_FILE}"

# 2. 基于上传成功的镜像列表文件进行签名
echo "All images have been uploaded successfully. Starting Cosign signing..." | tee -a "${SIGN_LOG}"

while IFS= read -r image; do
  image_tag=$(trim "${image}")  # 提取 image:tag 格式并去除空格

  # 构建签名对象:tag 格式
  repo_name="${image_tag#*/}"
  target_tag="${REGISTRY_URL}/${repo_name}"

  # 使用 Cosign 签名 tag
  echo "Signing image ${target_tag} ..." | tee -a "${SIGN_LOG}"
  cosign sign --key "${COSIGN_KEY}" --cert "${COSIGN_CERT}" --cert-chain="${COSIGN_CERT_CHAIN}" --tlog-upload=false "${target_tag}" &>> "${SIGN_LOG}"
  if [[ $? -eq 0 ]]; then
    echo "Successfully signed tag: ${target_tag}." | tee -a "${SIGN_LOG}"
  else
    echo "Failed to sign tag: ${target_tag}." | tee -a "${SIGN_LOG}"
  fi

done < "${UPLOADED_IMAGES_FILE}"

echo "All images have been signed successfully." | tee -a "${SIGN_LOG}"

# 检查上传和签名日志中的错误
echo "Checking logs for errors..." | tee -a "${UPLOAD_LOG}"
UPLOAD_ERRORS=$(grep -iE "failed|error|timeout" "${UPLOAD_LOG}")
SIGN_ERRORS=$(grep -iE "failed|error|timeout" "${SIGN_LOG}")

if [[ -n "$UPLOAD_ERRORS" || -n "$SIGN_ERRORS" ]]; then
  echo "Some issues were detected during the process:" | tee -a "${UPLOAD_LOG}"
  if [[ -n "$UPLOAD_ERRORS" ]]; then
    echo "Upload errors:" | tee -a "${UPLOAD_LOG}"
    echo "$UPLOAD_ERRORS" | tee -a "${UPLOAD_LOG}"
  fi
  if [[ -n "$SIGN_ERRORS" ]]; then
    echo "Sign errors:" | tee -a "${SIGN_LOG}"
    echo "$SIGN_ERRORS" | tee -a "${SIGN_LOG}"
  fi
else
  echo "No errors found in the log files." | tee -a "${UPLOAD_LOG}"
fi

echo "Process completed!"

离线Linux配置—上传并签名image

chmod +x push-image-by-skopeo.sh
./push-image-by-skopeo.sh

准备Docker Compose相关文件

我们采用Docker-Compose来运行KeyCloak、Omni和Image-Factroy、MinIO,其中KeyCloak用于SAML认证和用户管理,MinIO用于存储集群备份。

注意1:请修改相应的IP地址、密码和UUID 注意2:Omni容器中/etc/ssl/certs/目录是存放私有CA信任的位置,只需要把Omni的根CA映射到此目录,即可实现证书信任。

准备文件目录

chown -R ops /usr/local/omni
mkdir -p /usr/local/omni/data/{postgres,keycloak,minio}
mkdir -p /usr/local/omni/etcd
chmod 777 /usr/local/omni/data/{postgres,keycloak,minio}
# 由于omni需要使用tun,而系统默认没有打开tun,此处通过modprobe加载
sudo modprobe tun
lsmod |grep tun

准备.env文件,其用于Docker Compose启动参数

cat > /usr/local/omni/.env <<EOF
POSTGRES_VERSION=16
KC_VERSION=25.0
KC_PORT=8443
KC_LOG_LEVEL=INFO
KC_REALM_NAME=omni
KEYCLOAK_ADMIN=admin
KEYCLOAK_ADMIN_PASSWORD=VMware1!
PG_USER=keycloak
PG_PASSWORD=VMware1!
OMNI_IP=10.40.43.112
OMNI_VERSION=v0.42.3
#此UUID可以uuidgen生成
OMNI_ACCOUNT_UUID=b5841399-88f7-4129-98a5-49f4bff92671
MINIO_VERSION=RELEASE.2024-10-02T17-50-41Z
EOF

准备docker-compose.yml文件,用于启动环境

cat  /usr/local/omni/docker-compose.yml 
version: '3.8'
x-logging: &logging
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
services:
  postgres:
    image: postgres:${POSTGRES_VERSION}
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "keycloak"]
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: ${PG_USER}
      POSTGRES_PASSWORD: ${PG_PASSWORD}
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    networks:
      - omni
    logging: *logging

  keycloak:
    image: quay.io/keycloak/keycloak:${KC_VERSION}
    command: ["start", "--https-port=8443"]
    restart: unless-stopped
    environment:
      KC_DB: postgres
      KC_DB_USERNAME: ${PG_USER}
      KC_DB_PASSWORD: ${PG_PASSWORD}
      KC_DB_URL: "jdbc:postgresql://postgres:5432/keycloak"
      KC_METRICS_ENABLED: false
      KC_LOG_LEVEL: ${KC_LOG_LEVEL}
      KC_REALM_NAME: ${KC_REALM_NAME}
      KEYCLOAK_ADMIN: ${KEYCLOAK_ADMIN}
      KEYCLOAK_ADMIN_PASSWORD: ${KEYCLOAK_ADMIN_PASSWORD}
      KC_HOSTNAME: ${OMNI_IP}
      KC_TRUSTSTORE_PATHS: /opt/keycloak/certificates/truststore
      KC_HTTPS_CERTIFICATE_FILE: /opt/keycloak/certificates/omni-chain.pem
      KC_HTTPS_CERTIFICATE_KEY_FILE: /opt/keycloak/certificates/omni-key.pem
    ports:
      - ${KC_PORT}:8443
    volumes:
      - ./data/keycloak:/opt/keycloak/data:rw
      - ./certs/omni-chain.pem:/opt/keycloak/certificates/omni-chain.pem:ro
      - ./certs/omni-key.pem:/opt/keycloak/certificates/omni-key.pem:ro
      - ./certs/omni-ca.pem:/opt/keycloak/certificates/truststore/omni-ca.pem:ro
    networks:
      - omni
    depends_on:
      postgres:
        condition: service_healthy
        restart: true
    logging: *logging
  omni:
    image: ghcr.io/siderolabs/omni
    container_name: omni
    restart: unless-stopped
    network_mode: host
    cap_add:
      - NET_ADMIN
    volumes:
      - ./etcd:/_out/etcd
      - ./certs/omni-chain.pem:/omni-chain.pem
      - ./certs/omni-key.pem:/omni-key.pem
      - ./certs/omni-ca.pem:/etc/ssl/certs/omni-ca.pem
      - ./omni.asc:/omni.asc
    devices:
      - "/dev/net/tun:/dev/net/tun"
    command: >
      --account-id=${OMNI_ACCOUNT_UUID}
      --cert=/omni-chain.pem
      --key=/omni-key.pem
      --siderolink-api-cert=/omni-chain.pem
      --siderolink-api-key=/omni-key.pem
      --private-key-source=file:///omni.asc
      --event-sink-port=8091
      --bind-addr=0.0.0.0:443
      --siderolink-api-bind-addr=0.0.0.0:8090
      --k8s-proxy-bind-addr=0.0.0.0:8100
      --advertised-api-url=https://${OMNI_IP}:443/
      --siderolink-api-advertised-url=https://${OMNI_IP}:8090/
      --siderolink-wireguard-advertised-addr=${OMNI_IP}:50180
      --advertised-kubernetes-proxy-url=https://${OMNI_IP}:8100/
      --auth-auth0-enabled=false
      --auth-saml-enabled
      --auth-saml-url=https://${OMNI_IP}:8443/realms/omni/protocol/saml/descriptor
      --talos-installer-registry=${OMNI_IP}:5000/siderolabs/installer
      --kubernetes-registry=${OMNI_IP}:5000/siderolabs/kubelet
      --image-factory-address=http://${OMNI_IP}:8080
      --image-factory-pxe-address=http://${OMNI_IP}:8080
      --registry-mirror=docker.io=http://${OMNI_IP}:5000,gcr.io=http://${OMNI_IP}:5000,ghcr.io=http://${OMNI_IP}:5000,registry.k8s.io=http://${OMNI_IP}:5000,factory.talos.dev=http://${OMNI_IP}:8080
    depends_on:
      keycloak:
        condition: service_started
        restart: true
    logging: *logging
  image-factory:
    image: "ghcr.io/siderolabs/image-factory:v0.5.0"
    restart: unless-stopped
    container_name: image-factory
    privileged: true
    ports:
      - "8080:8080"
    environment:
      SIGSTORE_ROOT_FILE: /root.pem
      SIGSTORE_REKOR_PUBLIC_KEY: /cosign.pub
      SIGSTORE_CT_LOG_PUBLIC_KEY_FILE: /cosign.pub
    volumes:
      - /dev:/dev
      - /sys:/sys
      - ./cache-signing-key.key:/cache-signing-key.key
      - ./certs/import-cosign.key:/cosign.key
      - ./certs/import-cosign.pub:/cosign.pub
      - ./tmp:/tmp
      - ./certs/omni-ca.pem:/root.pem
      - ./secureboot_keys/uki-signing-key.pem:/uki-signing-key.pem
      - ./secureboot_keys/uki-signing-cert.pem:/uki-signing-cert.pem
      - ./secureboot_keys/pcr-signing-key.pem:/pcr-signing-key.pem
    command: >
      -http-port=0.0.0.0:8080
      -image-registry ${OMNI_IP}:5000
      -external-url http://${OMNI_IP}:8080
      -schematic-service-repository ${OMNI_IP}:5000/image-factory/schematic
      -installer-internal-repository ${OMNI_IP}:5000/siderolabs
      -installer-external-repository ${OMNI_IP}:5000/siderolabs
      -cache-repository ${OMNI_IP}:5000/cache
      -secureboot
      -secureboot-pcr-key-path /pcr-signing-key.pem
      -secureboot-signing-key-path /uki-signing-key.pem
      -secureboot-signing-cert-path /uki-signing-cert.pem
      -insecure-image-registry
      -insecure-cache-repository
      -insecure-schematic-service-repository
      -insecure-installer-internal-repository
      -cache-signing-key-path /cache-signing-key.key
      -container-signature-pubkey /cosign.pub
      -container-signature-issuer http://localhost
    logging: *logging
  minio:
    image: quay.io/minio/minio:RELEASE.2024-10-02T17-50-41Z
    container_name: minio
    restart: unless-stopped
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: VMware1!
    volumes:
      - ~/data/minio:/data
    command: server /data --console-address ":9001"
networks:
  omni:

离线Linux配置—打开防火墙端口

sudo firewall-cmd --permanent \
--add-port=8080/tcp \
--add-port=8090/tcp \
--add-port=8100/tcp \
--add-port=8091/tcp \
--add-port=8091/udp \
--add-port=8090/udp \
--add-port=8100/udp \
--add-port=50180/tcp \
--add-port=50180/udp \
--add-port=8092/tcp \
--add-port=8092/udp \
--add-port=10000/tcp \
--add-port=10000/udp \
--add-port=8093/tcp \
--add-port=8093/udp \
--add-port=50001/tcp \
--add-port=50001/udp \
--add-port=5000/tcp \
--add-port=8200/tcp \
--add-port=80/tcp \
--add-port=443/tcp \
--add-port=8443/tcp \
--add-port=9000/tcp \
--add-port=9001/tcp

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" forward-port port="80" protocol="tcp" to-port="443"'
sudo firewall-cmd --reload

启动环境

cd /usr/local/omni
docker compose up -d

配置Keycloak Realm

浏览器输入:https://${OMNI_IP}:8443,使用admin账户登录后,参考下面文档进行配置:https://omni.siderolabs.com/how-to-guides/self_hosted/_index

配置完成后,再次启用omni容器(因为Keycloak未配置时,Omni会启动失败)

docker compose up -d

此时可以看到Omini容器已经可以正常启动

测试Omni

您可以在Windows/Linux机器中添加omni-ca.pem的根证书信任,这样浏览器就不会再弹出警告,具备方法自行搜索,这里不做示例。 首先打开浏览器登录omn(https://10.40.43.112) ,自动跳转到keycloak认证 认证成功后,点击“Log In”登录 登录后,我们可以看到Omni的管理页面,包括API Endpoint信息等 先测试本地构建image是否可以工作,点击右侧的“Download Installation Media”,我们可以看到Omni自动识别了已在离线Registry上传的Talos Linux版本(1.7.4-1.8.0),第二个选项中可以选择Image类型(ISO、VMware、Openstack等),在添加所需的Extension,还可以添加额外标签、内核参数,也可以生成PXE启动URL。 由于我们使用了自建名证书,会导致节点启动后无法连接omni,这里需要把证书信任添加到Kernel参数中

echo "apiVersion: v1alpha1
kind: TrustedRootsConfig
name: root-ca
certificates: |
$(cat certs/omni-ca.pem | sed 's/^/    /')" > trusted-roots-config.yaml

cat trusted-roots-config.yaml |zstd --compress --ultra -22 | base64 -w 0

Kernel参数:

echo talos.config.inline=<base64>

这里我选择1.8.0和VMware平台,点击“Download” 构建完成后,浏览器会自动下载“vmware-amd64-omni-default-v1.8.0.ova” 通过vCenter部署OVA,不需要提供任何额外的参数,因为所有参数都通过kernel Arguments传递了。可以通过Console看到,虚拟机已经启动,并连接到Omni; 在Omni管理页面上,也可以看到机器; 下面,我们创建一个单节点K8S集群;我们可以选择Kubernetes的版本(Talos版本需要和运行的版本一致,此处选择1.8.0),Kubernetes的版本是从仓库中读取出来的,根据需要选择;选择可用的节点,点击“Create Cluster” 还需要提供“Config Patch”,用于信任Omni CA证书,否则Node重启后,无法连接到Omni;

echo "apiVersion: v1alpha1
kind: TrustedRootsConfig
name: root-ca
certificates: |
$(cat certs/omni-ca.pem | sed 's/^/    /')" > trusted-roots-config.yaml

echo trusted-root-config.yaml

最终的Config Path如下所示:

apiVersion: v1alpha1
kind: TrustedRootsConfig
name: omni-ca
certificates: |
    -----BEGIN CERTIFICATE-----
    MIIDmDCCAoCgAwIBAgIUbHcCbcWgVQytszXCKlgeP2YcR/8wDQYJKoZIhvcNAQEL
    BQAwYzELMAkGA1UEBhMCQ04xDjAMBgNVBAgTBUppbGluMRIwEAYDVQQHEwlDaGFu
    Z2NodW4xDzANBgNVBAoTBkRldk9wczENMAsGA1UECxMET01OSTEQMA4GA1UEAxMH
    T21uaSBDQTAgFw0yNDEwMDQxMzA3MDBaGA8yMDc0MDkyMjEzMDcwMFowYzELMAkG
    A1UEBhMCQ04xDjAMBgNVBAgTBUppbGluMRIwEAYDVQQHEwlDaGFuZ2NodW4xDzAN
    BgNVBAoTBkRldk9wczENMAsGA1UECxMET01OSTEQMA4GA1UEAxMHT21uaSBDQTCC
    ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMJqM71fjKYGuu2rn2glElBp
    lhz3I9czd3dZyhos2WMEvGKHZ59fzDJR/ft/rRwTg0ORwq3u++wLBSjdY8SPIZoQ
    JCKfBoL0n/wlDT8cD38efnj4OSxkzO1oMvwsdT6pr39BpcKax8miGKxX+V7i2ums
    ed8sxlqKhe9IP2nD25Z6KEiXilovZ3sj9Vrwx2OOY87eOHzHQvR+X2veoXk+qLTM
    TFc1R1Sf6w+4A67W0e5EznThzMKTVGFmteKYl1ZpuhK82VqE89XW+riqpXM5XwkL
    n4iJBeaSYilfQaq0+haBk30KjuTHoCpvZ9N/cn1Kmk2OUHs9sLLMVtYJOW3hA+0C
    AwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0O
    BBYEFHqh+kEAUo11lunyYbkGjRw/IcraMA0GCSqGSIb3DQEBCwUAA4IBAQAyhlH7
    CaXcbZxq2FM/1rSwy0q0y1Aud901RQY4OYds/6pSvIewNneFK4+hb1n39PujiMA4
    SfzmmibbrIYGfwjn60P2WqCY73xweJrmiSrxhz6Hp9DcFVrH4CwKhATClvc5NZan
    vVRRSfNhQE9fQqfqY4kZ0P2C76od0GUB4U71b2Ih2gXD1ciWSBSz3WuXQGF7qNsY
    hkBWzJz4zKd3JRjmNrsCKgzWFwmaSKXRIbHnx6gKav1INiteM9iH8PKbzW2wvmCr
    vL96XSkMf6tLpZELpEQNO3wCgOBesS1KuLLdVZ19agTOj7PUifzyBQ4T4Jrzglze
    jmu/SjbzXlVHIkZe
    -----END CERTIFICATE-----

稍等片刻,Cluster创建完成。 虚拟机Console显示“Healthy”状态 通Omni查看Node的状态、控制台日志和配置

完成

由于Omni和Talos涉及大量技术依赖,离线环境部署还是比较复杂的,所以,本博客的内容也非常多,我也是尽量把有用的内容放进来,减少大家再去研究。

当然,这也不仅仅适用于Omni,如果单独运行Talos Linux的离线环境,也适用,只需要做好Image部分的离线就可以。


发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注