[Study][GitOps] ArgoCD 운영 구성 - 4

이전에 작성한 글에 이어서 ArgoCD 운영을 염두에 둔 기능들을 테스트해보았다.

• ApplicationSet Generators

 

ArgoCD 실습환경

이전과 동일하게 Windows 11의 WSL2를 통해서 Ubuntu 24.04를 실행하고 Kind를 통해서 구성하였다.

• Host : Windows 11
• WSL2 : Ubuntu 24.04
  
  • kind / kubectl / git / helm / argocd CLI 설치
• GitHub Private Repo 사용
• Private Repository 접근 권한이 있는 GitHub Fine-grained PAT(Personal Access Token)

 

kind 기반 멀티 클러스터(mgmt/dev/prd) 구성

mgmt 클러스터 생성 + ingress-nginx

# mgmt 클러스터 생성
kind create cluster --name mgmt --image kindest/node:v1.32.8 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  labels:
    ingress-ready: "true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
  - containerPort: 30000
    hostPort: 30000
EOF

# ingress-nginx 설치
kubectl apply -f \
  https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

# SSL Passthrough 옵션 추가
kubectl get deployment ingress-nginx-controller -n ingress-nginx -o yaml \
| sed '/- --publish-status-address=localhost/a\
        - --enable-ssl-passthrough' | kubectl apply -f -

Argo CD용 self-signed 인증서 및 Secret 생성

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout argocd.example.com.key \
  -out argocd.example.com.crt \
  -subj "/CN=argocd.example.com/O=argocd"

kubectl create ns argocd

kubectl -n argocd create secret tls argocd-server-tls \
  --cert=argocd.example.com.crt \
  --key=argocd.example.com.key

Helm으로 Argo CD 설치

cat <<EOF > argocd-values.yaml
global:
  domain: argocd.example.com

server:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    tls: true
EOF

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm install argocd argo/argo-cd \
  --version 9.0.5 \
  -f argocd-values.yaml \
  --namespace argocd

hosts 등록 (Windows)

PS C:\Windows\System32> notepad C:\Windows\System32\drivers\etc\hosts

PS C:\Windows\System32> cat C:\Windows\System32\drivers\etc\hosts
# Copyright (c) 1993-2009 Microsoft Corp.
...
127.0.0.1 argocd.example.com

접속테스트

PS C:\Windows\System32> curl -vk https://argocd.example.com/
* Host argocd.example.com:443 was resolved.
* IPv6: (none)
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:443...
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* Connected to argocd.example.com (127.0.0.1) port 443
...
* schannel: SSL/TLS connection renegotiated
* Request completely sent off
< HTTP/1.1 200 OK
...

초기 패스워드 확인 & CLI 로그인

# 초기 admin 패스워드 확인
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d ; echo

# CLI 로그인
ARGOPW=<위에서 출력된 비밀번호>
argocd login argocd.example.com --insecure \
  --username admin --password $ARGOPW

# (필요 시) 비밀번호 변경
argocd account update-password \
  --current-password $ARGOPW \
  --new-password <변경할 비밀번호>

dev / prd 클러스터 생성 & 컨텍스트 설정

# Too many open files 에러 회피용
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512

# dev 클러스터
kind create cluster --name dev --image kindest/node:v1.32.8 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 31000
    hostPort: 31000
EOF

# prd 클러스터
kind create cluster --name prd --image kindest/node:v1.32.8 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 32000
    hostPort: 32000
EOF

# 컨텍스트 확인
(⎈|kind-prd:N/A) root@DESKTOP-O4EPQ9T:~# kubectl config get-contexts
CURRENT   NAME        CLUSTER     AUTHINFO    NAMESPACE
          kind-dev    kind-dev    kind-dev
          kind-mgmt   kind-mgmt   kind-mgmt
*         kind-prd    kind-prd    kind-prd

# alias 편의 설정 (선택)
echo "alias k8s1='kubectl --context kind-mgmt'" >> ~/.bashrc
echo "alias k8s2='kubectl --context kind-dev'"  >> ~/.bashrc
echo "alias k8s3='kubectl --context kind-prd'"  >> ~/.bashrc
source ~/.bashrc

(⎈|kind-prd:N/A) root@DESKTOP-O4EPQ9T:~# k8s1 get node -owide
NAME                 STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                     CONTAINER-RUNTIME
mgmt-control-plane   Ready    control-plane   20m   v1.32.8   172.18.0.2    <none>        Debian GNU/Linux 12 (bookworm)   6.6.87.2-microsoft-standard-WSL2   containerd://2.1.3

(⎈|kind-prd:N/A) root@DESKTOP-O4EPQ9T:~# k8s2 get node -owide
NAME                STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                     CONTAINER-RUNTIME
dev-control-plane   Ready    control-plane   6m50s   v1.32.8   172.18.0.3    <none>        Debian GNU/Linux 12 (bookworm)   6.6.87.2-microsoft-standard-WSL2   containerd://2.1.3

(⎈|kind-prd:N/A) root@DESKTOP-O4EPQ9T:~# k8s3 get node -owide
NAME                STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                     CONTAINER-RUNTIME
prd-control-plane   Ready    control-plane   98s   v1.32.8   172.18.0.4    <none>        Debian GNU/Linux 12 (bookworm)   6.6.87.2-microsoft-standard-WSL2   containerd://2.1.3

Argo CD에 dev/prd 클러스터 등록

아무런 작업없이 등록하려고 하면 kubeconfig 상에서 각 클러스터 server가 127.0.0.1:<Port>로 잡혀 있고, ArgoCD에서는 해당 주소로 접속이 안된다. 이를 위해 아래 kubeconfig 수정 작업이 필요하다.

# kind 도커 네트워크에서 각 클러스터 IP 확인
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# docker network inspect kind | grep -E 'Name|IPv4Address'
        "Name": "kind",
                "Name": "mgmt-control-plane",
                "IPv4Address": "172.18.0.2/16",
                "Name": "dev-control-plane",
                "IPv4Address": "172.18.0.3/16",
                "Name": "prd-control-plane",
                "IPv4Address": "172.18.0.4/16",
                
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl config view --minify --context kind-dev | grep server
    server: https://127.0.0.1:41557
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl config view --minify --context kind-prd | grep server
    server: https://127.0.0.1:39563

# kubeconfig 백업 후 dev/prd의 server를 컨테이너 IP로 변경
cp ~/.kube/config ~/.kube/config.bak

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# cat ~/.kube/config | grep server
    server: https://127.0.0.1:41557
    server: https://127.0.0.1:44367
    server: https://127.0.0.1:39563

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# cat ~/.kube/config | grep server -A 1
    server: https://172.18.0.3:6443
  name: kind-dev
--
    server: https://127.0.0.1:44367
  name: kind-mgmt
--
    server: https://172.18.0.4:6443
  name: kind-prd

# dev 클러스터 등록
argocd cluster add kind-dev --name dev-k8s --yes

# prd 클러스터 등록
argocd cluster add kind-prd --name prd-k8s --yes

# 확인
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd cluster list
SERVER                          NAME        VERSION  STATUS      MESSAGE                                                  PROJECT
https://172.18.0.3:6443         dev-k8s              Unknown     Cluster has no applications and is not being monitored.
https://172.18.0.4:6443         prd-k8s     1.32     Successful                                                         
https://kubernetes.default.svc  in-cluster           Unknown     Cluster has no applications and is not being monitored.

 

ApplicationSet Generators

ApplicationSet은 하나의 템플릿(Application)과, 여러 개의 “파라미터 세트”를 조합해서 다수의 Argo CD Application을 자동으로 만들어주는 컨트롤러이다.

• spec.generators: 파라미터를 만들어내는 부분 (리스트, 클러스터 목록, Git 디렉터리 구조 등)
•  spec.template: Application 템플릿 (repoURL, path, destination 등)

제너레이터 종류에 따라 “파라미터를 생성하는 방식”이 달라진다고 한다.

• List Generator – YAML 안에서 직접 파라미터 목록 선언
• Cluster Generator – 등록된 클러스터 목록 기반
• Git Directory Generator – Git 디렉터리 구조 스캔
• Git File Generator – Git 내 JSON/YAML 파일 파싱
• Matrix Generator – 여러 Generator의 조합(product) 생성
• Merge Generator – base + override 방식으로 파라미터 병합

아래는 실습해보진 않은 Argo CD에서 지원하는 제너레이터 목록이다.

• SCM Provider Generator – GitHub/GitLab 등의 repo 목록 기반
• Pull Request Generator – PR마다 preview 환경 자동 생성
• Cluster Decision Generator – 외부 스케줄러 기반 클러스터 선택
• Plugin Generator – 외부 API 기반 파라미터 생성 (최고 확장성)

Generators - Argo CD - Declarative GitOps CD for Kubernetes

실습에 사용한 source는 https://github.com/argoproj/argo-cd.git 저장소에서 제공하는 샘플코드를 사용한다.

k8s1 config use-context kind-mgmt 2>/dev/null || kubectl config use-context kind-mgmt

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# docker network inspect kind | grep -E 'Name|IPv4Address'
        "Name": "kind",
                "Name": "mgmt-control-plane",
                "IPv4Address": "172.18.0.2/16",
                "Name": "dev-control-plane",
                "IPv4Address": "172.18.0.3/16",
                "Name": "prd-control-plane",
                "IPv4Address": "172.18.0.4/16",

# 실제 출력된 IP에 맞게 아래 값 설정
DEVK8SIP=172.18.0.3
PRDK8SIP=172.18.0.4

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# echo $DEVK8SIP $PRDK8SIP
172.18.0.3 172.18.0.4

List Generator

• 역할 YAML 안에 elements 배열로 직접 리스트를 정의하고, 각 element마다 하나의 Application 생성.
• 장점
  • 구조가 단순해서 ApplicationSet 개념 파악에 최적.
  • 소규모/PoC/스터디용으로 좋음.
• 단점
  • 엔트리가 많아질수록 YAML이 금방 지저분해짐.
  • 완전히 수동(동적 X).
• 주요 옵션
  • generators[0].list.elements 아래에 자유롭게 key/value 구성.
  • 템플릿에서 {{.name}}, {{.url}}, {{.pathDir}} 등으로 접근

dev/prd guestbook 배포

각 환경(dev/prd)에 다른 Git path를 사용하고, 네임스페이스는 list-guestbook-<env>로 분리

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-list
  namespace: argocd
  labels:
    appset-demo: list
spec:
  goTemplate: true
  generators:
  - list:
      elements:
      - name: dev
        url: https://$DEVK8SIP:6443
        pathDir: applicationset/examples/list-generator/guestbook/engineering-dev
      - name: prd
        url: https://$PRDK8SIP:6443
        pathDir: applicationset/examples/list-generator/guestbook/engineering-prod
  template:
    metadata:
      name: 'list-{{.name}}-guestbook'
      labels:
        appset-demo: list
        environment: '{{.name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: '{{.pathDir}}'
      destination:
        server: '{{.url}}'
        namespace: 'list-guestbook-{{.name}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
EOF

# ApplicationSet
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applicationset -n argocd -l appset-demo=list
NAME             AGE
guestbook-list   88s

# Argo CD Applications (argocd CLI)
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd app list --selector appset-demo=list
NAME                       CLUSTER                  NAMESPACE           PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                               TARGET
argocd/list-dev-guestbook  https://172.18.0.3:6443  list-guestbook-dev  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev   HEAD
argocd/list-prd-guestbook  https://172.18.0.4:6443  list-guestbook-prd  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-prod  HEAD

# 쿠버네티스 Application 리소스
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=list
NAME                 SYNC STATUS   HEALTH STATUS
list-dev-guestbook   Synced        Healthy
list-prd-guestbook   Synced        Healthy

# dev/prd 네임스페이스 리소스
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# k8s2 get pod,svc -n list-guestbook-dev
NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-7cf4fd7cb9-v7xhb   1/1     Running   0          2m42s

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.224.103   <none>        80/TCP    2m43s

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# k8s3 get pod,svc -n list-guestbook-prd
NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-7cf4fd7cb9-9j8sr   1/1     Running   0          2m58s

NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.236.3   <none>        80/TCP    2m59s

Cluster Generator

• 역할
  • Argo CD에 등록된 클러스터 Secret 목록을 기반으로 Application 생성.
  • clusters: {} → 모든 클러스터, selector → 특정 라벨 가진 클러스터만.
• 장점
  • 클러스터를 새로 등록/삭제하면 ApplicationSet이 자동으로 App 목록을 조정.
  • 멀티 클러스터 GitOps에서 가장 자주 쓰이는 패턴.
• 단점 어떤 클러스터를 등록할지/라벨을 어떻게 붙일지” 전략이 필요.
• 주요 옵션
  • clusters: {} → 등록된 모든 클러스터
  • clusters.selector.matchLabels: → 특정 라벨 가진 클러스터만 대상

Argo CD가 아는 모든 클러스터에 guestbook 배포

네임스페이스는 cluster-all-guestbook 사용

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-clusters
  namespace: argocd
  labels:
    appset-demo: cluster-all
spec:
  goTemplate: true
  generators:
  - clusters: {}   # 등록된 모든 클러스터
  template:
    metadata:
      name: '{{.name}}-cluster-all-guestbook'
      labels:
        appset-demo: cluster-all
    spec:
      project: "default"
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: applicationset/examples/list-generator/guestbook/engineering-dev
      destination:
        server: '{{.server}}'
        namespace: cluster-all-guestbook
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
EOF

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applicationset -n argocd -l appset-demo=cluster-all
NAME                 AGE
guestbook-clusters   6s

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd app list --selector appset-demo=cluster-all
NAME                                     CLUSTER                         NAMESPACE              PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                              TARGET
argocd/dev-k8s-cluster-all-guestbook     https://172.18.0.3:6443         cluster-all-guestbook  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev  HEAD
argocd/in-cluster-cluster-all-guestbook  https://kubernetes.default.svc  cluster-all-guestbook  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev  HEAD
argocd/prd-k8s-cluster-all-guestbook     https://172.18.0.4:6443         cluster-all-guestbook  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev  HEAD

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=cluster-all
NAME                               SYNC STATUS   HEALTH STATUS
dev-k8s-cluster-all-guestbook      Synced        Healthy
in-cluster-cluster-all-guestbook   Synced        Healthy
prd-k8s-cluster-all-guestbook      Synced        Healthy

특정(dev) 클러스터에만 배포

dev 클러스터 Secret에 라벨 추가

kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster

NAME                            TYPE     DATA   AGE
cluster-172.18.0.3-4100004299   Opaque   3      117m
cluster-172.18.0.4-568336172    Opaque   3      117m


# 위 출력에서 "dev 클러스터"에 해당하는 secret 이름을 확인한 뒤 아래에 대입
# DEV_CLUSTER_SECRET=<dev-클러스터-secret-이름>
DEV_CLUSTER_SECRET=cluster-172.18.0.3-4100004299

kubectl label secret -n argocd $DEV_CLUSTER_SECRET env=dev --overwrite

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get secret -n argocd -l env=dev
NAME                            TYPE     DATA   AGE
cluster-172.18.0.3-4100004299   Opaque   3      119m

네임스페이스 cluster-dev-guestbook에 클러스터 라벨 env=dev를 selector로 사용해서 배포

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-clusters-dev-only
  namespace: argocd
  labels:
    appset-demo: cluster-dev-only
spec:
  goTemplate: true
  generators:
  - clusters:
      selector:
        matchLabels:
          env: "dev"
  template:
    metadata:
      name: 'dev-only-guestbook'
      labels:
        appset-demo: cluster-dev-only
        env: dev
    spec:
      project: "default"
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: applicationset/examples/list-generator/guestbook/engineering-dev
      destination:
        server: '{{.server}}'
        namespace: cluster-dev-guestbook
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
EOF

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applicationset -n argocd -l appset-demo=cluster-dev-only
NAME                          AGE
guestbook-clusters-dev-only   11s

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd app list --selector appset-demo=cluster-dev-only
NAME                       CLUSTER                  NAMESPACE              PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                              TARGET
argocd/dev-only-guestbook  https://172.18.0.3:6443  cluster-dev-guestbook  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev  HEAD

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=cluster-dev-only
NAME                 SYNC STATUS   HEALTH STATUS
dev-only-guestbook   Synced        Healthy

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# k8s2 get pod,svc -n cluster-dev-guestbook
NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-7cf4fd7cb9-9xlw6   1/1     Running   0          27s

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.225.228   <none>        80/TCP    27s

Git Directory Generator

• 역할 Git 레포의 디렉터리 목록을 스캔해서 각 디렉터리마다 Application 생성.
• 장점
  • Git 디렉터리 구조 = 배포 단위 구조 로 사용하는 GitOps 패턴에 적합.
  • 디렉터리 추가만으로 새 앱이 자동 생성.
• 단점 리포 구조 설계를 잘 해야 함(폴더 구조가 곧 플랫폼 구조).
• 주요 옵션
  • directories.path: 디렉터리 패턴 (예: apps/*, */overlays/*)
  • directories.exclude: 제외 디렉터리 패턴

cluster-addons 예제

git-dir-<디렉터리명>으로 네임스페이스를 사용, prometheus-operator가 너무 무겁고 네임스페이스를 생성해줘야해서 해당 부분은 제외하고 구성하도록 수정했다.

• cluster-addons/* 전체를 포함하면서
• cluster-addons/prometheus-operator만 **exclude: true**로 제외

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
  labels:
    appset-demo: git-dir
spec:
  goTemplate: true
  generators:
  - git:
      repoURL: https://github.com/argoproj/argo-cd.git
      revision: HEAD
      directories:
      # 1) cluster-addons 아래 모든 디렉터리 후보
      - path: applicationset/examples/git-generator-directory/cluster-addons/*
      # 2) 그 중 prometheus-operator 디렉터리는 제외
      - path: applicationset/examples/git-generator-directory/cluster-addons/prometheus-operator
        exclude: true
  template:
    metadata:
      name: '{{.path.basename}}'
      labels:
        appset-demo: git-dir
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'git-dir-{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
EOF

# ApplicationSet 확인
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applicationset -n argocd -l appset-demo=git-dir
NAME             AGE
cluster-addons   57s

# 이 ApplicationSet이 만든 Application만 보기
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd app list --selector appset-demo=git-dir
NAME                   CLUSTER                         NAMESPACE               PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                                           TARGET
argocd/argo-workflows  https://kubernetes.default.svc  git-dir-argo-workflows  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/git-generator-directory/cluster-addons/argo-workflows  HEAD

# 쿠버네티스 Application 리소스
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=git-dir
NAME             SYNC STATUS   HEALTH STATUS
argo-workflows   Synced        Healthy

# 실제 배포된 워크로드 (argo-workflows 쪽)
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=git-dir
NAME             SYNC STATUS   HEALTH STATUS
argo-workflows   Synced        Healthy
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get pod,svc -n git-dir-argo-workflows
NAME                                       READY   STATUS    RESTARTS   AGE
pod/argo-server-66569f88d9-m92f4           1/1     Running   0          94s
pod/workflow-controller-5fbb6d96f5-nz59m   1/1     Running   0          94s

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/argo-server   ClusterIP   10.96.255.13   <none>        2746/TCP   95s

Git File Generator

• 역할 Git 레포 내 특정 패턴에 매칭되는 파일(YAML) 하나를 Application 파라미터 하나로 사용.
• 장점 “클러스터 정의 파일 모음”, “앱 정의 파일 모음” 등을 한 번에 읽어 처리 가능.
• 단점 파일 포맷이 제너레이터가 기대하는 구조여야 함.
• 주요 옵션
  • files.path: 파일 glob 패턴
  • 템플릿 변수: {{.file.path}}, {{.file.basename}}

공식 git-generator-files 예제

네임스페이스는 git-file-guestbook-<클러스터명> 을 사용

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: git-file-discovery
  namespace: argocd
  labels:
    appset-demo: git-file-discovery
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - git:
      repoURL: https://github.com/argoproj/argo-cd.git
      revision: HEAD
      files:
      - path: "applicationset/examples/git-generator-files-discovery/cluster-config/**/config.json"
  template:
    metadata:
      name: '{{.cluster.name}}-guestbook-gitfile'
      labels:
        appset-demo: git-file-discovery
        cluster: '{{.cluster.name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: "applicationset/examples/git-generator-files-discovery/apps/guestbook"
      destination:
        # discovery 예제에서는 일단 in-cluster(mgmt)에 배포
        server: https://kubernetes.default.svc
        # 클러스터 이름별로 네임스페이스를 분리
        namespace: 'git-file-guestbook-{{.cluster.name}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true
EOF

# ApplicationSet 확인
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applicationset -n argocd -l appset-demo=git-file-discovery
NAME                 AGE
git-file-discovery   16s

# 이 ApplicationSet에서 생성된 Application들만 보기
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd app list --selector appset-demo=git-file-discovery
NAME                                       CLUSTER                         NAMESPACE                            PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                                  TARGET
argocd/engineering-dev-guestbook-gitfile   https://kubernetes.default.svc  git-file-guestbook-engineering-dev   default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/git-generator-files-discovery/apps/guestbook  HEAD
argocd/engineering-prod-guestbook-gitfile  https://kubernetes.default.svc  git-file-guestbook-engineering-prod  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/git-generator-files-discovery/apps/guestbook  HEAD

# 쿠버네티스 Application 리소스
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=git-file-discovery
NAME                                 SYNC STATUS   HEALTH STATUS
engineering-dev-guestbook-gitfile    Synced        Healthy
engineering-prod-guestbook-gitfile   Synced        Healthy


(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get pod,svc -n git-file-guestbook-engineering-dev   || true
NAME                                          READY   STATUS    RESTARTS   AGE
pod/kustomize-guestbook-ui-7cf4fd7cb9-4mhrn   1/1     Running   0          108s

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kustomize-guestbook-ui   ClusterIP   10.96.98.117   <none>        80/TCP    108s
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~#
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get pod,svc -n git-file-guestbook-engineering-prod   || true
NAME                                          READY   STATUS    RESTARTS   AGE
pod/kustomize-guestbook-ui-7cf4fd7cb9-wpllk   1/1     Running   0          2m2s

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kustomize-guestbook-ui   ClusterIP   10.96.84.204   <none>        80/TCP    2m2s

Matrix Generator

• 역할
  • 두 개(이상)의 제너레이터 결과를 곱(product) 해서 조합 생성.
  • 예: env(dev/prd) × 애드온 목록.
• 장점 “환경 × 애플리케이션”, “클러스터 × 애플리케이션” 조합 만들 때 강력.
• 단점 조합 수 폭발에 주의(설계 필요).
• 주요 옵션 matrix.generators[]: 조합에 사용할 서브 제너레이터들

env(dev/prd) × guestbook 으로 배포

네임스페이스 matrix-<env>-<basename>-guestbook 으로 배포 

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: matrix-sample
  namespace: argocd
  labels:
    appset-demo: matrix
spec:
  goTemplate: true
  generators:
  - matrix:
      generators:
      # 1) Git Directory Generator: guestbook 하위 디렉터리들
      - git:
          repoURL: https://github.com/argoproj/argo-cd.git
          revision: HEAD
          directories:
          - path: applicationset/examples/list-generator/guestbook/*
      # 2) List Generator: env(dev/prd)
      - list:
          elements:
          - env: dev
          - env: prd
  template:
    metadata:
      name: 'matrix-{{.env}}-{{.path.basename}}-guestbook'
      labels:
        appset-demo: matrix
        env: '{{.env}}'
        app: '{{.path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'matrix-{{.env}}-{{.path.basename}}-guestbook'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
EOF

# ApplicationSet
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applicationset -n argocd -l appset-demo=matrix
NAME            AGE
matrix-sample   16s

# Argo CD Applications
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd app list --selector appset-demo=matrix
NAME                                          CLUSTER                         NAMESPACE                              PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                               TARGET
argocd/matrix-dev-engineering-dev-guestbook   https://kubernetes.default.svc  matrix-dev-engineering-dev-guestbook   default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev   HEAD
argocd/matrix-dev-engineering-prod-guestbook  https://kubernetes.default.svc  matrix-dev-engineering-prod-guestbook  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-prod  HEAD
argocd/matrix-prd-engineering-dev-guestbook   https://kubernetes.default.svc  matrix-prd-engineering-dev-guestbook   default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev   HEAD
argocd/matrix-prd-engineering-prod-guestbook  https://kubernetes.default.svc  matrix-prd-engineering-prod-guestbook  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-prod  HEAD

# Kubernetes Application 리소스
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=matrix
NAME                                    SYNC STATUS   HEALTH STATUS
matrix-dev-engineering-dev-guestbook    Synced        Healthy
matrix-dev-engineering-prod-guestbook   Synced        Healthy
matrix-prd-engineering-dev-guestbook    Synced        Healthy
matrix-prd-engineering-prod-guestbook   Synced        Healthy

# 실제 생성된 네임스페이스
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get ns | grep 'matrix-.*-guestbook' || true
matrix-dev-engineering-dev-guestbook    Active   52s
matrix-dev-engineering-prod-guestbook   Active   52s
matrix-prd-engineering-dev-guestbook    Active   52s
matrix-prd-engineering-prod-guestbook   Active   52s

# 네임스페이스나 별 리소스 확인
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get pod,svc -n matrix-dev-engineering-dev-guestbook   || true
NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-7cf4fd7cb9-cnnmf   1/1     Running   0          58s

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.241.133   <none>        80/TCP    58s
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~#
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~#
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get pod,svc -n matrix-prd-engineering-prod-guestbook  || true
NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-7cf4fd7cb9-m5lsd   1/1     Running   0          64s

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.146.177   <none>        80/TCP    64s
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~#
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get pod,svc -n matrix-dev-engineering-prod-guestbook   || true
NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-7cf4fd7cb9-ml8mq   1/1     Running   0          86s

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.117.119   <none>        80/TCP    86s
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~#
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get pod,svc -n matrix-prd-engineering-dev-guestbook  || true
NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-7cf4fd7cb9-cp5c7   1/1     Running   0          99s

NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.83.48   <none>        80/TCP    99s​

추가사항) 기존 실습의 cluster-addon을 사용할 때 아래의 내용을 주의

• PriorityClass, ClusterRole, ClusterRoleBinding, CustomResourceDefinition 같은 건 클러스터 스코프 리소스라서
  → “한 클러스터에서 하나의 Application만 관리”하게 두는 게 안전하다고 생각됨.
• Matrix/Directory 제너레이터는 가능하면 namespaced 리소스를 가진 매니페스트를 대상으로 쓰는 게 좋아보임.

Merge Generator

• 역할
  • 여러 제너레이터가 만든 파라미터를 mergeKeys 기준으로 병합(override).
  • 앞쪽 제너레이터 = base, 뒤쪽 = override 로 생각.
• 장점 공통 설정과 환경별/클러스터별 override를 분리해서 관리 가능.
• 단점 머지 규칙이 복잡해지면 디버깅 어려울 수 있음.
• 주요 옵션
  • merge.mergeKeys: 어떤 키 기준으로 합칠지 (예: name, cluster, app)
  • merge.generators: base → override 순서대로 제너레이터 나열

dev/prd 중 prd만 replicas override

네임스페이스는 merge-guestbook-<env>를 사용, name 키 기준으로 base/override 머지.

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: merge-sample
  namespace: argocd
  labels:
    appset-demo: merge
spec:
  goTemplate: true
  generators:
  - merge:
      mergeKeys:
      - name
      generators:
      - list:
          elements:
          - name: dev
            replicas: 1
            url: https://$DEVK8SIP:6443
          - name: prd
            replicas: 1
            url: https://$PRDK8SIP:6443
      - list:
          elements:
          - name: prd
            replicas: 3
  template:
    metadata:
      name: 'merge-{{.name}}-r{{.replicas}}-guestbook'
      labels:
        appset-demo: merge
        env: '{{.name}}'
        replicas: '{{.replicas}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: applicationset/examples/list-generator/guestbook/engineering-dev
      destination:
        server: '{{.url}}'
        namespace: 'merge-guestbook-{{.name}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
EOF

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applicationset -n argocd -l appset-demo=merge
NAME           AGE
merge-sample   15s

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd app list --selector appset-demo=merge
NAME                        CLUSTER                  NAMESPACE            PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                              TARGET
argocd/merge-dev-guestbook  https://172.18.0.3:6443  merge-guestbook-dev  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev  HEAD
argocd/merge-prd-guestbook  https://172.18.0.4:6443  merge-guestbook-prd  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev  HEAD

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=merge
NAME                  SYNC STATUS   HEALTH STATUS
merge-dev-guestbook   Synced        Healthy
merge-prd-guestbook   Synced        Healthy

(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# k8s2 get ns | grep merge-guestbook- || true
merge-guestbook-dev     Active   88s
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# k8s3 get ns | grep merge-guestbook- || true
merge-guestbook-prd     Active   90s

List Generator로만 처리하려면 dev/prd용 리스트를 전부 개별적으로 써야 하지만 Merge Generator를 사용하면

• base 목록: 모든 환경의 공통값
• override 목록: 특정 환경만 수정할 값

을 mergeKeys 기준으로 합쳐서 하나의 환경별 파라미터로 만들어준다.

이 실습에서는 prd의 replicas의 3 항목만 prd에 병합되어 반영된다. 다만 list generator 예제의 guestbook 매니페스트(Deployment)에서는 replicas: {{ .Values.replicas }} 와 같은 식으로 반영하지 않고 있기 때문에 실제 pod의 개수가 변경되진 않은것이다.

# 앱 이름에 r1 / r3가 들어갔는지 확인
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# argocd app list --selector appset-demo=merge
NAME                           CLUSTER                  NAMESPACE            PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                              TARGET
argocd/merge-dev-r1-guestbook  https://172.18.0.3:6443  merge-guestbook-dev  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev  HEAD
argocd/merge-prd-r3-guestbook  https://172.18.0.4:6443  merge-guestbook-prd  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev  HEAD

# 라벨에 replicas 값이 다른지 확인
(⎈|kind-mgmt:N/A) root@DESKTOP-O4EPQ9T:~# kubectl get applications -n argocd -l appset-demo=merge -o jsonpath='{range .items[*]}{.metadata.name}{" => replicas="}{.metadata.labels.replicas}{"\n"}{end}'
merge-dev-r1-guestbook => replicas=1
merge-prd-r3-guestbook => replicas=3

 

테스트에 사용한 ApplicationSet 삭제

kubectl config use-context kind-mgmt

kubectl delete applicationset guestbook-list -n argocd --ignore-not-found
kubectl delete applicationset guestbook-clusters -n argocd --ignore-not-found
kubectl delete applicationset guestbook-clusters-dev-only -n argocd --ignore-not-found
kubectl delete applicationset cluster-addons -n argocd --ignore-not-found
kubectl delete applicationset git-files-example -n argocd --ignore-not-found
kubectl delete applicationset git-file-discovery -n argocd --ignore-not-found
kubectl delete applicationset matrix-sample -n argocd --ignore-not-found
kubectl delete applicationset merge-sample -n argocd --ignore-not-found