k8s 折腾笔记

部署 GitLab 并支持基于 22 端口的 SSH

在 M1 部署 GitLab 真不容易，不过好在成功了。

构建适用于 M1 的 GitLab 镜像

一般情况下，使用 sameersbn/docker-gitlabopen in new window 这个项目提供的镜像即可，但该镜像不支持 arm64 架构，因此需要自己构建。

首先，拉取源码：

git clone [email protected]:sameersbn/docker-gitlab.git

根据这个 PRopen in new window 修改 assets/build/install.sh 文件

然后，运行如下构建命令：

docker buildx build -t listenzz/gitlab:16.1.0 --platform linux/arm64 .

大概需要十几分钟，构建完成后，运行如下命令，将镜像推送到 Docker Hub：

docker push listenzz/gitlab:16.1.0

listenzz 是作者在 Docker Hub 上的用户名

16.1.0 是作者 clone 该项目时的最新版本

构建 PostgreSQL 镜像

sameersbn/docker-gitlabopen in new window 的作者还提供了一个 PostgreSQL 镜像，但该镜像也不支持 arm64 架构，因此也需要自己构建。

首先，拉取源码：

git clone [email protected]:sameersbn/docker-postgresql.git

然后，运行如下构建命令：

docker build -t listenzz/postgresql:14 .

docker build 默认使用和当前 CPU 架构相同的架构，作者在 M1 上构建，就会得到一个 arm64 架构的镜像。

最后，运行如下命令，将镜像推送到 Docker Hub：

docker push listenzz/postgresql:14

部署 GitLab

镜像构建好后，就可以开始部署 GitLab 了。

部署参考 sameersbn/docker-gitlabopen in new window 项目 kubernetes 目录下的清单，但需要做一些修改。尤其是将镜像修改为自己构建的镜像。

创建 NFS 存储

前往 NFS 服务器，在 /mnt/nfs 目录下创建 git 子目录

mkdir -p /mnt/nfs/git/redis
mkdir -p /mnt/nfs/git/postgresql
mkdir -p /mnt/nfs/git/data

创建名字空间

kubectl create ns gitlab

部署 redis

## redis-storage.yml
## PV
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis
  labels:
    app: redis
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    server: nfs.todoit.tech
    path: /mnt/nfs/git/redis
---
## PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: redis
spec:
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteOnce
  selector:
    matchLabels:
      app: redis

# redis-rc.yml
apiVersion: v1
kind: ReplicationController
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    name: redis
  template:
    metadata:
      name: redis
      labels:
        name: redis
    spec:
      containers:
        - name: redis
          image: redis:6.2.6
          ports:
            - name: redis
              containerPort: 6379
          volumeMounts:
            - mountPath: /var/lib/redis
              name: data
          livenessProbe:
            exec:
              command:
                - redis-cli
                - ping
            initialDelaySeconds: 30
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command:
                - redis-cli
                - ping
            initialDelaySeconds: 5
            timeoutSeconds: 1
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: redis

# redis-svc.yml
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    name: redis
spec:
  ports:
    - name: redis
      port: 6379
      targetPort: redis
  selector:
    name: redis

部署 postgresql

# postgresql-storage.yml
## PV
apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgresql
  labels:
    app: postgresql
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    server: nfs.todoit.tech
    path: /mnt/nfs/git/postgresql
---
## PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgresql
spec:
  resources:
    requests:
      storage: 20Gi
  accessModes:
    - ReadWriteOnce
  selector:
    matchLabels:
      app: postgresql

# postgresql-rc.yml
apiVersion: v1
kind: ReplicationController
metadata:
  name: postgresql
spec:
  replicas: 1
  selector:
    name: postgresql
  template:
    metadata:
      name: postgresql
      labels:
        name: postgresql
    spec:
      containers:
        - name: postgresql
          image: listenzz/postgresql:14
          env:
            - name: DB_USER
              value: gitlab
            - name: DB_PASS
              value: passw0rd
            - name: DB_NAME
              value: gitlab_production
            - name: DB_EXTENSION
              value: pg_trgm
          ports:
            - name: postgres
              containerPort: 5432
          volumeMounts:
            - mountPath: /var/lib/postgresql
              name: data
          livenessProbe:
            exec:
              command:
                - pg_isready
                - -h
                - localhost
                - -U
                - postgres
            initialDelaySeconds: 30
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command:
                - pg_isready
                - -h
                - localhost
                - -U
                - postgres
            initialDelaySeconds: 5
            timeoutSeconds: 1
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: postgresql

















 







































# postgresql-svc.yml
apiVersion: v1
kind: Service
metadata:
  name: postgresql
  labels:
    name: postgresql
spec:
  ports:
    - name: postgres
      port: 5432
      targetPort: postgres
  selector:
    name: postgresql

部署 gitlab

# gitlab-storage.yml
## PV
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gitlab
  labels:
    app: gitlab
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    server: nfs.todoit.tech
    path: /mnt/nfs/git/data
---
## PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gitlab
spec:
  resources:
    requests:
      storage: 100Gi
  accessModes:
    - ReadWriteOnce
  selector:
    matchLabels:
      app: gitlab

# gitlab-rc.yml
apiVersion: v1
kind: ReplicationController
metadata:
  name: gitlab
spec:
  replicas: 1
  selector:
    name: gitlab
  template:
    metadata:
      name: gitlab
      labels:
        name: gitlab
    spec:
      containers:
        - name: gitlab
          image: listenzz/gitlab:16.1.0
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: GITLAB_TIMEZONE
              value: Beijing

            - name: GITLAB_SECRETS_DB_KEY_BASE
              value: long-and-random-alpha-numeric-string
            - name: GITLAB_SECRETS_SECRET_KEY_BASE
              value: long-and-random-alpha-numeric-string
            - name: GITLAB_SECRETS_OTP_KEY_BASE
              value: long-and-random-alpha-numeric-string

            - name: GITLAB_ROOT_PASSWORD
              value: 'iam59!z$'
            - name: GITLAB_ROOT_EMAIL
              value: [email protected]

            - name: GITLAB_HOST
              value: git.todoit.tech
            - name: GITLAB_PORT
              value: '80'
            - name: GITLAB_SSH_PORT
              value: '22'

            - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
              value: 'true'
            - name: GITLAB_NOTIFY_PUSHER
              value: 'false'

            - name: GITLAB_BACKUP_SCHEDULE
              value: daily
            - name: GITLAB_BACKUP_TIME
              value: 01:00

            - name: DB_TYPE
              value: postgres
            - name: DB_HOST
              value: postgresql
            - name: DB_PORT
              value: '5432'
            - name: DB_USER
              value: gitlab
            - name: DB_PASS
              value: passw0rd
            - name: DB_NAME
              value: gitlab_production

            - name: REDIS_HOST
              value: redis
            - name: REDIS_PORT
              value: '6379'

            - name: SMTP_ENABLED
              value: 'false'
            - name: SMTP_DOMAIN
              value: www.example.com
            - name: SMTP_HOST
              value: smtp.gmail.com
            - name: SMTP_PORT
              value: '587'
            - name: SMTP_USER
              value: [email protected]
            - name: SMTP_PASS
              value: password
            - name: SMTP_STARTTLS
              value: 'true'
            - name: SMTP_AUTHENTICATION
              value: login

            - name: IMAP_ENABLED
              value: 'false'
            - name: IMAP_HOST
              value: imap.gmail.com
            - name: IMAP_PORT
              value: '993'
            - name: IMAP_USER
              value: [email protected]
            - name: IMAP_PASS
              value: password
            - name: IMAP_SSL
              value: 'true'
            - name: IMAP_STARTTLS
              value: 'false'
          ports:
            - name: https
              containerPort: 443
            - name: http
              containerPort: 80
            - name: ssh
              containerPort: 22
          volumeMounts:
            - mountPath: /home/git/data
              name: data
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 180
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            timeoutSeconds: 1
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: gitlab

















 














































































































# gitlab-svc.yml
apiVersion: v1
kind: Service
metadata:
  name: gitlab
  labels:
    name: gitlab
spec:
  type: ClusterIP
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: https
    - name: http
      protocol: TCP
      port: 80
      targetPort: http
    - name: ssh
      port: 22
      targetPort: ssh
  selector:
    name: gitlab

配置域名和证书

kubectl apply -f ingress.yaml

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab-ingress
  namespace: gitlab
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-dns01 # 配置自动生成 https 证书
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/proxy-body-size: '0'
spec:
  tls:
    - hosts:
        - 'git.todoit.tech'
      secretName: gitlab-letsencrypt-tls
  rules:
    - host: git.todoit.tech
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gitlab
                port:
                  number: 80

域名映射

README-2022-06-23-17-27-52

等待部署完成

kubectl get pods -n gitlab -w
# NAME                      READY   STATUS    RESTARTS   AGE
# gitlab-79cbb585c4-kgk5w   0/1     Running   0          2m8s
# gitlab-79cbb585c4-kgk5w   1/1     Running   0          3m11s

等待证书生成

kubectl get certificate -n gitlab -w
# NAME                     READY   SECRET                   AGE
# gitlab-letsencrypt-tls   True    gitlab-letsencrypt-tls   3m34s

现在可以通过域名 git.todoit.techopen in new window 访问 GitLab 了。

首先通过用户名，密码登录进去

用户名和密码都在上面的 yaml 文件中配置

README-2022-06-23-18-05-25

配置 GitLab

修改默认分支

README-2022-06-23-18-09-01

自定义 clone URL，记得保存变更。

README-2022-06-23-20-08-23

支持 SSH

配置 ingress-nginx 22 端口

修改 ingress-ngnix 配置，添加 ssh 端口

ingress-nginx 的安装请参考这里

# nginx/values.yaml
tcp:
  22: 'gitlab/gitlab:22'

然后更新 ingress-nginx

# 在 nginx/values.yaml 所在目录下执行
helm upgrade ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx -f ./values.yaml --version 4.5.2

添加 SSH KEY 到 GitLab

如何配置 SSH KEY，可以参考这里open in new window

配置好 SSH KEY 后，打开终端，输入如下命令，成功后，就可以愉快使用 SSH 了。

ssh -T [email protected]

README-2022-06-23-20-12-03

参考：

上次更新: