使用gitlab自带的ci/cd实现.net6部署到k8s

第一步: 安装gitlab,这个自然不用多说

第二步: 使用docker容器安装gitlab-runner组件

新建一个docker-compose.yml文件,内容如下,在linux系统下使用 docker-compose up -d 命令跑起来就好(如果不会,可度娘搜索安装和使用)

version: "3.3"
services:
  gitlab-runner:
    image:  gitlab/gitlab-runner:latest
    # user: root
    container_name: gitlab-runner
    #总是重启后启动
    restart: always
    privileged: true
    volumes:
      - ./config:/etc/gitlab-runner
      - /root/.docker:/var/gitlab_home/.docker
      - /usr/bin/docker:/usr/bin/docker
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/docker/certs.d/reg.frame4j.local/reg.frame4j.local.crt:/certs/reg.frame4j.local.crt
      - /etc/docker/daemon.json:/etc/docker/daemon.json
    networks:
      - gitlab_runner_net

networks:
  gitlab_runner_net:
    driver: bridge

其中

- /etc/docker/certs.d/reg.frame4j.local/reg.frame4j.local.crt:/certs/reg.frame4j.local.crt 为挂载的私服harbor证书.

第三步: 安装完之后,需要将安装好的gitlab-runner注册到gitlab中去,这里我们就需要用到如下图中红色框框内的token令牌

温馨提示:从图1中进入图2中红色框框项勾选上,则可以即使提交代码时没有tag标签也可以支持构建

四、添加gitlab-runner账号授权

1、添加一个 gitlab-runner 用户(一般这个用户是不存在的)

sudo adduser gitlab-runner

2、将gitlab-runner用户添加到docker组内

sudo gpasswd -a gitlab-runner docker

3、查看docker组中是否已经成功(如果出现:docker:x:973:root,gitlab-runner则成功)

cat /etc/group |grep docker

4、重启docker容器服务

sudo systemctl restart docker

5、给docker.sock设置权限

sudo chmod a+rw /var/run/docker.sock

五、创建文件名为 .gitlab-ci.yml 的文件

在项目根目录下创建文件名为 .gitlab-ci.yml 的文件,此文件是用来构建打包部署的

stages:
   - build
   - deploy

build:
  stage: build
  script:
     - cd ./
     - docker build -f "./Dockerfile"  -t  reg.frame4j.local/nslxh/hsapi-gitlabcicd:latest --label "com.microsoft.created-by=lxh" --label "com.microsoft.visual-studio.project-name=HansWebApi" "./"
     - docker login -uadmin -pDz666666 reg.frame4j.local
     - docker push reg.frame4j.local/nslxh/hsapi-gitlabcicd:latest
     - docker rmi reg.frame4j.local/nslxh/hsapi-gitlabcicd:latest
     - docker image prune -f
  only:
     - master


deploy:
  stage: deploy
  script:
     - cd ./  
     - pwd
     - ls 
     - whoami
     - scp ./deployment.yml root@192.168.9.100:/root/
     - ssh root@192.168.9.100 'kubectl apply -f /root/deployment.yml'
     - ssh root@192.168.9.100 'kubectl rollout restart  deployment erapi'
     - ssh root@192.168.9.100 'kubectl rollout restart  deployment erapi2'
     - ssh root@192.168.9.100 'kubectl rollout restart  deployment erapi3' 
  only:
     - master

deployment.yml文件内容如下:

# 第一个k3cloud账号部署webapi
apiVersion: apps/v1
kind: Deployment
metadata:
  name: erapi
  labels:
    app: erapi
spec:
  # 配置rs,初始副本数
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: erapi
  ## 配置是POD模板
  template:
    metadata:
      labels:
        app: erapi
        env: erapi-test
    spec:
      containers:
      - name: erapi-container
        volumeMounts:
          - name: harbor-certs
            mountPath: /etc/harbor-certs #harbor自定义证书在k8s中的挂载
            readOnly: true
        imagePullPolicy: Always
        # image: registry.cn-hangzhou.aliyuncs.com/nslxh/hsapi-bl:latest
        # image: reg.frame4j.local/nslxh/hsapi-bl:latest
        image: reg.frame4j.local/nslxh/hsapi-gitlabcicd:latest
        # 容器资源限制
        resources:
          limits:
            cpu: "1"
            memory: "1Gi"
          requests:
            cpu: "0.5"
            memory: "512Mi"
        ports:
        - name: erapi-port
        # 必须和dockerfile中暴露端口一致
          containerPort: 80 
      imagePullSecrets:
      # harbor登录在k8s中的生成的secret: kubectl create secret docker-registry harborusrpwd --docker-server=reg.frame4j.local --docker-username=admin --docker-password=Dz666666 --docker-email=haikuang@126.com
        - name: harborusrpwd 
      volumes:
        - name: harbor-certs
          secret:
           # harbor自定义证书在k8s中的挂载: kubectl create secret generic harborkey  --from-file=/usr/local/share/ca-certificates/reg.frame4j.local.crt
           # ,其中reg.frame4j.local.crt  为harbor生成的自定义证书
            secretName: harborkey 

---
apiVersion: v1
kind: Service
metadata:
  name: erapi
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30301
  selector:
    app: erapi

---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: erapi-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: erapi
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

# 第一个k3cloud账号部署 ingress负载均衡器
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: erapi-ingress
spec:
  rules:
    - host: k8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: erapi
                port:
                  number: 80

# 第二个个k3cloud账号部署webapi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: erapi2
  labels:
    app: erapi2
spec:
  # 配置rs,初始副本数
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: erapi2
  ## 配置是POD模板
  template:
    metadata:
      labels:
        app: erapi2
        env: erapi2-test
    spec:
      containers:
      - name: erapi2-container
        volumeMounts:
          - name: harbor-certs
            mountPath: /etc/harbor-certs #harbor自定义证书在k8s中的挂载
            readOnly: true
        imagePullPolicy: Always
        # image: registry.cn-hangzhou.aliyuncs.com/nslxh/hsapi-bl:latest
         # image: reg.frame4j.local/nslxh/hsapi-bl:latest
        image: reg.frame4j.local/nslxh/hsapi-gitlabcicd:latest
        # 容器资源限制
        resources:
          limits:
            cpu: "1"
            memory: "1Gi"
          requests:
            cpu: "0.5"
            memory: "512Mi"
        ports:
        - name: hsapi2-port
        # 必须和dockerfile中暴露端口一致
          containerPort: 80 
      imagePullSecrets:
      # harbor登录在k8s中的生成的secret: kubectl create secret docker-registry harborusrpwd --docker-server=reg.frame4j.local --docker-username=admin --docker-password=Dz666666 --docker-email=haikuang@126.com
        - name: harborusrpwd 
      volumes:
        - name: harbor-certs
          secret:
           # harbor自定义证书在k8s中的挂载: kubectl create secret generic harborkey  --from-file=/usr/local/share/ca-certificates/reg.frame4j.local.crt
           # ,其中reg.frame4j.local.crt  为harbor生成的自定义证书
            secretName: harborkey 

---
apiVersion: v1
kind: Service
metadata:
  name: erapi2
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30302
  selector:
    app: erapi2


---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: erapi2-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: erapi2
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

# 第二个k3cloud账号部署ingress负载均衡器
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: erapi2-ingress
spec:
  rules:
    - host: k2.k8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: erapi2
                port:
                  number: 80

# 第三个个k3cloud账号部署webapi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: erapi3
  labels:
    app: erapi3
spec:
  # 配置rs,初始副本数
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: erapi3
  ## 配置是POD模板
  template:
    metadata:
      labels:
        app: erapi3
        env: erapi3-test
    spec:
      containers:
      - name: erapi3-container
        volumeMounts:
          - name: harbor-certs
            mountPath: /etc/harbor-certs #harbor自定义证书在k8s中的挂载
            readOnly: true
        imagePullPolicy: Always
        # image: registry.cn-hangzhou.aliyuncs.com/nslxh/hsapi-bl:latest
        # image: reg.frame4j.local/nslxh/hsapi-bl:latest
        image: reg.frame4j.local/nslxh/hsapi-gitlabcicd:latest
        # 容器资源限制
        resources:
          limits:
            cpu: "1"
            memory: "1Gi"
          requests:
            cpu: "0.5"
            memory: "512Mi"
        ports:
        - name: erapi3-port
        # 必须和dockerfile中暴露端口一致
          containerPort: 80 
      imagePullSecrets:
      # harbor登录在k8s中的生成的secret: kubectl create secret docker-registry harborusrpwd --docker-server=reg.frame4j.local --docker-username=admin --docker-password=Dz666666 --docker-email=haikuang@126.com
        - name: harborusrpwd 
      volumes:
        - name: harbor-certs
          secret:
           # harbor自定义证书在k8s中的挂载: kubectl create secret generic harborkey  --from-file=/usr/local/share/ca-certificates/reg.frame4j.local.crt
           # ,其中reg.frame4j.local.crt  为harbor生成的自定义证书
            secretName: harborkey 

---
apiVersion: v1
kind: Service
metadata:
  name: erapi3
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30303
  selector:
    app: erapi3


---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: erapi3-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: erapi3
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

# 第三个k3cloud账号部署ingress负载均衡器
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: erapi3-ingress
spec:
  rules:
    - host: k3.k8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: erapi3
                port:
                  number: 80

我这里把构建分为了两个阶段,一个是编译,另外一个是部署

1、build阶段负责把源代码编译成一个容器镜像.这里需要注意的地方就是 cd 只cd到根目录下,不要直接cd到Dockerfile文件所在目录,而是在编译命令中指定Dockerfile文件目录

2、根据编译好的容器镜像启动容器.这里需要注意的是需要大家多一步判断如果容器为新建则要排除停止和删除容器,否则会出现部署不通过的错误

六、使用git提交代码就会自动触发构建

当每一步都是绿色时,则表示构建成功，如下图

补充:

1.在DOCKER容器中安装 docker-compose 的文档地址:https://docs.docker.com/compose/install/

2.解决 gitlab-runner容器部署时,使用scp,ssh命令时 Host key verification failed 的报错问题

安装 gitlab-runner 构建机默认会将用户设置为 gitlab-runner，该设置会导致.gitlab-ci.yml 脚本运行时出现一些权限问题。

a.为了解决这些权限问题，将 gitlab-runner 构建机上的默认用户设置为 root

b或者在gitlab-runner容器内切换到gitlab-runner用户,然后再执行如下命令:

#换到gitlab-runner用户

su gitlab-runner

#gitlab-runner容器生成密钥对

ssh-keygen -t rsa

#远程拷贝到k8s的master机器，建立免密

ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.9.100 #复制密钥

#执行如下ssh命令,并输入远程目标主机的登录密码,后面再CICD就OK了.

ssh root@192.168.9.100

3.

chmod 777 /var/run/docker.sock

最终api部署成功: