使用Operator Sdk开发k8s Operator

使用operator sdk开发k8s Operator

环境安装

创建项目

  • 创建项目
    1
    operator-sdk new operator-demo --repo github.com/qianbaidu/operator-demo
  • 创建Api / controller 
    1
    2
    3
    cd operator-demo
    operator-sdk add api --api-version=app.qipajun.com/v1alpha1 --kind=AppService
    operator-sdk add controller --api-version=app.qipajun.com/v1alpha1 --kind=AppService
  • 生成目录结构
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    ➜  operator-demo tree .
    .
    ├── build # 构建编译目录
    │   ├── Dockerfile
    │   └── bin
    │       ├── entrypoint
    │       └── user_setup
    ├── cmd # 启动入口
    │   └── manager
    │       └── main.go
    ├── deploy # 部署配置目录
    │   ├── crds
    │   │   ├── app.qipajun.com_appservices_crd.yaml
    │   │   └── app.qipajun.com_v1alpha1_appservice_cr.yaml
    │   ├── operator.yaml
    │   ├── role.yaml
    │   ├── role_binding.yaml
    │   └── service_account.yaml
    ├── go.mod
    ├── go.sum
    ├── pkg
    │   ├── apis
    │   │   ├── addtoscheme_app_v1alpha1.go
    │   │   ├── apis.go
    │   │   └── app
    │   │       ├── group.go
    │   │       └── v1alpha1
    │   │           ├── appservice_types.go  # api struct定义
    │   │           ├── doc.go
    │   │           ├── register.go
    │   │           └── zz_generated.deepcopy.go
    │   └── controller
    │       ├── add_appservice.go
    │       ├── appservice
    │       │   └── appservice_controller.go # 控制器
    │       └── controller.go
    ├── tools.go
    └── version
        └── version.go
  • 生成项目主要目录介绍：
    • build： 编译构建目录
    • cmd： 项目启动入口
    • deploy： 部署构建配置文件，生成k8s资源定义和配置yaml
    • pkg/apis: api资源定义
    • pkg/controller：资源控制器，实现资源控制操作

这里主要有两块关注：api和controller，

  • api定义好资源类型
  • controller中主要实现Reconcile方法，实现接口，实现配置读取，资源生命周期维护管理，这点和基于kubebuilder开发CRD一样

开始开发：

  • 调试、运行思路：
    • 1、创建k8s资源： deploy目录下已经有资源配置了，apply到k8s集群中，创建资源
    • 2、运行项目（本地开发能连入测试k8s集群），项目运行后会watch k8s资源变更可实时获取变更，即可代码调试、开发了
    • 3、开发调试完成后、编译打包镜像，修改yaml镜像地址，即可部署到k8s集群

部署运行开发测试环境：

  • 1、创建角色权限

    1
    2
    3
    4
    5
    6
    ➜  kubectl apply -f role.yaml
    role.rbac.authorization.k8s.io/operator-demo created
    ➜  kubectl apply -f role_binding.yaml
    rolebinding.rbac.authorization.k8s.io/operator-demo created
    ➜  kubectl apply -f service_account.yaml
    serviceaccount/operator-demo created

  • 2、创建自定义资源

    1
    2
    ➜  kubectl apply -f app.qipajun.com_appservices_crd.yaml
    customresourcedefinition.apiextensions.k8s.io/appservices.app.qipajun.com created

  • 查看资源

    1
    2
    3
    ➜  kubectl get crd -A
    NAME                          CREATED AT
    appservices.app.qipajun.com   2020-04-19T04:19:27Z

  • 3、运行项目代码

    • 先加几个log点方便查看
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      # log组件暂时用自己导入的 import中添加导入
      	loger "github.com/sirupsen/logrus"

      #  pkg/controller/appservice/appservice_controller.go Reconcile方法中将get到的instace log打印出来
      	err := r.client.Get(context.TODO(), request.NamespacedName, instance)
      	if err != nil {
      		loger.Error(err, "client get instance error")
      		if errors.IsNotFound(err) {
      			// Request object not found, could have been deleted after reconcile request.
      			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
      			// Return and don't requeue
      			loger.Error(err, "client get instance not found")
      			return reconcile.Result{}, nil
      		}
      		// Error reading the object - requeue the request.
      		return reconcile.Result{}, err
      	} else {
      		i, _ := json.Marshal(instance)
      		loger.Info("get instance info :", string(i))
      	}

  • 运行代码

    1
    2
    3
    4
    ➜  make run
    go mod download
    export WATCH_NAMESPACE=default && go run cmd/manager/main.go
    ...

  • 创建一个自定义的operator AppService类型资源

    1
    2
    ➜ kubectl apply -f deploy/crds/app.qipajun.com_v1alpha1_appservice_cr.yaml
    appservice.app.qipajun.com/example-appservice created

  • 此时查看代码运行log

    1
    2
    3
    4
    5
    INFO[0001] Reconcile
    INFO[0001] get instance info :{"kind":"AppService","apiVersion":"app.qipajun.com/v1alpha1","metadata":{"name":"example-appservice","namespace":"default","selfLink":"/apis/app.qipajun.com/v1alpha1/namespaces/default/appservices/example-appservice","uid":"d52ae6db-8200-11ea-9367-0800275e4e86","resourceVersion":"143948","generation":1,"creationTimestamp":"2020-04-19T05:44:27Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"app.qipajun.com/v1alpha1\",\"kind\":\"AppService\",\"metadata\":{\"annotations\":{},\"name\":\"example-appservice\",\"namespace\":\"default\"},\"spec\":{\"size\":3}}\n"}},"spec":{},"status":{}}
    INFO[0001] Set AppService instance as the owner and controller success
    {"level":"info","ts":1587275589.3622541,"logger":"controller_appservice","msg":"Skip reconcile: Pod already exists","Request.Namespace":"default","Request.Name":"example-appservice","Pod.Namespace":"default","Pod.Name":"example-appservice-pod"}
    {"level":"info","ts":1587275589.362294,"logger":"controller_appservice","msg":"Reconciling AppService","Request.Namespace":"default","Request.Name":"example-appservice"}

  • 当创建、更新、删除时都会触Reconcile发这个方法，所以可以根据自己需求在这个方式实现资源创建、更新、删除、容错判断处理

构建部署自己的operator部署到k8s

开发完需求代码后，执行编译、构建docker镜像，并修改opreator yaml镜像地址部署到集群就可以实时watch资源变更执行一系列操作了

  • 构建、推送镜像
    1
    2
    operator-sdk build liuhaogui/operator-demo
    docker push liuhaogui/operator-demo
  • 本地docker推送来回拉网络太慢使用docker save和load来加载镜像
    1
    2
    3
    4
    5
    6
    7
    # 导出镜像、scp到目标node上
    docker save liuhaogui/operator-demo:v0.0.1 > operator-demo:v0.0.1.tar
    scp ./operator-demo:v0.0.1.tar root@192.168.99.101:/root

    # 目标node上加载docker镜像
    docker load -i operator-demo\:v0.0.1.tar

修改镜像地址，部署自定义operator

  • 修改镜像地址

    1
    2
    3
    4
    # replace image url
    sed -i "s|REPLACE_IMAGE|liuhaogui/operator-demo:v0.0.1|g" deploy/operator.yaml
    # On OSX use:
    sed -i "" "s|REPLACE_IMAGE|liuhaogui/operator-demo:v0.0.1|g" deploy/operator.yaml

  • 由于是使用自己scp过去的镜像，连不上外网docker hub，所以修改下镜像拉取策略

    1
    2
    3
              imagePullPolicy: Always
    # 修改成 IfNotPresent（本地优先）或者 Never
              imagePullPolicy: IfNotPresent

  • 在k8s中部署operator

    1
    2
    ➜  kubectl apply -f deploy/operator.yaml
    deployment.apps/operator-demo created

  • 查看运行

    1
    2
    3
    4
    ➜  operator-demo kubectl get po -A|grep operator
    default       operator-demo-84dd848bbb-8866f       1/1     Running            0          78m
    default       operator-demo-84dd848bbb-9n4b5       1/1     Running            0          64m

  • 清理资源

    1
    2
    3
    4
    5
    6
    7
    8
    9
    ➜   cd deploy && kubectl delete -f .
    deployment.apps "operator-demo" deleted
    role.rbac.authorization.k8s.io "operator-demo" deleted
    rolebinding.rbac.authorization.k8s.io "operator-demo" deleted
    serviceaccount "operator-demo" deleted

    ➜  cd crds && kubectl delete -f .
    customresourcedefinition.apiextensions.k8s.io "appservices.app.qipajun.com" deleted
    appservice.app.qipajun.com "example-appservice" deleted

完整代码地址： https://github.com/qianbaidu/operator-demo