使用operator sdk开发k8s Operator

使用operator sdk开发k8s Operator

环境安装

创建项目

  • 创建项目
operator-sdk new operator-demo --repo github.com/qianbaidu/operator-demo
  • 创建Api / controller
cd operator-demo
operator-sdk add api --api-version=app.qipajun.com/v1alpha1 --kind=AppService
operator-sdk add controller --api-version=app.qipajun.com/v1alpha1 --kind=AppService
  • 生成目录结构
➜  operator-demo tree .
.
├── build # 构建编译目录
│   ├── Dockerfile
│   └── bin
│       ├── entrypoint
│       └── user_setup
├── cmd # 启动入口
│   └── manager
│       └── main.go
├── deploy # 部署配置目录
│   ├── crds
│   │   ├── app.qipajun.com_appservices_crd.yaml
│   │   └── app.qipajun.com_v1alpha1_appservice_cr.yaml
│   ├── operator.yaml
│   ├── role.yaml
│   ├── role_binding.yaml
│   └── service_account.yaml
├── go.mod
├── go.sum
├── pkg
│   ├── apis
│   │   ├── addtoscheme_app_v1alpha1.go
│   │   ├── apis.go
│   │   └── app
│   │       ├── group.go
│   │       └── v1alpha1
│   │           ├── appservice_types.go  # api struct定义
│   │           ├── doc.go
│   │           ├── register.go
│   │           └── zz_generated.deepcopy.go
│   └── controller
│       ├── add_appservice.go
│       ├── appservice
│       │   └── appservice_controller.go # 控制器
│       └── controller.go
├── tools.go
└── version
    └── version.go
  • 生成项目主要目录介绍:
    • build: 编译构建目录
    • cmd: 项目启动入口
    • deploy: 部署构建配置文件,生成k8s资源定义和配置yaml
    • pkg/apis: api资源定义
    • pkg/controller:资源控制器,实现资源控制操作

这里主要有两块关注:api和controller,

  • api定义好资源类型
  • controller中主要实现Reconcile方法,实现接口,实现配置读取,资源生命周期维护管理,这点和基于kubebuilder开发CRD一样

开始开发:

  • 调试、运行思路:
    • 1、创建k8s资源: deploy目录下已经有资源配置了,apply到k8s集群中,创建资源
    • 2、运行项目(本地开发能连入测试k8s集群),项目运行后会watch k8s资源变更可实时获取变更,即可代码调试、开发了
    • 3、开发调试完成后、编译打包镜像,修改yaml镜像地址,即可部署到k8s集群

部署运行开发测试环境:

  • 1、创建角色权限
➜  kubectl apply -f role.yaml
role.rbac.authorization.k8s.io/operator-demo created
➜  kubectl apply -f role_binding.yaml
rolebinding.rbac.authorization.k8s.io/operator-demo created
➜  kubectl apply -f service_account.yaml
serviceaccount/operator-demo created
  • 2、创建自定义资源
➜  kubectl apply -f app.qipajun.com_appservices_crd.yaml
customresourcedefinition.apiextensions.k8s.io/appservices.app.qipajun.com created
  • 查看资源
➜  kubectl get crd -A
NAME                          CREATED AT
appservices.app.qipajun.com   2020-04-19T04:19:27Z
  • 3、运行项目代码
    • 先加几个log点方便查看
# log组件暂时用自己导入的 import中添加导入
	loger "github.com/sirupsen/logrus"

#  pkg/controller/appservice/appservice_controller.go Reconcile方法中将get到的instace log打印出来
	err := r.client.Get(context.TODO(), request.NamespacedName, instance)
	if err != nil {
		loger.Error(err, "client get instance error")
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			loger.Error(err, "client get instance not found")
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err
	} else {
		i, _ := json.Marshal(instance)
		loger.Info("get instance info :", string(i))
	}

  • 运行代码
➜  make run
go mod download
export WATCH_NAMESPACE=default && go run cmd/manager/main.go
...
  • 创建一个自定义的operator AppService类型资源
➜ kubectl apply -f deploy/crds/app.qipajun.com_v1alpha1_appservice_cr.yaml
appservice.app.qipajun.com/example-appservice created
  • 此时查看代码运行log
INFO[0001] Reconcile
INFO[0001] get instance info :{"kind":"AppService","apiVersion":"app.qipajun.com/v1alpha1","metadata":{"name":"example-appservice","namespace":"default","selfLink":"/apis/app.qipajun.com/v1alpha1/namespaces/default/appservices/example-appservice","uid":"d52ae6db-8200-11ea-9367-0800275e4e86","resourceVersion":"143948","generation":1,"creationTimestamp":"2020-04-19T05:44:27Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"app.qipajun.com/v1alpha1\",\"kind\":\"AppService\",\"metadata\":{\"annotations\":{},\"name\":\"example-appservice\",\"namespace\":\"default\"},\"spec\":{\"size\":3}}\n"}},"spec":{},"status":{}}
INFO[0001] Set AppService instance as the owner and controller success
{"level":"info","ts":1587275589.3622541,"logger":"controller_appservice","msg":"Skip reconcile: Pod already exists","Request.Namespace":"default","Request.Name":"example-appservice","Pod.Namespace":"default","Pod.Name":"example-appservice-pod"}
{"level":"info","ts":1587275589.362294,"logger":"controller_appservice","msg":"Reconciling AppService","Request.Namespace":"default","Request.Name":"example-appservice"}
  • 当创建、更新、删除时都会触Reconcile发这个方法,所以可以根据自己需求在这个方式实现资源创建、更新、删除、容错判断处理

构建部署自己的operator部署到k8s

开发完需求代码后,执行编译、构建docker镜像,并修改opreator yaml镜像地址部署到集群就可以实时watch资源变更执行一系列操作了

  • 构建、推送镜像
operator-sdk build liuhaogui/operator-demo
docker push liuhaogui/operator-demo
  • 本地docker推送来回拉网络太慢使用docker save和load来加载镜像
# 导出镜像、scp到目标node上
docker save liuhaogui/operator-demo:v0.0.1 > operator-demo:v0.0.1.tar
scp ./operator-demo:v0.0.1.tar root@192.168.99.101:/root

# 目标node上加载docker镜像
docker load -i operator-demo\:v0.0.1.tar

修改镜像地址,部署自定义operator

  • 修改镜像地址
# replace image url
sed -i "s|REPLACE_IMAGE|liuhaogui/operator-demo:v0.0.1|g" deploy/operator.yaml
# On OSX use:
sed -i "" "s|REPLACE_IMAGE|liuhaogui/operator-demo:v0.0.1|g" deploy/operator.yaml
  • 由于是使用自己scp过去的镜像,连不上外网docker hub,所以修改下镜像拉取策略
          imagePullPolicy: Always
# 修改成 IfNotPresent(本地优先)或者 Never
          imagePullPolicy: IfNotPresent
  • 在k8s中部署operator
➜  kubectl apply -f deploy/operator.yaml
deployment.apps/operator-demo created
  • 查看运行
➜  operator-demo kubectl get po -A|grep operator
default       operator-demo-84dd848bbb-8866f       1/1     Running            0          78m
default       operator-demo-84dd848bbb-9n4b5       1/1     Running            0          64m

  • 清理资源
➜   cd deploy && kubectl delete -f .
deployment.apps "operator-demo" deleted
role.rbac.authorization.k8s.io "operator-demo" deleted
rolebinding.rbac.authorization.k8s.io "operator-demo" deleted
serviceaccount "operator-demo" deleted

➜  cd crds && kubectl delete -f .
customresourcedefinition.apiextensions.k8s.io "appservices.app.qipajun.com" deleted
appservice.app.qipajun.com "example-appservice" deleted

完整代码地址: https://github.com/qianbaidu/operator-demo