kube-api log ListAndWatch error: failed to list redis.redis.opstreelabs.in/v1beta1, Kind=RedisReplication: conversion webhook for redis.redis.opstreelabs.in/v1beta2

[ktzsolt]

What version of redis operator are you using?

redis-operator chart 0.15.10 version from https://ot-container-kit.github.io/helm-charts

Does this issue reproduce with the latest release?

latest helm chart is not available, only 0.15.10.
Latest image published to quay is 0.15.1

What did you do?

helm repo add ot-container-kit  https://ot-container-kit.github.io/helm-charts/
helm upgrade --install redis-operator ot-container-kit/redis-operator --version 0.15.10 --namespace redis-operator --create-namespace  

What did you expect to see?

No errors by the kube-api pods

What did you see instead?

E0529 10:31:11.738631 1 cacher.go:470] cacher (redisreplications.redis.redis.opstreelabs.in): unexpected ListAndWatch error: failed to list redis.redis.opstreelabs.in/v1beta1, Kind=RedisReplication: conversion webhook for redis.redis.opstreelabs.in/v1beta2, Kind=RedisReplication failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.118.14:9443, code 502: 502 Bad Gateway; reinitializing...

[MoZadro]

We had same issue on multiple clusters, redis operator helm chart version 0.15.9

We edited redis crd's:

redis.redis.redis.opstreelabs.in
redisclusters.redis.redis.opstreelabs.in
redisreplications.redis.redis.opstreelabs.in
redissentinels.redis.redis.opstreelabs.in

And deleted this part:

conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: webhook-service
          namespace: redis-operator
          path: /convert
          port: 443
      conversionReviewVersions:
      - v1beta1
      - v1beta2

[ignlg]

We're also dealing with this issue. Thanks a lot, @MoZadro, for sharing your workaround!

[ktzsolt]

I was able to upgrade to 0.18.0 chart now that the image is pushed to quay.io and the issue (error messages in kube-api pods) is gone now

[ktzsolt]

I am reopening this.

In dev cluster the issue went away after upgrading the operator to 0.18.1 but in test cluster we still see this happening even after upgrading to 0.18.0. I even tried to nuke the operator and the redis-replication and redis-sentinel resources to no avail.

The version of the api for redis resources are redis.redis.opstreelabs.in/v1beta2 in both cluster:

$ kubectl api-resources | grep redis
redis                                                   redis.redis.opstreelabs.in/v1beta2          true         Redis
redisclusters                                           redis.redis.opstreelabs.in/v1beta2          true         RedisCluster
redisreplications                                       redis.redis.opstreelabs.in/v1beta2          true         RedisReplication
redissentinels                                          redis.redis.opstreelabs.in/v1beta2          true         RedisSentinel

TBH I don't know what the difference is between the 2 cluster that makes this happen.

[ktzsolt]

based on @MoZadro comment, I found the difference in our clusters,
in dev we have this for the CRD:

spec:
  conversion:
    strategy: None

but in test we had this:

spec:
  conversion:
      strategy: Webhook
      webhook:
        clientConfig:
          service:
            name: webhook-service
            namespace: redis-operator
            path: /convert
            port: 443
        conversionReviewVersions:
        - v1beta1
        - v1beta2

We installed the 0.18.1 redis-operator in both cluster without any parameters so I don't get it why we have this difference.