Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error in DecoratorController of metacontroller rock: failed to sync kubeflow-pipelines-profile-controller #157

Open
mvlassis opened this issue Mar 6, 2025 · 5 comments · May be fixed by #158
Open

Error in DecoratorController of metacontroller rock: failed to sync kubeflow-pipelines-profile-controller #157

mvlassis opened this issue Mar 6, 2025 · 5 comments · May be fixed by #158
Labels
bug Something isn't working

Comments

@mvlassis
Copy link
Contributor

mvlassis commented Mar 6, 2025

Bug Description

This issue occurred while running the integration tests for kserve-operators

To Reproduce

  • Create a new PR on the kserve-operators repo
  • You should find an error similar to: lightkube.core.exceptions.ApiError: secrets "kserve-controller-s3" not found

Environment

  • Juju 3.6
  • Metacontroller rock version: v4.11.22

Relevant Log Output

{"level":"Level(-4)","ts":"2025-03-06T14:49:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"Sync","object":{"name":"manos"}}
{"level":"Level(-4)","ts":"2025-03-06T14:49:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-pipelines-profile-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"manos"}}
{"level":"error","ts":"2025-03-06T14:49:00Z","msg":"Unhandled Error","logger":"UnhandledError","error":"failed to sync kubeflow-pipelines-profile-controller 'v1:Namespace::manos': sync hook failed: http error: Post \"http://kfp-profile-controller.kubeflow/sync\": dial tcp: lookup kfp-profile-controller.kubeflow on 10.152.183.10:53: no such host","stacktrace":"metacontroller/pkg/controller/decorator.(*decoratorController).processNextWorkItem\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:320\nmetacontroller/pkg/controller/decorator.(*decoratorController).worker\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:308\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:161\nmetacontroller/pkg/controller/decorator.(*decoratorController).Start.func1.1\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:282"}
{"level":"info","ts":"2025-03-06T14:49:01Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:49:01Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:49:06Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:49:06Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:49:54Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:49:54Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"Level(-4)","ts":"2025-03-06T14:50:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"Sync","object":{"name":"manos"}}
{"level":"Level(-4)","ts":"2025-03-06T14:50:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-pipelines-profile-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"manos"}}
{"level":"error","ts":"2025-03-06T14:50:00Z","msg":"Unhandled Error","logger":"UnhandledError","error":"failed to sync kubeflow-pipelines-profile-controller 'v1:Namespace::manos': sync hook failed: http error: Post \"http://kfp-profile-controller.kubeflow/sync\": dial tcp: lookup kfp-profile-controller.kubeflow on 10.152.183.10:53: no such host","stacktrace":"metacontroller/pkg/controller/decorator.(*decoratorController).processNextWorkItem\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:320\nmetacontroller/pkg/controller/decorator.(*decoratorController).worker\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:308\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:161\nmetacontroller/pkg/controller/decorator.(*decoratorController).Start.func1.1\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:282"}
{"level":"info","ts":"2025-03-06T14:50:06Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:50:06Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:50:30Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:50:30Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"Level(-4)","ts":"2025-03-06T14:50:46Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"Sync","object":{"name":"manos"}}
{"level":"Level(-4)","ts":"2025-03-06T14:50:46Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-pipelines-profile-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"manos"}}
{"level":"error","ts":"2025-03-06T14:50:46Z","msg":"Unhandled Error","logger":"UnhandledError","error":"failed to sync kubeflow-pipelines-profile-controller 'v1:Namespace::manos': sync hook failed: http error: Post \"http://kfp-profile-controller.kubeflow/sync\": dial tcp: lookup kfp-profile-controller.kubeflow on 10.152.183.10:53: no such host","stacktrace":"metacontroller/pkg/controller/decorator.(*decoratorController).processNextWorkItem\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:320\nmetacontroller/pkg/controller/decorator.(*decoratorController).worker\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:308\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:161\nmetacontroller/pkg/controller/decorator.(*decoratorController).Start.func1.1\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:282"}
{"level":"info","ts":"2025-03-06T14:50:59Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:50:59Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"Level(-4)","ts":"2025-03-06T14:51:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"Sync","object":{"name":"manos"}}
{"level":"Level(-4)","ts":"2025-03-06T14:51:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-pipelines-profile-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"manos"}}
{"level":"error","ts":"2025-03-06T14:51:00Z","msg":"Unhandled Error","logger":"UnhandledError","error":"failed to sync kubeflow-pipelines-profile-controller 'v1:Namespace::manos': sync hook failed: http error: Post \"http://kfp-profile-controller.kubeflow/sync\": dial tcp: lookup kfp-profile-controller.kubeflow on 10.152.183.10:53: no such host","stacktrace":"metacontroller/pkg/controller/decorator.(*decoratorController).processNextWorkItem\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:320\nmetacontroller/pkg/controller/decorator.(*decoratorController).worker\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:308\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:161\nmetacontroller/pkg/controller/decorator.(*decoratorController).Start.func1.1\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:282"}
{"level":"info","ts":"2025-03-06T14:51:09Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:51:09Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:51:32Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:51:32Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:51:42Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:51:42Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"Level(-4)","ts":"2025-03-06T14:52:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"Sync","object":{"name":"manos"}}
{"level":"Level(-4)","ts":"2025-03-06T14:52:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-pipelines-profile-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"manos"}}
{"level":"error","ts":"2025-03-06T14:52:00Z","msg":"Unhandled Error","logger":"UnhandledError","error":"failed to sync kubeflow-pipelines-profile-controller 'v1:Namespace::manos': sync hook failed: http error: Post \"http://kfp-profile-controller.kubeflow/sync\": dial tcp: lookup kfp-profile-controller.kubeflow on 10.152.183.10:53: no such host","stacktrace":"metacontroller/pkg/controller/decorator.(*decoratorController).processNextWorkItem\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:320\nmetacontroller/pkg/controller/decorator.(*decoratorController).worker\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:308\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:161\nmetacontroller/pkg/controller/decorator.(*decoratorController).Start.func1.1\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:282"}
{"level":"info","ts":"2025-03-06T14:52:18Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:52:18Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:52:30Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:52:30Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"Level(-4)","ts":"2025-03-06T14:53:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"Sync","object":{"name":"manos"}}
{"level":"Level(-4)","ts":"2025-03-06T14:53:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-pipelines-profile-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"manos"}}
{"level":"error","ts":"2025-03-06T14:53:00Z","msg":"Unhandled Error","logger":"UnhandledError","error":"failed to sync kubeflow-pipelines-profile-controller 'v1:Namespace::manos': sync hook failed: http error: Post \"http://kfp-profile-controller.kubeflow/sync\": dial tcp: lookup kfp-profile-controller.kubeflow on 10.152.183.10:53: no such host","stacktrace":"metacontroller/pkg/controller/decorator.(*decoratorController).processNextWorkItem\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:320\nmetacontroller/pkg/controller/decorator.(*decoratorController).worker\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:308\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:161\nmetacontroller/pkg/controller/decorator.(*decoratorController).Start.func1.1\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:282"}
{"level":"info","ts":"2025-03-06T14:53:17Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:53:17Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:53:28Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:53:28Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:53:58Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:53:58Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"Level(-4)","ts":"2025-03-06T14:54:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"Sync","object":{"name":"manos"}}
{"level":"Level(-4)","ts":"2025-03-06T14:54:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-pipelines-profile-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"manos"}}
{"level":"error","ts":"2025-03-06T14:54:00Z","msg":"Unhandled Error","logger":"UnhandledError","error":"failed to sync kubeflow-pipelines-profile-controller 'v1:Namespace::manos': sync hook failed: http error: Post \"http://kfp-profile-controller.kubeflow/sync\": dial tcp: lookup kfp-profile-controller.kubeflow on 10.152.183.10:53: no such host","stacktrace":"metacontroller/pkg/controller/decorator.(*decoratorController).processNextWorkItem\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:320\nmetacontroller/pkg/controller/decorator.(*decoratorController).worker\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:308\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:161\nmetacontroller/pkg/controller/decorator.(*decoratorController).Start.func1.1\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:282"}
{"level":"info","ts":"2025-03-06T14:54:15Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:54:15Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:54:45Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:54:45Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: poddefaults.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"poddefaults\" in API group \"kubeflow.org\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-03-06T14:54:53Z","msg":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
{"level":"error","ts":"2025-03-06T14:54:53Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: serviceaccounts is forbidden: User \"system:serviceaccount:kubeflow:metacontroller-operator-charm\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/root/go/pkg/mod/k8s.io/client-go@v0.32.0/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/wait.go:72"}
{"level":"Level(-4)","ts":"2025-03-06T14:55:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"Sync","object":{"name":"manos"}}
{"level":"Level(-4)","ts":"2025-03-06T14:55:00Z","logger":"decorator.kubeflow-pipelines-profile-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-pipelines-profile-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"manos"}}
{"level":"error","ts":"2025-03-06T14:55:00Z","msg":"Unhandled Error","logger":"UnhandledError","error":"failed to sync kubeflow-pipelines-profile-controller 'v1:Namespace::manos': sync hook failed: http error: Post \"http://kfp-profile-controller.kubeflow/sync\": dial tcp: lookup kfp-profile-controller.kubeflow on 10.152.183.10:53: no such host","stacktrace":"metacontroller/pkg/controller/decorator.(*decoratorController).processNextWorkItem\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:320\nmetacontroller/pkg/controller/decorator.(*decoratorController).worker\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:308\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/wait/backoff.go:161\nmetacontroller/pkg/controller/decorator.(*decoratorController).Start.func1.1\n\t/root/parts/metacontroller/build/pkg/controller/decorator/controller.go:282"}

Additional Context

This issue seems related to the upgrade of the metacontroller rock: canonical/metacontroller-rock#24
@mvlassis mvlassis added the bug Something isn't working label Mar 6, 2025
@syncronize-issues-to-jira Syncronize issues to Jira
Copy link

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-7099.

This message was autogenerated

@mvlassis mvlassis mentioned this issue Mar 6, 2025
Open
@DnPlas
Copy link
Contributor

DnPlas commented Mar 6, 2025 •
edited
Loading

Thanks for reporting this.

Reproducing the issue

The test case you mention in the description of this issue is tests/integration/test_charm.py::test_new_user_namespace_has_manifests , which is looking into the test-namespace-resource-dispatcher namespace for a secret called kserve-controller-s3, but it fails because it cannot find it:

httpx:_client.py:1038 HTTP Request: GET https://10.1.0.10:16443/api/v1/namespaces/test-namespace-resource-dispatcher/secrets/kserve-controller-s3 "HTTP/1.1 404 Not Found"

This secret is requested by the kserve-controller to the resource-dispatcher to be applied on all namespaces labelled as user.kubeflow.org/enabled=true. This should happen automatically after relating both charms using the secrets interface.

I tried reproducing the issue locally and in fact I wasn't able to see the secrets (or service accounts) that kserve-controller is requesting despite the relation being established:

$ kubectl get secrets -ntest-namespace-resource-dispatcher
No resources found in test-namespace-resource-dispatcher namespace.

$ kubectl get serviceaccounts -ntest-namespace-resource-dispatcher
NAME      SECRETS   AGE
default   0         84m

$ juju status resource-dispatcher kserve-controller --relations
Model     Controller  Cloud/Region        Version  SLA          Timestamp
kubeflow  uk8s        microk8s/localhost  3.6.3    unsupported  12:57:21-06:00

App                  Version  Status  Scale  Charm                Channel      Rev  Address         Exposed  Message
kserve-controller             active      1  kserve-controller    latest/edge  800  10.152.183.233  no
resource-dispatcher           active      1  resource-dispatcher  latest/edge  280  10.152.183.45   no

Unit                    Workload  Agent  Address      Ports  Message
kserve-controller/0*    active    idle   10.1.15.119
resource-dispatcher/0*  active    idle   10.1.15.118

Integration provider                  Requirer                            Interface              Type     Message
istio-pilot:gateway-info              kserve-controller:ingress-gateway   istio-gateway-info     regular
knative-serving:local-gateway         kserve-controller:local-gateway     serving-local-gateway  regular
minio:object-storage                  kserve-controller:object-storage    object-storage         regular
resource-dispatcher:secrets           kserve-controller:secrets           kubernetes_manifest    regular
resource-dispatcher:service-accounts  kserve-controller:service-accounts  kubernetes_manifest    regular

Though I can see the resource dispatcher Pod has those resources in the ./resource directory as expected:

$ kubectl exec -ti -nkubeflow resource-dispatcher-0 -cresource-dispatcher -- /bin/bash

root@resource-dispatcher-0:/app# cat resources/service-accounts/kserve-controller-s3.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kserve-controller-s3
secrets:
- name: kserve-controller-s3
root@resource-dispatcher-0:/app# cat resources/secrets/kserve-controller-s3.yaml
apiVersion: v1
kind: Secret
metadata:
  annotations:
    serving.kserve.io/s3-endpoint: minio.kubeflow:9000
    serving.kserve.io/s3-region: us-east-1
    serving.kserve.io/s3-useanoncredential: 'false'
    serving.kserve.io/s3-usehttps: '0'
  name: kserve-controller-s3
stringData:
  AWS_ACCESS_KEY_ID: minio
  AWS_SECRET_ACCESS_KEY: TOCPGMA1KAQ7R8PLV5E7Y4C38A6JVG
type: Opaque

What's causing the issue?

I believe this issue is caused by 5e1ddb8 as it removed this ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/name: {{ namespace }}-{{ app_name }}-charm
  name: {{ namespace }}-{{ app_name }}-charm
rules:
- apiGroups:
  - "*"
  resources:
  - "*"
  verbs:
  - "*"

We can see that's the case because now the metacontroller workload cannot perform any operations (e.g. list, patch, delete) on poddefaults, serviceaccounts, and secrets.

You can check this by running:

$ kubectl auth can-i list serviceaccounts --as=system:serviceaccount:kubeflow:metacontroller-operator-charm --namespace=kubeflow

no

Important to note: the ClusterRole that allows "*" on "*" is not in upstream kubeflow, but it is in upstream metacontroller. I think this may cause confusion as we want to bring the same changes as upstream Kubeflow, but in our case, the functionality of the resource-dispatcher expects more access to certain resources; on the contrary, this is not required at all in Kubeflow upstream.

Fix

Either bring back the very permissive ClusterRole that was removed or ensure that the ClusterRole has "*" access for poddefaults, serviceaccounts and secrets. This is necessary because we (in contrast with upstream), have a very specific use case for those resources and need the metacontroller workload to have access to those.

Observations

After applying the following diff on metacontroller-operator, the metacontroller workload still logged something when trying to create the secret and serviceaccount that the kserve-controller was requesting:

diff 
diff --git a/src/files/manifests/metacontroller-rbac.yaml b/src/files/manifests/metacontroller-rbac.yaml
index 20b0b2d..d5717f0 100644
--- a/src/files/manifests/metacontroller-rbac.yaml
+++ b/src/files/manifests/metacontroller-rbac.yaml
@@ -14,102 +14,11 @@ metadata:
   name: {{ namespace }}-{{ app_name }}-charm
 rules:
 - apiGroups:
-  - ""
+  - "*"
   resources:
-  - namespaces
+  - "*"
   verbs:
-  - get
-  - list
-  - watch
-  - update
-- apiGroups:
-  - ""
-  resources:
-  - namespaces/status
-  verbs:
-  - get
-  - list
-  - watch
-  - update
-  - patch
-- apiGroups:
-  - ""
-  resources:
-  - secrets
-  - configmaps
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
-- apiGroups:
-  - apps
-  resources:
-  - deployments
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
-- apiGroups:
-  - ""
-  resources:
-  - services
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
-- apiGroups:
-  - networking.istio.io
-  resources:
-  - destinationrules
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
-- apiGroups:
-  - security.istio.io
-  resources:
-  - authorizationpolicies
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
-- apiGroups:
-  - metacontroller.k8s.io
-  resources:
-  - compositecontrollers
-  - controllerrevisions
-  - decoratorcontrollers
-  verbs:
-  - get
-  - list
-  - watch
-- apiGroups:
-  - ""
-  resources:
-  - events
-  verbs:
-  - create
-  - patch
+  - "*"
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
$ kubectl logs -nkubeflow metacontroller-operator-charm-0

{"level":"Level(-4)","ts":"2025-03-06T20:21:56Z","logger":"decorator.kubeflow-resource-dispatcher-controller","msg":"DecoratorController updating","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-resource-dispatcher-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"test-namespace-resource-dispatcher"}}
{"level":"info","ts":"2025-03-06T20:21:56Z","msg":"unknown field \"status.resources-ready\""}
{"level":"Level(-4)","ts":"2025-03-06T20:22:06Z","logger":"decorator.kubeflow-resource-dispatcher-controller","msg":"Sync","object":{"name":"test-namespace-resource-dispatcher"}}
{"level":"Level(-4)","ts":"2025-03-06T20:22:06Z","logger":"decorator.kubeflow-resource-dispatcher-controller","msg":"DecoratorController sync","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-resource-dispatcher-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"test-namespace-resource-dispatcher"}}
{"level":"info","ts":"2025-03-06T20:22:06Z","msg":"unknown field \"status.resources-ready\""}
{"level":"Level(-4)","ts":"2025-03-06T20:22:06Z","logger":"decorator.kubeflow-resource-dispatcher-controller","msg":"DecoratorController updating","controller":{"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","name":"kubeflow-resource-dispatcher-controller"},"parent":{"apiVersion":"v1","kind":"Namespace","name":"test-namespace-resource-dispatcher"}}
{"level":"info","ts":"2025-03-06T20:22:06Z","msg":"unknown field \"status.resources-ready\""}

Not sure if this is causing another issue, but it seems to be preventing the resources to be created:

$ kubectl get secrets -ntest-namespace-resource-dispatcher

No resources found in test-namespace-resource-dispatcher namespace.

$ kubectl get sa -ntest-namespace-resource-dispatcher
NAME      SECRETS   AGE
default   0         16m

@misohu misohu mentioned this issue Mar 7, 2025
Open
@kimwnasptd
Copy link
Contributor

@DnPlas thank you for the thorough exploration!

It's not clear to me though why we get 404 for the secret, since the manifests were giving permissions for secrets:

metacontroller-operator/src/files/manifests/metacontroller-rbac.yaml

Lines 35 to 47 in ce36513

- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

For the PodDefaults and ServiceAccounts it makes sense, since those permissions were removed.

@kimwnasptd kimwnasptd mentioned this issue Mar 7, 2025
Open
@DnPlas
Copy link
Contributor

DnPlas commented Mar 7, 2025

@DnPlas thank you for the thorough exploration!

It's not clear to me though why we get 404 for the secret, since the manifests were giving permissions for secrets:

metacontroller-operator/src/files/manifests/metacontroller-rbac.yaml

Lines 35 to 47 in ce36513

  • apiGroups:
    • ""
      resources:
    • secrets
    • configmaps
      verbs:
    • get
    • list
    • watch
    • create
    • update
    • patch
    • delete

For the PodDefaults and ServiceAccounts it makes sense, since those permissions were removed.

When I was doing the exploration, it looked like the metacontroller workload wouldn't work until it had access to all three resources, so it wouldn't work correctly until proper auth was set for all.

@kimwnasptd
Copy link
Contributor

Can confirm that by adding permissions for serviceaccounts and poddefaults (already has permissions for secrets) the PodDefaults for KFP, and MLflow (MinIO) are created as expected.

Also the secrets for MLflow's MinIO and kserve-controller-s3 are created as expected.

So I'll send a PR with including only those (and a comment so we don't accidentally remove them in the future) and avoid the * permissions.

@kimwnasptd kimwnasptd linked a pull request Mar 7, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: include necessary RBAC canonical/metacontroller-operator
3 participants
@kimwnasptd @DnPlas @mvlassis