trivy-operator not scanning all running containers images neither generating vuln report

[farizz1]

Question

I have set up the latest version of Trivy Operator (0.24.1) in a Rancher-managed cluster. However, the operator is not scanning all running container images, particularly those in system namespaces like kube-system. The scan job is being created, and I can see that it successfully downloads the Trivy DB and Trivy checks. Despite this, the Trivy Operator does not proceed with scanning the images and it doesn't generate the Vuln report at all. What could be causing this issue?

Target

Container Image

Scanner

Vulnerability

Output Format

None

Mode

Standalone

Operating System

No response

Version

trivy-operator version: 0.24.1
Trivy: 0.59.1

[farizz1]

I can see in logs that a scan job is submitted but no report is generated
DEBUG reconciler.vulnerabilityreport Getting workload from cache {"kind": "Job", "name": {"name":"helm-install-rke2-ingress-nginx","namespace":"kube-system"}, "workload": "helm-install-rke2-ingress-nginx"}
2025-02-27T08:52:15Z DEBUG reconciler.vulnerabilityreport Submitting a scan for the workload {"kind": "Job", "name": {"name":"helm-install-rke2-ingress-nginx","namespace":"kube-system"}, "workload": "helm-install-rke2-ingress-nginx"}

I am struggling to identify the issue but can't do it and it already took alot of time to make it work and scan all the container images within cluster.
As an info: my cluster is quite big, it has around 650 running pods, is there a limit that a trivy-operator can scan

other than default configuration in values.yaml file, I have made these changes

custom-values.yaml file

serviceMonitor:
enabled: true
trivy:
command: image
ignoreUnfixed: true
httpsProxy: http://proxy-url.com:8080
httpProxy: http://proxy-url.com:8080
noProxy: "local enterprise domain"

timeout: "10m0s"

insecureRegistries:
internalRegistry: private-registry.com
artifactoryRegistry: private-registry2.com
slow: false
vulnType: "os,library"
dbRepositoryInsecure: true

nonSslRegistries:
internalRegistry: private-registry.com
artifactoryRegistry: private-registry2.com

policiesBundle:
insecure: true

resources:
limits:
cpu: 500m
memory: 1000Mi
requests:
cpu: 200m
memory: 400Mi

nodeCollector:
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
- key: "node-role.kubernetes.io/etcd"
operator: "Exists"
effect: "NoExecute"
- key: "app"
operator: "Equal"
value: "longhorn"
effect: "NoSchedule"

operator:
metricsVulnIdEnabled: true
metricsConfigAuditInfo: true
metricsRbacAssessmentInfo: true
metricsInfraAssessmentInfo: true
metricsExposedSecretInfo: true
metricsImageInfo: true
metricsClusterComplianceInfo: true
scanJobsConcurrentLimit: 10
scannerReportTTL: "24h"
privateRegistryScanSecretsNames:
{ "trivy-system": "artifactory-registry-secret,bcr-registry-secret" }
logDevMode: true

trivyOperator:
scanJobPodTemplateContainerSecurityContext:
runAsUser: 0

[tom1299]

I don't know whether the log extract is complete. But it seems like you got debug logging enabled. So if a job is created you should also see a log message like:

DEBUG	reconciler.vulnerabilityreport	Creating scan job for the workload	{"kind": "....

Can you see that message in the logs ? Otherwise no scan job would be have been created.
In addition it would be interesting to know what the scan job's main containers output is. Is there any error ?

[farizz1]

Hi @tom1299, Thanks for looking into it, I don't see such log but I do see that scan jobs are created
There is no error in the jobs container, it downloads the trivy-db, trivy-java-db successfully and than it terminates after 5m (timeout) without scanning the image and generating report.
The configauditreports and rbacassessmentreports are generated for all the configured resources even in system namespaces as well
sometime it does hit the job limit (10) which is fine, i can understand but it should atleast scan it within 24 hrs.

2025-02-28T07:46:55Z DEBUG reconciler.vulnerabilityreport Submitting a scan for the workload {"kind": "DaemonSet", "name": {"name":"pushprox-kube-scheduler-client","namespace":"cattle-monitoring-system"}, "workload": "pushprox-kube-scheduler-client"}
2025-02-28T07:46:55Z DEBUG reconciler.vulnerabilityreport Getting workload from cache {"kind": "Job", "name": {"name":"helm-install-rke2-snapshot-controller","namespace":"kube-system"}, "workload": "helm-install-rke2-snapshot-controller"}
2025-02-28T07:46:55Z DEBUG reconciler.vulnerabilityreport Submitting a scan for the workload {"kind": "Job", "name": {"name":"helm-install-rke2-snapshot-controller","namespace":"kube-system"}, "workload": "helm-install-rke2-snapshot-controller"}
2025-02-28T07:46:55Z DEBUG reconciler.vulnerabilityreport Getting workload from cache {"kind": "ReplicaSet", "name": {"name":"node-red-599b7567c4","namespace":"mel-connectivity-layer"}, "workload": "node-red-599b7567c4"}
2025-02-28T07:46:55Z DEBUG reconciler.vulnerabilityreport Submitting a scan for the workload {"kind": "ReplicaSet", "name": {"name":"node-red-599b7567c4","namespace":"mel-connectivity-layer"}, "workload": "node-red-599b7567c4"}
2025-02-28T07:46:55Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"service-vault-secrets-operator-metrics-service","namespace":"radium-op-vault-secrets-operator"}, "durationToTTLExpiration": "-240h22m14.162877871s"}
2025-02-28T07:46:55Z DEBUG reconciler.vulnerabilityreport Getting workload from cache {"kind": "Pod", "name": {"name":"kube-proxy-rngvm01245","namespace":"kube-system"}, "workload": "kube-proxy-rngvm01245"}
2025-02-28T07:46:55Z DEBUG reconciler.vulnerabilityreport Submitting a scan for the workload {"kind": "Pod", "name": {"name":"kube-proxy-rngvm01245","namespace":"kube-system"}, "workload": "kube-proxy-rngvm01245"}
2025-02-28T07:46:55Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"service-pushprox-kube-scheduler-proxy","namespace":"cattle-monitoring-system"}, "durationToTTLExpiration": "-240h22m43.262741935s"}
2025-02-28T07:46:55Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"service-cattle-project-p-6dd7z-monitoring-grafana","namespace":"cattle-project-p-6dd7z-monitoring"}, "durationToTTLExpiration": "-240h27m23.362876217s"}
2025-02-28T07:46:55Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"statefulset-dummyapp","namespace":"me-cyclechron-d-nginxtestapp"}, "durationToTTLExpiration": "-240h51m49.462865181s"}
2025-02-28T07:46:55Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"replicaset-84f7d46d7","namespace":"argo-bci-cn-demo-d-ias-cn-ies"}, "durationToTTLExpiration": "-240h51m10.563229611s"}
2025-02-28T07:46:55Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"replicaset-cattle-project-p-ncfpl-monitoring-grafana-799b6b5d68","namespace":"cattle-project-p-ncfpl-monitoring"}, "durationToTTLExpiration": "-240h34m30.663003848s"}
2025-02-28T07:46:55Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"resourcequota-default-hngzc","namespace":"cni-testing"}, "durationToTTLExpiration": "-240h35m58.762305242s"}
2025-02-28T07:46:55Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"resourcequota-default-ptcsj","namespace":"radium-operators"}, "durationToTTLExpiration": "-240h36m34.862840793s"}
2025-02-28T07:46:56Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"resourcequota-default-5gh8z","namespace":"cattle-project-p-dh6f4"}, "durationToTTLExpiration": "-240h35m39.962987234s"}
2025-02-28T07:46:56Z DEBUG reconciler.ttlreport RequeueAfter {"report": {"name":"pod-instance-manager-5d81d250011283c96cba2461dfc7efc3","namespace":"longhorn-system"}, "durationToTTLExpiration": "-240h42m37.062592924s"}
2025-02-28T07:46:56Z DEBUG reconciler.vulnerabilityreport Creating scan job for the workload {"kind": "StatefulSet", "name": "prometheus-cattle-project-p-dh6f4-mon-prometheus", "namespace": "cattle-project-p-dh6f4-monitoring", "podSpecHash": "dc65bf8dc"}
2025-02-28T07:46:56Z INFO KubeAPIWarningLogger spec.template.spec.containers[0].env[13]: hides previous definition of "TRIVY_INSECURE"
2025-02-28T07:46:56Z INFO KubeAPIWarningLogger spec.template.spec.containers[1].env[13]: hides previous definition of "TRIVY_INSECURE"
2025-02-28T07:46:56Z INFO KubeAPIWarningLogger spec.template.spec.containers[2].env[13]: hides previous definition of "TRIVY_INSECURE"
2025-02-28T07:46:56Z INFO KubeAPIWarningLogger spec.template.spec.containers[3].env[13]: hides previous definition of "TRIVY_INSECURE"

[simar7]

hi @farizz1 - I see that you're running a Rancher-managed cluster, Trivy operator makes no guarantees on managed Kubernetes flavors, so it's quite possible we don't fully support it.

Having said that, if you are able to post the full logs for us somewhere like gist.github.com we maybe able to help you a bit better. From what you've posted so far, I'm unable to see a cause why the operator isn't scanning.