kubeEtcd.enabled: false in kube-prometheus-stack also removes the etcd alerts

On Talos, the chart's etcd Service selects no pods, so it looks like dead weight. Turning the whole kubeEtcd block off also drops the etcd alert rules and the Grafana dashboard. Turn off only the Service and ServiceMonitor.

A health check of my cluster flagged a Service with no endpoints: prometheus-kube-prometheus-kube-etcd. The kube-prometheus-stack chart creates it to find etcd pods by the label component=etcd. On Talos, etcd isn’t a pod. It’s a Talos system service, so the selector matches nothing.

That didn’t mean etcd went unmonitored. I’d already added a scrape job of my own for it. Talos serves etcd’s metrics unauthenticated on port 2381 once listen-metrics-urls is set to listen beyond loopback, and the job pointed at the three node addresses. All three targets were up. To check, I read the active targets through the API server’s service proxy, because the Prometheus image has no wget:

kubectl get --raw \
  "/api/v1/namespaces/monitoring/services/http:prometheus-kube-prometheus-prometheus:9090/proxy/api/v1/targets?state=active"

So the chart’s Service and ServiceMonitor were dead duplicates. The obvious cleanup is one line in the values:

kubeEtcd:
  enabled: false

What that line also removes

In the chart version I run (87.21), the same flag also gates the chart’s etcd alert rules (13 of them, such as etcdNoLeader, etcdHighFsyncDurations and etcdMembersDown) and the etcd Grafana dashboard. The rules match job=~".*etcd.*", so they work perfectly well with a custom job named talos-etcd. Turning the block off would have quietly removed the alerts I care most about on a cluster whose etcd shares a consumer NVMe.

To drop only the empty Service and its ServiceMonitor, switch those off and leave the block enabled:

kubeEtcd:
  enabled: true
  service:
    enabled: false
  serviceMonitor:
    enabled: false

Check the result before you merge it. helm template with your values should still render the etcd PrometheusRule group and the dashboard ConfigMap, and no longer the Service:

helm template kps prometheus-community/kube-prometheus-stack --version <v> -f values.yaml \
  | grep -E 'name: .*etcd'

The chart also supports the other direction: point kubeEtcd.endpoints at the node addresses with service.port and targetPort 2381, and let the chart’s own job scrape them instead of a custom one. Either works, as long as the job name contains etcd.