Choosing an etcd fsync alert threshold that means something

The stock etcdHighFsyncDurations warning fired about 200 times in a week on a homelab. Why histogram bucket edges decide what a threshold means, how kube-prometheus-stack lets you change it, and why raising it hides a real disk problem.

etcdHighFsyncDurations was the noisiest alert I had. The chart’s rule fires when the 99th percentile of WAL fsync time is above half a second for ten minutes. Over seven days that was about two hundred episodes, on a cluster that was working fine. Before changing anything I wanted to know what a sensible threshold would be, and whether raising it was just sweeping a real problem under the rug.

Buckets decide what a threshold means

The fsync histogram has power-of-two bucket edges: 0.256, 0.512, 1.024, 2.048 seconds and so on. histogram_quantile interpolates inside a bucket, so a p99 of “0.6 s” really says the 99th percentile sits somewhere between 0.512 and 1.024. A threshold of 0.5 sits just under the 0.512 edge, so any p99 in that bucket trips it. Thresholds only make sense near an edge.

What the numbers looked like

I counted episodes over seven days by taking the max across the three nodes and counting changes of a boolean subquery, divided by two: changes((expr > X)[7d:1m]). The result depended a lot on the level and the for duration:

The p99 was bimodal. In quiet hours it was 20 to 70 ms, and in busy hours it sat around 0.6 to 0.7 s. Peaks reached about five seconds. A trick I found in a blog post, x == 1 unless x offset 1m == 1 inside a subquery, returned zero for me, so I would not trust it.

Changing it in kube-prometheus-stack

The chart’s customRules block only overrides for, severity and labels. It cannot touch the expression. To change a threshold, disable the stock rule with defaultRules.disabled.etcdHighFsyncDurations: true and add your own PrometheusRule carrying the label your Prometheus selects on (release: <chart release name> here). Without that label the operator ignores the rule silently.

I ended up with a warning at one second and a critical at two, both after 15 minutes.

The uncomfortable part

Raising the threshold makes the pager quiet. It does not make the disk faster. etcd’s guidance is a p99 WAL fsync under 10 ms, and mine was exceeding half a second during busy hours. That is a storage problem, not an alerting problem, and I had a consumer NVMe with no power-loss protection under three etcd members.

So I treated the new thresholds as “alert when it is actually painful”, and kept an eye on etcdHighNumberOfLeaderChanges, which is the alert that tracks real harm. Fifty leader changes in a week with none in the last day was my baseline for whether the disk was hurting.