Different day, another Iceberg sink connector in a bad state — but this one taught a different lesson than yesterday’s. Yesterday was a genuine bytecode bug. Today was a checkpointing config interaction that no amount of extra compute could have fixed, and the graph that made that obvious only showed up once I stopped looking at node-level CPU and went looking at one specific thread.
The starting point
A sink connector’s internal control-topic consumer group had gone completely flat — zero committed-offset progress for over an hour, with one partition’s backlog climbing past 90k messages. The team’s first response, before I got involved, was the reasonable one: throw more compute at it. Moved the connector to a bigger node group and scaled its worker replicas up.
Didn’t help. Offsets were still frozen at the same values they’d been stuck at before the change. Whatever was wrong, it wasn’t a capacity problem.
Finding the actual bottleneck
Pulled a live thread dump off the JVM. The connector’s control-topic consumer runs on a single dedicated thread — a config knob that looked like it should parallelize consumption actually only parallelizes the downstream per-table commit calls, not the reading of the control topic itself. One thread, no way around it.
That thread was pegged at 60-70% of a single core, sustained, sitting inside reflection-based Avro decoding while it replayed the backlog on every recovery attempt. At node scale, one thread eating most of one core out of dozens available is invisible — CPU utilization graphs looked calm the entire time. The bottleneck was wall-clock time on a single thread, not aggregate compute, and no dashboard built around node or pod CPU percentages was ever going to surface that.
The actual mechanism
The logs had the real story, once I knew where to look:
Failing OffsetCommit request since the consumer is not part of an active group
Two settings were fighting each other:
- The poll-interval timeout that governs how long a consumer can go between polls before the broker kicks it out of the group was unset, silently defaulting to five minutes.
- The connector’s own commit-timeout setting, controlling how long a commit round is allowed to run, was configured well past that five-minute default.
So every commit round that took its full allotted time was already guaranteed to blow through the poll-interval default first. The broker would evict the consumer from the group before it ever got a chance to commit an offset. The consumer would rejoin fresh — with the same unadvanced offset — and immediately start re-decoding the same, now-larger backlog from scratch. A self-reinforcing loop that had nothing to do with load: more backlog meant a slower recovery pass, which meant it was even less likely to finish before the next eviction.
The fix
No pod restart, no connector recreation — just a config update, reconciled in place by the operator managing it:
| Setting | Before | After |
|---|---|---|
| poll-interval timeout | unset (5 min default) | 30 min |
| commit timeout | ~8.3 min | ~13.3 min |
| commit interval | 10 min | 15 min |
The core fix is just making sure the poll-interval ceiling is comfortably above the commit timeout, so a slow-but-progressing commit round never gets treated as a dead consumer.
Watching it actually recover
First commit round under the new config ran for its full new timeout window, then did what it was designed to do under pressure: forced through a commit of everything it had accumulated as a batch of row-level deltas instead of hanging indefinitely. Real writes landed a few minutes later — the first genuine write activity in the whole incident. Offsets caught up:
| Partition | Lag before | Lag after |
|---|---|---|
| 0 | ~7,700 | 1 |
| 1 | ~93,800 | 13 |
| 2 | ~2,200 | 1 |
Next commit cycle came through clean and on schedule, no stalling, steady state held for the rest of the observation window.
Takeaway
The instinct to scale up compute when something looks stuck is usually right, but it assumes the bottleneck is parallelizable. Here it wasn’t — one consumer thread, hard-serialized by design, meant the fix was entirely about the relationship between two timeout values, not about how many cores or replicas were available. Worth remembering: when a “backlog that won’t drain” keeps regenerating itself after every restart, check whether the recovery process itself is what’s causing the eviction, before assuming the backlog is a cause rather than a symptom.
Also flagged for later, not yet acted on: the per-table commit path leaks a file-IO handle on each table load (harmless, garbage-collected via finalizer, but worth cleaning up), and one of the tables involved has accumulated enough delete-files from merge-on-read writes that a compaction pass is probably due soon to keep commit latency in check.
Smaller thing from today: spent some time earlier fixing an observability gap in a shared CI rollback workflow — a set of GitHub Actions reused across several services. Two apps that shared the same deploy pipeline but had different environments looked identical in the Actions job list, the approval screen, and the Slack notification, because the job name and the notification only showed the environment, not which app it belonged to. Renamed the matrix jobs to include the app name and threaded an optional app-name field through the shared Slack notification action so it shows up as “app / environment” instead of just “environment.” Also noticed that a secondary deploy path used by at least one service never wired up the same rollback auto-dispatch that the primary path had, so it silently had no equivalent safety net — fixed that too, gated so it only affects callers whose rollback workflow actually declares the new field.