Spent today root-causing a Kafka Connect sink connector that had gone into a bad state: it loaded its target table once, then never committed anything again, spinning until it fell out of its consumer group roughly every five minutes and restarted — forever. Same connector, same jar, same config as a dozen others that had been running fine for months. Read-only investigation the whole way — production access, but nothing to mutate until the actual fix landed elsewhere.

Ruling things out first

The obvious first guess was a poison record — some CDC event with a malformed field choking the connector. I pulled real records off the source topic and checked them by hand: types were fine, no nulls where they shouldn’t be, nothing structurally wrong. One mildly interesting pattern (one row getting hammered by frequent updates, consistent with bot-like activity) but nothing that explains a hang.

Second guess: the target table’s metadata was missing the Iceberg-level identifier-field-ids attribute, which the connector’s upsert path is supposed to rely on for row identity. Checked it — sure enough, missing. Looked like a real lead until I checked a known-healthy table using the exact same connector jar and config as a baseline. Same missing attribute. That table had been doing successful upsert commits for months. Hypothesis dead on arrival — but a useful lesson in itself: a plausible-looking anomaly isn’t a root cause until you’ve checked whether a working instance has the same “anomaly” and still works fine.

Down to the source

At this point I needed to actually see what the connector’s write path was doing when it hung, so I pulled a thread dump. The stuck thread was parked inside a RecordProjection constructor — an internal class in the Kafka Connect sink’s data-writer layer, responsible for projecting a full record onto its identifier-column subset for building delete/upsert keys.

Went looking for the source. Found what looked like the right file in the public upstream repo the project descends from — read through it, reasoned about its loop bounds, concluded it structurally couldn’t spin forever on a flat schema. Except the package name in the thread dump didn’t match what I’d just read. Wrong file. The actual class living in the deployed jar turned out not to exist anywhere in the public repo’s history at all, at any tag, at any commit, in any branch reachable via a full recursive tree search. It’s an internally-built jar, and this particular file was never upstreamed.

So: no source, no public reference implementation, just a compiled .class file sitting inside a running pod.

Reading bytecode without a JDK

Neither the pod nor my laptop had a working JDK — no javap, nothing. Uploading a proprietary compiled class to some online decompiler was off the table on principle; that’s internal code, it doesn’t leave the building. The fix was simpler than expected: pull a JDK image with Docker, mount the extracted .class file read-only, and run javap -c -v inside the container. Fully local, nothing transmitted anywhere, no permanent install needed on the laptop either.

The disassembly of the constructor made the bug obvious once I actually looked at the right bytes. The method has a small loop that linearly scans a record’s fields looking for the one matching a target field ID:

for (int i = 0; !found && i < dataFields.size(); i += 1) {
    // check dataFields.get(i) against the target field
}

In the compiled bytecode, the loop’s increment step was:

iconst_1
istore   8      // i = 1  (a hardcoded constant!)
goto     <loop-condition>

Not iinc. Not iload; iconst_1; iadd; istore. A hardcoded i = 1, every single pass through the loop. I checked every write to that local variable across the whole method — there were exactly two: the initial i = 0, and this one. No path anywhere increments it properly.

Which means: once the scan index reaches 1 without a match, it can never move to 2. If the field it’s hunting for isn’t at position 0 or 1 of the record’s schema, the loop spins on index 1 forever. Not a slow leak, not a timeout — a genuine infinite loop, permanently parked, 100% of one core, until something external kills the thread.

Confirming it against real data

Checked the broken table’s actual schema field order: the identifier column sat at position 11. Checked the long-running healthy table used as an earlier baseline: identifier column at position 0. Checked the three other tables sharing the same connector: also position 0, all safe. That’s the entire difference between “runs fine for months” and “never commits, ever” — not the connector, not the config, just where one column happened to land when the table was first created.

The broken table had been backfilled by a batch job that preserved the source database’s original column order, which put the identifier column deep in the schema. Every other table, by what turned out to be pure convention rather than design, had it first.

The fix

Updated the backfill script to place the identifier column first before the table’s initial write, so any future table it creates lands in the loop’s reachable range. Rebuilt the table, connector came up clean, has been committing normally since.

Takeaway

The bytecode was the ground truth the whole time — the config was fine, the data was fine, even most of my early reasoning about “the code” was reasoning about the wrong code, because the deployed artifact didn’t match any source I could find. Two things carried the day: checking a working instance instead of trusting a plausible-looking anomaly, and being willing to go all the way to disassembly with tools that respect where proprietary bytecode is allowed to travel (local Docker container, not a hosted decompiler) rather than stopping at “I can’t read Java bytecode without a JDK.”