Why your indexer resumes from the same block

Restart an indexer built on a consume loop and it prints something like this:

resuming from 8637064:1

Which is fine, except it printed the same thing yesterday, and the chain tip is forty thousand blocks past that cursor now. It's tempting to read this as a stuck consumer: surely "continue where you left off" should mean the last place you checked, not the last place you found something.

The consumer is caught up. Those really are two different places, and expecting the checkpoint to track the last place you looked rather than the last place you kept something is where the confusion starts.

A consumer's health endpoint reports three different heights, and the whole story lives in the gaps between them:

{
  "ok": true,
  "checkpoint": "8637064:1",
  "last_delivered_height": 8637064,
  "scanned_height": 8676502,
  "tip_height": 8676502,
  "blocks_behind": 0
}

tip_height is the chain tip; easy. last_delivered_height is the block of the last event that matched this consumer's filter and landed in its database. And scanned_height is how far the consumer has verified: the height below which it can prove there is nothing addressed to it. Laid on the chain itself, the three heights tell the story of a marketplace that went quiet:

On a busy feed all three heights travel together, which is why the distinction is easy to never notice: it only becomes visible when a filter goes quiet. The backlog is zero either way, and the parked cursor is the correct durable state.

A consumer never reads blocks, on restart or at any other time. The server already read every block, decoded every event, and filed the results in a sorted index: a card catalog that took millions of blocks of effort to build, maintained so nobody has to walk the shelves again. The consumer's entire relationship with the chain is one question, asked every 500 milliseconds:

GET /v1/index/contract-calls
    ?contract_id=SP...marketplace-v4
    &function_name=purchase-asset
    &cursor=8637064:1
    &limit=200

When the answer is "no rows," the consumer has not skipped forty thousand blocks of work; there was no work sitting there to skip, just one question with a short answer. And because a seek costs the same at any cursor age^¹1. B-tree index on (filter, cursor); the seek is O(log n) over the event count, not the block count, so cursor age never enters the cost., restarting "from" 8,637,064 costs exactly what polling from the tip would cost:

The gap between cursor and tip contains, provably per the index, nothing addressed to this consumer, which is why re-asking about it costs nothing.

There are two things a checkpoint could record. A bookmark says where I was looking. A receipt says what I have safely stored. They sound interchangeable, and on a busy feed they behave identically, so the difference only surfaces in what each one costs when the process dies.

So only the receipt gets a transaction, committed atomically with the rows, which is the entire correctness story of a sink. The bookmark gets recomputed, because persisting it would mean a disk write every 500 milliseconds, forever, on every idle consumer: permanent rent paid to remember a fact the catalog re-tells you for free. The contract says it plainly:

That quote is about reorgs, and it reveals the receipt's second property: it points at a row you physically hold. If the chain reorganizes, "the last event I stored" is still a meaningful coordinate: rollback deletes above the fork and the receipt rewinds with it, one transaction again. A bookmark parked at the unfinalized tip can point at a block that stops existing.

If your instinct says the checkpoint should advance through empty ranges anyway, that instinct probably comes from systems where it does, and in those systems, it should. Subsquid processors and graph-node subgraphs checkpoint the last processed block. Kafka consumers commit offsets that march through every message. Those are the same rule applied to a different shape of work, because in those systems the client does the walking: the traversal is real, client-side work, and losing your place means redoing it.

So the two pointers are really one pointer whenever everything gets delivered. Kafka delivers every offset, a block walker processes every block, the two coincide, and nobody needs two words for them. They only split when a server-side filter makes delivery sparse, which is exactly when the catalog is doing the walking for you and "the last thing you gave me" falls behind "the last thing you checked." That gap in the log line is the seam where the walk moved off your machine.

The split also means there are two honest ways to measure distance from the tip, and they diverge exactly when the feed goes quiet. Event age (tip − last_delivered_height) is how old the newest matching event is: on our quiet marketplace it climbs all day and that is a fact about the contract, not the consumer. Backlog (tip − scanned_height) is how much verified ground the consumer still has to cover: it falls through a backfill and holds at zero once caught up. This is what blocks_behind reports.

The distinction matters most for monitoring. A dashboard that alerts on event age will page someone about a consumer doing exactly its job. Gate liveness on ok (page recency) and capacity on blocks_behind (backlog); read last_delivered_height as the age of your data, which the chain controls and your infrastructure doesn't.

scanned_height is also conservative on purpose. A capped server scan only claims the cursor it returned; finalizedOnly consumers stop at the finalized boundary, because the unfinalized tail is deliberately re-read until it settles; and a reorg rolls the verified position back below the fork, because the new chain above it is unread.

The behaviors that follow from this design, so none of them read as failures:

• Restart prints the same cursor when the feed is quiet. The checkpoint is the receipt for the last delivered row; it moves when delivery moves. Resuming re-verifies the empty range in one query.
• A caught-up tail is silent. No matching events means no output and no checkpoint writes; idle_s and ok on the health endpoint are the proof of life, not the log.
• blocks_behind falls during a backfill and reads ~0 at the tail, even while event age grows, because the two numbers measure different things.
• Checkpoint writes are transactional with your rows. A crash mid-batch leaves neither rows nor cursor; the batch is simply re-read. That contract, and how to implement it for your own store, is the sink interface.

Checkpoint design mostly comes down to one question: what can you afford to lose? Facts that are expensive to lose (delivered data) get durable, transactional storage. Facts that are free to re-derive (where the scan had gotten to) get recomputed and merely reported. When a surface like this is confusing, it is often two facts with different loss-costs sharing one name, and giving them separate names is most of the fix.

There's plenty I've skipped: exactly-once delivery (a lie, but a useful one: cursors as idempotency keys get you the honest version), what reorg rollback actually deletes, and why the cursor commits in the same transaction as your rows rather than after them. Those live in the sinks docs.

Anyway: next time a dashboard number feels wrong, it's worth asking which fact it's actually measuring before assuming something broke.