The checkpoint is a receipt, not a bookmark
Restart a consumer and it logs the same cursor it logged an hour ago, forty thousand blocks behind the tip. Nothing is stuck. Here's how to read every number your indexer reports, and why the design works this way.
Restart an indexer built on a consume loop and it prints something like this:
resuming from 8637064:1Which is fine, except it printed the same thing yesterday, and the chain tip is forty thousand blocks past that cursor now. It's tempting to read this as a stuck consumer: surely "continue where you left off" should mean the last place you checked, not the last place you found something.
It isn't stuck, and the distinction between those two places is the single most useful thing to understand about how checkpointed consumers work. This guide walks the whole mechanism: what the three heights on the health endpoint mean, what a poll actually costs, why the checkpoint stays parked on a quiet feed, and what behavior to expect from your own consumers.
A consumer's health endpoint reports three different heights, and the whole story lives in the gaps between them:
{
"ok": true,
"checkpoint": "8637064:1",
"last_delivered_height": 8637064,
"scanned_height": 8676502,
"tip_height": 8676502,
"blocks_behind": 0
}tip_height is the chain tip; easy. last_delivered_height is the block of the last event that matched this consumer's filter and landed in its database. And scanned_height is how far the consumer has verified: the height below which it can prove there is nothing addressed to it. Laid on the chain itself, the three heights tell the story of a marketplace that went quiet:
On a busy feed all three heights travel together, which is why the distinction is easy to never notice: it only becomes visible when a filter goes quiet. The backlog is zero either way. What follows explains why that's true, and why the parked cursor is the correct durable state.
The key fact behind everything else: a consumer never reads blocks. Not on restart, not ever. The server already read every block, decoded every event, and filed the results in a sorted index: a card catalog that took millions of blocks of effort to build, maintained so nobody has to walk the shelves again. The consumer's entire relationship with the chain is one question, asked every 500 milliseconds:
GET /v1/index/contract-calls
?contract_id=SP...marketplace-v4
&function_name=purchase-asset
&cursor=8637064:1
&limit=200When the answer is "no rows," the consumer has not skipped forty thousand blocks of work. There was never forty thousand blocks of work. There was one question, and the answer was no. And because a seek costs the same at any cursor age1. B-tree index on (filter, cursor); the seek is O(log n) over the event count, not the block count, so cursor age never enters the cost., restarting "from" 8,637,064 costs exactly what polling from the tip would cost:
The gap between cursor and tip contains, provably per the index, nothing addressed to this consumer. Re-asking is free. Everything else in this guide follows from that.
There are two things a checkpoint could record. A bookmark says where I was looking. A receipt says what I have safely stored. They sound interchangeable. They are not, and the difference is what each one costs when the process dies.
bookmark (where I was looking)
Recovery is one free question: "anything after my last receipt?", answered in a millisecond by the first poll after restart.
cost: ~1ms, once
receipt (what I have stored)
Recovery is data corruption: re-deliver rows you already stored (duplicates), or assume delivery that never happened (a silent gap you find weeks later, which is worse).
cost: your data
So only the receipt gets a transaction, committed atomically with the rows, which is the entire correctness story of a sink. The bookmark gets recomputed, because persisting it would mean a disk write every 500 milliseconds, forever, on every idle consumer: permanent rent paid to remember a fact the catalog re-tells you for free. The contract says it plainly:
Roll the projection back to the fork: delete everything at or above the fork point and commit the rewound cursor in the same transaction. Deleting without the rewound cursor is the classic silent-gap bug.
That quote is about reorgs, and it reveals the receipt's second property: it points at a row you physically hold. If the chain reorganizes, "the last event I stored" is still a meaningful coordinate: rollback deletes above the fork and the receipt rewinds with it, one transaction again. A bookmark parked at the unfinalized tip can point at a block that stops existing.
If your instinct says the checkpoint should advance through empty ranges anyway, that instinct probably comes from systems where it does, and in those systems, it should. Subsquid processors and graph-node subgraphs checkpoint the last processed block. Kafka consumers commit offsets that march through every message. Not design mistakes: the same rule applied to a different shape of work. In those systems the client does the walking: the traversal is real, client-side work, and losing your place means redoing it.
(kafka / block walker)sparse filter
(one contract’s sales)
Here's the unification, and it's the one idea to leave with. A receipt and a bookmark are the same pointer whenever everything is delivered. Kafka delivers every offset; a block walker processes every block; the two coincide and nobody needs two words. They only split when a server-side filter makes delivery sparse: when the catalog walks so you don't have to, and "the last thing you gave me" falls behind "the last thing you checked." The split is the visible seam of moving the walk server-side. It shows up in a log line and looks like a bug; it's the design working.
The split also means there are two honest ways to measure distance from the tip, and they diverge exactly when the feed goes quiet. Event age (tip − last_delivered_height) is how old the newest matching event is: on our quiet marketplace it climbs all day and that is a fact about the contract, not the consumer. Backlog (tip − scanned_height) is how much verified ground the consumer still has to cover: it falls through a backfill and holds at zero once caught up. This is what blocks_behind reports.
The distinction matters most for monitoring. A dashboard that alerts on event age will page someone about a consumer doing exactly its job. Gate liveness on ok (page recency) and capacity on blocks_behind (backlog); read last_delivered_height as what it is: the age of your data, driven by the chain, not by your infrastructure.
scanned_height is also conservative on purpose. A capped server scan only claims the cursor it returned; finalizedOnly consumers stop at the finalized boundary, because the unfinalized tail is deliberately re-read until it settles; and a reorg rolls the verified position back below the fork, because the new chain above it is unread.
The behaviors that follow from this design, so none of them read as failures:
- Restart prints the same cursor when the feed is quiet. The checkpoint is the receipt for the last delivered row; it moves when delivery moves. Resuming re-verifies the empty range in one query.
- A caught-up tail is silent. No matching events means no output and no checkpoint writes;
idle_sandokon the health endpoint are the proof of life, not the log. blocks_behindfalls during a backfill and reads ~0 at the tail, even while event age grows. Different facts.- Checkpoint writes are transactional with your rows. A crash mid-batch leaves neither rows nor cursor; the batch is simply re-read. That contract, and how to implement it for your own store, is the sink interface.
Checkpoint design compresses to one question: what can you afford to lose? Facts that are expensive to lose (delivered data) get durable, transactional storage. Facts that are free to re-derive (where the scan had gotten to) get recomputed and merely reported. Most of the confusing surfaces in distributed systems are two facts with different loss-costs wearing the same name, and most of the fix is giving them different names.
There's plenty this guide skips: exactly-once delivery (a lie, but a useful one: cursors as idempotency keys get you the honest version), what reorg rollback actually deletes, and why the cursor commits in the same transaction as your rows rather than after them. Those live in the sinks docs.
Anyway: if your indexer greets you with the same cursor every restart, you can now read it as the receipt it is. And when a number on a dashboard feels wrong, it's worth asking which fact it's actually measuring.