<!-- source: https://secondlayer.tools/docs/streams -->

# Streams

The raw firehose, for when you want to decode it yourself. Most people want [Index](/docs/index); this is what Index is built on.

Hosted reads are keyed ([Keys and billing](/docs/authentication)); on your own box see [Self-host](/docs/self-host#instance-token). Cold history beyond what your instance holds restores from the signed archive (credits).

## Build an indexer on it

`events.consume` is the checkpointed consumer with automatic reorg rewind; the loop is the one on [Index](/docs/index#build-your-index-on-it).

```ts indexer.ts figure
await sl.streams.events.consume({
  types: ["ft_transfer"],
  // Null on first run.
  fromCursor: await db.loadCheckpoint(),
  onBatch: async (events, envelope, ctx) => {
    await db.transaction(async (tx) => {
      await tx.insertEvents(events);
      // Commits with the rows.
      await tx.saveCheckpoint(ctx.cursor);
    });
    return ctx.cursor;
  },
  // Delete from the fork height, inclusive.
  onReorg: (reorg) => db.deleteFromHeight(reorg.fork_point_height),
});
```

A consumer opens at your checkpoint, writes rows and cursor in one transaction, and rewinds when a reorg lands. Each page ends with `next_cursor`; hand it back as `fromCursor` and the next page starts right after it.

Where to run it: [Deploy](/docs/deploy). Dated changes land in the [changelog](/docs/changelog).

## Read it directly

The loop is this call on repeat. A live page from mainnet, trimmed:

```bash
curl "$SECONDLAYER_API_URL/v1/streams/events?types=ft_transfer&limit=100" \
  -H "Authorization: Bearer $SECONDLAYER_API_KEY"
```

Pass the response's `next_cursor` back as `cursor`. Filters are exact-match, and event types without a given field don't match.

| Filter | Narrows to |
| --- | --- |
| `types`, `not_types` | Event types, included or excluded |
| `contract_id` | One contract |
| `sender`, `recipient` | The two sides of a transfer |
| `asset_identifier` | One asset |
| `filters` | Several named concerns on one cursor ([Filters](#filters)) |
| `from_height`, `to_height` | A block-height range |

Full grammar: [API reference](/docs/api/streams#list-stream-events).

```json response.json status=200
{
  "events": [
    {
      "cursor": "9210757:0",
      "block_height": 9210757,
      "tx_id": "0x0d70db6001a4f477fa032622a6aa51e570874a1e1586987dae5533aab71b13bc",
      "event_type": "ft_transfer",
      "contract_id": "SM3VDXK3WZZSA84XXFKAFAF15NNZX32CTSG82JFQ4.sbtc-token",
      "payload": {
        "amount": "12354",
        "sender": "SP4SZE494VC2YC5JYG7AYFQ44F5Q4PYV7DVMDPBG.rewards-pox5-v1",
        "recipient": "SP4SZE494VC2YC5JYG7AYFQ44F5Q4PYV7DVMDPBG.stbtc-reserve"
      },
      "canonical": true,
      "finalized": true
    }
  ],
  "next_cursor": "9210757:0",
  "tip": { "block_height": 9211740, "finalized_height": 9211449, "lag_seconds": 11 },
  "reorgs": []
}
```

## Check it

The SDK checks every frame's ed25519 signature before `onBatch` sees it; pin the key and you trust only the key ([Per-frame signatures](#per-frame-signatures)). To check what you stored, compare it with the signed archive. A divergence reads as it does on [Archive](/docs/archive#check-it).

```text ~/secondlayer exit=0
$ secondlayer verify raw --against https://archive.secondlayer.tools/latest.json
✓ Local data matches the archive across 183 ranges.
  reference signature verified
```

## Reorg behavior

Reorgs rewind to the fork point and re-deliver, inclusive of the fork block. Delivery is at-least-once, so keep writes idempotent.

- Only forks at or below your checkpoint roll back; a fork above it has nothing to undo.
- A rewind deeper than `maxRollbackDepth` (default 1000 blocks) is refused before anything is deleted.
- On empty pages the loop polls `reorgs.list`, so a fork that lands while idle at the tip is caught too. The cost is one extra request per idle poll.
- A fork at genesis hands `onReorg` a `null` cursor.

## Cold history

`events.replay` backfills from the signed bulk dumps, then tails live from the manifest's `latest_finalized_cursor`, no gap or dupe at the seam:

```ts backfill.ts
await streams.events.replay({
  // Covers every window the dumps hold.
  from: "genesis",
  async onDumpFile(file, { from }) {
    // sha256-verified.
    const bytes = await streams.dumps.download(file);
    await ingestParquet(bytes, { after: from }); // file-granular, at-least-once: skip rows at or below `from`
  },
  async onBatch(events, envelope) {
    await db.insertEvents(events);
    return envelope.next_cursor;
  },
});
```

`from: "genesis"` covers every window the dumps hold; the manifest's `coverage` reports that range, which starts where the dump program began, not chain block 1. Pass a cursor for a partial backfill.

## Dumps and DuckDB

Plain parquet. Pull a verified range and query it locally:

```text ~/indexer
$ secondlayer streams dumps --to ./dump   # sha256-verified against the signed manifest
$ duckdb -c "SELECT event_type, count(*) AS events
             FROM read_parquet('./dump/**/*.parquet')
             GROUP BY 1 ORDER BY 2 DESC;"
┌──────────────┬────────┐
│  event_type  │ events │
├──────────────┼────────┤
│ print        │ 610417 │
│ nft_mint     │ 107479 │
│ stx_transfer │ 104690 │
│ ft_transfer  │  40294 │
│ …            │      … │
└──────────────┴────────┘
```

> **Manifest signature is the root of trust**
>

The bulk manifest is itself ed25519-signed and verified before any per-file hash is trusted. Listing dumps and `events.replay()` both enforce it; opt out with `verifyDumpsManifest: false`.

## From the CLI

Every read has a `secondlayer streams` subcommand that prints JSON.

| Command | Does |
| --- | --- |
| `tip` | Current canonical tip. |
| `events` | One page of events; takes the REST filters as flags. |
| `by-tx <txId>` | Every event a transaction emitted. |
| `block-events <heightOrHash>` | Every event in one block. |
| `canonical <height>` | Canonical block metadata at a height. |
| `consume` | Long-running pull from a cursor, one event per line, until SIGINT or `--max-pages`. |
| `reorgs` | Recent reorgs, cursor-paginated. |
| `dumps` | Download finalized parquet dumps to a local dir, sha256-verified. |

## Bitcoin events (Runes)

`chain=bitcoin` serves Runes events on their own cursor space.

| Event types |
| --- |
| `rune_etch` |
| `rune_mint` |
| `rune_transfer` |
| `rune_burn` |

```bash
curl "$SECONDLAYER_API_URL/v1/streams/events?chain=bitcoin&rune=840000:3" \
  -H "Authorization: Bearer $SECONDLAYER_API_KEY"
```

- A bitcoin cursor only resumes with `chain=bitcoin` set again.
- Stacks-only filters (`contract_id`, `sender`, `recipient`, `asset_identifier`, `filters`, `clock`) answer 400 here.
- `rune` and `address` are bitcoin-only.
- The tip and canonical block drop the burn-chain fields Stacks carries, since Bitcoin has no burn chain of its own.
- Events carry `ts`, the block's time.

The rune catalog, per-rune activity feed, and balances live on [Index](/docs/index#runes); this is the raw event log underneath them.

### Index Runes into your own database

`events.consume` takes `chain: "bitcoin"` and runs the same checkpointed loop as [Build an indexer on it](#build-an-indexer-on-it). Rows and cursor commit together, and a Bitcoin reorg deletes from the fork height.

```ts runes-indexer.ts
await sl.streams.events.consume({
  chain: "bitcoin",
  types: ["rune_etch", "rune_mint", "rune_transfer", "rune_burn"],
  fromCursor: await db.loadCheckpoint(),
  onBatch: async (events, envelope, ctx) => {
    await db.transaction(async (tx) => {
      await tx.insertRuneEvents(events);
      await tx.saveCheckpoint(ctx.cursor);
    });
    return ctx.cursor;
  },
  onReorg: (reorg) => db.deleteFromHeight(reorg.fork_point_height),
});
```

- `rune` (an id or name) or `address` narrows the feed.
- `finalizedOnly: true` waits out the 6 confirmations instead of handling reorgs.

## Filters

One `filters` object can follow several concerns on one cursor.

### Two concerns, one loop

A flat filter is one AND-ed set, so tracking sBTC mints *and* treasury spends
meant two consume loops, two cursors, two checkpoints, or fetching the union
and throwing half of it away. `filters` names each concern and the server ORs
them into a single scan:

```ts
await sl.streams.events.consume({
  filters: {
    peg: { types: ["ft_transfer"], assetIdentifier: `${SBTC}::sbtc-token` },
    treasury: { types: ["stx_transfer"], sender: TREASURY },
  },
  on: {
    peg: (events) => events.map((event) => event.payload.asset_identifier),
    treasury: (events) => events.map((event) => event.payload.sender),
  },
});
```

Each label's declared types narrow its handler, so `event.payload` is typed
without an `event_type` guard.

A label added to `filters` is a compile error until `on` handles it.

- A label's own events arrive in cursor order.
- Events of different labels don't interleave within a page.
- Add `onBatch` for the whole page when you need strict global order.

One cursor covers all of it. The position is computed over the block's full
event set, not the filtered one, so you can add or drop labels between restarts
and keep the checkpoint.

Over REST it's one JSON query param, and each event echoes the labels it hit:

```bash
curl -G --data-urlencode 'filters={"peg":{"types":["ft_transfer"]}}' \
  -H "Authorization: Bearer $SECONDLAYER_API_KEY" \
  "$SECONDLAYER_API_URL/v1/streams/events"
```

```json response.json status=200
{ "cursor": "8054704:12", "event_type": "ft_transfer", "matched": ["peg"] }
```

SSE takes the same param, so `subscribe` narrows the same way.

Deliveries are idempotent, so retries never double-count.

- Fully-finalized pages are served `Cache-Control: public, immutable` with an `ETag` (`304` on `If-None-Match`).
- The moving tip stays short-lived.

## Real-time firehose (SSE)

```bash
curl -N "$SECONDLAYER_API_URL/v1/streams/events/stream?types=ft_transfer" \
  -H "Authorization: Bearer $SECONDLAYER_API_KEY"
```

`GET /v1/streams/events/stream` takes the same filters as `/events`.

| Param | Does |
| --- | --- |
| `cursor` (or `from_cursor`), `from_height` | Start position. With none it live-tails from the reorg-clamped tip |
| `ping` | Keepalive, every 20s |

Wire contract: [REST API](/docs/rest-api#streaming-sse).

> **Near-real-time, not instant**
>

The server polls roughly every 1.5s from a tip held a couple of blocks back for reorg safety, so events land seconds after they appear on chain.

`client.events.subscribe(...)` auto-reconnects from the last delivered cursor and verifies per-frame signatures; verification modes are under [per-frame signatures](#per-frame-signatures). Fan out to your backend instead with [Webhooks](/docs/webhooks).

## Per-frame signatures

Each frame body is `{ event, sig, key_id }`. The signature is ed25519 over `JSON.stringify(event)`.

The SDK verifies every frame by default, against the key at `/public/streams/signing-key`. Lenient, strict, skip, and pinned-key modes: [SDK](/docs/sdk#verifying-responses).

## Notes for agents

- Firehose: 'GET $SECONDLAYER_API_URL/v1/streams/events' with 'Authorization: Bearer $SECONDLAYER_API_KEY'. Filter by types, contract_id, sender; page forward with next_cursor; deliveries are idempotent, so loop to stay live.
- Indexer: streams.events.consume. Write rows in onBatch and return ctx.cursor (or nothing), never envelope.next_cursor; a finalizedOnly consumer throws on it; roll back in onReorg from reorg.fork_point_height (inclusive).
- Cold history: events.replay({ from: 'genesis' }), then tail live at the seam.
- Bulk dumps: 'secondlayer streams dumps' (sha256-verified against the signed manifest); query locally in DuckDB with read_parquet('./**/*.parquet'). No indexer required.

## Setup

Ensure setup once, skipping any step already done:
- Skill: `bunx skills add ryanwaits/secondlayer --skill secondlayer -y`
- CLI: `bun add -g @secondlayer/cli`
- Instance: `secondlayer init` writes `INSTANCE_TOKEN`. Loopback `/v1` reads need no key.
- Hosted/archive: export `SECONDLAYER_API_KEY` (`sk-sl_*`) or buy credits with `secondlayer credits buy` (`balance`, `refill`), then `secondlayer whoami`