# Self-host The same product on your Postgres: one container, no cluster, no queue. Bring your own node, or bundle one. Install the CLI with `bun add -g @secondlayer/cli`. Sizing: [Hardware](#hardware); [Runes](#runes) adds a Bitcoin node. ## Start it ### Configure ```bash git clone https://github.com/ryanwaits/secondlayer cd secondlayer/docker/oss # Set POSTGRES_PASSWORD in .env. cp .env.example .env # Writes the instance token and signing keys. secondlayer init --network mainnet cat .env.local >> .env ``` ### Start ```bash docker compose pull docker compose up -d curl http://127.0.0.1:3800/health ``` - `init` generates the instance token and signing keys. Appending them puts everything compose reads in one `.env`. - `pull` fetches the published images. Without it, compose reuses any older copy already on the box. - If `/health` never answers, `docker compose logs secondlayer` says why, most often a [floor](#floors-enforced-at-start). ### Use these docs on your box Every example in these docs reads `SECONDLAYER_API_URL` (and `SECONDLAYER_API_KEY` where a header is shown). On your box, point the URL at your own API. ```bash export SECONDLAYER_API_URL=http://127.0.0.1:3800 ``` - Reads on a loopback bind need no key. A header carrying an unknown or empty key is ignored there, so the same curl lines work unchanged. - Writes over curl, and any read past loopback, send `Authorization: Bearer $INSTANCE_TOKEN`. - The CLI, SDK and MCP send `INSTANCE_TOKEN` for you on any URL that is not the hosted API. - MCP needs the token in the client config's `env`, not just your shell. So `secondlayer subgraphs deploy` needs only the token in your env: ```bash export INSTANCE_TOKEN=… # from .env.local secondlayer subgraphs deploy ./my-subgraph.ts ``` Leave `SECONDLAYER_API_KEY` for the hosted account: archive quote, fetch and credits read it as the account key, so setting it to the instance token breaks them. Keep `POSTGRES_PORT` and `INDEXER_PORT` on loopback. ### Fill it with history A new box holds only what your node sends from now on. Restore verified history from the [archive](/docs/archive): ```bash export DATABASE_URL=postgres://secondlayer:@127.0.0.1:5432/secondlayer secondlayer bootstrap --against https://archive.secondlayer.tools/latest.json ``` ```text ~/secondlayer $ secondlayer bootstrap --against coverage genesis → 8,745,422 signature verified sync from genesis ~19.4 days (estimate) restore from archive ~2.3 hours ``` The official archive quotes a price in [credits](/docs/authentication#archive-fetches) before it fetches. Verify is free. ### Check it Compare what you restored with the signed archive. Free, and read-only. ```text ~/secondlayer exit=0 $ secondlayer verify raw --against https://archive.secondlayer.tools/latest.json ✓ Local data matches the archive across 183 ranges. reference signature verified ``` ### Stop it ```bash docker compose down ``` The index, chainstate and keys survive. Never add `-v`: it deletes the index volume. ## Production Ready for real traffic: your node, a pinned version, backups, upgrades. Starting out? Run it locally first with [Start it](#start-it). ### Profiles | Profile | What runs | | --- | --- | | default | `postgres` + `secondlayer` (API, ingest, decoder, subgraph, webhooks) | | `full-node` | + bundled Stacks node and bitcoind | | `bitcoin` | + `runes`, the [Runes](#runes) indexer | Set them in `.env` so every `docker compose` command picks them up: ```bash .env COMPOSE_PROFILES=full-node ``` | `NODE_MODE` | You bring | Also set | | --- | --- | --- | | `external` (default) | Your Stacks node | | | `stacks` | Your Stacks node, same as `external` | | | `full` | Nothing: both nodes run in the box | `BITCOIN_RPC_PASSWORD`, the `full-node` profile | There is no bundled Stacks-only mode: a bundled node needs Bitcoin data, and there is no public RPC default for it. Sizing for each: [Hardware](#hardware). ### Pin a version `docker compose up` runs `latest`. Pin a commit so a restart never changes what you run: ```bash .env SECONDLAYER_VERSION=975e0b9 ``` - Images are `ghcr.io/ryanwaits/secondlayer-runtime` and `secondlayer-bitcoin`, tagged by short commit. - `docker compose up -d --build` builds your checkout instead. ### Connect your node With `NODE_MODE=external`, this entry in your node's `Config.toml` sends Secondlayer every block. ```toml Config.toml [[events_observer]] endpoint = "10.0.0.5:3700" events_keys = ["*"] timeout_ms = 500 disable_retries = true ``` `secondlayer observer --mode signer-shared --endpoint 10.0.0.5:3700` prints it. | Field | `indexer` | `signer-shared` | Effect | | --- | --- | --- | --- | | `endpoint` | `host:port` | `host:port` | Where the node posts blocks. No scheme. | | `events_keys` | `["*"]` | `["*"]` | Every standard event. Unknown keys stop a stock node from starting. | | `timeout_ms` | `2000` | `500` | How long the node waits on each delivery. | | `disable_retries` | `false` | `true` | `false` retries until delivered, holding up the node meanwhile. | Node on another host: | Setting | Value | Why | | --- | --- | --- | | `INDEXER_PORT` in `.env` | `10.0.0.5:3700` | The default `127.0.0.1:3700` is unreachable off-box. | | `STACKS_NODE_RPC_URL` in `.env` | `http://:20443` | Polled for the tip when pushes stop. | | Firewall | `3700` open to the node's IP only | The observer port takes unauthenticated posts. | > **Signer on the same node** > Use `signer-shared`, so a stalled Secondlayer can't hold up the signer. Tradeoff: a missed block stays a gap until `secondlayer repair` pays [credits](/docs/authentication#archive-fetches) to fetch it. ### Config Six required non-secrets, plus one compose derives for you. Secrets come from `secondlayer init`. | Key | Default | Notes | | --- | --- | --- | | `NETWORK` | `mainnet` | `mainnet`, `testnet`, `devnet` | | `DATABASE_URL` | compose | Injected; one Postgres | | `NODE_MODE` | `external` | `external`, `stacks`, `full` | | `DATA_DIR` | `/data` | Handler + runtime files | | `API_PORT` | `127.0.0.1:3800` | Publish spec for the API port | | `INDEXER_PORT` | `127.0.0.1:3700` | Publish spec for the observer port | | `API_PUBLISH_ADDR` | compose | Injected from the same `API_PORT` value the port is published with | Unknown keys and contradictions fail before ingest. ### Instance token The instance token opens your box. `init` writes it to `.env.local`. - The variable is `INSTANCE_TOKEN`. It is not the account key from [Keys and billing](/docs/authentication). - Never put `sk-sl_*` in `INSTANCE_TOKEN`, or the hex token in `SECONDLAYER_API_KEY`. | Surface | Credential | Behavior | | --- | --- | --- | | `/v1` on `127.0.0.1` | none | your Postgres, no meter | | `/v1` past loopback | `INSTANCE_TOKEN` | same header on Index, Streams, subgraphs | | Writes (`/api/subgraphs`, `/api/webhooks`, `/api/node`) | `INSTANCE_TOKEN` | loopback included | ```bash # Local read: no key curl "http://127.0.0.1:3800/v1/index/events?limit=5" # Write: instance token, from 127.0.0.1 too curl -X POST http://127.0.0.1:3800/api/subgraphs/my-balances/reindex \ -H "Authorization: Bearer $INSTANCE_TOKEN" ``` Loopback is the publish spec (`API_PUBLISH_ADDR`), not the caller's address. On a keyless read, a wrong key is ignored; a matching token is used. Compose requires the token, for two reasons: - The container process has to bind `0.0.0.0` or the published port never reaches it. - An instance that binds past loopback with no token refuses to start. `API_PUBLISH_ADDR` is how the API learns which of those two facts to serve on. It carries the publish spec: - With the default `API_PORT` of `127.0.0.1:3800`, the `/v1` reads stay keyless. - Changing `API_PORT` to `0.0.0.0:3800` makes every `/v1` request carry the token. - Writes take the token either way. ### What differs on your box | Area | On your box | | --- | --- | | Metering | None. Your Postgres, no meter, no rate limit | | Subgraph sources | Any. Hosted takes event filters, `contract_call` and `contract_deploy` | | Typed ORM schemas | `secondlayer codegen subgraph` needs direct Postgres, so it is self-host only | | Handler RPC calls | Set `STACKS_NODE_RPC_URL` on the subgraph processor, pointing at the node you already run | | History below your bootstrap floor | A metered [archive backfill](/docs/authentication#archive-fetches) | | Webhook limits | Any `max_retries` and `timeout_ms`, no rate limits, egress as your network allows | | Proof routes | Witness, burn and Bitcoin-header routes answer `503 PROOFS_UNAVAILABLE` until `PROOF_SIDECAR_URL` points at a proof sidecar. The Bitcoin transaction route reads the bundled bitcoind (full-node profile, `txindex=1`) or the node at `BITCOIN_RPC_URL`, and answers `502 PROOF_SOURCE_ERROR` when it can't reach one | | Bitcoin output spends | `/v1/bitcoin/outspend` reads the same node and needs Bitcoin Core 24 or later. It answers `503 BITCOIN_UNAVAILABLE` until `BITCOIN_RPC_URL` is set, and `502 BITCOIN_SOURCE_ERROR` when it can't read the node | | Error reports | No `/v1/feedback`. Error bodies link to a GitHub issue instead | Consumers running on another box: [Deploy your app](/docs/deploy). ### Verify and repair ```bash secondlayer verify all --against secondlayer repair --against # plan secondlayer repair --against --apply ``` [Verified archive](/docs/archive#flags) owns flags and exit codes. ### Archive trust root Every archive manifest is checked against one key before the CLI acts on it. The key comes from, in order: 1. `--public-key` 2. `ARCHIVE_SIGNING_PUBLIC_KEY` in your env 3. The key compiled into the CLI release - It applies to `bootstrap`, `verify` and `repair`. - `init` writes the resolved key into `.env.local`, so the instance verifies with no path to any Secondlayer host. - Plaintext `http://` key sources are never consulted. - A signed `latest.json` pointer has to verify against the same key before its snapshot is followed. The tradeoff: rotating the archive key means a CLI release until a key ceremony lands. To pin a different archive: - Replace `ARCHIVE_SIGNING_PUBLIC_KEY` in `.env`. - A value you set there survives `init` re-runs. ### Running other teams' subgraphs A subgraph handler is code you deploy into the instance, so it runs with the instance's trust. That is fine for your own subgraphs. To run another team's on one box, give each team its own stack instead of sharing one: - Its own Postgres and its own network. - Its `api` and `subgraph-processor` containers under gVisor (`runtime: runsc`), as a non-root user. - A read-only root filesystem, with bounded memory, CPU and process count. [`docker/workload/tenant.compose.yml`](https://github.com/ryanwaits/secondlayer/blob/main/docker/workload/tenant.compose.yml) is the working shape. gVisor can't reach Docker's embedded DNS, so give each stack a static address for Postgres and a plain `resolv.conf`. The file explains both. ### Not in the box - Multi-tenant catalog - A managed stack: [hosted subgraphs](/docs/subgraphs#deploy) and [webhooks](/docs/webhooks) run on `api.secondlayer.tools` with an account key - Archive publisher (separate image) - Pay-per-call metering on your instance (the x402 rail was removed) Support is community: [github.com/ryanwaits/secondlayer](https://github.com/ryanwaits/secondlayer). ## Runes Index every rune etch, mint, transfer and burn since block 840,000 on your own box, read through the same [Runes](/docs/index#runes) routes as hosted. Add these lines to `.env`, pointing at a Bitcoin node with `txindex=1`: ```bash .env COMPOSE_PROFILES=bitcoin BITCOIN_RPC_URL=http://10.0.0.6:8332 BITCOIN_RPC_USER=bitcoin BITCOIN_RPC_PASSWORD=… ``` ```bash docker compose pull docker compose up -d curl "http://127.0.0.1:3800/v1/index/runes/doggotothemoon" ``` The first sync reads every block from 840,000 and takes about half a day. `tip.block_height` in each response shows how far it has read. The tradeoff: Runes has no archive yet, so a new box rebuilds from your node instead of downloading, and `secondlayer verify` covers Stacks only. ### Bitcoin node | Node | `.env` | Notes | | --- | --- | --- | | Yours | `BITCOIN_RPC_URL`, `BITCOIN_RPC_USER`, `BITCOIN_RPC_PASSWORD` | Needs `txindex=1`, so not pruned | | Bundled | `COMPOSE_PROFILES=full-node,bitcoin`, no `BITCOIN_RPC_URL` | Also runs the Stacks node; bitcoind syncs first, which takes days | ### Runes only No Stacks node? Skip [Fill it with history](#fill-it-with-history) and connect nothing. The Stacks side still runs and stays empty; only `runes` reads a chain. ### Upgrades with Runes A schema migration on a full Runes database can run for hours on upgrade. The Runes tip holds still until it finishes; that is expected, not a hang. ## Hardware What a self-hosted box needs, so you size it before the sync starts rather than after. Mainnet full history is large: the reference index measures ~500 GB. - About 250 GB is `blocks`, `transactions`, and `events`. The rest is the decoded plane. - Budget roughly 3 GB per 100k blocks for core datasets, or 6 GB with a broad decoder set. - Those are averages over history that is much denser at the tip. | Setup | RAM | Disk | | --- | --- | --- | | App services, external node, mainnet | 4 GB | 600 GB | | App services, external node, testnet/devnet | 4 GB | 80 GB | | Bundled node, mainnet (floor) | 96 GB | 2.5 TB | | Bundled node, mainnet (recommended) | 128 GB | 3 TB NVMe | ### Postgres Postgres `max_connections` ≥ 50. - Compose sets the Postgres cache to 1 GB (`POSTGRES_SHARED_BUFFERS`). - For a full-chain sync, give it about 25% of RAM. At the stock 128 MB, inserts wait on index reads from disk and the sync runs slower. - The bundled-node figure is dominated by the chain itself: Stacks chainstate alone measures 1.3 TB, and bitcoind is additional. ### With Runes [Runes](#runes) adds to the rows above: | Bitcoin node | RAM | Disk | | --- | --- | --- | | Yours, elsewhere | +22 GB | +100 GB | | bitcoind on the same box | +22 GB | +1.1 TB | `runes` keeps Runes state in memory: 7 to 12 GB while following the tip, up to 17 GB during the first sync, capped at 22 GB. ### Floors enforced at start The runtime checks RAM, disk, and `max_connections` at start and refuses to boot below these floors, so an undersized box finds out before it spends hours acquiring an index it cannot hold. Disk is total filesystem capacity, not free space, since a healthy instance legitimately fills most of its disk. An unmeasurable dimension is reported and skipped rather than assumed bad. To run below a floor anyway: ```bash SECONDLAYER_ALLOW_UNDERSIZED=true ``` That downgrades the refusal to a warning at every start. It exists so a box that was already undersized can still come back up after an upgrade: you keep the ability to run your own hardware, you just do it knowingly. ## Upgrades Back up first, then move to a newer release. ### Backup `secondlayer backup` writes one bundle: the index, plus the keys encrypted under a passphrase you supply. ```bash secondlayer backup --out ./backups/$(date +%F) secondlayer restore --from ./backups/2026-08-16 # dry run secondlayer restore --from ./backups/2026-08-16 --apply ``` | Flag or env | Effect | | --- | --- | | `--passphrase` or `SECONDLAYER_BACKUP_PASSPHRASE` | The passphrase the keys are encrypted under | | `--no-secrets` | Writes a data-only bundle | | `--force` (restore) | Lets a restore write over a database that already holds chain data | A restore refuses before writing anything if the secrets key the instance will use is not the key the data was encrypted with. A restore is all or nothing: - `pg_restore` runs in one transaction and stops on the first error, so a disk that fills mid-load leaves the target as it was rather than half replaced. - The tradeoff is no parallel restore. - After the load the canonical block range is compared with the manifest's scope. A dump that stops short of what it promised exits 1 instead of reporting success. - The dump digest is streamed, so a multi-gigabyte bundle hashes without being read into memory. The bundle carries three keys: - `INSTANCE_TOKEN` - `SECONDLAYER_SECRETS_KEY` - `STREAMS_SIGNING_PRIVATE_KEY` Losing those costs you more than losing the index. The index can be rebuilt from the [archive](/docs/archive); the keys cannot. Store the bundle somewhere your Postgres backups are not. Confirm any restore with `secondlayer verify all --against ` before trusting the result. Take a backup before you [upgrade](#upgrade). ### Upgrade Move from the commit you run (X) to a later one (Y), keeping the database and the keys. ```bash curl -s "$SECONDLAYER_API_URL/health" docker compose images secondlayer --version ``` One jump to Y; read every [changelog](/docs/changelog) entry dated after X first, since that page owns what broke. The CLI is a separate npm version: bump it to match the instance you land on. The index can be rebuilt from the [archive](/docs/archive). The keys cannot. | Keep | Why | | --- | --- | | `postgres_data` volume | chain, subgraph tables, webhook rows | | `subgraphs_data` volume | compiled handlers | | `SECONDLAYER_SECRETS_KEY` | decrypts webhook signing secrets in Postgres | | `INSTANCE_TOKEN` | callers; required at boot | | `STREAMS_SIGNING_PRIVATE_KEY`, `SECONDLAYER_WEBHOOK_SIGNING_PRIVATE_KEY` | platform signature on deliveries | | `POSTGRES_PASSWORD` | the volume already uses it | Losing `SECONDLAYER_SECRETS_KEY` makes every existing `whsec_` unreadable; rotate each webhook rather than minting a new master key (signing secrets). A full dump of a mainnet index takes hours, so a filesystem snapshot of the volume is the faster [backup](#backup) if the host can take one. > **Do not wipe the volume** > - `docker compose down -v` deletes `postgres_data`. - `secondlayer init --force` mints a new secrets key that leaves existing webhook secrets unreadable. - Stop with `docker compose down` (no `-v`). - Bootstrap is not an upgrade: it refuses a database that already has blocks. Check out Y for its compose file, pin the same commit, then restart: ```bash git fetch && git checkout # .env: SECONDLAYER_VERSION= docker compose pull docker compose up -d --remove-orphans ``` `--remove-orphans` stops services Y's compose file no longer lists. Migrations run at boot under an advisory lock, forward-only. Then confirm `/health` is ok and verify against the archive: ```bash curl "$SECONDLAYER_API_URL/health" secondlayer verify all --against ``` To roll back, pin X again. A migrated schema stays migrated, so roll back the image, not the database. ## Notes for agents - Start: 'git clone https://github.com/ryanwaits/secondlayer', 'cd secondlayer/docker/oss', 'cp .env.example .env' (set POSTGRES_PASSWORD), 'secondlayer init --network mainnet' (writes .env.local), 'cat .env.local >> .env', 'docker compose pull', 'docker compose up -d', then 'curl http://127.0.0.1:3800/health'. If health never answers, 'docker compose logs secondlayer' names the refusal (usually a disk or RAM floor; SECONDLAYER_ALLOW_UNDERSIZED=true in .env overrides). There is no 'secondlayer setup' or 'secondlayer uninstall'. - History: export DATABASE_URL=postgres://secondlayer:@127.0.0.1:5432/secondlayer, then 'secondlayer bootstrap --against https://archive.secondlayer.tools/latest.json' (quotes the price in credits first) and 'secondlayer verify all --against '. For an external node, 'secondlayer observer --mode indexer' prints the Config.toml stanza. - Profiles and version live in .env: COMPOSE_PROFILES=full-node|bitcoin (comma-separated), SECONDLAYER_VERSION= (default latest). - Point the docs examples at the box: export SECONDLAYER_API_URL=http://127.0.0.1:3800. Loopback reads need no key; writes and any read past loopback send 'Authorization: Bearer $INSTANCE_TOKEN' (the CLI and SDK send it). - Leave SECONDLAYER_API_KEY alone: it is the hosted account key that credits and archive fetches read. - Published images: 'ghcr.io/ryanwaits/secondlayer-runtime' and 'secondlayer-bitcoin', tagged by short commit; compose pulls them. 'docker compose up -d --build' builds the checkout instead. - Genesis sync needs a Stacks node that is itself syncing from genesis (an already-synced node only sends new blocks). Start with TIP_FOLLOWER_ENABLED=false, track 'curl http://localhost:3700/health | jq .block_height' against the chain tip, re-enable the tip follower, run 'secondlayer verify all --against ', then deploy with SECONDLAYER_API_URL=http://127.0.0.1:3800. - Backup: 'secondlayer backup --out ./backups/$(date +%F)' with --passphrase or SECONDLAYER_BACKUP_PASSPHRASE. The bundle carries INSTANCE_TOKEN, SECONDLAYER_SECRETS_KEY and STREAMS_SIGNING_PRIVATE_KEY, so store it apart from Postgres backups. Restore: 'secondlayer restore --from ' is a dry run; --apply writes; confirm with 'secondlayer verify all --against '. - Upgrade X to Y: identify X with 'curl -s $SECONDLAYER_API_URL/health', 'docker compose images', 'secondlayer --version'. Ask for Y (a git commit). Read changelog entries dated after X. Keep postgres_data, subgraphs_data and .env (a new SECONDLAYER_SECRETS_KEY makes existing whsec_ secrets unreadable). 'git checkout ', set SECONDLAYER_VERSION= in .env, 'docker compose pull', 'docker compose up -d --remove-orphans'. Confirm /health and verify all. Rollback pins X; the schema stays migrated, so only the image rolls back. - Runes: add COMPOSE_PROFILES=bitcoin and BITCOIN_RPC_URL/USER/PASSWORD (node with txindex=1) to .env, then 'docker compose up -d'. Bundled bitcoind: COMPOSE_PROFILES=full-node,bitcoin with BITCOIN_RPC_URL unset. First sync starts at block 840,000 and takes about half a day; tip.block_height on any /v1/index/runes response is the indexed height (0 = not started). Runes has no archive: 'secondlayer verify' and 'repair' do not cover it. Size: +22 GB RAM, +100 GB disk, plus about 1 TB if bitcoind runs on the same box. - Never 'docker compose down -v'. Never 'secondlayer init --force' on a running instance (new secrets key). - Floors: app services with an external node need 4 GB RAM and a 600 GB SSD on mainnet (80 GB on testnet and devnet). A bundled node needs 96 GB RAM and 2.5 TB; 128 GB and 3 TB NVMe recommended. - Postgres max_connections must be at least 50. Set POSTGRES_SHARED_BUFFERS to about 25% of RAM for a full-chain sync. - The runtime refuses to boot below the floors unless SECONDLAYER_ALLOW_UNDERSIZED=true. ## Setup Ensure setup once, skipping any step already done: - Skill: `bunx skills add ryanwaits/secondlayer --skill secondlayer -y` - CLI: `bun add -g @secondlayer/cli` - Instance: `secondlayer init` writes `INSTANCE_TOKEN`. Loopback `/v1` reads need no key. - Hosted/archive: export `SECONDLAYER_API_KEY` (`sk-sl_*`) or buy credits with `secondlayer credits buy` (`balance`, `refill`), then `secondlayer whoami`