Serving RPC at scale

A single node serves a surprising amount of traffic and then stops doing so in a specific order. Knowing the order is most of the work, because each failure has a different fix and three of the four are not “add another node”.

• One method starves the rest. Log queries and traces are orders of magnitude more expensive than a balance read. A handful of unbounded ones will make every cheap call on the same node slow, and the graph will look like the node is overloaded when one query is.
• Subscriptions consume the connection budget. WebSocket subscriptions are long-lived state on the node. Thousands of clients each holding one is a different resource problem from request throughput and hits a different ceiling.
• The node needs maintenance. Upgrades, resyncs and disk work all take it offline. With one node that is downtime; the second node is bought for this at least as much as for load.
• Then, eventually, throughput. Genuinely running out of CPU is the last of the four to arrive and the first that people plan for.

A generic HTTP balancer will happily send traffic to a node that is answering perfectly and is four hundred blocks behind. Every request succeeds; every answer is wrong. Three things separate an RPC-aware balancer from one that merely distributes load.

Health checks that ask the right question

Not “is the port open”. Ask the node for its head and compare it to the other nodes’ heads, and take out any node more than a small number of blocks behind the best. This is the same comparison this platform makes when it judges an answer stale — against a cohort reference rather than an absolute, because there is no absolute available from inside one node.

Copy

# The check worth running. A TCP probe would pass on all of these.
curl -s http://127.0.0.1:8545 -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"eth_blockNumber"}' \
  | jq -r '.result' | xargs printf '%d\n'

Stickiness where state is node-local

Filters are the trap. A filter created with eth_newFilter exists on one node; polling it through a round-robin balancer hits a different node, which has never heard of it, and returns an error that looks like the filter expired. Either pin a filter’s whole lifecycle to one node or do not offer filters through the balancer at all.

Retries that know what is safe

Reads are idempotent and safe to retry elsewhere. A transaction submission is not — a retry after an ambiguous timeout can broadcast it twice. Retry reads freely, retry writes never, and let the caller decide by transaction hash whether it landed.

Once there is more than one node, they do not have to be the same node. Routing by method is the single highest-leverage thing in this guide, because it turns one expensive homogeneous fleet into a small expensive pool and a large cheap one.

• Head reads — balances, calls at latest, block numbers, receipts for recent transactions. The overwhelming majority of traffic. Any full node, any of them, cheapest hardware.
• Historical state — a call or a balance at an old block. Needs archive. Route these to the two expensive nodes rather than requiring every node to be one.
• Log ranges and traces — the methods that starve the others. Give them their own pool so that when somebody asks for a million-block range, the damage is contained to the people asking for million-block ranges.
• Transaction submission — wants the best-connected node, and often wants to go to more than one place at once.

The method mix is measurable and usually surprising. Before building any of this, log which methods your application actually calls and in what proportion; the answer is normally that three methods are ninety percent of the volume and the expensive ones are a rounding error by count and a majority of the cost. The method matrix here classifies each method by how expensive it is to serve, which is a reasonable starting weight if you have no measurements of your own yet.

Caching RPC is unusually rewarding and unusually easy to get wrong, because the correctness rule is not about time. It is about whether the answer can still change.

Cacheable forever, once finalised

A block by hash, a transaction by hash, a receipt for a finalised transaction, the code at an address at a specific historical block, the chain ID. These are immutable facts. Cache them indefinitely, keyed by their identifier — not by the request, which may name the same block in three different ways.

Cacheable for a moment

Anything at latest: balances, calls, gas price, block number. A one-second cache in front of a chain with a twelve-second block time removes most duplicate load and costs almost nothing in freshness. On a chain producing blocks several times a second, the same second is most of a block.

Do not cache

Nonces used for building transactions, anything at pending, and subscription streams. A stale nonce produces a transaction that will not land, and the failure appears far away from the cache that caused it.

Failing over between your own nodes is easy: they run the same software with the same configuration and differ only in which machine they are on. Failing over between two providers is not, and the differences are the ones that produce the strange bugs.

• They are not at the same height. Two healthy providers legitimately differ by a block or two. Switch mid-sequence and your application can read a lower block number than it just read — time appears to go backwards. Keep the highest block number you have seen and refuse to act on anything lower.
• They do not support the same methods. Tracing and other non-standard methods vary by client and by tier. A fallback that cannot answer the call you failed over for is not a fallback; check what each one answers before nominating one.
• They do not have the same history. Archive depth differs sharply between endpoints. A historical query that works against one provider can fail against another that is otherwise perfectly healthy.
• They fail at different times, which is the point. Independent failure is the entire value of the second provider, and it only exists if the second one is genuinely independent — a different company, not a second URL from the same one.

Subscriptions look like a cheaper way to stay current than polling, and they are, right up until you have several thousand of them. The scaling model is different from request traffic and needs its own thinking.

• Fan out from one upstream subscription. One connection to the node, many connections to your users, and your process in the middle. Ten thousand users should not be ten thousand subscriptions on your node, and if you are paying a provider per connection they certainly should not be.
• Assume the stream drops and plan the gap. A reconnect that resumes from the current head silently loses everything that happened while it was disconnected. Record the last height you processed and backfill from it by ordinary request before resuming live delivery.
• Do not trust arrival order across providers. Two providers announcing the same block will do so milliseconds apart — measurably, and it is measured here — so a consumer merging two streams must deduplicate by height and hash rather than by arrival.