Skip to main content

Routing and manifests

The hard problem in a network without a coordinator is not executing a query. It is deciding who to ask. Gnarl solves it with manifests.

What a manifest is​

A manifest is a small, signed document that describes what a node can answer — never what it holds. It is the only thing that crosses the network before a query does.

{
  "index": "imagery",
  "node": "4f2c9a7e1b83d05c6ea41f9027bd3e58c1740a9f6b2d85e3c09a7f41d62be8b7",
  "version": 7,
  "doc_count": 48219,
  "fields": ["title", "captured_at", "sensor"],
  "summary": {
    "time_range": ["2019-03-01T00:00:00Z", "2026-07-28T00:00:00Z"],
    "bbox": [-124.7, 24.5, -66.9, 49.4],
    "sensor": ["landsat-8", "sentinel-2", "planetscope"],
    "terms": "bloom:8kb:base64…"
  },
  "signature": "ed25519:9c41…2f7e"
}

Typical size is a few kilobytes regardless of corpus size — a 50-million-document node advertises no more than a 50-thousand-document one.

Elimination, not lookup​

Planning is a process of provable elimination. The planner never asks "who has this document?" It asks "who can I rule out?"

Predicate in queryManifest field consultedEliminates
captured_at > 2024summary.time_rangeNodes whose newest document predates 2024
within(bbox)summary.bboxNodes with no overlapping footprint
sensor = "landsat-8"summary.sensorNodes that have never seen that sensor
match("glacier")summary.terms (Bloom filter)Nodes whose vocabulary lacks the term
anypolicyNodes that would refuse you anyway

Whatever survives elimination gets asked. False positives cost one wasted request; false negatives are impossible, because every summary is conservative by construction — a Bloom filter may say "maybe" but never a wrongful "no".

Fan-out control​

Asking everyone is correct but not always affordable. Queries carry a budget.

The budget is not something a caller sets today. The planner ranks surviving candidates by expected contribution — manifest match strength, recent latency, and recent success rate — asks the best ones, and reports what it got:

"coverage": {
  "expected_claims": 4,
  "served_claims": 3,
  "skipped_claims": ["…"]
},
"partial": true

That is the contract: a partial answer is always visibly partial. When partial is not acceptable, say so and let the query fail instead — gnarl search "…" --require-complete, which exits non-zero rather than handing back less than you asked for.

Manifest propagation​

Manifests gossip. A node pushes a new manifest version to a random subset of peers, which forward it onward until the network converges — typically within a few seconds on networks of hundreds of nodes.

Versions are monotonic and signed, so a stale manifest can never overwrite a fresh one, and a peer cannot forge a manifest on someone else's behalf.

Tuning​

There is no config file, and most of routing is not a dial you turn. The one exposed knob is how often a node reconciles with its peers:

gnarl start --check-in-interval-secs 5     # 60 on an --edge peer

Shorter means faster convergence and more chatter; longer is why a phone on battery is not the same cost as a rack.

See also​