Skip to main content

Routing and manifests

The hard problem in a network without a coordinator is not executing a query. It is deciding who to ask. Gnarl solves it with manifests.

What a manifest is​

A manifest is a small, signed document that describes what a node can answer — never what it holds. It is the only thing that crosses the network before a query does.

{
  "index": "imagery",
  "node": "n1qh7f2m9x8p4vd6kkzq3r7wjn5tbc2a8",
  "version": 7,
  "doc_count": 48219,
  "fields": ["title", "captured_at", "sensor"],
  "summary": {
    "time_range": ["2019-03-01T00:00:00Z", "2026-07-28T00:00:00Z"],
    "bbox": [-124.7, 24.5, -66.9, 49.4],
    "sensor": ["landsat-8", "sentinel-2", "planetscope"],
    "terms": "bloom:8kb:base64…"
  },
  "signature": "ed25519:9c41…2f7e"
}

Typical size is a few kilobytes regardless of corpus size — a 50-million-document node advertises no more than a 50-thousand-document one.

Elimination, not lookup​

Planning is a process of provable elimination. The planner never asks "who has this document?" It asks "who can I rule out?"

Predicate in queryManifest field consultedEliminates
captured_at > 2024summary.time_rangeNodes whose newest document predates 2024
within(bbox)summary.bboxNodes with no overlapping footprint
sensor = "landsat-8"summary.sensorNodes that have never seen that sensor
match("glacier")summary.terms (Bloom filter)Nodes whose vocabulary lacks the term
anypolicyNodes that would refuse you anyway

Whatever survives elimination gets asked. False positives cost one wasted request; false negatives are impossible, because every summary is conservative by construction — a Bloom filter may say "maybe" but never a wrongful "no".

Fan-out control​

Asking everyone is correct but not always affordable. Queries carry a budget.

{
  "scope": {
    "max_nodes": 64,
    "deadline_ms": 750,
    "min_coverage": 0.9
  }
}

When more nodes survive elimination than max_nodes allows, the planner ranks candidates by expected contribution — manifest match strength, recent latency, and recent success rate — and asks the best ones first. The response always reports what fraction of eligible nodes actually answered.

Manifest propagation​

Manifests gossip. A node pushes a new manifest version to a random subset of peers, which forward it onward until the network converges — typically within a few seconds on networks of hundreds of nodes.

Versions are monotonic and signed, so a stale manifest can never overwrite a fresh one, and a peer cannot forge a manifest on someone else's behalf.

Tuning​

gnarl.yaml
routing:
  gossip_interval: 5s
  manifest_ttl: 15m       # re-ask a peer whose manifest went stale
  candidate_cap: 128      # hard ceiling on fan-out per query
  latency_window: 5m      # how much history feeds candidate ranking
  bloom_bits_per_doc: 10  # larger = fewer false positives, bigger manifest

See also​