Introducing Gnarl — search without the cluster
Every distributed search system built in the last twenty years starts from the same assumption: before you can search data, you must move it into a cluster you control.
That assumption used to be free. It is not free anymore.
The bill for centralization
Three things changed at once.
Data got heavier than the compute. A petabyte imagery archive costs more to move than to search. Ingestion is no longer a setup step you do once — it is a permanent tax you pay forever, in egress, in duplicate storage, and in the lag between when data lands and when it becomes searchable.
Boundaries became legal, not architectural. Residency rules, partner agreements, and classification levels are not problems you solve with a faster pipeline. Some data will never be allowed into your cluster, no matter how good the cluster is.
The blast radius stopped being acceptable. One coordinator, one shard rebalance at the wrong moment, one config push — and the search surface for the whole organization goes dark together.
What we are building instead
Gnarl replaces the cluster with a network. Every place your data already lives becomes a node: it indexes locally, publishes a signed summary of what it can answer, and answers queries against its own data.
gnarl node up --data ./corpus
gnarl join --network acme-internal
gnarl search "glacier retreat"
There is no coordinator in that network. The node a query happens to arrive at plans the fan-out for that query, and stops being special the moment it returns results.
The manifest is the whole trick
The hard part of a network without a coordinator is not executing a query. It is deciding who to ask without a central index of who has what.
Every node publishes a manifest — a few kilobytes describing what it can answer. Time ranges. Bounding boxes. Field names. A Bloom filter over its vocabulary. Never the documents.
Planning is then a process of elimination rather than lookup. The planner does not ask "who has this document?" It asks "who can I prove cannot help?" and asks everyone who is left. Summaries are conservative by construction, so a false positive costs one wasted request and a false negative cannot happen.
Partial answers, honestly reported
In a network, some nodes will be slow, some will be offline, and some will refuse you. That is not an outage — it is one fewer answer. Every response says so:
{
"coverage": {"eligible": 41, "answered": 39, "timed_out": 2, "complete": false}
}
We think this is strictly more honest than a cluster that quietly returns results from the shards that happened to be up. You can require completeness, retry the stragglers, or degrade on purpose. What you cannot do is mistake a partial answer for a whole one.
Where this is going
Gnarl is early. The wire protocol and manifest format are documented and stable, and the client SDKs you build against are open source at github.com/lucenia/gnarl-client. The node itself is a commercial product from Lucenia, with a free tier for running a single peer.
If your data is somewhere it cannot leave, we would like to hear about it.