Building stellars

A research step for your coding agent. Four million repos and papers, hybrid search, and no LLM anywhere on the server.

Filip Nowak · 11 min read

Everyone is building loops now. Your agent plans, writes code, runs it, reads the error, tries again.

Running the loop is the easy part. Making sure the loop goes in the right direction is not. A loop that starts from the wrong approach will happily run all night and hand you a working implementation of the wrong thing.

So when your agent hits "how have others solved this?" - what does it actually do?

It has two moves, and both are bad. It can answer from the model's memory, which is months stale and confidently wrong exactly where your problem is unusual. Or it can search the web, which gives you blog posts about the problem instead of the code that solved it. GitHub search only helps if you already guessed the word the maintainer picked.

So I built the missing step. stellars is a search engine over open source code and research papers that your agent calls before it writes anything. It's a home cooked tool I've been using on my own engineering problems for months, it's now a North Star member benefit, and this is how it works under the hood.

The whole thing rests on one bet: if you give a model enough genuinely relevant context, it'll average out on a breakthrough solution instead of the median of everything it ever read.

What you actually get back

One endpoint, one job. You send a problem, you get back ranked sources with a similarity score and a confidence label.

curl -X POST https://stellars.dev/api/v1/research \
  -H "Authorization: Bearer $KEY" \
  -d '{"problem": "distributed rate limiting", "k": 20}'

{
  "confidence": "strong",   // strong | thin | none
  "sources": [
    {
      "id": "repo:owner/name",
      "type": "repository",
      "similarity": 0.61,
      "stars": 1234,
      "url": "…",
      "snippet": "…"
    }
  ]
}

Notice what's missing? There's no prose and no recommendation anywhere in there. You get evidence and a number, and your own agent turns that into an approach. I'll come back to why that split matters more than anything else here.

Where four million things come from

Right now the index holds 3,992,554 items - 769,660 repos and 3,179,806 papers.

The repos come from anvaka/map-of-github, which is a lovely MIT-licensed dataset of ~690,000 notable repos filtered by real stargazer signal. It also ships a co-star graph - basically "teams who starred X also starred Y". That's a signal keyword search simply cannot produce, and I got it for free.

The papers were harder. I started with OpenAlex because it's CC0, which makes it the only paper corpus you can build a product on without a lawyer in the room. But OpenAlex snapshots are citation-weighted, and citations lag by years. So the corpus was great on 2019 and nearly blind on the AI and robotics work from the last eighteen months. A vision-language-action query returned nothing from 2025. Not ideal for a tool whose whole job is telling you what's proven!

The fix was a second pipeline that ignores citations completely: a direct arXiv harvest over OAI-PMH, 719,878 papers across sixteen categories going back to 2007, plus a daily feed for new preprints at 06:30 UTC. A paper with zero citations from last Tuesday is exactly what you want when you're picking an approach, and it's precisely what a citation-weighted snapshot throws in the bin.

Everything gets embedded with BAAI/bge-m3 at 1024 dimensions on a Modal L4 (about a dollar an hour), then stored as a LanceDB dataset and served off a single ARM box at Hetzner. Bulk embedding on CPU is hopeless at this size. Serving on CPU is completely fine.

One thing I deliberately don't do: stellars indexes no source code. READMEs and in-repo /docs only. The engine finds the three repos that matter and your agent reads their actual code with its own tools. I'm not going to vouch for code quality I never read.

Cosine similarity on its own does not work

This is the failure that took me longest to accept.

Next.js describes itself as "The React Framework". That's it. That's the whole text - five words. So when you search "react framework", pure cosine similarity put it around 83rd, underneath a long tail of tutorial repos whose READMEs happened to be shaped like a question. Ollama sat around 380th for "run LLMs locally".

Think about why. Semantic search rewards documents shaped like questions, and canonical projects are terse because they don't need to explain themselves. Next.js doesn't have to sell you anything.

So the path has four stages now:

1. Two searches at once. A dense leg over the vector index (IVF_PQ, 24 probes, refine factor 12, pool of 600) and a lexical BM25 leg over a full text index. LanceDB's Rust core releases the GIL, so both run in threads and I pay for the slower one instead of both.
2. Reciprocal rank fusion to merge them. The lexical leg catches exact names the vectors under-rank - it's how you find ghostty when you searched "terminal emulator".
3. A small popularity nudge, 0.04 · log10(popularity), and only after candidates clear a similarity gate. It reorders near-ties and nothing else. This is what drags Next.js and Ollama back where they belong.
4. A cross-encoder rerank over an over-retrieved pool - six times the requested k, capped at 60 pairs so latency stays sane.

That last one needed a correction I didn't see coming. A cross-encoder reads the query and the document together instead of scoring them separately, so in theory it judges relevance much better. In practice the small MS-MARCO model is obsessed with literal keyword overlap, and when I let it decide alone it buried the canonical repos under query-shaped toys all over again.

So it doesn't decide alone. The final score keeps 0.85 on the fusion order and gives the cross-encoder 0.15. It refines the ranking instead of replacing it. Not a glamorous conclusion, and I'd never have got there without measuring it.

Two smaller things I like. Every repo carries two vector rows - one for its crisp description, one for its README prose - so a query can match either. The twins collapse into one canonical id before the pool gets cut, so a single project never eats two slots. And results are interleaved by source type, because a mixed top-k lets three million papers bulldoze every repo.

Confidence is just maths

This is the part I care about most, and it took two shipped bugs to get right.

Your agent can't do anything with a bare list of results. It needs to know whether the list means anything. And here's the thing - the dangerous failure isn't returning nothing. It's returning four plausible-looking sources for a problem that has no precedent, which your agent then treats as proven and builds on for the next three hours.

So confidence is computed, with no model involved anywhere:

• Anything under 0.45 cosine similarity gets dropped as noise.
• strong needs at least three sources at 0.55 or above, plus a coherence check.
• thin means the best source clears 0.50.
• Otherwise it's none.

That coherence check is the interesting bit, and it exists because similarity thresholds alone shipped a false strong twice. A vague query pulled a set where every source scored ~0.55 against the query and none of them had anything to do with each other - container tooling, quantum computing, molecular dynamics, all in one answer. Every single hit was individually defensible. The answer was garbage.

So coherence is the mean pairwise cosine of the top eight source vectors. It asks whether the sources agree with each other, not just with the query. Real answers land at 0.60-0.66. Scattered grab-bags come in under 0.55, so strong now demands 0.58. A pile of unrelated hits can't be strong no matter how nicely each one matched.

And none is a real answer, not a failure. "I found no strong precedent" is useful. A confident guess poisons everything downstream of it.

Why there's no LLM on the server

There used to be one. It called Claude to write a playbook server-side. I deleted it.

Three reasons, in the order they actually convinced me.

An LLM sitting in the middle of a retrieval pipeline is exactly where hallucination gets in, and it gets in wearing your product's authority. That synthesis step was the piece I could verify least and the piece users would trust most. Bad combination!

Second - you already have a model. It's probably better than whatever I'd pay to run, and it's already holding the full context of the codebase you're asking about. Synthesising on my server means throwing all of that away and charging you for the privilege.

Third, and this is the one that sold me: taking it out made the output auditable. Same query, same sources, same confidence number, every time. I can put that behind a nightly eval and know the moment it regresses. You can't regression test a paragraph.

What moved to the client isn't a thin wrapper, though. The skill tells your agent to expand a problem into 2-4 differently shaped queries before searching at all - the problem framing (what outcome you want), the mechanism framing (the technique you suspect), the artifact framing (the concrete thing to go find), and optionally a constraint-tight variant. Then it fuses the results itself and writes the approach, citing a source id for every claim it makes.

Query planning turned out to be the single biggest quality lever in the whole system, and it belongs where the reasoning already lives.

The nine tests that gate a release

Nine labeled cases, all from problems I actually had, run against the pipeline every night at 05:00 UTC.

Five are genuine - VR teleoperation of a robot arm with inverse kinematics, real-time control for a cheap air hockey arm, edge inference for drone detection on a Jetson. Those must never come back none.

Four are traps that must never come back strong. One has no precedent because nothing like it exists: "purple elephant quantum bakery scheduler for left-handed dentists". One is too specific to be real. One is deliberate scatter bait, vague enough to pull unrelated junk: "a good general purpose open source tool for developers". They're in there because returning strong over an incoherent grab-bag is the bug that shipped twice, and this gate is how it doesn't ship a third time.

For a tool like this, precision matters far more than recall. A search engine that pads its results is worse than one that shuts up, because your agent has no way to tell the difference.

The map

The corpus is the product, so I made it visible. The map is a UMAP projection of 1,529,812 items above 50 stars or 30 citations, sorted into 391 regions across five zoom tiers. The top two tiers have hand-written names, because automatic labels are only ever good enough to be embarrassing.

Honestly, building it paid for itself in debugging alone. My first pass produced region names full of align, center, img and githubusercontent - which is what you get when you run TF-IDF over READMEs and forget that badge HTML is also text. That same garbage was sitting in the search index. The map is just where I could finally see it.

Most of the engineering went into first load. The label payload started at 163 MB, which is not a website. Stripping the descriptions out and fetching them lazily on hover got it to 61 MB gzipped.

What I got wrong

Before launching I ran a twenty-agent review across my own repo. It found things I'd rather it hadn't!

The synthesis "moat" was one 85-line prompt with no enforcement around it whatsoever. My README proudly claimed "uncited claims are flagged" - and nothing, anywhere in the code, checked whether a cited id existed in the retrieved set. LanceDB was handing me similarity scores and I was throwing them away, which is why confidence was a model's opinion instead of a measurement.

The eval was the worst of it. The seed queries I used to build the test index leaked 1:1 into the test set, so I had built an index to answer exactly the questions I was grading it on. Every positive result was inflated. Lovely.

Most of this post is just the response to that review: delete the server model, compute confidence from distances that were already sitting there, write negative cases, gate on them nightly.

Two limits I haven't solved. Map of GitHub is a roughly biennial manual snapshot rather than a live feed, so repo freshness depends on my own re-crawls. And the co-star graph is relatedness, not relevance - it leans toward popular projects, so it sits behind a quality filter as one ranking input among several. It's not the moat I once told people it was.

What I stole

anvaka/map-of-github gave me both the repo universe and the whole visual grammar of the map, and it's why the co-star signal exists at all. OpenAlex is the only paper corpus with a licence that lets you build commercially without hedging. And the idea of shipping one tool that does real work, instead of a suite of thin API wrappers, came from watching agent tools fail in the other direction over and over.

My favourite one though: when I hit the abstention problem, I used stellars to research it. It surfaced the HyDE, ANN-cutoff and selective-abstention papers that the coherence gate is now built on. The tool found the papers that fixed the tool : )

What's next

There are two things on the list.

Autoresearch - the same retriever in a loop instead of a single call, for when a question deserves an hour rather than one query.

Private corpus fusion - and this is the one I actually want. Blend in your own solved problems and internal notes alongside the public evidence, so "how do we do this" includes how you already did it. Public prior art is a commodity and it's only getting cheaper. Your own history isn't.

Hugging Face models and Spaces are already in the index as source types. MTEB scores as ranking metadata and datasets are queued behind them.

Go try it

The map is free and open, and you can fire queries at a subset straight from the search bar. The skill is a copy-paste definition for Claude Code. API keys are a monthly subscription over at stellars.dev.

If you're a North Star member it's free - 2,000 calls per 30-day cycle, claimable from the benefits tab at atlas.ns2agi.com. If you're not, that's the cheaper way in, and honestly the community is worth more than the tool.

Then tell me where it breaks. A none on a problem that obviously has precedent is the bug I most want to hear about - and it's the one I can't find on my own.

References

1. anvaka/map-of-github - repo universe and co-star graph (MIT).
2. OpenAlex - open, CC0 catalogue of scholarly works.
3. arXiv OAI-PMH interface - bulk metadata harvesting.
4. BGE-M3 - the embedding model, 1024 dimensions.
5. LanceDB - embedded vector store on columnar Lance files.
6. Cormack, G. et al. (2009). Reciprocal Rank Fusion.
7. Gao, L. et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE).
8. Modal - GPU pricing for the embedding runs.