The model was never the hard part

I restarted this company in December 2025, which happened to be the middle of a stretch where the major AI labs were shipping something significant most weeks. New models, new capabilities, a genuinely different set of things being possible every month.

Most accounts of that period treat it as a problem to be managed. Everything you built last quarter is now the slow way of doing it; the ground moves under you faster than you can lay anything on it.

I did not experience it that way, and I do not think that was temperament. I had somewhere to put it.

A topic makes new capability legible

When you have a concrete problem you understand deeply, every new release arrives as a specific question: does this help with the thing I am stuck on? You read the announcement and within an hour you know whether it changes anything for you. Most of the time it does not, and you carry on. Occasionally it does, and you know exactly where it goes.

Without a problem like that, you end up reading everything, retaining little and feeling perpetually behind, because you have no test that separates what matters from what is merely impressive. So the pace was useful to me because I had a narrow enough problem to test the news against.

The problem, stated plainly

What this company does rests on reading an enormous number of documents that nobody wants to read.

Capital improvement plans. Board minutes. Budgets. Rate studies. Engineering reports. They are public, they are scattered across tens of thousands of municipal websites, and they are mostly PDFs: some of them scanned, many formatted for a printer rather than a parser, almost none of them structured in any way that a machine finds convenient. That is where the whole difficulty sits, and choosing a model is a small part of it.

Where it started

I solved the first version of the reading problem with NotebookLM, a consumer tool, used the way anyone would use it, because the question at that point was whether the analysis was possible at all rather than whether it could be automated.

It was possible. So the question became how to do it at the scale of a sales territory rather than a document at a time, and that is a different exercise entirely.

From there it was OCR techniques, open-source models, and a lot of testing about efficiency and speed. That work is not glamorous and does not demo well, and it is where most of the engineering has gone.

Two problems, not one

It took me a while to see that this splits cleanly in half, and that the halves need different things.

Getting the raw material. Finding the documents, pulling them in, and turning whatever format they arrive in into something machine-readable at acceptable cost and speed. Parsing PDFs efficiently is a genuine research problem, and how well you do it sets the ceiling for everything downstream.

Extracting the signal. Having the text is not having the answer. The question is always something like what is this utility likely to procure in the next two to three years, and why. Answering it means finding the handful of passages across thousands of pages that bear on it, then reasoning over them.

Getting the documents in is plumbing, and finding the signal takes judgment. Treating the two as one problem is why a lot of document-AI demos work beautifully once and then fall over on the second utility.

It is not a vector database

The obvious architecture for this is: embed everything, put it in a vector store, retrieve by similarity. That is where I started and it is not sufficient.

What has worked is a hybrid: a vector database for semantic retrieval, alongside ordinary relational tables for the things that are simply facts and should be queried as facts, and graph traversal for the relationships between entities, which is often where the answer lives. Which utility, which contractor, which project, which document, which decision, and how they connect.

Semantic search alone is very good at finding text that sounds relevant. It is noticeably worse at answering a question whose answer depends on a chain of associations rather than a passage.

The thing I stopped building

Early on, we were building our own model to do this.

It is the assumption I would have defended hardest at the time. If the difficult part of your product is understanding a specialized corpus, it feels obvious that the differentiated thing to own is a model that understands it.

What ended that was not failure. It was the rate of improvement everywhere else. The general-purpose models were getting better faster than we could improve ours, and every month the gap moved in the wrong direction. That said less about our work than about who we were up against: AI labs spending more on a single training run than this company will raise in its life.

So we stopped, and became deliberately agnostic: treat the large language model as a component, call whichever one is currently best, and put our effort into the parts that are ours.

That is where the title of this piece comes from. I arrived at it by spending months building a model and watching the industry make it beside the point.

The decision I have not made yet

More recently the motion has partly reversed, which I find interesting enough to write down while it is still unresolved.

We are looking at open-source models for some core components (the chat agent is the live example), and the driver there is cost. Frontier model calls are a running expense that scales with usage, and at some volume the arithmetic of hosting your own stops being theoretical.

The hosting question that comes with it has a different driver: data residency. Running on a US-based server is about where the data physically sits, which in this sector is a real constraint rather than a preference.

I am keeping those as two separate decisions, because they can be answered independently. You can host a frontier provider's model in a way that satisfies residency, and you can run an open model somewhere that does not. Bundling them is how people end up rebuilding more than they needed to.

I have not settled either yet.

What I would note is that the earlier decision is what makes this one available. Because we stayed agnostic rather than coupling ourselves to one provider, swapping a component is an engineering question rather than a rebuild. We did not stay agnostic for the optionality, but that has turned out to be the main benefit.

How I know it is working

For now, I read it.

We are building an eval framework and do not have one yet. What happens today is that when something changes in retrieval or extraction, I generate a profile and read the output, and I form a view about whether it got better.

I am aware of what is wrong with that. It does not scale. It cannot catch a regression in the cases I did not happen to look at. And the judgment deciding whether the output improved is the same judgment that designed the thing being tested, which is not a controlled experiment by any definition.

But I would not dismiss it either, because at this stage it is doing work an automated eval would not. I know this sector. When a profile asserts an opportunity that is not really there, or misses one that is, I can see it. That is exactly the signal that is hardest to encode. A retrieval metric tells you whether you fetched the passages you expected. It does not tell you whether the resulting claim about a utility is true.

I suspect this is the right order to do it in. You cannot write a good evaluation for a judgment you are not yet able to make yourself, and the manual reading is how I am working out what the framework should measure.

Eight or nine months of iterations

That stack has been rebuilt several times, and the reason is the thing I opened with: the tooling underneath kept improving. Better parsing, better models, better retrieval patterns. Each was worth the rework, and each arrived because the field moved, not because I had a new idea. It is a strange way to build a company and I do not entirely recommend it, but that is what the last eight or nine months looked like.

Almost none of that work has been about model choice. The model is the most discussed and least differentiating part of what I do, because everyone has access to the same ones. The work that decides whether an answer is useful or only plausible is getting the documents in cleanly and finding the four paragraphs that matter. That is where most of the engineering at AquaIntel goes.