Why We Run Our Own Models

For about a year, the most important product decisions at VeteranAI weren't made by me. They were made by a per-token price list.

Not in any dramatic way. It shows up as small, reasonable-sounding calls. Cap chat usage at some number of messages. Truncate the context window on long documents. Route the cheap model at the hard question and the expensive one only where you can justify it. Every one of those has a plausible product rationale attached to it after the fact. Every one of them was actually a margin decision.

That's the thing nobody writes about when they write about AI costs. The bill doesn't just cost you money. It quietly writes your roadmap.

The structural problem

We sell a subscription. Pro is $25 a month. That number is fixed the moment a veteran signs up.

Metered inference is not fixed. It scales with exactly the behavior we most want — a veteran who engages deeply, uploads a 1,500-page C-File, asks follow-up questions, comes back six weeks later when the decision letter arrives. Our best users were our most expensive users. Our worst users were free money.

Any company with that shape ends up rationing. You can be principled about it and ration late, or you can be sloppy and ration early, but you ration. And the rationing lands hardest on the veteran with the complicated file, which is to say the veteran who needed the most help.

I won't publish our numbers, and the size of the bill was never the interesting part. The direction was. It grew with engagement, and engagement had just doubled.

What we actually did

We deployed an open-weight model on dedicated hardware we control, running around the clock. Six, our assistant, now serves off that endpoint.

The bill is the same at a thousand messages a day and at ten million. Marginal cost per token is zero.

I want to be precise about what that did and didn't do, because there's a version of this post that overclaims and I'd rather not write it.

Note what isn't in that table: a smaller number. Self-hosting did not make our inference cheaper, and anyone here for that headline should stop reading. It changed the slope. Cost stopped being a function of usage and became a function of capacity, and those are completely different things to manage. One of them caps your product. The other one is an engineering problem with a purchase order attached.

The practical consequence: usage caps are now a throughput question, not a COGS question. We can raise them when the hardware allows it rather than when the spreadsheet allows it. We can put a free tier back in front of veterans who aren't ready to pay — and for a population where a meaningful share of people are deciding between a subscription and something else that month, that matters more than it sounds like it does.

The privacy argument, including the part you're about to ask about

What moves through this system: service treatment records. C-Files. Mental health narratives. Descriptions of the worst thing that ever happened to somebody, typed out at 2 a.m. because that's when it's easiest to type.

On a metered API, that data leaves your infrastructure. There are contracts and there are commitments and I'm not suggesting those are worthless. But there's a difference between a promise about data handling and an architecture where the question doesn't arise. Weights on hardware we control, in a pipeline we control, is the second kind.

People will ask which model. We don't publish that, for the same reason we don't publish our infrastructure diagram. There's a version of that answer that sounds evasive, so here's the real one.

With open weights, the question changes shape. A closed API bundles two things you have to trust a vendor on simultaneously: how the model was built, and where your data goes. Open weights separate them. The weights are a file. We run that file on machines inside our own network boundary. There is no phone home, because there is nobody to phone.

So we can be as skeptical as we want about any given model's provenance — and we are; we select on tool-calling reliability and hallucination rate, and we've dropped candidates over both — without that skepticism having anything to do with whether a veteran's records left our infrastructure. They didn't. That isn't a policy commitment we're asking you to take on faith. It's an architecture in which the question doesn't arise.

Those two things get collapsed constantly. They are not the same question, and only one of them is the kind you can solve by reading a contract.

Why any of this matters for claims work

A VA disability claim is not a transaction. It runs two to five years. A veteran files, waits, gets a decision, files a supplemental, develops a secondary condition, requests a higher-level review. The file grows the entire time.

Software that's useful across that arc has to hold an enormous amount of context about one person and keep re-reading it. Every re-read is tokens. At metered rates, the cost of being genuinely useful to one veteran over three years is a number that doesn't work — so you build something that's useful for one session instead, and call it a product.

Zero marginal cost is what makes the long version arithmetic instead of a pitch deck. It doesn't build it for you. But it removes the reason it can't exist.

And the constraint that stays fixed regardless: the veteran decides what gets filed. We don't submit anything. We're not accredited, we don't represent anyone before the VA, and nothing about cheaper inference changes that — a veteran reviews what's been drafted and a veteran files it. Lowering our costs buys more analysis, not more authority.

About the price you're paying now

Worth saying plainly, for anyone building on metered inference: the rate card is not a price. It's a promotional rate.

The frontier labs are, by their own reported numbers, losing money serving inference. OpenAI is projected to burn roughly $14B this year. Sector gross margins have come in under plan because inference costs ran higher than forecast — Anthropic's 2025 margins landed around 40%, about ten points below their own projection. Consumer and low-tier API pricing is the most heavily subsidized slice of it. Meanwhile average cost per million tokens across major providers fell from roughly $10 to $2.50 in a single year, which looks like deflation and is at least partly a customer-acquisition budget.

If your unit economics work at today's rates, you should know whether they work at 3× today's rates, because you don't set that number and neither does anyone who sold you on it.

Which brings up the part of the "we should slow down at the frontier" conversation that doesn't make it onto the slide. I don't think those arguments are made in bad faith — some of the people making them are serious, and the risks they're describing aren't imaginary. But they're being argued by parties with a balance sheet in the fight, and open weights are the one thing that structurally caps what any of them can eventually charge. Both of those can be true simultaneously. Only one of them tends to get said out loud, and the quiet one is the one that determines whether a company like ours can exist.

What it costs

Self-hosting is not free in the way people mean when they say "free."

You inherit an ops burden. The endpoint has states, and cold starts are real, so we run a provider router that falls back to a commercial API the moment the self-hosted endpoint is anything other than running — no user waits on a boot. You give up frontier-model quality on the genuinely hard reasoning, which is why some workloads still run on commercial models and probably always will. You take on a capacity question instead of a budget question, and we do not yet have a settled answer on how far our current configuration scales. That's an open engineering problem, not a solved one.

It's a mixed fleet and it will stay a mixed fleet. The point was never to purge commercial models. The point was to stop letting a rate card decide how much help a veteran gets.