or “How to serve 1 Billion Tokens per day in a 45 kW rack”

Designing and Planning for Enterprise Sovereign AI running On-Prem with an MSP — Image from nvidia.com

Designing and Planning for Enterprise Sovereign AI running On-Prem with an MSP — Image from nvidia.com

Sep 18, 2026 — Madkat Omega

The pitch

We are proposing to buy two racks and put frontier-class artificial intelligence inside our own building, under our own control, for roughly the price we would otherwise pay a cloud provider for the same volume of tokens — and for a class of work we cannot legally or commercially send to a cloud provider at all.

The unit is a 60 kW pod: one 45 kW rack of inference hardware and one 15 kW rack of supporting services. It fits on an existing datacenter floor. It does not require a new building, a substation, or a greenfield row of 120 kW liquid-cooled cabinets. It runs the best open-weight models available today — models that score within a few points of the strongest commercial systems on the same public benchmarks — and it runs them on hardware we own, against data that never leaves the premises.

One pod delivers about 1 billion tokens a day. That is the scale at which a large enterprise stops experimenting and starts depending: every knowledge worker with a grounded assistant that can actually read the company’s documents, agents that run multi-step work overnight, engineers querying schematics and specifications by conversation instead of by search. At that volume the economics invert. Below roughly 730 million tokens a day, renting is cheaper. Above it, owning is cheaper — and the gap widens every day the system is used harder.

Three things make this the right moment rather than a speculative bet.

The open models got good enough. In September 2026 the strongest downloadable models sit roughly three points behind the leading commercial frontier systems on the Artificial Analysis Intelligence Index. Eighteen months ago that gap was a chasm and self-hosting meant accepting a visibly weaker assistant. It no longer does.

The hardware got specialized. The industry worked out that generating text and reading text are different computational problems, and started building different machines for each. Splitting those two jobs across purpose-fit hardware — rather than buying one enormous general-purpose cluster and letting most of it idle — is what brings a credible enterprise AI platform down from 120 kW to 45 kW, and therefore onto floors we already have.

The compliance question stopped being theoretical. For a growing set of industries, “our data is contractually protected in a cloud tenancy” is no longer an answer that survives an audit. The requirement is architectural proof, not a contractual assurance. You cannot buy that from a provider. You have to build it.

What follows is the argument in full: why sovereignty matters and for whom, why open weights specifically, what the money actually looks like, a walk through every significant component in the build and what it does, the objections worth taking seriously, and a 90-day plan with the specific companies to call.

Why sovereignty stopped being a preference

Every major cloud AI service now offers zero-data-retention terms, customer-managed encryption keys, private network peering, and a stack of compliance certifications. For most companies, most of the time, that is genuinely sufficient. It would be dishonest to argue otherwise, and any vendor who tells you every enterprise needs an air gap is selling you something.

The distinction that matters is between a promise and a property.

A promise is a contractual commitment that a third party will behave in a particular way, backed by audit reports and indemnities. A property is a fact about the physical world: the data cannot leave because there is no path by which it could leave. Promises are enforced after the fact, through litigation and remediation. Properties are enforced continuously, by physics.

For a growing set of situations, the difference is the whole question.

Regulated industries where the auditor asks a different question. Defense and government contractors working under controlled-unclassified-information rules, classified programs, and export-controlled technical data are not asked whether their provider promises confidentiality. They are asked to demonstrate the architecture. “Our vendor holds a certification” is a different answer from “there is no route from this system to any external network, here is the diagram and here is the test result.” Only the second answer closes the finding.

Industries where the data is the company. In pharmaceutical research, the clinical trial dataset and the molecule pipeline are the entire enterprise value. In advanced manufacturing and robotics, the process parameters, the control algorithms and the mechanical designs are what competitors would most like to have. In finance, the trading model and the position data are the franchise. When the asset being fed into an AI system is the crown jewel, the calculus changes: the question is not whether the provider is trustworthy but whether you want the crown jewels to exist, in decrypted form, in a machine you do not own, even for the milliseconds of an inference call.

Cross-border and jurisdictional exposure. European operations under GDPR and the AI Act, healthcare data under national health-data rules, and any business operating where data-localization law has teeth face a recurring problem: cloud AI capacity is concentrated in a handful of regions, and the newest and best models often land in one jurisdiction first. A sovereign pod sidesteps the question entirely — the data is in the building it is legally required to be in.

Legal hold and discovery. A subpoena served on a cloud provider can compel production of data the provider holds, on a timeline and through a process you do not control. Data in your own building is subject to legal process served on you, where your counsel manages the response.

What a contract cannot buy

Even when a provider honors every commitment perfectly, three exposures remain that no contractual language addresses:

The prompt is the leak. Enterprises worry about training data and rarely think carefully about inference traffic. At the volumes we are discussing, roughly 800 million input tokens a day — two and a half to three and a half terabytes of internal documents, specifications, contracts, patient records or source code, every single day — crosses the network boundary and is decrypted inside somebody else’s memory. Nothing in a zero-retention agreement changes the fact that it was there.

The model can change underneath you. Hosted endpoints update checkpoints, adjust quantization, and revise safety behavior continuously, usually without announcement. A prompt that reliably produces valid structured output in March can begin failing in April. For a chat interface, that is an annoyance. For an autonomous agent executing a hundred sequential tool calls against production systems, it is a reliability incident with no root cause you can investigate, because you cannot see the thing that changed.

Access can be withdrawn. Models are withdrawn, deprecated, repriced, and occasionally restricted by policy shifts entirely outside commercial control — including export-control actions that have, within the past year, suspended access to frontier models for reasons unrelated to any customer’s conduct. A downloaded and pinned set of open weights cannot be recalled. It is on our storage array. That is a strategic property, not a technical detail: it means the capability we build our operations around cannot be taken away from us by someone else’s decision.

That is the case for sovereignty. It is not a claim that clouds are unsafe. It is a claim that for certain data, in certain industries, an architectural guarantee is the only kind that counts — and that for everyone, owning the model is the only way to own the roadmap.

Why open weights, and not just a private cloud

There is a middle option that looks attractive and mostly is not: rent dedicated capacity from a hyperscaler, get a private tenancy, and run a commercial frontier model on hardware nobody else touches. It solves the multi-tenancy worry. It solves nothing else.

In that arrangement the model is still a black box you are permitted to query. You cannot inspect it, you cannot freeze it, you cannot take it with you, and you cannot run it if the relationship ends. You have rented isolation, not sovereignty.

Open weights are different in kind. “Open weights” means the trained parameters of the model are published as files you download, and you may run them on your own hardware for commercial purposes. It does not usually mean the training data is public — most of these are open-weight rather than fully open-source, and the distinction is worth being precise about internally so nobody overclaims to an auditor. What matters operationally is that the artifact is yours: a file on our storage array, verified by cryptographic digest, that runs whether or not anyone else cooperates.

Four consequences follow, and each one maps to a different executive’s problem.

Immutability — the CTO’s problem. We choose exactly which version runs, and it stays that version until we decide otherwise. We pin the weights, the tokenizer, the chat template, the serving engine and the quantization recipe together as one matched set, and we validate that set against our own evaluation suite before it goes near production. When a new model arrives, we test it and choose. Nothing changes under an agent at three in the morning. For any team running long-horizon autonomous work, this is the difference between an engineering discipline and a hope.

Portability — the CEO’s problem. The capability is an asset on our balance sheet rather than a subscription. If a vendor relationship sours, a price changes, an export control lands, or a model is withdrawn, our operations continue unchanged. We are also not negotiating from a position where the counterparty knows that switching costs us our AI program.

Specialization — the CIO’s problem. Open weights can be fine-tuned on our own corpus, on our own hardware, without that corpus ever being transmitted to a third party for training. An assistant that has genuinely absorbed our internal terminology, part numbering, document conventions and engineering practice is worth considerably more than a general one, and building it is only possible when the weights are ours to adapt.

Inspectability — the principal engineer’s problem. When something goes wrong we can attach a debugger to it. We can measure which expert layers activate, quantify how quantization affected a specific behavior, reproduce a failure deterministically, and red-team the exact checkpoint that is serving traffic rather than a description of it. Anyone who has tried to root-cause an intermittent failure in a hosted endpoint understands what this is worth.

The honest tradeoff

Open weights are not free capability. They come three to six points behind the best commercial systems on aggregate benchmarks. Serving them is our problem: the batching, the memory management, the tail latency, the upgrade choreography. Licenses vary and need reading — some are Apache 2.0 with no strings, others carry conditions that bind large model-as-a-service providers but not an internal enterprise deployment. And the models we would run today are predominantly the output of a small number of labs, several of them outside the United States, which is a procurement conversation worth having deliberately rather than discovering later.

The design deliberately hedges that last point: the primary tier is the strongest open model available, and a fully permissive US-origin model of comparable architecture is maintained as a live, tested fallback that can carry the entire workload if procurement policy changes. That is not a hypothetical contingency. It is a configuration we validate at acceptance and re-validate every release.

The money

The financial case has one shape worth internalizing before any of the detail: cloud costs are flat per token, and owned infrastructure costs are fixed. Divide a fixed cost by a rising volume and the unit cost falls. That single fact determines everything.

At 1 billion tokens a day, one pod costs $2.92 per million tokens over three years, all in — hardware, colocation or floor space, power, networking, staff, and the sovereignty controls. The same volume against frontier-tier cloud APIs costs $4.11 per million. Over three years that is $3.20M against $4.50M: a $1.3M saving, alongside capability we would not otherwise have.

The crossover sits at roughly 730 million tokens a day. Below it, renting wins and building is an expensive way to make a point. Above it, owning wins and the advantage compounds. This is the number to defend in front of a board, because it is also the number that disqualifies the project if our real volume turns out to be lower — and the plan in the final section spends six weeks measuring it before anyone signs a purchase order.

The comparison a good CFO will make

A sharp finance partner will not compare us to frontier pricing. They will point out that most traffic does not need a frontier model, that a well-routed cloud deployment sending 80% of requests to cheap mid-tier models costs about $1.34 per million tokens, and that this beats the pod. They are right on the arithmetic, and the answer is that the arithmetic describes a price we cannot actually buy at this scale.

Rate limits are the catch. Pay-as-you-go cloud pricing assumes bursty consumption. Sustaining tens of thousands of tokens per second at a business-hours peak exceeds standard tier limits, and the provider’s remedy is reserved capacity — provisioned throughput units billed at a flat monthly rate whether consumed or not. Buying enough reserved capacity for our peak drives realistic cloud spend into the $2.5M–$3.5M range, which is the pod’s territory, without any of the pod’s other properties.

Egress is not free. Two and a half to three and a half terabytes a day of internal context leaving the building is a bandwidth bill and a WAN engineering programme, and neither appears in the per-token comparison.

The pod also does the retrieval. The cloud comparison prices tokens only. We would still be running the vector database, the embedding pipeline, the document ingestion and the agent orchestration ourselves, on hardware we pay for either way. The pod’s second rack absorbs that work; the cloud column quietly assumes it happens somewhere free.

What we are actually buying

That third row is the strongest argument against us and should be stated plainly rather than buried: hosted inference prices have fallen sharply and repeatedly, and a pod bought today is depreciating silicon competing against a cost curve that keeps dropping. At 50 million tokens a day this consideration is fatal — the same pod would cost over $40 per million tokens and be indefensible. At 1 billion a day, with a sovereignty requirement that cloud cannot satisfy at any price, it is a manageable risk mitigated by consumption-based MSP terms that include hardware refresh rather than leaving us holding obsolete equipment.

The structure matters here. This is not a seven-figure capital request. It is a monthly subscription to a managed service that happens to live in our building, with the hardware risk carried by the provider.

What one pod covers

Before the engineering, the question every executive asks second: how much is this, in people?

One pod’s realistic sustained ceiling is 2.5 to 3 billion tokens a day. The 1 billion we are designing for is the comfortable operating point, not the limit — it leaves the system at 30 to 60% utilization, which is where latency stays good and a failed node degrades capacity instead of taking the service down.

The difference between 1.5 billion and 2.8 billion is not hardware. It is scheduling. A pod that only serves people typing during business hours gives up half its capacity to the nightly trough; a pod that also runs document indexing, overnight agent work and scheduled analysis fills that trough and reaches the top of the range. This is worth noticing because it is the same fact that answers the “you will pay for idle hardware” objection — the work that fills the nights is the work that justifies the machine.

Turning tokens into people requires knowing what a person consumes, and the spread is enormous. A light user — fifteen chat turns and a few grounded questions a day — costs about 90,000 tokens. A regular knowledge worker running forty turns, half of them against real documents, costs about 540,000. An engineer running four agent tasks a day costs about 1.9 million. One power user is worth roughly twenty-one light users.

So the honest answer to “how many employees” is: somewhere between 2,400 and 6,600 at our design volume, and up to 16,500 at the ceiling — and which end we land on has almost nothing to do with headcount. A single team running continuous autonomous agents can consume more than several thousand people doing chat.

That reframes the capacity question usefully. It is not “how many employees do we have,” it is “how many agent-hours a day do we intend to run,” which is a roadmap decision rather than an HR number. It is also the strongest argument for measuring before buying: the 30-day traffic capture in the plan below replaces all three of the assumptions behind these figures — the input-to-output ratio, the per-user volume, and the share of employees active on a given day — with numbers from our own systems.

Walking the build, part one: the front door

The pod is two racks. Rack B, the 15 kW air-cooled one, is where a request arrives and where the sovereignty boundary is enforced. Nothing in it is exotic. It is worth understanding first, because most of what determines whether the system is useful lives here, and most of what determines whether it is safe lives here too.

The gateway and ingress nodes

What it is: two dual-socket AMD EPYC 9005-class servers with 1.5 TB of memory and four 400 Gb network cards each, running a containerized API proxy.

What it does: every request enters here. The gateway authenticates the caller, checks what that person is entitled to see, scans the request for sensitive data that policy says must be redacted, decides which model tier should handle it, and enforces rate limits.

Why it matters: two reasons, one obvious and one not. The obvious one is that this is where access control lives, and an AI system without it is a very efficient way to help people read documents they should not see. The non-obvious one is that agentic work spends a surprising fraction of its time outside the neural network entirely — parsing API schemas, validating structured output, managing state, deciding what to do next. That is ordinary CPU work, and if it is slow, expensive accelerators sit idle waiting for it. Under-specifying these nodes is one of the most common and most expensive mistakes in an on-premise AI build.

The router

What it is: software on the gateway, plus a small classification model, that decides which of our models answers each request.

Why it matters: this is the single most important economic decision in the system. Roughly 80% of enterprise requests — summarize this, reformat that, answer a straightforward question about a retrieved document — do not require the largest model. They are handled by a model roughly one-sixteenth as expensive to run that scores within a few points of its larger sibling on the tasks that matter. The remaining 20% escalate.

The escalation is not a guess. The cheaper model’s answer is checked — did it produce valid structured output, did the tool call make sense, is it self-consistent — and only if the check fails does the request re-run on the large reasoning model. We monitor the escalation rate continuously. If it climbs past 30%, the routing policy is wrong and our capacity assumptions no longer hold. This routing is what makes 1 billion tokens a day fit inside 45 kW. Without it, the same workload would need roughly three times the hardware.

The semantic cache

What it is: an in-memory datastore holding recent question-answer pairs, keyed by meaning rather than exact text.

Why it matters: in any large organization, the same questions get asked constantly — in slightly different words, by different people, all week. A cache that recognizes “what is the warranty period on the conveyor drive” and “how long is the drive unit under warranty” as the same question answers the second one instantly, for free, without touching the expensive rack at all. Hit rates of 20–40% on enterprise chat traffic are ordinary. This is the cheapest capacity in the building.

Retrieval: the RAG tier

This is the part that makes the system know anything about our company. It deserves more attention than it usually gets, because a mediocre model with excellent retrieval outperforms an excellent model with mediocre retrieval, every time, on enterprise work.

The embedding nodes. Two servers with low-power inference cards convert documents into vectors — numerical representations of meaning — so that similar concepts land near each other. Every document we want the system to know about passes through here once, and every question passes through here on the way in.

The vector database. A self-hosted store (Qdrant, Milvus or similar) holding those vectors with millisecond lookup. Self-hosted matters: a managed vector service would mean shipping embeddings of our documents to a third party, and embeddings are reversible enough to be treated as the documents themselves.

Hybrid search. Vector search alone fails badly on exactly the content enterprises care about: part numbers, document identifiers, internal acronyms, error codes. “Meaning-similar” is the wrong tool for finding assembly 47-A-2291. So we run vector search alongside traditional keyword search and combine the results, then pass them through a reranking model that reads each candidate against the actual question and reorders them. Three stages, because each covers the others’ blind spots.

Access-filtered retrieval. Entitlements are applied before ranking, not after. Filtering afterwards leaks the existence of documents through timing and result counts, which is a finding waiting to happen.

Multimodal indexing. Documents are indexed as page images as well as extracted text. This is what lets an engineer ask a question about a schematic, a scanned drawing, or a table trapped in a PDF, and get a real answer. For a manufacturing or engineering organization this is not a nice-to-have; it is most of the value, because most institutional knowledge is in drawings.

The agent runtime

What it is: four application servers running the orchestration layer — the thing that lets the model take multi-step action rather than just answer.

Why it matters: an agent generates a few tokens, calls a database, waits, reads the result, calls an API, waits, and continues, for dozens or hundreds of steps. Four properties are non-negotiable: sessions survive restarts and long tool waits without reloading context; every tool runs sandboxed with narrowly scoped credentials; irreversible actions require human approval enforced in the runtime rather than requested in a prompt; and content fetched from outside is permanently tagged untrusted and can never itself authorize a tool call.

That last one is subtle and important. An agent that reads a public web page and then follows instructions written on that page has handed control of our tooling to whoever wrote it. The tag travels with the data through summarization and embedding, and the tool-authorization layer checks it.

Storage

What it is: a 2U all-flash NVMe array, at least 200 TB usable, serving over the network fabric using NVMe-over-Fabrics.

Why it matters: it holds the model weights, the document corpus, the vector index and the audit log. The performance requirement comes from an operational scenario: when a node fails or a model needs swapping, we want weights loaded across the fabric in under fifteen seconds rather than having the rack sit idle for minutes. Over a three-year deployment those minutes are the difference between meeting the availability target and missing it.

Walking the build, part two: the engine

Rack A is the 45 kW liquid-cooled rack, and its design rests on a single insight that took the industry several years to act on.

The insight: reading and writing are different jobs

When a model answers a question, it does two completely different kinds of computation.

Prefill is reading. The model ingests the entire prompt — the system instructions, the 48,000 tokens of retrieved documents, the conversation history — and builds an internal representation of it. This is dense matrix arithmetic. It saturates the arithmetic units of a processor and is limited by raw computational throughput.

Decode is writing. The model emits one token, then uses that token to emit the next, and so on, one at a time, sequentially. Each step requires streaming the model’s weights out of memory to compute what is effectively a single small multiplication. It barely uses the arithmetic units at all; it is limited entirely by how fast memory can be read.

Running both on the same hardware means one of them is always wasting the machine. During decode — which is the overwhelming majority of elapsed time in chat and agent work — the expensive matrix units sit idle for over 90% of clock cycles while still drawing power. That stranded capacity is the single largest source of waste in a conventional AI rack, and eliminating it is what takes this build from 120 kW to 45 kW.

Worse, the two phases interfere. In a shared system, one user submitting a 32,000-token document stalls token generation for everyone else on that machine — the “stutter” users notice and complain about. Separating the phases physically eliminates it by construction.

The prefill node

What it is: one 8-accelerator node, 288 GB of high-bandwidth memory per accelerator — 2.3 TB in a single tightly-coupled domain — direct-liquid-cooled, power-capped to 13.5 kW. An NVIDIA HGX B300-class system or the AMD Instinct MI355X equivalent.

What it does: reads prompts and multimodal content, builds the internal representation, hands it off, and immediately picks up the next batch. It never generates tokens.

Why it matters: this is where the 48,000-token RAG prompt and the scanned engineering drawing get processed, and where responsiveness is won or lost — time-to-first-token is what a user experiences as “is this thing working?”. The 2.3 TB of memory is what lets the reasoning model, the multimodal model and the embedding models all stay resident simultaneously, so nothing has to be loaded from storage on the critical path.

The decode nodes

What it is: two more 8-accelerator nodes of the same family, power-capped to 11 kW each, running the token-generation loop with speculative decoding.

Why the same hardware as prefill: there is a genuinely tempting alternative here, and the reasoning for rejecting it as the baseline is worth stating. Specialized inference chips — Tenstorrent’s Galaxy systems, Qualcomm’s AI200 cards with 768 GB of memory each, NVIDIA’s Groq-derived LPX racks — are meaningfully more efficient per watt at decode. Some post genuinely striking numbers. But they achieve that efficiency by hardwiring assumptions about model architecture, and the models we plan to run break those assumptions: sparse mixture-of-experts routing, hybrid linear attention, speculative decoding that changes the arithmetic shape mid-flight. A chip taped out two years ago for dense transformer attention loses much of its advantage against a general-purpose accelerator whose behavior can be changed in software in an afternoon.

The practical argument is simpler still: identical hardware across prefill and decode means one driver stack, one firmware matrix, one spares pool, and the freedom to re-role a decode node as a prefill node when the workload shifts. Given that we do not yet know our true traffic mix, that flexibility is worth more than a percentage of efficiency. Specialized silicon is specified as a permitted alternative that a bidder may propose — but they must demonstrate our entire model portfolio running at spec on it before we buy.

Speculative decoding deserves a mention because it is nearly free performance: a small fast model guesses the next several tokens, the large model verifies them in a single pass, and accepted guesses come out at a fraction of the cost. Correctly tuned it raises throughput by 1.5–2.5x with no change in output quality.

The CXL memory pool — the unglamorous hero

What it is: two memory expansion sleds holding 4 TB of ordinary DDR5, attached over CXL 3.0, drawing 1.6 kW.

Why it matters: this is the component most likely to be cut for cost and most likely to be regretted. Here is the scenario it exists for. An agent generates a query, calls a database, and waits 400 milliseconds — or, in a real workflow, ten minutes for a batch job. During that wait, its accumulated context sits in the most expensive memory in the building, doing nothing, blocking other work from entering the system.

The pool gives that state somewhere cheap to wait. Context is pushed out to commodity DDR5 in microseconds and pulled back when the tool returns, with no recomputation. Without it, we choose between wasting premium memory on idle sessions or recomputing context every time a tool returns — and at hundreds of concurrent agent sessions, both choices are expensive. A 200-session workload at 48,000 tokens of context each needs roughly 430 GB just for parked state.

The network fabric

What it is: two 64-port 800 GbE switches running RoCEv2 — remote direct memory access over Ethernet — with flow control and congestion notification carefully tuned. Plus 400 GbE leaf switches in Rack B, a 400 Gb link between the racks, and a physically separate 1 GbE management network.

Why it matters: disaggregating prefill from decode means the internal representation built by the prefill node must physically travel to a decode node, many times a second. That transfer is the dominant traffic on this network and it must complete in single-digit milliseconds or the whole architecture loses to a simpler design. RDMA moves it memory-to-memory without involving either processor.

We chose Ethernet over InfiniBand deliberately: our network team already knows it, it costs less, and it avoids proprietary fabric lock-in. The catch, and this is the one to hold a bidder to, is that lossless Ethernet must be tuned, not merely configured. Priority flow control and congestion notification set to defaults are the single most common cause of mysterious tail-latency problems in disaggregated inference. We require measured results, not a configuration file.

Power and cooling

48V DC busbar. Instead of a power supply in every server, a bank of 48 V DC rectifier shelves feeds the whole rack over a copper busbar. This cuts conversion losses by 6–8%, removes a large number of chassis fans, and simplifies maintenance. At 45 kW those percentages are real money and real heat. The rail is three 1OU ORv3 HPR shelves — 99 kW of raw capacity against a 42.1 kW DC load, which keeps the rack up with an entire shelf and a further rectifier out of service. The 1OU battery unit beneath them covers transfer ride-through in seconds — the 10-minute battery backup requirement is a facility UPS or a rack battery cabinet, a separate issue from this shelf.

Direct-to-chip liquid cooling. Coolant is pumped directly across the accelerator packages and the heat is rejected through a rear-door heat exchanger or facility water — about 65 litres per minute at a 10 °C temperature rise. No air can move 45 kW out of one rack. A rear-door exchanger is a closed loop that fits an existing raised floor; it is the specific technology that makes this a brownfield retrofit rather than a construction project.

Electrical feeds. Rack A takes two 80 A 480 V three-phase circuits, each able to carry the whole rack on its own. The arithmetic behind the number: 45 kW at 480 V three-phase draws 54.1 A, and electrical code derates a continuous load to 80% of circuit capacity, so the circuit has to be rated for at least 67.7 A. 80 A is the next standard size above that, and it leaves room to grow into the full 45 kW envelope without re-pulling conductors. Two feeds, A and B, because a rack that loses its supply loses the service outright — this is the cheapest redundancy in the pod. It is also among the longest-lead items on the facilities side, which is why it belongs in the site survey on day one rather than in a change order three weeks after go-live.

Power capping. Each accelerator node is capped in firmware at its budgeted figure and the cap is monitored rather than assumed. Uncapped, nodes of this class draw 14–15 kW each; three of them would exceed the rack envelope by roughly 8 kW. The caps are what make 45 kW a real number, and we test it with all three nodes saturated simultaneously before we accept the system.

What actually runs on it

The hardware is the easy part to explain and the models are what people will ask about. The portfolio is deliberately plural: no single model is the right tool for every request, and pretending otherwise is how organizations end up paying frontier prices to reformat a paragraph.

On the model we deliberately excluded. Kimi K3 ties GLM-5.3 for the strongest open model available. It is not in this design: its weights run to roughly 1.56 TB, serving it wants on the order of sixteen top-end GPUs, it generates around 38 tokens per second — below our interactive threshold — and its license requires a separate commercial agreement above $20M in annual revenue. It is an impressive model and the wrong shape for a 45 kW rack. Being able to explain why a well-known model is absent is part of the diligence.

Two disciplines govern all of them. First, every model is pinned as a matched set — weights, tokenizer, chat template, serving engine, quantization recipe — verified by cryptographic digest, and nothing upgrades without passing our own evaluation suite. This is the same discipline we already apply to our build images, extended to model artifacts. Second, we quantize in-house from original weights rather than downloading community-compressed versions. A public repository is an unverified supply chain, and some circulating derivatives have had their safety behavior deliberately stripped at the weight level — which is exactly the sort of thing that does not announce itself.

If procurement rules out non-US weights

Four of the six choices above come from labs outside the United States. That is a procurement conversation worth having deliberately, at the start, rather than discovering it in a contract review nine months in. So here is the same portfolio built only from US-origin models under licenses that permit commercial enterprise use — with the compromise each one carries stated plainly, because there is a compromise in every row but one.

The pattern is worth naming. The restriction does not cost us one big thing; it costs us a little on every tier, and the losses compound in an unhelpful direction. A weaker workhorse escalates more often, and it escalates to a reasoning tier that needs more memory and more active parameters per unit of capability than the one it replaced. Cost per token goes up under the restriction, not down — which is the opposite of what most people assume when they imagine trading capability for compliance.

The retrieval row is the one to watch. It attracts the least attention and does the most damage: a grounded assistant that cannot find the right paragraph fails in a way no amount of reasoning capability repairs. If the restriction is adopted, the engineering budget shifts toward hybrid search and reranking to claw back what the embedder gives up.

None of this is permanent. This list has a half-life measured in months — models leapfrog each other faster than procurement cycles close, and the specific names here will be stale before the hardware ships. What survives is the requirement: US origin, a commercially permissive license, and a passing score on our own evaluation set. Write that into the specification and re-run the selection at each model refresh; do not write model names into a contract.

The objections worth taking seriously

Any proposal of this size attracts objections. These are the ones that are actually correct, stated as strongly as their proponents would state them, with our answer.

“Enterprise AI usage follows working hours. You will pay for hardware that idles three-quarters of the time.”

True for interactive chat, and this is the strongest argument against dedicated hardware in general. Our answer is that the workload we are sizing for is not primarily interactive. Continuous document indexing, overnight agentic runs, code and drawing analysis, and scheduled research are steady-state work that fills the nights and weekends — and they are the workloads with the highest value and the worst fit for metered pricing. If our usage profile turns out to be 9-to-5 chat and nothing else, the diurnal objection wins and we should not build this. The 30-day measurement in the plan below is designed to answer exactly that question before we commit.

“Cloud inference prices keep falling. You will be holding depreciating silicon.”

Also true, and unavoidable. Three mitigations: the consumption-based managed-service structure puts hardware refresh risk on the provider rather than on our balance sheet; the sovereignty requirement is not price-sensitive, because no price makes a cloud endpoint an air gap; and the crossover analysis shows we are comfortably on the right side of the line at our volume, with margin. We should nonetheless review this annually and be honest if the picture changes.

“The MLOps burden is the real cost, and it is bigger than the hardware.”

Correct, and the most commonly underestimated line in any on-premise AI business case. Keeping a distributed serving stack performant, patched and secure takes real engineers at real salaries. We have budgeted 1.5 fully burdened engineers, and the managed-service structure exists precisely to keep the firmware, drivers, fabric tuning and cluster lifecycle off our team’s plate so those 1.5 people work on models, retrieval quality and agent behavior instead of on hardware.

“Agents are brittle on anything smaller than a frontier model.”

This was decisively true eighteen months ago and is why so many on-premise pilots disappointed. It is much less true now: the open models at the top of this portfolio are specifically built for long-horizon agentic work, and one of them sustains hundreds of consecutive tool calls without intervention. But it is not fully resolved, which is why the evaluation gate exists — we test agentic success rates on our own real tasks, before purchase, and if the models fail our bar we do not proceed.

“Long documents will exhaust the memory and starve everyone else.”

This is a genuine failure mode of naive deployments and it is exactly what the CXL memory pool and the prefill/decode separation are for. It is also what the acceptance testing measures: peak concurrency with long contexts, sustained for hours, with tail latency recorded.

When we should not build this

A proposal is more credible when it names its own disqualifying conditions. We should stop if:

  • Measured volume comes in below roughly 730 million tokens a day with no credible path upward. The economics simply do not work, and at 50 million tokens a day this same pod would cost over $40 per million tokens — roughly ten times what renting costs.
  • The models fail our task-level evaluation. Benchmark scores are not a substitute for success on our own work.
  • No sovereignty requirement genuinely exists. If every workload could legitimately run in a cloud tenancy and the only motivation is cost, the case is thinner and depends entirely on sustained volume.
  • We cannot staff it. A pod without the 1.5 engineers becomes an expensive, slowly degrading liability.

The action plan

Nothing here requires a capital commitment for ninety days. The first two phases exist to answer the two questions that decide whether we proceed at all: do we actually have the volume, and do the models actually do our work. Both answers arrive before anyone signs a purchase order.

If all three phases pass, hardware procurement runs 8–16 weeks on long-lead items, installation and commissioning four weeks, pilot six weeks. Realistic first production traffic: seven to nine months from today. Anyone promising materially faster is not accounting for accelerator lead times.

Who owns what

  • Executive sponsor (CIO or CTO): owns the decision, the budget line, and the relationship with the MSP.
  • Principal engineer / platform lead: owns the technical specification, bid evaluation, acceptance testing and the routing policy. This role is the difference between a working platform and an expensive one.
  • Security and compliance: owns the sovereignty requirement, the guard policy, and the annual re-test.
  • Domain experts: own the evaluation set. Without their real tasks and real answers, every other decision is guesswork.
  • Facilities: owns power, cooling and the physical survey — and should be in the room from day one, not consulted in month five when the electrical drawing arrives.

Who to call

These are the organizations to engage and the specific question to put to each. All of them have enterprise sales organizations with a standard intake path; none require a warm introduction. Where possible, go in with the design specification attached — a vendor responding to a written requirement behaves very differently from one responding to “tell us about your AI solutions.”

Turnkey managed platforms — start here

These are the primary bidders. Each sells a consumption-based private AI platform, which is the commercial structure this proposal depends on.

A useful third-party comparison of these platforms is IntuitionLabs’ on-premise AI infrastructure review; read it before the first meeting so the vendor is not defining the frame.

Colocation, if the server facility will not take it

Go to these only after the facilities survey. Ask for high-density, liquid-ready or rear-door-exchanger-ready cages, dedicated tenancy, and 80 A 480 V redundant feeds — and ask what they charge per rack per month at 45 kW, since high-density pricing differs sharply from standard.

  • Equinix — enterprise sales; ask about liquid-cooling-enabled IBX facilities in our region
  • Digital Realty — enterprise sales; same question
  • Our existing colocation provider, if we have one — always check first, since an existing master agreement usually beats a new one

Silicon, for direct technical engagement

These conversations are worth having in parallel with the MSPs, because the accelerator vendors will tell you things the integrators will not, and their field engineers are the ones who know where the real bottlenecks are.

  • NVIDIA — enterprise/data-center team. Ask about B300-class node availability and lead times, and about the Groq 3 LPX decode accelerator: specifically whether any configuration exists below full rack scale, since the current product is a complete MGX rack
  • AMD — Instinct data-center team. Ask for a like-for-like MI355X quote against the NVIDIA configuration and honest guidance on ROCm maturity for the specific models in our portfolio. This is a real price lever even if we do not choose them
  • Tenstorrent — direct sales; their Galaxy systems are air-cooled and list at $110,000. Ask which of our four models are in their verified set and what porting the others would actually take
  • Qualcomm — Dragonfly AI200 data-center team. Ask whether any deployment below their 140–160 kW rack unit is possible; 768 GB per card is genuinely compelling for agent workloads if it can be packaged to fit

Cooling and power

  • Vertiv, Motivair (now Schneider Electric), CoolIT — rear-door heat exchangers and in-rack liquid distribution. Bring them the facilities survey and ask what our specific floor supports
  • Our electrical contractor — the 80 A 480 V three-phase feed question, early. This is frequently the longest-lead item in the entire project and it is boring enough to get forgotten

Software and open-source

Mostly no procurement required, but worth knowing who to call when something breaks at 2 a.m.

  • vLLM and SGLang — the serving engines; open-source with active communities, and commercial support available through Red Hat (which stewards vLLM) and several vendors
  • Red Hat — OpenShift AI, if we want a supported Kubernetes AI platform rather than assembling one
  • Qdrant or Zilliz (Milvus) — vector databases; both offer self-hosted enterprise editions with support, which matters because this component sits on the critical path for every request

Specialist consultancies

Worth engaging if we lack in-house depth on disaggregated inference specifically — a scarcer skill than general MLOps. Look for firms with named, contactable references running prefill/decode-disaggregated deployments in production, not firms with an AI practice page. Ask every one of them the same opening question: how many disaggregated inference deployments have you put into production, and may we speak to two of those customers? The answer sorts the field quickly.

Pod Rack Elevation

Sources

This essay summarizes the Sovereign AI Inference Pod design specification, which carries the full requirement set, sizing arithmetic, acceptance gates and cost model. Figures here are drawn from it.

Models and benchmarks

Hardware

Vendors and platforms