What hardware for a local AI server

Memory first, compute second

On a conventional server you look at the processor. On an inference machine you look first at the memory accessible to the graphics processor, because it determines which models can run at all. Compute then determines how fast the answers come.

The order matters. A very powerful machine with little memory cannot run a large model, however powerful it is. A more modest machine with plenty of memory runs it, more slowly, but it runs it.

Dedicated memory or unified memory

The classic architecture separates processor memory from graphics card memory. Every exchange between the two crosses a bus, which costs time. To go beyond a certain model size you then have to stack several cards, with the cost, power draw and complexity that follow.

Unified architecture removes that separation: processor and graphics side address the same memory space. The NVIDIA GB10 Grace Blackwell superchip follows this principle with 128 GB unified, which loads models in a desktop enclosure that previously required a multi-card configuration.

The line items everyone forgets

The budget for an in-house AI server almost always exceeds the initial estimate, because several line items go unanticipated.

• Power supply: a multi-GPU configuration frequently exceeds what a standard office socket can deliver.
• Cooling and noise: these machines are designed for technical rooms, not for an open-plan floor.
• Fast storage: models weigh tens of gigabytes and are loaded often. Slow storage penalises every start.
• The software layer: drivers, inference engine, model updates. This is the line item that consumes the most internal time over the years.

Sizing by number of users

Inference occupies the machine for as long as it takes to produce an answer. But usage is intermittent: you ask, you read, you work, you come back. Sizing is therefore based on real concurrency, not on the number of accounts.

For a team of twenty to thirty people, a single well-sized machine is usually enough. DIWY is calibrated for around twenty concurrent users on everyday work, with 1 PFLOP at FP4 precision, 128 GB of unified memory and 4 TB of NVMe.

Buy, build or rent

Building it yourself gives the best control and the worst ratio of time invested to result, unless you already have MLOps skills in-house. Hardware is only the visible part: it is the software layer that demands continuous attention.

Buying a ready-made machine removes the assembly but leaves operations and obsolescence on your side, in a field where hardware moves fast.

Renting shifts that risk to the vendor. DIWY is offered at 500 € per month with a twelve-month commitment, or 750 € per month with no commitment, with a refundable 750 € deposit at checkout. Hardware, Devana OS, Suite 366 and Devana are included, and the box is delivered within 72 hours.