In a bold move, Hetzner has entered the field of AI inference, highlighting a significant shift in how AI workloads are managed and the potential cost savings involved. Hetzner Inference is an OpenAI-compatible API hosted on Hetzner's infrastructure and currently offers a single model, Qwen3.6-35B-A3B-FP8, for experimentation. This initiative marks an exploration into cost-effective AI inference options, challenging the traditional reliance on expensive hardware investments.
The Cost of Inference
Traditionally, AI inference, particularly for large language models (LLMs), has required substantial hardware investments, often involving advanced GPUs and extensive infrastructure. This has made AI deployment costly, particularly for startups and smaller companies. However, Hetzner's experiment suggests a potential shift towards more accessible and affordable AI services. By utilizing a 35-billion-parameter Mixture-of-Experts model with FP8-quantized weights, Hetzner demonstrates that significant AI tasks can be performed without a massive GPU cluster, thus reducing costs.
Hetzner's Experimentation
Hetzner's foray into AI inference is still experimental, with no billing, SLA, or production guarantees in place, as noted in a HackerNews discussion. The test involves a model that accepts both text and images, boasting a 262K context window. By providing an OpenAI-compatible API, Hetzner encourages users to experiment with the service, collecting valuable data on system performance and user demand.
The model, while not groundbreaking in its performance—missing simple arithmetic questions, for example—serves its purpose as a testbed for understanding the demand and potential scaling of AI inference services. As the Typebulb report illustrates through cross-entropy comparisons, models like Qwen are not designed to lead in performance but to explore infrastructural capabilities.
Implications for Developers
For developers, the implications of such experiments are substantial. Reduced reliance on high-cost hardware could democratize access to AI capabilities, enabling more developers to integrate AI into their applications without prohibitive costs. This is particularly relevant as the industry sees a rise in AI applications across various sectors, necessitating more cost-effective infrastructure solutions.
A Shift in AI Infrastructure Strategy
Hetzner's initiative could pioneer a shift in how AI inference is approached, moving away from traditional hardware dependency towards more innovative, cost-efficient models. While the current experiment is not ready for production, it signals a potential future where AI services are not only more accessible but also economically viable for a broader range of enterprises. The real challenge will be maintaining performance and reliability as these services scale.
Overall, Hetzner's experiment is a promising indicator of future trends in AI infrastructure, suggesting a move towards cost reduction and increased accessibility without sacrificing capability.