Introduction
Large language models (LLMs) are at the forefront of many recent AI innovations. Hetzner, known for its robust hosting solutions, is now exploring LLM inference. This article looks at Hetzner's ongoing experiment, its potential implications, and how it could redefine LLM usage.
What is Hetzner LLM Inference?
Hetzner Inference is an OpenAI-compatible API running on Hetzner's infrastructure. The concept is straightforward: allow users to test a language model without the usual complexity associated with managing computing resources. The current model, Qwen/Qwen3.6-35B-A3B-FP8, is a 35-billion-parameter Mixture-of-Experts model with 3 billion active parameters. It accepts text and images, has a 262K context window, and uses FP8-quantized weights.
Why Does It Matter?
The significance lies in the implications for tech businesses: accessibility and flexibility. Companies can test real workloads on an LLM model without immediately investing in expensive infrastructure. It's an opportunity to explore AI with minimal risk.
Hardware Challenges
Hardware is a crucial part of the equation. Currently, Hetzner does not offer a production-ready solution. There is no billing, SLA, or production guarantee. The goal is to learn how the system behaves, which features matter, and what kind of load it can handle.
Experiments and Initial Tests
The initial integrations with the API are straightforward thanks to OpenAI compatibility. For example, integrating this API requires just a few lines of Python code. A basic test might look like this:
```python pip install openai from openai import OpenAI
client = OpenAI( base_url="https://inference.hetzner.com/api/v1", api_key="YOUR_TOKEN", ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[{"role": "user", "content": "Explain why the sky is blue in one sentence."}], ) print(response.choices[0].message.content) ```
Opportunities and Future
Should Hetzner decide to push this project beyond experimentation, it could offer companies a viable new option for LLM inference, particularly for those seeking to avoid the high costs associated with cloud giants.
Conclusion
Hetzner's initiative with LLM inference is a bold step that could redefine AI solutions for many companies. Although it remains an experiment, it deserves the attention of tech decision-makers. To discuss how this technology can integrate into your project, let's discuss your project in 15 minutes.