Hot French startup ZML releases free product to speed inference across lots of AI chips

While Nvidia remains a dominant force in the market, new competitors and alternatives are emerging from various sectors.

ZML, a rapidly growing French AI startup backed by Turing Award recipient Yann LeCun, has introduced inference-performance software. This technology enables numerous open-source large language models to operate on diverse chips, including those from Nvidia, AMD, Google’s TPU, Apple Metal, and Intel Arc.

The newly launched LLM inference server by ZML, known as ZML/LLMD, aims to dismantle existing barriers and optimize the use of various chips for AI applications, potentially surpassing their current speeds, according to ZML founder Steeve Morin in a conversation with JS.

As AI becomes more integrated into daily activities and work environments, the optimization of inference, which refers to prompt processing, is gaining priority over model training. However, this process often faces challenges due to software and architecture limitations, leading to vendor lock-in, Morin explained.

Achieving optimal performance across different chips is not only a technological milestone but could also disrupt the market amid growing concerns about AI-related expenses.

ZML seeks to offer enterprises and cloud services the flexibility to utilize a combination of chips, some of which could be more cost-effective or energy-efficient. “The idea is to give people back the power to create their own system and achieve real efficiency gains that allow [AI] to be disseminated,” Morin remarked.

This software innovation could support emerging AI chip manufacturers, particularly those based in Europe, Morin noted, mentioning companies like Axelera, Fractile, Kalray, OLIX, Q.ANT, SiPearl, SpiNNcloud, and VSORA. Ultimately, the focus is on collaborating on “things that haven’t been done before anywhere in the world,” he emphasized.

Morin remains optimistic about Nvidia, largely due to its current supply. He shared with JS that ZML maintains a positive relationship with the AI chip leader, which is preparing for the increased demand for inference.

The area of inference has seen such significant investment that it is being referred to as the “inference gold rush.” ZML faces competition from companies like Baseten, valued at $13 billion; Inferact, creators of the open-source project vLLM; and RadixArk, the company behind SGLang.

Although vLLM and SGLang compete partially with LLMD, Morin’s vision for ZML extends further. “We have reached the point where we are co-designing silicon,” he stated. He attributed ZML’s rapid progress to its small team of 20, which allows the Paris-based startup to innovate quickly, with more developments in the pipeline.

This compact team has also benefited from substantial funding relative to its size. With Morin’s background as VP of engineering at Zenly, which was acquired by Snapchat for a significant sum in 2017, he secured $20 million from investors, including Harry Stebbings’ 20VC, >commit, AALVC, Drysdale Ventures, Xavier Niel’s Kima Ventures, Kindred Capital, LocalGlobe, and Puzzle Ventures.

Unlike ZML’s initial public venture, an inference-focused ML framework launched in 2024 and updated in March, ZML/LLMD is not open source. However, it is being released for free to gather usage insights. “I’d rather measure and [then generate revenue] where it is most effective without hindering my growth stupidly because I have been too greedy from the get-go,” Morin expressed.

It is still uncertain when ZML/LLMD will transition to a paid model and how it will be adopted. Nonetheless, the startup’s investor list indicates significant interest, including attention from Dagger and Docker founder Solomon Hykes, Clément Delangue and Julien Chaumond from Hugging Face, and LeCun, now with AMI Labs. This suggests that European AI startups are increasingly able to grow locally. “I couldn’t do ZML anywhere but in Paris,” Morin shared.