Hugging Face Releases Low-Latency Voice Agent Pipeline as Open Source

Hugging Face has released “speech-to-speech,” a low-latency, fully modular voice agent pipeline, as open source. The project adopts a cascade architecture that starts with Voice Activity Detection (VAD), followed by Speech to Text (STT), Language Model (LLM), and Text to Speech (TTS). Each component runs in its own thread and is connected via queues. The API is exposed as an OpenAI Realtime-compatible WebSocket interface, allowing developers to leverage existing client assets.

Technical Features

The standout feature of speech-to-speech is its modularity. It uses Silero VAD v5 for voice activity detection—handling speech boundaries and turn-taking. The default STT is Parakeet TDT, which also supports live partial transcription. The LLM slot communicates via an OpenAI-compatible protocol, so you can specify a hosted provider, Hugging Face Inference Providers, or your own hardware running vLLM or llama.cpp servers. The default TTS is Qwen3-TTS, utilizing a GGML backend.

According to Hugging Face’s official blog, “This pipeline is already in production as the conversational backend for thousands of Reachy Mini robots.” This means it is not just an experimental demo but is built to withstand real-world deployment.

Installation and Quick Start

Installation is straightforward. Python 3.10 or newer is required, and you can set it up with the following command:

pip install speech-to-speech

Next, set your OpenAI API key as an environment variable and start the server:

export OPENAI_API_KEY=...
speech-to-speech

This starts an OpenAI Realtime-compatible server at ws://localhost:8765/v1/realtime. You can then run a client script from another terminal to actually have a conversation:

python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765

If you want to use a local LLM, launch Gemma 4 with llama.cpp and specify that server with --responses_api_base_url when starting speech-to-speech, achieving a fully local stack.

Module Swapping and Extensibility

The design philosophy of speech-to-speech is that “all components are swappable.” CLI flags let you freely switch backends for each stage. The code is designed to prioritize models available through Transformers and the Hugging Face Hub, making it easy to modify.

Current default combinations are:

• STT: Parakeet TDT
• LLM: OpenAI-compatible API (any provider depending on settings)
• TTS: Qwen3-TTS (GGML backend, mlx-audio on Apple Silicon)

Dependencies for macOS and non-macOS are automatically resolved via platform markers in pyproject.toml. Linux environments require the CUDA 12.8 runtime; if you don’t have it, you can pre-install the appropriate wheel from Hugging Face’s wheelhouse.

Production Use Cases

As mentioned, this pipeline is already in production on thousands of Reachy Mini robots. This demonstrates that speech-to-speech is a viable option for deploying voice agents in production. It is especially promising for edge devices and robotics, where a low-latency, fully local voice pipeline is essential.

Additionally, having an OpenAI Realtime-compatible API is a major advantage for developers. Existing OpenAI Realtime API clients can connect directly, lowering migration costs.

Context of the Open Source Strategy

Hugging Face has carved out a unique position as an open source AI platform. CEO Clément Delangue has consistently advocated for the value of open source models. The release of speech-to-speech can be seen as part of that strategy. By offering an option that doesn’t rely on proprietary voice agent APIs and allows building a completely closed pipeline on your own hardware, it aims to meet enterprise data sovereignty and privacy requirements.

As we previously noted in “Hugging Face CEO on the Value of Open Source AI,” the company is focused on expanding the open ecosystem.

Editorial Opinion

In the short term, this speech-to-speech pipeline will become a strong option for developers in robotics and edge AI. Its ability to run fully locally while maintaining compatibility with the OpenAI Realtime API is likely to be appreciated in use cases sensitive to data security and latency. As the community expands the components within the Hugging Face ecosystem, further adoption can be expected.

From a long-term perspective, it could boost the presence of open source in the voice agent market. Currently, voice agent APIs are offered by various companies in a proprietary manner. If open, fully swappable options like this project become widespread, companies can more easily avoid vendor lock-in. This could be especially relevant in privacy-regulated sectors like healthcare and finance.

However, challenges remain for real-world deployment. The overall quality depends directly on the performance of individual components—such as VAD accuracy, STT noise robustness, and LLM latency.

References

• “huggingface /

  speech-to-speech", by **huggingface** — GitHub Trending, 2026-07-31 (ARR)

• Source URL: https://github.com/huggingface/speech-to-speech