Open-Source Conversational AI with SpeechBrain 1.0

Mirco Ravanelli; Titouan Parcollet; Adel Moumen; Sylvain de Langen; Cem Subakan; Peter Plantinga; Yingzhi Wang; Pooneh Mousavi; Luca Della Libera; Artem Ploujnikov; Francesco Paissan; Davide Borra; Salah Zaiem; Zeyu Zhao; Shucong Zhang; Georgios Karakasidis; Sung Lin Yeh; Pierre Champion; Aku Rouhe; Rudolf Braun; Florian Mai; Juan Zuluaga-Gomez; Seyed Mahed Mousavi; Andreas Nautsch; Ha Nguyen; Xuechen Liu; Sangeet Sagar; Jarod Duret; Salima Mdhaffar; Gaëlle Laperrière; Mickael Rouvier; Renato De Mori; Yannick Estève

Open-Source Conversational AI with SpeechBrain 1.0

Mirco Ravanelli
, Titouan Parcollet
, Adel Moumen
, Sylvain de Langen
, Cem Subakan
, Peter Plantinga
, Yingzhi Wang
, Pooneh Mousavi
, Luca Della Libera
, Artem Ploujnikov
, Francesco Paissan
, Davide Borra
, Salah Zaiem
, Zeyu Zhao
, Shucong Zhang
, Georgios Karakasidis
, Sung Lin Yeh
, Pierre Champion
, Aku Rouhe
, Rudolf Braun

Florian Mai, Juan Zuluaga-Gomez, Seyed Mahed Mousavi, Andreas Nautsch, Ha Nguyen, Xuechen Liu, Sangeet Sagar, Jarod Duret, Salima Mdhaffar, Gaëlle Laperrière, Mickael Rouvier, Renato De Mori, Yannick Estève

Concordia University
Mila - Quebec Artificial Intelligence Institute
University of Montreal
Samsung
University of Cambridge
Avignon Université
Université Laval
Zaion
Fondazione Bruno Kessler
Aalto University
University of Bologna
Ecole Nationale Superieure des Telecommunications
University of Edinburgh
Institut national de recherche en informatique et en automatique
Silo AI
Idiap
KU Leuven
EPFL
University of Trento
Research Organization of Information and Systems, National Institute of Informatics
Saarland University
McGill University

Research output: Contribution to journal › Article › peer-review

50 Citations (Scopus)

Abstract

SpeechBrain¹ is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech recognition, speech enhancement, speaker recognition, text-to-speech, and much more. It promotes transparency and replicability by releasing both the pre-trained models and the complete “recipes” of code and algorithms required for training them. This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face. SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.

Original language	English
Journal	Journal of Machine Learning Research
Volume	25
Publication status	Published - 2024
Externally published	Yes

Keywords

Conversational AI
deep learning
open-source
speech processing

Cite this

@article{6b3f9b9d1ea24e50b2460bd23a9a64ca,

title = "Open-Source Conversational AI with SpeechBrain 1.0",

abstract = "SpeechBrain1 is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech recognition, speech enhancement, speaker recognition, text-to-speech, and much more. It promotes transparency and replicability by releasing both the pre-trained models and the complete {\textquotedblleft}recipes{\textquotedblright} of code and algorithms required for training them. This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face. SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.",

keywords = "Conversational AI, deep learning, open-source, speech processing",

author = "Mirco Ravanelli and Titouan Parcollet and Adel Moumen and \{de Langen\}, Sylvain and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Libera, \{Luca Della\} and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Yeh, \{Sung Lin\} and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Mousavi, \{Seyed Mahed\} and Andreas Nautsch and Ha Nguyen and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Ga{\"e}lle Laperri{\`e}re and Mickael Rouvier and \{De Mori\}, Renato and Yannick Est{\`e}ve",

note = "Publisher Copyright: {\textcopyright} 2024 Mirco Ravanelli, Titouan Parcollet, et al.",

year = "2024",

language = "English",

volume = "25",

journal = "Journal of Machine Learning Research",

issn = "1532-4435",

}

Ravanelli, M, Parcollet, T, Moumen, A, de Langen, S, Subakan, C, Plantinga, P, Wang, Y, Mousavi, P, Libera, LD, Ploujnikov, A, Paissan, F, Borra, D, Zaiem, S, Zhao, Z, Zhang, S, Karakasidis, G, Yeh, SL, Champion, P, Rouhe, A, Braun, R, Mai, F, Zuluaga-Gomez, J, Mousavi, SM, Nautsch, A, Nguyen, H, Liu, X, Sagar, S, Duret, J, Mdhaffar, S, Laperrière, G, Rouvier, M, De Mori, R & Estève, Y 2024, 'Open-Source Conversational AI with SpeechBrain 1.0', Journal of Machine Learning Research, vol. 25.

TY - JOUR

T1 - Open-Source Conversational AI with SpeechBrain 1.0

AU - Ravanelli, Mirco

AU - Parcollet, Titouan

AU - Moumen, Adel

AU - de Langen, Sylvain

AU - Subakan, Cem

AU - Plantinga, Peter

AU - Wang, Yingzhi

AU - Mousavi, Pooneh

AU - Libera, Luca Della

AU - Ploujnikov, Artem

AU - Paissan, Francesco

AU - Borra, Davide

AU - Zaiem, Salah

AU - Zhao, Zeyu

AU - Zhang, Shucong

AU - Karakasidis, Georgios

AU - Yeh, Sung Lin

AU - Champion, Pierre

AU - Rouhe, Aku

AU - Braun, Rudolf

AU - Mai, Florian

AU - Zuluaga-Gomez, Juan

AU - Mousavi, Seyed Mahed

AU - Nautsch, Andreas

AU - Nguyen, Ha

AU - Liu, Xuechen

AU - Sagar, Sangeet

AU - Duret, Jarod

AU - Mdhaffar, Salima

AU - Laperrière, Gaëlle

AU - Rouvier, Mickael

AU - De Mori, Renato

AU - Estève, Yannick

PY - 2024

Y1 - 2024

N2 - SpeechBrain1 is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech recognition, speech enhancement, speaker recognition, text-to-speech, and much more. It promotes transparency and replicability by releasing both the pre-trained models and the complete “recipes” of code and algorithms required for training them. This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face. SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.

AB - SpeechBrain1 is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech recognition, speech enhancement, speaker recognition, text-to-speech, and much more. It promotes transparency and replicability by releasing both the pre-trained models and the complete “recipes” of code and algorithms required for training them. This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face. SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.

KW - Conversational AI

KW - deep learning

KW - open-source

KW - speech processing

UR - https://www.scopus.com/pages/publications/105018580751

M3 - Article

AN - SCOPUS:105018580751

SN - 1532-4435

VL - 25

JO - Journal of Machine Learning Research

JF - Journal of Machine Learning Research

ER -

Open-Source Conversational AI with SpeechBrain 1.0

Abstract

Keywords

Other files and links

Cite this