Découverte non supervisée de mots à partir de la parole

Unsupervised word discovery in speech data

• Auteur / Autrice : Robin Algayres
• Direction : Emmanuel Dupoux, Benoît Sagot
• Type : Projet de thèse
• Discipline(s) : Sciences cognitives
• Date : Inscription en doctorat le ; Soutenance le 26/09/2023
• Etablissement(s) : Université Paris sciences et lettres
• Ecole(s) doctorale(s) : École doctorale Cerveau, cognition, comportement
• Partenaire(s) de recherche : Laboratoire : Laboratoire de sciences cognitives et psycholinguistique (1985-....)
• Partenaire(s) de recherche : Equipe de recherche : CoML - Cognitive Machine Learning
• Partenaire(s) de recherche : établissement opérateur d'inscription : École normale supérieure (Paris ; 1985-....)
• Jury : Président / Présidente : Sharon Goldwater
• Jury : Examinateurs / Examinatrices : Emmanuel Dupoux, Herman Kamper, Karen Livescu, Benoît Sagot, Wei-Ning Hsu
• Jury : Rapporteurs / Rapporteuses : Herman Kamper, Karen Livescu

Mots clés

Mots clés libres

Apprentissage automatique non supervisé

Développement du langage chez l'enfant

Language en faible ressource

Infant language development

Low resource languages

Unsupervised machine learning

Résumé

En science des données, les appareils de reconnaissance vocale automatique permettent de transcrire à l’écrit une phrase dite à l’oral. Pour cela, ces systèmes ont besoin d’une grande quantité de parole annotée, c’est à dire que chaque enregistrement audio doit être accompagné de sa transcription textuelle. Cependant, pour la plupart des langages, nous ne possédons que des enregistrements audio et peu, voire aucune annotation. Or, on sait que les nouveau-nés acquièrent leur langage sans l’aide d’annotation textuelle. Par conséquent, dans le but de créer un système de modélisation du langage, nous avons développé un modèle de découverte de mots à partir de la parole qui n'utilise pas de données textuelles.

Inspired by how infants learn their mother tongue(s), we tackle the problem of Spoken Word Discovery without using linguistic data and the usage of discovered words for unsupervised Spoken Language Modelling. Spoken Word Discovery is the unveiling of the notion of words directly from the speech signal. To tackle this problem, we pose two technical challenges. The first one is to learn Speech Sequence Embedding (SSE) which are fixed-size vectors that represent variable-size sequences of speech. Our method is to train a neural network on top of self-supervised features (Wav2vec2.0) using data augmentation, contrastive learning and iterative self-training. The second technical challenge is to perform speech segmentation into words. To achieve this, we introduce DP-Parse, a Dirichlet Process non-parametric Bayesian model inspired from goldwater2009wordseg. DP-Parse finds frequent patterns in the speech signals using density estimation on speech fragments represented with SSEs. Those frequent patterns serve as anchors to perform full-sentence segmentation into word tokens. Then, we further improve segmentation performances by fine-tuning an XLS-R model to predict DP-Parse boundaries. To evaluate SSEs, we introduce the Mean Average Precision (MAP) computed over spoken phoneme-n-grams (i.e. speech fragments that are not necessarily words) instead of spoken words. We prove that this metric is correlated with performances on the frequency estimation task, which is a core element of the DP-Parse algorithm. To evaluate DP-Parse performances, we use F1 scores on correctly predicted word boundaries and tokens. Regarding MAP and F1 scores, our SSE model and DP-Parse reached the level of the current state-of-the-art. After fine-tuning XLS-R on DP-Parse boundaries, we set a significantly higher state-of-the-art in speech segmentation into words. We tackle Spoken Language Modelling as a downstream task that leverages our work on Spoken Word Discovery. Our goal is to train a spoken language model (SLM) on speech recordings that have been segmented by DP-Parse and represented into sequences of SSEs. Instead of discretizing the SSEs with clustering, which we argue is a very hard task because of Zipf's Law, we decided to adapt the training and inference steps of our SLM to the continuous SSEs. In order to evaluate the semantic and syntactic representations learned by our SLM, we introduce two ABX metrics that are sensitive to the quality of the input speech segmentation. Regarding spoken generation, our SLM is able to generate utterances that are on par with the state-of-the-art in intelligibility and meaningfulness as assessed by automatic metrics as well as human raters.