{"id":2425,"date":"2023-11-19T00:10:44","date_gmt":"2023-11-18T23:10:44","guid":{"rendered":"https:\/\/theroamingworkshop.cloud\/b\/?p=2425"},"modified":"2024-02-01T18:58:37","modified_gmt":"2024-02-01T17:58:37","slug":"%f0%9f%90%b8coqui-ai-tts-ultra-fast-voice-generation-and-cloning-from-multilingual-text","status":"publish","type":"post","link":"https:\/\/theroamingworkshop.cloud\/b\/en\/2425\/%f0%9f%90%b8coqui-ai-tts-ultra-fast-voice-generation-and-cloning-from-multilingual-text\/","title":{"rendered":"\ud83d\udc38Coqui-AI\/TTS: ultra fast voice generation and cloning from multilingual text"},"content":{"rendered":"\n<p>A few months ago I brought TorToiSe-TTS repo, which made it easy to generate text-to-speech although it only worked with english models.<\/p>\n\n\n\n<p><a href=\"https:\/\/theroamingworkshop.cloud\/b\/en\/2083\/%f0%9f%90%a2tortoise-tts-ai-text-to-speech-generation\/\">https:\/\/theroamingworkshop.cloud\/b\/en\/2083\/%f0%9f%90%a2tortoise-tts-ai-text-to-speech-generation\/<\/a><\/p>\n\n\n\n<p>But AI world is moving so fast that today I&#8217;m bringing an evolution that completely exceeds the previous post, with complex voice generation and cloning in a matter of seconds and multilingual: <strong>Coqui-AI TTS<\/strong>.<\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/coqui-ai\/TTS\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/github.com\/coqui-ai\/TTS<\/a><\/p>\n\n\n\n<div id=\"menu\" style=\"padding:20px 20px 20px 20px; border-left:2px solid darkgrey;\">\n<p style=\"font-weight:bold;\">Content<\/p>\n<\/div>\n<script>\nvar text;\nvar element;\n\nfunction fillmenu(){\n\nfor (let i=0; i<window.document.getElementsByTagName(\"h2\").length; i++){\nelement = window.document.getElementsByTagName(\"h2\")[i];\ntext = \"\u25b9 \"+element.innerHTML;\nvar newelement = document.createElement(\"a\");\nnewelement.innerHTML=text;\nvar postid=window.document.getElementsByTagName(\"article\")[0].id;\nvar url = \"https:\/\/theroamingworkshop.cloud\/b\/?p=\"+postid.substr(5,postid.length)+\"#\"+element.id;\nnewelement.setAttribute(\"href\", url);\nnewelement.appendChild(document.createElement(\"br\"));\nwindow.document.getElementById(\"menu\").appendChild(newelement);\nconsole.log(text);\n}\n}\nif(window.location.href.includes(\"tag\")){\n\n}else{\nwindow.setTimeout(fillmenu,2000);\n}\n\n<\/script>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"web-version\">Web version<\/h2>\n\n\n\n<p>If you're in a rush and don't want trouble, you can use the free huggingface space and get your cloned voice in a few seconds:<\/p>\n\n\n\n<p><a href=\"https:\/\/huggingface.co\/spaces\/coqui\/xtts\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/huggingface.co\/spaces\/coqui\/xtts<\/a><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Write the text to be generated<\/li>\n\n\n\n<li>Select language<\/li>\n\n\n\n<li>Upload your reference file<\/li>\n\n\n\n<li>Configure the other options (tick the boxes: Cleanup Reference Voice, Do not use language auto-detect, Agree)<\/li>\n\n\n\n<li>Request cloning to the server (Send)<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"installation\">Installation<\/h2>\n\n\n\n<p>Another strength of <strong>Coqui-AI TTS <\/strong>is the almost instant installation:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You'll need <strong>python<\/strong> &gt; 3.9, &lt; 3.12.<\/li>\n\n\n\n<li>RAM: not as much as for image generation. <strong>4GB<\/strong> should be enough.<\/li>\n\n\n\n<li>Create a project folder, for example \"text-2-speech\". Using a Linux terminal:<br><kbd>mkdir text-2-speech<\/kbd><\/li>\n\n\n\n<li>It's convenient to create a specific python environment to avoid package incompatibilities, so you need <strong>python3-venv<\/strong>. I'll create an environemtn called TTSenv:<br><kbd>cd text-2-speech<\/kbd><br><kbd>python3 -m venv TTSenv<\/kbd><\/li>\n\n\n\n<li>Activate the environment in the terminal:<br><kbd>source TTSenv\/bin\/activate<\/kbd><\/li>\n\n\n\n<li>If you only need voice generation (without cloning or training), install TTS directly with python:<br><kbd>pip install TTS<\/kbd><\/li>\n\n\n\n<li>Otherwise, install the full repo from <strong>Coqui-AI TTS<\/strong> github:<br><kbd>git clone https:\/\/github.com\/coqui-ai\/TTS<\/kbd><br><kbd>cd TTS<\/kbd><br><kbd>pip install -e .[all]<\/kbd><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"check-models\">Checking language models and voices<\/h2>\n\n\n\n<p>First thing you can do is to check the available models to transform text into voice in different languages.<\/p>\n\n\n\n<p>Type the following in your terminal:<\/p>\n\n\n\n<p><kbd>tts --list_models<\/kbd><\/p>\n\n\n\n<p style=\"width:100%;height:200px;overflow-y:visible;overflow-x:hidden;background-color:lightgrey;padding:5px 5px 5px 5px;\"><code style=\"background-color:transparent\">\nNo API token found for \ud83d\udc38Coqui Studio voices - https:\/\/coqui.ai \n<br>Visit \ud83d\udd17https:\/\/app.coqui.ai\/account to get one.\n<br>Set it as an environment variable `export COQUI_STUDIO_TOKEN=<token>`\n<br>\n<br>\n<br> Name format: type\/language\/dataset\/model\n<br> 1: tts_models\/multilingual\/multi-dataset\/xtts_v2 [already downloaded]\n<br> 2: tts_models\/multilingual\/multi-dataset\/xtts_v1.1 [already downloaded]\n<br> 3: tts_models\/multilingual\/multi-dataset\/your_tts\n<br> 4: tts_models\/multilingual\/multi-dataset\/bark [already downloaded]\n<br> 5: tts_models\/bg\/cv\/vits\n<br> 6: tts_models\/cs\/cv\/vits\n<br> 7: tts_models\/da\/cv\/vits\n<br> 8: tts_models\/et\/cv\/vits\n<br> 9: tts_models\/ga\/cv\/vits\n<br> 10: tts_models\/en\/ek1\/tacotron2\n<br> 11: tts_models\/en\/ljspeech\/tacotron2-DDC\n<br> 12: tts_models\/en\/ljspeech\/tacotron2-DDC_ph\n<br> 13: tts_models\/en\/ljspeech\/glow-tts\n<br> 14: tts_models\/en\/ljspeech\/speedy-speech\n<br> 15: tts_models\/en\/ljspeech\/tacotron2-DCA\n<br> 16: tts_models\/en\/ljspeech\/vits\n<br> 17: tts_models\/en\/ljspeech\/vits--neon\n<br> 18: tts_models\/en\/ljspeech\/fast_pitch\n<br> 19: tts_models\/en\/ljspeech\/overflow\n<br> 20: tts_models\/en\/ljspeech\/neural_hmm\n<br> 21: tts_models\/en\/vctk\/vits\n<br> 22: tts_models\/en\/vctk\/fast_pitch\n<br> 23: tts_models\/en\/sam\/tacotron-DDC\n<br> 24: tts_models\/en\/blizzard2013\/capacitron-t2-c50\n<br> 25: tts_models\/en\/blizzard2013\/capacitron-t2-c150_v2\n<br> 26: tts_models\/en\/multi-dataset\/tortoise-v2\n<br> 27: tts_models\/en\/jenny\/jenny\n<br> 28: tts_models\/es\/mai\/tacotron2-DDC [already downloaded]\n<br> 29: tts_models\/es\/css10\/vits [already downloaded]\n<br> 30: tts_models\/fr\/mai\/tacotron2-DDC\n<br> 31: tts_models\/fr\/css10\/vits\n<br> 32: tts_models\/uk\/mai\/glow-tts\n<br> 33: tts_models\/uk\/mai\/vits\n<br> 34: tts_models\/zh-CN\/baker\/tacotron2-DDC-GST\n<br> 35: tts_models\/nl\/mai\/tacotron2-DDC\n<br> 36: tts_models\/nl\/css10\/vits\n<br> 37: tts_models\/de\/thorsten\/tacotron2-DCA\n<br> 38: tts_models\/de\/thorsten\/vits\n<br> 39: tts_models\/de\/thorsten\/tacotron2-DDC\n<br> 40: tts_models\/de\/css10\/vits-neon\n<br> 41: tts_models\/ja\/kokoro\/tacotron2-DDC\n<br> 42: tts_models\/tr\/common-voice\/glow-tts\n<br> 43: tts_models\/it\/mai_female\/glow-tts\n<br> 44: tts_models\/it\/mai_female\/vits\n<br> 45: tts_models\/it\/mai_male\/glow-tts\n<br> 46: tts_models\/it\/mai_male\/vits\n<br> 47: tts_models\/ewe\/openbible\/vits\n<br> 48: tts_models\/hau\/openbible\/vits\n<br> 49: tts_models\/lin\/openbible\/vits\n<br> 50: tts_models\/tw_akuapem\/openbible\/vits\n<br> 51: tts_models\/tw_asante\/openbible\/vits\n<br> 52: tts_models\/yor\/openbible\/vits\n<br> 53: tts_models\/hu\/css10\/vits\n<br> 54: tts_models\/el\/cv\/vits\n<br> 55: tts_models\/fi\/css10\/vits\n<br> 56: tts_models\/hr\/cv\/vits\n<br> 57: tts_models\/lt\/cv\/vits\n<br> 58: tts_models\/lv\/cv\/vits\n<br> 59: tts_models\/mt\/cv\/vits\n<br> 60: tts_models\/pl\/mai_female\/vits\n<br> 61: tts_models\/pt\/cv\/vits\n<br> 62: tts_models\/ro\/cv\/vits\n<br> 63: tts_models\/sk\/cv\/vits\n<br> 64: tts_models\/sl\/cv\/vits\n<br> 65: tts_models\/sv\/cv\/vits\n<br> 66: tts_models\/ca\/custom\/vits\n<br> 67: tts_models\/fa\/custom\/glow-tts\n<br> 68: tts_models\/bn\/custom\/vits-male\n<br> 69: tts_models\/bn\/custom\/vits-female\n<br> 70: tts_models\/be\/common-voice\/glow-tts\n<br>\n<br> Name format: type\/language\/dataset\/model\n<br> 1: vocoder_models\/universal\/libri-tts\/wavegrad\n<br> 2: vocoder_models\/universal\/libri-tts\/fullband-melgan [already downloaded]\n<br> 3: vocoder_models\/en\/ek1\/wavegrad\n<br> 4: vocoder_models\/en\/ljspeech\/multiband-melgan\n<br> 5: vocoder_models\/en\/ljspeech\/hifigan_v2\n<br> 6: vocoder_models\/en\/ljspeech\/univnet\n<br> 7: vocoder_models\/en\/blizzard2013\/hifigan_v2\n<br> 8: vocoder_models\/en\/vctk\/hifigan_v2\n<br> 9: vocoder_models\/en\/sam\/hifigan_v2\n<br> 10: vocoder_models\/nl\/mai\/parallel-wavegan\n<br> 11: vocoder_models\/de\/thorsten\/wavegrad\n<br> 12: vocoder_models\/de\/thorsten\/fullband-melgan\n<br> 13: vocoder_models\/de\/thorsten\/hifigan_v1\n<br> 14: vocoder_models\/ja\/kokoro\/hifigan_v1\n<br> 15: vocoder_models\/uk\/mai\/multiband-melgan\n<br> 16: vocoder_models\/tr\/common-voice\/hifigan\n<br> 17: vocoder_models\/be\/common-voice\/hifigan\n\n<br> Name format: type\/language\/dataset\/model\n<br> 1: voice_conversion_models\/multilingual\/vctk\/freevc24 [already downloaded]\n<\/code><\/p>\n\n\n\n<p>Or filter the result with <strong>grep<\/strong>, for example to get spanish models:<\/p>\n\n\n\n<p><kbd>tts --list_models | grep \"\/es\"<\/kbd><\/p>\n\n\n\n<p style=\"width:100%;height:50px;overflow-y:visible;overflow-x:hidden;background-color:lightgrey;padding:5px 5px 5px 5px;\"><code style=\"background-color:transparent\">\n 28: tts_models\/es\/mai\/tacotron2-DDC [already downloaded]\n<br> 29: tts_models\/es\/css10\/vits [already downloaded]\n<\/code><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"tts\">Text to speech<\/h2>\n\n\n\n<p>With all this you're ready to turn text into speech in a matter of seconds and in the language of your choice.<\/p>\n\n\n\n<p>In the previous terminal, write the following, specifying the right model name:<\/p>\n\n\n\n<p><kbd><kbd>tts --text \"Ahora puedo hablar en espa\u00f1ol!\" --model_name \"tts_models\/es\/css10\/vits\" --out_path output\/tts-es.wav<\/kbd><\/kbd><\/p>\n\n\n\n<p>Make sure that the output folder exists, then check your result. The first time you'll get several files downloaded, and you'll have to accept Coqui-AI license. Next, voice generation only takes a few seconds:<\/p>\n\n\n\n<iframe style=\"border:0; width:100%;height:250px\" src=\"https:\/\/theroamingworkshop.cloud\/aM\/tts-es.php\"><\/iframe>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"cloning\">Voice cloning<\/h2>\n\n\n\n<p>Lastly, the most amazing feature of this model is the voice cloning from only a few seconds of audio recording.<\/p>\n\n\n\n<p>Like in the previous post, I took some 30 seconds of Ultron's voice from the film  <em>Avengers: Age of Ultron<\/em>.<\/p>\n\n\n\n<p>Sample in spanish:<\/p>\n\n\n\n<iframe style=\"border:0; width:100%;height:250px\" src=\"https:\/\/theroamingworkshop.cloud\/aM\/ultron-es.php\"><\/iframe>\n\n\n\n<p>Sample in english:<\/p>\n\n\n\n<iframe style=\"border:0; width:100%;height:250px\" src=\"https:\/\/theroamingworkshop.cloud\/aM\/ultron-en.php\"><\/iframe>\n\n\n\n<p>Now, let's prepare a python script to set all needed parameters, which will do the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Import torch and TTS<br><kbd>import torch<br>from TTS.api import TTS<\/kbd><\/li>\n\n\n\n<li>Define memory device (cuda or cpu). Using cpu should be enough (cuda might probably crash).<br><kbd>device=\"cpu\"<\/kbd><\/li>\n\n\n\n<li>Define text to be generated.<br><kbd>txt=\"Voice generated from text\"<\/kbd><\/li>\n\n\n\n<li>Define the reference audio sample (a .wav file of about 30 seconds)<br><kbd>sample=\"\/voice-folder\/voice.wav\"<\/kbd><\/li>\n\n\n\n<li>Call to TTS model<br><kbd>tts1=TTS(\"model_name\").to(device)<\/kbd><\/li>\n\n\n\n<li>File creation<br><kbd>tts1.tts_to_file(txt, speaker_wav=sample, language=\"es\", file_path=\"output-folder\/output-file.wav\")<\/kbd><\/li>\n<\/ul>\n\n\n\n<p>I called a script <strong>TRW-clone.py<\/strong> looking like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import torch\nfrom TTS.api import TTS\n\n# Get device ('cuda' or 'cpu')\ndevice=\"cpu\"\n\n#Define text\ntxt=\"Bienvenido a este nuevo art\u00edculo del blog. Disfruta de tu visita.\"\n#txt=\"Welcome to this new block post... Enjoy your visit!\"\n\n#Define audio sample\nsample=\"..\/my-voices\/ultron-es\/mix.wav\"\n#sample=\"..\/my-voices\/ultron-en\/mix.wav\"\n\n#Run cloning\ntts1 = TTS(\"tts_models\/multilingual\/multi-dataset\/xtts_v2\").to(device)\n\ntts1.tts_to_file(txt, speaker_wav=sample, language=\"es\", file_path=\"..\/output\/ultron-es.wav\")<\/code><\/pre>\n\n\n\n<p>Run it from the TTS folder where the repo was installed:<\/p>\n\n\n\n<p><kbd>cd TTS<br>python3 TRW-clone.py<\/kbd><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"results\">Results<\/h2>\n\n\n\n<p>Here I drop the results I got on my first tests.<\/p>\n\n\n\n<p>Spanish:<\/p>\n\n\n\n<iframe style=\"border:0; width:100%;height:250px\" src=\"https:\/\/theroamingworkshop.cloud\/aM\/ultron-es2.php\"><\/iframe>\n\n\n\n<p>English:<\/p>\n\n\n\n<iframe style=\"border:0; width:100%;height:250px\" src=\"https:\/\/theroamingworkshop.cloud\/aM\/ultron-en2.php\"><\/iframe>\n\n\n\n<p>And with a couple of iterations you can get really amazing results.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"804\" style=\"aspect-ratio: 1920 \/ 804;\" width=\"1920\" controls src=\"https:\/\/theroamingworkshop.cloud\/b\/wp-content\/uploads\/2023\/11\/Ultron-TTS.mp4\"><\/video><\/figure>\n\n\n\n<p>Any doubts or comments you can still drop me a line on <strong>Twitter\/X<\/strong><\/p>\n\n\n\n<p>\ud83d\udc26 <code><a rel=\"noreferrer noopener\" href=\"https:\/\/twitter.com\/roamingworkshop\" target=\"_blank\">@RoamingWorkshop<\/a><\/code><\/p>\n","protected":false},"excerpt":{"rendered":"<p>A few months ago I brought TorToiSe-TTS repo, which made it easy to generate text-to-speech although it only worked with english models. https:\/\/theroamingworkshop.cloud\/b\/en\/2083\/%f0%9f%90%a2tortoise-tts-ai-text-to-speech-generation\/ But AI world is moving so fast that today I&#8217;m bringing an evolution that completely exceeds the previous post, with complex voice generation and cloning in a matter of seconds and multilingual: [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":2423,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[592,44],"tags":[339,341,343,345,347,349,351,353,355,357],"class_list":["post-2425","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-software","tag-ai-en","tag-artifitial-intelligence-en","tag-ia-en","tag-inteligencia-artificial-en","tag-text-to-speech-en","tag-texto-a-voz-en","tag-tortoise-en","tag-tts-en","tag-voces-en","tag-voices-en","post-preview"],"_links":{"self":[{"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/posts\/2425","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/comments?post=2425"}],"version-history":[{"count":10,"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/posts\/2425\/revisions"}],"predecessor-version":[{"id":2591,"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/posts\/2425\/revisions\/2591"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/media\/2423"}],"wp:attachment":[{"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/media?parent=2425"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/categories?post=2425"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/theroamingworkshop.cloud\/b\/wp-json\/wp\/v2\/tags?post=2425"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}