Share with others:
What is AudioCraft, the new AI tool under Meta?
AudioCraft is an open source program that creates effects and music from text prompts, much like these AI image or video generators. AudioCraft has three models available:
- MusicGen for composing music
- AudioGen for creating sound effects
- EnCodec to help in sound compression
MusicGen was previously known among music creators and AI hobbyists. But now Meta has revealed the code for this model, which allows users to enrich it with their own music data. Understandably, ethical and also legal questions immediately arose, because most AI music works were immediately reported by music publishers as infringement of intellectual property.
Meta specifically stated that it created the default model only based on company-owned music and their licensed music. More specifically, they used 20,000 hours of audio and 400,000 recordings along with text descriptions and metadata, all under the umbrella of the Meta Music Initiative Sound Collection, Shutterstock and Pond5 platforms. They also removed all the vocals before the release, with the aim of preventing imitation of the voices of the creators.
The second model, AudioGen, is dedicated to creating ambient sounds and sound effects. AudioGen is a diffusion-based model, like most modern image generators (DALL-E 2, Stable Diffusion …). In diffusion, the model learns how to gradually remove noise from the initial data, which consists entirely of noise – for example sound or images – and thus moves him step by step closer to the target prompt.
In addition to the effects, AudioGen was also created to generate speech, which Meta admits could be misused by some to spoof voices. Despite the concerns, at least for now, they have not set specific restrictions regarding the different ways of using the AudioCraft application.
The third model, EnCodec, is an improvement on the previous Meta model for creating music with fewer artifacts. Meta claims to more efficiently model audio sequences and capture different levels of information when training data audio waveforms to assist; in creating a new sound.
Meta envisioned AudioCraft as a tool for musicians and creators who could create new compositions without having to physically play instruments. They also targeted developers with a more limited budget, who could use AudioCraft to create different sounds for virtual worlds, and Instagram/TikTok creators, for example, to create the most suitable sounds for their posts.
At least for now, AudioCraft's license does not allow commercial use.
How to install and test AudioCraft AI tool?
The code is on Github, and to install you have moreÄ possibilities. You can use the program Pinokio (https://pinokio.computer) which will more or less automatically install the AI music tool for you. You need to select the AudioGradio module from their library, install it (takes a few minutes) and finally you will get a local IP to test AudioCraft with.
Other methods require pre-installed Python, Pip, Anacondo, minicondo or similar programs. Good and easy to understand guideÄ was posted on GitHub (https://bit.ly/GHglasba) by user mberman84 and is considered a miniconda program. The end result is the same. You will get an IP that you enter in your browser and you can start experimenting.
