Getting Started with Audio Programming

Books and other learning resources for new audio developers.

Are you interested in audio programming but don’t know where to begin? Here are my recommendations.

In summary, you need:

Audio programming is a lot of fun, but it requires specialized know-how that may take some time to acquire. But hey, learning is part of the fun!

Learning C++

The audio programmer’s language of choice is C++, pronounced “C plus plus”. Equally hated and loved, it’s the only practical language for writing real-time audio software. It is possible to write the non-audio parts of your program in other languages, but the DSP code needs to be C++.

A few things to know about C++:

There are lots of books and online tutorials on C++, some better than others. The Definitive C++ Book Guide and List has an overview of the best available books.

Here are my tips:

C++ Primer (5th Edition) by Lippman, Lajoie & Moo. If you have no programming experience at all, this is usually the recommended book to start with.

A Tour of C++ by Bjarne Stroustrup, the inventor of C++. A good starting point for intermediate programmers who are new to C++. Also worth reading if you programmed in C++ ages ago and want a refresher on all the new features in the language.

The Cherno on YouTube. If you’re more into watching videos than reading books, this is a very good series of videos on how to learn C++.

cppreference.com This is not a tutorial but an indispensable reference guide for when you need to look up something in the language or the standard library. Bookmark this!

I recommend learning C++ and audio programming together. Studying C++ in a vacuum is not particulary fruitful — it’s better to put your newly acquired knowledge into practice right away by writing real audio code, even if it’s only simple projects.

💡 I recently wrote a book, The Complete Beginner’s Guide to Audio Plug-in Development, that explains the basics of C++ in the context of writing a real audio plug-in. If you’re new to software development, I’d recommend this book as a starting point.

Can I use my favorite language instead of C++?

Probably not… Other languages are often not suitable for real-time audio, as they cannot guarantee consistent timing. Anything that uses garbage collection is a no-no, for example.

The exception here may be Rust, but this language is simply not popular enough (yet?) and doesn’t have the same audio ecosystem that C++ has. I wouldn’t suggest it to a beginner unless they were already proficient in Rust.

You could use C but only if you really really don’t like C++. Keep in mind that almost everything in the audio development ecosystem expects you to interface with C++ at some point.

That said, there are a number of useful audio prototyping languages, such as: Max/MSP, Pure Data, Plug’n Script, REAPER’s JSFX, Python, MATLAB. Mocking up a new idea is usually much quicker in these languages than in C++.

Many materials on audio programming will have code examples that are not in C++. Often they will use Python or MATLAB or some kind of pseudocode. My suggestion is to learn just enough Python and MATLAB so that you can convert such examples into C++. Once you know C++, it is not that hard to pick up new languages.

Tip: I can recommend learning Python as an additional language in any case. It’s great for writing small scripts and helper programs. I wouldn’t bother getting serious with MATLAB unless your employer or school is already paying for a license.

Development environments

To write and compile C++ code you need an IDE or Integrated Development Environment. Which one depends on your operating system and your own preferences. Most IDEs can be downloaded for free.

Xcode for macOS, downloadable from the Mac App Store.

Visual Studio for Windows. The Community Edition is free and does everything you need.

VSCode, works on any platform. This is a popular editor but it’s not a full-featured IDE and getting the C++ compiler toolchain to work can be tricky if you’re inexperienced. Get it here

CLion from JetBrains, works on any platform. This is a paid product and I have no experience with it, but many developers absolutely love it.

If you’re new to C++ or programming in general, I’d advise starting with Xode if you’re on Mac or with Visual Studio if you’re on Windows. Note that you want the regular Visual Studio, not “Visual Studio Code” a.k.a. VSCode — these are completely different products!

An IDE is a complex tool and it’s worth spending some time learning how to use your tools, especially the debugger.

Audio frameworks

Every operating system and plug-in format has its own proprietary audio APIs, and these are all incompatible. As an audio programmer you want to focus on the fun stuff, not worry about the annoying platform-specific implementation details. Ideally, you’d write your audio code just once and have it work on Windows, Mac, and Linux — and in all the different DAWs.

JUCE. My recommendation is to start with the JUCE framework, used by many professional audio companies in their products. Besides making it easy to write audio code, JUCE has cross-platform support for creating user interfaces. It also comes with a library of ready-made DSP components such as filters, delay lines, reverb, and much more. A potential downside of JUCE is that you have to pay for it unless the software you’re making is open source or for your own personal use. juce.com

iPlug2. This is another open source cross-platform audio framework, but perhaps a little less beginner-friendly than JUCE — the documentation is not great. iPlug2 is free to use in closed source projects, which is why certain developers may prefer it over JUCE. iplug2.github.io

PortAudio. A popular cross-platform audio I/O library. Suitable for making standalone apps but not plug-ins (JUCE and iPlug2 can do both). portaudio.com

AudioKit. Recommended if you primarily intend to make iOS apps. audiokitpro.com

Wwise and FMOD. Learn these if you’re interested in doing audio in the games industry.

Mathematics

Math, who doesn’t love it? 😜

Audio programming involves a fair bit of math. You don’t have to be a math wizard to be able to write audio software, but it’s nice not to panic when you encounter some equations.

To make the most out of your audio development career, you’ll want to get a basic foundation in arithmetic, algebra, trigonometry, calculus, and linear algebra. If you’re interested in AI and machine learning, add statistics and probability. Some dark corners of the audio programming world require heavy-hitter math like differential equations.

Don’t feel bad if you flunked math in school. Here you’ll be using math for something that’s actually fun: making your computer do cool things with sound.

If you need to brush up on any of this, check out the following:

Math Overboard! (Basic Math for Adults) by Colin W. Clark. This is a set of two books that explain how math works from the ground up, written for adults who want to learn math.

An Introduction to the Mathematics of Digital Signal Processing by F. R. Moore. These are two articles published in the Computer Music Journal in 1978 but they’re still relevant. The PDFs are behind a paywall but you can easily find them with a bit of googling.

Khan Academy has excellent free online courses on high school and college math — and pretty much everything else. khanacademy.org

I would suggest getting a couple of books from the next section first, and if you get stuck, level up the areas of math you’re having problems with. Don’t go and study math on its own, do it in the context of digital signal processing.

Digital signal processing (DSP)

Audio programming is a form of DSP, so it’s a good idea to get a solid grounding in how digital signal processing works. Here is a short selection of quality DSP books:

Think DSP by Allen B. Downey. Available in print but also online for free. A great introduction to the basic principles of digital signal processing. Written for beginners and takes it easy on the math. Especially nice is that most of the examples use audio signals. If you haven’t studied any DSP at all yet, I highly recommend starting with this book. It uses Python for the code examples.

The Scientist and Engineer’s Guide to Digital Signal Processing (dspguide.com) by Steven W. Smith. This is a fantastic book written for people who are new to DSP. Not too much math but still goes deeply into the subject. This book is about DSP in general, not only audio, but it’s all stuff you’ll need to know anyway. The code examples are in BASIC pseudocode. The book can be read for free online, although I suggest that you download the PDFs and read those.

Understanding Digital Signal Processing by Richard G. Lyons. Another well-regarded DSP beginner book. This has a bit more math but at some point you’ll have to get dirty with the math anyway, it’s unavoidable. A good book to read after the above two.

There are many other introductory DSP books. Just be aware that many of these are written as textbooks for university students. Such books are more suitable for use in a classroom than for self-study.

Audio programming

General DSP books usually don’t cover audio-specific techniques, but fortunately there are a number of resources that do. I’m going to start by pimping my own books:

The Complete Beginner’s Guide to Audio Plug-in Development by Matthijs Hollemans (that’s me). If you know absolutely nothing about software development yet, this book is written for you. It explains the entire process of creating an audio plug-in from scratch, including the C++ language, setting up the development environment (IDE), installing the JUCE framework, and writing the actual DSP code. The aim of this book is to lower the learning curve of getting started making plug-ins. The book is available as PDF and EPUB from The Audio Programmer or as paperback from Amazon.

Creating Synthesizer Plug-Ins with C++ and JUCE by me again. This e-book explains step-by-step how to create a complete software synthesizer plug-in. You’ll learn the theory of how synthesizers work and how to implement the underlying DSP algorithms in C++. The book is available as PDF, EPUB, and paperback. The source code is on GitHub, so you can already check out the finished synthesizer project.

The Audio Programmer If you learn best by watching videos, I can heartily recommend Joshua Hodge’s YouTube channel. He explains how to use JUCE to make audio effects and synths. Here you can also find the videos and live streams of monthly meetups where Josh and his co-host interview experts from the audio development world.

Designing Software Synthesizer Plugins in C++ and Designing Audio Effect Plugins in C++ by Will C. Pirkle. These books contain a wealth of relevant information, although the explanations are sometimes confusing (you may want to read a regular DSP book first) and I’m not a fan of the coding style. Personally, I found the first edition of the synth book to be easier to read than the second edition, but either edition is definitely worth getting. Lots of good stuff here!

The Computer Music Tutorial by Curtis Roads. A massive book that discusses all facets of computer music, including synthesis methods and audio processing. Not a programming book but it does describe the algorithms in detail. A great reference for anything related to computer audio and very readable. Every time I open it, I learn something new. (Tip: You may want to wait a few months to buy this one; a new edition is coming out later in 2023.)

Hack Audio by Eric Tarr. An accessible introduction to audio processing algorithms. The code examples are in MATLAB but Python versions can be found on GitHub. You can also find useful video tutorials on Eric’s website.

Audio Effects: Theory, Implementation and Application by Joshua D. Reiss and Andrew McPherson. A good practical book that explains a variety of audio effects. The code is in C++. Covers largely the same material as Pirkle’s audio effects book and Hack Audio, but a worthy addition to your bookshelf all the same.

Musimathics I & II by Gareth Loy. Not programming books but quite interesting nonetheless. Volume 1 explains the mathematical foundations of music and sound. Volume 2 is about the math behind digital audio and DSP.

Useful skills to have

Working with a DAW. It seems silly to say it, but you need to have some familiarity with music making, audio editing, sound design, composition, and so on. This is what the industry calls “domain knowledge”. It’s hard to write audio software if you don’t know how things work in the audio world or are confused by the terminology. (Example: do you know what a “sidechain” is?)

Instrument design. Audio programming isn’t only about implementing DSP algorithms. You need to make decisions about which building blocks to use and how to combine them into something that makes sense. This requires an understanding of how effects and instruments typically work — both on the inside and on the outside.

UI design. Great software is easy and intuitive to use, and making good user interfaces is a skill in its own. You may have the greatest DSP in the world, but if the UI is a mess, people won’t use it.

Knowing how to ask for help. There are plenty of experienced people online who are willing to help newcomers, but only if you make it easy for them to help you. Read this

Other resources

The Audio Programmer Community. A Discord server with smart and friendly people who love to chat about audio development all day long. This Discord is very welcoming to newbies but also has members who work for audio companies that you know and love. Come and join, it’s free and you will learn tons.

The JUCE website has tutorials and API documentation. If you haven’t looked at the tutorials yet, you definitely should. They don’t just explain how to use JUCE, but also how to do practical audio programming. Also check out the JUCE forums. And don’t forget the example projects that come with JUCE in the examples folder (also accessible from Projucer).

Academic papers. A lot of the knowledge in the audio field is in the form of academic papers that are published in journals such as the AES Journal and at conferences such as DAFX. You can often find these papers in PDF format online by googling their title. Sometimes the papers are behind a paywall. Once you’ve mastered the basics of audio development, it’s a good idea to get into reading papers.

Awesome lists. Members from the community have collected their favorite audio DSP links. Check them out here:

Audio Developer Conference (ADC). In the field of audio, conferences and organizations swing towards the academic side of things, but ADC is a great yearly event that focuses on code and practical solutions. It is organized by the people behind JUCE. You can find videos of previous ADC talks on the JUCE YouTube channel.