How code made the music in the launch video

First: what is a marimba?

A marimba is a big wooden xylophone. It has a row of wooden bars. You hit the bars with soft mallets, like sticks with balls of yarn on the end.

Long bars make low notes. Short bars make high notes. Under each bar hangs a metal tube. The tube makes the note louder and warmer, like cupping your hands around your mouth.

Try it. Tap the bars.

Marimba music sounds warm, round and bouncy. Each note starts with a soft "tok" and then fades quickly. The instrument comes from Africa and Central America. Today it is the national instrument of Guatemala. You may know one marimba sound already: the first iPhone's default ringtone was called "Marimba".

Because each note is short and clear, marimba music is great for a fast video. The notes hit the beat and get out of the way.

The problem

The video changes picture almost every second. We want the music to "hit" each change: a ding when the light bulb pops, a beat when the new site slides in.

How do people usually get music?

• Download a song. But the song does not know where our picture changes are. The beats land in the wrong places.
• Play a real marimba and record it. But we have no marimba, no player and no microphone. Also, Claude cannot hear. It cannot listen and say "a bit softer, please".

Root causeWe need sound that we can place to the exact millisecond, using only code, without ears.

So how can we make a marimba-like sound with only a computer?

To answer that, we first need to know what a sound is.

What is sound?

Throw a stone in a pond. Rings of ripples move out. Sound is like that, but in the air. Something shakes, and it pushes the air in tiny waves. The waves reach your ear, and you hear them.

A speaker makes these waves with a thin cone that moves in and out. The computer tells the cone where to be. It does this 44,100 times each second. Each "where to be" is one number.

LikeA flip book. Each page is one number. Flip 44,100 pages each second and you get a sound.

Key ideaA sound is just a long list of numbers. If we can write the list, we can make any sound. We do not need an instrument.

What list of numbers sounds like a marimba?

First, a quick look at the tools. Then we build one, step by step.

The tools in the box

The program is one Python file, about 100 lines. You can read all of it near the end of this page. It uses only a few common tools. There is no music library, no sound samples and no music app.

Every sound starts the same way: a list of times, one for each of the 44,100 numbers in a second.

import numpy as np

SR = 44100                              # numbers per second ("sample rate")
t = np.arange(int(1.2 * SR)) / SR       # 0.0, 0.0000227, 0.0000454, ... up to 1.2 s
print(len(t))                           # 52920 numbers for 1.2 seconds

Build a marimba-like note in four steps

Each button adds one new thing. Listen to each one, and watch the picture of the sound change.

Step 1: a pure tone

The simplest sound is a smooth wave that goes up and down, again and again, like a swing. How fast it swings is the pitch. 523 swings each second is the note C.

It sounds like a microwave oven's beep. Not like wood.

f = 523                                  # the note C: 523 swings each second
sound = np.sin(2 * np.pi * f * t)        # one smooth wave, the same from start to end

Why is a real marimba different from a beep?

Root causeA wooden bar makes many notes at the same time, not one. And its sound does not stay the same: it is loud at the hit and then it gets quiet.

Step 2: add overtones

When the mallet hits the bar, the bar wobbles in more than one way at once. It sings its main note, plus some higher, quieter notes. These are overtones.

LikeA smoothie. Mostly banana (the main note), a bit of strawberry, and a tiny bit of mint. The mix gives the taste. The mix of overtones gives the "wooden" taste of the sound.

Marimba makers carve an arch under each bar to tune these overtones. Our recipe adds one overtone about 4 times higher (at about a quarter of the loudness) and one about 9 times higher (very quiet). See the three ingredients in the picture.

Now it is brighter. But it never stops. It sounds like an organ.

sound = (      np.sin(2 * np.pi * f        * t)     # main note
       + 0.28 * np.sin(2 * np.pi * f * 3.93 * t)     # about 4x higher, quieter
       + 0.07 * np.sin(2 * np.pi * f * 9.2  * t))    # about 9x higher, very quiet

Step 3: add the fade

After you hit a bar, it rings and then it slowly goes quiet, like a bouncing ball that bounces lower and lower.

So we make the wave smaller and smaller over time. The high overtones fade faster than the main note, just like on a real bar. After a moment, you hear mostly the warm main note.

# np.exp(-t * k) starts at 1 and slides towards 0. A bigger k means a faster fade.
sound = (      np.sin(2 * np.pi * f        * t) * np.exp(-t * 3.2)    # slow fade
       + 0.28 * np.sin(2 * np.pi * f * 3.93 * t) * np.exp(-t * 13)     # faster
       + 0.07 * np.sin(2 * np.pi * f * 9.2  * t) * np.exp(-t * 32))    # fastest

Step 4: add the mallet "tok"

A real hit starts with a tiny "tok": the mallet touching the wood. It is not a note. It is a very short noise, like a spoon tapping a table.

So we add 4 thousandths of a second of random numbers at the very start. Your ear hears it as the hit. Now it sounds like a marimba, or close to one.

tok = int(0.004 * SR)                                    # 4 ms = 176 numbers
noise = np.random.normal(0, 0.15, tok)                   # random numbers = noise
sound[:tok] += noise * np.linspace(1, 0, tok)            # loud at first, then gone

def marimba(note, length=1.2):
    t = np.arange(int(length * 44100)) / 44100      # the time of each number
    f = 440 * 2 ** ((note - 69) / 12)                # note number to pitch (Hz)
    s = (np.sin(2*np.pi*f*t)            * np.exp(-t*3.2)   # main note, slow fade
       + 0.28*np.sin(2*np.pi*f*3.93*t)  * np.exp(-t*13)    # small overtone, faster fade
       + 0.07*np.sin(2*np.pi*f*9.2*t)   * np.exp(-t*32))   # tiny overtone, fastest fade
    tok = int(0.004 * 44100)                         # 4 ms
    s[:tok] += np.random.normal(0, .15, tok) * np.linspace(1, 0, tok)
    return s

From one note to music

We can make one marimba-like note. How do we make a song?

A song is just the right notes at the right times. So the program needs two lists: which notes, and when.

• Which note. Notes are like steps on a ladder. Each step up is about 6% higher. Twelve steps make the same note again, twice as high. The code turns a step number into a pitch.
• When. A clock ticks 120 times each minute, so one tick is half a second. Every sound sits on a tick, or halfway between two ticks.
• Which notes sound good together. Groups of notes that sound nice together are chords. The song loops four chords: C, G, A minor, F. Very many pop songs use these four, because they sound happy.
• Drums, made the same way with numbers. A kick drum is a low tone that drops fast. A clap is three quick bursts of noise. A shaker is a very short "tss".

def hz(note):                       # a step on the ladder -> pitch
    return 440 * 2 ** ((note - 69) / 12)

hz(69)   # 440.0  (A)
hz(72)   # 523.3  (C)
hz(84)   # 1046.5 (the next C, twice as high)

song = np.zeros(16 * SR)                  # 16 seconds of silence to fill

def add(sound, at):                       # put a sound into the song at a time (seconds)
    i = int(at * SR)
    song[i:i + len(sound)] += sound[:len(song) - i]

BEAT = 0.5                                # 120 beats each minute
CHORDS = {3.5: [72, 76, 79, 76, 72, 67, 76, 79],    # C
          5.5: [71, 74, 79, 74, 71, 67, 74, 79],    # G
          7.5: [72, 76, 81, 76, 72, 69, 76, 81],    # A minor
          9.5: [69, 72, 77, 72, 69, 65, 72, 77]}    # F
for start, notes in CHORDS.items():
    for k, n in enumerate(notes):
        add(marimba(n), start + k * BEAT / 2)       # 8 notes, half a beat apart

def kick():                               # a low tone that drops fast in pitch
    t = np.arange(int(0.35 * SR)) / SR
    pitch = 48 + 110 * np.exp(-t * 38)    # 158 Hz falling to 48 Hz
    return np.sin(2 * np.pi * np.cumsum(pitch) / SR) * np.exp(-t * 9)

def shaker():                             # a very short, very high "tss"
    t = np.arange(int(0.06 * SR)) / SR
    tss = np.random.normal(0, 1, len(t)) * np.exp(-t * 70)
    return highpass(tss, 6000)            # SciPy filter: keep only the high hiss

Match the music to the video

This is the part a downloaded song cannot do. The video was finished first. Then each sound got a time, in seconds, from the video. These are the dots under the video at the top.

Because the times are just numbers, changes are easy. When the phone part of the video got one second longer, we added one more bar of music, moved the last chord one second later, and ran the program again.

add(glock(84), 2.5)                       # light bulb: bell "ding"
add(whoosh(), 4.18)                       # first post pushed away
add(whoosh(), 5.18)                       # second push
add(slide_whistle(1300, 420), 6.15)       # down the rope
add(glock(88), 8.1);  add(glock(91), 9.1) # filter taps
add(big_c_chord(), 13.5)                  # end card

Mix it and save it

1. Add everything up, one list for the left speaker and one for the right.
2. Round off the loudest bits, so nothing crackles.
3. Set the volume to a normal level for online video, and save it as a sound file.
4. Glue it to the video with a tool called ffmpeg.

from scipy.io import wavfile

mix = np.tanh(song * 1.1) / np.tanh(1.1)          # round off the loudest bits
mix = mix * 0.89 / np.max(np.abs(mix))            # loudest point just under the top
wavfile.write("music.wav", SR, (mix * 32767).astype(np.int16))   # numbers -> sound file

ffmpeg -framerate 30 -i frames/%04d.jpg -i music.wav \
       -af loudnorm=I=-14 -c:v libx264 -c:a aac launch.mp4

The full code

Here is the whole soundtrack program: 105 lines, including comments. The marimba-like note is only 8 of them. The rest makes the drums and bells, puts every sound at its time, and mixes it all. Run it with Python, NumPy and SciPy, and it writes the exact track from the video.

"""The 16-second soundtrack of the notes.rr2.dev launch video, made only from numbers.
120 BPM. 0-2.5 s thinking (tick-tock), 2.5 s light-bulb ding, 3.5 s the beat starts,
whooshes for the pushed posts, a slide whistle down the rope, small bells for the
filters and dark mode, and a big C chord under the end card at 13.5 s.
Needs: numpy, scipy. Writes soundtrack.wav."""
import numpy as np
from scipy.signal import butter, sosfilt
from scipy.io import wavfile

SR, DUR, BPM = 44100, 16.0, 120
BEAT = 60 / BPM
N = int(SR * DUR)
L = np.zeros(N); R = np.zeros(N)
rng = np.random.default_rng(7)

def add(sig, t, gain=1.0, pan=0.0):
    i = int(t * SR); j = min(N, i + len(sig))
    if i >= N: return
    l = np.cos((pan + 1) * np.pi / 4); r = np.sin((pan + 1) * np.pi / 4)
    L[i:j] += sig[: j - i] * gain * l * 1.414; R[i:j] += sig[: j - i] * gain * r * 1.414

def tt(sec): return np.arange(int(sec * SR)) / SR
def hz(note): return 440 * 2 ** ((note - 69) / 12)
def bp(x, lo, hi): return sosfilt(butter(2, [lo, hi], btype="band", fs=SR, output="sos"), x)
def hp(x, f): return sosfilt(butter(2, f, btype="high", fs=SR, output="sos"), x)

def marimba(note, length=1.2, bright=1.0):   # the marimba-like note
    t = tt(length); f = hz(note)
    s = (np.sin(2 * np.pi * f * t) * np.exp(-t * 3.2)
         + 0.28 * bright * np.sin(2 * np.pi * f * 3.93 * t) * np.exp(-t * 13)
         + 0.07 * bright * np.sin(2 * np.pi * f * 9.2 * t) * np.exp(-t * 32))
    s *= np.minimum(1, t / 0.002)
    s[: int(0.004 * SR)] += rng.normal(0, 0.15, int(0.004 * SR)) * np.linspace(1, 0, int(0.004 * SR))
    return s

def glock(note, length=2.5):
    t = tt(length); f = hz(note)
    return (np.sin(2 * np.pi * f * t) + 0.35 * np.sin(2 * np.pi * f * 2.76 * t) * np.exp(-t * 4)) * np.exp(-t * 1.6) * np.minimum(1, t / 0.001)

def kick():
    t = tt(0.35); f = 48 + 110 * np.exp(-t * 38)
    return np.sin(2 * np.pi * np.cumsum(f) / SR) * np.exp(-t * 9)

def clap():
    out = np.zeros(int(0.25 * SR))
    for k, off in enumerate((0, 0.011, 0.022)):
        b = rng.normal(0, 1, int(0.009 * SR)) * np.linspace(1, 0.3, int(0.009 * SR)); i = int(off * SR)
        out[i:i + len(b)] += b * (0.8 if k < 2 else 1)
    tail = rng.normal(0, 1, len(out)) * np.exp(-tt(0.25) * 22); out[int(0.022 * SR):] += 0.6 * tail[: len(out) - int(0.022 * SR)]
    return bp(out, 900, 2600)

def shaker(): return hp(rng.normal(0, 1, int(0.06 * SR)) * np.exp(-tt(0.06) * 70), 6000)

def whoosh(length=0.35):
    t = tt(length); env = np.sin(np.pi * t / length) ** 2
    return bp(rng.normal(0, 1, len(t)), 500, 3500) * env

def slide_whistle(t0, t1, f0, f1):
    t = tt(t1 - t0); f = np.geomspace(f0, f1, len(t)) * (1 + 0.012 * np.sin(2 * np.pi * 6 * t))
    env = np.minimum(1, t / 0.06) * np.minimum(1, (t[-1] - t) / 0.12)
    return (np.sin(2 * np.pi * np.cumsum(f) / SR) + 0.15 * np.sin(4 * np.pi * np.cumsum(f) / SR)) * env

def wood(pitch=1.0):
    t = tt(0.12); f = 820 * pitch
    return (np.sin(2*np.pi*f*t) + 0.4*np.sin(2*np.pi*f*2.7*t)) * np.exp(-t*55)

C4, D4, E4, F4, G4, A4, B4, C5, D5, E5, G5, A5, C6, E6, G6 = 60, 62, 64, 65, 67, 69, 71, 72, 74, 76, 79, 81, 84, 88, 91
# 0-2.5 s: thinking. Tick-tock under a few curious notes.
for k in range(5): add(wood(1.0 if k % 2 == 0 else 0.75), 0.25 + k * BEAT, 0.28, 0.3 if k % 2 else -0.3)
for t, n in ((0.5, E5), (1.0, D5), (1.5, F4 + 12), (2.0, E5)): add(marimba(n, 0.9, 0.5), t, 0.2, 0.15)
# 2.5 s: the idea. Bright ding with a sparkle.
add(glock(C6, 2.0), 2.5, 0.32, 0.1); add(glock(G6, 1.6), 2.56, 0.16, 0.35); add(glock(E6, 1.4), 2.62, 0.12, -0.3)
# 2.75-3.5 s: a quick run up into the drop
for k, n in enumerate([C5, D5, E5, G5, A5, C6]): add(marimba(n, 0.5), 2.94 + k * 0.09, 0.22 + 0.02 * k, -0.3 + 0.12 * k)
# 3.5-11.5 s: the groove, four bars
CH = {3.5: (48, [C5, E5, G5, E5, C5, G4, E5, G5]), 5.5: (43, [B4, D5, G5, D5, B4, G4, D5, G5]),
      7.5: (45, [C5, E5, A5, E5, C5, A4, E5, A5]), 9.5: (41, [A4, C5, F4 + 12, C5, A4, F4, C5, F4 + 12]),
      11.5: (43, [B4, D5, G5, D5, B4, D5, G5, B4 + 12])}
for t0, (bass, arp) in CH.items():
    for k, n in enumerate(arp):
        add(marimba(n, 0.9), t0 + k * BEAT / 2, 0.30 if k % 2 == 0 else 0.22, -0.35 if k % 2 else 0.35)
    add(marimba(bass, 1.6, 0.4), t0, 0.55); add(marimba(bass, 1.0, 0.4), t0 + 2 * BEAT, 0.4)
    for b in range(4):
        t = t0 + b * BEAT
        if b in (0, 2): add(kick(), t, 0.85)
        if b in (1, 3): add(clap(), t, 0.42, 0.1)
        add(shaker(), t + BEAT / 2, 0.10, -0.3); add(shaker(), t, 0.05, 0.3)
add(whoosh(0.5), 3.3, 0.22, 0.5)                                   # the wipe
for t in (4.18, 5.18): add(whoosh(), t, 0.30, -0.4)               # posts pushed away
add(slide_whistle(0, 1.5, 1300, 420), 6.15, 0.14, 0.15)           # down the rope
for t, n in ((8.1, E6), (9.1, G6)): add(glock(n, 0.8), t, 0.12, 0.3)   # chip taps
add(glock(C6, 0.8), 10.2, 0.14, -0.3)                              # dark mode
add(whoosh(0.3), 10.87, 0.18, 0.4)                                 # cut to the phone
# 13.5 s: end card
for k, n in enumerate([48, C4, E4, G4, C5, E5]): add(marimba(n, 2.6, 0.7), 13.5 + k * 0.035, 0.36, -0.3 + 0.12 * k)
add(glock(C6, 2.5), 13.58, 0.22, 0.2); add(glock(G5, 2.2), 13.95, 0.12, -0.2)

mix = np.stack([L, R], 1)
fade = np.ones(N); f0 = int(15.0 * SR); fade[f0:] = np.linspace(1, 0, N - f0) ** 1.5
fade[: int(0.01 * SR)] = np.linspace(0, 1, int(0.01 * SR))
mix *= fade[:, None]
mix = np.tanh(mix * 1.1) / np.tanh(1.1)
mix *= 0.89 / np.max(np.abs(mix))
wavfile.write("soundtrack.wav", SR, (mix * 32767).astype(np.int16))
print("ok")

How close is it to a real marimba?

Close, but not the same. Think of it as a good drawing of a marimba, not a photo.

What is the same:

• The notes. Each pitch comes from the same formula a tuned instrument uses. A C here is the same C a real marimba plays.
• The shape of each note. A soft knock, a strong main tone, quieter high tones, and a fade where the high parts go first. This shape is most of what your ear calls "marimba".

What is simpler:

• The overtones are a little off. Makers tune a real bar so its overtones are very close to 4 and 10 times the main note. The recipe uses 3.93 and 9.2 times. That is nearer to a plain wooden bar, so it sounds a bit like a xylophone or a kalimba.
• No tube. On a real marimba, the tube under each bar makes the main note louder and warmer, mostly on low notes. The recipe has no tube.
• Only three ingredients. A real bar has many more faint tones, and no two hits are the same. Here, each hit of a note is the same, except for the tiny random "tok".
• One fade speed. On a real marimba, low notes ring longer than high notes. In the recipe, all notes fade at about the same speed.

For a 16-second launch video, this was enough. To get closer, the recipe could use 4 and 10 times, add a small boost for the tube, and let low notes ring longer. Or a recording of a real marimba could replace it, but then a person played the music, not the numbers.

The chain of questions

1. We want music that hits each moment of the video. A downloaded song or a recording cannot do that from code.
2. What is sound? A long list of numbers that moves a speaker.
3. The simplest list is a smooth wave. It sounds like a beep.
4. Why? A wooden bar sings many notes at once, and it fades. So we add overtones and a fade.
5. Add a tiny "tok" for the mallet. Now it sounds like a marimba.
6. Put notes on a ticking clock, in happy chords, with drums. Now it is music.
7. Give each sound a time from the video. Now it fits.

Back to the redesign post.