Preamble

preamble
Historical Overview of Audio Spectral Modeling

Julius Smith
CCRMA, Stanford University
Music 421 Applications Lecture
January 7, 2014
Julius Smith
Music 421 Applications Lecture 1 / 53
Milestones in Audio Spectral Modeling

Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith
Fouriers theorem (1822)

Telharmonium (1898)
Voder (1920s)
Vocoder (1920s)
Hammond Organ (1930s)
Phase Vocoder (1966)
Digital Organ (1968)
Additive Synthesis (1969)
FM Brass Synthesis (1970)
Synclavier 8-bit FM/Additive synthesizer (1975)
FM singing voice (1978)
Sinusoidal Modeling (1985)
Sines + Noise (1988)
Sines + Noise + Transients (1988,1996,1998,2000)
Inverse FFT synthesis (1992)
Future Directions
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Telharmonium (1898)
Julius Smith
Telharmonium (Cahill 1898)

U.S. patent 580,035:
Art of and Apparatus for Generating and Distributing Music Electrically
Julius Smith
Telharmonium Rheotomes
Forerunner of the Hammond Organ Tone Wheels
Julius Smith
Telharmonium Rotor: Forerunner of Hammond Organ Tone

Wheels
Outline
Telharmonium
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
The Voder (1939)
Julius Smith
The Voder (Homer Dudley 1939 Worlds Fair)
[<
http://davidszondy.com/future/robot/voder.htm
Julius Smith
Voder Keyboard
http://www.acoustics.hut.fi/publications/files/theses/
lemmetty mst/chap2.html (from Klatt 1987)
Julius Smith
Voder Schematic
http://ptolemy.eecs.berkeley.edu/~eal/audio/voder.html
Julius Smith
Voder Demos
Outline
Video
Telharmonium
Voder
Audio
Voder Keyboard
Voder Schematic
Voder Demos
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
The Channel Vocoder (1928)

(Voice Coder)
Julius Smith
Vocoder Analysis & Resynthesis (Dudley 1928)

Analysis:
Outline
Telharmonium
Voder
Ten analog bandpass filters between 250 and 3000 Hz:

Bandpass rectifier lowpass filter amplitude envelope
Channel Vocoder
Vocoder Examples
Phase Vocoder
Additive Synthesis
Voiced/Unvoiced decision made

Fundamental frequency F0 measured for voiced case
FM Synthesis
Sinusoidal Modeling
Synthesis:
Future
Ten matching bandpass filters driven by a

buzz source (voiced), or
hiss source (unvoiced)
Bands were scaled by amplitude envelopes and summed
Said to have an unpleasant electrical accent
Julius Smith
Related Speech Models:
Vocoder Filter Bank Analysis/Resynthesis

magnitude, or
magnitude and
phase extraction
Outline
Telharmonium
Voder
A
x0 (t)
Channel Vocoder
Vocoder Examples
Phase Vocoder
Additive Synthesis
x
0 (t)
x(t)
FM Synthesis
x1 (t)
f
Data Compression,
Transmission,
Storage,
Manipulation,
Noise reduction, ...
x
1 (t)
x
(t)
Sinusoidal Modeling
Future
xN 1
x
N 1
Analysis
Julius Smith
Processing
Synthesis
Channel Vocoder Sound Examples

Outline
Telharmonium
Original
Voder
Channel Vocoder
Vocoder Examples
Phase Vocoder
Additive Synthesis
10 channels, sine carriers

10 channels, narrowband-noise carriers
FM Synthesis
Sinusoidal Modeling
Future
26 channels, sine carriers

26 channels, narrowband-noise carriers
26 channels, narrowband-noise carriers, channels reversed
Phase Vocoder: Identity system in absence of modifications

Julius Smith
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
The Phase Vocoder (1966)
Julius Smith
Phase Vocoder Analysis for Additive Synthesis (1976)

Analysis Model
Outline
Telharmonium
ak(t)
Out
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
k+k(t)
A
F
Sine Osc
Channel Vocoder
Phase Vocoder
ak
Channel Filter
Response
Voder
Synthesis Model
Early channel vocoder implementations (hardware) only

measured amplitude ak (t) (Dudley 1939)
Future
The phase vocoder (Flanagan and Golden 1966) added

phase tracking in each channel
Portnoff (1976) developed the FFT phase vocoder,

which replaced the heterodyne comb in computer-music
additive-synthesis analysis (James A. Moorer)
Inverse FFT synthesis (Rodet and Depalle 1992) gave faster

sinusoidal oscillator banks
Julius Smith
Amplitude and Frequency Envelopes

Outline
ak (t)
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
k (t) = k (t)
Julius Smith
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith
Classic Additive-Synthesis Analysis (Heterodyne Comb)

Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
Additive Analysis
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
John Grey 1975 CCRMA Tech. Reports 1 & 2

(CCRMA STANM reports available online)
Julius Smith
Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)

Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
Additive Analysis
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
John Grey 1975 CCRMA Tech. Reports 1 & 2

(CCRMA STANM reports available online)
Julius Smith
Classic Additive Synthesis Diagram (Computer Music, 1960s)

Outline
A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
Additive Analysis
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
noise
FIR
y(t) =
4
X
i=1
Julius Smith
Ai (t) sin
Z
i (t)dt + i (0)
0
Classic Additive-Synthesis Examples

Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
Additive Analysis
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Bb Clarinet
Eb Clarinet
Oboe
Bassoon
Tenor Saxophone
Trumpet
English Horn
French Horn
Flute
All of the above

Independently synthesized set
(Synthesized from original John Grey data)
Julius Smith
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Frequency Modulation Synthesis

(1973)
Julius Smith
Frequency Modulation (FM) Synthesis

Outline
FM synthesis is normally used as a spectral modeling technique.
Telharmonium
Voder
Channel Vocoder
Discovered and developed (1970s) by John M. Chowning

(CCRMA Founding Director)
Phase Vocoder
Additive Synthesis
Key paper: JAES 1973 (vol. 21, no. 7)
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling
Commercialized by Yamaha Corporation:

DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
Future
On the physical modeling front, synthesis of vibrating-string

waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
Julius Smith

Outline
Telharmonium
Voder
Channel Vocoder

Phase Vocoder
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling

Future

Julius Smith

Outline
Telharmonium
Voder
Channel Vocoder

Phase Vocoder
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling

Future

Julius Smith

Outline
Telharmonium
Voder
Channel Vocoder

Phase Vocoder
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling

Future

Julius Smith

Outline
Telharmonium
Voder
Channel Vocoder

Phase Vocoder
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling

Future

Julius Smith

Outline
Telharmonium
Voder
Channel Vocoder

Phase Vocoder
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling

Future

Julius Smith

Outline
Telharmonium
Voder
Channel Vocoder

Phase Vocoder
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling

Future

Julius Smith

Outline
Telharmonium
Voder
Channel Vocoder

Phase Vocoder
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling

Future

Julius Smith
FM Formula
Outline
x(t) = Ac sin[c t + c + Am sin(m t + m )]
Telharmonium
Voder
where
Channel Vocoder
Phase Vocoder
(Ac , c , c ) specify the carrier sinusoid
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
(Am , m , m ) specify the modulator sinusoid

Can also be called phase modulation
Sinusoidal Modeling
Future
Julius Smith
Simple FM Brass Patch (Chowning 1970)

Outline
Telharmonium
Jean-Claude Risset observation (19641969):

Brass bandwidth amplitude
Voder
fm = f0
Channel Vocoder
Phase Vocoder
g
A
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling
fc = f0
A
Future
Out
Julius Smith
FM Harmonic Amplitudes (Bessel Function of First Kind)

Outline
Harmonic number k , FM index :
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
0.5
Jk()
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling
Future
2
-0.5
0
4
5
10
15
20
25
30
10
Order k
Argument
Julius Smith
Frequency Modulation (FM) Examples

All examples by John Chowning unless otherwise noted:
Outline
Telharmonium
FM brass synthesis
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling
Future
Low Brass example

Dexter Morrils FM Trumpet
FM singing voice (1978)
Each formant synthesized using an FM operator pair
(two sinusoidal oscillators)
Chorus
Voices
Basso Profundo
Other early FM synthesis
Clicks and Drums
Big Bell
String Canon
Julius Smith
FM Voice
Outline
Telharmonium
FM voice synthesis can be viewed as compressed modeling of

spectral formants
Voder
Channel Vocoder
Carrier 1
Additive Synthesis
FM Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sinusoidal Modeling
Future
Julius Smith
Carrier 2
Carrier 3
Magnitude
Phase Vocoder
0
Frequency
Modulation Frequency (all three)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Sinusoidal Modeling Synthesis

(1988)
Julius Smith
Tracking Spectral Peaks in the Short-Time Fourier Transform

Outline
atan
Telharmonium
Voder
Channel Vocoder
s(t)
Phases
FFT
Frequencies
Phase Vocoder
Additive Synthesis
FM Synthesis
dB mag
Peak
tracking
Quadratic
Peak
Interpolation
window w(n)
Amplitudes
Sinusoidal Modeling
Sinusoidal Modeling
Spectral Trajectories
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
STFT peak tracking at CCRMA: mid-1980s (PARSHL program)

Motivated by vocoder analysis of piano tones
Influences: STFT (Allen and Rabiner 1977),
ADEC (1977), MAPLE (1979)
Independently developed for speech coding by McAulay and
Quatieri at Lincoln Labs (1985)
Future
Julius Smith
Example Spectral Trajectories

Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
Future
Julius Smith
Parametric Spectral Modeling

Outline
A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
white noise
u(t)
y(t) =
filter(t)
ht ( )
4
P
i=1
Ai (t) cos
hR
t
0
i (t)dt + i (0) + (ht u)(t)
Future
Julius Smith
Sines + Noise Sound Examples

Outline
Xavier Serra thesis demos (Sines + Noise signal modeling)
Telharmonium
Voder
Piano
Channel Vocoder
Phase Vocoder
Additive Synthesis
Original
Sinusoids alone
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
Residual after sinusoids removed

Sines + noise model
Voice
Original
Sinusoids
Residual
Synthesis
Future
Julius Smith
Musical Effects with Sines+Noise Models

Outline
Telharmonium
Voder
Piano Effects
Pitch downshift one octave
Channel Vocoder
Phase Vocoder
Additive Synthesis
Pitch flattened
Varying partial stretching
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
Voice Effects
Frequency-scale by 0.6
Frequency-scale by 0.4 and stretch partials
Variable time-scaling, deterministic to stochastic
Future
Julius Smith
Cross-Synthesis with Sines+Noise Models

Outline
Telharmonium
Voice modulator
Voder
Channel Vocoder
Phase Vocoder
Creaking ships mast carrier
Additive Synthesis
FM Synthesis
Voice-modulated creaking mast
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
Same with modified spectral envelopes
Future
Julius Smith
Sines + Transients Sound Examples

Outline
Telharmonium
In this technique, the sinusoidal sum is phase-matched at the

cross-over point only (with no cross-fade).
Voder
Channel Vocoder
Marimba
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
Original
Sinusoidal model
Original attack, followed by sinusoidal model
Piano
Original
Sinusoidal model
Original attack, followed by sinusoidal model
Future
Julius Smith
Multiresolution Sines + Noise + Transients

Outline
Why Model Transients Separately?
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
Sinusoids efficiently model spectral peaks over time

Filtered noise efficiently models spectral residual vs. t
Neither is good for abrupt transients in the waveform
Phase-matched oscillators are expensive
More efficient to switch to a transient model during transients
Need sinusoidal phase matching at the switching times
Transient models:
Original waveform slice (1988)

Wavelet expansion (Ali 1996)
MPEG-2 AAC (with short window) (Levine 1998)
Frequency-domain LPC
(time-domain amplitude envelope) (Verma 2000)
Future
Julius Smith
Time Scale Modification of Sines + Noise + Transients Models
original signal
Voder
sines +
noise
sines +
noise
transients
Telharmonium
transients
Outline
sines +
noise
Channel Vocoder
Phase Vocoder
time-scaled
signal
sines +
noise
sines +
noise
transients
FM Synthesis
transients
Additive Synthesis
sines +
noise
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
time
Time-Scale Modification (TSM) becomes well defined:
Transients are translated in time

Sinusoidal envelopes are scaled in time
Noise-filter envelopes also scaled in time
Dual of TSM is frequency scaling
Future
Julius Smith
Sines + Noise + Transients Time-Frequency Map

16
Outline
14
Telharmonium
Channel Vocoder
Phase Vocoder
Additive Synthesis
12
frequency [kHz]
Voder
10
8
6
FM Synthesis
Sinusoidal Modeling
2
0
50
100
150
200
250
50
100
150
200
250
amplitude
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
0.5
0
-0.5
-1
time [milliseconds]
(Levine 1998)
Future
Julius Smith
Outline
Telharmonium
transient
Corresponding Analysis Windows
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
low octave
Phase Vocoder
amplitude
Channel Vocoder
middle octave high octave
Voder
50
100
150
200
250
time [milliseconds]
Future
Julius Smith
Quasi-Constant-Q (Wavelet) Time-Frequency Map

16
Outline
14
Telharmonium
Channel Vocoder
Phase Vocoder
Additive Synthesis
12
frequency [kHz]
Voder
FM Synthesis
8
6
4
Sinusoidal Modeling
2
0
0x 104
50
100
150
200
250
50
100
150
200
250
amplitude
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
10
1
0
-1
-2
time [milliseconds]
Future
Julius Smith
Bark-Band Noise Modeling at High Frequencies (Levine 1998)

Outline
85
Telharmonium
80
Voder
75
Channel Vocoder
Phase Vocoder
70
Additive Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
65
magnitude [dB]
FM Synthesis
60
55
50
45
40
35
30
500
1000
1500
2000
2500
3000
3500
4000
4500
frequency [Hz]
Future
Julius Smith
Amplitude Envelope for One Noise Band

80
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Original Mag. [dB]
Outline
70
60
50
Additive Synthesis
40
FM Synthesis
50
100
50
100
150
200
250
300
350
150
200
250
300
350
Sinusoidal Modeling
80
75
LSA Mag. [dB]
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
70
65
60
55
50
45
time [milliseconds]
Future
Julius Smith
Sines + Noise + Transients Sound Examples

Outline
Telharmonium
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)

(http://ccrma.stanford.edu/~scottl/thesis.html)
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
It Takes Two by Rob Base & DJ E-Z Rock
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
Original
MPEG-AAC at 32 kbps
Sines+transients+noise at 32 kbps
Multiresolution sinusoids
Residual Bark-band noise
Transform-coded transients (AAC)
Bark-band noise above 5 kHz
Future
Julius Smith
Time Scale Modification using Sines + Noise + Transients

Outline
Telharmonium
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)

(http://ccrma.stanford.edu/~scottl/thesis.html)
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
Time-Scale Modification (pitch unchanged)
FM Synthesis
Sinusoidal Modeling
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5]
S+N+T Pitch Shifting (timing unchanged)
Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12]
Future
Julius Smith
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Summary and Future Prospects
Julius Smith
Spectral Modeling History Highlights
Bernoullis modal sums (1733)

Fouriers initial theorem (1822)
Telharmonium (1906)
Hammond organ (1930s)
Channel Vocoder (1939)
Phase Vocoder (1966)
FFT Phase Vocoder (1976)
Sinusoidal Modeling
(1977,1979,1985)
Sines+Noise (1989)
Sines+Transients (1989)
TF Reassignment (1995)
Sines+Noise+Transients
(1998)
Julius Smith
Perceptual audio coding:

Princen-Bradley filterbank
(1986)
K. Brandenburg thesis (1989)
Auditory masking usage
Dolby AC2
Musicam
ASPEC
MPEG-I,II,IV
(S+N+T parametric sounds)
Telharmonium (Cahill 1898)

U.S. patent 580,035:
Art of and Apparatus for Generating and Distributing Music Electrically
Julius Smith
Telharmonium Rheotomes
Forerunner of the Hammond Organ Tone Wheels
Julius Smith
Telharmonium Rotor: Forerunner of Hammond Organ Tone

Wheels
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
SM Summary
Telharmonium
Spec Future
Julius Smith
Future Prospects
Observations:
Sinusoidal modeling of sound is Unreasonably Effective

Basic auditory masking discards 90% information
Interesting neuroscience observation:
... most neurons in the primary auditory cortex A1 are silent
most of the time ...
(from Sparse Time-Frequency Representations, Gardner and
Magnesco, PNAS:103(16), April 2006)
What is the right psychospectral model for sound?

The cochlea of the ear is a real-time spectrum analyzer
How is the ears spectrogram represented at higher levels?
Julius Smith
Future Prospects
Observations:


Julius Smith
Future Prospects
Observations:


Julius Smith
Future Prospects
Observations:


Julius Smith
Future Prospects
Observations:


Julius Smith
Future Prospects
Observations:


Julius Smith
Future Prospects
Observations:


Julius Smith

Preamble

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Preamble

Hochgeladen von

Copyright:

Verfügbare Formate

preamble

Historical Overview of Audio Spectral Modeling

Music 421 Applications Lecture 1 / 53

Milestones in Audio Spectral Modeling

Fouriers theorem (1822)

Music 421 Applications Lecture 2 / 53

Music 421 Applications Lecture 3 / 53

Telharmonium (Cahill 1898)

Music 421 Applications Lecture 4 / 53

Music 421 Applications Lecture 5 / 53

Telharmonium Rotor: Forerunner of Hammond Organ Tone

Music 421 Applications Lecture 6 / 53

The Voder (1939)

Music 421 Applications Lecture 7 / 53

The Voder (Homer Dudley 1939 Worlds Fair)

Music 421 Applications Lecture 8 / 53

Music 421 Applications Lecture 9 / 53

Music 421 Applications Lecture 10 / 53

Music 421 Applications Lecture 11 / 53

The Channel Vocoder (1928)

Music 421 Applications Lecture 12 / 53

Vocoder Analysis & Resynthesis (Dudley 1928)

Ten analog bandpass filters between 250 and 3000 Hz:

Voiced/Unvoiced decision made

Ten matching bandpass filters driven by a

Related Speech Models:

Music 421 Applications Lecture 13 / 53

Vocoder Filter Bank Analysis/Resynthesis

Music 421 Applications Lecture 14 / 53

Channel Vocoder Sound Examples

10 channels, sine carriers

26 channels, sine carriers

Phase Vocoder: Identity system in absence of modifications

Music 421 Applications Lecture 15 / 53

The Phase Vocoder (1966)

Music 421 Applications Lecture 16 / 53

Phase Vocoder Analysis for Additive Synthesis (1976)

Early channel vocoder implementations (hardware) only

The phase vocoder (Flanagan and Golden 1966) added

Portnoff (1976) developed the FFT phase vocoder,

Inverse FFT synthesis (Rodet and Depalle 1992) gave faster

Music 421 Applications Lecture 17 / 53

Amplitude and Frequency Envelopes

Music 421 Applications Lecture 18 / 53

Additive Synthesis (1969)

Music 421 Applications Lecture 19 / 53

Classic Additive-Synthesis Analysis (Heterodyne Comb)

John Grey 1975 CCRMA Tech. Reports 1 & 2

Music 421 Applications Lecture 20 / 53

Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)

John Grey 1975 CCRMA Tech. Reports 1 & 2

Music 421 Applications Lecture 21 / 53

Classic Additive Synthesis Diagram (Computer Music, 1960s)

A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)

Music 421 Applications Lecture 22 / 53

Classic Additive-Synthesis Examples

All of the above

Music 421 Applications Lecture 23 / 53

Frequency Modulation Synthesis

Music 421 Applications Lecture 24 / 53

Frequency Modulation (FM) Synthesis

FM synthesis is normally used as a spectral modeling technique.

Discovered and developed (1970s) by John M. Chowning