INFT1004 Visual Programming

Lecture 6 Sound and Arrays (Guzdial & Ericson chapters 6 and 7)

INFT1004 - SEMESTER 1 - 2012

Week 1 Week 2 Week 3 Week 4 Recess Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Week 11 Week 12 Week 13 Mar 4 Mar 11 Mar 18 Mar 25 Apr 1 - Apr 7 Apr 8 Apr 15 Apr 22 Apr 29 May 6 May 13 May 20 May 27 Jun 3 Introduction Programs, Arrays and Iteration Working with x and y coordinates Selection

LECTURE TOPICS

Mid Semester Recess Period More Picture Techniques Sound and Arrays More Sound and Arrays Program Design and Strings Lists, Files and Modules Web, Representations, Steganography Turtles and Other Classes Revision and Look Ahead No formal classes - MUST be available normal & supplementary period
Assignment due 3:00 Tuesday May 21 Practical Test 2 in Lab class Practical Test 1 in Lab class

Mid Year Examination Period

Lecture Topics and Lab topics are the same for each week

Revision Indentation
Indentation how far across the page each statement starts is absolutely integral to Python programming Statements that have bodies (eg def, if, for) end with colons The body of a statement must be indented further than the statement itself

Revision Indentation

2 or 3 spaces is good; its not too much to type - with only one space it can be hard to see the indentation. After the body, indentation must go back to the same as the statement it was the body of In all other cases, a statement must be indented the same amount as the statement before it

Revision Indentation

body of the function

Revision Indentation

body of the for loop

Revision Indentation

body of the if statement

Revision Indentation

Indentation is not just pretty, not just arbitrary It is what tells Python (and readers) the structure of the program. For the sake of readers, it helps if the comments are indented the same as their surrounding statements

Revision Indentation
Warning be very careful with the return statement when you are using loops if it accidentally gets to be part of the loop it will stop the loop on the first time through!

10

Revision Arrays
An array is a collection of items all have the same name but each has a different index, an identifying number

11

Revision Arrays
An array is a collection of items all have the same name but each has a different index, an identifying number eg. An array called height
height[2] height[0] if i has the value 4 this is height[i]

height index

1.6 1.7 2.1 1.9 1.7 . . .

4
12

Revision Arrays
An array is a collection of items all have the same name but each has a different index, an identifying number eg. An array called pixels
pixels = getPixels(picture) pixels[3]

pixels index
0 1 2 3 4

. . .
13

Revision - Functions
Programs are made up of one of more functions Functions can be called from the command area, but, more importantly, from other functions defined in the program If we want a function to take arguments, we include corresponding parameters when defining it Whenever we find the same piece of code appearing several times in a program, we should extract it, define it as a function, and replace the multiple occurrences with multiple calls to the function If there are variations between the occurrences, we use parameters and arguments to deal with the variations

14

Revision - getPixels() vs getPixel()

getPixels(pic), gets all of the pixels from a picture, in an array We can use for to loop through that array, thus processing all the pixels in the picture for px in getPixels(picture):

15

Revision - getPixels() vs getPixel()

getPixels(pic), gets all of the pixels from a picture, in an array We can use for to loop through that array, thus processing all the pixels in the picture for px in getPixels(picture): getPixel(pic, x, y) gets the single pixel at location (x, y) If we want to loop through a specific range of the pixels, not the whole lot, we must loop x and y through the required range, getting just one pixel at a time for x in range(startX, endX): for y in range(startY, endY): pix = getPixel(picture, x, y)

16

Revision Object Types

JES and Python know what types various objects are This determines what can be done with them The programmer also needs to know the types of all the objects in the program, or the programmer will do silly things with them Some of the object types weve used are file, picture, pixel, colour What type of object does getRed() apply to? You need to know these things if youre to successfully do the sort of programming weve been covering

17

Caution - Choosing names

Words in magenta are words that have some special meaning in JES You choose a name for a variable. When you type it, JES makes it magenta. Then you choose a bad name for your variable. Choose a different name! If JES lets you use one as a variable, you will change its meaning! This is something to avoid.

18

Caution - Choosing names

For example, you type red = 23 Later, when you type setColor(pixel, red) it set the pixels colour to your value of red(23), not JESs value (the color red)

19

JES Options
Some useful things you can do on the Edit/Options menu

1.Change Mode to Expert gives more help on errors, which might or might not be more helpful 2.Auto save on load means you dont have to agree to save every time you load the program

20

JES Options
Some useful things you can do on the Edit/Options menu

3. Consider turning off logging 4. Consider turning off saving backup copies 5. Look at the modulo option for pixel colour values; we discussed this in an earlier lecture
21

Working with Pictures

We see a picture as continuous patches of colour. But a digitised picture is broken into individual pixels, each representing the colour value at one small point

22

Working with Pictures

When the pixels are small enough and close enough together, it looks the same to us.

23

Working with Sound

Likewise, we hear a sound as a continuous stream

24

Working with Sound

Likewise, we hear a sound as a continuous stream

But a digitised sound is broken into individual samples, each representing the sound frequency at one small instant in time

When the samples are small enough and close enough together, it sounds the same to us

25

Working with Sound

index 0
1 2 3 4

...

samples

26

Features of Sound
Amplitude

cycles / second frequency

Amplitude, the height of the wave, relates to loudness Frequency, number of cycles per second, relates to pitch

27

Features of Sound

28

Features of Sound
Overtones are additional frequencies that turn pure sound into rich sound Real waves have different shapes (sine, square, triangle, indeterminate) Very few real sounds are pure in pitch or wave shape

29

Features of Sound
Sounds also have a quality called Timbre This is a quality of a sound that make them sound different (even though they have the same amplitude and frequency (e.g. flute compared to a violin, different voices)

30

Features of Sound
Loudness and pitch both relate logarithmically to amplitude and frequency:
Doubling the amplitude increases loudness by same amount Doubling the frequency increases pitch by same amount

31

Features of Sound
Loudness and pitch both relate logarithmically to amplitude and frequency:
Doubling the amplitude increases loudness by same amount Doubling the frequency increases pitch by same amount

Actually its more complex than this - we respond differently to low and high frequencies at low amplitudes (loudness button) - and responses to different frequencies change with age.

32

Sampling rates and Nyquist

Nyquist theorem: for a reasonable recording, sample at twice the rate of the highest frequency in the sample Human speech goes to 4,000Hz (cycles per second), so for a good speech recording we need to sample at 8,000Hz (8000 samples per second) Human hearing goes to 22,000Hz, so for a good music recording we need to sample at 44,000Hz (44,000 samples per second)

33

Binary numbers
Aside computers use binary, off/on representing 0/1

off 0 on 1

34

Decimal and Binary numbers

In the decimal system there are 10 digits, from 0 to 10 1 Column n (from the right) gives the number of times 10n1 is found in the number
column 3

40358 = 4 x 104 + 0 x 103 + 3 x 102 + 5 x 101 + 8 x 100

0 0 0 0 0 0

0 4 0 3 5 8

Decimal
base 10

10 4

10 0

10

10 5

10 8

0
35

Decimal and Binary numbers

In the binary system there are 2 digits, 0 and 1 Column n (from the right) gives the number of times 2n1 is found in the number
column 4

11011 = 1 x 24 + 1 x 23 + 0 x 22 + 0 x 21 + 1 x 20

0 0 0 0 0 0

0 1 1 0 0 1

Binary
base 2
(25 in base 10)

2 1

2 0

2 1

2 1

36

Decimal and Binary numbers

For sound we normally use 2 bytes (16 bits) to store the sound amplitude. One bit stores positive or negative The remaining 15 bits allows for numbers between 0 and 215 Possible values -32, 768 ----32, 767

37

JES objects pictures and sound

Picture file makePicture() Sound file - makeSound()

38

JES objects pictures and sound

Picture file makePicture() picture - explore() picture - show() Sound file - makeSound() sound - explore() sound - play()

39

JES objects pictures and sound

Picture file makePicture() picture - explore() picture - show() pixel Sound file - makeSound() sound - explore() sound - play() sample

40

JES objects pictures and sound

Picture file makePicture() picture - explore() picture - show() pixel colour Sound file - makeSound() sound - explore() sound - play() sample value (an integer)

41

JES objects pictures and sound

Picture file makePicture() picture - explore() picture - show() pixel colour pixels (an array) Sound file - makeSound() sound - explore() sound - play() sample value (an integer) samples (an array)
42

Things to do with a JES sound

print(snd) getSamplingRate(snd) # number samples/sec getLength(snd) # number of sample objects getSamples(snd) # an array of sample objects

index 0

...
43

Recording your own Sounds

JES works with wav files wav files from different software aren't all the same MediaTools, which comes with the media computation package, is fairly basic, but is designed to produce wav files with all the same settings that JES uses That is, sounds recorded with MediaTools should all work nicely in JES If you haven't yet set up MediaTools, remember that the instructions were provided much earlier in the course (you can download from blackboard/resources)
44

Things to do with a JES sound

print(snd) getSamplingRate(snd) # number samples/sec getLength(snd) # number of sample objects getSamples(snd) # an array of sample objects getSampleValueAt(snd, n) setSampleValueAt(snd, n, newVal) getSampleObjectAt(snd, n)

index 0

...
45

Things to do with a sample object

getSampleValue(samp) setSampleValue(samp, newVal)

...
46

More things to do with a JES sound

writeSoundTo(snd, file) play(snd) playAtRate(snd, rate) playAtRateDur(snd, rate, numSamples) blockingPlay(snd) explore(snd)

47

Processing Samples in an Array

Every element of a sample array is a sample object. Once youve got the array of samples you can access samples ..
sndArray = getSamples(snd) value = sndArray[i]

This might be easier than processing samples like this..

value = getSampleValueObjectAt(snd, i)

48

Increasing & decreasing amplitude

Amplitude

Once weve made a sound from a file, its easy to adjust the amplitude (and thus the volume)

49

Increasing & decreasing amplitude

Note that if we adjust too far, we get clipping the biggest (positive and negative) values are chopped off And we cant reverse it even if we reduce the amplitude the sound remains clipped.
50

Normalising sound
Normalising sound means increasing the amplitude just as far as we can without clipping

51

Normalising sound
Normalising sound means increasing the amplitude just as far as we can without clipping

1. We need to find the biggest amplitude (positive or negative) 2. Increase that up to the maximum possible (32767) 3. Multiply every other sample by the same amount

52

Normalising sound
Read the book and the code (lecture6sample.py) very carefully to understand how it works Why doesnt it look normalised? Because each sample is the average of many samples How would you find where it does reach the top or bottom?

53

Working in specific ranges

Just as with pictures, we can work in particular ranges of the array rather than the whole sound Just as with pictures, we use explore() to find the start and end of the range were interested in (drag to make a selection, then play it)

54

Working in specific ranges

Lets say we have a sound in which we want to increase the volume of three particular parts First we use explore() to find their start and finish values In a sound of 180,000 samples, we might want to boost the amplitude between 32,000 and 46,000 62,000 and 76,000 130,000 and 148,000
Series1 32,000 46,000 62,000 76,000 130,000 148,000

20000

40000

60000

80000

100000

120000

140000

160000

180000

55

Increase amplitude in a range

We write a function that increases the amplitude by a specified amount in a single range (note use of array)
def adjustAmp(aSound, multiplier, startIndex, endIndex): # Adjust the amplitude of aSound by multipler in the samples # from startIndex to endIndex # First we need the array samples = getSamples(aSound) # Now we can select the samples that need adjusting and adjust them for index in range(startIndex, endIndex): setSampleValue(samples[index], multiplier * getSampleValue(samples[index]))

56

Increase amplitude in a range

def adjustAmp(aSound, multiplier, startIndex, endIndex):

Then we write another function that calls that function three times
def boostThreeBits(aSound, mult, startA, endA, startB, endB, startC, endC): # A highly specific function to boost the amplitude by multiplier just in # three specific ranges adjustAmp(aSound, mult, startA, endA) adjustAmp(aSound, mult, startB, endB) adjustAmp(aSound, mult, startC, endC)

57

Increase amplitude in a range

def adjustAmp(aSound, multiplier, startIndex, endIndex): def boostThreeBits(aSound, mult, startA, endA, startB, endB, startC, endC):

And call the second one from the command area

boostThreeBits(sound,32,32000,46000, 62000,76000,130000,148000)

58

Assignment
Assignment has been posted. Please start now! I will go through it now. We will stop to look more closely at a few of the concepts described in this assignment 1. Thresholding grey-scale images to create binary images. 2. Using masks to process images (e.g. Sobel)
59

Thresholding a Grey-scale image

Remember grey-scale images have the same value on the blue, red and green channels. e.g. red=100, green = 100, blue = 100 To create a binary image (eg. black and white) you need to ensure that the only possible colours are black and white black red=0, green = 0, blue = 0 white red=255, green = 255, blue = 255

60

Thresholding a Grey-scale image

How do you decide which grey pixels to make white and which ones to make black You choose a threshold value eg. 120 Anything above 120 you make white and anything else black (remember you only need to check one value of a greyscale image as the red, green and blue are all the same) You dont have to make things black and white you could make it blue and white or something)
61

Using Masks with Images

Images are 2D arrays of pixels

p1 p4 p7

p2 p5 p8

p3 p6 p9

62

Using Masks with Images

p1 p2 p3 p4 p5 p6 p7 p8 p9

Masks usually 3 x 3 are placed over each pixel in the image one at a time.

The pixel under the centre of the mask is the one being processed.
63

Using Masks with Images

p1 p2 p3 p4 p5 p6 p7 p8 p9

64

Using Masks with Images

p1 p2 p3 p4 p5 p6 p7 p8 p9

65

Using Masks with Images

p1 p2 p3 p4 p5 p6 p7 p8 p9

66

Using Masks with Images

p1 p2 p3 p4 p5 p6 p7 p8 p9

67

Using Masks with Images

p1 p2 p3 p4 p5 p6 p7 p8 p9

68

Using Masks with Images

p1 p2 p3 p4 p5 p6 p7 p8 p9

69

Using Masks with Images

p1 p2 p3 p4 p5 p6 p7 p8 p9

etc, etc,

70

Using Masks with Images

Notice when you are finished only the inside pixels are processed. (You can just leave the outside ones as they are or set them to black)

71

Using Masks with Images

A mask for calculating vertical edges -1 -2 -1 0 0 0 1 2 1 p1 p4 p7 p2 p5 p8 p3 p6 p9
grey value at p6 you can use any channel eg. getBlue(p6)

weights for the vertical edge mask

Calculate a new grey value at p5 by using the weights and the original pixel grey values at (p1,p2,p3,p4,p5,p6,p7,p8,p9)

72

Using Masks with Images

A mask for calculating vertical edges -1 -2 -1 0 0 0 1 2 1 p1 p4 p7 p2 p5 p8 p3 p6 p9
Calculate a new grey value at p5 by using the weights and the original pixel grey values at (p1,p2,p3,p4,p5,p6,p7,p8,p9)

weights for the vertical edge mask

greyVertical = (p1 x -1) + (p2 x 0) + (p3 x 1) + (p4 x -2) + (p5 * 0) + (p6 x 2) + (p7 x -1) + (p8 x 0) + (p9 x 1) (you need to take the absolute value to make sure it stays positive)
73

Using Masks with Images

Just be careful that you dont change the grey value of the original image as you need to calculate each new pixel values using the grey-values from the original image.

greyVertical = (p1 x -1) + (p2 x 0) + (p3 x 1) + (p4 x -2) + (p5 * 0) + (p6 x 2) + (p7 x -1) + (p8 x 0) + (p9 x 1) (Dont change the original p5)
74

Using Masks with Images

A mask for calculating horizontal edges 1 0 -1 2 0 -2 1 0 -1 p1 p4 p7 p2 p5 p8 p3 p6 p9
Calculate a new greyscale value at p5 using the weights and the pixel grey value.
grey-value at p6 you can use any channel eg. getBlue(p6)

weights for the horizontal edge mask

greyHorizontal = (p1 x 1) + (p2 x 2) + (p3 x 1) + (p4 x 0) + (p5 * 0) + (p6 x 0) + (p7 x -1) + (p8 x -2) + (p9 x -1) (you need to take the absolute value to make sure it stays positive)
75

Using Masks with Images

A mask for calculating horizontal edges 1 0 -1 2 0 -2 1 0 -1 p1 p4 p7 p2 p5 p8 p3 p6 p9
Calculate a new greyscale value at p5 using the weights and the pixel grey value.
grey-value at p6 you can use any channel eg. getBlue(p6)

weights for the horizontal edge mask

greyHorizontal = (p1 x 1) + (p2 x 2) + (p3 x 1) + (p4 x 0) + (p5 * 0) + (p6 x 0) + (p7 x -1) + (p8 x -2) + (p9 x -1) (you need to take the absolute value to make sure it stays positive)
76

Sobel Mask
Combines both the vertical and horizontal calculation

greyHorizontal = (p1 x 1) + (p2 x 2) + (p3 x 1) + (p4 x 0) + (p5 * 0) + (p6 x 0) + (p7 x -1) + (p8 x -2) + (p9 x -1)

greyVertical = (p1 x -1) + (p2 x 0) + (p3 x 1) + (p4 x -2) + (p5 * 0) + (p6 x 2) + (p7 x -1) + (p8 x 0) + (p9 x 1)

sobelGrey =

greyHorizontal + greyVertical
Make sure gVertical is positive by taking absolute value of the result

Make sure gVertical is positive by taking absolute value of this result

77

Practical Test
You can pick up the mark sheets at the end of the lecture. I will go through the mark sheet now. If we have time I will also do the exam in front of your eyes.

78