In the early 1990s, I taught a Macintosh computer to play player piano
rolls.  The Macintosh would scan the rolls in using a TV camera, then
output the appropriate MIDI commands to a piano module in real time.

I've seen references in the MMD to earlier discussions regarding scan-
ning piano rolls using NTSC video.  I'm afraid I haven't read those
issues, so I apologize if this ground has already been covered.

The setup is: I returned to Worcester Polytechnic Institute (WPI)
nights to finish my BS in Computer Science (CS) in the early 1990s.
All I was missing was a thesis project.  I decided to do something
involving digital image processing.  After learning a survey of the
field, I needed a project to demonstrate this knowledge.  I decided to
do something fun: teach a Macintosh to play player piano rolls
optically.

The scheme I used was as follows:

(1) A player piano roll would scroll by a TV camera.  Of course, if you
already have a player piano, why do you need this project?  Because
this was the most convenient way to scroll the roll past the camera in
a fixture at a fixed rate.  Secondly, I used a camcorder, so I could
make a recording.  Then I didn't need to tie my computer to a piano to
write and debug the program.  I could just play the tape back in the
comfort of my own home.

(2) The video from the camera was input via a RasterOps video board
into the Macintosh computer.  Since the board also ran the 13-inch
monitor, it acquired and displayed full screen video (640 x 480
pixels).  The image was actually held in the video ram on the video
board.

(3) Selected portions of the image were transferred from the video
board to the main computer memory (ram).

(4) Pixels in the image corresponding to the tracker bar holes were
checked for the presence or absence of paper.

(5) When a hole was first found (black pixels in the image), a MIDI
command was sent to start playing the corresponding note.  When a hole
turned back to paper (white pixels in the image), another MIDI command
was sent to stop playing the corresponding note.

How some problems were solved:

(1) One problem is that just one screen image is 1.2 megabytes.

Standard American NTSC video (TV) is effectively 30 frames per second,
or 35 Mb/second.  That's a lot of memory to transfer from the video
board to main memory, and you don't need all of it.  Therefore, the
operator went through a couple setup steps:

First, a single image was displayed of the tracker bar with no roll
installed.  The user would zoom in 8x, then click on the pixels that
represented one tracker bar hole.

Second, the user would draw a line from the leftmost tracker bar hole
to the rightmost tracker bar hole.

This let the computer do two things:

(a) Find all the tracker bar holes along that line, given that it now
knew what one was supposed to look like.  (b) Only transfer the pixels
in the tracker bar hole area (about 30Kb per frame) from the video
board to main memory, instead of the whole frame (1.2Mb).

(2) The value when a hole is not present is not constant.

That is, roll paper is not "white" or even consistent in color.
Therefore, the first five seconds of playing a roll measured the
maximum value of "white" on the roll leader to avoid mistakenly finding
a hole when one was not present.

(3) Ampico tracker bars move!

Instead of moving the paper, the tracker bar moves left and right.
Therefore, the program had to "track the tracker bar".  The way it did
this was, each time it read an image in, after it read the holes, it
checked the "response" if it made believe the tracker bar had moved one
hole to the left or one hole to the right.

Again, due to the vagaries of NTSC video, this tracker bar tracking had
to be damped or else it would march off, right or left, into the weeds.
First, the "votes" of each hole's response to "move left", "move right",
or "not move" were tallied for one line.  Second, the sum of two
different votes to move right or left had to agree before the program
would consider it time to move one to the left or right in its tracking.

Things learned along the way:

(1) NTSC video was limited.  A single tracker bar hole was only about
3 x 5 pixels.  Not very large, eh?

(2) Because of the above, proper lighting was critical !  Despite this,
a hole was not pure black and the paper was not pure white -- not by a
long shot !

(3) Words on word rolls were mistaken for holes, resulting usually
in some "plinking" in the high end.  I did not have time during the
project to solve this problem, but it probably was not that difficult.

(4) MIDI commands also have a volume level.  It turns out the volume
levels from -128 to +128 do not translate to the same volume range from
one MIDI unit to the next.  There is not (was not?) a standard on this.

(5) Ampico expression was also modeled fairly easily.  As documented
elsewhere, the various expression holes represent distinct volume
levels or fast/slow crescendos/decrescendos.  It was easy to model in
software, counting up or down over the proper time period, or setting
the MIDI volume to a distinct level.  But it left me with two
questions:

(A) On various rolls, there would be periods of silence (no note holes)
for the entire width of the roll or just the left or right half of the
roll.  Yet, there would be all sorts of expression holes being punched
on and off.  Why?  What was it doing?

(B) Wayne Stahnke talked about modeling the Ampico expression system
for his "Window in Time" series of CDs.  Did he actually model in
software all the valves and pneumatics of the Ampico expression system,
or did he just do what I did?  If it was the former, why?

 [ Wayne used the model which you used, simply because its emulation
 [ is well within the variability of the pneumatic action.  In other
 [ words, it would be impossible to prove any better scheme!  -- Robbie

When I've told this story in the past, I often receive three reactions:

(1) Why did you use the TV camera?  Why didn't you just install some
photocells behind the tracker bar?

I did propose something like this to my professors at an early point.
However, in their eyes, this would have been a hardware project and my
objective was to execute a software project (the degree was Computer
Science, not Electrical Engineering.)

(2) Why don't you zoom the camera in so you get more pixels per tracker
bar hole?

Because then the left and right ends of the tracker bar don't fit in
the picture.

(3) What happens if you take a picture of something other than a piano
roll?

As it so happens, my computer was in my bedroom back then.  One day,
while working on this project, I hit the "record" button in the middle
of a piano roll video recording instead of the "play" button.  The
result was that, when demonstrated to friends, you would see the music
roll going by on the computer monitor while the music came out the
speakers from the attached MIDI piano unit.

When the video suddenly changed to a shot of my bed, the music changed
to something akin to a train wreck, as played on a piano.  (One friend
near fell on the floor, laughing so hard, saying like it was a scene
out of "Twin Peaks".)

Hope you found this interesting.

C. Jim Cook
Bolton, Mass.