Audio-Visual Sampling Synthesizer
In conjunction with the "Given Enough Eyeballs" show at Esther Klein, the PIFAS Acoustic Research Department is holding open meetings toward the construction of an audio-visual sampling synthesizer. This open-source project will have its source code released here along with a brief tutorial on its installation; it will rely heavily on other open source software and libraries including ffmpeg, vlc, wxPython, and pure data. The synthesizer is being developed in GNU/Linux using Ubuntu 7.04, but it should be able to run anywhere where the aforementioned libraries can be installed.

First Meeting

2-14, held a brief meeting explaining the basics of video encoding, the inspiration behind the project, and presenting what has been done so far. The script attached depends on wxPython and the vlc mediacontrol bindings for python - to get these, vlc must be compiled from source with the following flag included in the ./configure command:
--enable-mediacontrol-python-bindings

The script takes advantage of vlc's difficulty in seeking through wmv files, and presents these artifacts proudly.

At the top of the script you'll find the "dir" variable, which should be the full path to the wmv files you want to use for this, and a few other useful flags (all set either to 0 or 1) - FullScreen, UseTimer, which determines whether the script's internal clock should be used to seek through videos and skip to the next video, and SizeConst, which is a constant roughly representing the number of bytes per milisecond the video uses for encoding. For videos encoded in the 1000kbps range I've been using 75 as SizeConst - all this is used for is to determine how deep into the video we can skip. A smaller value will only seek through the earlier parts of the video, and a value too large will run the risk of overseeking, which does nothing.

Forthcoming will be more detailed instructions on how to install all dependencies, including ffmpeg, vlc w/ python bindings, and wxPython, as well as some other goodies to be used in the audio side of the program. For now, the user interface is fairly simple - hitting tab will randomly seek through the current video, piling up artifacts; hitting space will skip to the next video, and the escape key exits out of the program (don't forget this one). I'm looking into interfacing with the wiimote for controlling these functions & more but it remains to be seen if I can (1) come up with a sensible way to interpret wiimote data and (2) figure out how to keep the wiimote in the gallery without having it stolen.

First Meeting - Images
  • Screenshot
    Screenshot
First Meeting - Assets
  • synth1.py - First simple version of the video script
Installation Instructions

There are a number of dependencies which both the python script (the video portion) and the PD patch require. This is a guide to installing all of them on Ubuntu - this has been tested on multiple computers with Ubuntu 7.04 (Feisty Fawn).

Ubuntu 7.04 is available for free on the web as a downloadable disk image, which can then be burned, booted, and installed from. If you are having trouble booting from the CD, you may have to edit your BIOS to change the boot order to force booting from a CD before the internal hard drive.
Get Ubuntu here: Feisty Fawn

Once Ubuntu is installed and you have an internet connection, you can begin installing the dependencies. We can use the command apt-get for a lot of this. First, get the dependencies needed to build on Ubuntu.

sudo apt-get install build-essential checkinstall subversion libavcodec-dev libpostproc-dev python2.5 python2.5-dev

Now, we're going to build ffmpeg itself - we'll need it to convert our video files to the wmv format.

mkdir ~/src

cd ~/src

sudo apt-get build-dep ffmpeg

sudo apt-get install liblame-dev libfaad2-dev libfaac-dev libxvidcore4-dev liba52-0.7.4 liba52-0.7.4-dev libdts-dev

apt-get source ffmpeg

cd ~/src/ffmpeg-*/

#In 7.04 (these should be all on one line):

./configure --enable-gpl --enable-pp --enable-vorbis --enable-libogg --enable-a52 --enable-dts --enable-dc1394 --enable-libgsm --disable-debug --enable-mp3lame --enable-faad --enable-faac --enable-xvid --prefix=/usr

#Or, if you're using 7.10 (Gutsy Gibbon):

./configure --enable-gpl --enable-pp --enable-libvorbis --enable-libogg --enable-liba52 --enable-libdts --enable-dc1394 --enable-libgsm --disable-debug --enable-libmp3lame --enable-libfaad --enable-libfaac --enable-xvid --enable-shared --prefix=/usr

make

sudo make install

export LD_LIBRARY_PATH=/usr/local/lib/

#FFMPEG is installed! Now check what version we have:

ffmpeg -version

Next, we're going to build wxPython, which are the python bindings for wxWidgets. This will make it easy for us to build a python script with keyboard input, full screen access, and build a simple GUI if we choose to.

sudo apt-get build-dep python-opengl python-wxgtk2.8-dbg python-wxgtk2.8
sudo apt-get install python-opengl python-wxgtk2.8-dbg python-wxgtk2.8

Now, we're going to install the VLC library, which will allow us to decode the video files we'll be using and play them back. We're installing the library with the python bindings, so we'll be able to call it from our script

sudo apt-get install libmpeg2-4-dev libmad0-dev libasound2-dev libwxgtk2.8-0 libwxgtk2.8-dev

cd ~/src

wget

http://download.videolan.org/pub/videolan/vlc/0.8.6d/vlc-0.8.6d.tar.gz

tar -xvzf vlc-0.8.6d.tar.gz

cd vlc-0.8.6d/

#(these should be all on one line):

./configure --enable-x11 --enable-xvideo --disable-gtk --enable-sdl --enable-ffmpeg-faac --enable-mad --enable-libdvbpsi --enable-libmpeg2 --enable-dvdnav --enable-vorbis --enable-ogg --enable-theora --enable-faac --enable-mkv --enable-freetype --enable-fribidi --enable-speex --enable-flac --enable-caca --enable-skins --enable-skins2 --enable-alsa --disable-kde --enable-qt --enable-wxwidgets --enable-ncurses --enable-release --disable-a52 --enable-libmpeg2 --disable-dbus --disable-hal --enable-mediacontrol-python-bindings

make

sudo make install

Now we have everything we need for the video script. Finally, we'll install Pure Data. The PD extended package can be downloaded directly from sourceforge.

Now, to convert videos to a format we we can use with our program, we can use the following command:

ffmpeg -i in.avi -g 600 -b 5000 -y -an -ss 100 -t 60 out.wmv

This will take a video called in.avi and converts it to a wmv, which we'll use in our script. The flags we use are -n, to suppress audio output, -g 600, to limit keyframes to every 600 frames, -b 5000, to set the bitrate to 5000k if possible, -y to overwrite existing files, -ss to skip to 100 seconds into the video, and -t 60 to only convert 60 seconds. You can then place the output in whatever directory you specify as the wmv dir in the python script.

Departmental Associations
Project Members