Frontier Nerds: An ITP Blog

André Callot’s pithy survey of video art. Via Ben Kolak.

For the com lab final, I decided to revisit the blog-improvement assignment — might have been low hanging fruit given my prior experience on these fronts, but I wanted to try something new: tapping real-time sensor data to determine the aesthetic properties of my website.

I spend such an inordinate amount of time in front of screens, that bringing in a bit of the outside world onto said screens seemed like a nice thing to do (if just a bit sad).

So I decided to set up a camera pointed at the sky, 24 / 7, and use the average sky color to set the background on this website.

First, I needed a camera. I scrounged up an old webcam, and put it in a small Tupperware container since it was going to have to brave the elements on my windowsill.

Next up was how to get data from the camera to the web. I just found out about a website called Pachube (confoundingly pronounced “patch bay”) which provides a platform for the hosting and exchange of small bits of sensor data — ideal for this application. So I set up an input feed on Pachube to receive and store the sky color values.

I whipped up a Processing application to read from an old webcam, average the pixel values, and then send the value up to Pachube every minute. (The EEML Library for Processing made interfacing with Pachube completely painless.)

Getting the data out of Pachube and into my Drupal Blog needed another piece of glue code: the Pachube PHP Library. I created a Drupal module to wrap up the library, and another small module to query their servers, grab the current color value, pass it on to a bit of JavaScript, which then set the background color via jQuery.

Right now the background only updates on refresh (since the data changes quite slowly) but it wouldn’t be much more work to set up a timer and some AJAX to fetch the latest value at a regular interval.

With the software working…

And the camera situated…

I was ready for the data to start coming in. My webcam had other ideas, though. It turns out that even with auto-exposure and auto-gain turned on, the sensor doesn’t have enough dynamic range to capture the sky with any kind of accuracy. The sky is just too bright during the day, and too dark at night. There are a few interesting color values at dawn and dusk, but otherwise it’s either solid black or solid white.

Here’s a look at tonight’s sunset (the top of the gradient is about 6:00 PM ET, the bottom is 2:00 PM ET (the sun officially set at 4:30). There’s about an hour of meaningful color data between the blown-out day and solid-black night. I’m investigating more sensitive camera options, in hopes of getting meaningful values for more of the day.

Here’s the Processing code. It’s a bit verbose, since I entertained a few what-if scenarios and drew out some portions for legibility’s sake. The low frame rate helps keep CPU usage (and therefore CPU temperature and fan noise) within the realm of rationality.

// Sky color logger.

// Takes a photograph of the sky every second, then finds and stores the average color value.
// Each minute, average the one-second averages to find the average sky color over the last minute.
// Upload the result to Pachube (http://www.pachube.com/) as a hexadecimal color value.
// Built with the EEML Library for Processing, available at http://www.eeml.org/library/
import processing.video.*;
import eeml.*;

DataOut dataOut;
Capture cam;

int pixelCount;
color averageColor;
color minuteAverageColor;
int lastMinute;

int minuteAverageRed = 0;
int minuteAverageGreen = 0;
int minuteAverageBlue = 0;
int averageRed = 0;
int averageGreen = 0;
int averageBlue = 0;
int sampleCount = 0;

int[] averages; // Stores recent averages.

void setup() {
 size(640, 480);
 noStroke();
 frameRate(2);

 // Set up communication with Pachube.
 String feedURL = "http://www.pachube.com/api/feeds/3929.xml";
 String apiKey = "YOUR PACHUBE API KEY HERE";
 dataOut = new DataOut(this, feedURL, apiKey);
 dataOut.addData(0, "sky,color");

 // Set up the camera.
 String[] devices = Capture.list();
 println(devices);
 cam = new Capture(this, 320, 240, devices[1]);
 cam.frameRate(1); // Read 1 frame per second.
 pixelCount = cam.width * cam.height;

 // Take note of the time. We will upload once per minute.
 lastMinute = minute();

 // Store recent averages in an array.
 averages = new int[240];

 // Fill the averages array with white to start
 for(int i = 0; i < averages.length; i++) averages[i] = 255;
}

void draw() {
 background(0);

 // Show the live camera feed in the top left.
 fill(0);
 image(cam, 0, 0);

 // Show the latest average color in the top right.
 fill(averageColor);
 rect(320, 0, 320, 240);

 // Show the last minute average in the bottom left.
 // This is what was actually sent to Pachube.
 fill(minuteAverageColor);
 rect(0, 240, 320, 240);

 // Draw the recent averages as horizontal lines in the bottom right.
 // The latest values are pushed onto the top of the stack.
 for(int i = 0; i < averages.length; i++) {
   stroke(averages[i]);
   strokeWeight(1);
   line(320, 240 + i, 640, 240 + i);
 }
 noStroke();

 // Aggregate the averages and send to Pachube every minute.
 if((minute() != lastMinute) && (sampleCount > 0)) {

   // Find the aggregated average for each minute.
   minuteAverageRed = minuteAverageRed / sampleCount;
   minuteAverageGreen = minuteAverageGreen / sampleCount;
   minuteAverageBlue = minuteAverageBlue / sampleCount;

   minuteAverageColor = color(minuteAverageRed, minuteAverageGreen, minuteAverageBlue);

   // Convert to hex and send to Pachube.
   println("Sending sky value " + hex(minuteAverageColor, 6) + " to Pachube.");
   dataOut.update(0, hex(minuteAverageColor, 6));
   int response = dataOut.updatePachube();
   println("Pachube response: " + response); // Should be 200 if successful; 401 if unauthorized; 404 if feed doesn't exist.

   // Shift values in the averages array over one, and then add the latest to the beginning.
   for(int i = averages.length - 1; i > 0; i--) averages[i] = averages[i - 1];
   averages[0] = minuteAverageColor;

   // Reset the minute averages.
   minuteAverageRed = 0;
   minuteAverageGreen = 0;
   minuteAverageBlue = 0;

   // Reset the sample count.
   sampleCount = 0;

   // Reset the minute mark.
   lastMinute = minute();
 }
}

void captureEvent(Capture cam) {
 // Get the latest camera data.
 cam.read();
 cam.loadPixels();

 // Reset the averages for this frame.
 averageRed = 0;
 averageGreen = 0;
 averageBlue = 0;

 // Average the pixels.
 int pixelColor;

 for (int i = 0; i < pixelCount; i++) {
   pixelColor = cam.pixels[i];

   // Extract the RGB values.
   averageRed += (pixelColor >> 16) & 0xff;
   averageGreen += (pixelColor >> 8) & 0xff;
   averageBlue += pixelColor & 0xff;
 }

 // Take the average.
 averageRed = averageRed / pixelCount;
 averageGreen = averageGreen / pixelCount;
 averageBlue = averageBlue / pixelCount;

 // Note this average for the aggregated average taken at the minute mark.
 // It would be simpler not to store every single frame's average, and instead to average
 // a minute's worth of pixel data (e.g. average = averageChannel / (pixelCount * sampleCount))
 // but for the sake of the display it's useful to have average data from the latest frame as well.
 minuteAverageRed += averageRed;
 minuteAverageGreen += averageGreen;
 minuteAverageBlue += averageBlue;

 averageColor = color(averageRed, averageGreen, averageBlue);

 // Iterate the sample count... this is what we use to take the average
 // at the minute mark. With the camera at 1 FPS, it should always be 60, but
 // in case of timing issues it's worth keeping track.
 sampleCount++;
}

In preparation for an AfterEffects project, I set out to free the SimCity 2000 building tiles from their binary confines. I wanted every building in the game on a transparent background. Here’s the final working set I ended up with:

Getting all of the assets in shape for AfterEffects turned out to be a bit of an ordeal. My initial plan was to do something clever, like writing a parser for the mysterious “.mif” file format in which Maxis embedded their tile data. A thorough Googling turned up no documentation or existing hacks for the file type. Poking through the bytes with my trusty hex editor also failed to bring enlightenment. Since MIF is ostensibly a graphics format, in a final act of desperation I attempted to interpret the bytes as pixels via Photoshop, again with no success. It’s not even an autostereogram:

(This dead-end detour did bring to mind the need for a tool that can take the friction out of rapidly reinterpreting large binary files as pixel data… being able to “tune” the dimensions / header length / byte order of the image parser in real-time whilst looking for the emergence of some kind of pattern or representational content in the pixels could be useful, or at least of conceptual interest. I’ll add it to the project queue.)

In the end I resorted to a mandraulic approach: selecting each building in the Urban Renewal Kit, exporting a BMP for each, and sending the results through a number of Photoshop batches to place them on transparency, resize, etc.

Not exactly clever, but it worked.

Our survey of live video services last week proved that there’s plenty of content out there, but the bottleneck seemed to be a lack of discoverability in the interface. None of the sites we looked at really gave the sense of how much was happening at one, or provided a way to sift through the piles of live video to find something of relevance or interest. The major live webcam sites seem stuck in the YouTube / catalog approach. Luckily for us, JustinTV hosts an API that makes building experimental interfaces relatively painless.

(Ustream offers a similar but slightly less robust API.)

I wanted to see whether bandwidth / API terms-of-service / human perception would allow for a massive grid of live feeds. The answer turns out to be “sort of.” I wrote some code to take the most active JustinTV channels in the “lifecasting” category, and dump them into a grid.

The results are available here.

In case the system breaks, or I run through my API allocation, a video of the system in action is available below:

A few other re-workings of the catalog approach are of interest.

5000 Webcams places live feeds in geographical context on a google map. (Also available as a grid view.)