Great Party! (How'd You Do it?)

An interview with the technologists behind frog design’s augmented-reality SXSW Interactive fiesta.

Saturday’s opening night party marked the 12th year that frog has organized and hosted the official party of South by Southwest Interactive (SXSWi), and it was an augmented-reality humdinger. We’ve posted photos on this blog and on our Flickr site to give some flavor of the evening. The twitter chatter was telling as well. Even so, we’ve had quite a few people ask us how we did it — literally what was the technology behind the augmented reality Porta Potties? How did you create instantaneous 3D images of people dancing? What’s a Whuffie? People also wanted to know why we chose augmented reality (AR) as a theme.

frog is an innovation firm. We are designers, strategists, and technologists, so technology is part and parcel of what we can do. This year’s party was a way for frog to take its emerging technologies expertise and have some fun. It’s not often you get to see this kind of thing on such a grand scale and displayed in such a playful way. The basics were this: When each guest arrived, they were handed a card with an RFID tag glued to it. There were stations around the party where guests could scan their tags, essentially entering certain information data into a nearby computer. The data was collected, translated into a visual form, and then logged as statistics to be shown on the central “Whuffie Wall.”

The term “whuffie,” by the way, is a word coined by author Cory Doctorow in his book Down And Out In the Magic Kingdom. It refers to the measurement of respect or karma a person gains or looses in their lives. In Doctorow’s future, humans have implants in their brains that visually project their whuffie, which has replaced money as currency. The Whuffie Wall at the frog party took the place of Doctorow’s brain implants, and instead of karma, it measured and displayed attendees social “value.”

I spoke to the two frogs in our Austin studio who were responsible for both coming up with the theme and executing the experience: Principal Technologist Jared Ficklin and Senior Technologist Gregg Wygonik. They helped pull back the curtain on the set up.

design mind: Augmented reality is getting some buzz right now, but no one is quite sure where it’s going or even what relevance it has. Why did you decide to have a party with an AR theme?

Jared Ficklin: AR is a bit of a buzz because people have discovered that they can have an instant layer of information with it. But right now AR is centered on the low hanging fruit of the “you are here” map. With the party we wanted to explore other possibilities.

Gregg Wygonik: Yeah, most people think AR is some 3D sasquatch that you can see when you hold your beef jerky package to a webcam, but there are a couple of mobile offerings that are starting to blend data visualization with location based information. It’s starting to open people’s eyes that AR is not all just wiz-bang graphics; it’s the information overlay.

Jared Ficklin: Also, we wanted throw a party where we could use some of the toys we play with everyday at work, and put some of frog’s capabilities on display. AR just fit right now into that theme.

dm: So what would you say is frog’s technology offering?

JF: Technology at frog is an emerging technologies software design group. We explore create and develop new technologies and we fulfill the vision of the designer. Sometimes we’re ahead of the designers and sometimes they’re ahead of us, but either way we tend to work in emerging technologies. That means we’re usually pushing that technology into a new space.

GW: I think the one thing the technology group at frog does is that they’re closer to technology trends than design trends, and designers aren’t sometimes aware of what’s possible. We can help push the design by letting designers know that they can push things beyond what they’ve been doing in the past.

dm: So let’s talk about the party. What was the technology you used for the party and why did you use it?

JF: We used a few different technologies because we had several different needs. We had to gather and distribute data, we had to create components for the interaction, and we had to visualize the data.

For visuals we used XNA and Adobe Air because it created the user interface for the experience. For the dance floor, Canesta loaned us a 3D infrared camera and we used Computer Vision to make the data artistic and fun. We used Phidgets for the user interaction. Phidgets is an open hardware platform company. They have APIs that allow you to code — essentially plug and play readers that opens you up to new input output methodologies very rapidly. Way down at the base, we used a simple source control — SVN — along with an old Windows 2008 server box.

Each experience was writing data down as XML into individual folders, but once synced, the software experiences could all read from each other’s data. The result was a distributed data network for the party. When someone scanned their data, we could use it for all the other experiences. It was a hackish but elegant thing.

dm: One of the biggest hits of the party (and one of the most surprising) were the augmented reality Porta Potties. How did that setup work?

JF: A lot of people thought we had camera inside but we used Phidget sonar sensors. When someone stepped in the Porta Potty, we would get a value from the sensors between zero and 1,000. The door was a value of 78. The front of the seat was 38. That’s how we knew if a person was standing or sitting. We took that data using Adobe Air and created a user interface that I thought was funny. Also, the Air layer wrote XML files that we could use later for stats. To get the graphic on the potties, we used an ultra-short throw projector positioned about ten feet in the air to prevent shadows and we wrote our own free transform for bitmap code to correct for the resulting extreme keystoning. Keystoning is the term given to the image you get when you project at a sharp angle — the image gets severely distorted and looks like an architectural keystone. At this level we had to correct for it by writing our own software. That’s how we were able to use one projector, one laptop, and a wireless USB VGA connection to project across eight Porta Potties. I actually had a Porta Potty delivered to my house and I worked for two weeks in my driveway trying to figure it out. I can only imagine what my neighbors were thinking.

GW: I was not surprised the Porta Potties were the biggest hit. The day the concept came up there were very few indifferent opinions. It’s hilarious but it’s only hilarious because of the graphics presented. There’s also a practical side. It’s really useful to know, if you’re standing in a queue, which Porta Potty is free. So you can add a green or red color. That’s one type of information. The information changed as soon as we showed someone sitting or standing. That’s when it affected people. Maybe it takes their mind off of standing in line? Then, using the bathroom becomes a fun thing to do.

dm: The dance floor visualization was another big hit. You could see a colorful shape of yourself as you danced but if you looked closely you could see three dimensional depth as well.

JF: Yeah, we were able to do that with a 3D imaging camera loaned to us by a company called Canesta. The camera sees depth in addition to you and it does it at a very high resolution. It creates what you could call a data cloud.

GW: It’s actually an RGB image camera with an infrared lens. It projects infrared into the environment and it measures the position of things in the environment within a centimeter of accuracy. That’s essentially gestural technology. It’s very interesting stuff. For the party, we wanted to see the artistic side of the camera. There is so much potential from generating art from 3D data than hasn’t been explored, so that’s what we tried to do.

We used Microsoft’s XNA, their Xbox community game framework. It’s freely downloadable. Getting what we wanted was a matter of writing a layer between XNA and the camera, and the coding work was all about sorting the data so we could track individual or multiple people in the scene. It’s Computer Vision with depth added. Once we did that the sky was the limit on how we visualized it. We had a good supply of designers at frog giving us visual assets.

JF: People were up on stage and didn’t realize they were interacting with a computer by dancing, but that’s just what they were doing. And this kind of computer vision is going to be important to AR experiences moving forward. In the future. you will be able to gesture to interact with a computer in the same way humans gesture with each other. Computers will be able to pick up on your movements.

dm: The AR bar was fun. And what was with the old school graphics on the display?

JF: Well, we’re both a couple of eight-bit nerds so we gave a nod to Space Invaders in the visual. But the bar was just another information gathering and data visualization station. We lined up RFID readers along the bar, so when people scanned their tag after getting a drink, an animated beer would go up from the station and it would display the number of drinks that person had taken all night. The technology was the Phidgets on the front end to collect the data, and Adobe Air on the back end to create the visuals. The experiment was, will you take more or less drinks when everyone knows how much you’re drinking? The other thing about the bar was that at about 10:45pm, enough beer had inundated the technology readers until they stopped working. I have never, as a technologist, had to beer proof my technology.

dm: The Whuffie Wall was the center of the party. What was going on there?

GW: There’s no limit to what you can augment with data. The difficult part is getting the data with which to augment the reality. That’s certainly why some of the initial geo-location mobile AR experiences are based on public databases. That’s just what’s available. At the party, the key was gathering all the data to be able to show it.

JF: Yeah, and the way we showed how all the various stations or experiences came together was on the Whuffie Wall. In Cory Doctorow’s book people had to have implants to see this stuff. At the party, the RFID tags were the proxy for information gathering, and the projections on the wall were the proxy for Doctorow’s implants.

As soon as someone scanned their tag, a dot dedicated to them would appear on the wall. Then, every time they scanned the tag, that dot would grow in different ways. For every beer that people took from the bar, the blue area of the dot would expand. If they friended someone [by scanning their tag at one of the mannequin stations], the yellow ring around the dot would expand. Everything they “liked” resulted in a pink hearts. We were also able to show which dots were connected by drawing lines between them. So everything someone did would grow their circle in real time. There was a guy who stayed at the connect station all night, showing people how it worked and friending everyone that walked by, because he wanted to be the biggest dot on the wall. In a small way, the Whuffie Wall proves out Cory Doctorow’s thesis that this type of information, once it’s visible, is important to people, and it changes people’s behavior. The guy with the biggest dot on the wall would have done something different had that stuff not been there.

dm: What about the photo wall? How did you create photos that then became mosaics of other photos?

JF: That was the InfiniVision station, and for that we had a guest interactive artist named Craig Swann who owns a company called crash media. He very kindly provided us the source. It takes a picture and takes all the pictures taken that night and reproduce that picture as a mosaic. What we did was add the ability to record that. When someone scanned their RFID tag it did it’s thing, but it also allowed us to add that picture to their dot on the Whuffie Wall. We were also able to use the photos for a real time slideshow near the stat wall.. The entire party was about hyper-connectedness. In future situations, no one point of data will be for one purpose.

dm: What was your favorite takeaway from all this?

JF: I think having the party at SXSW Interactive was key because we had a tech savvy audience. They were really into figuring out what was going on and playing along. They understood what they were looking at and when things went down — and they did go down — they actually appreciated what we were doing more. One of the favorite things I overheard someone saying that night was “It’s really raw. It looks like they’re coding this right here.” And in some cases we were. There was one moment where the bar set up got overwhelmed with information, we load tested it pretty hard but never had 3000 people to throw at it. It just choked, so I had to rewrite parts of the interaction to be much thinner, and I did the installation right on the laptop that was feeding the bar visualization in the middle of the party. I didn’t have time to switch out of mirror mode, so everyone at the bar could see me writing the code and reinstalling it on the screen over the bar. When I finished, people applauded. That would only happen at SXSW. Here there seems to be a great understanding that technology can be playful and new and have foibles, and that that’s all part of it.

GW: For me it was the social part of the Canesta experience. One thing that totally fascinated me all night was seeing how people reacted to the technology. Most people would get in front of the camera and move around just to see themselves. But then, part of the Canesta experience was that it would switch to an interactive game mode. These games were intended to be single person experiences so if there was a group of people on stage, because of the way the technology was written, the camera doesn’t know where to go if it can’t focus on one thing so nothing would happen. What was great to see, though, was that people figured this out. When one of the games came up and there were five or six people on stage, at first everyone was moving around and nothing was happening, but then they really quickly learned that it was about all being together as one group. And they all ran back and forth together as a group. This happened over and over. It may have taken some time, but everyone got it. People modified their behavior.

dm: Thanks for your time. What about next year?

GW: Who knows? Even if we had an idea, it would change as the technology changes between now and then.

JF: Ask me in a month after I’ve had some sleep. Right now I’d say it’s going to be a keg of beer and a sleeve of red cups.

Sam is the director of publishing for frog where he oversees frog's global content, editorial, and digital publishing strategy. He is also the editor of design mind, frog's print and online media platform. Sam is the author of numerous books of non fiction and has written for Dwell, Metropolis, GOOD, and other magazines.