Matt (aka Stew) provide a bit more detail about our jpeg streaming that was discussed in our half-semester presentations:
Unity can not handle a stream of infinite length, and Anthony found the runtime class in Unity called WWW to facilitate our needs. He calls a function that returns an object, which can be applied as a texture to a plane. When we transition to the camera ‘feed’ in our experience, the script calls a URL that I have designated for him. The function that Anthony wrote refreshes this image as soon as it has downloaded the previous one, which simulates a live feed that runs at a few frames a second.
The camera is controlled via a set of computer gateway interface (CGI) commands provided by Panasonic. This set of instructions allows us to access the camera via a website, where the camera acts as the server. The camera has its local application files that translate the commands given by the website into the control language used to manipulate the pan, tilt, and zoom controls. My part of the workaround was to use the still camera functionality to capture an image of the designated resolution. 3-4 frames a second is not a video stream. But! We can get away with this because the camera will already be positioned on the child’s star before we transition to the live feed. The effect gives a ‘twinkle’ as the light subtly changes the RGB pixel values at this rate, furthering the imagined live stream.
Cassandra wrote up a summary of some progress the tech team made over the past few weeks:
We’ve made significant progress in connecting our network through the server and over the web. It was relatively simple to make the server pull data from the database and concatenate the data text into a URL that the camera would understand. There are many handy modules for Python to speak to MySQL database; in this case, we used PyMySQL. However, figuring out how to send that URL to the camera automatically was more challenging. Cassandra and Tom found the HTTP Protocol Client in the python library, which greatly simplified the process. One can send the values of pan, tilt, zoom, and focus by POSTing to the IP address and port of the camera. The function returns a number code to alert you as to the success of the action.
The server is also communicating with the iPad now, receiving strings of text. Anthony and Cassandra found C# scripting for socket client connections and, with a little tweaking, successfully sent strings – such as a 6 digit barcode – over the wireless router to the server. In the process, we’ve learned a lot about how servers and clients operate and how to derive a desired function from sockets. Looking forward, we’re establishing text codes to send for events on the iPad, such as click events or onscreen events. As we get these developed, we’ll be building the environmental elements for the world.
As part of our ongoing effort to highlight our technical achievements, Tom has some info on our system overview:
Guests using the Starscape application will be holding an iPad in their hands, but this is only one small part of the show. Behind the scenes there are a number of systems and devices communicating and working together to make this experience happen. Here is a brief description of the technology that makes Starscape run!
The iPad – Guests will explore and interact with the fairy realm using an Apple iPad 2. The Starscape application uses built-in sensors such as the gyroscope and compass to determine where it should look in the virtual world. When the guest interacts with an object or character inside the world, that generates an event that is sent out over a wireless network to the Server.
The Server – This computer works behind the scenes to keep everything running at the same time. During a show, it listens for events from the iPad, and uses that information to determine what sounds to play and what lighting events to send to the Show Control system. The Server stores the location of every star in the tower and when the guest scans their barcode at the entrance, that location is sent to the iPad and used to place the star in the virtual sky. The Server also hosts a web-based interface to allow the staff at Give Kids to record the stars when they place them on the tower walls.
The Show Control System – This machine controls the lights and sounds inside and outside of the tower. When the Server sends it a lighting cue during a show, the Show Control System will respond by sending signals to the equipment that controls the individual LED light fixtures in the tower. This system will run events in the tower during times when the show is not running. It also controls the sounds and effects for the outdoor show, where the fairy appears from the cupola on top of the tower each evening.
The Cameras – There are 9 networked cameras placed throughout the tower. At the end of a show, the guest sees a live image of their star, sent over an internet connection by one of these cameras. The Server uses its database to recall the location of the guest’s star, and sends a command to the appropriate camera to point and zoom onto that spot, then send the feed to the iPad.
Dani recaps our recent presentation. We’re looking into getting the video recording so that we can share the whole thing!
Last week we had our mid-semester presentations for faculty, staff, students and guests. Throughout the semester we have to make sure that every member of our team presents at least once, so since the boys and Hyemi spoke during our quarter presentations, Katherine, Rebecca, Dani, and Cassandra were up to bat. Because Anthony is a rockstar with the iPad, we also asked him to introduce the hardware and the working tech demo, which is featured in a previous post.
Our presentation and our project were very well-received.
Here’s an example of some of our faculty feedback:
“Great presentation, benefiting from a great team of 8 motivated students working with a client that has prior positive experiences with the ETC”
“Good tech explanation. Great tech demo”
“Clear discussion of the setting for this work”
Our team is very excited by the projects reception and we are even more motivated to work hard, meet our deadlines and create a fantastic end product.
Correction! The object circled is actually NOT the tower! The incomplete tower, without the copula, is the purple object to the left of the one circled. Oops!
Our partners on-site have passed along the following photographs:
In the first photo you can see the Star Fairy’s Copula has been added to the top, and in the second you can see the brick detailing that helps tie the Star Tower in with the existing facade of the Castle of Miracles. The interior is still being finished, but if it looks anywhere as nice as the outside it’s going to be fantastic!
Our very own Matt Stewart and Anthony Palma have been performing inception: putting the physical world (the star) made digital (a camera feed) inside a virtual world (on the iPad) all held by a guest in the physical world! Whew! They wrote up a modest summary of their achievements:
We are in the process of integrating network cameras to an iPad. During our show experience for the guest, the iPad will need to stream a live feed from the correct camera to present the child with a view of their star in the tower. In the week prior to GDC, we were able to successfully create this instance in our virtual world on the iPad. The hurdle came in working with Unity, which requires a stream of finite length. To accomplish this, Matt and Anthony problem solved to arrive at using the still capture functionality of the camera. Matt wrote the correct string to access the camera, and the program written by Anthony refreshes this image within the virtual world to create a JPEG stream by requesting a new image as soon as the previous is loaded.
Click For Video
This morning Tom waited patiently in line for the iPad 2, hoping that somebody ahead of him would give up their spot. He was ultimately successful, and when he mentioned to the sales clerk that it was to be used in a project for Give Kids The World, she was so impressed that she asked for a business card to pass along to management (perhaps we’ve won over another sponsor?). The release of the new iPad couldn’t come at a better time: it’s significantly lighter than the original and the sensitivity afforded by the gyroscope is a huge step up from that of the accelerometer. All in all, it will allow for a more comfortable, intuitive and smooth guest experience.
Posted in tech, video
Tagged tech, video
We’ve been busy here at Starscape, with members traveling to attending GDC (San Fransisco), SXSW (Austin) and the TEA Summit (Los Angeles). For most of us, Spring Break was anything but a break, but we’re jumping right back into the project after some nice weather and inspiring talks. This morning we managed to buy the very last iPad 2 in Pittsburgh — keep your eyes peeled for a tech update and demo video! — and we’re gearing up for our half-semester presentations on Friday afternoon.
Our partners out in Orlando have also been extremely busy, building an entire Star Tower! When we last posted about the Tower’s progress, you may remember, there was little more than a hole in the ground.
Jerry over at ITEC has been kind enough to send us updates and photos to keep us clued into the progress. As of this morning we’ve been notified that the shell is complete and they will begin applying the exterior stucco finish. We can’t wait to see a photo of the completed tower, but we’re even more excited to visit it in person when we go to install our finished product! In the meantime, enjoy some of the process photos:
Katherine and Dani Testing Wireframe UIs
Dani and Rebecca have been working to develop the user interface that allows the volunteers to easily place the stars in the tower and focus the live camera feeds in the appropriate location. This week they produced paper prototypes of the interface, using white paper for different screens, colored post-it notes for the stars and a staircase for the Genie the lifts volunteers to a height where stars can be placed.
After testing the designs with teammates, naive volunteers will be brought in to break the system.