The Show


The weakest parts were HERB not using fingers (like Yume Project at ETC seeing
that HANDS were very important and resorting to gloves to improve robot
performance), HERB not moving during actress lines for the most part (explained in
Q/A), and lack of any quick reactions.

The best parts for HERB were its backing up and reacting to the long
aggressive delivery of the actress lines (hints that a play where HERB would have
had time to react to others’ lines would have been better choice), and head pose
movement to make more contact with stage action. I liked the Hemingway limb
gestures and the big “I like to party” limb gesture. Big and expressive worked, the
first time used. If repeated too often, it would give away that HERB does not have
a huge set of combinatorial expressions like a human actor would.

–Mike Christel


One of the things that kind of struck me was timing, obviously there’s frustration with the robot, One the question I had was differentiating between using the technology of the robot to move the arm and take advantage of that, versus using the robot to create a good well-timed back-and-forth repertoire with the actress, and to basically provide technological transparency, so the audience wouldn’t be thinking the robot is not doing what it should be doing, but the audience will be thinking the robot is responding, and responding so fast that I’m now sure if it’s automated? Or someone is real-time puppeteering it, or it’s pre-scripted, you are not thinking about the technology you are thinking about response of the robot in the context of the performance.

–Sabrina Haskell


One of the perspectives from the performance side, I think you guys really did well on the technical side of stuff. The tool, the technology and how you get to work was the core, but the performance was the a thing that you have to do, but wasn’t the core.

–Dave Culyba


We are relieved a little bit but also really happy the way we finished this semester. I think the lab was . Jon and our team, we are thinking what even outside the performance.



Simon started animating using the blender plugin, he animated HERB in each case. They also realized which part of HERB is important, like fingers, in conveying emotions I think that’s the biggest take-away.




HERB Performance Q&A Documentation

Transcript by Robin Li

Q:(Sam) Before we end the show tonight, I think one the most exciting thing about the representation is the two diverse audience we have tonight, so I want to open up for questions, that can be about the theater side, the robotic side, and sort of see where the conversation will be going. I’m actually gonna start it though, I’m gonna be the first button, so Don, who has actually acted in high school, does this tonight feel like acting?
A:Yeah definitely. It was of course it’s different in that when you are on stage you have to make choices about what to do and when to do them at the same time, but here with the robot I already made choices about what to do since I already animated them beforehand. The choice that I have at the moment was when to do them. Of course sometimes HERB wasn’t quite cooperating.(audience laughter…) but after some time I kind of got more excited…

Q: You said HERB sometimes wasn’t cooperating, could you talk about what the pauses are?
A:(Don) When it pauses, it usually means that I started a queue before the last one finishes. It hangs. (Sam) Yeah, HERB has to return to the idle spot for most of the next ones. The bigger ones take longer to the idle spot, Don is finding the balance between waiting too long for a queue to settle

Q: Is your software supporting listening and speaking, is there a way HERB could listen to Olivia and hear when she stops speaking so the need for an operator will become less immediate after each queue?
A: (Garth) I’d love to do that, but it’s mostly a concern about reliability and robustness in the theater. Here theater is a much more demanding demo than robotic demos. The thought of having it work every time the best we can motivates us to keep a human in the loop. Also human is in the loop to have the high level interaction, Sam says no robot could ever understand that, that’s human understanding. So yes, it will be nice to work on speech recognition to be much more robust enough to at least answer a dialog at a time. The speech engine we build is actually capable of variation on the performance, the process of building a synthesized voice also involves analyzing the speech in a way that we could reconstruct it. We also like to extend more of our mean stretching the interpretation of the dialogue This was because basically the original interpretation from the actor in the sound booth was so good that we didn’t manipulate it very much, that was sort of the counterpart to this…

Q: Did you consider allowing HERB queued by the other actor? So essentially being able to reconstruct a one person show ?
A:It’s completely technically feasible, Don is like clicking one button…

Q: So this performance was not autonomous, sort of looking forward, would it be possible to have a robot improvising?
A:(Garth)You get these questions of human intelligence very quickly, there are these contents every year people write chat box or other programmers trying to interact and dialogue with people, basically simulating being a human, every year they get better, so yeah, technology is there with voice recognition and kind of dialogue recognition too, to build believable agent that could do the same, that’s an entire research path you can delve. We are as a lab kind of focusing more on the physical expression, the arm gesture, for me the interesting part here is taking the non-anthropomorphic or at least semi-anthropomorphic robot, then find movements in it without emphasizing the face and more emphasizing the arm and body language to create a visual vocabulary, instead of pushing that to the dialogue and story telling aspect, but that’s also a part of it.
(Sam) To elaborate from director’s perspective, I thought it’s more interesting to develop a robotic puppet than a robotic actor, who knows maybe in 30 or 40 years,…(can’t hear clearly here)

Q:What was the original learning objective of the project? (silence for 2 seconds, then burst into laughter, first Garth then everybody)
A: In some sense this project is the sort of fun project that evolves to see really where it could go. But at the same time it is serious, you got the questionnaires, there are obviously some intentions behind there. What Personal Robotic Lab is doing is trying to figure out how robot behaves in everyday life situation, the interaction with people is such a part of that, a condensation of drama in a specific form, out of that, we discovered something about how to tell a story, how to convince some human around about what you are to do, obviously it has nothing to do with normal daily activities, I would like to see now is HERB grab things at the table, not just does it in the way that accomplishes the task, but in a certain way that expresses itself as well, now we have the library of movements and ideas of how HERB can move. I also want to point out Sid is the PI of the lab, much of this is his agenda. More questions?

Q:(unclear but is about HERB’s joints in front, and how it affects the acting)
A:(Sam)That was sort of a learning experience in itself. The original thing that we tried was to simulate human movements as close as possible, like try to make HERB shrug like HERB can shrug as if he has shoulders(laughter). Then we started to see what is HERB’s equivalence of a shrug. Sort of what is a “non committal move that doesn’t require a lot of energy” figuring out how to move his elbows in the way that says “Nay”(Sam acting shrug)
(Garth) HERB is really designed for functional purposes, two arms can do bimanual functions, like moving objects with two hands. The head is actually added recently, responsible for practical tasks. But you know we can find emotions in it, that’s sort of the lesson with that, any interesting mechanism you explore it as a puppet and find the expressive emotions involved.

Q: I’m just curious in the animations HERB’s fingers never move? He has fingers
A: (Jon) I think the fingers were moving in the simulation.
(Aaron) HERB has three fingers, it can be kind of large, again he is designed for functional purposes, in some cases it was kind of tricky to open them up and do these gestures without looking like claws…
(Sam) He can demonstrate that…
(Don demonstrated a open fingers animation, audience burst into laughter…)

Q: I noticed that HERB didn’t have huge react of dialogues, the question for you is was that a technical issue or was that done by choice? Then I want to ask Olivia that question as well.
A:(Garth) You mean he isn’t directly responding to…(Q) While she is talking, he is doing nothing.
(Sam) There were actually a few times some smaller ones, it was during the monologues, it’s mainly a technical thing, most of time he was using the length of shorter lines to get back into the initiation for the next one. For the longer lines, he had a few looking arounds, looking downs, but for less than 2-3 seconds long will be pushing it get all those things.

Q: As an actress, how was it to perform with an actor who wasn’t reacting to you?
A:(Olivia)En… I think it’s a really interesting question, because it actually never occurred to me as an actress that he wasn’t, there’s obviously a point to which, this is not, as you said, a semi-anthropomorphic robot, there’s a obvious element of non-humanness, but certainly as an actress you have to accept what you are given, we talk about the reality that this is in the caffe which isn’t, in our reality it’s totally weird this robot is just coming out and talking to me. Or… some time in the future, it’s totally average that robot comes talking to me, or some other dimensions, some other reality where as before this is our world, but this is just another character in that world. I think that’s the ones that we landed on as such, while it would be really interesting to explore what human reactions would be to robot in situations like this…

Q: Kind of a question looking into the future, what kind of advances would you put on makes it easier to do today than back then?
A: I mean, honestly what it comes to down to, is the open source movement, because much of the software we have built has been based upon a wide variety of open source software, we leveraged blender the 3D animation package, using ROS the robotic operating system, at some sense all of the tools we are building are based on the works of thousands of people, arguably that’s more cultural , in terms of technology, we probably could run this on 10 year old computer in some respects, it wouldn’t be that different, the very arm on HERB was actually designed in the late 80’s. The technology has been there for long time. So I don’t think we are really using specific high right now, we are able to work in the eco system of robotic technology which is evolving, that’s the real difference.

Q: Well first of all, on behalf of school of drama, want to thank everyone who’s not in the school of drama to bring this in, and sharing with us and I think you guys have done a great service. My question is out of the context, I’m the guessing the primary purpose of the robot is helping disabled people in their homes, I’m wondering if you guys have done field tests to actually bring this to people at their homes, what their reactions have been and wether they had trouble accepting it?
A:(Garth) This is actually a question for Sid.
(Sid) Thank you for the question, it’s a great question, we’ve done several of these studies, the evolution of the robot itself has been motivated by a large number of these studies, to get the robot to work with people, not just scientists like us, but real people(audience laugh), But…the result has been startling sometimes, some people have found the robot endearing, some people have found the robot threatening. We are playtesting with children, actually children in the science center, 3 or 4 year old kids, who come by and some are immediately fond of the robot, and some…there’s a great anecdote, people were playing with the robot, the little kids, and then he started speaking, he said “Hi, I’m HERB”, these little kids started hugging each other (audience laughs), I’ve never seen such thing like that, the reactions have been very interesting, now we want to use the tool of the dramatic performance as a way to understand more about how the robot is not just being functional, but also be expressive and emotive in what he does.

Q: So what’s next in terms of making it more engaging for people?
A:We are building the scaffolding for dramatic performance, we are developing open source tools that can allow anybody, not just us, but even any kid with blender to create animations and mount them on the system, tremendous amount of tools we build we want them to be open source, and again like Garth was saying, we want to take the learning from this and automate the process, because all of you gave very excellent suggestions on what we should automate, we’ve been thinking about these, I want to take these and also put it into HERB’s everyday activities, just like if he’s picking up a coffee mug, if he picking up the coffee mug happily or sadly, or nervously, I think that’s what it really takes to get robot like this not just doing things in people’s homes but being accepted in people’s homes as an agent they want to work with. (Garth) One last question…how about you being the last one right here…

Q: You said you want to make all the softwares open source, how specific is the current software for HERB, or can it be adapted basically such that you can say this point of blender animation equally works on any other robot?
A: (Garth) Of course there are ad-hoc nature of the show, there are some kind of things that are still HERB-specific, at the core it’s pretty general, it’s based on ROS which is actually designed for a variety of robots, description of an arbitrary robot, we haven’t done it yet so we can’t promise, the goal is to have that, so with that I think we could close the formal session so people could go, oh sorry… (Olivia and HERB took a bow, audience applause…) please fill out the final part of your survey for us, come down ask questions if you like.

Transcript by Robin Li
There might be minor errors due to recording quality, corrections are welcome