A System of Animation Gestures for Effective Teaching Avatars

Project overview
	Almost a century ago, Edward Sapir noted that we “respond to gestures with an extreme alertness” according to “an elaborate and secret code that is written nowhere, known to none, and understood by all”. We are a team of computer science, computer graphics technology, and psychology researchers who have partnered with the long term goal of breaking the code of effective instructor gestures in education. In this first project we are developing a system of computer animation instructor avatars and we are using it to research instructor gesture effectiveness in the context of mathematical equivalence learning. We examine both gestures that might convey an appealing and engaging instructor personality and gestures that elucidate the mathematical concepts directly.
Publications
PDF	Animation Killed the Video Star In this position paper, we describe a novel approach for creating visual stimuli for research on gesture in instruction. The approach is based on a system of computer animation instructor avatars whose gesture is controlled with a script. Compared to video recording instructor actors, the approach has the advantage of efficiency—once the script is written, it is executed automatically by the avatar, without the delay of script memorization and of multiple takes, and the advantage of precision—gesture is controlled with high fidelity as required for each of many conditions, while all other experiment parameters (e.g. voice tone, secondary motion) are kept constant over all conditions, avoiding confounds. Voicu Popescu, Nicoletta Adamo-Villani, Meng-Lin Wu, Suren D. Rajasekaran, Martha W. Alibali, Mitchell Nathan, Susan Wagner Cook. ACM CHI Workshop on Gesture, 2014.
PDF VIDEO	An Animation Stimuli System for Research on Instructor Gestures in Education We present a system for creating animation stimuli for the study of instructor gestures in education. The system does not require artistic talent or programming expertise. The system provides an animation character that serves as an instructor avatar. The avatar can speak, write on a nearby white board, and make pointing, embodied cognition, and charisma gestures. The avatar is controlled with a text script that specifies when and what the avatar says and does. The script is executed automatically to create the animation stimuli. The system has been used so far in two studies. The first study investigated which type of charisma gestures makes the instructor more appealing by testing 18 gesture conditions. The second study compared the learning of mathematical equivalence after a lesson given by the avatar, with and without gestures. This shows that the system can be used efficiently to create accurate and effective stimuli for complex studies. Jian Cui, Voicu Popescu, Nicoletta Adamo-Villani, Susan Wagner Cook, Katherine Duggan, Howard Friedman. IEEE Computer Graphics and Applications, accepted for publication October 2016.
	Gestures and charisma: Lessons from an avatar Gestures are known predictors of interpersonal liking, attention, and learning, but the role of specific gestures has been difficult to study experimentally. Combining computer science with social and cognitive psychology, we developed an interactive software system with an avatar who moves and gestures. In the current study, we systematically manipulated the direction of gesture (vertical, inward-focused, or outward-focused) as well as the parallelism of arm movements (one arm, both arms moving asynchronously, or both arms moving in parallel) of the avatar in the context of a math lesson. Spoken language for the avatar was pre-recorded (and invariant across conditions). In a counterbalanced experimental design, undergraduates (N=56) viewed each math lesson and rated the avatar on a variety of interpersonal characteristics. Factor analyzing the ratings distilled two factors: charisma and attractiveness. Contrast analyses revealed that outward-focused parallel gestures were rated as the most charismatic, and that inward-focused asynchronous gestures were the least charismatic. Gestures were not related reliably to attractiveness ratings. This study confirms that certain key gestures are important predictors of charisma, with implications for interpersonal communication, both in-person and via emerging media technologies. This study also illustrates how new developments in animation technology can be employed in the experimental study of gesture in psychology and education. Duggan, K. A., Friedman, H. S., Cook, S. W., Schustack, M. W., Cui, K., & Popescu, V. (under review).
PDF	Hand Gesture and Mathematics Learning: Lessons from an Avatar When teachers gesture, students learn more. This beneficial effect of gesture on learning has been demonstrated in multiple domains, including mathematics, science and foreign languages. However, because gesture is known to co-vary with other non-verbal behaviors like eye gaze and prosody as well as other face, lip and body movements, it is possible the beneficial effect of gesture is instead attributable to other behaviors that reliably co-vary with gesture. We used a computer-generated avatar to create dynamic animated stimuli with controlled verbal and nonverbal behavior. Children viewed lessons on mathematical equivalence in which the avatar either gestured or did not gesture. Children who observed the gesturing avatar learned more than children who saw the avatar that did not gesture, and they solved problems more quickly. Moreover, those children who learned were more likely to transfer and generalize their knowledge. These findings provide converging evidence that gesture facilitates math learning, and also reveal the potential for using technology to study nonverbal behavior in controlled experiments. Susan Wagner Cook, Howard S. Friedman, Katherine A. Duggan, Jian Cui, Voicu Popescu. Cognitive Science, 2016, pp. 1-18. doi: 10.1111/cogs.12344.
PDF Scripted Animation VIDEO Manual Animation VIDEO	Scripted Animation towards Scalable Content Creation for eLearning—a Quality Analysis The success of eLearning depends on the broad availability of educational materials that provide a high-quality delivery of high-quality content. One approach for high-quality delivery is to rely on a computer animated instructor avatar that not only speaks, but that also gestures to elucidate novel concepts and to convey an engaging personality that captures and maintains the learners’ focus. The traditional approach of manual key frame animation does not scale, as it requires a substantial time investment as well as artistic talent. We have developed a system that allows animating an instructor avatar quickly and without the prerequisite of artistic talent through a text script. In this paper we quantify the speed / quality tradeoff made by our scripted animation by comparison to manual animation. Nicoletta Adamo-Villani, Jian Cui, Voicu Popescu. International Conference on E-Learning, E-Education, and Online Training, eLeot 2014, 1-9.
PDF VIDEO	Digital Learning Activities Delivered by Eloquent Instructor Avatars: Scaling With Problem Instance We present an approach for achieving scalable authoring of digital learning activities, without sacrificing delivery eloquence. A computer animation character serves as an instructor avatar that not only speaks but also makes deictic, iconic and charisma gestures. The avatar is controlled via a text script, without the prerequisites of computer programming or animation expertise. Given a script for a problem, the system automatically generates scripts for additional instances of the problem, by adapting the targets of the deictic gestures, the speech, and the synchronization between speech and gestures. Starting from initial learning activities of a few minutes, the system can automatically generate hours of quality on-line learning activities. An evaluation by computer graphics, computer animation, and education research experts reveals that the automatically generated learning activities have animation quality that is comparable to that of the original activities. Saikiran Anasingaraju, Meng-Lin Wu, Nicoletta Adamo-Villani, Voicu Popescu, Susan Wagner Cook, Mitchell Nathan, Martha Alibali. ACM SIGGRAPH Asia Symposium on Education 2016.
Presentations
Paper presented at International Society for Gesture Studies, Paris, France, July 2016.	The effect of temporal coordination on learning from speech and gesture. Pruner, T., Popescu, V. & Cook, S.W.
Poster presented at Cognitive Development Society, Columbus, OH, October, 2015.	The role of temporal synchrony in learning from hand gesture. Pruner, T., Popescu, V. & Cook, S.W.
Roundtable discussion at American Educational Research Association, 2015.	Math Learning from a Gesturing Avatar Cook S. W. & Popescu, V.
University of Chicago, Committee on Education, Workshop Series, 2015.	Space and Time on Our Hands: How Gesture Influences Learning and Memory Cook S. W.
Annual Meeting of the Society for Personality and Social Psychology (SPSP), 2015.	Priming personal charisma with a charismatic avatar Duggan, K. A., Friedman, H. S., Hanna, C. N., & Schustack, M. W.
Annual Meeting of the Association for Psychological Science (APS), 2015.	Influence effects of a gesturing avatar Duggan, K. A., Schustack, M. W., & Friedman, H. S.
2015 Keynote Speaker, Symposium on Embodied Cognition in Multimedia Learning, Radboud University Nijmegen/Erasmus University Rotterdam, Rotterdam, The Netherlands.	Cook, S.W.
2014 Keynote Speaker, EUCogIII Workshop on Embodied Cognition, Institute für Neuroimformatik, Ruhr Universität, Bochum, Germany.	Cook, S.W.
Thematic Panel: Teachers’ Gestures in Instruction: Attitudes, Behavior, and Research Approaches. 6^th Conference of the International Society for Gesture Studies, 2014.	Avatar-Based Research on Gesture in Instruction: Opportunities, Challenges, and Solutions Jian Cui, Meng-Lin Wu, Suren Deepak Rajasekaran, Nicoletta Adamo-Villani, and Voicu Popescu
Nonverbal Behavior Preconference, Austin, TX, February, 2014.	Charismatic avatars and Pandora's box. Friedman, H. S., & Duggan, K. A.
Social/Personality Psychological Brown Bag Series, University of California, Riverside, February 2014.	Nonverbal communication and charismatic avatars. Duggan, K. A.
Annual meeting of the Society for Personality and Social Psychology, Austin, TX, February 2014.	Computer avatars, gestures, and personal charisma. Duggan, K. A., Friedman, H. S., & Schustack, M. W.
Annual meeting of the American Psychological Association, Honolulu, HI, August 2013.	Charismatic avatars and nonverbal communication research in emerging media technology. Duggan, K. A., Friedman, H. S., & Schustack, M. W.
Studio Talk, ACM SIGGRAPH 2014.	Charismatic and Eloquent Instructor Avatars with Scriptable Gesture Jian Cui, Nicoletta Adamo-Villani, and Voicu Popescu
People
	Voicu Popescu, PI Nicoletta Adamo-Villani, Co-PI Jian Cui, (former) graduate student
	Susan Wagner Cook, PI
	Howard Friedman, PI Katherine Duggan, (former) graduate student
Contact cyberlearning at cs dot purdue dot edu
	This material is based upon work supported by the National Science Foundation through an EXP Cyberlearning collaborative grant awarded to three institutions: Purdue University (grant No. 1217215), Iowa University (grant No. 1217137), and University of California Riverside (grant No. 1216984). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.