Paul Craig

Studies show that face to face collaborative videogames have a great potential to improve the quality of education in the classroom of the future. Educational games can help pupils increase cognitive skills, increase motivation and reduce the time taken to reach learning objectives. Games used in a group environment can enrich the learning experience further still by helping develop group social skills and exploiting social dynamics toward achieving a common learning objective. Despite these advantages the uptake of collaborative games as learning tools in the classroom is still relatively low. While collaborative working has proven advantages they also suffer from an incompatibility with the ingrained individualism of traditional education. Moreover, collaborative working can disadvantage introverted students, suffer from conflicts within a group or allow less motivated students to avoid making a contribution. This work investigates the viability of these disadvantages being managed through the intervention and mediation of an intelligent embodied conversational agent with awareness of group activity acting as a virtual tutor.

Methodology

In order to investigate the role of Embodied Conversational Agents (ECAs) in collaborative learning games we undertook an experiment that allowed us to observe children using a variety of collaborative games with and without the assistance of an ECA. Before and after the experiments the children were given written tests to assess specific learning objectives and during the tests we recorded audio and video. We also asked the students to complete questionnaires to determine how they felt about the level and quality of collaboration.

Our experiments involved six groups of three children aged eight to ten years. Each group spent two hours playing educational videogames. Half of the groups were joined by Samuela, the ECA virtual teaching assistant, and half were unsupervised. Four groups were all female (two with and two without Samuela) and two groups all male (one with and one without). Each child was tested immediately before, immediately after and four days after their session with Samuela. The exams used for this testing included three five minute sessions testing mathematics, languages and reading. Three versions of each exam were created so that no two students in the same group would do the same exam at the same time and no one student would do the same exam twice. This helped ensure that any differences in difficulty between the questions would average out over the course of the experiment.

The set-up of the experiment is shown below. Games were played on a forty-two inch screen angled at forty-five degrees and raised between waist and head height. The children were observed through two-way glass with audio and video recorded throughout. When our ECA, Samuela, was involved in experiments, she was controlled using a wizard-of -oz set-up visible to the students on a separate 26 inch monitor. The students were not made aware that Samuela was operated by a human controller.

Educational Videogames

Three educational games were developed for our experiment. In order for the games to be both accessible and challenging for children with different levels of learning, we tried to incorporate a gradually increasing level of difficulty. Other key aspects of game design were cultural relevance and age appropriateness. Here we tried to ensure that the games were non-violent and did not enforce gender stereotypes while encouraging the children to identify with elements of their native Mixtec and Mexican culture.

The mathematics game is a simple ‘castle defense’ type game where the students have to solve mathematical equations to fire eagles and prevent the muertes from reaching the perimeter wall of their town and draining their energy. The character of the user, the jaguar, sits at the left hand side of the screen. Below this character is a keypad and to the right a list of sums. To the right of the sums is a vertical wall and beyond the wall are the muertes. Each muerte advances slowly from right to left toward a sum and if a muerte reaches the wall it stops and begins to drain the health of the user. When the health of the user reaches zero the game is over.  In order to stop muertes reaching the wall the user can answer sums to fire eagles. The user can press on different sums to answer them using the keypad. If a sum is answered incorrectly health is drained and if a sum is answered correctly an eagle is fired from the wall toward the right hand side of the screen. When a muerte is hit by an eagle it is pushed back away from the wall. Pushing back muertes also adds to the users score and causes the level-up bar to rise. When the level up bar is full, every sum fires an eagle to push all the muertes back and the user progresses to the next level. As the level increases, the muertes begin to speed up and it becomes gradually more difficult to do all the sums on time and keep the muertes away from the wall.

The second game encourages students to learn the names of animals in English. This game is a similar ‘castle defense’ type game to that used to learn mathematics described above. However, instead of doing sums to fire eagles the students have to match words in English to their Spanish translations. When words are matched correctly eagles are fired from both words and health is drained when words are matched incorrectly. The game begins with a small number of words for more common animals such as cats and dogs. As the game advances the difficulty level increases with a wider variety of gradually more obscure animal names. If the children are not already familiar with the names of these animals in English, they can normally find the translations out by trial and error and learn from their mistakes.

The third game aims to help the children with reading by asking them to complete a story by replacing missing verbs. Literature and authors are held in particular esteem in Mexico, and Latin America in general, with popular authors often becoming diplomats or politicians. This game encourages the children to explore the Mexican side of their identity by using an adaptation of the short story ‘Mi vida con la ola’ by Mexican Nobel laureate for literature Octavio Paz. During the game, text from the story with missing words moves toward the top of the screen. The user can replace missing words from a list of words at the right hand side of the screen. If a word is replaces successfully the score increases. If however the children attempt to enter the wrong verb into a space, health is drained. Likewise if the space for a missing word reaches the top of the screen without being filled, health is drained and the text stops scrolling until a correct word is entered. While the text is scrolling the health replenishes slowly and as the children progress through the text by replacing words, the missing words become more frequent so as to make the game more difficult.

Samuela the Embodied Conversational Agent

To emulate an ECA in our experiment we used a Wizard of Oz set-up. This involved an avatar controlled by an unseen human operator giving the impression of an autonomous system. This allowed us to emulate an ECA running under ideal conditions without any of the undesirable artifacts, such as slow or inappropriate responses, that might be evident if we decided to use an actual autonomous ECA. This also allowed us to avoid having to rely on non-mature technologies such as imperfect speech and emotional recognition that would be likely to become a cause of unnecessary confusion during the experiments.

The avatar used in our experiment, Samuela, was originally developed by Telefónica R&D using Hapteck Software for the European Union funded project Companions. The original version of Samuela used in the project spoke English and was northern European in appearance with blonde hair and blue eyes. To make Samuela more familiar and understandable for the students involved in our study based in the Mixteca region of Mexico we changed Samuela’s language to Latin-American Spanish and her hair color to dark brown (see figure 4). In order to make Samuela speak we developed a basic interface that allowed the operator to type words and have them sent to Samuela by pressing the return key. This ran on the same laptop computer as Samuela using a different monitor. The monitor to view Samuela was placed in the games room with the children and the laptop computer with the control interface was left in the observation room.

Results

The children’s performance did not improve much immediately after their session with the games. This was most likely due to the children being tired and over-stimulated after playing the games for two hours. When the students were tested again, four days after the tests, there was a significant improvement in their performance. This improvement was particularly marked for mathematics where the student’s performance showed an increase of 21.9%. The improvement for languages was 4.3% and the students regressed slightly in their reading (by 2.1%). In order statistically validate our results and account for inter-sample variance we performed a single-tailed t-test. This gave a p-value of 0.016 for the second test to indicate that it was highly likely the children’s improvement was due to their exposure to the games rather than variation of the children’s scores overall. The p-value for improvement in the mathematics test was 0.0018 indicating a greater probability that the children’s improvement was due to their exposure to the games. The p-value for the language test was marginal at 0.20 (0.089 for the boys) indicating that there is insufficient evidence to conclude that exposure to the games caused an improvement in the children’s results (using the standard p-value threshold of 0.05). Results also indicate that there is insufficient evidence to conclude that the drop in reading performance is due to exposure to the reading game (with a p-value of 0.38).

Despite the improvement in the children’s performance being more apparent in the final exam four days after the games session, the influence of Samuela appeared to be more apparent in the exam immediately after the test. Here the children seemed to perform about 2.7% better with Samuela. This was not however a statistically significant improvement and the trend actually appeared to be reversed in the exam applied four days after the test.

Conclusions

The work, so far, shows the potential of an embodied conversational agent (ECA) to improve the level of collaboration for students using interactive learning games. While our ECA was not found to have any significant impact on the realization of short term learning objectives, the quality of collaboration between children was seen to improve dramatically. Since collaboration is known to improve the achievement of learning objectives in the long term and the ability to work in a team is an important skill in itself, we strongly believe there would be significant benefits to the introduction of intelligent ECAs in collaborative learning environments. Our experiments also showed us that children tended to see an ECA in the role of a friendly teacher and were comfortable communicating with a synthetic character acting in this role. As future work we plan to annotate data from the experiments in order to visualize the dialogue along with indicators of collaboration to facilitate more rigorous analysis. This should help us uncover some of the mechanisms of collaboration between the children assisted by Samuela and contribute toward a more formal interaction strategy for the future implementation in a fully autonomous ECA.

Publications

P. Craig, et al., “A Cognitonics Approach to Computer Supported Learning in the Mexican State of Oaxaca,” Informatica: An International Journal of Computing and Informatics, vol. 38, pp. 263-271, 2015. Informatica Journal Article

Craig, Paul, Roa-Seïler, Néna, Martínez Díaz, Marcela, & Lara Rosano, Felipe. (2013). Assessing the potential of colaborative video games to improve education in la mixteca region of
mexico. Paper presented at the Third International Conference on Cognitonics, Slovenia. (Draft version)

Craig, Paul, Roa-Seïler, Néna, Rosano, Felipe Lara, & Díaz, Marcela Martínez. (2013). The role of embodied conversational agents in collaborative face to face computer supported learning games. Paper presented at the 26th International Conference on System Research, Informatics & Cybernetics, Baden Baden, Germany. Winner of the Outstanding Scholarly Contribution Award (Draft version)

Martínez García, D., Craig, P., Roa-Seiler, N. and Benítez Saucedo, A., “Validación de una estrategia de interacción de un agente corpóreo conversacional a través de la técnica del mago de Oz,” presented at the MexIHC, Mexico City, Mexico, 2012. ACM ISBN: 978-1-4503-1659-0
draft meihc_22