Multimodal Learning Analytics

Set of three aligned plots labeled a) gaze auto-recurrence rate, b) hand cross recurrence rate, and c) hand positions. The graph are all split into three segments labelled Exploration, Discovery, and Fluency. The first graph shows a low level of gaze autorecurrence until right before fluency. The second graph shows high hand cross recurrence initially, which drops during the discovery phase and spikes again right before fluency. The third graph shows left hand height in red and right hand height In blue. The hands move together at first, then explore varied relationships. Towards the end of Discovery, they start to repeat a pattern. In fluency, they slowly move together with the blue line at about half the height of the red. The graph also shows when the user received a certain kind of feedback, labeled green feedback here. The green feedback increases during Discovery and becomes constant during Fluency.

Embodied cognitive science suggests that our bodily activity is part of our cognitive dynamics. I work with multimodal data (including movement and eye tracking, speech, video, electrodermal activity, sensor and touchscreen data) to empirically test this view. I draw in nonlinear methods like Recurrence Quantification Analysis, new to education research, that can capture the complex dynamics of learning data. I apply these methods to study changing coordinations of bodily and ecological resources as learning. For example, I study how shifts in hand movement coordination precede conceptual breakthroughs, and how changing patterns of eye movement reflect new ways of perceiving and consequently, thinking.

Key Publications

Abdu, R., Tancredi, S., Abrahamson, D., & Balasubramaniam, R. (2023). A complex systems outlook on hand-eye coordination in mathematical learning. In M. Schindler, A. Shvarts, & A. Lilienthal. (Eds.), Eye-tracking research in mathematics education [Special issue]. Educational Studies in Mathematics.

abdu-et-al.-2023Download

Tancredi, S., Abdu, R., Balasubramaniam, R., & Abrahamson, D. (2022). Intermodality in multimodal learning analytics for cognitive theory development: A case from embodied design for mathematics learning. In M. Giannakos, D. Spikol, D. Di Mitri, K. Sharma, X. Ochoa, & R. Hammad (Eds.), Multimodal learning analytics. Springer.

tancredi_abdu_balasubramaniam_abrahamson-2022-intermodality-in-multimodal-learning-analytics-for-cognitive-theory-development-a-case-from-embodied-design-for-mathematics-learning_mmla_haDownload

Tancredi, S., Abdu, R., Abrahamson, D., & Balasubramaniam, R. (2021, 2021/04/03/). Modeling nonlinear dynamics of fluency development in an embodied-design mathematics learning environment with Recurrence Quantification Analysis. International Journal of Child-Computer Interaction, 100297. https://doi.org/10.1016/j.ijcci.2021.100297

tancredi-abdu-abrahamson-balasubramaniam-ijcci-2021Download