Designing and Developing Accessible and AI-enabled Learning Platforms that Improve the Outcomes of Children and Adults at Risk of Exclusion

Access to appropriate ICT-based solutions provides perhaps the only chance for students with a range of physical, sensory, communication and cognitive disabilities of participating in society and realising their full potential. A 2015 Deloitte and Ipsos MORI study prepared for the European Commission highlighted the need to integrate inclusive, personalised ICT solutions into learning environments for children.

Research by NTU’s Interactive Systems Research Group has improved the inclusion of students with learning disabilities and autism in special and inclusive classrooms via approaches to designing accessible and adaptive learning technologies, which improve and enhance their engagement in learning. These are incorporated into:

• An award-winning educational coding app (Pocket Code) that has been downloaded more than one million times across 180 countries.
• A dedicated version of Pocket Code with embedded accessibility features for formal learning environments (Create@School) with over 10,000 installs.
• Adaptive learning platforms (including MaTHiSiS and Pathway+) that have supported inclusive teaching practice, and improved engagement, achievement, and behaviour for students with learning disabilities and autism in schools in Spain, Italy, Bulgaria, Turkey, Belgium, and the UK.
• An app (BuddyConnectTM) to support the mental wellbeing of those experiencing workplace anxiety being developed by a world-leading IT services provider.

Our research in multimodal affect recognition has more recently contributed to the innovation and entrepreneurial activity of a world leading IT Services provider. In 2019 Fujitsu, who support 7.5 million end users globally, began to integrate NTU algorithms to infer workplace anxiety into its services to support workers operating in highly stressful situations. The ‘BuddyConnectTM’ App is under development. This uses Heart Rate and Heart Rate Variability sensor data and machine learning to infer anxiety, then launches recommendations for evidence based psychological interventions when the need arises for improved workplace mental wellbeing. Fujitsu has invested £50,000 in the development of this App

Led by Brown, research at NTU cutting across multimodal affect recognition, serious games, location based services, digital game-making, robotics and accessibility has focused on using Artificial Intelligence (AI) to understand how engaged learners are, and how best to target novel ICT interventions at pupils in the greatest need. The overarching aim is to ensure that everyone, especially those with specialised learning needs, has an opportunity to reach their full potential through the support of personalised digital learning platforms.

The development of guidelines, requirements, and user models to promote the accessibility of web content and interactive media by people with a range of cognitive, physical, and sensory impairments is a cross-cutting theme at NTU. NTU was one of two universities in the EU FP7 project AEGIS – Open Accessibility Everywhere (2008-2012), using its research experience in promoting accessibility of people with learning disabilities. An Open Accessibility Framework (OAF) was developed to embed accessibility solutions into mainstream ICT products. The aim was to place users and their needs at the centre of all ICT developments by making accessibility open, personalised, and configurable. Working within special schools in Nottingham, Brown led the development of ‘digital personas’ and accessibility guidelines to help ICT designers understand the needs of those with accessibility requirements and posited that their adoption has the potential to improve accessibility for all. This approach was subsequently extended to making digital game-making tools more accessible for students with learning disabilities.

Brown led a programme of research, supported by seven EU grants totalling £1.5M between 2006 and 2016, into serious games or games-based learning, demonstrating that their use can have a positive effect on deficits in decision-making skills that can otherwise hinder the inclusion of students with learning disabilities in society. Through EU project Game On Extra Time (EU Leonardo Transfer of Innovation Project, 2008-2010, coordinated by Brown), researchers developed 10 serious games to support people with learning disabilities and additional sensory impairments to obtain, and retain, a job. The games helped participants to learn skills that would assist them in their working day. The RECALL project, led by NTU, came in response to people with learning disabilities being excluded from learning opportunities and community events because they cannot travel independently. Brown’s group demonstrated the viability of combining games-based learning with location-based services to help young adults with intellectual disabilities plan travel routes within their local community facilitating the development of cognitive maps. In another EU project led by NTU, researchers found that engagement of children with intellectual disabilities can be significantly improved using programmable humanoid robots.

The combined research outlined above shaped the design and delivery of two key EU Horizon 2020 projects, the No One Left Behind project and MaTHiSiS. The No One Left Behind project involved a three-country controlled study that explored the benefits of using digital game-making tools within a formal learning environment. Children used a gaming toolkit called Pocket Code, which allows users to create, play and share games in a LEGO-style programming environment, to develop inclusive games on mobile devices. Brown led the UK element of the study involving 200 children with a range of learning disabilities, which found that digital game-making significantly improves the engagement and collaborative behaviours of students in special educational schools.

Sustainable learning in students with learning disabilities and autism can only occur when there is meaningful engagement, which, in turn, can help reduce stress and anxiety. The MaTHiSiS project (EU H2020; 18 partners from 9 EU countries) developed a novel AI-enabled learning platform capable of monitoring and responding to an individual learner’s mood and behaviour to provide a personalised learning experience. As part of a wider consortium coordinated by IT firm Atos Spain, Brown’s group demonstrated that engagement does increase when activities are tailored to the personal needs and emotional state of students with learning disabilities and autism using multimodal affect recognition, and that these affect-sensitive algorithms can also be used to infer anxiety.

The user experience and educational inclusion of students with learning, physical and sensory impairments have improved via research in accessibility

Through the No One Left Behind project, NTU researchers led by Brown worked with children with learning disabilities to build a detailed understanding of their needs in using the digital game-making app Pocket Code. As well as embedding accessibility features in Pocket Code itself, the educational potential of the Pocket Code platform was exploited by developing a dedicated version for formal learning environments called Create@School. Targeted at children aged 9 to 18, this app was designed to enable teachers to integrate gamification into the classroom, leading to improved academic and behavioural outcomes for students with learning disabilities. Based on the research findings, Brown’s group developed novel accessibility features for the Create@School app.

These included the option to create bespoke profiles for users with cognitive, physical and visual impairments, together with a range of other accessibility features such as: larger text, high contrast, additional icons, large icons, large spacing, drag´n´drop delay and Beginner Mode. Designing for disability commonly improves accessibility for a wider population and Catrobat (the app developer), also embedded the new accessibility features into the main Pocket Code app in 2016.

Academic and behavioural outcomes for students with learning disabilities have improved, and educational practices have changed via our research in digital game making, serious games, location-based services, robotics, and multimodal affect recognition

Both the Create@School and Pocket Code apps have provided teachers of students of all abilities with support in teaching coding and computational thinking. The No One Left Behind project created a game-making teaching framework comprising a three-step methodology to integrate the apps into classroom practice. The provision of personalised learning experiences through NTU’s research contributions in multimodal affect recognition in the MaTHiSiS platform resulted in higher levels of student engagement and improved learning outcomes. In collaboration with Nottingham Schools’ Trust, NTU has identified areas for the improvement of educational outcomes for all pupils by exploring the adoption of alternative and new ways of working using AI-enabled adaptive learning platforms, serious games and social robotics to improve the inclusion, and academic and behavioural outcomes for students with a range of learning disabilities.

• (Accessibility) Evett L and Brown DJ (2005), Text formats and web design for visually impaired and dyslexic readers—Clear Text for All. Interacting with Computers 17, pp.453-472, https://doi.org/10.1016/j.intcom.2005.04.001
• (Digital Game Making) Hughes-Roberts T, Brown DJ, Boulton H, Burton A, Shopland N and Martinovs D (2020), Examining the Potential Impact of Digital Game Making in Curricula Based Teaching: Initial Observations. Computers and Education. https://doi.org/10.1016/j.compedu.2020.103988
• (Serious Games) Standen PJ, Rees F, Brown DJ (2009), Effect of playing computer games on decision making in people with intellectual disabilities. Journal of Assistive Technologies. 3(2), pp.4-12, https://doi.org/10.1108/17549450200900011.
• (Location Based Services) Brown DJ, McHugh D, Standen P, Evett L, Shopland N and Battersby S (2011), Designing Location based Learning Experiences for People with Intellectual Disabilities and Additional Sensory Impairments. Computers and Education, 56(1), pp.11-20. ISSN 0360-1315. http://dx.doi.org/10.1016/j.compedu.2010.04.014.
• (Robotics) Hughes-Roberts T, Brown DJ, Standen P, Desideri L, Negrini M, Rouame A, Malavasi M, Wager G and Hasson C (2018), Examining Engagement and Achievement in Learners with Individual Needs through Robotic-Based Teaching Sessions. British Journal of Educational Technology. https://doi.org/10.1111/bjet.12722
• (Multimodal Affect Recognition) Standen PJ, Brown DJ, Shopland N, Burton A, Taheri M and Boulton H (2020). An evaluation of an adaptive learning system based on multimodal affect recognition for learners with intellectual disabilities. British Journal of Educational Technology Special section on AI and Deep Learning in Education Technology Research and Practice. https://doi.org/10.1111/bjet.13010