Improving Learning through Data Standards for Educational Technologies
The surge in education technology use in response to COVID-19 represents a massive natural experiment: an opportunity to learn what works at scale, for which students, and under which conditions. However, without the right data standards in place we risk incomplete or inaccurate inferences from this experiment.
The COVID-19 pandemic has dramatically increased use of educational technologies. There is evidence that this “emergency onlining” will lead to learning loss, especially among underserved communities. To understand and address the extent of learning loss—as well as to explore and support potential future uses of educational technologies—the U.S. Department of Education (ED) must systematically implement established open-data standards that allow us to understand how students engage with learning technologies. Widescale implementation of these standards will make it possible to combine and analyze validated data sets generated by multiple technologies. This in turn will provide unprecedented, on-demand reporting and research capabilities that can be used to precisely identify gaps and create targeted interventions. Specifically, we recommend that ED mandate the use of the open Experience API (xAPI) standard for educational technology purchased with federal funds. We further recommend that ED invest time, talent, and resources to further develop this standard and pilot efforts to leverage educational-technology data for insights through the Institute for Education Sciences (IES) and other agencies.
Challenge and Opportunity
The COVID-19 pandemic rapidly forced schools across the country to close physical campuses and convert all instruction to an “emergency online” modality for much of 2020. The situation will likely persist well into 2021. The emergency shift to online teaching meant that many teachers had insufficient preparation to successfully adapt classroom-teaching methods for digital formats. Moreover, many students—especially those from low-income families or from historically underserved racial and ethnic groups—lack access to high-speed broadband and technology assets needed to fully participate in online learning. These factors are combining to create learning losses that exacerbate our existing digital divides that may persist for years.
Robust educational research and development is needed to fully understand the extent and distribution of learning loss, as well as to develop interventions for addressing it. Educational technologies—which record all student interactions, from logins to mouse-clicks to assignment submissions—could provide a wealth of data on how online education is succeeding and/or falling short. Unfortunately, these data are frequently recorded in a way that is unique to each application. This lack of consistency makes it difficult to integrate educational data or make comparisons between institutions
The time is ripe to introduce new requirements for learning technology designed to ensure that parents, educators, administrators, and stakeholders at every level can assess where students are at, what they know, and what will best help them to advance. These insights could also significantly reduce the day-to-day demands on teachers’ time and attention, enabling them to focus on deeper student questions. The technology needed to implement such requirements are already available in the open-source xAPI standard, which is currently in the final stages of approval as an IEEE standard. Further, there are xAPI “profiles” that define specific data requirements for processes common to educational technologies, such as playing a video. While the concept of a learning-data standard was recommended by ED as early as 2015, adoption has been uneven in practice. This situation must change for us to immediately address COVID19 learning loss as quickly and accurately as possible.
Plan of Action
To address the challenges outlined above, we recommend including xAPI as a federal procurement requirement to encourage adoption among educational software and service providers. Widespread adoption will mean that most—if not ultimately all—providers consistently and automatically generate only the educational data that conforms to standards established by ED. Establishing consistent standards for educational data will make it easier for all parties to contribute meaningfully to key datasets, and for researchers to develop tools to track and exchange meaningful data. These outcomes together will deliver deeper understandings of how our nation’s students are doing, inform efforts to close achievement gaps, and facilitate tracking of changes over time. We also recommend investing ED time, talent, and resources into further developing the xAPI standard and participating in pilot projects that demonstrate its utility. Each of these recommendations is detailed further below.
Recommendation 1. Mandate use of the xAPI standard for ED-funded procurement.
We recommend that ED mandate use of xAPI for all educational technology purchased through ED directly as well as through federal grants. ED should also establish a process for ensuring compliance, including conducting conformance tests on educational software and services from different providers.
The IEEE is in the final stages of publishing the open-source xAPI standard. Mandating its adoption would demonstrate cutting-edge ED leadership. Widespread adoption of the standard will provide a common approach to collecting evidence about how students, parents, and teachers interact with education platforms, paving the way for much more rigorous, consistent, and reliable educational research.
Recommendation 2. Develop xAPI profiles to facilitate data integration and improve data quality for the educational sector.
IEEE is standardizing documents that help automate the data governance needs for a type of educational solution (“application profiles”). Standards developers are rarely familiar with learning sciences and educational research. As a result, xAPI profiles will tend to be general in nature unless domain-specific experts get involved. For instance, medical experts have worked to develop the MedBiquitious xAPI Profiles for Medical Education. Human-resources experts have developed the Human Resources Open Standards (HROS) xAPI Profile and, and work is ongoing for an Assessment xAPI Profile that supports the U.S. Chamber of Commerce T3 initiative.
ED should invest time, talent, and funding to develop xAPI profiles that are aligned with current research and national priorities. An xAPI Profile effort could help to normalize data collection from a spate of popular 5th grade mobile math applications, that properly identify the relevant ED standards, competencies or objectives are challenged by a student, which could provide such app developers with the automation that would simplify generating better, aligned data. Works like this could change online classrooms into opportunities to embed better pedagogy into practice at scale.
Recommendation 3. Invest in applied research and development.
ED should partner with schools and educational technology companies to invest in applied research that demonstrates insights from standardized educational data. ED should also work with partners to invest in public repositories of code to make it easier for all stakeholders to leverage insights. Such investments should focus both on the short term (e.g., providing immediate insights about use of educational technology and learning loss during the COVID-19 pandemic) and long term (e.g., providing examples of potential applications that could be scaled and replicated in the future). Such investments would not only advance our understanding of education, but would also help to develop a market for further development of data-based educational products. IES and ED’s Office of Educational Technology should partner to identify topics and approaches to conduct this cutting-edge research.
There are multiple examples of research using educational-process data that these investments could build on. IES recently issued a request for proposals (RFP) to use National Assessment of Educational Progress (NAEP) process data to identify students with disabilities, to understand how those student use available accommodations, and to determine which are most successful. Predictive models of student dropout risk, course design analytics to identify areas for improvement, and course-taking patterns are all being conducting using this data at relatively small scale through academic societies, such as the Society for Learning Analytics Research (SoLAR) and the International Educational Data Mining Society. All stand to benefit and leverage this data to dramatically improve research.
SoLAR recently published a position paper describing current challenges with these data.4 Larger educational-research societies such as the American Educational Research Association (AERA) and the National Council on Measurement in Education (NCME) have launched specialized groups focused on working with educational-process data.
By investing in this area, ED could help to nurture this area of research and make a difference in the lives of students, parents, teachers and schools across the country. This approach would help to motivate better data quality and enable technologists to build more robust learning applications, thereby helping us to stem the COVID-19 learning loss as quickly as possible using contemporary science and technology.
As cyber threats grow more complex and sophisticated, the nation’s ability to defend itself depends on developing a robust, adaptable, and highly skilled cybersecurity workforce.
For the United States to continue to be a competitive global power in technology and innovation, we need a workforce that understands how to use, apply, and develop new innovations using AI and Data Science.
Students, families and communities want and need more STEM learning experiences to realize the American Dream, and yet they cannot access them. Prioritizing STEM education must be an urgent priority for the federal government and the Department of Education.
The Department of Education must provide guidance for education decision-makers to evaluate AI solutions during procurement, to support EdTech developers to mitigate bias in their applications, and to develop new fairness methods.