THE PENNY DROPS
REIMAGINE NEWSLETTER: ISSUE 6
Reimagine Education is an initiative that constantly reiterates the need to reimagine teaching, learning, and the way in which educational institutions prepare students for the employment market. Those two words – teaching and learning – feature at the heart of our conversations, project submissions, and the plenaries and panels that occur at December’s conference. Each year, we are fortunate enough to receive a global overview of how educators are trying to transform these two interlinked processes.
This innovation comes in myriad forms and types. One of last year’s most inspiring panels at the 2016 Reimagine Education conference was that which focused on alternative university models, where the innovation is fundamental and radical. Ben Nelson and his Minerva Project seek to take students away from a single fixed campus and turn the university experience into a global tour. In similarly iconoclastic vein, Brittany Bir and 42 have dispensed with lectures, homework, and, perhaps most strikingly, formal exams.
For those – we presume the majority of readers – that have passed through the traditional educational system, this represents an extraordinary change. Education may well be a process – as we are increasingly becoming aware, a lifelong one – but it is a process that promises outcomes, and one of those outcomes is, typically, a quantitative measure of attainment, whether in the form of a letter-based or numerical grade.
This form of evaluating the quality of the teaching process has received its criticisms. Some point to the essential reductionism of the examination process, questioning the validity of the way that student learning is assessed. Others argue, perhaps less convincingly, that they place undue pressure on students. Less dubiously, teachers point out that the need for continuously evaluating student learning places immense time pressures on them, quite apart from any concerns about the utility of those evaluations.
Others express concerns that traditional forms of assessment are measuring forms of intelligence and/or competencies that are no longer relevant. Among them are the academics behind the Assessment and Teaching of 21st Century Skills Project (ATC21S). The international initiative, based at the University of Melbourne in Australia, is designed to ensure that students both acquire modern, hard-to-assess, ‘twenty-first century’ skills.
During term time the workload can be relentless – planning, marking, behaviour management, observations, parents’ evening and data tracking
Rachel Whyld, Teacher
Though Bir and her coding school have circumvented all of these problems by simply removing examinations from their educational process, there is no indication that 42’s model will become a norm for the foreseeable future. Traditional assessment is here to stay, and therefore any initiative that deals with any of the above issues (system efficiency, construct validity, reliability, the teaching and assessment of hard-to-assess twenty-first century skills) will be a welcome submission to Reimagine Education
Digiexam
Digiexam is one such project. Its goal is the improvement of the efficiency of the assessment process, taking the form of a platform that allows examinations to be taken and assessed in digital format.
The benefits of such a platform, rendered successfully, are clear. When an educator first encounters the Digiexam webpage, they’re asked three questions: (1) the number of classes that they teach each year, (2) the average number of students that they teach in each of those classes, and (3) how many tests they administer per year.
As one can imagine, this information allows the DigiExam platform to speedily make clear how many hours are lost to marking tests each year. In a world where teacher attrition is high and burnout a key contributory reason – almost a third of teachers quit within five years of qualifying in the UK state sector – the value of a platform that streamlines this process is clear. DigiExam promise as much, pledging that educators save up to half of the time that they spend assessing papers.
The value of DigiExam’s platform isn’t limited to the K12 and K5 sectors, either. Academics, torn between publish-and-perish culture on one hand and the imperative to deliver ‘world-class’ teaching on the other, are, increasingly, equally stretched. Though reducing grading time is by no means a panacea for these problems, an effective solution that achieves this would undoubtedly be of value.
DigiExam: the future of assessment?
And DigiExam, one of 2016’s Reimagine Education finalists in 2016 and the deserved winner of the Learning Assessment category, certainly is effective. Founded in 2011 by Johan Hägglund and Nima Marefat while the former was a student at the Stockholm School of Economics frustrated by taking exams, it’s grown to raise more than $3.5 million in venture capital. It’s facilitated more than 580,000 examinations, and has been adopted by 2300 schools at the time of going to print.
This is a testament to its intuitive interface, and the thoroughness with which its creators constructed the platform. Though predominantly cloud-based, DigiExam allows students to work both online and offline, preventing problems arising as a result of limited connectivity. The platform allows teachers to collate group results with ease. And – as the company themselves boast – the platform has value for those concerned about sustainability.
When evaluating DigiExam’s submission, these figures were impressive, but DigiExam is a strong example of another key Reimagine Education criterion: scalability. In speaking to Venturebeat last year, Joen Bonnier (owner of one of Sweden’s largest media companies) spoke of the inevitability of the move to digital examinations. Digitization will affect the assessment process as much as any other aspect of education, and DigiExam are well-placed to be at the heart of that process.
It’s not whether schools will move to digital assessments, it’s when they do it and how.
Joen Bonnier, Owner, Bonnier Group
AI and other possibilities
Or will they? DigiExam’s success is deserved, and its effectiveness unquestioned. However, it is possible to argue that the platform represents only the tip of the iceberg as the world seeks to reimagine assessment.
For example: the DigiExam platform permits both formative and score-based assessment. It allows teachers to administer multiple-choice exams, which its system is configured to facilitate with ease. It’s conscious of the potential for cheating, and its “Student computer lockdown” mode prevents students from accessing anything but the DigiExam app for the duration of the exam.
However, these are all questions of practice – not of theory; and the theory behind what assessment is designed to do, as mentioned at the start of this piece, is changing. The Assessment and Qualifications Alliance (AQA)’s “The Future of Assessment: 2025 and Beyond” paper (Ahmed et al, 2015) is designed to provide an overview of some of the challenges facing those seeking to prepare students for the modern world – and directions that assessment might take as a result. The paper provides a useful framework by which DigiExam might be evaluated.
The Future of Assessment: 2025 and Beyond
The report is at pains to point out that there is a need for the assessment process to become better-aligned with teaching and learning practice, and, in particular, to facilitate the evaluation of twenty-first century skills. The report focuses on a number of these, including collaborative problem-solving, group discussions, and other skills termed ‘hard-to-assess’. It seeks to provide an overview of, and some suggestions about, how these skills might be assessed. In doing so, it also serves as a useful tool for those seeking to consider how digital assessment and “presence assessment” might interact in the future.
By way of example, Ayesha Ahmed, one of the report’s compilers, notes that the OECD recently determined that collaborative-problem solving is one of the skills that it wishes its flagship PISA tests to examine. Ahmed explains how they go about assessing this competency: “In these tests the test-taker and a computer agent communicate via a chat box. The test-taker is assessed on his or her actions, communications with the agent and the products of the collaboration (solution to a problem).” The OECD, therefore, like DigiExam, have provided a digital solution to assessment problems.
As Ahmed notes, this form of assessment has its advantages. It reduces “concerns of reliability and fairness”: a computer does not suffer from assessor fatigue or boredom, nor is it inclined to exhibit bias or favouritism.
However, it is not clear that this form of assessment exhibits the desired level of ecological validity: Ahmed notes that the OECD’s procedure “ignores the critical social element of group work”. The student partaking in this assessment does not need to navigate the unique dynamics of a particular group, nor do they need to exhibit the ability to pick up on tone, mood. The ‘group’ modelled by this simulation does not require the student to recognize and overcome the biases that result when a working group is one with established hierarchies and personalities. It is a limited way of measuring a student’s ability to collaboratively solve problems.
Assessing group discussions
Similar problems face those who might seek to use technology to evaluate another of Ahmed et al’s ‘hard-to-assess’ skills: group discussions. As the ability to improve interdisciplinary, collaborative work becomes an increasingly desirable component of the modern worker (and researcher)’s skillset, assessment procedures need to be set up in a way that allows employers and universities to evaluate how well an individual functions in group discussions.
Yet, again, it is difficult to see how digital assessment might function here. Reasonably sophisticated voice recognition technology might permit, for example, a digital examination platform to assess how frequently a given individual contributed to a group session.
Yet this would again be an oversimplification of the construct under assessment, saying nothing about the quality of contribution. Nor would it allow the assessor to acknowledge that the term ‘quality of contribution’ is a multi-faceted one. A quality contribution is certainly one that takes the group closer to solving the problem at hand – a missing piece of evidence, or well-placed suggestion.
In addition, group discussions are unstructured and often emotionally-charged: the contributor that reminds the group that they are finding their way onto a tangent or calms two antagonistic parties down is also making a ‘quality contribution’. Though AI technology might one day reach the level of sophistication necessary to account for all of these issues, it is certainly unable to do so currently – and human assessors are still uniquely able to evaluate these constructs.
Measuring hard-to-assess skills
If there is one takeaway from the OECD’s experiment, then, it is that a full transition to digital assessment may be – at least in part, and temporarily – stymied as soon as it has begun. If the theoretical trend is towards constructing assessments that are able to assess complex skills in complex contexts – the skills that the modern worker needs – digital assessment will remain, for the time being, a step behind.
One might argue that this is only temporarily to be the case. AI will doubtless continue to advance, and, as human educators get better at defining the constructs that they seek to assess, the finest educational technologists of the age will be, progressively, better-able to set up technology that can measure performance according to those constructs.
Nor is this meant to detract from the excellence of DigiExam. DigiExam remains a highly-effective IT tool that allows teachers to create, administer, and mark exams in the fraction of the time that it previously took to do so. It is designed to revolutionise current practice, and it received its award as the result of our judges believing that it is able to do so.
Rather, this piece is designed to make clear how much scope there is for the ed tech industry to innovate in the field of assessment. If Bonnier’s belief that the future of assessment is digital has the quality of truth, Ahmed et al.’s paper makes it clear that it is only a partial truth – at least for the foreseeable future.
To make that future a present for the world’s students and educators, the type of collaboration that Reimagine Education is designed to foster is necessary. Employers must continue to identify the skills that they believe that their workforce will need, and must liaise with the academic community to define both precisely what those skills are, and how they might be measured.
The ed tech community – both within academia and outside – will then be empowered to use the talent at their disposal to ensure the vision that Ahmed et al., and DigiExam, articulate becomes a reality: valid, reliable assessment of complex skills, in a format that reduces the administrative and temporal burdens on the educators that seek to teach those very skills.
Jack Moran
Reimagine Education
03.05.2017
To find out more about DigiExam, join the “DigiExam: Digital Assessments Made Easy” webinar to discuss strategies for navigating and selecting digital assessment tools, hear about the benefits of digital assessments, and learn how to easily create, administer and grade digital assessments using DigiExam.