Australian Online Training Best Practice Guide for Regulators and RTO’s

Australian Online Training Best Practice Guide for Regulators and RTO’s – White Paper

Aust Online Training Best Practice Integrity Measures V6 .PDF 

Foreword

This white paper is intended to provide, policy writers, authors of units of competency, regulators and online training providers with integrity measures that must be in place to ensure the integrity of online learning outcomes.

 

In the absence of requirements for online integrity in either; the Standards for RTO’s 2015 or, units of competency, It is essential that regulators, policy writers and authors of units of competency, understand the importance of both the granularity of online training data and integrity measure features, so that they can specify them as mandatory requirements, improving integrity and the ability of the regular to act.

 

Contents

Australian Accredited online training Best practice integrity measures –  WHITE PAPER.. 1

Foreword. 1

Executive Summary. 2

The Future of Online Learning. 4

Benefits of high integrity online training. 4

High integrity online training costs. 4

Online Integrity Measures. 5

  1. Confirming Student Identity. 5

1.1.        Valid Photo Identification. 6

1.2.        Unique Student Identification. 6

1.3.        Website with HTTPS Security. 6

1.4.        Unique username. 6

1.5     Secure Password. 6

1.6     Identification during phone support 7

1.7     Terms and Conditions. 7

2        Assessing Student Competency. 7

2.1     Free text 7

2.2     Video Demonstration – 7

2.3     Lock out – 7

2.4     Multi choice wrong ratio. 7

2.5     Not given answer – 7

2.6     No direct answer link  – 8

2.7     Random questions. 8

2.8     Question banks. 8

3        Signing Off of Student Competency. 8

3.1     Use of qualified trainers – 8

3.2     Trainer Roster. 8

3.3     Trainer availability. 8

3.4     Human validation. 8

4        Minimising the Potential for Fraudulent Activity. 8

4.1     Capture the student’s IP address – 8

4.2     A system for monitoring cheating. 8

4.3     Student name fields must be locked. 8

4.5     Must be able to intervene. 9

4.6     Warn about fraudulent 9

4.7     Ability to report suspicious activity. 9

4.8     Deliver training via SSL.. 9

4.9     Agree to terms and conditions. 9

4.10       Statutory Declaration. 9

5        Course Content 9

5.1     Volume of Learning. 9

5.1.1      Minimum learning time. 10

5.1.2      LMS capability to manage time. 10

6        Issuing Statements of Attainment 10

6.1     No Name change. 10

6.2     Unique ID.. 10

6.3     Watermark. 10

6.4     Locked PDF.. 10

7        Learning Management System Data Capture Requirements. 10

7.2     Number of attempts. 10

7.3     Result of all attempts. 10

 

Executive Summary

Technological advancement is occurring in the Vocational Education and Training (VET) sector ahead of guidelines, legislation and quality control mechanisms by industry, government and regulators, to ensure that there is adequate integrity underpinning statements of attainment issued via an online mode of delivery.  In some cases this is leading to low cost and low quality, online learning outcomes as well as giving online training a poor reputation.

 

High-integrity online training has much to offer the training sector including; providing better access for remote and regional communities, and better flexibility for students, who may be working, studying or parents at home. In many cases high integrity online training can provide better learning outcomes through the use of engaging multi-media technologies that can be studied at the learner’s own pace.  High integrity online training can also improve evidence of assessment and demonstrate high compliance to regulations.

 

To ensure that learning outcomes are achieved in an online environment, certain technology with the capability to deliver minimum integrity measures, as outlined in this white paper, must be in place.  Specific online Integrity measures also need to be in place to reduce fraudulent activity.

 

Whilst examples of industry best practice guidelines for online training do exist, many training providers are either not aware of them or, in some cases, are deliberately not implementing them to cut costs and compete with the lowest cost providers in the market.  There are examples of regulators (e.g. state and territory liquor authorities) stipulating mandatory online integrity measures however, these requirements are often not enforced by the Australian Skills Quality Authority (ASQA) due to lack of resources or technical understanding.

 

Under the Federal Register of Legislation the Standards for Registered Training Organisations (RTOs) 2015 do not provide adequate guidance to ensure that online training has high integrity or effective learning outcomes.

 

Many employers have experienced lack of knowledge and poor skills form students who have gained qualifications from low-integrity, low-cost online training and may mistrust all online qualifications.

 

Despite advancement in technology such as virtual reality or simulated environments, rather than adding online integrity measures as a requirement to units of competency (i.e. to encourage high integrity online training),  some of the Industry Skills Councils have actively added requirements into units of competency to try to exclude the online mode of delivery.  The industry representatives on these skills councils may have limited understanding of online technologies and the need for high integrity measures.  This results in units of competency that can easily be delivered by low integrity providers who deliver poor learning outcomes.

 

Armed with an out of date Standard for Registered Training Organisations (RTO) 2015 and units of competency that do not specify online integrity measures, the Vocational Education and Training (VET) regulator, the Australian Skills Quality Authority (ASQA), is often unable to act against unscrupulous profiteering providers delivering low integrity online training.

 

Training providers that are cutting corners with online training are leading educators in a race to the bottom in terms of poor quality of training and assessment, and ultimately undermining the student’s value for money.  The result of lack of integrity is not only poor learning outcomes and inadequate training, leading to inefficiencies or dangerous practices in the workplace, but also damage to the reputation of the online or the ‘e-learning’ industry as a whole.

 

A small number of unscrupulous providers have taken over approximately 80% of some online training markets and are unfairly making it uneconomic for high-integrity and higher-cost training providers to compete.  Many online training providers that are not complying with best practice online integrity measures are producing what is known as ’page turning’ or ‘tick and flick’ learning with assessments such as true/ false or multiple choice that can be passed by guessing without restriction.  Based on the scope of some units of competency this conduct is endangering the lives of patrons and people in the community.

 

To bring to the fore the advantages of online training, and eliminate the risks from low integrity online training, policy writers, regulators, and authors of units of competency must ensure online integrity measures are specified in order to allow the regulator to act.

 

The Future of Online Learning

Automation in the training profession is continuing to advance at an ever-increasing speed. Examples include interactive videos, computer auto marking, interactive video assessment and virtual reality scenarios for the demonstration of skills.  The experience API is also allowing online training to move beyond the boundaries of formally structured content development and the classroom.

 

Many qualifications that are taught using sophisticated online training methods can already deliver the same, if not better learning outcomes than in-class training.  Since online training innovation is growing at an exponential rate it is advantageous if training providers are allowed to author their own learning materials to keep up with these changes.

 

Benefits of high integrity online training

The benefits of high integrity online training include but are not limited to;

  • Improved accessibility, particularly for students in remote or regional areas or parents with young children.
  • Students can learn in their own time and do not have to take time off work or study.
  • Reduced travel costs such as transport and parking during training and also reduced carbon emissions.
  • Greater consistency in terms of content delivered and the assessment method.
  • Quicker response times to updating learning content resulting in more up to date learning resources.
  • More empirical data can be collected for demonstrating competency and adherence to high integrity.
  • Reduced administrative and non-value-adding work such as marking quizzes or issuing certificates.
  • Less monotonous tasks for trainers and more time spent with students that need their assistance.
  • Reduced human error related to less reliance on monotonous and repetitive tasks.
  • Learners can work at their own pace and concentrate on areas they do not understand.
  • Students can spend as much time with qualified trainers as they need.
  • Students can access learning material ongoing and can also be provided with updates to the learning material.
  • Reduced room hire costs which are however, offset by online tools, online delivery, technical support and online course content development.

 

High integrity online training costs

There is a perception that online training is a way for training providers to cut costs such as eliminating trainers and class rooms and automating everything with a computer to increase profits.  However, if a training organisation is meeting the high integrity online measures outlined in this white paper, despite the reduced cost of hiring classrooms, the cost of delivering effective online training is equal to if not greater than in-class training.  This is because not only do all of the existing requirements that need to be in place for in-class training such as; qualified trainers and assessors, assessment material, mapping, internal audits, continuous improvement registers and so on, still need to be in place for online, but in addition to these costs, online providers need to purchase and maintain an online learning environment and build digital online learning material and robust assessments.

 

Whilst a 200 page PDF document available online and a quiz requiring a student to tick a few true / false questions meets the Standards for RTOS’ 2015 and most units of competency, effective and engaging online training includes videos, gamification, interactive scenarios, simulated environments and animations with voice overs and challenging exercises as well as additional reading materials or offline activities. If online training contains these kinds of elements then it can cost up to AUD$30,000 per hour of e-learning output.  Compare that to a power point presentation created for in class training and we are talking $180,000 vs a few thousand for a six hour course.

 

Many “online providers” are cutting corners by developing simple e-books or a PDF with a voice over and a next button between pages.  Whist it is possible to map such “online” material to the elements and criteria of a unit of a training package, there needs to be more emphasis on being able to demonstrate that a student can achieve competency from such material through rigorously linking it to high integrity online assessments.

 

Regulators that attempt to provide learning material to training providers may face increased costs and struggle to keep up with the rapidly evolving innovation.

 

An alternate approach that regulators may consider is to provide elements such as videos or interactive content that may be included by an online training provider as part of a course.  A good example of this is the “Behind the bar RSA interactive training video – Just one more” authored by the Office of Liquor and Gaming Regulation Queensland, who invite training providers to use the content in their courses royalty free.

 

From a training perspective, it is essential to realise that to achieve efficient and effective online learning,  that the ability for a training provider to edit content and assessment is inseparable.  This is because assessments are directly related to the learning material and in order to achieve accurate mapping to a unit of competency assessment questions must be written in such a way that may require regular edits to the content.  Furthermore, if an assessment question is causing confusion or returning a high rate of incorrect answers, it may be the case that the content needs to be edited rather than the assessment question.  Another issue that can occur with mandatory content provided by regulators is that if the government bodies are renamed or restructured, often they lose funding to maintain content and this content is then not able to be updated for name or content changes by training providers as they don’t have open source access.  A consideration for regulators providing such content is to provide open source access to authored content so that training providers and modify it as needed.

 

A key take home for regulators is that low sale prices for online courses are a likely reflection of the expenditure put into developing the content and also reflect the lack of real and qualified training resources behind the scenes and should be targeted for audits.

Online Integrity Measures

 

In the absence of any online integrity regulation, where possible, policy writers should add the following as mandatory requirements:

 

1.      Confirming Student Identity

Confirming student identify in an online setting is of paramount importance in reducing fraudulent activity, i.e. by ensuring that the person doing the training is the person obtaining the Statement of Attainment/Certificate.

Fingerprint and retina scans are becoming more prevalent however, until these are more widely accepted the following procedures and features are considered best practice to verify that a student, who is enrolled in an online course, is the person who also completes the course and receives the certification:

 

1.1.   Valid Photo Identification

Students must provide via a secure (https) online website, a copy of current government-issued photo ID which is reviewed and accepted or rejected by trained staff.  Details captured and verified must include the full name, date of birth, and contact details (including residential, valid email address, and contact phone number) of the student.

 

The residential address of the student must be validated.  This can be improved using an online address validation service or address look up tool.

 

Students must not be allowed to change personal details themselves – they need to lodge a request and provide evidence, such as government-issued photo ID, marriage certificate, change of name certificate, etc. where required.

 

For administrators the valid photo ID should be displayed against a student account when videos are being assessed, or during video conferencing so trainers can ascertain that they are speaking with the student who has made the recording.

 

In many circumstances this procedure may exceed identification integrity taking place in classroom training environments.

 

1.2.   Unique Student Identification

In Australia it is a government requirement for nationally recognised training, that before completing their enrolment students must obtain a Unique Student Identifier (USI) and the details must match their valid photo identification provided.  In Australia it is a government requirement that students supply their Unique Student Identifier (USI), the details of which must match their valid photo ID provided, before they can be issued with their Statement of Attainment/Certificate (qualification).

 

  • Website with HTTPS Security
    Students must only access online courses through a https secure website that requires a login and password.

 

1.4.   Unique username

Students must create their own unique username such as an email address

 

1.5  Secure Password

If an overly complex password is provided to the student then they will need to write it down and this reduces the security of the password so students must create a secure password themselves.

 

Students must  be provided with best practice information regarding setting up and maintaining the security of their passwords.

Student password recovery must be via clicking a link to recover their password that sends them details via their registered email. Passwords must not be able to be changed without the student logging into the account.

 

  • Identification during phone support
    When a student contacts a training provider by phone or email requesting help, the provider must have procedures in place to check their identity before providing any account details or assistance.
  • Terms and Conditions
    Students must agree to terms and conditions at the start of an online course, confirming that they are the person doing the training.  This can be done via a statutory declaration or equivalent where required.Once complete and before generating and accessing the final PDF Statement of Attainment/Certificate the student must declare that they were the person that did the training and that they did so without assistance.  This must be recorded in the electronic management system and be able to be provided as evidence for legal breaches.

2        Assessing Student Competency

Assessments must be mapped to a unit of competency to demonstrate that all of the required learning elements have been assessed.

 

In an online environment, assessments that are designed to provide a true reflection of the knowledge and skills that have been achieved are one method of proving competency has been gained.  Offline activities can also be undertaken and witnessed by a suitably qualified supervisor and their accounts included for consideration.

 

Policy writers must mandate that, In an online environment, in order for assessments to demonstrate that a student has achieved understanding and competency, they must have the following integrity measures:

 

  • Free text – Require at least 1 question in each assessment (ideally 25%) to be answered with free text sentences (ie at least 5 words or more, not a single word or number) allowing a proper critical response to questions.
  • Video Demonstration – In the case of demonstration of skills, require video footage to be submitted to capture a student’s ability to demonstrate a task or skill.
  • Lock out – If a student gets the same question wrong three (3) times they must be “locked out” and requiredto talk to the training organisation and if they need assistance they must be referred to a qualified trainer ie rather than keep guessing until they get the question correct through a process of elimination.
  • Multi choice wrong ratio – Multiple choice questions must have more wrong combinations than available attempts and designed so that students will be locked out before guessing through a process of elimination.
  • Not given answer – Students must not be provided with the correct answer eg before or after attempting an assessment before they get the answer correct themselves. If re attempting assessments, students must get the full range of options and not be shown which answers have already been attempted and marked as incorrect.
  • No direct answer link – Students must not be provided with links from the question directly to the correct answers.
  • Random questions – The order of assessments must be randomised to reduce the likelihood of cheat sheets.
  • Question banks -Because of the complexity of meeting mapping requirements if questions are randomised, banks of similar questions must be created and questions randomly selected from those banks to ensure that the particular questions achieve the required mapping.

 

In contrast to assessments that comply with the above high integrity measures, assessments that are 100% marked by a computer that are predominantly multiple choice by elimination or true/false by elimination, provide no requirement or opportunity for a student to demonstrate their comprehension of the learning material. Whilst it can be argued that by eventually getting a question right is a form of learning, allowing students to keep attempting questions until they get the correct answer does not deliver meaningful learning outcomes. Demonstrated competency through rigorous and auditable assessment, in combination with the above integrity measures, is considered an integral requirement for the demonstration of online learning outcomes.

 

3        Signing Off of Student Competency

Online training providers may try to cut corners and costs by using administration staff with no training qualifications to assist with training.  To cut costs training providers may also reduce access by students to qualified trainers to very short periods. It is essential that qualified trainers are available to students in an online environment.  These trainers may communicate with the students via:

  • Telephone
  • Email
  • Live chat
  • Webinars
  • Video conferencing

To improve Integrity measures policy writers must mandate the following:

  • Use of qualified trainers for assisting with training support. All trainers employed by a training organisation  must be on a training register. 
  • Trainer Roster – Total trainer hours must as a minimum reflect availability during normal office hours.
  • Trainer availability Trainers must be available at least during normal business hours and must assist learners requesting training within 30 minutes of a request.
  • Human validation – An online statement of attainment must not be issued until the enrolment details confirmed and assessment and learning activities of the student have been reviewed by a qualified trainer. This sign off process must be demonstrated by data in the learning management system.  

 

4        Minimising the Potential for Fraudulent Activity

Policy writers must ensure that the following integrity measures are in place to minimise the potential for fraudulent activity:

  • Capture the student’s IP addressand monitor if bulk users are coming from the one IP.
  • A system for monitoring cheating must be in place that flags when different students give the exact same free text answers to free text assessment questions.
  • Student name fields must be locked so that these can only be changed by administration staff upon request and subsequent verification of the student’s identification.
  • lock-out’ access –  to an assessment after failing a set number of attempts of each question (3)  and must speak with a qualified trainer to be unlocked if they do not know the answer.
  • Must be able to intervene and make direct contact with the student if required.
  • Warn about fraudulent activity – Notify students that certification may be voided if fraudulent information is provided or fraudulent activity is detected.
  • Ability to report suspicious activity – The learning system must have the ability to identify and report immediately any suspicious activity by students undertaking the course to the appropriate authority.
  • Deliver training via SSL – SSL to reduce third party fraudulent activity ie HTTPS at start of a domain..
  • Agree to terms and conditions – Capture electronic evidence that the student has agreed to terms and conditions relating to not being assisted in any way during assessment and that they are the one that undertook and completed the training.
  • Statutory Declaration – If adequate integrity measures are not in place, or a student has breached an integrity measure, students must sign and upload a witnessed statutory declaration that they are undertaking the training without assistance.

 

5        Course Content

 

5.1  Volume of Learning

The Australian Quality Framework says that “the volume of learning”  includes guided learning, individual study, research, learning activities in the workplace and assessment activities. It could be argued that rather than applying this at a qualification level, the volume of learning should be applied at a unit of competency level and rather than minimum duration the relationship between the quality of the content being able to map to a rigorous assessment should be the deciding factor in assessing competency.  As the AQF goes on to say:

 

“The duration of the delivery of the qualification may vary from the volume of learning specified for the qualification. Providers may offer the qualification in more or less time than the specified volume of learning, provided that delivery arrangements give students sufficient opportunity to achieve the learning outcomes for the qualification type, level and discipline.”

 

This could  be applied to a competency rather than the whole qualification.

 

When looking at VET training, we need to remember that students come from hugely various backgrounds and experience, from Year 11 students undertaking a VETiS program to someone with a PhD who is looking for a career change to an older person with a wide-ranging life experience.  This all affects how quickly they can become competent, not only based on their level of experience and in the skills they already have or need to learn, but their ability to undertake and comprehend the training in the first place.

 

While setting minimum durations based on the time a learner (who is new to the industry area) would be required to undertake supervised learning and assessment activities (ASQA report A review of issues relating to unduly short training page 15 has merit), the dilemma remains that some people undertaking formal study will already be to some degree competent and not require the same minimum duration.

In terms of auditing the appropriate volume of learning on a student by student basis, training providers using data from the AVETMISS enrolment should be able to demonstrate whether a student is new to the field and requires what could be considered as minimal learning hours, as opposed to someone who is requalifying and could reasonably be expected to complete a competency in a much shorter time frame.  I.e. based on evaluating the needs of the student from the language, literacy and numeracy (LL&N) together with the AVETMISS enrolment data a training provider should be able to specify the expected or minimum volume of learning required.   In a class room this could be part of the learning plan.  In an online environment, this tailored learning plan must be established and managed in a dynamic way, i.e. by having certain locks or minimum interactive learning hours completed before assessment is made available.

 

To improve course content Integrity measures policy writers must mandate;

  • Minimum learning time – Access to onlnline assessment should not be permitted until a minimum amount of learning that is determined on a student by student basis has been achieved
  • LMS capability to manage time – The LMS must be able to manage the minimum online learning time limits and must also be able to record the amount of time spent in learning activities.

 

6        Issuing Statements of Attainment

Copying or editing statements of attainment can be done no matter how they have been issued.  However, If statements of attainment are issued online, policy writers must mandate that the following integrity measures must be in place to minimise fraudulent activity:

 

  • No Name change – The student must not be able change their name online and an administrator should only be permitted to do this after identifying the student and then sighting an updated form of photographic identification meeting the identification integrity requirements with the new name.
  • Unique ID – The certificate must contain a unique number that is linked to the unique student username.
  • Watermark – The certificate must have a background water mark with the unique details of the certificate, e.g. student name, date of birth, issue date, certificate number, signature of issuing authority, expiration date over the top of the watermark(s).
  • Locked PDF – The certificate must be in a PDF format and the PDF must have security enabled that locks the PDF and prevents edits.

 

7        Learning Management System Data Capture Requirements

Without high granularity of data capture related to the digital learning environment, it is more difficult to demonstrate that integrity has been maintained.  Some Learning Management Systems (LMS’s) may only record the course sections that have or have not been accessed and that assessments have or have not been successfully completed, i.e. without capturing the granularity of the data such as the actual responses given to pass the assessment or the number of assessments or total learning engagement time.

 

In order to achieve integrity, a LMS must be used that has the sophistication to not only have integrity features in place, but also be able to collect data for integrity and track student activity and progress through the learning material and assessments including;

  • Content engagement – Record the amount of time that a student has interacted with learning content,
  • Number of attempts – Record how many attempts students have made for each assessment
  • Result of all attempts – Capture the result of each attempt not just correct answers,

 

Policy writers must specify that a learning management system used to deliver training online does so at a level of granularity that allows auditors to view every single piece of data submitted by the student in relation to assessment weather that submission is correct or not.  Also that the number of attempts and evidence of lockout after eg 3 attempts can be show with data evidencing how the lockout was overturned.

 

Do you have what it takes to create engaging elearning?

Ant Pugh is great at breaking things down into 4 minute videos and has some great knowledge to share about online training.

 

 

Assessment Automation v’s Validity of Assessment

LSM xapi assessment builder

“Assessment automation is a key step to – finally – unlocking the real potential of e-learning”

There’s growing recognition that automation, artificial intelligence, and electronic assessment tools have an important role in the future of e-learning. But e-learning company Enrolo sees assessment automation as a vital step towards transforming that future.

“Assessment automation is a key step to – finally – unlocking the real potential of e-learning,” says Enrolo CEO Adam Menary. “Like many companies, we’ve been exploring a range of new innovations and approaches to e-learning. Though we’ve also been approaching e-learning in a way that draws upon the best thinking and best practices from other industries.”

Beyond e-learning, Mr Menary has extensive experience in HACCP, risk and quality management systems. “These involve a systematic approach to assessing performance and achieving continuous improvement”, Mr Menary said. “This approach has been missing from e-learning because most data and systems are focussed on the learner. This can be useful, but ultimately, it’s a bit like trying to understand a tennis match when you’re seeing only one of the players and one side of the tennis court.”

So why does assessment automation matter? “It’s about advancing the assessment process with new technology-based tools, data and innovative approaches,” said Mr Menary. “Through our research projects, we’re starting to see that automation tools can transform the role of assessors, amplifying their ability to deliver value, both for students, and across the entire e-learning process.”

Enrolo CEO Adam Menary talked with -e-learning analyst Paul Wilson over coffees and Skype.

Question: What is it about assessment that has been missing from e-learning?

Enrolo CEO Adam Menary:                                             

Two things. Firstly, we’ve missed the opportunity to leverage the knowledge, value and insights of that assessors can provide.

In the drive to get e-learning content online and to get student numbers, the actual assessment process has received little attention. Without new assessment tools to match the pace and scale of work generated by learner-side technology advances, the ability of assessors to contribute value has been gradually diminishing. This can lead to a cost-focussed view of assessment, and with that, risks to learning outcomes – for example, through oversimplification of questions to reduce assessment costs, or reduced quality of assessment and feedback.

More broadly, in a competitive online sales environment this situation risks an unhealthy low cost and low value “race to the bottom” scenario for the future of e-learning.

However, it does not need to be this way.

The assessor is the person on the other side of the screen in a student’s e-learning experience. They can be the drivers of e-learning value and continuous improvement. They have front-line contact with students; they provide assessments and feedback that helps students confirm and improve their understanding; they also gain important insights into the strengths and weaknesses of e-learning content. They have the knowledge and insights to make a difference – and that’s usually the reason they’ve got into education in the first place.

Secondly, capturing rich data in the assessment process is essential to an integrated and scientific approach to continuous improvement across the entire e-learning process and organization. While other data is also important, data from the assessment process complements learner data, and helps provide a more complete view of overall e-learning performance.

Question: What do you mean by an “integrated and scientific approach”?

Enrolo CEO Adam Menary:

HACCP (food safety) systems use continuously-collected data to help ensure that food is safe to eat and free from contamination. Historically, e-learning has lacked that grade of data. For example, while SCORM involved collecting data that a student had successfully completed an assessment, SCORM might not be configured to keep track of which questions they answered, which they failed, where they got help or feedback, and whether that made a useful difference. This meant the data didn’t support efforts to statistically or scientifically track down areas where issues were occurring, or to identify whether changes were actually improvements.

The limitations of SCORM have been widely recognised, and xAPI emerged to address some of these. However, the focus remains on the learner and capturing learning-related data. Complementing this with data specifically about the assessment process can do two things: support work to streamline and improve assessment activities, and also provide insights about the overall performance of e-learning.

Question: Do you have any thoughts on how this might affect assessment from a learner’s perspective?

Enrolo CEO Adam Menary:

Yes. Multiple choice questions, true/false, checklists and the like are basic online forms of traditional assessment approaches but today they should be viewed by educators as simple check your progress tools rather than actual assessments. And sure, like others, Enrolo has done some interesting and innovative things with interactive check your progress. However, across the industry, e-learning has been constrained by a particular horizon of assessment possibilities, based on the costs, abilities, and practicalities of an assessor working in somewhat traditional manner. Automation, data and tools can help expand that horizon of assessment possibilities – and so offers further opportunities to transform e-learning.

And really, this is essential. Some of e-learning’s traditional assessment approaches are not that great. They can be blunt and ineffective tools for confirming a learner’s understanding.

We’re starting to explore ways to completely change assessment tasks, and learning content, based on the new possibilities of automation-enabled assessment. For example, what might it mean to have more free-form styles of assessment task, or much faster cycles of submission and feedback. Or imagine having assessment feedback going beyond comments – for example, using it the assessment feedback to dynamically reconfigure e-learning content to leverage a student’s strengths, or provide extra examples and explanations to address areas of weaknesses.

As the traditional true/false, pick from a list, or drag and drop type questions are relegated to check your progress aids I think we’re going to see some really important innovations and powerful new approaches to assessment and meaningful learning outcomes.

Disruptive change and the growing need for e-learning innovation

Today’s corporate leaders recognise that the pace of technological change is creating a major skills problem for their organizations. In a recent study, Deloitte found that 90% of CEOs believe their company is facing disruptive change driven by digital technologies, and 70 percent say their organization does not have the skills to adapt (Deloitte 2017: 30). Indicating some of the factors driving this problem, Deloitte point to the changing nature of a career, as follows.

The changing nature of a career (Deloitte 2017:30)

Clearly, with such frequent job changes and rapid skill obsolescence, there’s a significant demand placed on continuous learning and skills development. Thus, in many ways, individual and organizational performance now critically depends on the capability of e-learning providers to meet these demands. But how can e-learning leaders and innovators respond to this problem?

It’s useful to first get a little historical perspective.

Learning & Technological Change

The skills challenges outlined by Deloitte are the modern outcome of a long-term trend.

In earlier times, widely-used technologies and skills would remain mostly unchanged for generations. People could learn everything they needed from their parents and grandparents. Broader knowledge was sparse and slow-changing. For example, in Europe, Burke (1985) explains that Martianus Capella’s (5th century) book on the seven liberal arts became “all the education there was for the next 700 years”.

In later times, as knowledge and technologies started to change, schools and universities provided intensive bursts of education – ideally enough to last a lifetime. However, the pace of change has increased, leading to the situation described by Deloitte. For an overall perspective, the long-term trend can be understood by looking at the relationship between human lifespans and technology lifespans, as shown below.

Human lifespan vs technology lifespan (adapted from Scholte 1988:31) 

In modern times, things just move too quickly for many of our traditional approaches to education. Intensive bursts of formal education can no longer last a lifetime. Job descriptions change and blur. Key knowledge and skills become obsolete as entire industries fade and others emerge.

In these circumstances, there is a growing number of reasons why traditional education can’t keep up. And even faster-paced e-learning needs solid innovation and transformation.

The following sections explore ways that e-learning might respond to these challenges.

Rapid change & e-learning content development

It takes time to develop quality learning content – to make sense of a domain of activity, to distil the relevant insights, and to design content that helps establish these in the mind and practices of a learner. It can be time-consuming and costly. Given the rate of technological and skills change above, the challenges of content development are likely to increase. Deloitte (2017) points out one pathway forward:

“The good news is that an explosion of high-quality, free or low-cost content offers organizations and employees ready access to continuous learning. Thanks to tools such as YouTube and innovators such as Khan Academy, Udacity, Udemy, Coursera, NovoEd, edX, and others, a new skill is often only a mouse click away.”

This suggests a pathway involving something more along the line of e-learning content curation, drawing upon the best of what already exists and is freely available. Another related strategy might be to parallel the concept of “mashups”, creatively mixing this free content to support a specific learning need and context.

Such approaches will prove vital – but they will not be enough. This is because, historically, learning content authors have been able to draw insights from already-established and widely-accepted bodies of (mostly published) knowledge that may have been developed over generations. However this approach is under threat because it takes time for these bodies of knowledge to form; indeed, in times of rapid technological change, they may not get to even approach this point before another technological disruption occurs.  This suggests a future where learning shifts to the workplace front-line, actively supporting knowledge development and application in everyday activities.

Micro-learning and micro-credentials

Another response to the pace of change is to focus on a smaller and ongoing stream of workplace and on-demand learning activity – in essence, learning all the time, and using micro-learning and micro-credentials. Eades (2014) explains micro-learning as “delivering content in small, very specific bursts”; and on this same theme, micro-credentials involve providing formal recognition for skills and knowledge gained from on-the-job experience which can be “stacked” to form recognised qualifications (O’Keefe 2016). Practically, some of the main challenges of micro-learning will center upon:

  1. Identifying the workplace time and context for e-learning – for example, should it be when a person has struck a problem they need learning to address, or perhaps artificial intelligence could be used to anticipate what additional learning a person might need, such as before a scheduled client meeting. (For more on this, consider Friedman’s discussion of “Intelligent Assistants”, in Takahashi 2016).
  2. Ensuring that learning content is concise and relevant enough to leverage the learner’s current situation – for example, a long-winded explanation of marketing theory is likely to distract from the task-at-hand, and in contrast, perhaps, an analytical tool that helps a person address their immediate task or challenge. A key here seems to be leveraging the learner’s current situation, and their mental focus upon it, as the framing context for a learning activity.
  3. Ensuring cognitive continuity between workplace task and learning – for example, having to go to a specific training room is likely to be problematic, and in contrast, consider the value of mobile-based learning content that can be accessed immediately and within the current workplace context.

Tracking micro-learning

A challenge for the micro-learning approach is keeping track of an individual’s learning activities. One approach was explored in the Institute for the Future’s “Learning is Earning 2026” project. This project considered a very broad approach to learning and learning providers, with some interesting possibilities (see the project video, below) – and a central part of this they called “the Ledger”. They explain:

“In this future, the currency of learning is tracked and traded on a digital platform called the Ledger. It’s a complete record of everything you’ve ever learned, everyone you’ve learned from, and everyone who’s learned from you. The Ledger not only tracks what you know – it also tracks all of the projects, jobs, gigs, and challenges you’ve used that knowledge to complete.” (http://www.learningisearning2026.org/)

The foundation envisaged for the ledger concept is blockchain – the technology that underpins crypto-currencies like Bitcoin. There are various explanations of its potential for learning, but the essence is that it provides a way to keep an accurate and trusted records of historical activity – in the learning context, records of the activities a learner has completed.

The reality is that this technology faces some hurdles to being applied in a learning context. For example, the Bitcoin blockchain handles relatively simple and well-defined information: it deals with currency transactions. In contrast, a learning-focussed application of blockchain may need to record information about a learner, the person or authority certifying the learning, the deliverer of the learning, the content and version of the learning, and potentially submitted work and assessment information. Meanwhile, Flipo and Berne (2017) point out that the computer facilities that underpin the Bitcoin blockchain already consume electricity on a scale equivalent to the country of Ireland, and that even widespread adoption of Bitcoin would have grave energy consequences globally. A blockchain of complex learning data would, in all likelihood, quickly surpass Bitcoin’s energy impact.

Beyond these issues, the need and potential for strategies that address accelerated skills and learning needs remains. There will be ways forward. And while some details are yet to be established, the scenarios contemplated in the Institute for the Future’s project provide some interesting glimpses into ways that micro-learning, rapid skill development, and ubiquitous workplace and lifelong learning might play a role.

[Youtube video: https://www.youtube.com/watch?v=DcP78cLPGtE ]

Paul Wilson elearning Consultant

References

Burke J (1985), The Day the Universe Changed, BBC, London.

Deloitte University Press (2017), “Rewriting the rules for the digital age”, https://www2.deloitte.com/content/dam/Deloitte/global/Documents/HumanCapital/hc-2017-global-human-capital-trends-gx.pdf

Eades J (2014), “Why Microlearning is huge and how to be a part of it”, eLearningIndustry.com, accessed at: https://elearningindustry.com/why-microlearning-is-huge

O’Keefe D (September 21, 2016), “Old ways not enough to gain workplace credentials”, The Australian, accessed at: http://www.theaustralian.com.au/higher-education/old-ways-not-enough-to-gain-workplace-credentials/news-story/286a526a99fc866f96f99cdac2c6981a

Flipo F & Berne M (2017), “The bitcoin and blockchain: energy hogs”, The Conversation, accessed at: https://theconversation.com/the-bitcoin-and-blockchain-energy-hogs-77761

Learning is Earning 2026: http://www.learningisearning2026.org/

Scholtes PR (1988), The leader’s handbook: a guide to inspiring people and managing the daily workflow, McGraw-Hill, New York.

Takahashi D (December 7, 2016), “New York Times columnist Thomas Friedman tells us how to live in accelerated times”, Venture Beat, accessed at: https://venturebeat.com/2016/12/07/new-york-times-columnist-thomas-friedman-tells-us-how-to-live-in-accelerated-times/

Guide to online training Best practice integrity measures

30 Nov 2017 V2 Guidelines for online training best practice integrity measures

Contents

GUIDE TO ONLINE TRAINING BEST PRACTICE INTEGRITY MEASURES. 1

Executive Summary. 1

The Future of Online Learning. 2

Price of online training is a reflection of quality. 2

Learning Management System Requirements. 3

Integrity Measures. 3

Confirming Student Identity. 3

Valid Photo Identification. 3

Unique Student Identification. 3

Website with HTTPS Security. 4

Unique username. 4

Secure Password. 4

Identification during phone support 4

Terms and Conditions. 4

Assessing Student Competency. 4

Signing Off of Student Competency. 5

Minimising the Potential for Fraudulent Activity. 5

Course Content 6

Volume of Learning. 6

Issuing Statements of Attainment 7

 

Disclosure statement – Enrolo is a custom online training platform that complies with all of the best practice integrity measures detailed in this document.  At Enrolo we believe that for the betterment of the online training industry as a whole all online training providers of nationally recognised training must implement these integrity measures as a minimum.

Executive Summary

To ensure that learning outcomes are achieved in an online environment certain minimum integrity measures must be in place.  Integrity measures also need to be in place to reduce fraudulent activity. Whilst examples of industry best practice guidelines for online training do exist, many training providers are either not aware of them or, in some cases, are deliberately not implementing them to cut costs.  Whilst there are examples of regulators stipulating mandatory online integrity measures, these requirements are often not enforced due to lack of resources or technical understanding.  Training providers that are not complying with best practice online integrity measures are producing what is known as ’page turning’ or ‘tick and flick’ learning with assessments such as true/ false or multiple choice that can be passed by guessing.  Providers that are cutting corners with online training are leading a race to the bottom in terms of the quality of training, assessment and ultimately the student’s value for money.  The result of lack of integrity is not only poor learning outcomes and inadequate training, leading to inefficiencies in the workplace but also damage to the reputation of the online or the ‘e-learning’ industry as a whole.  This best practice guide is intended to provide regulators and online providers with integrity measures that must be in place to ensure the integrity of online learning outcomes.

 

The Future of Online Learning

Automation in the training profession is continuing to advance at an ever increasing speed. Examples include interactive videos, free text auto marking, interactive video assessment and virtual reality scenarios for the demonstration of skills.  The experience API is also allowing online training to move beyond the boundaries of formally structured content development.

Many qualifications that are taught using sophisticated online training methods can already deliver the same, if not better learning outcomes than in-class training.  Since online training innovation is growing at an exponential rate it is advantageous if training providers are allowed to author their own learning materials to keep up with these changes.  In the future regulators that attempt to provide learning material to training providers may face increased costs and struggle to keep up with the rapidly evolving innovation. An alternate approach that regulators could consider would be to provide elements such as videos or interactive content that must be included as part of a course.

From a training provider perspective, it is essential to realise that to achieve efficient and effective online learning,  content and assessment are inseparable.  This is because assessments are directly related to the learning material and in order to achieve accurate mapping to a unit of competency assessment questions must be written in such a way that may require regular edits to the content.  Furthermore, if an assessment question is causing confusion or returning a high rate of incorrect answers it may be the case that the content needs to be edited rather than the assessment question.  Another issue that can occur with content provided by third parties is if the third parties are renamed or restructured often they lose funding to maintain content and this content is then not able to be updated for name or content changes by training providers as there are often IP issues with authored content.

Price of online training is a reflection of quality

There is a perception that online training is a way for training providers to cut costs and increase revenues. However, if a training organisation is meeting the requirements for vocational education standards, despite the reduced cost of hiring classrooms, the cost of delivering effective online training is equal to if not greater than in-class training.  This is because not only do all of the existing requirements such as qualified trainers and assessors, assessment material, mapping, internal audits, continuous improvement registers and so on all need to be in place but in addition to these costs, online providers need to purchase and run an online learning environment and build online learning material and robust assessments.  Effective and engaging online training includes videos, games, interactive scenarios, and animations with voice overs and challenging exercises as well as additional reading materials. If the online content is achieving real learning outcomes then it can cost up to AUD$30,000 per hour of e-learning output.  Many “online providers” are cutting corners by developing simple e-books or a PDF with a voice over and a next button between pages.  Whist it is possible to map such material to the elements and criteria of a unit of training package there needs to be more emphasis on being able to demonstrate that a student can achieve competency from such material through rigorous assessment.  If high integrity online training does not reduce costs for the training providers then what are the benefits?

  • Improved accessibility particularly for remote or regional areas
  • Students can learn in their own time and do not have to take time off work
  • Reduced travel costs such as transport and parking during training and also reduced carbon emissions
  • Greater consistency in terms of content delivered and the assessment method
  • Quicker response times to updating learning content
  • More empirical data can be collected for demonstrating competency
  • Reduced administrative and non-value-adding work such as marking or issuing certificates
  • Less monotonous tasks for trainers and more time spent with students that need their assistance
  • Reduced human error related to monotonous and repetitive tasks
  • Learners can work at their own pace and concentrate on areas they do not understand
  • Students can spend as much time with qualified trainers as they need
  • Students can access learning material ongoing and can also be provided with updates to the learning material
  • Reduced room hire costs which are however, offset by online tools, online delivery, technical support and content development.

Learning Management System Requirements

In order to achieve integrity, a Learning Management System (LMS) must be used by training providers that has the sophistication to be able to collect data for integrity and track student activity and progress through the learning material and assessments. Whilst some LMS’s can track the amount of time that a student has interacted with learning content, how many attempts they have made for each assessment, and capture the result of each attempt, other LMS’s may only record the course sections that have or have not been accessed and that assessments have or have not been successfully completed, i.e. without capturing the granularity of the data. It is important that regulators understand these differences and to improve integrity by specifying the requirement for high granularity tracking of student interactions and assessment.  Without the high granularity of data related to the e-learning environment it is more difficult to demonstrate that integrity has been maintained.

Integrity Measures

Confirming Student Identity

Confirming student identify in an online setting is of paramount importance in reducing fraudulent activity, i.e. by ensuring that the person doing the training is the person obtaining the Statement of Attainment/Certificate.

Fingerprint and retina scans are becoming more prevalent however, until these are more widely accepted the following procedures and features are considered best practice to verify that a student who is enrolled in an online course is the person who also completes the course and receives the certification:

Valid Photo Identification

Students must provide via a secure online website a copy of current government-issued photo ID which is reviewed and accepted or rejected by trained staff.  Details captured and verified must include the full name, date of birth, and contact details (including residential, valid email address, and contact phone number) of the student.

The residential address of the student must be validated.  This can be improved using an online address validation service or address look up tool.

Students must not be allowed to change personal details themselves – they need to lodge a request and provide evidence, such as government-issued photo ID, marriage certificate, change of name certificate, etc. where required.

For administrators the valid photo ID should be displayed against a student account when videos are being assessed, or during video conferencing so trainers can ascertain that they are speaking with the student who has made the recording.

Unique Student Identification

In Australia it is a government requirement for nationally recognised training that before completing their enrolment students must obtain a Unique Student Identifier (USI) and the details must match their valid photo identification provided. In Australia it is a government requirement that students supply their Unique Student Identifier (USI), the details of which must match their valid photo ID provided, before they can be issued with their Statement of Attainment/Certificate (qualification).

Website with HTTPS Security
Students must only access online courses through a https secure website that requires a login and password.

Unique username

Students must create their own unique username such as an email address

Secure Password

If an overly complex password is provided to the student then they will need to write it down and this reduces the security of the password so students must create a secure password themselves.

Students must  be provided with best practice information regarding setting up and maintaining the security of their passwords.

Student password recovery must be via clicking a link to recover their password that sends them details via their registered email. Passwords must not be able to be changed without the student logging into the account.

Identification during phone support
When a student contacts a training provider by phone or email requesting help, the provider must have procedures in place to check their identity before providing any account details or assistance.

Terms and Conditions

Students must agree to terms and conditions at the start of an online course, confirming  that they are the person doing the training.  This can be done via a statutory declaration or equivalent.

Once complete and before generating and accessing the final PDF Statement of Attainment/Certificate the student must declare that they were the person that did the training and that they did so without assistance.  This must be recorded in the electronic management system and be able to be provided as evidence for legal breaches.

Assessing Student Competency

Assessments must be mapped to a unit of competency to demonstrate that all of the required learning elements have been assessed.

In an online environment assessments that are designed to provide a true reflection of the knowledge and skills that have been achieved are one method of proving competency has been gained.  Offline activities can also be undertaken and witnessed by a suitably qualified supervisor.

In an online environment in order for assessments to demonstrate that a student has achieved understanding and competency they must have the following integrity measures:

  1. Require at least 25% of questions to be answered with free text sentences allowing a proper critical response to questions, i.e. not single word answers or numbers
  2. In the case of demonstration of skills, require video footage to be submitted to capture a student’s ability to demonstrate a task or skill
  3. If a student gets the same question wrong 3 times they are “locked out” and required to talk to the training organisation and if they need assistance they are referred to a qualified trainer rather than keep guessing until they get the question correct through a process of elimination
  4. Multiple choice questions must have more wrong combinations than available attempts and designed so that students will be locked out before guessing through a process of elimination.
  5. Students must not be provided with the correct answer before they get the answer correct themselves.
  6. The order of assessments should be randomised to reduce the likelihood of cheat sheets.
  7. Because of the complexity of meeting mapping requirements if questions are randomised, banks of similar questions should be created and questions randomly selected from those banks to ensure that the particular questions achieve the required mapping.

In contrast to assessments that comply with the above integrity measures,  assessments that are 100% marked by a computer that are predominantly multiple choice by elimination or true/false by elimination, provide no requirement or opportunity for a student to demonstrate their comprehension of the learning material. Whilst it can be argued that by eventually getting a question right is a form of learning, allowing students to keep attempting questions until they get the correct answer does not deliver meaningful learning outcomes. Demonstrated competency through rigorous and auditable assessment in combination with the above integrity measures is considered an integral requirement for the demonstration of online learning outcomes.

Signing Off of Student Competency

Online training providers may try to cut corners and costs by using administration staff with no training qualifications to assist with training.  To cut costs training providers may also reduce access to qualified trainers to very short periods. It is essential that qualified trainers are available in an online environment.  These trainers may communicate with the students via:

  • Telephone
  • Email
  • Live chat
  • Webinars
  • Video conferencing
  • Skype

Integrity measures include:

  1. The use of qualified trainers for assisting with training support. All trainers employed by a training organisation must be on a training register.
  2. Total trainer hours must reflect availability during normal office hours.
  3. Trainers must be available during normal business hours and must assist learners requesting training within 30 minutes of a request.
  4. An online statement of attainment must not be issued until the enrolment details confirmed and assessment and learning activities of the student have been reviewed by a qualified trainer.

Minimising the Potential for Fraudulent Activity

The following integrity measures must be in place to minimise the potential for fraudulent activity

  1. Capture the student’s IP address and monitor if bulk users are coming from the one IP.
  2. A system for monitoring cheating must be in place that flags when different students give the exact same free text answers to free text assessment questions..
  3. Student detail fields must be locked so that these can only be changed by administration staff upon request and subsequent verification of the student’s identification.
  4. Students are ‘locked-out’ of being able to complete an assessment after failing a set number of attempts of each question and must speak with a qualified trainer to be unlocked if they do not know the answer.
  5. Must be able to intervene and make direct contact with the student if required.
  6. Notify students that certification may be voided if fraudulent information is provided or fraudulent activity is detected.
  7. Have the ability to report immediately any suspicious activity by students undertaking the course to the appropriate authority.
  8. Deliver training via SSL to reduce third party fraudulent activity.
  9. Capture electronic evidence that the student has agreed to terms and conditions relating to not being assisted in any way during assessment and that they are the one that undertook and completed the training.
  10. If adequate integrity measures are not in place, or a student has breached an integrity measure, students must sign and upload a witnessed statutory declaration that they are undertaking the training without assistance.

Course Content

Volume of Learning

The volume of learning includes guided learning, individual study, research, learning activities in the workplace and assessment activities. It could be argued that rather than applying this at a qualification level, the volume of learning should be applied at a unit of competency level and rather than minimum duration the relationship between the quality of the content being able to map to a rigorous assessment should be the deciding factor in assessing competency as the AQF goes on to say:

 

“The duration of the delivery of the qualification may vary from the volume of learning specified for the qualification. Providers may offer the qualification in more or less time than the specified volume of learning, provided that delivery arrangements give students sufficient opportunity to achieve the learning outcomes for the qualification type, level and discipline.”

 This could easily be applied to a competency rather than the whole qualification.

When looking at VET training, we need to remember that students come from hugely various backgrounds and experience, from Year 11 students undertaking a VETiS program to someone with a PhD who is looking for a career change to an older person with a wide-ranging life experience. This all affects how quickly they can become competent, not only based on their level of experience and in the skills they already have or need to learn but their ability to undertake and comprehend the training in the first place.

While setting minimum durations based on the time a learner (who is new to the industry area) would be required to undertake supervised learning and assessment activities (ASQA report A review of issues relating to unduly short training, page 15 https://goo.gl/z9v6jr) has merit, the dilemma remains that some people undertaking formal study will already be to some degree competent and not require the same minimum duration.

In terms of auditing the appropriate volume of learning on a student by student basis, training providers using data from the AVETMISS enrolment should be able to demonstrate whether a student is new to the field and requires what could be considered as minimal learning hours as opposed to someone who is requalifying and could reasonably be expected to complete a competency in a much shorter time frame.  I.e. based on evaluating the needs of the student from the language, literacy and numeracy (LL&N) together with the AVETMISS enrolment data a training provider should be able to specify the expected or minimum volume of learning required.   In a class room this could be part of the learning plan.  In an online environment this tailored learning plan must be established and managed in a dynamic way, i.e. by having certain locks or minimum interactive learning hours completed before assessment is made available.

Integrity measures

  1. Access to assessment should not be permitted until a minimum amount of learning that is determined on a student by student basis has been achieved
  2. The LMS must be able to manage the minimum time limits and must also be able to record the amount of time spent in learning activities.

Issuing Statements of Attainment

Copying or editing statements of attainment can be done no matter how they have been issued. However if statements of attainment are issued online the following integrity measures must be in place to minimise fraudulent activity:

  1. The student must not be able change their name online and an administrator should only be permitted to do this after identifying the student and then sighting an updated form of photographic identification meeting the identification integrity requirements with the new name.
  2. The certificate must contain a unique number that is linked to the unique student username.
  3. The certificate must have a background water mark with the unique details of the certificate, e.g. student name, date of birth, issue date, certificate number, signature of issuing authority, expiration date over the top of the watermark(s).
  4. The certificate must be in a PDF format and the PDF must have security enabled that locks the PDF and prevents edits.