Australian Online Training Best Practice Guide for Regulators and RTO’s – White Paper

Aust Online Training Best Practice Integrity Measures-V8.PDF 

Foreword

This white paper is intended to provide, policy writers, authors of units of competency, regulators and online training providers with integrity measures that must be in place to ensure the integrity of online learning outcomes.

In the absence of  explicit requirements for online integrity in either; the Standards for RTO’s 2015 or, units of competency, It is essential that regulators, policy writers and authors of units of competency, understand the importance of both the granularity of online training data and integrity measure features, so that they can specify them as mandatory requirements, improving integrity and the ability of the Training Regulator to act.

Executive Summary

Technological advancement is occurring in the Vocational Education and Training (VET) sector ahead of guidelines, legislation and quality control mechanisms by industry, government and regulators, to ensure that there is adequate integrity underpinning statements of attainment issued via an online mode of delivery.  In some cases this is leading to low cost and low quality, online learning outcomes as well as giving online training a poor reputation.

High-integrity online training has much to offer the training sector including; providing better access for remote and regional communities, and better flexibility for students, who may be working, studying or parents at home. In many cases high integrity online training can provide better learning outcomes through the use of engaging multi-media technologies that can be studied at the learner’s own pace.  High integrity online training can also improve evidence of assessment and demonstrate high compliance to regulations.

To ensure that learning outcomes are achieved in an online environment, certain technology with the capability to deliver minimum integrity measures, as outlined in this white paper, must be in place.  Specific online Integrity measures also need to be in place to reduce fraudulent activity.

Whilst examples of industry best practice guidelines for online training do exist, many training providers are either not aware of them or, in some cases, are deliberately not implementing them to cut costs and compete with the lowest cost providers in the market.  There are examples of regulators (e.g. state and territory liquor authorities) stipulating mandatory online integrity measures however, these requirements are often not enforced by the Australian Skills Quality Authority (ASQA) due to lack of resources or technical understanding.

Under the Federal Register of Legislation the Standards for Registered Training Organisations (RTOs) 2015 do not provide adequate guidance to ensure that online training has high integrity or effective learning outcomes.

Many employers have experienced lack of knowledge and poor skills form students who have gained qualifications from low-integrity, low-cost online training and may mistrust all online qualifications.

Despite advancement in technology such as virtual reality or simulated environments, rather than adding online integrity measures as a requirement to units of competency (i.e. to encourage high integrity online training),  some of the Industry Skills Councils have actively added requirements into units of competency to try to exclude the online mode of delivery.  The industry representatives on these skills councils may have limited understanding of online technologies and the need for high integrity measures.  This results in units of competency that can easily be delivered by low integrity providers who deliver poor learning outcomes.

Armed with an out of date Standard for Registered Training Organisations (RTO) 2015 and units of competency that do not specify online integrity measures, the Vocational Education and Training (VET) regulator, the Australian Skills Quality Authority (ASQA), is often unable to act against unscrupulous profiteering providers delivering low integrity online training.

Training providers that are cutting corners with online training are leading educators in a race to the bottom in terms of poor quality of training and assessment, and ultimately undermining the student’s value for money.  The result of lack of integrity is not only poor learning outcomes and inadequate training, leading to inefficiencies or dangerous practices in the workplace, but also damage to the reputation of the online or the ‘e-learning’ industry as a whole.

A small number of unscrupulous providers have taken over approximately 80% of some online training markets and are unfairly making it uneconomic for high-integrity and higher-cost training providers to compete.  Many online training providers that are not complying with best practice online integrity measures are producing what is known as ’page turning’ or ‘tick and flick’ learning with assessments such as true/ false or multiple choice that can be passed by guessing without restriction.  Based on the scope of some units of competency this conduct is endangering the lives of patrons and people in the community.

To bring to the fore the advantages of online training, and eliminate the risks from low integrity online training, policy writers, regulators, and authors of units of competency must ensure online integrity measures are specified in order to allow the regulator to act.

The Future of Online Learning

Automation in the training profession is continuing to advance at an ever-increasing speed. Examples include interactive videos, computer auto marking, interactive video assessment and virtual reality scenarios for the demonstration of skills.  The experience API is also allowing online training to move beyond the boundaries of formally structured content development and the classroom.

Many qualifications that are taught using sophisticated online training methods can already deliver the same, if not better learning outcomes than in-class training.  Since online training innovation is growing at an exponential rate it is advantageous if training providers are allowed to author their own learning materials to keep up with these changes.

Benefits of high integrity online training

The benefits of high integrity online training include but are not limited to;

• Improved accessibility, particularly for students in remote or regional areas or parents with young children.
• Students can learn in their own time and do not have to take time off work or study.
• Reduced travel costs such as transport and parking during training and also reduced carbon emissions.
• Greater consistency in terms of content delivered and the assessment method.
• Quicker response times to updating learning content resulting in more up to date learning resources.
• More empirical data can be collected for demonstrating competency and adherence to high integrity.
• Reduced administrative and non-value-adding work such as marking quizzes or issuing certificates.
• Less monotonous tasks for trainers and more time spent with students that need their assistance.
• Reduced human error related to less reliance on monotonous and repetitive tasks.
• Learners can work at their own pace and concentrate on areas they do not understand.
• Students can spend as much time with qualified trainers as they need.
• Students can access learning material ongoing and can also be provided with updates to the learning material.
• Reduced room hire costs which are however, offset by online tools, online delivery, technical support and online course content development.

High integrity online training costs

There is a perception that online training is a way for training providers to cut costs such as eliminating trainers and class rooms and automating everything with a computer to increase profits.  However, if a training organisation is meeting the high integrity online measures outlined in this white paper, despite the reduced cost of hiring classrooms, the cost of delivering effective online training is equal to if not greater than in-class training.  This is because not only do all of the existing requirements that need to be in place for in-class training such as; qualified trainers and assessors, assessment material, mapping, internal audits, continuous improvement registers and so on, still need to be in place for online, but in addition to these costs, online providers need to purchase and maintain an online learning environment and build digital online learning material and robust assessments.

Whilst a 200 page PDF document available online and a quiz requiring a student to tick a few true / false questions meets the Standards for RTOS’ 2015 and most units of competency, effective and engaging online training includes videos, gamification, interactive scenarios, simulated environments and animations with voice overs and challenging exercises as well as additional reading materials or offline activities. If online training contains these kinds of elements then it can cost up to AUD$30,000 per hour of e-learning output.  Compare that to a power point presentation created for in class training and we are talking $180,000 vs a few thousand for a six hour course.

Many “online providers” are cutting corners by developing simple e-books or a PDF with a voice over and a next button between pages.  Whist it is possible to map such “online” material to the elements and criteria of a unit of a training package, there needs to be more emphasis on being able to demonstrate that a student can achieve competency from such material through rigorously linking it to high integrity online assessments.

Regulators that attempt to provide learning material to training providers may face increased costs and struggle to keep up with the rapidly evolving innovation.

An alternate approach that regulators may consider is to provide elements such as videos or interactive content that may be included by an online training provider as part of a course.  A good example of this is the “Behind the bar RSA interactive training video – Just one more” authored by the Office of Liquor and Gaming Regulation Queensland, who invite training providers to use the content in their courses royalty free.

From a training perspective, it is essential to realise that to achieve efficient and effective online learning,  that the ability for a training provider to edit content and assessment is inseparable.  This is because assessments are directly related to the learning material and in order to achieve accurate mapping to a unit of competency assessment questions must be written in such a way that may require regular edits to the content.  Furthermore, if an assessment question is causing confusion or returning a high rate of incorrect answers, it may be the case that the content needs to be edited rather than the assessment question.  Another issue that can occur with mandatory content provided by regulators is that if the government bodies are renamed or restructured, often they lose funding to maintain content and this content is then not able to be updated for name or content changes by training providers as they don’t have open source access.  A consideration for regulators providing such content is to provide open source access to authored content so that training providers and modify it as needed.

A key take home for regulators is that low sale prices for online courses are a likely reflection of the expenditure put into developing the content and also reflect the lack of real and qualified training resources behind the scenes and should be targeted for audits.

Online Integrity Measures

In the absence of any online integrity regulation, where possible, policy writers should add the following as mandatory requirements:

1.      Confirming Student Identity

Confirming student identify in an online setting is of paramount importance in reducing fraudulent activity, i.e. by ensuring that the person doing the training is the person obtaining the Statement of Attainment/Certificate.

Fingerprint and retina scans are becoming more prevalent however, until these are more widely accepted the following procedures and features are considered best practice to verify that a student, who is enrolled in an online course, is the person who also completes the course and receives the certification:

1.1.   Valid Photo Identification

Students must provide via a secure (https) online website, a copy of current government-issued photo ID which is reviewed and accepted or rejected by trained staff.  Details captured and verified must include the full name, date of birth, and contact details (including residential, valid email address, and contact phone number) of the student.

The residential address of the student must be validated.  This can be improved using an online address validation service or address look up tool.

Students must not be allowed to change personal details themselves – they need to lodge a request and provide evidence, such as government-issued photo ID, marriage certificate, change of name certificate, etc. where required.

For administrators the valid photo ID should be displayed against a student account when videos are being assessed, or during video conferencing so trainers can ascertain that they are speaking with the student who has made the recording.

In many circumstances this procedure may exceed identification integrity taking place in classroom training environments.

1.2.   Unique Student Identification

In Australia it is a government requirement for nationally recognised training, that before completing their enrolment students must obtain a Unique Student Identifier (USI) and the details must match their valid photo identification provided.  In Australia it is a government requirement that students supply their Unique Student Identifier (USI), the details of which must match their valid photo ID provided, before they can be issued with their Statement of Attainment/Certificate (qualification).

• Website with HTTPS Security
  Students must only access online courses through a https secure website that requires a login and password.

1.4.   Unique username

Students must create their own unique username such as an email address

1.5  Secure Password

If an overly complex password is provided to the student then they will need to write it down and this reduces the security of the password so students must create a secure password themselves.

Students must  be provided with best practice information regarding setting up and maintaining the security of their passwords.

Student password recovery must be via clicking a link to recover their password that sends them details via their registered email. Passwords must not be able to be changed without the student logging into the account.

• Identification during phone support
  When a student contacts a training provider by phone or email requesting help, the provider must have procedures in place to check their identity before providing any account details or assistance.
• Terms and Conditions
  Students must agree to terms and conditions at the start of an online course, confirming that they are the person doing the training.  This can be done via a statutory declaration or equivalent where required.Once complete and before generating and accessing the final PDF Statement of Attainment/Certificate the student must declare that they were the person that did the training and that they did so without assistance.  This must be recorded in the electronic management system and be able to be provided as evidence for legal breaches.

2        Assessing Student Competency

Assessments must be mapped to a unit of competency to demonstrate that all of the required learning elements have been assessed.

In an online environment, assessments that are designed to provide a true reflection of the knowledge and skills that have been achieved are one method of proving competency has been gained.  Offline activities can also be undertaken and witnessed by a suitably qualified supervisor and their accounts included for consideration.

Policy writers must mandate that, In an online environment, in order for assessments to demonstrate that a student has achieved understanding and competency, they must have the following integrity measures:

• Free text – Require at least 1 question in each assessment (ideally 25%) to be answered with free text sentences (ie at least 5 words or more, not a single word or number) allowing a proper critical response to questions.
• Video Demonstration – In the case of demonstration of skills, require video footage to be submitted to capture a student’s ability to demonstrate a task or skill.
• Lock out – If a student gets the same question wrong three (3) times they must be “locked out” and requiredto talk to the training organisation and if they need assistance they must be referred to a qualified trainer ie rather than keep guessing until they get the question correct through a process of elimination.
• Multi choice wrong ratio – Multiple choice questions must have more wrong combinations than available attempts and designed so that students will be locked out before guessing through a process of elimination.
• Not given answer – Students must not be provided with the correct answer eg before or after attempting an assessment before they get the answer correct themselves. If re attempting assessments, students must get the full range of options and not be shown which answers have already been attempted and marked as incorrect.
• No direct answer link – Students must not be provided with links from the question directly to the correct answers.
• Random questions – The order of assessments must be randomised to reduce the likelihood of cheat sheets.
• Question banks -Because of the complexity of meeting mapping requirements if questions are randomised, banks of similar questions must be created and questions randomly selected from those banks to ensure that the particular questions achieve the required mapping.
• Prevent Concurrent viewing of Learning material and Assessment The online system must ensure that students cannot have the learning material open in one window and the assessment in the other allowing the student to copy and paste answers without critical thinking. This needs to be in place for multiple tabs, multiple browsers on the one device or multiple logins across multiple devices.
• Progress through the assessment is clearly indicated – complying with the fairness guidelines of the Standard’s for RTO’s.

In contrast to assessments that comply with the above high integrity measures, assessments that are 100% marked by a computer that are predominantly multiple choice by elimination or true/false by elimination, provide no requirement or opportunity for a student to demonstrate their comprehension of the learning material. Whilst it can be argued that by eventually getting a question right is a form of learning, allowing students to keep attempting questions until they get the correct answer does not deliver meaningful learning outcomes. Demonstrated competency through rigorous and auditable assessment, in combination with the above integrity measures, is considered an integral requirement for the demonstration of online learning outcomes.

3        Signing Off of Student Competency

Online training providers may try to cut corners and costs by using administration staff with no training qualifications to assist with training.  To cut costs training providers may also reduce access by students to qualified trainers to very short periods. It is essential that qualified trainers are available to students in an online environment.  These trainers may communicate with the students via:

• Telephone
• Email
• Live chat
• Webinars
• Video conferencing

To improve Integrity measures policy writers must mandate the following:

• Use of qualified trainers – for assisting with training support. All trainers employed by a training organisation  must be on a training register. 
• Trainer Roster – Total trainer hours must as a minimum reflect availability during normal office hours.
• Trainer availability – Trainers must be available at least during normal business hours and must assist learners requesting training within 30 minutes of a request.
• Human validation – An online statement of attainment must not be issued until the enrolment details confirmed and assessment and learning activities of the student have been reviewed by a qualified trainer. This sign off process must be demonstrated by data in the learning management system.  

4        Minimising the Potential for Fraudulent Activity

Policy writers must ensure that the following integrity measures are in place to minimise the potential for fraudulent activity:

• Capture the student’s IP address – and monitor if bulk users are coming from the one IP.
• A system for monitoring cheating must be in place that flags when different students give the exact same free text answers to free text assessment questions.
• Student name fields must be locked so that these can only be changed by administration staff upon request and subsequent verification of the student’s identification.
• ‘lock-out’ access –  to an assessment after failing a set number of attempts of each question (3)  and must speak with a qualified trainer to be unlocked if they do not know the answer.
• Must be able to intervene and make direct contact with the student if required.
• Warn about fraudulent activity – Notify students that certification may be voided if fraudulent information is provided or fraudulent activity is detected.
• Ability to report suspicious activity – The learning system must have the ability to identify and report immediately any suspicious activity by students undertaking the course to the appropriate authority.
• Deliver training via SSL – SSL to reduce third party fraudulent activity ie HTTPS at start of a domain..
• Agree to terms and conditions – Capture electronic evidence that the student has agreed to terms and conditions relating to not being assisted in any way during assessment and that they are the one that undertook and completed the training.
• Statutory Declaration – If adequate integrity measures are not in place, or a student has breached an integrity measure, students must sign and upload a witnessed statutory declaration that they are undertaking the training without assistance.

5        Course Content

5.1  Volume of Learning

The Australian Quality Framework says that “the volume of learning”  includes guided learning, individual study, research, learning activities in the workplace and assessment activities. It could be argued that rather than applying this at a qualification level, the volume of learning should be applied at a unit of competency level and rather than minimum duration the relationship between the quality of the content being able to map to a rigorous assessment should be the deciding factor in assessing competency.  As the AQF goes on to say:

“The duration of the delivery of the qualification may vary from the volume of learning specified for the qualification. Providers may offer the qualification in more or less time than the specified volume of learning, provided that delivery arrangements give students sufficient opportunity to achieve the learning outcomes for the qualification type, level and discipline.”

This could  be applied to a competency rather than the whole qualification.

When looking at VET training, we need to remember that students come from hugely various backgrounds and experience, from Year 11 students undertaking a VETiS program to someone with a PhD who is looking for a career change to an older person with a wide-ranging life experience.  This all affects how quickly they can become competent, not only based on their level of experience and in the skills they already have or need to learn, but their ability to undertake and comprehend the training in the first place.

While setting minimum durations based on the time a learner (who is new to the industry area) would be required to undertake supervised learning and assessment activities (ASQA report A review of issues relating to unduly short training page 15 has merit), the dilemma remains that some people undertaking formal study will already be to some degree competent and not require the same minimum duration.

In terms of auditing the appropriate volume of learning on a student by student basis, training providers using data from the AVETMISS enrolment should be able to demonstrate whether a student is new to the field and requires what could be considered as minimal learning hours, as opposed to someone who is requalifying and could reasonably be expected to complete a competency in a much shorter time frame.  I.e. based on evaluating the needs of the student from the language, literacy and numeracy (LL&N) together with the AVETMISS enrolment data a training provider should be able to specify the expected or minimum volume of learning required.   In a class room this could be part of the learning plan.  In an online environment, this tailored learning plan must be established and managed in a dynamic way, i.e. by having certain locks or minimum interactive learning hours completed before assessment is made available.

To improve course content Integrity measures policy writers must mandate;

• Minimum learning time – Access to onlnline assessment should not be permitted until a minimum amount of learning that is determined on a student by student basis has been achieved
• LMS capability to manage time – The LMS must be able to manage the minimum online learning time limits and must also be able to record the amount of time spent in learning activities.

6        Issuing Statements of Attainment

Copying or editing statements of attainment can be done no matter how they have been issued.  However, If statements of attainment are issued online, policy writers must mandate that the following integrity measures must be in place to minimise fraudulent activity:

• No Name change – The student must not be able change their name online and an administrator should only be permitted to do this after identifying the student and then sighting an updated form of photographic identification meeting the identification integrity requirements with the new name.
• Unique ID – The certificate must contain a unique number that is linked to the unique student username.
• Watermark – The certificate must have a background water mark with the unique details of the certificate, e.g. student name, date of birth, issue date, certificate number, signature of issuing authority, expiration date over the top of the watermark(s).
• Locked PDF – The certificate must be in a PDF format and the PDF must have security enabled that locks the PDF and prevents edits.

7        Learning Management System Data Capture Requirements

Without high granularity of data capture related to the digital learning environment, it is more difficult to demonstrate that integrity has been maintained.  Some Learning Management Systems (LMS’s) may only record the course sections that have or have not been accessed and that assessments have or have not been successfully completed, i.e. without capturing the granularity of the data such as the actual responses given to pass the assessment or the number of assessments or total learning engagement time.

In order to achieve integrity, a LMS must be used that has the sophistication to not only have integrity features in place, but also be able to collect data for integrity and track student activity and progress through the learning material and assessments including;

• Content engagement – Record the amount of time that a student has interacted with learning content,
• Number of attempts – Record how many attempts students have made for each assessment
• Result of all attempts – Capture the result of each attempt not just correct answers,

Policy writers must specify that a learning management system used to deliver training online does so at a level of granularity that allows auditors to view every single piece of data submitted by the student in relation to assessment weather that submission is correct or not.  Also that the number of attempts and evidence of lockout after eg 3 attempts can be show with data evidencing how the lockout was overturned.