Assessment Automation v’s Validity of Assessment

LSM xapi assessment builder

“Assessment automation is a key step to – finally – unlocking the real potential of e-learning”

There’s growing recognition that automation, artificial intelligence, and electronic assessment tools have an important role in the future of e-learning. But e-learning company Enrolo sees assessment automation as a vital step towards transforming that future.

“Assessment automation is a key step to – finally – unlocking the real potential of e-learning,” says Enrolo CEO Adam Menary. “Like many companies, we’ve been exploring a range of new innovations and approaches to e-learning. Though we’ve also been approaching e-learning in a way that draws upon the best thinking and best practices from other industries.”

Beyond e-learning, Mr Menary has extensive experience in HACCP, risk and quality management systems. “These involve a systematic approach to assessing performance and achieving continuous improvement”, Mr Menary said. “This approach has been missing from e-learning because most data and systems are focussed on the learner. This can be useful, but ultimately, it’s a bit like trying to understand a tennis match when you’re seeing only one of the players and one side of the tennis court.”

So why does assessment automation matter? “It’s about advancing the assessment process with new technology-based tools, data and innovative approaches,” said Mr Menary. “Through our research projects, we’re starting to see that automation tools can transform the role of assessors, amplifying their ability to deliver value, both for students, and across the entire e-learning process.”

Enrolo CEO Adam Menary talked with -e-learning analyst Paul Wilson over coffees and Skype.

Question: What is it about assessment that has been missing from e-learning?

Enrolo CEO Adam Menary:                                             

Two things. Firstly, we’ve missed the opportunity to leverage the knowledge, value and insights of that assessors can provide.

In the drive to get e-learning content online and to get student numbers, the actual assessment process has received little attention. Without new assessment tools to match the pace and scale of work generated by learner-side technology advances, the ability of assessors to contribute value has been gradually diminishing. This can lead to a cost-focussed view of assessment, and with that, risks to learning outcomes – for example, through oversimplification of questions to reduce assessment costs, or reduced quality of assessment and feedback.

More broadly, in a competitive online sales environment this situation risks an unhealthy low cost and low value “race to the bottom” scenario for the future of e-learning.

However, it does not need to be this way.

The assessor is the person on the other side of the screen in a student’s e-learning experience. They can be the drivers of e-learning value and continuous improvement. They have front-line contact with students; they provide assessments and feedback that helps students confirm and improve their understanding; they also gain important insights into the strengths and weaknesses of e-learning content. They have the knowledge and insights to make a difference – and that’s usually the reason they’ve got into education in the first place.

Secondly, capturing rich data in the assessment process is essential to an integrated and scientific approach to continuous improvement across the entire e-learning process and organization. While other data is also important, data from the assessment process complements learner data, and helps provide a more complete view of overall e-learning performance.

Question: What do you mean by an “integrated and scientific approach”?

Enrolo CEO Adam Menary:

HACCP (food safety) systems use continuously-collected data to help ensure that food is safe to eat and free from contamination. Historically, e-learning has lacked that grade of data. For example, while SCORM involved collecting data that a student had successfully completed an assessment, SCORM might not be configured to keep track of which questions they answered, which they failed, where they got help or feedback, and whether that made a useful difference. This meant the data didn’t support efforts to statistically or scientifically track down areas where issues were occurring, or to identify whether changes were actually improvements.

The limitations of SCORM have been widely recognised, and xAPI emerged to address some of these. However, the focus remains on the learner and capturing learning-related data. Complementing this with data specifically about the assessment process can do two things: support work to streamline and improve assessment activities, and also provide insights about the overall performance of e-learning.

Question: Do you have any thoughts on how this might affect assessment from a learner’s perspective?

Enrolo CEO Adam Menary:

Yes. Multiple choice questions, true/false, checklists and the like are basic online forms of traditional assessment approaches but today they should be viewed by educators as simple check your progress tools rather than actual assessments. And sure, like others, Enrolo has done some interesting and innovative things with interactive check your progress. However, across the industry, e-learning has been constrained by a particular horizon of assessment possibilities, based on the costs, abilities, and practicalities of an assessor working in somewhat traditional manner. Automation, data and tools can help expand that horizon of assessment possibilities – and so offers further opportunities to transform e-learning.

And really, this is essential. Some of e-learning’s traditional assessment approaches are not that great. They can be blunt and ineffective tools for confirming a learner’s understanding.

We’re starting to explore ways to completely change assessment tasks, and learning content, based on the new possibilities of automation-enabled assessment. For example, what might it mean to have more free-form styles of assessment task, or much faster cycles of submission and feedback. Or imagine having assessment feedback going beyond comments – for example, using it the assessment feedback to dynamically reconfigure e-learning content to leverage a student’s strengths, or provide extra examples and explanations to address areas of weaknesses.

As the traditional true/false, pick from a list, or drag and drop type questions are relegated to check your progress aids I think we’re going to see some really important innovations and powerful new approaches to assessment and meaningful learning outcomes.

Guide to online training Best practice integrity measures

Guidelines for online training best practice integrity measures.PDF

Foreword

This white paper provides, authors of units of competency, industry representatives, regulators, policy writers and Registered Training Organisations (RTO’s) an insight into the online integrity measures that MUST be in place to meet compliance with the Standards for Registered Training Organisations 2015.

Understanding the best practiced online integrity measures in this white paper will assist:

Authors of units of competency to reduce the likelihood of accidently restricting the ability for RTO’s to effectively use the online mode of delivery.

Industry Representatives to boost their confidence that learning outcomes can be achieved from online mode of delivery.

Regulators & policy writers, at a state or territory level to specify aspects of online integrity as mandatory requirements for state-based, approved training provider, conditions of approval. 

RTO’s in the interpretation of the Standards for RTO’s 2015 in relation to the online training mode of delivery and assist with training and assessment strategy development and compliance.

Students to receive value for money with real learning outcomes and valuable skills and knowledge for employment via a flexible, effective and efficient online mode of delivery

Disclosure: All of the integrity measures mentioned in this white paper including, enrolment, training, and assessment (including demonstration of skills in simulated online environments and serious games), were developed for several units of competency using the Enrolo learning management system (LMS).  These unit of competency were audited by ASQA, and successfully added to an RTO’s scope for distance learning via asynchronous and synchronous, 100% online learning mode of delivery.

Disclaimer:  Online training is a rapidly evolving mode of delivery and any information provided in this white paper are the opinion of the authors and are open to interpretation and are not to be relied on for compliance purposes.

Contents

Australian Accredited online training  Best practice integrity measures WHITE PAPER.. 1

Foreword. 1

Background. 4

The Future of Online Learning. 5

Benefits of high integrity online training. 5

High integrity online training costs. 6

Autonomy to Author Training and Assessment is Essential 7

Online Integrity Measures. 7

  1. Confirming Student Identity. 7

1.1.         Valid Photo Identification. 8

1.2.         Unique Student Identification. 8

1.3.         Website with HTTPS Security. 8

1.4.         Unique username. 8

1.5     Secure Password. 8

1.6     Identification during phone support 8

1.7     Terms and Conditions. 9

2        Assessing Student Competency. 9

2.1     Free text 9

2.2     Video Demonstration – 9

2.3     Lock out – 9

2.4     Multi choice wrong ratio. 9

2.5     Not given answer – 9

2.6     No direct answer link – 9

2.7     Random questions. 9

2.8     Question banks. 10

2.9     Prevent Concurrent viewing of Learning material and Assessment 10

2.10       Progress through the assessment is clearly indicated – complying with the fairness guidelines of the Standard’s for RTO’s. 10

3        Signing Off of Student Competency. 10

3.1     Use of qualified trainers – 10

3.2     Trainer Roster. 10

3.3     Trainer availability. 10

3.4     Human validation. 10

4        Minimising the Potential for Fraudulent Activity. 10

4.1     Capture the student’s IP address – 11

4.2     A system for monitoring cheating. 11

4.3     Student name fields must be locked. 11

4.5     Must be able to intervene. 11

4.6     Warn about fraudulent 11

4.7     Ability to report suspicious activity. 11

4.8     Deliver training via SSL.. 11

4.9     Agree to terms and conditions. 11

4.10       Statutory Declaration. 11

5        Course Content 11

5.1     Volume of Learning. 11

5.1.1      Minimum learning time. 12

5.1.2      LMS capability to manage time. 12

5.1.3      Active learning. 12

6        Demonstration of skills in online simulated environments. 12

7        Issuing Statements of Attainment 12

7.1     No Name change. 13

7.2     Unique ID.. 13

7.3     Watermark. 13

7.4     Locked PDF.. 13

8        Learning Management System Data Capture Requirements. 13

8.2     Number of attempts. 13

8.3     Result of all attempts. 13

8.4     Data base time stamp of data – 13

9        Auditing Approach for Online. 14

10      Feedback welcome and appreciated. 15

Background

In response to the Covid19 pandemic restrictions, there has been an unprecedented move by traditional in-class training providers to “online training” mostly using classroom, live streaming, webinar technology.  As Registered Training Organisations (RTO’s) progress to more sophisticated self-paced learning presentations and online assessments, many RTO’s are either; unaware of the online integrity requirements or, in some cases are deliberately not implementing them to cut costs and compete with the lowest cost, non-compliant providers already in the market.  In the meantime, following a strategic review into online training, ASQA are in a better position to audit these low integrity providers to help level the playing field for serious online training providers.

Many employers have experienced lack of knowledge and poor skills form students who have gained qualifications from low-integrity, low-cost, online training (whereby statements of attainment can be obtained by students in a matter of minutes by simply clicking without reading or engaging with course material, or randomly guessing until they pass an assessment that is automatically marked by a computer) and as a result may mistrust online qualifications.

Training providers that are cutting corners with online training are leading educators in a race to the bottom in terms of poor quality of training and assessment, and ultimately undermining the student’s value for money.  The result of lack of integrity is not only poor learning outcomes and inadequate training, leading to inefficiencies or dangerous practices in the workplace, but also damage to the reputation of the online or the ‘e-learning’ industry.  This is also putting the RTO at risk of compliance breaches with ASQA who are fully aware of online integrity requirements such as those outlined in this whitepaper.

Sophisticated online training has much to offer the training sector including providing better access for remote and regional communities, and better flexibility for students, who may be working, studying, or parenting at home. In many cases high integrity online training can provide better learning outcomes through the use of engaging multi-media technologies such as Scenario Based Training Interactives and Serious Games.  Another advantage of online training for students is that it can be studied at the learner’s own pace and focus on areas they are having comprehension difficulties without holding up the whole class.  High integrity online training allows students to have as much access to qualified trainers as they need, compared with in-class training where one-on-one contact is limited. High integrity online training where data is properly captured in an LMS can also improve evidence of training and assessment.

To date there has been little guidance from authors of the units of competency for online integrity who maintain that it is up to RTO’s to determine the mode of delivery of their training and self-determine how to achieve the learning and assessment outcomes of a unit of competency via their training and assessment strategy.  It is also assumed that ASQA will audit against the Standards for Registered Training Organisations (RTOs) 2015 to ensure integrity rather than adding online integrity measures as a requirement to units of competency.  However, the Standards for RTO’s 2015 do not provide implicit or obvious guidance for new online providers to ensure that their online training has high integrity.  Rather, the requirements for online integrity fall under the Standards for Registered Training Organisations 2015  Clauses 1.8 – 1.12 Conduct effective assessment including principles of assessment (Fairness, flexibility, Validity and reliability) and rules of evidence (Validity, Sufficiency, Authenticity and Currency).  To be compliant RTO’s must consider the integrity measures outlined in this white paper in that context.

There are examples of regulators (e.g. state and territory regulatory authorities) stipulating certain capability via conditions of approval to deliver minimum integrity measures, (many as outlined in this white paper), in order to be a state based approved provider.  This may have been in response to ASQA not previously ranking online integrity as a high risk or not fully auditing all online providers courses for regulatory completeness.  However, ASQA has recently conducted a strategic review into online training and is aware of and taking action on the integrity issues highlighted in this white paper.

In the past some of the Industry Skills Councils have actively added requirements into units of competency to try to exclude the online mode of delivery.  The industry representatives on these skills councils may have had members with bad experience with staff who completed low integrity online training, and they may have limited understanding of the availability of high integrity online technologies and the capability of technology to deliver high quality learning outcomes.  On the other hand, some skills councils have removed challenging aspects of the training unit because they incorrectly believed that online integrity was overly difficult to achieve online (such as demonstration of skills in a simulated industry environment) and added to cost of delivery and elevated price.  This has resulted in units of competency that deliver poor learning outcome but can easily be delivered online by low integrity providers at low cost.

In March 2022 ASQA was nominated as the ‘National Training Package Assurance Body’, and moving forward, armed with fresh insights into online integrity issues from the strategic review into online training,  the inconsistency between how the units of competency are being authored vs how integrity is managed, will be addressed.  ASQA will also be able to act against low integrity online providers, establishing a more level playing field for those that are taking integrity seriously via the online mode of delivery.

The Future of Online Learning

Automation in the training profession is continuing to advance at an ever-increasing speed heading to exponential growth. Examples include interactive videos, computer auto marking, interactive video assessment and virtual reality scenarios, Scenario Based Training Interactives (SBTI’s) and serious games for the demonstration of skills.  The experience API is also allowing online training to move beyond the boundaries of formally structured content development and the classroom.

Many qualifications that are taught using sophisticated online training methods can already deliver the same, if not better learning outcomes than in-class training.  Since online training innovation is growing at an exponential rate it is advantageous if high integrity online training providers are allowed to author their own learning materials to keep up with these changes.

Benefits of high integrity online training

The benefits of high integrity online training include but are not limited to;

  • Improved accessibility, particularly for students in remote or regional areas or parents with young children.
  • Students can learn in their own time and do not have to take time off work or study.
  • Provides online experiences that are relatable, can trigger behavioural change, facilitate decision making and increase critical thinking skills.
  • Provides understanding of the impact and consequence of their decisions and choices in a safe mode and gives them an approach to hone skills nada chive higher proficiency levels.
  • Crating sticking learning experiences, facilities problem solving, provides guided exploration, and a safe practice zone to gain proficiency and mastery.
  • A successful learning strategy because it can be designed to be: motivating, engaging, immersive, relevant and relatable, replicable, consistent, challenging, and rewarding.
  • Reduced travel costs such as transport and parking during training and reduced carbon emissions.
  • Greater consistency in terms of content delivered and the assessment method.
  • Quicker response times to updating learning content resulting in more up to date learning resources.
  • More empirical data can be collected for demonstrating competency and adherence to high integrity.
  • Reduced administrative and non-value-adding work such as marking quizzes or issuing certificates.
  • Less monotonous tasks for trainers and more time spent with students that need their assistance.
  • Reduced human error related to less reliance on monotonous and repetitive tasks.
  • Learners can work at their own pace and concentrate on areas they do not understand.
  • Students can spend as much time with qualified trainers as they need.
  • Students can access learning material ongoing and can also be provided with updates to the learning material.
  • Reduced room hire costs which are however, offset by online tools, online delivery, technical support and online course content development.

High integrity online training costs

There is a misperception with some training providers that online training is a way to cut costs such as eliminating trainers and classrooms and automating everything including assessment with a computer to increase profits.  However, if a training organisation is meeting the high integrity online measures outlined in this white paper and meeting the requirements of the Standards for RTO’s 2015, despite the reduced cost of classrooms, the cost of delivering effective online training is equal to if not greater than in-class training.  This is because not only do all of the existing requirements that need to be in place for in-class training such as; qualified trainers and assessors, mapping, internal audits, continuous improvement registers and so on, still need to be in place for online, but in addition to these costs, online providers need to purchase and maintain: an online learning environment (LMS), student management system (SMS) and build digital online learning material and robust digital assessments.  There is also a much higher requirement for one-on-one engagement between students and trainers and assessors when students need assistance with comprehension and get locked out of an assessment. This one-on-one approach dramatically changes the learning landscape and massively increases costs. This, is only the case if the design of the learning has high integrity which forces the student to contact the trainer when needed such as being locked out from continuing.   This is a paradigm shift where the learning management system does most of the training and the trainer and assessor assists those that need help as much as they need, so much so in this context they may also be considered a tutor.

Whilst some RTO’s believe that an online slide show with voice over followed by an online quiz requiring a student to guess a few true / false questions, that are automatically marked by a computer, meets the Standards for RTOS’ 2015, effective and engaging online training includes; videos, gamification, interactive scenarios, simulated environments and animations with voice overs and challenging exercises. Effective assessment requires students to be locked out from excessive guessing, essays, and free text responses as well as video demonstration of skills or offline activities. If online training contains these kinds of elements, then it can cost up to AUD$30,000 per hour of e-learning output.  Compare that to a power point presentation created for in-class training and high integrity online training providers are spending approximately $180,000 for a six-hour online course development vs a few thousand for an equivalent in-class presentation or even less for low integrity online courses.

Autonomy to Author Training and Assessment is Essential

From a training perspective, it is essential to realise that to achieve efficient and effective online learning, that the ability for a training provider to edit content and assessment is inseparable.  This is because assessments are directly related to the learning material and to achieve accurate mapping to a unit of competency assessment questions must be written in such a way that may require regular edits to the content.  Furthermore, if an assessment question is causing confusion or returning a high rate of incorrect answers, it may be the case that the content needs to be edited rather than the assessment question.  Another issue that can occur with mandatory content provided by regulators is that if the government bodies are renamed or restructured, often they lose funding to maintain content and this content is then not able to be updated for name or content changes by training providers as they don’t have open-source access.  Government bodies that attempt to provide learning material to training providers may face increased costs and struggle to keep up with the rapidly evolving innovation and this should be avoided.  However, there have been some good examples of authorities providing effective videos that RTOS have incorporated into their learning material.

Rather than being overly prescriptive for state requirements an alternate approach that regulators can consider is to provide elements such as videos or interactive content that may be included by an online training provider as part of a course.  A good example of this is the “Behind the bar RSA training video – Just one more” authored by the Office of Liquor and Gaming Regulation Queensland, who invite training providers to use the content in their courses royalty free.

A consideration for regulators providing such content is to provide open-source access to authored content so that training providers and modify it as needed.

A key take home for regulators risk assessment for online training is that low sale prices for online courses well below the average price being offered by others in the market, are a likely reflection of low expenditure in developing the content and therefore unlikely to meet the Standards for RTOs 2015.  Very low online course sale prices that are below the cost of being able to pay a trainer to be involved may reflect the lack of students being forced to access trainers due to comprehension issues or excessive guessing.

Online Integrity Measures

RTOs developing online training must ensure that all the following measures are in place to meet the requirements of the Standards for RTO’s 2015.

1.      Confirming Student Identity

Confirming student identify in an online setting is of paramount importance in reducing fraudulent activity, i.e. by ensuring that the person doing the training is the person obtaining the Statement of Attainment/Certificate.  Fingerprint and face scans are becoming more prevalent however, until these are more widely accepted the following procedures and features are considered best practice to verify that a student, who is enrolled in an online course, is the person who also completes the course and receives the certification:

1.1.   Valid Photo Identification

Students must provide via a secure (https) online website, a copy of current government-issued photo ID which is reviewed and accepted or rejected by trained staff.  Details captured and verified must include the full name, date of birth, and contact details (including residential, valid email address, and contact phone number) of the student.

The residential address of the student must be validated.  This can be improved using an online address validation service or address look up tool.

Students must not be allowed to change personal details themselves – they need to lodge a request and provide evidence, such as government-issued photo ID, marriage certificate, change of name certificate, etc. where required.

For video assessment, the valid photo ID should be displayed against a student account when videos are being assessed, or during video conferencing so trainers can ascertain that they are speaking with the student who has made the recording.

In many circumstances this procedure may exceed identification integrity taking place in classroom training environments.

1.2.   Unique Student Identification

In Australia it is a government requirement for nationally recognised training, that before completing their enrolment students must obtain a Unique Student Identifier (USI) and the details must match their valid photo identification provided before they can be issued with their Statement of Attainment/Certificate (qualification).

  • Website with HTTPS Security
    Students must only access online courses through a https secure website that requires a login and password.

1.4.   Unique username

Students must create their own unique username such as an email address

1.5  Secure Password

If an overly complex password is provided to the student then they will need to write it down and this reduces the security of the password so students must create a secure password themselves.

Students must  be provided with best practice information regarding setting up and maintaining the security of their passwords.

Student password recovery must be via clicking a link to recover their password that sends them details on how to do this via their registered email. Passwords must not be able to be changed without the student logging into the account or providing the answer to a secret question if they have lost email access.

  • Identification during phone support
    When a student contacts a training provider by phone or email requesting help, the provider must have procedures in place to check their identity before providing any account details or assistance such as an answer to a secret question.
  • Terms and Conditions
    To place a legal onus of responsibility on the student, the students must agree to terms and conditions at the start of an online course, confirming that they are the person doing the training. This can be done via a statutory declaration or equivalent where required.

    Once complete and before generating and accessing the final PDF Statement of Attainment/Certificate the student must declare that they were the person that did the training and that they did so without assistance.  This must be recorded in the electronic management system and be able to be provided as evidence for legal breaches.

2        Assessing Student Competency

Assessments must be mapped to a unit of competency to demonstrate that all the required learning elements have been assessed.

In an online environment, assessments that are designed to provide a true reflection of the knowledge and skills that have been achieved are one method of proving competency has been gained.  Offline activities can also be undertaken and witnessed by a suitably qualified supervisor and their accounts included for consideration.

For an RTO to be compliant in an online environment, (i.e. in order for assessments to demonstrate that a student has achieved understanding and competency), their online system must have the following integrity measures:

  • Free text – Require at least 1 question in each assessment (ideally 25%) to be answered with free text sentences (ie at least 5 words or more, not a single word or number) allowing a proper critical response to questions.
  • Video Demonstration – In the case of demonstration of skills, require video footage to be submitted to capture a student’s ability to demonstrate a task or skill.
  • Lock out – If a student gets the same question wrong three (3) times they must be “locked out” and required to talk to the training organisation and if they need assistance, they must be referred to a qualified trainer ie rather than keep guessing until they get the question correct through a process of elimination.
  • Multi choice wrong ratio – Multiple choice questions must have more wrong combinations than available attempts and designed so that students will be locked out before guessing through a process of elimination.
  • Not given answer – Students must not be provided with the correct answer eg before or after attempting an assessment before they get the answer correct themselves. If re-attempting assessments, students must get the full range of options and not be shown which answers have already been attempted and marked as incorrect.
  • No direct answer link – Students must not be provided with links from the question directly to the correct answers in the learning material.
  • Random questions – The order of assessments must be randomised to reduce the likelihood of cheat sheets.
  • Question banks -Because of the complexity of meeting mapping requirements if questions are randomised, banks of similar questions must be created, and questions randomly selected from those banks to ensure that the particular questions achieve the required mapping.
  • Prevent Concurrent viewing of Learning material and Assessment The online system must ensure that students cannot have the learning material open in one window and the assessment in the other allowing the student to copy and paste answers without critical thinking. This needs to be in place for multiple tabs, multiple browsers on the one device or multiple logins across multiple devices.
  • Progress through the assessment is clearly indicated – complying with the fairness guidelines of the Standard’s for RTO’s.

In contrast to assessments that comply with the above high integrity measures, assessments that are; 100% marked by a computer, predominantly multiple choice with unlimited elimination attempts or true/false by elimination with two attempts, provide no requirement or opportunity for a student to demonstrate their comprehension of the learning material. Whilst it may be argued that by eventually getting a question right though guessing is a form of learning, allowing students to keep attempting questions throughout and entire assessment until they get the correct answer, means the student does not even need to read the question or answer and does not deliver meaningful learning outcomes. Demonstrated competency through rigorous and auditable assessment, in combination with the above integrity measures, is considered an integral requirement for the demonstration of online learning outcomes.

3        Signing Off of Student Competency

Online training providers may try to cut corners and costs by using administration staff with no training qualifications to assist with training.  To cut costs training providers may also reduce access by students to qualified trainers to very short periods. It is essential that qualified trainers are available to students in an online environment.  These trainers may communicate with the students via:

  • Telephone
  • Email
  • Live chat
  • Webinars
  • Video conferencing

To improve Integrity measures RTO’s must implement the following:

  • Use of qualified trainers for assisting with training support. All trainers employed by a training organisation must be on a training register. 
  • Trainer Roster – Total trainer hours must as a minimum reflect availability during normal office hours.
  • Trainer availability Trainers must be available at least during normal business hours and must assist learners requesting training within 30 minutes of a request.
  • Human validation – An online statement of attainment must not be issued until the enrolment details confirmed and assessment and learning activities of the student have been reviewed by a qualified trainer. This sign off process must be demonstrated by data in the learning management system.  

4        Minimising the Potential for Fraudulent Activity

RTO’s must ensure that the following integrity measures are in place to minimise the potential for fraudulent activity:

  • Capture the student’s IP address and monitor if bulk users are coming from the one IP.
  • A system for monitoring cheating must be in place that flags when different students give the exact same free text answers to free text assessment questions.
  • Student name fields must be locked so that these can only be changed by administration staff upon request and subsequent verification of the student’s identification.
  • Lock-out’ access – to an assessment after failing a set number of attempts of each question (3)  and must speak with a qualified trainer to be unlocked if they do not know the answer.
  • Must be able to intervene and make direct contact with the student if required.
  • Warn about fraudulent activity – Notify students that certification may be voided if fraudulent information is provided or fraudulent activity is detected.
  • Ability to report suspicious activity – The learning system must have the ability to identify and report immediately any suspicious activity by students undertaking the course to the appropriate authority.
  • Deliver training via SSL – SSL to reduce third party fraudulent activity ie HTTPS at start of a domain..
  • Agree to terms and conditions – Capture electronic evidence that the student has agreed to terms and conditions relating to not being assisted in any way during assessment and that they are the one that undertook and completed the training.
  • Statutory Declaration – If adequate integrity measures are not in place, or a student has breached an integrity measure, students must sign and upload a witnessed statutory declaration that they are undertaking the training without assistance.

5        Course Content

5.1  Volume of Learning

The Australian Quality Framework says that “the volume of learning” includes guided learning, individual study, research, learning activities in the workplace and assessment activities. It could be argued that rather than applying this at a qualification level, the volume of learning should be applied at a unit of competency level and rather than minimum duration the relationship between the quality of the content being able to map to a rigorous assessment should be the deciding factor in assessing competency.  As the AQF goes on to say:

“The duration of the delivery of the qualification may vary from the volume of learning specified for the qualification. Providers may offer the qualification in more or less time than the specified volume of learning, provided that delivery arrangements give students sufficient opportunity to achieve the learning outcomes for the qualification type, level and discipline.”

This could be applied to a competency rather than the whole qualification.

When looking at VET training, we need to remember that students come from hugely various backgrounds and experience, from Year 11 students undertaking a VETiS program to someone with a PhD who is looking for a career change to an older person with a wide-ranging life and work experience.  This all affects how quickly they can become competent, not only based on their level of experience and in the skills they already have or need to learn, but their ability to undertake and comprehend the training in the first place.

While setting minimum durations based on the time a learner (who is new to the industry area) would be required to undertake supervised learning and assessment activities (ASQA report A review of issues relating to unduly short training page 15 has merit), the dilemma remains that some people undertaking formal study will already be to some degree competent and not require the same minimum duration.

In terms of auditing the appropriate volume of learning on a student-by-student basis, training providers using data from the AVETMISS enrolment should be able to demonstrate whether a student is new to the field and requires what could be considered as minimal learning hours, as opposed to someone who is requalifying and could reasonably be expected to complete a competency in a much shorter time frame.  I.e. based on evaluating the needs of the student from the language, literacy and numeracy (LL&N) together with the AVETMISS enrolment data a training provider should be able to specify the expected or minimum volume of learning required.   In a classroom this could be part of the learning plan.  In an online environment, this tailored learning plan must be established and managed in a dynamic way, i.e. by having certain locks or minimum interactive learning hours completed before assessment is made available.

To improve course content Integrity measures authors must stipulate:

  • Minimum learning time – Access to onlnline assessment should not be permitted until a minimum amount of learning that is determined on a student by student basis has been achieved
  • LMS capability to manage time – The LMS must be able to manage the minimum online learning time limits and must also be able to record the amount of time spent in learning activities.
  • Active learning the LMS must be able to determine whether the learner is actively learning and stop the clock if they are not eg if a presentation plays for an hour can the student take a break and come back and have their learning record show one hour?

6        Demonstration of skills in online simulated environments

Some units of competency require the demonstration of skills in a simulated industry environment.  With very sophisticated training and assessment strategies combined with advanced technological approaches such as Scenario Basted Training Interactives (SBTI), video capture and serious games this can be achieved however, regulators now know the difference between a valid implementation of these approaches and unsophisticated approaches including:

  • has a simulated environment been constructed either; in the online scenario or, by the student in their physical environment, meeting the requirements of equipment needed as specified by the unit of competency or implementation guide such as printing out signage or physical equipment being present?
  • Has the simulated environment been validated as adequate by a suitably qualified trainer/assessor and is there evidence of this?
  • Does the student provide evidence of their skills in the simulated environment e.g. by submitting video evidence as compared to just a voice recording which would not?
  • How is the video evidence checked for authenticity in terms of cheating or that the person in the submitted video is the student as enrolled with photo ID?
  • Are gamified learning or assessment tools sophisticated enough such as serious games that deliver meaningful skills training or assessment or just gimmicks? I.e. is the logic and sophistication of the data collected in such games adequate to prove competency?

7        Issuing Statements of Attainment

Fraudulently copying or editing statements of attainment can be done no matter the mode of delivery eg in-class or online.  However, If statements of attainment are digitally issued online, policy RTO’s must ensure that the following integrity measures are in place to minimise fraudulent activity:

  • No Name change – The student must not be able change their name online and an administrator should only be permitted to do this after identifying the student and then sighting an updated form of photographic identification meeting the identification integrity requirements with the new name.
  • Unique ID – The certificate must contain a unique number that is linked to the unique student username.
  • Watermark – The certificate must have a background water mark with the unique details of the certificate, e.g. student name, date of birth, issue date, certificate number, signature of issuing authority, expiration date over the top of the watermark(s).
  • Locked PDF – The certificate must be in a PDF format and the PDF must have security enabled that locks the PDF and prevents edits.

8        Learning Management System Data Capture Requirements

Without high granularity of data capture related to the digital learning environment, it is more difficult to demonstrate that integrity has been maintained.  Some Learning Management Systems (LMS’s) may only record the course sections that have or have not been accessed and that assessments have or have not been successfully completed, i.e. without capturing the granularity of the data such as the actual responses given to pass the assessment or the number of assessments or total learning engagement time.

In order to achieve integrity, a LMS must be used that has the sophistication to not only have integrity features in place, but also be able to collect data for integrity and track student activity and progress through the learning material and assessments including;

  • Content engagement – Record the amount of time that a student has meaningfully interacted with learning content,
  • Number of attempts – Record how many attempts students have made for each assessment
  • Result of all attempts – Capture the result of each attempt not just that a student at some point provide correct answers,
  • Data base time stamp of data –the regulator may interrogate the formula in the learning management system for determination of duration of learning and confirm that it is not fraudulently adding hours. There needs to be detailed scrutiny and proof of validation of duration data by the online system 

RTO’s must ensure that their learning management system used to deliver training online does so at a level of granularity that allows auditors to view every single piece of data submitted by the student in relation to assessment whether that submission is correct or not.  Also that the number of attempts and evidence of lockout after eg 3 attempts can be show with data evidencing how the lockout was overturned and by whom. Most importantly that evidence of duration is and the way that is has been calculated is valid.

All the integrity measures mentioned in this white paper including, enrolment, training, and assessment (including demonstration of skills in simulated online environments and serious games), were developed for several units of competency using the Enrolo learning management system (LMS).  These unit of competency were audited by ASQA, and successfully added to an RTO’s scope for distance learning via asynchronous and synchronous, 100% online learning mode of delivery.

9        Auditing Approach for Online

Student satisfaction scores for single units of competency delivered 100% online are not always a valid representation of the quality or integrity of online training.  Student satisfaction of very poor-quality online learning and low integrity training and assessment may be very high.  This is because students that need mandatory qualifications are often on a very short timeline, not interested in the quality of the learning but rather a short duration and will therefore give high satisfaction scores for cheap prices and short duration courses.

Because of the integrity issues highlighted above, asynchronous online training where learning and assessment is delivered in the absence of a trainer or assessor can be considered higher risk than e.g. a trainer delivering a lesion via a webinar.

The risk of training units can also be considered in terms of the amount of potential health risks to students or the community from inadequate training and the amount of regulatory oversight that exists in that industry sector.

Whilst a well written Training and Assessment Strategy (TAS) is the essential first point of reference during an audit of the validity of the online training the ONLY way to tell if the training and assessment strategy is being implemented in either the classroom or online is for the auditor to anonymously purchase the course, undertake the training and gain a statement of attainment.  (mystery shop) During the process of obtaining the statement of attainment the auditor will check that the course:

  • meets all of the items listed in this best practice document as well as the training and assessment strategy.
  • Prevents the student guessing their way through the assessment, that they get locked out after three wrong attempts
  • Allows the student to access meaningful and timely trainer support in accordance with the training and assessment strategy.
  • Has valid learning material as defined by the standards for RTO’s 2015 i.e. whereby the student is challenged throughout and is given an opportunity to put their knowledge into practice prior to final assessment otherwise presentations are not considered part of learning duration.
  • Online assessment and learning materials comply with the standards for RTO’s 2015 in relation to integrity, authenticity, reliability and fairness i.e. in relation to these best practice online integrity measures.
  • Reported duration matches the actual course duration as undertaken by the auditor
  • Does not have loopholes allowing shortcuts and confirm the impact on nonattendance on reported duration of training in the learning management system.

Because electronic data can be easily manipulated Auditors are more likely to ask for 100% of the data not samples of data and data collected during mystery shopping will be compared to what is being reported.

10   Feedback welcome and appreciated

We value feedback form industry, regulators, trainer/assessors, training organisations and students please let us know if you have any feedback in relation to this white paper so that we can consider it for our next update.

feedback@enrolo.com

Disclosure: All of the integrity measures mentioned in this white paper including, enrolment, training, and assessment (including demonstration of skills in simulated online environments and serious games), were developed for several units of competency using the Enrolo learning management system (LMS).  These unit of competency were audited by ASQA, and successfully added to an RTO’s scope for distance learning via asynchronous and synchronous, 100% online learning mode of delivery.

Disclaimer:  Online training is a rapidly evolving mode of delivery and any information provided in this white paper are the opinion of the authors and are open to interpretation and are not to be relied on for compliance purposes.

Australian Online Training Best Practice Guide for Regulators and RTO’s

Australian Online Training Best Practice Guide for Regulators and RTO’s – White Paper

Aust Online Training Best Practice Integrity Measures-V8.PDF 

Foreword

This white paper is intended to provide, policy writers, authors of units of competency, regulators and online training providers with integrity measures that must be in place to ensure the integrity of online learning outcomes.

In the absence of  explicit requirements for online integrity in either; the Standards for RTO’s 2015 or, units of competency, It is essential that regulators, policy writers and authors of units of competency, understand the importance of both the granularity of online training data and integrity measure features, so that they can specify them as mandatory requirements, improving integrity and the ability of the Training Regulator to act.

Executive Summary

Technological advancement is occurring in the Vocational Education and Training (VET) sector ahead of guidelines, legislation and quality control mechanisms by industry, government and regulators, to ensure that there is adequate integrity underpinning statements of attainment issued via an online mode of delivery.  In some cases this is leading to low cost and low quality, online learning outcomes as well as giving online training a poor reputation.

High-integrity online training has much to offer the training sector including; providing better access for remote and regional communities, and better flexibility for students, who may be working, studying or parents at home. In many cases high integrity online training can provide better learning outcomes through the use of engaging multi-media technologies that can be studied at the learner’s own pace.  High integrity online training can also improve evidence of assessment and demonstrate high compliance to regulations.

To ensure that learning outcomes are achieved in an online environment, certain technology with the capability to deliver minimum integrity measures, as outlined in this white paper, must be in place.  Specific online Integrity measures also need to be in place to reduce fraudulent activity.

Whilst examples of industry best practice guidelines for online training do exist, many training providers are either not aware of them or, in some cases, are deliberately not implementing them to cut costs and compete with the lowest cost providers in the market.  There are examples of regulators (e.g. state and territory liquor authorities) stipulating mandatory online integrity measures however, these requirements are often not enforced by the Australian Skills Quality Authority (ASQA) due to lack of resources or technical understanding.

Under the Federal Register of Legislation the Standards for Registered Training Organisations (RTOs) 2015 do not provide adequate guidance to ensure that online training has high integrity or effective learning outcomes.

Many employers have experienced lack of knowledge and poor skills form students who have gained qualifications from low-integrity, low-cost online training and may mistrust all online qualifications.

Despite advancement in technology such as virtual reality or simulated environments, rather than adding online integrity measures as a requirement to units of competency (i.e. to encourage high integrity online training),  some of the Industry Skills Councils have actively added requirements into units of competency to try to exclude the online mode of delivery.  The industry representatives on these skills councils may have limited understanding of online technologies and the need for high integrity measures.  This results in units of competency that can easily be delivered by low integrity providers who deliver poor learning outcomes.

Armed with an out of date Standard for Registered Training Organisations (RTO) 2015 and units of competency that do not specify online integrity measures, the Vocational Education and Training (VET) regulator, the Australian Skills Quality Authority (ASQA), is often unable to act against unscrupulous profiteering providers delivering low integrity online training.

Training providers that are cutting corners with online training are leading educators in a race to the bottom in terms of poor quality of training and assessment, and ultimately undermining the student’s value for money.  The result of lack of integrity is not only poor learning outcomes and inadequate training, leading to inefficiencies or dangerous practices in the workplace, but also damage to the reputation of the online or the ‘e-learning’ industry as a whole.

A small number of unscrupulous providers have taken over approximately 80% of some online training markets and are unfairly making it uneconomic for high-integrity and higher-cost training providers to compete.  Many online training providers that are not complying with best practice online integrity measures are producing what is known as ’page turning’ or ‘tick and flick’ learning with assessments such as true/ false or multiple choice that can be passed by guessing without restriction.  Based on the scope of some units of competency this conduct is endangering the lives of patrons and people in the community.

To bring to the fore the advantages of online training, and eliminate the risks from low integrity online training, policy writers, regulators, and authors of units of competency must ensure online integrity measures are specified in order to allow the regulator to act.

The Future of Online Learning

Automation in the training profession is continuing to advance at an ever-increasing speed. Examples include interactive videos, computer auto marking, interactive video assessment and virtual reality scenarios for the demonstration of skills.  The experience API is also allowing online training to move beyond the boundaries of formally structured content development and the classroom.

Many qualifications that are taught using sophisticated online training methods can already deliver the same, if not better learning outcomes than in-class training.  Since online training innovation is growing at an exponential rate it is advantageous if training providers are allowed to author their own learning materials to keep up with these changes.

Benefits of high integrity online training

The benefits of high integrity online training include but are not limited to;

  • Improved accessibility, particularly for students in remote or regional areas or parents with young children.
  • Students can learn in their own time and do not have to take time off work or study.
  • Reduced travel costs such as transport and parking during training and also reduced carbon emissions.
  • Greater consistency in terms of content delivered and the assessment method.
  • Quicker response times to updating learning content resulting in more up to date learning resources.
  • More empirical data can be collected for demonstrating competency and adherence to high integrity.
  • Reduced administrative and non-value-adding work such as marking quizzes or issuing certificates.
  • Less monotonous tasks for trainers and more time spent with students that need their assistance.
  • Reduced human error related to less reliance on monotonous and repetitive tasks.
  • Learners can work at their own pace and concentrate on areas they do not understand.
  • Students can spend as much time with qualified trainers as they need.
  • Students can access learning material ongoing and can also be provided with updates to the learning material.
  • Reduced room hire costs which are however, offset by online tools, online delivery, technical support and online course content development.

High integrity online training costs

There is a perception that online training is a way for training providers to cut costs such as eliminating trainers and class rooms and automating everything with a computer to increase profits.  However, if a training organisation is meeting the high integrity online measures outlined in this white paper, despite the reduced cost of hiring classrooms, the cost of delivering effective online training is equal to if not greater than in-class training.  This is because not only do all of the existing requirements that need to be in place for in-class training such as; qualified trainers and assessors, assessment material, mapping, internal audits, continuous improvement registers and so on, still need to be in place for online, but in addition to these costs, online providers need to purchase and maintain an online learning environment and build digital online learning material and robust assessments.

Whilst a 200 page PDF document available online and a quiz requiring a student to tick a few true / false questions meets the Standards for RTOS’ 2015 and most units of competency, effective and engaging online training includes videos, gamification, interactive scenarios, simulated environments and animations with voice overs and challenging exercises as well as additional reading materials or offline activities. If online training contains these kinds of elements then it can cost up to AUD$30,000 per hour of e-learning output.  Compare that to a power point presentation created for in class training and we are talking $180,000 vs a few thousand for a six hour course.

Many “online providers” are cutting corners by developing simple e-books or a PDF with a voice over and a next button between pages.  Whist it is possible to map such “online” material to the elements and criteria of a unit of a training package, there needs to be more emphasis on being able to demonstrate that a student can achieve competency from such material through rigorously linking it to high integrity online assessments.

Regulators that attempt to provide learning material to training providers may face increased costs and struggle to keep up with the rapidly evolving innovation.

An alternate approach that regulators may consider is to provide elements such as videos or interactive content that may be included by an online training provider as part of a course.  A good example of this is the “Behind the bar RSA interactive training video – Just one more” authored by the Office of Liquor and Gaming Regulation Queensland, who invite training providers to use the content in their courses royalty free.

From a training perspective, it is essential to realise that to achieve efficient and effective online learning,  that the ability for a training provider to edit content and assessment is inseparable.  This is because assessments are directly related to the learning material and in order to achieve accurate mapping to a unit of competency assessment questions must be written in such a way that may require regular edits to the content.  Furthermore, if an assessment question is causing confusion or returning a high rate of incorrect answers, it may be the case that the content needs to be edited rather than the assessment question.  Another issue that can occur with mandatory content provided by regulators is that if the government bodies are renamed or restructured, often they lose funding to maintain content and this content is then not able to be updated for name or content changes by training providers as they don’t have open source access.  A consideration for regulators providing such content is to provide open source access to authored content so that training providers and modify it as needed.

A key take home for regulators is that low sale prices for online courses are a likely reflection of the expenditure put into developing the content and also reflect the lack of real and qualified training resources behind the scenes and should be targeted for audits.

Online Integrity Measures

In the absence of any online integrity regulation, where possible, policy writers should add the following as mandatory requirements:

1.      Confirming Student Identity

Confirming student identify in an online setting is of paramount importance in reducing fraudulent activity, i.e. by ensuring that the person doing the training is the person obtaining the Statement of Attainment/Certificate.

Fingerprint and retina scans are becoming more prevalent however, until these are more widely accepted the following procedures and features are considered best practice to verify that a student, who is enrolled in an online course, is the person who also completes the course and receives the certification:

1.1.   Valid Photo Identification

Students must provide via a secure (https) online website, a copy of current government-issued photo ID which is reviewed and accepted or rejected by trained staff.  Details captured and verified must include the full name, date of birth, and contact details (including residential, valid email address, and contact phone number) of the student.

The residential address of the student must be validated.  This can be improved using an online address validation service or address look up tool.

Students must not be allowed to change personal details themselves – they need to lodge a request and provide evidence, such as government-issued photo ID, marriage certificate, change of name certificate, etc. where required.

For administrators the valid photo ID should be displayed against a student account when videos are being assessed, or during video conferencing so trainers can ascertain that they are speaking with the student who has made the recording.

In many circumstances this procedure may exceed identification integrity taking place in classroom training environments.

1.2.   Unique Student Identification

In Australia it is a government requirement for nationally recognised training, that before completing their enrolment students must obtain a Unique Student Identifier (USI) and the details must match their valid photo identification provided.  In Australia it is a government requirement that students supply their Unique Student Identifier (USI), the details of which must match their valid photo ID provided, before they can be issued with their Statement of Attainment/Certificate (qualification).

  • Website with HTTPS Security
    Students must only access online courses through a https secure website that requires a login and password.

1.4.   Unique username

Students must create their own unique username such as an email address

1.5  Secure Password

If an overly complex password is provided to the student then they will need to write it down and this reduces the security of the password so students must create a secure password themselves.

Students must  be provided with best practice information regarding setting up and maintaining the security of their passwords.

Student password recovery must be via clicking a link to recover their password that sends them details via their registered email. Passwords must not be able to be changed without the student logging into the account.

  • Identification during phone support
    When a student contacts a training provider by phone or email requesting help, the provider must have procedures in place to check their identity before providing any account details or assistance.
  • Terms and Conditions
    Students must agree to terms and conditions at the start of an online course, confirming that they are the person doing the training.  This can be done via a statutory declaration or equivalent where required.Once complete and before generating and accessing the final PDF Statement of Attainment/Certificate the student must declare that they were the person that did the training and that they did so without assistance.  This must be recorded in the electronic management system and be able to be provided as evidence for legal breaches.

2        Assessing Student Competency

Assessments must be mapped to a unit of competency to demonstrate that all of the required learning elements have been assessed.

In an online environment, assessments that are designed to provide a true reflection of the knowledge and skills that have been achieved are one method of proving competency has been gained.  Offline activities can also be undertaken and witnessed by a suitably qualified supervisor and their accounts included for consideration.

Policy writers must mandate that, In an online environment, in order for assessments to demonstrate that a student has achieved understanding and competency, they must have the following integrity measures:

  • Free text – Require at least 1 question in each assessment (ideally 25%) to be answered with free text sentences (ie at least 5 words or more, not a single word or number) allowing a proper critical response to questions.
  • Video Demonstration – In the case of demonstration of skills, require video footage to be submitted to capture a student’s ability to demonstrate a task or skill.
  • Lock out – If a student gets the same question wrong three (3) times they must be “locked out” and requiredto talk to the training organisation and if they need assistance they must be referred to a qualified trainer ie rather than keep guessing until they get the question correct through a process of elimination.
  • Multi choice wrong ratio – Multiple choice questions must have more wrong combinations than available attempts and designed so that students will be locked out before guessing through a process of elimination.
  • Not given answer – Students must not be provided with the correct answer eg before or after attempting an assessment before they get the answer correct themselves. If re attempting assessments, students must get the full range of options and not be shown which answers have already been attempted and marked as incorrect.
  • No direct answer link – Students must not be provided with links from the question directly to the correct answers.
  • Random questions – The order of assessments must be randomised to reduce the likelihood of cheat sheets.
  • Question banks -Because of the complexity of meeting mapping requirements if questions are randomised, banks of similar questions must be created and questions randomly selected from those banks to ensure that the particular questions achieve the required mapping.
  • Prevent Concurrent viewing of Learning material and Assessment The online system must ensure that students cannot have the learning material open in one window and the assessment in the other allowing the student to copy and paste answers without critical thinking. This needs to be in place for multiple tabs, multiple browsers on the one device or multiple logins across multiple devices.
  • Progress through the assessment is clearly indicated – complying with the fairness guidelines of the Standard’s for RTO’s.

In contrast to assessments that comply with the above high integrity measures, assessments that are 100% marked by a computer that are predominantly multiple choice by elimination or true/false by elimination, provide no requirement or opportunity for a student to demonstrate their comprehension of the learning material. Whilst it can be argued that by eventually getting a question right is a form of learning, allowing students to keep attempting questions until they get the correct answer does not deliver meaningful learning outcomes. Demonstrated competency through rigorous and auditable assessment, in combination with the above integrity measures, is considered an integral requirement for the demonstration of online learning outcomes.

3        Signing Off of Student Competency

Online training providers may try to cut corners and costs by using administration staff with no training qualifications to assist with training.  To cut costs training providers may also reduce access by students to qualified trainers to very short periods. It is essential that qualified trainers are available to students in an online environment.  These trainers may communicate with the students via:

  • Telephone
  • Email
  • Live chat
  • Webinars
  • Video conferencing

To improve Integrity measures policy writers must mandate the following:

  • Use of qualified trainers for assisting with training support. All trainers employed by a training organisation  must be on a training register. 
  • Trainer Roster – Total trainer hours must as a minimum reflect availability during normal office hours.
  • Trainer availability Trainers must be available at least during normal business hours and must assist learners requesting training within 30 minutes of a request.
  • Human validation – An online statement of attainment must not be issued until the enrolment details confirmed and assessment and learning activities of the student have been reviewed by a qualified trainer. This sign off process must be demonstrated by data in the learning management system.  

 

4        Minimising the Potential for Fraudulent Activity

Policy writers must ensure that the following integrity measures are in place to minimise the potential for fraudulent activity:

  • Capture the student’s IP addressand monitor if bulk users are coming from the one IP.
  • A system for monitoring cheating must be in place that flags when different students give the exact same free text answers to free text assessment questions.
  • Student name fields must be locked so that these can only be changed by administration staff upon request and subsequent verification of the student’s identification.
  • lock-out’ access –  to an assessment after failing a set number of attempts of each question (3)  and must speak with a qualified trainer to be unlocked if they do not know the answer.
  • Must be able to intervene and make direct contact with the student if required.
  • Warn about fraudulent activity – Notify students that certification may be voided if fraudulent information is provided or fraudulent activity is detected.
  • Ability to report suspicious activity – The learning system must have the ability to identify and report immediately any suspicious activity by students undertaking the course to the appropriate authority.
  • Deliver training via SSL – SSL to reduce third party fraudulent activity ie HTTPS at start of a domain..
  • Agree to terms and conditions – Capture electronic evidence that the student has agreed to terms and conditions relating to not being assisted in any way during assessment and that they are the one that undertook and completed the training.
  • Statutory Declaration – If adequate integrity measures are not in place, or a student has breached an integrity measure, students must sign and upload a witnessed statutory declaration that they are undertaking the training without assistance.

 

5        Course Content

5.1  Volume of Learning

The Australian Quality Framework says that “the volume of learning”  includes guided learning, individual study, research, learning activities in the workplace and assessment activities. It could be argued that rather than applying this at a qualification level, the volume of learning should be applied at a unit of competency level and rather than minimum duration the relationship between the quality of the content being able to map to a rigorous assessment should be the deciding factor in assessing competency.  As the AQF goes on to say:

“The duration of the delivery of the qualification may vary from the volume of learning specified for the qualification. Providers may offer the qualification in more or less time than the specified volume of learning, provided that delivery arrangements give students sufficient opportunity to achieve the learning outcomes for the qualification type, level and discipline.”

 

This could  be applied to a competency rather than the whole qualification.

When looking at VET training, we need to remember that students come from hugely various backgrounds and experience, from Year 11 students undertaking a VETiS program to someone with a PhD who is looking for a career change to an older person with a wide-ranging life experience.  This all affects how quickly they can become competent, not only based on their level of experience and in the skills they already have or need to learn, but their ability to undertake and comprehend the training in the first place.

While setting minimum durations based on the time a learner (who is new to the industry area) would be required to undertake supervised learning and assessment activities (ASQA report A review of issues relating to unduly short training page 15 has merit), the dilemma remains that some people undertaking formal study will already be to some degree competent and not require the same minimum duration.

In terms of auditing the appropriate volume of learning on a student by student basis, training providers using data from the AVETMISS enrolment should be able to demonstrate whether a student is new to the field and requires what could be considered as minimal learning hours, as opposed to someone who is requalifying and could reasonably be expected to complete a competency in a much shorter time frame.  I.e. based on evaluating the needs of the student from the language, literacy and numeracy (LL&N) together with the AVETMISS enrolment data a training provider should be able to specify the expected or minimum volume of learning required.   In a class room this could be part of the learning plan.  In an online environment, this tailored learning plan must be established and managed in a dynamic way, i.e. by having certain locks or minimum interactive learning hours completed before assessment is made available.

To improve course content Integrity measures policy writers must mandate;

  • Minimum learning time – Access to onlnline assessment should not be permitted until a minimum amount of learning that is determined on a student by student basis has been achieved
  • LMS capability to manage time – The LMS must be able to manage the minimum online learning time limits and must also be able to record the amount of time spent in learning activities.

6        Issuing Statements of Attainment

Copying or editing statements of attainment can be done no matter how they have been issued.  However, If statements of attainment are issued online, policy writers must mandate that the following integrity measures must be in place to minimise fraudulent activity:

 

  • No Name change – The student must not be able change their name online and an administrator should only be permitted to do this after identifying the student and then sighting an updated form of photographic identification meeting the identification integrity requirements with the new name.
  • Unique ID – The certificate must contain a unique number that is linked to the unique student username.
  • Watermark – The certificate must have a background water mark with the unique details of the certificate, e.g. student name, date of birth, issue date, certificate number, signature of issuing authority, expiration date over the top of the watermark(s).
  • Locked PDF – The certificate must be in a PDF format and the PDF must have security enabled that locks the PDF and prevents edits.

7        Learning Management System Data Capture Requirements

Without high granularity of data capture related to the digital learning environment, it is more difficult to demonstrate that integrity has been maintained.  Some Learning Management Systems (LMS’s) may only record the course sections that have or have not been accessed and that assessments have or have not been successfully completed, i.e. without capturing the granularity of the data such as the actual responses given to pass the assessment or the number of assessments or total learning engagement time.

In order to achieve integrity, a LMS must be used that has the sophistication to not only have integrity features in place, but also be able to collect data for integrity and track student activity and progress through the learning material and assessments including;

  • Content engagement – Record the amount of time that a student has interacted with learning content,
  • Number of attempts – Record how many attempts students have made for each assessment
  • Result of all attempts – Capture the result of each attempt not just correct answers,

Policy writers must specify that a learning management system used to deliver training online does so at a level of granularity that allows auditors to view every single piece of data submitted by the student in relation to assessment weather that submission is correct or not.  Also that the number of attempts and evidence of lockout after eg 3 attempts can be show with data evidencing how the lockout was overturned.

Do you have what it takes to create engaging elearning?

Ant Pugh is great at breaking things down into 4 minute videos and has some great knowledge to share about online training.

 

 

Disruptive change and the growing need for e-learning innovation

Today’s corporate leaders recognise that the pace of technological change is creating a major skills problem for their organizations. In a recent study, Deloitte found that 90% of CEOs believe their company is facing disruptive change driven by digital technologies, and 70 percent say their organization does not have the skills to adapt (Deloitte 2017: 30). Indicating some of the factors driving this problem, Deloitte point to the changing nature of a career, as follows.

The changing nature of a career (Deloitte 2017:30)

Clearly, with such frequent job changes and rapid skill obsolescence, there’s a significant demand placed on continuous learning and skills development. Thus, in many ways, individual and organizational performance now critically depends on the capability of e-learning providers to meet these demands. But how can e-learning leaders and innovators respond to this problem?

It’s useful to first get a little historical perspective.

Learning & Technological Change

The skills challenges outlined by Deloitte are the modern outcome of a long-term trend.

In earlier times, widely-used technologies and skills would remain mostly unchanged for generations. People could learn everything they needed from their parents and grandparents. Broader knowledge was sparse and slow-changing. For example, in Europe, Burke (1985) explains that Martianus Capella’s (5th century) book on the seven liberal arts became “all the education there was for the next 700 years”.

In later times, as knowledge and technologies started to change, schools and universities provided intensive bursts of education – ideally enough to last a lifetime. However, the pace of change has increased, leading to the situation described by Deloitte. For an overall perspective, the long-term trend can be understood by looking at the relationship between human lifespans and technology lifespans, as shown below.

Human lifespan vs technology lifespan (adapted from Scholte 1988:31) 

In modern times, things just move too quickly for many of our traditional approaches to education. Intensive bursts of formal education can no longer last a lifetime. Job descriptions change and blur. Key knowledge and skills become obsolete as entire industries fade and others emerge.

In these circumstances, there is a growing number of reasons why traditional education can’t keep up. And even faster-paced e-learning needs solid innovation and transformation.

The following sections explore ways that e-learning might respond to these challenges.

Rapid change & e-learning content development

It takes time to develop quality learning content – to make sense of a domain of activity, to distil the relevant insights, and to design content that helps establish these in the mind and practices of a learner. It can be time-consuming and costly. Given the rate of technological and skills change above, the challenges of content development are likely to increase. Deloitte (2017) points out one pathway forward:

“The good news is that an explosion of high-quality, free or low-cost content offers organizations and employees ready access to continuous learning. Thanks to tools such as YouTube and innovators such as Khan Academy, Udacity, Udemy, Coursera, NovoEd, edX, and others, a new skill is often only a mouse click away.”

This suggests a pathway involving something more along the line of e-learning content curation, drawing upon the best of what already exists and is freely available. Another related strategy might be to parallel the concept of “mashups”, creatively mixing this free content to support a specific learning need and context.

Such approaches will prove vital – but they will not be enough. This is because, historically, learning content authors have been able to draw insights from already-established and widely-accepted bodies of (mostly published) knowledge that may have been developed over generations. However this approach is under threat because it takes time for these bodies of knowledge to form; indeed, in times of rapid technological change, they may not get to even approach this point before another technological disruption occurs.  This suggests a future where learning shifts to the workplace front-line, actively supporting knowledge development and application in everyday activities.

Micro-learning and micro-credentials

Another response to the pace of change is to focus on a smaller and ongoing stream of workplace and on-demand learning activity – in essence, learning all the time, and using micro-learning and micro-credentials. Eades (2014) explains micro-learning as “delivering content in small, very specific bursts”; and on this same theme, micro-credentials involve providing formal recognition for skills and knowledge gained from on-the-job experience which can be “stacked” to form recognised qualifications (O’Keefe 2016). Practically, some of the main challenges of micro-learning will center upon:

  1. Identifying the workplace time and context for e-learning – for example, should it be when a person has struck a problem they need learning to address, or perhaps artificial intelligence could be used to anticipate what additional learning a person might need, such as before a scheduled client meeting. (For more on this, consider Friedman’s discussion of “Intelligent Assistants”, in Takahashi 2016).
  2. Ensuring that learning content is concise and relevant enough to leverage the learner’s current situation – for example, a long-winded explanation of marketing theory is likely to distract from the task-at-hand, and in contrast, perhaps, an analytical tool that helps a person address their immediate task or challenge. A key here seems to be leveraging the learner’s current situation, and their mental focus upon it, as the framing context for a learning activity.
  3. Ensuring cognitive continuity between workplace task and learning – for example, having to go to a specific training room is likely to be problematic, and in contrast, consider the value of mobile-based learning content that can be accessed immediately and within the current workplace context.

Tracking micro-learning

A challenge for the micro-learning approach is keeping track of an individual’s learning activities. One approach was explored in the Institute for the Future’s “Learning is Earning 2026” project. This project considered a very broad approach to learning and learning providers, with some interesting possibilities (see the project video, below) – and a central part of this they called “the Ledger”. They explain:

“In this future, the currency of learning is tracked and traded on a digital platform called the Ledger. It’s a complete record of everything you’ve ever learned, everyone you’ve learned from, and everyone who’s learned from you. The Ledger not only tracks what you know – it also tracks all of the projects, jobs, gigs, and challenges you’ve used that knowledge to complete.” (http://www.learningisearning2026.org/)

The foundation envisaged for the ledger concept is blockchain – the technology that underpins crypto-currencies like Bitcoin. There are various explanations of its potential for learning, but the essence is that it provides a way to keep an accurate and trusted records of historical activity – in the learning context, records of the activities a learner has completed.

The reality is that this technology faces some hurdles to being applied in a learning context. For example, the Bitcoin blockchain handles relatively simple and well-defined information: it deals with currency transactions. In contrast, a learning-focussed application of blockchain may need to record information about a learner, the person or authority certifying the learning, the deliverer of the learning, the content and version of the learning, and potentially submitted work and assessment information. Meanwhile, Flipo and Berne (2017) point out that the computer facilities that underpin the Bitcoin blockchain already consume electricity on a scale equivalent to the country of Ireland, and that even widespread adoption of Bitcoin would have grave energy consequences globally. A blockchain of complex learning data would, in all likelihood, quickly surpass Bitcoin’s energy impact.

Beyond these issues, the need and potential for strategies that address accelerated skills and learning needs remains. There will be ways forward. And while some details are yet to be established, the scenarios contemplated in the Institute for the Future’s project provide some interesting glimpses into ways that micro-learning, rapid skill development, and ubiquitous workplace and lifelong learning might play a role.

[Youtube video: https://www.youtube.com/watch?v=DcP78cLPGtE ]

Paul Wilson elearning Consultant

References

Burke J (1985), The Day the Universe Changed, BBC, London.

Deloitte University Press (2017), “Rewriting the rules for the digital age”, https://www2.deloitte.com/content/dam/Deloitte/global/Documents/HumanCapital/hc-2017-global-human-capital-trends-gx.pdf

Eades J (2014), “Why Microlearning is huge and how to be a part of it”, eLearningIndustry.com, accessed at: https://elearningindustry.com/why-microlearning-is-huge

O’Keefe D (September 21, 2016), “Old ways not enough to gain workplace credentials”, The Australian, accessed at: http://www.theaustralian.com.au/higher-education/old-ways-not-enough-to-gain-workplace-credentials/news-story/286a526a99fc866f96f99cdac2c6981a

Flipo F & Berne M (2017), “The bitcoin and blockchain: energy hogs”, The Conversation, accessed at: https://theconversation.com/the-bitcoin-and-blockchain-energy-hogs-77761

Learning is Earning 2026: http://www.learningisearning2026.org/

Scholtes PR (1988), The leader’s handbook: a guide to inspiring people and managing the daily workflow, McGraw-Hill, New York.

Takahashi D (December 7, 2016), “New York Times columnist Thomas Friedman tells us how to live in accelerated times”, Venture Beat, accessed at: https://venturebeat.com/2016/12/07/new-york-times-columnist-thomas-friedman-tells-us-how-to-live-in-accelerated-times/