POV09Report

=Project Output POV09= **Model for use of POV technology for remote assessment** toc

**RTO**
National Training P/L

partnering with

Business IT Training Specialists (BITTS) P/L as facilitators

**Contact**
Katrina Sommers - katrina [at] bitts4learning.com.au Mobile 0402 009 884

**MODEL for use of POV for Remote Assessment**
This model will provide you with a range of resources, recommendations and examples for conducting successful authentic remote assessment across any training package.

Overview of the process of conducting Assessment by POV Recording
The following flow chart tracks the critical processes involved in setting up a situation where remote assessment of workplace trainees can be successfully authenticated. Authentication is not an issue surrounding receiving a single item of video assessment material. Authentication is a process that is enabled through establishing a relationship with your trainee, and using a range of assessment tools that ensure valid, reliable current competency to be assessed.

Powerpoint presentation on project delivered at AUPOV Conference June 2009
media type="custom" key="4885323" You can download a pdf version of the presentation including notes with each slide for further information and reference.

Powerpoint presentation on project delivered at Exploring e-Innovations online conference November 2009
media type="custom" key="4886513" You can download a pdf version of the presentation including notes with each slide for further information. All video material referenced in this presentation is available for viewing in the video section below.

Digital Medial Release Forms
It is critical to obtain permission from both the workplace and anyone who may be captured on video when the technology is in use in the workplace.

Trial Assessment tools
Assessment tools were developed for use during the trial. Below are 2 examples of observation assessment tools used.

Feedback Forms
Every assessment conducted was followed by collection of feedback from both the Trainee and the Assessor. The analysis of this feedback is included in the Project Report at the end of this model.

Sample video -
A comprehensive listing of sample video is provided below to assist you in setting up your own POV Assessment routine. During our trial we came across many issues both with correct use of the technology and with assisting people to use the technology to its greatest advantage. The sample video provided here has been collected to give you a guide to solving or reducing issues that can arise. Due to the large file size of many videos the following samples are relevant snippets from video footage in most cases. media type="file" key="01 Hospitality Out of Focus.mov" width="300" height="300" media type="file" key="02 Bike lighting.mov" width="300" height="300" media type="file" key="03 lighting issues.mov" width="300" height="300"
 * This video was recorded very early in the trial and is an excellent example of NOT using the technology correctly. As an assessment video it would be useless because the trainee involved has not been able to focus the camera on what they are actually doing. The narration is excellent, however a remote assessor would not be able to assess the task.
 * This video was recorded to test potential issues of background noise, movement on a vehicle and moving from full sun to shadow. The purpose of testing these conditions is for the potential of use of the technology for assessments involving environments such as construction sites, using machinery such as forklifts or road machinery. We were very pleased with how the technology handled the conditions of the motorbike motor running, the movement even though it was faster than a forklift would be expected to be moving and found that the camera actually copes very well with natural lighting both full sun and shade.
 * This short clip demonstrates clearly the issues you can have from lighting particularly fluorescent lights. When recording inside always perform a 10 second test video until you have all lighting issues solved to avoid flickering and flashes of light.
 * We found it was critical to familiarise both Assessors and Trainees with the technology to prevent issues such as the first video example where the camera never captured anything the trainees hands were actually doing. We developed a simple task for trainees and assessors to perform to assist them become comfortable with how to move their head instead of their eyes when recording.

media type="file" key="04 demo tea.mov" width="300" height="300"
 * When a trainee had used the technology well for assessment and then came back to use it again after a few weeks (trials were done in rounds and some trainees used the technology 2 or 3 times during the project), it was always worthwhile to have them begin by doing a very shortened version of the task just to be sure they remembered how to focus on what their hands are doing as shown in the short clip below.

media type="file" key="05 Testing focus.mov" width="300" height="300"
 * As the trials progressed through the first rounds we discovered that many assessors were still uncomfortable with the technology and it was not enough to send them a pair of glasses and instructions. We ran a workshop to induct the assessors into the entire process which proved to be highly successful in overcoming many minor issues that could have prevented some assessors from taking the technology to their trainees. The assessors were given a mock assessment task and undertook the assessment, recording the task until they were satisfied with their footage. They then swapped hats and became assessors again and used observation checklists to assess their peers video. At the end of the workshop assessors felt they were now well enough equipped to take the technology to the workplace and instruct their trainees adequately to produce viable assessment material.

media type="file" key="06 Les paperfold 1.3gp" width="300" height="300" media type="file" key="07 Les paperfold 6.3gp" width="300" height="300" media type="file" key="08 Nigel paperfold 2.mov" width="300" height="300" media type="file" key="09 Dave paperfold 2.mov" width="300" height="300" media type="file" key="10 Trainee Id.mov" width="300" height="300" media type="file" key="11 OHS Assessment.mov" width="300" height="300" media type="file" key="12 OHS Inteview.mov" width="300" height="300" media type="file" key="13 TLI Assess 1.mov" width="300" height="300" media type="file" key="14 TLI Assess2.mov" width="300" height="300" media type="file" key="15 ICA Assess Fake.mov" width="300" height="300" media type="file" key="16 Correct Identification.mov" width="300" height="300"
 * Correct Identification at the beginning of an assessment task is considered to be similar to a trainee placing their name, date, unit code and name onto any submitted assessment task. However it is also important for authentication of who is performing the task. Trainees usually found it easier if they wrote a short intro script to use.
 * This video is an example section of a trainee conducting an assessment for ICAU3004B Apply Occupational Health and Safety Procedures from the ICA package. An important principle when paperwork is involved in an assessment task such as this, where the trainee has developed an OHS checklist for their workplace and is now demonstrating how to use the checklist to conduct a workplace audit is for the trainee to submit the completed checklist with their assessment video. This allows the assessor to see them conducting the audit, listening to their observations, and to be looking at the actual completed paperwork. This contributes to the authentication of the assessment evidence submitted. A third party observer is also able to wear the POV technology and conduct interviews with the trainee to capture them explaining processes in the workplace or any other supportive assessment evidence they may wish to submit. The second video here is a third party interview conducted after the trainee completed the workplace audits.
 * This video is an example section of a trainee conducting an assessment for TLIA3907B Receive and Store Stock from the TLI package. We were not able to obtain permission in this case to provide the actual workplace invoices etc from this task for public viewing but the same principal was involved in this assessment material being submitted along with the video of the task.
 * As we progressed through the stages of the trial we were very happy with the material being submitted. We were feeling that authentication was a fairly simple matter using the procedures we were using and the principles outlined in the project report. However we were still wondering what would happen if a trainee were to attempt to cheat by identifying themselves and then passing the glasses to another person to actually perform the assessment task whilst they narrated alongside them. We felt this was an unlikely scenario but not an impossible one and so set up a video to test out some of our assumptions. We felt that if we recorded an assessment task with a female identification and voice, but a man's arms and hands performing the task it would be obvious. We were stunned when we sent the following video to 5 different assessors and not a single one even suspected there was anything wrong! Of course they were not alerted that there was any issue to look for, and as assessors we are not normally required to look for this kind of deception. Watching this video knowing that this scenario was set up is likely to make the deception obvious to you.
 * As a result of this scenario we developed a more comprehensive identification method for trainees to use. A trainee can record this identification video before their first assessment task and submit it as a separate video file. The assessor then holds this on file and can refer to it at any time when assessing POV submitted assessment evidence to ensure authenticity.

Learning object samples -
The project also looked at the use of POV technology to record learning objects. The project report provides more detailed information on our work in this area. The following 4 samples are from the ICA package again for the unit ICAI3021A Connect Internal Hardware Components. Whilst we were able to use video recorded by some trainees for others in the same workplace to learn from, many trainees were uncomfortable with their footage being released on a public forum. The following footage demonstrates 4 critical stages involved in the installation of RAM chips in a 15 inch MacBook Pro computer. The task was deliberately broken into 4 stages as we were testing having short demonstration videos that would allow a learner to grab 'just in time' learning at need. For example if they needed to install RAM and were only unsure on how to open the RAM Access Door, they would only need to view the one short video. 1. Remove the battery 2. Open the RAM Access Door media type="file" key="01 ICA30105 Remove Battery.mov" width="300" height="300" media type="file" key="02 ICA30105 RAM Access Door.mov" width="300" height="300" 3. Remove existing RAM chips 4. Insert RAM chips media type="file" key="03 ICA30105 Remove RAM.mov" width="300" height="300" media type="file" key="04 ICA30105 Insert RAM.mov" width="300" height="300"

Project report
The project report completes this model by giving detailed information on establishing relationships with trainees, range of assessment tools, authentication, an analysis of feedback from our trials and creating learning objects.