Increasing apprenticeship course attendance through behavioural interventions

We found that if the employers of apprentices and trainees (henceforth "learners") were sent a text message detailing what the learners are studying at TAFE, we could significantly increase course attendance.

The issue

In 2015, the Premier Mike Baird set a state target of 65% completion rates for apprentices and trainees. Since November 2015, BIU has worked closely with Training Services NSW (TS NSW) and TAFE NSW to design and deliver a range of interventions to increase the proportion of people completing apprenticeships and traineeships. Our collaboration has resulted in three trials.

Today, we have published our report on our second trial, Strengthening Connections:  Increasing Apprenticeship Course Attendance Through Behavioural Interventions. 

Barriers to completion

Our previous fieldwork revealed two key barriers to apprenticeship and traineeship completion: a disconnect between formal study and on-the-job learning, and insufficient employer support.

• At work, learners often do not communicate what they are learning to their employers. Consequently, employers do not value the training their learners received.
• Employers are also often unaware of what their learners study at TAFE so they cannot give them a chance to practice their skills at work. Due to this disconnect, both employers and learners have negative perceptions about the value of the training they receive at TAFE.

SMS can improve learning

Prior behavioural research on school attendance shows that improving communication between teachers and parents using SMS has increased benefits for students, from preschool to high school.

Our trial sought to apply these empirical insights into a new setting for a different cohort: vocational learners and their employers.

We ran a trial

We ran a randomised control trial (RCT). One set of learners in the Treatment group received our behavioural intervention. Their employers received text messages for one semester. Each week, TAFE NSW teachers and head teachers wrote to our project team with a brief summary of what they taught in class. We then used this content to craft a behaviourally informed message, either prompting employers to give the apprentice a chance to practice that skill on the job, or to talk to their apprentice about that topic.  

The other set of students were in the Control group. They benefited from the same level of service all students already receive, but without these text messages.

Our text messages had a positive effect on class attendance

Our primary outcome measure was to increase class attendance. Our text messages to employers were effective in increasing the students’ class attendance by 3.1 percentage points. This effect was mainly driven by attendance in the last two weeks of Term. This finding translates to 147 out of 966 (15.2%) extra classes that students attend as a result of our treatment. Multiplied across the state, the impact of our trial could improve attendance for thousands of learners.

The effect may be understated because of the possibility of spillover. This refers to the possibility that learners in the Control group who worked with employers who also had students in the Treatment group (the Spillover group) may have also benefited from the treatment, as employers may have changed the way they treated all of their employees as a result of our treatment.   

Treatment did not encourage more interaction

We had two secondary outcome measures. First, we conducted a survey to to better understand the experience of receiving additional communication. Learners in the Treatment group did not report more frequent interactions with their employers, nor did they receive more opportunities to practice what they learnt on the job, in comparison to the Control group. Similarly, employers with learners in the Treatment group did not report higher understanding of what their learners were studying. It could be that receiving text messages increased class attendance simply due to greater perceived social pressure from employers to learners, rather than the specific content of the messages.

Nevertheless, our survey suggests all learners and employers were likely to recommend studying at TAFE (with the exception of carpentry).

Some employers were confused by relevance of the course information in the text messages and did not see the value. For example, during our trial, carpentry courses were focused on mathematics. A typical message was: ‘This week we learned about Pythagoras’ theorem.’ We suggest demonstrating the relevance of courses and include actionable messages that link theory to practice. For example, ‘This week we learned about Pythagoras’ theorem to show how to accurately lay out footing for a new building. Here’s a quick tip on how you can get Sarah to try this at work <include hyperlink to five-minute exercise easily achieved at work>.’

No impact on cancellation but 1/5 of learners who leave keep studying

Our other secondary outcome measure was to see the impact of messages on completions 12 months after treatment ended. Both the Treatment and Control groups showed similar levels of contract cancellation rates. The treatment ran for only three months in the first semester. This suggests the benefits of messages to class attendance wear off without ongoing reinforcement.

Still, we found that one-fifth of learners who cancelled their first contract continued studying under a subsequent contract with another employer. This implies that cancelling a contract does not mean that the learner drops out of their apprenticeship or traineeship.  

Policy implications

Based on the findings from our trial, we proposed the following policy recommendations.

Continue timely text messages with actions employers can take

• Create stronger links to translate curriculum (including theory) to on-the-job-learning.
• Enhance communication with employers using SMS at optimum times, detailing what learners are learning and providing succinct tips, ‘cheat-sheets,’ checklists and other actionable messages to boost class attendance.

Use behavioural insights (BI) to disseminate best practice to employers:

• Make behaviour change easy. Reduce the hassle for employers to learn better communication and supervision skills, with mobile-accessible resources and support. 
• Design attractive rewards and sanctions. Boost communication between learners and employers using incentives and disincentivise poor practices.
• Reinforce social norms of best practice. Disseminate best practice from employers with high completion rates. Make completions data available from top employers, using social comparison methods from behavioural science. This will stimulate peer influence and promote changed behaviour among low-performing employers.
• Maintain timeliness of messages. Effects of messages are short-lived and require ongoing reinforcement during difficult periods where learners may be considering quitting.

Apply lessons from the trial to better use data on completions.

As TAFE NSW has moved to the ‘One TAFE’ model and looks to standardise course content in 2019, the methods of our intervention show that getting access to data early and achieving consistency in reporting attendance and cancellations would benefit these aims.

• Better use of data allows for superior monitoring of negative patterns, targeting and timing of interventions.
• Proactively support learners using machine learning to identify problem patterns in class attendance and target interventions.
• Support employers’ supervision capability using electronic prompts. SMS can be used to update records and inform employers on learners’ ongoing outcomes and class attendance.

TAFE NSW will be incorporating this body of recommendations into broader activities that the organisation is undertaking to enhance customer experience. 

What’s next?

We are currently supporting TAFE NSW and TS NSW to continue testing SMS messages to boost attendance and on-the-job learning.  

Our trial would be of interest to other jurisdictions currently working on improving apprenticeship and traineeship attendance and completion rates.

Read our report.

Published: 31 May 2019