Step 6: Manage implementation of the workplan, including production of reports

Using good project management techniques

You use the same processes and skills to manage a program evaluation as you would when managing other types of projects. Good project management is about communicating well and communicating early, within your own team and with any external teams.

In all evaluation projects, the internal group is responsible for:

• managing the implementation
• keeping the evaluation on track
• ensuring emerging issues are dealt with in a timely way.

These tasks must be done without jeopardising the rigour or integrity of the project.

Tasks for the internal team include:

• reviewing drafts of plans and deliverables
• ensuring governance groups are engaged with the evaluation.

When an external provider is involved, the internal team manager's role is focused on keeping lines of communication open and ensuring milestones are met.

You may need to adapt the communication schedule developed in the project plan in response to emerging issues.

Collecting the data

Ensuring that data are collected using the agreed methods and within the specified timeframe is fundamental to executing a rigorous evaluation.

As the manager of an internal or an external team, you will need to be aware of how data collection is progressing. It is your responsibility to know about and respond to emerging issues, such as:

• delays to fieldwork
• low survey response rates
• incomplete administrative datasets.

If the scope or integrity of data collection is compromised, so too is the integrity of the evaluation.

Communicate often and early with your team, ensure they brief you on problems, possible solutions, and the implications for the overall evaluation timeframe, and importantly, rigour.

In all evaluations, but particularly with an external team, be aware the fieldwork team is likely to have substantial experience in data collection and can advise you on how to proceed. You will need to be prepared to revisit the scope of data collection, if required.

It is also the program manager's responsibility to know about data quality issues. To be effective, data quality checking should occur as data is being collected, for example:

• check administrative datasets for completeness when they are provided to you
• ask questions about spurious variables
• spot-check survey responses as they come in
• do random checks of any online systems to ensure that they are functioning properly.

Finding out about data quality issues early increases your ability to respond and overcome the problems with reduced impact on the evaluation.

Analysing the data

Analysing data to summarise it and looking for patterns is an important part of every evaluation.

Data cleaning and analysis will likely be done by members of your internal or external team, but you must understand how the data is being managed. Well-performed analyses use the right methods for the purpose, including technical appropriateness, as well as the most meaningful way to present the analysis depending on the audience and overall purpose of the project.

It is your responsibility to prompt internal analysts to conduct rigorous, meaningful analyses that will allow the evaluation questions to be answered. Similarly, if you are managing an external team, it is your responsibility to ensure the analysis is leading towards the final evaluation products.

Data visualisation is the process of representing data graphically to identify trends and patterns that would otherwise be unclear or difficult to discern. Data visualisation can bring clarity during analysis and be used to communicate. 

The final stage of analysis is synthesis, where findings from all data sources are brought together and integrated to address the evaluation questions. Combining qualitative and quantitative data can improve a program evaluation by ensuring that the limitations of one type of data are balanced by the strengths of another.

Combined data can be useful to arrange a preliminary findings presentation, where the internal or external analytic team discuss the key trends and findings. This process is particularly helpful if the team is external as it:

• helps you know what to expect from the final report
• highlights errors or inconsistencies
• can help the analytic team interpret their findings from a broader policy perspective.

Reporting

The final deliverable of a program evaluation project is typically a written report, although sometimes it is a PowerPoint presentation, positioning paper or communiqué.

The final evaluation report, either in full or summary form, needs to be meaningful to the intended audiences. The report should be readable, straight to the point, and use a writing style that promotes understanding.

A good evaluation report contains these basic components:

• an executive summary containing a condensed version of the most important aspects of the evaluation
• a description of the program and its context relevant to the evaluation
• a summary of the evaluation's scope and focus, including the purpose and key evaluation questions
• an outline of the evaluation design and methods
• the findings of the evaluation, with responses to the key evaluation questions, supported by relevant quantitative and qualitative evidence
• conclusions and recommendations
• appendices with additional information needed such as:
  • terminology
  • details of who was involved in the evaluation
  • detailed methodology
  • extended data summaries.

The final deliverable should explicitly respond to the key evaluation questions and present sufficient evidence to justify its position or recommendations. The final deliverable should be a synthesis of the data collected and present a balanced summary of all the evidence collected, noting where divergent viewpoints or edge cases exist and what these mean.

Ensuring the final deliverable is written clearly and concisely is important for how the report is used.

If you are writing the report, use plain English and data visualisation to highlight key trends and findings. Whether the final deliverable includes a summary of key findings or a list of recommendations will depend on the project context. If you have contracted the work externally, it is important to make clear your expectations about the format, style and content of deliverables.

You will need to review the deliverable for accuracy, and to ensure that the terms of reference or evaluation questions have been answered.

If the work is done:

• internally – you will work with your team to prepare the final deliverable 
• externally – you will review the deliverable, often with input from a governance group.

Many organisations and evaluation systems require the evaluation reports to be formally reviewed as part of quality control before they are publicly released.