Small business

Summative evaluation
Summative evaluation measures are taken at various levels of impact:
Reaction impact measures the reaction of immediate beneficiaries to the input, be it training, counselling
or a more direct service. This usually evaluates whether customers:
EFFECTIVE POLICIES FOR SMALL BUSINESS - © OECD 2004
14. Management, Monitoring, Evaluation and Control
80

---

• Like the service;
• Think it is relevant to their jobs/institutions;
• Think they have gained from it; 
• Like the service providers;
• Like the location and amenities;
• Benefit a lot or little;
• Find it easy to participate and contribute; 
• Would like more, and what:
In addition, the highs and the lows overall of the programme can be explored.
All the above except the last can be measured on a scale 1 to 5 representing high to low.
Appreciation/learning impact: This measures what has been gained from the input in more detail. In the
case of a training/counselling programme this would be a test of what has been learnt. In the case of a
service such as micro credit it would be how well the beneficiaries understand the process and how they
can gain from it. In the broader policy context the impact measured is how well the policy has been
internalised by the beneficiaries, and indeed the public as a whole.
Behaviour impact: This measures how and why people do things differently as a result of the programme. This
can be assessed by questions about the way individuals approach key parts of their work and what has changed
as a result of the input – especially, of course, what has improved. It can also be checked by observation or through
the perceptions of other stakeholders as to whether those benefiting from the programme have changed their
behaviour. In the case of policies and programmes specifically targeting personal behaviour (for example
through counselling or training) this may be tested by observation of events or video recording.
Intermediate organisational impact: This measures what has been changed in organisation practices. For
example, in the context of microfinance, are there new credit rating systems, new forms of marketing and
new ways of dealing with customers in arrears? In the case of start up training: have participants started a
business, how far down has the business proceeded?
Ultimate impact: This seeks to measure what impact on the ultimate performance of beneficiary has been.
In the case of micro credit this requires measuring improvements in lending (sales), profits, liquidity and
employment. The impact on the ultimate customers should also be measured.
At the higher levels of summative evaluation it is difficult to attribute results to any specific
programme/policy input, for two reasons:
• Programme impacts need time: it may be several years before a specific scheme to improve micro
credit has a full impact on its final customers;
• Numerous other factors in the economic and social environment have an impact on specific policies
and programmes: what is the exact effect of a microfinance training course attended by a few
individuals on the growth and profitability of a micro finance institution?
Cost-benefit measures of inputs are usually so contrived (or alternatively so simplistic) in these cases
as to be useless. The value of summative work lies in actively using the ‘hierarchy of impacts’ approach in
designing a programme or policy. In institutional development, it helps in setting immediate and ultimate
impact targets, targeting organisational changes, and identifying groups that need training as well as
behaviours that need to change.