A significant part of the time and attention at this year’s two meetings of the extended working group for coordinating the implementation of the SBA in BiH (we wrote about what the SBA is and how the assessment process and progress reporting takes place in a previous blog, which you can read here) was focused on the needs and opportunities to improve monitoring, evaluation and reporting. The occasion was the publication of the 6th report “SME Policy Index: Western Balkans and Turkey 2022: Assessment of the implementation of the Small Business Act for Europe, SME Policy Index”, which the OECD with partners (the European Commission, the European Bank for Reconstruction and Development and the European Training Foundation) in early July 2022. There were several reasons, here are the most important: still inadequate (weak) monitoring of the implementation of the SBA by (sub)dimensions by which the OECD makes assessment every three years, the lack of evaluation of targeted policies to support the development of SMEs at all levels in the country, and insufficiently spending of energy and time to collect data and information necessary for reporting.
In the next few paragraphs, we would like to briefly explain the needs and outline the possibilities for improvements, which were discussed at these meetings, held in July in Teslić and November in Jahorina. Let’s go in order.
By definition, monitoring in a strategic context implies the systematic and continuous collection, processing, analysis and use of data and indicators for the purpose of measuring progress in achieving set goals/priorities/measures, as well as progress in the use of available resources and undertaking appropriate activities with the aim of possible corrections. Monitoring, therefore, has a dual function: (1) checking whether work is being done in accordance with the plan and achieving the desired progress, and (2) signaling the need for corrections in case progress is lacking. Monitoring is based on the appropriate parts of the planning document (e.g. description of the activities of the measure or project and defined indicators with starting and target values) and on the data and information collected in connection with it (what was done, when, how, with what resources, and did it affect the state of the indicator). Without systematic and continuous monitoring, it is not possible to get a score higher than 3 in any (sub)dimension when assessing the implementation of SBA, nor is it possible to systematically work on improving and updating the measures and projects that are implemented. Monitoring, understood and organized in this way, has not yet been developed to the level of good practice in our institutions. Better said, we are in an early stage, characterized by an ad hoc approach, reacting usually only when it is time to report, statistical sources that generally do not follow what we really need, and similar.
Evaluation is linked to monitoring and is often referred to as a pair: M&E. By definition, evaluation is a procedure that asseses the success, effectiveness, progress and impact achieved by the implementation of a strategic document, i.e. policy, based on collected and analyzed data and knowledge obtained through the procedure of regular monitoring and additional data collection during the evaluation preparation process. Evaluation is usually done according to the aspects of relevance, efficiency, effectiveness, impact and sustainability. It can be carried out before the creation of a new strategic document (ex-ante), in the form of an evaluation of these aspects of the previous strategy, or after half of the implementation period (mid-term). In the first case, it creates better conditions for the development of a new strategy/policy, and in the second case, for corrections and revision of that strategy until its validity expires. Seen from the perspective of assigning grades when assessing the implementation of the SBA, the highest grade 5 can be obtained if the policy within the observed (sub)dimension is effectively implemented and if the findings/results of monitoring and evaluation are used for (re)designing and improving the implementation of the policy, i.e. the strategy. The chain rule applies here (the chain is as strong as the weakest link), and the experience of our institutions with evaluation is only in the initial phase, so both links are weak, monitoring as well as evaluation, with insufficient potential impact on improving planning/(re)design and policy implementation.
Monitoring and evaluation provide the necessary information for reporting. The reporting is annual and three-yearly, and includes many elements concerning the realization of planned activities, achievement of defined indicators, expenditure of financial resources and other resources, reasons for observed deviations from the plan, recommendations for improvement and similar. In our institutions, until now, reporting has had crucial importance and role, so with increased engagement in this work, when the time comes, they tried to neutralize the previously mentioned shortcomings in connection with monitoring and evaluation. But the chain rule applies here too, so good reporting cannot be expected if it is not based on good monitoring (and evaluation).
At this year’s very meaningful and constructive meetings of the extended SBA implementation coordination group (and this group is increasingly working as a team, but more on that another time) there was much more discussion about how to improve monitoring and reporting. Evaluation somehow remained on the back burner, and there was not much to say about the relationship and impact of interconnected monitoring and evaluation on planning and implementation. With this blog, we try to correct that imbalance.
Here are key ideas on what can be improved and how:
- To the extent possible, efforts should be made to integrate and simultaneously monitor the implementation of policies/measures from the SBA framework and SME development strategies. The assumptions for this approach have already been created to a considerable extent, as (sub)dimensions from the SBA framework have already been included or are now being included in the appropriate priorities and measures of SME development strategies, in accordance with the competences of the appropriate level of government. In this way, effectiveness (doing the right things) and efficiency (doing things the right way) are directly contributed to, because policies are planned and implemented that affect the implementation of SBA, in addition to the impact on SMEs, and duplication of similar work is avoided.
- Efficiency in the collection and processing of data and information in the monitoring process can be significantly improved by creating a digital platform for monitoring and reporting, which would be based on elements of SBA implementation assessment (mainly questions from the questionnaire used) and elements for monitoring the implementation of SME development strategies, and which would be adapted to the needs and constraints of each level of government, i.e. user. The assumptions for this improvement are being created, mainly with the support of the EU4DigitalSME project (GIZ) and the appropriate help of our team, primarily on the integration of elements from the SBA framework and strategies. The idea is to enter data and information in real-time as activities unfold and monitor the status of indicators, while specifying the source of verification. The initial versions of the terms of reference and the corresponding solutions have already been done, now they are being developed and adjusted further. After that, an important part of the work remains on their introduction, promotion, training of users and creating the habit of entering data and information at the time of their creation, instead of “campaign” report making only when the deadline comes.
- Building the capacity of institutions for evaluation can practically start at the beginning of next year, using the “learning by doing” approach, based on the examples of action plans for SME innovation, which were adopted in both entities and Brčko District of BiH for the period 2021-2023. Both external and internal evaluations can be carried out in parallel, and the findings and recommendations can be used for the preparation of new action plans for the period 2024-2027. The experience gained will be valuable when the time comes to carry out a mid-term evaluation of SME development strategies. In the Republic of Srpska, an evaluation of the Women Entrepreneurship Development Strategy for the period 2019-2023 can be carried out similarly, and its findings and recommendations can be used to prepare a new strategy.
- According to the new approach, the reporting process should be overlapped to a considerable extent with monitoring, so that the institutions that are responsible for the implementation of measures from SME development strategies enter information about the execution of activities and the realization of effects in real-time, via the previously mentioned digital platform, and that the ministries/departments that coordinate the implementation of strategies and reporting on the implementation of the SBA take over and consolidate this information and process it into appropriate annual and medium-term reports, according to predetermined dynamics. Along with the development and introduction of a digital platform for monitoring, an important prerequisite for the successful application of this approach is an advanced building of the capacities of all institutions that participate as the bearers of the implementation of measures from the strategies, as well as in the process of reporting on the implementation of SBA. In addition, special attention should be paid to the gradual formation of new habits among representatives of those institutions for up-to-date and responsible input of information into the system at the moment of its creation, and not later, when much of the content and value of that information is lost among other events and priorities.
All these are no longer just ideas, but things that are already being worked on or activities are being planned for the next year. Of particular value is the fact that these are not ideas recommended by experts “from the outside”, but challenges, problems and potential solutions that were reached by the representatives of the institutions participating in the meetings/workshops of the extended group for coordinating the implementation of SBA in Bosnia and Herzegovina.