COMDIS-HSD is a Research Programme Consortium funded by UK aid until 31st December 2018. We work with partner NGOs in low- and middle-income countries to provide evidence to policy makers to improve the way they deliver health services for their populations. Our focus is communicable diseases such as TB, MDR-TB, malaria, hepatitis, neglected tropical diseases and anti-microbial resistance, together with non-communicable diseases and urban health, with service delivery through primary and community health care.
This impact evaluation covers the period 2006-2018 incorporating evidence from two consortia, COMDIS[1] (COMDIS HSD’s predecessor) and COMDIS-HSD[2]. We interviewed former and current members of staff from our consortium partner organisations. We also interviewed policy makers and practitioners[3] who had interacted with our programme, through an on-line survey and in-depth interviews.
Key findings from the evaluation:
Building the capacity of our consortium partners

• Capacity building is an essential and integral component of being able to deliver a programme effectively.
• We have provided high quality capacity-building support. Coverage has been substantial over all areas of the consortium, including research skills; programme and project management; research uptake (RU) and communications; and operational areas such as finance, monitoring and evaluation (M&E), and governance.
• Our largest capacity building component has been through ‘learning by doing’ where partners worked together with in-house subject experts to gain in-depth knowledge and skills to enhance their abilities to successfully deliver each area of the programme. This method of capacity building is a key strength and has proved invaluable for our partners, especially when complemented by more structured training (e.g. ‘writing for publication’ workshops).
• Through our capacity building efforts we have contributed to building a highly skilled team across all partners. The programme has:
  • Contributed to enhancing skillsets and knowledge at individual and organisational level in all areas of our programme.
  • Contributed to strengthening career progression for some individuals.
  • Supported some of our partner organisations to build confidence, grow and become attractive to other funders, which has subsequently contributed to their longer term sustainability. Partners are able to secure funding from wider sources because of their credibility and very strong reputations.
  • Helped partners strengthen relationships with senior officials within their Ministries. Partners’ collaboration and advocacy approaches with their Ministries are exceptionally strong.
  • Contributed to identifying in-country priorities and addressing needs in a timely way.
  • Encouraged partners to work with in-country programmes to understand their priorities and the interventions needed, and embedding our high quality rigorous research within the programmes to meet these needs.
  • Provided opportunities to work with consortium partners from other countries to help understand different contexts, including political and cultural influences on health and service delivery.
• Longevity of funding is vital. The programme length and flexibility has enabled our consortium to assess scale-up and impact of our research interventions.

Influencing policy makers’ and practitioners’ ways of working

• We have engaged with policy makers and practitioners through our research activities, including stakeholder meetings and technical working groups to develop packages of care and oversee our research; support capacity building training for healthcare staff; support implementation activities; and hold dissemination and knowledge sharing events. All respondents to the on-line stakeholder survey and interviews said they had found the programme activities they had been involved with useful.
• We have substantially influenced policy and practice. 91% of respondents thought we had influenced policy locally, and 82% nationally; and 100% of respondents believed we had contributed to changes in their working practices, and 92% that we had contributed to changes in the interventions they deliver.
• 77% of respondents have accessed our website resources which include journal articles, research and policy briefs, case studies and learning papers, our tools and guides and social media channels. 96% of respondents said they had found our resources useful and all respondents found them easy to use.
• Overall, respondents highly rated their experience of working with the programme, with 62% giving a rating of 5 out of 5 and 31% of 4 out of 5.

Background
COMDIS has a wealth of experience in developing and running international programmes for 13 years across two consortia since inception in 2006. This has included developing systems and working practices; growing the capacity of partners to design and manage their research; and working closely with policy makers and practitioners to deliver research to improve health services for their populations.
At the beginning of 2018 we enhanced the method for evidencing our impact by modifying our theory of change into an impact pathway for the consortium (see Annex 1). This has helped us better understand the pathways to impact, how change happens, and how it can be evidenced.
In order to begin to consolidate what we have learnt as a consortium, and test our impact pathway, we carried out an evaluation to explore how we have built the capacity of our consortium partners; the effectiveness of our programme; how our research has influenced policy makers’ and practitioners’ ways of working; key lessons learned; and what difference we have made.
Methods
The evaluation explored two key themes:

• How COMDIS-HSD has contributed to building the capacity of consortium partners at individual and organisational level;
• How COMDIS-HSD research has contributed to influencing policy makers’ and practitioners’ ways of working.

Investigating our impact on beneficiaries (vulnerable men, women and children) is outside the scope of this evaluation. Due to the short time available it was not possible to interview beneficiaries directly to explore how our research has impacted them and their families. For further information on our impact, please see our COMDIS-HSD impact summary 2011-2018 which details our achievements and how our work is making a difference to the lives of many people living in low- and middle-income countries.
We conducted 15 semi-structured qualitative interviews with former and current members of staff from our partner organisations, covering a broad range of roles (i.e. research, operational, RU, management). Each interview explored the capacity building efforts and processes used during the programme, what difference the programme had made to individuals and organisations, and the key lessons they had learned.
We also conducted an on-line stakeholder survey, which was completed by 13 policy makers and practitioners from our partner countries. This was followed by two in-depth qualitative interviews, with one policy maker and one practitioner who had worked closely with us on our research activities in their respective countries. The survey and interviews explored how policy makers and practitioners had interacted with the programme and how our research had influenced changes to policy and their working practices.
This evaluation report also includes some reflections and experiences from partner staff and policy makers and practitioners interviewed who were involved with COMDIS-HSD’s predecessor (COMDIS).
Findings from the Evaluation
Capacity building and engagement activities
From the interviews with partner staff it was evident that our internal capacity building activities have been of high quality, covering all aspects of the consortium including research skills, programme and project management, research uptake and communications, finance, M&E and governance. Capacity building is viewed as an essential and integral component of being able to deliver a programme effectively.
In the early days of COMDIS capacity building needs were dealt with on an ad-hoc basis. From the start of COMDIS-HSD a more strategic approach to capacity building was developed and a capacity needs assessment was completed by each partner. This more clearly described partner’s capacity needs and the support required to address the needs. Although in some cases this produced a long list of requirements (which was reduced and focused over time), it provided an understanding of the bespoke requirements for each partner and the more common capacity needs across the consortium.
We addressed the internal capacity building needs across the consortium in two main ways. The first and largest component was through ‘learning by doing’ where partners worked together with in-house subject experts to gain in-depth knowledge and skills to be able to successfully deliver each area of the programme. Examples include, for research skills working step-by-step on the study design, data collection tools, analysis and write up; for research uptake approaches, providing bespoke support on communication plans, website development and writing. This method of learning has proved invaluable for all of our partners and is a key strength of how we work together as a consortium. The second component was delivery through more formalised structured training and workshops. This type of training has also proved very useful, offering more dedicated time away from the office to develop specific skills and knowledge. Examples include writing for publications workshops; conducting randomised control trials; and influencing behaviour change techniques in tobacco cessation.
Partner staff remarked that capacity building activities had been largely effective and where an activity was less useful it was due to issues with timing or preparation, or that they would have liked more frequency in training on a particular topic area.
Partner staff commented that the capacity building activities delivered by the programme had contributed to enhancing their knowledge and skillsets at an individual and organisational level in many areas including:

• Conducting rigorous and effective research (incorporating more advanced qualitative and quantitative methods and gender and intersectionality issues);
• writing case studies, briefs and publications;
• understanding and implementing research uptake approaches;
• collaboration and advocacy techniques;
• leadership skills;
• project and programme management methods and other operational skills such as finance, governance and M&E.

We engaged with policy makers and practitioners in a variety of ways through our research and capacity building activities. This included stakeholder meetings and technical working groups to develop the packages of care in-country and oversee our research; support with capacity building training for healthcare staff; support with research implementation activities; and holding dissemination and knowledge sharing events. All respondents to the on-line stakeholder survey and interviews said they had found the programme activities they had been involved with useful. 73% of respondents reported they had found activities very useful, 21% were reported as useful with 6% reported as somewhat useful. Communicating and sharing information across teams was identified as a key component.
77% of respondents to the survey confirmed they had accessed our resources either through the COMDIS-HSD website, consortium partner websites or via other sources. Research and policy briefs have proved to be the most popular of the resources accessed (see table below).

69% of respondents said they had found our resources very useful, with 21% finding them useful, 6% as somewhat useful and only 4% as not useful. All respondents found our resources easy to use with 40% finding them very easy to use and 60% finding them easy to use.
For those respondents who had not accessed our resources (23%), this was due to either not being aware of our programme earlier, being unsure where to find our website or being too busy focusing on the research implementation to read our resources.
Impact
Partner staff confirmed that through our capacity building efforts we have contributed to building a highly skilled team across all of our partners. The programme has contributed to enhancing knowledge and skillsets at an individual and organisational level across all areas, and has helped strengthen career progression for some individuals into more challenging roles. The programme has supported some of our partner organisations to build confidence, grow and become attractive to other funders, which subsequently contributed to their longer term sustainability. Partners are able to secure funding from wider sources because of their credibility and very strong reputations.
We have helped partners to strengthen relationships with senior officials within their Ministries and encouraged them to work closely with in-country programmes to understand their issues and interventions needed. This type of collaborative working and flexibility to pick up on areas that are essential to the country have been crucial to embedding our research within national health programmes and meeting these needs. Partners’ collaboration and advocacy approaches with their Ministries are exceptionally strong.
The programme has provided useful opportunities for partners to work together and learn from each other in different country contexts, including political and cultural influences on health and service delivery.
Some partner staff commented that, if the COMDIS-HSD programme (and its predecessor COMDIS) had not existed, their organisation would not have developed as quickly or in the way that it had. In some cases, accessing and building strong relationships with their Ministries would have been much more difficult. Country programmes might have continued with their in-country priorities securing funding from other sources. However, this might have been on a much slower trajectory and the rigour and standards of research studies might have suffered.
When participants were asked to consider what made the programme unique or unusual, they said they strongly believed that securing long term funding across two consortia had enabled effective trusting relationships to be built between partners and with stakeholders in country. It is important to note that partnerships do not grow overnight and need time to develop and mature. The programme length and flexibility allowed our partners to focus on demonstrating impact and scale-up of research interventions. Partners were able to take a lead in their respective countries, be nimble and responsive to the in-country programme priorities, and embed our research to address identified needs.
When asked whether the programme had contributed to outcomes outside its scope, some partner staff commented that they believed the scope was too broad from the start and there was a lack of focus. However, we tried to address this issue over time by reducing our research remit and working to develop more coherence across our research themes and activities.
We have substantially influenced policy and practice. 91% of respondents thought we had influenced policy locally, and 82% nationally; and 100% of respondents believed we had contributed to changes in their working practices, and 92% that we had contributed to changes in the interventions they deliver. See table below for further detail on the programmes level of contribution to policy and practice change.

Overall we have made really positive progress with a 64% significant or reasonable contribution to influencing policy change and an 81% significant or reasonable contribution to changing working practices.
Policy makers and practitioners commented that the programme had influenced changes to policy and their ways of working in a variety of ways:

• Through our work to raise awareness of different health issues and needs, we have supported the Ministries to review and re-focus on their in-country priorities.
• Our technical inputs have helped to improve the quality of the interventions delivered.
• We have supported enhancement of skillsets, and increased motivation and confidence to deliver the research interventions (e.g. through training of healthcare staff).
• Embedding our research with in-country priorities has helped to strengthen country programmes, with tailored research that is informing policy processes. An example is decentralising hospital-led care to local clinics, thereby increasing reach to high quality care, i.e. ‘taking the service to the people’.
• Our dissemination activities have helped to raise the profile of our research outcomes and impact.

Overall, survey respondents highly rated their experience of working with the programme, with 62% giving a rating of 5 out of 5 and 31% of 4 out of 5. The remaining 7% rated their experience as 3 out of 5.
Learning
Partner staff were asked to share lessons learned from their experiences of working on the programme. A summary of the main key learning points is given below.
Working with Ministries and wider stakeholders

• Develop resilient relationships – it is important to build relations with Ministries and wider stakeholders through regular communication and advocacy techniques.
• Work with in-country priorities – it is important to find a pragmatic balance on what you can work on and what the priorities are.
• Integrate the research within routine health programmes – ensure you know how to ‘embed’ research with the Ministries and in-country programmes.
• Research uptake is an essential component of the programme – it is important to communicate research findings using a variety of different media channels, to facilitate use of research evidence.
• Global advocacy and influence – consider developing strategies to access and build relationships with the relevant international organisations.

Carrying out research

• Coherence is key – from the outset, consistency and focus are needed on a shared research agenda across all partner countries.
• Ensure you understand the impact you want to achieve from the start – unless you can articulate why you want to do the work, it will be difficult as you’re not all on the same page.
• Planning is very important – try not to do ‘quick and dirty’ research as this doesn’t inform you very well.
• Volume of work – be clear and specific on what you can achieve, otherwise you can take on too much.
• Spend more time planning and producing outputs – don’t leave publications until the end, when they can be planned much earlier on.
• Gender and intersectionality – there are useful tools out there to help you think through these issues.
  Always be reflective – this allows you to be ready to learn new things, ready to understand gaps and needs and helps you consider how to improve.

Consortium partners

• It is important to have a large spread of capacity within the team – (research, leadership, RU and operational). Recognise the skills needed, identify clear roles and ensure clear decision-making from the start.
• Oversight of the programme – having a management team in place is key to cover all elements of the programme.
• Diversity of partners – this has helped us to know more so we can generate new areas of interest and deliver on our commitments.
• Partnerships do not grow overnight – they need to time to develop and mature and longevity of funding has helped build strong relationships over a long period of time.
• Cross collaboration between partners – start discussions early on about working on multi-country studies and consider a rolling approach to developing and adapting the packages of care across countries.
• Organisational systems and processes – it is important to have the right structures in place from the outset.

Conclusion
Investing in capacity strengthening, both internally and externally, is a vital component to successfully delivering a programme. Capacity building should not be seen as stand-alone but integral to everything we do. The longevity of funding is also a key element to delivery, offering time to build effective partnerships, influence changes to policy and practice, and ultimately demonstrate the impact of our interventions.
The results from this evaluation give evidence of our achievements in meeting the impact statements in the COMDIS-HSD impact pathway:

• Our programme has contributed to building sustainable resilient partners who are able to implement effective programmes of research.
• Policy makers and practitioners have been influenced by our research processes and findings

COMDIS-HSD (and its predecessor COMDIS) has offered valuable opportunities to learn and develop at an individual and organisational level and build effective long-lasting relationships across partners, with Ministries and with wider stakeholders. COMDIS-HSD has enabled us to be agile, respond rapidly to new health issues, be ready with our research evidence and packages of care, and have major impact on health services and the health of vulnerable people in low- and middle-income countries.
1. Communicable Disease Research Programme Consortium (COMDIS) 2006-2010.
2. Communicable Disease Health Service Delivery Research Programme Consortium (COMDIS-HSD) 2011-2018.
3. Policy Makers and Practitioners who completed the on-line survey were from the following types of organisations: Government; Non-governmental organisation (NGO)/Charity; Hospital/Health Centre; Educational Institution.

Annex 1: COMDIS-HSD Impact Pathway