A FRAMEWORK FOR EVALUATING PARTNERSHIPS HOW A COMPREHENSIVE

A FRAMEWORK FOR EVALUATING PARTNERSHIPS HOW A COMPREHENSIVE APPROACH CAN STRENGTHEN UNDERSTANDING AND APPLICATION

Overview l Partnerships as delivery of policy programs have long history l Development of effective frameworks and tools been lagging l Evaluators face technical challenges l Within international development also face conceptual challenge l l Case example of a Partnership Evaluation Framework Use of framework can strengthen understanding

A brief history and definitional issues l l Not a new concept No single, common definition – confusion (Carnwell and Carson, 2004; Dowling et al, 2004; Caplan et al, 2007) l l Different drivers and underlying theories Relatively recent emergence in international development (Piciotto, 2007)

Current evaluation approaches l Few studies to assess performance or effectiveness (Brinkerhoff, 2002; Jorgensen, 2006; Serafin, 2008) l l Focus primarily on outputs (Caplan et al, 2007) No agreement about elements to be assessed – different elements emphasised by different authors

Partnership assessment tools l l Include some of the elements emphasised in literature Primarily for developmental purposes Beneficial (Sunderland et al, 2009) Do NOT perform well across all elements (Halliday et al, 2004)

Partnership evaluation frameworks l l l Emerged in response to perceived shortfalls Range of domains + sub dimensions More comprehensive: – – l l developmental and evaluative Outcomes Questions, evidence criteria Mixed methods

Evaluation challenges l Technical - need to adjust methods (Conlin & Stittat, 2006; Jobin, 2008) l Conceptually (international development) – – l Uptake in short space of time Broad scale and scope, and more complex Promotion of sovereignty (Picciotto, 2007) Still grappling with purpose Involving partners – – Practical participatory evaluation Transformative participatory evaluation (Cousins and Whitmore, 1998)

Case example l l Review - Australia and Indonesia Partnership Maternal and Neonatal Health New modality Review – 2010; 2 years into partnership Effectiveness and clarification of partnership

Case example Partners still grappling with concept l No shared view l I wanted a tool or framework to further develop the concept and increase understanding l Partnership Assessment Tool (Hardy, 2003) promising BUT… l Frameworks more comprehensive BUT… l

Case example l Developed a framework based on mix of: – – l l Elements highlighted by the partners Salient and commonly used features from literature Drew heavily on Brinkerhoff (2002) Plus Caplan et al (2007), Jobin (2008),

Partnership Evaluation Framework Hind, 2010 PREREQUISITES Enabling environment • Political attitudes • Preoccupations and priorities • Commitment • Expectations • Stability of environment Drivers: Organisational & individual • Incentives • Obligations Institutional elements • Common objectives – public policy purpose • Shared governance • Written agreement STRUCTURE Formal partnership dimensions • Legitimacy • Resources • Responsibilities • Decision-making • Contribution • Compliance Informal partnership dimensions • Social capital Reputation Trust PROCESS Partnership practice • Nature of Interaction • Capacity development • Mutuality and equality Partnership performance • Relationships • Performance of representatives • Conflict resolution • Creating and strengthening success factors OUTCOMES Effectiveness • Achieving desired results – including Influence on: o. Administrative capacity o. Service delivery capacity o. Community capacity • Responsiveness Efficiency • Make good use of partner resources • Benefits • Costs • Added value

Areas of assessment l l Intended to develop assessment areas participatory Accepted a pragmatic approach – – – Practical Contextually responsive Consequential (Datta, 1997)

Enabling environment Extent to which: partnership is supported politically; has the backing of key stakeholders; there are real opportunities for collaboration; partners tolerate and encourage powersharing partnership meets current priorities partners are able to focus on the partnership given other work/priorities/demands partners provide leadership; there are champions; partners are willing to adapt; partners are future oriented there are clear and reasonable expectations; expectations are similar between partners environment is stable and conducive to collaborative working; key individuals are in place to facilitate partnership; external context is understood Prerequisites Areas for assessment Drivers: organisational and individual Extent to which: purpose, mandate, vision of partner organisations supports/facilitates partnership partners have inbuilt organisational culture, processes, systems, demands to facilitate partnership partners have legal and/or organisational requirements to partner decision making processes are clear and sufficient to facilitate partnership partners have processes in place to address lapses in commitment Institutional elements Extent to which: there are common, shared goals linked to relevant public policies; partners articulate what they want to achieve from the partnership formal and informal governance structures are in place and work; processes and structures have been developed collaboratively the partnership is articulated in writing yet flexible to adapt as required

Application of framework l l Chose all domains and dimensions Mixed methods q q q Document review Semi-structured interviews Observational visits Informal interaction with beneficiaries Workshops o o Presentations by partners Facilitated group discussion Partner surveys using 5 -point ranking scale Narrative writing

Effectiveness of the Framework R Sufficient breadth to: – – – address issues re: time, resources, data availability answer all salient evaluation questions make judgement using most not all dimensions Could be used flexibly within the local context R Drew on literature and salient features identified by partners R

Did it help strengthen the understanding of partnership? R Workshops with partners o o o R Opportunity to discuss and reflect on partnership New insights + promoted broader thinking Some further development in understanding Discussions during report drafting o Opportunity to examine the sub dimensions and their relevance advanced thinking of concept BUT… • More in-depth discussions were limited so potential constrained

Conclusion l l l Evaluating partnerships is essential Still in formative stage Concept poses a challenge therefore is a challenge for evaluators The framework was robust Need to adapt for local situation The partnership evaluation framework + techniques can help progress concept

Partnership assessment tools l The Partnership Analysis Tool: Vic. Health: http: //www. vichealth. vic. gov. au/en/Publications/Vic. H ealth-General-Publications/Partnerships-Analysis. Tool. aspx l The Partnership Self-Assessment Tool: Center for the Advancement of Collaborative Strategies in Health; www. partnershiptool. net l The Partnership Assessment Tool: Hardy et el, 2003, http: //www. communities. gov. uk/documents/localgove rnment/pdf/135112. pdf

Partnership evaluation frameworks l Brinkerhoff (2002) – domains linked through flow chart l Caplan et al (2007) – the ‘drivers’: external, organisational + individual contexts l Jobin (2008) –transaction cost theory; allows consideration of counterfactual; social capital