There is a well-documented interest in how insights from the study of complexity can be applied to policy evaluation. However, important questions remain as to how complexity is understood and used by policy evaluators. We present findings from semi-structured interviews with 30 UK policy evaluators working in food, energy, water and environment policy domains. We explore how they understand, use and approach complexity, and consider the implications for evaluation research and practice. Findings reveal understandings of complexity arising from contextual factors, scale-related issues and perceptions of unpredictability. The evidence indicates terminological and analogical use of complexity and its concepts by policy evaluators, but limited evidence of its literal use. Priorities for the future include framing complexity more pragmatically and as an opportunity not a cost. Communicating this up the policy hierarchy is the key to progressing complexity-appropriate evaluation – this can be enabled by strengthening links between policy evaluation and academic communities.
This paper investigates the role of evaluation commissioning in hindering the take-up of complexity-appropriate evaluation methods, using findings from interviews with 19 UK evaluation commissioners and contractors. We find, against a backdrop of a need to ‘do more with less’ and frustration with some traditional approaches, the commissioning process is perceived to hinder adoption of complexity-appropriate methods because of its inherent lack of time and flexibility, and assessment processes which struggle to compare methods fairly. Participants suggested a range of ways forward, including more scoping and dialogue in commissioning processes, more accommodation of uncertainty, fostering of demand from policy users, more robust business cases, and more radical overhauls of the commissioning process. Findings also emphasised the need to understand how the commissioning process interacts with the wider policy making environment and evidence culture, and how this manifests itself in different attitudes to risk in commissioning from different actors.
Despite 20 years of increasing acceptance, implementing complexity-appropriate methods for ex-post evaluation remains a challenge: instead of focusing on complex interventions, methods need to help evaluators better explore how policies (no matter how simple) take place in real-world, open, dynamic systems where many intertwined factors about the cases being targeted affect outcomes in numerous ways. To assist in this advance, we developed case-based scenario simulation, a new visually intuitive evaluation tool grounded in a data-driven, case-based, computational modelling approach, which evaluators can use to explore counterfactuals, status-quo trends, and what-if scenarios for some potential set of real or imagined interventions. To demonstrate the value and versatility of case-based scenario simulation we explore four published evaluations that differ in design (cross sectional, longitudinal, and experimental) and purpose (learning or accountability), and present a prospective view of how case-based scenario simulation could support and enhance evaluators’ efforts in these complex contexts.
Theory of Change diagrams are commonly used within evaluation. Due to their popularity and flexibility, Theories of Change can vary greatly, from the nuanced and nested, through to simplified and linear. We present a methodology for building genuinely holistic, complexity-appropriate, system-based Theory of Change diagrams, using Participatory Systems Mapping as a starting point. Participatory System Maps provide a general-purpose resource that can be used in many ways; however, knowing how to turn their complex view of a system into something actionable for evaluation purposes is difficult. The methodology outlined in this article gives this starting point and plots a path through from systems mapping to a Theory of Change evaluators can use. It allows evaluators to develop practical Theories of Change that take into account feedbacks, wider context and potential negative or unexpected outcomes. We use the example of the energy trilemma map presented elsewhere in this special issue to demonstrate.
The use of complexity science in evaluation has received growing attention over the last 20 years. We present the use of a novel complexity-appropriate method – Participatory Systems Mapping – in two real-world evaluation contexts and consider how this method can be applied more widely in evaluation. Participatory Systems Mapping involves the production of a causal map of a system by a diverse set of stakeholders. The map, once refined and validated, can be analysed and used in a variety of ways in an evaluation or in evaluation planning. The analysis approach combines network analysis with subjective information from stakeholders. We suggest Participatory Systems Mapping shows great potential to offer value to evaluators due to the unique insights it offers, the relative ease of its use, and its complementarity with existing evaluation approaches and methods.
The value of complexity science and related approaches in policy evaluation have been widely discussed over the last 20 years, not least in this journal. We are now at a crossroads; this Special Issue argues that the use of complexity science in evaluation could deepen and broaden rendering evaluations more practical and rigorous. The risk is that the drive to better evaluate policies from a complexity perspective could falter. This special issue is the culmination of 4 years’ work at this crossroads in the UK Centre for the Evaluation of Complexity Across the Nexus. It includes two papers which consider the cultural and organisational operating context for the use of complexity in evaluation and four methodological papers on developments and applications. Together, with a strong input from practitioners, these papers aim to make complexity actionable and expand the use of complexity ideas in evaluation and policy practice.
In this case study, CECAN, BEIS and CAG Consultants applied CECAN’s approach to Participatory Systems Mapping to support the evaluation of the Renewable Heat Incentive.
The energy trilemma describes the interaction in the energy system between sustainability and emissions, affordability and prices, and security of supply. The sheer number of programmes and policies with close interaction and overlap in this area has led to a crowded and complex policy landscape with a range of potentially complementary and conflicting aims. In this case study, CECAN and the Department for Business, Energy and Industrial Strategy (BEIS) worked together to build a richer understanding of this complex area by developing a participatory systems map of the energy trilemma.
Computational models are increasingly being used to assist in developing, implementing and evaluating public policy. This paper reports on the experience of the authors in designing and using computational models of public policy (‘policy models’, for short). The paper considers the role of computational models in policy making, and some of the challenges that need to be overcome if policy models are to make an effective contribution. It suggests that policy models can have an important place in the policy process because they could allow policy makers to experiment in a virtual world, and have many advantages compared with randomised control trials and policy pilots. The paper then summarises some general lessons that can be extracted from the authors’ experience with policy modelling. These general lessons include the observation that often the main benefit of designing and using a model is that it provides an understanding of the policy domain, rather than the numbers it generates; that care needs to be taken that models are designed at an appropriate level of abstraction; that although appropriate data for calibration and validation may sometimes be in short supply, modelling is often still valuable; that modelling collaboratively and involving a range of stakeholders from the outset increases the likelihood that the model will be used and will be fit for purpose; that attention needs to be paid to effective communication between modellers and stakeholders; and that modelling for public policy involves ethical issues that need careful consideration. The paper concludes that policy modelling will continue to grow in importance as a component of public policy making processes, but if its potential is to be fully realised, there will need to be a melding of the cultures of computational modelling and policy making.
D Anzola, P Barbrook-Johnson, M Salgado, Nigel Gilbert (2017)Sociology and Non-Equilibrium Social Science, In: Non-Equilibrium Social Science and Policy: Introduction and Essays on New and Changing Paradigms in Socio-Economic Thinking(4)pp. 59-69
Springer International Publishing
Abstract This chapter addresses the relationship between sociology and Non- Equilibrium Social Science (NESS). Sociology is a multiparadigmatic discipline with significant disagreement regarding its goals and status as a scientific discipline. Different theories and methods coexist temporally and geographically. However, it has always aimed at identifying the main factors that explain the temporal stability of norms, institutions and individuals’ practices; and the dynamics of institutional change and the conflicts brought about by power relations, economic and cultural inequality and class struggle. Sociologists considered equilibrium could not sufficiently explain the constitutive, maintaining and dissolving dynamics of society as a whole. As a move from the formal apparatus for the study of equilibrium, NESS does not imply a major shift from traditional sociological theory. Complex features have long been articulated in sociological theorization, and sociology embraces the complexity principles of NESS through its growing attention to complex adaptive systems and non-equilibrium sciences, with human societies seen as highly complex, path-dependent, far-from equilibrium, and selforganising systems. In particular, Agent-BasedModelling provides a more coherent inclusion of NESS and complexity principles into sociology. Agent-based sociology uses data and statistics to gauge the ‘generative sufficiency’ of a given microspecification by testing the agreement between ‘real-world’ and computer generated macrostructures.When the model cannot generate the outcome to be explained, the microspecification is not a viable candidate explanation. The separation between the explanatory and pragmatic aspects of social science has led sociologists to be highly critical about the implementation of social science in policy. However, ABM allows systematic exploration of the consequences of modelling assumptions and makes it possible to model much more complex phenomena than previously. ABM has proved particularly useful in representing socio-technical and socio-ecological systems, with the potential to be of use in policy. ABM offers formalized knowledge that can appear familiar to policymakers versed in the methods and language of economics, with the prospect of sociology becoming more influential in policy.
Nigel Gilbert, D Anzola, P Johnson, C Elsenbroich, T Balke, O Dilaver Kalkan (2015)Self-organizing dynamical systems, In: International Encyclopedia of the Social & Behavioral Sciences21pp. 529-534
The concept of self-organization in social science is reviewed. In the first two sections, some basic features of self-organizing dynamical systems in general science are presented and the origin of the concept is reconstructed, paying special attention to social science accounts of self-organization. Then, theoretical and methodological considerations regarding the current application of the concept and prospective challenges are examined.
Two leading camps for studying social complexity are case-based methods (CBM) and agent-based modelling (ABM). Despite the potential epistemological links between ‘cases’ and ‘agents,’ neither camp has leveraged their combined strengths. A bridge can be built, however, by drawing on Abbott’s insight that ‘agents are cases doing things’, Byrne’s suggestion that ‘cases are complex systems with agency’, and by viewing CBM and ABM within the broader trend towards computational modelling of cases. To demonstrate the utility of this bridge, we describe how CBM can utilise ABM to identify case-based trends; explore the interactions and collective behaviour of cases; and study different scenarios. We also describe how ABM can utilise CBM to identify agent types; construct agent behaviour rules; and link these to outcomes to calibrate and validate model results. To further demonstrate the bridge, we review a public health study that made initial steps in combining CBM and ABM.
In the UK, calls for the application of insights from the study of complex adaptive systems to public policy evaluation are beginning to be taken seriously in government. Policymakers and analysts are accepting the fallibility of overly simplistic, definitive, or linear analysis, or are finding traditional forms of analysis and evidence less appropriate or feasible. Through our work in CECAN (the Centre for the Evaluation of Complexity Across the Nexus), we reflect on our experiences and the practical challenges of using complexity-appropriate computational modeling with policy analysts and evaluators in UK central government. As an example, we discuss our work with the COMPLEX-IT toolkit, which uses a selection of case-based computational modeling approaches. We end by suggesting ways forward for applied complexity scientists, and policy evaluators and analysts to make more effective use of these methods.
Government communication is an important management tool during a public health crisis, but understanding its impact is difficult. Strategies may be adjusted in reaction to developments on the ground and it is challenging to evaluate the impact of communication separately from other crisis management activities. Agent-based modeling is a well-established research tool in social science to respond to similar challenges. However, there have been few such models in public health. We use the example of the TELL ME agent-based model to consider ways in which a non-predictive policy model can assist policy makers. This model concerns individuals’ protective behaviors in response to an epidemic, and the communication that influences such behavior. Drawing on findings from stakeholder workshops and the results of the model itself, we suggest such a model can be useful: (i) as a teaching tool, (ii) to test theory, and (iii) to inform data collection. We also plot a path for development of similar models that could assist with communication planning for epidemics.
Complexity science and its methodological applications have increased in popularity in social science during the last two decades. One key concept within complexity science is that of self-organization. Self-organization is used to refer to the emergence of stable patterns through autonomous and self-reinforcing dynamics at the micro-level. In spite of its potential relevance for the study of social dynamics, the articulation and use of the concept of self-organization has been kept within the boundaries of complexity science and links to and from mainstream social science are scarce. These links can be difficult to establish, even for researchers working in social complexity with a background in social science, because of the theoretical and conceptual diversity and fragmentation in traditional social science. This article is meant to serve as a first step in the process of overcoming this lack of cross-fertilization between complexity and mainstream social science. A systematic review of the concept of self-organization and a critical discussion of similar notions in mainstream social science is presented, in an effort to help practitioners within subareas of complexity science to identify literature from traditional social science that could potentially inform their research.
Melvin Lippe, Mike Bithell, Nick Gotts, Davide Natalini, Peter Barbrook-Johnson, Carlo Giupponi, Mareen Hallier, Gert Jan Hofstede, Christophe Le Page, Robin B. Matthews, Maja Schlüter, Peter Smith, Andrea Teglio, Kevin Thellmann (2019)Using agent-based modelling to simulate social-ecological systems across scales, In: GeoInformatica
Agent-based modelling (ABM) simulates Social-Ecological-Systems (SESs) based on the decision-making and actions of individual actors or actor groups, their interactions with each other, and with ecosystems. Many ABM studies have focused at the scale of villages, rural landscapes, towns or cities. When considering a geographical, spatially-explicit domain, current ABM architecture is generally not easily translatable to a regional or global context, nor does it acknowledge SESs interactions across scales sufficiently; the model extent is usually determined by pragmatic considerations, which may well cut across dynamical boundaries. With a few exceptions, the internal structure of governments is not included when representing them as agents. This is partly due to the lack of theory about how to represent such as actors, and because they are not static over the time-scales typical for social changes to have significant effects. Moreover, the relevant scale of analysis is often not known a priori, being dynamically determined, and may itself vary with time and circumstances. There is a need for ABM to cross the gap between micro-scale actors and larger-scale environmental, infrastructural and political systems in a way that allows realistic spatial and temporal phenomena to emerge; this is vital for models to be useful for policy analysis in an era when global crises can be triggered by small numbers of micro-level actors. We aim with this thought-piece to suggest conceptual avenues for implementing ABM to simulate SESs across scales, and for using big data from social surveys, remote sensing or other sources for this purpose.