This think tank session will be held at AEA on Thursday November 6 from 4:30 to 6:00. Please come and join the conversation.

ABSTRACT

Systems thinking, tools, techniques, and methods have been apparent at AEA for over five years. This Think Tank will explore some fundamental questions about systems and evaluation, including:
• What are “non-systemic” evaluations? When, if ever, are they a good choice?
• How are the multiple systems approaches similar to each other? Different from each other?
• What are good selection criteria when choosing among systems-based evaluation approaches?
• What are the risks and benefits of working with systems-based approaches as fundamental theories of change? As models of behavior and interaction? As methods of design, data collection, analysis, reporting? As tools to collect and analyze data?

In small discussion groups, participants will address these questions, document their differences and agreements, and share their findings through an article that will be submitted for publication.

Relevance Statement

In the beginning, when systems theory and practice entered the domain of evaluation, they were seen as multiple collections of arcane and complicated methods, tools, and techniques. Over time, systems approaches have adapted to the needs of evaluators. Computerized tools simplify systems dynamics modeling, social networking, and complex text analysis processes. Language of systems has become clearer and less intimidating, and systems proponents have begun to talk across their boundaries to overcome differences that do not make a difference in practice. At the same time, evaluators have incorporated some of the fundamental tenets of systems thinking into their everyday practice, including causality across levels, environmental interaction, sensitivity to initial conditions, unintended consequences, and the relationships between local and global patterns of behavior.

The boundaries between the theory and practice of systems and the theory and practice of evaluation have gotten fuzzy. Relationship between the two is beneficial, but the risk is that the significant distinctions will be lost. If all systems are evaluative and all evaluation is systemic, then scholars and practitioners lose the potential power of both. To be most effective, we need to be able to think of systemic and non-systemic evaluations. We also need to recognize that systems applications that inform program design and implementation are different from those that inform evaluation.

Discourse in the evaluation community is finally mature enough to tackle critical distinctions in the midst of generative similarities. This 90-minute think tank is designed to explore and articulate the critical similarities and distinctions between evaluation thinking and systems thinking with the fervent hope that we will all gain a richer, more coherent, and more useful framework for both systems and evaluation.

Participants will be asked to respond to open-ended questions and to capture the essence of their dialogues. Data generated during the session will be captured and incorporated into a manuscript that will be submitted for publication. All participants will be included as co-authors of the paper. Individuals who wish to be excluded from the list of authors will be offered that opportunity. The expectation is that such an engaged and emergent dialogue among theoreticians and practitioners of both systems and evaluation will inform an emergent foundation for future research and practice in both fields.