Mixing evaluation cultures in learning communities
By Paul Beenen, Matthias Guett, Ron Kersjes, & Ageeth Jorna
Knowledge institutions set themselves the challenge of positioning themselves in the (regional) social challenges. As Sterling succinctly noted: “Higher education has largely failed to meet the urgent demands of sustainability”. This is a call for urgent reflection, how we can become more performative in this major tasks ahead. Characteristic of these tasks is that they are complex in nature, which in any case means that it is not possible to define the outcomes of an often lengthy process at the front end.
Being able to anticipate these complex issues also requires a fundamental change in thinking and acting from the knowledge institutions. The infrastructure of a learning community lends itself well to this learning in complexity (Beenen et al, 2024). The starting point is an informal collective learning process in which various stakeholders , including the knowledge institutions, work together, learn, research and innovate.
This article describes the consequences for evaluating and monitoring the work on these complex issues. The article challenges to critically consider often one-sided evaluation and testing culture in education and its limitations. We argue that this evaluation culture is not sufficient for education that plays a role in (regional) social tasks. We then offer an alternative in the developmental evaluation culture and a pragmatic mix between both cultures.
EVALUATION CULTURES
The possibilities for evaluation are – partly due to the availability of data technology – gradually becoming endless. In practice, this means that we always have to make a selection or risk getting bogged down in details so that we can no longer see the wood for the trees. In addition to the availability and amount of information, we can also look at evaluation culture.
In the traditional way of evaluation, the starting point is usually that the intended outcomes of a (proposed) intervention are defined in advance and that it is checked (or checked) afterwards whether these have been achieved. We call this an audit culture.
In innovative and transformative processes, we don’t know exactly what the (distant) future will look like; Along the way, all kinds of things will happen that affect this. Specific outcomes cannot be defined at the front end, or can be defined to a limited extent. And we shouldn’t want to: it is precisely in the uncertain phase that there is an enormous collective learning and innovative power. In learning in complexity, we cherish this power. In other words; It is a complex process that is formulated as a transition for the social task (see box 1). To learn and evaluate in these processes, a more developmental culture is needed.
In this article, we distinguish between the above two cultures: the audit culture and the developmental culture. We argue that in learning in complexity there must be a place for both, and attention must be paid to the mutual coherence.
The term transitions is increasingly used to refer to large-scale disruptive changes that are expected to contribute to major societal challenges. They intervene profoundly in social systems and occur over a period of several decades. (After: Loorbach, 2017; Hölscher et al 2018)
Transitions can therefore be seen as major, urgent and long-term social system changes that go hand in hand with substantial changes in views on the world and on what appropriate methods and working methods are.
We can also consider the developments within Higher education in which we want to focus on learning in learning communities as a transition. In this book, the traditional ideas about education, knowledge conception and teacher roles are discussed (Topsectoren, 2019 a, b).
[Box 1]
Complex or complicated?
In the foregoing, we actually make a distinction between complex and complicated issues. Insight into the differences and between complex and complicated issues will help to understand how the two evaluation cultures differ and are interrelated. Complicated issues can be very difficult, for example because they require high-quality, specialist knowledge of different disciplines, but if you bring in the right expertise, they are easy to think out in advance and therefore predictable.
Complex issues have interdependencies and are unpredictable. Think of major social challenges such as caring for health, combating climate change and the transformation to a circular economy. They are long-term processes in which many factors and those involved constantly influence each other (systemically) and thus shape and distort the issue along the way. They cannot be defined and planned in advance. In figure 1 (see full article) we give a number of examples of both types of issues.
Complex and complicated
In complex issues, you often come across many complicated issues. These are the stable components that are short-term, clear and plannable. They have a more or less predictable and constant outcome. In learning communities, think of a recurring commitment of students from an educational curriculum or is called setting up a research project.
For the learning community, this commitment aims to contribute to the larger complex issue and therefore you cannot see them separately from each other in practice. Often many initiatives aimed at complex issues together form ‘the portfolio’ of the approach to complex issues. In a transition-oriented approach, you always walk on two legs, so to speak: from complicated to complex and back again. We will come back to this at the end of this article.
THE STRENGHT AND LIMITATION OF THE AUDIT CULTURE
Powerful instrument in a stable world
In a predictable world, the basis of evaluation is the measurement of fixed activities. This can define outcomes and criteria at the outset, which can then be measured after the intervention to see whether sufficient results have been achieved and can be adjusted. This creates a classic ‘Plan-Do-Check-Act cycle’ that optimizes the process and steers it towards the desired results. It is an instrument that monitors planned activities, is traceable and can be accounted for. Ultimately, a planned linear ‘impact pathway’ can then develop in terms of ‘input > activity > output > outcome > impact’. The plannability offers the possibility to link a budget to the activities and to set up a timeline with milestones that is easy to monitor and control. Thinking and acting in short-cycle projects fits in perfectly with this form of evaluation. The underlying idea of “measuring is knowing” is also the starting point in the action repertoire of professionals, and the education for it. By focusing on the desired behavior and outcomes, those involved are less distracted by all kinds of peripheral issues and can work in a result-oriented and efficient way. This audit culture has proven to be enormously successful in our Western thinking and has provided predictability, transparency and trust because we can account for our (future) actions by evaluating.
The limitation of pre-planned outcomes
The audit culture is not suitable for all processes. There is a risk that it will lead to blinders. You judge on the planned outcomes; In this view, what happens around it in the context is quickly seen as ‘noise’. Steering on generic outcome measures takes little account of the specificity of the environment and the influence of people who have a role within it. Consider, for example, a project by students from the knowledge institution. Often specific goals with very specific criteria are described, in this case in the form of learning outcomes. The question is how much room there is within such a project to involve other related factors that come to the surface in the learning community, for example previous products of students or results of other projects. In a learning community, the formal part of education always ends up in a continuous informal learning process. In this informal learning process, it is also important to pay attention to aspects that are difficult to measure, but essential aspects such as the use of one’s own experiences and the possibilities that the participants in the learning community have themselves or together to work towards the best possible direction in their specific situation.
The pitfall of focusing on the burden of proof
A pitfall is also that attention is mainly paid to collecting evidence for others such as the client or financier. This can distract from the original intent and potential value of the activity. Because the desired reality is already determined in advance, and therefore quite boarded up, there is little room for perceiving alternative possibilities and creativity. Prioritizing the achievement of a number of outcome measures is often at the expense of those involved to own their initiative and the bottom-up cooperation. Finally, it can also distract from paying attention to the contribution of the activity or the long-term mission and goals.
In conclusion
An audit culture works very well in top-down policy; In transition issues, it has its limitations. New opportunities for value creation along the way are missed due to predetermined outcomes and the focus on burden of proof can lead to a (perceived) lack of trust and participation among all parties involved who want to work together to arrive at new solutions.
THE CONTEXT OF KNOWLEDGE INSTITUTIONS
In recent decades, education has mainly been packaged as a standard product that can easily be taught to the consumer; the student, by means of a transfer. Knowledge products based on research and innovation are also understood as such, after which products from knowledge centres can be implemented or applied in practice. This view frames learning, researching and innovating knowledge institutions as a complicated activity. The described audit culture in which consumer satisfaction and efficiency are leading principles fits in perfectly with this. However, this view does not tell the whole story.
This proposition of knowledge institutions has been strongly criticized by, among others, educational science, pedagogy and implementation sciences (Smith & Seal, 2021; Sterling 2024; Greenhalgh, 2019). In other social domains, too, there is an increasing emphasis on the fact that knowledge, products and a learning process cannot be commodified or reduced to “products” for “users”. An increasingly strong counter-movement is visible in which participation and more collective production of value are coming to the fore. This applies not only to education but also to other social domains. Examples of this are the new service logic and ‘commons’ ideas of governance (Osborne, 2021; Thrush, 2023). It would take too long for this article to go into this in detail. We consider the acknowledgment of complexity and the inability to navigate complexity from the now dominant views of knowledge and associated methods that fit complicated issues to be at the heart of these critiques. The recognition of complexity also seems to be an important underpinning of various social transitions, including the transformation of education (Sterling, 2024).
If we really want to prepare students for the complexity of the issues they face, we need to make learning in complexity (e.g. in learning communities) part of the curricula of knowledge institutions. Formal education is then in line with the informal collective learning processes in learning communities. More specifically, students from different orientations (EQF levels) temporarily join an existing local situation in which different stakeholders are already working together.
This is a major shift in thinking and working, and requires a completely different approach to learning than we are used to (Netherlands Association of Universities of Applied Sciences, 2022). In fact, it means a transition in education itself (Sterling, 2021). However, it does offer the following benefits:
=> Study activities are more in line with the demand of practice and build better on activities of previous students;
=> Knowledge institutions are forced to develop learning (also) from the complex demand of practice;
=> The contribution of formal education (knowledge institutions) to local practice becomes clearer; students not only come to learn (the ‘practice as a means of the learning process’), they also contribute to research and solutions for local issues;
=> Students prepare for lifelong development from the learning community, in which they learn to appreciate a variety of backgrounds and (learning) orientations and are therefore better prepared for their role in society;
=> Practice gets to know knowledge institutions and formal learning processes as part of their informal, long-term and collective learning process. This could also open the way for the purchase of non-formal services and products from the knowledge institutions, such as courses and additional research.
THE VALUE OF A DEVELOPMENTAL CULTURE IN A TURBULENT WORLD
Back to the role of evaluation in which we now take complexity as a starting point. We often have expectations of an activity to be carried out, but we cannot yet determine its specific contribution and significance. Learning and development is therefore paramount.
The value of a collaborative learning process
Back to the role of evaluation in which we now take complexity as a starting point. We often have expectations of an activity to be carried out, but we cannot yet determine its specific contribution and significance. Learning and development is therefore paramount.
The power of a shared mission
Working on a transition is in the interest of all those involved. The aim is to involve them, to connect them and to generate energy on the formulation of a collectively desirable situation. We call this mission-driven work. Everyone is challenged to commit to the mission of the collective, beyond their individual interests (Mazucatto, 2019). In this way, the group of people makes a start with a developmental culture. The cohesive process of building a portfolio of activities and learning and navigating them with a view to the mission. A mission-driven way of working is suitable for the complex issues mentioned above, for which a new desirable situation in the future is sought over long periods of time, precisely because the desired long-term perspective is then central to the action. This also fits well with the agenda of Key Enabling Methodologies described by the top sectors, in which; ‘Through participation and co-creation in experimental environments, simple interventions and early prototypes, they can quickly be tried and experienced in the ‘real’ world’. Trying out and validating ideas (Clicknl, 2024). In addition, it must also be possible to test the effects of developed interventions on changes later in the long-term process and possibly make adjustments in monitoring and effect measurement (Clicknl, 2024).
Be open, navigate and try out
Within the collective, it is important to always navigate development in the direction of the mission and to nurture creativity along the way. In this way, the collective is constantly looking for and experimenting with possibilities. The dynamics that this brings about set the system in motion, it causes cracks. Opportunities and threats that were previously hidden are becoming visible. The trick is to be sensitive to this and to act actively (this is also called ‘sensemaking’). The starting points for those involved in the process are being open to innovation (novelty), daring to act and thus gaining experience (‘learning by doing’).
Systematic evaluation and agenda-setting
In order to systematically organize this collective learning process, a direct link to evaluation and reflection is desirable. Regular joint evaluation can be linked to a (learning) agenda so that a long-term monitoring process is created (see figure 3 in the full article).
Monitoring as a compass
When you use developmental evaluation periodically, short-cyclically, it always gives direction to the next steps to be taken. The evaluation is then not only an instrument for accountability afterwards, but it functions first and foremost as a compass with which all those involved keep the mission in mind along the way. In addition, insight is gained into the results of the planned activities, into the contribution of unplanned events and into the way in which they contribute to the joint mission. See Figure 4 (see full article).
Impact of lessons learned
The outcomes of this form of evaluation can take different forms, such as a concrete product or service, or a shared narrative with lessons learned and insights into working (or non-working) mechanisms.
With this form of monitoring, you set up a long-term ‘probing and sensing’ process, in which planned short-term activities and spontaneous events are always assessed for their value to the long-term mission. When monitoring takes place on a larger scale, it provides an overview and insight into the mutual contribution of short-term activities and their interdependence. The collective learning capacity can be increased in this way. The impact of initiatives will then be higher on the agenda; It prevents us from inventing the same wheels over and over again.
HOW CAN BOTH CULTURES RELATE TO EACH OTHER?
As can be seen from the previous description, there are many differences between the two evaluation cultures. Table 1 lists the most striking differences (see table in full article).
Mutual understanding and recognition of complementarity
Both evaluation cultures have their value and can reinforce each other in practice. The condition is that there is understanding and recognition of the value of both. This requires recognition of the existence of complexity and insight into its systemic significance, as well as recognition of the fact that issues can also be complicated within this complexity.
The evaluation of short-cycle planned activities benefits from an audit culture. Given the characteristics of a transition, this requires that it be embedded in a consideration of the complex system world and that an assessment is made of the contribution of the planned activities to the given mission of the developmental culture.
This brings us back to the image of walking on two legs when we work on and monitor complex issues. In a transition issue, this starts in a mission-oriented way and with an eye for the complexity of the issue. From there, the complicated issues are identified and addressed. In the monitoring, the task always remains to initiate the learning process in addition to regular audits in which the contribution of the complicated issue to the transition task is central. Due to the multitude of possible activities and events in these tasks, it is necessary to work methodically and systematically, so that the learning process is focused on the long term and the mission is always navigated. A learning evaluation and monitoring offers tools to do this. In another context, we have called this transition-oriented monitoring (Beenen et al, 2024).
TRANSITION-ORIENTATED MONITORING
In our complex society, which is characterised by enormous challenges, it is important to understand the effect of the prevailing audit culture on our actions. We are conditioned to operate within that culture. We like to think in terms of solutions and results, we often set up policy and organizational processes in short-cyclical terms. We like to think in terms of ‘projects’, because they are manageable. However, this means that essential power of change is lost. In this way, we treat issues as complicated problems and we ignore the complexity, uncertainty and interconnectedness. We see the recognition and use of precisely those characteristics of transitions as necessary to achieve systemic change.
Transition-orientated monitoring connects the audit culture of the stable, predictable world with the development-oriented culture of the turbulent world. From the idea of walking on two legs, this form of monitoring respects both cultures and builds on them. The starting point is a collective mission-driven learning process (Mazzucatu, 2018). This learning process is supported by evaluating results and by assessing the contribution (impact) of short-cyclical actions and initiatives to the long-term mission. It is a form of reflexive monitoring with a flexible range of activities, aimed at collecting and documenting results and lessons, and at anticipating and making action-oriented adjustments in an ever-changing societal context (Beers et al, 2019). Based on the evaluations, this action-oriented adjustment is also made collectively by aligning, prioritizing and formulating the agenda: an action-oriented agenda for both the short term and the long term.
DEMARCATION TO A MANAGABLE MONITOR
What do we have to develop?
We see the development of a good mix of evaluation cultures in the knowledge institutions supporting the transition of the knowledge institutions themselves. It is also conditional for their role in social transitions. The vast majority of knowledge institutions are currently failing to meet the urgent requirements for this role (Sterling, 2024). Embracing complex issues and developing appropriate methods are central to this. This starts, among other things, with leadership and a consistent paradigm shift.
Leadership in transitions
Learning evaluation requires leadership that looks beyond the short term and embraces complexity. This requires willingness and the ability to deal with the unknown and to relate to associated uncertainties and risks. Leadership also makes a moral appeal to the collective to look for value-driven alternatives. It is a choice to deny existing complex social transition issues, to let them take their course or to take them seriously and want to influence them positively. For those who choose the latter option, recognizing, understanding and living through complexity in transitions is a start. This also means as a leader skillfully dealing with all kinds of unfamiliar situations and being part of a larger whole as a wave surfer (see box 2). In many domains, this transition-oriented approach to social issues clashes with the prevailing conception of knowledge and the way in which we shape learning in education. In it, learning is mainly based on the transfer of a knowable world consisting of complicated issues. For a detailed view on this, we refer to an earlier article: https://ecolah.eu/leren-in-complexe-vraagstukken/
The leader of a hundred years ago was a lion king, who stood on top of a rock and told everyone what to do. That was in line with the spirit of the times and a worldview that is based on familiarity and predictability.
Today’s metaphor is the wave surfer, who moves effectively on forces that are stronger than himself. Together with others, and without being in charge of the sea and the waves (Peeters, 2016).
[Box 2]
Paradigm shift and readying
Thinking in terms of complexity, its consequences and the associated developmental evaluation is a true paradigm shift for most people. Often people do use the language, but they have difficulty acting consistently. In the words of Kuhn; “Paradigm shifts are a conversion experience that cannot be forced” (Kuhn, 1974). This process needs more attention to arrive at preparedness and in learning to think, act and be able to be in complexity. We call this a process of ‘readying’; a continuous development process that precedes a paradigm shift, supports the shift, makes it action-oriented and validates it as the new alternative. We emphasize that ‘readying’ is a verb (Bateson, 2022). A learning question is how we can facilitate ‘readying’ in both individual and collective learning within the complex transition-oriented issues. This is, in the aforementioned learning communities, a professionalization issue.
Paradigm shift and readying
Learning through evaluation and monitoring in knowledge institutes is still in its infancy and still requires a lot of practice and further research. If you are interested, you can contact us: onderzoeksgroep.tolch@org.hanze.nl
REFERENCES
See full article.
