Matthew Easterday
Cognitive games for teaching policy deliberation
My work on policy argument has made several contributions toward scientifically-based instruction for policy deliberation by: (a) defining a cognitive framework for policy deliberation, (b) showing that although confirmation bias often prevents students from recognizing when evidence contradicts their policy beliefs, causal diagrams can help students to make evidence-based arguments, (c) developing intelligent cognitive tutoring software that teaches policy deliberation, and (d) showing how embedding this tutor into a game affects motivation while maintaining efficiency in learning.
A cognitive framework for deliberation
Engaged citizenship requires a host of skills including critical thinking, speaking and debating, and organizing. With respect to policy deliberation problems, critical thinking can be thought of as a process of posing questions, searching for information, comprehending information, synthesizing information into causal models of the policy domain, and deciding upon recommendations. This process can be augmented with diagrams to improve reasoning in the same way that equations improve reasoning about algebra problems. Much of my work focuses on identifying and developing educational technology to teach the scores of cognitive sub skills required for to perform each of the steps in this process.
The deliberation framework describes policy deliberation as a process in which the citizen: (a) identifies a focus question such as "what should we do about childhood obesity?" (b) searches for raw information about that question, such as a report on the effects of junk food advertising on obesity, (c) comprehends and evaluates that information into some schematized mental representation, (d) constructs an external representation of the information such as a causal diagram, (e) synthesizes the new information with their preexisting knowledge, and finally (f) uses the external representation to decide upon a policy recommendation.
Bias in search and analysis of evidence
Where does bias occur during the policy deliberation process? In Easterday, Aleven, Scheines & Carver (2009b), undergraduates played a computer game in which they assumed the role of policy analysts who had to determine whether four different policies: reducing class size, increasing teacher qualifications, increased funding or providing vouchers, would increase school performance. Unbeknownst to the students, half the evidence mostly supported one of their policy beliefs, e.g., about class size, and half the evidence mostly undermined one of their policy beliefs, e.g. about teacher qualifications.
I found, unsurprisingly, that students' recommendations are biased by their prior beliefs, but not necessarily in the way predicted by the literature. Measurements of students' beliefs showed that students change their beliefs in response to particular reports in a roughly rational manner, increasing their confidence after reading a confirming report and decreasing their confidence after reading a disconfirming report. The problem is not so much that students discount disconfirming evidence as that they start with extremely high confidence in their initial beliefs. Furthermore, students do not maintain an accurate picture of the evidence read, so when asked to recall the evidence read, students answers merely rationalize their final position.
With respect to the theoretical framework, the study suggests that students do not have as much difficulty with search, comprehension and evaluation as they do with synthesis. More precisely, students' synthesized causal models do not include an accurate mental representation of the evidence making them susceptible to bias. This result suggests a possible need for external representations of the evidence.
Diagrams improve reasoning
Can diagrams improve reasoning? Easterday, Aleven, Scheines & Carver (2009a) examined whether causal diagrams help students coordinate a more complex set of causal claims.
Students were given a set of causal claims from multiple sources about a policy problem and asked them to determine which policy should be pursued, assuming that a given set of sources are credible. Some students received the claims as text, while other students received the claims as text plus a causal diagram, and other students received the claims as text with a diagram tool with which to construct their own diagrams.
This study found that causal diagrams do indeed improve student's ability to make evidence-based policy recommendations and also that practice constructing diagrams improves students' evidence comprehension skills. In other words, causal diagrams can be used as an equations language for policy reasoning. This study also revealed that students had great difficulty constructing accurate diagrams themselves.
A follow-up protocol study uncovered ways in which confirmation bias affects students' and experts' use of diagrams by comparing how several experts and novices analyze causal evidence. I found that reasoners are heavily influenced by background knowledge in both normative ways, such as when experts overrule information based on superior prior knowledge, and non-normative ways, such as when novices reinterpret the diagrams in a biased way to support their prior beliefs.
Policy World: A Cognitive game for teaching deliberation
Taken together, the framework and experimental results show that confirmation bias does indeed hinder students' synthesis of evidence and that while diagrams can help students coordinate multiple policy claims, diagram construction proves difficult. These findings suggest that a computer tutor for policy reasoning should focus on the construction of external representations to synthesize evidence.
Policy World is cognitive game (intelligent tutor + video game) that teaches students how to deliberate, that is, use evidence from multiple conflicting sources to make policy decisions, primarily by constructing causal diagrams (Easterday, in press; 2010).
A randomized controlled trial of Policy World showed that the more tutor-like versions of policy world (that gave more feedback and fewer penalties) increased learning and interest compared to a more game-like versions with less feedback and harsher penalties. (Easterday, Aleven, Scheines & Carver, 2011). This suggests not only that game-designers might need to change how they design games in order to improve learning, but even more importantly, we can increase learning without sacrificing fun.