I thought I’d try something different. I’m giving a paper at a conference before too long, and I thought I’d solicit feedback on it *before* I give it: I’ll write the thing in public. I am always such crap when it comes to properly formatting citations etc, and I have a mental block when it comes to words that sound alike… so please be gentle. Feedback in the comments, please.

This terrifies me, to some extent, but I watched a similar experiment unfold on Grand Text Auto a while back, which had excellent results. And so, I offer:

Rolling your Own: On Modding Commercial Games for Educational Goals

Shawn Graham – University of Manitoba, Grand Canyon University

Making modifications to existing commercial games is a strong and vibrant sub-culture in modern video gaming. Strictly speaking, ‘modding’ refers to actually changing the rules by which a game operates, but a less rigorous definition includes scenario building, or the set up of the pieces on the game board. Many publishers now provide tools to make this easier, as part of their marketing strategy. Talented individuals who make and release mods or scenarios for popular games such as the Civilization franchise have been been plucked from the fan communities to employment with the publishers (Jon Shafer, lead designer of the upcoming Civilization V, is one notable example).  Most scholars who have focused on Civilization have addressed its narrative of technological progress and American exceptionalism (REFS); others have concentrated on how the game can be employed in classroom settings, its anachronisms and theoretical outlook on history (Sid Meier famously stating that he did not set out to create a work of history, he wanted to create a game). In this piece, I wish to focus attention on the fansites as the locus for learning.

I too wanted to use Civilization for paedegogical ends in my online classroom. With the help of participants on civfanatics.com, I created a scenario (with one rule change; thus a mod) to address a problem I was having in my Introduction to Roman History class. I carefully crafted a scenario to reflect the events of AD 69, the Year of the Four Emperors, devised an assignment to go with it, and launched it on my students. It was a flop. Its lack of success I suspect is due to the ‘creepy treehouse’ phenomenon (Stein, 2008), referring to the urban legends surrounding treehouses built with no other purpose but to lure children. In online learning, the ‘creepy treehouse’ plays out as a use of some aspect of social media that does not emerge naturally from the class dynamic, but rather is imposed from on top and thus feels artificial to the participants – an instructor who requires every student to post 3 times a week to the class blog, for instance.

In this paper I explore just how my experiment with modding and scenario building ended with a ‘creepy treehouse’. That experience refocused my attention to the fansites themselves and the participants who helped me build my scenario. This points us as educators to an under appreciated value of game-based learning using commercial video games. When we ‘roll our own’, it is the aspect of creating it in public that might have the greatest educational impact. The nature of the fansites promotes the kind of learning we labor to facilitate in our online classrooms, spontaneously and from the bottom up: teaching without teachers.

The Year of the Four Emperors

The death of Nero launched the Empire into a period of turmoil and civil war, with four Emperors being declared in various parts of the Empire, in quick succession. My introduction to Roman history class were struggling with the period. Vespasian was the last of the four contenders to be declared Emperor by his troops. In looking at the period, my students were explaining Vespasian’s success in pacifying the Empire and consolidating his hold on Rome in terms of his later role as Emperor: “Of course Vespasian would win the civil war because Vespasian was the Emperor.” This is to put the cart before the horse. As I discussed the period with them, I realized that part of the problem, aside from confusion of cause and effect, was a poor understanding of the realities of Mediterranean geography and of the difficulties of communications in a pre-industrial world (factoring in the time it took for news to travel and how that influenced the political dynamic).

I wanted my students to understand the contingency of history, that Vespasian’s eventual triumph was not fore-ordained, and that physical and political geography played a role. Thus I embarked on the creation of a scenario, using Civilization IV. Civ IV comes with a piece of software for setting up scenarios, the ‘world builder’. I quickly became frustrated with using it. It is meant to allow the player to place all of the different pieces on the map, to set up the starting positions for the game. Many of its features are disabled, and cannot be unlocked until the player adds a line of code to the Civilization initialization file. This information is not provided by the game’s documentation, but rather comes from the fan sites. Trying to unlock the worldbuilder led me to the modding community (indeed my post relating what the unlock code is, is consistently the most visited post on my research blog).

As I became more and more excited about the possibilities of scenario building, I came to rely more and more on the fan sites for help (principally, www.civfanatics.com). Civilization IV was built using XML to describe nearly every object in the game. By adjusting the information in the XML, one can change the names of leaders or cultures (or add more); one can adjust the game calendar so that each turn represents a single day, week, or month. One can add ancillary information to set the scene for the scenario when it opens, or prevent certain kinds of technology from ever being ‘discovered’ (a world without gunpowder, for instance). How to find this information, how to change it, was all courtesy of the fans.

Eventually, with the help of ‘Carloquillo’, I had a working scenario of the Roman world in AD69. The aim was to outmanouevre the other claimants to the throne, whether politically or militarily (the ‘Senate’ would examine the balance of power in Italy periodically, and declare one or the other of the rivals to be the ‘Emperor’ – thus simulating the ineffectualness of the Senate at this period). The scenario was not perfect of course – Vespasian kept converting to Judaism if the artificial intelligence was allowed to play as him. I devised an assessment exercise for my online students. Instead of writing the final paper, I would instead have the students play the scenario through. At set intervals, they would take a screenshot of the world map, and record a narrative of what was going on in their counterfactual history (they would be its historians). Then, to conclude they would identify and address the similarities and differences between the game’s version of ‘history’ with what had in fact happened in the past (which would make a virtue of Vespasian’s conversion to Judaism, for instance). My hope was that in playing the scenario the students would begin to appreciate the difficulty of Vespasian’s position, initially; how difficult it was to act; what an accomplishment it was in fact to manage and control such an enormous hetergeneous territory; and by identifying anachronisms better understand the important concepts of the period.

This is the point where the scenario failed. A number of my online students did indeed have copies of Civilization IV. I offered the scenario to these students as an alternative to the final assignment, confirmed that some of them were playing it, and waited to see what would happen. While feedback on the scenario was positive – “this was a fun scenario, sir” – to a person, none took up the offer to play the game for credit, turning in standard essays instead. I asked why, and every response was evasive. I initially put it down to the conservatism of students: everyone understands how essays work as far as grading goes, but maybe a game-for-credit was a step to far. I have not tried this scenario again in a classroom setting because I now realize that the major error I made was that I sprung it on my students without any kind of preparation. There was no buy-in, because it was a ‘creepy treehouse’. I selected the period to model; I had made it; it was my representation. Of course there could be no buy-in.

But there was one mitigating factor. The thread I started on Civfanatics asking for help attracted the attention of 14 other players (very nearly the same size as my class). They helped me to build it, they asked questions about the period, and they suggested ways of accomplishing what I wished to model using the game. The scenario that I uploaded to that thread was play-tested by them, and has since been downloaded nearly 1000 times. On the Civfanatics site, my role as a university instructor did not put me in any privileged position vis-a-vis the other participants; I was just one of many people who enjoyed the game.

Learning did happen as a result of my experiment in scenario building. It just didn’t happen in my classroom.

Assessing the educational value of online discussion forums

The major learning management systems used by colleges and universities rely on a twenty year old metaphor: the bulletin board, or discussion forum. Students make posts, leaving messages commenting on some topic. Posts are organized into threads (thus mixing metaphors). Similarly, Civfanatics relies on posts and threads. Significantly, online courses rely on the instructor to keep the discussion flowing, to push it into the interesting areas, and to assess the students’ learning in the forums. While Civfanatics has ‘moderators’ who monitor the discussions, their role is solely to make sure that topics are in the right place – don’t post your wish list of features for Civilization IV in the area marked for scenario swapping, for instance. There is therefore no authority ‘in charge’ of any discussion on Civfanatics. What order there is is in a given thread is largely self-organized. The literature of formal online learning can usefully be explored to assess what kind of learning is taking place in these self-organized forums. In the thread that I started, I clearly learned about how to simulate using the game. But what of history?

In the classes that I teach, when I assess a discussion forum, I am looking for posts that demonstrate an understanding of the material; that engage with others’ thoughts and comments; and which push the conversation forward. In truth, my rubric is not overly elaborate. A more rigorous rubric and approach is proposed by Uzuner (2007). Uzuner makes a distinction between ‘educationally valuable talk’ (EVT) and ‘educationally less valuable talk’ (ELVT). He situates this distinction in the traditions of Vygotsky’s 1934 insights concerning language and how “knowledge building is created between/among people in their collaborative meaning-making through dialogue.” Uzuner’s approach therefore is firmly rooted in a constructivist approach to education. Uzuner suggests that EVT, in the context of discussion threads, is

a particular interactional pattern in online discussion threads characterized as dialogic exchanges whereby participants collaboratively display constructive, and at times, critical engagement with the ideas or key concepts that make up the topic of an online discussion, and build knowledge through reasoning, articulation, creativity, and reflection. (2007)

On the other hand, ELVT is talk “that lacks substance in regards to critical and meaningful engagement with the formal content or ideas that are discussed in the posts of others in an online discussion” (2007).  Uzuner then provides examples of different kinds of EVT and ELVT, with 11 different kinds of EVT, and 5 kinds of ELVT. I reproduce Uzuner’s two tables below:

Table 1. Online Conversations and Educationally Valuable Talk (EVT) Indicators

Indicator	Acronym	Defined	Examples	Source
Exploratory	EPL	Recognition of some confusion/curiosity or perplexity as a result of a problem/issue arising out of an experience/course readings; posing a problem and enticing others to take a step deeper into it.	“I wonder…….” “I am not sure if what the author suggests…….” “In the article X, the author said …. This brought up a few questions in my mind ….”	Mercer (1994); Garrison, Anderson & Archer (2001)
Invitational	INVT	Inviting others to think together, to ponder, to engage by asking questions, requiring information, opinion or approval.	“Jane says …….. What do you think?” “Do you think ……?” “The authors suggest …., no?	Uzuner & Mehta (2007)
Argumentational	ARG	Expressing reasoning (with analogies, causal, inductive and/or deductive reasoning etc) to trigger discussion	“If teachers ……., then ……..” “Teaching is like …………..” “X is important because …….”	Kumpulainen (1996)
Critical	CRT	Challenging or counter-challenging statements/ideas proposed by others OR playing devil’s advocate	“I agree that …. However, …….”	Uzuner & Mehta (2007)
Heuristic	HE	Expressing discovery (similar to “A ha!” moments or expressions like “I find it!”); directing others’ attention to a newly discovered idea.	“I did not know that there is a name for XXX. I think XXX is …..Has anyone experienced that too?”	Kumpulainen (1996)
Reflective	REF	Examination of past events, practices (why/how they happened) or understandings in relation to formal content	“I’ve noticed that I had a tendency to ….. After reading X’s article, I’ve learned not to ……”	Uzuner & Mehta (2007)
Interpretive	INTP	Interpretation of formal content through opinions that are supported by relevant examples, facts, or evidence.	“In my opinion X is …… Y is a good example of why …….”	Uzuner & Mehta (2007)
Analytical	ANL	Interpretation of content through the analysis, synthesis, and evaluation of others’ understanding	“The original question was … Joe said … Mary said … As for me ….”	Uzuner & Mehta (2007)
Informative	INF	Providing information from literature and relating it to course content/topic of discussion	“I read an article about X once and the author said …. You can find more information about this in …”	Kumpulainen (1996)
Explanatory	EXPL	Chain of connected messages intended to explain/make clear OR statements serving to elaborate on the ideas suggested in previous posts	“I want to build on your comment that ……..”	Uzuner & Mehta (2007)
Implicative	IMP	Assertions that call for action OR statements whereby participants formulate a proposal/decision about how to achieve a certain end based on the insights they gained from the course readings/discussions	“Teachers should / should not ….” “X must not be forced ….”	Uzuner & Mehta (2007)

Table 2. Online Conversations and Educationally Less Valuable Talk (ELVT) Indicators

Indicator	Acronym	Defined	Examples	Source
Affective	AF AA ASP	Short posts that ONLY contain a statement of personal feelings (likes & dislikes) Short posts that ONLY contain appraisal (praising & thanking someone) Questions or comments that add social presence to the discussion but do not contribute new information.	“I never liked Math either” “Thank you for offering your insights into ….” “I have been to your country once and I visited X, Y, Z when I was there”	Garrison, Anderson & Archer (2000)
Judgmental	JA JDA	Short posts that ONLY contain brief statements of agreement without elaboration Short posts that ONLY contain brief statements of disagreement without elaboration.	“Yes, I agree with you ….” “I do not think so”	Kumpulainen (1996)
Experiential	EXP	Posts that only contain personal experiences, narratives, descriptions that are not followed by reflection	“I did the same thing when I was teaching X. “I did A, B, C. It was fun”	Kumpulainen (1996)
Reproductional	REP	Repeating/reproducing the ideas mentioned/proposed in the previous posts without elaboration	“You are right, X is …… “ (followed by a sentence)	Kumpulainen (1996)
Miscellaneous	MIS	Opinions that seem to be off topic OR statements regarding technical problems/course logistics	“I am unable to open Jay’s file…”	Uzuner & Mehta (2007)

Uzuner’s schema thus provides a route for understanding the educational potential of the discussion forums on Civfanatics. I therefore assessed the posts in the most-viewed scenario in the Civfanatics.com Civilization IV – Scenarios forum, John Shafer’s WWI Scenario.[1] This scenario was first posted on May 6^th, 2006. To date, it has been viewed over 94 000 times; the most recent post was on January 19^th, 2009. There are 311 posts in this thread. I read each post, and tallied the kinds of educationally valuable or less valuable talk that was occurring.

Table 3. Educationally Valuable Talk in Shafer’s WWI Scenario thread

Kinds of Valuable Talk	# of instances
EPL	10
INVT	31
ARG	28
CRT	22
HE	3
REF	13
INTP	2
ANL	8
INF	18
EXPL	0
IMP	2

Table 4. Educationally LessValuable Talk in Shafer’s WWI Scenario thread

Kinds of Less Valuable Talk	# of instances
AF	2
AA	0
ASP	79
JA	11
JDA	11
EXP	14
REP	14
MIS	184

A straight tally would suggest that the ‘less educationally valuable talk’ carries the day, with 315 posts to ‘educationally valuable talk’s 137. But this misses some important dynamics. The ‘miscellaneous’ category captures two distinct kinds of posts – ‘how do I install this scenario / it didn’t work’ queries, and more complex play-throughs of the scenario, reporting what exactly took place. These latter posts are actually quite valuable, in that since the scenario is a kind of simulation, each play-through records a different kind of trajectory through all of the possible outcomes of the scenario. It’s a kind of sweeping of the scenario-as-simulation’s ‘behavior space’ (cf Graham 2009) and so provides important fodder for other kinds of educationally valuable talk.

The development of the forum follows a distinct trajectory. Shafer introduces the forum on May 6th. A flurry of appreciative posts and ‘how do I…’ technical queries ensues for about 50 posts, followed by a second phase of play-testing and reporting of bugs. Educationally valuable talk picks up in this second phase as various individuals pick up on items in the play-throughs. By post 79, the conversation has turned to how to best represent the carnage, social, and strategic impact of trench warfare given the procedural rhetorics (though not framed in those terms) of the game. This phase continues for around another 100 posts, and includes discussions on the real world impact of the Russian Revolution on the War, and how this should best be simulated. There is a strong concern throughout these posts for verisimilitude and ‘authenticity’ – but what constitutes authenticity is debated. A flame war erupts in post 92 on this very question, and is eventually quelled by Shafer who notes that this is just a game and is meant to be engaging. In post 103, another individual suggests modifications to the scenario, and actually begins another thread elsewhere on Civfanatics to improve and expand on Shafer’s work. In posts 171, the author uses the scenario to leap into counterfactual history, and propose quite a complex counterfactual based on his play throughs of the scenario. By September of 2006 most of the heat has gone out of the thread, and most subsequent posts are again of the ‘how do I make this work’ or the play-through variety. This continues until the thread goes dormant in January of 2009.

The other aspect that needs to be considered, to give fullness to Uzuner’s approach and Vygotsky’s insight, is the social aspect. Who is talking to whom? I mapped out the pattern of social interactions in the forum as a kind of network. If ‘DoctorG’ addressed ‘JLocke’, then I connected the two individuals. If ‘KobatheDread’ posted a note recounting a play-through, I mapped that as a response to Shafer’s original post (since everything posted is public, in a sense, every individual is connected to every other individual, and so for the sake of analysis and simplicity, can be disregarded from the network). If Shafer responded to Koba quoting JLocke, I connected all three together. The resulting network is more-or-less star shaped, with Shafer in the middle and everyone else radiating off as spokes. However, there are clumps of highly interconnected individuals representing sub-conversations and discussions that developed in the forum.

Figure 1: Conversation in the WWI Scenario Thread as a Social Network. Shafer is in the exact center. There are 59 individuals.

[network diagram of pattern of social interaction in the threads; mostly starshaped, but interesting cluster of connections spins out of it]

Using the Keyplayer program from Analytech (Borgatti, 2008), I assessed the most central individuals in this network; that is, the individuals whose removal from the forum would result in a disrupted graph, or would ‘break’ the conversation. Keyplayer reported that the removal of Shafer, Jlocke, Dom Pedro, Kitten of Chaos, and Koba the Dread would cause this network to fragment almost completely. These individuals between them account for a majority of the educationally valuable posts made in the forum. This is quite interesting from the standpoint of an online educator, in that it suggests that we can determine from structure alone the individuals who are making the greatest contribution to the learning going on in a forum.

This was a forum without an official leader, or any one acting in the role of ‘teacher’. The contrast with my own Year of the Four Emperors thread is striking.[2] My thread began on May 16 2006, and went stagnant by September. Fourteen individuals contributed, and noticeably, aside from my own initial post, there is a large absence of EVT, unless you count the technical ‘how-to’ posts I made, and the play-through reports. As a social network, the graph is entirely centered around me, in a star shape. Why the difference? I think I once again created a ‘creepy treehouse’. It was all about me. I was also very upfront about my identity and the use I wished to put the scenario, which made it more of a curiosity than a scenario that got people excited.

Rolling your Own: Lessons Learned?

If we are going to ‘roll our own’ scenarios or otherwise use commercial video games like Civilization in our teaching, we need to approach it more from the point of view of a fan, than as a teaching professional. Otherwise, we create artifacts that do not create the kind of response that we wish. Learning is obviously going on in the fan forums, and using tools like Uzuner’s typology is one way of assessing what kind of learning is happening. The pattern of social interaction, and the evolution of those discussions are also extremely important it would seem. One would need to study a much greater number of the threads to see the fuller picture – this is an area where text mining could be usefully employed. Perhaps we can emulate the way these discussions tend to evolve, and foster game-based learning in our classrooms that way. As an example to the online educational field, the idea that structure might be correlated with educational impact (and thus could be measured automatically) is intriguing, and needs to be explored further.

Borgatti, Steve (2008) Keyplayer http://www.analytictech.com/keyplayer/keyplayer.htm

Garrison, R., Anderson, T., & Archer, W. (2000). Critical Inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2 (2-3), 87-105.

Graham, Shawn (2007) ‘Vespasian, Civ IV, and Intro to Roman Culture’. http://planetcivilization.gamespy.com/View.php?view=Articles.Detail&id=33

Graham, Shawn (2007) “Re-Playing History: The Year of the Four Emperors and Civilization IV” Case Study, Higher Education Academy Subject Centre for History, Classics and Archaeology, United Kingdom.http://www.heacademy.ac.uk/hca/resources/detail/re_playing_history

Graham, Shawn (2009) Behaviour Space: Simulating Roman Social Life and Civil Violence. Digital Studies / Le Champ NuméRique, 1(2). Retrieved January 25, 2010, from http://www.digitalstudies.org/ojs/index.php/digital_studies/article/view/172/214

Kumpulainen, K. (1996). The nature of peer interaction in the social context created by the use of word processors. Learning and Instruction, 6(3), 243-261.

Mercer, N. (1994). The quality of talk in children’s joint activity at the computer. Journal of Computer Assisted Learning, 10, 24-32.

Stein, Jared (2008) ‘Defining the Creepy Treehouse’ http://flexknowlogy.learningfield.org/2008/04/09/defining-creepy-tree-house/

Uzuner, S. (2007) Educationally Valuable Talk: A New Concept for Determining the
Quality of Online Conversations Journal of Online Learning and Teaching 3.4 http://jolt.merlot.org/vol3no4/uzuner.htm

Uzuner, S. & Mehta, R. (2007, August) Aiming for educationally talk in online discussions. Paper presented at the Multimedia Educational Resource for Learning and Online Teaching – MERLOT Seventh International Conference, Sheraton New Orleans Hotel, New Orleans, Louisiana.

Vygotsky, L.S. (1962). Thought and language. Cambridge, MA: MIT Press. (Original work published in 1934).

---

[1] http://forums.civfanatics.com/showthread.php?t=170090

[2] http://forums.civfanatics.com/showthread.php?t=171164