Still playing with historical maps into minecraft

I managed to get my map of the zone between the Hogs’ back falls and Dow’s Lake (nee Swamp) into Minecraft. I completely screwed up the elevations though, so it’s a pretty ….interesting… landscape. I’ve trying again with a map of Lowertown, coupled with elevation data from a modern map. This clearly isn’t ideal, as the topography of the area has changed a lot with 150 years of urbanism. But it’s the best I have handy. Anyway, it’s nearly been working for me.

Nearly.

So I provide to you the elevation and features for your own enjoyment, see if you can make ‘em run with the generate_map.py script. If you get ‘key errors’, try editing the features file in Paint, make sure the blocks of colour are not fuzzy on the edges.

https://dl.dropboxusercontent.com/u/37716296/byward-market/market-maps.zip

 

 

Historical Maps into Minecraft

dowslakemap1847

Dow’s Lake area, settlement by 1847 Map Source: Bruce Elliott, Nepean, The City Beyond, page 23, posted on http://www.bytown.net/dowslake.htm

The folks over at the New York Public Library published an excellent & comprehensive tutorial for digitizing historical maps, and then importing them into Minecraft.

First: thank you!

Unfortunately, for me, it’s not working. I document here what I’ve been doing and ideally someone far more clever than me will figure out what needs to happen…

The first parts of the tutorial – working with QGIS & Inkscape – go very well (although there might be a problem with colours, but more on that anon). Let’s look at the python script for combining the elevation map (generated from QGIS) with the blocks map (generated from Inkscape). Oh, you also need to install imagemagick, which you then run from the command line, to convert SVG to TIF.

“The script for generating the worlds uses PIL to load the TIFF bitmaps into memory, and pymclevel to generate a Minecraft worlds, one block at a time. It’s run successfully on both Mac OS X and Linux.”

After digitizing, looks like this.

After digitizing, looks like this.

I’ve tried both Mac and Linux, with python installed, and PIL, and pymclevel. No joy (for the same reasons as for Windows, detailed below). Like most things computational, there are dependencies that we only uncover quite by accident…

Anyway, when you’ve got python installed on Windows, you can just type the python file name at the command prompt and you’re off. So I download pymclevel, unzip it, open a command prompt in that folder (shift + right click, ‘open command prompt here’), and type ‘setup.py’. Error message. Turns out, I need setuptools. Which I obtain from:

https://pypi.python.org/pypi/setuptools#windows-7-or-graphical-install

Download, install. Works. Ok, back to the pymclevel folder, setup.py, and new error message. Looks like I need something called ‘cython’.

http://cython.org/#download

I download, unzip, go to that folder, setup.py. Problem. Some file called ‘vcvarsall.bat’ is needed. Solution? Turns out I need to donwload Microsoft Visual Studio 10. Then, I needed to create an environment variable called ‘vs90comntools’, which I did by typing this at the command prompt:

set VS90COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\

Wunderbar. I go back to the pymclevel folder, I run setup.py again, and hooray! It installs. I had PIL installed from a previous foray into things pythonesque, so at least I didn’t have to fight with that again.

I copy the generate_map.py script into notepad++, change the file names within it (so that it finds my own elevation.tif and features.tif files, which are called hogs-elevation.tif and hogs-features.tif; the area I’m looking at is the Hogsback Falls section of the Rideau. In the script, just change ‘fort-washington’ to ‘hogs’ or whatever your files are called). In my folder, at the command prompt, I type generate_map.py and get a whole bunch of error messages: various ‘yaml’ files can’t be found.

Did I mention PyYaml has to be installed? Fortunately, it has a windows installer.  Oh, and by the way – PyWin is also needed; I got that error message at one point (something obscure about win32api), and downloading/installing from here solved it: http://sourceforge.net/projects/pywin32/files/pywin32/

Ok, so where were we? Right, missing yaml files, like ‘minecraft.yaml’ and ‘classic.yaml’, and ‘indev.yaml’ and ‘pocket.yaml’. These files were there in the original repository, but for whatever, they didn’t install into the pymclevel that now lives in the Python directory. So I went to the pymclevel repo on github, copied-and-pasted the code into new documents in notepad++, saved them as thus:

c:\Python27\Lib\site-packages\pymclevel-0.1-py2.7-win32.egg\pymclevel\minecraft.yaml

Phew. Back to where I was working on my maps, and have my generate_map.py, which I duly enter and…. error. can’t find ‘tree import Tree, treeObjs’.  Googling around to solve this is a fool’s errand: ‘tree’ is such a common word, concept in programming that I just can’t figure out what’s going on here. So I turned that line off with a # in the code. Run it again…. and it seems to work (but is this the key glitch that kills all that follows?).

(update: as Jonathan Goodwin points out, ‘tree.py’ is there, in the NYPL repo

…so I uncommented out the line in generate_map.py, saved tree.py in the same directory, and ran the script again. Everything that follows still happens. So perhaps there’s something screwed-up with my map itself.)

The script tells me I need to tell it whether I’m creating a creative mode map or a survival mode:

so for creative mode: c:>generate_map.py map

for survival: c:>generate_map.py game

And it chugs along. All is good with the world. Then: error message. KeyError: 255 in line 241, block_id, block_data, depth = block_id_lookup[block_id]. This is the bit of code that tells the script how to map minecraft blocks to the colour scheme I used in Inkcraft to paint the information from the map into my features.tif. Thing is, I never used RGB R value of 255. Where’s it getting this from? I go back over my drawing, inspecting each element, trying to figure it out. All seems good with the drawing. So I just add this line to the code in the table:

block_id_lookup = {

[..existing code...]

255 : (m.Water.ID, 0, 1),

}

And run it again. Now it’s 254. And then 253. Then 249. 246. 244. 241. Now 238.

And which point, I say piss on this, and I provide you with my features tif and elevation tif and if you can please tell me what I’m doing wrong, I’d be ever so appreciative (and here’s the svg with the drawing layers, for good measure).

….when I first saw the tutorial from the NYPL, I figured, hey! I could use this with my students! I think not, at least, not yet.

(update 2: have downloaded the original map tifs that the NYPL folks used, and am running the script on them. So far, so good: which shows that, once all this stuff is installed, that it’s my maps that are the problem. This is good to know!)

Part Two:

(updated about 30 minutes after initial post) So after some to-and-fro on Twitter, we’ve got the tree.py problem sorted out. Thinking that it’s the maps where the problem is, I’ve opened the original Fort Washinton features.tif in MS Paint (which is really an underappreciated piece of software). I’ve zoomed in on some of the features, and compared the edges with my own map (similarly opened and zoomed upon). In my map, there are extremely faint colour differentations/gradations where blocks of colour meet. This, I think, is what has gone wrong. So, back to Inkscape I go…

Update the Third: looks like I made (another) silly error – big strip of white on the left hand side of my features.tif. So I’ve stripped that out. But I can’t seem to suss the pixel antialiasing issue. Grrrrr! Am now adding all of the pixels into the dictionary, thus:

lock_id_lookup = {
0 : (m.Grass.ID, None, 2),
10 : (m.Dirt.ID, 1, 1), # blockData 1 == grass can’t spread
11 : (m.Dirt.ID, 1, 1), # blockData 1 == grass can’t spread
12 : (m.Dirt.ID, 1, 1), # blockData 1 == grass can’t spread
14 : (m.Dirt.ID, 1, 1), # blockData 1 == grass can’t spread
16 : (m.Grass.ID, None, 2),
20 : (m.Grass.ID, None, 2),
30 : (m.Cobblestone.ID, None, 1),
40 : (m.StoneBricks.ID, None, 3),
200 : (m.Water.ID, 0, 2), # blockData 0 == normal state of water
210 : (m.WaterActive.ID, 0, 1),
220 : (m.Water.ID, 0, 1),
49 : (m.StoneBricks.ID, None, 3),
43 : (m.StoneBricks.ID, None, 3),
}

…there’s probably a far more elegant way of dealing with this. Rounding? Range lookup? I’m not v. python-able…

Update, 2.20pm: Ok. I can run the script on the Fort Washington maps and end up with a playable map (yay!). But my own maps continue to contain pixels of colours the script doesn’t want to play with. I suppose I could just add 255 lines worth, as above, but that seems silly. The imagemagick command, I’m told, works fine on a mac, but doesn’t seem to achieve anything on my PC. So something to look into (and perhaps try this http://www.graphicsmagick.org/ instead). In the meantime, I’ve opened the Fort Washington map in good ol’ Paint, grabbing snippets of the colours to paste into my own map (also open in Paint). Then, I use Paint’s tools to clean up the colour gradients at the edges on my map. In essence, I trace the outlines.

Then, I save, run the script and…… success!

I have a folder with everything I need (and you can have it, too.) I move it to

C:\Users\[me]\AppData\Roaming\.minecraft\saves and fire up the game:

Rideau River in Minecraft!

Rideau River in Minecraft!

Does it actually look like the Hogs’ Back to Dow’s Lake section of the Rideau Canal and the Rideau River? Well, not quite. Some issues with my basic elevation points. But – BUT! – the workflow works! So now to find some better maps and to start again…

Desert Island Archaeologies

You’ve been castaway on an uncharted desert isle… but friendly dolphins deposit a steamer trunk full of books on the shore to keep you occupied, the exact ten you’d pick. Thus the premise of Lorna Richardson’s new public archaeology project: Desert Island Archaeologies. Turns out, I was the first castaway. You can read my ten picks alongside those of other castaways, or just keep reading here.

[... the sun beats down...]

Damn steamer trunks. Can’t lift it. All these archaeology books! What those dolphins must be eating, I ask you!

Let’s see. Ah. Here we go. Goodness: the exact ten books I would want to be reading. First up: Ray Laurence, Roman Pompeii: Space and Society, 1994. This was the book that convinced me to go to grad school – we had a whole seminar built on it in my final year, back in ’96. It was unlike anything else I was reading as an undergraduate, and showed me that there were ways of looking at something as well-trod as Pompeii that were completely askew of what I’d come to expect. The geek in me loved the space-syntax, the way of reading street life. Hell, it was fun!

Next,Stephen Shennan, Genes, Memes and Human History – Darwinian Archaeology and Cultural Evolution (2002). By the time I came across this, I was getting very much into complex systems and simulation, and this was something that helped me make sense of what I was doing. And it’s a fun read. Oh look, here’s Amanda Claridge’s Rome: An Oxford Archaeological Guide‘ (1998). I hear Amanda’s dry wit every time I open this thing. This was my constant companion on my first trip to Rome. I can’t imagine going there without it.

If I ever get off this island.

What else, what else… It’s interesting how nostalgic I am about these items. Each one seems tied to a particular chapter of my life. Matthew Johnson’s ‘Archaeological Theory‘ (1999) still makes me laugh and provides guidance through the thorny thickets of theory. Sybille Haynes’ ‘Etruscan Civilization‘ is a treat for sore eyes, filled with the beauty and magic of that people. I expect it can also be used for self-defence, in case of wild animal attack on this island. I used it for the first class I ever taught, at the school of continuing education at Reading.

Harry Evans, ‘Water Distribution in Ancient Rome‘ (1997) reminds me of adventures through the Roman countryside on a dangerously lunatic vespa, trying to identify the standing ruins, with A. Trevor Hodge’s ‘Roman Aqueducts and Water Supply (1992) in the other hand. Hodge’s book was as a bible for me writing my MA; I had the opportunity to meet Hodge at Carleton University shortly after I started working there. Sadly, a trivial over-long meeting prevented that from happening. Hodge died later that week. I will regret that always.

Back to Ray Laurence. The man has had a profound impact on me as a scholar. His ‘Roads of Roman Italy: Mobility and Cultural Change‘ (1999) and all that space-economy stuff: fantastic! Totally connected with the ORBIS simulation of the Roman world by Meeks and Scheidel, by the way, in terms of how it changes our perspective on the Roman world (ORBIS isn’t a book, but maybe there’s a tablet in this steamer trunk somewhere?) In the intro to Roads of Roman Italy, Laurence mentions my name, which was the first time I’d seen my name in print, in an academic context. A real thrill! No less of a thrill than how I came to be mentioned in the first place: driving the British School at Rome’s death-trap ducato for Ray as we explored the remains of the Roman roads in the outskirts of town. If there is no tablet in this steamer trunk (with wifi provided by an unseen Google blimp, obviously), I think the ‘Baths of Caracalla‘ by Janet DeLaine (1997) might be buried down here somewhere… ah, here it is. When I first pitched my MA idea to Janet, she kept finishing my sentences. I wanted to do a quanity survey of the Roman aqueducts. Turned out, she was waaaaay ahead of me. She let me use the manuscript to this as I puttered away on the Aqua Claudia and the Anio Novus. It’s actually quite a fun read, especially when you start thinking about nuts-and-bolts type questions like, how the hell did they build this damned thing anyway?

Final book? It’s not archaeological, but it’s a good read. Complexity: A Guided Tour‘ by Melanie Mitchell, 2011. I’m quite into simulation and games, and the emergent behaviours of both ai and humans when they conspire together to create (ancient) history (as distinct from the past). That’s a whole lot of interdisciplinariness, so this volume by Mitchell always provides clarity and illumination.

So… that’s what I’ve found in this steamer trunk. The bibliographic biography of a digital archaeologist. Neat!

 

Shouting into the Void?

Carleton has an annual ‘academic retreat’, which is happening this weekend. I’m not sure what, precisely, occurs there, but I’ve been asked to talk about things digital/history/archaeological. In the wake of the recent SAA #blogarch session, and in advance of the upcoming special issue in Internet Archaeology on blogging archaeology, I thought I’d talk about one aspect of what I found when I set out to map the shape of ‘roman archaeology’ on the web.  It’s an update to what I did in 2011, for the #blogarch session at that year’s SAAs (you can read what I thought then here.)

I give you, ‘Shouting into the Void? Social Media and the Construction of Cultural Heritage Knowledge Online’

The Web of Authors for Wikipedia’s Archaeology Page

I’m playing with a new toy, WikiImporter, which allows me to download the network of authorship on media-wiki powered sites. I fired it up, set it to grab the user-article network and “The Hyperlink Coauthorship network will analyze all the links found in the seed article and create an edge between each user that edited the article found in that link and the article”.

Naturally, I pointed it at ‘archaeology’ on Wikipedia.  I’ve posted the resulting two mode network on figshare for all and sundry to analyze.

I also asked it to download the article to article links (which is slightly different than my spidering results, as my spiders also included the wiki pages themselves, like the ‘this page is a stub’ or ‘this page needs citations’, which gives me an interesting perspective on the quality of the articles. More on that another day). This file is also on figshare here.

Just remember to cite the files. Enjoy!

 

Bless your little cotton socks: reflecting on Carleton’s Data Day

I went to Carleton’s ‘Big Data Day’ yesterday. My student, Hollis, had entered a poster in the poster competition, detailing his approach to data mining the audio recordings of THATCamp Accessibility. Looks like he and I were the only two from the humanities end of the spectrum at the event. The day opened with a panel discussion, then a series of presentations from faculty involved in Carleton’s new MA in Data Science (which it appears has support from IBM; there’s a webpage at Carleton but it has a warning on it to all and sundry not to share the link. So go to carleton.ca and do this search). Carleton has aspirations in this regard. The program, and the event, was a collaboration from Computer Science, Biology, Business, Systems and Computer Engineering, Economics, Geography and Environmental Studies.

(And of course, not to be confused with our MA in Digital Humanities, which does not have support from a major tech firm).

Hollis explains his research at Carleton’s Data Day

Anyway, some observations on what this particular approach to big data seems to be about:

1. ‘Big data’ is not defined in this forum.

2. Data are an unalloyed good.

3. Data have agency.

4. “You can have privacy, or you can have cool online stuff. Which do you want?” – this from the Shopify data analyst (biology grad, self-described hacker).

I had a question for the panel. I asked about algorithmic prisons: the uses to which data are put. I pointed out that the data, by themselves, mean nothing. Interpretation, power, and control, and the creation of algorithms that have a view of the world which may not in themselves be warranted. Wasn’t there a role for the humanities in all of this?

Well bless my little cotton socks. It is rare that I am on the receiving end of techno-condescenion. I hope to god I’ve never done this to some one else – make assumptions about your level of engagment or knowledge of the field this way –  but it was apparent that I lost the panel the moment I said, ‘hi, I’m from the history department’. The shopify guy proceeded to lecture me on disruption, and that even the book was a disruption; but we got over that. Unintended side-effects? Sure, there’s always those, but don’t worry.

Big data will save us.

Shopify guy did have one interesting point to make: it’s not ‘big data’, but ‘big interest in data’. He then framed the 20th century as the century of deterministic solutions, whereas the 21st century will be the one of statistical solutions.

Oh good.

On the privacy issue, one comment from the floor in response to the ‘you can have privacy or you can have cool online tools’ was, ‘you’re just not trying hard enough’. The day progressed.

There were good points throughout the day; one of the data librarians from Carleton talked about data research management plans.

That’s why you need humanities.

As I was sitting in the room, I was also following the tweets from #caa2014 and #saa2014.  At those two conferences, archaeologists are wrestling with big data issues themselves. Hell, archaeology could be the original big data (if measured in sheer metric tonnage… but also in more conventional measurements of data). But what was striking was the contrast between the two displicinary perspectives. The archaeologists were a lot more humble in the face of dealing with big data describing the human past. There were tweets about uncertainty (about how to model it; about how to think through its implications). There were tweets about power, control, ownership, engagement, community. Tweets about liberation, tweets about being chained. (I’m freewheeling right now, from memory, rather than linking through).

Humility was lacking, at data day.  More data, better data, and our algorithms are an unalloyed good.

I’m not against making money. But… but… but… not like this.

So. The takeaway from this day? The big data, analytics-for-business crowd, need the humanities. There’s space there already in their discussions, though they don’t seem to recognize it. Or at least not yesterday.

Yes.

 

Interview by Ben Meredith, for his article on procedurally generated archaeology sims

I was interviewed by Ben Meredith on procedurally generated game worlds and their affinities with archaeology, for Kill Screen Magazine. The piece was published this morning. It’s a good read, and an interesting take on one of the more interesting recent developments in gaming. I asked Ben if I could post the unedited communication we had, from which he drew on for his article. He said ‘yes!’, so here it is.

Hi Ben,

It seems to me that archaeology and video games share a number of affinities, not least of which because they are both procedurally generated. There is a method for field archaeology; follow the method, and you will have correctly excavated the site/surveyed the landscape/recorded the standing remains/etc. These procedures contain within them various ways of looking at the world, and emphasize certain kinds of values over others, which is why it is possible to have a marxist archaeology, or a gendered archaeology, or so on. Thus, it also seems obvious to me that you can have an archaeology within video games (not to be confused with media archaeology, or an archaeology of video games). A great example of this kind of work is Andrew Rheinhart’s exploration of the beta of Elder Scrolls Online – you should touch base with him, too.http://archaeogaming.wordpress.com/2014/01/22/beta-testing-archaeology-in-elder-scrolls-online-taken-down/

On to your questions!

What motivated you to become an archaeologist?

Romance, mystery, allure, the ‘other’, the desire to travel… my initial impetus for getting into archaeology comes from the fact that I’m ‘from the bush’ in rural Canada and as a teenager I wanted so much more from the world. I now recognize that there’s some amazing archaeology in my own backyard (as it were) but I was too young and immature to recognize it then. The Greek Bronze Age, the Mycenaean heroes, the Minoans, Thera… all these captured my imagination. And there was no snow!

Personally, what single facet of archaeology captures the spirit of the field most effectively?

Check out the work of Colleen Morgan http://middlesavagery.wordpress.com/2014/03/05/stop-saying-archaeology-is-actually-boring/ and Sophie Hay http://pompei79.wordpress.com/2014/03/05/scratching-the-surface/ and Lorna Richardson http://digipubarch.org/2014/03/14/all-the-swears-for-this/ If there is a ‘spirit of the field’, I think these three scholars capture it admirably. They are curious, reflective, aware of the impact that the doing of archaeology has in the wider world. Archaeology produces powerful narratives, powerful ways of framing our current situation regarding the past and the present. I aspire to be more like these three remarkable women.

Which game do you think, so far, best achieves this?

A hard question to answer. But I think I’d go with Minecraft, for its community and especially its ability to be adopted in educational circles, for the way it requires the player to build and engage with the environments created. The world is what you make it, in Minecraft. So too in archaeology.
If a game attempted to procedurally generate ancient civilizations, what do you think would be the three most important elements that had to be generated?
I’ve done a lot of agent-based simulation. http://www.graeworks.net/category/simulations/ . Such a game would have to be built on an agent-based framework, for the NPCs. Each NPC would have to be unique. Those rules of behaviours that describe how the NPCs interact with each other, the environment, and the player would have to accurately capture the target ancient civilization. You can’t just have an ‘ancient civilization'; you’ll have to consider one very particular culture in one very particular time and place. That’s what a procedural rhetoric is all about: an argument in code about how this aspect of the world worked/is/existed.
Would investigation play an integral part in a video game interpretation?
I’m not sure I follow. Procedural generation on its own still is meaningless; it would have to be interpreted. The act of playing the game (and see the work of Roger Travis on http://playthepast.org on practicomimetics) sings it into existence.
Conversely, for you would stumbling blindly upon a ruin diminish the effect?
If the world is procedurally generated, then there would be clues in the landscape that would attune the attentive player to the presence of the past in that location. If there is no rhyme or reason – we stumble blindly – then the procedures do not describe an ancient (or any) civilization.

Do you think an archaeology simulator would be best implemented in first person (e.g. Minecraft) or third person (e.g. Terraria)? Would it be more important to convey an intimate atmosphere or impressive scale?
I like first person, but on a screen, first person can just induce nausea in the player. Maybe with an Oculus Rift that’s not a concern, in which case I’d say go first person! On a screen, I think third is better. Why not go AR and put your procedurally generated civilization into the local landscape?