historical maps into Unity3d

This should work.

Say there’s a historical map that you want to digitize.  It may or may not have contour lines on it, but there is some indication of the topography (hatching or shading or what not). Say you wanted to digitize it such that a person could explore its conception of geography from a first person perspective.

Here’s a workflow for making that happen.

Some time ago, the folks at the NYPL put together a tutorial explaining how to turn such a map into a minecraft world. So let’s do the first part of their tutorial. In essence, what we do is take the georectified map (which you could georectify using something like the Harvard Map Warper), load that into QGIS, add elevation points, generate a surface from that elevation, turn it into grayscale, export that image, convert to raw format, import into Unity3d.

Easy peasy.

For the first part, we follow the NYPL:

Requirements

QGIS 2.2.0 ( http://qgis.org )

  • Activate Contour plugin
  • Activate GRASS plugin if not already activated

A map image to work from

  • We used a geo-rectified TIFF exported from this map but any high rez scan of a map with elevation data and features will suffice.

Process:

Layer > Add Raster Layer > [select rectified tiff]

  • Repeat for each tiff to be analyzed

Layer > New > New Shapefile Layer

  • Type: Point
  • New Attribute: add ‘elevation’ type whole number
  • remove id

Contour (plugin)

  • Vector Layer: choose points layer just created
  • Data field: elevation
  • Number: at least 20 (maybe.. number of distinct elevations + 2)
  • Layer name: default is fine

Export and import contours as vector layer:

  • right click save (e.g. port-washington-contours.shp)
  • May report error like “Only 19 of 20 features written.” Doesn’t seem to matter much

Layer > Add Vector Layer > [add .shp layer just exported]

Edit Current Grass Region (to reduce rendering time)

  • clip to minimal lat longs

Open Grass Tools

  • Modules List: Select “v.in.ogr.qgis”
  • Select recently added contours layer
  • Run, View output, and close

Open Grass Tools

  • Modules List: Select “v.to.rast.attr”
  • Name of input vector map: (layer just generated)
  • Attribute field: elevation
  • Run, View output, and close

Open Grass Tools

  • Modules List: Select “r.surf.contour”
  • Name of existing raster map containing colors: (layer just generated)
  • Run (will take a while), View output, and close

Hide points and contours (and anything else above bw elevation image) Project > Save as Image

You may want to create a cropped version of the result to remove un-analyzed/messy edges

As I noted a while ago, there are some “hidden, tacit bits [concerning] installing the Contour plugin, and working with GRASS tools (especially the bit about ‘editing the current grass region’, which always is fiddly, I find).”  Unhelpfully, I didn’t write down what these were.

Anyway, now that you have a grayscale image, open it in Gimp (or Photoshop; if you do have photoshop go watch this video and you’re done.).

For those of us without photoshop, this next bit comes from the addendum to a previous post of mine:L

    1. open the grayscale image in Gimp.
    2. resized the image as power of 2 + 1 (*shrug* everything indicates this is what you do, with unity); in this case I chose 1025.
    3. save as file type RAW. IMPORTANT: in the dialogue that opens, set ‘RGB save type to ‘planar’.
    4. Change the file extension from .data to .raw in mac Finder or windows Explorer.

Now you can import this historical elevation map in Unity. In Unity, add a gameobject -> 3d object -> terrain to the project. In the inspector window, there’s a cogwheel. Click this; it opens the settings. One of the options will be ‘import raw’. Click this.

Select your .raw grayscale image.

  1. On the import dialogue, change it to 8-bit image rather than 16-bit.
  2. Change the width, height, x and z to all be 1025. Changed the y to be 75 (yours will be different; look at the range in your original map of highest and lowest point, and input that. For reference, please also see this post which saved me: http://newton64.github.io/blog/2013-07-24-gimp-unity-terrain.html

Ta da – a white glacial landscape with your elevation data.

Screen Shot 2015-06-09 at 12.14.30 PMNow the fun stuff can happen. But – before someone can ‘walk’ around your landscape, you have to add controls to your project. So, in Unity 3d, go to:

Assets – Import package – characters.

Once that’s all done, you’ll drag-and-drop a ‘FPSController’ into your project. You’ll find it as below:

Screen Shot 2015-06-09 at 7.26.52 PM

Click and grab that blue box and move it up into your project (just drop it in the main window). Make sure that the control is above (and also, not intersecting any part of) your landscape, or when you go to play, you’ll either be stuck or indeed falling to the centre of the earth. We don’t want that. Also, delete the ‘camera’ from the hierarchy; the fpscontroller has its own camera. My interface looks like this:

Screen Shot 2015-06-09 at 7.30.47 PM

You do the grass and trees etc from the terrain inspector, as in the window there on the right. I’ll play some more with that aspect, report back soonish. Notice the column drum in the right foreground, and the tombstone in the back? Those were made with 3d photogrammetry; both are hosted on Sketchfab, as it happens. Anyway, in Meshlab I converted from .obj to .dae, after having reduced the polygons with quadratic edge decimation, to make them a bit simpler. You can add such models to your landscape by dropping the folder into the ‘assets’ folder of your Unity project (via the mac Finder or windows explorer).  Then, as you did with the fpscontroller block, you drag them into your scene and reposition them as you want.

Here’s my version, pushed to webGL

Enjoy!

(by the way, it occurs to me that you could use that workflow to visualize damned near anything that can be mapped, not just geography. Convert the output of a topic model into a grayscale elevation map; take a network and add elevation points to match betweeness metrics…)

Historical Maps into Minecraft: My Workflow

The folks at the New York Public Library have a workflow and python script for translating historical maps into Minecraft. It’s a three-step (quite big steps) process. First, they generate a DEM (digital elevation model) from the historical map, using QGIS. This is saved as ‘elevation.tiff’. Then, using Inkscape, they trace over the features from the historical map that they want to translate into Minecraft. Different colours equal different kinds of blocks. This is saved as ‘features.tiff’. Then, using a custom python script, the two layers are combined to create a minecraft map, which can either be in ‘creative’ mode or ‘survival’ mode.

There are a number of unspoken steps in that workflow, including a number of dependencies for the python script that have to be installed first. Similarly, QGIS and its plugins also have a steep (sometimes hidden) learning curve. As does Inkscape. And Imagemagick. This isn’t a criticism; it’s just the way this kind of thing works. The problem, from my perspective, is that if I want to use this in the classroom, I have to guide 40 students with widely varying degrees of digital fluency.* I’ve found in the past that many of my students “didn’t study history to have to work with computers” and that the payoff sometimes (to them) doesn’t seem to have (immediate) value. The pros and cons of that kind of work shall be the post for another day.

Right now, my immediate problem is, how can I smooth the gradient of the learning curve? I will do this by providing 3 separate paths for creating the digital elevation model.

Path 1, for when real world geography is not the most important aspect.

It may be that the shape of the world described by the historical map is what is of interest, rather than the current topography of the world. For example, I could imagine a student wanting to explore the historical geography of the Chats Falls before they were flooded by the building of a hydro dam. Current topographic maps and DEMs are not useful. For this path, the student will need to use the process described by the NYPL folks:

Requirements

QGIS 2.2.0 ( http://qgis.org )

  • Activate Contour plugin
  • Activate GRASS plugin if not already activated

A map image to work from

  • We used a geo-rectified TIFF exported from this map but any high rez scan of a map with elevation data and features will suffice.

Process:

Layer > Add Raster Layer > [select rectified tiff]

  • Repeat for each tiff to be analyzed

Layer > New > New Shapefile Layer

  • Type: Point
  • New Attribute: add ‘elevation’ type whole number
  • remove id

Contour (plugin)

  • Vector Layer: choose points layer just created
  • Data field: elevation
  • Number: at least 20 (maybe.. number of distinct elevations + 2)
  • Layer name: default is fine

Export and import contours as vector layer:

  • right click save (e.g. port-washington-contours.shp)
  • May report error like “Only 19 of 20 features written.” Doesn’t seem to matter much

Layer > Add Vector Layer > [add .shp layer just exported]

Edit Current Grass Region (to reduce rendering time)

  • clip to minimal lat longs

Open Grass Tools

  • Modules List: Select “v.in.ogr.qgis”
  • Select recently added contours layer
  • Run, View output, and close

Open Grass Tools

  • Modules List: Select “v.to.rast.attr”
  • Name of input vector map: (layer just generated)
  • Attribute field: elevation
  • Run, View output, and close

Open Grass Tools

  • Modules List: Select “r.surf.contour”
  • Name of existing raster map containing colors: (layer just generated)
  • Run (will take a while), View output, and close

Hide points and contours (and anything else above bw elevation image) Project > Save as Image

You may want to create a cropped version of the result to remove un-analyzed/messy edges

The hidden, tacit bits here involve installing the Countour plugin, and working with GRASS tools (especially the bit about ‘editing the current grass region’, which always is fiddly, I find). Students pursuing this path will need a lot of one-on-one.

Path 2, for when you already have a shapefile from a GIS:

This was cooked up for me by Joel Rivard, one of our GIS & Map specialists in the Library. He writes,

1) In the menu, go to Layer > Add Vector Layer. Find the point shapefile that has the elevation information.
Ensure that you select point in the file type.
2) In the menu, go to Raster > Interpolation. Select “Field 3” (this corresponds to the z or elevation field) for Interpolation attribute and click on “Add”.
Feel free to keep the rest as default and save the output file as an Image (.asc, bmp, jpg or any other raster – probably best to use .asc since that’s what MicroDEM likes.
We’ll talk about MicroDEM in a moment. I haven’t tested this path yet, myself. But it should work.

Path 3 For when modern topography is fine for your purposes

In this situation, modern topography is just what you need.

1. Grab Shuttle Radar Topography Mission data for the area you are interested in (it downloads as a tiff.)

2. Install MicroDEM and all of its bits and pieces (the installer wants a whole bunch of other supporting bits; just say yes. MicroDEM is PC software, but I’ve run it on a Mac within WineBottler).

3. This video tutorial covers working with MicroDEM and Worldpainter:

https://www.youtube.com/watch?v=Wha2m4_CPoo

But here’s some screenshots – basically, you open up your .tiff or your .asc image file within MicroDEM, crop to the area you are interested in, and then convert the image to grayscale:

MicroDEM: open image, crop image.

MicroDEM: open image, crop image.

Convert to grayscale

Convert to grayscale

Remove legends, marginalia

Remove legends, marginalia

Save your grayscaled image as a .tiff.
Regardless of the path you took (and think about the historical implications of those paths) you now have a gray scale DEM image that you can use to generate your mindcraft world.

Converting your grayscale DEM to a Minecraft World

At this point, the easiest thing to do is to use WorldPainter. It’s free, but you can donate to its developers to help them maintain and update it. Now, the video shown above shows how to load your DEM image into WorldPainter. It parses the black-to-white pixel values and turns them into elevations. You have the option of setting where ‘sea level’ is on your map (so elevations below that point are covered with water). There are many, many options here; play with it! Adam Clarke, who made the video, suggests scaling up your image to 900%, but I’ve found that that makes absolutely monstrous worlds. You’ll have to play around to see what makes most sense for you, but with real-world data of any area larger than a few kilometres on a side, I think 100 to 200% is fine.

Now, the crucial bit for us: you can import an image into WorldPainter to use as an overlay to guide the placement of blocks, terrain, buildings, whatever. So, rather than me simply regurgitating what Adam narrates, go watch the video. Save as a .world file for editing; export to Minecraft when you’re ready (be warned: big maps can take *a very long time* to render. That’s another reason why I don’t scale up the way Adam suggests).

Go play.

To get you started: here are a number of DEMs and WorldPainter world files that I’ve been playing with. Try ’em out for yourself.

 

* another problem I’ve encountered is that my features colours don’t map onto the index values for blocks in the script. I’ve tried modifying the script to allow for a bit of fuzziness (a kind of, ‘if the pixel value is between x and y, treat as z’). I end up with worlds filled with water. If I run the script on the Fort Washington maps provided by NYPL, it works perfectly. The script is supposed to only be looking at the R of the RGB values when it assigns blocks, but I wonder if there isn’t something else going on. I had it work once, correctly, for me – but I used MS Paint to recolour my image with the exact colours from the Fort Washington map. Tried it again, exact same workflow on a different map, nada. Nyet. Zip. Zilch. Just a whole of tears and heartache.

Mapping Witches

I’ve always quite enjoyed the work of Terry Pratchett. This Christmas, I was given a copy of his work, with Jacqueline  Simpson,  about the folklore which Pratchett happily transmogrifies from Earth to the Discworld. It’s quite an interesting book, and as I read the section on the witches of Lancre, it sparked some thoughts.

In Lancre, as on Earth, witches sometimes ‘borrow‘ and or transform themselves into animals.  A common story has a hunter shooting a hare or bird, which manages to escape; the next day the local witch will be seen limping or otherwise affected.  I never appreciated how widespread that story (and its variants) is (according to Folklore)… and then remembered that we have a variant of our own in this neck of the woods.

In 2007 I was hired to catalogue the intangible heritage of Gatineau Park. One story that I came across (recorded in Up the Gatineau! the journal of the Gatineau Valley Historical Society) was about the witches of Brown’s Lake. Apparently, in the early 20th century, there were two branches of the Brown family, and they were a-feuding.  The matriarchs of both branches were acknowledged witches. One day, the one Mrs. Brown was assaulted by a poltergeist in her own house. She eventually was able to find the source of the poltergeist, emanating from under the upstairs bed. She had her son shoot his rifle under the bed, and the paranormal activity ceased. At the exact moment that the rifle went off, the other Mrs. Brown was walking down the road to Wakefield village – and she dropped down dead on the spot.

So now I’m wondering… how widespread is this (dare I use the word) meme? If we mapped it, what patterns would we discover? A story so widespread must mean something? I’ve set up a Google map, with this first information point on it; anyone may edit it. Over time, perhaps something interesting will emerge…