An Open Research Notebook Workflow with Scrivener and Github

I like Scrivener. I *really* like being able to have my research and my writing in the same place, and most of all, I like being able to re-arrange the cards until I start to see the ideas fall into place.

I’m a bit of a visual learner, I suppose. (Which makes it ironic that I so rarely provide screenshots here. But I digress). What I’ve been looking for is a way to share my research, my lab notes, my digital ephemera in a single notebook. Lots of examples are out there, but another criterion is that I need to be able to set something up that my students might possibly be able to replicate.

So my requirements:

1. Visually see my notes, their layout, their possible logical connections. The ability to rearrange my notes provides the framework for my later written outputs.

2. Get my notes (but not all of the other bits and pieces) onto the web in such a way that each note becomes a citable object, with revision history freely available.

3. Ideally, that could then feed into some sort of shiny interface for others’ browsing – something like Jeckyll, I guess – but not really a big deal at the moment.

So #1 is taken care of with Scrivener. Number 2? I’m thinking Github. Number 3? We’ll worry about that some other day. There are Scrivener project templates that can be dropped into a Github repository (see previous post). You would create a folder/repo on your computer, drop the template into that, and write away to your hearts content, committing and syncing at the end of the day. This is what you’d get. All those slashes and curly brackets tell Scrivener what’s going on, but it’s not all that nice to read. (After all, that solution is about revision history, not open notebooks).

Now, it is possible to manually compile your whole document, or bits at a time, into markdown files and to commit/sync those. That’s nice, but time consuming. What I think I need is some way to turn Scrivener’s rtf’s into nice markdown. I found this, a collection of scripts by Sparkygetsthegirl as part of a Scrivener to Android tablet and back writing flow. Check it out! Here’s how it works. NB, this is all Mac based, today.

1. Make a new Scrivener project.

2. Sync it to dropbox. (which is nice: backups, portability via Dropbox, sharing via Github! see below)

3. drop the 4 scripts into the synced folder. Open a terminal window there. We’ll come back to that.

4. open Automator. What we’re going to do is create an application that will open the ‘drafts’ folder in the synced project, grab everything, then filter for just the markdown files we made, then move them over to our github repo, overwriting any pre-existing files there. Here’s a screenshot of what that application looks like in the Automator editing screen:

Remember, you're creating an 'application', not a 'workflow'
Remember, you’re creating an ‘application’, not a ‘workflow’

You drag the drafts folder into the ‘Get specified finder items’ box, get the folder contents, filter for files with file extension .md, and then copy to your github repo. Tick off the overwrite checkbox.

Back in scrivener, you start to write.

Write write write.

Here’s a screenshot of how I’m setting up a new project.

Screen Shot 2014-09-17 at 1.50.14 PM

In this screenshot, I’ve already moved my notecards from ‘research’ into ‘draft’. In a final compile, I’d edit things heavily, add bits and pieces to connect the thoughts, shuffle them around, etc. But right now, you can see one main card that identifies the project and the pertinent information surrounding it (like for instance, when I’m supposed to have this thing done). I can compile just that card into multimarkdown, and save it directly to the github repository as readme.md.

Now the day is done, I’m finished writing/researching/playing. I sync the project one last time. Then, in the terminal window, I can type

./rtf2md Draft/*.rtf

for everything in the draft folder, and

./rtf2md Notes/*.rtf

for everything in the notes folder. Mirabile dictu, the resulting md files will have the title of the notecard as their file name!

Screen Shot 2014-09-17 at 1.56.06 PM

Here, I’ve used some basic citation info as the name for each card; a better idea might be to include tags in there too. Hey, this is all still improv theatre.

Now, when I created that application using automator, I saved it to my desktop. I double-click on it, and it strains out the md files and moves them over to my github repository. I then commit & sync, and I now have an open lab notebook on the web. Now, there are still some glitches; my markdown syntax that I wrote in, in Scrivener, isn’t being recognized on github because I think Scrivener is adding backslashes here and there, which are working like escape characters?

Anyway, this seems a promising start. When I do further analysis in R, or build a model in Netlogo, I can record my observations this way, create an R notebook with knitr or a netlogo applet, and push these into subfolders in this repo. Thus the whole thing will stick together.

I think this works.

~o~
Update Sept 18. I’ve discovered that I might have messed something up with my syncing. It could be I’ve just done something foolish locally or it might be something with my workflow. I’m investigating, but the upshot is, I got an error when I synced and a new folder called ‘Trashed Files’, and well, I think I’m close to my ideal setup, but there’s still something wonky. Stay tuned.

Update Sept 19 Don’t write in Scrivener using markdown syntax! I had a ‘doh’ moment. Write in Scrivener using bold, italics, bullets, etc to mark up your text. Then, when the script converts to markdown, it’ll format it correctly – which means that github will render it more or less correctly, making your notes a whole lot easier to read. Click on ‘raw’ on this page to see what I mean!

6 thoughts on “An Open Research Notebook Workflow with Scrivener and Github

  1. I love scrivener too, but I find that collaborating/sharing with others is such a task unless you compile individually like you said – but that is such a pain too. I like how you worked around the web-sharing problem, maybe someday, the literature and latte folks will come up with a way to enable real-time collaboration within scrivener, kind of like Wunderlist. As an educator, I’d love to see how that would shape crowd-sourced learning. Awesome post!

    1. Thanks! I believe the Mac version of Scrivener can allow at least version control between multiple collaborators (if you’re saving to a dropbox). I’d have to dig into the documentation a bit more to make sure I don’t lead you astray. On our ‘Macroscope’ project (http://themacroscope.org) we had three collaborators using scrivener, but one used windows, one used mac, and the other flipped between platforms (I think). Anyway, we have a piece coming out next month about that workflow, and I’ll link to it from here when it does! Thanks again

  2. Hey mate,
    I’n the owner of the site you borrowed your code from (sparkygetsthegirl.com). I’m glad you too are getting some use from it.:) And thanks for the linkback.

    The issue with escape characters is that if you have something like a dash in your document (as most authors do as scene/section breaks), the Markdown formatter needs a way to represent these as being part of the text and not for formatting. That’s why you’ll see spare slashes floating around.

    This is all in accordance with the definition for Markdown at http://daringfireball.net/projects/markdown/. Check out the ‘Syntax’ section. If you scroll all the way down to the end, you’ll see the escape characters Markdown supports. I didn’t allow all of these (like a dot or period). My markdown editor didn’t seem to have a problem with fullstops, so I left them unescaped.

    I didn’t think to use Automator, but what i did do is use Crontab (available on all Mac/Linus/Unix systems). To convert the included code to Windows-based, you would need Windows equivalents of the Mac system programs I use in the shell files. Shouldn’t be too huge a task to port over.

    Anyhow, good luck with it and enjoy.
    SparkyGetsTheGirl

    1. Hi! I’m glad you shared the code – it was just what I was looking for, and I’m looking forward to trying it out fully following your workflow to my Android tablet. I’m ammending my text to give you fuller props for writing that. Next on my agenda is as you say to try to port over to Windows, so I can use my windows machine at work and mac machine at home.

      Thank you also for the pointers re markdown. I’m just getting to grips with markdown, in the hopes of building it into my teaching for the winter term.

      Cheers!

Comments are closed.