This is my first post on wordpress.com where I have to use the Gutenberg editor and man I hate it. I should probably migrate this damned thing – 14 years old! – out of wordpress. I’ve wasted a huge amount of time just trying to get things organized so I know what the hell is going on. Yeah, I can’t stand change, once I figure out how to do something the way I like it. But, on a similar theme… I’m still trying to figure out out to *take good notes*.

When I was a PhD student, I kept incredibly dense and largely useless notes. By ‘useless’ I mean – once I’d made an entry in my great big notebook, it was incredibly hard to find it again. To see which other materials it might speak to. I had several notebooks, where I’d duly copy out interesting passages and make my observations, and then shove slips of paper or post-it notes in on those pages where I thought ‘hey, this might be important and I really need to find this again’. Because I had no structure for getting into and through those notes, I’ve never used those notebooks since. Which is a waste.

Nobody ever taught me how to take notes.

When it came time to write, I’d get a big piece of paper and try to sketch out how the Big Idea I was writing about worked. I’d write down page numbers, cryptic directions to various pages in various notebooks, half-baked references, remembrances of important things I’d read, and draw circles and lines and swoops and eventually something would emerge out of that, but it was a messy, wasteful process.

What I’ve been searching for ever since was a way where I could capture the exciting ideas I was reading, the interesting thoughts I was having, in such a way that knowledge would crystallize out of the mess. With time, I’ve started to come up with a way that works on paper – a notebook with a line down the middle of the page; observations or important phrases copied on one side, with my reflection or thoughts on the other side. A citekey scribbled at the top of the page to connect to my reference manager (I’ve used all of the reference managers, it seems). An index page at the front of the notebook. Then, when it comes time to write, my Big Page is at least a little bit tidier with references to ‘orange nb p24 re bennett 32’.

There are a number of posts on this ol’ blog about taking notes, and different systems I’ve tried to cobble together to make ’em. In recent years, I’ve really become interested in the whole zettlekasten scene; the basic idea is one idea, one note. Sometimes I copy the relevant passage down, but most of the time, it’s just me riffing on something I’ve read; usually no more than three or four sentences. Then a system for indexing these so that notes can be compiled into larger overview notes or broken back down again. It doesn’t have to be digital, but of course, digital search and storage makes life easier. I’ve used everything from Notational Velocity through to plugins and mods for Sublime Text or Atom. And these all work in the sense that I’m able to pull together all of my relevant atomic notes and sometimes – if I’ve been really switched on – the compiled overview note goes into whatever I’m writing in its entirety. The note taking process is the writing process.

I really like when that happens.

Unfortunately, I find it hard to maintain the use of these different packages for zettles consistently. I think the reason for this is because, despite the ability to recombine, search, and find my atomic notes, I still can’t see the connections between things very well. But, with my most recent book project finally out of the way, and a bunch of other things finally having made their way through the publishing process, I’m ready to start again.

And boy, how the landscape has changed!

Roam, Foam, Org-roam, and Obsidian

It was a chance tweet I saw by Jonathan Reeve that sent me out on this latest note-taking odyssey, by the way:

I had to investigate. The major thing that has changed I think is the idea of ‘networked thought’ has really entered into the note taking space. And I think that’s what I’ve always missed in note taking process. The idea that if you make connections as appropriate between ideas, eventually larger structures emerge; these larger structures (network structures like shortest paths, clusters of various kinds, most-central nodes of various kinds) can give insight into the nature of your thoughts/nodes and perhaps suggest insights that you might not otherwise have spotted.

There is a wide array of editors to help you with this, all of which include network visualizations of links, backlinks, and tag structures. Some, like Roam  are subscription based and keep your notes somewhere on the cloud; others like  Foam  or Org-Roam are open source and keep your notes locally as markdown files (though Org-Roam is built on emacs and life’s too short). Then there’s  Obsidian which is not open source, but does keep your notes as separate markdown files. It has a pretty slick interface, and it will publish and host your ‘vault’ (folder of notes) as a website if you so wish (for a hosting fee, which seems pretty reasonable). If you ever read Caleb McDaniel’s ‘Open Notebook History‘ that feature will be quite interesting.*

I’ve been kicking the tires on Obsidian for the last week, and I have to say, I quite like it. I have a few community plugins installed that let me ‘refactor’ (break apart or merge together) notes as appropriate, that let me insert citations from my Zotero library (or create new notes from scratch on a given resource in my Zotero library) with links back to the original pdfs/resource, and a few cosmetic tweaks. New panes can be opened at will from a variety of places, and if you have the screen real-estate, organized however you like. I grabbed my existing folder of notes and opened it within Obsidian; I created a new index note to provide some consistent points of entry:

*I keep my notes, my ‘vault’, in a git-tracked folder, pushing online to a private git repo. I was also pushing to a public wikijs instance I host on Reclaim Cloud, but the importer broke and I can’t make it work any more. Anyway, that was probably too much – if I want to make my notes available online, I can probably just gh-pages them and that’ll serve. You can automate the process of pushing new notes to github; see this post by Bryan Jenks.

When you search for keywords or phrases or tags, the results of those searches can be turned into instant notes with wiki-style links. See that graph at top right? The green nodes are tags, the blue are notes, the red are notes that I’ve created while writing other notes that remain to be filled in.

Workflow

So here’s my workflow. I have Zotero and Zotfile installed, so I can send pdfs to a folder on my ipad. On the ipad, I use pdf reader to annotate. Zotfile retrieves these and pulls them back into Zotero. I use zotero-mdnotes to push the notes to my folder (‘vault’) of notes. (If I’m reading something physical, I can just mark it up or use my paper notebooks as before, and then transfer/consolidate notes into a new note in Obsidian.)

These I can then refactor into individual atomic notes as necessary. Using BetterBibTex for Zotero, I have also exported (with constant updating) my library’s bib file (as csl json) to the vault; I can then add the cite-key to any atomic note as appropriate. I add tags as appropriate. I link to other notes as appropriate. Obsidian shows me when a given note is referenced by another or mentions another and so I can use that to guide back-linking too.

Then, I can garden. By ‘gardening’, I mean, exploring my notes and their connections and thinking about what I’m seeing. Perhaps I add new notes. Perhaps I prune or delete notes. Perhaps I add more links or tags.

I love the graph feature. But I wish I could analyze it. There is a plugin that exports your graph to Neo4j for analysis, but that’s almost too much power for what I have in mind, and besides, you need to learn the cypher query language to make sense of that kind of thing. The ‘Infranodus‘ platform might be worth exploring here, as it does network metrics and text analysis too and can ingest your notes (see for instance this post) but I didn’t feel like signing up with credit card to something I just wanted to explore a bit (Infranodus can be installed locally, but it’s a beast of a thing to configure – it depends on Neo4j! – and after wasting the better part of a day on it, I threw in the towel).

No, good ol’ gephi or cytoscape or similar is all I need. So I did a bit of digging – where does Obsidian hold all of that info? It turns out, there is a json graph in a folder called ‘ObsidianCache’ that contains the current representation of your vault and its interlinkages:

Now, I’m certain that one could write a bit of python to grab each note and its links and tags, represent as a graph, and then do a few network metrics. But I don’t know how to do that in python – yet. But I can do it with jq , and reshape the json so that I end up with note – link and note – tag pairs. Gephi doesn’t ingest json, so I use a bit of R to turn it into graphml. Hey presto, a network I can explore in Gephi! What are the most central ideas? What kinds of ‘communities’ exist? I am imagining that knowing this information would help kick start my writing, or help me detect emergent ideas I hadn’t considered yet. (Other people feed their notes in Devonthink, which does some natural language magic to find connections in your notes. That’s another of the beautiful things about keeping your notes in plain text on your own machine).

The relevant jq query:

jq --raw-output '.metadata[] | {title: .frontmatter.title, tag: .tags[]?.tag}'

Then a bit of regex in sublime to put commas at the end of each line, wrap in square brackets, then a bit of R:

ibrary(igraph)
library(jsonlite)
setwd("~/Desktop")
thing <- fromJSON("tag-test.json") 
g <- graph_from_data_frame(thing, directed = FALSE)
write_graph(g, "tag-2mode.graphml", "graphml")

Open ‘er up in Gephi, using the multimode plugin to turn it from a network of notes to tags, to tags – tags by virtue of notes in common…

So obviously, there is some mucky data in my test vault, but interesting, eh? Incidentally, the ‘sg’ tag is for when I’ve had some inspiration that I want to come back to. And of course, maybe note to note by virtue of common tags would be a more interesting/useful view. Or perhaps, since a ‘tag’ could be considered as a kind of semantic note on its own, I just leave it notes – tags and treat it all as unimodal. Things to explore!

So we’ll see how things go. This morning, I spent a happy hour refactoring and building notes from a great article about archaeological photography at Dura Europos. Baird writes,

“Taking photographs, like drawing reconstructions, was a means by which the archaeologists could attempt to understand the object and the past and to rebuild the ruin. At Dura, photography was not a passive recording device as it is thought of in most histories of archaeology; rather, it was something that seems to have been an active means of constructing a particular past (fig. 5). Time in these photographs refers both to the practice of taking the photographs— the posing and framing—and the excavator’s construction of a time in the image; thus, they reflect a temporal breach that constructed an East in which modern peoples are equated with ancient.”

Active note taking, gardening our thoughts using these digital tools, seems to me a bit like how Baird writes about photography, perhaps. But I haven’t fully fleshed out that thought yet; perhaps its because it lets me build something new from others’ mental excavations of their own thought. Or I’m pushing the metaphor too far. Back to the garden I go!

Some useful videos

Below is a video of PhD student Courtney Applewhite describes how she uses Obsidian to study for her comps; something similar to this approach might be worth adapting.