Maps, Books, and Code

Recently, Tom MacWright tweeted about something near and dear to my heart:

Tom: though i ❤️ thinking about this problem, every time that linear-plain-text is proposed as the problem with programming, i wonder about books. would novels or nonfiction benefit from explanatory arrows or nonlinear spatial layouts?

Rasmus Andersson: For example, why don’t we use multiple columns to express concurrency in our program code? This is how most of us write code; in a linear fashion. Then we use our powerful brains to internally in our minds convert what’s on the screen to what is actually happening (arrows.)

Tom: i think i’m circling around the unpopular opinion that “visual thinking is wildly overrated”

I’m going to be a bit unfair and reply to these tweets in a longer blog post.

For Tom’s main question, “would novels or nonfiction benefit from explanatory arrows or nonlinear spatial layouts?” I think the answer is probably loosely “yes they would, but it’s complicated.” For starters, there are plenty of books which don’t require much beyond words (or some that are better because they’re only made of words), primarily works of fiction. These books are made entirely with words in mind, and words and the readers’s experience with those words is the point of these books. A novel “augmented” with a nonlinear spatial layout might be interesting, but it might also no longer be a novel.

But what about explanatory books? For these kinds of texts, while words alone might be passable, their explanations are augmented and enriched by images: pictures, graphics, diagrams, etc. And sure enough, there are tons of books like this (for a survey, check out some of Edward Tufte’s books cataloguing them). Many of these books are primarily linear, but often feature breaches in the linearity with images and cross references.

Comics especially excel at this, despite them being primarily “sequential art,” they benefit from the spatial arrangement of their panels, which can result in larger meanings. See this Nerdwriter video about a page of Art Spiegelman’s Maus. For more about the explanatory potential of the medium of comics, whose information is more rooted in image than in word, see Scott McCloud’s Understanding Comics and Nick Sousanis’s Unflattening.

(There are also choose your own adventure books, which are decidedly less linear works of fiction, but they’re aimed more at play than they are at explanation. See also Doug Dorst and JJ Abrams’s S “novel”.)

I’m certain Tom and anyone else reading this post knows there are benefits to explanatory books that are more than just words, and that there are books benefiting from more spatial, less linear arrangements. You might ask “then, why aren’t all books like this?” to which I’d say probably two reasons:

For starters, these kinds of books are more “expensive” to make, both in terms of production costs (images can be ink heavy), but more importantly in terms of skill. Modern Western education centres words, but marginalizes images as the fringe we call “art” (see Betty Edwards’s Drawing on the Right Side of the Brain (especially its first few chapters) and Max Kreminski’s thread about individual style in drawing for arguments in favour of drawing education as a general purpose thinking tool). True, ours is a culture with plenty of images on televisions and computers, but those images are made by a relatively small portion of the population.

And secondly, books are limited by the physical world they inhabit. It’s not arbitrary that most books are the generally small, generally hand-holdable size that they are. Most books tend toward a size and weight that can be carried around and held in somebody’s hand, comfortably. The limit of hand size and wrist strength, along with the resolution a typical human eye can read make it harder to conveniently fit useful spatial layouts. This is the reason why many explanatory books are published in larger trim sizes, and why some are relegated to the “coffee table book” status.

A paper map might be spatially and informationally rich, but you usually need to lay it out on a table to read it. And if you want to read some building stories, you may need a couple of tables.

Anyway, I think explanatory books would benefit from a post-text, post-linear world where more text is augmented not only by images, but also arrangement. It’s just hard, especially with paper, to do it.

Now back to our regular programming

Regular programming being, “linear-plain-text,” would that benefit from non-linear, spatial arrangements? You can probably guess that I’m going to say yes, but I’ll start by saying that program code is already non-linear in structure. Code cross references itself all over the place in every single source code file — just about every line of code could be thought of like a hyperlink to another piece of code (and indeed, many code editors let you navigate around your files as though the code statements were hyperlinks). But while the structure of code involves a ton of non-linearity, the presentation of code is more or less shackled to line-after-line, just like the teletypewriters our command lines (and thus code) emulate.

I think you could augment just about every aspect of our plain text code edited in linear text editors — there’s a whole orchard of low hanging fruit here. For starters, code editors seem resigned to monospaced type at a single font size, ignoring fundamental graphic design / typography principles (eg hierarchy and scale, etc).

Then there’s the linear layout of code. Rasmus’s multi-column with arrows mockup itself probably wouldn’t be that great, but I think the spirit of what he’s attempting is good: busting source code files out of their text-like layout. Code Bubbles (video demo) is another, perhaps more fleshed out prototype of a code editor not bound by the flow of text in a file.

I think this is a fertile area for research, as we’re just barely scratching the surface of how to explore and edit large, dynamic, non-linear systems. I also think breaking out of laptop (or even iMac-sized) screen real estates would help immensely too (see Dynamicland and Bret Victor in general, but also pre-computer systems-exploring tools for some starting points on this). I think arrows, columns, “live programming” might be necessary starts, but I don’t think they’re sufficient.

While Tom says he’s “circling around the unpopular opinion that ‘visual thinking is wildly overrated,’” I think it’s more true that today’s “programming skill” means excelling at not thinking visually, because the visuals just aren’t available to us. Or maybe put another way, programming today means excelling at thinking visually, inside your head, instead of thinking with visuals available to you from your tools.

So, it’s mostly true that skilled programmers can get by (though, I’d say, are tremendously hobbled) without visual thinking tools, but I’m of the opinion that if programming involved thinking with visual tools from the getgo, things might be a hell of a lot easier.

You could imagine a world where cartography never incorporated drawings of territories, and instead relied solely on written descriptions of land. “To the west is a mountainous range, with several large rivers emptying to a gulf in the south.” In such a world, there would no doubt be practised experts, capable of envisioning in their minds the described area. But these written maps would clearly suffer from a lack of depictions.

I think we’re in a similar place with programming, where we must rely solely on written descriptions, and seldomly if ever are we able to make use of drawn depictions. While cartographic maps might have a slightly easier time depicting the land that’s out there, there’s also the notion that maps are abstractions (ie the map is not the territory). What we model with programming is also an abstraction, a model of some territory, but we’re stuck picturing it in our heads. And I think the dynamic terroir of computer programming invites more exploration and experimentation to see what’s really out there.

Speed of Light