Literate Programming Writ Large

After crafting an Org file with code blocks, we tangle the file to create a source code file. To make sure the code in your org file matches the source code file, place the following code at the bottom of your Org file. This automatically tangles when you save the file:

# Local Variables:
# eval: (add-hook 'after-save-hook #'org-babel-tangle t t)
# End:

We typically use this to set file-local variables when a file is loaded, but the eval sequence allows us to call Emacs functions. Note that when you first load a file like this, it will ask you if you want to run the code, and one of the options is to permanently allow that. I’m not encouraging such risky behavior, but …

If you choose !, it adds the hook to the list of safe-local-variable-values by altering either your init.el, or your custom.el if you set the custom-file variable. I don’t want Emacs altering my init.el file like this, so I add this code in my early-init.el file:

(setq custom-file
      (expand-file-name "custom.el" user-emacs-directory))

(when (file-exists-p custom-file)
  (load custom-file))

When you are viewing the tangled file, call org-babel-tangle-jump-to-org to return to the original Org file.

A related idea…In the header-args property, set the comments header argument to link, to allow changes to your tangled code to update your original literate org file after calling M-x org-babel-detangle. For instance:

#+PROPERTY:    header-args: :tangle yes :comments link

This tangle/detangle allows you to edit either representation of your code. See also the org-tanglesync project for putting a bit of controls around this feature. Personally, I haven’t used this feature, for as the remainder of this essay shows, I’m happy editing solely in the Org file.

I assume you know how you can evaluate code block’s in Org files in the language you tag by typing C-c C-c anywhere in the block. Here are some of my more advanced tips.

In a large code base, we begin by organizing our code by library or module, and then each file contains a class composed of methods or functions and variables or fields. As I mentioned above, literate programming allows me to add a semantic organization layer where I can group related concepts under headlines (which can be nested).

With a few functions, I’ve created an user interface that allows me to search the header sections in my project and jump to that file.

In the screencast below, you notice that the file is the first part of the line, followed by the top-level header, then subheading under that.

Using the fuzzy matching capabilities, I can type any combination of file/headline to narrow the scope to matching headers.

Feel free to steal the code for this here.

What about jumping directly to a function or class name? Often, we can see some code that calls a method on a class or a function, and we issue a call to xref-find-definitions (M-. … or if using Evli, evil-goto-definition). But these functions don’t understand that the source code can be located in Org files.

When jumping around in code we typically have three types of jumps:

• Definitions. This is often a quick jump to where you have defined a symbol (function, variable, class) that the cursor is on. Often you will have one result.
• References. Usually a list of function calls or variable usage is presented, allowing you to choose the one you want.
• All Usages (called apropos in Emacs parlance). A combination of both of the above, plus references in comments, etc. Again, often presented as a list to choose.

I want to jump around my code like this whether I was in an Org code block or in the surrounding prose. Inspired by the dumb-jump project that uses git-grep or ripgrep to quickly search a code base by symbol in real time, I wrote an extension to the xref API.

The first challenge was determining what a symbol could be. In most languages, that is straight-forward. In Python, only sequences of alpha-numeric and underscores, but in Lisp, one needs context, as almost any character sequence, starting and terminating with spaces or paren/braces defines a symbol. With literate programming, I want to be able to describe a function name in the prose so any of the following is possible:

The symbol, ~foobar~, defines what is.
However, =foobar=, defines what isn't.
As mentioned, the symbol (foobar) states much.
But other functions, like foobar, are understated.

In the screencast below, you can see the point (cursor) on a symbol in the prose, and I can hit a sequence to jump to the definition in an org block:

My next challenge was deciding how to take advantage of searching for references and apropos references. I decided that references would be limited to Org code blocks. The following is an example of selecting xref-find-references (M-?) on the symbol, major-mode:

I’m pleased with the results, but it isn’t perfect, so I’m leaving it in my personal Emacs configuration file, but hope to make a nice package on Melpa. Until then, feel free to steal this code too.