Capturing Content for Emacs

Let’s suppose you are current investigating a new code base, system or other problem, and you are following my advice and copying code, storing output, and taking notes along the way. All of this gets stored into your engineering notebook, aka primary org mode file (for me, this is often my current Sprint page).

Sure, selecting code, switching buffers or windows, pasting the code (maybe even jotting down some notes), and then popping back to your original file, may not be many keystrokes, but it exacts a bit of mental tax that mounts.

The typical solution to this problem is to use the org-capture feature (If you are not familiar with this Org feature, check out my gentle introduction or see Step 3 of Sacha Chua’s essay, Learn how to take notes more efficiently in Org Mode). While org-capture makes copying content into your org file easy, I am trying to improve on it, and here some of my experiments.

The “Current” Task

One mentally taxing aspect of org-capture is determining where something should go. Do you have a dozen (file) reference destinations? I have found the (clock) reference ideal for altering a default destination. Specifically, I begin work on a task, and designate it the focus of my attention (i.e. the destination of my work), by clocking in, using org-clock-in (C-c C-x C-i or , I in Spacemacs).

Now, we can add the following to the org-capture list:

(add-to-list 'org-capture-templates
             `("c" "Item to Current Clocked Task" item
               (clock)
               "%i%?" :empty-lines 1))

This capture destination allows me to easily specify any header as a special destination with a simple clock in. However, we do have the mental interruption associated with creating a new buffer. Let’s minimize that by allowing us to put something on the kill ring, and send it to that clocked-in task:

(add-to-list 'org-capture-templates
             `("K" "Kill-ring to Current Clocked Task" plain
               (clock)
               "%c" :immediate-finish t :empty-lines 1))

The trick here is the use of :immediate-finish, where it doesn’t even bother with a buffer, but just injects the kill-ring contents to the clocked in task without even a sneeze. Don’t want the hassle of sending something to the kill-ring? With this one, you only have to select the text, then kick off the capture:

(add-to-list 'org-capture-templates
             `("C" "Contents to Current Clocked Task" plain
               (clock)
               "%i" :immediate-finish t :empty-lines 1))

In fact, create the following function and keybinding, and you can select text, and immediately copy it to your clocked in task without bothering with the org-capture menu:

(defun region-to-clocked-task (start end)
  "Copies the selected text to the currently clocked in org-mode task."
  (interactive "r")
  (org-capture-string (buffer-substring-no-properties start end) "C"))

(global-set-key (kbd "C-<F17>") 'region-to-clocked-task)

This is great for general textual content, but much of what I want to copy is code, which could bring along a bit of meta data.

Code References

Much of my ideas got started after reading this blog entry where the idea is to have a function gather meta data associated with the currently selected text, and help to leave a back trace to the original code file.

I wanted to copy both code and regular text, so I made ha/org-capture-clip-snippet for wrapping the region in an EXAMPLE:

(defun ha/org-capture-clip-snippet (f)
  "Given a file, F, this captures the currently selected text
within an Org EXAMPLE block and a backlink to the file."
  (with-current-buffer (find-buffer-visiting f)
    (ha/org-capture-fileref-snippet f "EXAMPLE" "" nil)))

And ha/org-capture-code-snippet for getting function name and the code type:

(defun ha/org-capture-code-snippet (f)
  "Given a file, F, this captures the currently selected text
within an Org SRC block with a language based on the current mode
and a backlink to the function and the file."
  (with-current-buffer (find-buffer-visiting f)
    (let ((org-src-mode (replace-regexp-in-string "-mode" "" (format "%s" major-mode)))
          (func-name (which-function)))
      (ha/org-capture-fileref-snippet f "SRC" org-src-mode func-name))))

Both of these function do not do much, but given some values to Nick’s original function (which I’ve modified the format to fit my personal style):

(defun ha/org-capture-fileref-snippet (f type headers func-name)
  (let* ((code-snippet
          (buffer-substring-no-properties (mark) (- (point) 1)))
         (file-name   (buffer-file-name))
         (file-base   (file-name-nondirectory file-name))
         (line-number (line-number-at-pos (region-beginning)))
         (initial-txt (if (null func-name)
                          (format "From [[file:%s::%s][%s]]:"
                                  file-name line-number file-base)
                        (format "From ~%s~ (in [[file:%s::%s][%s]]):"
                                func-name file-name line-number
                                file-base))))
    (format "
   %s

   #+BEGIN_%s %s
%s
   #+END_%s" initial-txt type headers code-snippet type)))

However, content I want to store in an org-mode comes from more than just Emacs buffers.

Output from Terminal Commands

What if the end result of a command sequence on the Terminal was a pipe to a program that could use cat to gather textual data from standard input, and then use emacsclient call org-capture to store it?

Yeah, and interesting idea when sent to the current clocked in task:

#!/bin/bash

TITLE="$*"
CONTENT="
     #+BEGIN_EXAMPLE
$(cat | sed 's/^/     /g')
     #+END_EXAMPLE
"

if [[ -n $TITLE ]]
then
  CONTENT="   - ${TITLE}\n${CONTENT}"
fi

/usr/local/bin/emacsclient -c -n \
  -e "(progn (org-capture-string \"$CONTENT\" \"C\") (delete-frame))"

Here I’m using our latest C capture template to that just takes textual context and stores is. Let’s try it in action by typing the following in a shell:

date | ec

Works like a charm:

#+BEGIN_EXAMPLE
Thu Jun  7 22:45:23 PDT 2018
#+END_EXAMPLE

Content from Browsers

Like many software people, I have a love-hate relationship with browsers. I often find myself copying/pasting information from a web site into my engineering notebook. Pasting text data into an org-mode file looses all text formatting as well as hyperlink references. But operating system clipboards can store some of this formatting data, so we just need to tap into it.

The downside is that accessing this information is operating system dependent…

Version for Mac

Let’s start from the top and work our way down. I need a way, outside of Emacs, to run a command to copy the selected region to the clipboard, and then use emacsclient to start a function to copy that information into the currently clocked in task.

I use Alfred to start a Workflow, as it will allow me to trigger these scripts in succession as shown in this diagram:

capturing-content-01.png

The trigger (in this case, just about every meta-key on a laptop), will start the first script that basically issues the Command-C to copy the selected text to the clipboard:

tell application "System Events" to keystroke "c" using command down

This works with any Mac application, including browsers.

The next script basically takes the contents of the clipboard (as HTML), render that to an org-compatible format with pandoc (which you’ll need to install), and then use emacsclient to call my org-capture routine with the “C” selection, so that the contents go directly to my clocked in task. My first attempt was a modified version from Roland Crosby:

query=$(osascript -e 'the clipboard as "HTML"' | \
            perl -ne 'print chr foreach unpack("C*",pack("H*",substr($_,11,-3)))' | \
            /usr/local/bin/pandoc -f html -t org | \
            sed 's/"//g' | sed 's/^/   /' )

/usr/local/bin/emacsclient -c -n \
                           -e "(progn (org-capture-string \"${query}\" \"C\") (delete-frame))"

While the above code works well from a browser, if I copy text from something else (like Slack), the as "HTML" bit throws an error, as the clipboard contains plain text. After working on a fairly involved Perl script (that didn’t handle Unicode characters well), I ended up writing the entire part in Emacs Lisp. Now the shell script is nothing more than:

/usr/local/bin/emacsclient -c -n -e "(ha/external-capture-to-org)"

Unpacking Apple’s encoding plus dealing with both HTML and Text content was a bit more involved, but you can see the functions on Gitlab. The end result is great. After selecting some text on the homepage at orgmode.org, and clocked this header as my current task, I ended up with this getting pasted:

The stable version of Org is *9.1.13*, as of May 2018. See the
[[https://orgmode.org/Changes.html][release notes]].

Get it with =M-x package-install RET org RET= (see
[[https://orgmode.org/elpa.html][Org ELPA]]).

Or download it as a [[https://orgmode.org/org-9.1.13.tar.gz][tar.gz]] or
[[https://orgmode.org/org-9.1.13.zip][zip]] archives.

Keep in mind, that this copy/pasting business happens completely in the background while I am still surfin’ the web.

Version for Linux

Like everything, I suppose, getting this feature working on my Linux laptop is both easier and harder. Unlike the Mac, I can find no way to automatically copy the current selection to the clipboard like I can with an Applescript. However, once the content is on the clipboard, I can more easily grab it and throw it into Emacs.

Under the Keyboard section of the Settings app (at least on Ubuntu), you can create shortcuts to run commands:

capturing-content-02.png

Begin by scrolling to the bottom of the panel, and selecting the + button, and assigning your favorite hot-key (I decided to hold down the Control, Alt, and Super keys along with V since that seems most memorable to me), and have it run our function:

capturing-content-03.png

Of course, we’ll need to modify our function to use xclip. This isn’t install by default on Ubuntu (but neither is pandoc), so install those first:

sudo apt install xclip pandoc -y

Now, let’s create a function to call xclip, and like before, we ask for HTML, and if it fails, we’ll get it as regular text:

(defun ha/get-linux-clipboard ()
  "Return the clipbaard for a Unix-based system. See `ha/get-clipboard'."
  (destructuring-bind (exit-code contents)
      (shell-command-with-exit-code "xclip" "-o" "-t" "text/html")
    (if (= 0 exit-code)
        (list :html contents)
      (list :text (shell-command-to-string "xclip -o")))))

Since I want my same code to work with both my Mac and my Linux systems, I create a simple little dispatcher:

(defun ha/get-clipboard ()
  "Returns a list where the first entry is the content type,
either :html or :text, and the second is the clipboard contents."
  (if (eq system-type 'darwin)
      (ha/get-mac-clipboard)
    (ha/get-linux-clipboard)))

The rest of the code is same. See Gitlab for file updates.

Summary

My workflow proposal amounts to gathering data from a web browser, shell commands, and source code, and be able to fling it into my engineering notebook without switching out of that application.

Later, I will return to my notebook in Emacs and clean up and summarize my capturing. Once clean, the issues or knowledge I wish to share can then be easily exported from org.

The side-benefit, is that I automatically remind myself to clock in to my task.