Death to the Shell

Really I just included this slide as click-bait for gray-hairs to get up-voted on HackerNews or something like that.

In reality, I bring this subject and my project to start a discussion, and HackerNews is actually the last place to have that conversation.

Because in reality, the shell, like a biological virus, has subversively embedded part of its DNA into my DNA, and it has shaped the way I do.

As Emacsians, we have a significant love/hate relationship with shells. Just look at all the terminal emulators we have available, like shell, term, ansi-term, eshell, and vterm. (If you still haven’t embedded terminal inside your Emacs workflow, you probably haven’t seen vterm, as that seems to be an answer for you).

But like magit and dired, we have better options for just about anything the shell can do.

This talk covers both ways we use the shell, interactively and programmatically.

I have created a project to address this, but I want to get a warning: This project is for tinkerers only (at least, at this point).

Allow me to begin discussing the first idea by relating a story that happened the other day. Someone asked me to jump on a development system and restart the OpenStack service. Well, OpenStack is not a single service, but a bunch of Python scripts supporting web interfaces, and I can’t remember all of their names.

Since all of the services start with the word, openstack, I could quickly type this out into the shell:

for S in $(systemctl --all | grep openstack | sed 's/\.service.*//' | cut -c3-)
do
    systemctl restart $S
done

If you believe that I just banged that out, you’d be deluded.

Notice the subshell section. Converting data by chaining small executables is incredibly flexible, but it has a dark side…

While I suppose one could call the tee command, but typically the flow of data through pipes isn’t viewable. Yeah, threading data through pipes like this is similar to function calls, but in the shell, we don’t have a debugger to expose each step.

If our problem is large, we typically skirt the issue by writing the initial data to a file, editing it in a descent enough editor, and then use that file for the final component.

This gave me an idea…

The shell’s flexibility makes it powerful, but anything the shell can do, I’d claim that Emacs can do better.

So my first idea is to eschew transforming data through commands like grep and sed to transforming a buffer of data using Emacs functions.

Once I can transform the data, I would like to send that data to some command in flexible ways like the shell can. I mean, there is only one way to get textual data out of a command, but lots of ways to get data in to those commands. I’ve come up with five specific use cases that if I could answer sufficiently, that should address 80 to 90% of use:

• Data piped in through a command’s standard in is the biggest, and easiest to address, think of Emacs’ function, shell-command-on-region does this.
• The xargs command converts a stream of lines as command line parameters.
• I write for loops when I want to run command repeated with each line as some parameter somewhere on the command line call.
• Often I need to copy data to clipboard so that I can paste this into my bug tracking software or some chat program, etc.
• Finally, I may want visually inspect it.

I suppose everything I’ve state has been too esoteric, so it is time for a demonstration. What I’m about to show is primariy just a user interface, that is a thin wrapper around existing Emacs functionality. Very little is unique.

I’d like to reproduce my previous story in real time by going through the following steps:

• Get a list of all services: systemctl --all
• Filter to just the openstack service names.
• Get the status of each service.

Oh, and let’s do this all remotely, shall we?

Remember how I mention that we use shell in two distinct ways, interactively and programmatically. I’ve shown my idea for the first, let’s chat about the second.

While I’ve been thinking about this for many years, not long ago, I came across a Reddit Question:

To which I responded that as a Lisp, Emacs Lisp is great as integrating any good idea that comes along, and stealing good ideas has made this system from the twentieth century still useful and almost modern.

The old adage that Emacs is an operating system is pretty true, it is just an operating system of functions, and as such, it really good for the hacker. You want to either overwrite or just advice every function. Global variables, normally a bad thing when you are sharing code with others, turns out to be really easy to prototype ideas when it is just you.

But think about all the really great libraries and support that we have that are new. I didn’t have these in the 1980s:

• Common Lisp library
• Dash library
• S library for strings
• F Library for files
• Web-focused Libraries for JSON, Yaml, XML

But I think the most compelling argument for using Emacs functions instead of shell scripts is the improved user interface. I’m serious. Think about all those command line options to each command you use, and how no amount of tab completion helps. In emacs, we can easily bind functions to keys, use two great completing systems (Ivy or Helm) that can give you history of the arguments, just not the command.

Finally, think of using Hydra and Magit’s interface, Transient, to really have a flexible approach to running commands.

Shell scripts, especially small ones can be really readable, and they are good as running commands and connecting them with streams, however, any script that grows to any size seems to become completely unreadable. The other issue is that since everything is a string, we can have conversion issues into numbers or needing to escape white space. Sure, modern versions of the shell have arrays, but they’re pretty kludgy at best.

So, can we take the best of both Lisp and Shell scripts? If we could, this could be a win for us using Emacs, as calling a function is better than a script.

What should we bring over from shells? The format function is fairly good, but the shell, along with most modern languages, have variable substitution within strings. This seems more readable. Also, since the shell has embedded itself in my DNA, I am comfortable with mkdir and cp. Besides, since Emacs doesn’t have namespaces, our functions tend to be pretty lengthy, which often makes our code is little more difficult to read.

But when we do, we can get a slew of Lispy goodness, like data structures, better iterations, logical variable scoping, and more functions than you can shake a stick.

The magic starts with a couple of macros that will convert the forms within to be more script-like. I have five specific goals for how this macro should behave. Let me explain each of these in details.

Should be easy to call a shell command, using the shell form:

(piper-script
 (shell "ls /tmp"))

This should also work well with files that have spaces:

(piper-script
 (shell "ls" "~/Google Drive File Stream/My Drive/"))

Give the shell form function multiple strings, and it will assume that all strings are parameters, and not do any parsing, but this function will split single strings on spaces.

How about making an alias for the shell function name?

(piper-script
 ($ "ls /tmp"))

Looks like a prompt character.

I use the same technique described earlier of using a temporary buffer, and running shell commands such that the contents of the buffer become standard in to a shell command, and the results of the command replace the contents of the buffer for the next command. This allows an obvious pipe-like approach:

(piper-script
 ($ "ls" "-1" "/tmp")
 (| "grep Temp")
 (| "tr [a-z] [A-Z]"))

Keep in mind you can use any Emacs function that works with buffers.

File expansion should work as expected:

(piper-script
 ($ "ls" "~/.emacs.d/*.el"))

Keep in mind that the piper-script looks at every string, so to make this work, if an expansion fails to find matching files, I just return the original string.

While format is fine, string-variable substitution is more readable, plus I want to be able to insert both Emacs variables as well as environment variables:

(let ((some-var "fooey"))
  (piper-script
   (let ((nudder-var "chop"))
     (echo "$HOME/projects/${nudder-var}/${some-var}"))))

Should return:

"/home/howard/projects/chop/fooey"

I’m not convinced, but perhaps when converted shell script should have shorter, more shell-like function names. In others words, I could use the function grep instead of keep-lines. In this case, if grep were in quotes and passed to the shell function it would call out to the executable, and if not, to the Emacs function, so maybe this would be more confusing.

However, I do need to write some functions, like sudo is a let wrapper around a modification to convert the default-directory variable to reference Tramp.

Let me run through a couple of examples, and for each, I want to show my original shell script, and how I converted it.

First, let’s see if I can get the resident memory size of two running browsers.

While the ps has a lot of command line options, I’m going to use grep to look for the top-level process, that is, the ones where the parent process is 1, and I want to ignore all the extra service frameworks that Chrome starts:

ps -u $USER -o ppid,rss,command | egrep 'Chrome|Firefox' | grep '^ *1' | grep -v Frameworks

The most prominent advantage is the use of Emacs’ rx macro for the regular expression (which I pass to the keep-lines alias):

(let ((browsers      (rx (or "Chrome" "Firefox")))
      (session-leads (rx line-start (one-or-more blank) "1")))

  (piper-script
   ($ "ps" "-u $USER" "-o ppid,rss,command")
   (grep browsers)
   (grep session-leads)
   (grep-v "Frameworks"))))

I’ve installed Ubuntu on an old Macbook laptop, and I noticed that after closing the lid, the computer would repeated turn itself on and off. The reason is that some devices are set to wake up the system, and magic process entry can be used to both read the state of all these devices, but if you write the name of the device back to this process entry, it will turn it off.

Why yes, this does seem like a job for a script, eh?

My original script uses cat to expose all the entries, which text I can massage with a little pipe sequence. Eventually storing each enabled device in a variable, and writing the value of that variable back into the process entry.

The piper-script version starts by looking a lot like the original, but I use a Lisp-looking looping macro I call for and a function that converts all the lines in this temporary buffer into a list of strings for it to consume:

(piper-script
 (sudo
  (cat "/proc/acpi/wakeup")
  (grep "enabled")
  (replace-regexp " .*" "")
  (for (device (read-all-lines))
    (write-into device "/proc/acpi/wakeup"))))

The magic is obviously a nifty macro. I simply use a function from the dash library to analyze and possibly change every element I find in the forms given to it, without changing the structure of those forms:

(defmacro piper-script (&rest forms)
  "Runs the FORMS in a shell-like DSL."
  (cons 'with-temp-buffer
        (-tree-map #'piper--script-transform forms)))

Since all command works with a buffer, we create one, run through the converted forms, and not shown here, but we return the contents of the buffer as a string.

The followup magic happens with a function that returns a potentially changed version of the form. It is just a lengthy cond statement, but you can see that at first, I change strings, so that I can expand any wildcards or variable substitution. Then I just swap out various function names for their alias. The way it is currently written, I can’t use any variables with matching names as I don’t currently distinguish their position. Did I mention that this was a proof of concept?

(defun piper--script-transform (element)
  "Helper for `piper-script' to convert forms and strings. "
  (cond
   ((stringp element) `(piper-script-get-files ,element))

   ((eq element '$) 'piper-script-shell)
   ((eq element '|) 'piper-script-shell)

   ((eq element 'echo) 'piper-script-echo)
   ((eq element 'ifsh) 'piper-script-shell-if)
   ((eq element 'for) 'piper-script-loop)

   ((eq element 'cat) 'insert-file-contents)
   ((eq element 'touch) 'f-touch)
   ((eq element 'ln-s) 'f-symlink)   ; Perhaps make-symbolic-link
   ((eq element 'mkdir) 'f-mkdir)    ; ...

   ((eq element 'write-into) 'piper-script-write-into)
   ((eq element 'to-clipboard) 'piper-script-to-clipboard)
   ((eq element 'read-all-lines) 'piper-script-read-all-lines)
   ;; ...
   (t element)))

A fun little project that I’m ready to show the world, but remember that this won’t be helpful unless you are ready to help me hack on the Lisp code right now. However, it does potentially show how Emacs can improve both ways we use the shell, by interactively loading data into a buffer that can be easily manipulated and in an improved way, sent to other commands.

I personally think that Emacs Lisp is better than any shell script, and modern libraries help. Some features, like global variables, are condemned, but in your own environment, make Emacs hackable. However, a simple macro can create a DSL within Emacs that makes converting shell scripts easier.

If you want to check out the code base, pop over to emacs-piper project on Gitlab.