Death to the Shell

I hate the Shell

— Howard Abrams @howardabrams

Well, I kinda hate the Shell

As Emacsians, we have a love/hate relationship with shells.

shell, term, ansi-term, eshell, vterm, oh my…
Like dired, we have better options

Agenda

I have two shell-related itches to scratch related to both shell uses:

Interactive Data transformations through pipes are good and bad.
Automative Personally, I want to replace 20+ years of shell scripts with Emacs Lisp.

Work in Progress

This project is for tinkers only (at least, at this point).

Let’s Start with a Story

Hey Howard, can you restart the Openstack service for me?

Uh, you mean, services?

for S in $(systemctl --all | grep openstack | sed 's/\.service.*//' | cut -c3-)
do
    systemctl restart $S
done

That was easy to bang out, right?

Converting data by chaining small executables is incredibly flexible.

Bad Parts of Pipes

The flow of data through pipes is inscrutable.

Similar to function calls: a | b | c → (c (b (a)))

Good thing we can use an editor to do the real dirty work, eh?

$ systemctl --all > /tmp/all-services
$ emacsclient /tmp/all-services
$ systemctl restart $(cat /tmp/all-services)

Wait a minute…

Merging Shell Pipes and Emacs

In the shell, we use commands, editors, and scripts based on personal preference.

This flexibility gives a shell its power.

But Emacs has this same flexibility.

First Idea:

Transform data flows through pipes from command to command to Transform a buffer of data from function to function

Sending Data to Commands

Transforming data in an Emacs buffer is easy.

How do we use that data in the shell?

As standard in to a command
As a series of command line arguments (aka xargs)
Repeatedly run command with line as argument (aka for loop)
Copy data to clipboard
Just to visually inspect it (aka less)

Demonstration: Piper

Primarily a user interface
Thin wrapper around existing Emacs functions

Steps:

Cat /proc/acpi/wakeup
Pair it down to just the enabled device names
Write each device back to /proc/acpi/wakeup

Steps:

Get a list of all services: service --status-all
Filter to just the service names.
Get a description of each service.

Note: We can talk about a better name later. ☻

Scripts

First way we use the shell is interactively.

The second way is programatically … you know, scripting.

Now, Let’s talk about the second way.

Reddit Question

And responded …

Emacs Lisp as Go To Language?

Emacs is an ecosystem of Lisp functions
Emacs Lisp is really hackable
As a Lisp, you can port great ideas:
- Common Lisp library
- Dash library
- String library
- Web-focused Libraries for JSON, Yaml, XML
Good user interface:
- Bind your program to a key chord/sequence
- Completing read with Ivy or Helm
- Hydra and Transient

Shell Scripts are both good and bad

Good Bad —————————————— ———————————————————

Can be readable • Can be unreadable
Executing programs • Hopefully no spaces in your filenames
Pretty good at data streams • Awful at data structures

Can we take the best of Lisp and Shell scripts?

As an Emacsian, calling a function is better than a script!

Maybe

What could we bring over from shells into Emacs Lisp?

Variables in strings:
Short, iterative, commands:

What could we bring over from Emacs Lisp?

Data structures
Better iterations
Better variable scoping
Functions, functions and more functions!

Desired Features

We will create a couple of macros to make Lisp look more scripty:

(piper-script
   @forms)

(defpiper-script name (params)
   :configuration
   @forms)

Shell Executables should just Work

(piper-script
 (shell "ls /tmp"))

And:

(piper-script
 (shell "ls" "/tmp"))

How about a short-cut?

(piper-script
 ($ "ls /tmp"))

Pipes and Data Transformations

Call a series of executable commands:

(piper-script
 ($ "ls" "-1 /tmp")
 (| "grep Temp")
 (| "tr [a-z] [A-Z]"))

Returns the string:

TEMP-19531244-A27D-4890-8D90-D6FDA91B6147
TEMP-F3C3C3F6-029D-442E-9441-6C676CEC3B62

The pipes between standard in and out, is just a temporary Emacs buffer.

Any other Emacs function can work too.

Wildcard Expansion

File expansion should work as expected:

(piper-script
 ($ "ls" "~/.emacs.d/*.el"))

Every string inside the piper-script automatically get converted with file-expand-wildcards.

(defun piper-expand (s)
  "Returns result of `file-expand-wildcards' if non-nil, otherwise, returns S."
  (or (file-expand-wildcards s) (list s)))

Embedded Strings

While format is fine, string-variable interpolation is more readable:

(let ((some-var "fooey"))
  (piper-script
   (let ((nudder-var "chop"))
     (echo "$HOME/projects/${nudder-var}/${some-var}"))))

Should return:

"/home/howard/projects/chop/fooey"

Notice we didn’t need to call get-env for the HOME value either.

Shell-Like Commands?

To make converting shell script to Emacs functions, should:

Write keep-lines as grep?
Write flush-lines as grep-v?
Use make-directory or mkdir?

I would like sudo to be a let wrapper around a modification to convert default-directory.

First Example

Let’s see process information about Google Chrome and Firefox…

My Original Script

Yeah, ps is flexible, but never flexible enough:

ps -u $USER -o ppid,rss,command | egrep 'Chrome|Firefox' | grep '^ *1' | grep -v Frameworks

Ya gotta love a series of regular expressions …

New Piper Script

Ooo, let’s use Emacs’ rx macro!

(let ((browsers      (rx (or "Chrome" "Firefox")))
      (session-leads (rx line-start (one-or-more blank) "1")))

  (piper-script
   ($ "ps" "-u $USER" "-o ppid,rss,command")
   (grep browsers)
   (grep session-leads)
   (grep-v "Frameworks"))))

Currently returns the string:

1 361356 /Applications/Firefox 2.app/Contents/MacOS/firefox
1 195164 /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Second Example

My laptop will turn on regularly after I close the lid.

Fix is simple, cat the “file” /proc/acpi/wakeup:

Device     S-state   Status   Sysfs node
P0P2         S3    *enabled  pci:0000:00:01.0
PEG1         S3    *disabled
EC   S4    *disabled  platform:PNP0C09:00
GMUX         S3    *disabled  pnp:00:03
...
RP05         S3    *disabled  pci:0000:00:1c.4
XHC1         S3    *enabled  pci:0000:00:14.0
ADP1         S4    *disabled  platform:ACPI0003:00
LID0         S4    *enabled  platform:PNP0C0D:00

Write the device name back to the “file” to toggle it:

echo LID0 > /proc/acpi/wakeup

My Original Script

The following script disables any ’device events’ that can wake my laptop:

cat "/proc/acpi/wakeup" | grep "enabled" | sed "s/ .*//" | while read DEVICE
do
  echo $DEVICE > /proc/acpi/wakeup
done

New Piper Script

Re-written with piper-script:

(piper-script
 (sudo
  (cat "/proc/acpi/wakeup")
  (grep "enabled")
  (replace-regexp " .*" "")
  (dolist (device (read-all-lines))
    (write-into device "/proc/acpi/wakeup"))))

Show us the Magic

The magic is obviously a nifty macro.

(defmacro piper-script (&rest forms)
  "Runs the FORMS in a shell-like DSL."
  (cons 'with-temp-buffer
        (-tree-map #'piper--script-transform forms)))

Since all command works with a buffer, we create one.

Next, we potentially change every element in the tree…

Piper Script Transform

(defun piper--script-transform (element)
  "Helper for `piper-script' to convert forms and strings. "
  (cond
   ((stringp element) `(piper-script-get-files ,element))

   ((eq element '$) 'piper-script-shell)
   ((eq element '|) 'piper-script-shell)

   ((eq element 'echo) 'piper-script-echo)
   ((eq element 'ifsh) 'piper-script-shell-if)
   ((eq element 'touch) 'piper-script-touch)

   ((eq element 'ln-s) 'make-symbolic-link)
   ((eq element 'mkdir) 'make-directory)
   ((eq element 'cat) 'insert-file-contents)
   ((eq element 'write-into) 'piper-script-write-into)
   ((eq element 'to-clipboard) 'piper-script-to-clipboard)
   ((eq element 'read-all-lines) 'piper-script-read-all-lines)
   ;; ...
   (t element)))

Summary

Transforming data from standard in to standard out is better in an Emacs buffer
Sending data to an executable can be improved:
- for loops
- xargs, etc.
Any Lisp is better than Shell’s language:
- Emacs Lisp is pretty hacky, and that’s a good thing
- Calling Emacs functions is trivial and pretty flexible
- Making Emacs Lisp as your go to scripting language is pretty fun

Discussion

Some potential sources for inspiration:

Should we come up with a memorable acronym?

SEAS
Stitching Emacs and Shell

HORSE
Howard’s Obvious Replacement of the Shell with Emacs
https://en.wikipedia.org/wiki/List_of_bagpipers
https://en.wikipedia.org/wiki/List_of_flautists
something related to the https://en.wikipedia.org/wiki/Hermit_crab , which leaves its shell
something related to the https://en.wikipedia.org/wiki/Pied_Piper_of_Hamelin
something related to plumbing or pipefitting
see if https://en.wikipedia.org/wiki/Piper triggers anything
https://en.wikipedia.org/wiki/Gheorghe_Zamfir , pan flautist with a place in the popular consciousness
translate “piper” to other languages:
- Scots gaelic: pìobaire
- Irish: píopaí
- Galician: gaiteiro
- Arabic: zamar, mizmari

(defun emacs-piper-presentation-reset ()
  (interactive)
  (setq piper--command-history '())
  (setq history-delete-duplicates t)
  (add-to-history 'piper--command-history "cut -f1")
  (add-to-history 'piper--command-history "service --status-all"))

(use-package demo-it
  :load-path "~/Other/demo-it"
  :config
  (demo-it-create :advanced-mode :single-window
                  (demo-it-presentation (buffer-file-name) 3 :both)))