Why use EShell?

While diving into the source code of Emacs’ Shell (eshell), I found some gems and thought I’d share the top 10 reasons why you should give Eshell a second chance.

What do I mean about a second chance? Most of us try it out and become frustrated. Keep in mind, Eshell is … a shell, not a Terminal Emulator. If you are using programs that make heavy use of terminal control codes, like tmux or even more modern replacements, use vterm. If you are doing a lot of file manipulation, use dired. Otherwise, learning the intricacies of eshell is rewarding and fun.

Emacsians complain that eshell has poor documentation, but I believe that between the chapter on mastering the eshell (which happens to be free). and my deep dive (oh, and hints on the Internet), I do feel the information is available. This essay is an attempt to show how cool and interesting eshell can be.

During EmacsConf 2022, I gave a lightning talk with 10 minutes (actually 14) to present the top 10 Reasons why you Retry Eshell. But the talk ballooned, and I felt the content was too condensed to be useful. This essay should fill in a lot of the details (as well as links to the extension code).

#1 … It’s an Emacs REPL

Eshell is an Emacs Lisp REPL. So typing Lisp expressions works:

$ (+ 1 2 3)
6

What makes this interesting? The parens are somewhat optional:

$ + 1 2 3
6

#2 … It’s also a Shell

While eshell may look like a shell, like Bash, you should view it as a REPL with parenthesis-less s-expressions.

Makes sense, because a shell command with options, like:

ls -d /tmp

Looks like this s-expression:

(ls "-d" "/tmp")

#3 … You can Mix these Two Modes

Shells can call subshells, which return their output like a function call, like this Bash command:

$ rg red $(cat interesting-files.txt)

In this Eshell example, I use a text file with filenames as command line arguments to ripgrep. Notice I use braces to state that the content inside the braces to evaluate as an Eshell expression:

$ rg -i red { cat interesting-files.txt }

But eshell has another subshell-like feature … Emacs Lisp expressions. And you can mix them together. In my talk, I gave the following overly contrived example: ‘

$ setq afile banana.org   # Notice good ol’ setq
banana.org

$ setq others apple.org cantaloupe.org # This doesn’t work
$ echo $others
apple.org

# The function `listify` is like `list`, but for eshell sections:
$ setq some { listify apple.org cantaloupe.org }
("apple.org" "cantaloupe.org")

# We could also do this, but it require quotes:
$ setq some (list "apple.org" "cantaloupe.org" )
("apple.org" "cantaloupe.org")

$ cons afile others   # Not quite, because eshell mode treats
("afile" . "others")  # words as strings, not variables.

# But surrounding it in parens activates Eshell mode:
$ rg -i red (cons others some)

I love that you can use cons as an Emacs Lisp function, but rg (ripgrep) is an executable.

Remember the differences:

With (...), eshell treats it as Lisp, like the last line in my example.
With {...}, eshell follows these shell-like rules:
- If it looks like a number, eshell converts it to a number
- Otherwise, eshell converts it to a string (quotes, like a shell, groups words)
- What about this mix between functions and executables?
  - Calls functions that begin with eshell/ first
  - Next are executables on your $PATH
  - Then matching Lisp functions. You can switch this order (see the eshell-prefer-lisp-functions variable).

#4 … Emacs is better than Shell

If the following works, why would you call expr or bc or dc?

$ + 17 5 (* 10 2)
42

Why call less or more when you could call view-file?

alias less 'view-file $1'

With the view-file-mode, you exit with q, like with less, but you get take advantage of Emacs collection of file modes.

Improvement: View Multiple Files

The problem with view-file is it takes a single file as an argument. In a shell, we might want to view more than one.

Let’s make a solution to that:

(defun eshell-fn-on-files (fun1 fun2 args)
  "Call FUN1 on the first element in list, ARGS.
Call FUN2 on all the rest of the elements in ARGS."
  (unless (null args)
    (let ((filenames (flatten-list args)))
      (funcall fun1 (car filenames))
      (when (cdr filenames)
        (mapcar fun2 (cdr filenames))))
    ;; Return an empty string, as the return value from `fun1'
    ;; probably isn't helpful to display in the `eshell' window.
    ""))

This allows me to make a version of less that calls view-file on the first function given, but open in another window for each remaining file given:

(defun eshell/less (&rest files)
  "Essentially an alias to the `view-file' function."
  (eshell-fn-on-files 'view-file 'view-file-other-window files))

And then this alias:

(defalias 'eshell/more 'eshell/less)

#5 … Better Regular Expressions

Can’t remember regular expressions when calling grep or other search commands? Use the rx macro:

$ rg (rx (1+ hex) "-")

banana.org
8::ID: 460545f7-b993-42ec-9736-4276e8035f63

cabbage.md
4:Sed bibendum.  Fed-add etiam vel tortor sodales tellus ultricies commodo.  Aliquam erat volutpat.  Donec vitae dolor.  Pellentesque tristique imperdiet tortor.  Lorem ipsum dolor sit amet, consectetuer adipiscing elit.

apple.org
8::ID: 2f71822d-1128-4214-a040-5d2c98656220

cantaloupe.org
8::ID: 8f64466c-e7ab-4abc-a627-b38f009caac4

This uses ripgrep to almost find UUIDs in files in the current directory.

Wait, why is that paragraph showing? Ah, the word fed-add is two hex strings with a dash.

We could re-run the search, but add something like ID: to the expression, but we could also expand our regular expression capabilities…

Improvement: Non-Emacs Regexps

While the rx macro is freaking cool for Emacs Lisp … it doesn’t always translate to regular expressions accepted by most programs.

The pcre2el project can convert from a Lisp regular expression to a PCRE (Perl Compatible Regular Expression), acceptable by ripgrep.

Let’s create a new macro, called prx, that translates the output of the rx macro. Here is the complete snippet I use in my Emacs configuration file:

(use-package pcre2el
  :straight (:host github :repo "joddie/pcre2el")
  :config
  (defmacro prx (&rest expressions)
    "Convert the rx-compatible regular EXPRESSIONS to PCRE.
  Most shell applications accept Perl Compatible Regular Expressions."
    `(rxt-elisp-to-pcre (rx ,@expressions))))

Now we can use the rx expression of (= 8 hex) to mean 8 hexadecimal characters, to expand our a regular expression for accurate UUIDs:

$ rg (prx (seq (= 8 hex) "-" (= 3 (seq (= 4 hex) "-")) (= 12 hex)))

banana.org
8::ID: 460545f7-b993-42ec-9736-4276e8035f63

apple.org
8::ID: 2f71822d-1128-4214-a040-5d2c98656220

cantaloupe.org
8::ID: 8f64466c-e7ab-4abc-a627-b38f009caac4

Is that easier to read, write and remember than?

$ rg [[:xdigit:]]{8}-(?:[[:xdigit:]]{4}-){3}[[:xdigit:]]{12}

Improvement: Adding Keywords to Regexps

Why not extend the rx macro with your own favorite key words:

(defmacro prx (&rest expressions)
    "Convert the rx-compatible regular EXPRESSIONS to PCRE.
  Most shell applications accept Perl Compatible Regular Expressions."
    `(rx-let ((integer (1+ digit))
              (float   (seq integer "." integer))
              (b256    (seq (optional (or "1" "2"))
                            (regexp "[0-9]\\{1,2\\}")))
              (ipaddr  (seq b256 "." b256 "." b256 "." b256))
              (time    (seq digit (optional digit) ":" (= 2 digit) (optional ":" (= 2 digit))))
              (email   (seq (1+ (regexp "[^,< ]")) "@" (1+ (seq (1+ (any alnum "-"))) ".") (1+ alnum)))
              (date    (seq (= 2 digit) (or "/" "-") (= 2 digit) (or "/" "-") (= 4 digit)))
              (ymd     (seq (= 4 digit) (or "/" "-") (= 2 digit) (or "/" "-") (= 2 digit)))
              (uuid    (seq (= 8 hex) "-" (= 3 (seq (= 4 hex) "-")) (= 12 hex)))
              (guid    (seq uuid)))
       (rxt-elisp-to-pcre (rx ,@expressions)))))

Now our command would be:

$ rg (prx uuid)

#6 … Loops are Better with Predicates

Let’s say you want to remove the execute bit from files that have it.

In a shell like bash, you need a for loop and an if. For instance:

for F in *.org
do
  if [[ $(stat --format %A $F | cut -c4) = "x" ]]
  then
    chmod -x $F
  fi
done

With eshell, use a predicate filter to combine into a single loop. The (x) after a file glob, like *.org, filters for files marked as executable:

for F in *.org(x) {
  chmod -x $F
}

Improvement: Looping like a Map

Since we often type loops to execute a command, what about creating a function that can do this all in one go?

This do function splits the arguments on a double colon .. where the left side is a single statement to run, and the right side is a list of files.

It loops through each file, creating an eshell command with the file appended.

With this, I can remove the execute bit on all CSV files that have it.

do chmod -x :: *.csv(x)

Here is the code:

(defun eshell/do (&rest args)
  "Execute commands over lst."
  (seq-let (cmd lst) (-split-on "::" args)
    (dolist (file
              (flatten-list (append lst)))
      (add-to-list 'cmd file)
      (eshell-named-command
            (car cmd) (cdr cmd)))))

Now that I see that my example wasn’t good, as most commands, like chown, can accept multiple command line arguments.

You get the idea. I do have a larger version that allows me to use an _ to be replaced by the filename.

#7 … Output of Last Command

Most shells have the special variables $? for the exit code of the last command.

While reading through the source code, I noticed that $$ refers to the output of the last command.

$ echo "Hello World!"
Hello World!
$ echo $$ | tr o u
Hellu Wurld!

But it doesn’t always work:

$ ls
apple.org  banana.org  cantaloupe.org  interesting-files.txt
$ echo $$
$

That is because the call to ls returns t or nil for external commands.

Improvement: Fixing $$

After running every command, Eshell sets these variables:

eshell-last-input-start
eshell-last-input-end
eshell-last-output-start
eshell-last-output-end

I created a function that I hook into Eshell via the eshell-post-command-hook:

(add-hook 'eshell-post-command-hook 'ha-eshell-store-last-output)

This makes Eshell call my function, ha-eshell-store-last-output, after every command.

Using buffer-substring, I store the output from the buffer into a global variable:

(defun ha-eshell-store-last-output ()
  "Store the output from the last eshell command.
Called after every command by connecting to the `eshell-post-command-hook'."
  (let ((output
         (buffer-substring-no-properties eshell-last-input-end eshell-last-output-start)))
    (setq ha-eshell-output output)))

And extend Eshell’s special variables list:

(add-to-list 'eshell-variable-aliases-list '("$"  ha-eshell-output))

In my Emacs configuration, I turned this into a ring, so while $$ works, so does array sub-scripting on that variable:

$ ls
apple.org  asparagus.md  banana.org  broccoli.md  cabbage.md  cantaloupe.org  interesting-files.txt

$ rg ID *
apple.org
6::ID: AF1ED3CD-0E05-4909-A7D9-26AD12551FA3

banana.org
8::ID: 83f5b53e-7a79-48fe-90da-469610543f67

cantaloupe.org
8::ID: 1e715bd7-a504-4eed-9ef0-444585d3d028

$ echo $$ | rg org
apple.org
banana.org
cantaloupe.org

$ echo $$[1] | rg ID
6::ID: AF1ED3CD-0E05-4909-A7D9-26AD12551FA3
8::ID: 83f5b53e-7a79-48fe-90da-469610543f67
8::ID: 1e715bd7-a504-4eed-9ef0-444585d3d028

The code is a bit long, so I wrote the details in my Emacs configuration file.

#8 … Redirect Back to Emacs

The output of a command can go to kill-ring:

$ rg ID >/dev/kill

Or the clipboard (for pasting into other applications):

$ rg ID >/dev/clip

You can send that output into a buffer:

$ rg ID > #<new-scratch>

Or into an existing buffer where the point is:

rg ID >>> #<scratch>

If you are taking notes, this requires you to coordinate two buffers (placing your cursor point at the location where the output should go).

Improvement: Engineering Notebook

Create a capture template that takes a string (or if called interactively, the region) and that does an :immediate-finish after inserting that string (with the %i string) to a file:

(add-to-list 'org-capture-templates
             `("e" "Contents to Engineering Notebook" plain
               (file+datetree org-default-notes-file "Notes")
               "#+begin_example\n%i\n#+end_example\n"
               :immediate-finish t :empty-lines 1))

Create a wraper function to call org-capture-string:

(defun eshell-to-engineering-notebook (string)
  "Write STRING to our enginnering notebook using org-capture."
  (org-capture-string string "e"))

Add our new function to :

(add-to-list 'eshell-virtual-targets
             '("/dev/e" eshell-to-engineering-notebook nil))

Now the following redirect writes into your org-default-notes-file under today’s date:

$ rg ID > /dev/e

#9 … Use Emacs Buffers

Why leave the results of eshell commands in the *eshell* buffer? Send the output into a buffer where you can use it. Here’s a call to ripgrep:

rg --no-heading (prx email) user-access.csv > #<almost-grep>

When you switch to almost-grep buffer, turn on grep-mode, you can jump around as if you called grep instead.

Perhaps I’m good with ripgrep, but not good with cut or awk and unsure how to extract the email address from all the people in my database? First, send it to a buffer:

rg -IN (prx email) user-access.csv > #<names-data>

Now I can switch to this buffer, and edit the data directly using Emacs commands. If you want to know more about my easy column technique that I used during the presentation, the code is in my Emacs Configuration file.

Improvement: bcat

Eshell doesn’t do pipes from buffers as standard in. I think that is an over-sight. This might be useful:

(defun eshell/bcat (&rest args)
  "Output the contents of one or more buffers as a string. "
  ;; Convert args (either buffer refs or strings) into buffers:
  (let ((buffers (mapcar #'get-buffer args)))
    ;; Combine output from all buffers, separated by newlines:
    (mapconcat (lambda (buf)
                 (save-window-excursion
                   (switch-to-buffer buf)
                   (buffer-substring-no-properties (point-min) (point-max))))
               buffers "\n")))

So the following works to grab my email addresses that I extracted, and send them out to another program:

$ bcat #<names-data> | ~/bin/code-review -s 'Issue #47' 3e729973

If you’re interested, I have a more elaborate (and yet simpler) workflow surrounding sending data back and forth from Eshell to Emacs buffers I call Ebb and Flow. A function, ebb, copies the command output to a buffer, and an alias of my bcat expansion, flow, brings it back to Eshell.

For instance:

$ rg -i red  # a too-simple search expression
banana.org
15:  - Red

cabbage.md
12:  * Red

apple.org
21:  - Red Delicious

asparagus.md
4:Nullam tempus. Red bug by moonlight. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.

$ ebb

This ebb call brings up a buffer, *emacs-edit*, where I can trim that down to:

banana.org
cabbage.md
apple.org

Hitting the Q key closes that buffer, returning to eshell, where we can type:

 $ rg -i pink { flow }
apple.org
13:  - Pink Lady

#10 … You can cd to Remote Systems

This command uses SSH to jump to my host, goblin, start a root session, and start in the etc directory.

$ cd /ssh:goblin|sudo:goblin:/etc

Remember that Tramp can be finicky if you start blinging your remote hosts with oh my zshell, and whatnot, so your mileage may vary.

Summary

Use eshell if want to:

Quick way to run commands and Emacs functions as a REPL, or
Run an OS program and process the output with Emacs functions

Be careful if you need any of the following:

Complicate curses-like output
Shell programming features or advanced pipe
Remote systems without key-based access (no passwords)

Like Emacs, make Eshell your own. Feel free to steal my configuration: https://is.gd/hamacs_eshell

Your homework (if you choose) is to:

Grab the extended prx macro
See if you want to use my array version of $$ variable
Build an org-capture function
Use bcat to bring buffers into Eshell
Check out my ebb and flow functions

Did I mentioned that I’m on the birdlessland of Mastodon: @howard@emacs.ch