www.howardism.org
Babblings of an aging geek in love with the Absurd, his family, and his own hubris.... oh, and Lisp.

Presenting the Eshell

The following is a transcript of the slides and demonstration of EShell I gave to both the PDX and London Emacs User groups. Hopefully this page will be easier to copy/paste…

Introduction

John Wiegley created EShell in 1998:

…as a way to provide a UNIX-like environment on a Windows NT machine.

Part of Emacs since v21.

Personally?

  • Started with ksh
  • Used a lot of shells…
  • Tried eshell soon after its birth
  • Shelved it since it wasn’t shell-enough
  • Rediscovered years later
  • Finally got it

Contents: What’s all this then?

  • What EShell really is
  • How to use it effectively
  • Hacking

Shell… The Good

  • Can be immensely powerful… at times
  • Pipes and redirection are a staple
  • Utilizing small, focused text-oriented executables
  • Complex command re-invocation
  • History okay … nothing like Emacs

Shell… The Bad

  • Commands? Like key sequences, only longer
  • Needing completion to run commands?
  • Loops? Not terrible
  • Copy and pasting … with a mouse!? (at least use M-x shell)

Shell… The WTF?

  • Best part: extensibility!
  • But what an awful language:

    if [ $(echo "$IN" | cut -c 1-3 ) == 'abc' ]; then
       # ...
    fi
    
  • May be Turing complete, but so what.

    But, but, but… we know the shell!

Seen Rich Hickey’s Simple Made Easy talk? The shell is easy because it is so close.

iPython

Python REPL with shell-like features.

  • Understands a current directory
  • Has some shell-like commands, cat
  • Doesn’t easily execute programs: system
  • Executes Python scripts: run

Slide Notes

Demonstrated the iPython approach by entering the following into an ipython REPL:

Python 2.7.12 (default, Nov 19 2016, 06:48:10)
Type "copyright", "credits" or "license" for more information.

IPython 2.4.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 2 ** 60
Out[1]: 1152921504606846976

In [2]: cd /tmp/testing
/tmp/testing

In [3]: ls
pi.py  pi.rb  README  src/  tests/

In [4]: cat pi.py
import math

print(math.pi)
print(math.cos(math.pi))

In [5]: chmod a+x pi.py
  File "<ipython-input-5-b9e30b9b8e31>", line 1
    chmod a+x pi.py
          ^
SyntaxError: invalid syntax


In [6]: run pi
3.14159265359
-1.0

In [7]: system ruby pi.rb
Out[7]: ['-0.9999999999964793', '-3.141592653589793']

In [8]: cat README
Ciao, all you cool cats.
Oh, and hey to all my dawgs.

In [9]: def cat(arg=None):
   ...:     return 'Meow!'
   ...:

In [10]: cat
Out[10]: <function __main__.cat>

In [11]: cat()
Out[11]: 'Meow!'

In [12]: cat README
  File "<ipython-input-12-01ccd53eccb4>", line 1
    cat README
             ^
SyntaxError: invalid syntax

In answer to your question, I haven’t looked to see why we have an array when calling the system function.

If you want to see what shell-like functions iPython has, type % and hit the Tab key, which shows something like:

In [13]: %
Display all 122 possibilities? (y or n)
%%!                      %doctest_mode            %pfile
%%HTML                   %ed                      %pinfo
%%SVG                    %edit                    %pinfo2
%%bash                   %env                     %popd
%%capture                %gui                     %pprint
%%debug                  %hist                    %precision
%%file                   %history                 %profile
%%html                   %install_default_config  %prun
%%javascript             %install_ext             %psearch
%%latex                  %install_profiles        %psource
%%perl                   %killbgscripts           %pushd
%%prun                   %ldir                    %pwd
%%pypy                   %less                    %pycat
%%python                 %lf                      %pylab
%%python2                %lk                      %quickref
%%python3                %ll                      %recall
%%ruby                   %load                    %rehashx
%%script                 %load_ext                %reload_ext
%%sh                     %loadpy                  %rep
%%svg                    %logoff                  %rerun
%%sx                     %logon                   %reset
%%system                 %logstart                %reset_selective
%%time                   %logstate                %rm
%%timeit                 %logstop                 %rmdir
%%writefile              %ls                      %run
%alias                   %lsmagic                 %save
%alias_magic             %lx                      %sc
%autocall                %macro                   %store
%autoindent              %magic                   %sx
%automagic               %man                     %system
%bookmark                %matplotlib              %tb
%cat                     %mkdir                   %time
%cd                      %more                    %timeit
%clear                   %mv                      %unalias
%colors                  %notebook                %unload_ext
%config                  %page                    %who
%cp                      %paste                   %who_ls
%cpaste                  %pastebin                %whos
%debug                   %pdb                     %xdel
%dhist                   %pdef                    %xmode
%dirs                    %pdoc

EShell as a Shell

  • Most “interactive language” interfaces choose:
    • Language-specific REPL, or
    • Shell-focused program worker
  • As a shell:
    • Concept of a current directory
    • popd, pushd, and dirs
    • Globbing Expressions
    • Quotes often optional
      • Do you care about spaces?
      • Double and single quotes are interchangeable
    • Aliases: alias ll 'ls -l'
  • Emacs shell interaction:
    • M-n / M-p scroll through history
    • M-r select from history
    • C-c C-p move to previous prompts
    • C-c C-l list history in buffer
  • Tempted to think eshell is like shell

Slide Notes

At this point, we start an eshell process, and demonstrate some of the shell-like features that we’d expect for something that ends with -shell:

$ cd /tmp/testing

$ pwd
/tmp/testing

$ ls
README  pi.py  pi.rb  src  tests

$ cat README
Ciao, all you cool cats.
Oh, and hey to all my dawgs.

$ ruby pi.rb
-0.9999999999964793
-3.141592653589793

$ python *.py
3.141592653589793
-1.0

$ echo "Hello"
Hello

$ echo 'Hello'
Hello

$ echo Hello
Hello

$ alias ee 'find-file-other-window $1'

$ ee pi.rb
#<buffer pi.rb>

This last example shows that ee opens a window in another window. Note, however, that the alias is actually calling an Emacs function, not another executable (although it could).

EShell as a REPL

  • Lisp expressions work within parens
  • Unlike shell, EShell:
    • Commands can be executables or Emacs functions
    • Distinguishes strings, numbers, and lists
  • EShell is marriage of two syntax parsers:
    • Shell Expressions
    • Lisp Expressions
  • A single line can mix the two!

Slide Notes

To demonstrate how eshell is a REPL, let’s type a simple Lisp expression:

$ (length "hi")
2

$ length "hi"      # ← Works since shell parser calls Lisp
2

$ length hi        # ← Works since shell reads as string
2

$ (+ 1 3)
4

$ + 1 3            # ← Works since shell reads as number
4

$ * 3 (+ 1 2)      # ← Shows both shell and lisp parsers
9

$ ls
README  pi.py  pi.rb  src  tests

$ length *        # Globs return a list
5

$ length *.py
1

$ touch 'and go.py'

$ ls *.py
and go.py  pi.py

$ echo *.py        # ← More clear that globs are lists
("and go.py" "pi.py")

Eshell’s Parsers

  • Lisp parser:
    • ( ... )
    • $( ... ) … useful for string evaluation
  • Shell parser:
    • no parens … in other words, the default
    • { ... }
    • ${ ... } … useful for string evaluation
  • In shell parser, reference variables with $

Slide Notes

To drive home the differences between shell and lisp parsers, let’s enter the following in eshell buffer:

$ setq ANSWER 42   # ← Normal Emacs variable
42

$ numberp $ANSWER  # ← Use $ to get value.
t

$ setq UNANSWER "41"
41

$ stringp $UNANSWER
t

$ mod ANSWER 5    # ← Forgot the $ with shell parser
Wrong type argument: number-or-marker-p, "ANSWER"

$ mod $ANSWER 5   # ← Math without expr
2

$ (mod ANSWER 5)  # ← Lisp doesn't need $ for vars
2

$ (mod $ANSWER 5)
Symbol's value as variable is void: $ANSWER

$ echo $UNANSWER:$ANSWER
41:42

$ echo $UNANSWER:(mod ANSWER 5)   # ← Let's talk about predicate filters later in the show
Malformed modification time modifier `m'

$ echo $UNANSWER:$(mod ANSWER 5)
41:2

$ echo $ANSWER:${mod $ANSWER 5}
42:2

Shell-like Loops

  • Syntactic sugar around loop.
  • Code following in is a generate list
  • Use trailing { ... } for side-effects

Slide Notes

Show off the for concept, by entering the following in an eshell buffer. Note that the do syntax for some shells doesn’t work. Loops look more like csh’s:

$ for F in *; do echo "I like $F"; done
Symbol's value as variable is void: do
done: command not found

$ for F in * { echo "I like $F" }
I like README
I like and go.py
I like pi.py
I like pi.rb
I like src/
I like tests/

# A list can be generated in any way, like with Lisp:
$ for N in (number-sequence 1 5) { % $ANSWER $N }
0
0
0
2
2

# Generate the list with eshell parser syntax:
$ for N in {number-sequence 1 5} { % $ANSWER $N }
0
0
0
2
2

# Note: Can't use Lisp as the loop's action:
$ for N in {number-sequence 1 5} (% ANSWER N)

# Unless you embed the Lisp in shell parser:
$ for N in {number-sequence 1 5} {(% ANSWER N)}
0
0
0
2
2

Function or Executable?

What about the executable find vs. Emacs’ find function?

Precedence Order:

  1. Eshell aliases
  2. Emacs functions that being with eshell/ prefix
  3. Normal Emacs functions (don’t need to be interactive)
  4. Shell executables

Of course, this is customizable:

  • eshell-prefer-lisp-functions prefer Lisp functions to external commands
  • eshell-prefer-lisp-variables prefer Lisp variables to environmentals

Slide Notes

To demonstrate the precedence order for eshell commands, I created a script called foobar that simply contains:

#!/bin/sh

echo "Called: executable"

Without anything else, this will be called:

$ which foobar
/home/howard/bin/foobar

$ foobar
Called: executable

We now create a regular Emacs function in Lisp (notice that it isn’t interactive):

(defun foobar ()
  "Called: function")

It now takes precedence over the executable:

$ which foobar
foobar is a Lisp function

$ foobar
Called: function

Create another Lisp function, this has the eshell/ prefix. Again, no need to make interactive:

(defun eshell/foobar ()
  "Called: eshell function")

And this new function over-shadows the others:

$ which foobar
eshell/foobar is a Lisp function

$ foobar
Called: eshell function

Finally, we define an alias, and demonstrate that it over-shadows all the others:

$ alias foobar 'echo "Called: alias"'

$ which foobar
foobar is an alias, defined as "echo "Called: alias""

$ foobar
Called: alias

Globbin’ Filters

  • The * glob-thing has filters
  • Great if you can remember the syntax:
    • . for files
    • / for directories
    • r if readable
    • w if writable
    • L filtering based on file size
    • m filtering on modification time
  • The filters can be stacked, e.g. .L
  • Can’t remember? C-c M-q Or: eshell-display-predicate-help

Slide Notes

Using a directory for this purpose, we can demonstrate EShell’s predicate filter feature. First, list all files:

$ ls *(.)
README  and go.py  pi.py  pi.rb

$ ls *(^/)   # ← Inverse of directories are often files
README  and go.py  pi.py  pi.rb

Demonstrate combining modifiers by listing all files with more than 50 bytes to them:

$ ls *(.L+50)
README  pi.py

After creating three files (using the touch executable), we can list all empty files (that is, those that have less than 1 byte):

$ ls *(L-1)
and go.py  goo.py  grip.py  swam.py

Or those modified less than 40 seconds ago:

$ ls *(.ms-40)
README  and go.py  goo.py  grip.py  pi.py  pi.rb  src  swam.py  tests

Modified after we modified goo.py:

$ ls *(.m-'goo.py')
grip.py  swam.py

And before we modified goo.py:

$ ls *(.m+'goo.py')
README  and go.py  pi.py  pi.rb

I love this. I can get a list of my journal entries larger than 5000 bytes, and open dired showing only those files:

$ dired ~/journal/2017*(L+5000)

Modifiers

Syntactic sugar to convert strings and lists.

Can’t remember? C-c M-m Or: eshell-display-modifier-help

Eshell filters and modifiers remind me of regular expressions

Don’t know the eshell-way? Just drizzle Lisp.

Slide Notes

Convert a string with a modifier:

$ echo "hello"(:U)
HELLO

Convert all files, as strings, in a list:

$ echo *(:U)
("README" "AND GO.PY" "GOO.PY" "GRIP.PY" "PI.PY" "PI.RB" "SRC/" "SWAM.PY" "TESTS/")

Modifiers can be combined with filters:

$ echo *(.L-1:U)
("AND GO.PY" "GOO.PY" "GRIP.PY" "SWAM.PY")

However, we often split these and use the for loop:

$ for F in *(.L-1) { mv $F $F(:U) }

$ ls
AND GO.PY  GOO.PY  GRIP.PY  README  SWAM.PY  pi.py  pi.rb  src  tests

Now, all empty files are in upper case.

You’d think you could reverse a list with:

$ echo ("hello" "cruel" "world")(:R)
No matches found: ("hello" "cruel" "world")

Since the shell parser doesn’t like that syntax, perhaps you could set the list to a variable and work with that?

$ setq BADDABING (list "hello" "cruel" "world")
("hello" "cruel" "world")

$ echo $BADDABING(:R)
("world" "cruel" "hello")

If you find this stuff too odd and confusing, you can always fall back to Lisp:

$ reverse (list "hello" "cruel" "world")
("world" "cruel" "hello")

EShell Hack Points

While offering similar shell experience, Eshell is really hackable!

Here are some ideas…

Write your Own Functions

  • Functions for Eshell: eshell/
  • They do not need to be interactive
  • Functions should assume &rest for arguments:

    (defun eshell/do-work (&rest args)
      "Do some work in an optional directory."
      (let ((some-dir (if args
                           (pop args)
                        default-directory)))
        (message "Work in %s" some-dir)))
    

Slide Notes

Using &rest allows your functions to behave more like shell functions:

$ do-work
Work in /tmp/testing/

$ do-work /home/howard/bin
Work in /home/howard/bin

Remote Connections

To have eshell work on a remote server:

(let ((default-directory "/ssh:your-host.com:public/"))
  (eshell))

My personal project:

  • Connect to my hypervisor controller
  • Download and store a list of virtual machines
  • Use ido-completing-read to select a host / ip
  • Generate a Tramp URL for default-directory
eshell-present-fav-hosts.png

Slide Notes

Here is a simplified example that might be a helpful start:

(defvar eshell-fav-hosts (make-hash-table :test 'equal)
  "Table of host aliases for IPs or other actual references.")

(puthash "web-server" "172.217.4.14" eshell-fav-hosts)
(puthash "slc-jumpbox" "10.93.254.176" eshell-fav-hosts)
;; ...

(defun eshell-favorite (hostname &optional root dir)
  "Start an shell experience on HOSTNAME, that can be an alias to
a virtual machine from my 'cloud' server. With prefix command,
opens the shell as the root user account."
  (interactive
   (list
    (ido-completing-read "Hostname: "
                         (hash-table-keys (eshell-fav-hosts)))))

  (when (equal current-prefix-arg '(4))
    (setq root t))
  (when (not dir)
    (setq dir ""))

  (let* ((ipaddr (gethash hostname eshell-fav-hosts hostname))
         (trampy (if (not root)
                     (format "/ssh:%s:%s"       ipaddr dir)
                   (format "/ssh:%s|sudo:%s:%s" ipaddr ipaddr dir)))
         (default-directory trampy))
    (eshell)))

Extending Predicates

The User predicate (U) could have been written:

(defun file-owned-current-uid-p (file)
  (when (file-exists-p file)
      (= (nth 2 (file-attributes file))
         (user-uid))))

Then add it:

(add-hook 'eshell-pred-load-hook (lambda ()
  (add-to-list 'eshell-predicate-alist
           '(?U . 'file-owned-current-uid-p))))

My engineering notebook is a directory of files.

Most of my files have #+TAGS entries.

I can filter based on these tag entries.

I have to parse text following predicate key.

Slide Notes

Assuming the pi.py script is owned by the root user, we can show which ones I own:

$ ls *(.U)
AND GO.PY  GOO.PY  GRIP.PY  README  SWAM.PY  pi.rb

And which ones I don’t:

$ ls *(^U)
pi.py

My engineering notes contains quite a few files:

$ length ~/technical/*(.)
598

For instance, one file in my engineering notebook starts with:

#+TITLE:  Perfect Square
#+AUTHOR: Howard Abrams
#+EMAIL:  howard.abrams@gmail.com
#+DATE:   2013 Jun 06
#+TAGS:   programming clojure

Haitao posed a question: How do you write a function that determines
if a number is a perfect square.  You know, 9 and 25 are both perfect
squares because their square roots are natural integers.

While I have a brute force approach with an imperative loop in my
head, I'm curious if I could do it with Lisp...

Notice the line that starts with #+TAGS: … I can get the files that contain a word on this line, with my new T predicate:

$ length ~/technical/*(.T'clojure')
45

Replacing Pipes

Pipes for shell are flexible, but…

  • Shell’s text processing is limited
  • Need arsenal of tiny, cryptic programs
  • Re-run many times since debug pipe steps

Emacs is pretty good at text processing

  • keep-lines / flush-lines instead of grep
  • replace-string, et. al instead of sed

In EShell, redirect output to Emacs buffer:

$ some-command > #<buffer buf-name>

After editing the buffer, use it:

$ bargs #<buf-name> mv % /tmp/testing

Reference buffers as #<buf-name> with:

(setq eshell-buffer-shorthand t)

Or use keybinding, C-c M-b

Slide Notes

To demonstrate this feature, I first put a string in a new buffer, fling:

$ echo hello > #<buffer fling>

And displayed the buffer contents just to be sure.

Next, let’s overwrite the contents of that buffer:

$ ls -1 > #<buffer fling>

Now I call keep-lines to choose only the python files, and flush-lines to remove all files that contain go.

To show that I can now get the remaining files, I pass them to echo with:

$ bargs #<buffer fling> echo
("GRIP.PY" "SWAM.PY" "pi.py")

I might actually do something like this with the function (notice the % character will be substituted with the list of files):

$ mkdir oddities

$ bargs #<buffer fling> mv % oddities

$ ls oddities
GRIP.PY  SWAM.PY  pi.py

Bargs Code

Initial implementation of bargs:

(defun eshell/-buffer-as-args (buffer separator command)
  "Takes the contents of BUFFER, and splits it on SEPARATOR, and
runs the COMMAND with the contents as arguments. Use an argument
`%' to substitute the contents at a particular point, otherwise,
they are appended."
  (let* ((lines (with-current-buffer buffer
                  (split-string
                   (buffer-substring-no-properties (point-min) (point-max))
                   separator)))
         (subcmd (if (-contains? command "%")
                     (-flatten (-replace "%" lines command))
                   (-concat command lines)))
         (cmd-str  (string-join subcmd " ")))
    (message cmd-str)
    (eshell-command-result cmd-str)))

(defun eshell/bargs (buffer &rest command)
  "Passes the lines from BUFFER as arguments to COMMAND."
  (eshell/-buffer-as-args buffer "\n" command))

(defun eshell/sargs (buffer &rest command)
  "Passes the words from BUFFER as arguments to COMMAND."
  (eshell/-buffer-as-args buffer nil command))

EShell Summary

  • Advantages:
    • Similar shell experience between operating systems
    • Much more extendable, hackable and funner
  • Disadvantages:
    • Pipes go through Emacs buffers… not efficient
    • Programs that need special displays:

      (add-to-list 'eshell-visual-commands "top")
      

      My goal was to inspire potential hackery…

Questions?