Ebb and Flow with Emacs’ Shell

Let’s suppose you are running commands in a shell, as one does, and you type something like the following, where the output contains a little snippet of data you need for follow-up commands. In this case, I need the ID of the Trackpad:

$ usbreset
Usage:
  usbreset PPPP:VVVV - reset by product and vendor id
  usbreset BBB/DDD   - reset by bus and device number
  usbreset "Product" - reset by product name

Devices:
  Number 001/006  ID 05ac:8289  Bluetooth USB Host Controller
  Number 001/003  ID 05ac:0262  Apple Internal Keyboard / Trackpad
  Number 001/002  ID 0a5c:4500  BRCM20702 Hub

Why sure, you could use the mouse to select the goodies (what, and leave the keyboard?) or retype the command and extract the data using a series of commands, e.g.

ID=$(usbreset | grep Trackpad | cut -d' ' -f4)
sudo usbreset $ID

But I would like to show a few other ways one can collect that data … all without using the mouse.

But what if you wanted to copy more than a single symbol, or even a list of various data objects?

Since my talk at last year’s EmacsConf, I would like to expand on one of the ideas and elaborate on my ebb and flow functions. Let’s continue using my example above.

After typing the command, usbreset, I type ebb to collect the output from the previous command and put it in an Emacs editor buffer. I then use editor commands to whittle away until the buffer contains only:

05ac:0262

I then return back to the shell, and type:

sudo usbreset { flow }

Where the { flow } sequence is replaced by the contents of that buffer I edited.

How can command-line executables, with limited interaction to the Terminal emulation program, extract and push data to an editor?¹ The ebb command is actually an Emacs function that works in Emacs’ shell, eshell.² Commands typed into eshell, first attempt to execute Emacs’ own functions before popping out to the file system to look for executables, and Emacs functions have access to Emacs buffers.

Once the output from the ebb command is in an Emacs buffer, I use standard Emacs commands, and including my fancy data-focused functions to extract the data and filter the data. For instance, let’s start with the buffer containing the command mentioned above:

Usage:
  usbreset PPPP:VVVV - reset by product and vendor id
  usbreset BBB/DDD   - reset by bus and device number
  usbreset "Product" - reset by product name

Devices:
  Number 001/006  ID 05ac:8289  Bluetooth USB Host Controller
  Number 001/003  ID 05ac:0262  Apple Internal Keyboard / Trackpad
  Number 001/002  ID 0a5c:4500  BRCM20702 Hub

I call the keep-lines function, which is like grep in that only matching lines are kept, which I bind to SPC d l k … the space is my leader key for calling functions in normal mode (or normal state in Evil parlance).³ After giving the keep-lines function the regular expression of Trackpad, I am left with the following contents in my Emacs buffer:

Number 001/003  ID 05ac:0262  Apple Internal Keyboard / Trackpad

Next, I call a function (which I bind to SPC d t k for keeping columns in a table). This function prompts for a separator character, which I give a space. It then asks for the columns to keep, and I just type 4 (yes, very similar to the cut shell command), which leaves my buffer containing:

05ac:0262

The buffer that my ebb function creates is a little special, as it supports a minor mode, ebbflow-mode, which supplies a keybinding, C-x C-q (or just Q in normal state) to return to my Eshell.

I can now type another command, using the flow function to bring the data I extracted (in my example, the ID of the Trackpad), as a value for a call to the usbreset command:

$ sudo usbreset { flow }
Resetting Apple Internal Keyboard / Trackpad ... ok

Instead of `..` (backticks) or $(…) to run a command and substitute the results as a string, in Eshell, this is done by surrounding the command in braces.

Let’s get all the IDs from a list of images so that I can delete the images. First, we call the openstack command and then call my ebb function to take the table and place it into a buffer:

$ openstack image list
+--------------------------------------+---------------------------------+--------+
| ID                                   | Name                            | Status |
+--------------------------------------+---------------------------------+--------+
| dfc1dfb0-d7bf-4fff-8994-319dd6f703d7 | cirros-0.3.5-x86_64-uec         | active |
| a3867e29-c7a1-44b0-9e7f-10db587cad20 | cirros-0.3.5-x86_64-uec-kernel  | active |
| 4b916fba-6775-4092-92df-f41df7246a6b | cirros-0.3.5-x86_64-uec-ramdisk | active |
| d07831df-edc3-4817-9881-89141f9134c3 | myCirrosImage                   | active |
+--------------------------------------+---------------------------------+--------+
$ ebb

The keep-lines function allows me to keep only the lines with IDs, by typing active into the regular expression prompt. The buffer now looks like:

| dfc1dfb0-d7bf-4fff-8994-319dd6f703d7 | cirros-0.3.5-x86_64-uec         | active |
| a3867e29-c7a1-44b0-9e7f-10db587cad20 | cirros-0.3.5-x86_64-uec-kernel  | active |
| 4b916fba-6775-4092-92df-f41df7246a6b | cirros-0.3.5-x86_64-uec-ramdisk | active |
| d07831df-edc3-4817-9881-89141f9134c3 | myCirrosImage                   | active |

Next, I use my table-oriented functions, that act a lot like the cut command. I specify a separator of | and a field to keep, 1, and I end up with a buffer containing only the IDs:

dfc1dfb0-d7bf-4fff-8994-319dd6f703d7
a3867e29-c7a1-44b0-9e7f-10db587cad20
4b916fba-6775-4092-92df-f41df7246a6b
d07831df-edc3-4817-9881-89141f9134c3

Mission accomplished. I return back to the Eshell buffer with C-c C-q, and use the flow function to insert that list of IDs as an option to another openstack command:

$ openstack image delete { flow }

A lot of commands now have the ability to output JSON data, which can then be filtered using the jq command. I created a wrapper function around jq, where I can take the JSON output, and repeatedly whittle it down to the parts I need. For instance, at work, I have a web service that we call from the command line that returns a JSON response. For instance:

$ portal list images
{"payload": {"data": {"links": [ { "ref":
"6bdbf25a-7008-4a38-83f1-5a21a6a5358c", "name": "fusce-sagittis",
"created": "1694058475"}, { "ref":
"b64b1912-a5ad-4423-ac0d-2a26f8d769ca", "name":
"libero-molestie-mollis", "created": "1694062142"}]},
// ...snip...
"results": "success"}}

To extract the list of UUIDs, I would call the command multiple times, passing it into jq until I have it right. I wrote a wrapper function that allows me to whittle the data, section by section (as long as the results are still valid JSON, I can call it again with the contents of the buffer).

For instance, after calling ebb —mode json, I could call my ha-json-buffer-to-jq function, and give it the query, .payload, and have the buffer show this:

{
  "data": {
    "links": [
      {
        "ref": "6bdbf25a-7008-4a38-83f1-5a21a6a5358c",
        "name": "fusce-sagittis",
        "created": "1694058475"
      },
      {
        "ref": "b64b1912-a5ad-4423-ac0d-2a26f8d769ca",
        "name": "libero-molestie-mollis",
        "created": "1694062142"
      },
      // ...snip...
    ]
  },
  "results": "success"
}

Now, I can see I need to call it again, with a query, .data to remove more:

{
  "links": [
    {
      "ref": "6bdbf25a-7008-4a38-83f1-5a21a6a5358c",
      "name": "fusce-sagittis",
      "created": "1694058475"
    },
    {
      "ref": "b64b1912-a5ad-4423-ac0d-2a26f8d769ca",
      "name": "libero-molestie-mollis",
      "created": "1694062142"
    },
    // ...snip...
  ]
}

Now, I can see it clearly, so I call my function one more time, with .links[].ref, to get this:

"6bdbf25a-7008-4a38-83f1-5a21a6a5358c"
"b64b1912-a5ad-4423-ac0d-2a26f8d769ca"
// ...snip...

Quick global replace to delete all quotes, and I can now return to my Eshell buffer to type:

$ portal image delete { flow }

I hope this gives an idea of the versatility of using an Editor to manipulate complicated data while still in a shell session.

The reason why my ebb and flow functions are better than re-running a command with pipes, or (heaven forbid) using a mouse to copy/paste, are:

1. The ebb function takes the output from the Eshell buffer window, so I don’t have to re-run a potentially long command, just to get its output.
2. The ebb function can be given text to add to the buffer, which means, command sequences like ebb { rg foobar } works as you’d expect.
3. While I’m pretty good at using grep, sed, tr, cut, etc. (I’ve had decades of practice), I am better at using my editor.
4. Debugging piped commands in the shell is difficult, since you can’t see the data going through the pipes, but in an editor window, I can, as I am initiating each change.
5. If I make a mistake in filtering my data, I can easily undo the change … we expect that from our editors.

6. The ebb function can combine the data from many different commands into my editing session. For instance, typing these two commands:

   ebb { rg foo }
   ebb -e { rg bar }
   

   Has the same effect as ebb { rg foo ; rg bar }

7. The ebb function can specify what type of data is being sent to my editing session, i.e. ebb -m json

I currently haven’t incorporated these functions into a package, as the source code only lives in my Emacs Init Files, however, if this essay has piqued your curiosity, perhaps I will. Let me know your thoughts on Mastodon, @howard@emacs.ch.