www.howardism.org
Babblings of an aging geek in love with the Absurd, his family, and his own hubris.... oh, and Lisp.

Examples of My Literate Devops

This file contains various examples of my Literate Devops ideas. Viewing an exported version of this file won’t be nearly as helpful as downloading the org-mode formatted file.

Linking Results between Two Source Code Blocks

This example in this section demonstrates how the output from one code block can be used as values to another code block.

First, we call ls to get a list of files in a directory, and name them as orgfiles:

ls -lok *.org
-rw-r–r–@ 1 howard 8497 Dec 24 10:43 eshell-fun.org
-rw-r–r–@ 1 howard 2976 Dec 24 10:43 eshell.org
-rw-r–r–@ 1 howard 5007 Dec 24 10:43 getting-started.org
-rw-r–r–@ 1 howard 8399 Dec 24 10:43 journaling-org.org
-rw-r–r–@ 1 howard 10285 Mar 1 09:13 lists-and-key-sequences.org
-rw-r–r–@ 1 howard 1545 Mar 15 12:09 literate-devops-examples.org
-rw-r–r–@ 1 howard 15924 Mar 15 12:09 literate-devops.org
-rw-r–r–@ 1 howard 2041 Dec 24 10:43 magit-squashing.org
-rw-r–r–@ 1 howard 6812 Feb 9 23:11 new-window-manager.org
-rw-r–r–@ 1 howard 12083 Mar 15 11:23 spreadsheet.org
-rw-r–r–@ 1 howard 17861 Mar 5 09:05 tao-of-emacs.org
-rw-r–r–@ 1 howard 9441 Dec 24 10:43 why-emacs.org

This blocks reads the orgfiles as a list, and just grabs the last column (the names) as the variable, $FILES:

wc -l $FILES
300 eshell-fun.org
84 eshell.org
105 getting-started.org
244 journaling-org.org
255 lists-and-key-sequences.org
51 literate-devops-examples.org
422 literate-devops.org
69 magit-squashing.org
189 new-window-manager.org
342 spreadsheet.org
437 tao-of-emacs.org
231 why-emacs.org
2729 total

Downloading RPM Keys

The following section demonstrates the noweb feature from Knuth’s original idea of literate programming, where code from one language, Ruby, is included as a string in a shell script.

Assuming we have a repository URL reference that returns an HTML file with all of the RPM-GPG-KEYs, we use wget to get this listing, and then pass the HTML to a Ruby script to parse it, and only return the URL references to the keys we want:

wget -O - $repo 2>&1 | ruby -e "repo='$repo'" -ne 'puts repo + $1 if ($_ =~ / href="(RPM-GPG-KEY[^"]+)"/ )'
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-3
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-4
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-7
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-Debug-6
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-Debug-7
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-Security-6
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-Testing-6
- http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-Testing-7
- http://mirror.centos.org/centos/RPM-GPG-KEY-beta
- http://mirror.centos.org/centos/RPM-GPG-KEY-centos4

While we could attempt an HTML parsing Gem or some sorts, might find that a simple regular expression is sufficient:

puts repo + $1 if ($_ =~ / href="(RPM-GPG-KEY[^"]+)"/ )

Note: This block is not executed, but is embedded in the previous shell block.

Let’s download every GPG key we found from the repository:

rm -rf RPM-GPG-KEY*

for URL in $LIST
do
   wget $URL
done

ls RPM-GPG-KEY*
- RPM-GPG-KEY-beta
- RPM-GPG-KEY-CentOS-3
- RPM-GPG-KEY-centos4
- RPM-GPG-KEY-CentOS-4
- RPM-GPG-KEY-CentOS-5
- RPM-GPG-KEY-CentOS-6
- RPM-GPG-KEY-CentOS-7
- RPM-GPG-KEY-CentOS-Debug-6
- RPM-GPG-KEY-CentOS-Debug-7
- RPM-GPG-KEY-CentOS-Security-6
- RPM-GPG-KEY-CentOS-Testing-6
- RPM-GPG-KEY-CentOS-Testing-7

Configuring my Virtual Server

I have a virtual machine that often gets deleted and redeployed. I want my personal environment to be installed easily. To quickly configure the machine, I do:

  • Change the IP address in the PROPERTIES section in this drawer (optional, as the IP doesn’t change often)
  • Tangle each of code block (with modifications) by typing C-u C-c C-v C-t (which creates the file via Tramp to the remote system)

Begin by making sure that I can connect to my server:

hostname

TMux Configuration: This code ends up configuring my tmux so that I can hit F2 to jump to the second tmux window:

set -g status-right "| #(hostname -i) "

# Don't rename the window based on the directory.
set-option -g allow-rename off

# Start windows and panes at 1, not 0
set -g base-index 1
set -g pane-base-index 1

# Bind function keys.
bind -n F1 select-window -t 1
bind -n F2 select-window -t 2
bind -n F3 select-window -t 3
bind -n F4 select-window -t 4
bind -n F5 select-window -t 5
bind -n F6 select-window -t 6
bind -n F7 select-window -t 7
bind -n F8 select-window -t 8
bind -n F9 select-window -t 9
bind -n F10 select-window -t 10

Bash Profile and Aliases: Is the basics for my shell, but when I am first starting into tmux, it displays all my aliases for running commands in other tmux windows:

# The following aliases have been installed on this machine.
#
# - `go` to make a new SSH connection to a remote host.

# - `nw` to create a new Tmux window. Give ‘er a command.
alias nw='tmux new-window'

# - `irb` for a TMUX window into an interactive Ruby session (v. 1.9)
alias irb='nw -n ruby "/opt/chef/embedded/bin/irb $*"'

# - `root` for a TMUX window as the root user.
alias root='nw -n root "sudo su -" '
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

PATH=$PATH:$HOME/bin

# IP=$(ifconfig | grep addr: | grep '10.' | cut -d: -f2 | cut -d' ' -f1)
IP=$(hostname -i)
export PS1="\[\e[0;32m\]$IP\[\e[m\]:\[\e[1;34m\]\w\[\e[m\] \[\e[1;32m\]\$\[\e[m\] "

function go {
    if [ "$1" = "-n" ]
    then
        NAME=$2
        shift 2
    else
        SCRIPT='if (/\.([0-9]+)$/) {
   print "host-$1";
} elsif (/ ([^ @]+)$/) {
   print $1;
} else {
   print $_;
}'
        NAME=$(echo "$*" | perl -ne "$SCRIPT")
    fi
    nw -n "$NAME" "ssh $*"
}

# If we have a tmux session running, attach to it, otherwise, start
# one up....oh, and let's start emacs as a daemon too.
if [ "$TERM" = "screen" ]
then
    if [ "$HOME/.bash_aliases" ]
    then
        source "$HOME/.bash_aliases"
        grep '^# ' "$HOME/.bash_aliases" | sed 's/^# *//'
        echo
    fi
fi

TMux Start Script: When I connect to my virtual server, I automatically start a script that either runs tmux or attaches to it, e.g.

ssh -L 7443:10.96.68.204:443 10.96.68.204 -t '~/bin/t'

The script is pretty simple:

TMUX=$(which tmux 2>/dev/null)
if [ -z "$TMUX" ]
then
    TMUX=$HOME/bin/tmux
fi

if [ "$TERM" != "screen" ]
then
    if [ -x $TMUX ]
    then
        $TMUX att || $TMUX \
            new -s Chef -n shell
    fi
fi

Make sure to actually install tmux prior to running the previous script.

yum install -y tmux

Delete all Neutron Ports

I collected some table formatting code for OpenStack’s nova command, and placed them in an Tower of Babel file that makes the :post commands in any files (like this one) able to use them. The os-table function, in that file, converts the ASCII tables from OpenStack’s CLI into org-mode tables that I can export, render, and feed into other code blocks as values. This feature came in handy today…

The backstory for code is that once, when I was bringing up an OpenStack cluster, I discovered one of my previous failures had left all the ports already allocated, so while nova list returned an empty list of virtual machines, neutron showed a few entries.

A problem arose when a colleague attempted to run the following script at the command line to remove them all:

for PORTID in $(neutron port-list | cut -d'|' -f1)
do
  neutron port-delete $PORTID
done

Seems that the neutron command acts differently than the nova command (probably due to the lack of a tty), so it spit out JSON instead of text…harder to parse.

I first ran following command in Emacs and stored the IDs as a table within my document, and named it ports:

neutron port-list
id name mac_address fixed_ips      
557963f3-87b3-4057-b1b2-21e3b4603f8a 557963f3-87b3-4057-b1b2-21e3b4603f8a 02:55:79:63:f3:87 subnet_id 0b819174-9adc-493f-8485-81ac019568d1 ip_address 10.96.68.203
b2f9c1e0-9dde-47ad-8365-dde3c06b9f6c b2f9c1e0-9dde-47ad-8365-dde3c06b9f6c 02:b2:f9:c1:e0:9d subnet_id 0b819174-9adc-493f-8485-81ac019568d1 ip_address 10.96.68.204
d3e6a458-2a78-4632-bd37-333189761975 d3e6a458-2a78-4632-bd37-333189761975 02:d3:e6:a4:58:2a subnet_id 0b819174-9adc-493f-8485-81ac019568d1 ip_address 10.96.68.200
d9c48d59-2d72-40e2-8593-2db26de8ce08 d9c48d59-2d72-40e2-8593-2db26de8ce08 02:d9:c4:8d:59:2d subnet_id 0b819174-9adc-493f-8485-81ac019568d1 ip_address 10.96.68.202
e3bb26a7-1abc-4e3a-8484-c5a66ae21cfa e3bb26a7-1abc-4e3a-8484-c5a66ae21cfa 02:e3:bb:26:a7:1a subnet_id 0b819174-9adc-493f-8485-81ac019568d1 ip_address 10.96.68.201

Now that I’ve extracted the ID values, I feed them through to this code block to delete them:

for ID in $IDS
do
    neutron port-delete $ID
done
Deleted port: 557963f3-87b3-4057-b1b2-21e3b4603f8a
Deleted port: b2f9c1e0-9dde-47ad-8365-dde3c06b9f6c
Deleted port: d3e6a458-2a78-4632-bd37-333189761975
Deleted port: d9c48d59-2d72-40e2-8593-2db26de8ce08
Deleted port: e3bb26a7-1abc-4e3a-8484-c5a66ae21cfa