Examples of My Literate Devops

This example in this section demonstrates how the output from one code block can be used as values to another code block.

First, we call ls to get a list of files in a directory, and name them as orgfiles:

ls -lok *.org

This blocks reads the orgfiles as a list, and just grabs the last column (the names) as the variable, $FILES:

wc -l $FILES

The following section demonstrates the noweb feature from Knuth’s original idea of literate programming, where code from one language, Ruby, is included as a string in a shell script.

Assuming we have a repository URL reference that returns an HTML file with all of the RPM-GPG-KEYs, we use wget to get this listing, and then pass the HTML to a Ruby script to parse it, and only return the URL references to the keys we want:

wget -O - $repo 2>&1 | ruby -e "repo='$repo'" -ne 'puts repo + $1 if ($_ =~ / href="(RPM-GPG-KEY[^"]+)"/ )'

While we could attempt an HTML parsing Gem or some sorts, might find that a simple regular expression is sufficient:

puts repo + $1 if ($_ =~ / href="(RPM-GPG-KEY[^"]+)"/ )

Note: This block is not executed, but is embedded in the previous shell block.

Let’s download every GPG key we found from the repository:

rm -rf RPM-GPG-KEY*

for URL in $LIST
do
   wget $URL
done

ls RPM-GPG-KEY*

I have a virtual machine that often gets deleted and redeployed. I want my personal environment to be installed easily. To quickly configure the machine, I do:

• Change the IP address in the PROPERTIES section in this drawer (optional, as the IP doesn’t change often)
• Tangle each of code block (with modifications) by typing C-u C-c C-v C-t (which creates the file via Tramp to the remote system)

Begin by making sure that I can connect to my server:

hostname

TMux Configuration: This code ends up configuring my tmux so that I can hit F2 to jump to the second tmux window:

set -g status-right "| #(hostname -i) "

# Don't rename the window based on the directory.
set-option -g allow-rename off

# Start windows and panes at 1, not 0
set -g base-index 1
set -g pane-base-index 1

# Bind function keys.
bind -n F1 select-window -t 1
bind -n F2 select-window -t 2
bind -n F3 select-window -t 3
bind -n F4 select-window -t 4
bind -n F5 select-window -t 5
bind -n F6 select-window -t 6
bind -n F7 select-window -t 7
bind -n F8 select-window -t 8
bind -n F9 select-window -t 9
bind -n F10 select-window -t 10

Bash Profile and Aliases: Is the basics for my shell, but when I am first starting into tmux, it displays all my aliases for running commands in other tmux windows:

# The following aliases have been installed on this machine.
#
# - `go` to make a new SSH connection to a remote host.

# - `nw` to create a new Tmux window. Give ‘er a command.
alias nw='tmux new-window'

# - `irb` for a TMUX window into an interactive Ruby session (v. 1.9)
alias irb='nw -n ruby "/opt/chef/embedded/bin/irb $*"'

# - `root` for a TMUX window as the root user.
alias root='nw -n root "sudo su -" '

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

PATH=$PATH:$HOME/bin

# IP=$(ifconfig | grep addr: | grep '10.' | cut -d: -f2 | cut -d' ' -f1)
IP=$(hostname -i)
export PS1="\[\e[0;32m\]$IP\[\e[m\]:\[\e[1;34m\]\w\[\e[m\] \[\e[1;32m\]\$\[\e[m\] "

function go {
    if [ "$1" = "-n" ]
    then
        NAME=$2
        shift 2
    else
        SCRIPT='if (/\.([0-9]+)$/) {
   print "host-$1";
} elsif (/ ([^ @]+)$/) {
   print $1;
} else {
   print $_;
}'
        NAME=$(echo "$*" | perl -ne "$SCRIPT")
    fi
    nw -n "$NAME" "ssh $*"
}

# If we have a tmux session running, attach to it, otherwise, start
# one up....oh, and let's start emacs as a daemon too.
if [ "$TERM" = "screen" ]
then
    if [ "$HOME/.bash_aliases" ]
    then
        source "$HOME/.bash_aliases"
        grep '^# ' "$HOME/.bash_aliases" | sed 's/^# *//'
        echo
    fi
fi

TMux Start Script: When I connect to my virtual server, I automatically start a script that either runs tmux or attaches to it, e.g.

ssh -L 7443:10.96.68.204:443 10.96.68.204 -t '~/bin/t'

The script is pretty simple:

TMUX=$(which tmux 2>/dev/null)
if [ -z "$TMUX" ]
then
    TMUX=$HOME/bin/tmux
fi

if [ "$TERM" != "screen" ]
then
    if [ -x $TMUX ]
    then
        $TMUX att || $TMUX \
            new -s Chef -n shell
    fi
fi

Make sure to actually install tmux prior to running the previous script.

yum install -y tmux

I collected some table formatting code for OpenStack’s nova command, and placed them in an Tower of Babel file that makes the :post commands in any files (like this one) able to use them. The os-table function, in that file, converts the ASCII tables from OpenStack’s CLI into org-mode tables that I can export, render, and feed into other code blocks as values. This feature came in handy today…

The backstory for code is that once, when I was bringing up an OpenStack cluster, I discovered one of my previous failures had left all the ports already allocated, so while nova list returned an empty list of virtual machines, neutron showed a few entries.

A problem arose when a colleague attempted to run the following script at the command line to remove them all:

for PORTID in $(neutron port-list | cut -d'|' -f1)
do
  neutron port-delete $PORTID
done

Seems that the neutron command acts differently than the nova command (probably due to the lack of a tty), so it spit out JSON instead of text…harder to parse.

I first ran following command in Emacs and stored the IDs as a table within my document, and named it ports:

neutron port-list

Now that I’ve extracted the ID values, I feed them through to this code block to delete them:

for ID in $IDS
do
    neutron port-delete $ID
done