I recently have started a process of migrating my website over to
blogger.com. One of the main reasons for this was because in my last
server move, I had broken my Movable Type installation, and found myself
too busy to fix it. I found I didn't want to spend my time fixing and
updating blogging software. I wanted to work on my projects, write them
up, and post them. It was time to move my content to an existing
platform that handled the back end. I looked at a few, and decided
blogger.com would be as good as any other service.
It only took a short time to setup a blog, point a CNAME at it, and then
to import my existing posts. When I started creating some new posts, I
immediately ran into some limitations.

1. You used to be able to edit permalinks on blogger. Now, you can only
   do that before you publish. The only way to change a permalink after
   publishing is to create a new post with the desired permalink and
   delete the old one.
2. Blogger has no built in formatting for code blocks. So if I want to
   show a config file, source code, or terminal session log, I have to
   fiddle with changing fonts, size, and "blockquote" to get it
   presentable. Even then, you run the risk of strange formatting of
   your raw text.

I found a solution that other bloggers use called SyntaxHighlighter.
This is a combination of javascript and css code that takes text within
your

; tags and gives you nice looking blocks of code, highlighted and (optionally) with line numbers. The catch is that your pre tags need to have a class name, along with the language (perl/c/bash/text) "brush" to use. If you go with pre tags, you have to change any angle bracks to their HTML escaped equivalents of < and >. They have a work-around using SCRIPT/CDATA, but it takes some getting used to. Adding this to your blog only requires a few steps.

I rather liked syntaxhighlighter, but it still seemed like I had to do a
lot of manual work with the code. Also, I had to select the brush each
time. Couldn't it guess? Notepad++ and some others will guess at what
language you're using and highlight accordingly. I found something
called prettify that does just that. You only need to load one js
file and one css file. Prettify works off of either

 or  tags and has similar limitations to SyntaxHighlighter regarding html tags. However, it has the advantage of being able to guess the language automatically.

Being able to use this code made my posts look much nicer, but the
entire process got me thinking. The way I "document" most of projects
typically invole using a notepad editor like Geany or
Notepad++. As I work, I add notes, copy in source code or shell
comands, and do everything in a plain text editor. Later, I add
commentary and clean up the document. I take this and paste it into the
WYSIWYG editor on blogger. Finally, I have to keep switching between
compose and html mode to get my text looking suitable. There are too
many steps for me to want to do this consistently. All I really want to
do is take my text file, add a little formatting in a few spots, a few
hyperlinks in others, and post it.
Enter markdown. Markdown is a text-to-HTML conversion tool for web
writers. Markdown allows you to write using an easy-to-read,
easy-to-write plain text format, then convert it to structurally valid
XHTML (or HTML). I have used this before, but didn't pay it close
enough attention. It's used on github and reddit, there are plugins for
it in dokuwiki and redmine. The idea is you write in text, adding
formatting using the markdown syntax. This format is both human readable
and machine readable. When read by the appropriate library, clean html
is generated. It also has a feature for wrapping blocks of code inside
of

&;lt;code> tags and html-escaping html inside of those tags.

Within the MarkDown project is a paged called "dingus" which means
"placeholder". You can paste your markdown text into one textarea and
get the generated html plus a preview back. I tested pasting that
generated html into Blogger's HTML box and it seems to work perfectly
fine. What this means is that I can type up my documentation completely
within my text editor of choice, save it locally, and then generate my
html code to paste into blogger.
Some of you may have realized that my

 tags are missing that class name (. Well, I copy the generated html, do a search and replace of  with  and then paste it, but that's adding more steps. Instead, I sought to make my own dingus that does this automatically. I found that there is an extension of markdown called Markdown Extra written in PHP. Extra adds a few features such as simple tables, but remains consistent with original Markdown formatting. Using that library, I was able to create my own dingus rather easily and alter the  tag with one line of code $render2 = str_replace("<pre>", "<pre class="prettyprint linenums">", $render);. In my experimentation, I made a parser that reads a text file and outputs html, and three dingus parsers. Dingus1 does straightforward conversion of markdown extra to html. Dingus2 and 3 provide the class names for prettified code, with #3 going the extra step of applying stylesheets for the preview.

With this setup, I can quickly paste in my text document and pull html
code to paste into blogger.com's html edit box. With some more research,
I can modify the dingus to interact with blogger's API and post on my
behalf. There are also some WYSIWYM live editors that show you an
instant render of your markdown as you type (you type in a textarea
while your html renders in a nearby div). This would be a good way to do
some tweaking to the markdown text before posting the html to the web.
My next plans are to make a better dingus, possibly with a live preview
and a "post to blogger" option.
Some other links:

• http://balupton.github.com/jquery-syntaxhighlighter/demo/
• http://code.google.com/p/pagedown/wiki/PageDown
• http://markitup.jaysalvat.com/examples/markdown/

Yesterday, I setup gateone to run as a non-root user. I also spent
some time looking at potential init scripts for starting and stopping
this. The gateone project does not currently provide any init scripts,
but this is planned for the future ([Issue #47]). I tried to use one
of the scripts in that thread, but I wasn't really pleased with them.
The big issue is that gateone.py doesn't fork. However, I believe there
is a better solution.

supervisord is a python script designed to control and monitor
non-daemonizing python scripts. As Gate One is a foreground only
process, it seems particularly suited to this task - more so than
writing my own script in python or daemontools.

Installation can be done with python-pip or easy_install. On newer
systems, pip is recommended.

sudo pip install supervisor

On Ubuntu, pip installs supervisord to /usr/local/bin. By default,
/usr/local/bin is not in root's path, so it makes sense (to me at least)
to create symlinks to /usr/sbin.

johnh@puppet2:~$ ls /usr/local/bin
echo_supervisord_conf  pidproxy  supervisorctl  
supervisordjohnh@puppet2:~$ sudo ln -s /usr/local/bin/supervisord /usr/sbin
johnh@puppet2:~$ sudo ln -s /usr/local/bin/supervisorctl /usr/sbin

Now, we need to create a configuration file. Supervisord has a utility
to generate a sample one.

echo_supervisord_conf  > supervisord.conf

To get started, we can use the sample configuration and just add a
couple lines to the bottom for gateone.

 [program:gateone]
 command=/opt/gateone/gateone.py
 directory=/opt/gateone
 ;user=johnh   ; Default is root. Add a user= to setuid

Now, copy supervisord.conf to /etc/supervisord.conf and start
supervisord. Make sure gateone.py is not currently running. Then we'll
run supervisorctl to test things out.

johnh@puppet2:~$ sudo cp supervisord.conf /etc
johnh@puppet2:~$ sudo supervisord
johnh@puppet2:~$ sudo supervisorctl status
gateone                          RUNNING    pid 9549, uptime 0:00:05
johnh@puppet2:~$ ps ax | grep gateone
 9549 ?        Sl     0:00 python /opt/gateone/gateone.py
johnh@puppet2:~$ sudo supervisorctl stop gateone
gateone: stopped
johnh@puppet2:~$ ps ax | grep gateone
 9605 ?        Ss     0:00 dtach -c /opt/gateone/tmp/gateone/../dtach_3 -E -z -r none /opt/gateone/plugins/ssh/scripts/ssh_connect.py -S /tmp/gateone/.../%SHORT_SOCKET% --sshfp -a -oUserKnownHostsFile=/opt/gateone/users/user1@puppet2/ssh/known_hosts
 9606 pts/3    Ss+    0:00 python /opt/gateone/plugins/ssh/scripts/ssh_connect.py -S /tmp/gateone/.../%SHORT_SOCKET% --sshfp -a -oUserKnownHostsFile=/opt/gateone/users/user1@puppet2/ssh/known_hosts

In this example, we see that gateone.py is started and stopped by
supervisorctl, but because we have dtach enabled, our sessions are still
in place. If we restart gateone.py, we can connect to it again and have
our sessions resumed. While we could probably configure supervisord to
kill these terminals, I believe we'd normally want to keep them running.
The few times I would want to stop those terminals would be a) manually
reconfiguring/troubleshooting opengate, b) updating software, or c)
rebooting the server. For a&b, running the command "gateone.py -kill"
will kill those terminals. For a server shutdown or reboot, the act of
shutting down the OS will kill these terminals.

Finally, we need a way to start and stop supervisord itself.
Fortunately, the supervisord project has provided a number of init
scripts. I was able to use the Debian script in Ubuntu with only
a few minor changes.

1. I had symlinked supervisord and supervisorctl to /usr/sbin. The
   script expects them in /usr/bin (but even says that /usr/sbin is a
   better location). I had to change /usr/bin to /usr/sbin.
   Alternatively, you can symlink the files into /usr/bin
2. I added a status option that runs $SUPERVISORCTL status
3. If you started supervisord manually, you must shut it down and start
   it with the script. The script won't be able to stop supervisord
   unless /var/run/supervisord.pid is current.

Here is my complete init script for Ubuntu:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          supervisord
# Required-Start:    $local_fs $remote_fs $networking
# Required-Stop:     $local_fs $remote_fs $networking
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts supervisord - see http://supervisord.org
# Description:       Starts and stops supervisord as needed 
### END INIT INFO
# Author: Leonard Norrgard 
# Version 1.0-alpha
# Based on the /etc/init.d/skeleton script in Debian.
# Please note: This script is not yet well tested. What little testing
# that actually was done was only on supervisor 2.2b1.
# Do NOT "set -e"
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Run a set of applications as daemons."
NAME=supervisord
# Supervisord is installed in /usr/bin by default, but /usr/sbin would 
# make more sense
DAEMON=/usr/sbin/$NAME   
SUPERVISORCTL=/usr/sbin/supervisorctl
PIDFILE=/var/run/$NAME.pid
DAEMON_ARGS="--pidfile ${PIDFILE}"
SCRIPTNAME=/etc/init.d/$NAME
# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0
# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions
#
# Function that starts the daemon/service
#
do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        [ -e $PIDFILE ] && return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- 
                $DAEMON_ARGS  
                || return 2
        # Add code here, if necessary, that waits for the process to be ready
        # to handle requests from services started subsequently which depend
        # on this one.  As a last resort, sleep for some time.
}
#
# Function that stops the daemon/service
#
do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        [ -e $PIDFILE ] || return 1
        # Stop all processes under supervisord control.
        $SUPERVISORCTL stop all
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE  
             --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
}
#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
        return 0
}
case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  #reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        #log_daemon_msg "Reloading $DESC" "$NAME"
        #do_reload
        #log_end_msg $?
        #;;
  restart|force-reload)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
                do_start
                case "$?" in
                        0) log_end_msg 0 ;;
                        1) log_end_msg 1 ;; # Old process is still running
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
          *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
  status)
        $SUPERVISORCTL status
        RETVAL=$?
        ;;
  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload|status}" >&2
        exit 3
        ;;
esac

And here is a complete copy of my supervisord.conf file:

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
[program:gateone]
command=/opt/gateone/gateone.py
directory=/opt/gateone
stdout_logfile=/opt/gateone/logs/supervisor.log
user=johnh

I came across a program called Gate One by LiftOff Software that
just amazed me. This is an open-source, web-based ssh terminal. It is
capable of multiple users, sessions, and bookmarks. I've tried a number
of AJAX terminals or Java applet based ones in the past. The javascript
ones usually did not have very good terminal emulation, while the Java
apps worked, but worked just like a local desktop app (making it's own
connection to port 22). Gate One uses WebSockets, allowing for full
duplex communication through your web browser over the same port 80 or
443 used to serve up the web page.

Installation

Gate One is a python application using the Tornado framework. As
such, at runs independently of an existing web server and handles
connections from browsers internally. If you already have a web server
running on your system, you will need to tell Gate One to use a
different IP or a different port.

Installation using pre-built binaries or the source is fairly
straightforward and detailed in the documentation.

The installer creates a directory of /opt/gateone and places all
necessary files there. You can run it by changing to that directory and
running gateone.py as root.

johnh@puppet2:/opt/gateone$ sudo ./gateone.py
[W 120801 13:52:06 terminal:166] Could not import the Python Imaging Library (PIL) so images will not be displayed in the terminal
[W 120801 13:52:06 gateone:2232] dtach command not found.  dtach support has been disabled.
[I 120801 13:52:06 gateone:1800] No authentication method configured. All users will be ANONYMOUS
[I 120801 13:52:06 gateone:1876] Loaded plugins: bookmarks, help, logging, logging_plugin, notice, playback, ssh
[I 120801 13:52:06 gateone:2329] Listening on https://*:443/

At this point, gateone is running in the foreground and you can view as
connections occur and any errors. Pressing Ctrl If you conect to gateone
using your webbrowser, you are logged in as user ANONYMOUS and can
connect to any ssh host, either localhost or something remote.

If you edit /opt/gateone/server.conf, you can change authentication to
"pam" or "google". Using pam will perform a Basic HTTP style
authenication requiring a system-level username and password. Using
google will log you in with your google account. Both of these "just
work" without complicated setup.

Running as a Non-Root

Before I put something like this in production, I wanted to apply some
additional security. First off, I want to see if I can get this to run
as a non-root user.

Since gateone ran as root user initially, it has files owned by rootOnly UID 0 can open ports below 1024.gateone may need permission to write to system directories

To solve the first one, I chowned the /opt/gateone directory to my
username. In the future, I'll want to run it under its own user, but
I'll use mine for now for simplicity. To solve the second and third, I
edited server.conf.

johnh@puppet2:/opt/gateone$ sudo chown -R johnh:johnh .
johnh@puppet2:/opt/gateone$ vi server.conf# change/add the following lines appropriatelyport = 2443session_dir = "/opt/gateone/tmp/gateone"pid_file = "/opt/gateone/tmp/gateone.pid"uid = 1000gid = 1000
johnh@puppet2:/opt/gateone$ ./gateone.py
[W 120801 14:06:01 terminal:166] Could not import the Python Imaging Library (PIL) so images will not be displayed in the terminal
[W 120801 14:06:01 gateone:2232] dtach command not found.  dtach support has been disabled.
[I 120801 14:06:01 gateone:1802] No authentication method configured. All users will be ANONYMOUS
[I 120801 14:06:01 gateone:1876] Loaded plugins: bookmarks, help, logging, logging_plugin, notice, playback, ssh
[I 120801 14:06:01 gateone:2329] Listening on https://*:2443/

Authentication

Running as a lower uid, you can use authentication of None or "google"
without issue. If you use "pam", you discover you can only login with
the username that gateone is running under. If you are the only intended
user of the service, this may not be an issue. But if you want to allow
other users, this becomes an issue. If you are fine with running as root
or using Google as your authentication provider, you can ignore this
next step.

Fortunately, pam is highly configurable. You aren't required to
authenticate against shadow passwords. You can also authenticate against
db4 files with pam_userdb, msyql, or even htpasswd files. To start off,
I'm going to use htpasswd files. Note that Ubuntu doesn't provide
pam_pwdfile.so by default. You need to install libpam-pwdfile ("sudo
apt-get install libpam-pwdfile").

Note - in testing, I discovered gateone uses Crypt encryption while htpasswd defaults to MD5. Use -d to switch to crypt encryption.

johnh@puppet2:/opt/gateone$ htpasswd -c -d users.passwd user1
New password:
Re-type new password:
Adding password for user user1
johnh@puppet2:/opt/gateone$ cat users.passwd
user1:KKEPyZtUf9sadf9

Create a pam module called gateone under /etc/pam.d

johnh@puppet2:/opt/gateone$ cat /etc/pam.d/gateone
#%PAM-1.0
# Login using a htpasswd file
@include common-sessionauth    
required pam_pwdfile.so          pwdfile /opt/gateone/users.passwdaccount 
required pam_permit.so

Modify server.conf to use pam and pam_service of gateone:

auth = "pam"
pam_service = "gateone"

Now start gateone and log in.

johnh@puppet2:~/g1/gateone$ ./gateone.py
[W 120801 14:59:16 terminal:168] Could not import the Python Imaging Library (PIL) so images will not be displayed in the terminal
[W 120801 14:59:16 gateone:2577] dtach command not found.  dtach support has been disabled.
[I 120801 14:59:16 gateone:2598] Connections to this server will be allowed from the following origins: 'http://localhost https://localhost http://127.0.0.1 https://127.0.0.1 https://puppet2 https://127.0.1.1 https://puppet2:2443'
[I 120801 14:59:16 gateone:2023] Using pam authentication
[I 120801 14:59:16 gateone:2101] Loaded plugins: bookmarks, help, logging, logging_plugin, mobile, notice, playback, ssh
[I 120801 14:59:16 gateone:2706] Listening on https://*:2443/
[I 120801 14:59:16 gateone:2710] Process running with pid 32591
[I 120801 14:59:17 gateone:949] WebSocket opened (user1@gateone).

One additional nice feature with authentication enabled is the ability
to resume sessions - even across different computers or browsers.

Reverse Proxy

(I failed on this part, but felt it was worth recording)

Once I got it working in single user mode, I wanted to go ahead and set
this up under a reverse proxy under Apache. This would allow me to
integrate it into my existing web server under a sub-directory.

First, I edited server.conf to use a URL prefix of /g1/

Second, I tried setting up a ReverseProxy in Apache.

# GateOne 
ProxySSLProxyEngine 
OnProxyPass /g1/ https://localhost:2443/g1/
ProxyPassReverse /g1/ https://localhost:2443/g1/
ProxyPassReverseCookieDomain localhost localhost
ProxyPassReverseCookiePath / /g1/

This almost worked. I had no errors, but the resulting page was
unreadable. However, at the bottom was a clue. "The WebSocket connection
was closed. Will attempt to reconnect every 5 seconds... NOTE: Some web
proxies do not work properly with WebSockets." The problem was Apache
not properly proxying my websocket connection. People have managed to
get this working under nginx, but not Apache.

Searching for a solution led me to a similar question on ServerFault, an
apache-websocket module on github, and a websocket tcp proxy based on
that module.

• http://serverfault.com/questions/290121/configuring-apache2-to-proxy-websocket
• https://github.com/disconnect/apache-websocket
• http://blog.alex.org.uk/2012/02/16/using-apache-websocket-to-proxy-tcp-connection/

In order to get this work, I'll need to download and compile some code.
The apxs command requires the apache-prefork-dev package in
Debian/Ubuntu. Install it with "sudo apt-get install
apache-prefork-dev".

Now we are ready to download the code and install the module:

johnh@puppet2:~$ git clone https://github.com/disconnect/apache-websocket.git
Cloning into 'apache-websocket'..... done
johnh@puppet2:~$ wget http://blog.alex.org.uk/wp-uploads/mod_websocket_tcp_proxy.tar.gz
johnh@puppet2:~$ cd apache-websocket
johnh@puppet2:~/apache-websocket$ sudo apxs2 -i -a -c mod_websocket.c*snip*
johnh@puppet2:~/apache-websocket$ sudo apxs2 -i -a -c mod_websocket_draft76.c*snip*
johnh@puppet2:~$ cd examples
johnh@puppet2:~$ tar -xzvf ../../mod_websocket_tcp_proxy.tar.gzmod_websocket_tcp_proxy.c
johnh@puppet2:~$ cd apache-websocket/examples/
johnh@puppet2:~/apache-websocket/examples$ sudo apxs2 -c -i -a -I.. mod_websocket_tcp_proxy.c
*snip*
chmod 644 /usr/lib/apache2/modules/mod_websocket_tcp_proxy.so
[preparing module `websocket_tcp_proxy' in /etc/apache2/mods-available/websocket_tcp_proxy.load]
Enabling module websocket_tcp_proxy.To activate the new configuration, you need to run:service apache2 restart
johnh@puppet2:~$

Before we restart, I want to remove my Proxy lines and replace them with
the mod_websocket_tcp_proxy lines.

SetHandler websocket-handler        
WebSocketHandler  /usr/lib/apache2/modules/mod_websocket_tcp_proxy.so tcp_proxy_init        
WebSocketTcpProxyBase64 on        
WebSocketTcpProxyHost 127.0.0.1        
WebSocketTcpProxyPort 2443        
WebSocketTcpProxyProtocol base64

Despite all this, I was still unable to get this to work. I even
attempted using the web root (/) as my location. If the Location matches
and your HTTP request is handled by mod_websocket, you get a 404. If
you use Proxy, then your websocket request is handled by mod_proxy.
Mod_proxy wins out over Location matches. Perhaps you can modify
gateone code to have one URL for the web interface and one for
websockets (or maybe it's already in place and we just need to know),
but I don't see a way at this time to get this working under Apache. I
may be able to work with the gateone author and the
mod_websocket_tcp_proxy.c author to come up with a solution. Or I
could try installing nginx. In the meantime, I can continue to run Open
Gate as a non-root user on a non-standard port. Alternatively, I could
find a wrapper to bring port 443 to 2443.

Tags: puppet selinux

Once I got a rough handle on setting up Puppet, I decided to get Puppet
Dashboard working on my puppetmaster (a Centos 6 server) to have a
sort of web interface to view puppet statuses. Since I already have the
puppetlabs repositories setup from when I installed puppet, this was
almost as simple as running "yum install puppet-dashboard".

You then go to /usr/share/puppet-dashboard and follow the install
guide to get the database working. Then you can run it locally using
the "sudo -u puppet-dashboard ./script/server -e production" command or
starting the service (service puppet-dashboard start). I recommed
running it manually the first few times to make sure everything is
working.

At this point, I was able to browse to http://puppetmaster:3000/ and
view a nice looking dashboard with no hosts checked in. I then made sure
to add the following to puppetmaster's puppet.conf [master] section and
restart puppet.

reports = http, store
reporturl = http://localhost:3000/reports/upload

I performed a "puppet agent --test" on one of my puppets, and here is
where I ran into trouble. Everything appeared to work, but no report or
"pending task" showed up in the Dashboard. I ran the Dashboard locally
so I could see the http request coming in. No request. I double checked
my configuration, everything looked good.

Finally, I ran my puppetmaster in debug/no-daemonize mode.
(/usr/sbin/puppetmaster --debug --no-daemonize) so I could watch what
was happening. However, in this mode, it worked fine. I ran
/usr/sbin/puppetmaster without debugging and it still worked. The
reports would get submitted to dashboard if I ran puppetmaster directly,
but not if I started it with the init scripts.

I couldn't find any differences between how the init script was starting
puppetmaster vs me starting it manually. However, I did come across this
entry in /var/log/messages:

 Jul 25 10:15:16 puppetmasterj puppet-master[11988]: Compiled catalog for puppet2.lab in environment production in 1.16 seconds
 Jul 25 10:15:17 puppetmasterj puppet-master[11988]: Report processor failed: Permission denied - connect(2)

Which led me to the following entry in audit.log

 type=AVC msg=audit(1343225819.078:1582): avc:  denied  { name_connect } for  pid=11988 comm="puppetmasterd" dest=3000 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket

This was an selinux issue. I quickly ascertained that turning enforcing
off in selinux allowed my reports to get through. I couldn't find anyone
else online encountering this issue. However, many people disable
selinux enforcing early on and I guess the cross-section of
puppet-dashboard users and those using selinux enforcing is somewhat
small.

How to fix this? There is a set of python programs called "audit2why"
and "audit2allow" as part of the policycoreutils-python package. These
tools will parse entries from the audit.log and either explain why an
action was denied or provide a solution. You can get these tools by
doing a "yum install policycoreutils-python".

Now we can use audit2allow to parse our audit.log error. You'll want to
run the tool, paste in your log entry, and then hit Ctrl+D on a blank
line.

 [root@puppetmasterj tmp]# audit2allow -m puppetmaster
 type=AVC msg=audit(1343232143.497:1617): avc:  denied  { name_connect } for  pid=12552 comm="puppetmasterd" dest=3000 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket
 module puppetmaster 1.0;
 require {
     type puppetmaster_t;
     type ntop_port_t;
     class tcp_socket name_connect;
 }
 #============= puppetmaster_t ==============
 allow puppetmaster_t ntop_port_t:tcp_socket name_connect;

The above gives you a textual view of a module you can create to allow
puppetmaster to make an outbound connection. audit2allow will even
generate that module with the -M option.

 [root@puppetmasterj tmp]# audit2allow -M puppetmaster
 type=AVC msg=audit(1343232143.497:1617): avc:  denied  { name_connect } for  pid=12552 comm="puppetmasterd" dest=3000 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket
 ******************** IMPORTANT ***********************
 To make this policy package active, execute:
 semodule -i puppetmaster.pp

That generated the module "puppetmaster.pp". Despite the pp extension,
this is not a puppet manifest, but an selinux binary module. Let's
install it.

 [root@puppetmasterj tmp]# semodule -i puppetmaster.pp

With that, puppetmaster can communicate with dashboard and reports are
flowing. The only remaining thing left to do is to file a bug report. As
it happened, someone had posted a bug report similar to this for
documentation on the puppetdb project. I decided to append to that
issue, but I suggested migrating the issue to the main puppet project.
Issue #15567.

This is a basic walkthrough of getting a slightly complex "step by step
to install" program like djbdns to install under puppet (in this case,
under Ubuntu 12.04). It shows building the manifest, testing it, and
some possible gotchas.

I am generally following the guide put together by Higher Logic[1], with
a few changes of my own.

Step 1: Installation
I use the dbndns fork of djbdns, which has a few patches installed that
djbdns lacks. In fact, the djbdns package in Debian/Ubuntu is a virtual
package that really install dbndns. To install it normally, you would
type "sudo apt-get install dbndns". This would also install daemontools
and daemontools-run. However, we'll also need make and ucspi-tcp.

We're going to do this the puppet way. I'm assuming my puppet
configuration in in /etc/puppet, node manifests are in
/etc/puppet/nodes, and modules are in /etc/puppet/modules.

a. Create the dbndns module with a package definition to install

sudo mkdir -p /etc/puppet/modules/dbndns/manifests
    sudo vi /etc/puppet/modules/dbndns/manifests/init.pp

        class dbndns {
            package {
                    dbndns:
                    ensure => present;

ucspi-tcp:
                    ensure => present;

make:
                    ensure => present;
            }

}

b. Create a file for your node (ie: puppet2.example.net)

sudo vi /etc/puppet/nodes/puppet2.example.net.pp

        node    'puppet2.lab.example.net' {
            include dbndns
        }

c. Test
Ok, to test on your puppet client, run "sudo puppet agent --test"

johnh@puppet2:~# sudo puppet agent --test
    info: Retrieving plugin
    info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
    info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
    info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
    info: Caching catalog for puppet2.lab.example.net
    info: Applying configuration version '1340213237'
    notice: /Stage[main]/Dbndns/Package[dbndns]/ensure: created
    notice: Finished catalog run in 3.39 seconds

Here we can see our dbndns package installed. But is it running? Well,
djbdns uses daemontools, which runs svscan, and some searching online
shows that in Ubuntu 12.04/Precise, this is now an upstart job. svscan
is not running. So let's make it run. Add the following to your init.pp
(within the module definition):

define the service to restart

        service { "svscan":
                ensure  => "running",
                provider => "upstart",
                require => Package["dbndns"],
        }

Now back on puppet2, let's test it.

johnh@puppet2:~# sudo puppet agent --test
    info: Retrieving plugin
    info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
    info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
    info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
    info: Caching catalog for puppet2.lab.example.net
    info: Applying configuration version '1340213237'
    notice: /Stage[main]/Dbndns/Service[svscan]/ensure: ensure changed
'stopped' to 'running'
    notice: Finished catalog run in 0.47 seconds

We now told puppet to ensure that svscan is running. The "provider"
option tells it to use upstart instead of /etc/init.d/ scripts or the
service command. Also, we make sure that it doesn't attempt to start
svscan unless dbndns is already installed.

Now we have daemontools running, but we haven't got it start our tinydns
service yet. To do that, we need to create some users and configure the
service.

Step 2: Create users

Going back to our guide, our next step is to create users. We can do
that in puppet as well.
    # Users for the chroot jail
    adduser --no-create-home --disabled-login --shell /bin/false dnslog
    adduser --no-create-home --disabled-login --shell /bin/false
tinydns
    adduser --no-create-home --disabled-login --shell /bin/false
dnscache

So in our init.pp module file, we need to define our users:

user { "dnslog":
            shell => "/bin/false",
            managehome => "no",
            ensure => "present",
        }

    user { "tinydns":
            shell => "/bin/false",
            managehome => "no",
            ensure => "present",
        }

    user { "dnscache":
            shell => "/bin/false",
            managehome => "no",
            ensure => "present",
        }

Back on puppet2, we can give that a test.

johnh@puppet2:~$ sudo puppet agent --test
    info: Retrieving plugin
    info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
    info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
    info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
    info: Caching catalog for puppet2.lab.example.net
    info: Applying configuration version '1340215757'
    notice: /Stage[main]/Dbndns/User[dnscache]/ensure: created
    notice: /Stage[main]/Dbndns/User[tinydns]/ensure: created
    notice: /Stage[main]/Dbndns/User[dnslog]/ensure: created
    notice: Finished catalog run in 0.86 seconds
    johnh@puppet2:~$ cat /etc/passwd | grep dns
    dnscache:x:1001:1001::/home/dnscache:/bin/false
    tinydns:x:1002:1002::/home/tinydns:/bin/false
    dnslog:x:1003:1003::/home/dnslog:/bin/false

So far, so good. Now we have to do the configuration, which will require
executing some commands.

Step 3 - Configuration
Our next step are the following commands:

Config

    tinydns-conf tinydns dnslog /etc/tinydns/ 1.2.3.4
    dnscache-conf dnscache dnslog /etc/dnscache 127.0.0.1
    cd /etc/dnscache; touch /etc/dnscache/root/ip/127.0.0
    mkdir /etc/service ; cd /etc/service ; ln -sf /etc/tinydns/ ; ln -sf
/etc/dnscache

The first two commands create our service directories. Authoratative
tinydns is set to listen on 1.2.3.4 and dnscache is set to listen on
127.0.0.1. The 3rd command creates a file that restricts dnscache to
only respond to requests from IPs starting with 127.0.0. This is isn't
necessary, but the challenge is interesting.

What we want to do first is see if /etc/tinydns and /etc/dnscache exist
and if not, run the -conf program. We also need to know the IP address.
Fortunately, puppet provides this as a variable "$ipaddress". Try
running the "facter" command.

Puppet has a property call creates that is ideal. If the directory
specified by creates does not exist, it will perform the associated
commands. Here are our new lines:

exec { "configure-tinydns":
            command => "/usr/bin/tinydns-conf tinydns dnslog
/etc/tinydns $ipaddress",
            creates => "/etc/tinydns",
            require => Package['dbndns'],
    }

exec { "configure-dnscache":
            command => "/usr/bin/dnscache-conf dnscache dnslog
/etc/dnscache 127.0.0.1",
            creates => "/etc/dnscache",
            require => Package['dbndns'],
    }

Thos will configure tinydns and dnscache, and then we can restrict
dnscache

file { "/etc/dnscache/root/ip/127.0.0":
            ensure => "present",
            owner => "dnscache",
            require => Exec["configure-dnscache"],
    }

Then, we need to create the /etc/service directory and bring tinydns and
dnscache under svscan's control.

    file { "/etc/service":
            ensure => "directory",
            require => Package["dbndns"],
    }

file { "/etc/service/tinydns":
            ensure => "link",
            target => "/etc/tinydns",
            require => [ File['/etc/service'],
Exec["configure-tinydns"], ],
    }

file { "/etc/service/dnscache":
            ensure => "link",
            target => "/etc/dnscache",
            require => [  File['/etc/service'],
Exec["configure-dnscache"]  ],
    }

And our tests:

johnh@puppet2:~$ sudo puppet agent --test
    info: Retrieving plugin
    info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
    info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
    info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
    info: Caching catalog for puppet2.lab.example.net
    info: Applying configuration version '1340218775'
    notice: /Stage[main]/Dbndns/Exec[configure-dnscache]/returns:
executed successfull
    notice:
/Stage[main]/Dbndns/File[/etc/dnscache/root/ip/127.0.0]/ensure: created
    notice: /Stage[main]/Dbndns/File[/etc/service/dnscache]/ensure:
created
    notice: /Stage[main]/Dbndns/Exec[configure-tinydns]/returns:
executed successfully
    notice: /Stage[main]/Dbndns/File[/etc/service/tinydns]/ensure:
created
    notice: Finished catalog run in 0.59 seconds
    johnh@puppet2:~$ ls /etc/service/tinydns/root/
    add-alias  add-alias6  add-childns  add-host  add-host6  add-mx 
add-ns  data  Makefile
    johnh@puppet2:~$ ps ax | grep supervise
     7932 ?        S      0:00 supervise dnscache
     7933 ?        S      0:00 supervise log
     7934 ?        S      0:00 supervise tinydns
     7935 ?        S      0:00 supervise log

Doing a dig www.example.net @localhost returns 192.0.43.10, so dnscache
works.

Now, let's check tinydns. No domains are configured yet, so let's put
example.com in there. Edit /etc/tinydns/root/data and put these lines in
it, substituting 10.100.0.178 for your own "public" IP address.

&example.com::ns0.example.com.:3600 
Zexample.com:ns0.example.com.:hostmaster.example.com.:1188079131:16384:2048:1048576:2560:2560
 +ns0.example.com:10.100.0.178:3600

Then "make" the data.cdb file:

cd /etc/tinydns/root ; sudo make

Now test:

johnh@puppet2:/etc/tinydns/root$ dig ns0.example.com @10.100.0.178

; <<>> DiG 9.8.1-P1 <<>> ns0.example.com @10.100.0.178
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25433
    ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL:
0
    ;; WARNING: recursion requested but not available

;; QUESTION SECTION:
    ;ns0.example.com.               IN      A

;; ANSWER SECTION:
    ns0.example.com.        3600    IN      A       10.100.0.178

;; AUTHORITY SECTION:
    example.com.            3600    IN      NS      ns0.example.com.

Ok, for a final test, let's remove everything and run it again.

sudo service svscan stop
    sudo apt-get purge daemontools daemontools-run ucspi-tcp dbndns
    sudo rm -rf /etc/service /etc/tinydns /etc/dnscache
    sudo userdel tinydns
    sudo userdel dnslog
    sudo userdel dnscache

Let's do this:

johnh@puppet2:~$ sudo puppet agent --test
    info: Retrieving plugin
    info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
    info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
    info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
    info: Caching catalog for puppet2.lab.example.net
    info: Applying configuration version '1340220032'
    notice: /Stage[main]/Dbndns/Service[svscan]/ensure: ensure changed
'stopped' to 'running'
    err: /Stage[main]/Dbndns/Exec[configure-dnscache]/returns: change
from notrun to 0 failed: /usr/bin/dnscache-conf dnscache dnslog
/etc/dnscache 127.0.0.1 returned 111 instead of one of [0] at
/etc/puppet/modules/dbndns/manifests/init.pp:47
    notice: /Stage[main]/Dbndns/User[dnscache]/ensure: created
    notice: /Stage[main]/Dbndns/User[tinydns]/ensure: created
    notice: /Stage[main]/Dbndns/File[/etc/service]/ensure: created
    notice: /Stage[main]/Dbndns/File[/etc/service/dnscache]: Dependency
Exec[configure-dnscache] has failures: true
    warning: /Stage[main]/Dbndns/File[/etc/service/dnscache]: Skipping
because of failed dependencies
    notice: /Stage[main]/Dbndns/User[dnslog]/ensure: created
    notice: /Stage[main]/Dbndns/File[/etc/dnscache/root/ip/127.0.0]:
Dependency Exec[configure-dnscache] has failures: true
    warning: /Stage[main]/Dbndns/File[/etc/dnscache/root/ip/127.0.0]:
Skipping because of failed dependencies
    notice: /Stage[main]/Dbndns/Exec[configure-tinydns]/returns:
executed successfully
    notice: /Stage[main]/Dbndns/File[/etc/service/tinydns]/ensure:
created
    notice: Finished catalog run in 0.98 seconds

Looks like we had something fail. Oops! configure-dnscache failed. We
see that the user dnscache and tinydns were created after. So we need to
make sure that the users are created before we can configure the
service. This needs to happen to tinydns as well as dnscache. Good thing
we did this test so it doesn't bite us in the future. Let's adjust our
init.pp

exec { "configure-tinydns":
                command => "/usr/bin/tinydns-conf tinydns dnslog
/etc/tinydns $ipaddress",
                creates => "/etc/tinydns",
                require => [ Package['dbndns'], User['dnscache'],
User['dnslog'] ],
        }

exec { "configure-dnscache":
                command => "/usr/bin/dnscache-conf dnscache dnslog
/etc/dnscache 127.0.0.1",
                creates => "/etc/dnscache",
                require => [ Package['dbndns'],  User['dnscache'],
User['dnslog'] ],
        }

Also, let's go ahead and run our commands above to get rid of everything
again.

johnh@puppet2:~$ sudo puppet agent --test
    info: Retrieving plugin
    info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
    info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
    info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
    info: Caching catalog for puppet2.lab.example.net
    info: Applying configuration version '1340220641'
    notice: /Stage[main]/Dbndns/Service[svscan]/ensure: ensure changed
'stopped' to 'running'
    notice: /Stage[main]/Dbndns/User[dnscache]/ensure: created
    notice: /Stage[main]/Dbndns/User[tinydns]/ensure: created
    notice: /Stage[main]/Dbndns/File[/etc/service]/ensure: created
    notice: /Stage[main]/Dbndns/User[dnslog]/ensure: created
    notice: /Stage[main]/Dbndns/Exec[configure-dnscache]/returns:
executed successfully
    notice: /Stage[main]/Dbndns/File[/etc/service/dnscache]/ensure:
created
    notice:
/Stage[main]/Dbndns/File[/etc/dnscache/root/ip/127.0.0]/ensure: created
    notice: /Stage[main]/Dbndns/Exec[configure-tinydns]/returns:
executed successfully
    notice: /Stage[main]/Dbndns/File[/etc/service/tinydns]/ensure:
created
    notice: Finished catalog run in 1.05 seconds

Everything looks good, but when we run "ps ax | grep svscan" we don't
see svscan running. So we check /var/log/syslog and see this

Jun 20 19:31:35 puppet2 kernel: [ 9646.348251] init: svscan main
process ended, respawning
    Jun 20 19:31:35 puppet2 kernel: [ 9646.359074] init: svscan
respawning too fast, stopped

If we start it by hand, it works, so what happened? /etc/service didn't
exist yet.

johnh@puppet2:~$ sudo service svscan start
    svscan start/running, process 9726
    johnh@puppet2:~$ ps ax | grep supervise
     9730 ?        S      0:00 supervise dnscache
     9731 ?        S      0:00 supervise log
     9732 ?        S      0:00 supervise tinydns
     9733 ?        S      0:00 supervise log

Let's fix that.

define the service to restart

        service { "svscan":
                ensure  => "running",
                provider => "upstart",
                require => [ Package["dbndns"], File["/etc/service"] ]
        }

Now, let's give it a go:

johnh@puppet2:~$ sudo puppet agent --test
    info: Retrieving plugin
    info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
    info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
    info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
    info: Caching catalog for puppet2.lab.example.net
    info: Applying configuration version '1340220885'
    notice: /Stage[main]/Dbndns/User[dnscache]/ensure: created
    notice: /Stage[main]/Dbndns/User[tinydns]/ensure: created
    notice: /Stage[main]/Dbndns/File[/etc/service]/ensure: created
    notice: /Stage[main]/Dbndns/Service[svscan]/ensure: ensure changed
'stopped' to 'running'
    notice: /Stage[main]/Dbndns/User[dnslog]/ensure: created
    notice: /Stage[main]/Dbndns/Exec[configure-dnscache]/returns:
executed successfully
    notice: /Stage[main]/Dbndns/File[/etc/service/dnscache]/ensure:
created
    notice:
/Stage[main]/Dbndns/File[/etc/dnscache/root/ip/127.0.0]/ensure: created
    notice: /Stage[main]/Dbndns/Exec[configure-tinydns]/returns:
executed successfully
    notice: /Stage[main]/Dbndns/File[/etc/service/tinydns]/ensure:
created
    notice: Finished catalog run in 1.24 seconds
    johnh@puppet2:~$ ps ax | grep svscan
    10613 ?        Ss     0:00 /bin/sh /usr/bin/svscanboot
    10615 ?        S      0:00 svscan /etc/service
    10639 pts/0    S+     0:00 grep --color=auto svscan
    johnh@puppet2:~$ ps ax | grep supervise
    10630 ?        S      0:00 supervise dnscache
    10631 ?        S      0:00 supervise log
    10632 ?        S      0:00 supervise tinydns
    10633 ?        S      0:00 supervise log
    10641 pts/0    S+     0:00 grep --color=auto supervise

Excellent! We now have a working puppet class that will install puppet,
configure it, and get it up and running. At this point, we don't have
any records being served by tinydns, but it wouldn't be hard to push a
file to /etc/tinydns/root/data and execute a command to perform the
make. In my case, I will be using VegaDNS's update-data.sh[2] to pull
the data remotely.

Here is our completed modules/dbndns/init.pp:

---

class dbndns {

package {
                dbndns:
                ensure => present;

ucspi-tcp:
                ensure => present;

make:
                ensure => present;
        }

define the service to restart

        service { "svscan":
                ensure  => "running",
                provider => "upstart",
                require => [ Package["dbndns"], File["/etc/service"] ]
        }

user { "dnslog":
                        shell => "/bin/false",
                        managehome => false,
                        ensure => "present",
                }

user { "tinydns":
                        shell => "/bin/false",
                        managehome => false,
                        ensure => "present",
                }

user { "dnscache":
                        shell => "/bin/false",
                        managehome => false,
                        ensure => "present",
                }

exec { "configure-tinydns":
                command => "/usr/bin/tinydns-conf tinydns dnslog
/etc/tinydns $ipaddress",
                creates => "/etc/tinydns",
                require => [ Package['dbndns'], User['dnscache'],
User['dnslog'] ],
        }

exec { "configure-dnscache":
                command => "/usr/bin/dnscache-conf dnscache dnslog
/etc/dnscache 127.0.0.1",
                creates => "/etc/dnscache",
                require => [ Package['dbndns'],  User['dnscache'],
User['dnslog'] ],
        }

file { "/etc/dnscache/root/ip/127.0.0":
                ensure => "present",
                owner => "dnscache",
                require => Exec["configure-dnscache"],
        }

file { "/etc/service":
                ensure => "directory",
                require => Package["dbndns"],
        }

file { "/etc/service/tinydns":
                ensure => "link",
                target => "/etc/tinydns",
                require => [ File['/etc/service'],
                                        Exec["configure-tinydns"],
                                ],
        }

file { "/etc/service/dnscache":
                ensure => "link",
                target => "/etc/dnscache",
                require => [  File['/etc/service'],
                                        Exec["configure-dnscache"]
                                ],
        }

}

---

[1]
http://higherlogic.com.au/2011/djbdns-on-ubuntu-10-04-server-migration-from-bind-and-zone-transfers-to-secondaries-bind/
[2] https://github.com/shupp/VegaDNS/blob/master/update-data.sh

In continuing working with XMPP and Python, I managed to get xmpppy
working.

This proved to be particularly tricky because xmpppy uses some outdated
modules like md5 and sha.  It also relies on some dns functions which no
longer seem to work.

I was able to use the sample script "xsend.py" that the project
provides.  The big thing is that I had to specify talk.google.com and
the port number directly in the connect command.

If I did not specify the servername, I would get the error:
 An error occurred while looking up _xmpp-client._tcp.example.com

NOTE: DNS is configured correctly for the domain I was practicing with,
but the xmpppy code was not able to resolve it.

To run the script, simply execute;

./xblast.py [email protected]  text to send

The Script:

!/usr/bin/python

$Id: xsend.py,v 1.8 2006/10/06 12:30:42 normanr Exp $

import sys,os,xmpp,time

if len(sys.argv) < 2:
    print "Syntax: xsend JID text"
    sys.exit(0)

tojid=sys.argv[1]
text=' '.join(sys.argv[2:])

jidparams={}

jidparams['jid']='[email protected]'
jidparams['password'] = '123456'

jid=xmpp.protocol.JID(jidparams['jid'])
cl=xmpp.Client(jid.getDomain(),debug=True)

con=cl.connect(('talk.google.com',5222),  use_srv=False)

if not con:
    print 'could not connect!'
    sys.exit()
print 'connected with',con
auth=cl.auth(jid.getNode(),jidparams['password'],resource=jid.getResource())
if not auth:
    print 'could not authenticate!'
    sys.exit()
print 'authenticated using',auth

cl.SendInitPresence(requestRoster=0)   # you may need to uncomment

this for old server
id=cl.send(xmpp.protocol.Message(tojid,text))
print 'sent message with id',id

time.sleep(1)   # some older servers will not send the message if you
disconnect immediately after sending

cl.disconnect()

---

I found that someone has written a perl script geared towards sending
alerts from Nagios to XMPP usernames.

http://www.gridpp.ac.uk/wiki/Nagios_jabber_notification

I have downloaded this, but have not yet got it working as of yet, but
it does look promising.  It took me a while to update all the
dependencies for it, but if those were in place, the installation itself
is rather simple.  That is, the notification script works, but I haven't
actually configured nagios as of yet.

This script has a shortcoming in that it only accepts the username
portion of the JID and not the domain name -- and this means that
notifications can only be sent between users of the same domain.

To illustrate, [email protected] can not send a message to
[email protected], but user1 and user2 on example.net can send to each
other.

# this command will fail:
ytjohn@monitor:/usr/local/bin$ ./notify_by_jabber.pl
[email protected] testing 
# while this one works
ytjohn@monitor:/usr/local/bin$ ./notify_by_jabber.pl yourtech
testing

And in the perl script, you have to specify the login credentials and
server you're connecting to:

## Configuration
# my $username = '[email protected]';   # does not work
my $username = 'system';
my $password = "password";
my $resource = "nagios";
## End of configuration

my $len = scalar @ARGV;
if ($len ne 2) {
   die "Usage...n $0 [jabberid] [message]n";
}
my @field=split(/,/,$ARGV[0]);
#------------------------------------

# Google Talk & Jabber parameters :

my $hostname = 'talk.google.com';
my $port = 5222;
# componentname is the second half of your JID:
my $componentname = 'example.com';
my $connectiontype = 'tcpip';
my $tls = 1;

I am getting more and more into the idea of using XMPP for sending
messages.  XMPP (also known as jabber) is a very open protocol.  Anyone
can throw up a server, and by publishing a few DNS records, your server
can interact with any other XMPP server out there (that is open).  For a
popular example, anyone can throw up their server with their domain and
send messages to someone using Google Chat.

I have a customer that has their domain email hosted by Google apps for
domains, which also means that each person has Google Talk, which is
based off of XMPP.  For a monitoring scenario, I decided I'd rather have
alerts go to an installed Google Talk client instead of to their email. 
This would make alerts more noticeable, but less intrusive to their
Inbox.  Google would also tie all their messages together (chat and
email) for searching later. So my idea was to have monitoring server
send the email alert to a local user, and have it piped through a
program that would send the XMPP alert.  In the future, perhaps an XMPP
bot could also accept commands to control the monitoring system.

I wanted to create a program called "jabblast.py" which would accept
either standard input, or a command line argument, and send a message to
a pre-defined list of recipients.

echo "sent from stdin" | jabblast.py
  jabblast.py -m "sent from a command line argument"

Looking at the python and XMPP options out there, I came across three
that looked promising. 

# xmpppy
  # sleekxmpp
  # twisted (the "words" extension)

I was really expecting to find exactly what I wanted to do already
written.  However, it seems anyone writing an XMPP script is creating a
bot that runs continuously and would need modified for a one-time
event.

Sleekxmpp was actually the easiest to understand and work with, however
the project could be in danger of abandonment.   Twisted looks the most
advanced, and I am drawn to it because it seems to offer the most
versatility.  In my mind, I suspect that if I learn twisted for this, I
can just keep using it for all the other stuff I want to do.  However,
it might be "too much" for this simple of an operation.  xmpppy is the
middle ground.  The project seems relatively active and popular. 

The problem with all three projects is a severe lack of documentation --
for twisted, this lack is primarily on the xmpp side.  One thing very
well hidden is how to specify the server name you want to connect to. 
Ideally, this is provided through a DNS record lookup, but that method
eliminates experimentation with local network ip addresses.   Sleek
never even approaches the concept that you might want your XMPP program
to end.   It has a call for disconnecting, but when I call that, sleek
immediately tries to reconnect.

I'm going to go ahead and create my program using all three methods and
decide later which I like best.

For these examples, I have hardcoded the login credentials, as well as
the "to" into the source code itself.  In the real world, I should be
pulling that from a configuration file, either XML or YAML (and
potentially from a remote web page).  Secondly, I only list one "to".  I
believe that going from sending a single message to sending multiple
messages should be a matter of a for loop.  Some libraries might even
offer a feature to send to multiple recipients at once.

For my first example, I have created a script in sleekxmpp.  I wrote
this by looking at the EchoBot example found here.  The script will
send a message and then hang for a while after disconnecting before it
finally exits.  I don't know why.  If you enable logging, an error is
thrown after you pass in the disconnect command.  But otherwise, a
relatively simple bit of code, easy to expand upon.

#!/usr/bin/env python

import sys
import logging
import sleekxmpp

# Uncomment the following line to turn on debugging
#logging.basicConfig(level=logging.DEBUG, format="%(levelname)-8s
%(message)s")

def main() :
  # configure jabber id credentials and to
  to = '[email protected]'
  myjid = '[email protected]/blastbot'
  mysecret = 'password'

args = sys.argv[1:]
  if not args:
    message = sys.stdin.read()
  else:
    message = sys.argv[1]

# print to, message
  # sys.exit()

bot = BlastBot(myjid, mysecret, to, message)
  bot.run()

class BlastBot :

def __init__(self, jid, password, to, message) :
    self.message = message
    self.to = to
    self.xmpp = sleekxmpp.ClientXMPP(jid, password)
    self.xmpp.add_event_handler("session_start",
self.handleXMPPConnected)
    self.xmpp.add_event_handler("disconnected",
self.handleXMPPDisconnected)

def run(self) :
    self.xmpp.connect()
    self.xmpp.process(threaded=False)

def handleXMPPDisconnected(self, event) :
    sys.exit()

def handleXMPPConnected(self, event) :
    self.xmpp.sendMessage(self.to, self.message)
    self.xmpp.disconnect(self)

if __name__ == "__main__" :
  main()

On one of my customer's devices, the backup software requires that the
backups go to a separate partition (or drive).  However, the customer
only has one raid array and the bulk of the space is in /home.   To work
around this limitation, I created a raw image file called backup.img,
which gets mounted as /backup.   After the software performs its local
backup, I use duplicity to backup /backup remotely to a backup server at
my location (with encryption).

Today I got an alert that /backup was running low on space.  It was an
80GB image and 61GB was in use, leaving only 15GB free.  Now, this
amount of free space should last quite a while.  However, the software
(cPanel)  has a known issue for years that the 80% limit is hardcoded
into the program.  I can change this, but every time cPanel updates, it
overwrites that change.

So to be proactive, I decided to go ahead and increase the image size.

In order to increase the size of an image, you simply unmount your raw
image and use the dd command.

# Increase by ~20GB
dd if=/dev/zero bs=1M count=20480 >> backup.img
# 20,480 is 20,480 MB or ~20GB

# check the filesystem
/sbin/e2fsck -f backup.img
# resize the filesystem
/sbin/resize2fs backup.img
# check the filesystem again
e2fsck -f backup.img               

I am reasonably familiar with Perl and PHP languages.  Recently though,
I have taken it upon myself to learn Python.  I've played with Python a
few times, and it seems to be very powerful with lots of useful
functionality under the hood.

For some basic practice, I recommend Learn Python The Hard Way,
which is free book by a reputable Python author, Zed Shaw.  He makes the
book itself available for free as a PDF download or you can purchase it
directly.  The book walks you through a number of examples, each one
building upon what you learned in the previous exercise.  You are
required to enter each exercise in by hand.  This uses the principle
that if you learn better by writing things down, you'll learn better by
typing it in.  Copying and pasting completely bypasses that manual entry
learning process, so if you want to learn, I suggest you follow Shaw's
instructions.  I found the book rather insightful, but I often felt that
certain aspects were not being properly explained to me.  To be fair,
I'm only up to exercise 15 in a 52 exercise book.

However, I also came across another way to learn Python.  Google has had
instructor Nick Parlante teach Python in a classroom setting.  The
class was video taped and put online along with the associated
learning aids (primarily a group of python files).  The class was taught
in 7 segments over 2 days.  So if you go to the class website, you can download the learning aids, and then follow along with
the lecture videos. 

I like this method better than following the static book.  Since it was
taught in an interactive classroom, it goes at a pace where you can keep
up with it from your own computer.  Also, students in the class were
able to ask questions, which allows the information to be represented in
a different way.  Finally, at the end of each segment, there are some
sample files you can open up in your editor.  Each file has a skeleton
template.  In comments, it tells you what a segment is supposed to do,
and you have to implement it in code based on what you learned while
watching the video.  The sample file also includes some bare bones "unit
tests" that call your functions, passing it input data and comparing it
to an expected output.  These unit tests will show what they received
from your function and what they expected.  If the two matched, you have
succeeded. 

I like these practice samples mainly because it immediately forces you
to apply the knowledge you learned.  You do have to think about it and
do some back and forth troubleshooting on your own code.  Since they
provided the unit tests, you can get instant feedback on if you're doing
it correctly or not.

If you want to learn Python and can dedicate an hour on any given day to
it, I recommend the Google course.  If you want to go in smaller 10-20
minute blocks, try using Learn Python The Hard Way.  Of course, you can
pause the videos and come back to them, but I think it's much better to
go through the entire video, following along on your own system as you
go.