Tags: python console, python, import, pysphere

This is a quick little python tip. When experimenting with python commands and modules, it's usually easiest to use the python console interactively, then create your programs later. The downside of this is that sometimes you have to do a bit of typing before you get to the specific command you want to try.

Imagine the following example:

johnh@puppet2:~$ python
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pysphere import *
>>> server = VIServer()
>>> server.connect("vc1.example.com", "username", "password")
>>> print server.get_server_type(), server.get_api_version()
VMware vCenter Server 5.0

Here, I had to type in 3 lines, including my password in plaintext, to test out querying the server. I can't demonstrate this live, because then I reveal my password. Well last week, I made a [test1.py] file that reads a yaml configuration file and does the commands I just did. Here's the smart bit. I can import that file directly into the python console. Once it imports, it runs each python command and leaves me in the console, ready to query the system again. The only caveat is that my "server" variable is now part of the test1 module as "test1.server".

johnh@puppet2:~$ python
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test1
VMware vCenter Server 5.0
puppet1-centos6 10.100.0.206
>>> test1.server.is_connected()
True
>>> vmlist = test1.server.get_registered_vms()
>>> for v in vmlist:
...     print v
...
[iscsi-244gb-raid10] rhel-puppet1/rhel-puppet1.vmx  
[iscsi-244gb-raid10] puppet2-ubuntu 12.04LTS server/puppet2-ubuntu 12.04LTS server.vmx
[iscsi-488gb-raid10] puppet3-solaris11/puppet3-solaris11.vmx

Tags: api, python, programming, vmware, pysphere

I do a good bit of work with VMWare vSphere and I've been wanting to
work more with their API. Everything (except console) that you can do in
the vSphere client, you should be able to do through a web-based API.
Unfortunately, it seems that VMWare does not provide an SDK for Python,
my current language of choice. I could work in Perl or Java, but I want
to develop a web application, which I don't want to do in Perl.
Fortunately, I found pysphere, which is a fairly active project of
implementing the VI API in python. It might not fully implement the API,
but it looks relatively stable and easy to implement. Plus, if I find
any functionality missing, I can extend the class directly.
I followed their Getting Started page to get it installed
and get connected, but I didn't like having my password right there in
my working code. This was easily resolved by installing PyYaml and
creating a config.yaml file. Then, it was just a matter of following
along with the examples to make a good test script.
My config.yaml:

server: esxi1.example.com
user: john
pass: password

My test.py:

#!/usr/bin/python

import yaml
from pysphere import *

f = open('config.yaml')
config = yaml.load(f)
f.close()

server = VIServer()
server.connect(config["server"], config["user"], config["pass"])

print server.get_server_type(), server.get_api_version()
vm1 = server.get_vm_by_name("puppet1-centos6")
print vm1.get_property('name'), vm1.get_property('ip_address')

And does it work?

$ ./test.py
VMware vCenter Server 5.0
puppet1-centos6 10.100.0.206

I was even able to go so far as cloning a vm (vm2 = vm1.clone('new vm'))
and can already see massive possibilities with this library in its
current state. The API can be queried much like a simple database, and
objects acted upon with simple statements. Operations like my vm clone
can be setup as a task and run asynchrously. I could easily see
integrating this with something like tornado, twisted, or even
cyclone to make a non-blocking web framework.

Currently, I have Redmine version redmine 1.3.3 installed via the
ondrej/redmine PPA. I have been wanting to upgrade to the 2.x series
of redmine, but no PPA currently exists for it. Redmine is officially
provided by Ubuntu, but the version for Precise is 1.3.2, and Ondřej's
PPA is on 1.4.3. While I usually prefer to have my software installation
and updates handled by packages, it looks like to get to the 2.x series,
I'll have to go back to source.

I will be following the official upgrade guide closely, but with a
few variations.

1. The apt-get/ppa version uses multiple file locations for source code
   and configuration. I'll have to consolidate to one place.
2. My ruby and passenger modules were installed and modified to work
   with the ppa version of redmine. Adjustments will be needed.

My ruby version is 1.8.7 (1.8.7 min), rails 2.3.14 (2.3.6 min) and gem
1.8.15 (1.8 min. Already having the minimum requirements makes this a
bit easier.

After performing a mysql backup (hint: database.yml is in
/etc/redmine/default), I downloaded redmine to /usr/local/redmine-2.0. I
also decided to stop Apache so that Passenger wouldn't be running the
existing redmine instance. If I had other sites running on this server,
I would have disabled this virtual host or put up a maintenance page.

cp /etc/default/database.yml /usr/local/redmine-2.0/config
cp /var/lib/redmine/default/files/* /usr/local/redmine-2.0/files

I didn't have any plugins, but if I did they would either be in
/usr/share/redmine/vendor/plugins or /usr/share/redmine/lib/plugins. I
do intend to install a couple plugins when I get into 2.x though.

I found in step 3 that the rake commands didn't work. This is probably
because I wasn't working from an existing rails directory. I went to the
Redmine Installer page, which gave me the answer. "Since 1.4.0,
Redmine uses Bundler to manage gems dependencies. You need to install
Bundler first.".

cp /etc/default/database.yml /usr/local/redmine-2.0/config
cp /var/lib/redmine/default/files/* /usr/local/redmine-2.0/files

I ran into an error when bundle was installing json 1.7.4.

gem install bundler
# run the next command from /usr/local/redmine-2.0
# it reads the file "Gemfile"
bundle install --without development test

According to an about.com page, I need build-essentials,
libopenssl-ruby, and ruby1.8-dev installed. The one I was missing was
ruby1.8-dev. This is easily fixed with an apt-get install ruby1.8-dev.

I had to install the following other packages for certain gems. The
Gemfile includes items for postgresql and sqlite, even if you don't use
it. The install guide lets you know that you can
skip these with the --without option. You would just add "pg sqlite
rmagick" to the end of your bundle install line (above).

• json: build-essentials, libopenssl-ruby, and ruby1.8-dev
• mysql: libmysqlclient-dev
• pg: libpq-dev (alternatively: add pg to the --without list)
• rmagick: libmagickcore-dev, libmagickwand-dev (alternatively: add
  rmagick to the --without list)
• sqlite: libsqlite3-dev

Once we got Bundler installed and all the required gems, we switch back
to the Upgrade Guide to update our session
store and migrate the database. I had no plugins, so I'm skipping that
step.

/usr/bin/ruby1.8 extconf.rb
extconf.rb:1:in `require': no such file to load -- mkmf (LoadError)
from extconf.rb:1

Let's start this locally before we mess with passenger or apache (be
sure to allow port 3000 via iptables or ufw).

rake generate_secret_token
rake db:migrate RAILS_ENV=production 
# unecessary, as this is a new directory, but why not clean up?
rake tmp:cache:clear
rake tmp:sessions:clear

This worked without a hitch for me. Now on to my passenger setup. I
already has this configured and installed previously, so all I have to
do is change my VirtualHost directory.

ServerName projects.example.com
DocumentRoot /usr/local/redmine-2.0/public
RailsSpawnMethod smart
# Keep the application instances alive longer. Default is 300 (seconds)
PassengerPoolIdleTime 1000
RailsAppSpawnerIdleTime 0
RailsFrameworkSpawnerIdleTime 0
PassengerMaxRequests 5000
PassengerStatThrottleRate 5

        AllowOverride all
        Options -MultiViews

I did have to change a few permisssions (all files installed as owned by
root)

chgrp -R www-data config
chown -R www-data files
chown -R www-data log

I recently have started a process of migrating my website over to
blogger.com. One of the main reasons for this was because in my last
server move, I had broken my Movable Type installation, and found myself
too busy to fix it. I found I didn't want to spend my time fixing and
updating blogging software. I wanted to work on my projects, write them
up, and post them. It was time to move my content to an existing
platform that handled the back end. I looked at a few, and decided
blogger.com would be as good as any other service.
It only took a short time to setup a blog, point a CNAME at it, and then
to import my existing posts. When I started creating some new posts, I
immediately ran into some limitations.

1. You used to be able to edit permalinks on blogger. Now, you can only
   do that before you publish. The only way to change a permalink after
   publishing is to create a new post with the desired permalink and
   delete the old one.
2. Blogger has no built in formatting for code blocks. So if I want to
   show a config file, source code, or terminal session log, I have to
   fiddle with changing fonts, size, and "blockquote" to get it
   presentable. Even then, you run the risk of strange formatting of
   your raw text.

I found a solution that other bloggers use called SyntaxHighlighter.
This is a combination of javascript and css code that takes text within
your

; tags and gives you nice looking blocks of code, highlighted and (optionally) with line numbers. The catch is that your pre tags need to have a class name, along with the language (perl/c/bash/text) "brush" to use. If you go with pre tags, you have to change any angle bracks to their HTML escaped equivalents of < and >. They have a work-around using SCRIPT/CDATA, but it takes some getting used to. Adding this to your blog only requires a few steps.

I rather liked syntaxhighlighter, but it still seemed like I had to do a
lot of manual work with the code. Also, I had to select the brush each
time. Couldn't it guess? Notepad++ and some others will guess at what
language you're using and highlight accordingly. I found something
called prettify that does just that. You only need to load one js
file and one css file. Prettify works off of either

 or  tags and has similar limitations to SyntaxHighlighter regarding html tags. However, it has the advantage of being able to guess the language automatically.

Being able to use this code made my posts look much nicer, but the
entire process got me thinking. The way I "document" most of projects
typically invole using a notepad editor like Geany or
Notepad++. As I work, I add notes, copy in source code or shell
comands, and do everything in a plain text editor. Later, I add
commentary and clean up the document. I take this and paste it into the
WYSIWYG editor on blogger. Finally, I have to keep switching between
compose and html mode to get my text looking suitable. There are too
many steps for me to want to do this consistently. All I really want to
do is take my text file, add a little formatting in a few spots, a few
hyperlinks in others, and post it.
Enter markdown. Markdown is a text-to-HTML conversion tool for web
writers. Markdown allows you to write using an easy-to-read,
easy-to-write plain text format, then convert it to structurally valid
XHTML (or HTML). I have used this before, but didn't pay it close
enough attention. It's used on github and reddit, there are plugins for
it in dokuwiki and redmine. The idea is you write in text, adding
formatting using the markdown syntax. This format is both human readable
and machine readable. When read by the appropriate library, clean html
is generated. It also has a feature for wrapping blocks of code inside
of

&;lt;code> tags and html-escaping html inside of those tags.

Within the MarkDown project is a paged called "dingus" which means
"placeholder". You can paste your markdown text into one textarea and
get the generated html plus a preview back. I tested pasting that
generated html into Blogger's HTML box and it seems to work perfectly
fine. What this means is that I can type up my documentation completely
within my text editor of choice, save it locally, and then generate my
html code to paste into blogger.
Some of you may have realized that my

 tags are missing that class name (. Well, I copy the generated html, do a search and replace of  with  and then paste it, but that's adding more steps. Instead, I sought to make my own dingus that does this automatically. I found that there is an extension of markdown called Markdown Extra written in PHP. Extra adds a few features such as simple tables, but remains consistent with original Markdown formatting. Using that library, I was able to create my own dingus rather easily and alter the  tag with one line of code $render2 = str_replace("<pre>", "<pre class="prettyprint linenums">", $render);. In my experimentation, I made a parser that reads a text file and outputs html, and three dingus parsers. Dingus1 does straightforward conversion of markdown extra to html. Dingus2 and 3 provide the class names for prettified code, with #3 going the extra step of applying stylesheets for the preview.

With this setup, I can quickly paste in my text document and pull html
code to paste into blogger.com's html edit box. With some more research,
I can modify the dingus to interact with blogger's API and post on my
behalf. There are also some WYSIWYM live editors that show you an
instant render of your markdown as you type (you type in a textarea
while your html renders in a nearby div). This would be a good way to do
some tweaking to the markdown text before posting the html to the web.
My next plans are to make a better dingus, possibly with a live preview
and a "post to blogger" option.
Some other links:

• http://balupton.github.com/jquery-syntaxhighlighter/demo/
• http://code.google.com/p/pagedown/wiki/PageDown
• http://markitup.jaysalvat.com/examples/markdown/

Yesterday, I setup gateone to run as a non-root user. I also spent
some time looking at potential init scripts for starting and stopping
this. The gateone project does not currently provide any init scripts,
but this is planned for the future ([Issue #47]). I tried to use one
of the scripts in that thread, but I wasn't really pleased with them.
The big issue is that gateone.py doesn't fork. However, I believe there
is a better solution.

supervisord is a python script designed to control and monitor
non-daemonizing python scripts. As Gate One is a foreground only
process, it seems particularly suited to this task - more so than
writing my own script in python or daemontools.

Installation can be done with python-pip or easy_install. On newer
systems, pip is recommended.

sudo pip install supervisor

On Ubuntu, pip installs supervisord to /usr/local/bin. By default,
/usr/local/bin is not in root's path, so it makes sense (to me at least)
to create symlinks to /usr/sbin.

johnh@puppet2:~$ ls /usr/local/bin
echo_supervisord_conf  pidproxy  supervisorctl  
supervisordjohnh@puppet2:~$ sudo ln -s /usr/local/bin/supervisord /usr/sbin
johnh@puppet2:~$ sudo ln -s /usr/local/bin/supervisorctl /usr/sbin

Now, we need to create a configuration file. Supervisord has a utility
to generate a sample one.

echo_supervisord_conf  > supervisord.conf

To get started, we can use the sample configuration and just add a
couple lines to the bottom for gateone.

 [program:gateone]
 command=/opt/gateone/gateone.py
 directory=/opt/gateone
 ;user=johnh   ; Default is root. Add a user= to setuid

Now, copy supervisord.conf to /etc/supervisord.conf and start
supervisord. Make sure gateone.py is not currently running. Then we'll
run supervisorctl to test things out.

johnh@puppet2:~$ sudo cp supervisord.conf /etc
johnh@puppet2:~$ sudo supervisord
johnh@puppet2:~$ sudo supervisorctl status
gateone                          RUNNING    pid 9549, uptime 0:00:05
johnh@puppet2:~$ ps ax | grep gateone
 9549 ?        Sl     0:00 python /opt/gateone/gateone.py
johnh@puppet2:~$ sudo supervisorctl stop gateone
gateone: stopped
johnh@puppet2:~$ ps ax | grep gateone
 9605 ?        Ss     0:00 dtach -c /opt/gateone/tmp/gateone/../dtach_3 -E -z -r none /opt/gateone/plugins/ssh/scripts/ssh_connect.py -S /tmp/gateone/.../%SHORT_SOCKET% --sshfp -a -oUserKnownHostsFile=/opt/gateone/users/user1@puppet2/ssh/known_hosts
 9606 pts/3    Ss+    0:00 python /opt/gateone/plugins/ssh/scripts/ssh_connect.py -S /tmp/gateone/.../%SHORT_SOCKET% --sshfp -a -oUserKnownHostsFile=/opt/gateone/users/user1@puppet2/ssh/known_hosts

In this example, we see that gateone.py is started and stopped by
supervisorctl, but because we have dtach enabled, our sessions are still
in place. If we restart gateone.py, we can connect to it again and have
our sessions resumed. While we could probably configure supervisord to
kill these terminals, I believe we'd normally want to keep them running.
The few times I would want to stop those terminals would be a) manually
reconfiguring/troubleshooting opengate, b) updating software, or c)
rebooting the server. For a&b, running the command "gateone.py -kill"
will kill those terminals. For a server shutdown or reboot, the act of
shutting down the OS will kill these terminals.

Finally, we need a way to start and stop supervisord itself.
Fortunately, the supervisord project has provided a number of init
scripts. I was able to use the Debian script in Ubuntu with only
a few minor changes.

1. I had symlinked supervisord and supervisorctl to /usr/sbin. The
   script expects them in /usr/bin (but even says that /usr/sbin is a
   better location). I had to change /usr/bin to /usr/sbin.
   Alternatively, you can symlink the files into /usr/bin
2. I added a status option that runs $SUPERVISORCTL status
3. If you started supervisord manually, you must shut it down and start
   it with the script. The script won't be able to stop supervisord
   unless /var/run/supervisord.pid is current.

Here is my complete init script for Ubuntu:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          supervisord
# Required-Start:    $local_fs $remote_fs $networking
# Required-Stop:     $local_fs $remote_fs $networking
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts supervisord - see http://supervisord.org
# Description:       Starts and stops supervisord as needed 
### END INIT INFO
# Author: Leonard Norrgard 
# Version 1.0-alpha
# Based on the /etc/init.d/skeleton script in Debian.
# Please note: This script is not yet well tested. What little testing
# that actually was done was only on supervisor 2.2b1.
# Do NOT "set -e"
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Run a set of applications as daemons."
NAME=supervisord
# Supervisord is installed in /usr/bin by default, but /usr/sbin would 
# make more sense
DAEMON=/usr/sbin/$NAME   
SUPERVISORCTL=/usr/sbin/supervisorctl
PIDFILE=/var/run/$NAME.pid
DAEMON_ARGS="--pidfile ${PIDFILE}"
SCRIPTNAME=/etc/init.d/$NAME
# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0
# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions
#
# Function that starts the daemon/service
#
do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        [ -e $PIDFILE ] && return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- 
                $DAEMON_ARGS  
                || return 2
        # Add code here, if necessary, that waits for the process to be ready
        # to handle requests from services started subsequently which depend
        # on this one.  As a last resort, sleep for some time.
}
#
# Function that stops the daemon/service
#
do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        [ -e $PIDFILE ] || return 1
        # Stop all processes under supervisord control.
        $SUPERVISORCTL stop all
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE  
             --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
}
#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
        return 0
}
case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  #reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        #log_daemon_msg "Reloading $DESC" "$NAME"
        #do_reload
        #log_end_msg $?
        #;;
  restart|force-reload)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
                do_start
                case "$?" in
                        0) log_end_msg 0 ;;
                        1) log_end_msg 1 ;; # Old process is still running
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
          *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
  status)
        $SUPERVISORCTL status
        RETVAL=$?
        ;;
  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload|status}" >&2
        exit 3
        ;;
esac

And here is a complete copy of my supervisord.conf file:

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
[program:gateone]
command=/opt/gateone/gateone.py
directory=/opt/gateone
stdout_logfile=/opt/gateone/logs/supervisor.log
user=johnh

I came across a program called Gate One by LiftOff Software that
just amazed me. This is an open-source, web-based ssh terminal. It is
capable of multiple users, sessions, and bookmarks. I've tried a number
of AJAX terminals or Java applet based ones in the past. The javascript
ones usually did not have very good terminal emulation, while the Java
apps worked, but worked just like a local desktop app (making it's own
connection to port 22). Gate One uses WebSockets, allowing for full
duplex communication through your web browser over the same port 80 or
443 used to serve up the web page.

Installation

Gate One is a python application using the Tornado framework. As
such, at runs independently of an existing web server and handles
connections from browsers internally. If you already have a web server
running on your system, you will need to tell Gate One to use a
different IP or a different port.

Installation using pre-built binaries or the source is fairly
straightforward and detailed in the documentation.

The installer creates a directory of /opt/gateone and places all
necessary files there. You can run it by changing to that directory and
running gateone.py as root.

johnh@puppet2:/opt/gateone$ sudo ./gateone.py
[W 120801 13:52:06 terminal:166] Could not import the Python Imaging Library (PIL) so images will not be displayed in the terminal
[W 120801 13:52:06 gateone:2232] dtach command not found.  dtach support has been disabled.
[I 120801 13:52:06 gateone:1800] No authentication method configured. All users will be ANONYMOUS
[I 120801 13:52:06 gateone:1876] Loaded plugins: bookmarks, help, logging, logging_plugin, notice, playback, ssh
[I 120801 13:52:06 gateone:2329] Listening on https://*:443/

At this point, gateone is running in the foreground and you can view as
connections occur and any errors. Pressing Ctrl If you conect to gateone
using your webbrowser, you are logged in as user ANONYMOUS and can
connect to any ssh host, either localhost or something remote.

If you edit /opt/gateone/server.conf, you can change authentication to
"pam" or "google". Using pam will perform a Basic HTTP style
authenication requiring a system-level username and password. Using
google will log you in with your google account. Both of these "just
work" without complicated setup.

Running as a Non-Root

Before I put something like this in production, I wanted to apply some
additional security. First off, I want to see if I can get this to run
as a non-root user.

Since gateone ran as root user initially, it has files owned by rootOnly UID 0 can open ports below 1024.gateone may need permission to write to system directories

To solve the first one, I chowned the /opt/gateone directory to my
username. In the future, I'll want to run it under its own user, but
I'll use mine for now for simplicity. To solve the second and third, I
edited server.conf.

johnh@puppet2:/opt/gateone$ sudo chown -R johnh:johnh .
johnh@puppet2:/opt/gateone$ vi server.conf# change/add the following lines appropriatelyport = 2443session_dir = "/opt/gateone/tmp/gateone"pid_file = "/opt/gateone/tmp/gateone.pid"uid = 1000gid = 1000
johnh@puppet2:/opt/gateone$ ./gateone.py
[W 120801 14:06:01 terminal:166] Could not import the Python Imaging Library (PIL) so images will not be displayed in the terminal
[W 120801 14:06:01 gateone:2232] dtach command not found.  dtach support has been disabled.
[I 120801 14:06:01 gateone:1802] No authentication method configured. All users will be ANONYMOUS
[I 120801 14:06:01 gateone:1876] Loaded plugins: bookmarks, help, logging, logging_plugin, notice, playback, ssh
[I 120801 14:06:01 gateone:2329] Listening on https://*:2443/

Authentication

Running as a lower uid, you can use authentication of None or "google"
without issue. If you use "pam", you discover you can only login with
the username that gateone is running under. If you are the only intended
user of the service, this may not be an issue. But if you want to allow
other users, this becomes an issue. If you are fine with running as root
or using Google as your authentication provider, you can ignore this
next step.

Fortunately, pam is highly configurable. You aren't required to
authenticate against shadow passwords. You can also authenticate against
db4 files with pam_userdb, msyql, or even htpasswd files. To start off,
I'm going to use htpasswd files. Note that Ubuntu doesn't provide
pam_pwdfile.so by default. You need to install libpam-pwdfile ("sudo
apt-get install libpam-pwdfile").

Note - in testing, I discovered gateone uses Crypt encryption while htpasswd defaults to MD5. Use -d to switch to crypt encryption.

johnh@puppet2:/opt/gateone$ htpasswd -c -d users.passwd user1
New password:
Re-type new password:
Adding password for user user1
johnh@puppet2:/opt/gateone$ cat users.passwd
user1:KKEPyZtUf9sadf9

Create a pam module called gateone under /etc/pam.d

johnh@puppet2:/opt/gateone$ cat /etc/pam.d/gateone
#%PAM-1.0
# Login using a htpasswd file
@include common-sessionauth    
required pam_pwdfile.so          pwdfile /opt/gateone/users.passwdaccount 
required pam_permit.so

Modify server.conf to use pam and pam_service of gateone:

auth = "pam"
pam_service = "gateone"

Now start gateone and log in.

johnh@puppet2:~/g1/gateone$ ./gateone.py
[W 120801 14:59:16 terminal:168] Could not import the Python Imaging Library (PIL) so images will not be displayed in the terminal
[W 120801 14:59:16 gateone:2577] dtach command not found.  dtach support has been disabled.
[I 120801 14:59:16 gateone:2598] Connections to this server will be allowed from the following origins: 'http://localhost https://localhost http://127.0.0.1 https://127.0.0.1 https://puppet2 https://127.0.1.1 https://puppet2:2443'
[I 120801 14:59:16 gateone:2023] Using pam authentication
[I 120801 14:59:16 gateone:2101] Loaded plugins: bookmarks, help, logging, logging_plugin, mobile, notice, playback, ssh
[I 120801 14:59:16 gateone:2706] Listening on https://*:2443/
[I 120801 14:59:16 gateone:2710] Process running with pid 32591
[I 120801 14:59:17 gateone:949] WebSocket opened (user1@gateone).

One additional nice feature with authentication enabled is the ability
to resume sessions - even across different computers or browsers.

Reverse Proxy

(I failed on this part, but felt it was worth recording)

Once I got it working in single user mode, I wanted to go ahead and set
this up under a reverse proxy under Apache. This would allow me to
integrate it into my existing web server under a sub-directory.

First, I edited server.conf to use a URL prefix of /g1/

Second, I tried setting up a ReverseProxy in Apache.

# GateOne 
ProxySSLProxyEngine 
OnProxyPass /g1/ https://localhost:2443/g1/
ProxyPassReverse /g1/ https://localhost:2443/g1/
ProxyPassReverseCookieDomain localhost localhost
ProxyPassReverseCookiePath / /g1/

This almost worked. I had no errors, but the resulting page was
unreadable. However, at the bottom was a clue. "The WebSocket connection
was closed. Will attempt to reconnect every 5 seconds... NOTE: Some web
proxies do not work properly with WebSockets." The problem was Apache
not properly proxying my websocket connection. People have managed to
get this working under nginx, but not Apache.

Searching for a solution led me to a similar question on ServerFault, an
apache-websocket module on github, and a websocket tcp proxy based on
that module.

• http://serverfault.com/questions/290121/configuring-apache2-to-proxy-websocket
• https://github.com/disconnect/apache-websocket
• http://blog.alex.org.uk/2012/02/16/using-apache-websocket-to-proxy-tcp-connection/

In order to get this work, I'll need to download and compile some code.
The apxs command requires the apache-prefork-dev package in
Debian/Ubuntu. Install it with "sudo apt-get install
apache-prefork-dev".

Now we are ready to download the code and install the module:

johnh@puppet2:~$ git clone https://github.com/disconnect/apache-websocket.git
Cloning into 'apache-websocket'..... done
johnh@puppet2:~$ wget http://blog.alex.org.uk/wp-uploads/mod_websocket_tcp_proxy.tar.gz
johnh@puppet2:~$ cd apache-websocket
johnh@puppet2:~/apache-websocket$ sudo apxs2 -i -a -c mod_websocket.c*snip*
johnh@puppet2:~/apache-websocket$ sudo apxs2 -i -a -c mod_websocket_draft76.c*snip*
johnh@puppet2:~$ cd examples
johnh@puppet2:~$ tar -xzvf ../../mod_websocket_tcp_proxy.tar.gzmod_websocket_tcp_proxy.c
johnh@puppet2:~$ cd apache-websocket/examples/
johnh@puppet2:~/apache-websocket/examples$ sudo apxs2 -c -i -a -I.. mod_websocket_tcp_proxy.c
*snip*
chmod 644 /usr/lib/apache2/modules/mod_websocket_tcp_proxy.so
[preparing module `websocket_tcp_proxy' in /etc/apache2/mods-available/websocket_tcp_proxy.load]
Enabling module websocket_tcp_proxy.To activate the new configuration, you need to run:service apache2 restart
johnh@puppet2:~$

Before we restart, I want to remove my Proxy lines and replace them with
the mod_websocket_tcp_proxy lines.

SetHandler websocket-handler        
WebSocketHandler  /usr/lib/apache2/modules/mod_websocket_tcp_proxy.so tcp_proxy_init        
WebSocketTcpProxyBase64 on        
WebSocketTcpProxyHost 127.0.0.1        
WebSocketTcpProxyPort 2443        
WebSocketTcpProxyProtocol base64

Despite all this, I was still unable to get this to work. I even
attempted using the web root (/) as my location. If the Location matches
and your HTTP request is handled by mod_websocket, you get a 404. If
you use Proxy, then your websocket request is handled by mod_proxy.
Mod_proxy wins out over Location matches. Perhaps you can modify
gateone code to have one URL for the web interface and one for
websockets (or maybe it's already in place and we just need to know),
but I don't see a way at this time to get this working under Apache. I
may be able to work with the gateone author and the
mod_websocket_tcp_proxy.c author to come up with a solution. Or I
could try installing nginx. In the meantime, I can continue to run Open
Gate as a non-root user on a non-standard port. Alternatively, I could
find a wrapper to bring port 443 to 2443.