Using the haproxy charm

The haproxy charm in the charmstore is deceptively powerful. I recently had a use case which I thought it would not handle. It turns out, it does.

The details are all in the services config value. In my case, I am replacing the apache charm using balancer and reverseproxy relations.

The apache charm has vhost_https_template and vhost_http_template which gets pasted in as apache httpd config. The haproxy charm services config value has service_options as yaml which works much the same.

In my case, port 80 redirects to 443, so I start with this in yaml:

- service_name: haproxy_service
  service_host: ""
  service_port: 80
  server_options: maxconn 100 cookie S{i} check
      - 'redirect scheme https code 301 if !{ ssl_fc }'

You’ll notice this is almost identical to the default value for the services config:

- service_name: haproxy_service 
  service_host: "" 
  service_port: 80 
  service_options: [balance leastconn, cookie SRVNAME insert] 
  server_options: maxconn 100 cookie S{i} check

I only changed the service_options entry with what would redirect haproxy in a frontend config.

This is where the magic of this haproxy charm happens. The haproxy charm knows which config values are for a frontend haproxy section and also for a backend haproxy section. The charm automatically puts the value in the right place.

The next thing which wasn’t obvious to me from reading the haproxy charm readme is that the Juju application related using reverseproxy relation becomes a backend section and its values will be merged from the services config.


$ juju add-relation my-app haproxy:reverseproxy
$ juju add-relation kibana haproxy:reverseproxy

I can use defaults, or I can make some tweaks to the my-app and kibana applications.

For my use case, I was using apache httpd config RewriteRule ^/?KIBANA/(.*)$ balancer://kibana/$1 [P,L]

The equivalent in haproxy config looks like this:

    acl path_kibana path -m beg  /KIBANA/
    use_backend kibana if path_kibana

and in the kibana backend:

   reqirep  ^([^\ :]*)\ /KIBANA/(.*)     \1\ /\2

The haproxy charm allows all of this to be configured using services config. The related application is automatically set as a service name and your config must match it via service_name yaml.

I use juju2’s juju config command to set the config directly from a yaml file. If you are using juju1 you’ll need to use juju set command. juju config haproxy services=@haproxy-config-services.yaml

- service_name: my-app
  service_host: ""
  service_port: 443
  crts: [DEFAULT]
      - balance leastconn
      - reqadd X-Forwarded-Proto:\ https
      - acl path_kibana path -m beg  /KIBANA/
      - use_backend kibana if path_kibana
  server_options: maxconn 100 cookie S{i} check
- service_name: kibana
      - balance leastconn
      - reqirep  ^([^\ :]*)\ /KIBANA/(.*)     \1\ /\2
      - rspirep ^Location:\ https?://[^/]+/(.*) Location:\ /KIBANA/\1
      - rspirep ^(Set-Cookie:.*)\ Path=(.*) \1\ Path=/KIBANA/\2
  server_options: maxconn 100 cookie S{i} check

The implication here is that there is another application, “my-app” which is also related to haproxy. This config tells haproxy to use my-app as the default application, but if the url starts with /KIBANA/, to use the kibana backend instead of the “my-app” backend. For completeness, I am including the equivalant of apache’s ProxyPassReverse and ProxyPassReverseCookiePath. These are the rspirep… Location and rspirep…Set-Cookie lines in the config respectively.

Continuous Delivery via Unattended Upgrades

I’ll be the first to admit that this is pretty slow for continuous delivery, as the default configuration for unattended upgrades is daily. Adjust the cron configuration at your discretion.


  • CI system which builds apt source packages and dputs them to a PPA.
  • Machine instances configured with these PPA and with unattended-upgrade

The unattended-upgrades package, by default only installs security updates. We can configure it to install updates to packages in our PPA by adding the correct package origin to the config. We get the package origin from apt-cache policy.


$ apt-cache policy

500 trusty/main amd64 Packages
release v=14.04,o=LP-PPA-evarlast-experimental,a=trusty,n=trusty,l=ex per ee m3nt4l,c=main

Extract that LP-PPA-evarlast-experimental from that output and add it to a new section in /etc/apt/apt.conf.d/50unattended-upgrades. If you want, use `cat >> /etc/apt/apt.conf.d/50unattended-upgrades`

Unattended-Upgrade::Origins-Pattern {

Now when unattended-upgrades run, packages from that PPA are considered important enough that they will be installed.

The details for the configuration are in the README here:


nodejs 7 on ubuntu

nodejs 7 was released and nodesource does an excellent job of creating packages of nodejs for use on many operating systems.

I refuse to curl $URL and pipe the results to bash. It scares me (maybe illogically) to trust a script on the internet with access to my local shell.

The commands without the curl pipe to shell are almost as short and run faster*. It is super easy to get nodejs 7.x installed your ubuntu xenial or yackety system.

curl -s | sudo\ apt-key add -
sudo add-apt-repository -u "deb $(lsb_release -c -s) main"
sudo apt install nodejs

* The apt update command is intentionally skipped. The -u option to add-apt-repository optimizes it to only merge available packages with the newly added repository. This is a bit faster, or a lot faster on older machines or slow cloud instances.


Ubuntu Kiosk

This post is a work in progress. I’ll update it as I tweak the solution.

Last Wednesday I was helping a friend build a Kiosk. We tried to follow but it didn’t work. It turns out between using the wrong version of ubuntu (16.04 instead of 14.04) and doing it in a virtual machine, we were all messed up.

There has to be a better way.

There is a secret to debian/ubuntu packages. If you aren’t trying to get them included in debian/ubuntu, you can break most of the rules and get them to do whatever you want. I figured I should be able to use this and make creating a kiosk as easy as apt install kiosk

TL;DR: you can try this by running these two commands on a new ubuntu-server installation:

add-apt-repository ppa:evarlast/kiosk
apt install --no-install-recommends kioskme

The rest of the post describes how I did this.

First, I’m going to create a new PPA on launchpad just for this, so that a user can `add-apt-repository ppa:evarlast/kiosk`

I visit and fill in the fields with kiosk and click activate.

Next, I start a new deb. I may as well build it from source. There might be a better way, but I’ve gotten to know dh (debhelper) a bit, so I’m going to use it.

$ mkdir kioskme ; cd kioskme
$ cat > Makefile
<tab>echo noop
<tab>install -d 755 ${DESTDIR}/usr/bin
<tab>install -m 755 kioskme ${DESTDIR}/usr/bin/kioskme
$ cat > kioskme
xset -dpms
xset s off 
openbox-session & 
while true; do 
  rm -rf ~/.{config,cache}/chromium/ 
  chromium-browser --kiosk --no-first-run '' 

Now debianize this script directory using dh_make:

dh_make -p kioskme_0.0.0 --createorig -s

Now customize the deb with a service, preinst for user creation and some dependencies:

$ cat > debian/service

ExecStart=/usr/bin/startx /etc/X11/Xsession /usr/bin/kioskme
$ cat > debian/preinst

set -e

. /usr/share/debconf/confmodule

case "$1" in
 if ! getent group kioskme >/dev/null; then
 addgroup --system kioskme >/dev/null
 if ! getent passwd kioskme >/dev/null; then
 adduser \
 --system \
 --disabled-login \
 --ingroup kioskme \
 --gecos kioskme \
 --shell /bin/false \
 kioskme >/dev/null
 mkdir -p /var/log/kioskme
 chown kioskme:kioskme /var/log/kioskme
 setfacl -m u:kioskme:rw /dev/tty0 /dev/tty7


 echo "preinst called with unknown argument \`$1'" >&2
 exit 1

# dh_installdeb will replace this with shell code automatically
# generated by other debhelper scripts.


exit 0

Alright, maybe that preinst is a bit big. I copy it around and fill it out like a template for services I put into debs.

Now edit the debian/control file to add dependencies, change the section to utils, fill in whatever else you want, set Depends to look like this:

Depends: ${shlibs:Depends}, ${misc:Depends}, Xorg, openbox, chromium-browser, pulseaudio

Now create the deb:

fakeroot debian/rules clean build binary

To test the deb, I copy it to a fresh ubuntu server install and dpkg -i to install it. I get a bunch of errors because dpkg -i doesn’t resolve dependencies, but I run apt install -f and the dependencies are installed.

Once I tested and tweaked and got things working, I updated the tarball `tar -Jcf ../kioskme_0.0.0-0.orig.tar.xz -C ..  –exclude=’debian’ kioskme` and I used dpkg-buildpackage -S to build a source package and then I used dput ppa:evarlast/kiosk ../kioskme_0.0.0-1_source.changes to upload to PPA.

Now, this still does not work in a VM. Ubuntu desktop installer must do some magic to make X work in a virtual machine with a driver which works with VMWare, VirtualBox, or Parallels.


Ubuntu Xenial 16.04 Has All The Good Stuff

A couple of days ago, Ubuntu Xenial was released. There is a press release with some good stuff in it.

I’ve been looking forward to this release for the following reasons:

  • Postgresql 9.5
  • systemd
  • haproxy 1.6.3
  • uwsgi 2.0.12
  • nginx 1.9.15

I know, it doesn’t look that exciting until you recall that the last LTS release of Ubuntu, Trusty, 14.04, was missing fabulous features OOTB in each of these components.

Postgresql 9.3 did not have the the awesome JSONB improvements of 9.4 and 9.5

haproxy 1.4 didn’t have ssl support.

uwsgi… well latest uwsgi is just always great to have.

nginx 1.9.15 has http2 support, out of the box!

Finally, while I loved upstart, systemd is nice and has been rock solid.

This is the greatest Ubuntu ever. I’ve not even mentioned how awesome lxd is on it. That is covered elsewhere. This is just my personal little list. Thanks Ubuntu.

Ubuntu 15.10 brings faster add-apt-repository

Wily Werewolf was released yesterday and with it many new things out of the box.

My favorite feature is something that is silly, simple, a tiny patch, and speeds up something I do often.

I work in the cloud and that means I am deploying new machine images many times a day. Anything to speed this up is something that I want.

In Wily, add-apt-repository now has a -u switch.

-u, –update Update package cache after adding

You’ll notice everywhere on the internet where add-apt-repository is used, the next line of instructions is `apt-get update`. This refreshes the package cache for ALL of the configured apt repositories. On a slow machine with slow IO or slow network, this can take more than just a few seconds, possibly a minute or two. This is too long to wait.

The -u option solves this problem. Not only does is remove the need to `apt-get update` by automatically doing it, but it only fetches the package cache for that newly added repository saving much time.

So anywhere on or where you see:

sudo add-apt-repository FOO
sudo apt-get update
sudo apt-get install BAR

add the -u and remove the update command:

sudo add-apt-repository -u FOO
sudo apt-get install BAR

Revel in the time you save.

Converting your existing ssh rsa key for use with Windows Azure

Oh Microsoft, it seems like you make simple things complex.

I could not find anything on converting an existing ssh key for use with Azure. Once I figured out what was needed and the commands available to me, it was easy. It only took me hours of fiddling with ssh-keygen and openssl.

The magic was learning that openssh stores its id_rsa in a format which openssl can read. This means I can use openssl directly to convert this private key.

openssl req -x509 -new -days 365 -key id_rsa -out id_rsa.x509req.pem

Type in your password for your private key (if you are not using a password, you should be.) Then fill out the certificate request fields.

Now you can boot your azure vm using id_rsa.x509req.pem

azure vm create jrwtest b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_3-LTS-amd64-server-20130916.1-en-us-30GB jwren –location “East US” -e -t id_rsa.x509req.pem

Now you can secure shell to your azure vm.


SWEET. 🙂  No generating new ssh keys for me.