All posts by jrwren

Ubuntu Cloud Image Based Containers with LXC

At a previous employer, we standardized on Ubuntu cloud images on AWS EC2 and in our OpenStack. You can find the images at http://cloud-images.ubuntu.com. If you are using Ubuntu on EC2 or another Certified Public Cloud, then its most likely one of these cloud images.

We leveraged cloud-init and extended an already existing simple management system to allow passing user-data to EC2 instances and OpenStack Nova instances. The use of ephemeral instances proved very powerful and influenced our thinking greatly. We came up with great solutions using these very simple techniques.

Even before I left that job, I longed for an easy way to do the same thing for myself. I played a bit with the AWS CLI tool (the newer python boto based tool) and yes, aws ec2 run-instances –user-data works. I always longed to get the same thing on my home server and on my laptop.

Finally, I figured out how to do this with LXC. Its simple yes, but I finally learned how to do what I want.

tl;dr example:

lxc-create -n crisp-Hadley -t ubuntu-cloud -- -r trusty -S ~/.ssh/id_rsa-.pub -u one.yaml

cat > one.yaml
#cloud-config
output:
 all: "|tee -a /tmp/cloud.out"

bootcmd:
 - rm -f /etc/dpkg/dpkg.cfg.d/multiarch
 - for i in 1 2 3 4 5 ; do curl -s http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add - && break ; sleep 2 ; done
apt_sources:
 - source: deb http://packages.elasticsearch.org/logstash/1.4/debian stable main
 - source: ppa:evarlast/experimental
packages:
 - mongodb
final_message: "The system is finally up, after $UPTIME seconds"
runcmd:
 - service myapp start

Background

Most LXC tutorials that I’ve seen walk the user through using the download template. The download template is not bad for new users, but I want something more powerful. It turns out there are a number of templates available by default in /usr/share/lxc/templates and you can even create your own.

The template I am interested in is the ubuntu-cloud template. These lxc templates are not so much templates at all as they are scripts. Some of them use other scripts called hooks defined in /usr/share/lxc/hooks. The ubuntu-cloud template, defined in /usr/share/lxc/templates/lxc-ubuntu-cloud.

Usage

The help for templates is a little hidden and lxc is a little stupid at letting you view the help. You COULD run lxc-create, use the — to pass options to the template and use -h. That has the unfortunate side effect of creating the container anyway. You’d have to lxc-destroy it even though you only used -h. Instead, it is easier to invoke the template directly and get help.

$ /usr/share/lxc/templates/lxc-ubuntu-cloud -h
LXC Container configuration for Ubuntu Cloud images.

Generic Options
[ -r | --release <release> ]: Release name of container, defaults to host
[ --rootfs <path> ]: Path in which rootfs will be placed
[ -a | --arch ]: Architecture of container, defaults to host architecture
[ -T | --tarball ]: Location of tarball
[ -d | --debug ]: Run with 'set -x' to debug errors
[ -s | --stream]: Use specified stream rather than 'tryreleased'

Additionally, clone hooks can be passed through (ie, --userdata). For those,
see:
 /usr/share/lxc/hooks/ubuntu-cloud-prep --help

Here, we see that if we don’t specify the -r option, it defaults to match the host. I’m running vivid on my host, but I’d really like to stick with trusty inside of containers. The -a is interesting, and I can only guess that it only works where compatible. -a i386 would let me use the i386 cloud image on an amd64 host. I can’t think of any other use where a mixing architecture would work in a container.

But there is nothing here about cloud-init

cloud-init via cloud-prep

The last line of help says clone hooks can be passed through. This is useful and IMO the most important item. Run the help for ubuntu-cloud-prep exactly as suggested.

$ /usr/share/lxc/hooks/ubuntu-cloud-prep --help
Usage: ubuntu-cloud-prep [options] root-dir

  root-dir is the root directory to operate on

  [ -C | --cloud  ]:       do not configure a datasource.  incompatible with
                           options marked '[ds]'
  [ -i | --instance-id]:   instance-id for cloud-init, defaults to random [ds]
  [ -L | --nolocales ]:    Do not copy host's locales into container
  [ -S | --auth-key ]:     ssh public key file for datasource [ds]
  [ -u | --userdata ]:     user-data file for cloud-init [ds]

Options for –userdata and –auth-key. Are those what I think they are? It turns out, yes, they work exactly like choosing a public key and user-data when starting an EC2 or Nova instance.

Putting all this together you can create cloud-config yaml files and specify and ssh key, and starting an LXC is just like starting a public cloud instance.

For example, want a postgresql server running?

$ cat > psql.yaml
#cloud-config
output:
 all: "|tee -a /tmp/cloud.out"
packages:
 - postgresql
runcmd:
 - echo "listen_addresses = '*'" >>/etc/postgresql/9.3/main/postgresql.conf
 - sudo -u postgres createuser -D -R -S myuser
 - sudo -u postgres createdb -E utf8 -O myuser mydb
 - echo host mydb myuser 10.0.0.0/8 trust >> /etc/postgresql/9.3/main/pg_hba.conf
 - service postgresql restart 
^D
$ lxc-create -n mypostgresql -t ubuntu-cloud -- -r trusty -S ~/.ssh/id_rsa.pub -u psql.yaml
$ lxc-start -n mypostgresql
$ lxc-info -n mypostgresql
Name:           mypostgresql
State:          RUNNING
PID:            6899
IP:             10.0.3.32
CPU use:        3.67 seconds
BlkIO use:      168.00 KiB
Memory use:     23.21 MiB
KMem use:       0 bytes
Link:           veth452FOE
 TX bytes:      3.84 KiB
 RX bytes:      20.36 KiB
 Total bytes:   24.20 KiB
$ psql -h 10.0.3.32 -U myuser -d mydb
psql (9.4.2, server 9.3.7)
SSL connection (protocol: TLSv1.2, cipher: DHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

mydb=> \q

One of my favorite things about using the cloud-image like this is that unlike the download images, openssh server is running and listening by default. The user ubuntu has the public key which you provided in its authorized_keys file. Everything is ready to go.

11 lines of config, 373 bytes is not much at all for a running postgresql server.

When I don’t want to use juju, this is my go to option.

logstash on ubuntu the easy way

The nice folks at elasticsearch package up logstash for debian and ubuntu. It is very easy to use.

$ curl -s http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb http://packages.elasticsearch.org/logstash/1.4/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list
$ sudo apt-get update
$ sudo apt-get install logstash

Now you have logstash.

Write a config file and fire it up.

/opt/logstash/bin/logstash -f logstash.conf

Zulu JRE from Azul Systems is a hidden gem

http://www.azulsystems.com/products/zulu Azul Systems, the company that Cliff Click works for, builds their own openjdk version.

If you don’t recall Cliff Click, I was first introduced to him via this awesome video: https://www.youtube.com/watch?v=agH7Cz5FSxY

If I have to run on the JVM, then this is how I want to run on the JVM.

Zulu isn’t Zing, and yet it is a hidden gem. No more stupid prompts from Oracle. No more being associated with the company that forces you to install the Ask toolbar and other spywear.

The download page is here: http://www.azulsystems.com/products/zulu/downloads

It is the best (only?) way to get openjdk onto your OSX mac.

Better still is the package install on Ubuntu.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0x219BD9C9
sudo apt-add-repository "deb http://repos.azulsystems.com/ubuntu stable main"
sudo apt-get update 
sudo apt-get install zulu-8

Zulu includes something called the CCK which says

The Commercial Compatibility Kit (CCK) for Zulu contains additional functionality that is not included in in the OpenJDK source, but which will help ensure compatibility in applications that take advantage of specific additional features that Oracle bundles into HotSpot.

curl -O http://cdn.azulsystems.com/zcck/2014-08-8-bin/zcck8-8.0.0.2-amd64.deb
sudo dpkg -i zcck8-8.0.0.2-amd64.deb

Do better Java on Ubuntu.

golang goals

When discussing the Go programming language, I find it useful to always reference the goals of the language. Discussion tends to devolve into a comparison of features of other programming languages which Go lacks. Without the context of these goals, the discussion ceases being useful.

Stolen from a Google tech talk that Rob Pike did back in 2009:

Goals

  • The efficiency of a statically-typed compiled language with the ease of programming of a dynamic language.
  • Safety: type-safe and memory-safe.
  • Good support for concurrency and communication.
  • Efficient, latency-free garbage collection.
  • High-speed compilation.

Watch the Go at Google video or read the article and you will get the impression that these goals are NOT in their order of importance. I suggest that the last item, High-speed compilation, trumps all the others.

Here is a short, 1:15 video demonstrating the speed of the go compiler: https://www.youtube.com/watch?v=wwoWei-GAPo

The first comment, over 2 years ago, at Lambda the Ultimate, about that Go at Google video, sums it up even better. It is a snapshot of another slide. This time instead of Go goals, the slide is “What makes large-scale development hard with C++ or Java (at least):”

  • slow builds
  • uncontrolled dependencies
  • each programmer using a different subset of the language
  • poor program understanding (documentation, etc.)
  • duplication of effort
  • cost of updates
  • version skew
  • difficulty of automation (auto rewriters etc.): tooling
  • cross-language builds

* Language features don’t usually address these.

 

It took me quite a while to keep these above things in mind when thinking about Go. In fact, I still tend to compare Go to my favorite programming languages, probably because I often forget some of those above drawbacks to C++/Java (read C# for me).

I’ll try to remember. I beg others to try to remember too.

Testing Out Apache All By Yourself

By all by yourself, I mean, without root.

This is on my Mac running OSX 10.10.

  1. Get yourself an httpd.conf – cp /private/etc/apache2/httpd.conf .
  2. Edit it to use a port >1024 and with user you – Listen 8081 & User jrwren & Group staff
  3. Log to a place you can write – ErrorLog /home/jrwren/errorlog & CustomLog /home/jrwren/access_log combined
  4. Use different pidfile –  PidFile /home/jrwren/httpd.pid Do this fter the Include /private/etc/apache2/extra/httpd-mpm.conf
  5. Accept mutex –  Mutex file:/home/jrwren
  6. Edit whatever else you want – ProxyPass / http://localhost:8080 & SetOutputFilter DEFLATE to see that Apache proxy does gzip for you
  7. Start httpd – httpd -d . -f httpd.conf -X

Faster mongodb deploys with Juju

I made a change to the official mongodb charm.

Before the change, the install step, when deploying the charm took 5 minutes.

2014-11-17 20:54:13 Installing mongodb
2014-11-17 21:04:12 Entering config_changed

After the change, the install step, when deploying took a tiny bit over 2 minutes.

2014-11-19 19:14:55 Installing mongodb
2014-11-19 19:16:06 Entering config_changed

Before: Fetched 46.2 MB
After: Fetched 14.6 MB

Why the huge change? The charm was previously running ‘apt-get install mongodb’ which Recommends the mongodb-dev package which Depends on development packages, ultimately pulling in a c++ compiler and boost dev libraries and header files.

I am happy to download 1/4 of what I previously was required to download for the same functionality.

When writing charms for juju, or automating your deploys using any other tool, remember consider what the system is really doing with the commands you give it.

I highly recommend always running apt-get install with the –no-install-recommends option when you are in a server environment. You’ll waste less time.

A note on the time comparison: At first I hesitated to write this, because the hardware on which I tested this is doing other things. I then realized that this is exactly what all hardware in the cloud is doing all the time. This isn’t a benchmark. This isn’t a timing test. This is an example of making a single case of something slow, a bit faster.

 

Netatalk Ubuntu Trusty Package

https://launchpad.net/ubuntu/+source/netatalk

This is a 3 year old piece of software in the latest Ubuntu LTS release. That is real bummer.

I don’t want to do all this:
http://netatalk.sourceforge.net/wiki/index.php/Install_Netatalk_3.1.6_on_Ubuntu_14.04_Trusty

I’d really just like to apt-get install netatalk and have the latest.

Using a 3.0+ version of Netatalk is especially nice since it uses filesystem extended attributes for AppleDouble instead of hidden files all over the place. http://netatalk.sourceforge.net/3.0/ReleaseNotes-3.0.html

First, the results:
$ sudo add-apt-repository ppa:evarlast/netatalk
$ sudo apt-get update && sudo apt-get install netatalk

A brief warning: config file syntax changed entirely from 2.2 netatalk to 3.0 netatalk. If you are upgrading from 2.x to 3.x you will need to audit your config files and test and make sure everything works.

How I did it:
On a refresh trusty install:
$ sudo apt-get install dpkg-dev devscripts libmysqlclient-dev libssl-dev systemtap-sdt-dev libdbus-glib-1-dev libglib2.0-dev tracker libtracker-sparql-0.16-dev libtracker-miner-0.16-dev libtdb-dev libevent-dev
$ sudo apt-get source netatalk
$ sudo apt-get build-dep netatalk
$ curl -o netatalk_3.1.6.orig.tar.bz2 -L ‘http://sourceforge.net/projects/netatalk/files/latest/download?source=files’
$ tar jxvf netatalk_3.1.6.orig.tar.bz2
$ cd netatalk-3.1.6
$ cp -a ../netatalk-2.2.2/debian .
$ dch -v 3.1.6 -D trusty
$ vim debian/patches/series # remove everything except the macusers patch
$ vim debian/control # edit and update with dependent packages we installed as listed http://netatalk.sourceforge.net/wiki/index.php/Install_Netatalk_3.1.6_on_Ubuntu_14.04_Trusty Be sure to add a final NETA_LDCONFIG=/bin/true to the configure flags
$ vim debian/rules # edit and update configure options as listed on wiki page
$ vim debian/atalk.docs # remove README line
$ debuild
$ dpkg-buildpackage -rfakeroot -S

Install the package or dput it into a PPA.

In putting this together I ran into an issue with a strange automake assumption that the build will be as root, or that the user running make can run ldconfig. This is not the case when building debian pacakges. Searching for NETA_LDCONFIG returned this url: http://oichinote.com/plus/2014/07/installing-debianized-netatalk-3-1-3-on-ubuntu-14-04.html

ctags for golang and vim; just the right things with godeps

I use vim.

I like to press ctrl-] to go to a tag and ctrl-t to pop up that tag stack.

I use ctags from homebrew to generate my tags file which vim reads. The OSX version of ctags is inadequate.

I often invoke ctags with -R . and a list of directory names to the libraries which I am using. When using ctags with python or C this works reasonably well, but I have to maintain the list of directories somewhere.

Go lets me handle this case slightly better.

godeps is a tool which does two things, in this case I only care about the first. godeps prints the source dependencies of named packages.

Combining godeps output and sending it to ctags means my tags file automatically has tokens from all of my dependent packages. I name this shell function goctags.

goctags () { godeps ./… | awk -v GOPATH=$GOPATH ‘{print GOPATH”/src/”$1}’ | xargs ctags -R .; }

I added it to my .bashrc.

Elasticsearch on Ubuntu

It sucks, but it doesn’t have to.

1. Import the GPG KEY from the elasticsearch repo.

 curl -s http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add –

2. Add the repo.

echo "deb http://packages.elasticsearch.org/elasticsearch/1.3/debian stable main" |sudo tee /etc/apt/sources.list.d/packages_elasticsearch_org_elasticsearch_1_3_debian.list

3. Update your apt cache.

sudo apt-get update

4. Install the elastic search package.

sudo apt-get install elasticsearch

If this is a server, then configure ES to run on system start and start ES now:

sudo update-rc.d elasticsearch defaults 95 10
sudo service elasticsearch

If this is a development environment, then the following may help.

Homebrew on MacOSX allows for ability to simply run “elasticsearch –config=myconfig.yml” and have different elasticsearch instances. I want this on my Linux dev system.

1. Copy elasticsearch shell script to a place in the path. $HOME/bin works just as good as /usr/local/bin here, if it is in your path. Then you can skip the sudo on these commands.

sudo cp /usr/share/elasticsearch/bin/elasticsearch /usr/local/bin/

2. Copy the in.sh file there too.

sudo cp /usr/share/elasticsearch/bin/elasticsearch.in.sh /usr/local/bin/

3. Set the ES_HOME in the in.sh file.

sudo sed -i ‘2 a ES_HOME=/usr/share/elasticsearch’ /usr/local/bin/elasticsearch.in.sh

4. DFSG don’t work if the app isn’t built correctly, so symlink the config back in place. Config won’t get used, but logging.yml will.

sudo ln -s /etc/elasticsearch/ /usr/share/elasticsearch/config

That shall do it. You can now test run a few different instances.

for $dir in a b ; do
mkdir $dir
pushd $dir
cat > config.yml <<EOM
cluster.name: cluster_$dir
path.data: ./data
path.logs: ./log/
network.host: 127.0.0.1
http.port: 1234
EOM
elasticsearch –config=config.yml &
popd
done

Now you have a slightly less terrible elasticsearch on your Linux system, about on par with what you get from homebrew on a Mac.

update:fixed path in sources list creation.
Note: you can substitute 1.4 or 1.5 to get those versions of elasticsearch. Also note, elasticsearch isn’t dependant on a JVM, so you will need to apt-get install openjdk-7-jre-headless or zulu-8. See http://jrwren.wrenfam.com/blog/2015/03/18/zulu-jre-from-azul-systems-is-a-hidden-gem/