All posts by jrwren

Testing Out Apache All By Yourself

By all by yourself, I mean, without root.

This is on my Mac running OSX 10.10.

  1. Get yourself an httpd.conf – cp /private/etc/apache2/httpd.conf .
  2. Edit it to use a port >1024 and with user you – Listen 8081 & User jrwren & Group staff
  3. Log to a place you can write – ErrorLog /home/jrwren/errorlog & CustomLog /home/jrwren/access_log combined
  4. Use different pidfile –  PidFile /home/jrwren/httpd.pid Do this fter the Include /private/etc/apache2/extra/httpd-mpm.conf
  5. Accept mutex –  Mutex file:/home/jrwren
  6. Edit whatever else you want – ProxyPass / http://localhost:8080 & SetOutputFilter DEFLATE to see that Apache proxy does gzip for you
  7. Start httpd – httpd -d . -f httpd.conf -X

Faster mongodb deploys with Juju

I made a change to the official mongodb charm.

Before the change, the install step, when deploying the charm took 5 minutes.

2014-11-17 20:54:13 Installing mongodb
2014-11-17 21:04:12 Entering config_changed

After the change, the install step, when deploying took a tiny bit over 2 minutes.

2014-11-19 19:14:55 Installing mongodb
2014-11-19 19:16:06 Entering config_changed

Before: Fetched 46.2 MB
After: Fetched 14.6 MB

Why the huge change? The charm was previously running ‘apt-get install mongodb’ which Recommends the mongodb-dev package which Depends on development packages, ultimately pulling in a c++ compiler and boost dev libraries and header files.

I am happy to download 1/4 of what I previously was required to download for the same functionality.

When writing charms for juju, or automating your deploys using any other tool, remember consider what the system is really doing with the commands you give it.

I highly recommend always running apt-get install with the –no-install-recommends option when you are in a server environment. You’ll waste less time.

A note on the time comparison: At first I hesitated to write this, because the hardware on which I tested this is doing other things. I then realized that this is exactly what all hardware in the cloud is doing all the time. This isn’t a benchmark. This isn’t a timing test. This is an example of making a single case of something slow, a bit faster.

 

Netatalk Ubuntu Trusty Package

https://launchpad.net/ubuntu/+source/netatalk

This is a 3 year old piece of software in the latest Ubuntu LTS release. That is real bummer.

I don’t want to do all this:
http://netatalk.sourceforge.net/wiki/index.php/Install_Netatalk_3.1.6_on_Ubuntu_14.04_Trusty

I’d really just like to apt-get install netatalk and have the latest.

Using a 3.0+ version of Netatalk is especially nice since it uses filesystem extended attributes for AppleDouble instead of hidden files all over the place. http://netatalk.sourceforge.net/3.0/ReleaseNotes-3.0.html

First, the results:
$ sudo add-apt-repository ppa:evarlast/netatalk
$ sudo apt-get update && sudo apt-get install netatalk

A brief warning: config file syntax changed entirely from 2.2 netatalk to 3.0 netatalk. If you are upgrading from 2.x to 3.x you will need to audit your config files and test and make sure everything works.

How I did it:
On a refresh trusty install:
$ sudo apt-get install dpkg-dev devscripts libmysqlclient-dev libssl-dev systemtap-sdt-dev libdbus-glib-1-dev libglib2.0-dev tracker libtracker-sparql-0.16-dev libtracker-miner-0.16-dev libtdb-dev libevent-dev
$ sudo apt-get source netatalk
$ sudo apt-get build-dep netatalk
$ curl -o netatalk_3.1.6.orig.tar.bz2 -L ‘http://sourceforge.net/projects/netatalk/files/latest/download?source=files’
$ tar jxvf netatalk_3.1.6.orig.tar.bz2
$ cd netatalk-3.1.6
$ cp -a ../netatalk-2.2.2/debian .
$ dch -v 3.1.6 -D trusty
$ vim debian/patches/series # remove everything except the macusers patch
$ vim debian/control # edit and update with dependent packages we installed as listed http://netatalk.sourceforge.net/wiki/index.php/Install_Netatalk_3.1.6_on_Ubuntu_14.04_Trusty Be sure to add a final NETA_LDCONFIG=/bin/true to the configure flags
$ vim debian/rules # edit and update configure options as listed on wiki page
$ vim debian/atalk.docs # remove README line
$ debuild
$ dpkg-buildpackage -rfakeroot -S

Install the package or dput it into a PPA.

In putting this together I ran into an issue with a strange automake assumption that the build will be as root, or that the user running make can run ldconfig. This is not the case when building debian pacakges. Searching for NETA_LDCONFIG returned this url: http://oichinote.com/plus/2014/07/installing-debianized-netatalk-3-1-3-on-ubuntu-14-04.html

ctags for golang and vim; just the right things with godeps

I use vim.

I like to press ctrl-] to go to a tag and ctrl-t to pop up that tag stack.

I use ctags from homebrew to generate my tags file which vim reads. The OSX version of ctags is inadequate.

I often invoke ctags with -R . and a list of directory names to the libraries which I am using. When using ctags with python or C this works reasonably well, but I have to maintain the list of directories somewhere.

Go lets me handle this case slightly better.

godeps is a tool which does two things, in this case I only care about the first. godeps prints the source dependencies of named packages.

Combining godeps output and sending it to ctags means my tags file automatically has tokens from all of my dependent packages. I name this shell function goctags.

goctags () { godeps ./… | awk -v GOPATH=$GOPATH ‘{print GOPATH”/src/”$1}’ | xargs ctags -R .; }

I added it to my .bashrc.

Elasticsearch on Ubuntu

It sucks, but it doesn’t have to.

1. Import the GPG KEY from the elasticsearch repo.

 curl -s http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

2. Add the repo.

echo “deb http://packages.elasticsearch.org/GPG-KEY-elasticsearch stable main” |sudo tee /etc/apt/sources.list.d/packages_elasticsearch_org_elasticsearch_1_3_debian.list

3. Update your apt cache.

sudo apt-get update

4. Install the elastic search package.

sudo apt-get install elasticsearch

If this is a server, then configure ES to run on system start and start ES now:

sudo update-rc.d elasticsearch defaults 95 10
sudo service elasticsearch

If this is a development environment, then the following may help.

Homebrew on MacOSX allows for ability to simply run “elasticsearch –config=myconfig.yml” and have different elasticsearch instances. I want this on my Linux dev system.

1. Copy elasticsearch shell script to a place in the path. $HOME/bin works just as good as /usr/local/bin here, if it is in your path. Then you can skip the sudo on these commands.

sudo cp /usr/share/elasticsearch/bin/elasticsearch /usr/local/bin/

2. Copy the in.sh file there too.

sudo cp /usr/share/elasticsearch/bin/elasticsearch.in.sh /usr/local/bin/

3. Set the ES_HOME in the in.sh file.

sudo sed -i ‘2 a ES_HOME=/usr/share/elasticsearch’ /usr/local/bin/elasticsearch.in.sh

4. DFSG don’t work if the app isn’t built correctly, so symlink the config back in place. Config won’t get used, but logging.yml will.

sudo ln -s /etc/elasticsearch/ /usr/share/elasticsearch/config

That shall do it. You can now test run a few different instances.

for $dir in a b ; do
mkdir $dir
pushd $dir
cat > config.yml <<EOM
cluster.name: cluster_$dir
path.data: ./data
path.logs: ./log/
network.host: 127.0.0.1
http.port: 1234
EOM
elasticsearch –config=config.yml &
popd
done

Now you have a slightly less terrible elasticsearch on your Linux system, about on par with what you get from homebrew on a Mac.

Setup a New Mac

A couple years later, and I find myself referring to my own guide

But in a different order of important things.

0. Caps Lock key is a Control key, Preferences->Keyboard, fix that, while I’m there, remove some of the -F keys from being bound, I’m going to need apps to see them.
1. I NEED THE SSH KEY, and copying a private key can be kind of a challenge, cuz… privacy.
2. Finder needs to see hidden files. http://lifehacker.com/188892/show-hidden-files-in-finder says – defaults write com.apple.finder AppleShowAllFiles TRUE ; killall Finder
3. AppStore has a nice purchases view, so I can easily find apps I have on the old/other Mac – click yes to Evernote.
4. Get iTerm2 – http://www.iterm2.com/#/section/home and DO NOT CLICK the big DOWNLOAD button… click the download tab/section at the top and get the Test Releases download, start it, and under profiles->default-> select the keys tab & click the Left option as +Esc selection
5. Copy .bashrc and .bash_profile
6. Copy Music/iTunes folder, maybe?
7. Manually inspect https://raw.githubusercontent.com/Homebrew/homebrew/go/install before actually going there and doing the http://brew.sh installation via ruby -switches $()… because… SECURITY!… also, I like to sudo mkdir /usr/local ; sudo chown $USER /usr/local ; so that home-brew install doesn’t run anything as root.
8.  System Preferences -> Keyboard -> Shortcuts -> All Controls (at the bottom) – so that I can tab to selections in dialog boxes
9.  MacVIM – because I like it
10.  vim things like python-mode, vim-fugitive
11.  firefox aurora and login with my firefox password
12.  brew install go
13.  add GOROOT to .bashrc and GOROOT/misc/vim to vim rtp
14.  … I don’t really know what else.

UGH, how could I forget: download Envy Code R and tell vim to use it: http://damieng.com/blog/2008/05/26/envy-code-r-preview-7-coding-font-released  Add set guifont=Envy\ Code\ R:h13 to .vimrc

Follow the rest of the old post http://jrwren.wrenfam.com/blog/2012/03/07/setting-up-a-new-mac/

Leaving a Great Job for a Great Job

Today was my last day as an employee at Arbor Networks.

Leaving  Arbor Networks was a tough decision. There are so many good people and interesting problems at Arbor. When I was approached by folks, I said, “I’m not done working on stuff at Arbor.”

The work, people, culture and environment really are that fun at Arbor. As a result, the last weeks and days of my working at Arbor have been super busy as we came together as a team and decided to get some stuff done. We set some goals and achieved them before I left.

On Monday, I start at Canonical working on Juju. I am super excited to work with a team of people who are behind making the best cloud orchestration tool, and contribute to making it even better.

I’ll be learning more about Go in my new roll. Expect me to write a bit about learning go. I learned a ton about Python at Arbor Networks. I didn’t write about it because I feel like everything I learned is very well documented. Go is a much younger language. Hopefully I can contribute to its documentation and share my learning experience in a meaningful way.

Getting a Windows Password for EC2 Instance

… without pasting your private key to ec2.

EC2 should never see your private key… because.. security!

I launched a Windows Server 2012 R2 instance in EC2 recently and while the AWS console does let you retrieve an Administrator password, it requires you to paste your PRIVATE key to AWS console to do it. I couldn’t bring myself to do it, so I learned how to use boto to get the encrypted password data and openssl cmdline to decrypt it to get the password.

Its a 2 step process with maybe the zeroth step being writing a .boto file with your aws credentials if you have never used boto.

import boto
import base64
ec2 = boto.connect_ec2()
inst = ec2.get_all_instances()[0].instance
data = ec2.get_password_data(inst.id)
open(‘ec2-admin-password’,’w’,write(base64.decodestring(data))

I’m assuming its the only instance running. If you have lots of others, use a list comprehension with if clause to filter to one on the get_all_instances() call, or just skip that call and paste an id string you see in AWS console for inst.id in the get_password_data call.

openssl rsautil -in ec2-admin-password -inkey .ssh/id_rsa -decrypt

You’ll be prompted for your private key password (and you MUST have a password. ssh-agent is easy) and then the Administrator password will be output to stdout.

 

GNU date luxuries

http://www.cyberciti.biz/tips/linux-unix-get-yesterdays-tomorrows-date.html

Has a pretty good summary of using GNU date’s strtotime implementation.

strtotime.y is an interesting piece of code. Its often reproduced and imitated. The header says

Originally written by Steven M. Bellovin <smb@research.att.com> while at the University of North Carolina at Chapel Hill.  Later tweaked by a couple of people on Usenet.  Completely overhauled by Rich $alz <rsalz@bbn.com> and Jim Berets <jberets@bbn.com> in August, 1990;

This grammar has 13 shift/reduce conflicts.

This code is in the public domain and has no copyright.

Thanks Steven!

It turns out GNU renames this to parse-datetime.y, and fixes some local DST issues, but you can see that original message still there.

Its a nice lex/yacc refresher when you have been away from those tools for a while, and a nice C refresher too. Using it is easier than understanding how it works.

One of the things I like is that you can combine expressions.

$ date -d ‘1 day ago’
Wed May 14 15:11:51 UTC 2014

Just leave out the english conjunction. So instead of 1 day ago and 2 hours ago, say 1 day ago 2 hours ago.

$ date -d ‘1 day ago 2 hours ago’
Wed May 14 13:12:00 UTC 2014

I should mention that these are correct, because the time right now is

$ date
Thu May 15 15:12:50 UTC 2014

One thing which is not really clear in the above tip page is that minus is just an alias for ago.

$ date -d ‘-1 day -2 hours’
Wed May 14 13:14:04 UTC 2014

Things one may wish to do is floor a result. e.g. making yesterday start at top of yesterday.

$ date -d ‘yesterday 00:00′
Wed May 14 00:00:00 UTC 2014

Finally, I was surprised to dig up ruby’s date_parse.c and find that it does not claim any heritage with the original strtotime.y.

 

update 3 hours later:

I was just doing some comparisons on systems which use strtotime and the question arose, what if I call it with empty string?

$ date -d ”
Thu May 15 00:00:00 UTC 2014

easy answer: its the midnight floor of todays date. The same as ‘now 00:00:00′, ‘now 00:00′, or just ‘now 0′.