Author: jrwren

Netatalk Ubuntu Trusty Package

https://launchpad.net/ubuntu/+source/netatalk

This is a 3 year old piece of software in the latest Ubuntu LTS release. That is real bummer.

I don’t want to do all this:
http://netatalk.sourceforge.net/wiki/index.php/Install_Netatalk_3.1.6_on_Ubuntu_14.04_Trusty

I’d really just like to apt-get install netatalk and have the latest.

Using a 3.0+ version of Netatalk is especially nice since it uses filesystem extended attributes for AppleDouble instead of hidden files all over the place. http://netatalk.sourceforge.net/3.0/ReleaseNotes-3.0.html

First, the results:
$ sudo add-apt-repository ppa:evarlast/netatalk
$ sudo apt-get update && sudo apt-get install netatalk

A brief warning: config file syntax changed entirely from 2.2 netatalk to 3.0 netatalk. If you are upgrading from 2.x to 3.x you will need to audit your config files and test and make sure everything works.

How I did it:
On a refresh trusty install:
$ sudo apt-get install dpkg-dev devscripts libmysqlclient-dev libssl-dev systemtap-sdt-dev libdbus-glib-1-dev libglib2.0-dev tracker libtracker-sparql-0.16-dev libtracker-miner-0.16-dev libtdb-dev libevent-dev
$ sudo apt-get source netatalk
$ sudo apt-get build-dep netatalk
$ curl -o netatalk_3.1.6.orig.tar.bz2 -L ‘http://sourceforge.net/projects/netatalk/files/latest/download?source=files’
$ tar jxvf netatalk_3.1.6.orig.tar.bz2
$ cd netatalk-3.1.6
$ cp -a ../netatalk-2.2.2/debian .
$ dch -v 3.1.6 -D trusty
$ vim debian/patches/series # remove everything except the macusers patch
$ vim debian/control # edit and update with dependent packages we installed as listed http://netatalk.sourceforge.net/wiki/index.php/Install_Netatalk_3.1.6_on_Ubuntu_14.04_Trusty Be sure to add a final NETA_LDCONFIG=/bin/true to the configure flags
$ vim debian/rules # edit and update configure options as listed on wiki page
$ vim debian/atalk.docs # remove README line
$ debuild
$ dpkg-buildpackage -rfakeroot -S

Install the package or dput it into a PPA.

In putting this together I ran into an issue with a strange automake assumption that the build will be as root, or that the user running make can run ldconfig. This is not the case when building debian pacakges. Searching for NETA_LDCONFIG returned this url: http://oichinote.com/plus/2014/07/installing-debianized-netatalk-3-1-3-on-ubuntu-14-04.html

ctags for golang and vim; just the right things with godeps

I use vim.

I like to press ctrl-] to go to a tag and ctrl-t to pop up that tag stack.

I use ctags from homebrew to generate my tags file which vim reads. The OSX version of ctags is inadequate.

I often invoke ctags with -R . and a list of directory names to the libraries which I am using. When using ctags with python or C this works reasonably well, but I have to maintain the list of directories somewhere.

Go lets me handle this case slightly better.

godeps is a tool which does two things, in this case I only care about the first. godeps prints the source dependencies of named packages.

Combining godeps output and sending it to ctags means my tags file automatically has tokens from all of my dependent packages. I name this shell function goctags.

goctags () { godeps ./… | awk -v GOPATH=$GOPATH ‘{print GOPATH”/src/”$1}’ | xargs ctags -R .; }

I added it to my .bashrc.

Elasticsearch on Ubuntu

It sucks, but it doesn’t have to.

1. Import the GPG KEY from the elasticsearch repo.

 curl -s http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add –

2. Add the repo.

echo "deb http://packages.elasticsearch.org/elasticsearch/1.3/debian stable main" |sudo tee /etc/apt/sources.list.d/packages_elasticsearch_org_elasticsearch_1_3_debian.list

3. Update your apt cache.

sudo apt-get update

4. Install the elastic search package.

sudo apt-get install elasticsearch

If this is a server, then configure ES to run on system start and start ES now:

sudo update-rc.d elasticsearch defaults 95 10
sudo service elasticsearch

If this is a development environment, then the following may help.

Homebrew on MacOSX allows for ability to simply run “elasticsearch –config=myconfig.yml” and have different elasticsearch instances. I want this on my Linux dev system.

1. Copy elasticsearch shell script to a place in the path. $HOME/bin works just as good as /usr/local/bin here, if it is in your path. Then you can skip the sudo on these commands.

sudo cp /usr/share/elasticsearch/bin/elasticsearch /usr/local/bin/

2. Copy the in.sh file there too.

sudo cp /usr/share/elasticsearch/bin/elasticsearch.in.sh /usr/local/bin/

3. Set the ES_HOME in the in.sh file.

sudo sed -i ‘2 a ES_HOME=/usr/share/elasticsearch’ /usr/local/bin/elasticsearch.in.sh

4. DFSG don’t work if the app isn’t built correctly, so symlink the config back in place. Config won’t get used, but logging.yml will.

sudo ln -s /etc/elasticsearch/ /usr/share/elasticsearch/config

That shall do it. You can now test run a few different instances.

for $dir in a b ; do
mkdir $dir
pushd $dir
cat > config.yml <<EOM
cluster.name: cluster_$dir
path.data: ./data
path.logs: ./log/
network.host: 127.0.0.1
http.port: 1234
EOM
elasticsearch –config=config.yml &
popd
done

Now you have a slightly less terrible elasticsearch on your Linux system, about on par with what you get from homebrew on a Mac.

update:fixed path in sources list creation.
Note: you can substitute 1.4 or 1.5 to get those versions of elasticsearch. Also note, elasticsearch isn’t dependant on a JVM, so you will need to apt-get install openjdk-7-jre-headless or zulu-8. See http://jrwren.wrenfam.com/blog/2015/03/18/zulu-jre-from-azul-systems-is-a-hidden-gem/

Setup a New Mac

A couple years later, and I find myself referring to my own guide

But in a different order of important things.

0. Caps Lock key is a Control key, Preferences->Keyboard, fix that, while I’m there, remove some of the -F keys from being bound, I’m going to need apps to see them.
1. I NEED THE SSH KEY, and copying a private key can be kind of a challenge, cuz… privacy.
2. Finder needs to see hidden files. http://lifehacker.com/188892/show-hidden-files-in-finder says – defaults write com.apple.finder AppleShowAllFiles TRUE ; killall Finder
3. AppStore has a nice purchases view, so I can easily find apps I have on the old/other Mac – click yes to Evernote.
4. Get iTerm2 – http://www.iterm2.com/#/section/home and DO NOT CLICK the big DOWNLOAD button… click the download tab/section at the top and get the Test Releases download, start it, and under profiles->default-> select the keys tab & click the Left option as +Esc selection
5. Copy .bashrc and .bash_profile
6. Copy Music/iTunes folder, maybe?
7. Manually inspect https://raw.githubusercontent.com/Homebrew/homebrew/go/install before actually going there and doing the http://brew.sh installation via ruby -switches $()… because… SECURITY!… also, I like to sudo mkdir /usr/local ; sudo chown $USER /usr/local ; so that home-brew install doesn’t run anything as root.
8.  System Preferences -> Keyboard -> Shortcuts -> All Controls (at the bottom) – so that I can tab to selections in dialog boxes
9.  MacVIM – because I like it
10.  vim things like python-mode, vim-fugitive
11.  firefox aurora and login with my firefox password
12.  brew install go
13.  add GOROOT to .bashrc and GOROOT/misc/vim to vim rtp
14.  … I don’t really know what else.

UGH, how could I forget: download Envy Code R and tell vim to use it: http://damieng.com/blog/2008/05/26/envy-code-r-preview-7-coding-font-released  Add set guifont=Envy\ Code\ R:h13 to .vimrc

Follow the rest of the old post http://jrwren.wrenfam.com/blog/2012/03/07/setting-up-a-new-mac/

Leaving a Great Job for a Great Job

Today was my last day as an employee at Arbor Networks.

Leaving  Arbor Networks was a tough decision. There are so many good people and interesting problems at Arbor. When I was approached by folks, I said, “I’m not done working on stuff at Arbor.”

The work, people, culture and environment really are that fun at Arbor. As a result, the last weeks and days of my working at Arbor have been super busy as we came together as a team and decided to get some stuff done. We set some goals and achieved them before I left.

On Monday, I start at Canonical working on Juju. I am super excited to work with a team of people who are behind making the best cloud orchestration tool, and contribute to making it even better.

I’ll be learning more about Go in my new roll. Expect me to write a bit about learning go. I learned a ton about Python at Arbor Networks. I didn’t write about it because I feel like everything I learned is very well documented. Go is a much younger language. Hopefully I can contribute to its documentation and share my learning experience in a meaningful way.

Getting a Windows Password for EC2 Instance

… without pasting your private key to ec2.

EC2 should never see your private key… because.. security!

I launched a Windows Server 2012 R2 instance in EC2 recently and while the AWS console does let you retrieve an Administrator password, it requires you to paste your PRIVATE key to AWS console to do it. I couldn’t bring myself to do it, so I learned how to use boto to get the encrypted password data and openssl cmdline to decrypt it to get the password.

Its a 2 step process with maybe the zeroth step being writing a .boto file with your aws credentials if you have never used boto.

import boto
import base64
ec2 = boto.connect_ec2()
inst = ec2.get_all_instances()[0].instance
data = ec2.get_password_data(inst.id)
open(‘ec2-admin-password’,’w’,write(base64.decodestring(data))

I’m assuming its the only instance running. If you have lots of others, use a list comprehension with if clause to filter to one on the get_all_instances() call, or just skip that call and paste an id string you see in AWS console for inst.id in the get_password_data call.

openssl rsautil -in ec2-admin-password -inkey .ssh/id_rsa -decrypt

You’ll be prompted for your private key password (and you MUST have a password. ssh-agent is easy) and then the Administrator password will be output to stdout.

 

GNU date luxuries

http://www.cyberciti.biz/tips/linux-unix-get-yesterdays-tomorrows-date.html

Has a pretty good summary of using GNU date’s strtotime implementation.

strtotime.y is an interesting piece of code. Its often reproduced and imitated. The header says

Originally written by Steven M. Bellovin <smb@research.att.com> while at the University of North Carolina at Chapel Hill.  Later tweaked by a couple of people on Usenet.  Completely overhauled by Rich $alz <rsalz@bbn.com> and Jim Berets <jberets@bbn.com> in August, 1990;

This grammar has 13 shift/reduce conflicts.

This code is in the public domain and has no copyright.

Thanks Steven!

It turns out GNU renames this to parse-datetime.y, and fixes some local DST issues, but you can see that original message still there.

Its a nice lex/yacc refresher when you have been away from those tools for a while, and a nice C refresher too. Using it is easier than understanding how it works.

One of the things I like is that you can combine expressions.

$ date -d ‘1 day ago’
Wed May 14 15:11:51 UTC 2014

Just leave out the english conjunction. So instead of 1 day ago and 2 hours ago, say 1 day ago 2 hours ago.

$ date -d ‘1 day ago 2 hours ago’
Wed May 14 13:12:00 UTC 2014

I should mention that these are correct, because the time right now is

$ date
Thu May 15 15:12:50 UTC 2014

One thing which is not really clear in the above tip page is that minus is just an alias for ago.

$ date -d ‘-1 day -2 hours’
Wed May 14 13:14:04 UTC 2014

Things one may wish to do is floor a result. e.g. making yesterday start at top of yesterday.

$ date -d ‘yesterday 00:00’
Wed May 14 00:00:00 UTC 2014

Finally, I was surprised to dig up ruby’s date_parse.c and find that it does not claim any heritage with the original strtotime.y.

 

update 3 hours later:

I was just doing some comparisons on systems which use strtotime and the question arose, what if I call it with empty string?

$ date -d ”
Thu May 15 00:00:00 UTC 2014

easy answer: its the midnight floor of todays date. The same as ‘now 00:00:00’, ‘now 00:00’, or just ‘now 0’.

 

Update about a year later:

I wanted to know when 6months and 8months after the iPhone 6 release was. I can type a date and the plus operator is implied.

$ date -d 'Sept 19, 2014 6 months'
Thu Mar 19 00:00:00 EDT 2015
$ date -d 'Sept 19, 2014 8 months'
Tue May 19 00:00:00 EDT 2015

Hey! That is today!

finding files by date

I got tired of looking this up every time, and I’ve not seen anyone write about this specific use case. When dealing with dev servers in the cloud, sometimes I forget the use case for one. I find it useful to be able to find all the changed files on a system by date.

find / -mount -printf ‘%T@\t%T+\t%s\t%p\n’ | sort -nr

This shows me config files which recently changed, log files which recently changed. I can examine those and get an idea of the state of the system.

I use ls -alrt often enough, but this uses find so I get a view of the entire filesystem at once instead of a single directory.

Book Review: BeagleBone Robotics Projects

I was asked by Packt Publishing if I would read and review this book. I’ve owned a BeagleBoneBlack for a little while now. My use case was not robotics. This book might shed some new light on my old Black, so I agreed to review it.

The book starts off very accessible. Chapter 1 covers just about everything I did with my BBB when I first received it, hooking it up like a PC, replacing the default distro, making sure I could SSH to it were all in there. The author, Richard Grimmett, goes a step further and installs XFCE gui and vncserver and walks through connecting from a Windows PC using vncclient. All in all, chapter 1 is a great super basic tour.

Chapter 2 dives into programming on the thing and introduces Python. It does it in a really weird (to me) way. It has the reader running emacs in a putty window remote connected to the device. This must just feel weird to me because I do a lot of remote programming and its never with emacs (I’m a vim guy) and its rarely remote. For a new user, it seems to me like it would have been simpler and more friendly to say “use an editor of your choice” and “here is notepad2 or sublime” along with “here is how you copy files to and from the device.” I think this is mostly my background causing me to see things differently. The emacs in putty walk-through is very adequate.

Its not a programming book, so this is really a nit pick, but technically some of the descriptions of python aren’t really true. For example, if __name__==”__main__”: does not “tell the program to begin its execution at this point.” Again I’m nit picking, but I do feel like a different phrase that isn’t so very false to someone who knows python could have been found. Still, its not a programming book. The beginning of the chapter does list many resources for learning python.

Ugh, and then the book moves on to C++ and has quotes like this, “C++ is the original language of Linux” I’ve used Linux for almost as long as I’ve programmed C, and I am very (perhaps overly?) sensitive to the difference between C and C++.

OMG what do you mean Speech Input and Output? Really?  Chapter 3 tackles it. Really. For real. Speech Input and Output on that tiny little board. I can make my own Siri! This is a really cool topic; espeek is something I’ve only played with a little bit prior to reading this. It looks fun.

Speech recognition is done with software I’ve never used before called PocketSphinx. It isn’t packaged and so one has to compile it. Pretty sweet BBB being able to compile stuff like that. (I’m thinking of iOS and Android where I’ve not seen a compiler run on device.) The demo walks through limiting the grammar of speech input so that you don’t have to train the recognizer.

I’m a programmer, so I’m going to nitpick programmer things. I really wish authors wouldn’t do this, “I like to make a copy of the current file into continuous.c.old, so I can always get back to the starting program if it is required.” I really do wish authors would just say “go read about version control systems.”

Whew, speech is fun. Next step is video. Hook up a webcam and let’s do some image recognition. The book walks through OpenCV and it is as this point that we are forced to do a bunch of Linux sysadmin stuff to make our SD have enough free space to have a dev environment. This really could have gone anywhere in the book. I kind of like that it put it off until it was necessary.

The python image tracking example using OpenCV looks pretty cool. It is a complete example without going too deep or going off in the weeds.

Making the Unit Mobile introduced me to mobile platforms. The Magician Chassis that the book shows first, I found online for under $20! I knew that this stuff was accessible, but this is downright cheap. I feel almost guilty NOT getting one and trying it out.

The motor controller tutorial looks very straightforward. I already have ideas for code changes. Immediately after the simple time based tutorial it goes into speech controlled movement, which is pretty sweet.

After the wheeled robot tutorial is a walking robot example. The author makes a compelling argument for this type of robot, and the Pulse Width Modulation servo motors are cool, but I have to admit, this type of robot just doesn’t excite me. The book also punts on the PWM, using a controller which interprets serial USB commands into the PWM for the servos. For beginners, this is certainly the right choice.

Incidentally, the –help output from UscCmd includes Version, Culture, PublicKeyToken values like a Mono program might. I wonder if it is written in C# and running via Mono. I’m going to assume it is. That is pretty sweet. Indeed the linked download page mentions C#. http://www.pololu.com/docs/0J40/3.b

The sonar sensors section is a straightforward and great introduction to the use of them. I never knew how those things worked or what kind of value they returned. Now I do. Mounting the sensor to a survo makes for a nice subsystem on the bot.

Next, a fully remote control system is built. I don’t know if I like the choice of using an LCD monitor. It seems like overkill, but depending on the particular robotic application it would be a good choice. For the applications I have in mind, I think I’ll skip it. A wireless usb keyboard and mouse makes for an obvious choice. At this point, I just keep thinking about bluetooth and using an extra Wiimote, mostly because I think it would be a more fun control.

Oh, a GPS receiver! This could be necessary for when I lose my robot in a parking lot or the woods. As with the LCD Monitor and KB chapter, I kind of feel like I know how to do this since I’ve looked into it before. It is great coverage and good intro to the topic.

Much of my day job is what would traditionally be called Systems Programming so Chapter 10 is kind of a duh to me. I’d have started there, but that is just how I think about coding these days. Its great to have this in a chapter to tie some things together. In other words, read this chapter!

Using the BBB in sea, air and submarine applications is an interesting idea. I don’t think it is for me yet, but the book gives introduction to some ideas on the topic. The introduction to feedback control is very welcome.

Overall this is a great book. It really gave me a lot of ideas. It also showed me how easy it is to get started, something which I’d been a little hesitant to do. I’m actually a little excited to dive in now. I’ll be doing a bunch of this stuff with my 6yo over the next few years.