Programming – Jay R. Wren – lazy dawg evarlast

A Swift for Storage for Dev/Test/QA in 2 Minutes

I was adding swift support to a project and it became apparent that things would be easier if I had a local swift to which I could connect.

I needed to familiarize myself with the API and its behavior and doing this locally rather than over a VPN on some production system with a QA tenant would make things a lot easier.

I could have used Devstack. Devstack is great if you are developing OpenStack. That is for what it is made. It uses the OpenStack source. That seemed overkill to me.

What I ended up with is a cloud-config file which I pass to cloud-init on system start. I use LXD to start a container. In less than 2 minutes later, on my 10 year old home server, I have swift up and running and responding to my commands.

$ lxc launch -e ubuntu:16.04 $(petname) -c user.user-data="$(cat swift.yaml)"
Creating testy-Abril
Starting testy-Abril
$ lxc list
+-----------------+---------+-----------------------+-+-----------+---+
| testy-Abril     | RUNNING | 10.0.5.169 (eth0)     | | EPHEMERAL | 0 |
+-----------------+---------+-----------------------+-+-----------+---+
$ swift --user admin:admin --key
admin -A http://10.0.5.169:8080/auth/v1.0 list
$ swift --user admin:admin --key admin -A http://10.0.5.169:8080/auth/v1.0 list
$ swift --user admin:admin --key admin -A http://10.0.5.169:8080/auth/v1.0 upload t README.md
$ swift --user admin:admin --key admin -A http://10.0.5.169:8080/auth/v1.0 list
t
$ swift --user admin:admin --key admin -A http://10.0.5.169:8080/auth/v1.0 list t
README.md

No need to ssh into the container at all. Just start it, wait a bit for things to install, and a swift API is up and running.

The swift.yaml file is here on github and the only change you should make is either remove the last line or change it to import your key so you can ssh to it.

What It Is Like To Be a Programmer

This is the most realistic video of what it is to be a programmer that I have ever seen. They read the docs and the source a majority of the time in order to be able to write what they are writing. This is rarely demonstrated when I hear people describe programming to new or would-be programmers.

Version from debian/changelog

Almost two years ago I did some scripting of updating debian/changelog and building a package to enable a CI environment for some software. I wanted to parse the changelog correctly and so I copied and changed some perl from the source of dpkg-buildpackage. This turned out to be the wrong solution.

There is a nice tool called dpkg-parsechangelog. You can get just the version for use in scripts with this simple awk:

dpkg-parsechangelog | awk ‘/Version/ { print $2 }’

I didn’t even think to write about it until I ran across someone else who’d written some perl to do exactly the same thing. Dear world, we need to stop reinventing this wheel.

Optimizing uwsgi for Many Many Threads and Processes

tl;dr : Consider optimizing uwsgi by setting `threads-stacksize = 64` or some small value in your uwsgi config. Python apps which do not use many C modules do not use the C stack very much. A smaller stack size mean threads use less memory and you can safely have more of them servicing requests.

Long story:

Years ago I was deploying a new flask web service using uwsgi. I needed it to scale to thousands of connections. I read a blog post (I searched and cannot find it now) which suggested 10 processes with 10 threads each to be able to serve 100 concurrent connections. After testing and tuning this particular app, we settled on 10 processes and 100 threads per process. It ran well.

Recently, a production app, which I helped deploy, fell on its face. It was performing very poorly, seemingly out of nowhere. This app was originally deployed with the same 10 processes, 100 threads per process configuration which I had used so successfully in the past. The ops team had already reduced the process count to 4 due to excessive memory use of the application. This means the application was only able to service 400 concurrent connections.

I still cannot entirely explain why the app ran for many months and then suddenly had problems. I’m guessing it is because of recent announcements driving more traffic to the site. The 400 threads were actually being used instead of sitting idle waiting for connections.

In the process of trying to restore service, our ops team wisely used a tool which I was not likely to have used (huge thanks to them). The tool is pmap and it shows mapped memory for a given process. I noticed something interesting in the output of pmap:

00007fcf75061000   8192K rw---   [ anon ]
00007fcf75861000      4K -----   [ anon ]
00007fcf75862000   8192K rw---   [ anon ]
00007fcf76062000      4K -----   [ anon ]

This was repeated with the same memory increment 50 times for a total of 100. It occurred to me that the default stack size of a thread in Linux is 8MB and that these memory maps were the stack of each thread. I was able to confirm this suspicion by running the app myself and adjusting the size by configuring uwsgi with –threads-stacksize.

I started by moving to 1MB which I know is the default Windows thread stack size, guessing it would still be plenty. Then I started to play limbo and see how low can I go. I started to get pretty happy when I broke the 256KB mark and our app was still functioning. Our app has the luxury of not having any deep calls. I might have been able to go lower, but once I got to 64KB, I didn’t see my point. Every order of magnitude decrease was smaller and smaller an improvement.

Moving from 8MB to 1MB took memory usage from 3.2GB to 400MB. Every halving of stack size halved overall memory usage of the thread stacks by this app. First 512KB/thread for 200MB, then 256KB/thread for 100MB, then 128KB/thread for 50MB, then 64KB/thread for 25MB. At this point, everything about the app was running exactly the same, the only difference being that I wasn’t wasting 3.2GB of memory in unused thread stacks.

Zulu JRE from Azul Systems is a hidden gem

http://www.azulsystems.com/products/zulu Azul Systems, the company that Cliff Click works for, builds their own openjdk version.

If you don’t recall Cliff Click, I was first introduced to him via this awesome video: https://www.youtube.com/watch?v=agH7Cz5FSxY

If I have to run on the JVM, then this is how I want to run on the JVM.

Zulu isn’t Zing, and yet it is a hidden gem. No more stupid prompts from Oracle. No more being associated with the company that forces you to install the Ask toolbar and other spywear.

The download page is here: http://www.azulsystems.com/products/zulu/downloads

It is the best (only?) way to get openjdk onto your OSX mac.

Better still is the package install on Ubuntu.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0x219BD9C9
sudo apt-add-repository "deb http://repos.azulsystems.com/ubuntu stable main"
sudo apt-get update 
sudo apt-get install zulu-8

Zulu includes something called the CCK which says

The Commercial Compatibility Kit (CCK) for Zulu contains additional functionality that is not included in in the OpenJDK source, but which will help ensure compatibility in applications that take advantage of specific additional features that Oracle bundles into HotSpot.

curl -O http://cdn.azulsystems.com/zcck/2014-08-8-bin/zcck8-8.0.0.2-amd64.deb
sudo dpkg -i zcck8-8.0.0.2-amd64.deb

Do better Java on Ubuntu.

golang goals

When discussing the Go programming language, I find it useful to always reference the goals of the language. Discussion tends to devolve into a comparison of features of other programming languages which Go lacks. Without the context of these goals, the discussion ceases being useful.

Stolen from a Google tech talk that Rob Pike did back in 2009:

Goals

The efficiency of a statically-typed compiled language with the ease of programming of a dynamic language.
Safety: type-safe and memory-safe.
Good support for concurrency and communication.
Efficient, latency-free garbage collection.
High-speed compilation.

Watch the Go at Google video or read the article and you will get the impression that these goals are NOT in their order of importance. I suggest that the last item, High-speed compilation, trumps all the others.

Here is a short, 1:15 video demonstrating the speed of the go compiler: https://www.youtube.com/watch?v=wwoWei-GAPo

The first comment, over 2 years ago, at Lambda the Ultimate, about that Go at Google video, sums it up even better. It is a snapshot of another slide. This time instead of Go goals, the slide is “What makes large-scale development hard with C++ or Java (at least):”

slow builds
uncontrolled dependencies
each programmer using a different subset of the language
poor program understanding (documentation, etc.)
duplication of effort
cost of updates
version skew
difficulty of automation (auto rewriters etc.): tooling
cross-language builds

* Language features don’t usually address these.

It took me quite a while to keep these above things in mind when thinking about Go. In fact, I still tend to compare Go to my favorite programming languages, probably because I often forget some of those above drawbacks to C++/Java (read C# for me).

I’ll try to remember. I beg others to try to remember too.

ctags for golang and vim; just the right things with godeps

I use vim.

I like to press ctrl-] to go to a tag and ctrl-t to pop up that tag stack.

I use ctags from homebrew to generate my tags file which vim reads. The OSX version of ctags is inadequate.

I often invoke ctags with -R . and a list of directory names to the libraries which I am using. When using ctags with python or C this works reasonably well, but I have to maintain the list of directories somewhere.

Go lets me handle this case slightly better.

godeps is a tool which does two things, in this case I only care about the first. godeps prints the source dependencies of named packages.

Combining godeps output and sending it to ctags means my tags file automatically has tokens from all of my dependent packages. I name this shell function goctags.

goctags () { godeps ./… | awk -v GOPATH=$GOPATH ‘{print GOPATH”/src/”$1}’ | xargs ctags -R .; }

I added it to my .bashrc.

Book Review: BeagleBone Robotics Projects

I was asked by Packt Publishing if I would read and review this book. I’ve owned a BeagleBoneBlack for a little while now. My use case was not robotics. This book might shed some new light on my old Black, so I agreed to review it.

The book starts off very accessible. Chapter 1 covers just about everything I did with my BBB when I first received it, hooking it up like a PC, replacing the default distro, making sure I could SSH to it were all in there. The author, Richard Grimmett, goes a step further and installs XFCE gui and vncserver and walks through connecting from a Windows PC using vncclient. All in all, chapter 1 is a great super basic tour.

Chapter 2 dives into programming on the thing and introduces Python. It does it in a really weird (to me) way. It has the reader running emacs in a putty window remote connected to the device. This must just feel weird to me because I do a lot of remote programming and its never with emacs (I’m a vim guy) and its rarely remote. For a new user, it seems to me like it would have been simpler and more friendly to say “use an editor of your choice” and “here is notepad2 or sublime” along with “here is how you copy files to and from the device.” I think this is mostly my background causing me to see things differently. The emacs in putty walk-through is very adequate.

Its not a programming book, so this is really a nit pick, but technically some of the descriptions of python aren’t really true. For example, if __name__==”__main__”: does not “tell the program to begin its execution at this point.” Again I’m nit picking, but I do feel like a different phrase that isn’t so very false to someone who knows python could have been found. Still, its not a programming book. The beginning of the chapter does list many resources for learning python.

Ugh, and then the book moves on to C++ and has quotes like this, “C++ is the original language of Linux” I’ve used Linux for almost as long as I’ve programmed C, and I am very (perhaps overly?) sensitive to the difference between C and C++.

OMG what do you mean Speech Input and Output? Really? Chapter 3 tackles it. Really. For real. Speech Input and Output on that tiny little board. I can make my own Siri! This is a really cool topic; espeek is something I’ve only played with a little bit prior to reading this. It looks fun.

Speech recognition is done with software I’ve never used before called PocketSphinx. It isn’t packaged and so one has to compile it. Pretty sweet BBB being able to compile stuff like that. (I’m thinking of iOS and Android where I’ve not seen a compiler run on device.) The demo walks through limiting the grammar of speech input so that you don’t have to train the recognizer.

I’m a programmer, so I’m going to nitpick programmer things. I really wish authors wouldn’t do this, “I like to make a copy of the current file into continuous.c.old, so I can always get back to the starting program if it is required.” I really do wish authors would just say “go read about version control systems.”

Whew, speech is fun. Next step is video. Hook up a webcam and let’s do some image recognition. The book walks through OpenCV and it is as this point that we are forced to do a bunch of Linux sysadmin stuff to make our SD have enough free space to have a dev environment. This really could have gone anywhere in the book. I kind of like that it put it off until it was necessary.

The python image tracking example using OpenCV looks pretty cool. It is a complete example without going too deep or going off in the weeds.

Making the Unit Mobile introduced me to mobile platforms. The Magician Chassis that the book shows first, I found online for under $20! I knew that this stuff was accessible, but this is downright cheap. I feel almost guilty NOT getting one and trying it out.

The motor controller tutorial looks very straightforward. I already have ideas for code changes. Immediately after the simple time based tutorial it goes into speech controlled movement, which is pretty sweet.

After the wheeled robot tutorial is a walking robot example. The author makes a compelling argument for this type of robot, and the Pulse Width Modulation servo motors are cool, but I have to admit, this type of robot just doesn’t excite me. The book also punts on the PWM, using a controller which interprets serial USB commands into the PWM for the servos. For beginners, this is certainly the right choice.

Incidentally, the –help output from UscCmd includes Version, Culture, PublicKeyToken values like a Mono program might. I wonder if it is written in C# and running via Mono. I’m going to assume it is. That is pretty sweet. Indeed the linked download page mentions C#. http://www.pololu.com/docs/0J40/3.b

The sonar sensors section is a straightforward and great introduction to the use of them. I never knew how those things worked or what kind of value they returned. Now I do. Mounting the sensor to a survo makes for a nice subsystem on the bot.

Next, a fully remote control system is built. I don’t know if I like the choice of using an LCD monitor. It seems like overkill, but depending on the particular robotic application it would be a good choice. For the applications I have in mind, I think I’ll skip it. A wireless usb keyboard and mouse makes for an obvious choice. At this point, I just keep thinking about bluetooth and using an extra Wiimote, mostly because I think it would be a more fun control.

Oh, a GPS receiver! This could be necessary for when I lose my robot in a parking lot or the woods. As with the LCD Monitor and KB chapter, I kind of feel like I know how to do this since I’ve looked into it before. It is great coverage and good intro to the topic.

Much of my day job is what would traditionally be called Systems Programming so Chapter 10 is kind of a duh to me. I’d have started there, but that is just how I think about coding these days. Its great to have this in a chapter to tie some things together. In other words, read this chapter!

Using the BBB in sea, air and submarine applications is an interesting idea. I don’t think it is for me yet, but the book gives introduction to some ideas on the topic. The introduction to feedback control is very welcome.

Overall this is a great book. It really gave me a lot of ideas. It also showed me how easy it is to get started, something which I’d been a little hesitant to do. I’m actually a little excited to dive in now. I’ll be doing a bunch of this stuff with my 6yo over the next few years.

When Reversing the Interview Process becomes How Would You Do Fun Things in C

My boss’s boss’s pals wrote this: http://blog.exodusintel.com/2012/09/18/reversing-the-interview-process/

Its a story about how someone was asked a crazy fun C question in an interview and how the new team decided to try it.

After reading this and discussing it with coworkers, I decided to try it and of course the first thing that came to my mind was a way to use tail recursion to do it.

 1 #include <stdio.h>
 2 
 3 int c;
 4 //auto func0 = [&] () -> int { c++; _strlen(s+1);};
 5 //auto funcn = [&] () -> int { return c; };
 6 int rs(char* s);
 7 int go(char* s) { c++; return rs(s+1); }
 8 int ret(char* s) { return c; }
 9 int rs(char* s) {
10     int (*func[2]) (char *s) = {ret,go};
11     char i = (*s>>7 | *s>>6 | *s>>5 | *s>>4 | 
        *s>>3 | *s>>2 | *s>>1) & 1;
12     return func[i](s);
13 }
14 int _strlen(char* s) {
15     c = 0;
16     return rs(s);
17 }
18 
19 int main(int argc, char* argv[]) {
20     printf("_strlen(%s): %d\n", argv[1], 
        _strlen(argv[1]));
21     return 0;
22 }

After writing it, I went and looked at the other fella solutions for the second time. I should also mention that I haven’t written C on the job in 11 years, and when I did then, it was one tiny program which was quickly replaced with perl. I have never been anything other than an intro beginner C programmer.

Things I noticed after going back is that my solution is somewhat similar to Brandon and Zef’s solution, but I think both my use of function pointers and bit shifting are more elementary. I’m still not sure about how some parts of their solution works.

Storyboard Custom Segue For Custom PushViewController Animation

While there are a lot of google hits when searching for custom pushViewController Animation, I found nothing regarding use of a Custom Segue to make it reusable and I also found a lot of misinformation like “it can’t be done with the default UINavigationController.” It can.

From your Button, View, Gesture Recognizer or whatever, instead of dragging from Push, drag from Custom.

Then select the Segue that is created and type in a class name the custom Segue.

Now we can create the FromTopReplaceSegue class. Use Add File or however you like to create new classes in XCode.

//FromTypeReplaceSegue.h

#import <UIKit/UIKit.h>
@interface FromTopReplaceSegue : UIStoryboardSegue
@end

//FromTypeReplaceSegue.m

#import “FromTopReplaceSegue.h”
@implementation FromTopReplaceSegue
-(void)perform{
UIViewController *dst = [self destinationViewController];
UIViewController *src = [self sourceViewController];
[dst viewWillAppear:NO];
[dst viewDidAppear:NO];

[src retain];

[src.view addSubview:dst.view];

CGRect original = dst.view.frame;

dst.view.frame = CGRectMake(dst.view.frame.origin.x, 0-dst.view.frame.size.height, dst.view.frame.size.width, dst.view.frame.size.height);

[UIView beginAnimations:nil context:nil];
dst.view.frame = CGRectMake(original.origin.x, original.origin.y, original.size.height, original.size.width);
[UIView commitAnimations];

[self performSelector:@selector(animationDone:) withObject:dst afterDelay:0.2f];
}
– (void)animationDone:(id)vc{
UIViewController *dst = (UIViewController*)vc;
UINavigationController *nav = [[self sourceViewController] navigationController];
[nav popViewControllerAnimated:NO];
[nav pushViewController:dst animated:NO];
[[self sourceViewController] release];
}
@end

In this CustomSegue not only are we doing custom animation from Top to Bottom (just like the default push navigation of Right to Left) but instead of pushing, we are replacing the top view controller.

In my current project I have a nearly identical FromButtomReplaceSegue that does the replace but animates from Button. I hope for a library of these with varying animation transitions and Push/Replace variants of each. Then anytime you want to use a different animation you can simply use a Custom Segue instead of writing a bunch of code in ViewDidLoad or wherever. Hurray Storyboard!

http://stackoverflow.com/questions/2215672/how-to-change-the-push-and-pop-animations-in-a-navigation-based-app
http://stackoverflow.com/questions/5878732/how-to-create-uinavigationcontroller-animation-top-to-bottom
http://dmunsie.wordpress.com/2009/08/07/custom-animations-between-uiviewcontrollers/