A Ceph+radosgw for Storage for Dev/Test/QA in a Few Minutes

It turns out when I was adding Swift (object storage) support to a project I really wanted to also have Ceph support via radosgw, another object storage system.

It turns out that deploying Ceph is not trivial or easy. However, Juju helps manage the complexity of Big Software.

There are a couple of tricks to running Ceph on LXD.

  1. Specify a full path which does not start with /dev for the osd-devices option.
  2. use-direct-io: false on ZFS.

You can add these options to a bundle and deploy with a single command using a single bundle file like this:

$ cat > ceph-bundle.yaml
services:
 ceph-mon:
 charm: cs:~openstack-charmers-next/xenial/ceph-mon
 num_units: 3
 to:
 - '1'
 - '2'
 - '3'
 ceph-osd:
 charm: cs:~openstack-charmers-next/xenial/ceph-osd
 num_units: 3
 options:
 osd-devices: /srv/ceph-osd
 osd-reformat: 'yes'
 use-direct-io: false
 to:
 - '1'
 - '2'
 - '3'
 ceph-radosgw:
 charm: cs:~openstack-charmers-next/xenial/ceph-radosgw
 num_units: 1
 options:
 use-embedded-webserver: true
 to:
 - 1
relations:
- - ceph-osd:mon
 - ceph-mon:osd
- - ceph-radosgw:mon
 - ceph-mon:radosgw
machines:
 '1':
 constraints: arch=amd64
 series: xenial
 '2':
 constraints: arch=amd64
 series: xenial
 '3':
 constraints: arch=amd64
 series: xenial
^D
$ juju deploy ceph-bundle.yaml

Wait a while. Watch juju status. Then, see if radosgw is up and try s3cmd

$ curl 10.0.5.117
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
$ juju ssh ceph-radosgw/0 'sudo radosgw-admin user create --uid="ubuntu" --display-name="Ubuntu Ceph"'
{
 "user_id": "ubuntu",
 "display_name": "Ubuntu Ceph",
 "email": "",
 "suspended": 0,
 "max_buckets": 1000,
 "auid": 0,
 "subusers": [],
 "keys": [
 {
 "user": "ubuntu",
 "access_key": "O5W6PMIQZ83ODYCGVIGJ",
 "secret_key": "6aqf5vyRONOvGFJkvH65xW7ttxZIKNZx0c2cPMTA"
 }
 ],
 "swift_keys": [],
 "caps": [],
 "op_mask": "read, write, delete",
 "default_placement": "",
 "placement_tags": [],
 "bucket_quota": {
 "enabled": false,
 "max_size_kb": -1,
 "max_objects": -1
 },
 "user_quota": {
 "enabled": false,
 "max_size_kb": -1,
 "max_objects": -1
 },
 "temp_url_keys": []
}
$ s3cmd --host=10.0.5.117 --host-bucket=10.0.5.117 --access_key=O5W6PMIQZ83ODYCGVIGJ --secret_key=6aqf5vyRONOvGFJkvH65xW7ttxZIKNZx0c2cPMTA --signature-v2 mb s3://testb
$ s3cmd --host=10.0.5.117 --host-bucket=10.0.5.117 --access_key=O5W6PMIQZ83ODYCGVIGJ --secret_key=6aqf5vyRONOvGFJkvH65xW7ttxZIKNZx0c2cPMTA --signature-v2 put ceph.yaml s3://testb/ceph.yaml
$ s3cmd --host=10.0.5.117 --host-bucket=10.0.5.117 --access_key=O5W6PMIQZ83ODYCGVIGJ --secret_key=6aqf5vyRONOvGFJkvH65xW7ttxZIKNZx0c2cPMTA --signature-v2 ls
2016-11-16 20:23  s3://testb
$ s3cmd --host=10.0.5.117 --host-bucket=10.0.5.117 --access_key=O5W6PMIQZ83ODYCGVIGJ --secret_key=6aqf5vyRONOvGFJkvH65xW7ttxZIKNZx0c2cPMTA --signature-v2 ls s3://testb
2016-11-16 20:24 2581 s3://testb/ceph.yaml

Dev and Test as you wish.

It should go without saying, but I will write it anyway: do not use Ceph on LXD like this in production. Ceph must be scaled out to different machine nodes.

nodejs 7 on ubuntu

nodejs 7 was released and nodesource does an excellent job of creating packages of nodejs for use on many operating systems.

I refuse to curl $URL and pipe the results to bash. It scares me (maybe illogically) to trust a script on the internet with access to my local shell.

The commands without the curl pipe to shell are almost as short and run faster*. It is super easy to get nodejs 7.x installed your ubuntu xenial or yackety system.

curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo\ apt-key add -
sudo add-apt-repository -u "deb https://deb.nodesource.com/node_7.x $(lsb_release -c -s) main"
sudo apt install nodejs

* The apt update command is intentionally skipped. The -u option to add-apt-repository optimizes it to only merge available packages with the newly added repository. This is a bit faster, or a lot faster on older machines or slow cloud instances.

 

A Swift for Storage for Dev/Test/QA in 2 Minutes

I was adding swift support to a project and it became apparent that things would be easier if I had a local swift to which I could connect.

I needed to familiarize myself with the API and its behavior and doing this locally rather than over a VPN on some production system with a QA tenant would make things a lot easier.

I could have used Devstack. Devstack is great if you are developing OpenStack. That is for what it is made. It uses the OpenStack source. That seemed overkill to me.

What I ended up with is a cloud-config file which I pass to cloud-init on system start. I use LXD to start a container. In less than 2 minutes later, on my 10 year old home server, I have swift up and running and responding to my commands.

$ lxc launch -e ubuntu:16.04 $(petname) -c user.user-data="$(cat swift.yaml)"
Creating testy-Abril
Starting testy-Abril
$ lxc list
+-----------------+---------+-----------------------+-+-----------+---+
| testy-Abril     | RUNNING | 10.0.5.169 (eth0)     | | EPHEMERAL | 0 |
+-----------------+---------+-----------------------+-+-----------+---+
$ swift --user admin:admin --key
admin -A http://10.0.5.169:8080/auth/v1.0 list
$ swift --user admin:admin --key admin -A http://10.0.5.169:8080/auth/v1.0 list
$ swift --user admin:admin --key admin -A http://10.0.5.169:8080/auth/v1.0 upload t README.md
$ swift --user admin:admin --key admin -A http://10.0.5.169:8080/auth/v1.0 list
t
$ swift --user admin:admin --key admin -A http://10.0.5.169:8080/auth/v1.0 list t
README.md

No need to ssh into the container at all. Just start it, wait a bit for things to install, and a swift API is up and running.

The swift.yaml file is here on github and the only change you should make is either remove the last line or change it to import your key so you can ssh to it.

Scaling Apache httpd as a ReverseProxy

We recently had the need to make sure our front end apache httpd reverse proxy and ssl termination server could handle the larger number of websocket connections we are going to use with it. Given websockets are longer lived connections, this is a different use of apache httpd and we want to get it right. The proxied service is capable of handling tens of thousands of concurrent connections, if not hundreds of thousands or more.

First, our testing tool is custom made, it makes all the websocket connections first and then proceeds to ping. This is important as it exercises the concurrent connections capabilities of httpd. When using it, the client system needs the ability to create enough sockets. The first limit I encountered was with my test client system. The shell environment defaults to 1024 open files limited. It is a soft limit, so use ulimit -S to adjust the limit. Even ab will show an error of “socket: Too many open files (24)” if you use -n 1050 and -c 1050 options.

$ ulimit -n
1024
$ ulimit -Hn
65536
$ ulimit -Sn 65536
$ ulimit -n
65536

Now, your testing tool can create more than 1024 connections. The next limit I ran into was that of connections on the httpd server. Even mpm_event uses thread per request (do not let the event name fool you). The default ubuntu apache2 mpm_event configuration allows for 150 concurrent connections:

 StartServers 2
 MinSpareThreads 25
 MaxSpareThreads 75
 ThreadLimit 64
 ThreadsPerChild 25
 MaxRequestWorkers 150
 MaxConnectionsPerChild 0

A tool like ab won’t halt at 150. A tool named slowhttptest is in xenial/universe. Run apt install slowhttptest to install it. It is a flexible tool and has a great man page and -h help output.

$ slowhttptest -c 1000 -H -g -o my_header_stats -i 10 -r 200 -t GET -u http://system.under.test.example.com/ -x 24 -p 3

slowhttptest version 1.6
– https://code.google.com/p/slowhttptest/ –
test type: SLOW HEADERS
number of connections: 1000
URL: http://system.under.test.example.com/
verb: GET
Content-Length header value: 4096
follow up data max size: 52
interval between follow up data: 10 seconds
connections per seconds: 200
probe connection timeout: 3 seconds
test duration: 240 seconds
using proxy: no proxy

Tue Sep 27 14:33:03 2016:
slow HTTP test status on 5th second:

initializing: 0
pending: 284
connected: 667
error: 0
closed: 0
service available: YES

This screen will update as connections are created until service available changes from YES to NO.

In my tests it closed: value was exactly 150. I can view the my_header_stats.csv file to see when max was reached.

Next, lets adjust Apache httpd to allow for more concurrent connections. My target is 15,000 connections, so I’ll increase numbers linearly 2 processes (StartServers) with 75 threads each (ThreadsPerChild) gave 150 connections. 20 processes with 750 threads each should give 15,000 connections.

Edit mpm_event.conf: ($ sudo vi /etc/apache2/mods-enabled/mpm_event.conf)

<IfModule mpm_event_module>
 StartServers 10
 MinSpareThreads 25
 MaxSpareThreads 750
 ThreadLimit 1000
 ThreadsPerChild 750
# MaxRequestWorkers aka MaxClients => ServerLimit *ThreadsPerChild
 MaxRequestWorkers 15000
 MaxConnectionsPerChild 0
 ServerLimit 20
 ThreadStackSize 524288
</IfModule>

Restart (full restart, not graceful – ThreadsPerChild change requires this) apache2 httpd and retry the slowhttptest. Notice service available is always YES.

Now turn up the slowhttptest numbers. Change the -c parameter to 15000 and the -r to 1500. It should take 10sec to ramp up the connections. In my use case I could not create that many connections so quickly. slowhttptest was maxing out a CPU core.

All of the above apache httpd config was done using the mpm_event processing module. The next issue I ran into was a case of mpm_worker not behaving as I expected. I have a doubly proxied system, because this is super real world where we route http things all over the place, sometimes in ways we shouldn’t but because we are lazy, or it is easier or… anyway…

In ubuntu/trusty with apache httpd 2.4.7 mpm_worker has a limit of 64 ThreadsPerChild even if you configure it with a larger number. There is no warning. You’d never know unless you take a look at the number of processes in a worker: $ ps -uwww-data -opid,ppid,nlwp  The fix is to switch from mpm_worker to mpm_event.

$ sudo a2dismod mpm_worker
$ sudo a2enmod mpm_event
$ sudo service apache2 restart

I thought that I’d need to do more, but this got me to where I needed to be.

Ubuntu Kiosk

This post is a work in progress. I’ll update it as I tweak the solution.

Last Wednesday I was helping a friend build a Kiosk. We tried to follow https://thepcspy.com/read/building-a-kiosk-computer-ubuntu-1404-chrome/ but it didn’t work. It turns out between using the wrong version of ubuntu (16.04 instead of 14.04) and doing it in a virtual machine, we were all messed up.

There has to be a better way.

There is a secret to debian/ubuntu packages. If you aren’t trying to get them included in debian/ubuntu, you can break most of the rules and get them to do whatever you want. I figured I should be able to use this and make creating a kiosk as easy as apt install kiosk

TL;DR: you can try this by running these two commands on a new ubuntu-server installation:

add-apt-repository ppa:evarlast/kiosk
apt install --no-install-recommends kioskme

The rest of the post describes how I did this.

First, I’m going to create a new PPA on launchpad just for this, so that a user can `add-apt-repository ppa:evarlast/kiosk`

I visit https://launchpad.net/~/+activate-ppa and fill in the fields with kiosk and click activate.

Next, I start a new deb. I may as well build it from source. There might be a better way, but I’ve gotten to know dh (debhelper) a bit, so I’m going to use it.

$ mkdir kioskme ; cd kioskme
$ cat > Makefile
build:
<tab>echo noop
install:
<tab>install -d 755 ${DESTDIR}/usr/bin
<tab>install -m 755 kioskme ${DESTDIR}/usr/bin/kioskme
^D
$ cat > kioskme
#!/bin/bash
xset -dpms
xset s off 
openbox-session & 
start-pulseaudio-x11 
while true; do 
  rm -rf ~/.{config,cache}/chromium/ 
  chromium-browser --kiosk --no-first-run 'http://duckduckgo.com' 
done
^D

Now debianize this script directory using dh_make:

dh_make -p kioskme_0.0.0 --createorig -s

Now customize the deb with a service, preinst for user creation and some dependencies:

$ cat > debian/service
[Unit]
Description=kioskme

[Service]
Type=simple
Restart=on-failure
User=kioskme
Group=kioskme
ExecStart=/usr/bin/startx /etc/X11/Xsession /usr/bin/kioskme
^D
$ cat > debian/preinst
#!/bin/sh

set -e

. /usr/share/debconf/confmodule

case "$1" in
 install|upgrade)
 if ! getent group kioskme >/dev/null; then
 addgroup --system kioskme >/dev/null
 fi
 if ! getent passwd kioskme >/dev/null; then
 adduser \
 --system \
 --disabled-login \
 --ingroup kioskme \
 --gecos kioskme \
 --shell /bin/false \
 kioskme >/dev/null
 fi
 mkdir -p /var/log/kioskme
 chown kioskme:kioskme /var/log/kioskme
 setfacl -m u:kioskme:rw /dev/tty0 /dev/tty7
 ;;

 abort-upgrade)
 ;;

 *)
 echo "preinst called with unknown argument \`$1'" >&2
 exit 1
 ;;
esac

# dh_installdeb will replace this with shell code automatically
# generated by other debhelper scripts.

#DEBHELPER#

exit 0

Alright, maybe that preinst is a bit big. I copy it around and fill it out like a template for services I put into debs.

Now edit the debian/control file to add dependencies, change the section to utils, fill in whatever else you want, set Depends to look like this:

Depends: ${shlibs:Depends}, ${misc:Depends}, Xorg, openbox, chromium-browser, pulseaudio

Now create the deb:

fakeroot debian/rules clean build binary

To test the deb, I copy it to a fresh ubuntu server install and dpkg -i to install it. I get a bunch of errors because dpkg -i doesn’t resolve dependencies, but I run apt install -f and the dependencies are installed.

Once I tested and tweaked and got things working, I updated the tarball `tar -Jcf ../kioskme_0.0.0-0.orig.tar.xz -C ..  –exclude=’debian’ kioskme` and I used dpkg-buildpackage -S to build a source package and then I used dput ppa:evarlast/kiosk ../kioskme_0.0.0-1_source.changes to upload to PPA.

Now, this still does not work in a VM. Ubuntu desktop installer must do some magic to make X work in a virtual machine with a driver which works with VMWare, VirtualBox, or Parallels.

 

Some LXD containers on a hidden net, others on your lan

Back in November I wrote about Converting eth0 to br0 and getting all your LXC or LXD onto your LAN

It works, but you might not want ALL of your LXD on your LAN.

You’ll still need your LAN interface to be a br0 instead of a device that isn’t a bridge. Go follow the Bridge your interface section of that post to convert your eth0 to br0.

I’ve fully converted to using LXD. I don’t even remember if LXC supports profiles. I think it does, so I think the same idea could be applied to LXC, but I’m only showing this for LXD.

First, copy the default profile:

lxc profile copy default lanbridge

Second, edit the new profile to use br0 instead of lxdbr0:

lxc profile device set lanbridge eth0 parent br0

Third and finally, start instances with that profile:

lxc launch ubuntu-xenial -p lanbridge

In my case, this instance is on my local lan AND on public ipv6 space (thanks Comcast).

heritable-gale    | RUNNING | 192.168.15.172 (eth0) | 2601:400:8000:5ab3:216:3eff:fe73:d242 (eth0)

 

Cloud-config with LXD

A year ago I wrote http://jrwren.wrenfam.com/blog/2015/05/26/ubuntu-cloud-image-based-containers-with-lxc/

Since then, LXD became the best way to use LXC.

By default, LXD already uses ubuntu-cloudimg images.

The lesser know feature is using cloud-config with LXD. It turns out it is very easy to pass user-data to an LXD instance when you start it, just like you would on any cloud provider.

LXD even has the -e option to make your LXD instance ephemeral. It will be deleted automatically when you stop it.

Just like in that previous blog post, I create a file named one.yaml. The name can be anything. Then i start it:

lxc launch ubuntu:14.04 crisp-Hadley -c user.user-data="$(cat one.yaml)"

That is all there is to it.

Here is an example of config similar to what I used recently to QA a build configuration:

#cloud-config
output:
 all: "|tee -a /tmp/cloud.out"
#hostname: {{ hostname }}
bootcmd:
 - rm -f /etc/dpkg/dpkg.cfg.d/multiarch
apt_sources:
 - source: ppa:yellow/ppa
ssh_import_id: [evarlast] # use -S option
packages:
 - make
final_message: "The system is finally up, after $UPTIME seconds"
runcmd:
 - cd /home/ubuntu
 - git clone https://www.github.com/jrwren/myproject
 - cd myproject
 - make deps run

Inverse Multimatch Source Address Blocking With iptables and ipset

I wanted to block all traffic on port 22 except for a few hosts that I use.

I was tried of seeing lots of stupid worm attack traffic on my EC2 host.

Jun  9 19:02:45 ip-172-30-4-108 sshd[5004]: Invalid user cisco from 218.85.133.73
Jun  9 19:02:45 ip-172-30-4-108 sshd[5004]: input_userauth_request: invalid user cisco [preauth]
Jun  9 19:02:46 ip-172-30-4-108 sshd[5004]: Connection closed by 218.85.133.73 port 9224 [preauth]

Yes, I could use security groups, but then I’d have to use security groups.

iptables ! -s with,more,than,one,address fails

iptables v1.6.0: ! not allowed with multiple source or destination IP addresses

The alternative is to use ipset. Its not hard!

ipset create ssh-ok hash:ip
ipset add ssh-ok mine.example.com
ipset list  # is this thing working, just checking.
ipset add ssh-ok myfriend.example.com
ipset add ssh-ok mywork.example.com
ipset list  # still working, ok looks good.

iptables -A INPUT -m set \! --match-set ssh-ok src -p tcp --dport 22 -j DROP

Thanks for the help:

http://daemonkeeper.net/781/mass-blocking-ip-addresses-with-ipset/

http://unix.stackexchange.com/questions/70917/iptables-multiple-exclusions-on-port-forwarding

What It Is Like To Be a Programmer

This is the most realistic video of what it is to be a programmer that I have ever seen. They read the docs and the source a majority of the time in order to be able to write what they are writing. This is rarely demonstrated when I hear people describe programming to new or would-be programmers.

jq Is the grep, sed and awk for json

The only problem with jq is that its not installed by default in ubuntu or ubuntu-server. Its not in the default ubuntu-cloudimg. One must apt-get install jq.

https://stedolan.github.io/jq/manual/ says, “jq is a lightweight and flexible command-line JSON processor.

In working with juju, we work with json formatted cookies in a ~/.go-cookies file. Sometimes we need to investigate these cookies to develop, verify, and debug our services.

An unexpired cookie value might be as good as a password or authentication token and so for the purpose of our debugging sometimes everything but the value is good enough. The jq filter ‘.[]|del(.Value)‘ strips all of the .Value properties from every object in the input array. This results in:

{
“Name”: “macaroon-a40e7abc65a78faf130dc652d45052c1c8b5b4aeff8181f44a15175b6525558f”,
“Domain”: “api.staging.example.com”,
“Path”: “/identity/”,
“Secure”: false,
“HttpOnly”: false,
“Persistent”: true,
“HostOnly”: true,
“Expires”: “2016-05-09T19:52:21Z”,
“Creation”: “2016-04-11T15:52:21.466266522-04:00”,
“LastAccess”: “2016-04-11T15:52:21.928768825-04:00”,
“Updated”: “2016-04-11T15:52:21.928768825-04:00”,
“CanonicalHost”: “api.staging.example.com”
}
{
“Name”: “macaroon-a605d07b7a95ba7e57a267ed507f673bce1188d0de7f544074f1c33ec4a8ff2a”,
“Domain”: “www.example.org”,
“Path”: “/identity/”,
“Secure”: false,
“HttpOnly”: false,
“Persistent”: true,
“HostOnly”: true,
“Expires”: “2016-05-03T21:46:35Z”,
“Creation”: “2016-04-05T17:46:36.351842179-04:00”,
“LastAccess”: “2016-05-02T15:11:10.525298848-04:00”,
“Updated”: “2016-04-05T17:46:36.351842179-04:00”,
“CanonicalHost”: “www.example.org”
}
{
“Name”: “macaroon-authn”,
“Domain”: “www.example.org”,
“Path”: “/NEENR/”,
“Secure”: false,
“HttpOnly”: false,
“Persistent”: true,
“HostOnly”: true,
“Expires”: “2016-05-03T19:11:09.794240373Z”,
“Creation”: “2016-05-02T15:11:10.592852105-04:00”,
“LastAccess”: “2016-05-02T15:23:26.813664654-04:00”,
“Updated”: “2016-05-02T15:11:10.592852105-04:00”,
“CanonicalHost”: “www.example.org”
}

Now lets say you want to remove the cookie with the Path value “/NEENR/”.

The jq filter: ‘.[] | select(.Path!=”/NEENR/”)’ does that job.

These examples show filter and map, but what about reduce?

Min, max, min_by and max_by are nice default reducers.

  • min_by(.Expires) shows the next expiring cookie.
  • max_by(.Created) shows the most recently created cookie.
  • [.[]|.Expires]|max if you don’t care about the rest of the cookie and just want the max date.
  • [.[]|.Expires]|min if you just want the min date.

See the Array Construction section of the manual for the details on the syntax. I like to think of it as the .[]|.NAME returns elements and if I want them in an array I wrap it in [] for array construction.

jq is a sweet tool that I’m glad to have in my toolbox.