First, our testing tool is custom made, it makes all the websocket connections first and then proceeds to ping. This is important as it exercises the concurrent connections capabilities of httpd. When using it, the client system needs the ability to create enough sockets. The first limit I encountered was with my test client system. The shell environment defaults to 1024 open files limited. It is a soft limit, so use ulimit -S to adjust the limit. Even ab will show an error of “socket: Too many open files (24)” if you use -n 1050 and -c 1050 options.

$ ulimit -n
1024
$ ulimit -Hn
65536
$ ulimit -Sn 65536
$ ulimit -n
65536

Now, your testing tool can create more than 1024 connections. The next limit I ran into was that of connections on the httpd server. Even mpm_event uses thread per request (do not let the event name fool you). The default ubuntu apache2 mpm_event configuration allows for 150 concurrent connections:

 StartServers 2
 MinSpareThreads 25
 MaxSpareThreads 75
 ThreadLimit 64
 ThreadsPerChild 25
 MaxRequestWorkers 150
 MaxConnectionsPerChild 0

A tool like ab won’t halt at 150. A tool named slowhttptest is in xenial/universe. Run apt install slowhttptest to install it. It is a flexible tool and has a great man page and -h help output.

$ slowhttptest -c 1000 -H -g -o my_header_stats -i 10 -r 200 -t GET -u http://system.under.test.example.com/ -x 24 -p 3

slowhttptest version 1.6
– https://code.google.com/p/slowhttptest/ –
test type: SLOW HEADERS
number of connections: 1000
URL: http://system.under.test.example.com/
verb: GET
Content-Length header value: 4096
follow up data max size: 52
interval between follow up data: 10 seconds
connections per seconds: 200
probe connection timeout: 3 seconds
test duration: 240 seconds
using proxy: no proxy

Tue Sep 27 14:33:03 2016:
slow HTTP test status on 5th second:

initializing: 0
pending: 284
connected: 667
error: 0
closed: 0
service available: YES

This screen will update as connections are created until service available changes from YES to NO.

In my tests it closed: value was exactly 150. I can view the my_header_stats.csv file to see when max was reached.

Next, lets adjust Apache httpd to allow for more concurrent connections. My target is 15,000 connections, so I’ll increase numbers linearly 2 processes (StartServers) with 75 threads each (ThreadsPerChild) gave 150 connections. 20 processes with 750 threads each should give 15,000 connections.

Edit mpm_event.conf: ($ sudo vi /etc/apache2/mods-enabled/mpm_event.conf)

<IfModule mpm_event_module>
 StartServers 10
 MinSpareThreads 25
 MaxSpareThreads 750
 ThreadLimit 1000
 ThreadsPerChild 750
# MaxRequestWorkers aka MaxClients => ServerLimit *ThreadsPerChild
 MaxRequestWorkers 15000
 MaxConnectionsPerChild 0
 ServerLimit 20
 ThreadStackSize 524288
</IfModule>

Restart (full restart, not graceful – ThreadsPerChild change requires this) apache2 httpd and retry the slowhttptest. Notice service available is always YES.

Now turn up the slowhttptest numbers. Change the -c parameter to 15000 and the -r to 1500. It should take 10sec to ramp up the connections. In my use case I could not create that many connections so quickly. slowhttptest was maxing out a CPU core.

All of the above apache httpd config was done using the mpm_event processing module. The next issue I ran into was a case of mpm_worker not behaving as I expected. I have a doubly proxied system, because this is super real world where we route http things all over the place, sometimes in ways we shouldn’t but because we are lazy, or it is easier or… anyway…

In ubuntu/trusty with apache httpd 2.4.7 mpm_worker has a limit of 64 ThreadsPerChild even if you configure it with a larger number. There is no warning. You’d never know unless you take a look at the number of processes in a worker: $ ps -uwww-data -opid,ppid,nlwp  The fix is to switch from mpm_worker to mpm_event.

$ sudo a2dismod mpm_worker
$ sudo a2enmod mpm_event
$ sudo service apache2 restart

I thought that I’d need to do more, but this got me to where I needed to be.

It works, but you might not want ALL of your LXD on your LAN.

You’ll still need your LAN interface to be a br0 instead of a device that isn’t a bridge. Go follow the Bridge your interface section of that post to convert your eth0 to br0.

I’ve fully converted to using LXD. I don’t even remember if LXC supports profiles. I think it does, so I think the same idea could be applied to LXC, but I’m only showing this for LXD.

First, copy the default profile:

lxc profile copy default lanbridge

Second, edit the new profile to use br0 instead of lxdbr0:

lxc profile device set lanbridge eth0 parent br0

Third and finally, start instances with that profile:

lxc launch ubuntu-xenial -p lanbridge

In my case, this instance is on my local lan AND on public ipv6 space (thanks Comcast).

heritable-gale    | RUNNING | 192.168.15.172 (eth0) | 2601:400:8000:5ab3:216:3eff:fe73:d242 (eth0)

I was tried of seeing lots of stupid worm attack traffic on my EC2 host.

Jun  9 19:02:45 ip-172-30-4-108 sshd[5004]: Invalid user cisco from 218.85.133.73
Jun  9 19:02:45 ip-172-30-4-108 sshd[5004]: input_userauth_request: invalid user cisco [preauth]
Jun  9 19:02:46 ip-172-30-4-108 sshd[5004]: Connection closed by 218.85.133.73 port 9224 [preauth]

Yes, I could use security groups, but then I’d have to use security groups.

iptables ! -s with,more,than,one,address fails

iptables v1.6.0: ! not allowed with multiple source or destination IP addresses

The alternative is to use ipset. Its not hard!

ipset create ssh-ok hash:ip
ipset add ssh-ok mine.example.com
ipset list  # is this thing working, just checking.
ipset add ssh-ok myfriend.example.com
ipset add ssh-ok mywork.example.com
ipset list  # still working, ok looks good.

iptables -A INPUT -m set \! --match-set ssh-ok src -p tcp --dport 22 -j DROP

Thanks for the help:

http://daemonkeeper.net/781/mass-blocking-ip-addresses-with-ipset/

http://unix.stackexchange.com/questions/70917/iptables-multiple-exclusions-on-port-forwarding

One thing which I’ve done which has made all LXC and LXD instances more valuable to me, in my home development environment, is to use a bridge to put them directly on my home LAN.

Normally, LXC creates its own device, lxc-br0, which is managed by the lxc-net service. The service creates the device, brings it up, manages the dnsmasq tied to it (which provides DHCP for the 10.0.3.0/24 range).

Bridge your interface

Instead of using lxc-br0, I create a br0. I add my eth0 (and in my case other devices) to that br0. Then I configure LXC and LXD to use br0 instead of lxc-br0. I go as far as stopping the lxc-net service, since I’m not using it.

There is one trick if you are going to do that on a remote home system, e.g. I have an old laptop I leave in the basement and I’m really lazy and I don’t want to walk down there and use its console when I screw up its networking. The trick is to make sure eth0 comes up on br0 when its added there.

Before you do anything, make sure bridge-utils is installed. It probably is if you are already using lxc, but if this is a fresh install, you’ll want to apt-get install bridge-utils

Edit your /etc/network/interfaces and disable eth0 by setting it to manual. Add it to br0 by adding a new br0 section and listing eth0 in bridge-ifaces and bridge-ports.

auto br0
iface br0 inet dhcp
    bridge-ifaces eth0
    bridge-ports eth0
    up ifconfig eth0 up

iface eth0 inet manual

Now run sudo ifup br0. At this point something magical happens, the DHCP lease is renewed but this time the IP address is bound to br0. The magical part is that br0 used the eth0 MAC to make the DHCP request and so you get the same IP address in response and even your SSH session stays open. YAY!

LXC can use any bridge

Now configure LXC to use this bridge.

apt-get install lxc
sed -i 's/lxc.network.link = lxcbr0/lxc.network.link = br0/' /etc/lxc/default.conf

TADA, now any LXC containers you start with lxc-start will use br0 and get an address from your household DHCP server. They will be accessible from any host in your home.

Now what about LXD?

LXD can use any bridge

It turns out, while LXD is a layer on top of LXC, it doesn’t use /etc/lxc/default.conf for its default config, but instead uses its own settings. These are editable with lxc profile edit default. Change the lxcbr0 in your editor and save and exit. You can check that it is correct by using lxc profile show default.

There you have it. LXD instances starting on your local LAN.

Now go read Wayne’s post again and use the Juju LXD provider.

EC2 should never see your private key… because.. security!

I launched a Windows Server 2012 R2 instance in EC2 recently and while the AWS console does let you retrieve an Administrator password, it requires you to paste your PRIVATE key to AWS console to do it. I couldn’t bring myself to do it, so I learned how to use boto to get the encrypted password data and openssl cmdline to decrypt it to get the password.

Its a 2 step process with maybe the zeroth step being writing a .boto file with your aws credentials if you have never used boto.

import boto
import base64
ec2 = boto.connect_ec2()
inst = ec2.get_all_instances()[0].instance
data = ec2.get_password_data(inst.id)
open(‘ec2-admin-password’,’w’,write(base64.decodestring(data))

I’m assuming its the only instance running. If you have lots of others, use a list comprehension with if clause to filter to one on the get_all_instances() call, or just skip that call and paste an id string you see in AWS console for inst.id in the get_password_data call.

openssl rsautil -in ec2-admin-password -inkey .ssh/id_rsa -decrypt

You’ll be prompted for your private key password (and you MUST have a password. ssh-agent is easy) and then the Administrator password will be output to stdout.

http://serverfault.com/a/410618/79028

It is a simple iptables ruleset which blocks most of the common rfc1918 addresses. You have probably heard of these, the 10/8, 192.168/16 and 172.16/12 address ranges. What might be new to you, is that there is a whole great many more ranges which one should never observe on the internet.

You might add to your list:

• TEST-NET(192.0.2.0/24) from rfc3330
• benchmarktest(198.18.0/25) from rfc2544
• protocol assignment(192.0.0.0/24)
• testnet2(198.51.100/24) and testnet3(203.0.113/24) from rfc5736 and 5737
• carrier grade nat(100.64/10) from rfc6598

Blocking addresses of these ranges is completely valid. IANA has not and will not assign them for use on the internet. They are reserved and non internet route-able.

Some notes:

• Still no 6rd support on OpenWRT AFAIK
• rc2 and rc3 are the same for the brcm-2.4 version of Openwrt 10.03.1
• brcm4700 doesn’t work well at all with my WRT54GL. I think the open source broadcom drivers still aren’t as stable as the proprietary ones that ship with 2.4
• nearly the same config scripts as the old kamikaze work

Once you flash the router with the firmware you will need to install some extra packages.

1. opkg update
2. opkg install ip kmod-ipv6 kmod-sit radvd
3. paste this code into a new startup script at /etc/init.d/comcast6to4

#!/bin/sh /etc/rc.common

inetip=`ip -4 addr show dev eth0.1 | awk ‘/inet / {print $2}’ | cut -d/ -f 1`
inetipspaced=`echo $inetip | tr . ‘ ‘`
local6prefix=`printf 2002:%02x%02x:%02x%02x $inetipspaced`

start() {
ip tunnel add c6to4 mode sit ttl 255 remote any local $inetip
ip link set c6to4 up
ip -6 addr add $local6prefix:0::1/64 dev c6to4
ip -6 addr add $local6prefix:1::1/64 dev br-lan
ip -6 route add 2000::/3 via ::192.88.99.1 dev c6to4
sysctl -w net.ipv6.conf.all.forwarding=1 > /dev/null
cat > /etc/radvd.conf <<EOF
interface br-lan
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
prefix $local6prefix:1::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
AdvValidLifetime 86400;
AdvPreferredLifetime 86400;
};
};
EOF
}

stop() {
  ip tunnel del c6to4
  ip -6 addr del $local6prefix:1::1/64 dev br-lan
}

4. pushd /etc/rc.d ; ln –s ../init.d/comcast6to4 S42comcast6to4

Then be glad you have ipv6.

This will actually work for ANY provider which uses the standard IPv6 6to4 address of 192.88.99.1, not just Comcast.

Now if only Comcast would open back up their trial so I could join my work to the ipv6 network.

I’m running Kamikaze 8.09.1 brcm-2.4 on my Linksys WRT54GL these instructions probably won’t work elsewhere. I’m guessing that IPv6 is a little different in a 2.6 kernel with a new iptables. If you have very new stuff you should be using 6rd instead of 6to4.

I’m writing this because much of the information I found out there for 6to4 on Linux didn’t work for me, or was only partly correct and I had to piece together suggestions from different sources.

make a script in /etc/rc.d with this content. I called mine comcast6to4

inetip=`ip -4 addr show dev eth0.1 | awk ‘/inet / {print $2}’ | cut -d/ -f 1`
inetipspaced=`echo $inetip | tr . ‘ ‘`
local6prefix=`printf 2002:%02x%02x:%02x%02x $inetipspaced`
ip tunnel add c6to4 mode sit ttl 255 remote any local $inetip
ip link set c6to4 up
ip -6 addr add $local6prefix:0::1/64 dev c6to4
ip -6 addr add $local6prefix:1::1/64 dev br-lan
ip -6 route add 2000::/3 via ::192.88.99.1 dev c6to4

Make your /etc/radvd.conf look like this:

interface br-lan
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
prefix $local6prefix:1::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
AdvValidLifetime 86400;
AdvPreferredLifetime 86400;
};
};

That is it. I’m not going to explain it. Read the links below for all of that.

Sorry this isn’t a complete solution. You’ll have to fill in that $local6prefix in radvd.conf yourself.

Works Cited:

http://www.reddit.com/r/linux/comments/dbobx/

http://www.comcast6.net/

http://wiki.debian.org/DebianIPv6#IPv66to4Configuration

http://tldp.org/HOWTO/Linux+IPv6-HOWTO/configuring-ipv6to4-tunnels.html

http://en.wikipedia.org/wiki/6to4

http://www.dslreports.com/forum/r24972279-IPv6-via-6in4

http://www.anyweb.co.nz/tutorial/v6Linux6to4

http://wlanchannelinfo.codeplex.com/

https://code.edge.launchpad.net/~evarlast/+junk/WlanChannelInfo

I wrote this because Wifi in my home is very slow. Its so slow I’m tempted to run a network cable to my couch so that even when I’m couch surfing I can have fast access to my server.

In an effort to diagnose my slow Wifi, I tried to see if my neighbors were causing interference by running Wifi on the same or overlapping channel as me. I downloaded netstumbler; it didn’t work. I downloaded some other tool; neither did it.

So I wondered how hard it would be to write my own. It turns out Windows 7 added to the Wlan* api to expose all of the necessary data. After some digging I found the managedwlan project on codeplex. Now I got to play.

Once I figured out the api, I was able to write the entire application with pretty much one LINQ expression:

var client = new WlanClient();
var retval =
from wlanIface in client.Interfaces
from bssentry in wlanIface.GetNetworkBssList()
from network in wlanIface.GetAvailableNetworkList(Wlan.WlanGetAvailableNetworkFlags.IncludeAllAdhocProfiles)
where InterfaceService.GetStringForSSID(network.dot11Ssid) == InterfaceService.GetStringForSSID(bssentry.dot11Ssid)
select new WifiInfo
{
bssentry = GetStringForSSID(bssentry.dot11Ssid),
channel = Wifi.FrequencyChannelMap[bssentry.chCenterFrequency],
frequency = bssentry.chCenterFrequency,
linqQuality = bssentry.linkQuality,
strength = bssentry.rssi,
signalQuality = network.wlanSignalQuality,
wifitype = network.dot11BssType
};

The result of that expression is directly databound to a WPF DataGrid and I can now view the data that I want to.

I really love the platform (C#+.NET) on which I work.