Inverse Multimatch Source Address Blocking With iptables and ipset

I wanted to block all traffic on port 22 except for a few hosts that I use.

I was tried of seeing lots of stupid worm attack traffic on my EC2 host.

Jun  9 19:02:45 ip-172-30-4-108 sshd[5004]: Invalid user cisco from 218.85.133.73
Jun  9 19:02:45 ip-172-30-4-108 sshd[5004]: input_userauth_request: invalid user cisco [preauth]
Jun  9 19:02:46 ip-172-30-4-108 sshd[5004]: Connection closed by 218.85.133.73 port 9224 [preauth]

Yes, I could use security groups, but then I’d have to use security groups.

iptables ! -s with,more,than,one,address fails

iptables v1.6.0: ! not allowed with multiple source or destination IP addresses

The alternative is to use ipset. Its not hard!

ipset create ssh-ok hash:ip
ipset add ssh-ok mine.example.com
ipset list  # is this thing working, just checking.
ipset add ssh-ok myfriend.example.com
ipset add ssh-ok mywork.example.com
ipset list  # still working, ok looks good.

iptables -A INPUT -m set \! --match-set ssh-ok src -p tcp --dport 22 -j DROP

Thanks for the help:

http://daemonkeeper.net/781/mass-blocking-ip-addresses-with-ipset/

http://unix.stackexchange.com/questions/70917/iptables-multiple-exclusions-on-port-forwarding

What It Is Like To Be a Programmer

This is the most realistic video of what it is to be a programmer that I have ever seen. They read the docs and the source a majority of the time in order to be able to write what they are writing. This is rarely demonstrated when I hear people describe programming to new or would-be programmers.

jq Is the grep, sed and awk for json

The only problem with jq is that its not installed by default in ubuntu or ubuntu-server. Its not in the default ubuntu-cloudimg. One must apt-get install jq.

https://stedolan.github.io/jq/manual/ says, “jq is a lightweight and flexible command-line JSON processor.

In working with juju, we work with json formatted cookies in a ~/.go-cookies file. Sometimes we need to investigate these cookies to develop, verify, and debug our services.

An unexpired cookie value might be as good as a password or authentication token and so for the purpose of our debugging sometimes everything but the value is good enough. The jq filter ‘.[]|del(.Value)‘ strips all of the .Value properties from every object in the input array. This results in:

{
“Name”: “macaroon-a40e7abc65a78faf130dc652d45052c1c8b5b4aeff8181f44a15175b6525558f”,
“Domain”: “api.staging.example.com”,
“Path”: “/identity/”,
“Secure”: false,
“HttpOnly”: false,
“Persistent”: true,
“HostOnly”: true,
“Expires”: “2016-05-09T19:52:21Z”,
“Creation”: “2016-04-11T15:52:21.466266522-04:00”,
“LastAccess”: “2016-04-11T15:52:21.928768825-04:00”,
“Updated”: “2016-04-11T15:52:21.928768825-04:00”,
“CanonicalHost”: “api.staging.example.com”
}
{
“Name”: “macaroon-a605d07b7a95ba7e57a267ed507f673bce1188d0de7f544074f1c33ec4a8ff2a”,
“Domain”: “www.example.org”,
“Path”: “/identity/”,
“Secure”: false,
“HttpOnly”: false,
“Persistent”: true,
“HostOnly”: true,
“Expires”: “2016-05-03T21:46:35Z”,
“Creation”: “2016-04-05T17:46:36.351842179-04:00”,
“LastAccess”: “2016-05-02T15:11:10.525298848-04:00”,
“Updated”: “2016-04-05T17:46:36.351842179-04:00”,
“CanonicalHost”: “www.example.org”
}
{
“Name”: “macaroon-authn”,
“Domain”: “www.example.org”,
“Path”: “/NEENR/”,
“Secure”: false,
“HttpOnly”: false,
“Persistent”: true,
“HostOnly”: true,
“Expires”: “2016-05-03T19:11:09.794240373Z”,
“Creation”: “2016-05-02T15:11:10.592852105-04:00”,
“LastAccess”: “2016-05-02T15:23:26.813664654-04:00”,
“Updated”: “2016-05-02T15:11:10.592852105-04:00”,
“CanonicalHost”: “www.example.org”
}

Now lets say you want to remove the cookie with the Path value “/NEENR/”.

The jq filter: ‘.[] | select(.Path!=”/NEENR/”)’ does that job.

These examples show filter and map, but what about reduce?

Min, max, min_by and max_by are nice default reducers.

  • min_by(.Expires) shows the next expiring cookie.
  • max_by(.Created) shows the most recently created cookie.
  • [.[]|.Expires]|max if you don’t care about the rest of the cookie and just want the max date.
  • [.[]|.Expires]|min if you just want the min date.

See the Array Construction section of the manual for the details on the syntax. I like to think of it as the .[]|.NAME returns elements and if I want them in an array I wrap it in [] for array construction.

jq is a sweet tool that I’m glad to have in my toolbox.

Version from debian/changelog

Almost two years ago I did some scripting of updating debian/changelog and building a package to enable a CI environment for some software. I wanted to parse the changelog correctly and so I copied and changed some perl from the source of dpkg-buildpackage. This turned out to be the wrong solution.

There is a nice tool called dpkg-parsechangelog. You can get just the version for use in scripts with this simple awk:

dpkg-parsechangelog | awk ‘/Version/ { print $2 }’

I didn’t even think to write about it until I ran across someone else who’d written some perl to do exactly the same thing. Dear world, we need to stop reinventing this wheel.

Ubuntu Xenial 16.04 Has All The Good Stuff

A couple of days ago, Ubuntu Xenial was released. There is a press release with some good stuff in it.

I’ve been looking forward to this release for the following reasons:

  • Postgresql 9.5
  • systemd
  • haproxy 1.6.3
  • uwsgi 2.0.12
  • nginx 1.9.15

I know, it doesn’t look that exciting until you recall that the last LTS release of Ubuntu, Trusty, 14.04, was missing fabulous features OOTB in each of these components.

Postgresql 9.3 did not have the the awesome JSONB improvements of 9.4 and 9.5

haproxy 1.4 didn’t have ssl support.

uwsgi… well latest uwsgi is just always great to have.

nginx 1.9.15 has http2 support, out of the box!

Finally, while I loved upstart, systemd is nice and has been rock solid.

This is the greatest Ubuntu ever. I’ve not even mentioned how awesome lxd is on it. That is covered elsewhere. This is just my personal little list. Thanks Ubuntu.

MacGyvering Windows 8.1 Remote Assistance

My Mother called me up rather frazzled this evening.

This isn’t too surprising. Since her stroke 16 years ago she can sometimes become confused or forget simple things, things she once knew.

Tonight, the cause of her frazzled state was her computer.

After listening to her rant and ramble about her computer, I quickly realized that she had some web browser pop-up phishing telling her she had a virus. Partly because of who she is, and partly because of brain damage from stroke, she called the phone number that the pop-up displayed. When they told her they can fix it for $199 and if she took it to Best Buy, they would charge her $350-$400, this fueled her worry.

After some calming I finally had her start the Windows Remote Assistance application, but unfortunately she has forgotten what saving files actually means and she has no email configured. So she is unable to save the remote assist file and she can’t use Windows Remote Assistance to automatically email the request to me. It was at this point that I suggested she mail the laptop to me. I also may have said, “never again!” when I agree to support a laptop that someone else gifted her.

But, I couldn’t let it go. This was a challenge and I love a challenge.

I searched around a bit and tried my hand at the msra.exe command line. After a bit of trial and error, I realized I can have her open a powershell and type

msra /saveasfile helpme 12345678

Yes, I’m ok with the 12345678 password in this case. Trying some other password over the phone and having her type it was error prone.

“Did you say bee?”

“No I said pee, like Paul.”

“Bee like ball?”

“No…”

I still needed a way to get a file to me. I’ve had an aversion to PowerShell ever since it launched, despite tech reviewing a very fine PowerShell book. I knew it was probably my best bet at getting a file to me. After a bit of poking I found the invoke-webrequest helper, thingy. I don’t know PowerShell terminology. It looks like a function to me.

I have my home server on the internet. Its running Ubuntu Linux  and I’ve had 4 line php upload scripts with html forms that let people send me files for years. Could I use this?

The shoelace was there. The paperclip was there. Did I also have some bubble gum?

All I really needed as an index.php in a /mom/ directory that looked like this:

<?php
file_put_contents('err.out', file_get_contents('php://input'));
?>

Wow that is some trivial stuff. Bland bubble gum, I guess.

Why an index.php and a /mom/? Well, because that will be easy for me to relay over the telephone.

I did some testing and found invoke-webrequest works nicely coupled with this http request body dumping php.

invoke-webrequest -uri jrwren.xmtp/mom/ -infile .\helpme
.msrcincident -method post

I was able to call my mom back, tell her, to press windows key-r, reminding her that windows key is usually between the ctrl and alt on the keyboard, and to type powershell and press enter.

“Powershell, P-O-W-E-R-S-H-E-L-L- no spaces?”

“Yup”

On first try, I tried to have her use the password 1234, but msra.exe complained that it was too short. Working through this mistake, I tried to have her use the up arrow to edit the previously executed command line in powershell.

“What is the up arrow?”

This honestly dumbfounded me and I had absolutely no idea what to do for a minute or so.

“The up arrow on my keyboard is on the right. There is an inverted tee of arrows, left right up down to the left of my left control key.”

Whew, I got lucky and she found it.

Once we had the msra.exe create the helpme file, I had her type out the invoke-webrequest command, prompting her to press tab after typing helpme to autocomplete the file extension.

The multiline color output of running the command shocked and surprised her. It maybe even scared her a little bit, but as she was reading it aloud, I heard her say, “200 OK”

“200 OK is great”, I said.

I checked my server and there was an err.out file along side the index.php. The only two files in the mom directory.

My home server always has samba setup. I used Windows Explorer to navigate to H:\public_html\mom and I renamed err.out to helpme.msrcincident. I double clicked it.

Mom said, “Oh what is this? jrwren wants to share your computer.”

I rejoiced inside.

The hard part being done, I was able to connect and control her computer. Microsoft has done a very nice job with Windows Remote Assist, ever since Windows 7. I’m impressed that my Windows 7 can connect flawlessly to her Windows 8.1. I’m thankful that PowerShell is out of the box in all versions of windows. I do not think I’d have been able to walk her through this over the phone with this few keystrokes without PowerShell.

To the evil con artists who extort money from poor little old disabled ladies who work two jobs: please stop.

Optimizing uwsgi for Many Many Threads and Processes

tl;dr : Consider optimizing uwsgi by setting `threads-stacksize = 64` or some small value in your uwsgi config. Python apps which do not use many C modules do not use the C stack very much. A smaller stack size mean threads use less memory and you can safely have more of them servicing requests.

Long story:

Years ago I was deploying a new flask web service using uwsgi. I needed it to scale to thousands of connections. I read a blog post (I searched and cannot find it now) which suggested 10 processes with 10 threads each to be able to serve 100 concurrent connections. After testing and tuning this particular app, we settled on 10 processes and 100 threads per process. It ran well.

Recently, a production app, which I helped deploy, fell on its face. It was performing very poorly, seemingly out of nowhere. This app was originally deployed with the same 10 processes, 100 threads per process configuration which I had used so successfully in the past. The ops team had already reduced the process count to 4 due to excessive memory use of the application. This means the application was only able to service 400 concurrent connections.

I still cannot entirely explain why the app ran for many months and then suddenly had problems. I’m guessing it is because of recent announcements driving more traffic to the site. The 400 threads were actually being used instead of sitting idle waiting for connections.

In the process of trying to restore service, our ops team wisely used a tool which I was not likely to have used (huge thanks to them). The tool is pmap and it shows mapped memory for a given process. I noticed something interesting in the output of pmap:

00007fcf75061000   8192K rw---   [ anon ]
00007fcf75861000      4K -----   [ anon ]
00007fcf75862000   8192K rw---   [ anon ]
00007fcf76062000      4K -----   [ anon ]

This was repeated with the same memory increment 50 times for a total of 100. It occurred to me that the default stack size of a thread in Linux is 8MB and that these memory maps were the stack of each thread. I was able to confirm this suspicion by running the app myself and adjusting the size by configuring uwsgi with –threads-stacksize.

I started by moving to 1MB which I know is the default Windows thread stack size, guessing it would still be plenty. Then I started to play limbo and see how low can I go. I started to get pretty happy when I broke the 256KB mark and our app was still functioning. Our app has the luxury of not having any deep calls. I might have been able to go lower, but once I got to 64KB, I didn’t see my point. Every order of magnitude decrease was smaller and smaller an improvement.

Moving from 8MB to 1MB took memory usage from 3.2GB to 400MB. Every halving of stack size halved overall memory usage of the thread stacks by this app. First 512KB/thread for 200MB, then 256KB/thread for 100MB, then 128KB/thread for 50MB, then 64KB/thread for 25MB. At this point, everything about the app was running exactly the same, the only difference being that I wasn’t wasting 3.2GB of memory in unused thread stacks.

 

I Welcome Parse Developers to Juju

Hello Parse developers,

I was curious how easy it would be to get the published parse-server-example to run with Juju. The end result is that there is a new juju charm named parse-server available.*

Deploying parse-server is as easy as running these commands in a bootstrapped juju environment. This means that it can run ANYWHERE.

juju deploy cs:~evarlast/trusty/parse-server-0
juju deploy mongodb
juju add-relation parse-server mongodb
juju expose parse-server

You’ll then be able to use the http api at port 1337.

For example:

curl -X POST -H “X-Parse-Application-Id: myAppId” -H “Content-Type: application/json” -d ‘{“whatever”:”data”}’ 10.0.3.247:1337/parse/functions/hello

If you wish to take a look at this charm, its in the charmed branch of my fork of the parse-server-example. I do not recommend using this charm as an example of writing a good production charm. This is an example of a quick and dirty hack of a charm which happens to work.

Part of what makes Juju awesome is the magic of application modeling. While my hack of a parse-server charm isn’t production ready, it is building on a very production ready mongodb charm, which can be scaled out and made HA very easily. Charms are reusable open source ops. The mongodb ops have been captured in the mongodb charm. Any required parse-server ops need to be capture in a parse-server charm. The only ones captured so far are configuring the mongodb relation. While its a hack of a demo charm, it is a start.

 

—-

* The real reason is that I have cloud envy and I saw the azure release at https://azure.microsoft.com/en-us/blog/azure-welcomes-parse-developers/ and I thought to myself, gee that is a lot of clicks, seems like there is a better way.

Converting eth0 to br0 and getting all your LXC or LXD onto your LAN

Wayne has a great post on the new juju lxd work. I’ve been using it a bit and it is awesome. It is super fast and I can create and destroy environments faster than creating and destroying with juju-local.

One thing which I’ve done which has made all LXC and LXD instances more valuable to me, in my home development environment, is to use a bridge to put them directly on my home LAN.

Normally, LXC creates its own device, lxc-br0, which is managed by the lxc-net service. The service creates the device, brings it up, manages the dnsmasq tied to it (which provides DHCP for the 10.0.3.0/24 range).

Bridge your interface

Instead of using lxc-br0, I create a br0. I add my eth0 (and in my case other devices) to that br0. Then I configure LXC and LXD to use br0 instead of lxc-br0. I go as far as stopping the lxc-net service, since I’m not using it.

There is one trick if you are going to do that on a remote home system, e.g. I have an old laptop I leave in the basement and I’m really lazy and I don’t want to walk down there and use its console when I screw up its networking. The trick is to make sure eth0 comes up on br0 when its added there.

Before you do anything, make sure bridge-utils is installed. It probably is if you are already using lxc, but if this is a fresh install, you’ll want to apt-get install bridge-utils

Edit your /etc/network/interfaces and disable eth0 by setting it to manual. Add it to br0 by adding a new br0 section and listing eth0 in bridge-ifaces and bridge-ports.

auto br0
iface br0 inet dhcp
    bridge-ifaces eth0
    bridge-ports eth0
    up ifconfig eth0 up

iface eth0 inet manual

Now run sudo ifup br0. At this point something magical happens, the DHCP lease is renewed but this time the IP address is bound to br0. The magical part is that br0 used the eth0 MAC to make the DHCP request and so you get the same IP address in response and even your SSH session stays open. YAY!

wikipedia network bridge

LXC can use any bridge

Now configure LXC to use this bridge.

apt-get install lxc
sed -i 's/lxc.network.link = lxcbr0/lxc.network.link = br0/' /etc/lxc/default.conf

TADA, now any LXC containers you start with lxc-start will use br0 and get an address from your household DHCP server. They will be accessible from any host in your home.

Now what about LXD?

LXD can use any bridge

It turns out, while LXD is a layer on top of LXC, it doesn’t use /etc/lxc/default.conf for its default config, but instead uses its own settings. These are editable with lxc profile edit default. Change the lxcbr0 in your editor and save and exit. You can check that it is correct by using lxc profile show default.

There you have it. LXD instances starting on your local LAN.

Now go read Wayne’s post again and use the Juju LXD provider.