The details are all in the services config value. In my case, I am replacing the apache charm https://jujucharms.com/apache2/ using balancer and reverseproxy relations.

The apache charm has vhost_https_template and vhost_http_template which gets pasted in as apache httpd config. The haproxy charm services config value has service_options as yaml which works much the same.

In my case, port 80 redirects to 443, so I start with this in yaml:

- service_name: haproxy_service
  service_host: "0.0.0.0"
  service_port: 80
  server_options: maxconn 100 cookie S{i} check
  service_options:
      - 'redirect scheme https code 301 if !{ ssl_fc }'

You’ll notice this is almost identical to the default value for the services config:

- service_name: haproxy_service 
  service_host: "0.0.0.0" 
  service_port: 80 
  service_options: [balance leastconn, cookie SRVNAME insert] 
  server_options: maxconn 100 cookie S{i} check

I only changed the service_options entry with what would redirect haproxy in a frontend config.

This is where the magic of this haproxy charm happens. The haproxy charm knows which config values are for a frontend haproxy section and also for a backend haproxy section. The charm automatically puts the value in the right place.

The next thing which wasn’t obvious to me from reading the haproxy charm readme is that the Juju application related using reverseproxy relation becomes a backend section and its values will be merged from the services config.

e.g.

$ juju add-relation my-app haproxy:reverseproxy
$ juju add-relation kibana haproxy:reverseproxy

I can use defaults, or I can make some tweaks to the my-app and kibana applications.

For my use case, I was using apache httpd config RewriteRule ^/?KIBANA/(.*)$ balancer://kibana/$1 [P,L]

The equivalent in haproxy config looks like this:

    acl path_kibana path -m beg  /KIBANA/
    use_backend kibana if path_kibana

and in the kibana backend:

   reqirep  ^([^\ :]*)\ /KIBANA/(.*)     \1\ /\2

The haproxy charm allows all of this to be configured using services config. The related application is automatically set as a service name and your config must match it via service_name yaml.

I use juju2’s juju config command to set the config directly from a yaml file. If you are using juju1 you’ll need to use juju set command. juju config haproxy services=@haproxy-config-services.yaml

- service_name: my-app
  service_host: "0.0.0.0"
  service_port: 443
  crts: [DEFAULT]
  service_options:
      - balance leastconn
      - reqadd X-Forwarded-Proto:\ https
      - acl path_kibana path -m beg  /KIBANA/
      - use_backend kibana if path_kibana
  server_options: maxconn 100 cookie S{i} check
- service_name: kibana
  service_options:
      - balance leastconn
      - reqirep  ^([^\ :]*)\ /KIBANA/(.*)     \1\ /\2
      - rspirep ^Location:\ https?://[^/]+/(.*) Location:\ /KIBANA/\1
      - rspirep ^(Set-Cookie:.*)\ Path=(.*) \1\ Path=/KIBANA/\2
  server_options: maxconn 100 cookie S{i} check

The implication here is that there is another application, “my-app” which is also related to haproxy. This config tells haproxy to use my-app as the default application, but if the url starts with /KIBANA/, to use the kibana backend instead of the “my-app” backend. For completeness, I am including the equivalant of apache’s ProxyPassReverse and ProxyPassReverseCookiePath. These are the rspirep… Location and rspirep…Set-Cookie lines in the config respectively.

First, our testing tool is custom made, it makes all the websocket connections first and then proceeds to ping. This is important as it exercises the concurrent connections capabilities of httpd. When using it, the client system needs the ability to create enough sockets. The first limit I encountered was with my test client system. The shell environment defaults to 1024 open files limited. It is a soft limit, so use ulimit -S to adjust the limit. Even ab will show an error of “socket: Too many open files (24)” if you use -n 1050 and -c 1050 options.

$ ulimit -n
1024
$ ulimit -Hn
65536
$ ulimit -Sn 65536
$ ulimit -n
65536

Now, your testing tool can create more than 1024 connections. The next limit I ran into was that of connections on the httpd server. Even mpm_event uses thread per request (do not let the event name fool you). The default ubuntu apache2 mpm_event configuration allows for 150 concurrent connections:

 StartServers 2
 MinSpareThreads 25
 MaxSpareThreads 75
 ThreadLimit 64
 ThreadsPerChild 25
 MaxRequestWorkers 150
 MaxConnectionsPerChild 0

A tool like ab won’t halt at 150. A tool named slowhttptest is in xenial/universe. Run apt install slowhttptest to install it. It is a flexible tool and has a great man page and -h help output.

$ slowhttptest -c 1000 -H -g -o my_header_stats -i 10 -r 200 -t GET -u http://system.under.test.example.com/ -x 24 -p 3

slowhttptest version 1.6
– https://code.google.com/p/slowhttptest/ –
test type: SLOW HEADERS
number of connections: 1000
URL: http://system.under.test.example.com/
verb: GET
Content-Length header value: 4096
follow up data max size: 52
interval between follow up data: 10 seconds
connections per seconds: 200
probe connection timeout: 3 seconds
test duration: 240 seconds
using proxy: no proxy

Tue Sep 27 14:33:03 2016:
slow HTTP test status on 5th second:

initializing: 0
pending: 284
connected: 667
error: 0
closed: 0
service available: YES

This screen will update as connections are created until service available changes from YES to NO.

In my tests it closed: value was exactly 150. I can view the my_header_stats.csv file to see when max was reached.

Next, lets adjust Apache httpd to allow for more concurrent connections. My target is 15,000 connections, so I’ll increase numbers linearly 2 processes (StartServers) with 75 threads each (ThreadsPerChild) gave 150 connections. 20 processes with 750 threads each should give 15,000 connections.

Edit mpm_event.conf: ($ sudo vi /etc/apache2/mods-enabled/mpm_event.conf)

<IfModule mpm_event_module>
 StartServers 10
 MinSpareThreads 25
 MaxSpareThreads 750
 ThreadLimit 1000
 ThreadsPerChild 750
# MaxRequestWorkers aka MaxClients => ServerLimit *ThreadsPerChild
 MaxRequestWorkers 15000
 MaxConnectionsPerChild 0
 ServerLimit 20
 ThreadStackSize 524288
</IfModule>

Restart (full restart, not graceful – ThreadsPerChild change requires this) apache2 httpd and retry the slowhttptest. Notice service available is always YES.

Now turn up the slowhttptest numbers. Change the -c parameter to 15000 and the -r to 1500. It should take 10sec to ramp up the connections. In my use case I could not create that many connections so quickly. slowhttptest was maxing out a CPU core.

All of the above apache httpd config was done using the mpm_event processing module. The next issue I ran into was a case of mpm_worker not behaving as I expected. I have a doubly proxied system, because this is super real world where we route http things all over the place, sometimes in ways we shouldn’t but because we are lazy, or it is easier or… anyway…

In ubuntu/trusty with apache httpd 2.4.7 mpm_worker has a limit of 64 ThreadsPerChild even if you configure it with a larger number. There is no warning. You’d never know unless you take a look at the number of processes in a worker: $ ps -uwww-data -opid,ppid,nlwp  The fix is to switch from mpm_worker to mpm_event.

$ sudo a2dismod mpm_worker
$ sudo a2enmod mpm_event
$ sudo service apache2 restart

I thought that I’d need to do more, but this got me to where I needed to be.

This is on my Mac running OSX 10.10.

1. Get yourself an httpd.conf – cp /private/etc/apache2/httpd.conf .
2. Edit it to use a port >1024 and with user you – Listen 8081 & User jrwren & Group staff
3. Log to a place you can write – ErrorLog /home/jrwren/errorlog & CustomLog /home/jrwren/access_log combined
4. Use different pidfile –  PidFile /home/jrwren/httpd.pid Do this fter the Include /private/etc/apache2/extra/httpd-mpm.conf
5. Accept mutex –  Mutex file:/home/jrwren
6. Edit whatever else you want – ProxyPass / http://localhost:8080 & SetOutputFilter DEFLATE to see that Apache proxy does gzip for you
7. Start httpd – httpd -d . -f httpd.conf -X