Docker Datacenter LAB – Load balancer for DTR and SWIFT storage backend – part VI

This post will show final steps for DDC LAB setup. This time I will implement and configure load balancer [LB] for DTR and add Swift [ OpenStack Object Storage] to the configuration. I am using HAProxy VM for load balancing of 3x DTR servers. As before HAProxy will monitor specific URL on the backend and determine if DTR service is ready for receiving the client requests.
So here’s how I configured DTR LB in my demo HA LAB.

/ LB server /

I have created new instance docker-lb2 for DTR load balancing and external service discovery. Specs are: 2GB of memory, 2VCPUs and 10GB root disk.

Red Hat subscription:

subscription-manager register —auto-attach
subscription-manager repos —disable=*
subscription-manager repos –enable=rhel-7-server-rpms
yum update -y

 

Docker UCP server certificates are created under:

/var/lib/docker/volumes/ucp-controller-server-certs/

 

Another option is to create self-sign certificate or use externally signed certificate. [my option]

cat server.crt server.key > server.pem
cp server.pem /etc/ssl/private/

 

Example of creating Self-sign certificate on HAProxy server:

openssl req -new -newkey rsa:2048 -x509 -days 365 -nodes -keyout server.key -out server.crt
cat server.crt server.key > dtr-ca.pem
cp dtr-ca.pem /etc/ssl/certs/
🙂 In /etc/haproxy directory is script to create dummy certs.

 

/ DTR – /etc/haproxy/haproxy.cfg /

Front end is listening on both 80 and 443 ports, but http traffic is redirected by default to https.
Newer documentation points that URL to listen to or monitor is /health but that did not work for me. I am using /load_balancer_status to determine status of DRT service on each server.

The http-request directives tells the proxy, in short, to forward original client IP over SSL layer.

...
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  myhttps
    bind *:80
    redirect scheme https if !{ ssl_fc }
    bind *:443 ssl crt /etc/ssl/certs/dtr-ca.pem
    default_backend dtr

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend dtr
    mode http
    balance roundrobin
    option forwardfor
#    option httpchk GET /health
    option httpchk GET /load_balancer_status
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    server docker-lab-4 10.236.13.17:443 check ssl
    server docker-lab-5 10.236.13.19:443 check ssl
    server docker-lab-6 10.236.13.20:443 check ssl

Enable and start HAProxy service

systemctl enable haproxy && systemctl start haproxy
systemctl status -l haproxy

 

/DTR Domain name/

Changing DTR Domain name to the FQDN of DTR master server which is in this case docker-lab-4-master.highvail.com.
This is not the proxy server, but first DTR node as any permutation with LB breaks the trust. If you have an advice, share.
But we are still going to use LB service.
Here is the list of registered DTRs:
screenshot

/ Trust me UCP  I am your DTR /

New sw update from Docker automatically registers UCP to trust newly configured DTR but we still need one more step if we want to prevent ‘x509 certificate signed by unknown authority‘ error.

On each UCP controller = swarm manager do:

#DO NOT SOURCE THE BUNDLE FILE

export DOMAIN_NAME=docker-lab-4.highvail.com

openssl s_client -connect $DOMAIN_NAME:443 -showcerts </dev/null 2>/dev/null | \
openssl x509 -outform PEM | \

sudo tee /etc/pki/ca-trust/source/anchors/${DOMAIN_NAME}.crt

sudo update-ca-trust

sudo /bin/systemctl restart docker.service

# from each controller node test login => should work :)
docker login -u ${ADMIN_USERNAME} -p ${ADMIN_PASSWORD} ${DOMAIN_NAME}

166db2d9-8bb2-4c89-a8cd-746b6e1ae42b

In order to test HA first configure storage on shared backend. I am using same Openstack setup, but now Swift will be used for registry persistent storage:
screenshot
Verify setup on OpenStack:
screenshot

/Now I can do some testing. /

 
I can say that Docker DC behaves very good and tolerates number of deadly scenarios starting with compete instance shutdown. Give it a bit of time so RAFT gets nodes checked and soon after service is back up. When failures occurs there is red or amber bar on top of the screen indicating that something is wrong, assuming you are using proxy to access the service.
Only scenario where my configuration lacks is when master DTR instance goes down. This is because I have specified node 4 as external service IP. But my docker apps were still unaffected. Another edge case is when primary UCP node [leader manager] is shut down then the UCP access was veryyyyyyyy slooooooooow. Bottom line is, even if you run into problems there is good support team to back you up. At least this was my experience.
Some examples:
screenshot 89ad3098-03b3-4efe-8239-0da912003e35 967df4af-bfd7-43b5-82c8-140675e2c3d3

/ Final thoughts /

 The goal of all this effort behind writing these posts is to help you start with DDC and more importantly coding, and to understand new way of app packaging, distributing [shipping] and delivery in the enterprise. The Docker is trying to solve most of this for you. What’s left for you to do is creative thinking. I am big proponent of creative thinking and empowering young minds while solving life and IT problems.
Now, to start coding you did not have to do any of this, but in the enterprise world everything is different. It is different beast altogether. Isn’t it?

Leave a Reply

Your email address will not be published. Required fields are marked *