code spot

NGINX as a Load Balancer using Ansible

This article is a straightforward demonstration of how we can easily setup or machines using Ansible. For this demonstration i used three machines – three raspberries. I deployed two dockerized microservices in asp net core on two of those machines and them set up NGINX as a load balancer on a third Raspberry Pi to serve as a frontend and balance the load across the other two machines.

What i used :

  • My laptop with windows
  • Three Raspberry Pis (i used one of each of those versions : 3,4,5)
  • Machine with Linux. Specifically Ubuntu 22.04.1 LTS Jammy Jellyfish)
  • Ansible 9.8.0
  • Python 3.12.3

The steps :

  • Setup ssh with public key, from you latop to the hosts
  • Check pyton version in all your hosts
  • Ansible installation
  • Ansible setup
    • Declare the hosts
    • Define the ssh private key in ansible
    • Ping your hosts
  • Create Ansible Playbook to setup docker
  • Create Ansible Playbook to spin microservice containers
  • Create Ansible Playbook to deploy NGINX as Load Balancer
  • Create Ansible Playbook to deploy NGINX as Docker container

Setup ssh using key from your machine to the hosts

Eventhough this step is kind of self-explanatory 🙂 i will leave the config file with my ssh configuration :

# config file from my personal laptop 

Host linux-server
Hostname 192.168...
User joaoc
IdentityFile id_ed...
ServerAliveInterval 60
ServerAliveCountMax 120

Host raspberry-3
Hostname 192.168...
User joaoc
IdentityFile id_ed...
ServerAliveInterval 60
ServerAliveCountMax 120

# the same for the request of the raspberries

Check pyton version in all your hosts

Ansible is pretty popular because is agentless and does not require the instalation of extra software on the managed nodes or hosts it controls. On the other hand ansible needs pyton to be present on those hosts to execute commands.

About this mind is important to make sure we have pyton and that the versions are aligned in all the managed hosts but also in the control node. However when you install ansible in the control node, python is installed if not present. Apart from that modern distributions of linux have already python installed by default. 

# ssh into my control node 
> ssh linux-server

> python3 --version
Python 3.12.3

# ssh into one of the raspberries 
> ssh raspberry-3

> python3 --version
Python 3.12.3

# do the same for the other raspberries/managed hosts...

Ansible Installation

The installation instructions will be different depending on the linux distribution. In my case, i using ubuntu :

> sudo apt install software-properties-common

> sudo apt-add-repository ppa:ansible/ansible

> sudo apt update 

> sudo apt install ansible

> ansible -version

Ansible Setup

The first thing to do is to declare our managed hosts.

> cd /etc/ansible

> sudo nano hosts
##...other stuff above

## [openSUSE]
## green.example.com
## blue.example.com

[webservers]
web1 ansible_host=192.168.50.101 ansible_user=joaoc
web2 ansible_host=192.168.50.51 ansible_user=joaoc
web3 ansible_host=192.168.50.217 ansible_user=joaoc

After this step, and before process to ping our servers to see if it works, is good idea to “tell” ansible where the private key used for ssh is. We can do this with following commands : 

# command to define where the ssh private key is for Ansible
> ansible-playbook -i inventory playbook.yml --private-key ~/.ssh/id_rsa

# after that command we can verify the change in ansible.cfg
> cat /etc/ansible/ansible.cfg

Now, we can try to ping our managed-hosts, to see if it works. All necessary is one command.

# command 
>/etc/ansible$ ansible all -m ping


raspberry5 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
raspberry4 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
raspberry3 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

Create Ansible Playbook to setup docker

Now, let’s create a ansible playbook to setup docker in our managed hosts.

> sudo nano /etc/ansible/playbooks/install_docker.yaml
# from "Continuous Delivery with Docker and Jenkins"
# ubuntu 20.04

- hosts: webservers
  become: yes
  become_method: sudo
  tasks:
  - name: Install required packages
    apt:
      name: "{{ item }}"
      state: latest
      update_cache: yes
    loop:
    - apt-transport-https
    - ca-certificates
    - curl
    - software-properties-common
    - python3-pip
    - virtualenv
    - python3-setuptools
    - python3-docker
  - name: Add Docker GPG apt Key
    apt_key:
      url: https://download.docker.com/linux/ubuntu/gpg
      state: present
  - name: Add Docker Repository
    apt_repository:
      repo: deb https://download.docker.com/linux/ubuntu focal stable
      state: present
  - name: Update apt and install docker-ce
    apt:
      name: docker-ce
      state: latest
      update_cache: yes
  - name: Install Docker Module for Python
    pip:
      name: docker
> ansible-playbook /etc/ansible/playbooks/install_docker.yaml

After running our playbook the output should be something like this : 

PLAY [webservers] 
//....

TASK [Gathering Facts] 
//...
ok: [raspberry5]
ok: [raspberry4]
ok: [raspberry3]

TASK [Install required packages] ********************************************************************************************************************************************************************************
ok: [raspberry5] => (item=apt-transport-https)
ok: [raspberry5] => (item=ca-certificates)
ok: [raspberry4] => (item=apt-transport-https)
//...

TASK [Add Docker GPG apt Key] 
//...

TASK [Add Docker Repository] 
//...

TASK [Update apt and install docker-ce]
//...

TASK [Create a virtual environment] 
//...

TASK [Install Docker Module for Python in virtualenv] 
//...

PLAY RECAP ******************************************************************************************************************************************************************************************************
raspberry3                 : ok=7    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
raspberry4                 : ok=7    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
raspberry5                 : ok=7    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

After that for tests, we can ssh into one of the managed hosts and spin a docker image :

> ssh raspberry-3

# check that docker is installed
> docker --version

# i have in my docker hub a simple image of the typical 
# asp net core weather api 
> docker run -it joaocampos07/weather-microservice:v3-net8-arm64

Output : 

Unable to find image 'joaocampos07/weather-microservice:v3-net8-arm64' locally
v3-net8-arm64: Pulling from joaocampos07/weather-microservice
aa6fbc30c84e: Already exists
41f249b81fe4: Already exists
7d82e16437b1: Already exists
ff2a57289b80: Already exists
e3705ce62d26: Already exists
db9e3d0b6276: Already exists
87e0e2bfe42c: Already exists
4f4fb700ef54: Already exists
d9eb19e32dff: Already exists
b26809020a55: Pull complete
Digest: sha256:1a6be4d604a7222ba811bac17cdd3e1f8071d37e35a5e2e4a0b0f5053b902f55
Status: Downloaded newer image for joaocampos07/weather-microservice:v3-net8-arm64
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:8080

Them we can hit the weather endpoint from the master node:

> curl -X GET "http://raspberry-3:8080/weatherforecast"

Create Ansible Playbook to spin weather microservice containers

Next, let’s automatize this, we don’t want to ssh into all our machines, one by one, and spin our microservice docker image. So, we need to create a ansible playbook to automate this . First, we need to install the ansible module for docker to them we create playbook.

# install ansible docker module
> ansible-galaxy collection install community.docker

# create playbook
> sudo nano /etc/ansible/playbooks/install_docker.yaml
---
- name: Deploy Docker Container on Multiple Hosts
  hosts: all
  become: yes  # This ensures that commands needing root access can run
  vars:
    docker_username: "[put here your username]"
    docker_password: "[put here your password]"
    image_name: "joaocampos07/weather-microservice:v3-net8-arm64"
    container_name: "weather-microservice"
    container_ports:
      - "8080:8080" # Adjust as necessary for your container's port mapping

  tasks:

    - name: Log in to Docker Hub
      community.docker.docker_login:
        username: "{{ docker_username }}"
        password: "{{ docker_password }}"
      register: login_result

    - name: Pull Docker image from Docker Hub
      community.docker.docker_image:
        name: "{{ image_name }}"
        source: pull
      when: login_result.changed

    - name: Stop and remove existing container if exists
      community.docker.docker_container:
        name: "{{ container_name }}"
        state: absent
        force_kill: yes
        keep_volumes: no

    - name: Run Docker container
      community.docker.docker_container:
        name: "{{ container_name }}"
        image: "{{ image_name }}"
        state: started
        restart_policy: always
        ports: "{{ container_ports }}"

In this ansible playbook we also install docker if is not present in the managed host, and only after that we spin the container. The output should look something like this 

> ansible-playbook /etc/ansible/playbooks/weather_microservice_playbook.yaml

## output
PLAY [Deploy Docker Container on Multiple Hosts] ****************************************************************************************************************************************************************

TASK [Gathering Facts] 
//...

TASK [Log in to Docker Hub] 
//...

TASK [Pull Docker image from Docker Hub] 
//...

TASK [Stop and remove existing container if exists] //...

TASK [Run Docker container] //...

PLAY RECAP ******************************************************************************************************************************************************************************************************
raspberry3                 : ok=4    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
raspberry4                 : ok=4    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
raspberry5                 : ok=4    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Let’s test this 

master-node> curl -X GET "http://raspberry-4:8080/weatherforecast"

master-node> curl -X GET "http://raspberry-5:8080/weatherforecast"

Create Ansible Playbook to deploy NGINX as Load Balancer

Finally, we need a load balancer. We will create a ansible playbook to deploy nginx and set it as a load balancer. The load balancer will be in raspberry-3. However, in order to deploy nginx with some custom configuration we need to create a config file, that will be reference in the ansible playbook.

> sudo nano /etc/ansible/playbooks/templates/load_balancer.conf.j2
# Define a custom log format with upstream information
log_format upstreamlog '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '[upstream: $upstream_addr] [upstream_status: $upstream_status]';

# Define the location for access logs using the custom format
access_log /var/log/nginx/access.log upstreamlog;


upstream backend_servers {
    server 192.168.50.51:8080;
    server 192.168.50.217:8080;
}

server {
    listen 80;

    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Now we are ready to create the ansible playbook.

> sudo nano /etc/ansible/playbooks/install_nginx.yaml
- name: Ensure Nginx is installed and configured as a load balancer
  hosts: raspberry3
  become: yes

  tasks:
    - name: Install Nginx
      apt:
        name: nginx
        state: present
        update_cache: yes

    - name: Start and enable Nginx service
      service:
        name: nginx
        state: started
        enabled: yes

    - name: Modify default Nginx site configuration for load balancing
      template:
        src: templates/load_balancer.conf.j2 # this is out template to configure nginx :)
        dest: /etc/nginx/sites-available/default
        owner: root
        group: root
        mode: '0644'
      notify:
        - Reload Nginx

  handlers:
    - name: Reload Nginx
      service:
        name: nginx
        state: reloaded
> ansible-playbook /etc/ansible/playbooks/install_nginx.yaml
PLAY [Ensure Nginx is installed and configured as a load balancer] **********************************************************************************************************************************************

TASK [Gathering Facts] 
//...

TASK [Install Nginx] 
//...

TASK [Start and enable Nginx service] 
//...

TASK [Modify default Nginx site configuration for load balancing] ***********************************************************************************************************************************************
changed: [raspberry3]
//...

RUNNING HANDLER [Reload Nginx] 
//...

PLAY RECAP ******************************************************************************************************************************************************************************************************
raspberry3                 : ok=5    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Now, let’s hit nginx from the master node with http requests and check on nginx logs if the requests are being splited between the 3 machines :

> curl -X GET "http://raspberry-3/weatherforecast"
> curl -X GET "http://raspberry-3/weatherforecast"
> curl -X GET "http://raspberry-3/weatherforecast"
//...

Let’s check nginx logs.

> ssh raspberry-3

# check nginx logs
> sudo tail -f /var/log/nginx/access.log

192.168.50.220 - - [16/Aug/2024:09:58:29 +0200] "GET /weatherforecast HTTP/1.1" 200 402 "-" "curl/8.7.1" "-" [upstream: raspberry-5:8080] [upstream_status: 200]

//...

192.168.50.220 - - [16/Aug/2024:09:58:33 +0200] "GET /weatherforecast HTTP/1.1" 200 409 "-" "curl/8.7.1" "-" [upstream: raspberry-4:8080] [upstream_status: 200]

Create Ansible Playbook to deploy NGINX as Docker container

To put nginx as a docker container with our configuration is a bit more tricky. Let’s have a look at the propose ansible playbook 

- name: Ensure Nginx Docker container is running and configured as a load balancer
  hosts: raspberry3
  become: yes

  tasks:
    - name: Ensure Docker service is started and enabled
      service:
        name: docker
        state: started
        enabled: yes

    - name: Create Nginx configuration directory on host
      file:
        path: /home/pi/nginx_conf
        state: directory
        owner: joaoc
        group: joaoc
        mode: '0755'

    - name: Copy custom Nginx configuration to the host
      template:
        src: templates/load_balancer.conf.j2
        dest: /home/pi/nginx_conf/default.conf # copy my nginx config to the host
        owner: joaoc
        group: joaoc
        mode: '0644'

    - name: Run Nginx container with custom configuration
      docker_container:
        name: nginx_load_balancer
        image: nginx
        platform: linux/arm64 # notice here, we need the nginx image for linux/arm64
        state: started
        restart_policy: always
        volumes:
          - /home/pi/nginx_conf/default.conf:/etc/nginx/conf.d/default.conf # map volume using previous directory here our config is
        ports:
          - "80:80"

    - name: Ensure Nginx container is running
      docker_container:
        name: nginx_load_balancer
        state: started
# run the playbook
> $ ansible-playbook /etc/ansible/playbooks/install_nginx_docker.yaml

Them, first, we can go to the raspberry-3 machine and check if the docker container is running and everthing looks okay :

master-node> ssh raspberry-3

# avoid the anoying permissons ask
raspberry-3> sudo -i

## check the docker containers
raspberry-3(root)> docker ps

CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                NAMES
845b23b369d2   nginx     "/docker-entrypoint.…"   38 minutes ago   Up 26 minutes   0.0.0.0:80->80/tcp   nginx_load_balancer

## go into the container 
> docker exec -it nginx_load_balancer /bin/bash

# inside container now...

## checking the main configuration file for Nginx
## we should have the nginx.conf default file
## like we had on our load_balancer.conf.j2 file.


nginx-container > cat /etc/nginx/nginx.conf
nginx-container > cat /etc/nginx/conf.d/default.conf

## confirm that nginx configuration is okay
> nginx -t 

## exit the container 
> exit


## them we can check the logs live as we are going to test and we want to see if the load balacing is working 
> docker logs --tail 100 -f nginx_load_balancer

We can open another terminal in the master node and test :

# terminal 2
> curl -X GET "http://192.168.50.101/weatherforecast"

> curl -X GET "http://192.168.50.101/weatherforecast"

> curl -X GET "http://192.168.50.101/weatherforecast"

> curl -X GET "http://192.168.50.101/weatherforecast"

Checking the other terminal 

## we should see something like that 
> docker logs --tail 100 -f nginx_load_balancer

## (...)
192.168.50.220 - - [17/Aug/2024:10:32:24 +0000] "GET /weatherforecast HTTP/1.1" 200 397 "-" "curl/8.7.1" "-" [upstream: 192.168.50.217:8080] [upstream_status: 200]

192.168.50.220 - - [17/Aug/2024:10:32:25 +0000] "GET /weatherforecast HTTP/1.1" 200 399 "-" "curl/8.7.1" "-"

192.168.50.220 - - [17/Aug/2024:10:32:25 +0000] "GET /weatherforecast HTTP/1.1" 200 399 "-" "curl/8.7.1" "-" [upstream: 192.168.50.51:8080] [upstream_status: 200]

192.168.50.220 - - [17/Aug/2024:10:32:25 +0000] "GET /weatherforecast HTTP/1.1" 200 399 "-" "curl/8.7.1" "-"

192.168.50.220 - - [17/Aug/2024:10:32:25 +0000] "GET /weatherforecast HTTP/1.1" 200 399 "-" "curl/8.7.1" "-" [upstream: 192.168.50.217:8080] [upstream_status: 200]

Leave a Reply

Your email address will not be published. Required fields are marked *