Server Configuration at Scale with Ansible

This covers how to structure Ansible inventories, roles, and playbooks so server configuration stays idempotent and reviewable, using an SSH-hardening and Nginx playbook as the running example, plus secrets encrypted with Vault.

Inventory: static hosts plus group_vars/host_vars

Resist the urge to put configuration values directly in playbooks. Inventory tells Ansible which hosts exist and how they're grouped; group_vars and host_vars tell it what's true about each group or host. That separation lets the same playbook run safely against staging and production.

# inventory/production.ini
[web]
web-01.example.internal
web-02.example.internal
 
[db]
db-01.example.internal
 
[web:vars]
nginx_worker_connections=2048
 
[production:children]
web
db

inventory/
├── production.ini
├── staging.ini
├── group_vars/
│   ├── all.yml
│   ├── web.yml
│   └── db/
│       ├── vars.yml
│       └── vault.yml
└── host_vars/
    └── db-01.example.internal.yml

group_vars/all.yml holds values true everywhere (NTP servers, the admin SSH key, log retention). group_vars/web.yml holds values true for every web node (Nginx tuning, the app port). host_vars/<hostname>.yml is for the genuine one-offs. And if that file starts accumulating more than a couple of keys, that's usually a sign the host doesn't belong in its current group.

Role layout: one responsibility per role

A role is the unit of reuse. Each one should do exactly one thing: "harden SSH" or "install and configure Nginx," not "set up a web server," which inevitably grows into an unstructured pile.

roles/
└── nginx/
    ├── tasks/
    │   └── main.yml
    ├── handlers/
    │   └── main.yml
    ├── templates/
    │   └── nginx.conf.j2
    ├── defaults/
    │   └── main.yml
    └── files/
        └── snakeoil-fallback.conf

defaults/main.yml sets sane values that callers can override from group_vars. tasks/main.yml is the actual work. handlers/main.yml holds actions that fire only when something actually changed.

A real playbook: SSH hardening and Nginx

Here's a trimmed but functional example covering both patterns: locking down SSH and deploying Nginx from a template.

# playbooks/site.yml
---
- name: Harden SSH and configure web servers
  hosts: web
  become: true
  roles:
    - ssh_hardening
    - nginx

# roles/ssh_hardening/tasks/main.yml
---
- name: Deploy SSH hardening drop-in
  ansible.builtin.copy:
    dest: /etc/ssh/sshd_config.d/00-hardening.conf
    content: |
      # Managed by Ansible, do not edit by hand
      PasswordAuthentication no
      KbdInteractiveAuthentication no
      PermitRootLogin no
    owner: root
    group: root
    mode: '0644'
    validate: '/usr/sbin/sshd -t -f %s'
  notify: Restart ssh
 
- name: Check the effective sshd configuration
  ansible.builtin.command: /usr/sbin/sshd -T
  register: sshd_effective
  changed_when: false
  when: not ansible_check_mode   # in --check the drop-in isn't written yet
  failed_when: >-
    sshd_effective.rc != 0 or
    'passwordauthentication no' not in sshd_effective.stdout or
    'permitrootlogin no' not in sshd_effective.stdout

# roles/ssh_hardening/handlers/main.yml
---
- name: Restart ssh
  ansible.builtin.service:
    name: ssh   # the unit is "ssh" on Debian/Ubuntu, "sshd" on RHEL-family
    state: restarted

Why a drop-in instead of editing /etc/ssh/sshd_config with lineinfile? On Debian 12 and Ubuntu 22.04+, the main config starts with Include /etc/ssh/sshd_config.d/*.conf, and sshd keeps the first value it reads for each keyword. On Ubuntu cloud images, cloud-init writes /etc/ssh/sshd_config.d/50-cloud-init.conf, and depending on the image and its ssh_pwauth setting that file can contain PasswordAuthentication yes. When it does, a PasswordAuthentication no in the main file is read later and ignored without any warning. Naming the drop-in 00-hardening.conf makes it load first, ahead of the cloud-init file and anything after it. You could also delete the cloud-init file or set ssh_pwauth: false in your cloud-init user-data, but the low-numbered drop-in keeps working whatever else ends up in that directory.

validate runs sshd -t against the new drop-in on its own before putting it in place, which catches a typo in the file. It can't see how the drop-in combines with the rest of the config, so the second task checks the merged configuration with sshd -T and fails the play if the settings didn't take effect. notify queues the handler, and handlers only run at the end of the play, once, however many tasks notified them. So the check always runs before sshd restarts, and if it fails, that host skips the restart (fix the file and rerun before anything else restarts sshd). That matters when the service you're changing is the one you SSH in through.

The Nginx role works the same way, with a Jinja2 template in place of a static file:

# roles/nginx/tasks/main.yml
---
- name: Install nginx
  ansible.builtin.apt:
    name: nginx
    state: present
    update_cache: true
 
- name: Deploy nginx configuration
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: '0644'
    validate: '/usr/sbin/nginx -t -c %s'
  notify: Reload nginx
 
- name: Ensure nginx is running and enabled
  ansible.builtin.service:
    name: nginx
    state: started
    enabled: true

# roles/nginx/defaults/main.yml
---
nginx_worker_connections: 1024
nginx_keepalive_timeout: 65
nginx_document_root: /var/www/html

# roles/nginx/handlers/main.yml
---
- name: Reload nginx
  ansible.builtin.service:
    name: nginx
    state: reloaded

# roles/nginx/templates/nginx.conf.j2
user www-data;
worker_processes auto;
 
events {
    worker_connections {{ nginx_worker_connections }};
}
 
http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
 
    server_tokens off;
    keepalive_timeout {{ nginx_keepalive_timeout }};
 
    server {
        listen 80;
        server_name {{ inventory_hostname }};
        root {{ nginx_document_root }};
 
        location / {
            try_files $uri $uri/ =404;
        }
    }
}

Every variable the template uses has a default in defaults/main.yml, and the nginx_worker_connections=2048 from the inventory overrides it for the web group. Don't drop the include of mime.types. Without it, every file is served with the same default type (text/plain unless you set default_type), and browsers refuse to apply stylesheets and scripts served that way. The path is absolute because validate runs nginx -t against a temporary copy of the file outside /etc/nginx, and nginx resolves relative includes against the directory of the config file it was given. A relative include mime.types; would fail validation.

template renders Jinja2 with the host's variables, compares the result against what's on disk, and reports a change (and fires notify) only when the rendered file actually differs. Run this playbook a second time with nothing changed upstream and every task should report ok, not changed. If it doesn't, something in that role isn't actually idempotent yet.

Secrets: Ansible Vault in group_vars

Database passwords, API keys, and TLS private keys don't belong in plaintext YAML, even in a private repo. Vault solves this by encrypting values in place while keeping them addressable exactly like any other variable.

ansible-vault encrypt_string --stdin-name 'db_password' \
  >> inventory/group_vars/db/vault.yml

--stdin-name reads the value from stdin instead of the command line, so the secret never ends up in your shell history or in ps output. Type the value, then press Ctrl-D twice without pressing Enter first. If you press Enter, the trailing newline becomes part of the encrypted secret.

I keep a plain group_vars/db/vars.yml for non-secret values and a separate group_vars/db/vault.yml for encrypted ones. Ansible merges everything under group_vars/db/* automatically, so a role just references {{ db_password }} and doesn't care which file it came from. Running the playbook then just needs --ask-vault-pass, or --vault-password-file pointed at a file kept out of the repo entirely.

In CI, store the vault password as a masked pipeline secret and never write it to disk. If the file passed to --vault-password-file is executable, Ansible runs it and uses its output as the password. That lets a two-line script committed to the repo read the password from the environment:

#!/usr/bin/env bash
# scripts/vault-pass.sh (commit it with chmod +x; it contains no secret)
printf '%s\n' "${ANSIBLE_VAULT_PASSWORD:?ANSIBLE_VAULT_PASSWORD is not set}"

ansible-playbook -i inventory/production.ini playbooks/site.yml \
  --vault-password-file scripts/vault-pass.sh

If you do need a password file, create it with mktemp under a strict umask and delete it with a trap in the same script, so it's removed even when the playbook fails:

umask 077
vault_pass_file=$(mktemp)
trap 'rm -f "$vault_pass_file"' EXIT
printf '%s\n' "$ANSIBLE_VAULT_PASSWORD" > "$vault_pass_file"
ansible-playbook -i inventory/production.ini playbooks/site.yml \
  --vault-password-file "$vault_pass_file"

Tags and --check for safe rollout

On a fleet of any real size, you rarely want to run the entire playbook every time; you want to reload Nginx configs across the fleet without touching SSH settings, or vice versa. Tags give you that control without splitting the playbook into a dozen files.

# roles/nginx/tasks/main.yml (the same task, now tagged)
- name: Deploy nginx configuration
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: '0644'
    validate: '/usr/sbin/nginx -t -c %s'
  notify: Reload nginx
  tags:
    - nginx
    - config

# only the nginx-tagged tasks
ansible-playbook -i inventory/production.ini playbooks/site.yml --tags nginx
 
# preview every change, apply nothing
ansible-playbook -i inventory/production.ini playbooks/site.yml --check --diff

--check runs the playbook without making changes and reports what would happen; --diff adds the actual before/after text for anything template or copy would rewrite. I run every playbook with --check --diff against production first, until running it for real becomes routine for a client's team.

Migrating from ad hoc scripts to roles is a decision that's cheaper to make before the fleet hits fifty hosts than after.