Indico

• Architecture
  • Nomad job specification
• Packaging Indico
• Deployment
  • Backend tier examples
• The rest of the 🦉
• References and footnotes

In this article, we’ll take a look at designing and deploying Indico as a Nomad workload. Indico is a very widely-used event management tool, and is also a great study in how to model stateful and complex workloads.

Architecture

Indico is a typical 3-tier application: frontend, app, backing services.

• The Frontend serves HTTP requests and consists of an Nginx server and a Redis cache.
• The Application Server runs the actual application, through an application server. In the case of Indico, this is a Python Flask application, and the application server is UWSGI
• Finally the Backing services consist of a (PostgreSQL) database and a Celery event queue.

Architecturally, it looks like this¹:

The user interacts with the web frontend, where a cache handles serving commonly-used content. Authentication is handed off to Flask Multipass which is configured to refer to specified identity providers for authorisation, before being proxied back to the UWSGI server running the application.

The application stores content in a claimed persistent storage, and application and user data in an attached database.

Tasks are scheduled by the application (either via its internal function, or due to user activity or external events) via the Celery worker which may process files, send email, etc.

Nomad job specification

Given this architecture, let’s consider a good Nomad deployment mode.

As usual, let’s consider a few scenarios:

• All-in-one deployment: One Nomad group, everything deployed on the same agent.
• Load-balanced application: The frontend group acts as a load-balancer to the application tier, which can independently scale
• Load-balanced application, elastic work queues: Same as before, but the celery work queue backend instances can independently scale, as tasks are placed on them.

The components of a Nomad job are “groups” of “tasks”. We can initially visualise thus three groups: frontend, application, backing, and each will be allocated to an agent or set thereof depending on the configuration of the group.

Groups are scheduled together on an agent, but we can “spread” a group with a “scaling” definition and an initial “count”.

The first draft of our job specification would look like:

// Indico job definition modelled as 3-tier workload
// Each tier gets a group
job "indico" {
  group "frontend" {
    count = 1
    // serves user requests, exposed to internet. Low resources
    task "nginx" {
       driver = "docker"
       config {
         image = "nginx:stable-alpine"
       }
    }
    task "redis" {
      driver = "docker"
      config {
        image = "redis:8-alpine"
      }
    }
  }

  group "application" {
    count = 1
    // Runs the actual application. High resources, disposable
    task "indico" {
      driver = "docker"
      config {
        image = "ghcr.io/hashi-at-home/indico" # We need to build this
      }
    }
  }

  group "backend" {
    // Handles events and persists data, high reliability and elasticity
    task "celery" {
      count = 1 // let's scale this guy eventually
      driver = "docker"
      # Reuse the same indico image as before, since it has celery
      # configured on it
      config {
        image = "ghcr.io/hashi-at-home/indico"
      }
    }

    task "db" {
      count = 1
      # This one is debatable - deploying the database together with the app
      # May be a recipe for disaster if things die.
      # Better to have an external database in prod
      driver = "docker"
      config {
        image = "postgres:18-alpine"
      }
    }
  }
}

Packaging Indico

Now we have a model that we can implement as a Nomad job. We can see how the application is integrated with its required backing services, and is exposed as web service. However, do not actually have a deployable artifact for the application yet.

We have identified Indico – the Python application served by the UWSGI application server – as the actual workload. In traditional terms, this application would be “installed” into a “machine”, and then “configured” with files or variables.

We are in a 12factor mindset.

We switch mindsets first from the concept of “machine” – there is none, there is only an instance of an application. The orchestrator takes care of the “configuration”, injecting it into the runtime via templates and environment variables. We can start as many of these instances as we like, as long as the configuration for backing services is sane, particularly when it comes to write operations to the database.

We will not realistically be doing this for now, our instance will probably never have to go beyond one concurrent instance except when we are doing blue-green deployments. Packaging the Indico application as a container (or some other self-contained artifact) allows us to pretend that it is just one special entity all by itself in the world, and deal with its externalities outside of it.

See this post for a longer discussion about how I actually build that image.

Deployment

With the job specification and the Indico application all packaged up, we can go a bit deeper into the actual deployment.

Deployment requires injecting configuration into the environment, and thus attaching backing services required for the application to run.

Indico itself respects this convention, allowing configuration to be set by environment variables or configuration files, and is well-documented

Looking at the startup from the application’s point of view, it will expect that the backing services are available². The launch sequence should be something like:

The database server needs to be available, and connections as the indico user to its application database need to be accepted an authorised. The same is true for the connections to the celery broker and the redis cache.

There are no conditions for the frontend though, we can start those (Nginx and Redis) immediately, however their configuration will be subject to change as the backend (UWSGI) becomes available. This is because the Nginx configuration file will be templated based on the Consul service registration of its intended upstream. Initially there will be no service registered, so no upstream will be registered, but as the service comes online, the template will be re-rendered, and Nginx will be automatically restarted. While the application is loading, we will just get a default HTML page, which we can configure.

The traditional procedure would be to define a few systemd units with dependencies between them, but this would require a global context. This also violates Factor 6 – “Processes” which states that all processes should be stateless and share nothing. In our case, we will define a few health checks which will allow the individual services to be registered in the catalogue as healthy before others could depend on it.

The last process to start should therefore be the uwsgi server with Indico itself. This should be done with the typical prestart hook to poll the necessary backing service availability.

Backend tier examples

Let’s take an illustrative look at what this would look like for the backing services tier. Recall that this is composed of two tasks :

• Postgres database
• Celery worker

At deployment time, we cannot make assumptions about the database, but we do know that the other components of the application do rely on credentials and existing databases, schemae etc. Therefore, we need to ensure that the database is properly provisioned. In a disposable environment, we can declare these in the runtime:

task "db" {
  driver = "docker"
  # Get a vault token for this job so that we can look up secrets.
  vault {}

  service {
    # Register the db service in Consul so that others can look it up
    # Port db is declared in the group network above
    port = "db"
  }
  resources {
    cpu = 1
    memory = 1024
  }
  config {
    image = "postgres:18-alpine"
    # expose the mapped db port to other services
    ports = ["db"]
    volumes = [
      "local/init-user-db.sh:/docker-entrypoint-initdb.d/init-user-db.sh"
    ]
  }
  template {
    # We can provision custom user scripts using the initdb.d pattern
    # Described by the packagers
    data =<<EOT
#!/usr/bin/env bash
set -e

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
GRANT ALL PRIVILEGES ON DATABASE indico TO postgres;
CREATE EXTENSION unaccent;
CREATE EXTENSION pg_trgm;
EOSQL
    EOT
    destination = "local/init-user-db.sh"

  }
  template {
    # Lookup the credentials in Vault and inject them into the environment
    data = <<EOT

{{ with secret "hashiatho.m2-v2/payloads/indico" }}
POSTGRES_PASSWORD="{{ .Data.data.db_password }}"
POSTGRES_USER="{{ .Data.data.db_username }}"
POSTGRES_DB="indico"
{{ end }}

    EOT
    destination = "secrets/db.env"
    env = true
  }
}

Once this task starts, we will have a Consul service registered like <job>-<group>-<task> i.e. indico-backend-db. Something similar for the frontend services is also registered – indico-frontend-redis and indico-frontend-nginx. Any tasks or services needing to connect to these can find them via DNS or a Consul lookup – e.g. to get the endpoint for the database we would do a lookup³ of the service:

{{ with range service "indico-backend-db" }}
{{ .Address }}:{{ .Port }}
{{ end }}

So far so good for the database - but that’s a zero-dependency service. The Celery worker actually depends on:

• Email exchange
• Redis cache
• Database

So, we need to inject configuration for these into the runtime:

// Handles events and persists data, high reliability and elasticity
task "celery" {
  constraint {
    attribute = "${attr.cpu.arch}"
    value = "amd64"
  }
  resources {
    cpu = 2
    memory = 4096
  }
  driver = "docker"
  # Reuse the same indico image as before, since it has celery
  # configured on it
  config {
    image = "ghcr.io/hashi-at-home/indico"
    entrypoint = ["/local/start_celery.sh"]
    auth {
      username = "${secret.github.gh_username}"
      password = "${secret.github.ghcr_token}"
    }
  }
  template {
    data = <<EOT
#!/bin/bash
# yes this should probably be in an init 1 process
mise exec -- indico celery worker -B
    EOT
    destination = "/local/start_celery.sh"
    perms = "0777"

  }
  template {
    # Template the celery-specific configuration for indico
    data = <<EOT
# Celery settings

# Lookup the secret and store it in a variable $secretData
{{ with secret "hashiatho.me-v2/payloads/indico" }}
{{ $secretData := Data.data }}
# Lookup the db service.
{{ range service "indico-backend-db" }}
SQLALCHEMY_DATABASE_URI = 'postgresql://${secretData.posgtgres_user}:${secretData.postgres_pass}@{{ .Address }}:{{ .Port }}/indico'
{{ end }} {{/* end service lookup */}}

SECRET_KEY = '${secretData.indico_secret}'
BASE_URL = 'http://0.0.0.0'

{{ range service "indico-frontend-redis" }}
CELERY_BROKER = 'redis://{{ .Address }}:{{ .Port }}/0'
REDIS_CACHE_URL = 'redis://{{ .Address }}:{{ .Port }}/1'
{{ end }} {{/* end service lookup */}}

{{ end }} {{/* end secret lookup context */}}

  EOT
    destination = "local/indico.conf"
    change_mode = "restart"
  }
  env {
    # Tell Indico where to find it's config
    INDICO_CONFIG = "/local/indico.conf"
  }
}

Hey presto, with this we can now deploy all the backing services and wait for the application to come online.

Here’s a short snippet the logs of the result:

Setting the alembic version to HEAD
Creating tables
Creating system user
Creating root category
Creating default ticket template for root category
Creating system oauth apps

 -------------- celery@0b220e3c5a8b v5.6.3 (recovery)
--- ***** -----
-- ******* ---- Linux-6.18.7-76061807-generic-x86_64-with-glibc2.39 2026-09-14 12:53:15
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         indico:0x74297f45aab0
- ** ---------- .> transport:   redis://192.168.1.10:21623/0
- ** ---------- .> results:     redis://192.168.1.10:21623/0
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

Astute readers will have noticed that there are warnings on missing configuration parameters since we have only configured the database and cache endpoints:

/opt/indico/.venv/lib/python3.12/site-packages/indico/core/config.py:253: UserWarning: Required config key NO_REPLY_EMAIL is not configured
  _validate_config(data)
/opt/indico/.venv/lib/python3.12/site-packages/indico/core/config.py:253: UserWarning: Required config key SUPPORT_EMAIL is not configured
  _validate_config(data)
/opt/indico/.venv/lib/python3.12/site-packages/indico/web/flask/app.py:481: UserWarning: Logging config file not found; using defaults. Copy /opt/indico/.venv/lib/python3.12/site-packages/indico/logging.yaml.sample to /local/logging.yaml to get rid of this warning.
  Logger.init(app)
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 06a037da1ec6
/opt/indico/.venv/lib/python3.12/site-packages/indico/core/config.py:253: UserWarning: Required config key NO_REPLY_EMAIL is not configured
  _validate_config(data)
/opt/indico/.venv/lib/python3.12/site-packages/indico/core/config.py:253: UserWarning: Required config key SUPPORT_EMAIL is not configured
  _validate_config(data)
/opt/indico/.venv/lib/python3.12/site-packages/indico/web/flask/app.py:481: UserWarning: Logging config file not found; using defaults. Copy /opt/indico/.venv/lib/python3.12/site-packages/indico/logging.yaml.sample to /local/logging.yaml to get rid of this warning.
  Logger.init(app)
2026-09-14 12:53:17,049  INFO     0000000000000000  -       celery.worker.consumer.connection Connected to redis://192.168.1.10:21623/0
2026-09-14 12:53:17,065  INFO     0000000000000000  -       celery.worker.consumer.mingle mingle: searching for neighbors
2026-09-14 12:53:18,092  INFO     0000000000000000  -       celery.worker.consumer.mingle mingle: all alone
2026-09-14 12:53:18,167  INFO     0000000000000000  -       celery.apps.worker        celery@0b220e3c5a8b ready.
2026-09-14 12:53:18,765  INFO     0000000000000000  -       celery.beat               beat: Starting...
2026-09-14 12:53:18,910  INFO     0000000000000000  -       celery.worker.strategy    Task heartbeat[0887e6ca-0474-4c46-952e-863e79181342] received
2026-09-14 12:53:19,029  INFO     0000000000000000  -       celery.app.trace          Task heartbeat[0887e6ca-0474-4c46-952e-863e79181342] succeeded in 0.11348885600455105s: None

The rest of the 🦉

Nevertheless the worker is up and communicating with the required backing services. We have a bunch more configuration to inject in here - the email, the authorisation, storage, etc. That is what is commonly known as

🖌️💢🦉

For now, we have cracked a nut, so let’s summarise:

1. Indico is a 3 tier app, with commodity bits on the front and back
2. In the middle is a Python application which needs to be packaged
3. We can launch the application scalably by injecting configuration into its runtime using Platform services (Vault, Consul, etc.)

Now with a single job definition indico-standalone.nomad.hcl we can deploy the full Indico stack with separately scaling frontend and backend groups, we can handle rolling updates and database migrations in the same definition, and most importantly entirely separate secrets from the deployment.

Find it soon in production.

References and footnotes

If this token does not have permission to read the services, the lookup will return a 403 or empty. If the service is registered in the Consul DNS, we could also use the DNS entry, indico-backend-db.service.consul but we wouldn’t have access to the port number that the service is exposed on – that would be the job of a router or ALB or something else.