High availability with app nodes

How it works

High availability lets the production of one Odoo database run on several servers at once. If one server stops answering, visitors are sent to the others.

A cluster is made of projects that run the same code:

• The primary: a project whose production instance runs Odoo and the database. Its database is opened to the app nodes, and to no one else.
• App nodes: other projects, each on a different production server, whose production instance runs Odoo only and uses the primary's database.

Visitors use the primary's address. CICDoo points that address at the primary and at every app node that is answering, and checks each node every few minutes. A node that stops answering is taken out of the address until it recovers.

The database itself still runs on one server, the primary's. App nodes make the Odoo side redundant; if the primary's server goes down, the site goes down with it.

Before you start

• Same code: every project in a cluster must use the same repository, default branch, Odoo version and edition. If Time Machine is on, the pinned commits must match too.
• Different servers: each app node must have its own production server, different from the primary's and from the other nodes'.
• Cloudflare proxy: spreading traffic across the nodes currently works for servers that have the Cloudflare proxy enabled. Use servers in the same region, since every page an app node serves talks to the primary's database.
• Shared filestore: every server in the cluster must mount the same shared storage for the Odoo filestore. Without it, an attachment uploaded through one node is missing on the others, and a visitor who lands on another node is logged out. Ask support if you need help setting this up.
• Database connections: each app node opens its own connections to the primary's database. As you add nodes, raise max_connections in the PostgreSQL Configuration of the primary's production instance, in the Settings tab of its console.

High availability settings are in Project Settings, on the Advanced tab, in the High Availability card. You need permission to change the project's settings.

Setting up the primary

Open the project that will run the database, go to Project Settings > Advanced, and in High Availability set Role to Primary (app + database). Click Apply and confirm.

The production instance restarts. When it comes back, its database accepts connections from the cluster's app nodes. Connections between servers are encrypted unless your servers share a private network.

The production instance must use its own bundled database. A project whose production instance points at an external database cannot be a primary.

Adding an app node

Create a project for the node as you normally would: same repository and default branch, same version and edition, and a production server of its own. Remove any staging and development instances it has: an app node only runs production, and staging and development branches live on the primary project.

Then open the node's Project Settings > Advanced, set Role to App node (uses a primary's database), pick the primary in Database source, click Apply and confirm. Only primaries in the same workspace that run the same code are offered.

The node does not start right away. The primary's server first has to let it in, which happens on its next check, usually within five minutes. The card shows "Waiting for the primary to allow this node" until then, and the node starts on its own as soon as it is allowed. A few minutes after it answers, it starts receiving visitors.

On the primary, the High Availability card lists its app nodes and whether the database server has applied the current access list.

What changes on an app node

• Its production instance has no database of its own. Its database settings are managed by CICDoo and cannot be changed in the instance's Odoo configuration.
• Scheduled jobs (Odoo crons) run on the primary only.
• Backups are taken on the primary. Back up and restore the cluster's database from the primary project.
• Pushing to an app node's branch does nothing on its own. The cluster is updated from the primary.

Deploying updates

Push to the primary's branch as usual. CICDoo then updates the whole cluster in order:

1. Each app node is stopped.
2. The primary is updated, including any module upgrade, while no other server is using the database.
3. Each app node is started again with the new code.

You can follow each step in the Queue tab of the instance it runs on. If the primary's update fails, the app nodes are not started again, and their queue entries say why. Fix the problem and deploy again.

Because module upgrades need the database to themselves, the site is unavailable while the primary updates, as it is for a single instance.

Leaving a cluster

To remove an app node, set its Role back to Standalone (own database) and click Apply. Its production instance stops, because its own database does not hold the cluster's data. Restore a backup into it before starting it again.

A primary cannot be changed while it still has app nodes. Remove the nodes first.

Troubleshooting

• The node keeps waiting to be allowed: check that the primary's production instance is running and that its server is online. The primary has to restart once after becoming a primary before any node is allowed.
• The node starts but Odoo does not come up: open the node's logs. A message saying the primary's database is not initialized means the primary has not finished its first start.
• Uploads or logins do not follow visitors between nodes: the servers are not sharing the filestore. See "Before you start" above.

See Branches and instances for how instances map to branches, and Using a load balancer for the server-level load balancer setting, which is a different feature.