# Deploying FlowFabric on Ubuntu 22.04

Two servers:

| Server | Runs |
|---|---|
| **App server** (Ubuntu 22.04) | Docker: the API, a background-job worker, the web app (nginx). Host nginx in front, later with HTTPS. Uploaded files. |
| **Database server** | PostgreSQL 18: FlowFabric's database and each Workspace's own database. |

```
browser ──80/443──▶ host nginx ──127.0.0.1:8080──▶ web (nginx: app files, /api, /site)
                                                    └──▶ backend (API, uvicorn :8000)
                                                         worker (jobs + scheduler)
                                    backend, worker ──5432──▶ PostgreSQL 18 (DB server)
```

Files used here, all in `deploy/`:

| File | What it is |
|---|---|
| `docker-compose.prod.yml` | the three containers |
| `.env.example` | every setting, to copy to `deploy/.env` |
| `nginx/flowfabric.conf` | the host's nginx site |
| `postgres/create-roles.sql` | the database roles, on the DB server |
| `scripts/update.sh` | pull, build, back up, migrate, restart |
| `scripts/backup-db.sh` | nightly dump of every database (DB server) |
| `scripts/backup-files.sh` | nightly copy of uploads and `.env` (app server) |

Replace these placeholders as you go:

- `APP_IP` — the app server's public IP (what people open in the browser)
- `APP_PRIVATE_IP` — the app server's IP as the DB server sees it
- `DB_HOST` — the DB server's private IP

> Both servers on a private network (same VPC/LAN) is best: the database
> connection isn't encrypted (the migration runner connects without TLS),
> so port 5432 should never be open to the internet.

---

## 0. Before you start: get the code to the server

The server builds from the repository at
`git@gitlab.saasvaap.com:vk/appstudio.git` (branch `main`), so **only
committed and pushed work is deployed**. Commit and push anything you want
on the server first.

The repository is private, so the server needs read access. Add the app
server's SSH public key as a **deploy key** (project → Settings →
Repository → Deploy keys, read-only) — or, to clone over HTTPS, make a
**deploy token** there (Deploy tokens, scope `read_repository`).

---

## 1. Database server: PostgreSQL 18

PostgreSQL 18 isn't in Ubuntu 22.04's own packages; use the PostgreSQL
project's repository (if the DB server isn't Ubuntu/Debian, install PG 18
the way its OS does and skip to 1.2).

### 1.1 Install

```bash
sudo apt update
sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo apt install -y postgresql-18
psql --version
```

### 1.2 Let the app server connect

In `/etc/postgresql/18/main/postgresql.conf`:

```
listen_addresses = 'localhost,DB_HOST'
password_encryption = scram-sha-256
```

At the end of `/etc/postgresql/18/main/pg_hba.conf` — `all` databases and
users, because each Workspace gets its own database and role:

```
host  all  all  APP_PRIVATE_IP/32  scram-sha-256
```

A firewall on the DB server, opening 5432 to the app server only:

```bash
sudo ufw allow OpenSSH
sudo ufw allow from APP_PRIVATE_IP to any port 5432 proto tcp
sudo ufw enable
sudo systemctl restart postgresql
```

### 1.3 Roles and database

Copy `deploy/postgres/create-roles.sql` to the DB server and run it. It asks
for two passwords — make them with `openssl rand -hex 24` (letters and digits
only, so they need no escaping in URLs):

```bash
sudo -u postgres psql -f create-roles.sql
```

It creates:

- **`flowfabric`** — the role the API connects as. An *ordinary* role
  (`NOSUPERUSER NOBYPASSRLS`): tenant tables use `FORCE ROW LEVEL SECURITY`,
  and a superuser would bypass it and see every tenant's rows. It owns the
  `flowfabric` database, so migrations can build the schema.
- **`ff_provisioner`** — a superuser the app uses **only** to create a new
  Workspace's database and role. Not `postgres`, so it can be rotated or
  removed on its own.

Check:

```bash
sudo -u postgres psql -c "SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname IN ('flowfabric','ff_provisioner')"
```

`flowfabric` must show `f | f`.

---

## 2. App server: Docker, nginx, firewall

### 2.1 Docker Engine and Compose (Docker's own repository)

```bash
sudo apt update
sudo apt install -y ca-certificates curl git nginx
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER   # then log out and back in
docker compose version
```

Building the images needs about 2 GB of RAM (the web app's build). On a
smaller server add swap:

```bash
sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```

### 2.2 Firewall

```bash
sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'    # 80 and 443
sudo ufw enable
```

The web container listens on `127.0.0.1:8080` only — Docker's published
ports go around ufw, so nothing is published on a public address.

Check the database is reachable from here:

```bash
nc -zv DB_HOST 5432
```

### 2.3 The code

```bash
sudo mkdir -p /opt/flowfabric && sudo chown $USER: /opt/flowfabric
# SSH, with the server's key added as a deploy key:
git clone git@gitlab.saasvaap.com:vk/appstudio.git /opt/flowfabric
# or HTTPS, with a deploy token:
# git clone https://<token-name>:<token>@gitlab.saasvaap.com/vk/appstudio.git /opt/flowfabric
cd /opt/flowfabric/deploy
```

### 2.4 Persistent uploads directory

Uploaded files live outside the containers, in a host directory owned by
the image's user (uid 10001):

```bash
sudo mkdir -p /srv/flowfabric/uploads
sudo chown -R 10001:10001 /srv/flowfabric/uploads
```

They are stored as `<tenant id>/…` — back this directory up (step 7).

---

## 3. Settings: `deploy/.env`

```bash
cd /opt/flowfabric/deploy
cp .env.example .env
chmod 600 .env
openssl rand -hex 32                      # → JWT_SECRET
docker run --rm python:3.14-slim sh -c "pip -q install cryptography && python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'"
                                          # → WORKSPACE_ENCRYPTION_KEY
nano .env
```

Fill in:

| Setting | Value |
|---|---|
| `DATABASE_URL` | `postgresql+asyncpg://flowfabric:<flowfabric password>@DB_HOST:5432/flowfabric` |
| `PROVISIONING_SUPERUSER` / `_PASSWORD` | `ff_provisioner` and its password |
| `JWT_SECRET` | the hex string above |
| `WORKSPACE_ENCRYPTION_KEY` | the Fernet key above. **Never change it once there is data**, and keep a copy off the server. |
| `APP_BASE_URL` | `http://APP_IP` (links in emails, the public site) |
| `PUBLIC_API_URL` | `http://APP_IP/api` (payment-gateway return links) |
| `CORS_ORIGINS` | `http://APP_IP` |

The API is started with `ENVIRONMENT=production`, which refuses to start
with the built-in development `JWT_SECRET` or `WORKSPACE_ENCRYPTION_KEY`.

Anything else in `.env.example` is optional. A Super Admin can later
override many settings from the UI (Super Admin → Platform settings) —
those are stored in the database and win over `.env`.

---

## 4. Build, create the schema, start

```bash
cd /opt/flowfabric/deploy
docker compose -f docker-compose.prod.yml build
# The empty database gets db/schema.sql (and every migration recorded as applied).
docker compose -f docker-compose.prod.yml run --rm --no-deps backend python -m app.db_migrate deploy
docker compose -f docker-compose.prod.yml up -d
docker compose -f docker-compose.prod.yml ps
```

`backend` should become `healthy`; `worker` and `web` `running`. Check:

```bash
curl -s http://127.0.0.1:8080/api/ready
docker compose -f docker-compose.prod.yml run --rm --no-deps backend python -m app.db_migrate status
```

### The background worker

`backend` only answers requests (`RUN_JOBS=false`). The `worker` container
(`python -m app.worker`) runs the job queue — notifications, emails and
webhooks, index and storage builds, workflow timers and SLA escalations —
and the scheduler. Jobs are rows in Postgres, so a restart loses nothing; a
job whose worker dies is picked up again. Its logs:

```bash
docker compose -f docker-compose.prod.yml logs -f worker
```

For more throughput, run more workers: `docker compose -f docker-compose.prod.yml up -d --scale worker=2`.
Failed jobs show under Scheduled Jobs → Background jobs, where they can be
retried.

---

## 5. Host nginx

```bash
sudo cp /opt/flowfabric/deploy/nginx/flowfabric.conf /etc/nginx/sites-available/flowfabric
sudo ln -s /etc/nginx/sites-available/flowfabric /etc/nginx/sites-enabled/flowfabric
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx
```

Open `http://APP_IP` — the sign-in page should load.

---

## 6. First accounts

A Super Admin (manages tenants, licenses, platform settings):

```bash
cd /opt/flowfabric/deploy
docker compose -f docker-compose.prod.yml exec backend \
  python scripts/create_platform_admin.py --name "Ops Admin" --email you@example.com
```

A tenant with its first Administrator — either from the Super Admin UI, or:

```bash
docker compose -f docker-compose.prod.yml exec backend \
  python scripts/provision_tenant.py --tenant-name "Acme Corp" \
  --admin-name "Alice Admin" --admin-email alice@acme.example --licensed
```

It prints the tenant ID and a generated password. Without a license
nobody in the tenant can sign in (`--licensed`, or the tenant's License tab).

**Creating a Workspace** (its own database): give the database host as
`DB_HOST` and port `5432` — the address the *containers* reach the DB
server on, not `localhost`.

---

## 7. Backups

Three things to keep: **the databases**, **the uploads directory**, and
**`deploy/.env`** (without `JWT_SECRET` and `WORKSPACE_ENCRYPTION_KEY`,
encrypted settings and workspace passwords in a restored database can't
be read).

**DB server** — every database (FlowFabric's and each Workspace's) plus the
roles, nightly at 02:15, kept 14 days:

```bash
sudo install -m 755 backup-db.sh /usr/local/bin/flowfabric-backup-db
echo '15 2 * * * postgres /usr/local/bin/flowfabric-backup-db' | sudo tee /etc/cron.d/flowfabric-backup-db
sudo -u postgres /usr/local/bin/flowfabric-backup-db    # try it once
```

**App server** — uploads and `.env`, nightly at 02:30:

```bash
echo '30 2 * * * root sh /opt/flowfabric/deploy/scripts/backup-files.sh' | sudo tee /etc/cron.d/flowfabric-backup-files
sudo sh /opt/flowfabric/deploy/scripts/backup-files.sh  # try it once
```

Both write to `/var/backups/flowfabric/`. **Copy that off both servers**
(rsync to another machine, rclone to object storage, …) — a backup on the
same disk is lost with it.

**Restoring** the main database (stop the app first):

```bash
docker compose -f docker-compose.prod.yml stop backend worker              # app server
sudo -u postgres pg_restore --clean --if-exists --no-owner --role=flowfabric \
  -d flowfabric /var/backups/flowfabric/db/<stamp>/flowfabric.dump          # DB server
docker compose -f docker-compose.prod.yml start backend worker             # app server
```

On a brand-new DB server, run `psql -f globals.sql` first (roles), then
`createdb -O <owner> <name>` and `pg_restore` for each `.dump`. Test a
restore once on a spare machine — an untested backup is a guess.

---

## 8. Updating

After pushing new commits to `main` on gitlab.saasvaap.com (the server's
`origin`):

```bash
sh /opt/flowfabric/deploy/scripts/update.sh
```

It pulls, builds, dumps the main database to
`/var/backups/flowfabric/pre-update/`, applies pending migrations
(`python -m app.db_migrate deploy`), restarts the containers and waits for
`/ready`. A failing migration stops it before the new code starts; the old
containers keep running.

Note: `db_migrate` updates the **main** database (`DATABASE_URL`).
Workspace databases are built from `db/schema.sql` when created and are not
migrated by it.

Useful commands (from `/opt/flowfabric/deploy`):

```bash
docker compose -f docker-compose.prod.yml logs -f --tail=100 backend
docker compose -f docker-compose.prod.yml run --rm --no-deps backend python -m app.db_migrate status
docker compose -f docker-compose.prod.yml restart backend worker
```

Rolling back code: `git -C /opt/flowfabric checkout <commit>`, then build
and `up -d`. Migrations don't roll back; restore the pre-update dump if a
migration has to be undone.

---

## 9. Later: a domain and HTTPS (Let's Encrypt)

1. Point the domain's DNS **A record** at `APP_IP` and wait until
   `dig +short app.example.com` shows it.
2. In `/etc/nginx/sites-available/flowfabric`, set
   `server_name app.example.com;`, then `sudo nginx -t && sudo systemctl reload nginx`.
3. Get the certificate (certbot edits the site to add HTTPS and the
   HTTP → HTTPS redirect, and renews it automatically):

   ```bash
   sudo snap install --classic certbot
   sudo ln -s /snap/bin/certbot /usr/bin/certbot
   sudo certbot --nginx -d app.example.com
   sudo certbot renew --dry-run
   ```

4. In `deploy/.env`:

   ```
   APP_BASE_URL=https://app.example.com
   PUBLIC_API_URL=https://app.example.com/api
   CORS_ORIGINS=https://app.example.com
   ```

   then `docker compose -f docker-compose.prod.yml up -d` (recreates the
   containers with the new values).

5. If reCAPTCHA, Google Maps or SSO (OIDC) are used, add the domain to
   their allowed domains / redirect URLs.

The host nginx passes `X-Forwarded-Proto: https` through the web container
to the API, so the links it builds use `https://`.

---

## Troubleshooting

| Symptom | Look at |
|---|---|
| `backend` keeps restarting | `docker compose … logs backend`. "ENVIRONMENT=production needs its own JWT_SECRET…" → `.env` secrets not filled in. Connection refused/timeout → `pg_hba.conf`, `listen_addresses`, ufw on the DB server. |
| Sign-in page loads, API calls fail (502) | `docker compose … ps` — is `backend` healthy? `curl http://127.0.0.1:8080/api/ready` |
| `/ready` lists pending migrations | `docker compose … run --rm --no-deps backend python -m app.db_migrate deploy` |
| Upload fails with permission denied | `sudo chown -R 10001:10001 /srv/flowfabric/uploads` |
| Upload fails with 413 | Both nginx configs allow 100 MB (`client_max_body_size`). |
| Emails/webhooks never go out | `docker compose … logs worker`; Scheduled Jobs → Background jobs |
| A tenant sees nothing / everything | The API must connect as `flowfabric`, not a superuser — check `DATABASE_URL` and step 1.3's query. |