Security Hardening Guide

This document describes the security measures implemented for the Geetanjali application infrastructure.

Table of Contents

  1. Server-Level Security
  2. Docker Container Security
  3. Network Security
  4. Application Security
  5. Secrets Management (SOPS + age)
  6. Security Checklist
  7. Incident Response

Server-Level Security

SSH Hardening

SSH is configured for key-based authentication only:

# /etc/ssh/sshd_config settings
PasswordAuthentication no
PermitRootLogin prohibit-password
PubkeyAuthentication yes

Best practices:

Firewall (UFW)

UFW is configured to allow only essential ports:

# View current rules
sudo ufw status verbose

# Expected configuration:
# 22/tcp    - SSH
# 80/tcp    - HTTP (redirects to HTTPS)
# 443/tcp   - HTTPS

All other ports are blocked. Docker services communicate internally via Docker network.

Fail2ban

Fail2ban protects against brute-force attacks:

# Configuration: /etc/fail2ban/jail.local
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 24h
findtime = 10m

Commands:

# Check status
sudo fail2ban-client status sshd

# View banned IPs
sudo fail2ban-client status sshd | grep "Banned IP"

# Unban an IP
sudo fail2ban-client set sshd unbanip <IP>

Audit Logging (auditd)

Auditd provides kernel-level audit logging:

# Check status
sudo systemctl status auditd

# View audit logs
sudo ausearch -ts today

# View login attempts
sudo ausearch -m USER_LOGIN

Docker Container Security

Linux Capabilities

Linux capabilities break down root privileges into ~40 distinct powers. We drop all capabilities except those explicitly needed:

Capability Description Containers Using
NET_BIND_SERVICE Bind to ports < 1024 Frontend (nginx)
CHOWN Change file ownership Redis, Frontend
SETUID Set user ID Redis, Frontend
SETGID Set group ID Redis, Frontend

Configuration in docker-compose.yml:

services:
  redis:
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true

Container-Specific Hardening

Container Runs As Capabilities PID Limit Extra Hardening
postgres postgres (non-root) cap_drop: ALL + minimal cap_add - no-new-privileges
redis redis (uid 999) cap_drop: ALL 64 read-only FS, tmpfs /data, no-new-privileges
chromadb chromauser (uid 1000) cap_drop: ALL 128 no-new-privileges
backend appuser (uid 1000) cap_drop: ALL 256 no-new-privileges
worker appuser (uid 1000) cap_drop: ALL 128 no-new-privileges, internal-only network
frontend nginx cap_drop: ALL + NET_BIND_SERVICE, SETUID, SETGID, CHOWN 64 no-new-privileges
ollama root (required) cap_drop: ALL + minimal cap_add - no-new-privileges, internal-only
prometheus nobody (uid 65534) cap_drop: ALL 64 no-new-privileges
grafana grafana (uid 472) cap_drop: ALL 64 no-new-privileges

no-new-privileges

The no-new-privileges security option prevents processes from gaining additional privileges through setuid binaries. This means even if an attacker exploits a vulnerability, they cannot escalate privileges.

Read-Only Filesystems

Redis runs with a read-only root filesystem (read_only: true), with only a tmpfs mount for /tmp. This prevents attackers from writing malicious files even if they gain access.

Docker Log Rotation

Docker is configured to prevent log files from consuming disk space:

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Network Security

Internal Docker Network

All services communicate via an internal Docker bridge network (geetanjali-network). External ports are exposed only for:

No external port exposure for:

Redis Authentication

Redis requires password authentication:

# Connection string format
redis://:${REDIS_PASSWORD}@redis:6379/0

Local Development

For local development, use docker-compose.override.yml to expose ports to localhost only:

services:
  postgres:
    ports:
      - "127.0.0.1:5432:5432"

Important: Never commit docker-compose.override.yml (it’s in .gitignore).

Application Security

Environment Variables

Sensitive values are stored in .env (git-ignored):

API Security

In production, enable secure cookies:

COOKIE_SECURE=true  # Requires HTTPS

Secrets Management (SOPS + age)

Production secrets are encrypted using SOPS with age encryption. This allows secrets to be safely committed to git while remaining encrypted.

How It Works

  1. .env.enc - Encrypted secrets file (safe to commit to git)
  2. .sops.yaml - SOPS configuration with public key
  3. ~/.config/sops/age/keys.txt - Private key (never commit!)

The deploy script automatically decrypts .env.enc to .env on the server during deployment.

Quick Reference

# View decrypted secrets (for debugging)
make secrets-view

# Edit encrypted secrets (opens in vim)
make secrets-edit

# Re-encrypt after editing .env.prod.backup
make secrets-encrypt

Editing Secrets

Option 1: Direct Edit (Recommended)

make secrets-edit
# Opens encrypted file in vim, saves encrypted

Option 2: Manual Workflow

# 1. Decrypt to temporary file
sops --decrypt --input-type dotenv --output-type dotenv .env.enc > .env.tmp

# 2. Edit the file
vim .env.tmp

# 3. Re-encrypt
sops --encrypt --input-type dotenv --output-type dotenv --output .env.enc .env.tmp

# 4. Clean up and commit
rm .env.tmp
git add .env.enc && git commit -m "chore: update secrets"

Key Locations

Location Purpose
Local: ~/.config/sops/age/keys.txt Private key for encryption/decryption
Server: ~/.config/sops/age/keys.txt Private key for decryption
Repo: .env.enc Encrypted secrets (safe to commit)
Repo: .sops.yaml SOPS config with public key

Emergency: Lost Private Key

If the private key is lost, you’ll need to:

  1. Generate a new key pair: age-keygen -o ~/.config/sops/age/keys.txt
  2. Update .sops.yaml with the new public key
  3. Re-encrypt all secrets from the plaintext backup
  4. Copy new private key to server

Important: Keep a secure backup of the private key outside of git!

Adding a New Secret

  1. Run make secrets-edit
  2. Add the new key=value line
  3. Save and quit (:wq)
  4. Commit: git add .env.enc && git commit -m "chore: add NEW_SECRET"
  5. Deploy: make deploy

Security Checklist

Pre-Deployment

Server Setup

Docker Deployment

Incident Response

If You Suspect a Breach

  1. Isolate: Take the affected service offline 
    docker compose stop <service>
  2. Preserve Evidence: Copy logs before making changes 
    docker logs geetanjali-<service> > /tmp/<service>-logs.txt 2>&1
  3. Check Audit Logs: 
    sudo ausearch -ts <timestamp>
sudo journalctl -u docker --since "1 hour ago"
  4. Rotate Credentials: Change all secrets
    • Database password
    • Redis password
    • JWT secret
    • API keys
  5. Review: Check fail2ban logs for attack patterns 
    sudo fail2ban-client status sshd

Recreating Containers with Fresh State

If data may be compromised:

# Stop and remove container with its volumes
docker compose down <service>
docker volume rm geetanjali_<volume>

# Recreate
docker compose up -d <service>

Updating Security Measures

System Updates

# Update packages (run periodically)
sudo apt update && sudo apt upgrade -y

# Restart services if needed
sudo systemctl restart fail2ban
sudo systemctl restart auditd

Docker Updates

# Rebuild containers with updated base images
docker compose build --no-cache
docker compose up -d

Backup and Restore Procedures

Automated Backups

PostgreSQL backups are automated via cron and stored locally:

# Backup location
/var/backups/geetanjali/

# Backup schedule (configured in server crontab)
# Daily at 2 AM: Full database backup
# Weekly on Sunday: Full backup with 4-week retention

Manual Backup

# Create immediate backup
docker exec geetanjali-postgres pg_dump -U geetanjali geetanjali > backup_$(date +%Y%m%d_%H%M%S).sql

# Backup with compression
docker exec geetanjali-postgres pg_dump -U geetanjali geetanjali | gzip > backup_$(date +%Y%m%d).sql.gz

Restore Procedures

⚠️ Warning: Restoring will overwrite all current data. Always verify the backup file first.

Step 1: Verify Backup File

# Check file size and header
ls -lh backup_file.sql
head -50 backup_file.sql

# For compressed backups
zcat backup_file.sql.gz | head -50

Step 2: Stop Application Services

# Stop services that write to the database
docker compose stop backend worker

Step 3: Restore Database

# Drop and recreate database
docker exec -it geetanjali-postgres psql -U geetanjali -c "DROP DATABASE IF EXISTS geetanjali;"
docker exec -it geetanjali-postgres psql -U geetanjali -c "CREATE DATABASE geetanjali;"

# Restore from backup
cat backup_file.sql | docker exec -i geetanjali-postgres psql -U geetanjali geetanjali

# For compressed backups
zcat backup_file.sql.gz | docker exec -i geetanjali-postgres psql -U geetanjali geetanjali

Step 4: Restart Services

docker compose up -d backend worker

Step 5: Verify Restore

# Check database health
docker exec geetanjali-backend curl -s http://localhost:8000/health

# Verify record counts
docker exec geetanjali-postgres psql -U geetanjali -c "SELECT COUNT(*) FROM cases;"
docker exec geetanjali-postgres psql -U geetanjali -c "SELECT COUNT(*) FROM users;"

Vector Store (ChromaDB) Backup

ChromaDB data is stored in a Docker volume and persists automatically:

# Backup ChromaDB volume
docker run --rm -v geetanjali_chroma_data:/data -v $(pwd):/backup alpine tar czf /backup/chroma_backup.tar.gz /data

# Restore ChromaDB volume
docker compose stop chromadb
docker run --rm -v geetanjali_chroma_data:/data -v $(pwd):/backup alpine sh -c "rm -rf /data/* && tar xzf /backup/chroma_backup.tar.gz -C /"
docker compose up -d chromadb

Backup Checklist

References