gniza4cp/CLAUDE.md

# agents.md — gniza Development Guide

> Reference for AI coding agents working on gniza. Describes architecture, conventions, and key patterns.

## Project Overview

gniza is a Bash CLI tool for cPanel server backup and disaster recovery. It runs `pkgacct` to export accounts, gzips SQL files, and transfers everything to one or more remote destinations using hardlink-based incremental snapshots. Supports three remote types: **SSH** (rsync with `--link-dest`), **Amazon S3** / S3-compatible (via rclone), and **Google Drive** (via rclone).

**Language:** Bash (bash 4+, `set -euo pipefail`)
**Target environment:** CentOS/AlmaLinux cPanel servers, running as root
**Install path:** `/usr/local/gniza/` with symlink at `/usr/local/bin/gniza`

## Repository Structure

```
bin/gniza                  # CLI entrypoint — command routing, argument parsing
lib/
├── constants.sh           # Version, exit codes, color codes, default values
├── utils.sh               # die(), require_root(), timestamp(), human_size/duration()
├── logging.sh             # Per-run log files (LOG_FILE), log_info/warn/error/debug
├── config.sh              # load_config(), validate_config()
├── locking.sh             # flock-based acquire_lock(), release_lock()
├── ssh.sh                 # build_ssh_opts(), remote_exec(), test_ssh_connection()
├── rclone.sh              # Rclone transport layer for S3/GDrive remotes
├── accounts.sh            # get_all_accounts(), filter_accounts(), account_exists()
├── pkgacct.sh             # run_pkgacct(), gzip_sql_files(), cleanup_pkgacct()
├── snapshot.sh            # get_snapshot_dir(), list_remote_snapshots(), finalize
├── transfer.sh            # rsync_to_remote(), transfer_pkgacct/homedir(), finalize_snapshot()
├── retention.sh           # enforce_retention() — prune old snapshots
├── verify.sh              # verify_account_backup(), verify_all_accounts()
├── notify.sh              # send_notification(), send_backup_report()
├── restore.sh             # restore_full_account/files/database/mailbox/server()
├── remotes.sh             # Multi-remote: list_remotes(), load_remote(), get_target_remotes()
└── schedule.sh            # Cron: decoupled schedules from schedules.d/
etc/
├── gniza.conf.example     # Main config template
├── remote.conf.example    # Remote destination config template
└── schedule.conf.example  # Schedule config template
scripts/
├── install.sh             # Install to /usr/local/gniza, create dirs/symlinks
└── uninstall.sh           # Remove install dir, symlink, cron entries, WHM plugin
tests/
└── test_utils.sh          # Unit tests for utils.sh, accounts.sh, config.sh
whm/
├── gniza-whm.conf         # WHM AppConfig registration
└── gniza-whm/
    ├── index.cgi          # Dashboard — overview, quick links, auto-redirect if unconfigured
    ├── setup.cgi          # 3-step setup wizard (SSH key → remote → schedule)
    ├── settings.cgi       # Main config editor (local settings only)
    ├── remotes.cgi        # Remote CRUD — add/edit/delete, SSH key guidance on add
    ├── schedules.cgi      # Schedule CRUD — add/edit/delete with remote checkboxes
    ├── restore.cgi        # Restore workflow — 4-step form (account → snapshot → confirm → execute)
    ├── assets/
    │   ├── gniza-whm.css          # Built Tailwind/DaisyUI CSS (committed, ~58KB)
    │   ├── gniza-logo.svg         # SVG logo (embedded as data URI in page header)
    │   └── src/
    │       ├── input.css           # Tailwind v4 entry point with DaisyUI plugin
    │       ├── safelist.html       # Class safelist for Tailwind content scanner
    │       └── package.json        # Build toolchain (tailwindcss + daisyui)
    └── lib/GnizaWHM/
        ├── Config.pm      # Pure Perl config parser/writer (KEY="value" files)
        ├── Validator.pm   # Input validation (mirrors lib/config.sh)
        ├── Cron.pm        # Cron read + allowlisted gniza schedule commands
        ├── Runner.pm      # Pattern-based safe CLI command runner for WHM
        └── UI.pm          # Nav, flash, CSRF, HTML escaping, CSS delivery
cpanel/
├── gniza/
│   ├── index.live.cgi     # Category grid — 8 restore type cards
│   ├── restore.live.cgi   # Multi-step restore workflow (4 steps)
│   ├── install.json       # cPanel plugin registration (Files section)
│   ├── assets/
│   │   ├── gniza-whm.css  # Built CSS (copy of WHM CSS)
│   │   └── gniza-logo.svg # Logo (copy of WHM logo)
│   └── lib/GnizaCPanel/
│       └── UI.pm          # Page wrapper, CSRF, flash, CSS delivery
└── admin/Gniza/
    ├── Restore            # AdminBin module (runs as root, privilege escalation)
    └── Restore.conf       # AdminBin config (mode=full)
```

## Architecture

### Global-Swapping Pattern (Multi-Remote)

All library functions (`ssh.sh`, `rclone.sh`, `transfer.sh`, `snapshot.sh`, `retention.sh`) read globals like `REMOTE_TYPE`, `REMOTE_HOST`, `REMOTE_PORT`, `REMOTE_USER`, `REMOTE_KEY`, `REMOTE_BASE`, `BWLIMIT`, `RETENTION_COUNT`, `RSYNC_EXTRA_OPTS`, plus cloud-specific globals (`S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_REGION`, `S3_ENDPOINT`, `S3_BUCKET`, `GDRIVE_SERVICE_ACCOUNT_FILE`, `GDRIVE_ROOT_FOLDER_ID`).

Rather than passing remote context through function arguments, `remotes.sh` provides:

- `_save_remote_globals()` — snapshot current globals
- `load_remote(name)` — source `/etc/gniza/remotes.d/<name>.conf`, overriding REMOTE_* globals
- `_restore_remote_globals()` — restore saved snapshot

This keeps the change set minimal — no existing function signatures needed modification.

**Critical pattern:** Always call `_save_remote_globals()` before a remote loop, `load_remote()` inside the loop, and `_restore_remote_globals()` after the loop:

```bash
_save_remote_globals
while IFS= read -r rname; do
    load_remote "$rname"
    # ... existing functions use current REMOTE_* globals ...
done <<< "$remotes"
_restore_remote_globals
```

### Backup Flow

```
cmd_backup()
├── load_config() + validate_config() + init_logging()
├── get_target_remotes(--remote flag)
├── _save_remote_globals()
├── Test connectivity to all targets upfront (SSH or rclone)
├── For each account:
│   ├── run_pkgacct()              # ONCE
│   ├── gzip_sql_files()           # ONCE
│   ├── For each remote:
│   │   ├── load_remote(name)
│   │   └── _backup_to_current_remote()
│   │       ├── clean_partial_snapshots()
│   │       ├── get_latest_snapshot()
│   │       ├── transfer_pkgacct()
│   │       ├── transfer_homedir()
│   │       ├── finalize_snapshot()
│   │       └── enforce_retention()
│   ├── _restore_remote_globals()
│   └── cleanup_pkgacct()
└── send_backup_report()
```

### Command Routing

`bin/gniza` main() parses the first arg and routes to `cmd_*()` functions. Each command handles its own `--config`, `--remote`, `--account` flags via `get_opt()` and `has_flag()`.

Commands: `backup`, `restore`, `list`, `verify`, `status`, `remote`, `schedule`, `init`, `version`, `help`

### Config Hierarchy

1. `lib/constants.sh` — `DEFAULT_*` readonly values
2. `/etc/gniza/gniza.conf` — main config: local settings only (accounts, logging, notifications)
3. `/etc/gniza/remotes.d/<name>.conf` — per-remote config (REMOTE_*, retention, transfer)
4. `/etc/gniza/schedules.d/<name>.conf` — per-schedule config (timing, target remotes)
5. CLI flags (`--debug`, `--config=PATH`)

### Snapshot Layout

- Remote path: `$REMOTE_BASE/<hostname>/accounts/<user>/snapshots/<timestamp>`
- Timestamp format: `YYYY-MM-DDTHHMMSS` (UTC)
- pkgacct content lives directly in the snapshot root (no `pkgacct/` wrapper)
- `homedir/` subdirectory sits alongside the pkgacct content

```
<timestamp>/
├── mysql/        ← pkgacct: SQL dumps (*.sql.gz)
├── mysql.sql     ← pkgacct: database grants
├── cp/           ← pkgacct: cPanel metadata
├── ...           ← other pkgacct files
└── homedir/      ← home directory
```

**SSH remotes:** In-progress snapshots have `.partial` suffix. `latest` symlink points to newest completed snapshot. Uses `rsync --link-dest` for deduplication.

**Cloud remotes (S3/GDrive):** Cloud storage has no atomic rename or symlinks. Instead:
- Uploads go directly to `<timestamp>/` (no `.partial` suffix)
- A `.complete` marker file is created on success
- `latest.txt` text file stores the newest timestamp (replaces symlink)
- Directories without `.complete` are treated as partials and purged on next run

**Backward compat:** Old snapshots may have a `pkgacct/` subdirectory. Verify and restore detect the format automatically (`test -d "$snap_path/pkgacct"` for SSH, `rclone_exists` for cloud) and adjust paths accordingly. `transfer_pkgacct()` link-dest also handles old-format previous snapshots.

### Decoupled Schedules

Schedules are independent from remotes. Each schedule lives in `/etc/gniza/schedules.d/<name>.conf` and defines when backups run and which remotes to target. This allows multiple schedules targeting different sets of remotes.

Cron entries are tagged with `# gniza:<name>` comment lines. `install_schedules()` strips old tagged lines and appends new ones. Format:

```
# gniza:nightly
0 2 * * * /usr/local/bin/gniza backup --remote=nas,offsite >> /var/log/gniza/cron-nightly.log 2>&1
```

### Comma-Separated Remote Targeting

`get_target_remotes()` accepts comma-separated remote names via `--remote=nas,offsite`. It splits on commas, verifies each remote exists, and outputs one name per line. This enables both CLI usage and schedule configs targeting multiple remotes.

### cPanel User Restore Plugin

Allows cPanel account owners to restore their own data (files, databases, email, etc.) without WHM admin access.

**Privilege escalation:** Uses cPanel's AdminBin framework. CGIs run as the logged-in cPanel user; the AdminBin module (`cpanel/admin/Gniza/Restore`) runs as root. The account parameter is always forced to `$ENV{'REMOTE_USER'}` (cPanel-authenticated), never from user input.

**Security model:**
- Account isolation: AdminBin forces the authenticated username — users can only restore their own data
- No `--terminate`: AdminBin never passes the terminate flag, preventing destructive full restores
- Remote filtering: `USER_RESTORE_REMOTES` config controls which remotes users can access (`"all"`, comma-separated names, or empty to disable)
- Strict regex validation on all arguments (mirrors `GnizaWHM::Runner` patterns)
- Per-user CSRF tokens at `/tmp/.gniza-cpanel-csrf-$user`

**Install locations:**
- CGIs: `/usr/local/cpanel/base/frontend/jupiter/gniza/`
- AdminBin: `/usr/local/cpanel/bin/admin/Gniza/`
- Plugin registration: via `install_plugin` with `install.json`

**Workflow:** Category grid (`index.live.cgi`) → 4-step restore (`restore.live.cgi`): select remote/snapshot → select items → confirm → execute

### GnizaCPanel::UI

| Function | Description |
|----------|-------------|
| `esc($str)` | HTML-escape a string |
| `get_current_user()` | Returns `$ENV{'REMOTE_USER'}` |
| `page_header($title)` | Inline CSS + `data-theme="gniza"` wrapper + logo |
| `page_footer()` | Close wrapper div |
| `set_flash($type, $text)` | Store flash message for next page load |
| `render_flash()` | Render and consume stored flash message |
| `csrf_hidden_field()` | Generate CSRF token + hidden input |
| `verify_csrf_token($token)` | Validate submitted CSRF token |
| `render_errors(\@errors)` | Render error list as HTML |

### AdminBin Module (Gniza::Restore)

Actions: `LIST_ALLOWED_REMOTES`, `LIST_SNAPSHOTS`, `LIST_DATABASES`, `LIST_MAILBOXES`, `LIST_FILES`, `LIST_DBUSERS`, `LIST_CRON`, `LIST_DNS`, `LIST_SSL`, `RESTORE_ACCOUNT`, `RESTORE_FILES`, `RESTORE_DATABASE`, `RESTORE_MAILBOX`, `RESTORE_CRON`, `RESTORE_DBUSERS`, `RESTORE_DOMAINS`, `RESTORE_SSL`

Called from CGI via: `Cpanel::AdminBin::Call::call('Gniza', 'Restore', 'ACTION', @args)`

## Coding Conventions

### Bash Style

- `set -euo pipefail` at top of entrypoint
- Functions use `local` for all variables
- Error paths: `log_error` + `return 1` (library) or `die "message"` (CLI)
- Guard-include pattern for constants: `[[ -n "${_GNIZA_CONSTANTS_LOADED:-}" ]] && return 0`
- `((count++)) || true` to avoid `set -e` traps on zero-to-one arithmetic

### Naming

- Libraries: `lib/<module>.sh` — each file focuses on one responsibility
- Public functions: `snake_case` (e.g., `transfer_pkgacct`, `list_remote_snapshots`)
- Private/helper functions: `_prefixed` (e.g., `_backup_to_current_remote`, `_save_remote_globals`)
- CLI commands: `cmd_<name>()` in `bin/gniza`
- Constants: `UPPER_SNAKE_CASE`, prefixed with `DEFAULT_` for defaults
- Globals: `UPPER_SNAKE_CASE` (e.g., `REMOTE_HOST`, `LOG_LEVEL`)

### Error Handling

- Single account failures don't abort the run (continue loop)
- In multi-remote mode, failure on one remote doesn't block others
- rsync retries with exponential backoff: `sleep $((attempt * 10))`
- Exit codes: `0` OK, `1` fatal, `2` locked, `5` partial failure
- Lock via `flock` on `/var/run/gniza.lock`

### cPanel API Policy

**All cPanel operations MUST use native cPanel APIs (UAPI, cpapi2, whmapi1).** Never use raw system commands (`mysql`, `crontab -u`, `rndc`, direct `cp` to cPanel paths) when a cPanel API exists. No fallbacks — if an API call fails, log the error and return failure.

| Operation | API | Command |
|-----------|-----|---------|
| Create database | UAPI | `uapi --user=$user Mysql create_database name=$dbname` |
| Create DB user | UAPI | `uapi --user=$user Mysql create_user name=$dbuser password=$pass` |
| Set DB privileges | UAPI | `uapi --user=$user Mysql set_privileges_on_database user=$dbuser database=$dbname privileges=ALL` |
| Get DB prefix | UAPI | `uapi --user=$user Mysql get_restrictions` → `prefix:` field |
| List DB users | UAPI | `uapi --user=$user Mysql list_users` |
| Create mailbox | UAPI | `uapi --user=$user Email add_pop email=$mailuser domain=$domain password=$pass quota=0` |
| Add cron job | cpapi2 | `cpapi2 --user=$user Cron add_line minute=... hour=... day=... month=... weekday=... command=...` |
| List cron jobs | cpapi2 | `cpapi2 --user=$user Cron listcron` |
| Remove cron job | cpapi2 | `cpapi2 --user=$user Cron remove_line linekey=$key` |
| Install SSL cert | whmapi1 | `whmapi1 installssl domain=$domain crt=$cert key=$key cab=$cab` |
| Create DNS zone | whmapi1 | `whmapi1 adddns domain=$domain trueowner=$user` |
| Add DNS record | whmapi1 | `whmapi1 addzonerecord domain=$domain name=$name type=$type address=$value ttl=$ttl` |
| Rebuild Apache | script | `/usr/local/cpanel/scripts/rebuildhttpdconf` |
| Rebuild user domains | script | `/usr/local/cpanel/scripts/updateuserdomains` |

**Prefix handling:** cPanel enforces a DB name/user prefix per account (e.g., `username_`). Get it via `uapi Mysql get_restrictions`. Names that don't match the prefix must be skipped with a warning — UAPI will reject them.

**Allowed exceptions (no cPanel API exists):**
- SQL dump import: `mysql $dbname < dump.sql` — only place raw `mysql` is permitted
- cPanel user config: `cp` to `/var/cpanel/users/$user` — no API for wholesale config file replacement
- Userdata files: `cp` to `/var/cpanel/userdata/$user/` — no API for writing raw userdata

**Forbidden (cPanel APIs exist — always use them):**
- `mysql` for CREATE DATABASE, CREATE USER, GRANT — use UAPI Mysql
- `crontab -u` — use cpapi2 Cron
- `cp` to `/var/named/` + `rndc reload` — use whmapi1 adddns/addzonerecord
- `cp` to `/var/cpanel/ssl/` + `checkallsslcerts` — use whmapi1 installssl
- Direct mailbox file creation — use UAPI Email add_pop first

### SSH/Rsync (REMOTE_TYPE=ssh)

- All SSH operations go through `build_ssh_opts()` / `remote_exec()` in `ssh.sh`
- rsync uses: `-aHAX --numeric-ids --delete --rsync-path="rsync --fake-super" --link-dest=<prev>`
- `--fake-super` stores real uid/gid/permissions as xattrs on the remote, preserving ownership even when the remote user is not root
- Bandwidth limiting: `--bwlimit=$BWLIMIT`
- Extra opts: `$RSYNC_EXTRA_OPTS` (split by word)

### Rclone Transport (REMOTE_TYPE=s3 or gdrive)

- All cloud operations go through `lib/rclone.sh`
- `_is_rclone_mode()` returns true when `REMOTE_TYPE` is `s3` or `gdrive`
- Each library function (snapshot, transfer, retention, verify, restore) checks `_is_rclone_mode` at the top and dispatches to `rclone_*` equivalents
- Temp rclone config generated per-operation from stored globals (`_build_rclone_config()`), cleaned up after
- S3 path: `remote:${S3_BUCKET}${REMOTE_BASE}/<hostname>/accounts/...`
- GDrive path: `remote:${REMOTE_BASE}/<hostname>/accounts/...`
- Bandwidth limiting: `--bwlimit=${BWLIMIT}k`
- Retries with exponential backoff mirror rsync behavior

## Configuration Files

### Main Config (`/etc/gniza/gniza.conf`)

Contains only local settings. Remote destinations are configured in `remotes.d/`.

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `TEMP_DIR` | No | `/usr/local/gniza/workdir` | Local working directory |
| `INCLUDE_ACCOUNTS` | No | (all) | Comma-separated account list |
| `EXCLUDE_ACCOUNTS` | No | `nobody` | Comma-separated exclusions |
| `LOG_DIR` | No | `/var/log/gniza` | Log directory |
| `LOG_LEVEL` | No | `info` | `debug\|info\|warn\|error` |
| `LOG_RETAIN` | No | `90` | Days to keep log files |
| `NOTIFY_EMAIL` | No | (disabled) | Notification email |
| `NOTIFY_ON` | No | `failure` | `always\|failure\|never` |
| `LOCK_FILE` | No | `/var/run/gniza.lock` | Lock file path |
| `SSH_TIMEOUT` | No | `30` | SSH connection timeout (seconds) |
| `SSH_RETRIES` | No | `3` | rsync retry attempts |
| `RSYNC_EXTRA_OPTS` | No | (empty) | Extra rsync options |
| `USER_RESTORE_REMOTES` | No | `all` | Remotes for cPanel user restore (`all`, comma-separated names, or empty to disable) |

### Remote Config (`/etc/gniza/remotes.d/<name>.conf`)

**Common (all types):**

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `REMOTE_TYPE` | No | `ssh` | `ssh`, `s3`, or `gdrive` |
| `REMOTE_BASE` | No | `/backups` | Remote base directory/path |
| `BWLIMIT` | No | `0` | Bandwidth limit KB/s |
| `RETENTION_COUNT` | No | `30` | Snapshots to keep |

**SSH-specific (REMOTE_TYPE=ssh):**

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `REMOTE_HOST` | Yes | — | Remote hostname/IP |
| `REMOTE_PORT` | No | `22` | SSH port |
| `REMOTE_USER` | No | `root` | SSH user |
| `REMOTE_AUTH_METHOD` | No | `key` | `key` or `password` |
| `REMOTE_KEY` | Yes (key) | — | SSH private key path |
| `REMOTE_PASSWORD` | Yes (password) | — | SSH password (requires sshpass) |
| `RSYNC_EXTRA_OPTS` | No | (empty) | Extra rsync options |

**S3-specific (REMOTE_TYPE=s3):**

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `S3_ACCESS_KEY_ID` | Yes | — | S3 access key |
| `S3_SECRET_ACCESS_KEY` | Yes | — | S3 secret key |
| `S3_REGION` | No | `us-east-1` | AWS region |
| `S3_ENDPOINT` | No | (empty) | Custom endpoint for S3-compatible services |
| `S3_BUCKET` | Yes | — | Bucket name |

**Google Drive-specific (REMOTE_TYPE=gdrive):**

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `GDRIVE_SERVICE_ACCOUNT_FILE` | Yes | — | Path to service account JSON key file |
| `GDRIVE_ROOT_FOLDER_ID` | No | (empty) | Root folder ID |

### Schedule Config (`/etc/gniza/schedules.d/<name>.conf`)

Schedules are decoupled from remotes. Each schedule targets one or more remotes.

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `SCHEDULE` | Yes | — | `hourly\|daily\|weekly\|monthly\|custom` |
| `SCHEDULE_TIME` | No | `02:00` | HH:MM (24-hour) |
| `SCHEDULE_DAY` | Conditional | — | Hours interval (1-23) for hourly, day-of-week (0-6) for weekly, day-of-month (1-28) for monthly |
| `SCHEDULE_CRON` | Conditional | — | Full 5-field cron expr (for `custom` only) |
| `REMOTES` | No | (all) | Comma-separated remote names to target |

## Key Functions Reference

### rclone.sh

| Function | Description |
|----------|-------------|
| `_is_rclone_mode()` | True if `REMOTE_TYPE` is `s3` or `gdrive` |
| `_build_rclone_config()` | Generate temp rclone.conf from current globals, return path |
| `_cleanup_rclone_config(path)` | Remove temp config file |
| `_rclone_remote_path(subpath)` | Build `remote:bucket/base/subpath` or `remote:base/subpath` |
| `_rclone_cmd(subcmd, args...)` | Run rclone with temp config, manage lifecycle |
| `rclone_to_remote(src, dest)` | Upload directory to remote (mirrors `rsync_to_remote`) |
| `rclone_from_remote(src, dest)` | Download from remote to local directory |
| `rclone_list_remote_snapshots(user)` | List snapshots with `.complete` marker |
| `rclone_get_latest_snapshot(user)` | Read `latest.txt` or fall back to sorted list |
| `rclone_finalize_snapshot(user, ts)` | Create `.complete` marker + write `latest.txt` |
| `rclone_clean_partial_snapshots(user)` | Purge dirs without `.complete` |
| `rclone_resolve_snapshot(user, ts)` | Check dir exists + has `.complete` |
| `rclone_ensure_dir(path)` | Create remote directory |
| `rclone_purge(path)` | Recursive delete |
| `rclone_exists(path)` | Check if remote path exists |
| `rclone_size(path)` | Get size via `rclone size --json` |
| `rclone_list_files(path)` | List files via `rclone lsf` |
| `rclone_list_dirs(path)` | List directories via `rclone lsf --dirs-only` |
| `rclone_cat(path)` | Read remote file content |
| `rclone_rcat(path, content)` | Write content to remote file |
| `test_rclone_connection()` | Verify credentials with `rclone lsd` |

### remotes.sh

| Function | Description |
|----------|-------------|
| `list_remotes()` | List remote names from `remotes.d/*.conf` |
| `has_remotes()` | Check if any remote configs exist |
| `load_remote(name)` | Source config, override REMOTE_*/S3_*/GDRIVE_* globals |
| `validate_remote(name)` | Load + validate a remote config (dispatches by REMOTE_TYPE) |
| `get_target_remotes(flag)` | Resolve `--remote=NAME[,NAME2]` (comma-separated) or return all; errors if none configured |
| `_save_remote_globals()` | Save current REMOTE_*/S3_*/GDRIVE_* globals |
| `_restore_remote_globals()` | Restore saved globals |

### schedule.sh

Reads schedules from `/etc/gniza/schedules.d/` (decoupled from remotes).

| Function | Description |
|----------|-------------|
| `list_schedules()` | List schedule names from `schedules.d/*.conf` |
| `has_schedules()` | Check if any schedule configs exist |
| `load_schedule(name)` | Source config, set SCHEDULE/SCHEDULE_REMOTES globals |
| `schedule_to_cron(name)` | Convert SCHEDULE vars to 5-field cron expression |
| `build_cron_line(name)` | Full cron line with gniza command, `--remote=` flag, and log redirect |
| `install_schedules()` | Strip old gniza cron entries, add new from all `schedules.d/` |
| `show_schedules()` | Display current gniza cron entries |
| `remove_schedules()` | Remove all gniza cron entries |

### restore.sh

All restore functions dispatch by `_is_rclone_mode` — using `rclone_from_remote` for cloud or rsync for SSH.

| Function | Description |
|----------|-------------|
| `restore_full_account(user, ts, terminate, exclude)` | Full account restore from snapshot. If `terminate=true`, removes existing account via `/scripts/removeacct` before restoring. Otherwise merges with `--force`. |
| `restore_files(user, ts, path)` | Restore specific files/directories |
| `restore_database(user, ts, dbname)` | Restore a MySQL database from snapshot |
| `restore_mailbox(user, email, ts)` | Restore a mailbox (parses email → mail/domain/user path) |
| `list_snapshot_databases(user, ts)` | List `*.sql.gz` in snapshot's mysql/ dir (SSH: remote_exec, cloud: rclone_list_files) |
| `list_snapshot_mailboxes(user, ts)` | List domain/user dirs in mail/ (SSH: remote_exec, cloud: rclone_list_dirs) |
| `_rsync_download(src, dest)` | Download helper — dispatches rclone_from_remote or rsync |
| `_detect_pkgacct_base(user, ts)` | Detect old vs new snapshot format (SSH or cloud) |

### bin/gniza (CLI helpers)

| Function | Description |
|----------|-------------|
| `get_opt(name, args...)` | Extract `--name=VALUE` from args |
| `has_flag(name, args...)` | Check for `--name` boolean flag |
| `_backup_to_current_remote(user, ts)` | Transfer + finalize + retention for one account on current remote |
| `_restore_load_remote(flag)` | Load remote context for restore (always requires `--remote`) |
| `_list_current_remote(account)` | Display listing for current remote context |
| `_test_connection()` | Dispatch `test_rclone_connection` or `test_ssh_connection` by type |
| `_status_ssh_and_disk()` | Connection test + disk/storage usage display (SSH: df, cloud: rclone about) |
| `_init_remote(name)` | Interactive remote destination setup |
| `cmd_remote()` | Remote management: list, delete |
| `cmd_schedule()` | Schedule CRUD: add, delete, list, install, show, remove |

### GnizaWHM::UI (WHM plugin)

| Function | Description |
|----------|-------------|
| `is_configured()` | True if any remote configs exist in `remotes.d/` |
| `detect_ssh_keys()` | Scan `/root/.ssh/` for key files, return arrayref of hashes |
| `render_ssh_guidance()` | HTML block: detected keys + keygen/ssh-copy-id instructions |
| `has_remotes()` | Check if `/etc/gniza/remotes.d/` has `.conf` files |
| `list_remotes()` | Return sorted list of remote names |
| `has_schedules()` | Check if `/etc/gniza/schedules.d/` has `.conf` files |
| `list_schedules()` | Return sorted list of schedule names |
| `schedule_conf_path($name)` | Return path to schedule config file |
| `esc($str)` | HTML-escape a string |
| `render_nav($page)` | DaisyUI tab navigation bar (tabs-lift) |
| `set_flash($type, $text)` | Store flash message for next page load |
| `render_flash()` | Render and consume stored flash message |
| `csrf_hidden_field()` | Generate CSRF token + hidden input |
| `verify_csrf_token($token)` | Validate submitted CSRF token |
| `render_errors(\@errors)` | Render error list as HTML |
| `page_header($title)` | Inline CSS + `data-theme="light"` wrapper + page title |
| `page_footer()` | Close the `data-theme` wrapper div |
| `_unwrap_layers($css)` | Strip `@layer` wrappers from Tailwind CSS for WHM compatibility |
| `get_cpanel_accounts()` | Parse `/etc/trueuserdomains` for account list |
| `test_rclone_connection(%args)` | Test S3/GDrive connection via rclone (generates temp config, runs `rclone lsd`) |

### GnizaWHM::Runner (WHM plugin)

Pattern-based command runner for safe CLI execution from the WHM UI. Each allowed command has regex patterns per argument position.

| Function | Description |
|----------|-------------|
| `run($cmd, $subcmd, \@args, \%opts)` | Validate against allowlist and execute gniza CLI |

Allowed commands: `restore account/files/database/mailbox/list-databases/list-mailboxes`, `list`.
Named option patterns: `--remote`, `--timestamp`, `--path`, `--account`, `--terminate`, `--exclude`.

### GnizaWHM::Config

| Array | Description |
|-------|-------------|
| `@MAIN_KEYS` | Main config keys (local settings only, no REMOTE_*) |
| `@REMOTE_KEYS` | Remote config keys (REMOTE_TYPE, SSH, S3, GDrive, transfer, retention — no SCHEDULE*) |
| `@SCHEDULE_KEYS` | Schedule config keys (SCHEDULE, SCHEDULE_TIME, SCHEDULE_DAY, SCHEDULE_CRON, REMOTES) |

### GnizaWHM::Validator

| Function | Description |
|----------|-------------|
| `validate_main_config(\%data)` | Validate main config values |
| `validate_remote_config(\%data)` | Validate remote config by REMOTE_TYPE (ssh: host+key/password, s3: credentials+bucket, gdrive: service account) |
| `validate_remote_name($name)` | Validate remote name (alphanumeric + hyphens/underscores) |
| `validate_schedule_config(\%data)` | Validate schedule config (SCHEDULE required, conditional fields) |
| `validate_schedule_name($name)` | Validate schedule name |

## Testing

Run existing tests:

```bash
bash tests/test_utils.sh
```

Tests cover: `timestamp()` format, `human_size()`, `human_duration()`, `require_cmd()`, `filter_accounts()`, `validate_config()`.

Tests use a simple `assert_eq`/`assert_ok`/`assert_fail` framework defined in `test_utils.sh`.

## Common Tasks

### Adding a new library function

1. Add to the appropriate `lib/<module>.sh`
2. Functions are automatically available — libraries are sourced in `bin/gniza`
3. Use `local` for all variables, `log_*` for output, `return 1` for errors

### Adding a new command

1. Add `cmd_<name>()` function in `bin/gniza`
2. Add routing in `main()` case statement
3. Update `cmd_usage()` help text
4. Update `README.md` commands table

### Adding a new config variable

1. Add `DEFAULT_<NAME>` to `lib/constants.sh`
2. Add to `load_config()` in `lib/config.sh` with fallback
3. Add validation in `validate_config()` if needed
4. Add to `etc/gniza.conf.example`
5. Document in `README.md` and this file

### Making a function remote-aware

If a function needs to work across multiple remotes, wrap calls in the save/load/restore pattern:

```bash
_save_remote_globals
while IFS= read -r rname; do
    load_remote "$rname"
    your_function_here    # uses current REMOTE_* globals
done <<< "$remotes"
_restore_remote_globals
```

### Adding a new WHM plugin page

1. Create `whm/gniza-whm/<name>.cgi` following the pattern of existing CGIs
2. Use same boilerplate: shebang, `use lib`, `Whostmgr::HTMLInterface`, `Cpanel::Form`, `GnizaWHM::UI`
3. Route by `$form->{'action'}` or similar param
4. Use `GnizaWHM::UI::page_header()`, `render_nav()`, `render_flash()`, `csrf_hidden_field()`, `page_footer()`
5. Validate POST with `verify_csrf_token()`, redirect with 302 after success
6. No AppConfig change needed — `url=/cgi/gniza-whm/` covers all CGIs in the directory
7. Add any new DaisyUI/Tailwind classes to `assets/src/safelist.html` and rebuild CSS
8. Add the page to `@NAV_ITEMS` in `UI.pm` if it should appear in the tab bar

### WHM CSS Policy

**NEVER write custom CSS.** Always use Tailwind utility classes and DaisyUI components exclusively. All styling must be done through class attributes in HTML — no custom CSS rules, no `<style>` blocks (except the auto-generated inline delivery in `page_header()`), no CSS files other than the Tailwind build output.

### WHM Theme & Color Palette

The WHM plugin uses a custom DaisyUI theme named `gniza` (defined in `assets/src/input.css`). Light-only, no dark mode.

| Role | OKLCH Value | Approx Color |
|------|-------------|-------------|
| **Primary** | `oklch(38.2% 0.145 259.4)` | Deep navy blue |
| **Primary content** | `oklch(100% 0 0)` | White |
| **Secondary** | `oklch(69.5% 0.169 47.8)` | Warm copper/orange |
| **Secondary content** | `oklch(100% 0 0)` | White |
| **Accent / Warning** | `oklch(86.4% 0.177 90.8)` | Soft gold/yellow |
| **Accent content / Warning content** | `oklch(30.9% 0.116 258.9)` | Dark navy |
| **Neutral / Base content** | `oklch(30.9% 0.116 258.9)` | Dark navy |
| **Neutral content** | `oklch(100% 0 0)` | White |
| **Base 100** | `transparent` | Transparent (inherits WHM background) |
| **Base 200** | `oklch(97% 0 0)` | Near-white gray |
| **Base 300** | `oklch(89.8% 0 0)` | Light gray |
| **Info** | `oklch(69% 0.083 217.5)` | Muted blue |
| **Success** | `oklch(65% 0.25 140)` | Vivid green |
| **Error** | `oklch(57.7% 0.245 27.3)` | Red |

**Typography:** `'Helvetica Neue', Helvetica, Arial, sans-serif`
**Border radius:** `0.5rem` (boxes/selectors), `0.25rem` (fields)
**Base font size:** `1.6rem` (set on the `data-theme` wrapper to match WHM's sizing)

### WHM CSS Build System (Tailwind v4 + DaisyUI v5)

All WHM pages use Tailwind CSS v4 with DaisyUI v5 for styling. The CSS is built from source and committed.

**Build:**
```bash
cd whm/gniza-whm/assets && npm install && npm run build:css
```

**Key files:**
- `assets/src/input.css` — Tailwind entry point with DaisyUI plugin config
- `assets/src/safelist.html` — Class safelist (required because Tailwind v4 scanner doesn't recognize `.cgi`/`.pm` file extensions)
- `assets/gniza-whm.css` — Built output (committed to repo)

**WHM CSS delivery quirks:**
- WHM's CGI directory cannot serve static files directly
- WHM URLs require session token prefix (`/cpsessXXXXX/`)
- CSS is inlined via `<style>` tag in `page_header()` — reads from disk, strips `@layer` wrappers, embeds inline
- Tailwind v4 wraps all CSS in `@layer` directives which have lower specificity than WHM's un-layered CSS — `_unwrap_layers()` strips these
- `@import "tailwindcss" important;` adds `!important` to all utilities so they override WHM's styles

**Adding new CSS classes:**
1. Add the class to `assets/src/safelist.html` (since Tailwind can't scan `.cgi`/`.pm` files)
2. Rebuild: `cd whm/gniza-whm/assets && npm run build:css`
3. Commit the updated `gniza-whm.css`

### Repository

| | URL |
|---|-----|
| **Git (SSH)** | `ssh://git@192.168.100.100:2222/shukivaknin/gniza.git` |
| **Git (HTTP)** | `http://192.168.100.100:3001/shukivaknin/gniza.git` |
| **Web UI** | http://192.168.100.100:3001/shukivaknin/gniza |