shukivaknin/gniza4linux
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite README with comprehensive feature documentation

Browse Source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
shuki
2026-03-07 04:52:33 +02:00
parent 09b7dd184e
commit 8e8d367233
3 changed files with 286 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files

340
README.md
View File

@@ -1,6 +1,8 @@
# gniza - Linux Backup Manager
A generic Linux backup tool with a Python Textual TUI, web GUI, and CLI interface. Define named backup sources (sets of directories), configure backup destinations (SSH, local, S3, Google Drive), and run incremental backups with rsync `--link-dest` deduplication.
A complete Linux backup solution that works as a **stand-alone backup tool** or a **centralized backup server**. Pull files from local directories, remote SSH servers, S3 buckets, or Google Drive, and push them to any combination of SSH, local, S3, or Google Drive destinations — all with incremental rsync snapshots, hardlink deduplication, and automatic retention.
Manage everything through a terminal UI, web dashboard, or CLI.
```
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
@@ -24,21 +26,32 @@ A generic Linux backup tool with a Python Textual TUI, web GUI, and CLI interfac
## Features
- **Source-based backups** - Define named profiles with sets of directories to back up
- **Include/exclude filters** - Rsync include or exclude patterns per source (comma-separated)
- **MySQL backup** - Dump all or selected databases alongside directory backups
- **Multiple destination types** - SSH, local (USB/NFS), S3, Google Drive
- **Incremental snapshots** - rsync `--link-dest` for space-efficient deduplication
- **Disk space safety** - Abort backup if destination disk usage exceeds configurable threshold (default 95%)
- **Textual TUI** - Beautiful terminal UI powered by [Textual](https://textual.textualize.io/)
- **Web dashboard** - Access the full TUI from any browser with HTTP Basic Auth
- **CLI interface** - Scriptable commands for automation and cron
- **Atomic snapshots** - `.partial` directory during backup, renamed on success
- **Retention policies** - Automatic pruning of old snapshots
- **Pre/post hooks** - Run custom commands before/after backups
- **Email notifications** - SMTP or system mail on success/failure
- **Root and user mode** - Works as root (system-wide) or regular user (per-user)
- **Cron scheduling** - Manage cron jobs through TUI or CLI
- **Stand-alone or backup server** — Back up the local machine, or pull from remote servers without installing anything on them
- **Remote sources** — Pull files from SSH servers, S3 buckets, or Google Drive before backing up
- **Multiple destination types** — Push to SSH, local drives (USB/NFS), S3, or Google Drive
- **Incremental snapshots** — rsync `--link-dest` hardlink deduplication across snapshots
- **MySQL/MariaDB backup** — Dump all or selected databases with grants, routines, and triggers
- **Atomic snapshots** — `.partial` directory during transfer, renamed on success
- **Retention policies** — Automatic pruning per-destination or per-source with snapshot pinning
- **Disk space safety** — Abort if destination usage exceeds threshold (default 95%)
- **Pre/post hooks** — Run shell commands before and after each backup
- **Cron scheduling** — Hourly, daily, weekly, monthly, or custom cron expressions
- **Email notifications** — SMTP (TLS/SSL) or system mail on failure or every run
- **Bandwidth limiting** — Global or per-destination KB/s cap
- **Retry logic** — Automatic SSH reconnection with exponential backoff
- **Include/exclude filters** — Rsync glob patterns per source
- **Terminal UI** — Full-featured TUI powered by [Textual](https://textual.textualize.io/)
- **Web dashboard** — Access the TUI from any browser with HTTP Basic Auth
- **CLI** — Scriptable commands for automation and cron
- **Root and user mode** — System-wide (`/etc/gniza`) or per-user (`~/.config/gniza`)
## Use Cases
**Stand-alone backup** — Install gniza on any Linux server or workstation. Define local folders as sources and back them up to an SSH server, USB drive, S3, or Google Drive.
**Backup server** — Install gniza on a central server. Define remote SSH sources pointing to your production servers. gniza pulls their files and stores snapshots on local drives or cloud storage — no agent needed on the source machines.
**Hybrid** — Mix local and remote sources in the same installation. Back up local configs alongside files pulled from multiple remote servers, all managed from one place.
## Installation
@@ -66,26 +79,28 @@ bash scripts/install.sh # user mode
Root mode installs to `/usr/local/gniza`. User mode installs to `~/.local/share/gniza`.
The installer detects dependencies, sets up config directories, and optionally launches a setup wizard.
### Dependencies
- **Required**: bash 4+, rsync
- **Optional**: ssh, curl (SMTP notifications), rclone (S3/GDrive)
- **Optional**: ssh, curl (SMTP notifications), sshpass (password auth), rclone (S3/Google Drive)
- **TUI/Web**: python3, textual, textual-serve (installed automatically)
## Quick Start
```bash
# Launch TUI
# Launch the TUI
gniza
# Or use CLI
# Or use the CLI
gniza targets add --name=mysite --folders=/var/www,/etc/nginx
gniza remotes add --name=backup-server # (edit config manually)
gniza remotes add --name=backup-server
gniza --cli backup --target=mysite
gniza --cli backup --all
```
## Usage
## CLI Reference
```
gniza [OPTIONS] [COMMAND]
@@ -99,13 +114,16 @@ Options:
Commands:
backup [--target=NAME] [--remote=NAME] [--all]
restore --target=NAME [--snapshot=TS] [--remote=NAME] [--dest=DIR]
targets list|add|delete|show [--name=NAME] [--folders=PATHS]
remotes list|add|delete|show|test|disk-info-short [--name=NAME]
restore --target=NAME --snapshot=TS [--remote=NAME] [--dest=DIR] [--skip-mysql]
targets list | add | delete | show [--name=NAME] [--folders=PATHS]
remotes list | add | delete | show | test | disk-info-short [--name=NAME]
snapshots list [--target=NAME] [--remote=NAME]
browse --target=NAME --snapshot=TS [--remote=NAME]
retention [--target=NAME] [--remote=NAME] [--all]
schedule install|show|remove
schedule install | show | remove
logs [--last] [--tail=N]
web start | install-service | remove-service | status [--port=PORT]
uninstall
```
## Configuration
@@ -117,24 +135,76 @@ Commands:
Config subdirectories: `targets.d/*.conf`, `remotes.d/*.conf`, `schedules.d/*.conf`
### Target Config (`targets.d/mysite.conf`)
### Global Settings (`gniza.conf`)
```ini
BWLIMIT=0 # Bandwidth limit in KB/s (0 = unlimited)
RETENTION_COUNT=30 # Default snapshots to keep
LOG_LEVEL="info" # info or debug
LOG_RETAIN=90 # Days to keep log files
DISK_USAGE_THRESHOLD=95 # Abort if destination >= this % (0 = disabled)
SSH_TIMEOUT=30 # SSH connection timeout in seconds
SSH_RETRIES=3 # Number of retry attempts
RSYNC_EXTRA_OPTS="" # Additional rsync flags
WORK_DIR="/tmp" # Temp directory for staging
# Notifications
NOTIFY_ON="failure" # never | failure | always
NOTIFY_EMAIL="" # Comma-separated recipients
SMTP_HOST=""
SMTP_PORT=587
SMTP_USER=""
SMTP_PASSWORD=""
SMTP_FROM=""
SMTP_SECURITY="tls" # tls | ssl | none
# Web dashboard
WEB_USER="admin"
WEB_API_KEY="" # Generated during install
```
### Source Config (`targets.d/mysite.conf`)
A **source** defines what to back up: a set of folders, optional filters, hooks, and MySQL settings.
```ini
TARGET_NAME="mysite"
TARGET_FOLDERS="/var/www,/etc/nginx"
TARGET_EXCLUDE="*.log,*.tmp,.cache"
TARGET_INCLUDE=""
TARGET_REMOTE=""
TARGET_RETENTION=""
TARGET_PRE_HOOK=""
TARGET_POST_HOOK=""
TARGET_REMOTE="" # Pin to a specific destination
TARGET_RETENTION="" # Override retention count
TARGET_PRE_HOOK="" # Shell command before backup
TARGET_POST_HOOK="" # Shell command after backup
TARGET_ENABLED="yes"
# MySQL backup (optional)
# Remote source (pull from another machine)
TARGET_SOURCE_TYPE="local" # local | ssh | s3 | gdrive
# SSH source
TARGET_SOURCE_HOST=""
TARGET_SOURCE_PORT="22"
TARGET_SOURCE_USER="root"
TARGET_SOURCE_AUTH_METHOD="key" # key | password
TARGET_SOURCE_KEY=""
TARGET_SOURCE_PASSWORD=""
# S3 source
TARGET_SOURCE_S3_BUCKET=""
TARGET_SOURCE_S3_REGION="us-east-1"
TARGET_SOURCE_S3_ENDPOINT=""
TARGET_SOURCE_S3_ACCESS_KEY_ID=""
TARGET_SOURCE_S3_SECRET_ACCESS_KEY=""
# Google Drive source
TARGET_SOURCE_GDRIVE_SERVICE_ACCOUNT_FILE=""
TARGET_SOURCE_GDRIVE_ROOT_FOLDER_ID=""
# MySQL backup
TARGET_MYSQL_ENABLED="no"
TARGET_MYSQL_MODE="all"
TARGET_MYSQL_DATABASES=""
TARGET_MYSQL_EXCLUDE=""
TARGET_MYSQL_MODE="all" # all | selected
TARGET_MYSQL_DATABASES="" # Comma-separated (when mode=selected)
TARGET_MYSQL_EXCLUDE="" # Databases to skip
TARGET_MYSQL_USER=""
TARGET_MYSQL_PASSWORD=""
TARGET_MYSQL_HOST="localhost"
@@ -142,76 +212,224 @@ TARGET_MYSQL_PORT="3306"
TARGET_MYSQL_EXTRA_OPTS="--single-transaction --routines --triggers"
```
**Include vs Exclude**: Set `TARGET_INCLUDE` to back up only matching files (e.g. `*.conf,*.sh`). When include is set, everything else is excluded. If only `TARGET_EXCLUDE` is set, matching files are skipped. Patterns are comma-separated and support rsync glob syntax.
**Include vs Exclude**: Set `TARGET_INCLUDE` to back up only matching files (e.g. `*.conf,*.sh`). When include is set, everything else is excluded. If only `TARGET_EXCLUDE` is set, matching files are skipped. Patterns use rsync glob syntax.
### Remote Config (`remotes.d/backup-server.conf`)
### Destination Config (`remotes.d/backup-server.conf`)
A **destination** defines where snapshots are stored.
```ini
REMOTE_TYPE="ssh"
REMOTE_TYPE="ssh" # ssh | local | s3 | gdrive
REMOTE_HOST="backup.example.com"
REMOTE_PORT=22
REMOTE_USER="root"
REMOTE_AUTH_METHOD="key"
REMOTE_KEY="/root/.ssh/backup_key"
REMOTE_AUTH_METHOD="key" # key | password
REMOTE_KEY="/root/.ssh/backup_key" # Defaults to ~/.ssh/id_rsa
REMOTE_PASSWORD=""
REMOTE_BASE="/backups"
BWLIMIT=0
RETENTION_COUNT=30
BWLIMIT=0 # Override global bandwidth limit
RETENTION_COUNT=30 # Override global retention
```
For local remotes (USB/NFS):
**Local destination** (USB drive, NFS mount):
```ini
REMOTE_TYPE="local"
REMOTE_BASE="/mnt/backup-drive"
```
**S3 destination**:
```ini
REMOTE_TYPE="s3"
S3_BUCKET="my-backups"
S3_ACCESS_KEY_ID="AKIA..."
S3_SECRET_ACCESS_KEY="..."
S3_REGION="us-east-1"
S3_ENDPOINT="" # For S3-compatible (MinIO, DigitalOcean Spaces)
```
**Google Drive destination**:
```ini
REMOTE_TYPE="gdrive"
GDRIVE_SERVICE_ACCOUNT_FILE="/path/to/service-account.json"
GDRIVE_ROOT_FOLDER_ID="" # Optional folder ID
```
### Schedule Config (`schedules.d/nightly.conf`)
```ini
SCHEDULE="daily" # hourly | daily | weekly | monthly | custom
SCHEDULE_TIME="02:00" # HH:MM
SCHEDULE_DAY="" # Day of week (0-6) or day of month (1-28)
SCHEDULE_CRON="" # Full cron expression (when SCHEDULE=custom)
SCHEDULE_ACTIVE="yes"
TARGETS="" # Comma-separated sources (empty = all)
REMOTES="" # Comma-separated destinations (empty = all)
```
## How Incremental Backups Work
GNIZA uses rsync's `--link-dest` option to create space-efficient incremental backups using **hardlinks**.
gniza uses rsync's `--link-dest` option to create space-efficient incremental backups using **hardlinks**.
**The first backup** copies every file from source to destination. This takes the most time and disk space, since every file must be transferred in full. Depending on the size of your data and network speed, this initial backup may take a long time — this is normal.
**The first backup** copies every file from source to destination. This takes the most time and disk space. Depending on data size and network speed, the initial backup may take a long time — this is normal.
**Every backup after the first** is significantly faster. Rsync compares each file against the previous snapshot. Files that haven't changed are not transferred again — instead, rsync creates a **hardlink** to the same data block on disk from the previous snapshot. Only new or modified files are actually copied.
**Every subsequent backup** is significantly faster. Rsync compares each file against the previous snapshot. Unchanged files are not transferred — instead, rsync creates a **hardlink** to the same data block from the previous snapshot. Only new or modified files are copied.
This means:
- Each snapshot appears as a full, complete directory tree — you can browse or restore any snapshot independently.
- Unchanged files share disk space between snapshots through hardlinks, so 10 snapshots of 50 GB with only minor changes might use 55 GB total instead of 500 GB.
- Deleting an old snapshot only frees space for files that are not referenced by any other snapshot.
- Subsequent backups typically finish in seconds or minutes rather than hours, since only the differences are transferred.
- Each snapshot appears as a **complete directory tree** — browse or restore any snapshot independently
- Unchanged files share disk space through hardlinks, so 10 snapshots of 50 GB with minor changes might use 55 GB total instead of 500 GB
- Deleting an old snapshot only frees space for files not referenced by other snapshots
- Subsequent backups typically finish in seconds or minutes rather than hours
> **Example**: A first backup of 20 GB takes 45 minutes over SSH. The next day, only 200 MB of files changed — the second backup takes under 2 minutes and uses only 200 MB of additional disk space, while still appearing as a complete 20 GB snapshot.
> **Example**: A first backup of 20 GB takes 45 minutes over SSH. The next day, only 200 MB changed — the second backup takes under 2 minutes and uses only 200 MB of additional disk space, while still appearing as a complete 20 GB snapshot.
## Remote Sources
gniza can pull files from remote machines **without installing anything on them**. This turns gniza into a centralized backup server.
### SSH Source
Back up a remote server by pulling files over SSH:
1. Create a source with `TARGET_SOURCE_TYPE="ssh"`
2. Set the SSH connection details (`TARGET_SOURCE_HOST`, etc.)
3. Set `TARGET_FOLDERS` to the remote paths you want to back up (e.g. `/var/www,/etc`)
gniza connects to the remote server, pulls the specified folders to a local staging area, then pushes the data to the configured destination using the standard snapshot pipeline.
### S3 / Google Drive Source
Pull files from cloud storage before backing them up:
- **S3**: Set `TARGET_SOURCE_TYPE="s3"` with bucket, region, and credentials
- **Google Drive**: Set `TARGET_SOURCE_TYPE="gdrive"` with a service account JSON file
Requires `rclone` to be installed.
## Snapshot Structure
```
$BASE/<hostname>/targets/<target>/snapshots/<YYYY-MM-DDTHHMMSS>/
├── meta.json
├── manifest.txt
├── var/www/
$BASE/<hostname>/targets/<source>/snapshots/<YYYY-MM-DDTHHMMSS>/
├── meta.json # Metadata (source, timestamp, duration, pinned)
├── manifest.txt # File listing
├── var/www/ # Backed-up directories
├── etc/nginx/
└── _mysql/ # MySQL dumps (if enabled)
└── _mysql/ # MySQL dumps (if enabled)
├── dbname.sql.gz
└── _grants.sql.gz
```
During transfer, snapshots are stored in a `.partial` directory. On success, the directory is renamed to the final timestamp. Interrupted backups leave no incomplete snapshots.
## Retention
Retention policies control how many snapshots to keep per source per destination.
- **Global default**: `RETENTION_COUNT` in `gniza.conf` (default: 30)
- **Per-destination override**: `RETENTION_COUNT` in the destination config
- **Per-source override**: `TARGET_RETENTION` in the source config
- **Snapshot pinning**: Pin individual snapshots in `meta.json` to preserve them indefinitely
Retention runs automatically after each successful backup. Run it manually with:
```bash
gniza --cli retention --all
```
## MySQL Backup
gniza can dump MySQL/MariaDB databases alongside file backups.
- **All databases**: Set `TARGET_MYSQL_MODE="all"` to dump every user database
- **Selected databases**: Set `TARGET_MYSQL_MODE="selected"` and list them in `TARGET_MYSQL_DATABASES`
- **Exclude databases**: Use `TARGET_MYSQL_EXCLUDE` to skip specific databases
- **Grants**: User grants are automatically dumped to `_grants.sql.gz`
- **Compression**: All dumps are gzip-compressed
- **Restore**: MySQL dumps are automatically restored unless `--skip-mysql` is passed
Auto-detects `mysqldump` or `mariadb-dump`.
## Scheduling
gniza manages cron entries for automated backups.
```bash
# Via CLI
gniza --cli schedule install # Install all schedules to crontab
gniza --cli schedule show # Show current cron entries
gniza --cli schedule remove # Remove gniza cron entries
```
Cron entries are tagged with `# gniza4linux:<name>` for clean install/removal. Each schedule can target specific sources and destinations.
## Notifications
Email notifications on backup success or failure.
**SMTP** (recommended): Configure `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD`, `SMTP_FROM`, and `SMTP_SECURITY` in `gniza.conf`. Supports TLS, SSL, and plaintext.
**System mail**: Falls back to `mail` or `sendmail` if SMTP is not configured.
**Report includes**: Status, source count (total/succeeded/failed), duration, failed source list, log file path, hostname, and timestamp.
## Web Dashboard
The TUI can be served in a browser via textual-serve with HTTP Basic Auth:
Serve the full TUI in a browser via textual-serve with HTTP Basic Auth.
```bash
# Enable during install (generates admin password)
curl -sSL .../install.sh | sudo bash
# Answer "y" to "Enable web dashboard?"
# Or manually
# Or set up manually:
gniza web install-service # Install systemd service (port 2323)
gniza web start # Start the service
gniza web stop # Stop the service
gniza web status # Check status
```
Access at `http://<server-ip>:2323`. Credentials are stored in `gniza.conf` as `WEB_USER` and `WEB_API_KEY`.
Supports both root (system service) and user (user service) modes.
## Terminal UI
Launch with `gniza` (no arguments). The TUI provides:
- **Sources** — Create, edit, delete backup sources with folder browser
- **Destinations** — Configure SSH, local, S3, or Google Drive destinations with connection testing
- **Backup** — Run backups with source/destination selection
- **Restore** — Browse snapshots and restore to original location or custom directory
- **Running Tasks** — Monitor active backup/restore jobs with live log output
- **Schedules** — Manage cron schedules with time/day pickers
- **Snapshots** — Browse and manage stored snapshots
- **Logs** — View backup history with pagination
- **Settings** — Configure global options
- **Setup Wizard** — Guided first-run configuration
The TUI adapts to terminal width, with an inline documentation panel on wide screens and a help modal on narrow ones.
## Disk Space Safety
gniza checks destination disk usage before and during backups. If usage reaches the configured threshold (default 95%), the backup aborts to prevent filling the disk.
```ini
DISK_USAGE_THRESHOLD=95 # Set to 0 to disable
```
Works with SSH and local destinations.
## Pre/Post Hooks
Run shell commands before and after each backup:
```ini
TARGET_PRE_HOOK="systemctl stop myapp"
TARGET_POST_HOOK="systemctl start myapp"
```
- **Pre-hook failure** aborts the backup
- **Post-hook failure** is logged as a warning
## Testing
```bash

7
lib/remotes.sh
View File

@@ -118,7 +118,7 @@ load_remote() {
REMOTE_PORT="${REMOTE_PORT:-$DEFAULT_REMOTE_PORT}"
REMOTE_USER="${REMOTE_USER:-$DEFAULT_REMOTE_USER}"
REMOTE_AUTH_METHOD="${REMOTE_AUTH_METHOD:-$DEFAULT_REMOTE_AUTH_METHOD}"
REMOTE_KEY="${REMOTE_KEY:-}"
REMOTE_KEY="${REMOTE_KEY:-$HOME/.ssh/id_rsa}"
REMOTE_PASSWORD="${REMOTE_PASSWORD:-}"
REMOTE_BASE="${REMOTE_BASE:-$DEFAULT_REMOTE_BASE}"
BWLIMIT="${BWLIMIT:-$DEFAULT_BWLIMIT}"
@@ -184,10 +184,7 @@ validate_remote() {
((errors++)) || true
fi
else
if [[ -z "$REMOTE_KEY" ]]; then
log_error "Remote '$name': REMOTE_KEY is required"
((errors++)) || true
elif [[ ! -f "$REMOTE_KEY" ]]; then
if [[ ! -f "$REMOTE_KEY" ]]; then
log_error "Remote '$name': REMOTE_KEY file not found: $REMOTE_KEY"
((errors++)) || true
fi

7
tui/widgets/folder_picker.py
View File

@@ -38,7 +38,7 @@ class FolderPicker(ModalScreen[str | None]):
tree = self.query_one("#fp-tree", DirectoryTree)
node = tree.cursor_node
if node and node.data and node.data.path:
return node.data.path
return node.data.path.resolve()
return None
def on_button_pressed(self, event: Button.Pressed) -> None:
@@ -52,6 +52,9 @@ class FolderPicker(ModalScreen[str | None]):
else:
self.dismiss(None)
def on_directory_tree_directory_selected(self, event: DirectoryTree.DirectorySelected) -> None:
self.query_one("#fp-search", Input).value = str(event.path.resolve())
def on_input_submitted(self, event: Input.Submitted) -> None:
if event.input.id == "fp-search":
self._go_to_path()
@@ -60,7 +63,7 @@ class FolderPicker(ModalScreen[str | None]):
raw = self.query_one("#fp-search", Input).value.strip()
if not raw:
return
target = Path(raw).expanduser()
target = Path(raw).expanduser().resolve()
if not target.is_dir():
self.notify(f"Not a directory: {target}", severity="error")
return