Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  Florian WildeandClaude Opus 5.5
|
47ccf90b56 |
@@ -0,0 +1,37 @@
|
||||
# Prüfungen bei jedem Push und Pull Request (Gitea Actions, act_runner).
|
||||
# Gleiche Befehle wie lokal vor einem Release – siehe README, „Entwicklung“.
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
tags: ["v*"]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
pruefen:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
# 3.12 = Mindestversion, 3.13 = Debian 13 (Zielsystem)
|
||||
python: ["3.12", "3.13"]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python }}
|
||||
- name: Abhängigkeiten
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -e ".[dev]"
|
||||
- name: ruff
|
||||
run: |
|
||||
ruff check .
|
||||
ruff format --check .
|
||||
- name: mypy
|
||||
run: mypy src
|
||||
- name: shellcheck
|
||||
run: shellcheck deploy/*.sh
|
||||
- name: pytest (inkl. Migrationen auf befüllter Datenbank)
|
||||
run: pytest -q
|
||||
+32
-1
@@ -8,6 +8,36 @@ Die Version steht an genau einer Stelle: `src/netmover/__init__.py`.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.10.0] – 2026-10-23
|
||||
|
||||
### Hinzugefügt
|
||||
- **Tägliche Datensicherung** (*System → Sicherung*): Datenbank, `secret.key`,
|
||||
`session.key` und `netmover.env` als mit Passphrase verschlüsselte Datei
|
||||
(scrypt + Fernet) auf eine Anbindung (SMB, NFS oder SFTP). Momentaufnahme über die
|
||||
SQLite-Backup-API mit Integritätsprüfung; die Datei wird vor dem Hochladen
|
||||
probeweise geöffnet und atomar geschrieben; ältere NetMover-Sicherungen über die
|
||||
eingestellte Anzahl hinaus werden gelöscht, andere Dateien nie. Verpasste Termine
|
||||
werden nachgeholt, nach einem Fehler stündlich neuer Versuch, erster Fehlschlag
|
||||
per E-Mail/Webhook. Knopf „Jetzt sichern“.
|
||||
- `netmover restore <datei>`: spielt eine Sicherung ein (nur bei gestopptem Dienst),
|
||||
legt die bisherigen Dateien vorher nach `vor-wiederherstellung-<zeit>/` und
|
||||
migriert die Datenbank. `netmover backup <ziel>` sichert von Hand in eine Datei.
|
||||
- Befehl `netmover` im Pfad (`/usr/local/sbin/netmover`, lädt `netmover.env`).
|
||||
- **Kennzahlen für Uptime Kuma**: `/healthz` und `/healthz/rule/<id>` liefern unter
|
||||
`metrics` Dateien, Fehler, Quarantäne und abgebrochene Läufe der letzten 24 Stunden
|
||||
sowie die letzte verschobene Datei – für Kuma-Monitore vom Typ „JSON-Abfrage“.
|
||||
Der Push meldet z. B. „OK · 24 h: 12 Dateien“.
|
||||
- `/healthz/backup`: eigener Monitor; eine seit zwei Tagen überfällige Sicherung gilt
|
||||
als Problem (Statusampel, `/healthz`, Push).
|
||||
- **CI in Gitea** (`.gitea/workflows/ci.yml`): ruff, mypy, shellcheck und Tests auf
|
||||
Python 3.12 und 3.13 bei jedem Push.
|
||||
- Test-Pflicht für Migrationen: jede Migration wird automatisch auf einer befüllten
|
||||
Datenbank hoch- und heruntergestuft.
|
||||
|
||||
### Geändert
|
||||
- README: HTTP im internen Netz als bewusste Entscheidung dokumentiert; TLS über
|
||||
Reverse Proxy bleibt der Weg für nicht vertrauenswürdige Netze.
|
||||
|
||||
## [0.9.0] – 2026-10-15
|
||||
|
||||
### Hinzugefügt
|
||||
@@ -277,7 +307,8 @@ Erste Version.
|
||||
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
||||
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
||||
|
||||
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.9.0...HEAD
|
||||
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.10.0...HEAD
|
||||
[0.10.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.9.0...v0.10.0
|
||||
[0.9.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.8.0...v0.9.0
|
||||
[0.8.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.7.0...v0.8.0
|
||||
[0.7.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.6.0...v0.7.0
|
||||
|
||||
@@ -23,10 +23,39 @@ Zeiten in Ortszeit (Europe/Berlin).
|
||||
| 2026-10-12 | 00:14 | 00:34 | 0:20 | – | Archiv je Regel (0.7.0) |
|
||||
| 2026-10-12 | 00:46 | 00:55 | 0:09 | – | Anbindungen kopieren (0.8.0) |
|
||||
| 2026-10-15 | 09:17 | 09:26 | 0:09 | – | Regeln kopieren (0.9.0) |
|
||||
| **Summe** | | | **4:07** | | |
|
||||
| 2026-10-23 | 13:42 | 14:10 | 0:28 | – | Sicherung, Kuma-Kennzahlen, CI (0.10.0) |
|
||||
| **Summe** | | | **4:35** | | |
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-23 · 13:42–14:10 · Sicherung, Kuma-Kennzahlen, CI (0.10.0)
|
||||
|
||||
**Bestandsaufnahme Live-Server** – 0.9.0 seit 15.10. ohne Neustart, 287 Dateien
|
||||
verschoben, keine Datei gescheitert oder in Quarantäne, Archivkopien vollständig
|
||||
(159 + 122). 18 kurze Pausen, alle wegen SSH zu den Paperless-Servern
|
||||
(„No existing session“, „protocol banner“, Timeout) – automatisches Pausieren und
|
||||
Fortsetzen hat jedes Mal gegriffen. Ursache liegt vermutlich auf den Paperless-Hosts
|
||||
→ Roadmap 0.11.0.
|
||||
|
||||
**Erledigt** – `backup.py` (Momentaufnahme per SQLite-Backup-API, tar.gz,
|
||||
scrypt+Fernet, Probe-Öffnen vor dem Upload, Upload über `Destination.commit`,
|
||||
Ausdünnen), Seite „Sicherung“, `/healthz/backup`, `netmover backup`/`restore`,
|
||||
Wrapper `/usr/local/sbin/netmover`; Kennzahlen unter `metrics` in `/healthz`;
|
||||
Gitea-CI; generischer Migrationstest. 381 Tests. Gegenproben: Ausdünnen ohne
|
||||
Namensfilter und Sicherung ohne Verschlüsselung schlagen jeweils an (2 Tests).
|
||||
|
||||
**Entscheidungen**
|
||||
- *Passphrase verschlüsselt in der DB*: sonst bräuchte die tägliche Sicherung eine
|
||||
Eingabe. Auf dem NAS schützt die Passphrase; auf dem Server ist ohnehin alles da.
|
||||
- *Termin per „fällig?“-Prüfung alle 5 min* statt Cron-Job: holt verpasste Termine
|
||||
nach, braucht kein Umplanen bei Änderungen und regelt den stündlichen Neuversuch.
|
||||
- *Ziel über dieselbe Fabrik wie Regelziele* (`destination_factory`): gleiche
|
||||
Mount-Prüfung, `.part`-Logik und SFTP-Unterstützung, kein zweiter Schreibweg.
|
||||
- *Ausdünnen nur nach Namensschema* – fremde Dateien im Ordner bleiben unangetastet.
|
||||
- *Kein TLS* im internen Netz (Nutzerentscheidung), im README begründet.
|
||||
- *Kuma*: Endpunkte gab es seit 0.3.0; neu sind auslesbare Kennzahlen für
|
||||
„JSON-Abfrage“-Monitore, z. B. Alarm bei `metrics.quarantined_24h != 0`.
|
||||
|
||||
## 2026-10-15 · 09:17–09:26 · Regeln kopieren (0.9.0)
|
||||
|
||||
**Erledigt** – Knopf „Kopieren“, Route `/rules/{id}/copy` (GET Formular, POST
|
||||
|
||||
@@ -388,12 +388,20 @@ findmnt -R /mnt/netmover # was ist wirklich eingehängt?
|
||||
## Sicherheit
|
||||
|
||||
- **Nicht ins Internet stellen** (siehe [Reverse Proxy](#reverse-proxy-mit-tls)).
|
||||
- **HTTP im internen Netz ist eine bewusste Entscheidung**, wenn das Netz
|
||||
vertrauenswürdig ist (Heimnetz, eigenes VLAN): Die Oberfläche läuft dann
|
||||
unverschlüsselt auf Port 8080, die Anmeldung kann per Netz abgeschaltet werden
|
||||
(*Einstellungen → Anmeldung*). Wer im selben Netz mitlesen kann, könnte Passwörter
|
||||
sehen, die man in Formulare eingibt. Für fremde oder gemischte Netze den
|
||||
[Reverse Proxy mit TLS](#reverse-proxy-mit-tls) davorsetzen.
|
||||
- Zugangsdaten liegen nur verschlüsselt in der Datenbank (Fernet, Schlüssel
|
||||
`secret.key`, 0600). Die Oberfläche zeigt sie nie an; in der Prozessliste stehen
|
||||
sie ebenfalls nie, weil `mount.cifs` eine kurzlebige Credentials-Datei bekommt.
|
||||
Ausgaben von `mount` werden geschwärzt, bevor sie protokolliert werden.
|
||||
- **Sichere `secret.key` zusammen mit der Datenbank.** Ohne den Schlüssel sind die
|
||||
gespeicherten Passwörter verloren.
|
||||
gespeicherten Passwörter verloren. Das erledigt die tägliche, mit Passphrase
|
||||
verschlüsselte [Sicherung](docs/ANLEITUNG.md#sicherung) (*System → Sicherung*);
|
||||
Wiederherstellen mit `netmover restore <datei>`.
|
||||
- Anmeldung:
|
||||
- Ein Administratorkonto mit Argon2-Passworthash.
|
||||
- Sperre nach 5 Fehlversuchen.
|
||||
@@ -430,7 +438,8 @@ Aufbau:
|
||||
| `mover/` | Scan, Stabilität, atomares Schreiben, Kollisionen, Buchführung, Regellauf |
|
||||
| `runs.py`, `scheduler.py`, `health.py`, `notify.py` | Protokoll, Zeitsteuerung, Gesundheit, Webhook |
|
||||
| `admin.py`, `keys.py`, `transfer.py`, `web/` | Formulare, Schlüsselverwaltung, Export/Import, Oberfläche |
|
||||
| `health.py`, `kuma.py` | Gesundheitsprüfung, Uptime-Kuma-Push |
|
||||
| `health.py`, `kuma.py` | Gesundheitsprüfung, Kennzahlen, Uptime-Kuma-Push |
|
||||
| `backup.py` | tägliche verschlüsselte Sicherung, `netmover backup`/`restore` |
|
||||
| `docs/` | Anleitung und Bilder; wird als Hilfe-Seite mit ausgeliefert |
|
||||
|
||||
Die gesamte Mover-Logik ist mit dem `FakeMountBackend` ohne root testbar. SFTP
|
||||
@@ -443,6 +452,17 @@ NETMOVER_DATA_DIR=$PWD/data .venv/bin/alembic revision --autogenerate -m "…"
|
||||
```
|
||||
|
||||
Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt.
|
||||
`tests/test_migrations.py` stuft **jede** Migration automatisch auf einer
|
||||
Datenbank mit verknüpften Daten hoch und wieder herunter – eine neue Migration ist
|
||||
damit ohne weiteres Zutun abgedeckt; ergänzt sie Spalten mit Daten, gehört ein
|
||||
eigener Test dazu.
|
||||
|
||||
### CI
|
||||
|
||||
`.gitea/workflows/ci.yml` führt bei jedem Push und Pull Request ruff, mypy,
|
||||
shellcheck und alle Tests aus (Python 3.12 und 3.13). Voraussetzung: Actions sind
|
||||
im Repository aktiviert und ein `act_runner` mit dem Label `ubuntu-latest` ist
|
||||
registriert.
|
||||
|
||||
### Arbeitsweise
|
||||
|
||||
@@ -450,7 +470,7 @@ Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt
|
||||
(`feat:`, `fix:`, `docs:`, `test:`, `chore:` …).
|
||||
- Jede nutzerrelevante Änderung kommt unter `[Unreleased]` in
|
||||
[CHANGELOG.md](CHANGELOG.md). Bei einem Release:
|
||||
1. Den Abschnitt `[Unreleased]` mit Versionsnummer versehen.
|
||||
1. CI grün? Dann den Abschnitt `[Unreleased]` mit Versionsnummer versehen.
|
||||
2. Die Version in `src/netmover/__init__.py` anheben.
|
||||
3. Einen Tag `vX.Y.Z` setzen.
|
||||
- Arbeitszeit und Entscheidungen stehen in [DEVLOG.md](DEVLOG.md).
|
||||
|
||||
+13
-4
@@ -10,10 +10,19 @@ Legende: ✅ erledigt · 🔨 in Arbeit · 📋 geplant · 💡 Idee (noch nicht
|
||||
|
||||
| Status | Thema | Warum |
|
||||
|---|---|---|
|
||||
| 📋 | **CI in Gitea** (Actions): Tests, ruff, mypy, shellcheck bei jedem Push | Fehler fallen vor dem Update auf, nicht auf dem Server |
|
||||
| 📋 | **Migrationstest-Pflicht**: jede neue Migration mit Test auf befüllter Datenbank | Lehre aus dem gescheiterten Update auf 0.2.0 |
|
||||
| 📋 | **Reverse Proxy mit TLS** auf der Ziel-VM (Caddy), danach Bindung an `127.0.0.1` | Anmeldung nicht unverschlüsselt im LAN |
|
||||
| 📋 | **SSH-Ausfälle bei Paperless klären**, Anbindungen auf SSH-Schlüssel umstellen | 18 kurze Ausfälle seit 11.10. („No existing session“, „protocol banner“) |
|
||||
| 📋 | **Statusseite schneller bei hängenden Mounts**: Füllstand zwischenspeichern | Heute bis zu 2 s Wartezeit, wenn ein NFS-Server weg ist |
|
||||
| 📋 | **Benachrichtigung bei Quarantäne** einer Datei | Unbeaufsichtigter Betrieb; heute über Kuma (`metrics.quarantined_24h`) sichtbar |
|
||||
| 📋 | **Kompatibilitätszusage für 1.0**: Einstellungen, Export-Format, Update aus jeder 0.x | Danach keine brechenden Änderungen ohne 2.0 |
|
||||
|
||||
### Weg zu 1.0
|
||||
|
||||
0.10.0 Sicherung + CI → 0.11.0 restliche Punkte oben → ein bis zwei Wochen ruhiger
|
||||
Betrieb → 1.0.0 ohne neue Funktionen.
|
||||
|
||||
**Entschieden:** Kein TLS im internen Netz – HTTP auf Port 8080 ist eine bewusste
|
||||
Entscheidung (README, „Sicherheit“). Der Reverse Proxy bleibt dokumentiert für
|
||||
nicht vertrauenswürdige Netze.
|
||||
|
||||
## Ideen
|
||||
|
||||
@@ -22,12 +31,12 @@ Legende: ✅ erledigt · 🔨 in Arbeit · 📋 geplant · 💡 Idee (noch nicht
|
||||
| 💡 | **Eine Regel, mehrere Ziele** je Quell-Unterordner | Heute: eine Regel pro Unterordner |
|
||||
| 💡 | Metriken für Prometheus (`/metrics`) | Ergänzt `/healthz`, z. B. Dateien pro Tag |
|
||||
| 💡 | Deinstallationsskript | Gegenstück zu `install.sh` |
|
||||
| 💡 | Benachrichtigung bei Quarantäne einer Datei | Heute nur im Protokoll sichtbar |
|
||||
|
||||
## Erledigt
|
||||
|
||||
| Version | Thema |
|
||||
|---|---|
|
||||
| 0.10.0 | Tägliche verschlüsselte Sicherung + `netmover restore`; Kennzahlen für Kuma; CI in Gitea; Migrationstest-Pflicht |
|
||||
| 0.9.0 | Regeln kopieren (Kopie zunächst deaktiviert) |
|
||||
| 0.8.0 | Anbindungen kopieren inklusive Zugangsdaten |
|
||||
| 0.7.0 | Archiv je Regel: Kopie ins Ziel und ins Archiv, Quelle erst danach löschen |
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@
|
||||
services:
|
||||
netmover:
|
||||
build: .
|
||||
image: netmover:0.9.0
|
||||
image: netmover:0.10.0
|
||||
container_name: netmover
|
||||
restart: unless-stopped
|
||||
cap_add:
|
||||
|
||||
+10
-1
@@ -33,8 +33,17 @@ if [ ! -f /etc/netmover/netmover.env ]; then
|
||||
fi
|
||||
install -d -m 0755 /mnt/netmover
|
||||
|
||||
echo "==> Update-Skript"
|
||||
echo "==> Update-Skript und Kommandozeile"
|
||||
install -m 0755 "$(dirname "$0")/update.sh" /usr/local/sbin/netmover-update
|
||||
# netmover backup/restore/… mit derselben Konfiguration wie der Dienst
|
||||
cat > /usr/local/sbin/netmover <<EOF
|
||||
#!/bin/sh
|
||||
set -a
|
||||
[ -r /etc/netmover/netmover.env ] && . /etc/netmover/netmover.env
|
||||
set +a
|
||||
exec "$PREFIX/venv/bin/netmover" "\$@"
|
||||
EOF
|
||||
chmod 0755 /usr/local/sbin/netmover
|
||||
|
||||
echo "==> systemd"
|
||||
install -m 0644 "$(dirname "$0")/netmover.service" /etc/systemd/system/netmover.service
|
||||
|
||||
+76
-2
@@ -10,7 +10,8 @@ stehen im [README](../README.md).
|
||||
**Inhalt:** [Begriffe](#begriffe) · [Erste Anmeldung](#erste-anmeldung) ·
|
||||
[Zugangsdaten](#zugangsdaten) · [Anbindungen](#anbindungen) · [Regeln](#regeln) · [Archiv](#archiv) ·
|
||||
[Probelauf](#probelauf) · [Status](#status) · [Wartungsmodus](#wartungsmodus) · [Protokoll](#protokoll) ·
|
||||
[Einstellungen](#einstellungen) · [Anmeldung](#anmeldung) · [Uptime Kuma](#uptime-kuma) · [Beispiele](#beispiele) · [Häufige Fragen](#fragen)
|
||||
[Einstellungen](#einstellungen) · [Anmeldung](#anmeldung) · [Uptime Kuma](#uptime-kuma) ·
|
||||
[Sicherung](#sicherung) · [Beispiele](#beispiele) · [Häufige Fragen](#fragen)
|
||||
|
||||
## Die vier Begriffe {#begriffe}
|
||||
|
||||
@@ -384,7 +385,36 @@ HTTP-Monitore*. Mit einem Monitor pro Anbindung und Regel sieht man in Kuma sofo
|
||||
|
||||
Als Problem gilt: Scheduler steht, eine aktive Anbindung ist im Fehlerzustand, eine
|
||||
aktive Regel hatte länger als das Dreifache ihres Intervalls keinen Erfolg oder ist
|
||||
dreimal hintereinander gescheitert.
|
||||
dreimal hintereinander gescheitert, oder die [Sicherung](#sicherung) ist überfällig.
|
||||
|
||||
**3. Werte auslesen (JSON-Abfrage)** – `/healthz` und `/healthz/rule/<nummer>`
|
||||
liefern unter `metrics` Kennzahlen der letzten 24 Stunden (Probeläufe zählen nicht):
|
||||
|
||||
| Feld | Bedeutung |
|
||||
|---|---|
|
||||
| `files_24h` | verschobene bzw. kopierte Dateien |
|
||||
| `failed_24h` | Dateien, die gescheitert sind (werden erneut versucht) |
|
||||
| `quarantined_24h` | Dateien, die in Quarantäne verschoben wurden |
|
||||
| `errors_24h` | abgebrochene Läufe (z. B. Anbindung nicht erreichbar) |
|
||||
| `pending_failures` | Dateien, die gerade auf einen neuen Versuch warten |
|
||||
| `last_moved_at` | Zeitpunkt der letzten verschobenen Datei (UTC) |
|
||||
|
||||
`/healthz` enthält die Summe und unter `metrics.rules.<nummer>` jede Regel einzeln.
|
||||
In Kuma: Typ *HTTP(s) – JSON-Abfrage*, dann Ausdruck, Bedingung und erwarteter Wert,
|
||||
zum Beispiel:
|
||||
|
||||
| Monitor | Adresse | Ausdruck | Bedingung | Wert |
|
||||
|---|---|---|---|---|
|
||||
| Gesamtzustand | `/healthz` | `status` | `==` | `ok` |
|
||||
| Alarm bei Quarantäne | `/healthz` | `metrics.quarantined_24h` | `==` | `0` |
|
||||
| Regel ohne Dateifehler | `/healthz/rule/1` | `metrics.failed_24h` | `==` | `0` |
|
||||
| Sicherung | `/healthz/backup` | `status` | `==` | `ok` |
|
||||
|
||||
Ältere Kuma-Versionen (1.x) kennen nur „erwarteter Wert = Ergebnis“; die Beispiele
|
||||
funktionieren dort genauso. Antwortet NetMover mit 503 (irgendein Problem), meldet
|
||||
Kuma auch einen Werte-Monitor als gestört – das ist gewollt.
|
||||
|
||||
Beim Push zeigt Kuma als Nachricht z. B. „OK · 24 h: 12 Dateien“.
|
||||
|
||||
### Anmeldung {#anmeldung}
|
||||
|
||||
@@ -416,6 +446,50 @@ Ausgesperrt oder Passwort vergessen? Auf dem Server
|
||||
`/opt/netmover/venv/bin/netmover reset-password` – setzt das Passwort zurück, schaltet
|
||||
die Anmeldung überall wieder ein und hebt Netz-Sperren auf.
|
||||
|
||||
### Sicherung {#sicherung}
|
||||
|
||||
*System → Sicherung*. NetMover sichert täglich seine Datenbank **und** den Schlüssel
|
||||
`secret.key` – also alle Anbindungen, Regeln, Zugangsdaten und Einstellungen – auf
|
||||
eine Anbindung, am besten eine eigene Freigabe auf dem NAS.
|
||||
|
||||
1. Auf dem NAS eine Freigabe anlegen (z. B. `Backup`), in NetMover als Anbindung
|
||||
eintragen und testen.
|
||||
2. *Sicherung*: Anbindung wählen, Unterordner (z. B. `netmover`), Uhrzeit, wie viele
|
||||
Sicherungen behalten werden, und eine **Passphrase** (mindestens 12 Zeichen).
|
||||
3. *Tägliche Sicherung einschalten*, speichern, **Jetzt sichern**.
|
||||
4. **Passphrase im Passwortmanager ablegen.** Ohne sie ist die Sicherung wertlos.
|
||||
|
||||
Die Sicherung ist mit der Passphrase verschlüsselt (scrypt + Fernet): Wer nur Zugriff
|
||||
aufs NAS hat, kommt nicht an die Zugangsdaten. Vor dem Hochladen öffnet NetMover die
|
||||
Datei probeweise und prüft die Datenbank; hochgeladen wird wie bei Regeln über eine
|
||||
`.part`-Datei. Danach werden ältere NetMover-Sicherungen über die eingestellte Anzahl
|
||||
hinaus gelöscht – andere Dateien im Ordner nie.
|
||||
|
||||
Fällt der Termin aus (Server aus, NAS weg), holt NetMover ihn nach; nach einem
|
||||
Fehler versucht es stündlich erneut und meldet den ersten Fehlschlag per E-Mail bzw.
|
||||
Webhook. Gab es zwei Tage lang keine erfolgreiche Sicherung, gilt das als Problem
|
||||
(Statusampel, `/healthz`, eigener Monitor `/healthz/backup`).
|
||||
|
||||
Eine neue Passphrase gilt nur für künftige Sicherungen; ältere brauchen weiter die
|
||||
alte.
|
||||
|
||||
**Wiederherstellen** (neuer Server, gleiche oder neuere NetMover-Version):
|
||||
|
||||
```sh
|
||||
# NetMover wie im README installieren, dann:
|
||||
systemctl stop netmover
|
||||
netmover restore /pfad/zu/netmover-<host>-<datum>.nmbak # fragt nach der Passphrase
|
||||
systemctl start netmover
|
||||
```
|
||||
|
||||
Die bisherigen Dateien legt `restore` vorher in
|
||||
`/var/lib/netmover/vor-wiederherstellung-<zeit>/` ab. Die Datenbank wird anschließend
|
||||
auf den Stand der installierten Version migriert. Die Sicherungsdatei holst du z. B.
|
||||
mit `scp` vom NAS oder hängst die Freigabe kurz von Hand ein.
|
||||
|
||||
Eine Sicherung von Hand in eine Datei: `netmover backup /root/` (verwendet die
|
||||
Passphrase aus den Einstellungen).
|
||||
|
||||
### Export und Import {#export-import}
|
||||
|
||||
*Konfiguration exportieren* lädt Anbindungen, Regeln und Einstellungen als JSON
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
||||
|
||||
__version__ = "0.9.0"
|
||||
__version__ = "0.10.0"
|
||||
|
||||
@@ -7,21 +7,25 @@ Ende: Scheduler stoppen, Mounts **nicht** aushängen (andere könnten sie nutzen
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from datetime import datetime
|
||||
|
||||
from netmover import kuma
|
||||
from netmover.backup import STALE_AFTER, BackupError, BackupService, load_state
|
||||
from netmover.config import Settings
|
||||
from netmover.crypto import SecretBox, load_or_create_key
|
||||
from netmover.db import make_engine, make_session_factory, upgrade
|
||||
from netmover.health import Health, check_health
|
||||
from netmover.db import make_engine, make_session_factory, session_scope, upgrade
|
||||
from netmover.health import Health, backup_check, check_health
|
||||
from netmover.health import metrics as health_metrics
|
||||
from netmover.logging_setup import event, get_logger
|
||||
from netmover.models import utcnow
|
||||
from netmover.models import Mount, utcnow
|
||||
from netmover.mounting.backend import MountBackend
|
||||
from netmover.mounting.fake import FakeMountBackend
|
||||
from netmover.mounting.linux import LinuxMountBackend
|
||||
from netmover.mounting.manager import MountManager, RestoreResult
|
||||
from netmover.mover.destinations import Destination
|
||||
from netmover.mover.runner import RuleRunner
|
||||
from netmover.notify import Notifier
|
||||
from netmover.notify import Notification, Notifier
|
||||
from netmover.runs import RunService
|
||||
from netmover.scheduler import RuleScheduler
|
||||
from netmover.settings_store import load_settings
|
||||
@@ -60,11 +64,19 @@ class NetMover:
|
||||
)
|
||||
self.runner = RuleRunner(self.session_factory, self.mounts)
|
||||
self.runs = RunService(self.session_factory, self.runner, self.notifier)
|
||||
self.backups = BackupService(
|
||||
settings,
|
||||
self.session_factory,
|
||||
self.backup_destination,
|
||||
self.secrets,
|
||||
notify=lambda kind, title, msg: self.notifier.send(Notification(kind, title, msg)),
|
||||
)
|
||||
self.scheduler = RuleScheduler(
|
||||
self.session_factory,
|
||||
self.runs,
|
||||
retry_mounts=self.mounts.retry_pending,
|
||||
push_health=self.push_health,
|
||||
backup_tick=self.backups.tick,
|
||||
)
|
||||
self.started_at: datetime = utcnow()
|
||||
self.restore_results: list[RestoreResult] = []
|
||||
@@ -85,13 +97,35 @@ class NetMover:
|
||||
"""Scheduler-Job (jede Minute): Zustand an Uptime Kuma melden, falls eingerichtet."""
|
||||
url = load_settings(self.session_factory).kuma_push_url
|
||||
if url:
|
||||
kuma.push(url, self.health())
|
||||
kuma.push(url, self.health(), metrics=health_metrics(self.session_factory))
|
||||
|
||||
def backup_destination(self, mount_id: int, path: str) -> Callable[[], Destination]:
|
||||
"""Ziel der Datensicherung – dieselbe Prüfung wie für Regelziele."""
|
||||
with session_scope(self.session_factory) as s:
|
||||
m = s.get(Mount, mount_id)
|
||||
if m is None:
|
||||
raise BackupError("Die Anbindung für die Sicherung existiert nicht mehr")
|
||||
mtype, name = m.type, m.name
|
||||
mp = self.mounts.check_alive(mount_id)
|
||||
return self.runner.destination_factory("backup_path", mount_id, mtype, name, path, mp, None)
|
||||
|
||||
def health(self) -> Health:
|
||||
return check_health(
|
||||
cfg = load_settings(self.session_factory)
|
||||
health = check_health(
|
||||
self.session_factory,
|
||||
scheduler_running=self.scheduler.running,
|
||||
last_heartbeat=self.scheduler.last_heartbeat,
|
||||
service_started=self.started_at,
|
||||
maintenance=load_settings(self.session_factory).maintenance,
|
||||
maintenance=cfg.maintenance,
|
||||
)
|
||||
st = load_state(self.session_factory)
|
||||
health.add(
|
||||
backup_check(
|
||||
enabled=cfg.backup_enabled and bool(cfg.backup_mount_id),
|
||||
last_success=st.last_success,
|
||||
last_error=st.last_error,
|
||||
stale_after=STALE_AFTER,
|
||||
service_started=self.started_at,
|
||||
)
|
||||
)
|
||||
return health
|
||||
|
||||
@@ -0,0 +1,514 @@
|
||||
"""Datensicherung: Datenbank und Schlüssel täglich auf eine Anbindung (z. B. das NAS).
|
||||
|
||||
Ohne ``secret.key`` sind alle gespeicherten Zugangsdaten verloren – die Sicherung
|
||||
enthält deshalb Datenbank **und** Schlüssel. Damit die Sicherung auf dem NAS nicht
|
||||
selbst zum Schlüsselbund für alle Freigaben wird, verschlüsselt NetMover sie mit
|
||||
einer Passphrase (scrypt + Fernet). Die Passphrase liegt verschlüsselt in der
|
||||
Datenbank, damit die tägliche Sicherung ohne Eingabe läuft; zum Wiederherstellen
|
||||
braucht man sie aus dem Passwortmanager.
|
||||
|
||||
Ablauf einer Sicherung:
|
||||
|
||||
1. Momentaufnahme der Datenbank über die SQLite-Backup-API (konsistent, auch
|
||||
während Regeln laufen) und ``PRAGMA integrity_check``.
|
||||
2. tar.gz mit ``manifest.json``, Datenbank, Schlüsseln und ``netmover.env``.
|
||||
3. Verschlüsseln und sofort probeweise wieder öffnen – eine Sicherung, die sich
|
||||
nicht lesen lässt, wird gar nicht erst hochgeladen.
|
||||
4. Hochladen wie jede andere Datei (``.part`` → prüfen → atomar umbenennen).
|
||||
5. Älteste Sicherungen über ``backup_keep`` hinaus löschen – erst nach Erfolg.
|
||||
|
||||
Format verschlüsselt (``.nmbak``): ``MAGIC`` + 16 Byte Salz + Fernet-Token.
|
||||
Unverschlüsselt: ein gewöhnliches ``.tar.gz``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import sqlite3
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from cryptography.fernet import Fernet, InvalidToken
|
||||
from cryptography.hazmat.primitives.kdf.scrypt import Scrypt
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover import __version__
|
||||
from netmover.config import Settings
|
||||
from netmover.crypto import SecretBox
|
||||
from netmover.logging_setup import event, get_logger
|
||||
from netmover.models import CollisionPolicy, utcnow
|
||||
from netmover.mover.destinations import MetaOptions
|
||||
from netmover.settings_store import AppSettings, get_value, load_settings, set_value
|
||||
from netmover.timeouts import CancelToken, run_with_watchdog
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
MAGIC = b"NETMOVER-BACKUP-1\n"
|
||||
SALT_BYTES = 16
|
||||
SCRYPT_N = 2**15
|
||||
SCRYPT_R = 8
|
||||
SCRYPT_P = 1
|
||||
MIN_PASSPHRASE = 12
|
||||
|
||||
SUFFIX_ENC = ".nmbak"
|
||||
SUFFIX_PLAIN = ".tar.gz"
|
||||
NAME_RE = re.compile(r"^netmover-[A-Za-z0-9._-]+-\d{8}-\d{6}(\.nmbak|\.tar\.gz)$")
|
||||
|
||||
DB_NAME = "netmover.db"
|
||||
KEY_FILES = ("secret.key", "session.key")
|
||||
ENV_NAME = "netmover.env"
|
||||
|
||||
# Wann eine Sicherung als überfällig gilt (für /healthz und Kuma): zwei Tage plus
|
||||
# Puffer – ein einzelner Ausfall des NAS über Nacht ist noch kein Alarm.
|
||||
STALE_AFTER = timedelta(hours=50)
|
||||
RETRY_AFTER = timedelta(hours=1)
|
||||
IDLE_TIMEOUT = 120.0
|
||||
|
||||
STATE_PREFIX = "backup."
|
||||
|
||||
|
||||
class BackupError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
# -- Verschlüsselung -----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _derive(passphrase: str, salt: bytes) -> bytes:
|
||||
kdf = Scrypt(salt=salt, length=32, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P)
|
||||
return base64.urlsafe_b64encode(kdf.derive(passphrase.encode()))
|
||||
|
||||
|
||||
def encrypt(data: bytes, passphrase: str) -> bytes:
|
||||
salt = os.urandom(SALT_BYTES)
|
||||
return MAGIC + salt + Fernet(_derive(passphrase, salt)).encrypt(data)
|
||||
|
||||
|
||||
def is_encrypted(blob: bytes) -> bool:
|
||||
return blob.startswith(MAGIC)
|
||||
|
||||
|
||||
def decrypt(blob: bytes, passphrase: str) -> bytes:
|
||||
if not is_encrypted(blob):
|
||||
raise BackupError("Keine verschlüsselte NetMover-Sicherung")
|
||||
salt = blob[len(MAGIC) : len(MAGIC) + SALT_BYTES]
|
||||
token = blob[len(MAGIC) + SALT_BYTES :]
|
||||
try:
|
||||
return Fernet(_derive(passphrase, salt)).decrypt(token)
|
||||
except InvalidToken:
|
||||
raise BackupError("Passphrase falsch oder Sicherung beschädigt") from None
|
||||
|
||||
|
||||
def passphrase_problem(passphrase: str, confirm: str) -> str | None:
|
||||
if len(passphrase) < MIN_PASSPHRASE:
|
||||
return f"Passphrase: mindestens {MIN_PASSPHRASE} Zeichen."
|
||||
if passphrase != confirm:
|
||||
return "Passphrasen stimmen nicht überein."
|
||||
return None
|
||||
|
||||
|
||||
# -- Archiv bauen und lesen ------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class Archive:
|
||||
manifest: dict[str, Any]
|
||||
files: dict[str, bytes]
|
||||
|
||||
|
||||
def snapshot_db(db_path: Path, target: Path) -> str:
|
||||
"""Konsistente Kopie der laufenden Datenbank; liefert die Alembic-Revision."""
|
||||
src = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
|
||||
dst = sqlite3.connect(target)
|
||||
try:
|
||||
src.backup(dst)
|
||||
status = dst.execute("PRAGMA integrity_check").fetchone()[0]
|
||||
if status != "ok":
|
||||
raise BackupError(f"Datenbank beschädigt: {status}")
|
||||
row = dst.execute("SELECT version_num FROM alembic_version").fetchone()
|
||||
finally:
|
||||
dst.close()
|
||||
src.close()
|
||||
return str(row[0]) if row else ""
|
||||
|
||||
|
||||
def _add(tar: tarfile.TarFile, name: str, data: bytes, mtime: float) -> None:
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size = len(data)
|
||||
info.mode = 0o600
|
||||
info.mtime = int(mtime)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
|
||||
|
||||
def build_archive(settings: Settings, now: datetime | None = None) -> tuple[bytes, dict[str, Any]]:
|
||||
"""tar.gz mit Datenbank, Schlüsseln und Umgebungsdatei (unverschlüsselt)."""
|
||||
now = now or utcnow()
|
||||
with tempfile.TemporaryDirectory(dir=settings.data_dir, prefix=".backup-") as tmp:
|
||||
snap = Path(tmp) / DB_NAME
|
||||
revision = snapshot_db(settings.db_path, snap)
|
||||
files: dict[str, bytes] = {DB_NAME: snap.read_bytes()}
|
||||
for name in KEY_FILES:
|
||||
path = settings.data_dir / name
|
||||
if path.is_file():
|
||||
files[name] = path.read_bytes()
|
||||
if settings.config_file.is_file():
|
||||
files[ENV_NAME] = settings.config_file.read_bytes()
|
||||
manifest: dict[str, Any] = {
|
||||
"format": 1,
|
||||
"version": __version__,
|
||||
"revision": revision,
|
||||
"created": now.isoformat(),
|
||||
"host": socket.gethostname(),
|
||||
"files": sorted(files),
|
||||
}
|
||||
buf = io.BytesIO()
|
||||
stamp = time.time()
|
||||
with tarfile.open(fileobj=buf, mode="w:gz") as tar:
|
||||
_add(tar, "manifest.json", json.dumps(manifest, indent=2).encode(), stamp)
|
||||
for name, data in files.items():
|
||||
_add(tar, name, data, stamp)
|
||||
return buf.getvalue(), manifest
|
||||
|
||||
|
||||
def open_archive(blob: bytes, passphrase: str | None) -> Archive:
|
||||
"""Sicherung lesen und prüfen (Manifest, Datenbank-Integrität)."""
|
||||
if is_encrypted(blob):
|
||||
if not passphrase:
|
||||
raise BackupError("Sicherung ist verschlüsselt – Passphrase angeben")
|
||||
blob = decrypt(blob, passphrase)
|
||||
files: dict[str, bytes] = {}
|
||||
try:
|
||||
with tarfile.open(fileobj=io.BytesIO(blob), mode="r:gz") as tar:
|
||||
for member in tar.getmembers():
|
||||
if not member.isfile() or "/" in member.name or member.name.startswith("."):
|
||||
raise BackupError(f"Unerwarteter Eintrag in der Sicherung: {member.name}")
|
||||
fh = tar.extractfile(member)
|
||||
if fh is not None:
|
||||
files[member.name] = fh.read()
|
||||
except (tarfile.TarError, OSError, EOFError) as exc:
|
||||
raise BackupError(f"Keine lesbare Sicherung: {exc}") from None
|
||||
try:
|
||||
manifest = json.loads(files.pop("manifest.json"))
|
||||
except (KeyError, ValueError):
|
||||
raise BackupError("Sicherung ohne gültiges manifest.json") from None
|
||||
if manifest.get("format") != 1:
|
||||
raise BackupError(f"Unbekanntes Sicherungsformat {manifest.get('format')!r}")
|
||||
if DB_NAME not in files or "secret.key" not in files:
|
||||
raise BackupError("Sicherung unvollständig: Datenbank oder secret.key fehlt")
|
||||
_check_db_bytes(files[DB_NAME])
|
||||
return Archive(manifest, files)
|
||||
|
||||
|
||||
def _check_db_bytes(data: bytes) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / DB_NAME
|
||||
path.write_bytes(data)
|
||||
try:
|
||||
con = sqlite3.connect(path)
|
||||
try:
|
||||
status = con.execute("PRAGMA integrity_check").fetchone()[0]
|
||||
finally:
|
||||
con.close()
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise BackupError(f"Datenbank in der Sicherung unlesbar: {exc}") from None
|
||||
if status != "ok":
|
||||
raise BackupError(f"Datenbank in der Sicherung beschädigt: {status}")
|
||||
|
||||
|
||||
def backup_name(now: datetime, encrypted: bool, host: str | None = None) -> str:
|
||||
host = re.sub(r"[^A-Za-z0-9._-]", "-", host or socket.gethostname()) or "host"
|
||||
local = now.astimezone()
|
||||
return f"netmover-{host}-{local:%Y%m%d-%H%M%S}{SUFFIX_ENC if encrypted else SUFFIX_PLAIN}"
|
||||
|
||||
|
||||
def make_backup(
|
||||
settings: Settings, passphrase: str | None, now: datetime | None = None
|
||||
) -> tuple[bytes, dict[str, Any]]:
|
||||
"""Fertige Sicherung (verschlüsselt, falls Passphrase) – bereits probeweise geöffnet."""
|
||||
raw, manifest = build_archive(settings, now)
|
||||
blob = encrypt(raw, passphrase) if passphrase else raw
|
||||
open_archive(blob, passphrase)
|
||||
return blob, manifest
|
||||
|
||||
|
||||
def prune_names(names: list[str], keep: int) -> list[str]:
|
||||
"""Die zu löschenden Sicherungen: alles über die ``keep`` neuesten hinaus.
|
||||
Nur Dateien mit NetMover-Namensschema – sonst wird nichts angefasst."""
|
||||
ours = sorted((n for n in names if NAME_RE.match(n)), key=_stamp_of, reverse=True)
|
||||
return ours[max(keep, 1) :]
|
||||
|
||||
|
||||
def _stamp_of(name: str) -> str:
|
||||
m = re.search(r"-(\d{8}-\d{6})\.", name)
|
||||
return m.group(1) if m else ""
|
||||
|
||||
|
||||
# -- Wiederherstellen ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _version_tuple(v: str) -> tuple[int, ...]:
|
||||
return tuple(int(p) for p in re.findall(r"\d+", v)[:3])
|
||||
|
||||
|
||||
def restore(settings: Settings, archive: Archive, now: datetime | None = None) -> Path:
|
||||
"""Spielt eine Sicherung ein. Der Dienst muss gestoppt sein. Vorhandene Dateien
|
||||
wandern vorher nach ``data_dir/vor-wiederherstellung-<zeit>/`` – nichts geht
|
||||
verloren. Liefert diesen Ordner."""
|
||||
version = str(archive.manifest.get("version", "0"))
|
||||
if _version_tuple(version) > _version_tuple(__version__):
|
||||
raise BackupError(
|
||||
f"Sicherung stammt von NetMover {version}, installiert ist {__version__} – "
|
||||
"erst aktualisieren, dann wiederherstellen."
|
||||
)
|
||||
now = now or utcnow()
|
||||
data_dir = settings.data_dir
|
||||
data_dir.mkdir(parents=True, exist_ok=True)
|
||||
keep = data_dir / f"vor-wiederherstellung-{now.astimezone():%Y%m%d-%H%M%S}"
|
||||
keep.mkdir(mode=0o700)
|
||||
for name in (DB_NAME, f"{DB_NAME}-wal", f"{DB_NAME}-shm", *KEY_FILES):
|
||||
path = data_dir / name
|
||||
if path.exists():
|
||||
path.rename(keep / name)
|
||||
for name, data in archive.files.items():
|
||||
if name == ENV_NAME:
|
||||
continue
|
||||
_write_private(data_dir / name, data)
|
||||
if ENV_NAME in archive.files:
|
||||
env = settings.config_file
|
||||
if env.is_file() and env.read_bytes() != archive.files[ENV_NAME]:
|
||||
_write_private(keep / ENV_NAME, env.read_bytes())
|
||||
if env.parent.is_dir():
|
||||
_write_private(env, archive.files[ENV_NAME])
|
||||
event(log, "backup.restored", version=version, kept=str(keep))
|
||||
return keep
|
||||
|
||||
|
||||
def _write_private(path: Path, data: bytes) -> None:
|
||||
tmp = path.with_name(f".{path.name}.restore")
|
||||
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "wb") as fh:
|
||||
fh.write(data)
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
# -- Dienst: tägliche Sicherung ------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class BackupState:
|
||||
last_attempt: datetime | None = None
|
||||
last_success: datetime | None = None
|
||||
last_error: str = ""
|
||||
last_file: str = ""
|
||||
last_size: int = 0
|
||||
failures: int = 0
|
||||
|
||||
|
||||
def _dt(raw: str | None) -> datetime | None:
|
||||
if not raw:
|
||||
return None
|
||||
value = json.loads(raw)
|
||||
return datetime.fromisoformat(value) if value else None
|
||||
|
||||
|
||||
def load_state(factory: sessionmaker[Session]) -> BackupState:
|
||||
def get(key: str) -> str | None:
|
||||
return get_value(factory, STATE_PREFIX + key)
|
||||
|
||||
return BackupState(
|
||||
last_attempt=_dt(get("last_attempt")),
|
||||
last_success=_dt(get("last_success")),
|
||||
last_error=json.loads(get("last_error") or '""'),
|
||||
last_file=json.loads(get("last_file") or '""'),
|
||||
last_size=json.loads(get("last_size") or "0"),
|
||||
failures=json.loads(get("failures") or "0"),
|
||||
)
|
||||
|
||||
|
||||
def save_state(factory: sessionmaker[Session], st: BackupState) -> None:
|
||||
values: dict[str, Any] = {
|
||||
"last_attempt": st.last_attempt.isoformat() if st.last_attempt else None,
|
||||
"last_success": st.last_success.isoformat() if st.last_success else None,
|
||||
"last_error": st.last_error,
|
||||
"last_file": st.last_file,
|
||||
"last_size": st.last_size,
|
||||
"failures": st.failures,
|
||||
}
|
||||
for key, value in values.items():
|
||||
set_value(factory, STATE_PREFIX + key, json.dumps(value))
|
||||
|
||||
|
||||
def parse_time(value: str) -> tuple[int, int]:
|
||||
m = re.fullmatch(r"([01]?\d|2[0-3]):([0-5]\d)", value.strip())
|
||||
if not m:
|
||||
raise ValueError("Uhrzeit als HH:MM, z. B. 03:30")
|
||||
return int(m.group(1)), int(m.group(2))
|
||||
|
||||
|
||||
def due_slot(cfg: AppSettings, now: datetime) -> datetime:
|
||||
"""Der letzte Sicherungstermin (Ortszeit des Servers) vor ``now``."""
|
||||
hour, minute = parse_time(cfg.backup_time)
|
||||
local = now.astimezone()
|
||||
slot = local.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||
if slot > local:
|
||||
slot -= timedelta(days=1)
|
||||
return slot
|
||||
|
||||
|
||||
def is_due(cfg: AppSettings, st: BackupState, now: datetime) -> bool:
|
||||
if not cfg.backup_enabled or not cfg.backup_mount_id:
|
||||
return False
|
||||
slot = due_slot(cfg, now)
|
||||
if st.last_success is not None and st.last_success >= slot:
|
||||
return False
|
||||
# Nach einem Fehlschlag stündlich erneut, nicht bei jedem Takt
|
||||
return st.last_attempt is None or now - st.last_attempt >= RETRY_AFTER
|
||||
|
||||
|
||||
@dataclass
|
||||
class BackupResult:
|
||||
ok: bool
|
||||
message: str
|
||||
file: str = ""
|
||||
size: int = 0
|
||||
deleted: int = 0
|
||||
|
||||
|
||||
Notify = Callable[[str, str, str], None] # (event, title, message)
|
||||
|
||||
|
||||
class BackupService:
|
||||
def __init__(
|
||||
self,
|
||||
settings: Settings,
|
||||
session_factory: sessionmaker[Session],
|
||||
destination_for: Callable[[int, str], Callable[[], Any]],
|
||||
secrets: SecretBox,
|
||||
notify: Notify | None = None,
|
||||
) -> None:
|
||||
self.settings = settings
|
||||
self.session_factory = session_factory
|
||||
self._destination_for = destination_for
|
||||
self.secrets = secrets
|
||||
self._notify = notify
|
||||
self._running = False
|
||||
|
||||
@property
|
||||
def running(self) -> bool:
|
||||
return self._running
|
||||
|
||||
def passphrase(self, cfg: AppSettings) -> str | None:
|
||||
if not cfg.backup_passphrase_enc:
|
||||
return None
|
||||
return self.secrets.decrypt(cfg.backup_passphrase_enc)
|
||||
|
||||
def tick(self, now: datetime | None = None) -> BackupResult | None:
|
||||
"""Scheduler-Job: sichert, wenn ein Termin fällig ist."""
|
||||
now = now or utcnow()
|
||||
cfg = load_settings(self.session_factory)
|
||||
if not is_due(cfg, load_state(self.session_factory), now):
|
||||
return None
|
||||
return self.run(now)
|
||||
|
||||
def run(self, now: datetime | None = None) -> BackupResult:
|
||||
if self._running:
|
||||
return BackupResult(False, "Eine Sicherung läuft bereits.")
|
||||
self._running = True
|
||||
try:
|
||||
return self._run(now or utcnow())
|
||||
finally:
|
||||
self._running = False
|
||||
|
||||
def _run(self, now: datetime) -> BackupResult:
|
||||
cfg = load_settings(self.session_factory)
|
||||
st = load_state(self.session_factory)
|
||||
st.last_attempt = now
|
||||
try:
|
||||
if not cfg.backup_mount_id:
|
||||
raise BackupError("Kein Sicherungsziel eingestellt")
|
||||
if not cfg.backup_passphrase_enc:
|
||||
raise BackupError("Keine Passphrase gesetzt")
|
||||
result = self._write(cfg, now)
|
||||
except Exception as exc:
|
||||
message = str(exc) or type(exc).__name__
|
||||
st.failures += 1
|
||||
st.last_error = message
|
||||
save_state(self.session_factory, st)
|
||||
event(log, "backup.failed", level=40, error=message, failures=st.failures)
|
||||
if st.failures == 1 and self._notify:
|
||||
self._notify(
|
||||
"backup.failed",
|
||||
"NetMover: Sicherung fehlgeschlagen",
|
||||
f"{message}\nNächster Versuch in einer Stunde.",
|
||||
)
|
||||
return BackupResult(False, message)
|
||||
recovered = st.failures > 0
|
||||
st.failures = 0
|
||||
st.last_error = ""
|
||||
st.last_success = now
|
||||
st.last_file = result.file
|
||||
st.last_size = result.size
|
||||
save_state(self.session_factory, st)
|
||||
event(log, "backup.done", file=result.file, size=result.size, deleted=result.deleted)
|
||||
if recovered and self._notify:
|
||||
self._notify(
|
||||
"backup.recovered",
|
||||
"NetMover: Sicherung läuft wieder",
|
||||
f"Sicherung {result.file} geschrieben.",
|
||||
)
|
||||
return result
|
||||
|
||||
def _write(self, cfg: AppSettings, now: datetime) -> BackupResult:
|
||||
passphrase = self.passphrase(cfg)
|
||||
blob, _manifest = make_backup(self.settings, passphrase, now)
|
||||
name = backup_name(now, passphrase is not None)
|
||||
make_dest = self._destination_for(cfg.backup_mount_id, cfg.backup_path)
|
||||
|
||||
def work(token: CancelToken) -> BackupResult:
|
||||
dest = make_dest()
|
||||
try:
|
||||
dest.prepare()
|
||||
with tempfile.TemporaryDirectory(dir=self.settings.data_dir) as tmp:
|
||||
local = Path(tmp) / name
|
||||
local.write_bytes(blob)
|
||||
ns = time.time_ns()
|
||||
result = dest.commit(
|
||||
local,
|
||||
len(blob),
|
||||
name,
|
||||
CollisionPolicy.RENAME,
|
||||
MetaOptions(ns, ns, 0o600, None),
|
||||
lambda: None,
|
||||
token,
|
||||
)
|
||||
deleted = 0
|
||||
for old in prune_names(dest.list_names(), cfg.backup_keep):
|
||||
token.check()
|
||||
with contextlib.suppress(OSError):
|
||||
dest.remove(old)
|
||||
deleted += 1
|
||||
return BackupResult(
|
||||
True,
|
||||
f"Sicherung {dest.describe(result.final_rel)} geschrieben",
|
||||
result.final_rel,
|
||||
len(blob),
|
||||
deleted,
|
||||
)
|
||||
finally:
|
||||
dest.close()
|
||||
|
||||
return run_with_watchdog(work, IDLE_TIMEOUT, "Sicherung")
|
||||
+116
-3
@@ -1,14 +1,18 @@
|
||||
"""Kommandozeile: ``netmover serve`` / ``netmover migrate`` / ``netmover version``
|
||||
/ ``netmover reset-password``."""
|
||||
/ ``netmover reset-password`` / ``netmover backup`` / ``netmover restore``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Sequence
|
||||
from collections.abc import Callable, Sequence
|
||||
from pathlib import Path
|
||||
|
||||
from netmover import __version__
|
||||
from netmover.config import get_settings
|
||||
from netmover.config import Settings, get_settings
|
||||
from netmover.db import upgrade
|
||||
from netmover.logging_setup import event, get_logger, setup_logging
|
||||
|
||||
@@ -25,6 +29,17 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
"reset-password",
|
||||
help="Administratorpasswort löschen – beim nächsten Aufruf startet die Einrichtung",
|
||||
)
|
||||
b = sub.add_parser("backup", help="Sicherung in eine lokale Datei oder einen Ordner schreiben")
|
||||
b.add_argument("out", type=Path, help="Zieldatei oder Ordner")
|
||||
b.add_argument(
|
||||
"--unverschluesselt",
|
||||
action="store_true",
|
||||
help="ohne Passphrase (die Datei enthält dann alle Zugangsdaten lesbar)",
|
||||
)
|
||||
r = sub.add_parser("restore", help="Sicherung einspielen (Dienst vorher stoppen)")
|
||||
r.add_argument("file", type=Path, help="Sicherungsdatei (.nmbak oder .tar.gz)")
|
||||
r.add_argument("--passphrase-file", type=Path, help="Passphrase aus Datei statt Abfrage")
|
||||
r.add_argument("--ja", action="store_true", help="ohne Rückfrage einspielen")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
settings = get_settings()
|
||||
@@ -77,8 +92,106 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
engine.dispose()
|
||||
print("Passwort gelöscht. Dienst neu starten; der Einrichtungscode steht im Log.")
|
||||
return 0
|
||||
if args.cmd == "backup":
|
||||
return cmd_backup(settings, args.out, encrypted=not args.unverschluesselt)
|
||||
if args.cmd == "restore":
|
||||
return cmd_restore(settings, args.file, args.passphrase_file, assume_yes=args.ja)
|
||||
return 2
|
||||
|
||||
|
||||
def service_active() -> bool:
|
||||
if shutil.which("systemctl") is None:
|
||||
return False
|
||||
result = subprocess.run(
|
||||
["systemctl", "is-active", "--quiet", "netmover"], # noqa: S607
|
||||
check=False,
|
||||
)
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def _ask_passphrase(confirm: bool) -> str:
|
||||
from netmover.backup import passphrase_problem
|
||||
|
||||
first = getpass.getpass("Passphrase: ")
|
||||
if not confirm:
|
||||
return first
|
||||
problem = passphrase_problem(first, getpass.getpass("Wiederholen: "))
|
||||
if problem:
|
||||
raise SystemExit(problem)
|
||||
return first
|
||||
|
||||
|
||||
def cmd_backup(settings: Settings, out: Path, *, encrypted: bool) -> int:
|
||||
from netmover.backup import backup_name, make_backup
|
||||
from netmover.crypto import SecretBox, load_or_create_key
|
||||
from netmover.db import make_engine, make_session_factory
|
||||
from netmover.models import utcnow
|
||||
from netmover.settings_store import load_settings
|
||||
|
||||
passphrase: str | None = None
|
||||
if encrypted:
|
||||
engine = make_engine(settings.effective_database_url)
|
||||
cfg = load_settings(make_session_factory(engine))
|
||||
engine.dispose()
|
||||
if cfg.backup_passphrase_enc:
|
||||
box = SecretBox(load_or_create_key(settings.secret_key_path))
|
||||
passphrase = box.decrypt(cfg.backup_passphrase_enc)
|
||||
print("Verschlüsselt mit der Passphrase aus den Einstellungen.")
|
||||
else:
|
||||
passphrase = _ask_passphrase(confirm=True)
|
||||
now = utcnow()
|
||||
blob, manifest = make_backup(settings, passphrase, now)
|
||||
target = out / backup_name(now, encrypted) if out.is_dir() else out
|
||||
fd_path = target.with_name(f".{target.name}.part")
|
||||
fd_path.write_bytes(blob)
|
||||
fd_path.chmod(0o600)
|
||||
fd_path.replace(target)
|
||||
print(f"Sicherung geschrieben: {target} ({len(blob)} Bytes, NetMover {manifest['version']})")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_restore(
|
||||
settings: Settings,
|
||||
file: Path,
|
||||
passphrase_file: Path | None,
|
||||
*,
|
||||
assume_yes: bool,
|
||||
active: Callable[[], bool] = service_active,
|
||||
) -> int:
|
||||
from netmover.backup import BackupError, is_encrypted, open_archive, restore
|
||||
|
||||
if active():
|
||||
print("Der Dienst läuft – erst `systemctl stop netmover`, dann wiederherstellen.")
|
||||
return 1
|
||||
blob = file.read_bytes()
|
||||
passphrase: str | None = None
|
||||
if is_encrypted(blob):
|
||||
passphrase = (
|
||||
passphrase_file.read_text().strip() if passphrase_file else _ask_passphrase(False)
|
||||
)
|
||||
try:
|
||||
archive = open_archive(blob, passphrase)
|
||||
except BackupError as exc:
|
||||
print(f"Fehler: {exc}")
|
||||
return 1
|
||||
m = archive.manifest
|
||||
print(
|
||||
f"Sicherung von {m.get('host')} vom {m.get('created')} (NetMover {m.get('version')}),"
|
||||
f" enthält: {', '.join(sorted(archive.files))}"
|
||||
)
|
||||
if not assume_yes and input(f"Nach {settings.data_dir} einspielen? [ja/nein] ") != "ja":
|
||||
print("Abgebrochen, nichts verändert.")
|
||||
return 1
|
||||
try:
|
||||
kept = restore(settings, archive)
|
||||
except BackupError as exc:
|
||||
print(f"Fehler: {exc}")
|
||||
return 1
|
||||
upgrade(settings.effective_database_url)
|
||||
print(f"Wiederhergestellt. Bisherige Dateien liegen in {kept}.")
|
||||
print("Jetzt `systemctl start netmover`.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
||||
@@ -21,6 +21,8 @@ class Settings(BaseSettings):
|
||||
data_dir: Path = Path("/var/lib/netmover")
|
||||
run_dir: Path = Path("/run/netmover")
|
||||
mount_base: Path = Path("/mnt/netmover")
|
||||
# Umgebungsdatei – wird mitgesichert und wiederhergestellt
|
||||
config_file: Path = Path("/etc/netmover/netmover.env")
|
||||
|
||||
host: str = "127.0.0.1"
|
||||
port: int = Field(default=8080, ge=1, le=65535)
|
||||
|
||||
+104
-3
@@ -19,11 +19,21 @@ from __future__ import annotations
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.db import session_scope
|
||||
from netmover.models import Mount, MountState, Rule, utcnow
|
||||
from netmover.models import (
|
||||
FileFailure,
|
||||
ItemStatus,
|
||||
Mount,
|
||||
MountState,
|
||||
Rule,
|
||||
Run,
|
||||
RunItem,
|
||||
RunStatus,
|
||||
utcnow,
|
||||
)
|
||||
|
||||
STALE_FACTOR = 3
|
||||
FAILURES_ALERT = 3
|
||||
@@ -32,7 +42,7 @@ HEARTBEAT_MAX_AGE = timedelta(minutes=3)
|
||||
|
||||
@dataclass
|
||||
class Check:
|
||||
kind: str # "scheduler" | "mount" | "rule"
|
||||
kind: str # "scheduler" | "mount" | "rule" | "backup"
|
||||
name: str
|
||||
ok: bool
|
||||
message: str
|
||||
@@ -146,3 +156,94 @@ def check_health(
|
||||
health.stale_rules.add(r.id)
|
||||
health.add(check)
|
||||
return health
|
||||
|
||||
|
||||
def backup_check(
|
||||
*,
|
||||
enabled: bool,
|
||||
last_success: datetime | None,
|
||||
last_error: str,
|
||||
stale_after: timedelta,
|
||||
service_started: datetime,
|
||||
now: datetime | None = None,
|
||||
) -> Check:
|
||||
"""Datensicherung: überfällig, wenn länger als ``stale_after`` keine gelungen ist
|
||||
(Bezug wie bei Regeln: spätester von letztem Erfolg und Dienststart)."""
|
||||
now = now or utcnow()
|
||||
|
||||
def mk(ok: bool, message: str) -> Check:
|
||||
return Check("backup", "Sicherung", ok, message)
|
||||
|
||||
if not enabled:
|
||||
return mk(True, "Sicherung ausgeschaltet")
|
||||
reference = max(d for d in (last_success, service_started) if d is not None)
|
||||
if now - reference > stale_after:
|
||||
when = "noch nie" if last_success is None else f"seit {last_success:%Y-%m-%d %H:%M} UTC"
|
||||
detail = f": {last_error}" if last_error else ""
|
||||
return mk(False, f"Sicherung überfällig – keine erfolgreiche {when}{detail}")
|
||||
if last_success is None:
|
||||
return mk(True, "Sicherung eingerichtet, noch keine gelaufen")
|
||||
note = f" (letzter Versuch fehlgeschlagen: {last_error})" if last_error else ""
|
||||
return mk(True, f"Letzte Sicherung {last_success:%Y-%m-%d %H:%M} UTC{note}")
|
||||
|
||||
|
||||
METRICS_WINDOW = timedelta(hours=24)
|
||||
COUNTERS = ("files_24h", "failed_24h", "quarantined_24h", "errors_24h", "pending_failures")
|
||||
|
||||
|
||||
def metrics(
|
||||
session_factory: sessionmaker[Session], now: datetime | None = None
|
||||
) -> dict[str, object]:
|
||||
"""Kennzahlen für Uptime Kuma („HTTP(s) – JSON-Abfrage“) und andere Monitore:
|
||||
gesamt und je Regel (Schlüssel = Regel-ID als Text). Probeläufe zählen nicht."""
|
||||
since = (now or utcnow()) - METRICS_WINDOW
|
||||
counts: dict[int, dict[str, int]] = {}
|
||||
last_moved: dict[int, datetime] = {}
|
||||
|
||||
def entry(rule_id: int) -> dict[str, int]:
|
||||
return counts.setdefault(rule_id, dict.fromkeys(COUNTERS, 0))
|
||||
|
||||
with session_scope(session_factory) as s:
|
||||
for (rid,) in s.execute(select(Rule.id)):
|
||||
entry(rid)
|
||||
real = (Run.dry_run.is_(False), Run.started_at >= since)
|
||||
for rid, moved, failed, errors in s.execute(
|
||||
select(
|
||||
Run.rule_id,
|
||||
func.coalesce(func.sum(Run.moved), 0),
|
||||
func.coalesce(func.sum(Run.failed), 0),
|
||||
func.count().filter(Run.status == RunStatus.ERROR),
|
||||
)
|
||||
.where(*real)
|
||||
.group_by(Run.rule_id)
|
||||
):
|
||||
entry(rid).update(files_24h=moved, failed_24h=failed, errors_24h=errors)
|
||||
for rid, n in s.execute(
|
||||
select(Run.rule_id, func.count())
|
||||
.join(RunItem, RunItem.run_id == Run.id)
|
||||
.where(*real, RunItem.status == ItemStatus.QUARANTINED)
|
||||
.group_by(Run.rule_id)
|
||||
):
|
||||
entry(rid)["quarantined_24h"] = n
|
||||
for rid, n in s.execute(
|
||||
select(FileFailure.rule_id, func.count()).group_by(FileFailure.rule_id)
|
||||
):
|
||||
entry(rid)["pending_failures"] = n
|
||||
for rid, latest in s.execute(
|
||||
select(Run.rule_id, func.max(Run.finished_at))
|
||||
.where(Run.dry_run.is_(False), Run.moved > 0)
|
||||
.group_by(Run.rule_id)
|
||||
):
|
||||
if latest is not None:
|
||||
last_moved[rid] = latest
|
||||
|
||||
def iso(d: datetime | None) -> str | None:
|
||||
return d.isoformat() if d else None
|
||||
|
||||
total: dict[str, object] = {k: sum(c[k] for c in counts.values()) for k in COUNTERS}
|
||||
total["last_moved_at"] = iso(max(last_moved.values(), default=None))
|
||||
total["rules"] = {
|
||||
str(rid): {**c, "last_moved_at": iso(last_moved.get(rid))}
|
||||
for rid, c in sorted(counts.items())
|
||||
}
|
||||
return total
|
||||
|
||||
+20
-4
@@ -26,9 +26,20 @@ MAX_MESSAGE = 250
|
||||
Getter = Callable[..., Any]
|
||||
|
||||
|
||||
def push_url(url: str, health: Health) -> str:
|
||||
def ok_message(metrics: dict[str, object] | None) -> str:
|
||||
"""Kurzer Text für die Kuma-Ansicht, z. B. „OK · 24 h: 12 Dateien“."""
|
||||
if not metrics:
|
||||
return "OK"
|
||||
parts = [f"24 h: {metrics.get('files_24h', 0)} Dateien"]
|
||||
failed = metrics.get("failed_24h") or 0
|
||||
if failed:
|
||||
parts.append(f"{failed} Fehler")
|
||||
return "OK · " + ", ".join(parts)
|
||||
|
||||
|
||||
def push_url(url: str, health: Health, metrics: dict[str, object] | None = None) -> str:
|
||||
"""Push-URL mit aktuellem Zustand (vorhandene Parameter werden ersetzt)."""
|
||||
msg = "OK" if health.ok else "; ".join(health.problems)
|
||||
msg = ok_message(metrics) if health.ok else "; ".join(health.problems)
|
||||
if len(msg) > MAX_MESSAGE:
|
||||
msg = msg[: MAX_MESSAGE - 1] + "…"
|
||||
target = httpx.URL(url)
|
||||
@@ -37,11 +48,16 @@ def push_url(url: str, health: Health) -> str:
|
||||
return str(target)
|
||||
|
||||
|
||||
def push(url: str, health: Health, getter: Getter | None = None) -> bool:
|
||||
def push(
|
||||
url: str,
|
||||
health: Health,
|
||||
getter: Getter | None = None,
|
||||
metrics: dict[str, object] | None = None,
|
||||
) -> bool:
|
||||
"""Meldet den Zustand an Kuma. Fehler werden protokolliert, nie weitergereicht –
|
||||
die URL nicht, weil sie das Token enthält."""
|
||||
try:
|
||||
resp = (getter or httpx.get)(push_url(url, health), timeout=10)
|
||||
resp = (getter or httpx.get)(push_url(url, health, metrics), timeout=10)
|
||||
resp.raise_for_status()
|
||||
except Exception as exc:
|
||||
event(log, "kuma.push_failed", level=30, error=type(exc).__name__)
|
||||
|
||||
@@ -109,6 +109,13 @@ class Destination(ABC):
|
||||
def close(self) -> None: # noqa: B027 – absichtlich optional
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def list_names(self) -> list[str]:
|
||||
"""Dateinamen direkt im Zielordner (für das Ausdünnen von Sicherungen)."""
|
||||
|
||||
def remove(self, rel: str) -> None:
|
||||
self._remove(rel)
|
||||
|
||||
# -- Primitive der Unterklassen -------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
@@ -235,6 +242,9 @@ class LocalDestination(Destination):
|
||||
def exists(self, rel: str) -> bool:
|
||||
return os.path.lexists(self._p(rel))
|
||||
|
||||
def list_names(self) -> list[str]:
|
||||
return [e.name for e in os.scandir(self.root) if e.is_file(follow_symlinks=False)]
|
||||
|
||||
def free_bytes(self) -> int | None:
|
||||
st = os.statvfs(self.root)
|
||||
return st.f_bavail * st.f_frsize
|
||||
@@ -421,6 +431,13 @@ class SftpDestination(Destination):
|
||||
return False
|
||||
return True
|
||||
|
||||
def list_names(self) -> list[str]:
|
||||
return [
|
||||
a.filename
|
||||
for a in self.client.listdir_attr(self.base)
|
||||
if a.st_mode is not None and stat.S_ISREG(a.st_mode)
|
||||
]
|
||||
|
||||
def free_bytes(self) -> int | None:
|
||||
usage = self.session.statvfs(self.base)
|
||||
return usage[1] if usage else None
|
||||
|
||||
@@ -198,7 +198,7 @@ class RuleRunner:
|
||||
try:
|
||||
make_source = self._source_factory(plan, src_mp)
|
||||
src_root = src_mp / plan.src_subpath if src_mp and plan.src_subpath else src_mp
|
||||
make_dest = self._target(
|
||||
make_dest = self.destination_factory(
|
||||
"dst_path",
|
||||
plan.dst_mount_id,
|
||||
plan.dst_type,
|
||||
@@ -208,7 +208,7 @@ class RuleRunner:
|
||||
src_root,
|
||||
)
|
||||
make_archive = (
|
||||
self._target(
|
||||
self.destination_factory(
|
||||
"archive_path",
|
||||
plan.archive_mount_id,
|
||||
plan.archive_type,
|
||||
@@ -291,7 +291,7 @@ class RuleRunner:
|
||||
|
||||
return make_local
|
||||
|
||||
def _target(
|
||||
def destination_factory(
|
||||
self,
|
||||
field: str,
|
||||
mount_id: int | None,
|
||||
@@ -302,7 +302,7 @@ class RuleRunner:
|
||||
src_root: Path | None,
|
||||
) -> Callable[[], Destination]:
|
||||
"""Ziel oder Archiv: lokaler Pfad (``mount_id`` None), SFTP oder Kernel-Mount."""
|
||||
what = "Archiv" if field == "archive_path" else "Ziel"
|
||||
what = {"archive_path": "Archiv", "backup_path": "Sicherungsziel"}.get(field, "Ziel")
|
||||
if mount_id is None:
|
||||
root = Path(safe_abspath(field, path))
|
||||
if self._under_mount_base(root):
|
||||
|
||||
@@ -6,7 +6,8 @@
|
||||
* Die Läufe selbst sind blockierend und laufen in einem eigenen Thread-Pool,
|
||||
nie im Event-Loop. Überlappung regelt der Lock im :class:`RuleRunner`, der den
|
||||
übersprungenen Termin auch protokolliert – deshalb ``max_instances`` > 1.
|
||||
* Herzschlag-Job für ``/healthz`` und stündliches Aufräumen der Protokolle.
|
||||
* Herzschlag-Job für ``/healthz``, stündliches Aufräumen der Protokolle und alle
|
||||
5 Minuten die Frage, ob die tägliche Datensicherung fällig ist.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -34,6 +35,7 @@ HEARTBEAT_SECONDS = 30
|
||||
CLEANUP_SECONDS = 3600
|
||||
RETRY_SECONDS = 60
|
||||
PUSH_SECONDS = 60
|
||||
BACKUP_CHECK_SECONDS = 300
|
||||
|
||||
|
||||
def _job_id(rule_id: int) -> str:
|
||||
@@ -48,6 +50,7 @@ class RuleScheduler:
|
||||
*,
|
||||
retry_mounts: Callable[[], object] | None = None,
|
||||
push_health: Callable[[], object] | None = None,
|
||||
backup_tick: Callable[[], object] | None = None,
|
||||
rng: random.Random | None = None,
|
||||
workers: int = 8,
|
||||
) -> None:
|
||||
@@ -55,6 +58,7 @@ class RuleScheduler:
|
||||
self.service = service
|
||||
self._retry_mounts = retry_mounts
|
||||
self._push_health = push_health
|
||||
self._backup_tick = backup_tick
|
||||
self._rng = rng or random.Random() # noqa: S311 – nur Startversatz
|
||||
self._executor = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="nm-run")
|
||||
self.scheduler = AsyncIOScheduler(
|
||||
@@ -85,6 +89,14 @@ class RuleScheduler:
|
||||
self.scheduler.add_job(
|
||||
self._push, IntervalTrigger(seconds=PUSH_SECONDS), id="kuma-push"
|
||||
)
|
||||
if self._backup_tick is not None:
|
||||
self.scheduler.add_job(
|
||||
self._backup,
|
||||
IntervalTrigger(
|
||||
seconds=BACKUP_CHECK_SECONDS, start_date=utcnow() + timedelta(minutes=2)
|
||||
),
|
||||
id="backup",
|
||||
)
|
||||
self.sync()
|
||||
event(log, "scheduler.started", rules=len(self._intervals))
|
||||
|
||||
@@ -166,6 +178,11 @@ class RuleScheduler:
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(self._executor, self._push_health)
|
||||
|
||||
async def _backup(self) -> None:
|
||||
if self._backup_tick is not None:
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(self._executor, self._backup_tick)
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(self._executor, cleanup, self.session_factory)
|
||||
|
||||
@@ -52,6 +52,13 @@ class AppSettings:
|
||||
auth_mode: str = "always"
|
||||
trusted_networks: str = ""
|
||||
allowed_networks: str = ""
|
||||
# Datensicherung (backup.py); Passphrase Fernet-verschlüsselt
|
||||
backup_enabled: bool = False
|
||||
backup_mount_id: int = 0 # 0 = kein Ziel
|
||||
backup_path: str = ""
|
||||
backup_time: str = "03:30"
|
||||
backup_keep: int = 14
|
||||
backup_passphrase_enc: str = ""
|
||||
|
||||
|
||||
_PREFIX = "app."
|
||||
|
||||
+126
-3
@@ -50,6 +50,9 @@ from netmover.admin import (
|
||||
save_rule,
|
||||
)
|
||||
from netmover.app_state import NetMover
|
||||
from netmover.backup import load_state as load_backup_state
|
||||
from netmover.backup import parse_time as parse_backup_time
|
||||
from netmover.backup import passphrase_problem
|
||||
from netmover.config import Settings, get_settings
|
||||
from netmover.credentials import (
|
||||
create_password,
|
||||
@@ -62,6 +65,7 @@ from netmover.credentials import (
|
||||
update_password,
|
||||
)
|
||||
from netmover.db import session_scope
|
||||
from netmover.health import metrics as health_metrics
|
||||
from netmover.kuma import push as kuma_push
|
||||
from netmover.logging_setup import event, get_logger
|
||||
from netmover.mail import SECURITY_MODES, MailConfigError, parse_recipients, valid_address
|
||||
@@ -85,6 +89,7 @@ from netmover.mounting.report import CheckReport
|
||||
from netmover.notify import Notification
|
||||
from netmover.settings_store import AppSettings, NotifyMode, load_settings, save_settings
|
||||
from netmover.transfer import apply_imported_settings, export_config, import_config
|
||||
from netmover.validation import ValidationError, safe_relpath
|
||||
from netmover.web import auth
|
||||
|
||||
log = get_logger(__name__)
|
||||
@@ -186,6 +191,7 @@ HELP_ANCHORS = {
|
||||
"rules": "regeln",
|
||||
"log": "protokoll",
|
||||
"settings": "einstellungen",
|
||||
"backup": "sicherung",
|
||||
}
|
||||
|
||||
|
||||
@@ -199,6 +205,7 @@ class NavState:
|
||||
rules: int
|
||||
rules_active: int
|
||||
maintenance: bool = False
|
||||
backup_ok: bool = True
|
||||
|
||||
|
||||
def _nav_state(request: Request) -> NavState | None:
|
||||
@@ -223,6 +230,7 @@ def _nav_state(request: Request) -> NavState | None:
|
||||
rules=len(rules),
|
||||
rules_active=sum(1 for (en,) in rules if en),
|
||||
maintenance=health.maintenance,
|
||||
backup_ok=(c := health.find("backup")) is None or c.ok,
|
||||
)
|
||||
|
||||
|
||||
@@ -306,16 +314,20 @@ async def healthz(request: Request) -> JSONResponse:
|
||||
"last_heartbeat": beat.isoformat() if beat else None,
|
||||
},
|
||||
"checks": [c.as_dict() for c in health.checks],
|
||||
"metrics": await run_in_threadpool(health_metrics, state.session_factory),
|
||||
}
|
||||
return JSONResponse(body, status_code=200 if health.ok else 503)
|
||||
|
||||
|
||||
async def _single_check(request: Request, kind: str, **key: Any) -> JSONResponse:
|
||||
async def _single_check(
|
||||
request: Request, kind: str, extra: dict[str, Any] | None = None, **key: Any
|
||||
) -> JSONResponse:
|
||||
health = await run_in_threadpool(nm(request).health)
|
||||
check = health.find(kind, **key)
|
||||
if check is None:
|
||||
return JSONResponse({"status": "unknown", "message": "nicht gefunden"}, status_code=404)
|
||||
body = {"status": "ok" if check.ok else "error", "name": check.name, "message": check.message}
|
||||
body.update(extra or {})
|
||||
return JSONResponse(body, status_code=200 if check.ok else 503)
|
||||
|
||||
|
||||
@@ -327,8 +339,24 @@ async def healthz_mount(request: Request, slug: str) -> JSONResponse:
|
||||
|
||||
@public.get("/healthz/rule/{rule_id}")
|
||||
async def healthz_rule(request: Request, rule_id: int) -> JSONResponse:
|
||||
"""Eigener Kuma-Monitor pro Regel."""
|
||||
return await _single_check(request, "rule", id=rule_id)
|
||||
"""Eigener Kuma-Monitor pro Regel – mit Kennzahlen der letzten 24 Stunden."""
|
||||
all_metrics = await run_in_threadpool(health_metrics, nm(request).session_factory)
|
||||
rules = all_metrics["rules"]
|
||||
rule_metrics = rules.get(str(rule_id)) if isinstance(rules, dict) else None
|
||||
return await _single_check(request, "rule", {"metrics": rule_metrics}, id=rule_id)
|
||||
|
||||
|
||||
@public.get("/healthz/backup")
|
||||
async def healthz_backup(request: Request) -> JSONResponse:
|
||||
"""Eigener Kuma-Monitor für die Datensicherung."""
|
||||
st = await run_in_threadpool(load_backup_state, nm(request).session_factory)
|
||||
extra = {
|
||||
"last_success": st.last_success.isoformat() if st.last_success else None,
|
||||
"last_file": st.last_file or None,
|
||||
"last_size": st.last_size,
|
||||
"failures": st.failures,
|
||||
}
|
||||
return await _single_check(request, "backup", extra)
|
||||
|
||||
|
||||
@public.get("/setup", response_class=HTMLResponse)
|
||||
@@ -1128,6 +1156,7 @@ def _monitor_urls(request: Request) -> list[tuple[str, str]]:
|
||||
(f"Regel „{n}“", f"{base}/healthz/rule/{rid}")
|
||||
for rid, n in s.execute(select(Rule.id, Rule.name).order_by(Rule.name))
|
||||
]
|
||||
urls.append(("Sicherung", f"{base}/healthz/backup"))
|
||||
return urls
|
||||
|
||||
|
||||
@@ -1213,6 +1242,100 @@ async def settings_save(request: Request) -> Response:
|
||||
return redirect("/settings")
|
||||
|
||||
|
||||
# -- Datensicherung -------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _backup_page(
|
||||
request: Request, cfg: AppSettings, errors: dict[str, str], status_code: int = 200
|
||||
) -> Response:
|
||||
state = nm(request)
|
||||
with state.session_factory() as s:
|
||||
mounts = [
|
||||
(m.id, f"{m.name} ({m.type.value})")
|
||||
for m in s.scalars(select(Mount).order_by(Mount.name))
|
||||
]
|
||||
health = state.health()
|
||||
return render(
|
||||
request,
|
||||
"backup.html",
|
||||
status_code=status_code,
|
||||
active="backup",
|
||||
cfg=cfg,
|
||||
errors=errors,
|
||||
mounts=mounts,
|
||||
st=load_backup_state(state.session_factory),
|
||||
check=health.find("backup"),
|
||||
running=state.backups.running,
|
||||
monitor_url=f"{str(request.base_url).rstrip('/')}/healthz/backup",
|
||||
)
|
||||
|
||||
|
||||
@protected.get("/backup", response_class=HTMLResponse)
|
||||
async def backup_page(request: Request) -> Response:
|
||||
return _backup_page(request, load_settings(nm(request).session_factory), {})
|
||||
|
||||
|
||||
@protected.post("/backup", response_class=HTMLResponse)
|
||||
async def backup_save(request: Request) -> Response:
|
||||
state = nm(request)
|
||||
form = _form_dict(await request.form())
|
||||
cfg = load_settings(state.session_factory)
|
||||
errors: dict[str, str] = {}
|
||||
cfg.backup_enabled = form.get("backup_enabled") == "on"
|
||||
try:
|
||||
cfg.backup_mount_id = int(form.get("backup_mount_id") or 0)
|
||||
except ValueError:
|
||||
cfg.backup_mount_id = 0
|
||||
with state.session_factory() as s:
|
||||
if cfg.backup_mount_id and s.get(Mount, cfg.backup_mount_id) is None:
|
||||
errors["backup_mount_id"] = "Anbindung existiert nicht."
|
||||
try:
|
||||
cfg.backup_path = safe_relpath("backup_path", form.get("backup_path") or "")
|
||||
except ValidationError as exc:
|
||||
errors["backup_path"] = str(exc)
|
||||
cfg.backup_time = (form.get("backup_time") or "03:30").strip()
|
||||
try:
|
||||
parse_backup_time(cfg.backup_time)
|
||||
except ValueError as exc:
|
||||
errors["backup_time"] = str(exc)
|
||||
try:
|
||||
cfg.backup_keep = int(form.get("backup_keep") or 14)
|
||||
if not 1 <= cfg.backup_keep <= 365:
|
||||
raise ValueError
|
||||
except ValueError:
|
||||
errors["backup_keep"] = "Zwischen 1 und 365."
|
||||
passphrase = form.get("passphrase") or ""
|
||||
if passphrase or form.get("passphrase_confirm"):
|
||||
problem = passphrase_problem(passphrase, form.get("passphrase_confirm") or "")
|
||||
if problem:
|
||||
errors["passphrase"] = problem
|
||||
else:
|
||||
cfg.backup_passphrase_enc = state.secrets.encrypt(passphrase)
|
||||
if cfg.backup_enabled and not cfg.backup_mount_id:
|
||||
errors["backup_mount_id"] = "Für die Sicherung eine Anbindung wählen."
|
||||
if cfg.backup_enabled and not cfg.backup_passphrase_enc and "passphrase" not in errors:
|
||||
errors["passphrase"] = "Für die Sicherung eine Passphrase festlegen." # noqa: S105
|
||||
if errors:
|
||||
return _backup_page(request, cfg, errors, status_code=400)
|
||||
save_settings(state.session_factory, cfg)
|
||||
flash(request, "Sicherung gespeichert.")
|
||||
if passphrase:
|
||||
flash(
|
||||
request,
|
||||
"Passphrase jetzt im Passwortmanager ablegen – ohne sie lässt sich keine "
|
||||
"Sicherung wiederherstellen.",
|
||||
"warn",
|
||||
)
|
||||
return redirect("/backup")
|
||||
|
||||
|
||||
@protected.post("/backup/run")
|
||||
async def backup_run(request: Request) -> Response:
|
||||
result = await run_in_threadpool(nm(request).backups.run)
|
||||
flash(request, result.message, "ok" if result.ok else "error")
|
||||
return redirect("/backup")
|
||||
|
||||
|
||||
def auth_mask(url: str) -> str:
|
||||
"""Webhook-URL nur gekürzt anzeigen – sie enthält oft ein Token."""
|
||||
if not url:
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import field, select, check, csrf_field %}
|
||||
{% block title %}Sicherung{% endblock %}
|
||||
{% block content %}
|
||||
<h1>Sicherung</h1>
|
||||
<p class="muted">Täglich Datenbank und Schlüssel – also alle Anbindungen, Regeln, Zugangsdaten und Einstellungen –
|
||||
verschlüsselt auf eine Anbindung, z. B. eine eigene Freigabe auf dem NAS.</p>
|
||||
|
||||
<div class="card stack">
|
||||
<h2>Zustand</h2>
|
||||
{% if check %}<p class="msg {{ 'ok' if check.ok else 'error' }}">{{ check.message }}</p>{% endif %}
|
||||
<table class="small"><tbody>
|
||||
<tr><td>Letzte erfolgreiche</td><td>{{ st.last_success|localdt if st.last_success else '–' }}</td></tr>
|
||||
<tr><td>Datei</td><td class="mono">{{ st.last_file or '–' }}{% if st.last_size %} ({{ st.last_size|bytes }}){% endif %}</td></tr>
|
||||
<tr><td>Letzter Versuch</td><td>{{ st.last_attempt|localdt if st.last_attempt else '–' }}</td></tr>
|
||||
{% if st.last_error %}<tr><td>Letzter Fehler</td><td>{{ st.last_error }}{% if st.failures > 1 %} ({{ st.failures }}× in Folge){% endif %}</td></tr>{% endif %}
|
||||
</tbody></table>
|
||||
<form class="row" method="post" action="/backup/run">
|
||||
{{ csrf_field(csrf) }}
|
||||
<button {% if running or not cfg.backup_mount_id or not cfg.backup_passphrase_enc %}disabled{% endif %}>{{ 'Sicherung läuft …' if running else 'Jetzt sichern' }}</button>
|
||||
<span class="hint">Kuma-Monitor: <code>{{ monitor_url }}</code></span>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
<h2>Einstellungen <a class="legend-help" href="/help#sicherung" title="Hilfe">?</a></h2>
|
||||
<form class="form card stack" method="post" action="/backup">
|
||||
{{ csrf_field(csrf) }}
|
||||
<div class="fields">
|
||||
{{ check('backup_enabled', 'Tägliche Sicherung einschalten', cfg.backup_enabled, wide=true) }}
|
||||
{{ select('backup_mount_id', 'Ziel-Anbindung', [(0, '– bitte wählen –')] + mounts, cfg.backup_mount_id, errors=errors, hint='SMB, NFS oder SFTP – am besten eine eigene Freigabe, getrennt von Scans und Archiv.') }}
|
||||
{{ field('backup_path', 'Unterordner', cfg.backup_path, errors=errors, placeholder='z. B. netmover', hint='Wird angelegt, falls nötig. Leer = direkt in der Anbindung.') }}
|
||||
{{ field('backup_time', 'Uhrzeit', cfg.backup_time, type='time', errors=errors, hint='Ortszeit des Servers. Verpasste Sicherungen werden nachgeholt, nach einem Fehler stündlich neuer Versuch.') }}
|
||||
{{ field('backup_keep', 'Anzahl behalten', cfg.backup_keep, type='number', errors=errors, attrs='min="1" max="365"', hint='Ältere NetMover-Sicherungen im Ordner werden gelöscht – andere Dateien nie.') }}
|
||||
<div class="field{{ ' has-error' if errors.get('passphrase') }}"><label for="f-passphrase">Passphrase</label>
|
||||
<input id="f-passphrase" type="password" name="passphrase" autocomplete="new-password" minlength="12" placeholder="{{ '••••••' if cfg.backup_passphrase_enc }}">
|
||||
{% if errors.get('passphrase') %}<span class="error">{{ errors['passphrase'] }}</span>{% endif %}
|
||||
<span class="hint">{{ 'Leer lassen = unverändert. ' if cfg.backup_passphrase_enc }}Mindestens 12 Zeichen.</span></div>
|
||||
<div class="field"><label for="f-passphrase2">Passphrase wiederholen</label>
|
||||
<input id="f-passphrase2" type="password" name="passphrase_confirm" autocomplete="new-password"></div>
|
||||
</div>
|
||||
<p class="msg warn small">Die Passphrase gehört in deinen Passwortmanager. Fällt der Server aus, ist sie der einzige Weg an die Sicherung –
|
||||
NetMover kann sie dann nicht mehr verraten. Eine neue Passphrase gilt für künftige Sicherungen; ältere Dateien brauchen weiter die alte.</p>
|
||||
<div class="row"><button class="primary">Speichern</button></div>
|
||||
</form>
|
||||
|
||||
<h2>Wiederherstellen</h2>
|
||||
<div class="card stack small">
|
||||
<p>Auf dem (neu installierten) Server, gleiche oder neuere NetMover-Version:</p>
|
||||
<pre class="mono">systemctl stop netmover
|
||||
netmover restore /pfad/zur/netmover-….nmbak
|
||||
systemctl start netmover</pre>
|
||||
<p class="muted">Die bisherigen Dateien bleiben in <code>/var/lib/netmover/vor-wiederherstellung-…</code> erhalten.
|
||||
Ausführlich: <a href="/help#sicherung">Hilfe → Sicherung</a>.</p>
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -47,6 +47,7 @@
|
||||
<div class="nav-group">
|
||||
<div class="nav-title">System</div>
|
||||
{{ item('settings', '/settings', 'Einstellungen') }}
|
||||
{{ item('backup', '/backup', 'Sicherung', '' if not nav or nav.backup_ok else 'überfällig', '' if not nav or nav.backup_ok else 'bad') }}
|
||||
{{ item('help', '/help', 'Hilfe') }}
|
||||
</div>
|
||||
<div class="nav-foot">
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
{%- elif name == 'rules' -%}<path d="M4 7h13l-3-3M20 17H7l3 3"/>
|
||||
{%- elif name == 'log' -%}<path d="M8 6h13M8 12h13M8 18h13M3 6h.01M3 12h.01M3 18h.01"/>
|
||||
{%- elif name == 'settings' -%}<circle cx="12" cy="12" r="3"/><path d="M12 2v3M12 19v3M4.2 4.2l2.1 2.1M17.7 17.7l2.1 2.1M2 12h3M19 12h3M4.2 19.8l2.1-2.1M17.7 6.3l2.1-2.1"/>
|
||||
{%- elif name == 'backup' -%}<path d="M4 7c0-1.7 3.6-3 8-3s8 1.3 8 3-3.6 3-8 3-8-1.3-8-3zM4 7v10c0 1.7 3.6 3 8 3s8-1.3 8-3V7M4 12c0 1.7 3.6 3 8 3s8-1.3 8-3"/>
|
||||
{%- elif name == 'help' -%}<circle cx="12" cy="12" r="9"/><path d="M9.5 9a2.5 2.5 0 1 1 3.5 2.3c-.7.3-1 .9-1 1.7M12 17h.01"/>
|
||||
{%- elif name == 'logout' -%}<path d="M15 4h4v16h-4M10 8l-4 4 4 4M6 12h11"/>
|
||||
{%- elif name == 'menu' -%}<path d="M4 6h16M4 12h16M4 18h16"/>
|
||||
|
||||
@@ -0,0 +1,307 @@
|
||||
"""0.10.0: Datensicherung – verschlüsselt auf eine Anbindung, Wiederherstellen per CLI."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import sqlite3
|
||||
import tarfile
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from netmover import __version__, backup
|
||||
from netmover.app_state import NetMover
|
||||
from netmover.backup import (
|
||||
BackupError,
|
||||
is_due,
|
||||
load_state,
|
||||
make_backup,
|
||||
open_archive,
|
||||
prune_names,
|
||||
)
|
||||
from netmover.cli import cmd_backup, cmd_restore
|
||||
from netmover.config import Settings
|
||||
from netmover.models import Mount, MountType
|
||||
from netmover.mounting.fake import FakeMountBackend
|
||||
from netmover.settings_store import AppSettings, load_settings, save_settings
|
||||
from tests.webutil import post
|
||||
|
||||
PASS = "korrekt-pferd-batterie"
|
||||
NOW = datetime(2026, 10, 23, 12, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
def _nas(state: NetMover, backend: FakeMountBackend) -> tuple[int, Path]:
|
||||
"""CIFS-Anbindung „Backup“ einhängen; liefert (id, Freigabe auf dem „NAS“)."""
|
||||
share = backend.export("nas", "backup")
|
||||
with state.session_factory() as s:
|
||||
m = Mount(
|
||||
name="Backup",
|
||||
slug="backup",
|
||||
type=MountType.CIFS,
|
||||
server="nas",
|
||||
share="backup",
|
||||
mountpoint=str(state.settings.mount_base.resolve() / "backup"),
|
||||
)
|
||||
s.add(m)
|
||||
s.commit()
|
||||
mid = m.id
|
||||
state.mounts.mount(mid)
|
||||
return mid, share
|
||||
|
||||
|
||||
def _configure(state: NetMover, mount_id: int, **kw: Any) -> AppSettings:
|
||||
cfg = load_settings(state.session_factory)
|
||||
cfg.backup_enabled = True
|
||||
cfg.backup_mount_id = mount_id
|
||||
cfg.backup_path = "netmover"
|
||||
cfg.backup_passphrase_enc = state.secrets.encrypt(PASS)
|
||||
for k, v in kw.items():
|
||||
setattr(cfg, k, v)
|
||||
save_settings(state.session_factory, cfg)
|
||||
return cfg
|
||||
|
||||
|
||||
# -- Format ----------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_encrypted_roundtrip_and_wrong_passphrase(state: NetMover) -> None:
|
||||
blob, manifest = make_backup(state.settings, PASS)
|
||||
assert backup.is_encrypted(blob)
|
||||
# Weder Schlüssel noch Datenbank im Klartext
|
||||
key = state.settings.secret_key_path.read_bytes().strip()
|
||||
assert key not in blob and b"SQLite format 3" not in blob
|
||||
archive = open_archive(blob, PASS)
|
||||
assert archive.manifest["version"] == __version__
|
||||
assert archive.manifest["revision"]
|
||||
assert archive.files["secret.key"].strip() == key
|
||||
assert archive.files["netmover.db"].startswith(b"SQLite format 3")
|
||||
assert manifest["files"] == sorted(archive.files)
|
||||
with pytest.raises(BackupError, match="Passphrase falsch"):
|
||||
open_archive(blob, "falsche-passphrase")
|
||||
with pytest.raises(BackupError, match="verschlüsselt"):
|
||||
open_archive(blob, None)
|
||||
|
||||
|
||||
def test_tampered_backup_is_rejected(state: NetMover) -> None:
|
||||
blob = bytearray(make_backup(state.settings, PASS)[0])
|
||||
blob[-10] ^= 0x01
|
||||
with pytest.raises(BackupError):
|
||||
open_archive(bytes(blob), PASS)
|
||||
|
||||
|
||||
def test_env_file_is_included(state: NetMover, tmp_path: Path) -> None:
|
||||
env = tmp_path / "netmover.env"
|
||||
env.write_text("NETMOVER_PORT=8080\n")
|
||||
settings = state.settings.model_copy(update={"config_file": env})
|
||||
archive = open_archive(make_backup(settings, None)[0], None)
|
||||
assert archive.files["netmover.env"] == b"NETMOVER_PORT=8080\n"
|
||||
|
||||
|
||||
def test_archive_without_secret_key_is_rejected() -> None:
|
||||
buf = io.BytesIO()
|
||||
with tarfile.open(fileobj=buf, mode="w:gz") as tar:
|
||||
data = b'{"format": 1}'
|
||||
info = tarfile.TarInfo("manifest.json")
|
||||
info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
with pytest.raises(BackupError, match="unvollständig"):
|
||||
open_archive(buf.getvalue(), None)
|
||||
|
||||
|
||||
def test_prune_keeps_newest_and_ignores_foreign_files() -> None:
|
||||
ours = [f"netmover-h-202610{d:02d}-033000.nmbak" for d in range(1, 8)]
|
||||
names = [*ours, "urlaub.jpg", "netmover-notizen.txt"]
|
||||
assert sorted(prune_names(names, 3)) == ours[:4]
|
||||
assert prune_names(names, 10) == []
|
||||
|
||||
|
||||
# -- Termin ----------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _cfg(time: str = "03:30") -> AppSettings:
|
||||
return AppSettings(backup_enabled=True, backup_mount_id=1, backup_time=time)
|
||||
|
||||
|
||||
def test_due_after_slot_once_per_day_and_hourly_retry() -> None:
|
||||
local = datetime(2026, 10, 23, 2, 0).astimezone() # 02:00 Ortszeit
|
||||
st = backup.BackupState()
|
||||
yesterday_ok = backup.BackupState(last_success=local - timedelta(hours=3))
|
||||
assert is_due(_cfg(), st, local) # noch nie gesichert → nachholen
|
||||
assert not is_due(_cfg(), yesterday_ok, local) # 23:00 gestern nach 03:30 gestern
|
||||
later = local + timedelta(hours=2) # 04:00 → heutiger Termin ist fällig
|
||||
assert is_due(_cfg(), yesterday_ok, later)
|
||||
failed = backup.BackupState(
|
||||
last_success=yesterday_ok.last_success, last_attempt=later - timedelta(minutes=10)
|
||||
)
|
||||
assert not is_due(_cfg(), failed, later) # gerade gescheitert → nicht jeden Takt
|
||||
assert is_due(_cfg(), failed, later + timedelta(hours=1))
|
||||
assert not is_due(AppSettings(backup_enabled=False, backup_mount_id=1), st, later)
|
||||
with pytest.raises(ValueError):
|
||||
backup.parse_time("25:00")
|
||||
|
||||
|
||||
# -- Dienst ----------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_service_writes_encrypted_backup_to_mount_and_prunes(
|
||||
state: NetMover, backend: FakeMountBackend
|
||||
) -> None:
|
||||
mid, share = _nas(state, backend)
|
||||
_configure(state, mid, backup_keep=3)
|
||||
(share / "netmover").mkdir()
|
||||
(share / "netmover" / "fremd.txt").write_text("bleibt")
|
||||
for i in range(5):
|
||||
result = state.backups.run(NOW + timedelta(days=i))
|
||||
assert result.ok, result.message
|
||||
files = sorted(p.name for p in (share / "netmover").iterdir())
|
||||
assert "fremd.txt" in files
|
||||
backups = [f for f in files if f.endswith(".nmbak")]
|
||||
assert len(backups) == 3
|
||||
assert not [f for f in files if f.endswith(".part")]
|
||||
newest = share / "netmover" / backups[-1]
|
||||
assert open_archive(newest.read_bytes(), PASS).manifest["version"] == __version__
|
||||
st = load_state(state.session_factory)
|
||||
assert st.last_success == NOW + timedelta(days=4)
|
||||
assert st.last_file == backups[-1] and st.last_size == newest.stat().st_size
|
||||
check = state.health().find("backup")
|
||||
assert check is not None and check.ok
|
||||
|
||||
|
||||
def test_failure_is_recorded_notified_once_and_becomes_problem(
|
||||
settings: Settings, backend: FakeMountBackend
|
||||
) -> None:
|
||||
sent: list[tuple[str, str, str]] = []
|
||||
state = NetMover(settings, backend=backend)
|
||||
mid, _share = _nas(state, backend)
|
||||
_configure(state, mid)
|
||||
state.backups._notify = lambda *a: sent.append(a)
|
||||
backend.vanish(state.mounts.mountpoint_for("backup"))
|
||||
backend.fail_next_mount(times=10)
|
||||
for i in range(3):
|
||||
result = state.backups.run(NOW + timedelta(hours=i))
|
||||
assert not result.ok
|
||||
st = load_state(state.session_factory)
|
||||
assert st.failures == 3 and "eingehängt" in st.last_error
|
||||
assert [k for k, _t, _m in sent] == ["backup.failed"]
|
||||
# Überfällig erst nach STALE_AFTER – dann rot in /healthz und Kuma
|
||||
state.started_at = datetime.now(UTC) - backup.STALE_AFTER * 2
|
||||
check = state.health().find("backup")
|
||||
assert check is not None and not check.ok and "überfällig" in check.message
|
||||
|
||||
|
||||
def test_tick_runs_only_when_due(state: NetMover, backend: FakeMountBackend) -> None:
|
||||
mid, _ = _nas(state, backend)
|
||||
_configure(state, mid)
|
||||
assert state.backups.tick(NOW) is not None # noch nie gesichert
|
||||
assert state.backups.tick(NOW + timedelta(minutes=5)) is None
|
||||
|
||||
|
||||
def test_missing_passphrase_refuses(state: NetMover, backend: FakeMountBackend) -> None:
|
||||
mid, _ = _nas(state, backend)
|
||||
_configure(state, mid, backup_passphrase_enc="")
|
||||
result = state.backups.run(NOW)
|
||||
assert not result.ok and "Passphrase" in result.message
|
||||
|
||||
|
||||
# -- Wiederherstellen ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _mount_names(db: Path) -> list[str]:
|
||||
con = sqlite3.connect(db)
|
||||
try:
|
||||
return [r[0] for r in con.execute("select name from mounts order by name")]
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
|
||||
def test_cli_backup_and_restore_roundtrip(
|
||||
state: NetMover, backend: FakeMountBackend, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
settings = state.settings
|
||||
mid, _ = _nas(state, backend)
|
||||
_configure(state, mid)
|
||||
out = tmp_path / "out"
|
||||
out.mkdir()
|
||||
assert cmd_backup(settings, out, encrypted=True) == 0
|
||||
(file,) = out.iterdir()
|
||||
assert file.suffix == ".nmbak" and file.stat().st_mode & 0o077 == 0
|
||||
old_key = settings.secret_key_path.read_bytes()
|
||||
state.stop()
|
||||
|
||||
# Nach der Sicherung geht etwas schief: Anbindung gelöscht, Schlüssel weg
|
||||
con = sqlite3.connect(settings.db_path)
|
||||
con.execute("delete from mounts")
|
||||
con.commit()
|
||||
con.close()
|
||||
settings.secret_key_path.write_bytes(b"kaputt")
|
||||
phrase = tmp_path / "phrase"
|
||||
phrase.write_text(PASS + "\n")
|
||||
|
||||
assert cmd_restore(settings, file, phrase, assume_yes=True, active=lambda: True) == 1
|
||||
assert "Dienst läuft" in capsys.readouterr().out
|
||||
assert cmd_restore(settings, file, phrase, assume_yes=True, active=lambda: False) == 0
|
||||
|
||||
assert _mount_names(settings.db_path) == ["Backup"]
|
||||
assert settings.secret_key_path.read_bytes() == old_key
|
||||
assert settings.secret_key_path.stat().st_mode & 0o077 == 0
|
||||
(kept,) = settings.data_dir.glob("vor-wiederherstellung-*")
|
||||
assert (kept / "secret.key").read_bytes() == b"kaputt"
|
||||
assert _mount_names(kept / "netmover.db") == []
|
||||
# Gespeicherte Zugangsdaten sind mit dem zurückgeholten Schlüssel wieder lesbar
|
||||
restarted = NetMover(settings, backend=backend)
|
||||
cfg = load_settings(restarted.session_factory)
|
||||
assert restarted.backups.passphrase(cfg) == PASS
|
||||
restarted.stop()
|
||||
|
||||
|
||||
def test_restore_refuses_newer_version(state: NetMover, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
archive = open_archive(make_backup(state.settings, None)[0], None)
|
||||
archive.manifest["version"] = "99.0.0"
|
||||
with pytest.raises(BackupError, match="erst aktualisieren"):
|
||||
backup.restore(state.settings, archive)
|
||||
|
||||
|
||||
# -- Oberfläche ------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_backup_page_save_run_and_health(
|
||||
client: TestClient, state: NetMover, backend: FakeMountBackend
|
||||
) -> None:
|
||||
mid, share = _nas(state, backend)
|
||||
page = client.get("/backup")
|
||||
assert page.status_code == 200 and 'href="/backup"' in page.text
|
||||
form = {
|
||||
"backup_enabled": "on",
|
||||
"backup_mount_id": str(mid),
|
||||
"backup_path": "netmover",
|
||||
"backup_time": "03:30",
|
||||
"backup_keep": "14",
|
||||
}
|
||||
r = post(client, "/backup", form)
|
||||
assert r.status_code == 400 and "Passphrase festlegen" in r.text
|
||||
r = post(client, "/backup", {**form, "passphrase": PASS, "passphrase_confirm": "anders"})
|
||||
assert r.status_code == 400 and "stimmen nicht" in r.text
|
||||
r = post(client, "/backup", {**form, "backup_path": "../raus", "passphrase": PASS})
|
||||
assert r.status_code == 400
|
||||
r = post(client, "/backup", {**form, "passphrase": PASS, "passphrase_confirm": PASS})
|
||||
assert r.status_code == 303
|
||||
with state.session_factory() as s:
|
||||
from netmover.models import Setting
|
||||
|
||||
raw = " ".join(v for (v,) in s.execute(select(Setting.value)))
|
||||
assert PASS not in raw # Passphrase nur verschlüsselt gespeichert
|
||||
# Passphrase leer lassen = unverändert
|
||||
assert post(client, "/backup", {**form, "backup_keep": "7"}).status_code == 303
|
||||
assert state.backups.passphrase(load_settings(state.session_factory)) == PASS
|
||||
|
||||
assert post(client, "/backup/run").status_code == 303
|
||||
assert "geschrieben" in client.get("/backup").text
|
||||
assert len(list((share / "netmover").glob("*.nmbak"))) == 1
|
||||
r = client.get("/healthz/backup")
|
||||
assert r.status_code == 200 and r.json()["status"] == "ok"
|
||||
assert "/healthz/backup" in client.get("/settings").text
|
||||
assert any(c["kind"] == "backup" for c in client.get("/healthz").json()["checks"])
|
||||
@@ -175,3 +175,47 @@ def test_upgrade_from_0002_keeps_everything_linked_to_rules(tmp_path: Path) -> N
|
||||
assert c.execute("SELECT waiting_since FROM rules").fetchone()[0] is None
|
||||
assert c.execute("SELECT username FROM mounts WHERE id = 2").fetchone()[0] == "paperless"
|
||||
c.close()
|
||||
|
||||
|
||||
# -- Pflicht für jede (auch künftige) Migration -------------------------------------------------
|
||||
|
||||
|
||||
def _revisions() -> list[str]:
|
||||
from alembic.script import ScriptDirectory
|
||||
|
||||
script = ScriptDirectory.from_config(alembic_config("sqlite://"))
|
||||
return [r.revision for r in reversed(list(script.walk_revisions()))]
|
||||
|
||||
|
||||
def _invariants(db: Path) -> tuple[object, ...]:
|
||||
st = _state(db)
|
||||
assert st["tmp_tables"] == [], "Reste einer Batch-Migration"
|
||||
assert st["fk_violations"] == [], "Fremdschlüssel verletzt"
|
||||
return st["mounts"], st["rules"], st["run_items"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("rev", _revisions()[1:])
|
||||
def test_every_migration_up_and_down_on_populated_db(tmp_path: Path, rev: str) -> None:
|
||||
"""Jede Migration einzeln: hoch und wieder runter auf einer Datenbank mit Daten –
|
||||
ohne Fremdschlüsselfehler, ohne Reste, ohne Datenverlust. Neue Migrationen werden
|
||||
hier automatisch erfasst."""
|
||||
revs = _revisions()
|
||||
prev = revs[revs.index(rev) - 1]
|
||||
db = tmp_path / "pflicht.db"
|
||||
url = f"sqlite:///{db}"
|
||||
upgrade(url, revs[0])
|
||||
_seed(db)
|
||||
upgrade(url, prev)
|
||||
before = _invariants(db)
|
||||
upgrade(url, rev)
|
||||
assert _state(db)["version"] == rev
|
||||
assert _invariants(db) == before
|
||||
command.downgrade(alembic_config(url), prev)
|
||||
assert _state(db)["version"] == prev
|
||||
assert _invariants(db) == before
|
||||
|
||||
|
||||
def test_migration_files_are_named_after_their_revision() -> None:
|
||||
versions = Path(__file__).parent.parent / "src/netmover/migrations/versions"
|
||||
names = sorted(p.name.split("_", 1)[0] for p in versions.glob("[0-9]*.py"))
|
||||
assert names == _revisions()
|
||||
|
||||
@@ -216,3 +216,47 @@ def test_navigation_groups_health_and_help_links(client: TestClient, state: Any)
|
||||
def test_logged_out_pages_have_no_menu(raw_client: TestClient) -> None:
|
||||
html = raw_client.get("/setup").text
|
||||
assert 'id="sidebar"' not in html
|
||||
|
||||
|
||||
# -- Kennzahlen (0.10.0) ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_metrics_in_healthz_and_rule_endpoint(client: TestClient, state: Any) -> None:
|
||||
from netmover.models import FileFailure, ItemStatus, Run, RunItem, RunStatus
|
||||
|
||||
mid = _add_mount(state, "NAS", "nas", MountState.MOUNTED)
|
||||
now = utcnow()
|
||||
with session_scope(state.session_factory) as s:
|
||||
a = Rule(name="A", src_mount_id=mid, dst_path="/tmp", last_success_at=now)
|
||||
b = Rule(name="B", src_mount_id=mid, dst_path="/tmp", last_success_at=now)
|
||||
s.add_all([a, b])
|
||||
s.flush()
|
||||
|
||||
def run(rule: Rule, ago: timedelta, **kw: Any) -> Run:
|
||||
r = Run(rule_id=rule.id, started_at=now - ago, finished_at=now - ago, **kw)
|
||||
s.add(r)
|
||||
s.flush()
|
||||
return r
|
||||
|
||||
run(a, timedelta(hours=1), status=RunStatus.OK, moved=3)
|
||||
q = run(a, timedelta(hours=2), status=RunStatus.PARTIAL, moved=1, failed=2)
|
||||
s.add(RunItem(run_id=q.id, src="x.pdf", status=ItemStatus.QUARANTINED))
|
||||
run(a, timedelta(hours=3), status=RunStatus.OK, moved=50, dry_run=True) # Probelauf
|
||||
run(a, timedelta(hours=30), status=RunStatus.OK, moved=7) # älter als 24 h
|
||||
run(b, timedelta(hours=4), status=RunStatus.ERROR)
|
||||
s.add(FileFailure(rule_id=b.id, src_relpath="y.pdf", size=1, mtime_ns=1, attempts=1))
|
||||
aid, bid = a.id, b.id
|
||||
|
||||
m = client.get("/healthz").json()["metrics"]
|
||||
assert (m["files_24h"], m["failed_24h"], m["quarantined_24h"]) == (4, 2, 1)
|
||||
assert m["errors_24h"] == 1 and m["pending_failures"] == 1
|
||||
assert m["last_moved_at"].startswith(str((now - timedelta(hours=1)).date()))
|
||||
ra = client.get(f"/healthz/rule/{aid}").json()["metrics"]
|
||||
assert ra["files_24h"] == 4 and ra["quarantined_24h"] == 1
|
||||
rb = client.get(f"/healthz/rule/{bid}").json()["metrics"]
|
||||
assert rb["files_24h"] == 0 and rb["errors_24h"] == 1 and rb["last_moved_at"] is None
|
||||
|
||||
|
||||
def test_push_ok_message_shows_files() -> None:
|
||||
url = push_url("https://k/api/push/x", _health(True), {"files_24h": 12, "failed_24h": 1})
|
||||
assert httpx.URL(url).params["msg"] == "OK · 24 h: 12 Dateien, 1 Fehler"
|
||||
|
||||
Reference in New Issue
Block a user