Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
47ccf90b56 | ||
|
|
b7577c4552 | ||
|
|
43b634a135 | ||
|
|
660bb70c5d | ||
|
|
e59932ab75 | ||
|
|
cf7a3bcc88 |
@@ -0,0 +1,37 @@
|
|||||||
|
# Prüfungen bei jedem Push und Pull Request (Gitea Actions, act_runner).
|
||||||
|
# Gleiche Befehle wie lokal vor einem Release – siehe README, „Entwicklung“.
|
||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
tags: ["v*"]
|
||||||
|
pull_request:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
pruefen:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
strategy:
|
||||||
|
fail-fast: false
|
||||||
|
matrix:
|
||||||
|
# 3.12 = Mindestversion, 3.13 = Debian 13 (Zielsystem)
|
||||||
|
python: ["3.12", "3.13"]
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: ${{ matrix.python }}
|
||||||
|
- name: Abhängigkeiten
|
||||||
|
run: |
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
- name: ruff
|
||||||
|
run: |
|
||||||
|
ruff check .
|
||||||
|
ruff format --check .
|
||||||
|
- name: mypy
|
||||||
|
run: mypy src
|
||||||
|
- name: shellcheck
|
||||||
|
run: shellcheck deploy/*.sh
|
||||||
|
- name: pytest (inkl. Migrationen auf befüllter Datenbank)
|
||||||
|
run: pytest -q
|
||||||
+79
-1
@@ -8,6 +8,79 @@ Die Version steht an genau einer Stelle: `src/netmover/__init__.py`.
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.10.0] – 2026-10-23
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Tägliche Datensicherung** (*System → Sicherung*): Datenbank, `secret.key`,
|
||||||
|
`session.key` und `netmover.env` als mit Passphrase verschlüsselte Datei
|
||||||
|
(scrypt + Fernet) auf eine Anbindung (SMB, NFS oder SFTP). Momentaufnahme über die
|
||||||
|
SQLite-Backup-API mit Integritätsprüfung; die Datei wird vor dem Hochladen
|
||||||
|
probeweise geöffnet und atomar geschrieben; ältere NetMover-Sicherungen über die
|
||||||
|
eingestellte Anzahl hinaus werden gelöscht, andere Dateien nie. Verpasste Termine
|
||||||
|
werden nachgeholt, nach einem Fehler stündlich neuer Versuch, erster Fehlschlag
|
||||||
|
per E-Mail/Webhook. Knopf „Jetzt sichern“.
|
||||||
|
- `netmover restore <datei>`: spielt eine Sicherung ein (nur bei gestopptem Dienst),
|
||||||
|
legt die bisherigen Dateien vorher nach `vor-wiederherstellung-<zeit>/` und
|
||||||
|
migriert die Datenbank. `netmover backup <ziel>` sichert von Hand in eine Datei.
|
||||||
|
- Befehl `netmover` im Pfad (`/usr/local/sbin/netmover`, lädt `netmover.env`).
|
||||||
|
- **Kennzahlen für Uptime Kuma**: `/healthz` und `/healthz/rule/<id>` liefern unter
|
||||||
|
`metrics` Dateien, Fehler, Quarantäne und abgebrochene Läufe der letzten 24 Stunden
|
||||||
|
sowie die letzte verschobene Datei – für Kuma-Monitore vom Typ „JSON-Abfrage“.
|
||||||
|
Der Push meldet z. B. „OK · 24 h: 12 Dateien“.
|
||||||
|
- `/healthz/backup`: eigener Monitor; eine seit zwei Tagen überfällige Sicherung gilt
|
||||||
|
als Problem (Statusampel, `/healthz`, Push).
|
||||||
|
- **CI in Gitea** (`.gitea/workflows/ci.yml`): ruff, mypy, shellcheck und Tests auf
|
||||||
|
Python 3.12 und 3.13 bei jedem Push.
|
||||||
|
- Test-Pflicht für Migrationen: jede Migration wird automatisch auf einer befüllten
|
||||||
|
Datenbank hoch- und heruntergestuft.
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- README: HTTP im internen Netz als bewusste Entscheidung dokumentiert; TLS über
|
||||||
|
Reverse Proxy bleibt der Weg für nicht vertrauenswürdige Netze.
|
||||||
|
|
||||||
|
## [0.9.0] – 2026-10-15
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Regeln kopieren** (Knopf „Kopieren“): alle Einstellungen inklusive Archiv
|
||||||
|
übernommen, Name „… (Kopie)“. Die Kopie ist zunächst **deaktiviert**, damit sie nicht
|
||||||
|
mit dem Original um dieselben Dateien konkurriert; Protokoll und Buchführung
|
||||||
|
beginnen neu.
|
||||||
|
|
||||||
|
## [0.8.0] – 2026-10-12
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Anbindungen kopieren** (Knopf „Kopieren“): Formular vorausgefüllt, Name
|
||||||
|
„… (Kopie)“. Eigene Zugangsdaten der Vorlage werden übernommen (eigenes Exemplar),
|
||||||
|
solange kein neues Passwort eingegeben wird; ein eigener SSH-Schlüssel wird
|
||||||
|
weiterverwendet; gespeicherte Zugangsdaten bleiben ausgewählt. „Testen“ im
|
||||||
|
Kopie-Formular nutzt die übernommenen Zugangsdaten. Der SFTP-Hostschlüssel wird nur
|
||||||
|
bei gleichem Server und Port übernommen.
|
||||||
|
|
||||||
|
## [0.7.0] – 2026-10-12
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Archiv je Regel:** zusätzliche Kopie in ein Archiv (Anbindung SMB/NFS/SFTP oder
|
||||||
|
lokaler Pfad), Ablage „wie in der Quelle“ oder „nach Datum“ (`Jahr/Monat/…`).
|
||||||
|
Reihenfolge: erst Archiv, dann Ziel, **erst dann wird die Quelle gelöscht**.
|
||||||
|
Scheitert das Ziel nach dem Archiv, verhindert eine Buchführung
|
||||||
|
(`archived_files`) eine zweite Archivkopie. Im Archiv wird nie überschrieben.
|
||||||
|
Nicht erreichbares Archiv → Regel wartet. Platzprüfung auch fürs Archiv,
|
||||||
|
Probelauf zeigt das Archivziel. Migration 0004.
|
||||||
|
- Export/Import enthalten die Archiv-Einstellungen; eine als Archiv benutzte
|
||||||
|
Anbindung lässt sich nicht löschen.
|
||||||
|
|
||||||
|
## [0.6.0] – 2026-10-11
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- **Anmeldung je Netz konfigurierbar** (*Einstellungen → Anmeldung*): „Ohne Passwort
|
||||||
|
aus diesen Netzen“ und „Mit Passwort aus diesen Netzen“ (leer = alle übrigen).
|
||||||
|
Ist die zweite Liste ausgefüllt, werden alle anderen Netze abgewiesen (403, auch die
|
||||||
|
Anmeldeseite); `/healthz` und statische Dateien bleiben erreichbar. Ersetzt die
|
||||||
|
Auswahl „immer / vertraute Netze / nie“ aus 0.5.0 (`0.0.0.0/0, ::/0` = ganz aus;
|
||||||
|
gespeicherter Modus „nie“ gilt weiter).
|
||||||
|
- Schutz vor Aussperren: Eine Einstellung, mit der die eigene Adresse keinen Zugang
|
||||||
|
mehr hätte, wird nicht gespeichert. `netmover reset-password` hebt auch Netz-Sperren auf.
|
||||||
|
|
||||||
## [0.5.0] – 2026-10-11
|
## [0.5.0] – 2026-10-11
|
||||||
|
|
||||||
### Hinzugefügt
|
### Hinzugefügt
|
||||||
@@ -234,7 +307,12 @@ Erste Version.
|
|||||||
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
||||||
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
||||||
|
|
||||||
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.5.0...HEAD
|
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.10.0...HEAD
|
||||||
|
[0.10.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.9.0...v0.10.0
|
||||||
|
[0.9.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.8.0...v0.9.0
|
||||||
|
[0.8.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.7.0...v0.8.0
|
||||||
|
[0.7.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.6.0...v0.7.0
|
||||||
|
[0.6.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.5.0...v0.6.0
|
||||||
[0.5.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.4.0...v0.5.0
|
[0.5.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.4.0...v0.5.0
|
||||||
[0.4.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.3.0...v0.4.0
|
[0.4.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.3.0...v0.4.0
|
||||||
[0.3.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.2.1...v0.3.0
|
[0.3.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.2.1...v0.3.0
|
||||||
|
|||||||
@@ -19,10 +19,104 @@ Zeiten in Ortszeit (Europe/Berlin).
|
|||||||
| 2026-10-11 | 00:13 | 00:32 | 0:19 | – | Menü, Anleitung, Roadmap, Uptime Kuma (0.3.0) |
|
| 2026-10-11 | 00:13 | 00:32 | 0:19 | – | Menü, Anleitung, Roadmap, Uptime Kuma (0.3.0) |
|
||||||
| 2026-10-11 | 00:31 | 00:50 | 0:19 | – | Pausieren, Wartungsmodus, E-Mail (0.4.0) |
|
| 2026-10-11 | 00:31 | 00:50 | 0:19 | – | Pausieren, Wartungsmodus, E-Mail (0.4.0) |
|
||||||
| 2026-10-11 | 00:54 | 01:16 | 0:22 | – | SFTP-Quelle, Zugangsdaten, Anmeldung (0.5.0) |
|
| 2026-10-11 | 00:54 | 01:16 | 0:22 | – | SFTP-Quelle, Zugangsdaten, Anmeldung (0.5.0) |
|
||||||
| **Summe** | | | **3:17** | | |
|
| 2026-10-11 | 08:43 | 08:55 | 0:12 | – | Anmeldung je Netz (0.6.0) |
|
||||||
|
| 2026-10-12 | 00:14 | 00:34 | 0:20 | – | Archiv je Regel (0.7.0) |
|
||||||
|
| 2026-10-12 | 00:46 | 00:55 | 0:09 | – | Anbindungen kopieren (0.8.0) |
|
||||||
|
| 2026-10-15 | 09:17 | 09:26 | 0:09 | – | Regeln kopieren (0.9.0) |
|
||||||
|
| 2026-10-23 | 13:42 | 14:10 | 0:28 | – | Sicherung, Kuma-Kennzahlen, CI (0.10.0) |
|
||||||
|
| **Summe** | | | **4:35** | | |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 2026-10-23 · 13:42–14:10 · Sicherung, Kuma-Kennzahlen, CI (0.10.0)
|
||||||
|
|
||||||
|
**Bestandsaufnahme Live-Server** – 0.9.0 seit 15.10. ohne Neustart, 287 Dateien
|
||||||
|
verschoben, keine Datei gescheitert oder in Quarantäne, Archivkopien vollständig
|
||||||
|
(159 + 122). 18 kurze Pausen, alle wegen SSH zu den Paperless-Servern
|
||||||
|
(„No existing session“, „protocol banner“, Timeout) – automatisches Pausieren und
|
||||||
|
Fortsetzen hat jedes Mal gegriffen. Ursache liegt vermutlich auf den Paperless-Hosts
|
||||||
|
→ Roadmap 0.11.0.
|
||||||
|
|
||||||
|
**Erledigt** – `backup.py` (Momentaufnahme per SQLite-Backup-API, tar.gz,
|
||||||
|
scrypt+Fernet, Probe-Öffnen vor dem Upload, Upload über `Destination.commit`,
|
||||||
|
Ausdünnen), Seite „Sicherung“, `/healthz/backup`, `netmover backup`/`restore`,
|
||||||
|
Wrapper `/usr/local/sbin/netmover`; Kennzahlen unter `metrics` in `/healthz`;
|
||||||
|
Gitea-CI; generischer Migrationstest. 381 Tests. Gegenproben: Ausdünnen ohne
|
||||||
|
Namensfilter und Sicherung ohne Verschlüsselung schlagen jeweils an (2 Tests).
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- *Passphrase verschlüsselt in der DB*: sonst bräuchte die tägliche Sicherung eine
|
||||||
|
Eingabe. Auf dem NAS schützt die Passphrase; auf dem Server ist ohnehin alles da.
|
||||||
|
- *Termin per „fällig?“-Prüfung alle 5 min* statt Cron-Job: holt verpasste Termine
|
||||||
|
nach, braucht kein Umplanen bei Änderungen und regelt den stündlichen Neuversuch.
|
||||||
|
- *Ziel über dieselbe Fabrik wie Regelziele* (`destination_factory`): gleiche
|
||||||
|
Mount-Prüfung, `.part`-Logik und SFTP-Unterstützung, kein zweiter Schreibweg.
|
||||||
|
- *Ausdünnen nur nach Namensschema* – fremde Dateien im Ordner bleiben unangetastet.
|
||||||
|
- *Kein TLS* im internen Netz (Nutzerentscheidung), im README begründet.
|
||||||
|
- *Kuma*: Endpunkte gab es seit 0.3.0; neu sind auslesbare Kennzahlen für
|
||||||
|
„JSON-Abfrage“-Monitore, z. B. Alarm bei `metrics.quarantined_24h != 0`.
|
||||||
|
|
||||||
|
## 2026-10-15 · 09:17–09:26 · Regeln kopieren (0.9.0)
|
||||||
|
|
||||||
|
**Erledigt** – Knopf „Kopieren“, Route `/rules/{id}/copy` (GET Formular, POST
|
||||||
|
speichern), `rule_copy_name`. 5 Tests (362 gesamt); Gegenprobe „Kopie aktiv“ schlägt
|
||||||
|
an (2 Tests).
|
||||||
|
|
||||||
|
**Entscheidung** – Kopie startet deaktiviert: Mit identischer Quelle würden Original
|
||||||
|
und Kopie sonst sofort parallel dieselben Dateien bearbeiten (doppelte Lieferung bzw.
|
||||||
|
Kollisionen). Protokoll und Buchführung (`copied_files`, `archived_files`) werden
|
||||||
|
bewusst nicht mitkopiert – die Kopie ist eine neue Regel.
|
||||||
|
|
||||||
|
## 2026-10-12 · 00:46–00:55 · Anbindungen kopieren (0.8.0)
|
||||||
|
|
||||||
|
**Erledigt** – Knopf „Kopieren“, Route `/mounts/{id}/copy` (GET Formular, POST
|
||||||
|
speichern), `save_mount(copy_from=…)`, Formular-Test mit Vorlage. 7 Tests (357 gesamt);
|
||||||
|
Gegenprobe ohne Übernahme der Zugangsdaten: 3 Tests schlagen an.
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- Eigene Zugangsdaten werden als eigenes Exemplar kopiert (nicht geteilt) – eigene
|
||||||
|
Zugangsdaten verschwinden mit ihrer Anbindung, eine geteilte Nutzung wäre
|
||||||
|
dafür der Fall „Zugangsdaten“.
|
||||||
|
- Eigenes Schlüsselpaar bleibt gleich (schon in `authorized_keys`).
|
||||||
|
- Hostschlüssel nur bei gleichem Server/Port übernehmen, sonst würde der Test an
|
||||||
|
einem anderen Server mit „Hostschlüssel geändert“ scheitern.
|
||||||
|
|
||||||
|
## 2026-10-12 · 00:14–00:34 · Archiv je Regel (0.7.0)
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- Archivfelder an der Regel, Tabelle `archived_files` (Migration 0004).
|
||||||
|
- Engine: Archiv zuerst, dann Ziel, dann Quelle löschen; Archiv immer „umbenennen“.
|
||||||
|
- Runner: Ziel- und Archivaufbau über eine gemeinsame Fabrik (`_target`).
|
||||||
|
- 14 Tests (350 gesamt); Mutationsprobe 4/4 erkannt (Buchführung, Archiv
|
||||||
|
übersprungen, Kollisionsregel, Platzprüfung).
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- Archiv *vor* Ziel: Paperless holt Dateien sofort ab – andersherum würde ein
|
||||||
|
gescheitertes Archiv beim Neuversuch zur doppelten Lieferung führen.
|
||||||
|
- Archiv überschreibt nie (unabhängig von der Kollisionsregel des Ziels).
|
||||||
|
- Ablage „nach Datum“ nach Verarbeitungsdatum (nicht Datei-mtime) – eindeutig
|
||||||
|
nachvollziehbar, wann eine Datei durchgelaufen ist.
|
||||||
|
|
||||||
|
**Stolperstein**
|
||||||
|
- Downgrade von 0004 scheiterte an den CHECK-Regeln der neuen Auswahlfelder
|
||||||
|
(verweisen nach dem Entfernen der Spalten ins Leere) – nur dank der Migrations-
|
||||||
|
tests mit Daten aufgefallen; Downgrade entfernt sie jetzt zuerst.
|
||||||
|
|
||||||
|
## 2026-10-11 · 08:43–08:55 · Anmeldung je Netz (0.6.0)
|
||||||
|
|
||||||
|
**Anlass** – Wunsch: Anmeldung abschalten, und zwar konfigurierbar, aus welchen Netzen
|
||||||
|
mit und ohne Passwort. Zugriffe kamen aus 10.0.10.0/24 und 10.1.5.0/24 (Access-Log).
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- `auth.access_decision()` als einzige Entscheidung (ohne PW / mit PW / abgewiesen),
|
||||||
|
benutzt von Middleware (403 für fremde Netze), Guard und Anmeldeseite.
|
||||||
|
- Schutz vor Aussperren beim Speichern; `reset-password` hebt Sperren auf.
|
||||||
|
- 19 Tests für die Anmeldung (336 gesamt).
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- `/healthz` und `/static` nie sperren – Uptime Kuma kann in einem anderen Netz stehen.
|
||||||
|
- Nicht-IP-Clients (z. B. Unix-Socket) gelten als „in keinem Netz“.
|
||||||
|
|
||||||
## 2026-10-11 · 00:54–01:16 · SFTP-Quelle, Zugangsdaten, Anmeldung abschaltbar (0.5.0)
|
## 2026-10-11 · 00:54–01:16 · SFTP-Quelle, Zugangsdaten, Anmeldung abschaltbar (0.5.0)
|
||||||
|
|
||||||
**Erledigt**
|
**Erledigt**
|
||||||
|
|||||||
@@ -388,12 +388,20 @@ findmnt -R /mnt/netmover # was ist wirklich eingehängt?
|
|||||||
## Sicherheit
|
## Sicherheit
|
||||||
|
|
||||||
- **Nicht ins Internet stellen** (siehe [Reverse Proxy](#reverse-proxy-mit-tls)).
|
- **Nicht ins Internet stellen** (siehe [Reverse Proxy](#reverse-proxy-mit-tls)).
|
||||||
|
- **HTTP im internen Netz ist eine bewusste Entscheidung**, wenn das Netz
|
||||||
|
vertrauenswürdig ist (Heimnetz, eigenes VLAN): Die Oberfläche läuft dann
|
||||||
|
unverschlüsselt auf Port 8080, die Anmeldung kann per Netz abgeschaltet werden
|
||||||
|
(*Einstellungen → Anmeldung*). Wer im selben Netz mitlesen kann, könnte Passwörter
|
||||||
|
sehen, die man in Formulare eingibt. Für fremde oder gemischte Netze den
|
||||||
|
[Reverse Proxy mit TLS](#reverse-proxy-mit-tls) davorsetzen.
|
||||||
- Zugangsdaten liegen nur verschlüsselt in der Datenbank (Fernet, Schlüssel
|
- Zugangsdaten liegen nur verschlüsselt in der Datenbank (Fernet, Schlüssel
|
||||||
`secret.key`, 0600). Die Oberfläche zeigt sie nie an; in der Prozessliste stehen
|
`secret.key`, 0600). Die Oberfläche zeigt sie nie an; in der Prozessliste stehen
|
||||||
sie ebenfalls nie, weil `mount.cifs` eine kurzlebige Credentials-Datei bekommt.
|
sie ebenfalls nie, weil `mount.cifs` eine kurzlebige Credentials-Datei bekommt.
|
||||||
Ausgaben von `mount` werden geschwärzt, bevor sie protokolliert werden.
|
Ausgaben von `mount` werden geschwärzt, bevor sie protokolliert werden.
|
||||||
- **Sichere `secret.key` zusammen mit der Datenbank.** Ohne den Schlüssel sind die
|
- **Sichere `secret.key` zusammen mit der Datenbank.** Ohne den Schlüssel sind die
|
||||||
gespeicherten Passwörter verloren.
|
gespeicherten Passwörter verloren. Das erledigt die tägliche, mit Passphrase
|
||||||
|
verschlüsselte [Sicherung](docs/ANLEITUNG.md#sicherung) (*System → Sicherung*);
|
||||||
|
Wiederherstellen mit `netmover restore <datei>`.
|
||||||
- Anmeldung:
|
- Anmeldung:
|
||||||
- Ein Administratorkonto mit Argon2-Passworthash.
|
- Ein Administratorkonto mit Argon2-Passworthash.
|
||||||
- Sperre nach 5 Fehlversuchen.
|
- Sperre nach 5 Fehlversuchen.
|
||||||
@@ -430,7 +438,8 @@ Aufbau:
|
|||||||
| `mover/` | Scan, Stabilität, atomares Schreiben, Kollisionen, Buchführung, Regellauf |
|
| `mover/` | Scan, Stabilität, atomares Schreiben, Kollisionen, Buchführung, Regellauf |
|
||||||
| `runs.py`, `scheduler.py`, `health.py`, `notify.py` | Protokoll, Zeitsteuerung, Gesundheit, Webhook |
|
| `runs.py`, `scheduler.py`, `health.py`, `notify.py` | Protokoll, Zeitsteuerung, Gesundheit, Webhook |
|
||||||
| `admin.py`, `keys.py`, `transfer.py`, `web/` | Formulare, Schlüsselverwaltung, Export/Import, Oberfläche |
|
| `admin.py`, `keys.py`, `transfer.py`, `web/` | Formulare, Schlüsselverwaltung, Export/Import, Oberfläche |
|
||||||
| `health.py`, `kuma.py` | Gesundheitsprüfung, Uptime-Kuma-Push |
|
| `health.py`, `kuma.py` | Gesundheitsprüfung, Kennzahlen, Uptime-Kuma-Push |
|
||||||
|
| `backup.py` | tägliche verschlüsselte Sicherung, `netmover backup`/`restore` |
|
||||||
| `docs/` | Anleitung und Bilder; wird als Hilfe-Seite mit ausgeliefert |
|
| `docs/` | Anleitung und Bilder; wird als Hilfe-Seite mit ausgeliefert |
|
||||||
|
|
||||||
Die gesamte Mover-Logik ist mit dem `FakeMountBackend` ohne root testbar. SFTP
|
Die gesamte Mover-Logik ist mit dem `FakeMountBackend` ohne root testbar. SFTP
|
||||||
@@ -443,6 +452,17 @@ NETMOVER_DATA_DIR=$PWD/data .venv/bin/alembic revision --autogenerate -m "…"
|
|||||||
```
|
```
|
||||||
|
|
||||||
Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt.
|
Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt.
|
||||||
|
`tests/test_migrations.py` stuft **jede** Migration automatisch auf einer
|
||||||
|
Datenbank mit verknüpften Daten hoch und wieder herunter – eine neue Migration ist
|
||||||
|
damit ohne weiteres Zutun abgedeckt; ergänzt sie Spalten mit Daten, gehört ein
|
||||||
|
eigener Test dazu.
|
||||||
|
|
||||||
|
### CI
|
||||||
|
|
||||||
|
`.gitea/workflows/ci.yml` führt bei jedem Push und Pull Request ruff, mypy,
|
||||||
|
shellcheck und alle Tests aus (Python 3.12 und 3.13). Voraussetzung: Actions sind
|
||||||
|
im Repository aktiviert und ein `act_runner` mit dem Label `ubuntu-latest` ist
|
||||||
|
registriert.
|
||||||
|
|
||||||
### Arbeitsweise
|
### Arbeitsweise
|
||||||
|
|
||||||
@@ -450,7 +470,7 @@ Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt
|
|||||||
(`feat:`, `fix:`, `docs:`, `test:`, `chore:` …).
|
(`feat:`, `fix:`, `docs:`, `test:`, `chore:` …).
|
||||||
- Jede nutzerrelevante Änderung kommt unter `[Unreleased]` in
|
- Jede nutzerrelevante Änderung kommt unter `[Unreleased]` in
|
||||||
[CHANGELOG.md](CHANGELOG.md). Bei einem Release:
|
[CHANGELOG.md](CHANGELOG.md). Bei einem Release:
|
||||||
1. Den Abschnitt `[Unreleased]` mit Versionsnummer versehen.
|
1. CI grün? Dann den Abschnitt `[Unreleased]` mit Versionsnummer versehen.
|
||||||
2. Die Version in `src/netmover/__init__.py` anheben.
|
2. Die Version in `src/netmover/__init__.py` anheben.
|
||||||
3. Einen Tag `vX.Y.Z` setzen.
|
3. Einen Tag `vX.Y.Z` setzen.
|
||||||
- Arbeitszeit und Entscheidungen stehen in [DEVLOG.md](DEVLOG.md).
|
- Arbeitszeit und Entscheidungen stehen in [DEVLOG.md](DEVLOG.md).
|
||||||
|
|||||||
+17
-4
@@ -10,10 +10,19 @@ Legende: ✅ erledigt · 🔨 in Arbeit · 📋 geplant · 💡 Idee (noch nicht
|
|||||||
|
|
||||||
| Status | Thema | Warum |
|
| Status | Thema | Warum |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| 📋 | **CI in Gitea** (Actions): Tests, ruff, mypy, shellcheck bei jedem Push | Fehler fallen vor dem Update auf, nicht auf dem Server |
|
| 📋 | **SSH-Ausfälle bei Paperless klären**, Anbindungen auf SSH-Schlüssel umstellen | 18 kurze Ausfälle seit 11.10. („No existing session“, „protocol banner“) |
|
||||||
| 📋 | **Migrationstest-Pflicht**: jede neue Migration mit Test auf befüllter Datenbank | Lehre aus dem gescheiterten Update auf 0.2.0 |
|
|
||||||
| 📋 | **Reverse Proxy mit TLS** auf der Ziel-VM (Caddy), danach Bindung an `127.0.0.1` | Anmeldung nicht unverschlüsselt im LAN |
|
|
||||||
| 📋 | **Statusseite schneller bei hängenden Mounts**: Füllstand zwischenspeichern | Heute bis zu 2 s Wartezeit, wenn ein NFS-Server weg ist |
|
| 📋 | **Statusseite schneller bei hängenden Mounts**: Füllstand zwischenspeichern | Heute bis zu 2 s Wartezeit, wenn ein NFS-Server weg ist |
|
||||||
|
| 📋 | **Benachrichtigung bei Quarantäne** einer Datei | Unbeaufsichtigter Betrieb; heute über Kuma (`metrics.quarantined_24h`) sichtbar |
|
||||||
|
| 📋 | **Kompatibilitätszusage für 1.0**: Einstellungen, Export-Format, Update aus jeder 0.x | Danach keine brechenden Änderungen ohne 2.0 |
|
||||||
|
|
||||||
|
### Weg zu 1.0
|
||||||
|
|
||||||
|
0.10.0 Sicherung + CI → 0.11.0 restliche Punkte oben → ein bis zwei Wochen ruhiger
|
||||||
|
Betrieb → 1.0.0 ohne neue Funktionen.
|
||||||
|
|
||||||
|
**Entschieden:** Kein TLS im internen Netz – HTTP auf Port 8080 ist eine bewusste
|
||||||
|
Entscheidung (README, „Sicherheit“). Der Reverse Proxy bleibt dokumentiert für
|
||||||
|
nicht vertrauenswürdige Netze.
|
||||||
|
|
||||||
## Ideen
|
## Ideen
|
||||||
|
|
||||||
@@ -22,12 +31,16 @@ Legende: ✅ erledigt · 🔨 in Arbeit · 📋 geplant · 💡 Idee (noch nicht
|
|||||||
| 💡 | **Eine Regel, mehrere Ziele** je Quell-Unterordner | Heute: eine Regel pro Unterordner |
|
| 💡 | **Eine Regel, mehrere Ziele** je Quell-Unterordner | Heute: eine Regel pro Unterordner |
|
||||||
| 💡 | Metriken für Prometheus (`/metrics`) | Ergänzt `/healthz`, z. B. Dateien pro Tag |
|
| 💡 | Metriken für Prometheus (`/metrics`) | Ergänzt `/healthz`, z. B. Dateien pro Tag |
|
||||||
| 💡 | Deinstallationsskript | Gegenstück zu `install.sh` |
|
| 💡 | Deinstallationsskript | Gegenstück zu `install.sh` |
|
||||||
| 💡 | Benachrichtigung bei Quarantäne einer Datei | Heute nur im Protokoll sichtbar |
|
|
||||||
|
|
||||||
## Erledigt
|
## Erledigt
|
||||||
|
|
||||||
| Version | Thema |
|
| Version | Thema |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
| 0.10.0 | Tägliche verschlüsselte Sicherung + `netmover restore`; Kennzahlen für Kuma; CI in Gitea; Migrationstest-Pflicht |
|
||||||
|
| 0.9.0 | Regeln kopieren (Kopie zunächst deaktiviert) |
|
||||||
|
| 0.8.0 | Anbindungen kopieren inklusive Zugangsdaten |
|
||||||
|
| 0.7.0 | Archiv je Regel: Kopie ins Ziel und ins Archiv, Quelle erst danach löschen |
|
||||||
|
| 0.6.0 | Anmeldung je Netz: ohne Passwort / mit Passwort / abgewiesen, Schutz vor Aussperren |
|
||||||
| 0.5.0 | SFTP als Quelle; gespeicherte Zugangsdaten für SMB/SFTP-Passwörter (Seite „Zugangsdaten“); Anmeldung abschaltbar (vertrauenswürdige Netze / ganz) |
|
| 0.5.0 | SFTP als Quelle; gespeicherte Zugangsdaten für SMB/SFTP-Passwörter (Seite „Zugangsdaten“); Anmeldung abschaltbar (vertrauenswürdige Netze / ganz) |
|
||||||
| 0.4.0 | Automatisch pausieren/fortsetzen bei nicht erreichbaren Anbindungen; Wartungsmodus; E-Mail-Benachrichtigung; eine Meldung pro ausgefallener Anbindung |
|
| 0.4.0 | Automatisch pausieren/fortsetzen bei nicht erreichbaren Anbindungen; Wartungsmodus; E-Mail-Benachrichtigung; eine Meldung pro ausgefallener Anbindung |
|
||||||
| 0.3.0 | Neues Menü (gruppierte Seitenleiste, Statusampel, Handy-Menü); Anleitung im Repo und als Hilfe-Seite; Roadmap; Uptime Kuma (Einzel-Endpunkte, Push-Monitor) |
|
| 0.3.0 | Neues Menü (gruppierte Seitenleiste, Statusampel, Handy-Menü); Anleitung im Repo und als Hilfe-Seite; Roadmap; Uptime Kuma (Einzel-Endpunkte, Push-Monitor) |
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@
|
|||||||
services:
|
services:
|
||||||
netmover:
|
netmover:
|
||||||
build: .
|
build: .
|
||||||
image: netmover:0.5.0
|
image: netmover:0.10.0
|
||||||
container_name: netmover
|
container_name: netmover
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
cap_add:
|
cap_add:
|
||||||
|
|||||||
+10
-1
@@ -33,8 +33,17 @@ if [ ! -f /etc/netmover/netmover.env ]; then
|
|||||||
fi
|
fi
|
||||||
install -d -m 0755 /mnt/netmover
|
install -d -m 0755 /mnt/netmover
|
||||||
|
|
||||||
echo "==> Update-Skript"
|
echo "==> Update-Skript und Kommandozeile"
|
||||||
install -m 0755 "$(dirname "$0")/update.sh" /usr/local/sbin/netmover-update
|
install -m 0755 "$(dirname "$0")/update.sh" /usr/local/sbin/netmover-update
|
||||||
|
# netmover backup/restore/… mit derselben Konfiguration wie der Dienst
|
||||||
|
cat > /usr/local/sbin/netmover <<EOF
|
||||||
|
#!/bin/sh
|
||||||
|
set -a
|
||||||
|
[ -r /etc/netmover/netmover.env ] && . /etc/netmover/netmover.env
|
||||||
|
set +a
|
||||||
|
exec "$PREFIX/venv/bin/netmover" "\$@"
|
||||||
|
EOF
|
||||||
|
chmod 0755 /usr/local/sbin/netmover
|
||||||
|
|
||||||
echo "==> systemd"
|
echo "==> systemd"
|
||||||
install -m 0644 "$(dirname "$0")/netmover.service" /etc/systemd/system/netmover.service
|
install -m 0644 "$(dirname "$0")/netmover.service" /etc/systemd/system/netmover.service
|
||||||
|
|||||||
+157
-17
@@ -8,9 +8,10 @@ Diese Anleitung erklärt die Bedienung. Installation, Updates und technische Det
|
|||||||
stehen im [README](../README.md).
|
stehen im [README](../README.md).
|
||||||
|
|
||||||
**Inhalt:** [Begriffe](#begriffe) · [Erste Anmeldung](#erste-anmeldung) ·
|
**Inhalt:** [Begriffe](#begriffe) · [Erste Anmeldung](#erste-anmeldung) ·
|
||||||
[Zugangsdaten](#zugangsdaten) · [Anbindungen](#anbindungen) · [Regeln](#regeln) ·
|
[Zugangsdaten](#zugangsdaten) · [Anbindungen](#anbindungen) · [Regeln](#regeln) · [Archiv](#archiv) ·
|
||||||
[Probelauf](#probelauf) · [Status](#status) · [Wartungsmodus](#wartungsmodus) · [Protokoll](#protokoll) ·
|
[Probelauf](#probelauf) · [Status](#status) · [Wartungsmodus](#wartungsmodus) · [Protokoll](#protokoll) ·
|
||||||
[Einstellungen](#einstellungen) · [Anmeldung](#anmeldung) · [Uptime Kuma](#uptime-kuma) · [Beispiele](#beispiele) · [Häufige Fragen](#fragen)
|
[Einstellungen](#einstellungen) · [Anmeldung](#anmeldung) · [Uptime Kuma](#uptime-kuma) ·
|
||||||
|
[Sicherung](#sicherung) · [Beispiele](#beispiele) · [Häufige Fragen](#fragen)
|
||||||
|
|
||||||
## Die vier Begriffe {#begriffe}
|
## Die vier Begriffe {#begriffe}
|
||||||
|
|
||||||
@@ -72,6 +73,22 @@ Stabilitätsfenster dort mindestens 2 Sekunden betragen.
|
|||||||
| **Beim Start automatisch einhängen** | Nach einem Neustart des Servers wieder einhängen. Fällt das NAS aus, versucht NetMover es außerdem jede Minute erneut. |
|
| **Beim Start automatisch einhängen** | Nach einem Neustart des Servers wieder einhängen. Fällt das NAS aus, versucht NetMover es außerdem jede Minute erneut. |
|
||||||
| **Aktiv** | Deaktivierte Anbindungen werden von Regeln nicht benutzt. |
|
| **Aktiv** | Deaktivierte Anbindungen werden von Regeln nicht benutzt. |
|
||||||
|
|
||||||
|
### Kopieren {#anbindung-kopieren}
|
||||||
|
|
||||||
|
Für eine ähnliche Anbindung – etwa eine zweite Freigabe auf demselben NAS – bei der
|
||||||
|
Vorlage auf **Kopieren** klicken. Das Formular ist mit allen Werten vorausgefüllt
|
||||||
|
(Name „… (Kopie)“); meist muss nur die Freigabe geändert werden.
|
||||||
|
|
||||||
|
- **Zugangsdaten werden übernommen**, solange das Passwortfeld leer bleibt. Die Kopie
|
||||||
|
bekommt ein eigenes Exemplar – das Löschen der Vorlage betrifft sie nicht.
|
||||||
|
Gespeicherte Zugangsdaten sind einfach vorausgewählt.
|
||||||
|
- Ein **eigener SSH-Schlüssel** der Vorlage wird weiter benutzt; ein neuer Eintrag in
|
||||||
|
`authorized_keys` ist nicht nötig.
|
||||||
|
- **Testen** funktioniert im Kopie-Formular bereits mit den übernommenen Zugangsdaten.
|
||||||
|
|
||||||
|
Tipp: Nutzen mehrere Anbindungen dieselben Zugangsdaten, lohnt sich ein Eintrag unter
|
||||||
|
[Zugangsdaten](#zugangsdaten) – dann gilt ein neues Passwort sofort überall.
|
||||||
|
|
||||||
### Testen {#anbindung-testen}
|
### Testen {#anbindung-testen}
|
||||||
|
|
||||||
**Testen** prüft drei Dinge einzeln und sagt im Klartext, was nicht klappt:
|
**Testen** prüft drei Dinge einzeln und sagt im Klartext, was nicht klappt:
|
||||||
@@ -160,6 +177,34 @@ Gespeicherte Zugangsdaten lassen sich nur löschen, wenn sie keine Anbindung meh
|
|||||||
| **Aktion** | **verschieben** (Quelle wird nach erfolgreicher Übertragung gelöscht) oder **kopieren** (Quelle bleibt; jede Dateiversion wird nur einmal kopiert). |
|
| **Aktion** | **verschieben** (Quelle wird nach erfolgreicher Übertragung gelöscht) oder **kopieren** (Quelle bleibt; jede Dateiversion wird nur einmal kopiert). |
|
||||||
| **Wenn Datei existiert** | **umbenennen** (`scan.pdf` → `scan-1.pdf`, `scan-2.pdf` …), **überspringen** oder **überschreiben**. |
|
| **Wenn Datei existiert** | **umbenennen** (`scan.pdf` → `scan-1.pdf`, `scan-2.pdf` …), **überspringen** oder **überschreiben**. |
|
||||||
|
|
||||||
|
### Archiv {#archiv}
|
||||||
|
|
||||||
|
Optional bekommt jede Datei zusätzlich eine **Kopie ins Archiv** – z. B. Paperless als
|
||||||
|
Ziel und ein Archivordner auf dem NAS. **Die Quelle wird erst gelöscht, wenn die Datei
|
||||||
|
im Ziel UND im Archiv angekommen ist.**
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **Archiv** | kein Archiv / eine Anbindung (SMB, NFS, SFTP) / ein lokaler Pfad |
|
||||||
|
| **Archivordner** | Anbindung: relativer Unterordner (wird angelegt). Lokal: absoluter Pfad, muss existieren. |
|
||||||
|
| **Ablage** | **wie in der Quelle** oder **nach Datum** – `2026/10/scan001.pdf` (Monat der Verarbeitung) |
|
||||||
|
|
||||||
|
So läuft es pro Datei:
|
||||||
|
|
||||||
|
1. **Zuerst ins Archiv** – mit denselben Sicherungen wie beim Ziel.
|
||||||
|
2. **Dann ins Ziel.**
|
||||||
|
3. **Dann erst** wird die Quelle gelöscht (bei „verschieben“).
|
||||||
|
|
||||||
|
Das Archiv kommt zuerst, weil ein Ziel wie Paperless Dateien sofort abholt: Scheitert
|
||||||
|
das Archiv, wurde noch nichts geliefert, und die Datei bleibt in der Quelle. Scheitert
|
||||||
|
danach das Ziel, merkt sich NetMover, dass die Datei schon archiviert ist – der nächste
|
||||||
|
Versuch legt keine zweite Archivkopie an.
|
||||||
|
|
||||||
|
Im Archiv wird **nie überschrieben**: Gleiche Namen werden immer umbenannt
|
||||||
|
(`scan-1.pdf`), egal was die Regel fürs Ziel vorsieht. Ist das Archiv nicht erreichbar,
|
||||||
|
wartet die Regel wie bei Quelle und Ziel. Im Kopiermodus bekommen Ziel und Archiv je
|
||||||
|
eine Kopie, die Quelle bleibt.
|
||||||
|
|
||||||
### Sicherheit und Grenzen {#regel-sicherheit}
|
### Sicherheit und Grenzen {#regel-sicherheit}
|
||||||
|
|
||||||
| Feld | Bedeutung | Empfehlung |
|
| Feld | Bedeutung | Empfehlung |
|
||||||
@@ -182,6 +227,17 @@ Synology-Verwaltungsordner (`@eaDir`, `#recycle`).
|
|||||||
| **Modus setzen** | z. B. `0644`; leer = wie die Quelldatei |
|
| **Modus setzen** | z. B. `0644`; leer = wie die Quelldatei |
|
||||||
| **Quarantäneordner** | Dateien, die mehrmals hintereinander scheitern, werden dorthin verschoben (relativ zur Quell-Anbindung), statt es endlos zu versuchen. Leer = keine Quarantäne. |
|
| **Quarantäneordner** | Dateien, die mehrmals hintereinander scheitern, werden dorthin verschoben (relativ zur Quell-Anbindung), statt es endlos zu versuchen. Leer = keine Quarantäne. |
|
||||||
|
|
||||||
|
### Regel kopieren {#regel-kopieren}
|
||||||
|
|
||||||
|
Für eine ähnliche Regel – etwa für einen zweiten Quell-Unterordner – bei der Vorlage
|
||||||
|
auf **Kopieren** klicken. Alle Einstellungen (Filter, Ziel, Archiv, Sicherheit,
|
||||||
|
Rechte) sind übernommen, der Name lautet „… (Kopie)“.
|
||||||
|
|
||||||
|
Die Kopie ist zunächst **deaktiviert**: Mit gleicher Quelle würde sie sonst sofort mit
|
||||||
|
dem Original um dieselben Dateien konkurrieren. Also Quelle bzw. Ziel anpassen,
|
||||||
|
speichern, **Probelauf**, dann **Aktivieren**. Protokoll und Buchführung beginnen bei
|
||||||
|
der Kopie neu.
|
||||||
|
|
||||||
### Probelauf {#probelauf}
|
### Probelauf {#probelauf}
|
||||||
|
|
||||||
**Probelauf** zeigt in einer Tabelle, welche Datei wohin ginge – **ohne etwas zu
|
**Probelauf** zeigt in einer Tabelle, welche Datei wohin ginge – **ohne etwas zu
|
||||||
@@ -329,29 +385,110 @@ HTTP-Monitore*. Mit einem Monitor pro Anbindung und Regel sieht man in Kuma sofo
|
|||||||
|
|
||||||
Als Problem gilt: Scheduler steht, eine aktive Anbindung ist im Fehlerzustand, eine
|
Als Problem gilt: Scheduler steht, eine aktive Anbindung ist im Fehlerzustand, eine
|
||||||
aktive Regel hatte länger als das Dreifache ihres Intervalls keinen Erfolg oder ist
|
aktive Regel hatte länger als das Dreifache ihres Intervalls keinen Erfolg oder ist
|
||||||
dreimal hintereinander gescheitert.
|
dreimal hintereinander gescheitert, oder die [Sicherung](#sicherung) ist überfällig.
|
||||||
|
|
||||||
|
**3. Werte auslesen (JSON-Abfrage)** – `/healthz` und `/healthz/rule/<nummer>`
|
||||||
|
liefern unter `metrics` Kennzahlen der letzten 24 Stunden (Probeläufe zählen nicht):
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| `files_24h` | verschobene bzw. kopierte Dateien |
|
||||||
|
| `failed_24h` | Dateien, die gescheitert sind (werden erneut versucht) |
|
||||||
|
| `quarantined_24h` | Dateien, die in Quarantäne verschoben wurden |
|
||||||
|
| `errors_24h` | abgebrochene Läufe (z. B. Anbindung nicht erreichbar) |
|
||||||
|
| `pending_failures` | Dateien, die gerade auf einen neuen Versuch warten |
|
||||||
|
| `last_moved_at` | Zeitpunkt der letzten verschobenen Datei (UTC) |
|
||||||
|
|
||||||
|
`/healthz` enthält die Summe und unter `metrics.rules.<nummer>` jede Regel einzeln.
|
||||||
|
In Kuma: Typ *HTTP(s) – JSON-Abfrage*, dann Ausdruck, Bedingung und erwarteter Wert,
|
||||||
|
zum Beispiel:
|
||||||
|
|
||||||
|
| Monitor | Adresse | Ausdruck | Bedingung | Wert |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Gesamtzustand | `/healthz` | `status` | `==` | `ok` |
|
||||||
|
| Alarm bei Quarantäne | `/healthz` | `metrics.quarantined_24h` | `==` | `0` |
|
||||||
|
| Regel ohne Dateifehler | `/healthz/rule/1` | `metrics.failed_24h` | `==` | `0` |
|
||||||
|
| Sicherung | `/healthz/backup` | `status` | `==` | `ok` |
|
||||||
|
|
||||||
|
Ältere Kuma-Versionen (1.x) kennen nur „erwarteter Wert = Ergebnis“; die Beispiele
|
||||||
|
funktionieren dort genauso. Antwortet NetMover mit 503 (irgendein Problem), meldet
|
||||||
|
Kuma auch einen Werte-Monitor als gestört – das ist gewollt.
|
||||||
|
|
||||||
|
Beim Push zeigt Kuma als Nachricht z. B. „OK · 24 h: 12 Dateien“.
|
||||||
|
|
||||||
### Anmeldung {#anmeldung}
|
### Anmeldung {#anmeldung}
|
||||||
|
|
||||||
Standardmäßig fragt NetMover immer nach dem Passwort. Unter *Einstellungen →
|
Unter *Einstellungen → Anmeldung* legst du fest, **aus welchen Netzen** es mit und
|
||||||
Anmeldung* lässt sich das ändern (Bestätigung mit dem aktuellen Passwort):
|
ohne Passwort geht (Bestätigung mit dem aktuellen Passwort):
|
||||||
|
|
||||||
| Einstellung | Wirkung |
|
| Feld | Wirkung |
|
||||||
|---|---|
|
|---|---|
|
||||||
| **immer** (empfohlen) | Anmeldung bei jedem Zugriff |
|
| **Ohne Passwort aus diesen Netzen** | z. B. `10.0.10.0/24, 10.1.5.0/24` – von dort keine Anmeldung. Leer = niemand. `0.0.0.0/0, ::/0` = Anmeldung ganz aus. |
|
||||||
| **nicht aus vertrauenswürdigen Netzen** | ohne Passwort aus den eingetragenen Netzen, z. B. `10.0.10.0/24`; von überall sonst mit Passwort |
|
| **Mit Passwort aus diesen Netzen** | Leer = **alle übrigen** Netze mit Passwort. Ausgefüllt = nur von dort mit Passwort, **alle anderen Netze werden abgewiesen** (auch die Anmeldeseite). |
|
||||||
| **nie** | keine Anmeldung – nur hinter einem Reverse Proxy mit eigener Anmeldung (z. B. Authelia, Authentik) |
|
|
||||||
|
|
||||||
Solange ohne Anmeldung gearbeitet wird, steht oben gelb „ohne Anmeldung“. Der Schutz
|
Beispiel: Ohne Passwort aus dem Heimnetz `10.0.10.0/24`, mit Passwort aus dem
|
||||||
|
VPN-Netz `10.8.0.0/24`, alles andere gesperrt.
|
||||||
|
|
||||||
|
- Einzelne Adressen gehen auch (`192.168.1.20`).
|
||||||
|
- Die Seite zeigt deine aktuelle Adresse. Eine Einstellung, mit der du dich selbst
|
||||||
|
aussperren würdest, wird nicht gespeichert.
|
||||||
|
- `/healthz` bleibt für die Überwachung von überall erreichbar.
|
||||||
|
- Solange du ohne Passwort arbeitest, steht oben gelb „ohne Anmeldung“. Der Schutz
|
||||||
gegen untergeschobene Formulare bleibt immer aktiv.
|
gegen untergeschobene Formulare bleibt immer aktiv.
|
||||||
|
|
||||||
> NetMover läuft als root und kennt die Zugangsdaten deiner Freigaben. Wer die
|
> NetMover läuft als root und kennt die Zugangsdaten deiner Freigaben. Wer die
|
||||||
> Oberfläche ohne Anmeldung erreicht, kann alles ändern. Läuft ein Reverse Proxy auf
|
> Oberfläche ohne Passwort erreicht, kann alles ändern – Netze „ohne Passwort“ daher
|
||||||
> demselben Server, kommen alle Anfragen von `127.0.0.1` – dann `127.0.0.1` nicht als
|
> so eng wie möglich wählen. Läuft ein Reverse Proxy auf demselben Server, kommen
|
||||||
> vertrauenswürdig eintragen und `NETMOVER_FORWARDED_ALLOW_IPS` setzen.
|
> alle Anfragen von `127.0.0.1`: dann `127.0.0.1` nicht eintragen und
|
||||||
|
> `NETMOVER_FORWARDED_ALLOW_IPS` setzen.
|
||||||
|
|
||||||
Zurück zur Anmeldung: dieselbe Einstellung auf „immer“, oder auf dem Server
|
Ausgesperrt oder Passwort vergessen? Auf dem Server
|
||||||
`/opt/netmover/venv/bin/netmover reset-password` (setzt auch die Anmeldung zurück).
|
`/opt/netmover/venv/bin/netmover reset-password` – setzt das Passwort zurück, schaltet
|
||||||
|
die Anmeldung überall wieder ein und hebt Netz-Sperren auf.
|
||||||
|
|
||||||
|
### Sicherung {#sicherung}
|
||||||
|
|
||||||
|
*System → Sicherung*. NetMover sichert täglich seine Datenbank **und** den Schlüssel
|
||||||
|
`secret.key` – also alle Anbindungen, Regeln, Zugangsdaten und Einstellungen – auf
|
||||||
|
eine Anbindung, am besten eine eigene Freigabe auf dem NAS.
|
||||||
|
|
||||||
|
1. Auf dem NAS eine Freigabe anlegen (z. B. `Backup`), in NetMover als Anbindung
|
||||||
|
eintragen und testen.
|
||||||
|
2. *Sicherung*: Anbindung wählen, Unterordner (z. B. `netmover`), Uhrzeit, wie viele
|
||||||
|
Sicherungen behalten werden, und eine **Passphrase** (mindestens 12 Zeichen).
|
||||||
|
3. *Tägliche Sicherung einschalten*, speichern, **Jetzt sichern**.
|
||||||
|
4. **Passphrase im Passwortmanager ablegen.** Ohne sie ist die Sicherung wertlos.
|
||||||
|
|
||||||
|
Die Sicherung ist mit der Passphrase verschlüsselt (scrypt + Fernet): Wer nur Zugriff
|
||||||
|
aufs NAS hat, kommt nicht an die Zugangsdaten. Vor dem Hochladen öffnet NetMover die
|
||||||
|
Datei probeweise und prüft die Datenbank; hochgeladen wird wie bei Regeln über eine
|
||||||
|
`.part`-Datei. Danach werden ältere NetMover-Sicherungen über die eingestellte Anzahl
|
||||||
|
hinaus gelöscht – andere Dateien im Ordner nie.
|
||||||
|
|
||||||
|
Fällt der Termin aus (Server aus, NAS weg), holt NetMover ihn nach; nach einem
|
||||||
|
Fehler versucht es stündlich erneut und meldet den ersten Fehlschlag per E-Mail bzw.
|
||||||
|
Webhook. Gab es zwei Tage lang keine erfolgreiche Sicherung, gilt das als Problem
|
||||||
|
(Statusampel, `/healthz`, eigener Monitor `/healthz/backup`).
|
||||||
|
|
||||||
|
Eine neue Passphrase gilt nur für künftige Sicherungen; ältere brauchen weiter die
|
||||||
|
alte.
|
||||||
|
|
||||||
|
**Wiederherstellen** (neuer Server, gleiche oder neuere NetMover-Version):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# NetMover wie im README installieren, dann:
|
||||||
|
systemctl stop netmover
|
||||||
|
netmover restore /pfad/zu/netmover-<host>-<datum>.nmbak # fragt nach der Passphrase
|
||||||
|
systemctl start netmover
|
||||||
|
```
|
||||||
|
|
||||||
|
Die bisherigen Dateien legt `restore` vorher in
|
||||||
|
`/var/lib/netmover/vor-wiederherstellung-<zeit>/` ab. Die Datenbank wird anschließend
|
||||||
|
auf den Stand der installierten Version migriert. Die Sicherungsdatei holst du z. B.
|
||||||
|
mit `scp` vom NAS oder hängst die Freigabe kurz von Hand ein.
|
||||||
|
|
||||||
|
Eine Sicherung von Hand in eine Datei: `netmover backup /root/` (verwendet die
|
||||||
|
Passphrase aus den Einstellungen).
|
||||||
|
|
||||||
### Export und Import {#export-import}
|
### Export und Import {#export-import}
|
||||||
|
|
||||||
@@ -370,12 +507,15 @@ deaktiviert, Zugangsdaten müssen neu eingegeben werden.
|
|||||||
3. Anbindung **Paperless** (SFTP, Pfad `/opt/paperless_data/consume`, Schlüssel
|
3. Anbindung **Paperless** (SFTP, Pfad `/opt/paperless_data/consume`, Schlüssel
|
||||||
wählen, Benutzer = der Paperless-Benutzer), testen.
|
wählen, Benutzer = der Paperless-Benutzer), testen.
|
||||||
4. Regel **Scans → Paperless**: Quelle NAS, Filter `*.pdf`, Ziel Paperless,
|
4. Regel **Scans → Paperless**: Quelle NAS, Filter `*.pdf`, Ziel Paperless,
|
||||||
verschieben, umbenennen, 30 s. **Probelauf**, dann aktivieren.
|
verschieben, umbenennen, 30 s. Optional **Archiv**: Anbindung NAS, Ordner
|
||||||
|
`Archiv/Scans`, Ablage nach Datum. **Probelauf**, dann aktivieren.
|
||||||
|
|
||||||
### Jede Person in einen eigenen Ordner {#beispiel-unterordner}
|
### Jede Person in einen eigenen Ordner {#beispiel-unterordner}
|
||||||
|
|
||||||
Scannt jede Person in einen eigenen Unterordner (`Scans/max`, `Scans/erika`): pro
|
Scannt jede Person in einen eigenen Unterordner (`Scans/max`, `Scans/erika`): pro
|
||||||
Person eine Regel mit eigenem **Unterordner** in der Quelle und eigenem Ziel.
|
Person eine Regel mit eigenem **Unterordner** in der Quelle und eigenem Ziel. Am
|
||||||
|
schnellsten: die erste Regel einrichten, dann **Kopieren** und nur Unterordner und
|
||||||
|
Ziel ändern.
|
||||||
|
|
||||||
### Kopie als zusätzliche Sicherung {#beispiel-kopie}
|
### Kopie als zusätzliche Sicherung {#beispiel-kopie}
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
||||||
|
|
||||||
__version__ = "0.5.0"
|
__version__ = "0.10.0"
|
||||||
|
|||||||
+100
-6
@@ -18,6 +18,8 @@ from sqlalchemy.orm import Session
|
|||||||
|
|
||||||
from netmover.crypto import SecretBox, generate_ssh_keypair
|
from netmover.crypto import SecretBox, generate_ssh_keypair
|
||||||
from netmover.models import (
|
from netmover.models import (
|
||||||
|
ArchiveKind,
|
||||||
|
ArchiveLayout,
|
||||||
CollisionPolicy,
|
CollisionPolicy,
|
||||||
Credential,
|
Credential,
|
||||||
CredentialKind,
|
CredentialKind,
|
||||||
@@ -292,7 +294,11 @@ def save_mount(
|
|||||||
mounts: MountManager,
|
mounts: MountManager,
|
||||||
data: MountInput,
|
data: MountInput,
|
||||||
mount_id: int | None = None,
|
mount_id: int | None = None,
|
||||||
|
copy_from: Mount | None = None,
|
||||||
) -> tuple[Mount | None, Errors]:
|
) -> tuple[Mount | None, Errors]:
|
||||||
|
"""``copy_from``: Vorlage beim Kopieren – eigene Zugangsdaten (Passwort bzw.
|
||||||
|
Schlüsselpaar) werden übernommen, solange kein neues Passwort eingegeben ist;
|
||||||
|
die Kopie bekommt ein eigenes Exemplar."""
|
||||||
errors: Errors = {}
|
errors: Errors = {}
|
||||||
m = s.get(Mount, mount_id) if mount_id is not None else None
|
m = s.get(Mount, mount_id) if mount_id is not None else None
|
||||||
if mount_id is not None and m is None:
|
if mount_id is not None and m is None:
|
||||||
@@ -347,6 +353,14 @@ def save_mount(
|
|||||||
setattr(m, f, getattr(data, f))
|
setattr(m, f, getattr(data, f))
|
||||||
if not data.type.is_kernel_mount:
|
if not data.type.is_kernel_mount:
|
||||||
m.auto_mount = False
|
m.auto_mount = False
|
||||||
|
if (
|
||||||
|
mount_id is None
|
||||||
|
and copy_from is not None
|
||||||
|
and data.type is MountType.SFTP
|
||||||
|
and copy_from.type is MountType.SFTP
|
||||||
|
and (copy_from.server, copy_from.port) == (data.server, data.port)
|
||||||
|
):
|
||||||
|
m.host_key = copy_from.host_key # gleicher Server – Hostschlüssel bleibt gültig
|
||||||
|
|
||||||
# Zugangsdaten: entweder ein gespeicherter Schlüssel aus der Schlüsselverwaltung
|
# Zugangsdaten: entweder ein gespeicherter Schlüssel aus der Schlüsselverwaltung
|
||||||
# (geteilt, wird hier nie verändert) oder eigene Zugangsdaten nur dieser Anbindung.
|
# (geteilt, wird hier nie verändert) oder eigene Zugangsdaten nur dieser Anbindung.
|
||||||
@@ -384,21 +398,32 @@ def save_mount(
|
|||||||
else:
|
else:
|
||||||
m.username = None
|
m.username = None
|
||||||
cred = own
|
cred = own
|
||||||
|
template = (
|
||||||
|
copy_from.credential
|
||||||
|
if copy_from is not None
|
||||||
|
and copy_from.credential is not None
|
||||||
|
and not copy_from.credential.managed
|
||||||
|
and copy_from.credential.kind is data.auth
|
||||||
|
else None
|
||||||
|
)
|
||||||
if cred is None or cred.kind is not data.auth:
|
if cred is None or cred.kind is not data.auth:
|
||||||
if data.auth is CredentialKind.SSH_KEY:
|
if template is not None and (data.auth is CredentialKind.SSH_KEY or not data.password):
|
||||||
private, public = generate_ssh_keypair(f"netmover@{m.slug}")
|
# Kopie: dasselbe Passwort bzw. Schlüsselpaar (steht schon in authorized_keys)
|
||||||
secret, pub = private, public
|
secret_enc, pub = template.secret_enc, template.public_key
|
||||||
|
elif data.auth is CredentialKind.SSH_KEY:
|
||||||
|
private, pub = generate_ssh_keypair(f"netmover@{m.slug}")
|
||||||
|
secret_enc = secrets.encrypt(private)
|
||||||
else:
|
else:
|
||||||
if not data.password:
|
if not data.password:
|
||||||
errors["password"] = "Passwort fehlt." # noqa: S105
|
errors["password"] = "Passwort fehlt." # noqa: S105
|
||||||
s.rollback()
|
s.rollback()
|
||||||
return None, errors
|
return None, errors
|
||||||
secret, pub = data.password, None
|
secret_enc, pub = secrets.encrypt(data.password), None
|
||||||
if cred is None:
|
if cred is None:
|
||||||
cred = Credential(name=f"mount:{m.slug}", managed=False)
|
cred = Credential(name=f"mount:{m.slug}", managed=False)
|
||||||
s.add(cred)
|
s.add(cred)
|
||||||
cred.kind = data.auth
|
cred.kind = data.auth
|
||||||
cred.secret_enc = secrets.encrypt(secret)
|
cred.secret_enc = secret_enc
|
||||||
cred.public_key = pub
|
cred.public_key = pub
|
||||||
elif data.password and data.auth is CredentialKind.PASSWORD:
|
elif data.password and data.auth is CredentialKind.PASSWORD:
|
||||||
cred.secret_enc = secrets.encrypt(data.password)
|
cred.secret_enc = secrets.encrypt(data.password)
|
||||||
@@ -419,12 +444,26 @@ def save_mount(
|
|||||||
return m, errors
|
return m, errors
|
||||||
|
|
||||||
|
|
||||||
|
def copy_name(s: Session, name: str) -> str:
|
||||||
|
"""„NAS (Kopie)“, „NAS (Kopie 2)“, … – frei in der Datenbank."""
|
||||||
|
base = f"{name} (Kopie)"[:100]
|
||||||
|
candidate, n = base, 1
|
||||||
|
while s.scalar(select(Mount.id).where(Mount.name == candidate)) is not None:
|
||||||
|
n += 1
|
||||||
|
candidate = f"{name} (Kopie {n})"[:100]
|
||||||
|
return candidate
|
||||||
|
|
||||||
|
|
||||||
def delete_mount(s: Session, mounts: MountManager, mount_id: int) -> None:
|
def delete_mount(s: Session, mounts: MountManager, mount_id: int) -> None:
|
||||||
m = s.get(Mount, mount_id)
|
m = s.get(Mount, mount_id)
|
||||||
if m is None:
|
if m is None:
|
||||||
return
|
return
|
||||||
used = s.scalars(
|
used = s.scalars(
|
||||||
select(Rule.name).where((Rule.src_mount_id == mount_id) | (Rule.dst_mount_id == mount_id))
|
select(Rule.name).where(
|
||||||
|
(Rule.src_mount_id == mount_id)
|
||||||
|
| (Rule.dst_mount_id == mount_id)
|
||||||
|
| (Rule.archive_mount_id == mount_id)
|
||||||
|
)
|
||||||
).all()
|
).all()
|
||||||
if used:
|
if used:
|
||||||
raise AdminError(f"Wird noch von Regeln benutzt: {', '.join(used)}")
|
raise AdminError(f"Wird noch von Regeln benutzt: {', '.join(used)}")
|
||||||
@@ -465,6 +504,10 @@ class RuleInput:
|
|||||||
dry_run: bool = False
|
dry_run: bool = False
|
||||||
quarantine_path: str = ""
|
quarantine_path: str = ""
|
||||||
quarantine_after: int = 3
|
quarantine_after: int = 3
|
||||||
|
archive_kind: ArchiveKind = ArchiveKind.NONE
|
||||||
|
archive_mount_id: int | None = None
|
||||||
|
archive_path: str = ""
|
||||||
|
archive_layout: ArchiveLayout = ArchiveLayout.FLAT
|
||||||
extra: dict[str, str] = field(default_factory=dict)
|
extra: dict[str, str] = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
@@ -492,6 +535,10 @@ def rule_input_from_model(r: Rule) -> RuleInput:
|
|||||||
dry_run=r.dry_run,
|
dry_run=r.dry_run,
|
||||||
quarantine_path=r.quarantine_path or "",
|
quarantine_path=r.quarantine_path or "",
|
||||||
quarantine_after=r.quarantine_after,
|
quarantine_after=r.quarantine_after,
|
||||||
|
archive_kind=r.archive_kind,
|
||||||
|
archive_mount_id=r.archive_mount_id,
|
||||||
|
archive_path=r.archive_path,
|
||||||
|
archive_layout=r.archive_layout,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -585,9 +632,52 @@ def parse_rule_form(s: Session, form: Mapping[str, str]) -> tuple[RuleInput, Err
|
|||||||
d.quarantine_path = safe_relpath("quarantine_path", d.quarantine_path)
|
d.quarantine_path = safe_relpath("quarantine_path", d.quarantine_path)
|
||||||
except ValidationError as exc:
|
except ValidationError as exc:
|
||||||
errors["quarantine_path"] = str(exc)
|
errors["quarantine_path"] = str(exc)
|
||||||
|
|
||||||
|
# Archiv
|
||||||
|
try:
|
||||||
|
d.archive_kind = ArchiveKind(form.get("archive_kind") or "none")
|
||||||
|
d.archive_layout = ArchiveLayout(form.get("archive_layout") or "flat")
|
||||||
|
except ValueError:
|
||||||
|
errors["archive_kind"] = "Ungültige Auswahl."
|
||||||
|
d.archive_path = (form.get("archive_path") or "").strip()
|
||||||
|
if d.archive_kind is ArchiveKind.MOUNT:
|
||||||
|
d.archive_mount_id = _int(form, "archive_mount_id", errors)
|
||||||
|
if d.archive_mount_id is None or s.get(Mount, d.archive_mount_id) is None:
|
||||||
|
errors["archive_mount_id"] = "Archiv-Anbindung wählen."
|
||||||
|
try:
|
||||||
|
d.archive_path = safe_relpath("archive_path", d.archive_path)
|
||||||
|
except ValidationError as exc:
|
||||||
|
errors["archive_path"] = str(exc)
|
||||||
|
if (
|
||||||
|
"archive_path" not in errors
|
||||||
|
and d.archive_mount_id == d.dst_mount_id
|
||||||
|
and d.dst_kind == "mount"
|
||||||
|
and d.archive_path == d.dst_path
|
||||||
|
):
|
||||||
|
errors["archive_path"] = "Archiv und Ziel sind derselbe Ordner."
|
||||||
|
elif d.archive_kind is ArchiveKind.LOCAL:
|
||||||
|
d.archive_mount_id = None
|
||||||
|
try:
|
||||||
|
d.archive_path = safe_abspath("archive_path", d.archive_path)
|
||||||
|
except ValidationError as exc:
|
||||||
|
errors["archive_path"] = str(exc)
|
||||||
|
if d.dst_kind == "local" and d.archive_path == d.dst_path:
|
||||||
|
errors["archive_path"] = "Archiv und Ziel sind derselbe Ordner."
|
||||||
|
else:
|
||||||
|
d.archive_mount_id, d.archive_path = None, ""
|
||||||
return d, errors
|
return d, errors
|
||||||
|
|
||||||
|
|
||||||
|
def rule_copy_name(s: Session, name: str) -> str:
|
||||||
|
"""„Scans (Kopie)“, „Scans (Kopie 2)“, … – frei in der Datenbank."""
|
||||||
|
base = f"{name} (Kopie)"[:100]
|
||||||
|
candidate, n = base, 1
|
||||||
|
while s.scalar(select(Rule.id).where(Rule.name == candidate)) is not None:
|
||||||
|
n += 1
|
||||||
|
candidate = f"{name} (Kopie {n})"[:100]
|
||||||
|
return candidate
|
||||||
|
|
||||||
|
|
||||||
def save_rule(s: Session, d: RuleInput, rule_id: int | None = None) -> tuple[Rule | None, Errors]:
|
def save_rule(s: Session, d: RuleInput, rule_id: int | None = None) -> tuple[Rule | None, Errors]:
|
||||||
dup = s.scalar(select(Rule.id).where(Rule.name == d.name))
|
dup = s.scalar(select(Rule.id).where(Rule.name == d.name))
|
||||||
if dup is not None and dup != rule_id:
|
if dup is not None and dup != rule_id:
|
||||||
@@ -617,6 +707,10 @@ def save_rule(s: Session, d: RuleInput, rule_id: int | None = None) -> tuple[Rul
|
|||||||
"on_collision",
|
"on_collision",
|
||||||
"dry_run",
|
"dry_run",
|
||||||
"quarantine_after",
|
"quarantine_after",
|
||||||
|
"archive_kind",
|
||||||
|
"archive_mount_id",
|
||||||
|
"archive_path",
|
||||||
|
"archive_layout",
|
||||||
):
|
):
|
||||||
setattr(r, f, getattr(d, f))
|
setattr(r, f, getattr(d, f))
|
||||||
r.set_owner = d.set_owner or None
|
r.set_owner = d.set_owner or None
|
||||||
|
|||||||
@@ -7,21 +7,25 @@ Ende: Scheduler stoppen, Mounts **nicht** aushängen (andere könnten sie nutzen
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from netmover import kuma
|
from netmover import kuma
|
||||||
|
from netmover.backup import STALE_AFTER, BackupError, BackupService, load_state
|
||||||
from netmover.config import Settings
|
from netmover.config import Settings
|
||||||
from netmover.crypto import SecretBox, load_or_create_key
|
from netmover.crypto import SecretBox, load_or_create_key
|
||||||
from netmover.db import make_engine, make_session_factory, upgrade
|
from netmover.db import make_engine, make_session_factory, session_scope, upgrade
|
||||||
from netmover.health import Health, check_health
|
from netmover.health import Health, backup_check, check_health
|
||||||
|
from netmover.health import metrics as health_metrics
|
||||||
from netmover.logging_setup import event, get_logger
|
from netmover.logging_setup import event, get_logger
|
||||||
from netmover.models import utcnow
|
from netmover.models import Mount, utcnow
|
||||||
from netmover.mounting.backend import MountBackend
|
from netmover.mounting.backend import MountBackend
|
||||||
from netmover.mounting.fake import FakeMountBackend
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
from netmover.mounting.linux import LinuxMountBackend
|
from netmover.mounting.linux import LinuxMountBackend
|
||||||
from netmover.mounting.manager import MountManager, RestoreResult
|
from netmover.mounting.manager import MountManager, RestoreResult
|
||||||
|
from netmover.mover.destinations import Destination
|
||||||
from netmover.mover.runner import RuleRunner
|
from netmover.mover.runner import RuleRunner
|
||||||
from netmover.notify import Notifier
|
from netmover.notify import Notification, Notifier
|
||||||
from netmover.runs import RunService
|
from netmover.runs import RunService
|
||||||
from netmover.scheduler import RuleScheduler
|
from netmover.scheduler import RuleScheduler
|
||||||
from netmover.settings_store import load_settings
|
from netmover.settings_store import load_settings
|
||||||
@@ -60,11 +64,19 @@ class NetMover:
|
|||||||
)
|
)
|
||||||
self.runner = RuleRunner(self.session_factory, self.mounts)
|
self.runner = RuleRunner(self.session_factory, self.mounts)
|
||||||
self.runs = RunService(self.session_factory, self.runner, self.notifier)
|
self.runs = RunService(self.session_factory, self.runner, self.notifier)
|
||||||
|
self.backups = BackupService(
|
||||||
|
settings,
|
||||||
|
self.session_factory,
|
||||||
|
self.backup_destination,
|
||||||
|
self.secrets,
|
||||||
|
notify=lambda kind, title, msg: self.notifier.send(Notification(kind, title, msg)),
|
||||||
|
)
|
||||||
self.scheduler = RuleScheduler(
|
self.scheduler = RuleScheduler(
|
||||||
self.session_factory,
|
self.session_factory,
|
||||||
self.runs,
|
self.runs,
|
||||||
retry_mounts=self.mounts.retry_pending,
|
retry_mounts=self.mounts.retry_pending,
|
||||||
push_health=self.push_health,
|
push_health=self.push_health,
|
||||||
|
backup_tick=self.backups.tick,
|
||||||
)
|
)
|
||||||
self.started_at: datetime = utcnow()
|
self.started_at: datetime = utcnow()
|
||||||
self.restore_results: list[RestoreResult] = []
|
self.restore_results: list[RestoreResult] = []
|
||||||
@@ -85,13 +97,35 @@ class NetMover:
|
|||||||
"""Scheduler-Job (jede Minute): Zustand an Uptime Kuma melden, falls eingerichtet."""
|
"""Scheduler-Job (jede Minute): Zustand an Uptime Kuma melden, falls eingerichtet."""
|
||||||
url = load_settings(self.session_factory).kuma_push_url
|
url = load_settings(self.session_factory).kuma_push_url
|
||||||
if url:
|
if url:
|
||||||
kuma.push(url, self.health())
|
kuma.push(url, self.health(), metrics=health_metrics(self.session_factory))
|
||||||
|
|
||||||
|
def backup_destination(self, mount_id: int, path: str) -> Callable[[], Destination]:
|
||||||
|
"""Ziel der Datensicherung – dieselbe Prüfung wie für Regelziele."""
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
m = s.get(Mount, mount_id)
|
||||||
|
if m is None:
|
||||||
|
raise BackupError("Die Anbindung für die Sicherung existiert nicht mehr")
|
||||||
|
mtype, name = m.type, m.name
|
||||||
|
mp = self.mounts.check_alive(mount_id)
|
||||||
|
return self.runner.destination_factory("backup_path", mount_id, mtype, name, path, mp, None)
|
||||||
|
|
||||||
def health(self) -> Health:
|
def health(self) -> Health:
|
||||||
return check_health(
|
cfg = load_settings(self.session_factory)
|
||||||
|
health = check_health(
|
||||||
self.session_factory,
|
self.session_factory,
|
||||||
scheduler_running=self.scheduler.running,
|
scheduler_running=self.scheduler.running,
|
||||||
last_heartbeat=self.scheduler.last_heartbeat,
|
last_heartbeat=self.scheduler.last_heartbeat,
|
||||||
service_started=self.started_at,
|
service_started=self.started_at,
|
||||||
maintenance=load_settings(self.session_factory).maintenance,
|
maintenance=cfg.maintenance,
|
||||||
)
|
)
|
||||||
|
st = load_state(self.session_factory)
|
||||||
|
health.add(
|
||||||
|
backup_check(
|
||||||
|
enabled=cfg.backup_enabled and bool(cfg.backup_mount_id),
|
||||||
|
last_success=st.last_success,
|
||||||
|
last_error=st.last_error,
|
||||||
|
stale_after=STALE_AFTER,
|
||||||
|
service_started=self.started_at,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return health
|
||||||
|
|||||||
@@ -0,0 +1,514 @@
|
|||||||
|
"""Datensicherung: Datenbank und Schlüssel täglich auf eine Anbindung (z. B. das NAS).
|
||||||
|
|
||||||
|
Ohne ``secret.key`` sind alle gespeicherten Zugangsdaten verloren – die Sicherung
|
||||||
|
enthält deshalb Datenbank **und** Schlüssel. Damit die Sicherung auf dem NAS nicht
|
||||||
|
selbst zum Schlüsselbund für alle Freigaben wird, verschlüsselt NetMover sie mit
|
||||||
|
einer Passphrase (scrypt + Fernet). Die Passphrase liegt verschlüsselt in der
|
||||||
|
Datenbank, damit die tägliche Sicherung ohne Eingabe läuft; zum Wiederherstellen
|
||||||
|
braucht man sie aus dem Passwortmanager.
|
||||||
|
|
||||||
|
Ablauf einer Sicherung:
|
||||||
|
|
||||||
|
1. Momentaufnahme der Datenbank über die SQLite-Backup-API (konsistent, auch
|
||||||
|
während Regeln laufen) und ``PRAGMA integrity_check``.
|
||||||
|
2. tar.gz mit ``manifest.json``, Datenbank, Schlüsseln und ``netmover.env``.
|
||||||
|
3. Verschlüsseln und sofort probeweise wieder öffnen – eine Sicherung, die sich
|
||||||
|
nicht lesen lässt, wird gar nicht erst hochgeladen.
|
||||||
|
4. Hochladen wie jede andere Datei (``.part`` → prüfen → atomar umbenennen).
|
||||||
|
5. Älteste Sicherungen über ``backup_keep`` hinaus löschen – erst nach Erfolg.
|
||||||
|
|
||||||
|
Format verschlüsselt (``.nmbak``): ``MAGIC`` + 16 Byte Salz + Fernet-Token.
|
||||||
|
Unverschlüsselt: ein gewöhnliches ``.tar.gz``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import socket
|
||||||
|
import sqlite3
|
||||||
|
import tarfile
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from cryptography.fernet import Fernet, InvalidToken
|
||||||
|
from cryptography.hazmat.primitives.kdf.scrypt import Scrypt
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover import __version__
|
||||||
|
from netmover.config import Settings
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
|
from netmover.logging_setup import event, get_logger
|
||||||
|
from netmover.models import CollisionPolicy, utcnow
|
||||||
|
from netmover.mover.destinations import MetaOptions
|
||||||
|
from netmover.settings_store import AppSettings, get_value, load_settings, set_value
|
||||||
|
from netmover.timeouts import CancelToken, run_with_watchdog
|
||||||
|
|
||||||
|
log = get_logger(__name__)
|
||||||
|
|
||||||
|
MAGIC = b"NETMOVER-BACKUP-1\n"
|
||||||
|
SALT_BYTES = 16
|
||||||
|
SCRYPT_N = 2**15
|
||||||
|
SCRYPT_R = 8
|
||||||
|
SCRYPT_P = 1
|
||||||
|
MIN_PASSPHRASE = 12
|
||||||
|
|
||||||
|
SUFFIX_ENC = ".nmbak"
|
||||||
|
SUFFIX_PLAIN = ".tar.gz"
|
||||||
|
NAME_RE = re.compile(r"^netmover-[A-Za-z0-9._-]+-\d{8}-\d{6}(\.nmbak|\.tar\.gz)$")
|
||||||
|
|
||||||
|
DB_NAME = "netmover.db"
|
||||||
|
KEY_FILES = ("secret.key", "session.key")
|
||||||
|
ENV_NAME = "netmover.env"
|
||||||
|
|
||||||
|
# Wann eine Sicherung als überfällig gilt (für /healthz und Kuma): zwei Tage plus
|
||||||
|
# Puffer – ein einzelner Ausfall des NAS über Nacht ist noch kein Alarm.
|
||||||
|
STALE_AFTER = timedelta(hours=50)
|
||||||
|
RETRY_AFTER = timedelta(hours=1)
|
||||||
|
IDLE_TIMEOUT = 120.0
|
||||||
|
|
||||||
|
STATE_PREFIX = "backup."
|
||||||
|
|
||||||
|
|
||||||
|
class BackupError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# -- Verschlüsselung -----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _derive(passphrase: str, salt: bytes) -> bytes:
|
||||||
|
kdf = Scrypt(salt=salt, length=32, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P)
|
||||||
|
return base64.urlsafe_b64encode(kdf.derive(passphrase.encode()))
|
||||||
|
|
||||||
|
|
||||||
|
def encrypt(data: bytes, passphrase: str) -> bytes:
|
||||||
|
salt = os.urandom(SALT_BYTES)
|
||||||
|
return MAGIC + salt + Fernet(_derive(passphrase, salt)).encrypt(data)
|
||||||
|
|
||||||
|
|
||||||
|
def is_encrypted(blob: bytes) -> bool:
|
||||||
|
return blob.startswith(MAGIC)
|
||||||
|
|
||||||
|
|
||||||
|
def decrypt(blob: bytes, passphrase: str) -> bytes:
|
||||||
|
if not is_encrypted(blob):
|
||||||
|
raise BackupError("Keine verschlüsselte NetMover-Sicherung")
|
||||||
|
salt = blob[len(MAGIC) : len(MAGIC) + SALT_BYTES]
|
||||||
|
token = blob[len(MAGIC) + SALT_BYTES :]
|
||||||
|
try:
|
||||||
|
return Fernet(_derive(passphrase, salt)).decrypt(token)
|
||||||
|
except InvalidToken:
|
||||||
|
raise BackupError("Passphrase falsch oder Sicherung beschädigt") from None
|
||||||
|
|
||||||
|
|
||||||
|
def passphrase_problem(passphrase: str, confirm: str) -> str | None:
|
||||||
|
if len(passphrase) < MIN_PASSPHRASE:
|
||||||
|
return f"Passphrase: mindestens {MIN_PASSPHRASE} Zeichen."
|
||||||
|
if passphrase != confirm:
|
||||||
|
return "Passphrasen stimmen nicht überein."
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# -- Archiv bauen und lesen ------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Archive:
|
||||||
|
manifest: dict[str, Any]
|
||||||
|
files: dict[str, bytes]
|
||||||
|
|
||||||
|
|
||||||
|
def snapshot_db(db_path: Path, target: Path) -> str:
|
||||||
|
"""Konsistente Kopie der laufenden Datenbank; liefert die Alembic-Revision."""
|
||||||
|
src = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
|
||||||
|
dst = sqlite3.connect(target)
|
||||||
|
try:
|
||||||
|
src.backup(dst)
|
||||||
|
status = dst.execute("PRAGMA integrity_check").fetchone()[0]
|
||||||
|
if status != "ok":
|
||||||
|
raise BackupError(f"Datenbank beschädigt: {status}")
|
||||||
|
row = dst.execute("SELECT version_num FROM alembic_version").fetchone()
|
||||||
|
finally:
|
||||||
|
dst.close()
|
||||||
|
src.close()
|
||||||
|
return str(row[0]) if row else ""
|
||||||
|
|
||||||
|
|
||||||
|
def _add(tar: tarfile.TarFile, name: str, data: bytes, mtime: float) -> None:
|
||||||
|
info = tarfile.TarInfo(name)
|
||||||
|
info.size = len(data)
|
||||||
|
info.mode = 0o600
|
||||||
|
info.mtime = int(mtime)
|
||||||
|
tar.addfile(info, io.BytesIO(data))
|
||||||
|
|
||||||
|
|
||||||
|
def build_archive(settings: Settings, now: datetime | None = None) -> tuple[bytes, dict[str, Any]]:
|
||||||
|
"""tar.gz mit Datenbank, Schlüsseln und Umgebungsdatei (unverschlüsselt)."""
|
||||||
|
now = now or utcnow()
|
||||||
|
with tempfile.TemporaryDirectory(dir=settings.data_dir, prefix=".backup-") as tmp:
|
||||||
|
snap = Path(tmp) / DB_NAME
|
||||||
|
revision = snapshot_db(settings.db_path, snap)
|
||||||
|
files: dict[str, bytes] = {DB_NAME: snap.read_bytes()}
|
||||||
|
for name in KEY_FILES:
|
||||||
|
path = settings.data_dir / name
|
||||||
|
if path.is_file():
|
||||||
|
files[name] = path.read_bytes()
|
||||||
|
if settings.config_file.is_file():
|
||||||
|
files[ENV_NAME] = settings.config_file.read_bytes()
|
||||||
|
manifest: dict[str, Any] = {
|
||||||
|
"format": 1,
|
||||||
|
"version": __version__,
|
||||||
|
"revision": revision,
|
||||||
|
"created": now.isoformat(),
|
||||||
|
"host": socket.gethostname(),
|
||||||
|
"files": sorted(files),
|
||||||
|
}
|
||||||
|
buf = io.BytesIO()
|
||||||
|
stamp = time.time()
|
||||||
|
with tarfile.open(fileobj=buf, mode="w:gz") as tar:
|
||||||
|
_add(tar, "manifest.json", json.dumps(manifest, indent=2).encode(), stamp)
|
||||||
|
for name, data in files.items():
|
||||||
|
_add(tar, name, data, stamp)
|
||||||
|
return buf.getvalue(), manifest
|
||||||
|
|
||||||
|
|
||||||
|
def open_archive(blob: bytes, passphrase: str | None) -> Archive:
|
||||||
|
"""Sicherung lesen und prüfen (Manifest, Datenbank-Integrität)."""
|
||||||
|
if is_encrypted(blob):
|
||||||
|
if not passphrase:
|
||||||
|
raise BackupError("Sicherung ist verschlüsselt – Passphrase angeben")
|
||||||
|
blob = decrypt(blob, passphrase)
|
||||||
|
files: dict[str, bytes] = {}
|
||||||
|
try:
|
||||||
|
with tarfile.open(fileobj=io.BytesIO(blob), mode="r:gz") as tar:
|
||||||
|
for member in tar.getmembers():
|
||||||
|
if not member.isfile() or "/" in member.name or member.name.startswith("."):
|
||||||
|
raise BackupError(f"Unerwarteter Eintrag in der Sicherung: {member.name}")
|
||||||
|
fh = tar.extractfile(member)
|
||||||
|
if fh is not None:
|
||||||
|
files[member.name] = fh.read()
|
||||||
|
except (tarfile.TarError, OSError, EOFError) as exc:
|
||||||
|
raise BackupError(f"Keine lesbare Sicherung: {exc}") from None
|
||||||
|
try:
|
||||||
|
manifest = json.loads(files.pop("manifest.json"))
|
||||||
|
except (KeyError, ValueError):
|
||||||
|
raise BackupError("Sicherung ohne gültiges manifest.json") from None
|
||||||
|
if manifest.get("format") != 1:
|
||||||
|
raise BackupError(f"Unbekanntes Sicherungsformat {manifest.get('format')!r}")
|
||||||
|
if DB_NAME not in files or "secret.key" not in files:
|
||||||
|
raise BackupError("Sicherung unvollständig: Datenbank oder secret.key fehlt")
|
||||||
|
_check_db_bytes(files[DB_NAME])
|
||||||
|
return Archive(manifest, files)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_db_bytes(data: bytes) -> None:
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = Path(tmp) / DB_NAME
|
||||||
|
path.write_bytes(data)
|
||||||
|
try:
|
||||||
|
con = sqlite3.connect(path)
|
||||||
|
try:
|
||||||
|
status = con.execute("PRAGMA integrity_check").fetchone()[0]
|
||||||
|
finally:
|
||||||
|
con.close()
|
||||||
|
except sqlite3.DatabaseError as exc:
|
||||||
|
raise BackupError(f"Datenbank in der Sicherung unlesbar: {exc}") from None
|
||||||
|
if status != "ok":
|
||||||
|
raise BackupError(f"Datenbank in der Sicherung beschädigt: {status}")
|
||||||
|
|
||||||
|
|
||||||
|
def backup_name(now: datetime, encrypted: bool, host: str | None = None) -> str:
|
||||||
|
host = re.sub(r"[^A-Za-z0-9._-]", "-", host or socket.gethostname()) or "host"
|
||||||
|
local = now.astimezone()
|
||||||
|
return f"netmover-{host}-{local:%Y%m%d-%H%M%S}{SUFFIX_ENC if encrypted else SUFFIX_PLAIN}"
|
||||||
|
|
||||||
|
|
||||||
|
def make_backup(
|
||||||
|
settings: Settings, passphrase: str | None, now: datetime | None = None
|
||||||
|
) -> tuple[bytes, dict[str, Any]]:
|
||||||
|
"""Fertige Sicherung (verschlüsselt, falls Passphrase) – bereits probeweise geöffnet."""
|
||||||
|
raw, manifest = build_archive(settings, now)
|
||||||
|
blob = encrypt(raw, passphrase) if passphrase else raw
|
||||||
|
open_archive(blob, passphrase)
|
||||||
|
return blob, manifest
|
||||||
|
|
||||||
|
|
||||||
|
def prune_names(names: list[str], keep: int) -> list[str]:
|
||||||
|
"""Die zu löschenden Sicherungen: alles über die ``keep`` neuesten hinaus.
|
||||||
|
Nur Dateien mit NetMover-Namensschema – sonst wird nichts angefasst."""
|
||||||
|
ours = sorted((n for n in names if NAME_RE.match(n)), key=_stamp_of, reverse=True)
|
||||||
|
return ours[max(keep, 1) :]
|
||||||
|
|
||||||
|
|
||||||
|
def _stamp_of(name: str) -> str:
|
||||||
|
m = re.search(r"-(\d{8}-\d{6})\.", name)
|
||||||
|
return m.group(1) if m else ""
|
||||||
|
|
||||||
|
|
||||||
|
# -- Wiederherstellen ----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _version_tuple(v: str) -> tuple[int, ...]:
|
||||||
|
return tuple(int(p) for p in re.findall(r"\d+", v)[:3])
|
||||||
|
|
||||||
|
|
||||||
|
def restore(settings: Settings, archive: Archive, now: datetime | None = None) -> Path:
|
||||||
|
"""Spielt eine Sicherung ein. Der Dienst muss gestoppt sein. Vorhandene Dateien
|
||||||
|
wandern vorher nach ``data_dir/vor-wiederherstellung-<zeit>/`` – nichts geht
|
||||||
|
verloren. Liefert diesen Ordner."""
|
||||||
|
version = str(archive.manifest.get("version", "0"))
|
||||||
|
if _version_tuple(version) > _version_tuple(__version__):
|
||||||
|
raise BackupError(
|
||||||
|
f"Sicherung stammt von NetMover {version}, installiert ist {__version__} – "
|
||||||
|
"erst aktualisieren, dann wiederherstellen."
|
||||||
|
)
|
||||||
|
now = now or utcnow()
|
||||||
|
data_dir = settings.data_dir
|
||||||
|
data_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
keep = data_dir / f"vor-wiederherstellung-{now.astimezone():%Y%m%d-%H%M%S}"
|
||||||
|
keep.mkdir(mode=0o700)
|
||||||
|
for name in (DB_NAME, f"{DB_NAME}-wal", f"{DB_NAME}-shm", *KEY_FILES):
|
||||||
|
path = data_dir / name
|
||||||
|
if path.exists():
|
||||||
|
path.rename(keep / name)
|
||||||
|
for name, data in archive.files.items():
|
||||||
|
if name == ENV_NAME:
|
||||||
|
continue
|
||||||
|
_write_private(data_dir / name, data)
|
||||||
|
if ENV_NAME in archive.files:
|
||||||
|
env = settings.config_file
|
||||||
|
if env.is_file() and env.read_bytes() != archive.files[ENV_NAME]:
|
||||||
|
_write_private(keep / ENV_NAME, env.read_bytes())
|
||||||
|
if env.parent.is_dir():
|
||||||
|
_write_private(env, archive.files[ENV_NAME])
|
||||||
|
event(log, "backup.restored", version=version, kept=str(keep))
|
||||||
|
return keep
|
||||||
|
|
||||||
|
|
||||||
|
def _write_private(path: Path, data: bytes) -> None:
|
||||||
|
tmp = path.with_name(f".{path.name}.restore")
|
||||||
|
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||||
|
with os.fdopen(fd, "wb") as fh:
|
||||||
|
fh.write(data)
|
||||||
|
fh.flush()
|
||||||
|
os.fsync(fh.fileno())
|
||||||
|
os.replace(tmp, path)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Dienst: tägliche Sicherung ------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class BackupState:
|
||||||
|
last_attempt: datetime | None = None
|
||||||
|
last_success: datetime | None = None
|
||||||
|
last_error: str = ""
|
||||||
|
last_file: str = ""
|
||||||
|
last_size: int = 0
|
||||||
|
failures: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
def _dt(raw: str | None) -> datetime | None:
|
||||||
|
if not raw:
|
||||||
|
return None
|
||||||
|
value = json.loads(raw)
|
||||||
|
return datetime.fromisoformat(value) if value else None
|
||||||
|
|
||||||
|
|
||||||
|
def load_state(factory: sessionmaker[Session]) -> BackupState:
|
||||||
|
def get(key: str) -> str | None:
|
||||||
|
return get_value(factory, STATE_PREFIX + key)
|
||||||
|
|
||||||
|
return BackupState(
|
||||||
|
last_attempt=_dt(get("last_attempt")),
|
||||||
|
last_success=_dt(get("last_success")),
|
||||||
|
last_error=json.loads(get("last_error") or '""'),
|
||||||
|
last_file=json.loads(get("last_file") or '""'),
|
||||||
|
last_size=json.loads(get("last_size") or "0"),
|
||||||
|
failures=json.loads(get("failures") or "0"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def save_state(factory: sessionmaker[Session], st: BackupState) -> None:
|
||||||
|
values: dict[str, Any] = {
|
||||||
|
"last_attempt": st.last_attempt.isoformat() if st.last_attempt else None,
|
||||||
|
"last_success": st.last_success.isoformat() if st.last_success else None,
|
||||||
|
"last_error": st.last_error,
|
||||||
|
"last_file": st.last_file,
|
||||||
|
"last_size": st.last_size,
|
||||||
|
"failures": st.failures,
|
||||||
|
}
|
||||||
|
for key, value in values.items():
|
||||||
|
set_value(factory, STATE_PREFIX + key, json.dumps(value))
|
||||||
|
|
||||||
|
|
||||||
|
def parse_time(value: str) -> tuple[int, int]:
|
||||||
|
m = re.fullmatch(r"([01]?\d|2[0-3]):([0-5]\d)", value.strip())
|
||||||
|
if not m:
|
||||||
|
raise ValueError("Uhrzeit als HH:MM, z. B. 03:30")
|
||||||
|
return int(m.group(1)), int(m.group(2))
|
||||||
|
|
||||||
|
|
||||||
|
def due_slot(cfg: AppSettings, now: datetime) -> datetime:
|
||||||
|
"""Der letzte Sicherungstermin (Ortszeit des Servers) vor ``now``."""
|
||||||
|
hour, minute = parse_time(cfg.backup_time)
|
||||||
|
local = now.astimezone()
|
||||||
|
slot = local.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||||
|
if slot > local:
|
||||||
|
slot -= timedelta(days=1)
|
||||||
|
return slot
|
||||||
|
|
||||||
|
|
||||||
|
def is_due(cfg: AppSettings, st: BackupState, now: datetime) -> bool:
|
||||||
|
if not cfg.backup_enabled or not cfg.backup_mount_id:
|
||||||
|
return False
|
||||||
|
slot = due_slot(cfg, now)
|
||||||
|
if st.last_success is not None and st.last_success >= slot:
|
||||||
|
return False
|
||||||
|
# Nach einem Fehlschlag stündlich erneut, nicht bei jedem Takt
|
||||||
|
return st.last_attempt is None or now - st.last_attempt >= RETRY_AFTER
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class BackupResult:
|
||||||
|
ok: bool
|
||||||
|
message: str
|
||||||
|
file: str = ""
|
||||||
|
size: int = 0
|
||||||
|
deleted: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
Notify = Callable[[str, str, str], None] # (event, title, message)
|
||||||
|
|
||||||
|
|
||||||
|
class BackupService:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
settings: Settings,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
destination_for: Callable[[int, str], Callable[[], Any]],
|
||||||
|
secrets: SecretBox,
|
||||||
|
notify: Notify | None = None,
|
||||||
|
) -> None:
|
||||||
|
self.settings = settings
|
||||||
|
self.session_factory = session_factory
|
||||||
|
self._destination_for = destination_for
|
||||||
|
self.secrets = secrets
|
||||||
|
self._notify = notify
|
||||||
|
self._running = False
|
||||||
|
|
||||||
|
@property
|
||||||
|
def running(self) -> bool:
|
||||||
|
return self._running
|
||||||
|
|
||||||
|
def passphrase(self, cfg: AppSettings) -> str | None:
|
||||||
|
if not cfg.backup_passphrase_enc:
|
||||||
|
return None
|
||||||
|
return self.secrets.decrypt(cfg.backup_passphrase_enc)
|
||||||
|
|
||||||
|
def tick(self, now: datetime | None = None) -> BackupResult | None:
|
||||||
|
"""Scheduler-Job: sichert, wenn ein Termin fällig ist."""
|
||||||
|
now = now or utcnow()
|
||||||
|
cfg = load_settings(self.session_factory)
|
||||||
|
if not is_due(cfg, load_state(self.session_factory), now):
|
||||||
|
return None
|
||||||
|
return self.run(now)
|
||||||
|
|
||||||
|
def run(self, now: datetime | None = None) -> BackupResult:
|
||||||
|
if self._running:
|
||||||
|
return BackupResult(False, "Eine Sicherung läuft bereits.")
|
||||||
|
self._running = True
|
||||||
|
try:
|
||||||
|
return self._run(now or utcnow())
|
||||||
|
finally:
|
||||||
|
self._running = False
|
||||||
|
|
||||||
|
def _run(self, now: datetime) -> BackupResult:
|
||||||
|
cfg = load_settings(self.session_factory)
|
||||||
|
st = load_state(self.session_factory)
|
||||||
|
st.last_attempt = now
|
||||||
|
try:
|
||||||
|
if not cfg.backup_mount_id:
|
||||||
|
raise BackupError("Kein Sicherungsziel eingestellt")
|
||||||
|
if not cfg.backup_passphrase_enc:
|
||||||
|
raise BackupError("Keine Passphrase gesetzt")
|
||||||
|
result = self._write(cfg, now)
|
||||||
|
except Exception as exc:
|
||||||
|
message = str(exc) or type(exc).__name__
|
||||||
|
st.failures += 1
|
||||||
|
st.last_error = message
|
||||||
|
save_state(self.session_factory, st)
|
||||||
|
event(log, "backup.failed", level=40, error=message, failures=st.failures)
|
||||||
|
if st.failures == 1 and self._notify:
|
||||||
|
self._notify(
|
||||||
|
"backup.failed",
|
||||||
|
"NetMover: Sicherung fehlgeschlagen",
|
||||||
|
f"{message}\nNächster Versuch in einer Stunde.",
|
||||||
|
)
|
||||||
|
return BackupResult(False, message)
|
||||||
|
recovered = st.failures > 0
|
||||||
|
st.failures = 0
|
||||||
|
st.last_error = ""
|
||||||
|
st.last_success = now
|
||||||
|
st.last_file = result.file
|
||||||
|
st.last_size = result.size
|
||||||
|
save_state(self.session_factory, st)
|
||||||
|
event(log, "backup.done", file=result.file, size=result.size, deleted=result.deleted)
|
||||||
|
if recovered and self._notify:
|
||||||
|
self._notify(
|
||||||
|
"backup.recovered",
|
||||||
|
"NetMover: Sicherung läuft wieder",
|
||||||
|
f"Sicherung {result.file} geschrieben.",
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def _write(self, cfg: AppSettings, now: datetime) -> BackupResult:
|
||||||
|
passphrase = self.passphrase(cfg)
|
||||||
|
blob, _manifest = make_backup(self.settings, passphrase, now)
|
||||||
|
name = backup_name(now, passphrase is not None)
|
||||||
|
make_dest = self._destination_for(cfg.backup_mount_id, cfg.backup_path)
|
||||||
|
|
||||||
|
def work(token: CancelToken) -> BackupResult:
|
||||||
|
dest = make_dest()
|
||||||
|
try:
|
||||||
|
dest.prepare()
|
||||||
|
with tempfile.TemporaryDirectory(dir=self.settings.data_dir) as tmp:
|
||||||
|
local = Path(tmp) / name
|
||||||
|
local.write_bytes(blob)
|
||||||
|
ns = time.time_ns()
|
||||||
|
result = dest.commit(
|
||||||
|
local,
|
||||||
|
len(blob),
|
||||||
|
name,
|
||||||
|
CollisionPolicy.RENAME,
|
||||||
|
MetaOptions(ns, ns, 0o600, None),
|
||||||
|
lambda: None,
|
||||||
|
token,
|
||||||
|
)
|
||||||
|
deleted = 0
|
||||||
|
for old in prune_names(dest.list_names(), cfg.backup_keep):
|
||||||
|
token.check()
|
||||||
|
with contextlib.suppress(OSError):
|
||||||
|
dest.remove(old)
|
||||||
|
deleted += 1
|
||||||
|
return BackupResult(
|
||||||
|
True,
|
||||||
|
f"Sicherung {dest.describe(result.final_rel)} geschrieben",
|
||||||
|
result.final_rel,
|
||||||
|
len(blob),
|
||||||
|
deleted,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
dest.close()
|
||||||
|
|
||||||
|
return run_with_watchdog(work, IDLE_TIMEOUT, "Sicherung")
|
||||||
+125
-7
@@ -1,14 +1,18 @@
|
|||||||
"""Kommandozeile: ``netmover serve`` / ``netmover migrate`` / ``netmover version``
|
"""Kommandozeile: ``netmover serve`` / ``netmover migrate`` / ``netmover version``
|
||||||
/ ``netmover reset-password``."""
|
/ ``netmover reset-password`` / ``netmover backup`` / ``netmover restore``."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import getpass
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
from collections.abc import Sequence
|
from collections.abc import Callable, Sequence
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
from netmover import __version__
|
from netmover import __version__
|
||||||
from netmover.config import get_settings
|
from netmover.config import Settings, get_settings
|
||||||
from netmover.db import upgrade
|
from netmover.db import upgrade
|
||||||
from netmover.logging_setup import event, get_logger, setup_logging
|
from netmover.logging_setup import event, get_logger, setup_logging
|
||||||
|
|
||||||
@@ -25,6 +29,17 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
"reset-password",
|
"reset-password",
|
||||||
help="Administratorpasswort löschen – beim nächsten Aufruf startet die Einrichtung",
|
help="Administratorpasswort löschen – beim nächsten Aufruf startet die Einrichtung",
|
||||||
)
|
)
|
||||||
|
b = sub.add_parser("backup", help="Sicherung in eine lokale Datei oder einen Ordner schreiben")
|
||||||
|
b.add_argument("out", type=Path, help="Zieldatei oder Ordner")
|
||||||
|
b.add_argument(
|
||||||
|
"--unverschluesselt",
|
||||||
|
action="store_true",
|
||||||
|
help="ohne Passphrase (die Datei enthält dann alle Zugangsdaten lesbar)",
|
||||||
|
)
|
||||||
|
r = sub.add_parser("restore", help="Sicherung einspielen (Dienst vorher stoppen)")
|
||||||
|
r.add_argument("file", type=Path, help="Sicherungsdatei (.nmbak oder .tar.gz)")
|
||||||
|
r.add_argument("--passphrase-file", type=Path, help="Passphrase aus Datei statt Abfrage")
|
||||||
|
r.add_argument("--ja", action="store_true", help="ohne Rückfrage einspielen")
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
settings = get_settings()
|
settings = get_settings()
|
||||||
@@ -64,16 +79,119 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
row = s.get(Setting, PASSWORD_KEY)
|
row = s.get(Setting, PASSWORD_KEY)
|
||||||
if row is not None:
|
if row is not None:
|
||||||
s.delete(row)
|
s.delete(row)
|
||||||
# Anmeldung wieder einschalten (falls abgeschaltet)
|
# Anmeldung wieder überall einschalten und Netz-Sperren aufheben
|
||||||
mode = s.get(Setting, "app.auth_mode")
|
for key, value in (
|
||||||
if mode is not None:
|
("app.auth_mode", '"always"'),
|
||||||
mode.value = '"always"'
|
("app.trusted_networks", '""'),
|
||||||
|
("app.allowed_networks", '""'),
|
||||||
|
):
|
||||||
|
setting = s.get(Setting, key)
|
||||||
|
if setting is not None:
|
||||||
|
setting.value = value
|
||||||
s.commit()
|
s.commit()
|
||||||
engine.dispose()
|
engine.dispose()
|
||||||
print("Passwort gelöscht. Dienst neu starten; der Einrichtungscode steht im Log.")
|
print("Passwort gelöscht. Dienst neu starten; der Einrichtungscode steht im Log.")
|
||||||
return 0
|
return 0
|
||||||
|
if args.cmd == "backup":
|
||||||
|
return cmd_backup(settings, args.out, encrypted=not args.unverschluesselt)
|
||||||
|
if args.cmd == "restore":
|
||||||
|
return cmd_restore(settings, args.file, args.passphrase_file, assume_yes=args.ja)
|
||||||
return 2
|
return 2
|
||||||
|
|
||||||
|
|
||||||
|
def service_active() -> bool:
|
||||||
|
if shutil.which("systemctl") is None:
|
||||||
|
return False
|
||||||
|
result = subprocess.run(
|
||||||
|
["systemctl", "is-active", "--quiet", "netmover"], # noqa: S607
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
return result.returncode == 0
|
||||||
|
|
||||||
|
|
||||||
|
def _ask_passphrase(confirm: bool) -> str:
|
||||||
|
from netmover.backup import passphrase_problem
|
||||||
|
|
||||||
|
first = getpass.getpass("Passphrase: ")
|
||||||
|
if not confirm:
|
||||||
|
return first
|
||||||
|
problem = passphrase_problem(first, getpass.getpass("Wiederholen: "))
|
||||||
|
if problem:
|
||||||
|
raise SystemExit(problem)
|
||||||
|
return first
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_backup(settings: Settings, out: Path, *, encrypted: bool) -> int:
|
||||||
|
from netmover.backup import backup_name, make_backup
|
||||||
|
from netmover.crypto import SecretBox, load_or_create_key
|
||||||
|
from netmover.db import make_engine, make_session_factory
|
||||||
|
from netmover.models import utcnow
|
||||||
|
from netmover.settings_store import load_settings
|
||||||
|
|
||||||
|
passphrase: str | None = None
|
||||||
|
if encrypted:
|
||||||
|
engine = make_engine(settings.effective_database_url)
|
||||||
|
cfg = load_settings(make_session_factory(engine))
|
||||||
|
engine.dispose()
|
||||||
|
if cfg.backup_passphrase_enc:
|
||||||
|
box = SecretBox(load_or_create_key(settings.secret_key_path))
|
||||||
|
passphrase = box.decrypt(cfg.backup_passphrase_enc)
|
||||||
|
print("Verschlüsselt mit der Passphrase aus den Einstellungen.")
|
||||||
|
else:
|
||||||
|
passphrase = _ask_passphrase(confirm=True)
|
||||||
|
now = utcnow()
|
||||||
|
blob, manifest = make_backup(settings, passphrase, now)
|
||||||
|
target = out / backup_name(now, encrypted) if out.is_dir() else out
|
||||||
|
fd_path = target.with_name(f".{target.name}.part")
|
||||||
|
fd_path.write_bytes(blob)
|
||||||
|
fd_path.chmod(0o600)
|
||||||
|
fd_path.replace(target)
|
||||||
|
print(f"Sicherung geschrieben: {target} ({len(blob)} Bytes, NetMover {manifest['version']})")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_restore(
|
||||||
|
settings: Settings,
|
||||||
|
file: Path,
|
||||||
|
passphrase_file: Path | None,
|
||||||
|
*,
|
||||||
|
assume_yes: bool,
|
||||||
|
active: Callable[[], bool] = service_active,
|
||||||
|
) -> int:
|
||||||
|
from netmover.backup import BackupError, is_encrypted, open_archive, restore
|
||||||
|
|
||||||
|
if active():
|
||||||
|
print("Der Dienst läuft – erst `systemctl stop netmover`, dann wiederherstellen.")
|
||||||
|
return 1
|
||||||
|
blob = file.read_bytes()
|
||||||
|
passphrase: str | None = None
|
||||||
|
if is_encrypted(blob):
|
||||||
|
passphrase = (
|
||||||
|
passphrase_file.read_text().strip() if passphrase_file else _ask_passphrase(False)
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
archive = open_archive(blob, passphrase)
|
||||||
|
except BackupError as exc:
|
||||||
|
print(f"Fehler: {exc}")
|
||||||
|
return 1
|
||||||
|
m = archive.manifest
|
||||||
|
print(
|
||||||
|
f"Sicherung von {m.get('host')} vom {m.get('created')} (NetMover {m.get('version')}),"
|
||||||
|
f" enthält: {', '.join(sorted(archive.files))}"
|
||||||
|
)
|
||||||
|
if not assume_yes and input(f"Nach {settings.data_dir} einspielen? [ja/nein] ") != "ja":
|
||||||
|
print("Abgebrochen, nichts verändert.")
|
||||||
|
return 1
|
||||||
|
try:
|
||||||
|
kept = restore(settings, archive)
|
||||||
|
except BackupError as exc:
|
||||||
|
print(f"Fehler: {exc}")
|
||||||
|
return 1
|
||||||
|
upgrade(settings.effective_database_url)
|
||||||
|
print(f"Wiederhergestellt. Bisherige Dateien liegen in {kept}.")
|
||||||
|
print("Jetzt `systemctl start netmover`.")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
sys.exit(main())
|
sys.exit(main())
|
||||||
|
|||||||
@@ -21,6 +21,8 @@ class Settings(BaseSettings):
|
|||||||
data_dir: Path = Path("/var/lib/netmover")
|
data_dir: Path = Path("/var/lib/netmover")
|
||||||
run_dir: Path = Path("/run/netmover")
|
run_dir: Path = Path("/run/netmover")
|
||||||
mount_base: Path = Path("/mnt/netmover")
|
mount_base: Path = Path("/mnt/netmover")
|
||||||
|
# Umgebungsdatei – wird mitgesichert und wiederhergestellt
|
||||||
|
config_file: Path = Path("/etc/netmover/netmover.env")
|
||||||
|
|
||||||
host: str = "127.0.0.1"
|
host: str = "127.0.0.1"
|
||||||
port: int = Field(default=8080, ge=1, le=65535)
|
port: int = Field(default=8080, ge=1, le=65535)
|
||||||
|
|||||||
+104
-3
@@ -19,11 +19,21 @@ from __future__ import annotations
|
|||||||
from dataclasses import asdict, dataclass, field
|
from dataclasses import asdict, dataclass, field
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
from sqlalchemy import select
|
from sqlalchemy import func, select
|
||||||
from sqlalchemy.orm import Session, sessionmaker
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
from netmover.models import Mount, MountState, Rule, utcnow
|
from netmover.models import (
|
||||||
|
FileFailure,
|
||||||
|
ItemStatus,
|
||||||
|
Mount,
|
||||||
|
MountState,
|
||||||
|
Rule,
|
||||||
|
Run,
|
||||||
|
RunItem,
|
||||||
|
RunStatus,
|
||||||
|
utcnow,
|
||||||
|
)
|
||||||
|
|
||||||
STALE_FACTOR = 3
|
STALE_FACTOR = 3
|
||||||
FAILURES_ALERT = 3
|
FAILURES_ALERT = 3
|
||||||
@@ -32,7 +42,7 @@ HEARTBEAT_MAX_AGE = timedelta(minutes=3)
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class Check:
|
class Check:
|
||||||
kind: str # "scheduler" | "mount" | "rule"
|
kind: str # "scheduler" | "mount" | "rule" | "backup"
|
||||||
name: str
|
name: str
|
||||||
ok: bool
|
ok: bool
|
||||||
message: str
|
message: str
|
||||||
@@ -146,3 +156,94 @@ def check_health(
|
|||||||
health.stale_rules.add(r.id)
|
health.stale_rules.add(r.id)
|
||||||
health.add(check)
|
health.add(check)
|
||||||
return health
|
return health
|
||||||
|
|
||||||
|
|
||||||
|
def backup_check(
|
||||||
|
*,
|
||||||
|
enabled: bool,
|
||||||
|
last_success: datetime | None,
|
||||||
|
last_error: str,
|
||||||
|
stale_after: timedelta,
|
||||||
|
service_started: datetime,
|
||||||
|
now: datetime | None = None,
|
||||||
|
) -> Check:
|
||||||
|
"""Datensicherung: überfällig, wenn länger als ``stale_after`` keine gelungen ist
|
||||||
|
(Bezug wie bei Regeln: spätester von letztem Erfolg und Dienststart)."""
|
||||||
|
now = now or utcnow()
|
||||||
|
|
||||||
|
def mk(ok: bool, message: str) -> Check:
|
||||||
|
return Check("backup", "Sicherung", ok, message)
|
||||||
|
|
||||||
|
if not enabled:
|
||||||
|
return mk(True, "Sicherung ausgeschaltet")
|
||||||
|
reference = max(d for d in (last_success, service_started) if d is not None)
|
||||||
|
if now - reference > stale_after:
|
||||||
|
when = "noch nie" if last_success is None else f"seit {last_success:%Y-%m-%d %H:%M} UTC"
|
||||||
|
detail = f": {last_error}" if last_error else ""
|
||||||
|
return mk(False, f"Sicherung überfällig – keine erfolgreiche {when}{detail}")
|
||||||
|
if last_success is None:
|
||||||
|
return mk(True, "Sicherung eingerichtet, noch keine gelaufen")
|
||||||
|
note = f" (letzter Versuch fehlgeschlagen: {last_error})" if last_error else ""
|
||||||
|
return mk(True, f"Letzte Sicherung {last_success:%Y-%m-%d %H:%M} UTC{note}")
|
||||||
|
|
||||||
|
|
||||||
|
METRICS_WINDOW = timedelta(hours=24)
|
||||||
|
COUNTERS = ("files_24h", "failed_24h", "quarantined_24h", "errors_24h", "pending_failures")
|
||||||
|
|
||||||
|
|
||||||
|
def metrics(
|
||||||
|
session_factory: sessionmaker[Session], now: datetime | None = None
|
||||||
|
) -> dict[str, object]:
|
||||||
|
"""Kennzahlen für Uptime Kuma („HTTP(s) – JSON-Abfrage“) und andere Monitore:
|
||||||
|
gesamt und je Regel (Schlüssel = Regel-ID als Text). Probeläufe zählen nicht."""
|
||||||
|
since = (now or utcnow()) - METRICS_WINDOW
|
||||||
|
counts: dict[int, dict[str, int]] = {}
|
||||||
|
last_moved: dict[int, datetime] = {}
|
||||||
|
|
||||||
|
def entry(rule_id: int) -> dict[str, int]:
|
||||||
|
return counts.setdefault(rule_id, dict.fromkeys(COUNTERS, 0))
|
||||||
|
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
for (rid,) in s.execute(select(Rule.id)):
|
||||||
|
entry(rid)
|
||||||
|
real = (Run.dry_run.is_(False), Run.started_at >= since)
|
||||||
|
for rid, moved, failed, errors in s.execute(
|
||||||
|
select(
|
||||||
|
Run.rule_id,
|
||||||
|
func.coalesce(func.sum(Run.moved), 0),
|
||||||
|
func.coalesce(func.sum(Run.failed), 0),
|
||||||
|
func.count().filter(Run.status == RunStatus.ERROR),
|
||||||
|
)
|
||||||
|
.where(*real)
|
||||||
|
.group_by(Run.rule_id)
|
||||||
|
):
|
||||||
|
entry(rid).update(files_24h=moved, failed_24h=failed, errors_24h=errors)
|
||||||
|
for rid, n in s.execute(
|
||||||
|
select(Run.rule_id, func.count())
|
||||||
|
.join(RunItem, RunItem.run_id == Run.id)
|
||||||
|
.where(*real, RunItem.status == ItemStatus.QUARANTINED)
|
||||||
|
.group_by(Run.rule_id)
|
||||||
|
):
|
||||||
|
entry(rid)["quarantined_24h"] = n
|
||||||
|
for rid, n in s.execute(
|
||||||
|
select(FileFailure.rule_id, func.count()).group_by(FileFailure.rule_id)
|
||||||
|
):
|
||||||
|
entry(rid)["pending_failures"] = n
|
||||||
|
for rid, latest in s.execute(
|
||||||
|
select(Run.rule_id, func.max(Run.finished_at))
|
||||||
|
.where(Run.dry_run.is_(False), Run.moved > 0)
|
||||||
|
.group_by(Run.rule_id)
|
||||||
|
):
|
||||||
|
if latest is not None:
|
||||||
|
last_moved[rid] = latest
|
||||||
|
|
||||||
|
def iso(d: datetime | None) -> str | None:
|
||||||
|
return d.isoformat() if d else None
|
||||||
|
|
||||||
|
total: dict[str, object] = {k: sum(c[k] for c in counts.values()) for k in COUNTERS}
|
||||||
|
total["last_moved_at"] = iso(max(last_moved.values(), default=None))
|
||||||
|
total["rules"] = {
|
||||||
|
str(rid): {**c, "last_moved_at": iso(last_moved.get(rid))}
|
||||||
|
for rid, c in sorted(counts.items())
|
||||||
|
}
|
||||||
|
return total
|
||||||
|
|||||||
+20
-4
@@ -26,9 +26,20 @@ MAX_MESSAGE = 250
|
|||||||
Getter = Callable[..., Any]
|
Getter = Callable[..., Any]
|
||||||
|
|
||||||
|
|
||||||
def push_url(url: str, health: Health) -> str:
|
def ok_message(metrics: dict[str, object] | None) -> str:
|
||||||
|
"""Kurzer Text für die Kuma-Ansicht, z. B. „OK · 24 h: 12 Dateien“."""
|
||||||
|
if not metrics:
|
||||||
|
return "OK"
|
||||||
|
parts = [f"24 h: {metrics.get('files_24h', 0)} Dateien"]
|
||||||
|
failed = metrics.get("failed_24h") or 0
|
||||||
|
if failed:
|
||||||
|
parts.append(f"{failed} Fehler")
|
||||||
|
return "OK · " + ", ".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def push_url(url: str, health: Health, metrics: dict[str, object] | None = None) -> str:
|
||||||
"""Push-URL mit aktuellem Zustand (vorhandene Parameter werden ersetzt)."""
|
"""Push-URL mit aktuellem Zustand (vorhandene Parameter werden ersetzt)."""
|
||||||
msg = "OK" if health.ok else "; ".join(health.problems)
|
msg = ok_message(metrics) if health.ok else "; ".join(health.problems)
|
||||||
if len(msg) > MAX_MESSAGE:
|
if len(msg) > MAX_MESSAGE:
|
||||||
msg = msg[: MAX_MESSAGE - 1] + "…"
|
msg = msg[: MAX_MESSAGE - 1] + "…"
|
||||||
target = httpx.URL(url)
|
target = httpx.URL(url)
|
||||||
@@ -37,11 +48,16 @@ def push_url(url: str, health: Health) -> str:
|
|||||||
return str(target)
|
return str(target)
|
||||||
|
|
||||||
|
|
||||||
def push(url: str, health: Health, getter: Getter | None = None) -> bool:
|
def push(
|
||||||
|
url: str,
|
||||||
|
health: Health,
|
||||||
|
getter: Getter | None = None,
|
||||||
|
metrics: dict[str, object] | None = None,
|
||||||
|
) -> bool:
|
||||||
"""Meldet den Zustand an Kuma. Fehler werden protokolliert, nie weitergereicht –
|
"""Meldet den Zustand an Kuma. Fehler werden protokolliert, nie weitergereicht –
|
||||||
die URL nicht, weil sie das Token enthält."""
|
die URL nicht, weil sie das Token enthält."""
|
||||||
try:
|
try:
|
||||||
resp = (getter or httpx.get)(push_url(url, health), timeout=10)
|
resp = (getter or httpx.get)(push_url(url, health, metrics), timeout=10)
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
event(log, "kuma.push_failed", level=30, error=type(exc).__name__)
|
event(log, "kuma.push_failed", level=30, error=type(exc).__name__)
|
||||||
|
|||||||
@@ -0,0 +1,103 @@
|
|||||||
|
"""archive
|
||||||
|
|
||||||
|
Revision ID: 0004
|
||||||
|
Revises: 0003
|
||||||
|
Create Date: 2026-10-12
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
import netmover.models
|
||||||
|
|
||||||
|
revision: str = "0004"
|
||||||
|
down_revision: str | None = "0003"
|
||||||
|
branch_labels: str | Sequence[str] | None = None
|
||||||
|
depends_on: str | Sequence[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table(
|
||||||
|
"archived_files",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("rule_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("src_relpath", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("size", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("mtime_ns", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("archive_path", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("archived_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["rule_id"],
|
||||||
|
["rules.id"],
|
||||||
|
name=op.f("fk_archived_files_rule_id_rules"),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_archived_files")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"rule_id",
|
||||||
|
"src_relpath",
|
||||||
|
"size",
|
||||||
|
"mtime_ns",
|
||||||
|
name=op.f("uq_archived_files_rule_id_src_relpath_size_mtime_ns"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
with op.batch_alter_table("rules", schema=None) as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"archive_kind",
|
||||||
|
sa.Enum(
|
||||||
|
"none",
|
||||||
|
"mount",
|
||||||
|
"local",
|
||||||
|
name="archive_kind",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
server_default="none",
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(sa.Column("archive_mount_id", sa.Integer(), nullable=True))
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("archive_path", sa.String(length=1024), server_default="", nullable=False)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"archive_layout",
|
||||||
|
sa.Enum(
|
||||||
|
"flat",
|
||||||
|
"date",
|
||||||
|
name="archive_layout",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
server_default="flat",
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.create_foreign_key(
|
||||||
|
batch_op.f("fk_rules_archive_mount_id_mounts"),
|
||||||
|
"mounts",
|
||||||
|
["archive_mount_id"],
|
||||||
|
["id"],
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("rules", schema=None) as batch_op:
|
||||||
|
batch_op.drop_constraint(batch_op.f("fk_rules_archive_mount_id_mounts"), type_="foreignkey")
|
||||||
|
# CHECK-Regeln der Auswahlfelder zuerst entfernen – sonst verweisen sie beim
|
||||||
|
# Neuaufbau der Tabelle auf Spalten, die es nicht mehr gibt.
|
||||||
|
batch_op.drop_constraint(batch_op.f("ck_rules_archive_kind"), type_="check")
|
||||||
|
batch_op.drop_constraint(batch_op.f("ck_rules_archive_layout"), type_="check")
|
||||||
|
batch_op.drop_column("archive_layout")
|
||||||
|
batch_op.drop_column("archive_path")
|
||||||
|
batch_op.drop_column("archive_mount_id")
|
||||||
|
batch_op.drop_column("archive_kind")
|
||||||
|
|
||||||
|
op.drop_table("archived_files")
|
||||||
@@ -10,6 +10,7 @@ Abweichungen bzw. Ergänzungen gegenüber dem Mindestumfang aus der Spezifikatio
|
|||||||
* ``credentials.managed`` + ``mounts.username`` (Migration 0002) –
|
* ``credentials.managed`` + ``mounts.username`` (Migration 0002) –
|
||||||
Schlüsselverwaltung: gespeicherte SSH-Schlüssel, die mehrere Anbindungen mit
|
Schlüsselverwaltung: gespeicherte SSH-Schlüssel, die mehrere Anbindungen mit
|
||||||
jeweils eigenem Benutzernamen nutzen.
|
jeweils eigenem Benutzernamen nutzen.
|
||||||
|
* ``rules.archive_*`` + Tabelle ``archived_files`` (Migration 0004) – Archivkopie.
|
||||||
* ``rules.waiting_since`` / ``rules.waiting_reason`` (Migration 0003) – Regel wartet
|
* ``rules.waiting_since`` / ``rules.waiting_reason`` (Migration 0003) – Regel wartet
|
||||||
auf eine nicht erreichbare Anbindung (automatisch pausiert).
|
auf eine nicht erreichbare Anbindung (automatisch pausiert).
|
||||||
* ``mounts.state_since`` – für „hängt seit …“ / „gestört seit …“ auf der Statusseite.
|
* ``mounts.state_since`` – für „hängt seit …“ / „gestört seit …“ auf der Statusseite.
|
||||||
@@ -138,6 +139,17 @@ class CollisionPolicy(enum.StrEnum):
|
|||||||
OVERWRITE = "overwrite"
|
OVERWRITE = "overwrite"
|
||||||
|
|
||||||
|
|
||||||
|
class ArchiveKind(enum.StrEnum):
|
||||||
|
NONE = "none"
|
||||||
|
MOUNT = "mount"
|
||||||
|
LOCAL = "local"
|
||||||
|
|
||||||
|
|
||||||
|
class ArchiveLayout(enum.StrEnum):
|
||||||
|
FLAT = "flat" # wie in der Quelle
|
||||||
|
DATE = "date" # Jahr/Monat/…, nach Tag der Verarbeitung
|
||||||
|
|
||||||
|
|
||||||
class RunStatus(enum.StrEnum):
|
class RunStatus(enum.StrEnum):
|
||||||
RUNNING = "running"
|
RUNNING = "running"
|
||||||
OK = "ok" # keine Dateifehler
|
OK = "ok" # keine Dateifehler
|
||||||
@@ -259,6 +271,18 @@ class Rule(Base):
|
|||||||
quarantine_path: Mapped[str | None] = mapped_column(String(1024))
|
quarantine_path: Mapped[str | None] = mapped_column(String(1024))
|
||||||
quarantine_after: Mapped[int] = mapped_column(Integer, default=3)
|
quarantine_after: Mapped[int] = mapped_column(Integer, default=3)
|
||||||
|
|
||||||
|
# Archiv (Migration 0004): zusätzliche Kopie; Quelle erst löschen, wenn Ziel UND Archiv da
|
||||||
|
archive_kind: Mapped[ArchiveKind] = mapped_column(
|
||||||
|
_enum(ArchiveKind, "archive_kind"), default=ArchiveKind.NONE, server_default="none"
|
||||||
|
)
|
||||||
|
archive_mount_id: Mapped[int | None] = mapped_column(
|
||||||
|
ForeignKey("mounts.id", ondelete="RESTRICT")
|
||||||
|
)
|
||||||
|
archive_path: Mapped[str] = mapped_column(String(1024), default="", server_default="")
|
||||||
|
archive_layout: Mapped[ArchiveLayout] = mapped_column(
|
||||||
|
_enum(ArchiveLayout, "archive_layout"), default=ArchiveLayout.FLAT, server_default="flat"
|
||||||
|
)
|
||||||
|
|
||||||
last_run_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
last_run_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
last_success_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
last_success_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
consecutive_failures: Mapped[int] = mapped_column(Integer, default=0)
|
consecutive_failures: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
@@ -269,6 +293,7 @@ class Rule(Base):
|
|||||||
|
|
||||||
src_mount: Mapped[Mount] = relationship(foreign_keys=[src_mount_id])
|
src_mount: Mapped[Mount] = relationship(foreign_keys=[src_mount_id])
|
||||||
dst_mount: Mapped[Mount | None] = relationship(foreign_keys=[dst_mount_id])
|
dst_mount: Mapped[Mount | None] = relationship(foreign_keys=[dst_mount_id])
|
||||||
|
archive_mount: Mapped[Mount | None] = relationship(foreign_keys=[archive_mount_id])
|
||||||
runs: Mapped[list[Run]] = relationship(
|
runs: Mapped[list[Run]] = relationship(
|
||||||
back_populates="rule", cascade="all, delete-orphan", passive_deletes=True
|
back_populates="rule", cascade="all, delete-orphan", passive_deletes=True
|
||||||
)
|
)
|
||||||
@@ -330,6 +355,22 @@ class CopiedFile(Base):
|
|||||||
copied_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
copied_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
|
|
||||||
|
class ArchivedFile(Base):
|
||||||
|
"""Welche Dateiversion schon im Archiv liegt – damit ein Neuversuch nach einem
|
||||||
|
gescheiterten Ziel keine zweite Archivkopie anlegt."""
|
||||||
|
|
||||||
|
__tablename__ = "archived_files"
|
||||||
|
__table_args__ = (UniqueConstraint("rule_id", "src_relpath", "size", "mtime_ns"),)
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
rule_id: Mapped[int] = mapped_column(ForeignKey("rules.id", ondelete="CASCADE"))
|
||||||
|
src_relpath: Mapped[str] = mapped_column(String(2048))
|
||||||
|
size: Mapped[int] = mapped_column(BigInteger)
|
||||||
|
mtime_ns: Mapped[int] = mapped_column(BigInteger)
|
||||||
|
archive_path: Mapped[str] = mapped_column(String(2048))
|
||||||
|
archived_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
|
|
||||||
class FileFailure(Base):
|
class FileFailure(Base):
|
||||||
"""Fehlversuche pro Datei, Grundlage für die Quarantäne."""
|
"""Fehlversuche pro Datei, Grundlage für die Quarantäne."""
|
||||||
|
|
||||||
|
|||||||
@@ -109,6 +109,13 @@ class Destination(ABC):
|
|||||||
def close(self) -> None: # noqa: B027 – absichtlich optional
|
def close(self) -> None: # noqa: B027 – absichtlich optional
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def list_names(self) -> list[str]:
|
||||||
|
"""Dateinamen direkt im Zielordner (für das Ausdünnen von Sicherungen)."""
|
||||||
|
|
||||||
|
def remove(self, rel: str) -> None:
|
||||||
|
self._remove(rel)
|
||||||
|
|
||||||
# -- Primitive der Unterklassen -------------------------------------------------
|
# -- Primitive der Unterklassen -------------------------------------------------
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
@@ -235,6 +242,9 @@ class LocalDestination(Destination):
|
|||||||
def exists(self, rel: str) -> bool:
|
def exists(self, rel: str) -> bool:
|
||||||
return os.path.lexists(self._p(rel))
|
return os.path.lexists(self._p(rel))
|
||||||
|
|
||||||
|
def list_names(self) -> list[str]:
|
||||||
|
return [e.name for e in os.scandir(self.root) if e.is_file(follow_symlinks=False)]
|
||||||
|
|
||||||
def free_bytes(self) -> int | None:
|
def free_bytes(self) -> int | None:
|
||||||
st = os.statvfs(self.root)
|
st = os.statvfs(self.root)
|
||||||
return st.f_bavail * st.f_frsize
|
return st.f_bavail * st.f_frsize
|
||||||
@@ -421,6 +431,13 @@ class SftpDestination(Destination):
|
|||||||
return False
|
return False
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
def list_names(self) -> list[str]:
|
||||||
|
return [
|
||||||
|
a.filename
|
||||||
|
for a in self.client.listdir_attr(self.base)
|
||||||
|
if a.st_mode is not None and stat.S_ISREG(a.st_mode)
|
||||||
|
]
|
||||||
|
|
||||||
def free_bytes(self) -> int | None:
|
def free_bytes(self) -> int | None:
|
||||||
usage = self.session.statvfs(self.base)
|
usage = self.session.statvfs(self.base)
|
||||||
return usage[1] if usage else None
|
return usage[1] if usage else None
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ import posixpath
|
|||||||
import time
|
import time
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
@@ -72,6 +73,7 @@ class RuleSnapshot:
|
|||||||
set_mode: int | None = None
|
set_mode: int | None = None
|
||||||
quarantine_after: int = 3
|
quarantine_after: int = 3
|
||||||
dst_root_squash: bool | None = None # nur NFS-Ziele: Ergebnis des Schreibtests
|
dst_root_squash: bool | None = None # nur NFS-Ziele: Ergebnis des Schreibtests
|
||||||
|
archive_layout: str = "flat" # "flat" | "date" (Jahr/Monat/…)
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -159,6 +161,7 @@ class Engine:
|
|||||||
dry_run: bool = False,
|
dry_run: bool = False,
|
||||||
quarantine_rel: str | None = None,
|
quarantine_rel: str | None = None,
|
||||||
quarantine_root: Path | None = None,
|
quarantine_root: Path | None = None,
|
||||||
|
archive: Destination | None = None,
|
||||||
) -> RunResult:
|
) -> RunResult:
|
||||||
"""``source``: eine :class:`Source` oder (Kurzform) ein lokales Verzeichnis.
|
"""``source``: eine :class:`Source` oder (Kurzform) ein lokales Verzeichnis.
|
||||||
Quarantäne relativ zur Wurzel der Anbindung (``quarantine_rel``) oder – bei
|
Quarantäne relativ zur Wurzel der Anbindung (``quarantine_rel``) oder – bei
|
||||||
@@ -168,7 +171,7 @@ class Engine:
|
|||||||
if quarantine_root is not None and isinstance(src, LocalSource):
|
if quarantine_root is not None and isinstance(src, LocalSource):
|
||||||
quarantine_rel = str(quarantine_root.relative_to(src.root))
|
quarantine_rel = str(quarantine_root.relative_to(src.root))
|
||||||
try:
|
try:
|
||||||
self._execute(rule, src, dest, token, result, dry_run, quarantine_rel)
|
self._execute(rule, src, dest, token, result, dry_run, quarantine_rel, archive)
|
||||||
except Cancelled:
|
except Cancelled:
|
||||||
result.status = RunStatus.ERROR
|
result.status = RunStatus.ERROR
|
||||||
result.message = "Lauf abgebrochen"
|
result.message = "Lauf abgebrochen"
|
||||||
@@ -195,10 +198,13 @@ class Engine:
|
|||||||
result: RunResult,
|
result: RunResult,
|
||||||
dry_run: bool,
|
dry_run: bool,
|
||||||
quarantine_rel: str | None,
|
quarantine_rel: str | None,
|
||||||
|
archive: Destination | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
src.check_root()
|
src.check_root()
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
dest.prepare()
|
dest.prepare()
|
||||||
|
if archive is not None:
|
||||||
|
archive.prepare()
|
||||||
token.check()
|
token.check()
|
||||||
|
|
||||||
exclude = frozenset({quarantine_rel}) if quarantine_rel else frozenset()
|
exclude = frozenset({quarantine_rel}) if quarantine_rel else frozenset()
|
||||||
@@ -212,8 +218,12 @@ class Engine:
|
|||||||
# Kopier-Buchführung (Fallstrick 11). Gilt auch für „verschieben“, falls
|
# Kopier-Buchführung (Fallstrick 11). Gilt auch für „verschieben“, falls
|
||||||
# dort einmal die Quelle nicht gelöscht werden konnte.
|
# dort einmal die Quelle nicht gelöscht werden konnte.
|
||||||
done = self.ledger.copied_keys(rule.id)
|
done = self.ledger.copied_keys(rule.id)
|
||||||
|
archived = self.ledger.archived_keys(rule.id) if archive is not None else set()
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
self.ledger.prune_copied(rule.id, {c.relpath for c in candidates})
|
present = {c.relpath for c in candidates}
|
||||||
|
self.ledger.prune_copied(rule.id, present)
|
||||||
|
if archive is not None:
|
||||||
|
self.ledger.prune_archived(rule.id, present)
|
||||||
fresh = [c for c in candidates if (c.relpath, c.state.size, c.state.mtime_ns) not in done]
|
fresh = [c for c in candidates if (c.relpath, c.state.size, c.state.mtime_ns) not in done]
|
||||||
unchanged = len(candidates) - len(fresh)
|
unchanged = len(candidates) - len(fresh)
|
||||||
|
|
||||||
@@ -271,7 +281,11 @@ class Engine:
|
|||||||
failures = self.ledger.failures(rule.id)
|
failures = self.ledger.failures(rule.id)
|
||||||
for c in stable:
|
for c in stable:
|
||||||
token.check()
|
token.check()
|
||||||
result.add(self._process(rule, c, src, dest, token, dry_run, quarantine_rel, failures))
|
result.add(
|
||||||
|
self._process(
|
||||||
|
rule, c, src, dest, token, dry_run, quarantine_rel, failures, archive, archived
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
if unchanged:
|
if unchanged:
|
||||||
result.message = f"{unchanged} bereits übertragen"
|
result.message = f"{unchanged} bereits übertragen"
|
||||||
@@ -309,8 +323,14 @@ class Engine:
|
|||||||
dry_run: bool,
|
dry_run: bool,
|
||||||
quarantine_rel: str | None,
|
quarantine_rel: str | None,
|
||||||
failures: dict[str, FailureInfo],
|
failures: dict[str, FailureInfo],
|
||||||
|
archive: Destination | None = None,
|
||||||
|
archived: set[tuple[str, int, int]] | None = None,
|
||||||
) -> ItemResult:
|
) -> ItemResult:
|
||||||
size = c.state.size
|
size = c.state.size
|
||||||
|
arch_rel = self._archive_rel(c.relpath, rule.archive_layout)
|
||||||
|
need_archive = archive is not None and (
|
||||||
|
(c.relpath, size, c.state.mtime_ns) not in (archived or set())
|
||||||
|
)
|
||||||
failure = failures.get(c.relpath)
|
failure = failures.get(c.relpath)
|
||||||
|
|
||||||
# Quarantäne für wiederholt scheiternde Dateien
|
# Quarantäne für wiederholt scheiternde Dateien
|
||||||
@@ -349,9 +369,14 @@ class Engine:
|
|||||||
else None
|
else None
|
||||||
)
|
)
|
||||||
verb = "kopieren" if rule.action is RuleAction.COPY else "verschieben"
|
verb = "kopieren" if rule.action is RuleAction.COPY else "verschieben"
|
||||||
return ItemResult(
|
msg = msg or f"würde {verb}"
|
||||||
c.relpath, ItemStatus.PLANNED, dest.describe(planned), size, msg or f"würde {verb}"
|
if archive is not None:
|
||||||
)
|
if need_archive:
|
||||||
|
arch_planned = archive.plan(arch_rel, CollisionPolicy.RENAME)
|
||||||
|
msg += f"; Archiv: {archive.describe(arch_planned or arch_rel)}"
|
||||||
|
else:
|
||||||
|
msg += "; bereits archiviert"
|
||||||
|
return ItemResult(c.relpath, ItemStatus.PLANNED, dest.describe(planned), size, msg)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
free = dest.free_bytes()
|
free = dest.free_bytes()
|
||||||
@@ -367,10 +392,51 @@ class Engine:
|
|||||||
f"benötigt {_mb(int(size * SPACE_RESERVE))})",
|
f"benötigt {_mb(int(size * SPACE_RESERVE))})",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
if need_archive and archive is not None:
|
||||||
|
try:
|
||||||
|
afree = archive.free_bytes()
|
||||||
|
except OSError:
|
||||||
|
afree = None
|
||||||
|
if afree is not None and afree < size * SPACE_RESERVE:
|
||||||
|
return ItemResult(
|
||||||
|
c.relpath,
|
||||||
|
ItemStatus.FAILED,
|
||||||
|
archive.describe(arch_rel),
|
||||||
|
size,
|
||||||
|
f"zu wenig Platz im Archiv (frei {_mb(afree)}, "
|
||||||
|
f"benötigt {_mb(int(size * SPACE_RESERVE))})",
|
||||||
|
)
|
||||||
|
|
||||||
def verify() -> None:
|
def verify() -> None:
|
||||||
if src.state(c.relpath) != c.state:
|
if src.state(c.relpath) != c.state:
|
||||||
raise SourceChanged("Quelle hat sich während der Übertragung geändert")
|
raise SourceChanged("Quelle hat sich während der Übertragung geändert")
|
||||||
|
|
||||||
|
# Archiv ZUERST: scheitert es, ist noch nichts ans Ziel (z. B. Paperless)
|
||||||
|
# geliefert. Scheitert danach das Ziel, verhindert die Buchführung eine
|
||||||
|
# zweite Archivkopie beim nächsten Versuch.
|
||||||
|
archive_note: str | None = None
|
||||||
|
if need_archive and archive is not None:
|
||||||
|
try:
|
||||||
|
meta, _ = self._meta(rule, src, c.relpath, dest)
|
||||||
|
ares = archive.commit(
|
||||||
|
_SourceFile(src, c.relpath),
|
||||||
|
size,
|
||||||
|
arch_rel,
|
||||||
|
CollisionPolicy.RENAME,
|
||||||
|
MetaOptions(meta.atime_ns, meta.mtime_ns, meta.mode, None),
|
||||||
|
verify,
|
||||||
|
token,
|
||||||
|
)
|
||||||
|
except CollisionSkip: # kann bei RENAME nicht auftreten
|
||||||
|
return ItemResult(c.relpath, ItemStatus.SKIPPED, None, size, "Archiv belegt")
|
||||||
|
except SourceChanged as exc:
|
||||||
|
return ItemResult(c.relpath, ItemStatus.SKIPPED, None, size, str(exc))
|
||||||
|
except (TransferError, OSError) as exc:
|
||||||
|
return self._fail(rule, c, exc, dry_run, prefix="Archiv: ")
|
||||||
|
archive_path = archive.describe(ares.final_rel)
|
||||||
|
self.ledger.record_archive(rule.id, c.relpath, size, c.state.mtime_ns, archive_path)
|
||||||
|
archive_note = f"archiviert: {archive_path}"
|
||||||
|
|
||||||
try:
|
try:
|
||||||
meta, notes = self._meta(rule, src, c.relpath, dest)
|
meta, notes = self._meta(rule, src, c.relpath, dest)
|
||||||
res = None
|
res = None
|
||||||
@@ -401,6 +467,8 @@ class Engine:
|
|||||||
return self._fail(rule, c, exc, dry_run)
|
return self._fail(rule, c, exc, dry_run)
|
||||||
|
|
||||||
notes += res.notes
|
notes += res.notes
|
||||||
|
if archive_note:
|
||||||
|
notes.insert(0, archive_note)
|
||||||
target = dest.describe(res.final_rel)
|
target = dest.describe(res.final_rel)
|
||||||
if c.relpath in failures:
|
if c.relpath in failures:
|
||||||
self.ledger.clear_failure(rule.id, c.relpath)
|
self.ledger.clear_failure(rule.id, c.relpath)
|
||||||
@@ -425,10 +493,12 @@ class Engine:
|
|||||||
)
|
)
|
||||||
return ItemResult(c.relpath, ItemStatus.MOVED, target, size, "; ".join(notes) or None)
|
return ItemResult(c.relpath, ItemStatus.MOVED, target, size, "; ".join(notes) or None)
|
||||||
|
|
||||||
def _fail(self, rule: RuleSnapshot, c: Candidate, exc: Exception, dry_run: bool) -> ItemResult:
|
def _fail(
|
||||||
|
self, rule: RuleSnapshot, c: Candidate, exc: Exception, dry_run: bool, prefix: str = ""
|
||||||
|
) -> ItemResult:
|
||||||
if isinstance(exc, OSError) and exc.errno in MOUNT_ERRNOS:
|
if isinstance(exc, OSError) and exc.errno in MOUNT_ERRNOS:
|
||||||
raise RunAborted(f"Anbindung gestört: {_describe_os_error(exc)}") from exc
|
raise RunAborted(f"{prefix}Anbindung gestört: {_describe_os_error(exc)}") from exc
|
||||||
msg = _describe_os_error(exc) if isinstance(exc, OSError) else str(exc)
|
msg = prefix + (_describe_os_error(exc) if isinstance(exc, OSError) else str(exc))
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
attempts = self.ledger.record_failure(
|
attempts = self.ledger.record_failure(
|
||||||
rule.id, c.relpath, c.state.size, c.state.mtime_ns, msg
|
rule.id, c.relpath, c.state.size, c.state.mtime_ns, msg
|
||||||
@@ -437,6 +507,12 @@ class Engine:
|
|||||||
event(log, "file.failed", level=30, rule=rule.name, file=c.relpath, error=msg)
|
event(log, "file.failed", level=30, rule=rule.name, file=c.relpath, error=msg)
|
||||||
return ItemResult(c.relpath, ItemStatus.FAILED, None, c.state.size, msg)
|
return ItemResult(c.relpath, ItemStatus.FAILED, None, c.state.size, msg)
|
||||||
|
|
||||||
|
def _archive_rel(self, relpath: str, layout: str) -> str:
|
||||||
|
if layout == "date":
|
||||||
|
t = datetime.fromtimestamp(self._clock())
|
||||||
|
return f"{t:%Y}/{t:%m}/{relpath}"
|
||||||
|
return relpath
|
||||||
|
|
||||||
def _quarantine(
|
def _quarantine(
|
||||||
self, rule: RuleSnapshot, c: Candidate, src: Source, q_rel: str, attempts: int
|
self, rule: RuleSnapshot, c: Candidate, src: Source, q_rel: str, attempts: int
|
||||||
) -> ItemResult:
|
) -> ItemResult:
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ from sqlalchemy import delete, select
|
|||||||
from sqlalchemy.orm import Session, sessionmaker
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
from netmover.models import CopiedFile, FileFailure, utcnow
|
from netmover.models import ArchivedFile, CopiedFile, FileFailure, utcnow
|
||||||
|
|
||||||
CopyKey = tuple[str, int, int] # (relpath, size, mtime_ns)
|
CopyKey = tuple[str, int, int] # (relpath, size, mtime_ns)
|
||||||
|
|
||||||
@@ -72,6 +72,53 @@ class Ledger:
|
|||||||
s.execute(delete(CopiedFile).where(CopiedFile.id.in_(stale)))
|
s.execute(delete(CopiedFile).where(CopiedFile.id.in_(stale)))
|
||||||
return len(stale)
|
return len(stale)
|
||||||
|
|
||||||
|
# -- Archiv ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def archived_keys(self, rule_id: int) -> set[CopyKey]:
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
rows = s.execute(
|
||||||
|
select(ArchivedFile.src_relpath, ArchivedFile.size, ArchivedFile.mtime_ns).where(
|
||||||
|
ArchivedFile.rule_id == rule_id
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
return {(r[0], r[1], r[2]) for r in rows}
|
||||||
|
|
||||||
|
def record_archive(
|
||||||
|
self, rule_id: int, relpath: str, size: int, mtime_ns: int, archive_path: str
|
||||||
|
) -> None:
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
exists = s.scalar(
|
||||||
|
select(ArchivedFile.id).where(
|
||||||
|
ArchivedFile.rule_id == rule_id,
|
||||||
|
ArchivedFile.src_relpath == relpath,
|
||||||
|
ArchivedFile.size == size,
|
||||||
|
ArchivedFile.mtime_ns == mtime_ns,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if exists is None:
|
||||||
|
s.add(
|
||||||
|
ArchivedFile(
|
||||||
|
rule_id=rule_id,
|
||||||
|
src_relpath=relpath,
|
||||||
|
size=size,
|
||||||
|
mtime_ns=mtime_ns,
|
||||||
|
archive_path=archive_path,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def prune_archived(self, rule_id: int, present: set[str]) -> int:
|
||||||
|
"""Wie :meth:`prune_copied`: nur nach einem vollständigen Scan aufrufen."""
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
rows = s.execute(
|
||||||
|
select(ArchivedFile.id, ArchivedFile.src_relpath).where(
|
||||||
|
ArchivedFile.rule_id == rule_id
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
stale = [r[0] for r in rows if r[1] not in present]
|
||||||
|
if stale:
|
||||||
|
s.execute(delete(ArchivedFile).where(ArchivedFile.id.in_(stale)))
|
||||||
|
return len(stale)
|
||||||
|
|
||||||
def failures(self, rule_id: int) -> dict[str, FailureInfo]:
|
def failures(self, rule_id: int) -> dict[str, FailureInfo]:
|
||||||
with session_scope(self.session_factory) as s:
|
with session_scope(self.session_factory) as s:
|
||||||
rows = s.scalars(select(FileFailure).where(FileFailure.rule_id == rule_id)).all()
|
rows = s.scalars(select(FileFailure).where(FileFailure.rule_id == rule_id)).all()
|
||||||
|
|||||||
@@ -56,6 +56,11 @@ class RulePlan:
|
|||||||
quarantine_path: str | None
|
quarantine_path: str | None
|
||||||
op_timeout: int
|
op_timeout: int
|
||||||
dry_run: bool
|
dry_run: bool
|
||||||
|
archive_kind: str = "none"
|
||||||
|
archive_mount_id: int | None = None
|
||||||
|
archive_type: MountType | None = None
|
||||||
|
archive_name: str | None = None
|
||||||
|
archive_path: str = ""
|
||||||
|
|
||||||
|
|
||||||
def parse_owner(value: str | None) -> tuple[int, int] | None:
|
def parse_owner(value: str | None) -> tuple[int, int] | None:
|
||||||
@@ -92,6 +97,7 @@ def load_plan(session: Session, rule_id: int) -> RulePlan:
|
|||||||
set_mode=parse_mode(r.set_mode),
|
set_mode=parse_mode(r.set_mode),
|
||||||
quarantine_after=r.quarantine_after,
|
quarantine_after=r.quarantine_after,
|
||||||
dst_root_squash=dst.root_squash if dst is not None else None,
|
dst_root_squash=dst.root_squash if dst is not None else None,
|
||||||
|
archive_layout=r.archive_layout.value,
|
||||||
)
|
)
|
||||||
return RulePlan(
|
return RulePlan(
|
||||||
snapshot=snap,
|
snapshot=snap,
|
||||||
@@ -109,6 +115,11 @@ def load_plan(session: Session, rule_id: int) -> RulePlan:
|
|||||||
else None,
|
else None,
|
||||||
op_timeout=r.op_timeout_seconds,
|
op_timeout=r.op_timeout_seconds,
|
||||||
dry_run=r.dry_run,
|
dry_run=r.dry_run,
|
||||||
|
archive_kind=r.archive_kind.value,
|
||||||
|
archive_mount_id=r.archive_mount_id if r.archive_kind.value == "mount" else None,
|
||||||
|
archive_type=r.archive_mount.type if r.archive_mount is not None else None,
|
||||||
|
archive_name=r.archive_mount.name if r.archive_mount is not None else None,
|
||||||
|
archive_path=r.archive_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -176,24 +187,52 @@ class RuleRunner:
|
|||||||
if plan.dst_mount_id is not None
|
if plan.dst_mount_id is not None
|
||||||
else None
|
else None
|
||||||
)
|
)
|
||||||
|
arch_mp = (
|
||||||
|
self.mounts.check_alive(plan.archive_mount_id)
|
||||||
|
if plan.archive_mount_id is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
except MountUnavailable as exc:
|
except MountUnavailable as exc:
|
||||||
return RunResult(RunStatus.ERROR, dry, message=str(exc), unavailable=True)
|
return RunResult(RunStatus.ERROR, dry, message=str(exc), unavailable=True)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
make_source = self._source_factory(plan, src_mp)
|
make_source = self._source_factory(plan, src_mp)
|
||||||
src_root = src_mp / plan.src_subpath if src_mp and plan.src_subpath else src_mp
|
src_root = src_mp / plan.src_subpath if src_mp and plan.src_subpath else src_mp
|
||||||
make_dest = self._dest_factory(plan, dst_mp, src_root)
|
make_dest = self.destination_factory(
|
||||||
|
"dst_path",
|
||||||
|
plan.dst_mount_id,
|
||||||
|
plan.dst_type,
|
||||||
|
plan.dst_name,
|
||||||
|
plan.dst_path,
|
||||||
|
dst_mp,
|
||||||
|
src_root,
|
||||||
|
)
|
||||||
|
make_archive = (
|
||||||
|
self.destination_factory(
|
||||||
|
"archive_path",
|
||||||
|
plan.archive_mount_id,
|
||||||
|
plan.archive_type,
|
||||||
|
plan.archive_name,
|
||||||
|
plan.archive_path,
|
||||||
|
arch_mp,
|
||||||
|
src_root,
|
||||||
|
)
|
||||||
|
if plan.archive_kind != "none"
|
||||||
|
else None
|
||||||
|
)
|
||||||
except (ValidationError, TransferError, SftpError) as exc:
|
except (ValidationError, TransferError, SftpError) as exc:
|
||||||
return RunResult(RunStatus.ERROR, dry, message=str(exc))
|
return RunResult(RunStatus.ERROR, dry, message=str(exc))
|
||||||
|
|
||||||
def work(token: CancelToken) -> RunResult:
|
def work(token: CancelToken) -> RunResult:
|
||||||
|
opened: list[Source | Destination] = []
|
||||||
|
try:
|
||||||
source = make_source()
|
source = make_source()
|
||||||
try:
|
opened.append(source)
|
||||||
dest = make_dest()
|
dest = make_dest()
|
||||||
except BaseException:
|
opened.append(dest)
|
||||||
source.close()
|
archive = make_archive() if make_archive is not None else None
|
||||||
raise
|
if archive is not None:
|
||||||
try:
|
opened.append(archive)
|
||||||
return self.engine.execute(
|
return self.engine.execute(
|
||||||
plan.snapshot,
|
plan.snapshot,
|
||||||
source,
|
source,
|
||||||
@@ -201,10 +240,11 @@ class RuleRunner:
|
|||||||
token,
|
token,
|
||||||
dry_run=dry,
|
dry_run=dry,
|
||||||
quarantine_rel=plan.quarantine_path,
|
quarantine_rel=plan.quarantine_path,
|
||||||
|
archive=archive,
|
||||||
)
|
)
|
||||||
finally:
|
finally:
|
||||||
dest.close()
|
for item in reversed(opened):
|
||||||
source.close()
|
item.close()
|
||||||
|
|
||||||
def finish() -> None:
|
def finish() -> None:
|
||||||
self._started.pop(rule_id, None)
|
self._started.pop(rule_id, None)
|
||||||
@@ -251,41 +291,50 @@ class RuleRunner:
|
|||||||
|
|
||||||
return make_local
|
return make_local
|
||||||
|
|
||||||
def _dest_factory(
|
def destination_factory(
|
||||||
self, plan: RulePlan, dst_mp: Path | None, src_root: Path | None
|
self,
|
||||||
|
field: str,
|
||||||
|
mount_id: int | None,
|
||||||
|
mount_type: MountType | None,
|
||||||
|
mount_name: str | None,
|
||||||
|
path: str,
|
||||||
|
mp: Path | None,
|
||||||
|
src_root: Path | None,
|
||||||
) -> Callable[[], Destination]:
|
) -> Callable[[], Destination]:
|
||||||
if plan.dst_mount_id is None:
|
"""Ziel oder Archiv: lokaler Pfad (``mount_id`` None), SFTP oder Kernel-Mount."""
|
||||||
root = Path(safe_abspath("dst_path", plan.dst_path))
|
what = {"archive_path": "Archiv", "backup_path": "Sicherungsziel"}.get(field, "Ziel")
|
||||||
|
if mount_id is None:
|
||||||
|
root = Path(safe_abspath(field, path))
|
||||||
if self._under_mount_base(root):
|
if self._under_mount_base(root):
|
||||||
base = self.mounts.mount_base
|
base = self.mounts.mount_base
|
||||||
raise ValidationError(
|
raise ValidationError(
|
||||||
"dst_path",
|
field,
|
||||||
f"Lokales Ziel liegt unter {base} – bitte die Anbindung als Ziel wählen, "
|
f"Lokales {what} liegt unter {base} – bitte die Anbindung als {what} wählen, "
|
||||||
"damit geprüft wird, ob sie eingehängt ist.",
|
"damit geprüft wird, ob sie eingehängt ist.",
|
||||||
)
|
)
|
||||||
kind = "local"
|
kind = "local"
|
||||||
local_root = root
|
local_root = root
|
||||||
create = False
|
create = False
|
||||||
elif plan.dst_type is MountType.SFTP:
|
elif mount_type is MountType.SFTP:
|
||||||
scfg = self.mounts.sftp_config_for(plan.dst_mount_id)
|
scfg = self.mounts.sftp_config_for(mount_id)
|
||||||
sub = safe_relpath("dst_path", plan.dst_path)
|
sub = safe_relpath(field, path)
|
||||||
remote = posixpath.join(scfg.base_path, sub) if sub else scfg.base_path
|
remote = posixpath.join(scfg.base_path, sub) if sub else scfg.base_path
|
||||||
label = plan.dst_name or "sftp"
|
label = mount_name or "sftp"
|
||||||
|
|
||||||
def make_sftp() -> Destination:
|
def make_sftp() -> Destination:
|
||||||
return SftpDestination(connect(scfg), remote, label=label)
|
return SftpDestination(connect(scfg), remote, label=label)
|
||||||
|
|
||||||
return make_sftp
|
return make_sftp
|
||||||
else:
|
else:
|
||||||
if dst_mp is None:
|
if mp is None:
|
||||||
raise TransferError("Ziel-Anbindung ist nicht eingehängt")
|
raise TransferError(f"{what}-Anbindung ist nicht eingehängt")
|
||||||
sub = safe_relpath("dst_path", plan.dst_path)
|
sub = safe_relpath(field, path)
|
||||||
local_root = dst_mp / sub if sub else dst_mp
|
local_root = mp / sub if sub else mp
|
||||||
kind = "cifs" if plan.dst_type is MountType.CIFS else "nfs"
|
kind = "cifs" if mount_type is MountType.CIFS else "nfs"
|
||||||
create = True
|
create = True
|
||||||
|
|
||||||
if src_root is not None and local_root.resolve() == src_root.resolve():
|
if src_root is not None and local_root.resolve() == src_root.resolve():
|
||||||
raise ValidationError("dst_path", "Quelle und Ziel sind derselbe Ordner")
|
raise ValidationError(field, f"Quelle und {what} sind derselbe Ordner")
|
||||||
|
|
||||||
def make_local() -> Destination:
|
def make_local() -> Destination:
|
||||||
return LocalDestination(local_root, kind, create_root=create) # type: ignore[arg-type]
|
return LocalDestination(local_root, kind, create_root=create) # type: ignore[arg-type]
|
||||||
|
|||||||
@@ -6,7 +6,8 @@
|
|||||||
* Die Läufe selbst sind blockierend und laufen in einem eigenen Thread-Pool,
|
* Die Läufe selbst sind blockierend und laufen in einem eigenen Thread-Pool,
|
||||||
nie im Event-Loop. Überlappung regelt der Lock im :class:`RuleRunner`, der den
|
nie im Event-Loop. Überlappung regelt der Lock im :class:`RuleRunner`, der den
|
||||||
übersprungenen Termin auch protokolliert – deshalb ``max_instances`` > 1.
|
übersprungenen Termin auch protokolliert – deshalb ``max_instances`` > 1.
|
||||||
* Herzschlag-Job für ``/healthz`` und stündliches Aufräumen der Protokolle.
|
* Herzschlag-Job für ``/healthz``, stündliches Aufräumen der Protokolle und alle
|
||||||
|
5 Minuten die Frage, ob die tägliche Datensicherung fällig ist.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -34,6 +35,7 @@ HEARTBEAT_SECONDS = 30
|
|||||||
CLEANUP_SECONDS = 3600
|
CLEANUP_SECONDS = 3600
|
||||||
RETRY_SECONDS = 60
|
RETRY_SECONDS = 60
|
||||||
PUSH_SECONDS = 60
|
PUSH_SECONDS = 60
|
||||||
|
BACKUP_CHECK_SECONDS = 300
|
||||||
|
|
||||||
|
|
||||||
def _job_id(rule_id: int) -> str:
|
def _job_id(rule_id: int) -> str:
|
||||||
@@ -48,6 +50,7 @@ class RuleScheduler:
|
|||||||
*,
|
*,
|
||||||
retry_mounts: Callable[[], object] | None = None,
|
retry_mounts: Callable[[], object] | None = None,
|
||||||
push_health: Callable[[], object] | None = None,
|
push_health: Callable[[], object] | None = None,
|
||||||
|
backup_tick: Callable[[], object] | None = None,
|
||||||
rng: random.Random | None = None,
|
rng: random.Random | None = None,
|
||||||
workers: int = 8,
|
workers: int = 8,
|
||||||
) -> None:
|
) -> None:
|
||||||
@@ -55,6 +58,7 @@ class RuleScheduler:
|
|||||||
self.service = service
|
self.service = service
|
||||||
self._retry_mounts = retry_mounts
|
self._retry_mounts = retry_mounts
|
||||||
self._push_health = push_health
|
self._push_health = push_health
|
||||||
|
self._backup_tick = backup_tick
|
||||||
self._rng = rng or random.Random() # noqa: S311 – nur Startversatz
|
self._rng = rng or random.Random() # noqa: S311 – nur Startversatz
|
||||||
self._executor = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="nm-run")
|
self._executor = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="nm-run")
|
||||||
self.scheduler = AsyncIOScheduler(
|
self.scheduler = AsyncIOScheduler(
|
||||||
@@ -85,6 +89,14 @@ class RuleScheduler:
|
|||||||
self.scheduler.add_job(
|
self.scheduler.add_job(
|
||||||
self._push, IntervalTrigger(seconds=PUSH_SECONDS), id="kuma-push"
|
self._push, IntervalTrigger(seconds=PUSH_SECONDS), id="kuma-push"
|
||||||
)
|
)
|
||||||
|
if self._backup_tick is not None:
|
||||||
|
self.scheduler.add_job(
|
||||||
|
self._backup,
|
||||||
|
IntervalTrigger(
|
||||||
|
seconds=BACKUP_CHECK_SECONDS, start_date=utcnow() + timedelta(minutes=2)
|
||||||
|
),
|
||||||
|
id="backup",
|
||||||
|
)
|
||||||
self.sync()
|
self.sync()
|
||||||
event(log, "scheduler.started", rules=len(self._intervals))
|
event(log, "scheduler.started", rules=len(self._intervals))
|
||||||
|
|
||||||
@@ -166,6 +178,11 @@ class RuleScheduler:
|
|||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
await loop.run_in_executor(self._executor, self._push_health)
|
await loop.run_in_executor(self._executor, self._push_health)
|
||||||
|
|
||||||
|
async def _backup(self) -> None:
|
||||||
|
if self._backup_tick is not None:
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
await loop.run_in_executor(self._executor, self._backup_tick)
|
||||||
|
|
||||||
async def _cleanup(self) -> None:
|
async def _cleanup(self) -> None:
|
||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
await loop.run_in_executor(self._executor, cleanup, self.session_factory)
|
await loop.run_in_executor(self._executor, cleanup, self.session_factory)
|
||||||
|
|||||||
@@ -46,9 +46,19 @@ class AppSettings:
|
|||||||
smtp_password_enc: str = ""
|
smtp_password_enc: str = ""
|
||||||
smtp_from: str = ""
|
smtp_from: str = ""
|
||||||
smtp_to: str = ""
|
smtp_to: str = ""
|
||||||
# Anmeldung: "always" | "trusted" (ohne Anmeldung aus trusted_networks) | "off"
|
# Anmeldung: Netze ohne Passwort (trusted_networks), Netze mit Passwort
|
||||||
|
# (allowed_networks, leer = alle übrigen); alle anderen Netze werden abgewiesen.
|
||||||
|
# auth_mode "off" = ohne Anmeldung von überall (aus 0.5.0, weiterhin gültig).
|
||||||
auth_mode: str = "always"
|
auth_mode: str = "always"
|
||||||
trusted_networks: str = ""
|
trusted_networks: str = ""
|
||||||
|
allowed_networks: str = ""
|
||||||
|
# Datensicherung (backup.py); Passphrase Fernet-verschlüsselt
|
||||||
|
backup_enabled: bool = False
|
||||||
|
backup_mount_id: int = 0 # 0 = kein Ziel
|
||||||
|
backup_path: str = ""
|
||||||
|
backup_time: str = "03:30"
|
||||||
|
backup_keep: int = 14
|
||||||
|
backup_passphrase_enc: str = ""
|
||||||
|
|
||||||
|
|
||||||
_PREFIX = "app."
|
_PREFIX = "app."
|
||||||
|
|||||||
@@ -16,6 +16,8 @@ from sqlalchemy.orm import Session
|
|||||||
|
|
||||||
from netmover import __version__
|
from netmover import __version__
|
||||||
from netmover.models import (
|
from netmover.models import (
|
||||||
|
ArchiveKind,
|
||||||
|
ArchiveLayout,
|
||||||
CollisionPolicy,
|
CollisionPolicy,
|
||||||
Mount,
|
Mount,
|
||||||
MountState,
|
MountState,
|
||||||
@@ -72,6 +74,9 @@ RULE_FIELDS = (
|
|||||||
"dry_run",
|
"dry_run",
|
||||||
"quarantine_path",
|
"quarantine_path",
|
||||||
"quarantine_after",
|
"quarantine_after",
|
||||||
|
"archive_kind",
|
||||||
|
"archive_path",
|
||||||
|
"archive_layout",
|
||||||
)
|
)
|
||||||
SETTING_FIELDS = (
|
SETTING_FIELDS = (
|
||||||
"retention_days",
|
"retention_days",
|
||||||
@@ -99,6 +104,7 @@ def export_config(s: Session, settings: AppSettings) -> dict[str, Any]:
|
|||||||
**{f: _plain(getattr(r, f)) for f in RULE_FIELDS},
|
**{f: _plain(getattr(r, f)) for f in RULE_FIELDS},
|
||||||
"src_mount": names[r.src_mount_id],
|
"src_mount": names[r.src_mount_id],
|
||||||
"dst_mount": names.get(r.dst_mount_id) if r.dst_mount_id else None,
|
"dst_mount": names.get(r.dst_mount_id) if r.dst_mount_id else None,
|
||||||
|
"archive_mount": names.get(r.archive_mount_id) if r.archive_mount_id else None,
|
||||||
}
|
}
|
||||||
for r in s.scalars(select(Rule).order_by(Rule.name))
|
for r in s.scalars(select(Rule).order_by(Rule.name))
|
||||||
],
|
],
|
||||||
@@ -165,13 +171,16 @@ def import_config(s: Session, mounts: MountManager, data: dict[str, Any]) -> Imp
|
|||||||
try:
|
try:
|
||||||
src = ids[raw["src_mount"]]
|
src = ids[raw["src_mount"]]
|
||||||
dst = ids[raw["dst_mount"]] if raw.get("dst_mount") else None
|
dst = ids[raw["dst_mount"]] if raw.get("dst_mount") else None
|
||||||
r = Rule(src_mount_id=src, dst_mount_id=dst)
|
arch = ids[raw["archive_mount"]] if raw.get("archive_mount") else None
|
||||||
|
r = Rule(src_mount_id=src, dst_mount_id=dst, archive_mount_id=arch)
|
||||||
for f in RULE_FIELDS:
|
for f in RULE_FIELDS:
|
||||||
if f in raw:
|
if f in raw:
|
||||||
setattr(r, f, raw[f])
|
setattr(r, f, raw[f])
|
||||||
r.pattern_type = PatternType(r.pattern_type)
|
r.pattern_type = PatternType(r.pattern_type)
|
||||||
r.action = RuleAction(r.action)
|
r.action = RuleAction(r.action)
|
||||||
r.on_collision = CollisionPolicy(r.on_collision)
|
r.on_collision = CollisionPolicy(r.on_collision)
|
||||||
|
r.archive_kind = ArchiveKind(r.archive_kind or "none")
|
||||||
|
r.archive_layout = ArchiveLayout(r.archive_layout or "flat")
|
||||||
r.src_subpath = safe_relpath("src_subpath", r.src_subpath or "")
|
r.src_subpath = safe_relpath("src_subpath", r.src_subpath or "")
|
||||||
if int(r.interval_seconds) < 10:
|
if int(r.interval_seconds) < 10:
|
||||||
raise ValueError("Intervall < 10 s")
|
raise ValueError("Intervall < 10 s")
|
||||||
|
|||||||
+273
-41
@@ -38,16 +38,21 @@ from netmover.admin import (
|
|||||||
AdminError,
|
AdminError,
|
||||||
MountInput,
|
MountInput,
|
||||||
RuleInput,
|
RuleInput,
|
||||||
|
copy_name,
|
||||||
credential_for_test,
|
credential_for_test,
|
||||||
delete_mount,
|
delete_mount,
|
||||||
mount_input_from_model,
|
mount_input_from_model,
|
||||||
parse_mount_form,
|
parse_mount_form,
|
||||||
parse_rule_form,
|
parse_rule_form,
|
||||||
|
rule_copy_name,
|
||||||
rule_input_from_model,
|
rule_input_from_model,
|
||||||
save_mount,
|
save_mount,
|
||||||
save_rule,
|
save_rule,
|
||||||
)
|
)
|
||||||
from netmover.app_state import NetMover
|
from netmover.app_state import NetMover
|
||||||
|
from netmover.backup import load_state as load_backup_state
|
||||||
|
from netmover.backup import parse_time as parse_backup_time
|
||||||
|
from netmover.backup import passphrase_problem
|
||||||
from netmover.config import Settings, get_settings
|
from netmover.config import Settings, get_settings
|
||||||
from netmover.credentials import (
|
from netmover.credentials import (
|
||||||
create_password,
|
create_password,
|
||||||
@@ -60,6 +65,7 @@ from netmover.credentials import (
|
|||||||
update_password,
|
update_password,
|
||||||
)
|
)
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
|
from netmover.health import metrics as health_metrics
|
||||||
from netmover.kuma import push as kuma_push
|
from netmover.kuma import push as kuma_push
|
||||||
from netmover.logging_setup import event, get_logger
|
from netmover.logging_setup import event, get_logger
|
||||||
from netmover.mail import SECURITY_MODES, MailConfigError, parse_recipients, valid_address
|
from netmover.mail import SECURITY_MODES, MailConfigError, parse_recipients, valid_address
|
||||||
@@ -83,6 +89,7 @@ from netmover.mounting.report import CheckReport
|
|||||||
from netmover.notify import Notification
|
from netmover.notify import Notification
|
||||||
from netmover.settings_store import AppSettings, NotifyMode, load_settings, save_settings
|
from netmover.settings_store import AppSettings, NotifyMode, load_settings, save_settings
|
||||||
from netmover.transfer import apply_imported_settings, export_config, import_config
|
from netmover.transfer import apply_imported_settings, export_config, import_config
|
||||||
|
from netmover.validation import ValidationError, safe_relpath
|
||||||
from netmover.web import auth
|
from netmover.web import auth
|
||||||
|
|
||||||
log = get_logger(__name__)
|
log = get_logger(__name__)
|
||||||
@@ -184,6 +191,7 @@ HELP_ANCHORS = {
|
|||||||
"rules": "regeln",
|
"rules": "regeln",
|
||||||
"log": "protokoll",
|
"log": "protokoll",
|
||||||
"settings": "einstellungen",
|
"settings": "einstellungen",
|
||||||
|
"backup": "sicherung",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -197,6 +205,7 @@ class NavState:
|
|||||||
rules: int
|
rules: int
|
||||||
rules_active: int
|
rules_active: int
|
||||||
maintenance: bool = False
|
maintenance: bool = False
|
||||||
|
backup_ok: bool = True
|
||||||
|
|
||||||
|
|
||||||
def _nav_state(request: Request) -> NavState | None:
|
def _nav_state(request: Request) -> NavState | None:
|
||||||
@@ -221,6 +230,7 @@ def _nav_state(request: Request) -> NavState | None:
|
|||||||
rules=len(rules),
|
rules=len(rules),
|
||||||
rules_active=sum(1 for (en,) in rules if en),
|
rules_active=sum(1 for (en,) in rules if en),
|
||||||
maintenance=health.maintenance,
|
maintenance=health.maintenance,
|
||||||
|
backup_ok=(c := health.find("backup")) is None or c.ok,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -269,7 +279,7 @@ async def guard(request: Request) -> None:
|
|||||||
if not auth.password_is_set(state.session_factory):
|
if not auth.password_is_set(state.session_factory):
|
||||||
raise auth.LoginRequired(setup=True)
|
raise auth.LoginRequired(setup=True)
|
||||||
cfg = load_settings(state.session_factory)
|
cfg = load_settings(state.session_factory)
|
||||||
bypass = auth.login_bypass(request, cfg.auth_mode, cfg.trusted_networks)
|
bypass = auth.login_bypass(request, cfg.auth_mode, cfg.trusted_networks, cfg.allowed_networks)
|
||||||
if bypass:
|
if bypass:
|
||||||
request.state.auth_bypass = bypass
|
request.state.auth_bypass = bypass
|
||||||
elif not auth.is_logged_in(request):
|
elif not auth.is_logged_in(request):
|
||||||
@@ -304,16 +314,20 @@ async def healthz(request: Request) -> JSONResponse:
|
|||||||
"last_heartbeat": beat.isoformat() if beat else None,
|
"last_heartbeat": beat.isoformat() if beat else None,
|
||||||
},
|
},
|
||||||
"checks": [c.as_dict() for c in health.checks],
|
"checks": [c.as_dict() for c in health.checks],
|
||||||
|
"metrics": await run_in_threadpool(health_metrics, state.session_factory),
|
||||||
}
|
}
|
||||||
return JSONResponse(body, status_code=200 if health.ok else 503)
|
return JSONResponse(body, status_code=200 if health.ok else 503)
|
||||||
|
|
||||||
|
|
||||||
async def _single_check(request: Request, kind: str, **key: Any) -> JSONResponse:
|
async def _single_check(
|
||||||
|
request: Request, kind: str, extra: dict[str, Any] | None = None, **key: Any
|
||||||
|
) -> JSONResponse:
|
||||||
health = await run_in_threadpool(nm(request).health)
|
health = await run_in_threadpool(nm(request).health)
|
||||||
check = health.find(kind, **key)
|
check = health.find(kind, **key)
|
||||||
if check is None:
|
if check is None:
|
||||||
return JSONResponse({"status": "unknown", "message": "nicht gefunden"}, status_code=404)
|
return JSONResponse({"status": "unknown", "message": "nicht gefunden"}, status_code=404)
|
||||||
body = {"status": "ok" if check.ok else "error", "name": check.name, "message": check.message}
|
body = {"status": "ok" if check.ok else "error", "name": check.name, "message": check.message}
|
||||||
|
body.update(extra or {})
|
||||||
return JSONResponse(body, status_code=200 if check.ok else 503)
|
return JSONResponse(body, status_code=200 if check.ok else 503)
|
||||||
|
|
||||||
|
|
||||||
@@ -325,8 +339,24 @@ async def healthz_mount(request: Request, slug: str) -> JSONResponse:
|
|||||||
|
|
||||||
@public.get("/healthz/rule/{rule_id}")
|
@public.get("/healthz/rule/{rule_id}")
|
||||||
async def healthz_rule(request: Request, rule_id: int) -> JSONResponse:
|
async def healthz_rule(request: Request, rule_id: int) -> JSONResponse:
|
||||||
"""Eigener Kuma-Monitor pro Regel."""
|
"""Eigener Kuma-Monitor pro Regel – mit Kennzahlen der letzten 24 Stunden."""
|
||||||
return await _single_check(request, "rule", id=rule_id)
|
all_metrics = await run_in_threadpool(health_metrics, nm(request).session_factory)
|
||||||
|
rules = all_metrics["rules"]
|
||||||
|
rule_metrics = rules.get(str(rule_id)) if isinstance(rules, dict) else None
|
||||||
|
return await _single_check(request, "rule", {"metrics": rule_metrics}, id=rule_id)
|
||||||
|
|
||||||
|
|
||||||
|
@public.get("/healthz/backup")
|
||||||
|
async def healthz_backup(request: Request) -> JSONResponse:
|
||||||
|
"""Eigener Kuma-Monitor für die Datensicherung."""
|
||||||
|
st = await run_in_threadpool(load_backup_state, nm(request).session_factory)
|
||||||
|
extra = {
|
||||||
|
"last_success": st.last_success.isoformat() if st.last_success else None,
|
||||||
|
"last_file": st.last_file or None,
|
||||||
|
"last_size": st.last_size,
|
||||||
|
"failures": st.failures,
|
||||||
|
}
|
||||||
|
return await _single_check(request, "backup", extra)
|
||||||
|
|
||||||
|
|
||||||
@public.get("/setup", response_class=HTMLResponse)
|
@public.get("/setup", response_class=HTMLResponse)
|
||||||
@@ -364,7 +394,7 @@ async def login_form(request: Request) -> Response:
|
|||||||
if not auth.password_is_set(state.session_factory):
|
if not auth.password_is_set(state.session_factory):
|
||||||
return redirect("/setup")
|
return redirect("/setup")
|
||||||
cfg = load_settings(state.session_factory)
|
cfg = load_settings(state.session_factory)
|
||||||
if auth.login_bypass(request, cfg.auth_mode, cfg.trusted_networks):
|
if auth.login_bypass(request, cfg.auth_mode, cfg.trusted_networks, cfg.allowed_networks):
|
||||||
return redirect("/")
|
return redirect("/")
|
||||||
return render(request, "login.html")
|
return render(request, "login.html")
|
||||||
|
|
||||||
@@ -422,13 +452,24 @@ class RuleView:
|
|||||||
target: str
|
target: str
|
||||||
|
|
||||||
|
|
||||||
|
def _archive_target(r: Rule) -> str | None:
|
||||||
|
if r.archive_kind.value == "none":
|
||||||
|
return None
|
||||||
|
if r.archive_kind.value == "local" or r.archive_mount is None:
|
||||||
|
place = r.archive_path
|
||||||
|
else:
|
||||||
|
place = f"{r.archive_mount.name}:/{r.archive_path}"
|
||||||
|
return place + (" (nach Datum)" if r.archive_layout.value == "date" else "")
|
||||||
|
|
||||||
|
|
||||||
def _rule_target(r: Rule) -> str:
|
def _rule_target(r: Rule) -> str:
|
||||||
src = f"{r.src_mount.name}:/{r.src_subpath}" if r.src_subpath else f"{r.src_mount.name}:/"
|
src = f"{r.src_mount.name}:/{r.src_subpath}" if r.src_subpath else f"{r.src_mount.name}:/"
|
||||||
if r.dst_mount is None:
|
if r.dst_mount is None:
|
||||||
dst = r.dst_path
|
dst = r.dst_path
|
||||||
else:
|
else:
|
||||||
dst = f"{r.dst_mount.name}:/{r.dst_path}" if r.dst_path else f"{r.dst_mount.name}:/"
|
dst = f"{r.dst_mount.name}:/{r.dst_path}" if r.dst_path else f"{r.dst_mount.name}:/"
|
||||||
return f"{src} → {dst}"
|
archive = _archive_target(r)
|
||||||
|
return f"{src} → {dst}" + (f" + Archiv {archive}" if archive else "")
|
||||||
|
|
||||||
|
|
||||||
async def _status_context(request: Request) -> dict[str, Any]:
|
async def _status_context(request: Request) -> dict[str, Any]:
|
||||||
@@ -580,10 +621,12 @@ def _mount_form(
|
|||||||
errors: dict[str, str],
|
errors: dict[str, str],
|
||||||
mount: Mount | None = None,
|
mount: Mount | None = None,
|
||||||
status_code: int = 200,
|
status_code: int = 200,
|
||||||
|
copy_from: Mount | None = None,
|
||||||
) -> Response:
|
) -> Response:
|
||||||
|
owner = mount or copy_from
|
||||||
pub = (
|
pub = (
|
||||||
mount.credential.public_key
|
owner.credential.public_key
|
||||||
if mount is not None and mount.credential and not mount.credential.managed
|
if owner is not None and owner.credential and not owner.credential.managed
|
||||||
else None
|
else None
|
||||||
)
|
)
|
||||||
return render(
|
return render(
|
||||||
@@ -596,8 +639,9 @@ def _mount_form(
|
|||||||
mount=mount,
|
mount=mount,
|
||||||
public_key=pub,
|
public_key=pub,
|
||||||
has_secret=bool(
|
has_secret=bool(
|
||||||
mount is not None and mount.credential is not None and not mount.credential.managed
|
owner is not None and owner.credential is not None and not owner.credential.managed
|
||||||
),
|
),
|
||||||
|
copy_from=copy_from,
|
||||||
key_options=_key_options(request),
|
key_options=_key_options(request),
|
||||||
password_options=_password_options(request),
|
password_options=_password_options(request),
|
||||||
)
|
)
|
||||||
@@ -617,16 +661,33 @@ async def mount_edit(request: Request, mount_id: int) -> Response:
|
|||||||
return _mount_form(request, mount_input_from_model(m), {}, m)
|
return _mount_form(request, mount_input_from_model(m), {}, m)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/mounts/{source_id}/copy", response_class=HTMLResponse)
|
||||||
|
async def mount_copy(request: Request, source_id: int) -> Response:
|
||||||
|
"""Formular für eine neue Anbindung, vorausgefüllt aus einer bestehenden."""
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
src = s.get(Mount, source_id)
|
||||||
|
if src is None:
|
||||||
|
return redirect("/mounts")
|
||||||
|
data = mount_input_from_model(src)
|
||||||
|
data.name = copy_name(s, src.name)
|
||||||
|
return _mount_form(request, data, {}, copy_from=src)
|
||||||
|
|
||||||
|
|
||||||
@protected.post("/mounts/new", response_class=HTMLResponse)
|
@protected.post("/mounts/new", response_class=HTMLResponse)
|
||||||
@protected.post("/mounts/{mount_id}/edit", response_class=HTMLResponse)
|
@protected.post("/mounts/{mount_id}/edit", response_class=HTMLResponse)
|
||||||
async def mount_save(request: Request, mount_id: int | None = None) -> Response:
|
@protected.post("/mounts/{source_id}/copy", response_class=HTMLResponse)
|
||||||
|
async def mount_save(
|
||||||
|
request: Request, mount_id: int | None = None, source_id: int | None = None
|
||||||
|
) -> Response:
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
data, errors = parse_mount_form(_form_dict(await request.form()))
|
data, errors = parse_mount_form(_form_dict(await request.form()))
|
||||||
with session_scope(state.session_factory) as s:
|
with session_scope(state.session_factory) as s:
|
||||||
existing = s.get(Mount, mount_id) if mount_id is not None else None
|
copy_from = s.get(Mount, source_id) if source_id is not None else None
|
||||||
if not errors:
|
if not errors:
|
||||||
try:
|
try:
|
||||||
m, errors = save_mount(s, state.secrets, state.mounts, data, mount_id)
|
m, errors = save_mount(
|
||||||
|
s, state.secrets, state.mounts, data, mount_id, copy_from=copy_from
|
||||||
|
)
|
||||||
except AdminError as exc:
|
except AdminError as exc:
|
||||||
errors = {"_": str(exc)}
|
errors = {"_": str(exc)}
|
||||||
m = None
|
m = None
|
||||||
@@ -635,7 +696,10 @@ async def mount_save(request: Request, mount_id: int | None = None) -> Response:
|
|||||||
if errors or m is None:
|
if errors or m is None:
|
||||||
s.rollback()
|
s.rollback()
|
||||||
existing = s.get(Mount, mount_id) if mount_id is not None else None
|
existing = s.get(Mount, mount_id) if mount_id is not None else None
|
||||||
return _mount_form(request, data, errors, existing, status_code=400)
|
copy_from = s.get(Mount, source_id) if source_id is not None else None
|
||||||
|
return _mount_form(
|
||||||
|
request, data, errors, existing, status_code=400, copy_from=copy_from
|
||||||
|
)
|
||||||
saved_id, name = m.id, m.name
|
saved_id, name = m.id, m.name
|
||||||
is_sftp_key = m.type is MountType.SFTP and data.auth is CredentialKind.SSH_KEY
|
is_sftp_key = m.type is MountType.SFTP and data.auth is CredentialKind.SSH_KEY
|
||||||
flash(request, f"Anbindung „{name}“ gespeichert.")
|
flash(request, f"Anbindung „{name}“ gespeichert.")
|
||||||
@@ -657,12 +721,17 @@ async def mount_test_form(request: Request) -> Response:
|
|||||||
data, errors = parse_mount_form(form)
|
data, errors = parse_mount_form(form)
|
||||||
if errors:
|
if errors:
|
||||||
return render(request, "partials/check_result.html", errors=errors, report=None)
|
return render(request, "partials/check_result.html", errors=errors, report=None)
|
||||||
existing_id = int(form["mount_id"]) if form.get("mount_id", "").isdigit() else None
|
ref = form.get("mount_id") or form.get("copy_from") or ""
|
||||||
|
existing_id = int(ref) if ref.isdigit() else None
|
||||||
with state.session_factory() as s:
|
with state.session_factory() as s:
|
||||||
existing = s.get(Mount, existing_id) if existing_id else None
|
existing = s.get(Mount, existing_id) if existing_id else None
|
||||||
cred = credential_for_test(data, existing, state.secrets, s)
|
cred = credential_for_test(data, existing, state.secrets, s)
|
||||||
cfg = data.config()
|
cfg = data.config()
|
||||||
if data.type is MountType.SFTP and existing is not None:
|
if (
|
||||||
|
data.type is MountType.SFTP
|
||||||
|
and existing is not None
|
||||||
|
and (existing.server, existing.port) == (data.server, data.port)
|
||||||
|
):
|
||||||
cfg = dataclasses.replace(cfg, host_key=existing.host_key)
|
cfg = dataclasses.replace(cfg, host_key=existing.host_key)
|
||||||
report = await run_in_threadpool(state.mounts.check, cfg, cred, None)
|
report = await run_in_threadpool(state.mounts.check, cfg, cred, None)
|
||||||
return render(request, "partials/check_result.html", report=report, errors={})
|
return render(request, "partials/check_result.html", report=report, errors={})
|
||||||
@@ -862,6 +931,7 @@ def _rule_form(
|
|||||||
errors: dict[str, str],
|
errors: dict[str, str],
|
||||||
rule_id: int | None,
|
rule_id: int | None,
|
||||||
status_code: int = 200,
|
status_code: int = 200,
|
||||||
|
copy_of: str | None = None,
|
||||||
) -> Response:
|
) -> Response:
|
||||||
with nm(request).session_factory() as s:
|
with nm(request).session_factory() as s:
|
||||||
mounts = s.scalars(select(Mount).order_by(Mount.name)).all()
|
mounts = s.scalars(select(Mount).order_by(Mount.name)).all()
|
||||||
@@ -875,6 +945,7 @@ def _rule_form(
|
|||||||
d=d,
|
d=d,
|
||||||
errors=errors,
|
errors=errors,
|
||||||
rule_id=rule_id,
|
rule_id=rule_id,
|
||||||
|
copy_of=copy_of,
|
||||||
src_mounts=mounts,
|
src_mounts=mounts,
|
||||||
src_options=[("", "– wählen –")] + [(m.id, f"{m.name} ({m.type.value})") for m in mounts],
|
src_options=[("", "– wählen –")] + [(m.id, f"{m.name} ({m.type.value})") for m in mounts],
|
||||||
dst_options=[("", "– wählen –")] + [(m.id, f"{m.name} ({m.type.value})") for m in mounts],
|
dst_options=[("", "– wählen –")] + [(m.id, f"{m.name} ({m.type.value})") for m in mounts],
|
||||||
@@ -902,25 +973,47 @@ async def rule_edit(request: Request, rule_id: int) -> Response:
|
|||||||
return _rule_form(request, d, {}, rule_id)
|
return _rule_form(request, d, {}, rule_id)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/rules/{source_id}/copy", response_class=HTMLResponse)
|
||||||
|
async def rule_copy(request: Request, source_id: int) -> Response:
|
||||||
|
"""Neue Regel, vorausgefüllt aus einer bestehenden – zunächst deaktiviert, damit sie
|
||||||
|
nicht sofort mit dem Original um dieselben Dateien konkurriert."""
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
src = s.get(Rule, source_id)
|
||||||
|
if src is None:
|
||||||
|
return redirect("/rules")
|
||||||
|
d = rule_input_from_model(src)
|
||||||
|
d.name = rule_copy_name(s, src.name)
|
||||||
|
d.enabled = False
|
||||||
|
source_name = src.name
|
||||||
|
return _rule_form(request, d, {}, None, copy_of=source_name)
|
||||||
|
|
||||||
|
|
||||||
@protected.post("/rules/new", response_class=HTMLResponse)
|
@protected.post("/rules/new", response_class=HTMLResponse)
|
||||||
@protected.post("/rules/{rule_id}/edit", response_class=HTMLResponse)
|
@protected.post("/rules/{rule_id}/edit", response_class=HTMLResponse)
|
||||||
async def rule_save(request: Request, rule_id: int | None = None) -> Response:
|
@protected.post("/rules/{source_id}/copy", response_class=HTMLResponse)
|
||||||
|
async def rule_save(
|
||||||
|
request: Request, rule_id: int | None = None, source_id: int | None = None
|
||||||
|
) -> Response:
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
form = _form_dict(await request.form())
|
form = _form_dict(await request.form())
|
||||||
with session_scope(state.session_factory) as s:
|
with session_scope(state.session_factory) as s:
|
||||||
|
source = s.get(Rule, source_id) if source_id is not None else None
|
||||||
|
copy_of = source.name if source is not None else None
|
||||||
d, errors = parse_rule_form(s, form)
|
d, errors = parse_rule_form(s, form)
|
||||||
r = None
|
r = None
|
||||||
if not errors:
|
if not errors:
|
||||||
try:
|
try:
|
||||||
r, errors = save_rule(s, d, rule_id)
|
r, errors = save_rule(s, d, rule_id) # Kopie: rule_id ist None → neue Regel
|
||||||
except AdminError as exc:
|
except AdminError as exc:
|
||||||
errors = {"_": str(exc)}
|
errors = {"_": str(exc)}
|
||||||
if errors or r is None:
|
if errors or r is None:
|
||||||
s.rollback()
|
s.rollback()
|
||||||
return _rule_form(request, d, errors, rule_id, status_code=400)
|
return _rule_form(request, d, errors, rule_id, status_code=400, copy_of=copy_of)
|
||||||
name = r.name
|
name, enabled = r.name, r.enabled
|
||||||
state.scheduler.sync()
|
state.scheduler.sync()
|
||||||
flash(request, f"Regel „{name}“ gespeichert.")
|
flash(request, f"Regel „{name}“ gespeichert.")
|
||||||
|
if copy_of and not enabled:
|
||||||
|
flash(request, "Die Kopie ist deaktiviert. Probelauf machen, dann aktivieren.", "info")
|
||||||
return redirect("/rules")
|
return redirect("/rules")
|
||||||
|
|
||||||
|
|
||||||
@@ -1063,6 +1156,7 @@ def _monitor_urls(request: Request) -> list[tuple[str, str]]:
|
|||||||
(f"Regel „{n}“", f"{base}/healthz/rule/{rid}")
|
(f"Regel „{n}“", f"{base}/healthz/rule/{rid}")
|
||||||
for rid, n in s.execute(select(Rule.id, Rule.name).order_by(Rule.name))
|
for rid, n in s.execute(select(Rule.id, Rule.name).order_by(Rule.name))
|
||||||
]
|
]
|
||||||
|
urls.append(("Sicherung", f"{base}/healthz/backup"))
|
||||||
return urls
|
return urls
|
||||||
|
|
||||||
|
|
||||||
@@ -1148,6 +1242,100 @@ async def settings_save(request: Request) -> Response:
|
|||||||
return redirect("/settings")
|
return redirect("/settings")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Datensicherung -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _backup_page(
|
||||||
|
request: Request, cfg: AppSettings, errors: dict[str, str], status_code: int = 200
|
||||||
|
) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
with state.session_factory() as s:
|
||||||
|
mounts = [
|
||||||
|
(m.id, f"{m.name} ({m.type.value})")
|
||||||
|
for m in s.scalars(select(Mount).order_by(Mount.name))
|
||||||
|
]
|
||||||
|
health = state.health()
|
||||||
|
return render(
|
||||||
|
request,
|
||||||
|
"backup.html",
|
||||||
|
status_code=status_code,
|
||||||
|
active="backup",
|
||||||
|
cfg=cfg,
|
||||||
|
errors=errors,
|
||||||
|
mounts=mounts,
|
||||||
|
st=load_backup_state(state.session_factory),
|
||||||
|
check=health.find("backup"),
|
||||||
|
running=state.backups.running,
|
||||||
|
monitor_url=f"{str(request.base_url).rstrip('/')}/healthz/backup",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/backup", response_class=HTMLResponse)
|
||||||
|
async def backup_page(request: Request) -> Response:
|
||||||
|
return _backup_page(request, load_settings(nm(request).session_factory), {})
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/backup", response_class=HTMLResponse)
|
||||||
|
async def backup_save(request: Request) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
errors: dict[str, str] = {}
|
||||||
|
cfg.backup_enabled = form.get("backup_enabled") == "on"
|
||||||
|
try:
|
||||||
|
cfg.backup_mount_id = int(form.get("backup_mount_id") or 0)
|
||||||
|
except ValueError:
|
||||||
|
cfg.backup_mount_id = 0
|
||||||
|
with state.session_factory() as s:
|
||||||
|
if cfg.backup_mount_id and s.get(Mount, cfg.backup_mount_id) is None:
|
||||||
|
errors["backup_mount_id"] = "Anbindung existiert nicht."
|
||||||
|
try:
|
||||||
|
cfg.backup_path = safe_relpath("backup_path", form.get("backup_path") or "")
|
||||||
|
except ValidationError as exc:
|
||||||
|
errors["backup_path"] = str(exc)
|
||||||
|
cfg.backup_time = (form.get("backup_time") or "03:30").strip()
|
||||||
|
try:
|
||||||
|
parse_backup_time(cfg.backup_time)
|
||||||
|
except ValueError as exc:
|
||||||
|
errors["backup_time"] = str(exc)
|
||||||
|
try:
|
||||||
|
cfg.backup_keep = int(form.get("backup_keep") or 14)
|
||||||
|
if not 1 <= cfg.backup_keep <= 365:
|
||||||
|
raise ValueError
|
||||||
|
except ValueError:
|
||||||
|
errors["backup_keep"] = "Zwischen 1 und 365."
|
||||||
|
passphrase = form.get("passphrase") or ""
|
||||||
|
if passphrase or form.get("passphrase_confirm"):
|
||||||
|
problem = passphrase_problem(passphrase, form.get("passphrase_confirm") or "")
|
||||||
|
if problem:
|
||||||
|
errors["passphrase"] = problem
|
||||||
|
else:
|
||||||
|
cfg.backup_passphrase_enc = state.secrets.encrypt(passphrase)
|
||||||
|
if cfg.backup_enabled and not cfg.backup_mount_id:
|
||||||
|
errors["backup_mount_id"] = "Für die Sicherung eine Anbindung wählen."
|
||||||
|
if cfg.backup_enabled and not cfg.backup_passphrase_enc and "passphrase" not in errors:
|
||||||
|
errors["passphrase"] = "Für die Sicherung eine Passphrase festlegen." # noqa: S105
|
||||||
|
if errors:
|
||||||
|
return _backup_page(request, cfg, errors, status_code=400)
|
||||||
|
save_settings(state.session_factory, cfg)
|
||||||
|
flash(request, "Sicherung gespeichert.")
|
||||||
|
if passphrase:
|
||||||
|
flash(
|
||||||
|
request,
|
||||||
|
"Passphrase jetzt im Passwortmanager ablegen – ohne sie lässt sich keine "
|
||||||
|
"Sicherung wiederherstellen.",
|
||||||
|
"warn",
|
||||||
|
)
|
||||||
|
return redirect("/backup")
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/backup/run")
|
||||||
|
async def backup_run(request: Request) -> Response:
|
||||||
|
result = await run_in_threadpool(nm(request).backups.run)
|
||||||
|
flash(request, result.message, "ok" if result.ok else "error")
|
||||||
|
return redirect("/backup")
|
||||||
|
|
||||||
|
|
||||||
def auth_mask(url: str) -> str:
|
def auth_mask(url: str) -> str:
|
||||||
"""Webhook-URL nur gekürzt anzeigen – sie enthält oft ein Token."""
|
"""Webhook-URL nur gekürzt anzeigen – sie enthält oft ein Token."""
|
||||||
if not url:
|
if not url:
|
||||||
@@ -1235,39 +1423,54 @@ async def settings_password(request: Request) -> Response:
|
|||||||
|
|
||||||
@protected.post("/settings/auth", response_class=HTMLResponse)
|
@protected.post("/settings/auth", response_class=HTMLResponse)
|
||||||
async def settings_auth(request: Request) -> Response:
|
async def settings_auth(request: Request) -> Response:
|
||||||
"""Anmeldung abschalten bzw. auf vertrauenswürdige Netze beschränken.
|
"""Netze ohne / mit Passwort festlegen. Nur mit dem aktuellen Passwort und nie so,
|
||||||
Nur mit dem aktuellen Passwort."""
|
dass man sich mit der eigenen Adresse aussperrt."""
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
form = _form_dict(await request.form())
|
form = _form_dict(await request.form())
|
||||||
|
back = redirect("/settings#anmeldung")
|
||||||
if not await run_in_threadpool(
|
if not await run_in_threadpool(
|
||||||
auth.verify_password, state.session_factory, form.get("current", "")
|
auth.verify_password, state.session_factory, form.get("current", "")
|
||||||
):
|
):
|
||||||
flash(request, "Aktuelles Passwort falsch – Anmeldung unverändert.", "error")
|
flash(request, "Aktuelles Passwort falsch – Anmeldung unverändert.", "error")
|
||||||
return redirect("/settings#anmeldung")
|
return back
|
||||||
mode = form.get("auth_mode", auth.AUTH_ALWAYS)
|
trusted = (form.get("trusted_networks") or "").strip()
|
||||||
networks = (form.get("trusted_networks") or "").strip()
|
allowed = (form.get("allowed_networks") or "").strip()
|
||||||
if mode not in auth.AUTH_MODES:
|
|
||||||
flash(request, "Ungültige Auswahl.", "error")
|
|
||||||
return redirect("/settings#anmeldung")
|
|
||||||
try:
|
try:
|
||||||
nets = auth.parse_networks(networks)
|
auth.parse_networks(trusted)
|
||||||
|
auth.parse_networks(allowed)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
flash(request, str(exc), "error")
|
flash(request, str(exc), "error")
|
||||||
return redirect("/settings#anmeldung")
|
return back
|
||||||
if mode == auth.AUTH_TRUSTED and not nets:
|
me = auth.client_host(request)
|
||||||
flash(request, "Bitte mindestens ein vertrauenswürdiges Netz angeben.", "error")
|
if auth.access_decision(me, auth.AUTH_ALWAYS, trusted, allowed) == auth.ACCESS_DENY:
|
||||||
return redirect("/settings#anmeldung")
|
flash(
|
||||||
|
request,
|
||||||
|
f"Nicht gespeichert: Damit wärst du selbst ausgesperrt (deine Adresse {me}). "
|
||||||
|
"Trage dein Netz bei „mit“ oder „ohne Passwort“ ein.",
|
||||||
|
"error",
|
||||||
|
)
|
||||||
|
return back
|
||||||
cfg = load_settings(state.session_factory)
|
cfg = load_settings(state.session_factory)
|
||||||
cfg.auth_mode, cfg.trusted_networks = mode, networks
|
cfg.trusted_networks, cfg.allowed_networks = trusted, allowed
|
||||||
|
cfg.auth_mode = auth.AUTH_TRUSTED if trusted else auth.AUTH_ALWAYS
|
||||||
save_settings(state.session_factory, cfg)
|
save_settings(state.session_factory, cfg)
|
||||||
event(log, "auth.mode_changed", level=30, mode=mode, networks=networks)
|
event(
|
||||||
msgs = {
|
log,
|
||||||
auth.AUTH_ALWAYS: "Anmeldung ist wieder immer erforderlich.",
|
"auth.access_changed",
|
||||||
auth.AUTH_TRUSTED: f"Ohne Anmeldung aus: {networks}. Von überall sonst mit Passwort.",
|
level=30,
|
||||||
auth.AUTH_OFF: "Anmeldung abgeschaltet – NetMover ist ohne Passwort bedienbar.",
|
without_password=trusted,
|
||||||
}
|
with_password=allowed or "alle übrigen",
|
||||||
flash(request, msgs[mode], "info" if mode == auth.AUTH_ALWAYS else "warn")
|
)
|
||||||
return redirect("/settings#anmeldung")
|
parts = [f"ohne Passwort aus: {trusted}" if trusted else "überall mit Passwort"]
|
||||||
|
parts.append(
|
||||||
|
f"mit Passwort nur aus: {allowed} – alle anderen Netze abgewiesen"
|
||||||
|
if allowed
|
||||||
|
else "alle übrigen Netze mit Passwort"
|
||||||
|
)
|
||||||
|
flash(
|
||||||
|
request, "Anmeldung gespeichert: " + "; ".join(parts) + ".", "warn" if trusted else "info"
|
||||||
|
)
|
||||||
|
return back
|
||||||
|
|
||||||
|
|
||||||
@protected.get("/settings/export")
|
@protected.get("/settings/export")
|
||||||
@@ -1371,6 +1574,35 @@ def create_app(settings: Settings | None = None, state: NetMover | None = None)
|
|||||||
openapi_url=None,
|
openapi_url=None,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@app.middleware("http")
|
||||||
|
async def network_access(
|
||||||
|
request: Request, call_next: Callable[[Request], Awaitable[Response]]
|
||||||
|
) -> Response:
|
||||||
|
path = request.url.path
|
||||||
|
if not (path.startswith("/healthz") or path.startswith("/static/")):
|
||||||
|
app_state = getattr(request.app.state, "nm", None)
|
||||||
|
if app_state is not None:
|
||||||
|
cfg = load_settings(app_state.session_factory)
|
||||||
|
decision = auth.access_decision(
|
||||||
|
auth.client_host(request),
|
||||||
|
cfg.auth_mode,
|
||||||
|
cfg.trusted_networks,
|
||||||
|
cfg.allowed_networks,
|
||||||
|
)
|
||||||
|
if decision == auth.ACCESS_DENY:
|
||||||
|
event(
|
||||||
|
log,
|
||||||
|
"auth.denied_network",
|
||||||
|
level=30,
|
||||||
|
client=auth.client_host(request) or "?",
|
||||||
|
)
|
||||||
|
return HTMLResponse(
|
||||||
|
"<!doctype html><meta charset=utf-8><title>NetMover</title>"
|
||||||
|
"<p>Zugriff aus diesem Netz ist nicht erlaubt.</p>",
|
||||||
|
status_code=403,
|
||||||
|
)
|
||||||
|
return await call_next(request)
|
||||||
|
|
||||||
@app.middleware("http")
|
@app.middleware("http")
|
||||||
async def security_headers(
|
async def security_headers(
|
||||||
request: Request, call_next: Callable[[Request], Awaitable[Response]]
|
request: Request, call_next: Callable[[Request], Awaitable[Response]]
|
||||||
|
|||||||
+47
-13
@@ -51,25 +51,59 @@ def parse_networks(raw: str) -> list[ipaddress.IPv4Network | ipaddress.IPv6Netwo
|
|||||||
return nets
|
return nets
|
||||||
|
|
||||||
|
|
||||||
def login_bypass(request: Request, mode: str, trusted: str) -> str | None:
|
ACCESS_NOPW, ACCESS_PW, ACCESS_DENY = "nopw", "pw", "deny"
|
||||||
"""Grund, warum keine Anmeldung nötig ist (``"off"``/``"trusted"``), sonst ``None``.
|
|
||||||
|
|
||||||
|
def _address(host: str | None) -> ipaddress.IPv4Address | ipaddress.IPv6Address | None:
|
||||||
|
if not host:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return ipaddress.ip_address(host)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _networks_or_empty(raw: str) -> list[ipaddress.IPv4Network | ipaddress.IPv6Network]:
|
||||||
|
try:
|
||||||
|
return parse_networks(raw)
|
||||||
|
except ValueError:
|
||||||
|
return [] # gespeicherte Werte sind beim Speichern geprüft
|
||||||
|
|
||||||
|
|
||||||
|
def access_decision(host: str | None, mode: str, trusted: str, allowed: str) -> str:
|
||||||
|
"""Wie darf diese Adresse auf die Oberfläche zugreifen?
|
||||||
|
|
||||||
|
1. aus ``trusted`` (oder Modus „off“): **ohne Passwort**,
|
||||||
|
2. ``allowed`` leer oder Adresse darin: **mit Passwort**,
|
||||||
|
3. sonst: **abgewiesen**.
|
||||||
|
|
||||||
Hinter einem Reverse Proxy ist die Client-Adresse nur dann echt, wenn
|
Hinter einem Reverse Proxy ist die Client-Adresse nur dann echt, wenn
|
||||||
``NETMOVER_FORWARDED_ALLOW_IPS`` den Proxy kennt – sonst sähe jede Anfrage aus,
|
``NETMOVER_FORWARDED_ALLOW_IPS`` den Proxy kennt – sonst sähe jede Anfrage aus,
|
||||||
als käme sie vom Proxy selbst."""
|
als käme sie vom Proxy selbst."""
|
||||||
if mode == AUTH_OFF:
|
if mode == AUTH_OFF:
|
||||||
return AUTH_OFF
|
return ACCESS_NOPW
|
||||||
if mode != AUTH_TRUSTED or request.client is None:
|
addr = _address(host)
|
||||||
|
if addr is not None and any(addr in n for n in _networks_or_empty(trusted)):
|
||||||
|
return ACCESS_NOPW
|
||||||
|
allowed_nets = _networks_or_empty(allowed)
|
||||||
|
if allowed_nets and (addr is None or not any(addr in n for n in allowed_nets)):
|
||||||
|
return ACCESS_DENY
|
||||||
|
return ACCESS_PW
|
||||||
|
|
||||||
|
|
||||||
|
def client_host(request: Request) -> str | None:
|
||||||
|
return request.client.host if request.client else None
|
||||||
|
|
||||||
|
|
||||||
|
def login_bypass(request: Request, mode: str, trusted: str, allowed: str = "") -> str | None:
|
||||||
|
"""Grund, warum keine Anmeldung nötig ist (``"off"``/``"trusted"``), sonst ``None``."""
|
||||||
|
if access_decision(client_host(request), mode, trusted, allowed) != ACCESS_NOPW:
|
||||||
return None
|
return None
|
||||||
try:
|
return AUTH_OFF if mode == AUTH_OFF or _covers_everything(trusted) else AUTH_TRUSTED
|
||||||
addr = ipaddress.ip_address(request.client.host)
|
|
||||||
except ValueError:
|
|
||||||
return None
|
def _covers_everything(raw: str) -> bool:
|
||||||
try:
|
return any(n.prefixlen == 0 for n in _networks_or_empty(raw))
|
||||||
nets = parse_networks(trusted)
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
return AUTH_TRUSTED if any(addr in n for n in nets) else None
|
|
||||||
|
|
||||||
|
|
||||||
def includes_loopback(raw: str) -> bool:
|
def includes_loopback(raw: str) -> bool:
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
{% from "macros.html" import field, select, check, csrf_field %}
|
||||||
|
{% block title %}Sicherung{% endblock %}
|
||||||
|
{% block content %}
|
||||||
|
<h1>Sicherung</h1>
|
||||||
|
<p class="muted">Täglich Datenbank und Schlüssel – also alle Anbindungen, Regeln, Zugangsdaten und Einstellungen –
|
||||||
|
verschlüsselt auf eine Anbindung, z. B. eine eigene Freigabe auf dem NAS.</p>
|
||||||
|
|
||||||
|
<div class="card stack">
|
||||||
|
<h2>Zustand</h2>
|
||||||
|
{% if check %}<p class="msg {{ 'ok' if check.ok else 'error' }}">{{ check.message }}</p>{% endif %}
|
||||||
|
<table class="small"><tbody>
|
||||||
|
<tr><td>Letzte erfolgreiche</td><td>{{ st.last_success|localdt if st.last_success else '–' }}</td></tr>
|
||||||
|
<tr><td>Datei</td><td class="mono">{{ st.last_file or '–' }}{% if st.last_size %} ({{ st.last_size|bytes }}){% endif %}</td></tr>
|
||||||
|
<tr><td>Letzter Versuch</td><td>{{ st.last_attempt|localdt if st.last_attempt else '–' }}</td></tr>
|
||||||
|
{% if st.last_error %}<tr><td>Letzter Fehler</td><td>{{ st.last_error }}{% if st.failures > 1 %} ({{ st.failures }}× in Folge){% endif %}</td></tr>{% endif %}
|
||||||
|
</tbody></table>
|
||||||
|
<form class="row" method="post" action="/backup/run">
|
||||||
|
{{ csrf_field(csrf) }}
|
||||||
|
<button {% if running or not cfg.backup_mount_id or not cfg.backup_passphrase_enc %}disabled{% endif %}>{{ 'Sicherung läuft …' if running else 'Jetzt sichern' }}</button>
|
||||||
|
<span class="hint">Kuma-Monitor: <code>{{ monitor_url }}</code></span>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2>Einstellungen <a class="legend-help" href="/help#sicherung" title="Hilfe">?</a></h2>
|
||||||
|
<form class="form card stack" method="post" action="/backup">
|
||||||
|
{{ csrf_field(csrf) }}
|
||||||
|
<div class="fields">
|
||||||
|
{{ check('backup_enabled', 'Tägliche Sicherung einschalten', cfg.backup_enabled, wide=true) }}
|
||||||
|
{{ select('backup_mount_id', 'Ziel-Anbindung', [(0, '– bitte wählen –')] + mounts, cfg.backup_mount_id, errors=errors, hint='SMB, NFS oder SFTP – am besten eine eigene Freigabe, getrennt von Scans und Archiv.') }}
|
||||||
|
{{ field('backup_path', 'Unterordner', cfg.backup_path, errors=errors, placeholder='z. B. netmover', hint='Wird angelegt, falls nötig. Leer = direkt in der Anbindung.') }}
|
||||||
|
{{ field('backup_time', 'Uhrzeit', cfg.backup_time, type='time', errors=errors, hint='Ortszeit des Servers. Verpasste Sicherungen werden nachgeholt, nach einem Fehler stündlich neuer Versuch.') }}
|
||||||
|
{{ field('backup_keep', 'Anzahl behalten', cfg.backup_keep, type='number', errors=errors, attrs='min="1" max="365"', hint='Ältere NetMover-Sicherungen im Ordner werden gelöscht – andere Dateien nie.') }}
|
||||||
|
<div class="field{{ ' has-error' if errors.get('passphrase') }}"><label for="f-passphrase">Passphrase</label>
|
||||||
|
<input id="f-passphrase" type="password" name="passphrase" autocomplete="new-password" minlength="12" placeholder="{{ '••••••' if cfg.backup_passphrase_enc }}">
|
||||||
|
{% if errors.get('passphrase') %}<span class="error">{{ errors['passphrase'] }}</span>{% endif %}
|
||||||
|
<span class="hint">{{ 'Leer lassen = unverändert. ' if cfg.backup_passphrase_enc }}Mindestens 12 Zeichen.</span></div>
|
||||||
|
<div class="field"><label for="f-passphrase2">Passphrase wiederholen</label>
|
||||||
|
<input id="f-passphrase2" type="password" name="passphrase_confirm" autocomplete="new-password"></div>
|
||||||
|
</div>
|
||||||
|
<p class="msg warn small">Die Passphrase gehört in deinen Passwortmanager. Fällt der Server aus, ist sie der einzige Weg an die Sicherung –
|
||||||
|
NetMover kann sie dann nicht mehr verraten. Eine neue Passphrase gilt für künftige Sicherungen; ältere Dateien brauchen weiter die alte.</p>
|
||||||
|
<div class="row"><button class="primary">Speichern</button></div>
|
||||||
|
</form>
|
||||||
|
|
||||||
|
<h2>Wiederherstellen</h2>
|
||||||
|
<div class="card stack small">
|
||||||
|
<p>Auf dem (neu installierten) Server, gleiche oder neuere NetMover-Version:</p>
|
||||||
|
<pre class="mono">systemctl stop netmover
|
||||||
|
netmover restore /pfad/zur/netmover-….nmbak
|
||||||
|
systemctl start netmover</pre>
|
||||||
|
<p class="muted">Die bisherigen Dateien bleiben in <code>/var/lib/netmover/vor-wiederherstellung-…</code> erhalten.
|
||||||
|
Ausführlich: <a href="/help#sicherung">Hilfe → Sicherung</a>.</p>
|
||||||
|
</div>
|
||||||
|
{% endblock %}
|
||||||
@@ -47,6 +47,7 @@
|
|||||||
<div class="nav-group">
|
<div class="nav-group">
|
||||||
<div class="nav-title">System</div>
|
<div class="nav-title">System</div>
|
||||||
{{ item('settings', '/settings', 'Einstellungen') }}
|
{{ item('settings', '/settings', 'Einstellungen') }}
|
||||||
|
{{ item('backup', '/backup', 'Sicherung', '' if not nav or nav.backup_ok else 'überfällig', '' if not nav or nav.backup_ok else 'bad') }}
|
||||||
{{ item('help', '/help', 'Hilfe') }}
|
{{ item('help', '/help', 'Hilfe') }}
|
||||||
</div>
|
</div>
|
||||||
<div class="nav-foot">
|
<div class="nav-foot">
|
||||||
|
|||||||
@@ -7,6 +7,7 @@
|
|||||||
{%- elif name == 'rules' -%}<path d="M4 7h13l-3-3M20 17H7l3 3"/>
|
{%- elif name == 'rules' -%}<path d="M4 7h13l-3-3M20 17H7l3 3"/>
|
||||||
{%- elif name == 'log' -%}<path d="M8 6h13M8 12h13M8 18h13M3 6h.01M3 12h.01M3 18h.01"/>
|
{%- elif name == 'log' -%}<path d="M8 6h13M8 12h13M8 18h13M3 6h.01M3 12h.01M3 18h.01"/>
|
||||||
{%- elif name == 'settings' -%}<circle cx="12" cy="12" r="3"/><path d="M12 2v3M12 19v3M4.2 4.2l2.1 2.1M17.7 17.7l2.1 2.1M2 12h3M19 12h3M4.2 19.8l2.1-2.1M17.7 6.3l2.1-2.1"/>
|
{%- elif name == 'settings' -%}<circle cx="12" cy="12" r="3"/><path d="M12 2v3M12 19v3M4.2 4.2l2.1 2.1M17.7 17.7l2.1 2.1M2 12h3M19 12h3M4.2 19.8l2.1-2.1M17.7 6.3l2.1-2.1"/>
|
||||||
|
{%- elif name == 'backup' -%}<path d="M4 7c0-1.7 3.6-3 8-3s8 1.3 8 3-3.6 3-8 3-8-1.3-8-3zM4 7v10c0 1.7 3.6 3 8 3s8-1.3 8-3V7M4 12c0 1.7 3.6 3 8 3s8-1.3 8-3"/>
|
||||||
{%- elif name == 'help' -%}<circle cx="12" cy="12" r="9"/><path d="M9.5 9a2.5 2.5 0 1 1 3.5 2.3c-.7.3-1 .9-1 1.7M12 17h.01"/>
|
{%- elif name == 'help' -%}<circle cx="12" cy="12" r="9"/><path d="M9.5 9a2.5 2.5 0 1 1 3.5 2.3c-.7.3-1 .9-1 1.7M12 17h.01"/>
|
||||||
{%- elif name == 'logout' -%}<path d="M15 4h4v16h-4M10 8l-4 4 4 4M6 12h11"/>
|
{%- elif name == 'logout' -%}<path d="M15 4h4v16h-4M10 8l-4 4 4 4M6 12h11"/>
|
||||||
{%- elif name == 'menu' -%}<path d="M4 6h16M4 12h16M4 18h16"/>
|
{%- elif name == 'menu' -%}<path d="M4 6h16M4 12h16M4 18h16"/>
|
||||||
|
|||||||
@@ -1,12 +1,14 @@
|
|||||||
{% extends "base.html" %}
|
{% extends "base.html" %}
|
||||||
{% from "macros.html" import field, select, check %}
|
{% from "macros.html" import field, select, check %}
|
||||||
{% block title %}{{ 'Anbindung bearbeiten' if mount else 'Neue Anbindung' }}{% endblock %}
|
{% block title %}{{ 'Anbindung bearbeiten' if mount else ('Anbindung kopieren' if copy_from else 'Neue Anbindung') }}{% endblock %}
|
||||||
{% block content %}
|
{% block content %}
|
||||||
<h1>{{ 'Anbindung „' ~ mount.name ~ '“ bearbeiten' if mount else 'Neue Anbindung' }}</h1>
|
<h1>{% if mount %}Anbindung „{{ mount.name }}“ bearbeiten{% elif copy_from %}Kopie von „{{ copy_from.name }}“{% else %}Neue Anbindung{% endif %}</h1>
|
||||||
|
{% if copy_from %}<p class="msg info small">Alle Werte stammen aus „{{ copy_from.name }}“. Zugangsdaten werden übernommen, solange du das Passwort leer lässt – die Kopie bekommt ihr eigenes Exemplar.{% if copy_from.credential and copy_from.credential.public_key and not copy_from.credential.managed %} Der SSH-Schlüssel bleibt derselbe, ein neuer Eintrag in <code>authorized_keys</code> ist nicht nötig.{% endif %}</p>{% endif %}
|
||||||
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
||||||
<form class="form" method="post" data-dynamic>
|
<form class="form" method="post" data-dynamic>
|
||||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||||
{% if mount %}<input type="hidden" name="mount_id" value="{{ mount.id }}">{% endif %}
|
{% if mount %}<input type="hidden" name="mount_id" value="{{ mount.id }}">{% endif %}
|
||||||
|
{% if copy_from %}<input type="hidden" name="copy_from" value="{{ copy_from.id }}">{% endif %}
|
||||||
<fieldset><legend>Verbindung <a class="legend-help" href="/help#anbindung-felder" title="Hilfe">?</a></legend><div class="fields">
|
<fieldset><legend>Verbindung <a class="legend-help" href="/help#anbindung-felder" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ field('name', 'Name', d.name, errors=errors, attrs='required maxlength="100"', hint=('Verzeichnis: ' ~ mount.mountpoint if mount and mount.mountpoint else '')) }}
|
{{ field('name', 'Name', d.name, errors=errors, attrs='required maxlength="100"', hint=('Verzeichnis: ' ~ mount.mountpoint if mount and mount.mountpoint else '')) }}
|
||||||
{{ select('type', 'Typ', [('cifs', 'SMB/CIFS'), ('nfs4', 'NFS 4'), ('nfs3', 'NFS 3'), ('sftp', 'SFTP')], d.type.value, errors=errors) }}
|
{{ select('type', 'Typ', [('cifs', 'SMB/CIFS'), ('nfs4', 'NFS 4'), ('nfs3', 'NFS 3'), ('sftp', 'SFTP')], d.type.value, errors=errors) }}
|
||||||
|
|||||||
@@ -22,6 +22,7 @@
|
|||||||
{% endif %}
|
{% endif %}
|
||||||
{% endif %}
|
{% endif %}
|
||||||
<a class="btn small" href="/mounts/{{ m.id }}/edit">Bearbeiten</a>
|
<a class="btn small" href="/mounts/{{ m.id }}/edit">Bearbeiten</a>
|
||||||
|
<a class="btn small" href="/mounts/{{ m.id }}/copy" title="Neue Anbindung mit denselben Werten und Zugangsdaten">Kopieren</a>
|
||||||
<form class="inline" method="post" action="/mounts/{{ m.id }}/delete" data-confirm="„{{ m.name }}“ wirklich löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
<form class="inline" method="post" action="/mounts/{{ m.id }}/delete" data-confirm="„{{ m.name }}“ wirklich löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -1,8 +1,11 @@
|
|||||||
{% extends "base.html" %}
|
{% extends "base.html" %}
|
||||||
{% from "macros.html" import field, select, check %}
|
{% from "macros.html" import field, select, check %}
|
||||||
{% block title %}{{ 'Regel bearbeiten' if rule_id else 'Neue Regel' }}{% endblock %}
|
{% block title %}{{ 'Regel bearbeiten' if rule_id else ('Regel kopieren' if copy_of else 'Neue Regel') }}{% endblock %}
|
||||||
{% block content %}
|
{% block content %}
|
||||||
<h1>{{ 'Regel bearbeiten' if rule_id else 'Neue Regel' }}</h1>
|
<h1>{% if rule_id %}Regel bearbeiten{% elif copy_of %}Kopie von „{{ copy_of }}“{% else %}Neue Regel{% endif %}</h1>
|
||||||
|
{% if copy_of %}<p class="msg info small">Alle Einstellungen stammen aus „{{ copy_of }}“. Die Kopie ist zunächst
|
||||||
|
<strong>deaktiviert</strong>, damit sie nicht mit dem Original um dieselben Dateien konkurriert – Quelle bzw. Ziel
|
||||||
|
anpassen, speichern, Probelauf machen, dann aktivieren. Protokoll und Buchführung beginnen bei der Kopie neu.</p>{% endif %}
|
||||||
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
||||||
{% if not src_mounts %}<p class="msg info">Zuerst eine Anbindung als Quelle <a href="/mounts/new">anlegen</a>.</p>{% endif %}
|
{% if not src_mounts %}<p class="msg info">Zuerst eine Anbindung als Quelle <a href="/mounts/new">anlegen</a>.</p>{% endif %}
|
||||||
<form class="form" method="post" data-dynamic>
|
<form class="form" method="post" data-dynamic>
|
||||||
@@ -30,6 +33,13 @@
|
|||||||
{{ select('on_collision', 'Wenn Datei existiert', [('rename', 'umbenennen (scan-1.pdf)'), ('skip', 'überspringen'), ('overwrite', 'überschreiben')], d.on_collision.value, errors=errors) }}
|
{{ select('on_collision', 'Wenn Datei existiert', [('rename', 'umbenennen (scan-1.pdf)'), ('skip', 'überspringen'), ('overwrite', 'überschreiben')], d.on_collision.value, errors=errors) }}
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
|
|
||||||
|
<fieldset><legend>Archiv <a class="legend-help" href="/help#archiv" title="Hilfe">?</a></legend><div class="fields">
|
||||||
|
{{ select('archive_kind', 'Archiv', [('none', 'kein Archiv'), ('mount', 'Anbindung'), ('local', 'Lokaler Pfad')], d.archive_kind.value, errors=errors, hint='Zusätzliche Kopie. Die Quelle wird erst gelöscht, wenn Ziel UND Archiv sie haben.') }}
|
||||||
|
<div data-show-for="archive_kind:mount">{{ select('archive_mount_id', 'Archiv-Anbindung', dst_options, d.archive_mount_id or '', errors=errors) }}</div>
|
||||||
|
<div data-show-for="archive_kind:mount local">{{ field('archive_path', 'Archivordner', d.archive_path, errors=errors, hint='Anbindung: relativer Unterordner (wird angelegt). Lokal: absoluter Pfad, muss existieren.') }}</div>
|
||||||
|
<div data-show-for="archive_kind:mount local">{{ select('archive_layout', 'Ablage', [('flat', 'wie in der Quelle'), ('date', 'nach Datum (Jahr/Monat)')], d.archive_layout.value, errors=errors, hint='Im Archiv wird nie überschrieben – gleiche Namen werden umbenannt.') }}</div>
|
||||||
|
</div></fieldset>
|
||||||
|
|
||||||
<fieldset><legend>Sicherheit und Grenzen <a class="legend-help" href="/help#regel-sicherheit" title="Hilfe">?</a></legend><div class="fields">
|
<fieldset><legend>Sicherheit und Grenzen <a class="legend-help" href="/help#regel-sicherheit" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ field('stability_seconds', 'Stabilitätsfenster (s)', d.stability_seconds, type='number', errors=errors, attrs='min="0"', hint='Größe und Zeit müssen so lange gleich bleiben') }}
|
{{ field('stability_seconds', 'Stabilitätsfenster (s)', d.stability_seconds, type='number', errors=errors, attrs='min="0"', hint='Größe und Zeit müssen so lange gleich bleiben') }}
|
||||||
{{ field('min_age_seconds', 'Mindestalter (s)', d.min_age_seconds, type='number', errors=errors, attrs='min="0"') }}
|
{{ field('min_age_seconds', 'Mindestalter (s)', d.min_age_seconds, type='number', errors=errors, attrs='min="0"') }}
|
||||||
|
|||||||
@@ -18,6 +18,7 @@
|
|||||||
<button class="small" hx-post="/rules/{{ r.id }}/dryrun" hx-target="#dry-{{ r.id }}" hx-swap="innerHTML">Probelauf <span class="htmx-indicator">…</span></button>
|
<button class="small" hx-post="/rules/{{ r.id }}/dryrun" hx-target="#dry-{{ r.id }}" hx-swap="innerHTML">Probelauf <span class="htmx-indicator">…</span></button>
|
||||||
<form class="inline" method="post" action="/rules/{{ r.id }}/toggle">{{ csrf_field(csrf) }}<button class="small">{{ 'Deaktivieren' if r.enabled else 'Aktivieren' }}</button></form>
|
<form class="inline" method="post" action="/rules/{{ r.id }}/toggle">{{ csrf_field(csrf) }}<button class="small">{{ 'Deaktivieren' if r.enabled else 'Aktivieren' }}</button></form>
|
||||||
<a class="btn small" href="/rules/{{ r.id }}/edit">Bearbeiten</a>
|
<a class="btn small" href="/rules/{{ r.id }}/edit">Bearbeiten</a>
|
||||||
|
<a class="btn small" href="/rules/{{ r.id }}/copy" title="Neue Regel mit denselben Einstellungen">Kopieren</a>
|
||||||
<form class="inline" method="post" action="/rules/{{ r.id }}/delete" data-confirm="Regel „{{ r.name }}“ und ihr Protokoll löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
<form class="inline" method="post" action="/rules/{{ r.id }}/delete" data-confirm="Regel „{{ r.name }}“ und ihr Protokoll löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -66,20 +66,23 @@
|
|||||||
</form>
|
</form>
|
||||||
|
|
||||||
<h2 id="anmeldung">Anmeldung <a class="legend-help" href="/help#anmeldung" title="Hilfe">?</a></h2>
|
<h2 id="anmeldung">Anmeldung <a class="legend-help" href="/help#anmeldung" title="Hilfe">?</a></h2>
|
||||||
<form class="form card stack" method="post" action="/settings/auth" data-dynamic>
|
<form class="form card stack" method="post" action="/settings/auth">
|
||||||
{{ csrf_field(csrf) }}
|
{{ csrf_field(csrf) }}
|
||||||
|
<p class="small muted">Deine Adresse gerade: <code>{{ client_ip }}</code></p>
|
||||||
<div class="fields">
|
<div class="fields">
|
||||||
{{ select('auth_mode', 'Anmeldung erforderlich', [('always', 'immer (empfohlen)'), ('trusted', 'nicht aus vertrauten Netzen'), ('off', 'nie')], cfg.auth_mode) }}
|
<div class="field wide"><label for="f-nets">Ohne Passwort aus diesen Netzen</label>
|
||||||
<div class="field wide" data-show-for="auth_mode:trusted"><label for="f-nets">Vertrauenswürdige Netze</label>
|
<input id="f-nets" type="text" name="trusted_networks" value="{{ cfg.trusted_networks }}" placeholder="z. B. 10.0.10.0/24, 10.1.5.0/24">
|
||||||
<input id="f-nets" type="text" name="trusted_networks" value="{{ cfg.trusted_networks }}" placeholder="10.0.10.0/24, 192.168.1.20">
|
<span class="hint">Mit Komma getrennt; einzelne Adressen gehen auch. Leer = niemand ohne Passwort. <code>0.0.0.0/0, ::/0</code> = Anmeldung ganz aus.</span></div>
|
||||||
<span class="hint">Mit Komma getrennt. Deine Adresse gerade: <code>{{ client_ip }}</code>. Von allen anderen Adressen wird weiter nach dem Passwort gefragt.</span></div>
|
<div class="field wide"><label for="f-allowed">Mit Passwort aus diesen Netzen</label>
|
||||||
|
<input id="f-allowed" type="text" name="allowed_networks" value="{{ cfg.allowed_networks }}" placeholder="leer = alle übrigen Netze">
|
||||||
|
<span class="hint">Leer = von überall sonst mit Passwort. Ausgefüllt = <strong>alle anderen Netze werden abgewiesen</strong> (auch die Anmeldeseite). <code>/healthz</code> bleibt für die Überwachung erreichbar.</span></div>
|
||||||
<div class="field"><label for="f-cur-auth">Aktuelles Passwort (zur Bestätigung)</label>
|
<div class="field"><label for="f-cur-auth">Aktuelles Passwort (zur Bestätigung)</label>
|
||||||
<input id="f-cur-auth" type="password" name="current" autocomplete="current-password" required></div>
|
<input id="f-cur-auth" type="password" name="current" autocomplete="current-password" required></div>
|
||||||
</div>
|
</div>
|
||||||
<p class="msg warn small" data-show-for="auth_mode:off|auth_mode:trusted">NetMover läuft als root und kennt die Zugangsdaten deiner Freigaben.
|
<p class="msg warn small">NetMover läuft als root und kennt die Zugangsdaten deiner Freigaben. Wer die Oberfläche ohne Passwort erreicht,
|
||||||
Ohne Anmeldung kann jeder, der die Oberfläche erreicht, Anbindungen und Regeln ändern. „Nie“ nur hinter einem Reverse Proxy mit eigener Anmeldung (z. B. Authelia) verwenden.
|
kann alles ändern – Netze „ohne Passwort“ daher so eng wie möglich wählen. Hinter einem Reverse Proxy <code>NETMOVER_FORWARDED_ALLOW_IPS</code> setzen,
|
||||||
Hinter einem Proxy außerdem <code>NETMOVER_FORWARDED_ALLOW_IPS</code> setzen, sonst sieht jede Anfrage aus, als käme sie vom Proxy.</p>
|
sonst sieht jede Anfrage aus, als käme sie vom Proxy. Eine Einstellung, mit der du dich selbst aussperren würdest, wird nicht gespeichert.</p>
|
||||||
{% if loopback_trusted %}<p class="msg warn small">Die Liste enthält 127.0.0.1/::1. Läuft ein Reverse Proxy auf diesem Server, gilt dann <strong>jede</strong> Anfrage über den Proxy als vertrauenswürdig.</p>{% endif %}
|
{% if loopback_trusted %}<p class="msg warn small">„Ohne Passwort“ enthält 127.0.0.1/::1. Läuft ein Reverse Proxy auf diesem Server, gilt dann <strong>jede</strong> Anfrage über den Proxy als vertrauenswürdig.</p>{% endif %}
|
||||||
<div class="row"><button class="primary">Anmeldung speichern</button></div>
|
<div class="row"><button class="primary">Anmeldung speichern</button></div>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,351 @@
|
|||||||
|
"""0.7.0: Archiv – Kopie ans Ziel UND ins Archiv, Quelle erst danach löschen."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.models import (
|
||||||
|
ArchivedFile,
|
||||||
|
ArchiveKind,
|
||||||
|
CollisionPolicy,
|
||||||
|
ItemStatus,
|
||||||
|
Rule,
|
||||||
|
RuleAction,
|
||||||
|
RunStatus,
|
||||||
|
)
|
||||||
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
|
from netmover.mounting.manager import MountManager
|
||||||
|
from netmover.mover.destinations import LocalDestination, TransferError
|
||||||
|
from netmover.mover.engine import Engine, RuleSnapshot, RunResult
|
||||||
|
from netmover.mover.ledger import Ledger
|
||||||
|
from netmover.mover.runner import RuleRunner
|
||||||
|
from netmover.timeouts import CancelToken
|
||||||
|
from tests.conftest import MountFactory
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def rule_id(make_mount: MountFactory, session_factory: sessionmaker[Session]) -> int:
|
||||||
|
mid = make_mount()
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
r = Rule(name="r", src_mount_id=mid, dst_path="/x")
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
return r.id
|
||||||
|
|
||||||
|
|
||||||
|
FIXED = datetime(2026, 10, 12, 9, 30).timestamp()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def engine(session_factory: sessionmaker[Session]) -> Engine:
|
||||||
|
return Engine(Ledger(session_factory), sleep=lambda _: None, clock=lambda: FIXED)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def dirs(tmp_path: Path) -> tuple[Path, Path, Path]:
|
||||||
|
src, dst, arch = tmp_path / "src", tmp_path / "dst", tmp_path / "archiv"
|
||||||
|
for d in (src, dst, arch):
|
||||||
|
d.mkdir()
|
||||||
|
return src, dst, arch
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def cross_device(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""tmp_path liegt auf einem Gerät – realistischer Weg über Kopie erzwingen."""
|
||||||
|
monkeypatch.setattr(LocalDestination, "move_same_device", lambda *a, **k: None)
|
||||||
|
|
||||||
|
|
||||||
|
def snap(rule_id: int, **kw: object) -> RuleSnapshot:
|
||||||
|
return RuleSnapshot(id=rule_id, name="r", stability_seconds=0, **kw) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def run(
|
||||||
|
engine: Engine, rule: RuleSnapshot, src: Path, dst: Path, arch: Path | None, **kw: object
|
||||||
|
) -> RunResult:
|
||||||
|
archive = LocalDestination(arch) if arch is not None else None
|
||||||
|
return engine.execute(rule, src, LocalDestination(dst), CancelToken(), archive=archive, **kw) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def write(p: Path, data: bytes = b"inhalt") -> Path:
|
||||||
|
p.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
p.write_bytes(data)
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def names(d: Path) -> list[str]:
|
||||||
|
return sorted(str(p.relative_to(d)) for p in d.rglob("*") if p.is_file())
|
||||||
|
|
||||||
|
|
||||||
|
def test_move_delivers_to_target_and_archive_then_deletes_source(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
data = os.urandom(200_000)
|
||||||
|
write(src / "scan001.pdf", data)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert result.status is RunStatus.OK
|
||||||
|
item = result.items[0]
|
||||||
|
assert item.status is ItemStatus.MOVED and item.message and "archiviert:" in item.message
|
||||||
|
assert (dst / "scan001.pdf").read_bytes() == data == (arch / "scan001.pdf").read_bytes()
|
||||||
|
assert not (src / "scan001.pdf").exists()
|
||||||
|
assert names(dst) == ["scan001.pdf"] and names(arch) == ["scan001.pdf"] # keine .part-Reste
|
||||||
|
|
||||||
|
|
||||||
|
def test_date_layout(engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "max" / "a.pdf")
|
||||||
|
run(engine, snap(rule_id, recursive=True, archive_layout="date"), src, dst, arch)
|
||||||
|
assert names(arch) == ["2026/10/max/a.pdf"] and names(dst) == ["max/a.pdf"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_archive_failure_delivers_nothing_and_keeps_source(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path], monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf")
|
||||||
|
real = LocalDestination._link_noreplace
|
||||||
|
|
||||||
|
def fail_in_archive(self: LocalDestination, tmp: str, final: str) -> None:
|
||||||
|
if self.root == arch:
|
||||||
|
raise TransferError("Archiv voll")
|
||||||
|
real(self, tmp, final)
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_link_noreplace", fail_in_archive)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
item = result.items[0]
|
||||||
|
assert item.status is ItemStatus.FAILED and item.message == "Archiv: Archiv voll (Versuch 1)"
|
||||||
|
assert (src / "a.pdf").exists() and names(dst) == [] and names(arch) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_target_failure_after_archive_never_archives_twice(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
dirs: tuple[Path, Path, Path],
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf")
|
||||||
|
real = LocalDestination._link_noreplace
|
||||||
|
|
||||||
|
def fail_in_target(self: LocalDestination, tmp: str, final: str) -> None:
|
||||||
|
if self.root == dst:
|
||||||
|
raise TransferError("Ziel gesperrt")
|
||||||
|
real(self, tmp, final)
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_link_noreplace", fail_in_target)
|
||||||
|
first = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert first.items[0].status is ItemStatus.FAILED
|
||||||
|
assert (src / "a.pdf").exists() and names(arch) == ["a.pdf"] and names(dst) == []
|
||||||
|
with session_factory() as s:
|
||||||
|
assert len(s.scalars(select(ArchivedFile)).all()) == 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_link_noreplace", real) # Ziel wieder frei
|
||||||
|
second = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert second.items[0].status is ItemStatus.MOVED
|
||||||
|
assert names(arch) == ["a.pdf"] # keine zweite Archivkopie (a-1.pdf)
|
||||||
|
assert names(dst) == ["a.pdf"] and not (src / "a.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_archive_never_overwrites(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(arch / "scan.pdf", b"alt im Archiv")
|
||||||
|
write(src / "scan.pdf", b"neu")
|
||||||
|
run(engine, snap(rule_id, on_collision=CollisionPolicy.OVERWRITE), src, dst, arch)
|
||||||
|
assert (arch / "scan.pdf").read_bytes() == b"alt im Archiv"
|
||||||
|
assert (arch / "scan-1.pdf").read_bytes() == b"neu"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_space_in_archive_writes_nothing(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path], monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf", b"x" * 1000)
|
||||||
|
real = LocalDestination.free_bytes
|
||||||
|
monkeypatch.setattr(
|
||||||
|
LocalDestination, "free_bytes", lambda self: 10 if self.root == arch else real(self)
|
||||||
|
)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert "zu wenig Platz im Archiv" in (result.items[0].message or "")
|
||||||
|
assert names(dst) == [] and names(arch) == [] and (src / "a.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_changed_during_archive_copy(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path], monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
f = write(src / "a.pdf", b"original")
|
||||||
|
real = LocalDestination._apply_meta
|
||||||
|
|
||||||
|
def change(self: LocalDestination, rel: str, meta: object) -> list[str]:
|
||||||
|
if self.root == arch:
|
||||||
|
f.write_bytes(b"geaendert!!")
|
||||||
|
return real(self, rel, meta) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_apply_meta", change)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert result.items[0].status is ItemStatus.SKIPPED
|
||||||
|
assert names(dst) == [] and names(arch) == [] and f.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_mode_with_archive(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
f = write(src / "a.pdf", b"v1")
|
||||||
|
rule = snap(rule_id, action=RuleAction.COPY)
|
||||||
|
for _ in range(3):
|
||||||
|
run(engine, rule, src, dst, arch)
|
||||||
|
assert f.exists() and names(dst) == ["a.pdf"] and names(arch) == ["a.pdf"]
|
||||||
|
f.write_bytes(b"version 2")
|
||||||
|
run(engine, rule, src, dst, arch)
|
||||||
|
assert names(dst) == ["a-1.pdf", "a.pdf"] and names(arch) == ["a-1.pdf", "a.pdf"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dry_run_shows_archive_and_writes_nothing(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf")
|
||||||
|
result = run(engine, snap(rule_id, archive_layout="date"), src, dst, arch, dry_run=True)
|
||||||
|
assert result.items[0].status is ItemStatus.PLANNED
|
||||||
|
assert f"Archiv: {arch / '2026/10/a.pdf'}" in (result.items[0].message or "")
|
||||||
|
assert names(dst) == [] and names(arch) == [] and (src / "a.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_without_archive_unchanged(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf")
|
||||||
|
result = run(engine, snap(rule_id), src, dst, None)
|
||||||
|
assert result.items[0].message is None and names(arch) == []
|
||||||
|
|
||||||
|
|
||||||
|
# -- Regellauf ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def setup(
|
||||||
|
manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
make_mount: MountFactory,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> Iterator[tuple[int, Path, Path, Path]]:
|
||||||
|
scans = fake_backend.export("nas", "scans")
|
||||||
|
arch_remote = fake_backend.export("nas", "archiv")
|
||||||
|
src_id = make_mount("scans", share="scans")
|
||||||
|
arch_id = make_mount("archiv", share="archiv")
|
||||||
|
manager.mount(src_id)
|
||||||
|
manager.mount(arch_id)
|
||||||
|
consume = tmp_path / "consume"
|
||||||
|
consume.mkdir()
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
r = Rule(
|
||||||
|
name="Scans → Paperless + Archiv",
|
||||||
|
src_mount_id=src_id,
|
||||||
|
dst_path=str(consume),
|
||||||
|
stability_seconds=0,
|
||||||
|
archive_kind=ArchiveKind.MOUNT,
|
||||||
|
archive_mount_id=arch_id,
|
||||||
|
archive_path="scans",
|
||||||
|
)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
rid = r.id
|
||||||
|
yield rid, scans, consume, arch_remote
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_with_archive_mount(
|
||||||
|
setup: tuple[int, Path, Path, Path],
|
||||||
|
manager: MountManager,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
rid, scans, consume, arch = setup
|
||||||
|
write(scans / "doc.pdf", b"%PDF")
|
||||||
|
result = RuleRunner(session_factory, manager, sleep=lambda _: None).run(rid)
|
||||||
|
assert result.status is RunStatus.OK, result.message
|
||||||
|
assert (
|
||||||
|
(consume / "doc.pdf").read_bytes() == b"%PDF" == (arch / "scans" / "doc.pdf").read_bytes()
|
||||||
|
)
|
||||||
|
assert not (scans / "doc.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_unavailable_archive_makes_rule_wait(
|
||||||
|
setup: tuple[int, Path, Path, Path],
|
||||||
|
manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
rid, scans, consume, _arch = setup
|
||||||
|
write(scans / "doc.pdf")
|
||||||
|
fake_backend.vanish(manager.mountpoint_for("archiv"))
|
||||||
|
result = RuleRunner(session_factory, manager, sleep=lambda _: None).run(rid)
|
||||||
|
assert result.unavailable and result.status is RunStatus.ERROR
|
||||||
|
assert (scans / "doc.pdf").exists() and list(consume.iterdir()) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_mount_used_as_archive_cannot_be_deleted(
|
||||||
|
setup: tuple[int, Path, Path, Path],
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
manager: MountManager,
|
||||||
|
) -> None:
|
||||||
|
from netmover.admin import AdminError, delete_mount
|
||||||
|
from netmover.models import Mount
|
||||||
|
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
arch_id = s.scalar(select(Mount.id).where(Mount.name == "archiv"))
|
||||||
|
assert arch_id is not None
|
||||||
|
with pytest.raises(AdminError, match="Wird noch von Regeln benutzt"):
|
||||||
|
delete_mount(s, manager, arch_id)
|
||||||
|
|
||||||
|
|
||||||
|
def test_rule_form_archive_validation(
|
||||||
|
session_factory: sessionmaker[Session], make_mount: MountFactory
|
||||||
|
) -> None:
|
||||||
|
from netmover.admin import parse_rule_form, save_rule
|
||||||
|
|
||||||
|
mid = make_mount()
|
||||||
|
base = {
|
||||||
|
"name": "x",
|
||||||
|
"src_mount_id": str(mid),
|
||||||
|
"dst_kind": "local",
|
||||||
|
"dst_path": "/srv/consume",
|
||||||
|
"interval_seconds": "30",
|
||||||
|
"stability_seconds": "5",
|
||||||
|
"op_timeout_seconds": "120",
|
||||||
|
}
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
_, err = parse_rule_form(
|
||||||
|
s, {**base, "archive_kind": "local", "archive_path": "/srv/consume"}
|
||||||
|
)
|
||||||
|
assert "derselbe Ordner" in err["archive_path"]
|
||||||
|
_, err = parse_rule_form(s, {**base, "archive_kind": "mount", "archive_path": "a"})
|
||||||
|
assert "archive_mount_id" in err
|
||||||
|
_, err = parse_rule_form(s, {**base, "archive_kind": "local", "archive_path": "relativ"})
|
||||||
|
assert "absoluten Pfad" in err["archive_path"]
|
||||||
|
d, err = parse_rule_form(
|
||||||
|
s,
|
||||||
|
{
|
||||||
|
**base,
|
||||||
|
"archive_kind": "mount",
|
||||||
|
"archive_mount_id": str(mid),
|
||||||
|
"archive_path": "archiv/scans",
|
||||||
|
"archive_layout": "date",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert err == {}
|
||||||
|
r, _ = save_rule(s, d)
|
||||||
|
assert r is not None and r.archive_kind is ArchiveKind.MOUNT
|
||||||
|
assert r.archive_path == "archiv/scans" and r.archive_layout.value == "date"
|
||||||
+119
-66
@@ -1,4 +1,4 @@
|
|||||||
"""0.5.0: Anmeldung abschaltbar – immer / nicht aus vertrauenswürdigen Netzen / nie."""
|
"""Anmeldung je Netz: ohne Passwort / mit Passwort / abgewiesen."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -7,15 +7,16 @@ from typing import Any
|
|||||||
import pytest
|
import pytest
|
||||||
from fastapi.testclient import TestClient
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
from netmover.settings_store import load_settings
|
from netmover.settings_store import load_settings, save_settings
|
||||||
from netmover.web.auth import includes_loopback, parse_networks
|
from netmover.web.auth import (
|
||||||
from tests.webutil import PASSWORD, csrf_of, post
|
ACCESS_DENY,
|
||||||
|
ACCESS_NOPW,
|
||||||
|
ACCESS_PW,
|
||||||
def _set_mode(client: TestClient, mode: str, nets: str = "", current: str = PASSWORD) -> Any:
|
access_decision,
|
||||||
return post(
|
includes_loopback,
|
||||||
client, "/settings/auth", {"auth_mode": mode, "trusted_networks": nets, "current": current}
|
parse_networks,
|
||||||
)
|
)
|
||||||
|
from tests.webutil import PASSWORD, csrf_of
|
||||||
|
|
||||||
|
|
||||||
def _other(client: TestClient, host: str) -> TestClient:
|
def _other(client: TestClient, host: str) -> TestClient:
|
||||||
@@ -23,70 +24,124 @@ def _other(client: TestClient, host: str) -> TestClient:
|
|||||||
return TestClient(client.app, follow_redirects=False, client=(host, 51000))
|
return TestClient(client.app, follow_redirects=False, client=(host, 51000))
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
def _save(c: TestClient, trusted: str = "", allowed: str = "", current: str = PASSWORD) -> Any:
|
||||||
def lan_client(client: TestClient) -> TestClient:
|
token = (
|
||||||
return _other(client, "10.0.10.50")
|
csrf_of(c.get("/settings").text)
|
||||||
|
if "X-Test-CSRF" not in c.headers
|
||||||
|
else c.headers["X-Test-CSRF"]
|
||||||
def test_default_requires_login(client: TestClient, lan_client: TestClient) -> None:
|
)
|
||||||
assert lan_client.get("/").status_code == 303
|
return c.post(
|
||||||
|
|
||||||
|
|
||||||
def test_mode_change_needs_current_password(client: TestClient, state: Any) -> None:
|
|
||||||
_set_mode(client, "off", current="falsch")
|
|
||||||
assert load_settings(state.session_factory).auth_mode == "always"
|
|
||||||
assert "Aktuelles Passwort falsch" in client.get("/settings").text
|
|
||||||
|
|
||||||
|
|
||||||
def test_trusted_networks(
|
|
||||||
client: TestClient, lan_client: TestClient, settings: Any, state: Any
|
|
||||||
) -> None:
|
|
||||||
_set_mode(client, "trusted", "10.0.10.0/24")
|
|
||||||
r = lan_client.get("/")
|
|
||||||
assert r.status_code == 200 and "ohne Anmeldung (vertrautes Netz)" in r.text
|
|
||||||
assert "Abmelden" not in r.text
|
|
||||||
assert lan_client.get("/login").headers["location"] == "/"
|
|
||||||
# CSRF gilt weiterhin
|
|
||||||
assert lan_client.post("/maintenance").status_code == 403
|
|
||||||
token = csrf_of(lan_client.get("/").text)
|
|
||||||
assert lan_client.post("/maintenance", data={"csrf_token": token}).status_code == 303
|
|
||||||
# andere Netze brauchen weiter das Passwort
|
|
||||||
assert _other(client, "192.168.5.5").get("/").status_code == 303
|
|
||||||
|
|
||||||
|
|
||||||
def test_off_and_back_on(client: TestClient, lan_client: TestClient) -> None:
|
|
||||||
_set_mode(client, "off")
|
|
||||||
r = lan_client.get("/rules")
|
|
||||||
assert r.status_code == 200 and "ohne Anmeldung" in r.text
|
|
||||||
token = csrf_of(r.text)
|
|
||||||
lan_client.post(
|
|
||||||
"/settings/auth",
|
"/settings/auth",
|
||||||
data={
|
data={
|
||||||
"csrf_token": token,
|
"csrf_token": token,
|
||||||
"auth_mode": "always",
|
"trusted_networks": trusted,
|
||||||
"trusted_networks": "",
|
"allowed_networks": allowed,
|
||||||
"current": PASSWORD,
|
"current": current,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
assert lan_client.get("/").status_code == 303
|
|
||||||
|
|
||||||
|
def _set(state: Any, **kw: str) -> None:
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
for k, v in kw.items():
|
||||||
|
setattr(cfg, k, v)
|
||||||
|
save_settings(state.session_factory, cfg)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Entscheidung ------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("host", "trusted", "allowed", "expected"),
|
||||||
|
[
|
||||||
|
("10.0.10.50", "", "", ACCESS_PW), # Standard: überall mit Passwort
|
||||||
|
("10.0.10.50", "10.0.10.0/24", "", ACCESS_NOPW),
|
||||||
|
("10.1.5.9", "10.0.10.0/24", "", ACCESS_PW), # übrige Netze mit Passwort
|
||||||
|
("10.1.5.9", "10.0.10.0/24", "10.1.5.0/24", ACCESS_PW),
|
||||||
|
("192.168.5.5", "10.0.10.0/24", "10.1.5.0/24", ACCESS_DENY),
|
||||||
|
("10.0.10.50", "10.0.10.0/24", "10.1.5.0/24", ACCESS_NOPW), # ohne PW schlägt Liste
|
||||||
|
("2001:db8::5", "0.0.0.0/0, ::/0", "", ACCESS_NOPW), # Anmeldung ganz aus
|
||||||
|
("testclient", "", "10.0.0.0/8", ACCESS_DENY), # keine IP-Adresse
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_access_decision(host: str, trusted: str, allowed: str, expected: str) -> None:
|
||||||
|
assert access_decision(host, "always", trusted, allowed) == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_off_mode_still_works() -> None:
|
||||||
|
assert access_decision("192.168.5.5", "off", "", "10.1.5.0/24") == ACCESS_NOPW
|
||||||
|
|
||||||
|
|
||||||
|
# -- Oberfläche --------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_requires_login(client: TestClient) -> None:
|
||||||
|
assert _other(client, "10.0.10.50").get("/").status_code == 303
|
||||||
|
|
||||||
|
|
||||||
|
def test_change_needs_current_password(client: TestClient, state: Any) -> None:
|
||||||
|
_save(client, trusted="10.0.10.0/24", current="falsch")
|
||||||
|
assert load_settings(state.session_factory).trusted_networks == ""
|
||||||
|
assert "Aktuelles Passwort falsch" in client.get("/settings").text
|
||||||
|
|
||||||
|
|
||||||
|
def test_without_password_from_trusted_network(client: TestClient, state: Any) -> None:
|
||||||
|
_save(client, trusted="10.0.10.0/24")
|
||||||
|
lan = _other(client, "10.0.10.50")
|
||||||
|
r = lan.get("/")
|
||||||
|
assert r.status_code == 200 and "ohne Anmeldung (vertrautes Netz)" in r.text
|
||||||
|
assert "Abmelden" not in r.text and lan.get("/login").headers["location"] == "/"
|
||||||
|
assert lan.post("/maintenance").status_code == 403 # CSRF gilt weiterhin
|
||||||
|
assert (
|
||||||
|
lan.post("/maintenance", data={"csrf_token": csrf_of(lan.get("/").text)}).status_code == 303
|
||||||
|
)
|
||||||
|
assert _other(client, "10.1.5.9").get("/").status_code == 303 # übrige mit Passwort
|
||||||
|
|
||||||
|
|
||||||
|
def test_other_networks_are_rejected_when_password_list_is_set(
|
||||||
|
client: TestClient, state: Any
|
||||||
|
) -> None:
|
||||||
|
_save(client, trusted="0.0.0.0/0") # damit der Test-Client (keine IP) weiterkommt …
|
||||||
|
_set(state, trusted_networks="10.0.10.0/24", allowed_networks="10.1.5.0/24") # … dann scharf
|
||||||
|
assert _other(client, "10.0.10.50").get("/").status_code == 200
|
||||||
|
assert _other(client, "10.1.5.9").get("/").headers["location"].startswith("/login")
|
||||||
|
blocked = _other(client, "192.168.5.5")
|
||||||
|
for url in ("/", "/login", "/setup", "/mounts"):
|
||||||
|
assert blocked.get(url).status_code == 403, url
|
||||||
|
assert blocked.get("/healthz").status_code == 200 # Überwachung bleibt erreichbar
|
||||||
|
assert blocked.get("/static/app.css").status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_cannot_lock_yourself_out(client: TestClient, state: Any) -> None:
|
||||||
|
lan = _other(client, "10.0.10.50")
|
||||||
|
_save(client, trusted="10.0.10.0/24")
|
||||||
|
# aus 10.0.10.50 eine Liste speichern, in der das eigene Netz fehlt
|
||||||
|
r = _save(lan, trusted="", allowed="10.1.5.0/24")
|
||||||
|
assert r.status_code == 303
|
||||||
|
assert "ausgesperrt" in lan.get("/settings").text
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
assert cfg.trusted_networks == "10.0.10.0/24" and cfg.allowed_networks == ""
|
||||||
|
# eigenes Netz bei „mit Passwort“ – erlaubt
|
||||||
|
_save(lan, trusted="10.0.10.0/24", allowed="10.1.5.0/24")
|
||||||
|
assert load_settings(state.session_factory).allowed_networks == "10.1.5.0/24"
|
||||||
|
|
||||||
|
|
||||||
|
def test_everything_without_password(client: TestClient) -> None:
|
||||||
|
_save(client, trusted="0.0.0.0/0, ::/0")
|
||||||
|
r = _other(client, "192.168.77.1").get("/rules")
|
||||||
|
assert r.status_code == 200 and ">ohne Anmeldung<" in r.text
|
||||||
|
|
||||||
|
|
||||||
def test_setup_still_required_without_password(raw_client: TestClient, state: Any) -> None:
|
def test_setup_still_required_without_password(raw_client: TestClient, state: Any) -> None:
|
||||||
from netmover.settings_store import save_settings
|
_set(state, trusted_networks="0.0.0.0/0")
|
||||||
|
|
||||||
cfg = load_settings(state.session_factory)
|
|
||||||
cfg.auth_mode = "off"
|
|
||||||
save_settings(state.session_factory, cfg)
|
|
||||||
assert raw_client.get("/").headers["location"] == "/setup"
|
assert raw_client.get("/").headers["location"] == "/setup"
|
||||||
|
|
||||||
|
|
||||||
def test_validation(client: TestClient, state: Any) -> None:
|
def test_validation_and_loopback_hint(client: TestClient, state: Any) -> None:
|
||||||
_set_mode(client, "trusted", "")
|
_save(client, trusted="10.0.10.0/24, quatsch")
|
||||||
assert "mindestens ein" in client.get("/settings").text
|
|
||||||
_set_mode(client, "trusted", "10.0.10.0/24, quatsch")
|
|
||||||
assert "kein gültiges Netz" in client.get("/settings").text
|
assert "kein gültiges Netz" in client.get("/settings").text
|
||||||
assert load_settings(state.session_factory).auth_mode == "always"
|
assert load_settings(state.session_factory).trusted_networks == ""
|
||||||
_set_mode(client, "trusted", "127.0.0.1")
|
_save(client, trusted="127.0.0.1")
|
||||||
assert "jede</strong> Anfrage über den Proxy" in client.get("/settings").text
|
assert "jede</strong> Anfrage über den Proxy" in client.get("/settings").text
|
||||||
|
|
||||||
|
|
||||||
@@ -102,13 +157,11 @@ def test_reset_password_turns_login_back_on(
|
|||||||
state: Any, settings: Any, monkeypatch: pytest.MonkeyPatch
|
state: Any, settings: Any, monkeypatch: pytest.MonkeyPatch
|
||||||
) -> None:
|
) -> None:
|
||||||
from netmover import cli
|
from netmover import cli
|
||||||
from netmover.settings_store import save_settings
|
|
||||||
from netmover.web.auth import set_password
|
from netmover.web.auth import set_password
|
||||||
|
|
||||||
set_password(state.session_factory, PASSWORD)
|
set_password(state.session_factory, PASSWORD)
|
||||||
cfg = load_settings(state.session_factory)
|
_set(state, auth_mode="off", trusted_networks="0.0.0.0/0", allowed_networks="10.1.5.0/24")
|
||||||
cfg.auth_mode = "off"
|
|
||||||
save_settings(state.session_factory, cfg)
|
|
||||||
monkeypatch.setattr(cli, "get_settings", lambda: settings)
|
monkeypatch.setattr(cli, "get_settings", lambda: settings)
|
||||||
assert cli.main(["reset-password"]) == 0
|
assert cli.main(["reset-password"]) == 0
|
||||||
assert load_settings(state.session_factory).auth_mode == "always"
|
cfg = load_settings(state.session_factory)
|
||||||
|
assert cfg.auth_mode == "always" and cfg.trusted_networks == "" and cfg.allowed_networks == ""
|
||||||
|
|||||||
@@ -0,0 +1,307 @@
|
|||||||
|
"""0.10.0: Datensicherung – verschlüsselt auf eine Anbindung, Wiederherstellen per CLI."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import sqlite3
|
||||||
|
import tarfile
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from netmover import __version__, backup
|
||||||
|
from netmover.app_state import NetMover
|
||||||
|
from netmover.backup import (
|
||||||
|
BackupError,
|
||||||
|
is_due,
|
||||||
|
load_state,
|
||||||
|
make_backup,
|
||||||
|
open_archive,
|
||||||
|
prune_names,
|
||||||
|
)
|
||||||
|
from netmover.cli import cmd_backup, cmd_restore
|
||||||
|
from netmover.config import Settings
|
||||||
|
from netmover.models import Mount, MountType
|
||||||
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
|
from netmover.settings_store import AppSettings, load_settings, save_settings
|
||||||
|
from tests.webutil import post
|
||||||
|
|
||||||
|
PASS = "korrekt-pferd-batterie"
|
||||||
|
NOW = datetime(2026, 10, 23, 12, 0, tzinfo=UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def _nas(state: NetMover, backend: FakeMountBackend) -> tuple[int, Path]:
|
||||||
|
"""CIFS-Anbindung „Backup“ einhängen; liefert (id, Freigabe auf dem „NAS“)."""
|
||||||
|
share = backend.export("nas", "backup")
|
||||||
|
with state.session_factory() as s:
|
||||||
|
m = Mount(
|
||||||
|
name="Backup",
|
||||||
|
slug="backup",
|
||||||
|
type=MountType.CIFS,
|
||||||
|
server="nas",
|
||||||
|
share="backup",
|
||||||
|
mountpoint=str(state.settings.mount_base.resolve() / "backup"),
|
||||||
|
)
|
||||||
|
s.add(m)
|
||||||
|
s.commit()
|
||||||
|
mid = m.id
|
||||||
|
state.mounts.mount(mid)
|
||||||
|
return mid, share
|
||||||
|
|
||||||
|
|
||||||
|
def _configure(state: NetMover, mount_id: int, **kw: Any) -> AppSettings:
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
cfg.backup_enabled = True
|
||||||
|
cfg.backup_mount_id = mount_id
|
||||||
|
cfg.backup_path = "netmover"
|
||||||
|
cfg.backup_passphrase_enc = state.secrets.encrypt(PASS)
|
||||||
|
for k, v in kw.items():
|
||||||
|
setattr(cfg, k, v)
|
||||||
|
save_settings(state.session_factory, cfg)
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
# -- Format ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_encrypted_roundtrip_and_wrong_passphrase(state: NetMover) -> None:
|
||||||
|
blob, manifest = make_backup(state.settings, PASS)
|
||||||
|
assert backup.is_encrypted(blob)
|
||||||
|
# Weder Schlüssel noch Datenbank im Klartext
|
||||||
|
key = state.settings.secret_key_path.read_bytes().strip()
|
||||||
|
assert key not in blob and b"SQLite format 3" not in blob
|
||||||
|
archive = open_archive(blob, PASS)
|
||||||
|
assert archive.manifest["version"] == __version__
|
||||||
|
assert archive.manifest["revision"]
|
||||||
|
assert archive.files["secret.key"].strip() == key
|
||||||
|
assert archive.files["netmover.db"].startswith(b"SQLite format 3")
|
||||||
|
assert manifest["files"] == sorted(archive.files)
|
||||||
|
with pytest.raises(BackupError, match="Passphrase falsch"):
|
||||||
|
open_archive(blob, "falsche-passphrase")
|
||||||
|
with pytest.raises(BackupError, match="verschlüsselt"):
|
||||||
|
open_archive(blob, None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_tampered_backup_is_rejected(state: NetMover) -> None:
|
||||||
|
blob = bytearray(make_backup(state.settings, PASS)[0])
|
||||||
|
blob[-10] ^= 0x01
|
||||||
|
with pytest.raises(BackupError):
|
||||||
|
open_archive(bytes(blob), PASS)
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_file_is_included(state: NetMover, tmp_path: Path) -> None:
|
||||||
|
env = tmp_path / "netmover.env"
|
||||||
|
env.write_text("NETMOVER_PORT=8080\n")
|
||||||
|
settings = state.settings.model_copy(update={"config_file": env})
|
||||||
|
archive = open_archive(make_backup(settings, None)[0], None)
|
||||||
|
assert archive.files["netmover.env"] == b"NETMOVER_PORT=8080\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_archive_without_secret_key_is_rejected() -> None:
|
||||||
|
buf = io.BytesIO()
|
||||||
|
with tarfile.open(fileobj=buf, mode="w:gz") as tar:
|
||||||
|
data = b'{"format": 1}'
|
||||||
|
info = tarfile.TarInfo("manifest.json")
|
||||||
|
info.size = len(data)
|
||||||
|
tar.addfile(info, io.BytesIO(data))
|
||||||
|
with pytest.raises(BackupError, match="unvollständig"):
|
||||||
|
open_archive(buf.getvalue(), None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_keeps_newest_and_ignores_foreign_files() -> None:
|
||||||
|
ours = [f"netmover-h-202610{d:02d}-033000.nmbak" for d in range(1, 8)]
|
||||||
|
names = [*ours, "urlaub.jpg", "netmover-notizen.txt"]
|
||||||
|
assert sorted(prune_names(names, 3)) == ours[:4]
|
||||||
|
assert prune_names(names, 10) == []
|
||||||
|
|
||||||
|
|
||||||
|
# -- Termin ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(time: str = "03:30") -> AppSettings:
|
||||||
|
return AppSettings(backup_enabled=True, backup_mount_id=1, backup_time=time)
|
||||||
|
|
||||||
|
|
||||||
|
def test_due_after_slot_once_per_day_and_hourly_retry() -> None:
|
||||||
|
local = datetime(2026, 10, 23, 2, 0).astimezone() # 02:00 Ortszeit
|
||||||
|
st = backup.BackupState()
|
||||||
|
yesterday_ok = backup.BackupState(last_success=local - timedelta(hours=3))
|
||||||
|
assert is_due(_cfg(), st, local) # noch nie gesichert → nachholen
|
||||||
|
assert not is_due(_cfg(), yesterday_ok, local) # 23:00 gestern nach 03:30 gestern
|
||||||
|
later = local + timedelta(hours=2) # 04:00 → heutiger Termin ist fällig
|
||||||
|
assert is_due(_cfg(), yesterday_ok, later)
|
||||||
|
failed = backup.BackupState(
|
||||||
|
last_success=yesterday_ok.last_success, last_attempt=later - timedelta(minutes=10)
|
||||||
|
)
|
||||||
|
assert not is_due(_cfg(), failed, later) # gerade gescheitert → nicht jeden Takt
|
||||||
|
assert is_due(_cfg(), failed, later + timedelta(hours=1))
|
||||||
|
assert not is_due(AppSettings(backup_enabled=False, backup_mount_id=1), st, later)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
backup.parse_time("25:00")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Dienst ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_service_writes_encrypted_backup_to_mount_and_prunes(
|
||||||
|
state: NetMover, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
mid, share = _nas(state, backend)
|
||||||
|
_configure(state, mid, backup_keep=3)
|
||||||
|
(share / "netmover").mkdir()
|
||||||
|
(share / "netmover" / "fremd.txt").write_text("bleibt")
|
||||||
|
for i in range(5):
|
||||||
|
result = state.backups.run(NOW + timedelta(days=i))
|
||||||
|
assert result.ok, result.message
|
||||||
|
files = sorted(p.name for p in (share / "netmover").iterdir())
|
||||||
|
assert "fremd.txt" in files
|
||||||
|
backups = [f for f in files if f.endswith(".nmbak")]
|
||||||
|
assert len(backups) == 3
|
||||||
|
assert not [f for f in files if f.endswith(".part")]
|
||||||
|
newest = share / "netmover" / backups[-1]
|
||||||
|
assert open_archive(newest.read_bytes(), PASS).manifest["version"] == __version__
|
||||||
|
st = load_state(state.session_factory)
|
||||||
|
assert st.last_success == NOW + timedelta(days=4)
|
||||||
|
assert st.last_file == backups[-1] and st.last_size == newest.stat().st_size
|
||||||
|
check = state.health().find("backup")
|
||||||
|
assert check is not None and check.ok
|
||||||
|
|
||||||
|
|
||||||
|
def test_failure_is_recorded_notified_once_and_becomes_problem(
|
||||||
|
settings: Settings, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
sent: list[tuple[str, str, str]] = []
|
||||||
|
state = NetMover(settings, backend=backend)
|
||||||
|
mid, _share = _nas(state, backend)
|
||||||
|
_configure(state, mid)
|
||||||
|
state.backups._notify = lambda *a: sent.append(a)
|
||||||
|
backend.vanish(state.mounts.mountpoint_for("backup"))
|
||||||
|
backend.fail_next_mount(times=10)
|
||||||
|
for i in range(3):
|
||||||
|
result = state.backups.run(NOW + timedelta(hours=i))
|
||||||
|
assert not result.ok
|
||||||
|
st = load_state(state.session_factory)
|
||||||
|
assert st.failures == 3 and "eingehängt" in st.last_error
|
||||||
|
assert [k for k, _t, _m in sent] == ["backup.failed"]
|
||||||
|
# Überfällig erst nach STALE_AFTER – dann rot in /healthz und Kuma
|
||||||
|
state.started_at = datetime.now(UTC) - backup.STALE_AFTER * 2
|
||||||
|
check = state.health().find("backup")
|
||||||
|
assert check is not None and not check.ok and "überfällig" in check.message
|
||||||
|
|
||||||
|
|
||||||
|
def test_tick_runs_only_when_due(state: NetMover, backend: FakeMountBackend) -> None:
|
||||||
|
mid, _ = _nas(state, backend)
|
||||||
|
_configure(state, mid)
|
||||||
|
assert state.backups.tick(NOW) is not None # noch nie gesichert
|
||||||
|
assert state.backups.tick(NOW + timedelta(minutes=5)) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_passphrase_refuses(state: NetMover, backend: FakeMountBackend) -> None:
|
||||||
|
mid, _ = _nas(state, backend)
|
||||||
|
_configure(state, mid, backup_passphrase_enc="")
|
||||||
|
result = state.backups.run(NOW)
|
||||||
|
assert not result.ok and "Passphrase" in result.message
|
||||||
|
|
||||||
|
|
||||||
|
# -- Wiederherstellen ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _mount_names(db: Path) -> list[str]:
|
||||||
|
con = sqlite3.connect(db)
|
||||||
|
try:
|
||||||
|
return [r[0] for r in con.execute("select name from mounts order by name")]
|
||||||
|
finally:
|
||||||
|
con.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_backup_and_restore_roundtrip(
|
||||||
|
state: NetMover, backend: FakeMountBackend, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
settings = state.settings
|
||||||
|
mid, _ = _nas(state, backend)
|
||||||
|
_configure(state, mid)
|
||||||
|
out = tmp_path / "out"
|
||||||
|
out.mkdir()
|
||||||
|
assert cmd_backup(settings, out, encrypted=True) == 0
|
||||||
|
(file,) = out.iterdir()
|
||||||
|
assert file.suffix == ".nmbak" and file.stat().st_mode & 0o077 == 0
|
||||||
|
old_key = settings.secret_key_path.read_bytes()
|
||||||
|
state.stop()
|
||||||
|
|
||||||
|
# Nach der Sicherung geht etwas schief: Anbindung gelöscht, Schlüssel weg
|
||||||
|
con = sqlite3.connect(settings.db_path)
|
||||||
|
con.execute("delete from mounts")
|
||||||
|
con.commit()
|
||||||
|
con.close()
|
||||||
|
settings.secret_key_path.write_bytes(b"kaputt")
|
||||||
|
phrase = tmp_path / "phrase"
|
||||||
|
phrase.write_text(PASS + "\n")
|
||||||
|
|
||||||
|
assert cmd_restore(settings, file, phrase, assume_yes=True, active=lambda: True) == 1
|
||||||
|
assert "Dienst läuft" in capsys.readouterr().out
|
||||||
|
assert cmd_restore(settings, file, phrase, assume_yes=True, active=lambda: False) == 0
|
||||||
|
|
||||||
|
assert _mount_names(settings.db_path) == ["Backup"]
|
||||||
|
assert settings.secret_key_path.read_bytes() == old_key
|
||||||
|
assert settings.secret_key_path.stat().st_mode & 0o077 == 0
|
||||||
|
(kept,) = settings.data_dir.glob("vor-wiederherstellung-*")
|
||||||
|
assert (kept / "secret.key").read_bytes() == b"kaputt"
|
||||||
|
assert _mount_names(kept / "netmover.db") == []
|
||||||
|
# Gespeicherte Zugangsdaten sind mit dem zurückgeholten Schlüssel wieder lesbar
|
||||||
|
restarted = NetMover(settings, backend=backend)
|
||||||
|
cfg = load_settings(restarted.session_factory)
|
||||||
|
assert restarted.backups.passphrase(cfg) == PASS
|
||||||
|
restarted.stop()
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_refuses_newer_version(state: NetMover, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
archive = open_archive(make_backup(state.settings, None)[0], None)
|
||||||
|
archive.manifest["version"] = "99.0.0"
|
||||||
|
with pytest.raises(BackupError, match="erst aktualisieren"):
|
||||||
|
backup.restore(state.settings, archive)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Oberfläche ------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_backup_page_save_run_and_health(
|
||||||
|
client: TestClient, state: NetMover, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
mid, share = _nas(state, backend)
|
||||||
|
page = client.get("/backup")
|
||||||
|
assert page.status_code == 200 and 'href="/backup"' in page.text
|
||||||
|
form = {
|
||||||
|
"backup_enabled": "on",
|
||||||
|
"backup_mount_id": str(mid),
|
||||||
|
"backup_path": "netmover",
|
||||||
|
"backup_time": "03:30",
|
||||||
|
"backup_keep": "14",
|
||||||
|
}
|
||||||
|
r = post(client, "/backup", form)
|
||||||
|
assert r.status_code == 400 and "Passphrase festlegen" in r.text
|
||||||
|
r = post(client, "/backup", {**form, "passphrase": PASS, "passphrase_confirm": "anders"})
|
||||||
|
assert r.status_code == 400 and "stimmen nicht" in r.text
|
||||||
|
r = post(client, "/backup", {**form, "backup_path": "../raus", "passphrase": PASS})
|
||||||
|
assert r.status_code == 400
|
||||||
|
r = post(client, "/backup", {**form, "passphrase": PASS, "passphrase_confirm": PASS})
|
||||||
|
assert r.status_code == 303
|
||||||
|
with state.session_factory() as s:
|
||||||
|
from netmover.models import Setting
|
||||||
|
|
||||||
|
raw = " ".join(v for (v,) in s.execute(select(Setting.value)))
|
||||||
|
assert PASS not in raw # Passphrase nur verschlüsselt gespeichert
|
||||||
|
# Passphrase leer lassen = unverändert
|
||||||
|
assert post(client, "/backup", {**form, "backup_keep": "7"}).status_code == 303
|
||||||
|
assert state.backups.passphrase(load_settings(state.session_factory)) == PASS
|
||||||
|
|
||||||
|
assert post(client, "/backup/run").status_code == 303
|
||||||
|
assert "geschrieben" in client.get("/backup").text
|
||||||
|
assert len(list((share / "netmover").glob("*.nmbak"))) == 1
|
||||||
|
r = client.get("/healthz/backup")
|
||||||
|
assert r.status_code == 200 and r.json()["status"] == "ok"
|
||||||
|
assert "/healthz/backup" in client.get("/settings").text
|
||||||
|
assert any(c["kind"] == "backup" for c in client.get("/healthz").json()["checks"])
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
"""0.8.0: Anbindungen kopieren – inklusive Zugangsdaten."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from netmover.admin import MountInput, copy_name, delete_mount, save_mount
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
|
from netmover.models import Credential, CredentialKind, Mount, MountType
|
||||||
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
|
from netmover.mounting.manager import MountManager
|
||||||
|
from tests.webutil import post
|
||||||
|
|
||||||
|
|
||||||
|
def _cifs(**kw: object) -> MountInput:
|
||||||
|
data: dict[str, object] = {
|
||||||
|
"name": "NAS",
|
||||||
|
"type": MountType.CIFS,
|
||||||
|
"server": "10.0.10.6",
|
||||||
|
"share": "Scans",
|
||||||
|
"username": "scanner",
|
||||||
|
"password": "Pa$$wort",
|
||||||
|
}
|
||||||
|
data.update(kw)
|
||||||
|
return MountInput(**data) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_reuses_own_password_as_independent_copy(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
src, _ = save_mount(session, secretbox, manager, _cifs())
|
||||||
|
assert src is not None and src.credential is not None
|
||||||
|
copy, err = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_cifs(name="Archiv", share="Archiv", password=""),
|
||||||
|
copy_from=src,
|
||||||
|
)
|
||||||
|
assert not err and copy is not None and copy.credential is not None
|
||||||
|
assert copy.credential.id != src.credential.id and not copy.credential.managed
|
||||||
|
assert secretbox.decrypt(copy.credential.secret_enc) == "Pa$$wort"
|
||||||
|
assert copy.credential.username == "scanner"
|
||||||
|
# Vorlage löschen – die Kopie behält ihre Zugangsdaten
|
||||||
|
delete_mount(session, manager, src.id)
|
||||||
|
session.flush()
|
||||||
|
assert secretbox.decrypt(copy.credential.secret_enc) == "Pa$$wort"
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_with_new_password_uses_new_one(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
src, _ = save_mount(session, secretbox, manager, _cifs())
|
||||||
|
copy, _ = save_mount(
|
||||||
|
session, secretbox, manager, _cifs(name="B", share="B", password="anders"), copy_from=src
|
||||||
|
)
|
||||||
|
assert copy is not None and copy.credential is not None
|
||||||
|
assert secretbox.decrypt(copy.credential.secret_enc) == "anders"
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_with_stored_credentials_shares_them(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
from netmover.credentials import create_password
|
||||||
|
|
||||||
|
cred = create_password(session, secretbox, "NAS-Scanner", "scanner", "x")
|
||||||
|
src, _ = save_mount(
|
||||||
|
session, secretbox, manager, _cifs(username="", password="", cred_id=str(cred.id))
|
||||||
|
)
|
||||||
|
copy, _ = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_cifs(name="B", share="B", username="", password="", cred_id=str(cred.id)),
|
||||||
|
copy_from=src,
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
copy is not None and copy.credential is cred and src is not None and src.credential is cred
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_sftp_keeps_keypair_and_host_key_only_for_same_server(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
sftp = {
|
||||||
|
"type": MountType.SFTP,
|
||||||
|
"server": "10.0.10.17",
|
||||||
|
"share": "/consume",
|
||||||
|
"port": 22,
|
||||||
|
"username": "paperless",
|
||||||
|
"auth": CredentialKind.SSH_KEY,
|
||||||
|
"password": "",
|
||||||
|
}
|
||||||
|
src, _ = save_mount(session, secretbox, manager, _cifs(name="Paperless", **sftp))
|
||||||
|
assert src is not None and src.credential is not None
|
||||||
|
src.host_key = "ssh-ed25519 AAAAhost"
|
||||||
|
same, _ = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_cifs(name="Paperless 2", **{**sftp, "share": "/consume2"}),
|
||||||
|
copy_from=src,
|
||||||
|
)
|
||||||
|
other, _ = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_cifs(name="Anderer", **{**sftp, "server": "10.0.10.99"}),
|
||||||
|
copy_from=src,
|
||||||
|
)
|
||||||
|
assert same is not None and other is not None
|
||||||
|
for m in (same, other):
|
||||||
|
assert m.credential is not None and m.credential.id != src.credential.id
|
||||||
|
assert m.credential.public_key == src.credential.public_key # gleiches Schlüsselpaar
|
||||||
|
assert m.credential.secret_enc == src.credential.secret_enc
|
||||||
|
assert same.host_key == "ssh-ed25519 AAAAhost" and other.host_key is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_name(session: Session, secretbox: SecretBox, manager: MountManager) -> None:
|
||||||
|
save_mount(session, secretbox, manager, _cifs())
|
||||||
|
assert copy_name(session, "NAS") == "NAS (Kopie)"
|
||||||
|
save_mount(session, secretbox, manager, _cifs(name="NAS (Kopie)", share="x"))
|
||||||
|
assert copy_name(session, "NAS") == "NAS (Kopie 2)"
|
||||||
|
|
||||||
|
|
||||||
|
# -- Oberfläche --------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
MOUNT = {
|
||||||
|
"name": "NAS",
|
||||||
|
"type": "cifs",
|
||||||
|
"server": "nas",
|
||||||
|
"share": "scans",
|
||||||
|
"username": "scanner",
|
||||||
|
"password": "Geheim,123",
|
||||||
|
"auto_mount": "on",
|
||||||
|
"enabled": "on",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _mid(state: Any, name: str) -> int:
|
||||||
|
with state.session_factory() as s:
|
||||||
|
mid = s.scalar(select(Mount.id).where(Mount.name == name))
|
||||||
|
assert mid is not None
|
||||||
|
return mid
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_button_form_test_and_save(
|
||||||
|
client: TestClient, state: Any, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
backend.export("nas", "scans")
|
||||||
|
backend.export("nas", "archiv")
|
||||||
|
post(client, "/mounts/new", MOUNT)
|
||||||
|
src_id = _mid(state, "NAS")
|
||||||
|
assert f'href="/mounts/{src_id}/copy"' in client.get("/mounts").text
|
||||||
|
|
||||||
|
form = client.get(f"/mounts/{src_id}/copy").text
|
||||||
|
assert "Kopie von „NAS“" in form and 'value="NAS (Kopie)"' in form
|
||||||
|
assert 'value="scanner"' in form and "Geheim,123" not in form
|
||||||
|
assert f'name="copy_from" value="{src_id}"' in form
|
||||||
|
|
||||||
|
# Testen mit übernommenem Passwort (Feld leer)
|
||||||
|
copy_form = {
|
||||||
|
**MOUNT,
|
||||||
|
"name": "NAS (Kopie)",
|
||||||
|
"share": "archiv",
|
||||||
|
"password": "",
|
||||||
|
"copy_from": str(src_id),
|
||||||
|
}
|
||||||
|
r = post(client, "/mounts/test", copy_form)
|
||||||
|
assert "Einhängen erfolgreich" in r.text
|
||||||
|
spec = [c for c in backend.calls if c.action == "mount"][-1].spec
|
||||||
|
assert spec is not None and spec.credentials is not None
|
||||||
|
assert spec.credentials.password == "Geheim,123"
|
||||||
|
|
||||||
|
r = post(client, f"/mounts/{src_id}/copy", copy_form)
|
||||||
|
assert r.status_code == 303, r.text
|
||||||
|
with state.session_factory() as s:
|
||||||
|
copy = s.get(Mount, _mid(state, "NAS (Kopie)"))
|
||||||
|
assert copy is not None and copy.share == "archiv" and copy.slug == "nas-kopie"
|
||||||
|
assert copy.credential is not None
|
||||||
|
assert state.secrets.decrypt(copy.credential.secret_enc) == "Geheim,123"
|
||||||
|
assert len(s.scalars(select(Credential)).all()) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_validation_error_keeps_copy_context(client: TestClient, state: Any) -> None:
|
||||||
|
post(client, "/mounts/new", MOUNT)
|
||||||
|
src_id = _mid(state, "NAS")
|
||||||
|
r = post(client, f"/mounts/{src_id}/copy", {**MOUNT, "name": "NAS", "password": ""})
|
||||||
|
assert r.status_code == 400 and "bereits vergeben" in r.text and "Kopie von „NAS“" in r.text
|
||||||
@@ -0,0 +1,168 @@
|
|||||||
|
"""0.9.0: Regeln kopieren – vorausgefüllt, zunächst deaktiviert."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.models import (
|
||||||
|
ArchiveKind,
|
||||||
|
ArchiveLayout,
|
||||||
|
CopiedFile,
|
||||||
|
Mount,
|
||||||
|
MountType,
|
||||||
|
Rule,
|
||||||
|
RuleAction,
|
||||||
|
Run,
|
||||||
|
)
|
||||||
|
from tests.webutil import post
|
||||||
|
|
||||||
|
|
||||||
|
def _setup(state: Any, tmp_path: Path) -> tuple[int, int]:
|
||||||
|
consume = tmp_path / "consume"
|
||||||
|
consume.mkdir()
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
m = Mount(
|
||||||
|
name="NAS",
|
||||||
|
slug="nas",
|
||||||
|
type=MountType.CIFS,
|
||||||
|
server="nas",
|
||||||
|
share="scans",
|
||||||
|
mountpoint=str(tmp_path / "mnt" / "nas"),
|
||||||
|
)
|
||||||
|
a = Mount(
|
||||||
|
name="Archiv",
|
||||||
|
slug="archiv",
|
||||||
|
type=MountType.CIFS,
|
||||||
|
server="nas",
|
||||||
|
share="archiv",
|
||||||
|
mountpoint=str(tmp_path / "mnt" / "archiv"),
|
||||||
|
)
|
||||||
|
s.add_all([m, a])
|
||||||
|
s.flush()
|
||||||
|
r = Rule(
|
||||||
|
name="Scans → Paperless",
|
||||||
|
src_mount_id=m.id,
|
||||||
|
src_subpath="max",
|
||||||
|
dst_path=str(consume),
|
||||||
|
pattern="*.pdf;*.jpg",
|
||||||
|
recursive=True,
|
||||||
|
interval_seconds=45,
|
||||||
|
stability_seconds=7,
|
||||||
|
action=RuleAction.COPY,
|
||||||
|
set_mode="0640",
|
||||||
|
quarantine_path="_q",
|
||||||
|
archive_kind=ArchiveKind.MOUNT,
|
||||||
|
archive_mount_id=a.id,
|
||||||
|
archive_path="scans",
|
||||||
|
archive_layout=ArchiveLayout.DATE,
|
||||||
|
)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
s.add(Run(rule_id=r.id))
|
||||||
|
s.add(CopiedFile(rule_id=r.id, src_relpath="a.pdf", size=1, mtime_ns=1, dst_path="x"))
|
||||||
|
return r.id, m.id
|
||||||
|
|
||||||
|
|
||||||
|
def _rule(state: Any, name: str) -> Rule:
|
||||||
|
with state.session_factory() as s:
|
||||||
|
r = s.scalars(select(Rule).where(Rule.name == name)).one()
|
||||||
|
s.expunge(r)
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
def _form_from_page(html: str) -> dict[str, str]:
|
||||||
|
"""Wie ein Browser: alle Felder des Formulars mit ihren vorausgefüllten Werten."""
|
||||||
|
import re
|
||||||
|
|
||||||
|
form: dict[str, str] = {}
|
||||||
|
for m in re.finditer(r'<input[^>]*name="([^"]+)"[^>]*>', html):
|
||||||
|
tag, name = m.group(0), m.group(1)
|
||||||
|
if 'type="checkbox"' in tag:
|
||||||
|
if " checked" in tag:
|
||||||
|
form[name] = "on"
|
||||||
|
else:
|
||||||
|
v = re.search(r'value="([^"]*)"', tag)
|
||||||
|
form[name] = v.group(1) if v else ""
|
||||||
|
for m in re.finditer(r'<select[^>]*name="([^"]+)"[^>]*>(.*?)</select>', html, re.S):
|
||||||
|
sel = re.search(r'<option value="([^"]*)" selected>', m.group(2))
|
||||||
|
form[m.group(1)] = sel.group(1) if sel else ""
|
||||||
|
return form
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_form_is_prefilled_and_disabled(
|
||||||
|
client: TestClient, state: Any, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
rid, _ = _setup(state, tmp_path)
|
||||||
|
assert f'href="/rules/{rid}/copy"' in client.get("/rules").text
|
||||||
|
html = client.get(f"/rules/{rid}/copy").text
|
||||||
|
assert "Kopie von „Scans → Paperless“" in html and "deaktiviert" in html
|
||||||
|
form = _form_from_page(html)
|
||||||
|
assert form["name"] == "Scans → Paperless (Kopie)"
|
||||||
|
assert "enabled" not in form # Häkchen „Aktiv“ nicht gesetzt
|
||||||
|
assert form["pattern"] == "*.pdf;*.jpg" and form["src_subpath"] == "max"
|
||||||
|
assert form["archive_kind"] == "mount" and form["archive_layout"] == "date"
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_saves_new_rule_with_all_settings(
|
||||||
|
client: TestClient, state: Any, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
rid, _ = _setup(state, tmp_path)
|
||||||
|
form = _form_from_page(client.get(f"/rules/{rid}/copy").text)
|
||||||
|
form["src_subpath"] = "erika" # typische Anpassung
|
||||||
|
r = post(client, f"/rules/{rid}/copy", form)
|
||||||
|
assert r.status_code == 303, r.text
|
||||||
|
assert "Die Kopie ist deaktiviert" in client.get("/rules").text
|
||||||
|
|
||||||
|
orig, copy = _rule(state, "Scans → Paperless"), _rule(state, "Scans → Paperless (Kopie)")
|
||||||
|
assert copy.id != orig.id and copy.enabled is False and orig.enabled is True
|
||||||
|
assert copy.src_subpath == "erika" and orig.src_subpath == "max"
|
||||||
|
for f in (
|
||||||
|
"src_mount_id",
|
||||||
|
"dst_path",
|
||||||
|
"pattern",
|
||||||
|
"recursive",
|
||||||
|
"interval_seconds",
|
||||||
|
"stability_seconds",
|
||||||
|
"action",
|
||||||
|
"set_mode",
|
||||||
|
"quarantine_path",
|
||||||
|
"archive_kind",
|
||||||
|
"archive_mount_id",
|
||||||
|
"archive_path",
|
||||||
|
"archive_layout",
|
||||||
|
):
|
||||||
|
assert getattr(copy, f) == getattr(orig, f), f
|
||||||
|
# Protokoll und Buchführung beginnen neu
|
||||||
|
with state.session_factory() as s:
|
||||||
|
assert s.scalars(select(Run).where(Run.rule_id == copy.id)).all() == []
|
||||||
|
assert s.scalars(select(CopiedFile).where(CopiedFile.rule_id == copy.id)).all() == []
|
||||||
|
assert state.scheduler.next_run(copy.id) is None # deaktiviert → kein Termin
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_names_count_up(client: TestClient, state: Any, tmp_path: Path) -> None:
|
||||||
|
rid, _ = _setup(state, tmp_path)
|
||||||
|
for expected in ("Scans → Paperless (Kopie)", "Scans → Paperless (Kopie 2)"):
|
||||||
|
form = _form_from_page(client.get(f"/rules/{rid}/copy").text)
|
||||||
|
assert form["name"] == expected
|
||||||
|
post(client, f"/rules/{rid}/copy", form)
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_validation_error_keeps_context(
|
||||||
|
client: TestClient, state: Any, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
rid, _ = _setup(state, tmp_path)
|
||||||
|
form = _form_from_page(client.get(f"/rules/{rid}/copy").text)
|
||||||
|
form["name"] = "Scans → Paperless" # schon vergeben
|
||||||
|
r = post(client, f"/rules/{rid}/copy", form)
|
||||||
|
assert r.status_code == 400 and "bereits vergeben" in r.text and "Kopie von" in r.text
|
||||||
|
with state.session_factory() as s:
|
||||||
|
assert len(s.scalars(select(Rule)).all()) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_of_missing_rule_redirects(client: TestClient) -> None:
|
||||||
|
assert client.get("/rules/4711/copy").headers["location"] == "/rules"
|
||||||
@@ -175,3 +175,47 @@ def test_upgrade_from_0002_keeps_everything_linked_to_rules(tmp_path: Path) -> N
|
|||||||
assert c.execute("SELECT waiting_since FROM rules").fetchone()[0] is None
|
assert c.execute("SELECT waiting_since FROM rules").fetchone()[0] is None
|
||||||
assert c.execute("SELECT username FROM mounts WHERE id = 2").fetchone()[0] == "paperless"
|
assert c.execute("SELECT username FROM mounts WHERE id = 2").fetchone()[0] == "paperless"
|
||||||
c.close()
|
c.close()
|
||||||
|
|
||||||
|
|
||||||
|
# -- Pflicht für jede (auch künftige) Migration -------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _revisions() -> list[str]:
|
||||||
|
from alembic.script import ScriptDirectory
|
||||||
|
|
||||||
|
script = ScriptDirectory.from_config(alembic_config("sqlite://"))
|
||||||
|
return [r.revision for r in reversed(list(script.walk_revisions()))]
|
||||||
|
|
||||||
|
|
||||||
|
def _invariants(db: Path) -> tuple[object, ...]:
|
||||||
|
st = _state(db)
|
||||||
|
assert st["tmp_tables"] == [], "Reste einer Batch-Migration"
|
||||||
|
assert st["fk_violations"] == [], "Fremdschlüssel verletzt"
|
||||||
|
return st["mounts"], st["rules"], st["run_items"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("rev", _revisions()[1:])
|
||||||
|
def test_every_migration_up_and_down_on_populated_db(tmp_path: Path, rev: str) -> None:
|
||||||
|
"""Jede Migration einzeln: hoch und wieder runter auf einer Datenbank mit Daten –
|
||||||
|
ohne Fremdschlüsselfehler, ohne Reste, ohne Datenverlust. Neue Migrationen werden
|
||||||
|
hier automatisch erfasst."""
|
||||||
|
revs = _revisions()
|
||||||
|
prev = revs[revs.index(rev) - 1]
|
||||||
|
db = tmp_path / "pflicht.db"
|
||||||
|
url = f"sqlite:///{db}"
|
||||||
|
upgrade(url, revs[0])
|
||||||
|
_seed(db)
|
||||||
|
upgrade(url, prev)
|
||||||
|
before = _invariants(db)
|
||||||
|
upgrade(url, rev)
|
||||||
|
assert _state(db)["version"] == rev
|
||||||
|
assert _invariants(db) == before
|
||||||
|
command.downgrade(alembic_config(url), prev)
|
||||||
|
assert _state(db)["version"] == prev
|
||||||
|
assert _invariants(db) == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_migration_files_are_named_after_their_revision() -> None:
|
||||||
|
versions = Path(__file__).parent.parent / "src/netmover/migrations/versions"
|
||||||
|
names = sorted(p.name.split("_", 1)[0] for p in versions.glob("[0-9]*.py"))
|
||||||
|
assert names == _revisions()
|
||||||
|
|||||||
@@ -216,3 +216,47 @@ def test_navigation_groups_health_and_help_links(client: TestClient, state: Any)
|
|||||||
def test_logged_out_pages_have_no_menu(raw_client: TestClient) -> None:
|
def test_logged_out_pages_have_no_menu(raw_client: TestClient) -> None:
|
||||||
html = raw_client.get("/setup").text
|
html = raw_client.get("/setup").text
|
||||||
assert 'id="sidebar"' not in html
|
assert 'id="sidebar"' not in html
|
||||||
|
|
||||||
|
|
||||||
|
# -- Kennzahlen (0.10.0) ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_metrics_in_healthz_and_rule_endpoint(client: TestClient, state: Any) -> None:
|
||||||
|
from netmover.models import FileFailure, ItemStatus, Run, RunItem, RunStatus
|
||||||
|
|
||||||
|
mid = _add_mount(state, "NAS", "nas", MountState.MOUNTED)
|
||||||
|
now = utcnow()
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
a = Rule(name="A", src_mount_id=mid, dst_path="/tmp", last_success_at=now)
|
||||||
|
b = Rule(name="B", src_mount_id=mid, dst_path="/tmp", last_success_at=now)
|
||||||
|
s.add_all([a, b])
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
def run(rule: Rule, ago: timedelta, **kw: Any) -> Run:
|
||||||
|
r = Run(rule_id=rule.id, started_at=now - ago, finished_at=now - ago, **kw)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
return r
|
||||||
|
|
||||||
|
run(a, timedelta(hours=1), status=RunStatus.OK, moved=3)
|
||||||
|
q = run(a, timedelta(hours=2), status=RunStatus.PARTIAL, moved=1, failed=2)
|
||||||
|
s.add(RunItem(run_id=q.id, src="x.pdf", status=ItemStatus.QUARANTINED))
|
||||||
|
run(a, timedelta(hours=3), status=RunStatus.OK, moved=50, dry_run=True) # Probelauf
|
||||||
|
run(a, timedelta(hours=30), status=RunStatus.OK, moved=7) # älter als 24 h
|
||||||
|
run(b, timedelta(hours=4), status=RunStatus.ERROR)
|
||||||
|
s.add(FileFailure(rule_id=b.id, src_relpath="y.pdf", size=1, mtime_ns=1, attempts=1))
|
||||||
|
aid, bid = a.id, b.id
|
||||||
|
|
||||||
|
m = client.get("/healthz").json()["metrics"]
|
||||||
|
assert (m["files_24h"], m["failed_24h"], m["quarantined_24h"]) == (4, 2, 1)
|
||||||
|
assert m["errors_24h"] == 1 and m["pending_failures"] == 1
|
||||||
|
assert m["last_moved_at"].startswith(str((now - timedelta(hours=1)).date()))
|
||||||
|
ra = client.get(f"/healthz/rule/{aid}").json()["metrics"]
|
||||||
|
assert ra["files_24h"] == 4 and ra["quarantined_24h"] == 1
|
||||||
|
rb = client.get(f"/healthz/rule/{bid}").json()["metrics"]
|
||||||
|
assert rb["files_24h"] == 0 and rb["errors_24h"] == 1 and rb["last_moved_at"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_ok_message_shows_files() -> None:
|
||||||
|
url = push_url("https://k/api/push/x", _health(True), {"files_24h": 12, "failed_24h": 1})
|
||||||
|
assert httpx.URL(url).params["msg"] == "OK · 24 h: 12 Dateien, 1 Fehler"
|
||||||
|
|||||||
Reference in New Issue
Block a user