Compare commits
@@ -0,0 +1,37 @@
|
|||||||
|
# Prüfungen bei jedem Push und Pull Request (Gitea Actions, act_runner).
|
||||||
|
# Gleiche Befehle wie lokal vor einem Release – siehe README, „Entwicklung“.
|
||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
tags: ["v*"]
|
||||||
|
pull_request:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
pruefen:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
strategy:
|
||||||
|
fail-fast: false
|
||||||
|
matrix:
|
||||||
|
# 3.12 = Mindestversion, 3.13 = Debian 13 (Zielsystem)
|
||||||
|
python: ["3.12", "3.13"]
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: ${{ matrix.python }}
|
||||||
|
- name: Abhängigkeiten
|
||||||
|
run: |
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
- name: ruff
|
||||||
|
run: |
|
||||||
|
ruff check .
|
||||||
|
ruff format --check .
|
||||||
|
- name: mypy
|
||||||
|
run: mypy src
|
||||||
|
- name: shellcheck
|
||||||
|
run: shellcheck deploy/*.sh
|
||||||
|
- name: pytest (inkl. Migrationen auf befüllter Datenbank)
|
||||||
|
run: pytest -q
|
||||||
+215
-1
@@ -8,6 +8,209 @@ Die Version steht an genau einer Stelle: `src/netmover/__init__.py`.
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.11.0] – 2026-10-24
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Mehrere Ziele je Regel**: neben dem Hauptziel bis zu 9 weitere Ziele (Anbindung
|
||||||
|
oder lokaler Pfad, je mit Bezeichnung, Ordner und Ablage flach/nach Datum) – z. B.
|
||||||
|
Rechnung ins DMS, in die Buchhaltung und ins Archiv. Weitere Ziele werden vor dem
|
||||||
|
Hauptziel beliefert, jedes mit eigener Buchführung; die Quelle wird erst gelöscht,
|
||||||
|
wenn alle Ziele die Datei haben.
|
||||||
|
- **Ein nicht erreichbares Ziel hält die anderen nicht auf**: Die erreichbaren Ziele
|
||||||
|
(auch das Hauptziel) werden beliefert, die Quelle bleibt liegen, bis das fehlende
|
||||||
|
Ziel zurück ist und nachbeliefert wurde. Die Regel gilt solange als wartend;
|
||||||
|
Wiederholungen während des Ausfalls erzeugen keine Protokolleinträge.
|
||||||
|
- Probelauf zeigt für jedes Ziel den künftigen Pfad.
|
||||||
|
- Im Kopiermodus bekommt ein neu hinzugefügtes Ziel auch die schon vorhandenen Dateien.
|
||||||
|
- Anleitung: Beispiel „Rechnungseingang: DMS, Buchhaltung und Archiv“.
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- Das Archiv ist jetzt ein weiteres Ziel. Migration 0005 übernimmt vorhandene Archive
|
||||||
|
als Ziel „Archiv“ samt Buchführung (`rule_targets`, `target_files` statt
|
||||||
|
`rules.archive_*`, `archived_files`). Downgrade: nur das erste weitere Ziel bleibt.
|
||||||
|
- Export enthält die Ziele (`targets`); Exporte älterer Versionen mit Archiv werden
|
||||||
|
beim Import umgewandelt.
|
||||||
|
- Regelformular: „Ziel“ heißt jetzt „Hauptziel“, „Archiv“ wird zu „Weitere Ziele“.
|
||||||
|
|
||||||
|
## [0.10.0] – 2026-10-23
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Tägliche Datensicherung** (*System → Sicherung*): Datenbank, `secret.key`,
|
||||||
|
`session.key` und `netmover.env` als mit Passphrase verschlüsselte Datei
|
||||||
|
(scrypt + Fernet) auf eine Anbindung (SMB, NFS oder SFTP). Momentaufnahme über die
|
||||||
|
SQLite-Backup-API mit Integritätsprüfung; die Datei wird vor dem Hochladen
|
||||||
|
probeweise geöffnet und atomar geschrieben; ältere NetMover-Sicherungen über die
|
||||||
|
eingestellte Anzahl hinaus werden gelöscht, andere Dateien nie. Verpasste Termine
|
||||||
|
werden nachgeholt, nach einem Fehler stündlich neuer Versuch, erster Fehlschlag
|
||||||
|
per E-Mail/Webhook. Knopf „Jetzt sichern“.
|
||||||
|
- `netmover restore <datei>`: spielt eine Sicherung ein (nur bei gestopptem Dienst),
|
||||||
|
legt die bisherigen Dateien vorher nach `vor-wiederherstellung-<zeit>/` und
|
||||||
|
migriert die Datenbank. `netmover backup <ziel>` sichert von Hand in eine Datei.
|
||||||
|
- Befehl `netmover` im Pfad (`/usr/local/sbin/netmover`, lädt `netmover.env`).
|
||||||
|
- **Kennzahlen für Uptime Kuma**: `/healthz` und `/healthz/rule/<id>` liefern unter
|
||||||
|
`metrics` Dateien, Fehler, Quarantäne und abgebrochene Läufe der letzten 24 Stunden
|
||||||
|
sowie die letzte verschobene Datei – für Kuma-Monitore vom Typ „JSON-Abfrage“.
|
||||||
|
Der Push meldet z. B. „OK · 24 h: 12 Dateien“.
|
||||||
|
- `/healthz/backup`: eigener Monitor; eine seit zwei Tagen überfällige Sicherung gilt
|
||||||
|
als Problem (Statusampel, `/healthz`, Push).
|
||||||
|
- **CI in Gitea** (`.gitea/workflows/ci.yml`): ruff, mypy, shellcheck und Tests auf
|
||||||
|
Python 3.12 und 3.13 bei jedem Push.
|
||||||
|
- Test-Pflicht für Migrationen: jede Migration wird automatisch auf einer befüllten
|
||||||
|
Datenbank hoch- und heruntergestuft.
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- README: HTTP im internen Netz als bewusste Entscheidung dokumentiert; TLS über
|
||||||
|
Reverse Proxy bleibt der Weg für nicht vertrauenswürdige Netze.
|
||||||
|
|
||||||
|
## [0.9.0] – 2026-10-15
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Regeln kopieren** (Knopf „Kopieren“): alle Einstellungen inklusive Archiv
|
||||||
|
übernommen, Name „… (Kopie)“. Die Kopie ist zunächst **deaktiviert**, damit sie nicht
|
||||||
|
mit dem Original um dieselben Dateien konkurriert; Protokoll und Buchführung
|
||||||
|
beginnen neu.
|
||||||
|
|
||||||
|
## [0.8.0] – 2026-10-12
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Anbindungen kopieren** (Knopf „Kopieren“): Formular vorausgefüllt, Name
|
||||||
|
„… (Kopie)“. Eigene Zugangsdaten der Vorlage werden übernommen (eigenes Exemplar),
|
||||||
|
solange kein neues Passwort eingegeben wird; ein eigener SSH-Schlüssel wird
|
||||||
|
weiterverwendet; gespeicherte Zugangsdaten bleiben ausgewählt. „Testen“ im
|
||||||
|
Kopie-Formular nutzt die übernommenen Zugangsdaten. Der SFTP-Hostschlüssel wird nur
|
||||||
|
bei gleichem Server und Port übernommen.
|
||||||
|
|
||||||
|
## [0.7.0] – 2026-10-12
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Archiv je Regel:** zusätzliche Kopie in ein Archiv (Anbindung SMB/NFS/SFTP oder
|
||||||
|
lokaler Pfad), Ablage „wie in der Quelle“ oder „nach Datum“ (`Jahr/Monat/…`).
|
||||||
|
Reihenfolge: erst Archiv, dann Ziel, **erst dann wird die Quelle gelöscht**.
|
||||||
|
Scheitert das Ziel nach dem Archiv, verhindert eine Buchführung
|
||||||
|
(`archived_files`) eine zweite Archivkopie. Im Archiv wird nie überschrieben.
|
||||||
|
Nicht erreichbares Archiv → Regel wartet. Platzprüfung auch fürs Archiv,
|
||||||
|
Probelauf zeigt das Archivziel. Migration 0004.
|
||||||
|
- Export/Import enthalten die Archiv-Einstellungen; eine als Archiv benutzte
|
||||||
|
Anbindung lässt sich nicht löschen.
|
||||||
|
|
||||||
|
## [0.6.0] – 2026-10-11
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- **Anmeldung je Netz konfigurierbar** (*Einstellungen → Anmeldung*): „Ohne Passwort
|
||||||
|
aus diesen Netzen“ und „Mit Passwort aus diesen Netzen“ (leer = alle übrigen).
|
||||||
|
Ist die zweite Liste ausgefüllt, werden alle anderen Netze abgewiesen (403, auch die
|
||||||
|
Anmeldeseite); `/healthz` und statische Dateien bleiben erreichbar. Ersetzt die
|
||||||
|
Auswahl „immer / vertraute Netze / nie“ aus 0.5.0 (`0.0.0.0/0, ::/0` = ganz aus;
|
||||||
|
gespeicherter Modus „nie“ gilt weiter).
|
||||||
|
- Schutz vor Aussperren: Eine Einstellung, mit der die eigene Adresse keinen Zugang
|
||||||
|
mehr hätte, wird nicht gespeichert. `netmover reset-password` hebt auch Netz-Sperren auf.
|
||||||
|
|
||||||
|
## [0.5.0] – 2026-10-11
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **SFTP als Quelle** mit denselben Garantien wie bei SMB/NFS: Stabilitätsprüfung,
|
||||||
|
atomares Schreiben, Löschen der Quelle erst nach vollständiger Übertragung,
|
||||||
|
Kopier-Buchführung, Quarantäne. Verbindungsabbrüche brechen den Lauf als gestörte
|
||||||
|
Anbindung ab. Bei SFTP-Quellen mindestens 2 s Stabilitätsfenster (Zeiten nur
|
||||||
|
sekundengenau).
|
||||||
|
- **Gespeicherte Zugangsdaten** für SMB und SFTP-Passwörter (Benutzer, Passwort,
|
||||||
|
Domäne); in Anbindungen auswählbar, ein neues Passwort gilt sofort für alle.
|
||||||
|
Die Seite „Schlüssel“ heißt jetzt **„Zugangsdaten“** (Passwörter + SSH-Schlüssel);
|
||||||
|
`/keys` leitet weiter.
|
||||||
|
- **Anmeldung abschaltbar** (*Einstellungen → Anmeldung*, Bestätigung mit Passwort):
|
||||||
|
immer / nicht aus vertrauenswürdigen Netzen / nie (z. B. hinter Reverse Proxy mit
|
||||||
|
eigener Anmeldung). Deutlicher Hinweis in der Kopfzeile; CSRF-Schutz bleibt;
|
||||||
|
`netmover reset-password` schaltet die Anmeldung wieder ein.
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- Mover intern auf eine Quell-Schnittstelle umgestellt (`LocalSource`, `SftpSource`);
|
||||||
|
bestehendes Verhalten unverändert (alle bisherigen Tests gelten weiter).
|
||||||
|
|
||||||
|
## [0.4.0] – 2026-10-11
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Automatisch pausieren und fortsetzen:** Ist Quelle oder Ziel nicht erreichbar,
|
||||||
|
wartet die Regel („wartet seit …“ mit Grund) – ein Protokolleintrag statt eines
|
||||||
|
Fehlers alle 30 s, „Fehler in Folge“ zählt nicht hoch. Sobald die Anbindung zurück
|
||||||
|
ist, läuft die Regel von selbst weiter („Fortgesetzt nach … Wartezeit“).
|
||||||
|
Migration 0003: `rules.waiting_since`, `rules.waiting_reason`.
|
||||||
|
- **Wartungsmodus** (Knopf auf der Statusseite): alle geplanten Läufe pausiert,
|
||||||
|
„Jetzt ausführen“ geht weiter, kein Alarm für Regeln in `/healthz`; gelber Hinweis
|
||||||
|
in der Kopfzeile.
|
||||||
|
- **E-Mail-Benachrichtigung** per SMTP (STARTTLS, SSL/TLS oder ohne für interne
|
||||||
|
Relays; Zertifikate werden geprüft), Passwort verschlüsselt, mehrere Empfänger,
|
||||||
|
Test-Knopf. Webhook und E-Mail lassen sich gleichzeitig nutzen.
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- Ausfälle von Anbindungen werden **einmal pro Anbindung** gemeldet, mit der Liste
|
||||||
|
der betroffenen Regeln, und mit Entwarnung, wenn sie zurück sind – auch wenn eine
|
||||||
|
Anbindung von außen ausgehängt wurde (vorher keine Meldung).
|
||||||
|
- Einstellungen neu gegliedert: Benachrichtigungen · Webhook · E-Mail · Überwachung.
|
||||||
|
- `/healthz`: Eine wartende Regel ist ein Problem (mit Grund); im Wartungsmodus nicht.
|
||||||
|
|
||||||
|
## [0.3.0] – 2026-10-11
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Anleitung** (`docs/ANLEITUNG.md`) mit Screenshots, zugleich in der Oberfläche
|
||||||
|
unter *Hilfe*; „Hilfe zu dieser Seite“ in der Kopfzeile und „?“-Links an den
|
||||||
|
Formularabschnitten führen direkt zum passenden Abschnitt.
|
||||||
|
- **Roadmap** (`ROADMAP.md`).
|
||||||
|
- **Uptime Kuma:** `/healthz` mit Version, Scheduler-Herzschlag und Einzelprüfungen;
|
||||||
|
Einzel-Endpunkte `/healthz/mount/<name>` und `/healthz/rule/<nummer>` (200/503/404);
|
||||||
|
Push an einen Kuma-Push-Monitor jede Minute (*Einstellungen → Überwachung*, URL
|
||||||
|
maskiert, Test-Knopf, fertige Adressen für HTTP-Monitore).
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- **Menü neu aufgebaut:** Seitenleiste mit Gruppen in der Reihenfolge der
|
||||||
|
Einrichtung (Übersicht · Verbindungen · Automatisierung · System), Symbole,
|
||||||
|
Zähler (z. B. „2 gestört“, „1/2 aktiv“), Statusampel in der Kopfzeile; auf dem
|
||||||
|
Handy ausklappbares Menü.
|
||||||
|
- SFTP-Formular zeigt nur noch passende Felder und Hinweise (kein Passwortfeld bei
|
||||||
|
Schlüssel-Anmeldung, keine SMB-Hinweise); kompaktere Schlüsselseite.
|
||||||
|
- Gesundheit: Eine aktive Regel mit drei fehlgeschlagenen Läufen in Folge gilt jetzt
|
||||||
|
ebenfalls als Problem (nicht nur „zu lange kein Erfolg“).
|
||||||
|
|
||||||
|
### Behoben
|
||||||
|
- `aria-current` im Menü und der gesperrte Löschknopf auf der Schlüsselseite
|
||||||
|
wurden von Jinja maskiert (`="`) und wirkten nicht; Test prüft alle Seiten.
|
||||||
|
|
||||||
|
## [0.2.1] – 2026-10-11
|
||||||
|
|
||||||
|
### Behoben
|
||||||
|
- **Update von 0.1.x auf 0.2.0 scheiterte**: Migration 0002 brach auf einer
|
||||||
|
Datenbank mit Anbindungen ab (`FOREIGN KEY constraint failed` beim Neuaufbau der
|
||||||
|
Tabelle `credentials`). Weil SQLite-Migrationen nicht in einer Transaktion liefen,
|
||||||
|
blieb dabei ein halber Zustand zurück. Migrationen laufen jetzt ohne
|
||||||
|
Fremdschlüsselprüfung, aber mit Integritätsprüfung vor dem Commit und vollständig
|
||||||
|
in einer Transaktion: Ein Abbruch hinterlässt die Datenbank unverändert.
|
||||||
|
Geprüft mit Tests auf verknüpften Daten und an einer Kopie einer echten Datenbank.
|
||||||
|
- `netmover-update` zeigt beim Zurückstufen keinen irreführenden Changelog-Auszug mehr.
|
||||||
|
|
||||||
|
## [0.2.0] – 2026-10-11
|
||||||
|
|
||||||
|
> ⚠️ **Nicht verwenden** – das Update von 0.1.x scheitert an Migration 0002.
|
||||||
|
> Stattdessen 0.2.1 installieren. Wer 0.2.0 schon versucht hat: Die Datenbank aus der
|
||||||
|
> Sicherung von `netmover-update` zurückspielen und auf 0.2.1 aktualisieren.
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Schlüsselverwaltung** (Seite „Schlüssel“): SSH-Schlüssel erzeugen (Ed25519) oder
|
||||||
|
importieren (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH/PEM, auch mit Passphrase),
|
||||||
|
Fingerabdruck und `authorized_keys`-Zeile mit SFTP-Beschränkung zum Kopieren,
|
||||||
|
Umbenennen, Löschen nur wenn unbenutzt. Private Schlüssel werden nie angezeigt.
|
||||||
|
- SFTP-Anbindungen können einen gespeicherten Schlüssel auswählen; der Benutzername
|
||||||
|
gehört zur Anbindung (Migration 0002: `credentials.managed`, `mounts.username`).
|
||||||
|
- **`netmover-update`**: Update aus dem Git-Repo mit Sicherung von Datenbank,
|
||||||
|
Schlüsseln und Einstellungen, Changelog-Auszug, Neustart und Healthcheck;
|
||||||
|
Versionswahl, Warnung beim Zurückstufen.
|
||||||
|
- Installation als Git-Klon unter `/opt/netmover/src`; `install.sh` installiert
|
||||||
|
`git` und das Update-Skript.
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- Klartext-Hinweis bei `mount error(95)` (keine gemeinsame SMB-Version) mit
|
||||||
|
Synology-Einstellung; aufgefallen bei der Anbindung des echten NAS.
|
||||||
|
- SFTP-Anbindungen gelten nach erfolgreichem Test als „erreichbar“ (vorher bis
|
||||||
|
zum ersten Regellauf „nicht geprüft“).
|
||||||
|
|
||||||
## [0.1.1] – 2026-10-10
|
## [0.1.1] – 2026-10-10
|
||||||
|
|
||||||
Korrekturen aus dem ersten Praxistest auf einer Debian-13-VM mit echten
|
Korrekturen aus dem ersten Praxistest auf einer Debian-13-VM mit echten
|
||||||
@@ -128,6 +331,17 @@ Erste Version.
|
|||||||
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
||||||
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
||||||
|
|
||||||
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.1...HEAD
|
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.11.0...HEAD
|
||||||
|
[0.11.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.10.0...v0.11.0
|
||||||
|
[0.10.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.9.0...v0.10.0
|
||||||
|
[0.9.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.8.0...v0.9.0
|
||||||
|
[0.8.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.7.0...v0.8.0
|
||||||
|
[0.7.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.6.0...v0.7.0
|
||||||
|
[0.6.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.5.0...v0.6.0
|
||||||
|
[0.5.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.4.0...v0.5.0
|
||||||
|
[0.4.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.3.0...v0.4.0
|
||||||
|
[0.3.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.2.1...v0.3.0
|
||||||
|
[0.2.1]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.2.0...v0.2.1
|
||||||
|
[0.2.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.1...v0.2.0
|
||||||
[0.1.1]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.0...v0.1.1
|
[0.1.1]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.0...v0.1.1
|
||||||
[0.1.0]: https://git.wilde-server.de/florian.wilde/netmover/releases/tag/v0.1.0
|
[0.1.0]: https://git.wilde-server.de/florian.wilde/netmover/releases/tag/v0.1.0
|
||||||
|
|||||||
@@ -14,10 +14,246 @@ Zeiten in Ortszeit (Europe/Berlin).
|
|||||||
| 2026-10-10 | 19:18 | 19:34 | 0:16 | 5 | Weboberfläche und Anmeldung |
|
| 2026-10-10 | 19:18 | 19:34 | 0:16 | 5 | Weboberfläche und Anmeldung |
|
||||||
| 2026-10-10 | 19:40 | 19:50 | 0:10 | 6 | Deployment, README, Release 0.1.0 |
|
| 2026-10-10 | 19:40 | 19:50 | 0:10 | 6 | Deployment, README, Release 0.1.0 |
|
||||||
| 2026-10-10 | 21:36 | 22:05 | 0:29 | – | Praxistest auf VM, Release 0.1.1 |
|
| 2026-10-10 | 21:36 | 22:05 | 0:29 | – | Praxistest auf VM, Release 0.1.1 |
|
||||||
| **Summe** | | | **1:35** | | |
|
| 2026-10-10 | 22:45 | 23:00 | 0:15 | – | Echtes NAS/Paperless anbinden |
|
||||||
|
| 2026-10-10 | 23:31 | 23:58 | 0:27 | – | Schlüsselverwaltung, Update-Skript, 0.2.x |
|
||||||
|
| 2026-10-11 | 00:13 | 00:32 | 0:19 | – | Menü, Anleitung, Roadmap, Uptime Kuma (0.3.0) |
|
||||||
|
| 2026-10-11 | 00:31 | 00:50 | 0:19 | – | Pausieren, Wartungsmodus, E-Mail (0.4.0) |
|
||||||
|
| 2026-10-11 | 00:54 | 01:16 | 0:22 | – | SFTP-Quelle, Zugangsdaten, Anmeldung (0.5.0) |
|
||||||
|
| 2026-10-11 | 08:43 | 08:55 | 0:12 | – | Anmeldung je Netz (0.6.0) |
|
||||||
|
| 2026-10-12 | 00:14 | 00:34 | 0:20 | – | Archiv je Regel (0.7.0) |
|
||||||
|
| 2026-10-12 | 00:46 | 00:55 | 0:09 | – | Anbindungen kopieren (0.8.0) |
|
||||||
|
| 2026-10-15 | 09:17 | 09:26 | 0:09 | – | Regeln kopieren (0.9.0) |
|
||||||
|
| 2026-10-23 | 13:42 | 14:10 | 0:28 | – | Sicherung, Kuma-Kennzahlen, CI (0.10.0) |
|
||||||
|
| 2026-10-24 | 00:10 | 01:00 | 0:50 | – | Mehrere Ziele je Regel (0.11.0) |
|
||||||
|
| **Summe** | | | **5:25** | | |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 2026-10-24 · 00:10–01:00 · Mehrere Ziele je Regel (0.11.0)
|
||||||
|
|
||||||
|
**Anlass** – Kundenanforderung Rechnungseingang: E-Rechnung ins DMS, in den
|
||||||
|
Importordner der Buchhaltung **und** ins Archiv – drei Ziele, Ziel + Archiv reichte
|
||||||
|
nicht mehr.
|
||||||
|
|
||||||
|
**Erledigt** – Tabellen `rule_targets`/`target_files` (Migration 0005, Archiv wird
|
||||||
|
Ziel „Archiv“), Engine mit Zielliste, Runner prüft Ziele einzeln, `RunResult.waiting_on`,
|
||||||
|
Formular mit 9 Plätzen (nächster Platz erscheint per `data-show-for`, kein neues JS),
|
||||||
|
Export/Import inkl. Altformat. 392 Tests. Gegenproben: Quelle trotz fehlendem Ziel
|
||||||
|
löschen → 3 Tests rot; Buchführung je Ziel weglassen → 6 Tests rot.
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- *Hauptziel bleibt in `rules`*, nur die weiteren Ziele in einer Tabelle: kleinster
|
||||||
|
Eingriff ins Schema, bewährte Reihenfolge (weitere Ziele zuerst, Hauptziel zuletzt,
|
||||||
|
dort weiter der schnelle Umbenennen-Weg) bleibt.
|
||||||
|
- *Nicht erreichbares Ziel blockiert die anderen nicht*: Bei 3–10 Zielen wäre sonst
|
||||||
|
jede gestörte Freigabe ein Stillstand. Das Hauptziel bekommt dann eine Kopie
|
||||||
|
(Buchführung in `copied_files`), die Quelle wird erst nach der Nachlieferung gelöscht.
|
||||||
|
Gelöscht wird nur in dem Lauf, in dem das letzte Ziel die Datei bekommt.
|
||||||
|
- *Kein Protokoll-Spam*: Fehlen nur noch nicht erreichbare Ziele, liefert `_process`
|
||||||
|
None; ein Lauf ohne Einträge wird wie „wartet“ behandelt (kein neuer Run).
|
||||||
|
- *Beim Speichern Ziele an Ort und Stelle aktualisieren*: Buchführung bleibt, außer
|
||||||
|
Anbindung oder Ordner eines Ziels ändern sich.
|
||||||
|
- *Weitere Ziele überschreiben nie* (wie bisher das Archiv) – kein eigenes
|
||||||
|
Kollisionsfeld je Ziel, weniger Fehlerquellen.
|
||||||
|
|
||||||
|
## 2026-10-23 · 13:42–14:10 · Sicherung, Kuma-Kennzahlen, CI (0.10.0)
|
||||||
|
|
||||||
|
**Bestandsaufnahme Live-Server** – 0.9.0 seit 15.10. ohne Neustart, 287 Dateien
|
||||||
|
verschoben, keine Datei gescheitert oder in Quarantäne, Archivkopien vollständig
|
||||||
|
(159 + 122). 18 kurze Pausen, alle wegen SSH zu den Paperless-Servern
|
||||||
|
(„No existing session“, „protocol banner“, Timeout) – automatisches Pausieren und
|
||||||
|
Fortsetzen hat jedes Mal gegriffen. Ursache liegt vermutlich auf den Paperless-Hosts
|
||||||
|
→ Roadmap 0.11.0.
|
||||||
|
|
||||||
|
**Erledigt** – `backup.py` (Momentaufnahme per SQLite-Backup-API, tar.gz,
|
||||||
|
scrypt+Fernet, Probe-Öffnen vor dem Upload, Upload über `Destination.commit`,
|
||||||
|
Ausdünnen), Seite „Sicherung“, `/healthz/backup`, `netmover backup`/`restore`,
|
||||||
|
Wrapper `/usr/local/sbin/netmover`; Kennzahlen unter `metrics` in `/healthz`;
|
||||||
|
Gitea-CI; generischer Migrationstest. 381 Tests. Gegenproben: Ausdünnen ohne
|
||||||
|
Namensfilter und Sicherung ohne Verschlüsselung schlagen jeweils an (2 Tests).
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- *Passphrase verschlüsselt in der DB*: sonst bräuchte die tägliche Sicherung eine
|
||||||
|
Eingabe. Auf dem NAS schützt die Passphrase; auf dem Server ist ohnehin alles da.
|
||||||
|
- *Termin per „fällig?“-Prüfung alle 5 min* statt Cron-Job: holt verpasste Termine
|
||||||
|
nach, braucht kein Umplanen bei Änderungen und regelt den stündlichen Neuversuch.
|
||||||
|
- *Ziel über dieselbe Fabrik wie Regelziele* (`destination_factory`): gleiche
|
||||||
|
Mount-Prüfung, `.part`-Logik und SFTP-Unterstützung, kein zweiter Schreibweg.
|
||||||
|
- *Ausdünnen nur nach Namensschema* – fremde Dateien im Ordner bleiben unangetastet.
|
||||||
|
- *Kein TLS* im internen Netz (Nutzerentscheidung), im README begründet.
|
||||||
|
- *Kuma*: Endpunkte gab es seit 0.3.0; neu sind auslesbare Kennzahlen für
|
||||||
|
„JSON-Abfrage“-Monitore, z. B. Alarm bei `metrics.quarantined_24h != 0`.
|
||||||
|
|
||||||
|
## 2026-10-15 · 09:17–09:26 · Regeln kopieren (0.9.0)
|
||||||
|
|
||||||
|
**Erledigt** – Knopf „Kopieren“, Route `/rules/{id}/copy` (GET Formular, POST
|
||||||
|
speichern), `rule_copy_name`. 5 Tests (362 gesamt); Gegenprobe „Kopie aktiv“ schlägt
|
||||||
|
an (2 Tests).
|
||||||
|
|
||||||
|
**Entscheidung** – Kopie startet deaktiviert: Mit identischer Quelle würden Original
|
||||||
|
und Kopie sonst sofort parallel dieselben Dateien bearbeiten (doppelte Lieferung bzw.
|
||||||
|
Kollisionen). Protokoll und Buchführung (`copied_files`, `archived_files`) werden
|
||||||
|
bewusst nicht mitkopiert – die Kopie ist eine neue Regel.
|
||||||
|
|
||||||
|
## 2026-10-12 · 00:46–00:55 · Anbindungen kopieren (0.8.0)
|
||||||
|
|
||||||
|
**Erledigt** – Knopf „Kopieren“, Route `/mounts/{id}/copy` (GET Formular, POST
|
||||||
|
speichern), `save_mount(copy_from=…)`, Formular-Test mit Vorlage. 7 Tests (357 gesamt);
|
||||||
|
Gegenprobe ohne Übernahme der Zugangsdaten: 3 Tests schlagen an.
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- Eigene Zugangsdaten werden als eigenes Exemplar kopiert (nicht geteilt) – eigene
|
||||||
|
Zugangsdaten verschwinden mit ihrer Anbindung, eine geteilte Nutzung wäre
|
||||||
|
dafür der Fall „Zugangsdaten“.
|
||||||
|
- Eigenes Schlüsselpaar bleibt gleich (schon in `authorized_keys`).
|
||||||
|
- Hostschlüssel nur bei gleichem Server/Port übernehmen, sonst würde der Test an
|
||||||
|
einem anderen Server mit „Hostschlüssel geändert“ scheitern.
|
||||||
|
|
||||||
|
## 2026-10-12 · 00:14–00:34 · Archiv je Regel (0.7.0)
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- Archivfelder an der Regel, Tabelle `archived_files` (Migration 0004).
|
||||||
|
- Engine: Archiv zuerst, dann Ziel, dann Quelle löschen; Archiv immer „umbenennen“.
|
||||||
|
- Runner: Ziel- und Archivaufbau über eine gemeinsame Fabrik (`_target`).
|
||||||
|
- 14 Tests (350 gesamt); Mutationsprobe 4/4 erkannt (Buchführung, Archiv
|
||||||
|
übersprungen, Kollisionsregel, Platzprüfung).
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- Archiv *vor* Ziel: Paperless holt Dateien sofort ab – andersherum würde ein
|
||||||
|
gescheitertes Archiv beim Neuversuch zur doppelten Lieferung führen.
|
||||||
|
- Archiv überschreibt nie (unabhängig von der Kollisionsregel des Ziels).
|
||||||
|
- Ablage „nach Datum“ nach Verarbeitungsdatum (nicht Datei-mtime) – eindeutig
|
||||||
|
nachvollziehbar, wann eine Datei durchgelaufen ist.
|
||||||
|
|
||||||
|
**Stolperstein**
|
||||||
|
- Downgrade von 0004 scheiterte an den CHECK-Regeln der neuen Auswahlfelder
|
||||||
|
(verweisen nach dem Entfernen der Spalten ins Leere) – nur dank der Migrations-
|
||||||
|
tests mit Daten aufgefallen; Downgrade entfernt sie jetzt zuerst.
|
||||||
|
|
||||||
|
## 2026-10-11 · 08:43–08:55 · Anmeldung je Netz (0.6.0)
|
||||||
|
|
||||||
|
**Anlass** – Wunsch: Anmeldung abschalten, und zwar konfigurierbar, aus welchen Netzen
|
||||||
|
mit und ohne Passwort. Zugriffe kamen aus 10.0.10.0/24 und 10.1.5.0/24 (Access-Log).
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- `auth.access_decision()` als einzige Entscheidung (ohne PW / mit PW / abgewiesen),
|
||||||
|
benutzt von Middleware (403 für fremde Netze), Guard und Anmeldeseite.
|
||||||
|
- Schutz vor Aussperren beim Speichern; `reset-password` hebt Sperren auf.
|
||||||
|
- 19 Tests für die Anmeldung (336 gesamt).
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- `/healthz` und `/static` nie sperren – Uptime Kuma kann in einem anderen Netz stehen.
|
||||||
|
- Nicht-IP-Clients (z. B. Unix-Socket) gelten als „in keinem Netz“.
|
||||||
|
|
||||||
|
## 2026-10-11 · 00:54–01:16 · SFTP-Quelle, Zugangsdaten, Anmeldung abschaltbar (0.5.0)
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- `mover/sources.py`: `Source` mit `LocalSource`/`SftpSource`; Engine greift nur noch
|
||||||
|
darüber auf die Quelle zu (Scan, frischer Zustand, Lesen, Löschen, Quarantäne).
|
||||||
|
Ziele bekommen statt `Path` ein „Readable“ (Adapter) – Destinations fast unverändert.
|
||||||
|
- `credentials.py` (vorher `keys.py`): gespeicherte Passwörter; Seite „Zugangsdaten“.
|
||||||
|
- Anmeldung: Modus immer / vertraute Netze / nie, zentral im `guard`.
|
||||||
|
- 26 neue Tests (325 gesamt); alle bisherigen Mover-Tests bestanden den Umbau
|
||||||
|
unverändert (nur ein Monkeypatch-Ziel verschoben).
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- SFTP-Quelle: Stabilität über Größe + sekundengenaue mtime, Mindestfenster 2 s;
|
||||||
|
Verbindungsabbruch → `ENOTCONN` → Lauf abgebrochen (Anbindung gestört).
|
||||||
|
- Gespeicherte Passwörter tragen den Benutzer selbst (anders als Schlüssel, deren
|
||||||
|
Benutzer zur Anbindung gehört).
|
||||||
|
- „Passworteingabe deaktivieren“ als Login-Modus umgesetzt, mit Bestätigung durch
|
||||||
|
das Passwort, Warnhinweis und Notausgang über `reset-password`. Ein Passwort bleibt
|
||||||
|
Pflicht (Ersteinrichtung), sonst ließe sich die Anmeldung nie wieder einschalten.
|
||||||
|
|
||||||
|
**Stolpersteine**
|
||||||
|
- paramiko `prefetch()` braucht `fstat`; der Test-Server konnte es nicht – jetzt
|
||||||
|
kann er es, und die Quelle übergibt die bekannte Größe (spart einen Umlauf).
|
||||||
|
- Zwei TestClients mit eigenem Lebenszyklus starteten den Scheduler doppelt →
|
||||||
|
zweiter Client ohne Lebenszyklus auf derselben App.
|
||||||
|
|
||||||
|
## 2026-10-11 · 00:31–00:50 · Pausieren, Wartungsmodus, E-Mail (0.4.0)
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- Wartezustand (`rules.waiting_since/_reason`, Migration 0003), Wartungsmodus,
|
||||||
|
E-Mail (`mail.py`, SMTP), eine Meldung pro Anbindung mit betroffenen Regeln.
|
||||||
|
- 0.4.0/0.5.0 aufgeteilt: kleinere Updates, schneller Nutzen.
|
||||||
|
- 21 neue Tests (299 gesamt), u. a. SMTP gegen echten Server (aiosmtpd) mit
|
||||||
|
STARTTLS, SSL, falschem Passwort und nicht vertrauenswürdigem Zertifikat.
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- Warten statt Deaktivieren; kein Fehlerzähler und nur ein Protokolleintrag beim
|
||||||
|
Eintritt; Ende des Wartens automatisch beim nächsten durchkommenden Lauf.
|
||||||
|
- Benachrichtigung pro Anbindung (nicht pro Regel) – sonst bei drei Regeln am NAS
|
||||||
|
drei gleiche Meldungen. Lücke geschlossen: Aushängen von außen meldete nichts.
|
||||||
|
- Wartungsmodus unterdrückt nur Regel-Alarme, Anbindungen werden weiter geprüft.
|
||||||
|
- SMTP prüft Zertifikate immer; für Tests ist nur `mail.tls_context()` ersetzbar.
|
||||||
|
|
||||||
|
**Stolpersteine (alle im Testaufbau)**
|
||||||
|
- Globales Monkeypatchen von `ssl.create_default_context` traf auch den
|
||||||
|
Testserver → eigene Funktion `tls_context()`.
|
||||||
|
- aiosmtpd: `AuthResult(success=False)` ohne `handled=False` schickt keine Antwort →
|
||||||
|
Client hängt bis zum Zeitlimit.
|
||||||
|
|
||||||
|
**Migrationen** – 0003 baut `rules` neu; getestet von 0001 und von 0002 mit allen
|
||||||
|
verknüpften Tabellen sowie an der befüllten Demo-Datenbank; vor dem Release an einer
|
||||||
|
Kopie der Live-Datenbank auf der VM.
|
||||||
|
|
||||||
|
## 2026-10-11 · 00:13–00:32 · Menü, Anleitung, Roadmap, Uptime Kuma (0.3.0)
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- Seitenleiste mit Gruppen, Symbolen (Inline-SVG), Zählern und Statusampel;
|
||||||
|
Handy-Menü per Knopf (ohne JS sichtbar).
|
||||||
|
- `docs/ANLEITUNG.md` (für Anwender) mit Sprungmarken; Hilfe-Seite rendert sie per
|
||||||
|
Python-Markdown, das Paket liefert `docs/` mit (hatch force-include).
|
||||||
|
- `ROADMAP.md`; Nutzerwünsche eingeplant: SFTP als Quelle, gespeicherte Zugangsdaten,
|
||||||
|
automatisches Pausieren/Fortsetzen, E-Mail (0.4.0).
|
||||||
|
- Uptime Kuma: ausführliches `/healthz`, Einzel-Endpunkte, Push-Job.
|
||||||
|
- 13 neue Tests (278 gesamt).
|
||||||
|
|
||||||
|
**Wie geprüft**
|
||||||
|
- Erstmals echte Screenshots mit Playwright (Chromium, Scratchpad-venv) gegen eine
|
||||||
|
Demo-Instanz (Fake-Backend), Desktop 1366 px und Handy 390 px, hell und dunkel.
|
||||||
|
Daraus gefunden und behoben: überflüssiges Passwortfeld/SMB-Hinweise bei SFTP,
|
||||||
|
sperriges Umbenennen auf der Schlüsselseite, Tabellen in der Hilfe ragten auf
|
||||||
|
dem Handy über den Rand. Dieselben Screenshots sind jetzt in der Anleitung
|
||||||
|
(128-Farben-PNG, zusammen ~150 KB; Demo-Pfad per DOM durch `/mnt/netmover` ersetzt).
|
||||||
|
- Ein echter Fehler, den erst der Blick auf das HTML zeigte: `{{ 'attr="x"' if … }}`
|
||||||
|
wird von Jinja maskiert → `aria-current` und `disabled title` wirkten nicht.
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- Automatisches Pausieren: kein Deaktivieren (müsste nach jedem NAS-Neustart von
|
||||||
|
Hand rückgängig gemacht werden), sondern „wartet auf Anbindung“ mit
|
||||||
|
automatischem Fortsetzen – für 0.4.0 vorgeschlagen.
|
||||||
|
- Regel mit 3 Fehlschlägen in Folge zählt jetzt als Problem in `/healthz`.
|
||||||
|
|
||||||
|
## 2026-10-10 · 23:31–23:58 · Schlüsselverwaltung, Update-Skript, Releases 0.2.0/0.2.1
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- Schlüsselverwaltung (`keys.py`, Seite „Schlüssel“, Auswahl in SFTP-Anbindungen),
|
||||||
|
Migration 0002 (`credentials.managed`, `mounts.username`).
|
||||||
|
- `deploy/update.sh` → `netmover-update`: Tags holen, Sicherung (SQLite-Online-
|
||||||
|
Backup + Schlüssel + env), Checkout, `install.sh`, Healthcheck, Rückweg-Anleitung.
|
||||||
|
- VM auf Git-Klon `/opt/netmover/src` umgestellt (Repo ist öffentlich lesbar).
|
||||||
|
|
||||||
|
**Zwischenfall beim Update der VM auf 0.2.0**
|
||||||
|
- Migration 0002 scheiterte: `FOREIGN KEY constraint failed` bei `DROP TABLE
|
||||||
|
credentials` (Alembic-Batch baut die Tabelle neu; `foreign_keys=ON`, und die
|
||||||
|
Anbindungen verweisen darauf). Danach crashte jeder Neustart an der
|
||||||
|
übriggebliebenen `_alembic_tmp_credentials` (DDL lief ohne Transaktion).
|
||||||
|
- `netmover-update` hat den Fehler erkannt und die Rückweg-Befehle genannt.
|
||||||
|
Sicherung zurückgespielt, mit `netmover-update v0.1.1` zurückgestuft – in
|
||||||
|
~1 Minute wieder lauffähig, NAS-Mount blieb durchgehend eingehängt.
|
||||||
|
- Ursache, warum die Tests es nicht fanden: Migrationen wurden nur auf **leeren**
|
||||||
|
Datenbanken getestet. Neu: `tests/test_migrations.py` mit verknüpften Daten und
|
||||||
|
mit Abbruch mitten in der Migration (Datenbank muss unverändert bleiben).
|
||||||
|
- Fix: eigene Migrationsverbindung (FK aus, `BEGIN IMMEDIATE`, `transactional_ddl`,
|
||||||
|
`foreign_key_check` vor dem Commit). Vor dem Release an einer Kopie der echten
|
||||||
|
Datenbank auf der VM geprüft.
|
||||||
|
- Tag v0.2.0 bleibt (veröffentlicht), im Changelog als „nicht verwenden“ markiert.
|
||||||
|
|
||||||
|
**Lehre**
|
||||||
|
- Jede künftige Migration bekommt einen Test auf einer befüllten Datenbank.
|
||||||
|
|
||||||
## 2026-10-10 · 21:36–22:05 · Praxistest auf der Debian-13-VM, Release 0.1.1
|
## 2026-10-10 · 21:36–22:05 · Praxistest auf der Debian-13-VM, Release 0.1.1
|
||||||
|
|
||||||
**Umgebung**
|
**Umgebung**
|
||||||
|
|||||||
@@ -3,11 +3,20 @@
|
|||||||
Selbst gehosteter Dienst mit Weboberfläche, der **SMB/CIFS- und NFS-Freigaben
|
Selbst gehosteter Dienst mit Weboberfläche, der **SMB/CIFS- und NFS-Freigaben
|
||||||
verwaltet** und **Dateien zeitgesteuert verschiebt** – „alle N Sekunden Ordner 1
|
verwaltet** und **Dateien zeitgesteuert verschiebt** – „alle N Sekunden Ordner 1
|
||||||
prüfen, passende Dateien nach Ordner 2 bringen“. Quelle und Ziel sind
|
prüfen, passende Dateien nach Ordner 2 bringen“. Quelle und Ziel sind
|
||||||
unabhängig voneinander per SMB, NFS, lokal oder (nur als Ziel) per SFTP angebunden.
|
unabhängig voneinander per SMB, NFS, SFTP oder (als Ziel) lokal angebunden.
|
||||||
|
|
||||||
Typischer Einsatz: Ein Synology-NAS nimmt Scans entgegen, NetMover schiebt sie alle
|
Typischer Einsatz: Ein Synology-NAS nimmt Scans entgegen, NetMover schiebt sie alle
|
||||||
30 Sekunden in den Eingangsordner von Paperless-ngx.
|
30 Sekunden in den Eingangsordner von Paperless-ngx.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
| Dokument | Inhalt |
|
||||||
|
|---|---|
|
||||||
|
| **[Anleitung](docs/ANLEITUNG.md)** | Bedienung: Anbindungen, Schlüssel, Regeln, Status, Protokoll, Uptime Kuma, Beispiele, Fragen – auch in der Oberfläche unter *Hilfe* |
|
||||||
|
| **[Roadmap](ROADMAP.md)** | Was in Arbeit und geplant ist |
|
||||||
|
| [Changelog](CHANGELOG.md) | Änderungen je Version |
|
||||||
|
| dieses README | Installation, Update, Betrieb, Fehlersuche, Sicherheit |
|
||||||
|
|
||||||
- [Wie NetMover Datenverlust verhindert](#wie-netmover-datenverlust-verhindert)
|
- [Wie NetMover Datenverlust verhindert](#wie-netmover-datenverlust-verhindert)
|
||||||
- [Installation](#installation) · [Erster Start](#erster-start)
|
- [Installation](#installation) · [Erster Start](#erster-start)
|
||||||
- [Beispiel: Synology → Paperless](#beispiel-synology--paperless)
|
- [Beispiel: Synology → Paperless](#beispiel-synology--paperless)
|
||||||
@@ -47,15 +56,36 @@ heißt deshalb immer **kopieren, prüfen, dann erst die Quelle löschen**:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
apt install -y git
|
apt install -y git
|
||||||
git clone https://git.wilde-server.de/florian.wilde/netmover.git
|
git clone https://git.wilde-server.de/florian.wilde/netmover.git /opt/netmover/src
|
||||||
cd netmover
|
cd /opt/netmover/src && git checkout "$(git tag -l 'v*' --sort=-v:refname | head -n 1)"
|
||||||
sudo ./deploy/install.sh
|
./deploy/install.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
Das Skript installiert `python3-venv`, `cifs-utils` und `nfs-common` und richtet
|
Das Skript installiert `python3-venv`, `cifs-utils`, `nfs-common` und `git`. Es
|
||||||
NetMover in `/opt/netmover/venv` ein. Danach installiert es
|
richtet NetMover in `/opt/netmover/venv` ein, installiert
|
||||||
[`deploy/netmover.service`](deploy/netmover.service) und startet den Dienst.
|
[`deploy/netmover.service`](deploy/netmover.service) und das Update-Skript
|
||||||
Ein Update ist derselbe Aufruf nach einem `git pull`.
|
`netmover-update` und startet den Dienst.
|
||||||
|
|
||||||
|
### Update
|
||||||
|
|
||||||
|
```bash
|
||||||
|
netmover-update # auf die neueste Version
|
||||||
|
netmover-update --list # verfügbare Versionen
|
||||||
|
netmover-update v0.2.0 # bestimmte Version
|
||||||
|
```
|
||||||
|
|
||||||
|
`netmover-update` geht so vor:
|
||||||
|
1. Neue Versions-Tags aus Gitea holen und die Änderungen laut Changelog anzeigen.
|
||||||
|
2. **Datenbank, `secret.key` und Einstellungen nach `/var/backups/netmover/` sichern**
|
||||||
|
(die letzten 10 bleiben).
|
||||||
|
3. Die Version auschecken, installieren und den Dienst neu starten. Neue
|
||||||
|
Datenbank-Migrationen laufen dabei automatisch.
|
||||||
|
4. `/healthz` prüfen.
|
||||||
|
|
||||||
|
Die Mounts bleiben die ganze Zeit eingehängt. Geht etwas schief, nennt das Skript
|
||||||
|
die Befehle, um die Sicherung zurückzuspielen und zur vorherigen Version zu wechseln.
|
||||||
|
**Zurückstufen** auf eine ältere Version geht nur mit der Datenbank-Sicherung von
|
||||||
|
vor dem Update, wenn das Update eine Migration enthielt; das Skript fragt deshalb nach.
|
||||||
|
|
||||||
Was wo liegt:
|
Was wo liegt:
|
||||||
|
|
||||||
@@ -146,14 +176,21 @@ Lege in NetMover die *Anbindung → Neue Anbindung* an:
|
|||||||
| Feld | Wert |
|
| Feld | Wert |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Name | `Paperless` |
|
| Name | `Paperless` |
|
||||||
| Typ | SFTP (nur Ziel) |
|
| Typ | SFTP |
|
||||||
| Server / SSH-Port | IP des LXC / `22` |
|
| Server / SSH-Port | IP des LXC / `22` |
|
||||||
| Freigabe / Pfad | `/opt/paperless_data/consume` |
|
| Freigabe / Pfad | `/opt/paperless_data/consume` |
|
||||||
| Anmeldeart | SSH-Schlüssel (NetMover erzeugt ihn) |
|
| Anmeldeart | SSH-Schlüssel |
|
||||||
|
| Schlüssel | einen gespeicherten Schlüssel wählen (siehe unten) oder „neuer Schlüssel nur für diese Anbindung“ |
|
||||||
| Benutzername | `paperless` (der Benutzer, unter dem Paperless läuft) |
|
| Benutzername | `paperless` (der Benutzer, unter dem Paperless läuft) |
|
||||||
|
|
||||||
Nach dem Speichern zeigt NetMover den **öffentlichen Schlüssel**. Trage ihn im LXC
|
**Schlüsselverwaltung:** Unter *Schlüssel* erzeugst du Ed25519-Schlüssel oder
|
||||||
ein, und zwar so, dass der Schlüssel **nur SFTP** darf:
|
importierst vorhandene (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH- oder PEM-Format,
|
||||||
|
auch mit Passphrase). Ein Schlüssel kann von mehreren Anbindungen genutzt werden; den
|
||||||
|
Benutzernamen legt jede Anbindung selbst fest. Private Schlüssel liegen verschlüsselt
|
||||||
|
in der Datenbank und werden nie angezeigt. Die Seite zeigt den Fingerabdruck und eine
|
||||||
|
fertige `authorized_keys`-Zeile zum Kopieren.
|
||||||
|
|
||||||
|
Trage den öffentlichen Schlüssel im LXC so ein, dass er **nur SFTP** darf:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# im Paperless-LXC als root
|
# im Paperless-LXC als root
|
||||||
@@ -312,8 +349,11 @@ Webhook-Vorlagen (Platzhalter `$event $title $message $rule $time $host`):
|
|||||||
| Gotify | `https://gotify.lan/message?token=…` | `application/json` | `{"title": "$title", "message": "$message", "priority": 5}` |
|
| Gotify | `https://gotify.lan/message?token=…` | `application/json` | `{"title": "$title", "message": "$message", "priority": 5}` |
|
||||||
| eigener Empfänger | beliebig | `application/json` | Standardvorlage |
|
| eigener Empfänger | beliebig | `application/json` | Standardvorlage |
|
||||||
|
|
||||||
`GET /healthz` liefert `200` oder `503` mit der Liste der Probleme (ohne Anmeldung),
|
**Überwachung:** `GET /healthz` liefert ohne Anmeldung `200` oder `503` mit allen
|
||||||
z. B. für Uptime Kuma.
|
Einzelprüfungen; `/healthz/mount/<name>` und `/healthz/rule/<nummer>` prüfen eine
|
||||||
|
einzelne Anbindung oder Regel. Alternativ meldet NetMover jede Minute an einen
|
||||||
|
Uptime-Kuma-Push-Monitor (*Einstellungen → Überwachung*). Details:
|
||||||
|
[Anleitung → Uptime Kuma](docs/ANLEITUNG.md#uptime-kuma).
|
||||||
|
|
||||||
## Fehlersuche
|
## Fehlersuche
|
||||||
|
|
||||||
@@ -327,6 +367,7 @@ findmnt -R /mnt/netmover # was ist wirklich eingehängt?
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `mount error(13): Permission denied` (CIFS) | Benutzer oder Passwort falsch, oder keine Rechte auf die Freigabe. |
|
| `mount error(13): Permission denied` (CIFS) | Benutzer oder Passwort falsch, oder keine Rechte auf die Freigabe. |
|
||||||
| `mount error(2): No such file or directory` | Freigabename falsch geschrieben (Groß/klein!). |
|
| `mount error(2): No such file or directory` | Freigabename falsch geschrieben (Groß/klein!). |
|
||||||
|
| `mount error(95): Operation not supported` | Keine gemeinsame SMB-Version (im Kernel-Log: „Dialect not supported by server“). Am NAS das maximale SMB-Protokoll auf SMB3 stellen oder Zusatzoption `vers=2.0`. |
|
||||||
| `mount error(112): Host is down` | NAS nicht erreichbar, oder SMB1 erwartet: `vers=3.0` bzw. `vers=2.1` probieren. |
|
| `mount error(112): Host is down` | NAS nicht erreichbar, oder SMB1 erwartet: `vers=3.0` bzw. `vers=2.1` probieren. |
|
||||||
| `… helper program` / `unknown filesystem type` | `apt install cifs-utils nfs-common` |
|
| `… helper program` / `unknown filesystem type` | `apt install cifs-utils nfs-common` |
|
||||||
| `Operation not permitted` / `only root can do that` | Kein `CAP_SYS_ADMIN`: unprivilegierter LXC oder Docker ohne `cap_add`. |
|
| `Operation not permitted` / `only root can do that` | Kein `CAP_SYS_ADMIN`: unprivilegierter LXC oder Docker ohne `cap_add`. |
|
||||||
@@ -347,12 +388,20 @@ findmnt -R /mnt/netmover # was ist wirklich eingehängt?
|
|||||||
## Sicherheit
|
## Sicherheit
|
||||||
|
|
||||||
- **Nicht ins Internet stellen** (siehe [Reverse Proxy](#reverse-proxy-mit-tls)).
|
- **Nicht ins Internet stellen** (siehe [Reverse Proxy](#reverse-proxy-mit-tls)).
|
||||||
|
- **HTTP im internen Netz ist eine bewusste Entscheidung**, wenn das Netz
|
||||||
|
vertrauenswürdig ist (Heimnetz, eigenes VLAN): Die Oberfläche läuft dann
|
||||||
|
unverschlüsselt auf Port 8080, die Anmeldung kann per Netz abgeschaltet werden
|
||||||
|
(*Einstellungen → Anmeldung*). Wer im selben Netz mitlesen kann, könnte Passwörter
|
||||||
|
sehen, die man in Formulare eingibt. Für fremde oder gemischte Netze den
|
||||||
|
[Reverse Proxy mit TLS](#reverse-proxy-mit-tls) davorsetzen.
|
||||||
- Zugangsdaten liegen nur verschlüsselt in der Datenbank (Fernet, Schlüssel
|
- Zugangsdaten liegen nur verschlüsselt in der Datenbank (Fernet, Schlüssel
|
||||||
`secret.key`, 0600). Die Oberfläche zeigt sie nie an; in der Prozessliste stehen
|
`secret.key`, 0600). Die Oberfläche zeigt sie nie an; in der Prozessliste stehen
|
||||||
sie ebenfalls nie, weil `mount.cifs` eine kurzlebige Credentials-Datei bekommt.
|
sie ebenfalls nie, weil `mount.cifs` eine kurzlebige Credentials-Datei bekommt.
|
||||||
Ausgaben von `mount` werden geschwärzt, bevor sie protokolliert werden.
|
Ausgaben von `mount` werden geschwärzt, bevor sie protokolliert werden.
|
||||||
- **Sichere `secret.key` zusammen mit der Datenbank.** Ohne den Schlüssel sind die
|
- **Sichere `secret.key` zusammen mit der Datenbank.** Ohne den Schlüssel sind die
|
||||||
gespeicherten Passwörter verloren.
|
gespeicherten Passwörter verloren. Das erledigt die tägliche, mit Passphrase
|
||||||
|
verschlüsselte [Sicherung](docs/ANLEITUNG.md#sicherung) (*System → Sicherung*);
|
||||||
|
Wiederherstellen mit `netmover restore <datei>`.
|
||||||
- Anmeldung:
|
- Anmeldung:
|
||||||
- Ein Administratorkonto mit Argon2-Passworthash.
|
- Ein Administratorkonto mit Argon2-Passworthash.
|
||||||
- Sperre nach 5 Fehlversuchen.
|
- Sperre nach 5 Fehlversuchen.
|
||||||
@@ -388,7 +437,10 @@ Aufbau:
|
|||||||
| `mounting/` | `MountBackend` (Linux/Fake), Mount-Verwaltung, Test, Wiederherstellung, SFTP |
|
| `mounting/` | `MountBackend` (Linux/Fake), Mount-Verwaltung, Test, Wiederherstellung, SFTP |
|
||||||
| `mover/` | Scan, Stabilität, atomares Schreiben, Kollisionen, Buchführung, Regellauf |
|
| `mover/` | Scan, Stabilität, atomares Schreiben, Kollisionen, Buchführung, Regellauf |
|
||||||
| `runs.py`, `scheduler.py`, `health.py`, `notify.py` | Protokoll, Zeitsteuerung, Gesundheit, Webhook |
|
| `runs.py`, `scheduler.py`, `health.py`, `notify.py` | Protokoll, Zeitsteuerung, Gesundheit, Webhook |
|
||||||
| `admin.py`, `transfer.py`, `web/` | Formulare, Export/Import, Oberfläche |
|
| `admin.py`, `keys.py`, `transfer.py`, `web/` | Formulare, Schlüsselverwaltung, Export/Import, Oberfläche |
|
||||||
|
| `health.py`, `kuma.py` | Gesundheitsprüfung, Kennzahlen, Uptime-Kuma-Push |
|
||||||
|
| `backup.py` | tägliche verschlüsselte Sicherung, `netmover backup`/`restore` |
|
||||||
|
| `docs/` | Anleitung und Bilder; wird als Hilfe-Seite mit ausgeliefert |
|
||||||
|
|
||||||
Die gesamte Mover-Logik ist mit dem `FakeMountBackend` ohne root testbar. SFTP
|
Die gesamte Mover-Logik ist mit dem `FakeMountBackend` ohne root testbar. SFTP
|
||||||
wird gegen einen echten SFTP-Server im Testprozess getestet.
|
wird gegen einen echten SFTP-Server im Testprozess getestet.
|
||||||
@@ -400,6 +452,17 @@ NETMOVER_DATA_DIR=$PWD/data .venv/bin/alembic revision --autogenerate -m "…"
|
|||||||
```
|
```
|
||||||
|
|
||||||
Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt.
|
Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt.
|
||||||
|
`tests/test_migrations.py` stuft **jede** Migration automatisch auf einer
|
||||||
|
Datenbank mit verknüpften Daten hoch und wieder herunter – eine neue Migration ist
|
||||||
|
damit ohne weiteres Zutun abgedeckt; ergänzt sie Spalten mit Daten, gehört ein
|
||||||
|
eigener Test dazu.
|
||||||
|
|
||||||
|
### CI
|
||||||
|
|
||||||
|
`.gitea/workflows/ci.yml` führt bei jedem Push und Pull Request ruff, mypy,
|
||||||
|
shellcheck und alle Tests aus (Python 3.12 und 3.13). Voraussetzung: Actions sind
|
||||||
|
im Repository aktiviert und ein `act_runner` mit dem Label `ubuntu-latest` ist
|
||||||
|
registriert.
|
||||||
|
|
||||||
### Arbeitsweise
|
### Arbeitsweise
|
||||||
|
|
||||||
@@ -407,7 +470,7 @@ Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt
|
|||||||
(`feat:`, `fix:`, `docs:`, `test:`, `chore:` …).
|
(`feat:`, `fix:`, `docs:`, `test:`, `chore:` …).
|
||||||
- Jede nutzerrelevante Änderung kommt unter `[Unreleased]` in
|
- Jede nutzerrelevante Änderung kommt unter `[Unreleased]` in
|
||||||
[CHANGELOG.md](CHANGELOG.md). Bei einem Release:
|
[CHANGELOG.md](CHANGELOG.md). Bei einem Release:
|
||||||
1. Den Abschnitt `[Unreleased]` mit Versionsnummer versehen.
|
1. CI grün? Dann den Abschnitt `[Unreleased]` mit Versionsnummer versehen.
|
||||||
2. Die Version in `src/netmover/__init__.py` anheben.
|
2. Die Version in `src/netmover/__init__.py` anheben.
|
||||||
3. Einen Tag `vX.Y.Z` setzen.
|
3. Einen Tag `vX.Y.Z` setzen.
|
||||||
- Arbeitszeit und Entscheidungen stehen in [DEVLOG.md](DEVLOG.md).
|
- Arbeitszeit und Entscheidungen stehen in [DEVLOG.md](DEVLOG.md).
|
||||||
|
|||||||
+58
@@ -0,0 +1,58 @@
|
|||||||
|
# Roadmap
|
||||||
|
|
||||||
|
Was NetMover kann, woran gearbeitet wird und was geplant ist. Erledigte Punkte
|
||||||
|
stehen mit Version im [CHANGELOG](CHANGELOG.md); Details zu Entscheidungen im
|
||||||
|
[DEVLOG](DEVLOG.md).
|
||||||
|
|
||||||
|
Legende: ✅ erledigt · 🔨 in Arbeit · 📋 geplant · 💡 Idee (noch nicht entschieden)
|
||||||
|
|
||||||
|
## Geplant
|
||||||
|
|
||||||
|
| Status | Thema | Warum |
|
||||||
|
|---|---|---|
|
||||||
|
| 📋 | **SSH-Ausfälle bei Paperless klären**, Anbindungen auf SSH-Schlüssel umstellen | 18 kurze Ausfälle seit 11.10. („No existing session“, „protocol banner“) |
|
||||||
|
| 📋 | **Statusseite schneller bei hängenden Mounts**: Füllstand zwischenspeichern | Heute bis zu 2 s Wartezeit, wenn ein NFS-Server weg ist |
|
||||||
|
| 📋 | **Benachrichtigung bei Quarantäne** einer Datei | Unbeaufsichtigter Betrieb; heute über Kuma (`metrics.quarantined_24h`) sichtbar |
|
||||||
|
| 📋 | **Kompatibilitätszusage für 1.0**: Einstellungen, Export-Format, Update aus jeder 0.x | Danach keine brechenden Änderungen ohne 2.0 |
|
||||||
|
|
||||||
|
### Weg zu 1.0
|
||||||
|
|
||||||
|
0.10.0 Sicherung + CI → 0.11.0 mehrere Ziele (Kundenanforderung; Datenformat muss vor
|
||||||
|
1.0 feststehen) → 0.12.0 restliche Punkte oben → ein bis zwei Wochen ruhiger Betrieb →
|
||||||
|
1.0.0 ohne neue Funktionen.
|
||||||
|
|
||||||
|
**Entschieden:** Kein TLS im internen Netz – HTTP auf Port 8080 ist eine bewusste
|
||||||
|
Entscheidung (README, „Sicherheit“). Der Reverse Proxy bleibt dokumentiert für
|
||||||
|
nicht vertrauenswürdige Netze.
|
||||||
|
|
||||||
|
## Ideen
|
||||||
|
|
||||||
|
| Status | Thema | Anmerkung |
|
||||||
|
|---|---|---|
|
||||||
|
| 💡 | **Eine Regel, mehrere Ziele** je Quell-Unterordner | Heute: eine Regel pro Unterordner |
|
||||||
|
| 💡 | Metriken für Prometheus (`/metrics`) | Ergänzt `/healthz`, z. B. Dateien pro Tag |
|
||||||
|
| 💡 | Deinstallationsskript | Gegenstück zu `install.sh` |
|
||||||
|
| 💡 | Prüfsumme (SHA-256) je Regel: Kopie zurücklesen und vergleichen | Zusätzlicher Nachweis, z. B. für Rechnungen (GoBD) |
|
||||||
|
|
||||||
|
## Erledigt
|
||||||
|
|
||||||
|
| Version | Thema |
|
||||||
|
|---|---|
|
||||||
|
| 0.11.0 | Mehrere Ziele je Regel (bis zu 10), unabhängig erreichbar; Archiv wird zum weiteren Ziel |
|
||||||
|
| 0.10.0 | Tägliche verschlüsselte Sicherung + `netmover restore`; Kennzahlen für Kuma; CI in Gitea; Migrationstest-Pflicht |
|
||||||
|
| 0.9.0 | Regeln kopieren (Kopie zunächst deaktiviert) |
|
||||||
|
| 0.8.0 | Anbindungen kopieren inklusive Zugangsdaten |
|
||||||
|
| 0.7.0 | Archiv je Regel: Kopie ins Ziel und ins Archiv, Quelle erst danach löschen |
|
||||||
|
| 0.6.0 | Anmeldung je Netz: ohne Passwort / mit Passwort / abgewiesen, Schutz vor Aussperren |
|
||||||
|
| 0.5.0 | SFTP als Quelle; gespeicherte Zugangsdaten für SMB/SFTP-Passwörter (Seite „Zugangsdaten“); Anmeldung abschaltbar (vertrauenswürdige Netze / ganz) |
|
||||||
|
| 0.4.0 | Automatisch pausieren/fortsetzen bei nicht erreichbaren Anbindungen; Wartungsmodus; E-Mail-Benachrichtigung; eine Meldung pro ausgefallener Anbindung |
|
||||||
|
| 0.3.0 | Neues Menü (gruppierte Seitenleiste, Statusampel, Handy-Menü); Anleitung im Repo und als Hilfe-Seite; Roadmap; Uptime Kuma (Einzel-Endpunkte, Push-Monitor) |
|
||||||
|
| 0.2.1 | Migrationen in einer Transaktion und ohne Fremdschlüsselfehler auf befüllten Datenbanken |
|
||||||
|
| 0.2.0 | Schlüsselverwaltung für SFTP; `netmover-update` mit Sicherung; Installation als Git-Klon |
|
||||||
|
| 0.1.1 | Mounts im System-Namespace sichtbar; frische Dateiattribute für die Stabilitätsprüfung; minütlicher Neuversuch nach NAS-Ausfall |
|
||||||
|
| 0.1.0 | Erste Version: SMB/NFS/SFTP, Regeln mit sicherem Verschieben, Scheduler, Protokoll, Weboberfläche, systemd/Docker |
|
||||||
|
|
||||||
|
## Wünsche einbringen
|
||||||
|
|
||||||
|
Neue Punkte bitte als Issue in Gitea anlegen oder direkt hier eintragen. Bei
|
||||||
|
Umsetzung wandert ein Punkt nach „In Arbeit“ und mit dem Release nach „Erledigt“.
|
||||||
+1
-1
@@ -4,7 +4,7 @@
|
|||||||
services:
|
services:
|
||||||
netmover:
|
netmover:
|
||||||
build: .
|
build: .
|
||||||
image: netmover:0.1.1
|
image: netmover:0.11.0
|
||||||
container_name: netmover
|
container_name: netmover
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
cap_add:
|
cap_add:
|
||||||
|
|||||||
+17
-3
@@ -1,8 +1,10 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# NetMover auf Debian 13 installieren bzw. aktualisieren (als root ausführen).
|
# NetMover auf Debian 13 installieren bzw. aktualisieren (als root ausführen).
|
||||||
#
|
#
|
||||||
# ./deploy/install.sh # aus diesem Verzeichnis installieren
|
# git clone https://git.wilde-server.de/florian.wilde/netmover.git /opt/netmover/src
|
||||||
# ./deploy/install.sh 'git+https://git.wilde-server.de/florian.wilde/netmover.git@v0.1.0'
|
# /opt/netmover/src/deploy/install.sh
|
||||||
|
#
|
||||||
|
# Spätere Updates: netmover-update (siehe deploy/update.sh).
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SOURCE="${1:-$(cd "$(dirname "$0")/.." && pwd)}"
|
SOURCE="${1:-$(cd "$(dirname "$0")/.." && pwd)}"
|
||||||
@@ -16,7 +18,7 @@ fi
|
|||||||
echo "==> Pakete"
|
echo "==> Pakete"
|
||||||
export DEBIAN_FRONTEND=noninteractive
|
export DEBIAN_FRONTEND=noninteractive
|
||||||
apt-get update -qq
|
apt-get update -qq
|
||||||
apt-get install -y -qq python3 python3-venv cifs-utils nfs-common >/dev/null
|
apt-get install -y -qq python3 python3-venv cifs-utils nfs-common git >/dev/null
|
||||||
|
|
||||||
echo "==> Virtuelle Umgebung in $PREFIX/venv"
|
echo "==> Virtuelle Umgebung in $PREFIX/venv"
|
||||||
install -d -m 0755 "$PREFIX"
|
install -d -m 0755 "$PREFIX"
|
||||||
@@ -31,6 +33,18 @@ if [ ! -f /etc/netmover/netmover.env ]; then
|
|||||||
fi
|
fi
|
||||||
install -d -m 0755 /mnt/netmover
|
install -d -m 0755 /mnt/netmover
|
||||||
|
|
||||||
|
echo "==> Update-Skript und Kommandozeile"
|
||||||
|
install -m 0755 "$(dirname "$0")/update.sh" /usr/local/sbin/netmover-update
|
||||||
|
# netmover backup/restore/… mit derselben Konfiguration wie der Dienst
|
||||||
|
cat > /usr/local/sbin/netmover <<EOF
|
||||||
|
#!/bin/sh
|
||||||
|
set -a
|
||||||
|
[ -r /etc/netmover/netmover.env ] && . /etc/netmover/netmover.env
|
||||||
|
set +a
|
||||||
|
exec "$PREFIX/venv/bin/netmover" "\$@"
|
||||||
|
EOF
|
||||||
|
chmod 0755 /usr/local/sbin/netmover
|
||||||
|
|
||||||
echo "==> systemd"
|
echo "==> systemd"
|
||||||
install -m 0644 "$(dirname "$0")/netmover.service" /etc/systemd/system/netmover.service
|
install -m 0644 "$(dirname "$0")/netmover.service" /etc/systemd/system/netmover.service
|
||||||
systemctl daemon-reload
|
systemctl daemon-reload
|
||||||
|
|||||||
Executable
+153
@@ -0,0 +1,153 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# NetMover aktualisieren (als root). Wird von install.sh als
|
||||||
|
# /usr/local/sbin/netmover-update installiert.
|
||||||
|
#
|
||||||
|
# netmover-update # auf die neueste Version (neuester Tag vX.Y.Z)
|
||||||
|
# netmover-update v0.2.0 # auf eine bestimmte Version (auch zurück)
|
||||||
|
# netmover-update --list # verfügbare Versionen anzeigen
|
||||||
|
#
|
||||||
|
# Ablauf: Tags holen → Datenbank und Schlüssel sichern → Version auschecken →
|
||||||
|
# install.sh (installiert, migriert beim Start, startet den Dienst neu) →
|
||||||
|
# Gesundheit prüfen. Die Mounts bleiben dabei eingehängt.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
SRC="${NETMOVER_SRC:-/opt/netmover/src}"
|
||||||
|
VENV=/opt/netmover/venv
|
||||||
|
ENV_FILE=/etc/netmover/netmover.env
|
||||||
|
BACKUP_ROOT=/var/backups/netmover
|
||||||
|
KEEP_BACKUPS=10
|
||||||
|
|
||||||
|
if [ "$(id -u)" -ne 0 ]; then
|
||||||
|
echo "Bitte als root ausführen." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if [ ! -d "$SRC/.git" ]; then
|
||||||
|
echo "Kein Git-Klon unter $SRC. Einmalig einrichten:" >&2
|
||||||
|
echo " git clone https://git.wilde-server.de/florian.wilde/netmover.git $SRC" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Einstellungen des Dienstes (Port, Datenverzeichnis)
|
||||||
|
NETMOVER_PORT=8080
|
||||||
|
NETMOVER_DATA_DIR=/var/lib/netmover
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
[ -f "$ENV_FILE" ] && . "$ENV_FILE"
|
||||||
|
|
||||||
|
cd "$SRC"
|
||||||
|
git fetch --tags --quiet origin
|
||||||
|
|
||||||
|
if [ "${1:-}" = "--list" ]; then
|
||||||
|
git tag -l 'v*' --sort=-v:refname
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
TARGET="${1:-$(git tag -l 'v*' --sort=-v:refname | head -n 1)}"
|
||||||
|
if ! git rev-parse -q --verify "refs/tags/$TARGET" >/dev/null; then
|
||||||
|
echo "Version $TARGET gibt es nicht. Verfügbar:" >&2
|
||||||
|
git tag -l 'v*' --sort=-v:refname | head -n 10 >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
CURRENT="v$("$VENV/bin/netmover" version 2>/dev/null || echo "?")"
|
||||||
|
echo "Installiert: $CURRENT"
|
||||||
|
echo "Ziel: $TARGET"
|
||||||
|
if [ "$CURRENT" = "$TARGET" ] && [ "${FORCE:-0}" != "1" ]; then
|
||||||
|
echo "Bereits auf dem gewünschten Stand (FORCE=1 erzwingt eine Neuinstallation)."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
NEWEST="$(printf '%s\n%s\n' "$CURRENT" "$TARGET" | sort -V | tail -n 1)"
|
||||||
|
if [ "$NEWEST" = "$CURRENT" ] && [ "$CURRENT" != "$TARGET" ]; then
|
||||||
|
echo
|
||||||
|
echo "!! Zurückstufen auf eine ältere Version: Wurde die Datenbank inzwischen migriert,"
|
||||||
|
echo " startet die alte Version nicht. Dann vorher die Sicherung von *vor* dem Update"
|
||||||
|
echo " zurückspielen (siehe $BACKUP_ROOT, Anleitung im README)."
|
||||||
|
printf " Trotzdem fortfahren? [j/N] "
|
||||||
|
read -r answer
|
||||||
|
[ "$answer" = "j" ] || [ "$answer" = "J" ] || exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$NEWEST" = "$TARGET" ]; then
|
||||||
|
echo
|
||||||
|
echo "==> Änderungen laut Changelog"
|
||||||
|
git show "$TARGET:CHANGELOG.md" | awk -v cur="${CURRENT#v}" '
|
||||||
|
/^## \[/ { if (index($0, "[" cur "]")) exit; show = ($0 !~ /Unreleased/) }
|
||||||
|
show' | head -n 60
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "==> Sicherung"
|
||||||
|
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||||
|
BACKUP="$BACKUP_ROOT/$STAMP-$CURRENT"
|
||||||
|
install -d -m 0700 "$BACKUP_ROOT" "$BACKUP"
|
||||||
|
# Konsistente Kopie auch bei laufendem Dienst (SQLite-Online-Backup)
|
||||||
|
"$VENV/bin/python" - "$NETMOVER_DATA_DIR/netmover.db" "$BACKUP/netmover.db" <<'EOF'
|
||||||
|
import sqlite3, sys
|
||||||
|
src, dst = sqlite3.connect(sys.argv[1]), sqlite3.connect(sys.argv[2])
|
||||||
|
with dst:
|
||||||
|
src.backup(dst)
|
||||||
|
src.close(); dst.close()
|
||||||
|
EOF
|
||||||
|
for f in secret.key session.key; do
|
||||||
|
[ -f "$NETMOVER_DATA_DIR/$f" ] && cp -p "$NETMOVER_DATA_DIR/$f" "$BACKUP/"
|
||||||
|
done
|
||||||
|
[ -f "$ENV_FILE" ] && cp -p "$ENV_FILE" "$BACKUP/"
|
||||||
|
chmod 0600 "$BACKUP"/*
|
||||||
|
echo " $BACKUP"
|
||||||
|
# alte Sicherungen aufräumen
|
||||||
|
find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -print | sort \
|
||||||
|
| head -n "-$KEEP_BACKUPS" | xargs -r rm -rf --
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "==> Version $TARGET auschecken und installieren"
|
||||||
|
git -c advice.detachedHead=false checkout --quiet "$TARGET"
|
||||||
|
"$SRC/deploy/install.sh" "$SRC" >/tmp/netmover-update.log 2>&1 || {
|
||||||
|
echo "install.sh ist fehlgeschlagen – Ausgabe: /tmp/netmover-update.log" >&2
|
||||||
|
tail -n 20 /tmp/netmover-update.log >&2
|
||||||
|
FAILED=1
|
||||||
|
}
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "==> Prüfung"
|
||||||
|
OK=0
|
||||||
|
if [ "${FAILED:-0}" != "1" ]; then
|
||||||
|
i=0
|
||||||
|
while [ $i -lt 30 ]; do
|
||||||
|
if "$VENV/bin/python" - "$NETMOVER_PORT" <<'EOF'
|
||||||
|
import json, sys, urllib.error, urllib.request
|
||||||
|
url = f"http://127.0.0.1:{sys.argv[1]}/healthz"
|
||||||
|
try:
|
||||||
|
body = urllib.request.urlopen(url, timeout=3).read()
|
||||||
|
except urllib.error.HTTPError as exc: # 503 = läuft, meldet aber Probleme
|
||||||
|
body = exc.read()
|
||||||
|
except OSError:
|
||||||
|
sys.exit(1)
|
||||||
|
data = json.loads(body)
|
||||||
|
print(f" Gesundheit: {data['status']}")
|
||||||
|
for p in data.get("problems", []):
|
||||||
|
print(f" – {p}")
|
||||||
|
EOF
|
||||||
|
then
|
||||||
|
OK=1
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
i=$((i + 1))
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$OK" = "1" ] && systemctl is-active --quiet netmover; then
|
||||||
|
echo " Dienst läuft: $("$VENV/bin/netmover" version)"
|
||||||
|
echo
|
||||||
|
echo "Fertig. Sicherung: $BACKUP"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "!! Der Dienst läuft nach dem Update nicht richtig." >&2
|
||||||
|
echo " Log: journalctl -u netmover -n 50" >&2
|
||||||
|
echo " Zurück auf $CURRENT (Datenbank aus der Sicherung, weil sie migriert sein kann):" >&2
|
||||||
|
echo " systemctl stop netmover" >&2
|
||||||
|
echo " cp $BACKUP/netmover.db $NETMOVER_DATA_DIR/netmover.db" >&2
|
||||||
|
echo " rm -f $NETMOVER_DATA_DIR/netmover.db-wal $NETMOVER_DATA_DIR/netmover.db-shm" >&2
|
||||||
|
echo " netmover-update $CURRENT" >&2
|
||||||
|
exit 1
|
||||||
@@ -0,0 +1,589 @@
|
|||||||
|
# NetMover – Anleitung {#anleitung}
|
||||||
|
|
||||||
|
NetMover holt Dateien regelmäßig aus einem Ordner und bringt sie sicher in einen
|
||||||
|
anderen. Typisch: Scans landen auf dem NAS, NetMover schiebt sie alle 30 Sekunden
|
||||||
|
in den Eingangsordner von Paperless.
|
||||||
|
|
||||||
|
Diese Anleitung erklärt die Bedienung. Installation, Updates und technische Details
|
||||||
|
stehen im [README](../README.md).
|
||||||
|
|
||||||
|
**Inhalt:** [Begriffe](#begriffe) · [Erste Anmeldung](#erste-anmeldung) ·
|
||||||
|
[Zugangsdaten](#zugangsdaten) · [Anbindungen](#anbindungen) · [Regeln](#regeln) · [Weitere Ziele](#weitere-ziele) ·
|
||||||
|
[Probelauf](#probelauf) · [Status](#status) · [Wartungsmodus](#wartungsmodus) · [Protokoll](#protokoll) ·
|
||||||
|
[Einstellungen](#einstellungen) · [Anmeldung](#anmeldung) · [Uptime Kuma](#uptime-kuma) ·
|
||||||
|
[Sicherung](#sicherung) · [Beispiele](#beispiele) · [Häufige Fragen](#fragen)
|
||||||
|
|
||||||
|
## Die vier Begriffe {#begriffe}
|
||||||
|
|
||||||
|
| Begriff | Was es ist | Beispiel |
|
||||||
|
|---|---|---|
|
||||||
|
| **Anbindung** | Ein Ort, an den NetMover herankommt: eine SMB-Freigabe, ein NFS-Export oder ein Server per SFTP | „NAS“ = `//10.0.10.6/Scans` |
|
||||||
|
| **Zugangsdaten** | Gespeicherter Benutzer mit Passwort oder SSH-Schlüssel, in mehreren Anbindungen nutzbar | „NAS-Scanner“, „paperless-lxc“ |
|
||||||
|
| **Regel** | „Alle N Sekunden aus Anbindung A nach Anbindung B“ – mit Filter, Sicherheitsprüfungen und Verhalten bei gleichen Namen | „Scans → Paperless“ |
|
||||||
|
| **Lauf** | Eine Ausführung einer Regel; steht mit allen Dateien im Protokoll | 23:41:07, 3 Dateien verschoben |
|
||||||
|
|
||||||
|
Die Reihenfolge beim Einrichten ist immer dieselbe – und so ist auch das Menü
|
||||||
|
aufgebaut: **(Zugangsdaten) → Anbindungen → Regeln → Status und Protokoll beobachten.**
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Erste Anmeldung {#erste-anmeldung}
|
||||||
|
|
||||||
|
Beim allerersten Aufruf verlangt NetMover ein Administratorpasswort (mindestens 10
|
||||||
|
Zeichen) und einen **Einrichtungscode**. Der Code steht im Log des Dienstes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
journalctl -u netmover | grep setup.required
|
||||||
|
```
|
||||||
|
|
||||||
|
Der Code verhindert, dass jemand anderes im Netz zuerst die Seite öffnet und das
|
||||||
|
Passwort festlegt. Passwort ändern: *System → Einstellungen*.
|
||||||
|
|
||||||
|
## Anbindungen {#anbindungen}
|
||||||
|
|
||||||
|
*Verbindungen → Anbindungen → Neue Anbindung*
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Typ wählen {#anbindung-typ}
|
||||||
|
|
||||||
|
| Typ | Wofür | Kann |
|
||||||
|
|---|---|---|
|
||||||
|
| **SMB/CIFS** | Windows-Freigaben, Synology, QNAP, … | Quelle und Ziel |
|
||||||
|
| **NFS 4 / NFS 3** | Linux-Server, NAS mit NFS | Quelle und Ziel |
|
||||||
|
| **SFTP** | Server, die nur per SSH erreichbar sind (z. B. ein LXC mit Paperless) | Quelle und Ziel |
|
||||||
|
|
||||||
|
SMB- und NFS-Anbindungen werden **eingehängt** und erscheinen unter
|
||||||
|
`/mnt/netmover/<name>`. SFTP wird nicht eingehängt, NetMover verbindet sich bei
|
||||||
|
jedem Lauf. Als **Quelle** gelten bei SFTP dieselben Sicherheitsprüfungen; weil
|
||||||
|
SFTP-Server Änderungszeiten nur sekundengenau melden, sollte das
|
||||||
|
Stabilitätsfenster dort mindestens 2 Sekunden betragen.
|
||||||
|
|
||||||
|
### Felder {#anbindung-felder}
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **Name** | Frei wählbar. Bestimmt auch das Verzeichnis unter `/mnt/netmover` (wird beim Anlegen festgelegt). |
|
||||||
|
| **Server** | IP-Adresse oder Hostname, z. B. `10.0.10.6` |
|
||||||
|
| **Freigabe / Pfad** | SMB: Freigabename, optional mit Unterordner (`Scans` oder `Scans/privat`). NFS: Exportpfad (`/volume1/scans`). SFTP: absoluter Ordner auf dem Server (`/opt/paperless_data/consume`). |
|
||||||
|
| **Benutzername / Passwort** | Bei SMB leer lassen für Gastzugang. Beim Bearbeiten heißt ein leeres Passwortfeld „unverändert“ – Passwörter werden nie angezeigt. |
|
||||||
|
| **Zugangsdaten / Anmeldeart** | Gespeicherte Zugangsdaten auswählen oder eigene Angaben nur für diese Anbindung; bei SFTP zusätzlich SSH-Schlüssel oder Passwort, siehe [Zugangsdaten](#zugangsdaten). |
|
||||||
|
| **uid, gid, file_mode, dir_mode** (SMB) | Wem die Dateien im eingehängten Ordner gehören. Meist leer lassen. |
|
||||||
|
| **Zusatzoptionen** | Für Sonderfälle, z. B. `vers=3.0` oder `nohandlecache`. Nur erlaubte Optionen; Zugangsdaten setzt NetMover selbst. |
|
||||||
|
| **Beim Start automatisch einhängen** | Nach einem Neustart des Servers wieder einhängen. Fällt das NAS aus, versucht NetMover es außerdem jede Minute erneut. |
|
||||||
|
| **Aktiv** | Deaktivierte Anbindungen werden von Regeln nicht benutzt. |
|
||||||
|
|
||||||
|
### Kopieren {#anbindung-kopieren}
|
||||||
|
|
||||||
|
Für eine ähnliche Anbindung – etwa eine zweite Freigabe auf demselben NAS – bei der
|
||||||
|
Vorlage auf **Kopieren** klicken. Das Formular ist mit allen Werten vorausgefüllt
|
||||||
|
(Name „… (Kopie)“); meist muss nur die Freigabe geändert werden.
|
||||||
|
|
||||||
|
- **Zugangsdaten werden übernommen**, solange das Passwortfeld leer bleibt. Die Kopie
|
||||||
|
bekommt ein eigenes Exemplar – das Löschen der Vorlage betrifft sie nicht.
|
||||||
|
Gespeicherte Zugangsdaten sind einfach vorausgewählt.
|
||||||
|
- Ein **eigener SSH-Schlüssel** der Vorlage wird weiter benutzt; ein neuer Eintrag in
|
||||||
|
`authorized_keys` ist nicht nötig.
|
||||||
|
- **Testen** funktioniert im Kopie-Formular bereits mit den übernommenen Zugangsdaten.
|
||||||
|
|
||||||
|
Tipp: Nutzen mehrere Anbindungen dieselben Zugangsdaten, lohnt sich ein Eintrag unter
|
||||||
|
[Zugangsdaten](#zugangsdaten) – dann gilt ein neues Passwort sofort überall.
|
||||||
|
|
||||||
|
### Testen {#anbindung-testen}
|
||||||
|
|
||||||
|
**Testen** prüft drei Dinge einzeln und sagt im Klartext, was nicht klappt:
|
||||||
|
|
||||||
|
1. **Einhängen** bzw. **Verbinden** (SFTP)
|
||||||
|
2. **Lesen** – Inhalt des Ordners auflisten
|
||||||
|
3. **Schreiben** – Testdatei anlegen, Eigentümer prüfen, wieder löschen
|
||||||
|
|
||||||
|
Daraus folgt, wofür die Anbindung taugt: **nur Quelle, kopieren** (nur lesen),
|
||||||
|
**Quelle mit Löschen** (lesen und löschen – nötig zum Verschieben) oder **Ziel**.
|
||||||
|
Man kann schon im Formular testen, bevor man speichert.
|
||||||
|
|
||||||
|
Bei SFTP hält der erste erfolgreiche Test den **Hostschlüssel** des Servers fest. Den
|
||||||
|
angezeigten Fingerabdruck einmal auf dem Server vergleichen:
|
||||||
|
`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`. Ändert er sich später,
|
||||||
|
verweigert NetMover die Verbindung (Schutz gegen untergeschobene Server). Wurde der
|
||||||
|
Server neu aufgesetzt: bei der Anbindung **Hostschlüssel zurücksetzen** und neu testen.
|
||||||
|
|
||||||
|
### Zustände {#anbindung-zustaende}
|
||||||
|
|
||||||
|
| Anzeige | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **eingehängt** / **erreichbar** | alles gut |
|
||||||
|
| **ausgehängt** / **nicht geprüft** | nicht in Benutzung |
|
||||||
|
| **gestört** | Antwortet nicht oder meldet „Stale file handle“; NetMover hängt gerade neu ein |
|
||||||
|
| **Fehler** | Wiederherstellung gescheitert; NetMover versucht es jede Minute weiter (bei „automatisch einhängen“) |
|
||||||
|
|
||||||
|
Regeln, deren Anbindung nicht verfügbar ist, **brechen ab, ohne etwas anzufassen**.
|
||||||
|
Es wird nie in ein leeres Verzeichnis geschrieben, nur weil die Freigabe fehlt.
|
||||||
|
Solche Regeln **warten** automatisch, bis die Anbindung zurück ist (siehe
|
||||||
|
[Status](#status)), und laufen dann von selbst weiter.
|
||||||
|
|
||||||
|
## Zugangsdaten {#zugangsdaten}
|
||||||
|
|
||||||
|
*Verbindungen → Zugangsdaten*
|
||||||
|
|
||||||
|
Benutzer und Passwörter für Freigaben sowie SSH-Schlüssel einmal hinterlegen und in
|
||||||
|
beliebig vielen Anbindungen auswählen. Passwörter und private Schlüssel liegen
|
||||||
|
verschlüsselt in der Datenbank und werden nie angezeigt.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Passwörter {#zugangsdaten-passwoerter}
|
||||||
|
|
||||||
|
Für SMB-Freigaben und SFTP-Server mit Passwort: Name, Benutzer, Passwort und
|
||||||
|
optional Domäne. In der Anbindung unter *Zugangsdaten* auswählen statt „eigene
|
||||||
|
Angaben“. **Ändern** setzt Benutzer, Domäne oder ein neues Passwort – das gilt sofort
|
||||||
|
für alle Anbindungen, die den Eintrag nutzen (praktisch, wenn das NAS-Passwort
|
||||||
|
wechselt).
|
||||||
|
|
||||||
|
### SSH-Schlüssel {#schluessel}
|
||||||
|
|
||||||
|
Für SFTP ist ein Schlüssel sicherer als ein Passwort. Einen Schlüssel kann man für
|
||||||
|
mehrere Anbindungen verwenden; den Benutzernamen legt jede Anbindung fest.
|
||||||
|
|
||||||
|
**Neuen Schlüssel erzeugen** – Namen eingeben, *Ed25519-Schlüssel erzeugen*. Dann:
|
||||||
|
|
||||||
|
1. Bei dem Schlüssel *Öffentlicher Schlüssel / authorized_keys* aufklappen und die
|
||||||
|
**authorized_keys-Zeile** kopieren. Sie beginnt mit
|
||||||
|
`restrict,command="/usr/lib/openssh/sftp-server"` – damit darf der Schlüssel
|
||||||
|
**nur SFTP**, keine Shell.
|
||||||
|
2. Auf dem Server an die Datei `~/.ssh/authorized_keys` des Benutzers anhängen
|
||||||
|
(Ordner `~/.ssh` mit Rechten `700`, Datei `600`, Eigentümer der Benutzer).
|
||||||
|
3. In der Anbindung *Anmeldeart: SSH-Schlüssel* und den Schlüssel wählen, **Testen**.
|
||||||
|
|
||||||
|
**Vorhandenen Schlüssel importieren** – privaten Schlüssel einfügen (OpenSSH- oder
|
||||||
|
PEM-Format; Ed25519, ECDSA oder RSA ab 2048 Bit). Hat er eine Passphrase, diese
|
||||||
|
angeben: Sie wird nur zum Entschlüsseln benutzt.
|
||||||
|
|
||||||
|
Gespeicherte Zugangsdaten lassen sich nur löschen, wenn sie keine Anbindung mehr nutzt.
|
||||||
|
|
||||||
|
## Regeln {#regeln}
|
||||||
|
|
||||||
|
*Automatisierung → Regeln → Neue Regel*
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Quelle und Ziel {#regel-quelle-ziel}
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **Quelle: Anbindung, Unterordner** | Woher. Unterordner relativ zur Freigabe; leer = Wurzel der Freigabe. |
|
||||||
|
| **Filter** | Glob: Dateiname, Groß/klein egal, mehrere mit `;` (`*.pdf;*.jpg`). Regulärer Ausdruck: gegen den relativen Pfad (`^rechnungen/.*\.pdf$`). |
|
||||||
|
| **Unterordner einbeziehen** | Auch in Unterordnern suchen; die Ordnerstruktur bleibt im Ziel erhalten. |
|
||||||
|
| **Ziel** | Eine Anbindung mit Zielordner (wird angelegt) oder ein **lokaler Pfad** auf dem NetMover-Server (muss existieren). |
|
||||||
|
| **Aktion** | **verschieben** (Quelle wird nach erfolgreicher Übertragung gelöscht) oder **kopieren** (Quelle bleibt; jede Dateiversion wird nur einmal kopiert). |
|
||||||
|
| **Wenn Datei existiert** | **umbenennen** (`scan.pdf` → `scan-1.pdf`, `scan-2.pdf` …), **überspringen** oder **überschreiben**. |
|
||||||
|
|
||||||
|
### Weitere Ziele (Archiv, Buchhaltung, …) {#weitere-ziele}
|
||||||
|
|
||||||
|
<span id="archiv"></span>Neben dem **Hauptziel** kann jede Regel bis zu **9 weitere
|
||||||
|
Ziele** beliefern – z. B. eine Rechnung ins DMS (Hauptziel), in den Importordner der
|
||||||
|
Buchhaltung und ins Archiv. **Die Quelle wird erst gelöscht, wenn alle Ziele die Datei
|
||||||
|
haben.**
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **Ziel 2 … 10** | Anbindung (SMB, NFS, SFTP) oder lokaler Pfad; „– kein weiteres Ziel –“ = Platz frei. Sobald ein Platz belegt ist, erscheint der nächste. |
|
||||||
|
| **Bezeichnung** | erscheint in Protokoll und Probelauf, z. B. „Archiv“ oder „Buchhaltung“; leer = Name der Anbindung |
|
||||||
|
| **Ordner** | Anbindung: relativer Unterordner (wird angelegt). Lokal: absoluter Pfad, muss existieren. |
|
||||||
|
| **Ablage** | **wie in der Quelle** oder **nach Datum** – `2026/10/scan001.pdf` (Monat der Verarbeitung). Für Ordner, die eine Software abholt (Buchhaltung), „wie in der Quelle“ wählen. |
|
||||||
|
|
||||||
|
Ein Ziel entfernen: Platz auf „– kein weiteres Ziel –“ stellen und speichern. Die
|
||||||
|
übrigen rücken nach.
|
||||||
|
|
||||||
|
So läuft es pro Datei:
|
||||||
|
|
||||||
|
1. **Zuerst die weiteren Ziele** – der Reihe nach, mit denselben Sicherungen wie beim
|
||||||
|
Hauptziel (`.part`-Datei, Größenprüfung, atomares Umbenennen).
|
||||||
|
2. **Dann das Hauptziel.**
|
||||||
|
3. **Dann erst** wird die Quelle gelöscht (bei „verschieben“).
|
||||||
|
|
||||||
|
Die weiteren Ziele kommen zuerst, weil ein Hauptziel wie Paperless Dateien sofort
|
||||||
|
abholt: Scheitert ein weiteres Ziel, ist am Hauptziel noch nichts angekommen, und die
|
||||||
|
Datei bleibt in der Quelle. NetMover merkt sich je Ziel, welche Datei es schon hat –
|
||||||
|
ein neuer Versuch legt nirgends eine zweite Kopie an.
|
||||||
|
|
||||||
|
**Ein Ziel ist nicht erreichbar** (NAS aus, Server weg): Die übrigen Ziele bekommen die
|
||||||
|
Datei trotzdem, auch das Hauptziel. Die Quelle bleibt liegen, bis das fehlende Ziel
|
||||||
|
zurück ist; dann wird es nachbeliefert, und erst danach wird die Quelle gelöscht. Die
|
||||||
|
Regel gilt solange als „wartet“ (Statusseite, `/healthz`, E-Mail beim Ausfall der
|
||||||
|
Anbindung). Weitere Läufe während des Ausfalls erzeugen keine Protokolleinträge.
|
||||||
|
|
||||||
|
In weiteren Zielen wird **nie überschrieben**: Gleiche Namen werden immer umbenannt
|
||||||
|
(`scan-1.pdf`), egal was die Regel fürs Hauptziel vorsieht. Im Kopiermodus bekommt jedes
|
||||||
|
Ziel eine Kopie, die Quelle bleibt; ein neu hinzugefügtes Ziel bekommt dabei auch die
|
||||||
|
Dateien, die schon in der Quelle liegen.
|
||||||
|
|
||||||
|
Dateien werden byte-genau kopiert – bei E-Rechnungen (ZUGFeRD/Factur-X) bleibt das
|
||||||
|
eingebettete XML unverändert.
|
||||||
|
|
||||||
|
### Sicherheit und Grenzen {#regel-sicherheit}
|
||||||
|
|
||||||
|
| Feld | Bedeutung | Empfehlung |
|
||||||
|
|---|---|---|
|
||||||
|
| **Intervall** | Wie oft geprüft wird (mind. 10 s) | 30 s |
|
||||||
|
| **Stabilitätsfenster** | Größe **und** Änderungszeit müssen so lange gleich bleiben. Schützt vor halb geschriebenen Scans. | 5 s, bei langsamen Scannern mehr |
|
||||||
|
| **Mindestalter** | Dateien erst ab diesem Alter anfassen | 0 |
|
||||||
|
| **Max. Größe** | Größere Dateien überspringen | leer |
|
||||||
|
| **Zeitlimit ohne Fortschritt** | Bricht einen Lauf ab, der hängt (z. B. NFS-Server weg). Große Dateien, die stetig übertragen werden, sind nicht betroffen. | 120 s |
|
||||||
|
|
||||||
|
Immer ignoriert werden versteckte Dateien (`.datei`), unfertige Übertragungen
|
||||||
|
(`.part`, `.tmp`, `.crdownload`, `.filepart`, `.!ut`), Verknüpfungen und
|
||||||
|
Synology-Verwaltungsordner (`@eaDir`, `#recycle`).
|
||||||
|
|
||||||
|
### Rechte und Quarantäne {#regel-rechte}
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **Eigentümer setzen** | `uid:gid`, z. B. `1000:1000`. Auf SMB-Zielen wirkungslos (dort bestimmen die Mount-Optionen den Eigentümer), auf NFS nur ohne Root-Squash. |
|
||||||
|
| **Modus setzen** | z. B. `0644`; leer = wie die Quelldatei |
|
||||||
|
| **Quarantäneordner** | Dateien, die mehrmals hintereinander scheitern, werden dorthin verschoben (relativ zur Quell-Anbindung), statt es endlos zu versuchen. Leer = keine Quarantäne. |
|
||||||
|
|
||||||
|
### Regel kopieren {#regel-kopieren}
|
||||||
|
|
||||||
|
Für eine ähnliche Regel – etwa für einen zweiten Quell-Unterordner – bei der Vorlage
|
||||||
|
auf **Kopieren** klicken. Alle Einstellungen (Filter, Ziele, Sicherheit,
|
||||||
|
Rechte) sind übernommen, der Name lautet „… (Kopie)“.
|
||||||
|
|
||||||
|
Die Kopie ist zunächst **deaktiviert**: Mit gleicher Quelle würde sie sonst sofort mit
|
||||||
|
dem Original um dieselben Dateien konkurrieren. Also Quelle bzw. Ziel anpassen,
|
||||||
|
speichern, **Probelauf**, dann **Aktivieren**. Protokoll und Buchführung beginnen bei
|
||||||
|
der Kopie neu.
|
||||||
|
|
||||||
|
### Probelauf {#probelauf}
|
||||||
|
|
||||||
|
**Probelauf** zeigt in einer Tabelle, welche Datei wohin ginge – **ohne etwas zu
|
||||||
|
verändern**. Vor dem ersten Aktivieren einer Regel immer einmal machen.
|
||||||
|
|
||||||
|
Das Häkchen **Nur Probelauf** in der Regel lässt sie regelmäßig laufen, ohne etwas zu
|
||||||
|
verändern – gut zum Beobachten über ein paar Tage.
|
||||||
|
|
||||||
|
### Wie verschoben wird {#wie-verschoben-wird}
|
||||||
|
|
||||||
|
Für jede Datei:
|
||||||
|
|
||||||
|
1. zweimal messen (Abstand = Stabilitätsfenster) – nur unveränderte Dateien weiter,
|
||||||
|
2. freien Platz am Ziel prüfen (Größe + 5 %),
|
||||||
|
3. unter einem versteckten Namen `.netmover-….part` ins Ziel schreiben und auf die
|
||||||
|
Platte zwingen,
|
||||||
|
4. prüfen, dass sich die Quelle in der Zwischenzeit nicht verändert hat,
|
||||||
|
5. in einem Schritt auf den Endnamen umbenennen – ohne etwas zu überschreiben,
|
||||||
|
6. **erst dann** die Quelle löschen.
|
||||||
|
|
||||||
|
Geht unterwegs etwas schief, bleibt die Quelle liegen und im Ziel steht nichts
|
||||||
|
Halbes. Ein Programm, das den Zielordner überwacht (Paperless), sieht nie eine
|
||||||
|
unfertige Datei.
|
||||||
|
|
||||||
|
## Status {#status}
|
||||||
|
|
||||||
|
*Übersicht → Status* ist die Startseite und aktualisiert sich alle 15 Sekunden.
|
||||||
|
|
||||||
|
- **Ampel** oben links: grün = alles in Ordnung, rot = mindestens ein Problem.
|
||||||
|
- **Anbindungen** mit Zustand, Füllstand und letztem Fehler.
|
||||||
|
- **Regeln** mit letztem Ergebnis, **letztem Erfolg** und nächstem Lauf.
|
||||||
|
|
||||||
|
Eine Regel wird **gelb**, wenn ihr letzter erfolgreicher Lauf länger als das
|
||||||
|
Dreifache ihres Intervalls zurückliegt. So fällt auf, wenn NetMover still ausfällt
|
||||||
|
– statt dass es aussieht, als gäbe es nichts zu tun.
|
||||||
|
|
||||||
|
### Regel wartet auf eine Anbindung {#wartet}
|
||||||
|
|
||||||
|
Ist die Quelle oder das Ziel einer Regel nicht erreichbar (NAS neu gestartet, Server
|
||||||
|
aus), **pausiert die Regel automatisch**: Sie zeigt „wartet seit …“ mit dem Grund.
|
||||||
|
|
||||||
|
- Im Protokoll steht **ein** Eintrag „Pausiert, bis die Anbindung zurück ist“ – nicht
|
||||||
|
alle 30 Sekunden ein neuer Fehler.
|
||||||
|
- Es kommt **eine** Benachrichtigung pro Anbindung, mit der Liste der betroffenen
|
||||||
|
Regeln.
|
||||||
|
- Sobald die Anbindung zurück ist, **läuft die Regel von selbst weiter**; der nächste
|
||||||
|
Lauf vermerkt „Fortgesetzt nach … Wartezeit“, und es kommt eine Entwarnung.
|
||||||
|
|
||||||
|
Man muss also nach einem NAS-Neustart nichts tun.
|
||||||
|
|
||||||
|
### Wartungsmodus {#wartungsmodus}
|
||||||
|
|
||||||
|
Der Knopf **Wartungsmodus** auf der Statusseite pausiert **alle geplanten Läufe** –
|
||||||
|
etwa während eines NAS-Updates. Oben erscheint dann gelb „Wartungsmodus“. „Jetzt
|
||||||
|
ausführen“ geht weiterhin. Für Regeln meldet `/healthz` in dieser Zeit keinen Alarm
|
||||||
|
(Anbindungen werden weiter geprüft). Mit **Wartungsmodus beenden** läuft alles
|
||||||
|
wieder nach Plan.
|
||||||
|
|
||||||
|
## Protokoll {#protokoll}
|
||||||
|
|
||||||
|
*Automatisierung → Protokoll* zeigt jeden Lauf: Zeitpunkt, Dauer, gesehen /
|
||||||
|
übertragen / übersprungen / Fehler. *Einzeldateien* aufklappen zeigt jede Datei mit
|
||||||
|
Ziel und Ergebnis. Leere Läufe sind ausgeblendet („auch leere Läufe“ zeigt sie).
|
||||||
|
|
||||||
|
Häufige Einträge:
|
||||||
|
|
||||||
|
| Eintrag | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| *wird noch geschrieben* | Datei ändert sich noch – im nächsten Lauf erneut |
|
||||||
|
| *Datei ist gerade in Benutzung* | Ein anderes Programm hält sie gesperrt |
|
||||||
|
| *existiert bereits im Ziel* | Regel steht auf „überspringen“ |
|
||||||
|
| *Quelle hat sich während der Übertragung geändert* | Abgebrochen, nichts gelöscht; nächster Lauf versucht es erneut |
|
||||||
|
| *n bereits übertragen* | Kopiermodus: diese Dateiversionen sind schon kopiert |
|
||||||
|
| *Termin übersprungen – voriger Lauf läuft noch* | Ein Lauf dauert länger als das Intervall |
|
||||||
|
| *Pausiert, bis die Anbindung zurück ist* | Quelle oder Ziel nicht erreichbar; die Regel wartet, siehe [Regel wartet](#wartet) |
|
||||||
|
| *Fortgesetzt nach … Wartezeit* | Die Anbindung ist zurück, die Regel läuft wieder |
|
||||||
|
|
||||||
|
Protokolle werden nach der eingestellten Frist gelöscht (Standard 30 Tage).
|
||||||
|
|
||||||
|
## Einstellungen {#einstellungen}
|
||||||
|
|
||||||
|
*System → Einstellungen*
|
||||||
|
|
||||||
|
### Benachrichtigungen {#benachrichtigungen}
|
||||||
|
|
||||||
|
NetMover meldet Probleme per **E-Mail** und/oder an einen **Webhook** (z. B. ntfy,
|
||||||
|
Gotify, Home Assistant). Beides lässt sich gleichzeitig nutzen.
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **Benachrichtigen** | nie / bei jedem fehlgeschlagenen Lauf / erst nach N Fehlschlägen in Folge. Fällt eine **Anbindung** aus, kommt die Meldung immer sofort (einmal, mit den betroffenen Regeln). |
|
||||||
|
| **Entwarnung** | Nachricht, wenn eine Regel oder Anbindung wieder funktioniert |
|
||||||
|
|
||||||
|
**E-Mail:**
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **SMTP-Server, Port, Verschlüsselung** | z. B. `smtp.example.org`, `587` mit STARTTLS oder `465` mit SSL/TLS. „keine“ nur für interne Relays. Zertifikate werden immer geprüft. |
|
||||||
|
| **Benutzer, Passwort** | Anmeldung am SMTP-Server; leer = ohne Anmeldung. Das Passwort wird verschlüsselt gespeichert und nie angezeigt. |
|
||||||
|
| **Absender, Empfänger** | Empfänger mit Komma getrennt |
|
||||||
|
|
||||||
|
*Test-E-Mail senden* prüft die gespeicherten Einstellungen und zeigt bei Fehlern die
|
||||||
|
Antwort des Servers.
|
||||||
|
|
||||||
|
**Webhook:**
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| **Webhook-URL** | Adresse des Empfängers. Wird nur gekürzt angezeigt, weil sie oft ein Token enthält. |
|
||||||
|
| **Content-Type, Vorlage** | Format der Nachricht; Platzhalter `$title`, `$message`, `$rule`, `$event`, `$time`, `$host` |
|
||||||
|
|
||||||
|
Beispiele: ntfy → URL `https://ntfy.sh/<thema>`, Content-Type `text/plain`, Vorlage
|
||||||
|
`$title: $message`. Gotify → URL `https://gotify.lan/message?token=…`, Content-Type
|
||||||
|
`application/json`, Vorlage `{"title": "$title", "message": "$message"}`.
|
||||||
|
*Test senden* prüft die gespeicherten Einstellungen.
|
||||||
|
|
||||||
|
### Überwachung mit Uptime Kuma {#uptime-kuma}
|
||||||
|
|
||||||
|
Zwei Wege, die sich gut ergänzen:
|
||||||
|
|
||||||
|
**1. Push (empfohlen)** – NetMover meldet sich jede Minute bei Kuma.
|
||||||
|
|
||||||
|
1. In Kuma *Neuer Monitor → Typ „Push“*, Heartbeat-Intervall 60 s. Kuma zeigt eine
|
||||||
|
Push-URL wie `https://kuma.lan/api/push/AbCdEf?status=up&msg=OK&ping=`.
|
||||||
|
2. In NetMover *System → Einstellungen → Überwachung* die URL eintragen, speichern,
|
||||||
|
*Push jetzt testen*.
|
||||||
|
|
||||||
|
NetMover meldet „up“ oder „down“ samt Problembeschreibung. Bleibt die Meldung ganz
|
||||||
|
aus – Dienst oder VM steht –, schlägt Kuma von selbst Alarm. Funktioniert auch, wenn
|
||||||
|
Kuma NetMover nicht erreichen kann.
|
||||||
|
|
||||||
|
**2. HTTP-Monitor** – Kuma fragt NetMover ab. Antwort **200** = in Ordnung,
|
||||||
|
**503** = Problem; ohne Anmeldung erreichbar.
|
||||||
|
|
||||||
|
| Adresse | Prüft |
|
||||||
|
|---|---|
|
||||||
|
| `http://<netmover>:8080/healthz` | alles: Scheduler, alle aktiven Anbindungen und Regeln |
|
||||||
|
| `http://<netmover>:8080/healthz/mount/<name>` | eine Anbindung (Name wie im Verzeichnis unter `/mnt/netmover`, z. B. `nas`) |
|
||||||
|
| `http://<netmover>:8080/healthz/rule/<nummer>` | eine Regel |
|
||||||
|
|
||||||
|
Die fertigen Adressen stehen unter *Einstellungen → Überwachung → Adressen für
|
||||||
|
HTTP-Monitore*. Mit einem Monitor pro Anbindung und Regel sieht man in Kuma sofort,
|
||||||
|
*was* gestört ist. `/healthz` liefert außerdem JSON mit allen Einzelprüfungen
|
||||||
|
(Kuma-Typ „HTTP(s) – JSON-Abfrage“, Ausdruck `status`, erwarteter Wert `ok`).
|
||||||
|
|
||||||
|
Als Problem gilt: Scheduler steht, eine aktive Anbindung ist im Fehlerzustand, eine
|
||||||
|
aktive Regel hatte länger als das Dreifache ihres Intervalls keinen Erfolg oder ist
|
||||||
|
dreimal hintereinander gescheitert, oder die [Sicherung](#sicherung) ist überfällig.
|
||||||
|
|
||||||
|
**3. Werte auslesen (JSON-Abfrage)** – `/healthz` und `/healthz/rule/<nummer>`
|
||||||
|
liefern unter `metrics` Kennzahlen der letzten 24 Stunden (Probeläufe zählen nicht):
|
||||||
|
|
||||||
|
| Feld | Bedeutung |
|
||||||
|
|---|---|
|
||||||
|
| `files_24h` | verschobene bzw. kopierte Dateien |
|
||||||
|
| `failed_24h` | Dateien, die gescheitert sind (werden erneut versucht) |
|
||||||
|
| `quarantined_24h` | Dateien, die in Quarantäne verschoben wurden |
|
||||||
|
| `errors_24h` | abgebrochene Läufe (z. B. Anbindung nicht erreichbar) |
|
||||||
|
| `pending_failures` | Dateien, die gerade auf einen neuen Versuch warten |
|
||||||
|
| `last_moved_at` | Zeitpunkt der letzten verschobenen Datei (UTC) |
|
||||||
|
|
||||||
|
`/healthz` enthält die Summe und unter `metrics.rules.<nummer>` jede Regel einzeln.
|
||||||
|
In Kuma: Typ *HTTP(s) – JSON-Abfrage*, dann Ausdruck, Bedingung und erwarteter Wert,
|
||||||
|
zum Beispiel:
|
||||||
|
|
||||||
|
| Monitor | Adresse | Ausdruck | Bedingung | Wert |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Gesamtzustand | `/healthz` | `status` | `==` | `ok` |
|
||||||
|
| Alarm bei Quarantäne | `/healthz` | `metrics.quarantined_24h` | `==` | `0` |
|
||||||
|
| Regel ohne Dateifehler | `/healthz/rule/1` | `metrics.failed_24h` | `==` | `0` |
|
||||||
|
| Sicherung | `/healthz/backup` | `status` | `==` | `ok` |
|
||||||
|
|
||||||
|
Ältere Kuma-Versionen (1.x) kennen nur „erwarteter Wert = Ergebnis“; die Beispiele
|
||||||
|
funktionieren dort genauso. Antwortet NetMover mit 503 (irgendein Problem), meldet
|
||||||
|
Kuma auch einen Werte-Monitor als gestört – das ist gewollt.
|
||||||
|
|
||||||
|
Beim Push zeigt Kuma als Nachricht z. B. „OK · 24 h: 12 Dateien“.
|
||||||
|
|
||||||
|
### Anmeldung {#anmeldung}
|
||||||
|
|
||||||
|
Unter *Einstellungen → Anmeldung* legst du fest, **aus welchen Netzen** es mit und
|
||||||
|
ohne Passwort geht (Bestätigung mit dem aktuellen Passwort):
|
||||||
|
|
||||||
|
| Feld | Wirkung |
|
||||||
|
|---|---|
|
||||||
|
| **Ohne Passwort aus diesen Netzen** | z. B. `10.0.10.0/24, 10.1.5.0/24` – von dort keine Anmeldung. Leer = niemand. `0.0.0.0/0, ::/0` = Anmeldung ganz aus. |
|
||||||
|
| **Mit Passwort aus diesen Netzen** | Leer = **alle übrigen** Netze mit Passwort. Ausgefüllt = nur von dort mit Passwort, **alle anderen Netze werden abgewiesen** (auch die Anmeldeseite). |
|
||||||
|
|
||||||
|
Beispiel: Ohne Passwort aus dem Heimnetz `10.0.10.0/24`, mit Passwort aus dem
|
||||||
|
VPN-Netz `10.8.0.0/24`, alles andere gesperrt.
|
||||||
|
|
||||||
|
- Einzelne Adressen gehen auch (`192.168.1.20`).
|
||||||
|
- Die Seite zeigt deine aktuelle Adresse. Eine Einstellung, mit der du dich selbst
|
||||||
|
aussperren würdest, wird nicht gespeichert.
|
||||||
|
- `/healthz` bleibt für die Überwachung von überall erreichbar.
|
||||||
|
- Solange du ohne Passwort arbeitest, steht oben gelb „ohne Anmeldung“. Der Schutz
|
||||||
|
gegen untergeschobene Formulare bleibt immer aktiv.
|
||||||
|
|
||||||
|
> NetMover läuft als root und kennt die Zugangsdaten deiner Freigaben. Wer die
|
||||||
|
> Oberfläche ohne Passwort erreicht, kann alles ändern – Netze „ohne Passwort“ daher
|
||||||
|
> so eng wie möglich wählen. Läuft ein Reverse Proxy auf demselben Server, kommen
|
||||||
|
> alle Anfragen von `127.0.0.1`: dann `127.0.0.1` nicht eintragen und
|
||||||
|
> `NETMOVER_FORWARDED_ALLOW_IPS` setzen.
|
||||||
|
|
||||||
|
Ausgesperrt oder Passwort vergessen? Auf dem Server
|
||||||
|
`/opt/netmover/venv/bin/netmover reset-password` – setzt das Passwort zurück, schaltet
|
||||||
|
die Anmeldung überall wieder ein und hebt Netz-Sperren auf.
|
||||||
|
|
||||||
|
### Sicherung {#sicherung}
|
||||||
|
|
||||||
|
*System → Sicherung*. NetMover sichert täglich seine Datenbank **und** den Schlüssel
|
||||||
|
`secret.key` – also alle Anbindungen, Regeln, Zugangsdaten und Einstellungen – auf
|
||||||
|
eine Anbindung, am besten eine eigene Freigabe auf dem NAS.
|
||||||
|
|
||||||
|
1. Auf dem NAS eine Freigabe anlegen (z. B. `Backup`), in NetMover als Anbindung
|
||||||
|
eintragen und testen.
|
||||||
|
2. *Sicherung*: Anbindung wählen, Unterordner (z. B. `netmover`), Uhrzeit, wie viele
|
||||||
|
Sicherungen behalten werden, und eine **Passphrase** (mindestens 12 Zeichen).
|
||||||
|
3. *Tägliche Sicherung einschalten*, speichern, **Jetzt sichern**.
|
||||||
|
4. **Passphrase im Passwortmanager ablegen.** Ohne sie ist die Sicherung wertlos.
|
||||||
|
|
||||||
|
Die Sicherung ist mit der Passphrase verschlüsselt (scrypt + Fernet): Wer nur Zugriff
|
||||||
|
aufs NAS hat, kommt nicht an die Zugangsdaten. Vor dem Hochladen öffnet NetMover die
|
||||||
|
Datei probeweise und prüft die Datenbank; hochgeladen wird wie bei Regeln über eine
|
||||||
|
`.part`-Datei. Danach werden ältere NetMover-Sicherungen über die eingestellte Anzahl
|
||||||
|
hinaus gelöscht – andere Dateien im Ordner nie.
|
||||||
|
|
||||||
|
Fällt der Termin aus (Server aus, NAS weg), holt NetMover ihn nach; nach einem
|
||||||
|
Fehler versucht es stündlich erneut und meldet den ersten Fehlschlag per E-Mail bzw.
|
||||||
|
Webhook. Gab es zwei Tage lang keine erfolgreiche Sicherung, gilt das als Problem
|
||||||
|
(Statusampel, `/healthz`, eigener Monitor `/healthz/backup`).
|
||||||
|
|
||||||
|
Eine neue Passphrase gilt nur für künftige Sicherungen; ältere brauchen weiter die
|
||||||
|
alte.
|
||||||
|
|
||||||
|
**Wiederherstellen** (neuer Server, gleiche oder neuere NetMover-Version):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# NetMover wie im README installieren, dann:
|
||||||
|
systemctl stop netmover
|
||||||
|
netmover restore /pfad/zu/netmover-<host>-<datum>.nmbak # fragt nach der Passphrase
|
||||||
|
systemctl start netmover
|
||||||
|
```
|
||||||
|
|
||||||
|
Die bisherigen Dateien legt `restore` vorher in
|
||||||
|
`/var/lib/netmover/vor-wiederherstellung-<zeit>/` ab. Die Datenbank wird anschließend
|
||||||
|
auf den Stand der installierten Version migriert. Die Sicherungsdatei holst du z. B.
|
||||||
|
mit `scp` vom NAS oder hängst die Freigabe kurz von Hand ein.
|
||||||
|
|
||||||
|
Eine Sicherung von Hand in eine Datei: `netmover backup /root/` (verwendet die
|
||||||
|
Passphrase aus den Einstellungen).
|
||||||
|
|
||||||
|
### Export und Import {#export-import}
|
||||||
|
|
||||||
|
*Konfiguration exportieren* lädt Anbindungen, Regeln und Einstellungen als JSON
|
||||||
|
herunter – **ohne** Passwörter, Schlüssel, Benutzernamen und Webhook-URL. Beim
|
||||||
|
Import werden vorhandene Namen übersprungen; importierte Regeln sind zunächst
|
||||||
|
deaktiviert, Zugangsdaten müssen neu eingegeben werden.
|
||||||
|
|
||||||
|
## Typische Einrichtungen {#beispiele}
|
||||||
|
|
||||||
|
### Scanner → NAS → Paperless {#beispiel-paperless}
|
||||||
|
|
||||||
|
1. Anbindung **NAS** (SMB, Freigabe `Scans`), testen, einhängen.
|
||||||
|
2. Unter *Zugangsdaten* den Schlüssel **paperless-lxc** erzeugen, Zeile in
|
||||||
|
`authorized_keys` im Paperless-LXC eintragen.
|
||||||
|
3. Anbindung **Paperless** (SFTP, Pfad `/opt/paperless_data/consume`, Schlüssel
|
||||||
|
wählen, Benutzer = der Paperless-Benutzer), testen.
|
||||||
|
4. Regel **Scans → Paperless**: Quelle NAS, Filter `*.pdf`, Ziel Paperless,
|
||||||
|
verschieben, umbenennen, 30 s. Optional als **weiteres Ziel** „Archiv“: Anbindung
|
||||||
|
NAS, Ordner `Archiv/Scans`, Ablage nach Datum. **Probelauf**, dann aktivieren.
|
||||||
|
|
||||||
|
### Jede Person in einen eigenen Ordner {#beispiel-unterordner}
|
||||||
|
|
||||||
|
Scannt jede Person in einen eigenen Unterordner (`Scans/max`, `Scans/erika`): pro
|
||||||
|
Person eine Regel mit eigenem **Unterordner** in der Quelle und eigenem Ziel. Am
|
||||||
|
schnellsten: die erste Regel einrichten, dann **Kopieren** und nur Unterordner und
|
||||||
|
Ziel ändern.
|
||||||
|
|
||||||
|
### Rechnungseingang: DMS, Buchhaltung und Archiv {#beispiel-rechnungen}
|
||||||
|
|
||||||
|
E-Rechnungen (PDF) landen in einem Eingangsordner und sollen ins DMS, in den
|
||||||
|
Importordner der Buchhaltungssoftware und ins Archiv.
|
||||||
|
|
||||||
|
1. Anbindungen: *Rechnungseingang*, *DMS*, *Buchhaltung*, *Archiv*.
|
||||||
|
2. Regel: Quelle *Rechnungseingang*, Filter `*.pdf;*.xml` (falls auch reine
|
||||||
|
XRechnung-XML kommt), **Hauptziel** *DMS*, Aktion verschieben.
|
||||||
|
3. **Weitere Ziele**: „Buchhaltung“ (Ablage **wie in der Quelle** – die Software sucht
|
||||||
|
nicht in Unterordnern) und „Archiv“ (Ablage **nach Datum**).
|
||||||
|
4. **Probelauf** mit ein, zwei Testrechnungen – er zeigt für jedes Ziel den künftigen
|
||||||
|
Pfad –, dann aktivieren.
|
||||||
|
|
||||||
|
Ist das DMS kurz weg, bekommt die Buchhaltung die Rechnung trotzdem; das DMS wird
|
||||||
|
nachbeliefert, sobald es zurück ist.
|
||||||
|
|
||||||
|
### Kopie als zusätzliche Sicherung {#beispiel-kopie}
|
||||||
|
|
||||||
|
Regel mit Aktion **kopieren**, Ziel z. B. ein zweites NAS, Intervall 3600 s. Jede
|
||||||
|
Dateiversion wird genau einmal kopiert; ändert sich eine Datei, wird die neue Version
|
||||||
|
nach der Kollisionsregel abgelegt.
|
||||||
|
|
||||||
|
## Häufige Fragen {#fragen}
|
||||||
|
|
||||||
|
**Eine Datei wird nicht verschoben.** – Im Protokoll der Regel (auch leere Läufe
|
||||||
|
anzeigen) steht der Grund. Häufig: Filter passt nicht (Groß/klein ist bei Glob egal,
|
||||||
|
bei regulären Ausdrücken nicht), Datei wird noch geschrieben oder ist größer als
|
||||||
|
„Max. Größe“.
|
||||||
|
|
||||||
|
**Die Anbindung zeigt „Fehler“.** – Die Meldung darunter nennt die Ursache und meist
|
||||||
|
die Abhilfe. Ist das NAS nur neu gestartet, hängt NetMover die Freigabe innerhalb
|
||||||
|
einer Minute selbst wieder ein; die Regeln warten so lange und laufen dann weiter.
|
||||||
|
|
||||||
|
**Ich mache gleich ein NAS-Update.** – Vorher auf der Statusseite *Wartungsmodus*
|
||||||
|
einschalten, danach wieder beenden. Nötig ist es nicht (die Regeln warten ohnehin
|
||||||
|
von selbst), aber so gibt es keine Alarme.
|
||||||
|
|
||||||
|
**„mount error(95): Operation not supported“** – NAS und NetMover finden keine
|
||||||
|
gemeinsame SMB-Version. Am NAS das maximale SMB-Protokoll auf SMB3 stellen
|
||||||
|
(Synology: Systemsteuerung → Dateidienste → SMB → Erweiterte Einstellungen) oder
|
||||||
|
Zusatzoption `vers=2.0`.
|
||||||
|
|
||||||
|
**„Root-Squash“ beim Testen einer NFS-Anbindung** – Der NFS-Server behandelt
|
||||||
|
NetMover wie einen Gast. Lesen und Schreiben gehen oft trotzdem, aber „Eigentümer
|
||||||
|
setzen“ nicht. Abhilfe am NAS in der NFS-Regel: *Squash → Keine Zuordnung*.
|
||||||
|
|
||||||
|
**Kann NetMover Dateien verlieren?** – Die Quelle wird erst gelöscht, wenn die Datei
|
||||||
|
vollständig und unter ihrem Endnamen im Ziel liegt. Fehlt eine Anbindung, wird nichts
|
||||||
|
angefasst. Einzig „überschreiben“ als Kollisionsregel ersetzt bewusst Dateien im Ziel.
|
||||||
|
|
||||||
|
**Wie aktualisiere ich NetMover?** – Auf dem Server `netmover-update`. Vorher wird
|
||||||
|
automatisch gesichert. Details im README.
|
||||||
|
|
||||||
|
**Passwort vergessen?** – Auf dem Server `/opt/netmover/venv/bin/netmover
|
||||||
|
reset-password`, Dienst neu starten, Einrichtung mit neuem Code aus dem Log.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
@@ -25,6 +25,7 @@ dependencies = [
|
|||||||
"itsdangerous>=2.2",
|
"itsdangerous>=2.2",
|
||||||
"httpx>=0.27",
|
"httpx>=0.27",
|
||||||
"paramiko>=3.4",
|
"paramiko>=3.4",
|
||||||
|
"markdown>=3.5",
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
@@ -33,6 +34,9 @@ dev = [
|
|||||||
"ruff>=0.6",
|
"ruff>=0.6",
|
||||||
"mypy>=1.11",
|
"mypy>=1.11",
|
||||||
"types-paramiko",
|
"types-paramiko",
|
||||||
|
"types-Markdown",
|
||||||
|
"shellcheck-py",
|
||||||
|
"aiosmtpd",
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
@@ -44,6 +48,10 @@ path = "src/netmover/__init__.py"
|
|||||||
[tool.hatch.build.targets.wheel]
|
[tool.hatch.build.targets.wheel]
|
||||||
packages = ["src/netmover"]
|
packages = ["src/netmover"]
|
||||||
|
|
||||||
|
# Die Anleitung liegt für Gitea unter docs/ und wird für die Hilfe-Seite mitgeliefert.
|
||||||
|
[tool.hatch.build.targets.wheel.force-include]
|
||||||
|
"docs" = "netmover/web/help"
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
[tool.pytest.ini_options]
|
||||||
testpaths = ["tests"]
|
testpaths = ["tests"]
|
||||||
addopts = "-ra"
|
addopts = "-ra"
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
||||||
|
|
||||||
__version__ = "0.1.1"
|
__version__ = "0.11.0"
|
||||||
|
|||||||
+262
-27
@@ -12,7 +12,7 @@ import re
|
|||||||
from collections.abc import Callable, Mapping
|
from collections.abc import Callable, Mapping
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
from sqlalchemy import select
|
from sqlalchemy import delete, select
|
||||||
from sqlalchemy.exc import IntegrityError
|
from sqlalchemy.exc import IntegrityError
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
@@ -27,6 +27,10 @@ from netmover.models import (
|
|||||||
PatternType,
|
PatternType,
|
||||||
Rule,
|
Rule,
|
||||||
RuleAction,
|
RuleAction,
|
||||||
|
RuleTarget,
|
||||||
|
TargetFile,
|
||||||
|
TargetKind,
|
||||||
|
TargetLayout,
|
||||||
)
|
)
|
||||||
from netmover.mounting.manager import CredentialData, MountConfig, MountManager
|
from netmover.mounting.manager import CredentialData, MountConfig, MountManager
|
||||||
from netmover.validation import (
|
from netmover.validation import (
|
||||||
@@ -48,6 +52,7 @@ from netmover.validation import (
|
|||||||
Errors = dict[str, str]
|
Errors = dict[str, str]
|
||||||
|
|
||||||
MIN_INTERVAL = 10
|
MIN_INTERVAL = 10
|
||||||
|
MAX_TARGETS = 9 # weitere Ziele zusätzlich zum Hauptziel
|
||||||
_OWNER_RE = re.compile(r"^\d{1,10}(:\d{1,10})?$")
|
_OWNER_RE = re.compile(r"^\d{1,10}(:\d{1,10})?$")
|
||||||
|
|
||||||
|
|
||||||
@@ -84,6 +89,10 @@ def _int(
|
|||||||
# -- Anbindungen ------------------------------------------------------------------------
|
# -- Anbindungen ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
NEW_KEY = "new"
|
||||||
|
OWN = "own"
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class MountInput:
|
class MountInput:
|
||||||
name: str = ""
|
name: str = ""
|
||||||
@@ -95,6 +104,11 @@ class MountInput:
|
|||||||
password: str = "" # leer = unverändert
|
password: str = "" # leer = unverändert
|
||||||
domain: str = ""
|
domain: str = ""
|
||||||
auth: CredentialKind = CredentialKind.PASSWORD # nur sftp: Schlüssel oder Passwort
|
auth: CredentialKind = CredentialKind.PASSWORD # nur sftp: Schlüssel oder Passwort
|
||||||
|
# nur sftp + Schlüssel: "new" = eigener Schlüssel nur für diese Anbindung,
|
||||||
|
# sonst ID eines Schlüssels aus der Schlüsselverwaltung
|
||||||
|
key_id: str = NEW_KEY
|
||||||
|
# Passwort-Anmeldung (SMB, SFTP): "own" = eigene Angaben, sonst ID gespeicherter Zugangsdaten
|
||||||
|
cred_id: str = OWN
|
||||||
uid: int | None = None
|
uid: int | None = None
|
||||||
gid: int | None = None
|
gid: int | None = None
|
||||||
file_mode: str | None = None
|
file_mode: str | None = None
|
||||||
@@ -126,9 +140,15 @@ def mount_input_from_model(m: Mount) -> MountInput:
|
|||||||
server=m.server,
|
server=m.server,
|
||||||
share=m.share,
|
share=m.share,
|
||||||
port=m.port,
|
port=m.port,
|
||||||
username=c.username if c else "",
|
username=m.username or (c.username if c else ""),
|
||||||
domain=(c.domain or "") if c else "",
|
domain=(c.domain or "") if c else "",
|
||||||
auth=c.kind if c else CredentialKind.PASSWORD,
|
auth=c.kind if c else CredentialKind.PASSWORD,
|
||||||
|
key_id=str(c.id)
|
||||||
|
if c is not None and c.managed and c.kind is CredentialKind.SSH_KEY
|
||||||
|
else NEW_KEY,
|
||||||
|
cred_id=str(c.id)
|
||||||
|
if c is not None and c.managed and c.kind is CredentialKind.PASSWORD
|
||||||
|
else OWN,
|
||||||
uid=m.uid,
|
uid=m.uid,
|
||||||
gid=m.gid,
|
gid=m.gid,
|
||||||
file_mode=m.file_mode,
|
file_mode=m.file_mode,
|
||||||
@@ -151,7 +171,15 @@ def parse_mount_form(form: Mapping[str, str]) -> tuple[MountInput, Errors]:
|
|||||||
extra_options=(form.get("extra_options") or "").strip(),
|
extra_options=(form.get("extra_options") or "").strip(),
|
||||||
auto_mount=_bool(form, "auto_mount"),
|
auto_mount=_bool(form, "auto_mount"),
|
||||||
enabled=_bool(form, "enabled"),
|
enabled=_bool(form, "enabled"),
|
||||||
|
key_id=(form.get("key_id") or NEW_KEY).strip(),
|
||||||
|
cred_id=(form.get("cred_id") or OWN).strip(),
|
||||||
)
|
)
|
||||||
|
if data.key_id != NEW_KEY and not data.key_id.isdigit():
|
||||||
|
errors["key_id"] = "Ungültige Schlüsselauswahl."
|
||||||
|
data.key_id = NEW_KEY
|
||||||
|
if data.cred_id != OWN and not data.cred_id.isdigit():
|
||||||
|
errors["cred_id"] = "Ungültige Auswahl."
|
||||||
|
data.cred_id = OWN
|
||||||
try:
|
try:
|
||||||
data.type = MountType(form.get("type", "cifs"))
|
data.type = MountType(form.get("type", "cifs"))
|
||||||
except ValueError:
|
except ValueError:
|
||||||
@@ -199,21 +227,50 @@ def parse_mount_form(form: Mapping[str, str]) -> tuple[MountInput, Errors]:
|
|||||||
data.username = check("username", lambda: validate_username(data.username), data.username)
|
data.username = check("username", lambda: validate_username(data.username), data.username)
|
||||||
if data.domain:
|
if data.domain:
|
||||||
check("domain", lambda: validate_username(data.domain), "")
|
check("domain", lambda: validate_username(data.domain), "")
|
||||||
if data.type is MountType.SFTP and not data.username:
|
if data.auth is CredentialKind.SSH_KEY or data.type in (MountType.NFS3, MountType.NFS4):
|
||||||
|
data.cred_id = OWN
|
||||||
|
uses_stored = data.cred_id != OWN
|
||||||
|
if uses_stored:
|
||||||
|
data.username = data.domain = data.password = "" # kommen aus den Zugangsdaten
|
||||||
|
errors.pop("username", None)
|
||||||
|
errors.pop("password", None)
|
||||||
|
if data.type is MountType.SFTP and not data.username and not uses_stored:
|
||||||
errors["username"] = "SFTP braucht einen Benutzernamen."
|
errors["username"] = "SFTP braucht einen Benutzernamen."
|
||||||
if data.type is not MountType.SFTP:
|
if data.type is not MountType.SFTP:
|
||||||
data.auth = CredentialKind.PASSWORD
|
data.auth = CredentialKind.PASSWORD
|
||||||
|
if data.auth is not CredentialKind.SSH_KEY:
|
||||||
|
data.key_id = NEW_KEY
|
||||||
return data, errors
|
return data, errors
|
||||||
|
|
||||||
|
|
||||||
def credential_for_test(
|
def credential_for_test(
|
||||||
data: MountInput, existing: Mount | None, secrets: SecretBox
|
data: MountInput, existing: Mount | None, secrets: SecretBox, s: Session | None = None
|
||||||
) -> CredentialData | None:
|
) -> CredentialData | None:
|
||||||
"""Zugangsdaten für „Testen“ aus dem Formular – leeres Passwort = gespeichertes."""
|
"""Zugangsdaten für „Testen“ aus dem Formular – leeres Passwort = gespeichertes,
|
||||||
if data.type in (MountType.NFS3, MountType.NFS4) or not data.username:
|
gewählter Schlüssel aus der Schlüsselverwaltung."""
|
||||||
|
if data.type in (MountType.NFS3, MountType.NFS4):
|
||||||
return None
|
return None
|
||||||
|
if data.cred_id != OWN and s is not None:
|
||||||
|
stored = s.get(Credential, int(data.cred_id))
|
||||||
|
if stored is None or not stored.managed or stored.kind is not CredentialKind.PASSWORD:
|
||||||
|
return None
|
||||||
|
return CredentialData(
|
||||||
|
CredentialKind.PASSWORD,
|
||||||
|
stored.username,
|
||||||
|
secrets.decrypt(stored.secret_enc),
|
||||||
|
stored.domain,
|
||||||
|
)
|
||||||
|
if not data.username:
|
||||||
|
return None
|
||||||
|
if data.auth is CredentialKind.SSH_KEY and data.key_id != NEW_KEY and s is not None:
|
||||||
|
key = s.get(Credential, int(data.key_id))
|
||||||
|
if key is None or not key.managed:
|
||||||
|
return None
|
||||||
|
return CredentialData(data.auth, data.username, secrets.decrypt(key.secret_enc))
|
||||||
secret = data.password
|
secret = data.password
|
||||||
stored = existing.credential if existing is not None else None
|
stored = existing.credential if existing is not None else None
|
||||||
|
if stored is not None and stored.managed:
|
||||||
|
stored = None # eigener Schlüssel gewählt, gespeicherter ist nicht gemeint
|
||||||
if not secret and stored is not None and stored.kind is data.auth:
|
if not secret and stored is not None and stored.kind is data.auth:
|
||||||
secret = secrets.decrypt(stored.secret_enc)
|
secret = secrets.decrypt(stored.secret_enc)
|
||||||
if not secret:
|
if not secret:
|
||||||
@@ -240,7 +297,11 @@ def save_mount(
|
|||||||
mounts: MountManager,
|
mounts: MountManager,
|
||||||
data: MountInput,
|
data: MountInput,
|
||||||
mount_id: int | None = None,
|
mount_id: int | None = None,
|
||||||
|
copy_from: Mount | None = None,
|
||||||
) -> tuple[Mount | None, Errors]:
|
) -> tuple[Mount | None, Errors]:
|
||||||
|
"""``copy_from``: Vorlage beim Kopieren – eigene Zugangsdaten (Passwort bzw.
|
||||||
|
Schlüsselpaar) werden übernommen, solange kein neues Passwort eingegeben ist;
|
||||||
|
die Kopie bekommt ein eigenes Exemplar."""
|
||||||
errors: Errors = {}
|
errors: Errors = {}
|
||||||
m = s.get(Mount, mount_id) if mount_id is not None else None
|
m = s.get(Mount, mount_id) if mount_id is not None else None
|
||||||
if mount_id is not None and m is None:
|
if mount_id is not None and m is None:
|
||||||
@@ -295,41 +356,88 @@ def save_mount(
|
|||||||
setattr(m, f, getattr(data, f))
|
setattr(m, f, getattr(data, f))
|
||||||
if not data.type.is_kernel_mount:
|
if not data.type.is_kernel_mount:
|
||||||
m.auto_mount = False
|
m.auto_mount = False
|
||||||
|
if (
|
||||||
|
mount_id is None
|
||||||
|
and copy_from is not None
|
||||||
|
and data.type is MountType.SFTP
|
||||||
|
and copy_from.type is MountType.SFTP
|
||||||
|
and (copy_from.server, copy_from.port) == (data.server, data.port)
|
||||||
|
):
|
||||||
|
m.host_key = copy_from.host_key # gleicher Server – Hostschlüssel bleibt gültig
|
||||||
|
|
||||||
# Zugangsdaten: jede Anbindung pflegt ihre eigenen.
|
# Zugangsdaten: entweder ein gespeicherter Schlüssel aus der Schlüsselverwaltung
|
||||||
cred = m.credential
|
# (geteilt, wird hier nie verändert) oder eigene Zugangsdaten nur dieser Anbindung.
|
||||||
|
current = m.credential
|
||||||
|
own = current if current is not None and not current.managed else None
|
||||||
|
stored_password = data.cred_id != OWN and (
|
||||||
|
data.type is MountType.CIFS
|
||||||
|
or (data.type is MountType.SFTP and data.auth is CredentialKind.PASSWORD)
|
||||||
|
)
|
||||||
needs_cred = data.type is MountType.SFTP or (
|
needs_cred = data.type is MountType.SFTP or (
|
||||||
data.type is MountType.CIFS and bool(data.username)
|
data.type is MountType.CIFS and (bool(data.username) or stored_password)
|
||||||
|
)
|
||||||
|
use_managed = (
|
||||||
|
data.type is MountType.SFTP
|
||||||
|
and data.auth is CredentialKind.SSH_KEY
|
||||||
|
and data.key_id != NEW_KEY
|
||||||
)
|
)
|
||||||
if not needs_cred:
|
if not needs_cred:
|
||||||
if cred is not None:
|
m.credential = None
|
||||||
m.credential = None
|
m.username = None
|
||||||
s.flush()
|
elif stored_password:
|
||||||
if not cred.mounts:
|
stored = s.get(Credential, int(data.cred_id))
|
||||||
s.delete(cred)
|
if stored is None or not stored.managed or stored.kind is not CredentialKind.PASSWORD:
|
||||||
|
s.rollback()
|
||||||
|
return None, {"cred_id": "Zugangsdaten existieren nicht."}
|
||||||
|
m.credential = stored
|
||||||
|
m.username = None
|
||||||
|
elif use_managed:
|
||||||
|
key = s.get(Credential, int(data.key_id))
|
||||||
|
if key is None or not key.managed or key.kind is not CredentialKind.SSH_KEY:
|
||||||
|
s.rollback()
|
||||||
|
return None, {"key_id": "Schlüssel existiert nicht."}
|
||||||
|
m.credential = key
|
||||||
|
m.username = data.username
|
||||||
else:
|
else:
|
||||||
kind_changed = cred is not None and cred.kind is not data.auth
|
m.username = None
|
||||||
if cred is None or kind_changed:
|
cred = own
|
||||||
if data.auth is CredentialKind.SSH_KEY:
|
template = (
|
||||||
private, public = generate_ssh_keypair(f"netmover@{m.slug}")
|
copy_from.credential
|
||||||
secret, pub = private, public
|
if copy_from is not None
|
||||||
|
and copy_from.credential is not None
|
||||||
|
and not copy_from.credential.managed
|
||||||
|
and copy_from.credential.kind is data.auth
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
if cred is None or cred.kind is not data.auth:
|
||||||
|
if template is not None and (data.auth is CredentialKind.SSH_KEY or not data.password):
|
||||||
|
# Kopie: dasselbe Passwort bzw. Schlüsselpaar (steht schon in authorized_keys)
|
||||||
|
secret_enc, pub = template.secret_enc, template.public_key
|
||||||
|
elif data.auth is CredentialKind.SSH_KEY:
|
||||||
|
private, pub = generate_ssh_keypair(f"netmover@{m.slug}")
|
||||||
|
secret_enc = secrets.encrypt(private)
|
||||||
else:
|
else:
|
||||||
if not data.password:
|
if not data.password:
|
||||||
errors["password"] = "Passwort fehlt." # noqa: S105
|
errors["password"] = "Passwort fehlt." # noqa: S105
|
||||||
s.rollback()
|
s.rollback()
|
||||||
return None, errors
|
return None, errors
|
||||||
secret, pub = data.password, None
|
secret_enc, pub = secrets.encrypt(data.password), None
|
||||||
if cred is None:
|
if cred is None:
|
||||||
cred = Credential(name=f"mount:{m.slug}")
|
cred = Credential(name=f"mount:{m.slug}", managed=False)
|
||||||
s.add(cred)
|
s.add(cred)
|
||||||
m.credential = cred
|
|
||||||
cred.kind = data.auth
|
cred.kind = data.auth
|
||||||
cred.secret_enc = secrets.encrypt(secret)
|
cred.secret_enc = secret_enc
|
||||||
cred.public_key = pub
|
cred.public_key = pub
|
||||||
elif data.password and data.auth is CredentialKind.PASSWORD:
|
elif data.password and data.auth is CredentialKind.PASSWORD:
|
||||||
cred.secret_enc = secrets.encrypt(data.password)
|
cred.secret_enc = secrets.encrypt(data.password)
|
||||||
cred.username = data.username
|
cred.username = data.username
|
||||||
cred.domain = data.domain or None
|
cred.domain = data.domain or None
|
||||||
|
m.credential = cred
|
||||||
|
# Eigene Zugangsdaten, die nicht mehr gebraucht werden, entfernen.
|
||||||
|
if own is not None and m.credential is not own:
|
||||||
|
s.flush()
|
||||||
|
if not own.mounts:
|
||||||
|
s.delete(own)
|
||||||
try:
|
try:
|
||||||
s.flush()
|
s.flush()
|
||||||
except IntegrityError:
|
except IntegrityError:
|
||||||
@@ -339,12 +447,26 @@ def save_mount(
|
|||||||
return m, errors
|
return m, errors
|
||||||
|
|
||||||
|
|
||||||
|
def copy_name(s: Session, name: str) -> str:
|
||||||
|
"""„NAS (Kopie)“, „NAS (Kopie 2)“, … – frei in der Datenbank."""
|
||||||
|
base = f"{name} (Kopie)"[:100]
|
||||||
|
candidate, n = base, 1
|
||||||
|
while s.scalar(select(Mount.id).where(Mount.name == candidate)) is not None:
|
||||||
|
n += 1
|
||||||
|
candidate = f"{name} (Kopie {n})"[:100]
|
||||||
|
return candidate
|
||||||
|
|
||||||
|
|
||||||
def delete_mount(s: Session, mounts: MountManager, mount_id: int) -> None:
|
def delete_mount(s: Session, mounts: MountManager, mount_id: int) -> None:
|
||||||
m = s.get(Mount, mount_id)
|
m = s.get(Mount, mount_id)
|
||||||
if m is None:
|
if m is None:
|
||||||
return
|
return
|
||||||
used = s.scalars(
|
used = s.scalars(
|
||||||
select(Rule.name).where((Rule.src_mount_id == mount_id) | (Rule.dst_mount_id == mount_id))
|
select(Rule.name).where(
|
||||||
|
(Rule.src_mount_id == mount_id)
|
||||||
|
| (Rule.dst_mount_id == mount_id)
|
||||||
|
| Rule.targets.any(RuleTarget.mount_id == mount_id)
|
||||||
|
)
|
||||||
).all()
|
).all()
|
||||||
if used:
|
if used:
|
||||||
raise AdminError(f"Wird noch von Regeln benutzt: {', '.join(used)}")
|
raise AdminError(f"Wird noch von Regeln benutzt: {', '.join(used)}")
|
||||||
@@ -353,13 +475,25 @@ def delete_mount(s: Session, mounts: MountManager, mount_id: int) -> None:
|
|||||||
cred = m.credential
|
cred = m.credential
|
||||||
s.delete(m)
|
s.delete(m)
|
||||||
s.flush()
|
s.flush()
|
||||||
if cred is not None and not cred.mounts:
|
# Eigene Zugangsdaten gehen mit; Schlüssel aus der Schlüsselverwaltung bleiben.
|
||||||
|
if cred is not None and not cred.managed and not cred.mounts:
|
||||||
s.delete(cred)
|
s.delete(cred)
|
||||||
|
|
||||||
|
|
||||||
# -- Regeln ----------------------------------------------------------------------------------
|
# -- Regeln ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TargetInput:
|
||||||
|
"""Ein weiteres Ziel im Regelformular (Felder ``target_<n>_…``)."""
|
||||||
|
|
||||||
|
name: str = ""
|
||||||
|
kind: str = "mount" # "mount" | "local"
|
||||||
|
mount_id: int | None = None
|
||||||
|
path: str = ""
|
||||||
|
layout: TargetLayout = TargetLayout.FLAT
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class RuleInput:
|
class RuleInput:
|
||||||
name: str = ""
|
name: str = ""
|
||||||
@@ -384,6 +518,7 @@ class RuleInput:
|
|||||||
dry_run: bool = False
|
dry_run: bool = False
|
||||||
quarantine_path: str = ""
|
quarantine_path: str = ""
|
||||||
quarantine_after: int = 3
|
quarantine_after: int = 3
|
||||||
|
targets: list[TargetInput] = field(default_factory=list)
|
||||||
extra: dict[str, str] = field(default_factory=dict)
|
extra: dict[str, str] = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
@@ -411,6 +546,9 @@ def rule_input_from_model(r: Rule) -> RuleInput:
|
|||||||
dry_run=r.dry_run,
|
dry_run=r.dry_run,
|
||||||
quarantine_path=r.quarantine_path or "",
|
quarantine_path=r.quarantine_path or "",
|
||||||
quarantine_after=r.quarantine_after,
|
quarantine_after=r.quarantine_after,
|
||||||
|
targets=[
|
||||||
|
TargetInput(t.name, t.kind.value, t.mount_id, t.path, t.layout) for t in r.targets
|
||||||
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -457,8 +595,6 @@ def parse_rule_form(s: Session, form: Mapping[str, str]) -> tuple[RuleInput, Err
|
|||||||
src = s.get(Mount, d.src_mount_id) if d.src_mount_id else None
|
src = s.get(Mount, d.src_mount_id) if d.src_mount_id else None
|
||||||
if src is None:
|
if src is None:
|
||||||
errors["src_mount_id"] = "Quelle wählen."
|
errors["src_mount_id"] = "Quelle wählen."
|
||||||
elif not src.type.is_kernel_mount:
|
|
||||||
errors["src_mount_id"] = "SFTP-Anbindungen sind nur als Ziel möglich."
|
|
||||||
try:
|
try:
|
||||||
d.src_subpath = safe_relpath("src_subpath", d.src_subpath)
|
d.src_subpath = safe_relpath("src_subpath", d.src_subpath)
|
||||||
except ValidationError as exc:
|
except ValidationError as exc:
|
||||||
@@ -491,6 +627,10 @@ def parse_rule_form(s: Session, form: Mapping[str, str]) -> tuple[RuleInput, Err
|
|||||||
errors["pattern"] = f"Ungültiger regulärer Ausdruck: {exc}"
|
errors["pattern"] = f"Ungültiger regulärer Ausdruck: {exc}"
|
||||||
if d.stability_seconds >= d.op_timeout_seconds:
|
if d.stability_seconds >= d.op_timeout_seconds:
|
||||||
errors["stability_seconds"] = "Muss kleiner als das Zeitlimit sein."
|
errors["stability_seconds"] = "Muss kleiner als das Zeitlimit sein."
|
||||||
|
elif src is not None and src.type is MountType.SFTP and d.stability_seconds < 2:
|
||||||
|
errors["stability_seconds"] = (
|
||||||
|
"Bei SFTP-Quellen mindestens 2 s – der Server meldet Zeiten nur sekundengenau."
|
||||||
|
)
|
||||||
if d.set_owner and not _OWNER_RE.fullmatch(d.set_owner):
|
if d.set_owner and not _OWNER_RE.fullmatch(d.set_owner):
|
||||||
errors["set_owner"] = "Als „uid:gid“ angeben, z. B. „1000:1000“."
|
errors["set_owner"] = "Als „uid:gid“ angeben, z. B. „1000:1000“."
|
||||||
try:
|
try:
|
||||||
@@ -502,9 +642,103 @@ def parse_rule_form(s: Session, form: Mapping[str, str]) -> tuple[RuleInput, Err
|
|||||||
d.quarantine_path = safe_relpath("quarantine_path", d.quarantine_path)
|
d.quarantine_path = safe_relpath("quarantine_path", d.quarantine_path)
|
||||||
except ValidationError as exc:
|
except ValidationError as exc:
|
||||||
errors["quarantine_path"] = str(exc)
|
errors["quarantine_path"] = str(exc)
|
||||||
|
|
||||||
|
d.targets = _parse_targets(s, form, d, errors)
|
||||||
return d, errors
|
return d, errors
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_targets(
|
||||||
|
s: Session, form: Mapping[str, str], d: RuleInput, errors: Errors
|
||||||
|
) -> list[TargetInput]:
|
||||||
|
"""Weitere Ziele aus den Plätzen ``target_0_…`` bis ``target_8_…``; leere Plätze
|
||||||
|
(Art „–“) fallen weg, Fehler gelten für die verbleibende Reihenfolge."""
|
||||||
|
folders: dict[tuple[int | None, str], str] = {}
|
||||||
|
if "dst_path" not in errors:
|
||||||
|
folders[(d.dst_mount_id if d.dst_kind == "mount" else None, d.dst_path)] = "Hauptziel"
|
||||||
|
result: list[TargetInput] = []
|
||||||
|
for slot in range(MAX_TARGETS):
|
||||||
|
p = f"target_{slot}_"
|
||||||
|
kind = form.get(p + "kind") or ""
|
||||||
|
if not kind:
|
||||||
|
continue
|
||||||
|
n = len(result)
|
||||||
|
e = f"target_{n}_"
|
||||||
|
t = TargetInput(
|
||||||
|
name=(form.get(p + "name") or "").strip()[:100],
|
||||||
|
kind=kind,
|
||||||
|
path=(form.get(p + "path") or "").strip(),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
t.layout = TargetLayout(form.get(p + "layout") or "flat")
|
||||||
|
except ValueError:
|
||||||
|
errors[e + "layout"] = "Ungültige Auswahl."
|
||||||
|
if kind == TargetKind.MOUNT:
|
||||||
|
t.mount_id = _int(form, p + "mount_id", errors)
|
||||||
|
m = s.get(Mount, t.mount_id) if t.mount_id else None
|
||||||
|
if m is None:
|
||||||
|
errors[e + "mount_id"] = "Anbindung wählen."
|
||||||
|
try:
|
||||||
|
t.path = safe_relpath(e + "path", t.path)
|
||||||
|
except ValidationError as exc:
|
||||||
|
errors[e + "path"] = str(exc)
|
||||||
|
if (
|
||||||
|
t.mount_id == d.src_mount_id
|
||||||
|
and "src_subpath" not in errors
|
||||||
|
and t.path == d.src_subpath
|
||||||
|
):
|
||||||
|
errors[e + "path"] = "Quelle und Ziel sind derselbe Ordner."
|
||||||
|
default_name = m.name if m is not None else ""
|
||||||
|
elif kind == TargetKind.LOCAL:
|
||||||
|
try:
|
||||||
|
t.path = safe_abspath(e + "path", t.path)
|
||||||
|
except ValidationError as exc:
|
||||||
|
errors[e + "path"] = str(exc)
|
||||||
|
default_name = t.path.rstrip("/").rsplit("/", 1)[-1]
|
||||||
|
else:
|
||||||
|
errors[e + "kind"] = "Ungültige Zielart."
|
||||||
|
default_name = ""
|
||||||
|
t.name = t.name or default_name or f"Ziel {n + 2}"
|
||||||
|
if e + "path" not in errors:
|
||||||
|
other = folders.get((t.mount_id, t.path))
|
||||||
|
if other:
|
||||||
|
errors[e + "path"] = f"Derselbe Ordner wie „{other}“."
|
||||||
|
folders[(t.mount_id, t.path)] = t.name
|
||||||
|
if any(x.name == t.name for x in result) or t.name == "Hauptziel":
|
||||||
|
errors[e + "name"] = "Name kommt in dieser Regel schon vor."
|
||||||
|
result.append(t)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def rule_copy_name(s: Session, name: str) -> str:
|
||||||
|
"""„Scans (Kopie)“, „Scans (Kopie 2)“, … – frei in der Datenbank."""
|
||||||
|
base = f"{name} (Kopie)"[:100]
|
||||||
|
candidate, n = base, 1
|
||||||
|
while s.scalar(select(Rule.id).where(Rule.name == candidate)) is not None:
|
||||||
|
n += 1
|
||||||
|
candidate = f"{name} (Kopie {n})"[:100]
|
||||||
|
return candidate
|
||||||
|
|
||||||
|
|
||||||
|
def _save_targets(s: Session, r: Rule, targets: list[TargetInput]) -> None:
|
||||||
|
"""Weitere Ziele an Ort und Stelle aktualisieren – so bleibt ihre Buchführung
|
||||||
|
erhalten. Ändert sich Anbindung oder Ordner eines Ziels, beginnt sie neu."""
|
||||||
|
existing = list(r.targets)
|
||||||
|
for pos, ti in enumerate(targets):
|
||||||
|
kind = TargetKind(ti.kind)
|
||||||
|
mount_id = ti.mount_id if kind is TargetKind.MOUNT else None
|
||||||
|
if pos < len(existing):
|
||||||
|
t = existing[pos]
|
||||||
|
if t.id is not None and (t.kind, t.mount_id, t.path) != (kind, mount_id, ti.path):
|
||||||
|
s.execute(delete(TargetFile).where(TargetFile.target_id == t.id))
|
||||||
|
else:
|
||||||
|
t = RuleTarget()
|
||||||
|
r.targets.append(t)
|
||||||
|
t.position, t.name, t.kind, t.mount_id = pos, ti.name, kind, mount_id
|
||||||
|
t.path, t.layout = ti.path, ti.layout
|
||||||
|
for t in existing[len(targets) :]:
|
||||||
|
r.targets.remove(t)
|
||||||
|
|
||||||
|
|
||||||
def save_rule(s: Session, d: RuleInput, rule_id: int | None = None) -> tuple[Rule | None, Errors]:
|
def save_rule(s: Session, d: RuleInput, rule_id: int | None = None) -> tuple[Rule | None, Errors]:
|
||||||
dup = s.scalar(select(Rule.id).where(Rule.name == d.name))
|
dup = s.scalar(select(Rule.id).where(Rule.name == d.name))
|
||||||
if dup is not None and dup != rule_id:
|
if dup is not None and dup != rule_id:
|
||||||
@@ -536,6 +770,7 @@ def save_rule(s: Session, d: RuleInput, rule_id: int | None = None) -> tuple[Rul
|
|||||||
"quarantine_after",
|
"quarantine_after",
|
||||||
):
|
):
|
||||||
setattr(r, f, getattr(d, f))
|
setattr(r, f, getattr(d, f))
|
||||||
|
_save_targets(s, r, d.targets)
|
||||||
r.set_owner = d.set_owner or None
|
r.set_owner = d.set_owner or None
|
||||||
r.set_mode = d.set_mode or None
|
r.set_mode = d.set_mode or None
|
||||||
r.quarantine_path = d.quarantine_path or None
|
r.quarantine_path = d.quarantine_path or None
|
||||||
|
|||||||
@@ -7,22 +7,28 @@ Ende: Scheduler stoppen, Mounts **nicht** aushängen (andere könnten sie nutzen
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
|
from netmover import kuma
|
||||||
|
from netmover.backup import STALE_AFTER, BackupError, BackupService, load_state
|
||||||
from netmover.config import Settings
|
from netmover.config import Settings
|
||||||
from netmover.crypto import SecretBox, load_or_create_key
|
from netmover.crypto import SecretBox, load_or_create_key
|
||||||
from netmover.db import make_engine, make_session_factory, upgrade
|
from netmover.db import make_engine, make_session_factory, session_scope, upgrade
|
||||||
from netmover.health import Health, check_health
|
from netmover.health import Health, backup_check, check_health
|
||||||
|
from netmover.health import metrics as health_metrics
|
||||||
from netmover.logging_setup import event, get_logger
|
from netmover.logging_setup import event, get_logger
|
||||||
from netmover.models import utcnow
|
from netmover.models import Mount, utcnow
|
||||||
from netmover.mounting.backend import MountBackend
|
from netmover.mounting.backend import MountBackend
|
||||||
from netmover.mounting.fake import FakeMountBackend
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
from netmover.mounting.linux import LinuxMountBackend
|
from netmover.mounting.linux import LinuxMountBackend
|
||||||
from netmover.mounting.manager import MountManager, RestoreResult
|
from netmover.mounting.manager import MountManager, RestoreResult
|
||||||
|
from netmover.mover.destinations import Destination
|
||||||
from netmover.mover.runner import RuleRunner
|
from netmover.mover.runner import RuleRunner
|
||||||
from netmover.notify import Notifier
|
from netmover.notify import Notification, Notifier
|
||||||
from netmover.runs import RunService
|
from netmover.runs import RunService
|
||||||
from netmover.scheduler import RuleScheduler
|
from netmover.scheduler import RuleScheduler
|
||||||
|
from netmover.settings_store import load_settings
|
||||||
|
|
||||||
log = get_logger(__name__)
|
log = get_logger(__name__)
|
||||||
|
|
||||||
@@ -48,7 +54,7 @@ class NetMover:
|
|||||||
self.session_factory = make_session_factory(self.db_engine)
|
self.session_factory = make_session_factory(self.db_engine)
|
||||||
self.secrets = SecretBox(load_or_create_key(settings.secret_key_path))
|
self.secrets = SecretBox(load_or_create_key(settings.secret_key_path))
|
||||||
self.backend = backend or make_backend(settings)
|
self.backend = backend or make_backend(settings)
|
||||||
self.notifier = notifier or Notifier(self.session_factory)
|
self.notifier = notifier or Notifier(self.session_factory, secrets=self.secrets)
|
||||||
self.mounts = MountManager(
|
self.mounts = MountManager(
|
||||||
self.session_factory,
|
self.session_factory,
|
||||||
self.backend,
|
self.backend,
|
||||||
@@ -58,8 +64,19 @@ class NetMover:
|
|||||||
)
|
)
|
||||||
self.runner = RuleRunner(self.session_factory, self.mounts)
|
self.runner = RuleRunner(self.session_factory, self.mounts)
|
||||||
self.runs = RunService(self.session_factory, self.runner, self.notifier)
|
self.runs = RunService(self.session_factory, self.runner, self.notifier)
|
||||||
|
self.backups = BackupService(
|
||||||
|
settings,
|
||||||
|
self.session_factory,
|
||||||
|
self.backup_destination,
|
||||||
|
self.secrets,
|
||||||
|
notify=lambda kind, title, msg: self.notifier.send(Notification(kind, title, msg)),
|
||||||
|
)
|
||||||
self.scheduler = RuleScheduler(
|
self.scheduler = RuleScheduler(
|
||||||
self.session_factory, self.runs, retry_mounts=self.mounts.retry_pending
|
self.session_factory,
|
||||||
|
self.runs,
|
||||||
|
retry_mounts=self.mounts.retry_pending,
|
||||||
|
push_health=self.push_health,
|
||||||
|
backup_tick=self.backups.tick,
|
||||||
)
|
)
|
||||||
self.started_at: datetime = utcnow()
|
self.started_at: datetime = utcnow()
|
||||||
self.restore_results: list[RestoreResult] = []
|
self.restore_results: list[RestoreResult] = []
|
||||||
@@ -76,10 +93,39 @@ class NetMover:
|
|||||||
self.scheduler.shutdown()
|
self.scheduler.shutdown()
|
||||||
self.db_engine.dispose()
|
self.db_engine.dispose()
|
||||||
|
|
||||||
|
def push_health(self) -> None:
|
||||||
|
"""Scheduler-Job (jede Minute): Zustand an Uptime Kuma melden, falls eingerichtet."""
|
||||||
|
url = load_settings(self.session_factory).kuma_push_url
|
||||||
|
if url:
|
||||||
|
kuma.push(url, self.health(), metrics=health_metrics(self.session_factory))
|
||||||
|
|
||||||
|
def backup_destination(self, mount_id: int, path: str) -> Callable[[], Destination]:
|
||||||
|
"""Ziel der Datensicherung – dieselbe Prüfung wie für Regelziele."""
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
m = s.get(Mount, mount_id)
|
||||||
|
if m is None:
|
||||||
|
raise BackupError("Die Anbindung für die Sicherung existiert nicht mehr")
|
||||||
|
mtype, name = m.type, m.name
|
||||||
|
mp = self.mounts.check_alive(mount_id)
|
||||||
|
return self.runner.destination_factory("backup_path", mount_id, mtype, name, path, mp, None)
|
||||||
|
|
||||||
def health(self) -> Health:
|
def health(self) -> Health:
|
||||||
return check_health(
|
cfg = load_settings(self.session_factory)
|
||||||
|
health = check_health(
|
||||||
self.session_factory,
|
self.session_factory,
|
||||||
scheduler_running=self.scheduler.running,
|
scheduler_running=self.scheduler.running,
|
||||||
last_heartbeat=self.scheduler.last_heartbeat,
|
last_heartbeat=self.scheduler.last_heartbeat,
|
||||||
service_started=self.started_at,
|
service_started=self.started_at,
|
||||||
|
maintenance=cfg.maintenance,
|
||||||
)
|
)
|
||||||
|
st = load_state(self.session_factory)
|
||||||
|
health.add(
|
||||||
|
backup_check(
|
||||||
|
enabled=cfg.backup_enabled and bool(cfg.backup_mount_id),
|
||||||
|
last_success=st.last_success,
|
||||||
|
last_error=st.last_error,
|
||||||
|
stale_after=STALE_AFTER,
|
||||||
|
service_started=self.started_at,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return health
|
||||||
|
|||||||
@@ -0,0 +1,514 @@
|
|||||||
|
"""Datensicherung: Datenbank und Schlüssel täglich auf eine Anbindung (z. B. das NAS).
|
||||||
|
|
||||||
|
Ohne ``secret.key`` sind alle gespeicherten Zugangsdaten verloren – die Sicherung
|
||||||
|
enthält deshalb Datenbank **und** Schlüssel. Damit die Sicherung auf dem NAS nicht
|
||||||
|
selbst zum Schlüsselbund für alle Freigaben wird, verschlüsselt NetMover sie mit
|
||||||
|
einer Passphrase (scrypt + Fernet). Die Passphrase liegt verschlüsselt in der
|
||||||
|
Datenbank, damit die tägliche Sicherung ohne Eingabe läuft; zum Wiederherstellen
|
||||||
|
braucht man sie aus dem Passwortmanager.
|
||||||
|
|
||||||
|
Ablauf einer Sicherung:
|
||||||
|
|
||||||
|
1. Momentaufnahme der Datenbank über die SQLite-Backup-API (konsistent, auch
|
||||||
|
während Regeln laufen) und ``PRAGMA integrity_check``.
|
||||||
|
2. tar.gz mit ``manifest.json``, Datenbank, Schlüsseln und ``netmover.env``.
|
||||||
|
3. Verschlüsseln und sofort probeweise wieder öffnen – eine Sicherung, die sich
|
||||||
|
nicht lesen lässt, wird gar nicht erst hochgeladen.
|
||||||
|
4. Hochladen wie jede andere Datei (``.part`` → prüfen → atomar umbenennen).
|
||||||
|
5. Älteste Sicherungen über ``backup_keep`` hinaus löschen – erst nach Erfolg.
|
||||||
|
|
||||||
|
Format verschlüsselt (``.nmbak``): ``MAGIC`` + 16 Byte Salz + Fernet-Token.
|
||||||
|
Unverschlüsselt: ein gewöhnliches ``.tar.gz``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import socket
|
||||||
|
import sqlite3
|
||||||
|
import tarfile
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from cryptography.fernet import Fernet, InvalidToken
|
||||||
|
from cryptography.hazmat.primitives.kdf.scrypt import Scrypt
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover import __version__
|
||||||
|
from netmover.config import Settings
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
|
from netmover.logging_setup import event, get_logger
|
||||||
|
from netmover.models import CollisionPolicy, utcnow
|
||||||
|
from netmover.mover.destinations import MetaOptions
|
||||||
|
from netmover.settings_store import AppSettings, get_value, load_settings, set_value
|
||||||
|
from netmover.timeouts import CancelToken, run_with_watchdog
|
||||||
|
|
||||||
|
log = get_logger(__name__)
|
||||||
|
|
||||||
|
MAGIC = b"NETMOVER-BACKUP-1\n"
|
||||||
|
SALT_BYTES = 16
|
||||||
|
SCRYPT_N = 2**15
|
||||||
|
SCRYPT_R = 8
|
||||||
|
SCRYPT_P = 1
|
||||||
|
MIN_PASSPHRASE = 12
|
||||||
|
|
||||||
|
SUFFIX_ENC = ".nmbak"
|
||||||
|
SUFFIX_PLAIN = ".tar.gz"
|
||||||
|
NAME_RE = re.compile(r"^netmover-[A-Za-z0-9._-]+-\d{8}-\d{6}(\.nmbak|\.tar\.gz)$")
|
||||||
|
|
||||||
|
DB_NAME = "netmover.db"
|
||||||
|
KEY_FILES = ("secret.key", "session.key")
|
||||||
|
ENV_NAME = "netmover.env"
|
||||||
|
|
||||||
|
# Wann eine Sicherung als überfällig gilt (für /healthz und Kuma): zwei Tage plus
|
||||||
|
# Puffer – ein einzelner Ausfall des NAS über Nacht ist noch kein Alarm.
|
||||||
|
STALE_AFTER = timedelta(hours=50)
|
||||||
|
RETRY_AFTER = timedelta(hours=1)
|
||||||
|
IDLE_TIMEOUT = 120.0
|
||||||
|
|
||||||
|
STATE_PREFIX = "backup."
|
||||||
|
|
||||||
|
|
||||||
|
class BackupError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# -- Verschlüsselung -----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _derive(passphrase: str, salt: bytes) -> bytes:
|
||||||
|
kdf = Scrypt(salt=salt, length=32, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P)
|
||||||
|
return base64.urlsafe_b64encode(kdf.derive(passphrase.encode()))
|
||||||
|
|
||||||
|
|
||||||
|
def encrypt(data: bytes, passphrase: str) -> bytes:
|
||||||
|
salt = os.urandom(SALT_BYTES)
|
||||||
|
return MAGIC + salt + Fernet(_derive(passphrase, salt)).encrypt(data)
|
||||||
|
|
||||||
|
|
||||||
|
def is_encrypted(blob: bytes) -> bool:
|
||||||
|
return blob.startswith(MAGIC)
|
||||||
|
|
||||||
|
|
||||||
|
def decrypt(blob: bytes, passphrase: str) -> bytes:
|
||||||
|
if not is_encrypted(blob):
|
||||||
|
raise BackupError("Keine verschlüsselte NetMover-Sicherung")
|
||||||
|
salt = blob[len(MAGIC) : len(MAGIC) + SALT_BYTES]
|
||||||
|
token = blob[len(MAGIC) + SALT_BYTES :]
|
||||||
|
try:
|
||||||
|
return Fernet(_derive(passphrase, salt)).decrypt(token)
|
||||||
|
except InvalidToken:
|
||||||
|
raise BackupError("Passphrase falsch oder Sicherung beschädigt") from None
|
||||||
|
|
||||||
|
|
||||||
|
def passphrase_problem(passphrase: str, confirm: str) -> str | None:
|
||||||
|
if len(passphrase) < MIN_PASSPHRASE:
|
||||||
|
return f"Passphrase: mindestens {MIN_PASSPHRASE} Zeichen."
|
||||||
|
if passphrase != confirm:
|
||||||
|
return "Passphrasen stimmen nicht überein."
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# -- Archiv bauen und lesen ------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Archive:
|
||||||
|
manifest: dict[str, Any]
|
||||||
|
files: dict[str, bytes]
|
||||||
|
|
||||||
|
|
||||||
|
def snapshot_db(db_path: Path, target: Path) -> str:
|
||||||
|
"""Konsistente Kopie der laufenden Datenbank; liefert die Alembic-Revision."""
|
||||||
|
src = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
|
||||||
|
dst = sqlite3.connect(target)
|
||||||
|
try:
|
||||||
|
src.backup(dst)
|
||||||
|
status = dst.execute("PRAGMA integrity_check").fetchone()[0]
|
||||||
|
if status != "ok":
|
||||||
|
raise BackupError(f"Datenbank beschädigt: {status}")
|
||||||
|
row = dst.execute("SELECT version_num FROM alembic_version").fetchone()
|
||||||
|
finally:
|
||||||
|
dst.close()
|
||||||
|
src.close()
|
||||||
|
return str(row[0]) if row else ""
|
||||||
|
|
||||||
|
|
||||||
|
def _add(tar: tarfile.TarFile, name: str, data: bytes, mtime: float) -> None:
|
||||||
|
info = tarfile.TarInfo(name)
|
||||||
|
info.size = len(data)
|
||||||
|
info.mode = 0o600
|
||||||
|
info.mtime = int(mtime)
|
||||||
|
tar.addfile(info, io.BytesIO(data))
|
||||||
|
|
||||||
|
|
||||||
|
def build_archive(settings: Settings, now: datetime | None = None) -> tuple[bytes, dict[str, Any]]:
|
||||||
|
"""tar.gz mit Datenbank, Schlüsseln und Umgebungsdatei (unverschlüsselt)."""
|
||||||
|
now = now or utcnow()
|
||||||
|
with tempfile.TemporaryDirectory(dir=settings.data_dir, prefix=".backup-") as tmp:
|
||||||
|
snap = Path(tmp) / DB_NAME
|
||||||
|
revision = snapshot_db(settings.db_path, snap)
|
||||||
|
files: dict[str, bytes] = {DB_NAME: snap.read_bytes()}
|
||||||
|
for name in KEY_FILES:
|
||||||
|
path = settings.data_dir / name
|
||||||
|
if path.is_file():
|
||||||
|
files[name] = path.read_bytes()
|
||||||
|
if settings.config_file.is_file():
|
||||||
|
files[ENV_NAME] = settings.config_file.read_bytes()
|
||||||
|
manifest: dict[str, Any] = {
|
||||||
|
"format": 1,
|
||||||
|
"version": __version__,
|
||||||
|
"revision": revision,
|
||||||
|
"created": now.isoformat(),
|
||||||
|
"host": socket.gethostname(),
|
||||||
|
"files": sorted(files),
|
||||||
|
}
|
||||||
|
buf = io.BytesIO()
|
||||||
|
stamp = time.time()
|
||||||
|
with tarfile.open(fileobj=buf, mode="w:gz") as tar:
|
||||||
|
_add(tar, "manifest.json", json.dumps(manifest, indent=2).encode(), stamp)
|
||||||
|
for name, data in files.items():
|
||||||
|
_add(tar, name, data, stamp)
|
||||||
|
return buf.getvalue(), manifest
|
||||||
|
|
||||||
|
|
||||||
|
def open_archive(blob: bytes, passphrase: str | None) -> Archive:
|
||||||
|
"""Sicherung lesen und prüfen (Manifest, Datenbank-Integrität)."""
|
||||||
|
if is_encrypted(blob):
|
||||||
|
if not passphrase:
|
||||||
|
raise BackupError("Sicherung ist verschlüsselt – Passphrase angeben")
|
||||||
|
blob = decrypt(blob, passphrase)
|
||||||
|
files: dict[str, bytes] = {}
|
||||||
|
try:
|
||||||
|
with tarfile.open(fileobj=io.BytesIO(blob), mode="r:gz") as tar:
|
||||||
|
for member in tar.getmembers():
|
||||||
|
if not member.isfile() or "/" in member.name or member.name.startswith("."):
|
||||||
|
raise BackupError(f"Unerwarteter Eintrag in der Sicherung: {member.name}")
|
||||||
|
fh = tar.extractfile(member)
|
||||||
|
if fh is not None:
|
||||||
|
files[member.name] = fh.read()
|
||||||
|
except (tarfile.TarError, OSError, EOFError) as exc:
|
||||||
|
raise BackupError(f"Keine lesbare Sicherung: {exc}") from None
|
||||||
|
try:
|
||||||
|
manifest = json.loads(files.pop("manifest.json"))
|
||||||
|
except (KeyError, ValueError):
|
||||||
|
raise BackupError("Sicherung ohne gültiges manifest.json") from None
|
||||||
|
if manifest.get("format") != 1:
|
||||||
|
raise BackupError(f"Unbekanntes Sicherungsformat {manifest.get('format')!r}")
|
||||||
|
if DB_NAME not in files or "secret.key" not in files:
|
||||||
|
raise BackupError("Sicherung unvollständig: Datenbank oder secret.key fehlt")
|
||||||
|
_check_db_bytes(files[DB_NAME])
|
||||||
|
return Archive(manifest, files)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_db_bytes(data: bytes) -> None:
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = Path(tmp) / DB_NAME
|
||||||
|
path.write_bytes(data)
|
||||||
|
try:
|
||||||
|
con = sqlite3.connect(path)
|
||||||
|
try:
|
||||||
|
status = con.execute("PRAGMA integrity_check").fetchone()[0]
|
||||||
|
finally:
|
||||||
|
con.close()
|
||||||
|
except sqlite3.DatabaseError as exc:
|
||||||
|
raise BackupError(f"Datenbank in der Sicherung unlesbar: {exc}") from None
|
||||||
|
if status != "ok":
|
||||||
|
raise BackupError(f"Datenbank in der Sicherung beschädigt: {status}")
|
||||||
|
|
||||||
|
|
||||||
|
def backup_name(now: datetime, encrypted: bool, host: str | None = None) -> str:
|
||||||
|
host = re.sub(r"[^A-Za-z0-9._-]", "-", host or socket.gethostname()) or "host"
|
||||||
|
local = now.astimezone()
|
||||||
|
return f"netmover-{host}-{local:%Y%m%d-%H%M%S}{SUFFIX_ENC if encrypted else SUFFIX_PLAIN}"
|
||||||
|
|
||||||
|
|
||||||
|
def make_backup(
|
||||||
|
settings: Settings, passphrase: str | None, now: datetime | None = None
|
||||||
|
) -> tuple[bytes, dict[str, Any]]:
|
||||||
|
"""Fertige Sicherung (verschlüsselt, falls Passphrase) – bereits probeweise geöffnet."""
|
||||||
|
raw, manifest = build_archive(settings, now)
|
||||||
|
blob = encrypt(raw, passphrase) if passphrase else raw
|
||||||
|
open_archive(blob, passphrase)
|
||||||
|
return blob, manifest
|
||||||
|
|
||||||
|
|
||||||
|
def prune_names(names: list[str], keep: int) -> list[str]:
|
||||||
|
"""Die zu löschenden Sicherungen: alles über die ``keep`` neuesten hinaus.
|
||||||
|
Nur Dateien mit NetMover-Namensschema – sonst wird nichts angefasst."""
|
||||||
|
ours = sorted((n for n in names if NAME_RE.match(n)), key=_stamp_of, reverse=True)
|
||||||
|
return ours[max(keep, 1) :]
|
||||||
|
|
||||||
|
|
||||||
|
def _stamp_of(name: str) -> str:
|
||||||
|
m = re.search(r"-(\d{8}-\d{6})\.", name)
|
||||||
|
return m.group(1) if m else ""
|
||||||
|
|
||||||
|
|
||||||
|
# -- Wiederherstellen ----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _version_tuple(v: str) -> tuple[int, ...]:
|
||||||
|
return tuple(int(p) for p in re.findall(r"\d+", v)[:3])
|
||||||
|
|
||||||
|
|
||||||
|
def restore(settings: Settings, archive: Archive, now: datetime | None = None) -> Path:
|
||||||
|
"""Spielt eine Sicherung ein. Der Dienst muss gestoppt sein. Vorhandene Dateien
|
||||||
|
wandern vorher nach ``data_dir/vor-wiederherstellung-<zeit>/`` – nichts geht
|
||||||
|
verloren. Liefert diesen Ordner."""
|
||||||
|
version = str(archive.manifest.get("version", "0"))
|
||||||
|
if _version_tuple(version) > _version_tuple(__version__):
|
||||||
|
raise BackupError(
|
||||||
|
f"Sicherung stammt von NetMover {version}, installiert ist {__version__} – "
|
||||||
|
"erst aktualisieren, dann wiederherstellen."
|
||||||
|
)
|
||||||
|
now = now or utcnow()
|
||||||
|
data_dir = settings.data_dir
|
||||||
|
data_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
keep = data_dir / f"vor-wiederherstellung-{now.astimezone():%Y%m%d-%H%M%S}"
|
||||||
|
keep.mkdir(mode=0o700)
|
||||||
|
for name in (DB_NAME, f"{DB_NAME}-wal", f"{DB_NAME}-shm", *KEY_FILES):
|
||||||
|
path = data_dir / name
|
||||||
|
if path.exists():
|
||||||
|
path.rename(keep / name)
|
||||||
|
for name, data in archive.files.items():
|
||||||
|
if name == ENV_NAME:
|
||||||
|
continue
|
||||||
|
_write_private(data_dir / name, data)
|
||||||
|
if ENV_NAME in archive.files:
|
||||||
|
env = settings.config_file
|
||||||
|
if env.is_file() and env.read_bytes() != archive.files[ENV_NAME]:
|
||||||
|
_write_private(keep / ENV_NAME, env.read_bytes())
|
||||||
|
if env.parent.is_dir():
|
||||||
|
_write_private(env, archive.files[ENV_NAME])
|
||||||
|
event(log, "backup.restored", version=version, kept=str(keep))
|
||||||
|
return keep
|
||||||
|
|
||||||
|
|
||||||
|
def _write_private(path: Path, data: bytes) -> None:
|
||||||
|
tmp = path.with_name(f".{path.name}.restore")
|
||||||
|
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||||
|
with os.fdopen(fd, "wb") as fh:
|
||||||
|
fh.write(data)
|
||||||
|
fh.flush()
|
||||||
|
os.fsync(fh.fileno())
|
||||||
|
os.replace(tmp, path)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Dienst: tägliche Sicherung ------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class BackupState:
|
||||||
|
last_attempt: datetime | None = None
|
||||||
|
last_success: datetime | None = None
|
||||||
|
last_error: str = ""
|
||||||
|
last_file: str = ""
|
||||||
|
last_size: int = 0
|
||||||
|
failures: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
def _dt(raw: str | None) -> datetime | None:
|
||||||
|
if not raw:
|
||||||
|
return None
|
||||||
|
value = json.loads(raw)
|
||||||
|
return datetime.fromisoformat(value) if value else None
|
||||||
|
|
||||||
|
|
||||||
|
def load_state(factory: sessionmaker[Session]) -> BackupState:
|
||||||
|
def get(key: str) -> str | None:
|
||||||
|
return get_value(factory, STATE_PREFIX + key)
|
||||||
|
|
||||||
|
return BackupState(
|
||||||
|
last_attempt=_dt(get("last_attempt")),
|
||||||
|
last_success=_dt(get("last_success")),
|
||||||
|
last_error=json.loads(get("last_error") or '""'),
|
||||||
|
last_file=json.loads(get("last_file") or '""'),
|
||||||
|
last_size=json.loads(get("last_size") or "0"),
|
||||||
|
failures=json.loads(get("failures") or "0"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def save_state(factory: sessionmaker[Session], st: BackupState) -> None:
|
||||||
|
values: dict[str, Any] = {
|
||||||
|
"last_attempt": st.last_attempt.isoformat() if st.last_attempt else None,
|
||||||
|
"last_success": st.last_success.isoformat() if st.last_success else None,
|
||||||
|
"last_error": st.last_error,
|
||||||
|
"last_file": st.last_file,
|
||||||
|
"last_size": st.last_size,
|
||||||
|
"failures": st.failures,
|
||||||
|
}
|
||||||
|
for key, value in values.items():
|
||||||
|
set_value(factory, STATE_PREFIX + key, json.dumps(value))
|
||||||
|
|
||||||
|
|
||||||
|
def parse_time(value: str) -> tuple[int, int]:
|
||||||
|
m = re.fullmatch(r"([01]?\d|2[0-3]):([0-5]\d)", value.strip())
|
||||||
|
if not m:
|
||||||
|
raise ValueError("Uhrzeit als HH:MM, z. B. 03:30")
|
||||||
|
return int(m.group(1)), int(m.group(2))
|
||||||
|
|
||||||
|
|
||||||
|
def due_slot(cfg: AppSettings, now: datetime) -> datetime:
|
||||||
|
"""Der letzte Sicherungstermin (Ortszeit des Servers) vor ``now``."""
|
||||||
|
hour, minute = parse_time(cfg.backup_time)
|
||||||
|
local = now.astimezone()
|
||||||
|
slot = local.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||||
|
if slot > local:
|
||||||
|
slot -= timedelta(days=1)
|
||||||
|
return slot
|
||||||
|
|
||||||
|
|
||||||
|
def is_due(cfg: AppSettings, st: BackupState, now: datetime) -> bool:
|
||||||
|
if not cfg.backup_enabled or not cfg.backup_mount_id:
|
||||||
|
return False
|
||||||
|
slot = due_slot(cfg, now)
|
||||||
|
if st.last_success is not None and st.last_success >= slot:
|
||||||
|
return False
|
||||||
|
# Nach einem Fehlschlag stündlich erneut, nicht bei jedem Takt
|
||||||
|
return st.last_attempt is None or now - st.last_attempt >= RETRY_AFTER
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class BackupResult:
|
||||||
|
ok: bool
|
||||||
|
message: str
|
||||||
|
file: str = ""
|
||||||
|
size: int = 0
|
||||||
|
deleted: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
Notify = Callable[[str, str, str], None] # (event, title, message)
|
||||||
|
|
||||||
|
|
||||||
|
class BackupService:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
settings: Settings,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
destination_for: Callable[[int, str], Callable[[], Any]],
|
||||||
|
secrets: SecretBox,
|
||||||
|
notify: Notify | None = None,
|
||||||
|
) -> None:
|
||||||
|
self.settings = settings
|
||||||
|
self.session_factory = session_factory
|
||||||
|
self._destination_for = destination_for
|
||||||
|
self.secrets = secrets
|
||||||
|
self._notify = notify
|
||||||
|
self._running = False
|
||||||
|
|
||||||
|
@property
|
||||||
|
def running(self) -> bool:
|
||||||
|
return self._running
|
||||||
|
|
||||||
|
def passphrase(self, cfg: AppSettings) -> str | None:
|
||||||
|
if not cfg.backup_passphrase_enc:
|
||||||
|
return None
|
||||||
|
return self.secrets.decrypt(cfg.backup_passphrase_enc)
|
||||||
|
|
||||||
|
def tick(self, now: datetime | None = None) -> BackupResult | None:
|
||||||
|
"""Scheduler-Job: sichert, wenn ein Termin fällig ist."""
|
||||||
|
now = now or utcnow()
|
||||||
|
cfg = load_settings(self.session_factory)
|
||||||
|
if not is_due(cfg, load_state(self.session_factory), now):
|
||||||
|
return None
|
||||||
|
return self.run(now)
|
||||||
|
|
||||||
|
def run(self, now: datetime | None = None) -> BackupResult:
|
||||||
|
if self._running:
|
||||||
|
return BackupResult(False, "Eine Sicherung läuft bereits.")
|
||||||
|
self._running = True
|
||||||
|
try:
|
||||||
|
return self._run(now or utcnow())
|
||||||
|
finally:
|
||||||
|
self._running = False
|
||||||
|
|
||||||
|
def _run(self, now: datetime) -> BackupResult:
|
||||||
|
cfg = load_settings(self.session_factory)
|
||||||
|
st = load_state(self.session_factory)
|
||||||
|
st.last_attempt = now
|
||||||
|
try:
|
||||||
|
if not cfg.backup_mount_id:
|
||||||
|
raise BackupError("Kein Sicherungsziel eingestellt")
|
||||||
|
if not cfg.backup_passphrase_enc:
|
||||||
|
raise BackupError("Keine Passphrase gesetzt")
|
||||||
|
result = self._write(cfg, now)
|
||||||
|
except Exception as exc:
|
||||||
|
message = str(exc) or type(exc).__name__
|
||||||
|
st.failures += 1
|
||||||
|
st.last_error = message
|
||||||
|
save_state(self.session_factory, st)
|
||||||
|
event(log, "backup.failed", level=40, error=message, failures=st.failures)
|
||||||
|
if st.failures == 1 and self._notify:
|
||||||
|
self._notify(
|
||||||
|
"backup.failed",
|
||||||
|
"NetMover: Sicherung fehlgeschlagen",
|
||||||
|
f"{message}\nNächster Versuch in einer Stunde.",
|
||||||
|
)
|
||||||
|
return BackupResult(False, message)
|
||||||
|
recovered = st.failures > 0
|
||||||
|
st.failures = 0
|
||||||
|
st.last_error = ""
|
||||||
|
st.last_success = now
|
||||||
|
st.last_file = result.file
|
||||||
|
st.last_size = result.size
|
||||||
|
save_state(self.session_factory, st)
|
||||||
|
event(log, "backup.done", file=result.file, size=result.size, deleted=result.deleted)
|
||||||
|
if recovered and self._notify:
|
||||||
|
self._notify(
|
||||||
|
"backup.recovered",
|
||||||
|
"NetMover: Sicherung läuft wieder",
|
||||||
|
f"Sicherung {result.file} geschrieben.",
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def _write(self, cfg: AppSettings, now: datetime) -> BackupResult:
|
||||||
|
passphrase = self.passphrase(cfg)
|
||||||
|
blob, _manifest = make_backup(self.settings, passphrase, now)
|
||||||
|
name = backup_name(now, passphrase is not None)
|
||||||
|
make_dest = self._destination_for(cfg.backup_mount_id, cfg.backup_path)
|
||||||
|
|
||||||
|
def work(token: CancelToken) -> BackupResult:
|
||||||
|
dest = make_dest()
|
||||||
|
try:
|
||||||
|
dest.prepare()
|
||||||
|
with tempfile.TemporaryDirectory(dir=self.settings.data_dir) as tmp:
|
||||||
|
local = Path(tmp) / name
|
||||||
|
local.write_bytes(blob)
|
||||||
|
ns = time.time_ns()
|
||||||
|
result = dest.commit(
|
||||||
|
local,
|
||||||
|
len(blob),
|
||||||
|
name,
|
||||||
|
CollisionPolicy.RENAME,
|
||||||
|
MetaOptions(ns, ns, 0o600, None),
|
||||||
|
lambda: None,
|
||||||
|
token,
|
||||||
|
)
|
||||||
|
deleted = 0
|
||||||
|
for old in prune_names(dest.list_names(), cfg.backup_keep):
|
||||||
|
token.check()
|
||||||
|
with contextlib.suppress(OSError):
|
||||||
|
dest.remove(old)
|
||||||
|
deleted += 1
|
||||||
|
return BackupResult(
|
||||||
|
True,
|
||||||
|
f"Sicherung {dest.describe(result.final_rel)} geschrieben",
|
||||||
|
result.final_rel,
|
||||||
|
len(blob),
|
||||||
|
deleted,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
dest.close()
|
||||||
|
|
||||||
|
return run_with_watchdog(work, IDLE_TIMEOUT, "Sicherung")
|
||||||
+126
-4
@@ -1,14 +1,18 @@
|
|||||||
"""Kommandozeile: ``netmover serve`` / ``netmover migrate`` / ``netmover version``
|
"""Kommandozeile: ``netmover serve`` / ``netmover migrate`` / ``netmover version``
|
||||||
/ ``netmover reset-password``."""
|
/ ``netmover reset-password`` / ``netmover backup`` / ``netmover restore``."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import getpass
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
from collections.abc import Sequence
|
from collections.abc import Callable, Sequence
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
from netmover import __version__
|
from netmover import __version__
|
||||||
from netmover.config import get_settings
|
from netmover.config import Settings, get_settings
|
||||||
from netmover.db import upgrade
|
from netmover.db import upgrade
|
||||||
from netmover.logging_setup import event, get_logger, setup_logging
|
from netmover.logging_setup import event, get_logger, setup_logging
|
||||||
|
|
||||||
@@ -25,6 +29,17 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
"reset-password",
|
"reset-password",
|
||||||
help="Administratorpasswort löschen – beim nächsten Aufruf startet die Einrichtung",
|
help="Administratorpasswort löschen – beim nächsten Aufruf startet die Einrichtung",
|
||||||
)
|
)
|
||||||
|
b = sub.add_parser("backup", help="Sicherung in eine lokale Datei oder einen Ordner schreiben")
|
||||||
|
b.add_argument("out", type=Path, help="Zieldatei oder Ordner")
|
||||||
|
b.add_argument(
|
||||||
|
"--unverschluesselt",
|
||||||
|
action="store_true",
|
||||||
|
help="ohne Passphrase (die Datei enthält dann alle Zugangsdaten lesbar)",
|
||||||
|
)
|
||||||
|
r = sub.add_parser("restore", help="Sicherung einspielen (Dienst vorher stoppen)")
|
||||||
|
r.add_argument("file", type=Path, help="Sicherungsdatei (.nmbak oder .tar.gz)")
|
||||||
|
r.add_argument("--passphrase-file", type=Path, help="Passphrase aus Datei statt Abfrage")
|
||||||
|
r.add_argument("--ja", action="store_true", help="ohne Rückfrage einspielen")
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
settings = get_settings()
|
settings = get_settings()
|
||||||
@@ -64,12 +79,119 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
row = s.get(Setting, PASSWORD_KEY)
|
row = s.get(Setting, PASSWORD_KEY)
|
||||||
if row is not None:
|
if row is not None:
|
||||||
s.delete(row)
|
s.delete(row)
|
||||||
s.commit()
|
# Anmeldung wieder überall einschalten und Netz-Sperren aufheben
|
||||||
|
for key, value in (
|
||||||
|
("app.auth_mode", '"always"'),
|
||||||
|
("app.trusted_networks", '""'),
|
||||||
|
("app.allowed_networks", '""'),
|
||||||
|
):
|
||||||
|
setting = s.get(Setting, key)
|
||||||
|
if setting is not None:
|
||||||
|
setting.value = value
|
||||||
|
s.commit()
|
||||||
engine.dispose()
|
engine.dispose()
|
||||||
print("Passwort gelöscht. Dienst neu starten; der Einrichtungscode steht im Log.")
|
print("Passwort gelöscht. Dienst neu starten; der Einrichtungscode steht im Log.")
|
||||||
return 0
|
return 0
|
||||||
|
if args.cmd == "backup":
|
||||||
|
return cmd_backup(settings, args.out, encrypted=not args.unverschluesselt)
|
||||||
|
if args.cmd == "restore":
|
||||||
|
return cmd_restore(settings, args.file, args.passphrase_file, assume_yes=args.ja)
|
||||||
return 2
|
return 2
|
||||||
|
|
||||||
|
|
||||||
|
def service_active() -> bool:
|
||||||
|
if shutil.which("systemctl") is None:
|
||||||
|
return False
|
||||||
|
result = subprocess.run(
|
||||||
|
["systemctl", "is-active", "--quiet", "netmover"], # noqa: S607
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
return result.returncode == 0
|
||||||
|
|
||||||
|
|
||||||
|
def _ask_passphrase(confirm: bool) -> str:
|
||||||
|
from netmover.backup import passphrase_problem
|
||||||
|
|
||||||
|
first = getpass.getpass("Passphrase: ")
|
||||||
|
if not confirm:
|
||||||
|
return first
|
||||||
|
problem = passphrase_problem(first, getpass.getpass("Wiederholen: "))
|
||||||
|
if problem:
|
||||||
|
raise SystemExit(problem)
|
||||||
|
return first
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_backup(settings: Settings, out: Path, *, encrypted: bool) -> int:
|
||||||
|
from netmover.backup import backup_name, make_backup
|
||||||
|
from netmover.crypto import SecretBox, load_or_create_key
|
||||||
|
from netmover.db import make_engine, make_session_factory
|
||||||
|
from netmover.models import utcnow
|
||||||
|
from netmover.settings_store import load_settings
|
||||||
|
|
||||||
|
passphrase: str | None = None
|
||||||
|
if encrypted:
|
||||||
|
engine = make_engine(settings.effective_database_url)
|
||||||
|
cfg = load_settings(make_session_factory(engine))
|
||||||
|
engine.dispose()
|
||||||
|
if cfg.backup_passphrase_enc:
|
||||||
|
box = SecretBox(load_or_create_key(settings.secret_key_path))
|
||||||
|
passphrase = box.decrypt(cfg.backup_passphrase_enc)
|
||||||
|
print("Verschlüsselt mit der Passphrase aus den Einstellungen.")
|
||||||
|
else:
|
||||||
|
passphrase = _ask_passphrase(confirm=True)
|
||||||
|
now = utcnow()
|
||||||
|
blob, manifest = make_backup(settings, passphrase, now)
|
||||||
|
target = out / backup_name(now, encrypted) if out.is_dir() else out
|
||||||
|
fd_path = target.with_name(f".{target.name}.part")
|
||||||
|
fd_path.write_bytes(blob)
|
||||||
|
fd_path.chmod(0o600)
|
||||||
|
fd_path.replace(target)
|
||||||
|
print(f"Sicherung geschrieben: {target} ({len(blob)} Bytes, NetMover {manifest['version']})")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_restore(
|
||||||
|
settings: Settings,
|
||||||
|
file: Path,
|
||||||
|
passphrase_file: Path | None,
|
||||||
|
*,
|
||||||
|
assume_yes: bool,
|
||||||
|
active: Callable[[], bool] = service_active,
|
||||||
|
) -> int:
|
||||||
|
from netmover.backup import BackupError, is_encrypted, open_archive, restore
|
||||||
|
|
||||||
|
if active():
|
||||||
|
print("Der Dienst läuft – erst `systemctl stop netmover`, dann wiederherstellen.")
|
||||||
|
return 1
|
||||||
|
blob = file.read_bytes()
|
||||||
|
passphrase: str | None = None
|
||||||
|
if is_encrypted(blob):
|
||||||
|
passphrase = (
|
||||||
|
passphrase_file.read_text().strip() if passphrase_file else _ask_passphrase(False)
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
archive = open_archive(blob, passphrase)
|
||||||
|
except BackupError as exc:
|
||||||
|
print(f"Fehler: {exc}")
|
||||||
|
return 1
|
||||||
|
m = archive.manifest
|
||||||
|
print(
|
||||||
|
f"Sicherung von {m.get('host')} vom {m.get('created')} (NetMover {m.get('version')}),"
|
||||||
|
f" enthält: {', '.join(sorted(archive.files))}"
|
||||||
|
)
|
||||||
|
if not assume_yes and input(f"Nach {settings.data_dir} einspielen? [ja/nein] ") != "ja":
|
||||||
|
print("Abgebrochen, nichts verändert.")
|
||||||
|
return 1
|
||||||
|
try:
|
||||||
|
kept = restore(settings, archive)
|
||||||
|
except BackupError as exc:
|
||||||
|
print(f"Fehler: {exc}")
|
||||||
|
return 1
|
||||||
|
upgrade(settings.effective_database_url)
|
||||||
|
print(f"Wiederhergestellt. Bisherige Dateien liegen in {kept}.")
|
||||||
|
print("Jetzt `systemctl start netmover`.")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
sys.exit(main())
|
sys.exit(main())
|
||||||
|
|||||||
@@ -21,6 +21,8 @@ class Settings(BaseSettings):
|
|||||||
data_dir: Path = Path("/var/lib/netmover")
|
data_dir: Path = Path("/var/lib/netmover")
|
||||||
run_dir: Path = Path("/run/netmover")
|
run_dir: Path = Path("/run/netmover")
|
||||||
mount_base: Path = Path("/mnt/netmover")
|
mount_base: Path = Path("/mnt/netmover")
|
||||||
|
# Umgebungsdatei – wird mitgesichert und wiederhergestellt
|
||||||
|
config_file: Path = Path("/etc/netmover/netmover.env")
|
||||||
|
|
||||||
host: str = "127.0.0.1"
|
host: str = "127.0.0.1"
|
||||||
port: int = Field(default=8080, ge=1, le=65535)
|
port: int = Field(default=8080, ge=1, le=65535)
|
||||||
|
|||||||
@@ -0,0 +1,308 @@
|
|||||||
|
"""Zugangsdaten-Verwaltung: gespeicherte Passwörter und SSH-Schlüssel.
|
||||||
|
|
||||||
|
Passwörter (SMB und SFTP): Benutzer, Passwort, optional Domäne – einmal gepflegt,
|
||||||
|
von beliebig vielen Anbindungen genutzt. Ein neues Passwort gilt sofort für alle.
|
||||||
|
|
||||||
|
SSH-Schlüssel:
|
||||||
|
|
||||||
|
* Erzeugen (Ed25519) oder Importieren (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH-
|
||||||
|
oder PEM-Format, optional mit Passphrase). Eine Passphrase wird beim Import
|
||||||
|
einmal benutzt; gespeichert wird der Schlüssel ohne Passphrase, aber mit
|
||||||
|
Fernet verschlüsselt – wie alle Geheimnisse.
|
||||||
|
* Der private Schlüssel verlässt NetMover nie: Die Oberfläche zeigt nur den
|
||||||
|
öffentlichen Schlüssel und den Fingerabdruck.
|
||||||
|
* Ein Schlüssel kann von mehreren Anbindungen genutzt werden; den Benutzernamen
|
||||||
|
legt jede Anbindung selbst fest. Löschen geht nur, solange ihn keine nutzt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import hashlib
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from cryptography.hazmat.primitives import serialization
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ec, ed25519, rsa
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from netmover.admin import AdminError
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
|
from netmover.models import Credential, CredentialKind, Mount
|
||||||
|
|
||||||
|
MAX_KEY_BYTES = 16 * 1024
|
||||||
|
MIN_RSA_BITS = 2048
|
||||||
|
SFTP_ONLY_PREFIX = 'restrict,command="/usr/lib/openssh/sftp-server" '
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class KeyInfo:
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
key_type: str
|
||||||
|
fingerprint: str
|
||||||
|
public_key: str
|
||||||
|
created_at: datetime
|
||||||
|
used_by: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def authorized_keys_line(self) -> str:
|
||||||
|
"""Zeile für ~/.ssh/authorized_keys, auf SFTP beschränkt."""
|
||||||
|
return SFTP_ONLY_PREFIX + self.public_key
|
||||||
|
|
||||||
|
|
||||||
|
def fingerprint(public_key: str) -> str:
|
||||||
|
"""SHA256-Fingerabdruck wie ``ssh-keygen -lf``."""
|
||||||
|
blob = base64.b64decode(public_key.split()[1])
|
||||||
|
return "SHA256:" + base64.b64encode(hashlib.sha256(blob).digest()).decode().rstrip("=")
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_name(s: Session, name: str, exclude_id: int | None = None) -> str:
|
||||||
|
name = name.strip()
|
||||||
|
if not name:
|
||||||
|
raise AdminError("Name fehlt.")
|
||||||
|
if len(name) > 100:
|
||||||
|
raise AdminError("Name höchstens 100 Zeichen.")
|
||||||
|
if name.startswith("mount:"):
|
||||||
|
raise AdminError("Namen mit „mount:“ sind für NetMover reserviert.")
|
||||||
|
dup = s.scalar(select(Credential.id).where(Credential.name == name))
|
||||||
|
if dup is not None and dup != exclude_id:
|
||||||
|
raise AdminError(f"Zugangsdaten mit dem Namen „{name}“ existieren bereits.")
|
||||||
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
def _comment(name: str) -> str:
|
||||||
|
return "netmover:" + "".join(c if c.isalnum() or c in "-_." else "-" for c in name)
|
||||||
|
|
||||||
|
|
||||||
|
PrivateKey = ed25519.Ed25519PrivateKey | ec.EllipticCurvePrivateKey | rsa.RSAPrivateKey
|
||||||
|
|
||||||
|
|
||||||
|
def _store(s: Session, secrets: SecretBox, name: str, key: PrivateKey) -> Credential:
|
||||||
|
private = key.private_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PrivateFormat.OpenSSH,
|
||||||
|
serialization.NoEncryption(),
|
||||||
|
).decode()
|
||||||
|
public = (
|
||||||
|
key.public_key()
|
||||||
|
.public_bytes(serialization.Encoding.OpenSSH, serialization.PublicFormat.OpenSSH)
|
||||||
|
.decode()
|
||||||
|
)
|
||||||
|
cred = Credential(
|
||||||
|
name=name,
|
||||||
|
kind=CredentialKind.SSH_KEY,
|
||||||
|
username="",
|
||||||
|
secret_enc=secrets.encrypt(private),
|
||||||
|
public_key=f"{public} {_comment(name)}",
|
||||||
|
managed=True,
|
||||||
|
)
|
||||||
|
s.add(cred)
|
||||||
|
s.flush()
|
||||||
|
return cred
|
||||||
|
|
||||||
|
|
||||||
|
def generate_key(s: Session, secrets: SecretBox, name: str) -> Credential:
|
||||||
|
return _store(s, secrets, _validate_name(s, name), ed25519.Ed25519PrivateKey.generate())
|
||||||
|
|
||||||
|
|
||||||
|
def import_key(
|
||||||
|
s: Session, secrets: SecretBox, name: str, private_key: str, passphrase: str = ""
|
||||||
|
) -> Credential:
|
||||||
|
name = _validate_name(s, name)
|
||||||
|
data = private_key.strip().encode()
|
||||||
|
if not data:
|
||||||
|
raise AdminError("Privaten Schlüssel einfügen.")
|
||||||
|
if len(data) > MAX_KEY_BYTES:
|
||||||
|
raise AdminError("Schlüssel ist zu groß.")
|
||||||
|
if b"PUBLIC KEY" in data or data.startswith((b"ssh-", b"ecdsa-")):
|
||||||
|
raise AdminError("Das ist ein öffentlicher Schlüssel – benötigt wird der private.")
|
||||||
|
password = passphrase.encode() if passphrase else None
|
||||||
|
loaders = (
|
||||||
|
[serialization.load_ssh_private_key]
|
||||||
|
if b"OPENSSH PRIVATE KEY" in data
|
||||||
|
else [serialization.load_pem_private_key]
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
key = loaders[0](data, password)
|
||||||
|
except TypeError:
|
||||||
|
raise AdminError(
|
||||||
|
"Der Schlüssel ist mit einer Passphrase geschützt – bitte Passphrase angeben."
|
||||||
|
if password is None
|
||||||
|
else "Der Schlüssel hat keine Passphrase – Feld bitte leer lassen."
|
||||||
|
) from None
|
||||||
|
except ValueError as exc:
|
||||||
|
msg = str(exc).lower()
|
||||||
|
if password is not None or "password" in msg or "decrypt" in msg:
|
||||||
|
# Bei OpenSSH-Schlüsseln meldet cryptography eine falsche Passphrase
|
||||||
|
# als „Corrupt data“ – beides lässt sich nicht unterscheiden.
|
||||||
|
raise AdminError("Passphrase falsch (oder Schlüssel beschädigt).") from None
|
||||||
|
raise AdminError(
|
||||||
|
"Kein lesbarer privater Schlüssel (erwartet OpenSSH- oder PEM-Format)."
|
||||||
|
) from None
|
||||||
|
except Exception: # unbekannte Formate/Algorithmen sauber melden
|
||||||
|
raise AdminError("Schlüsselformat wird nicht unterstützt.") from None
|
||||||
|
if not isinstance(
|
||||||
|
key, ed25519.Ed25519PrivateKey | ec.EllipticCurvePrivateKey | rsa.RSAPrivateKey
|
||||||
|
):
|
||||||
|
raise AdminError("Unterstützt werden Ed25519, ECDSA und RSA.")
|
||||||
|
if isinstance(key, rsa.RSAPrivateKey) and key.key_size < MIN_RSA_BITS:
|
||||||
|
raise AdminError(f"RSA-Schlüssel mit {key.key_size} Bit ist zu schwach (mind. 2048).")
|
||||||
|
return _store(s, secrets, name, key)
|
||||||
|
|
||||||
|
|
||||||
|
def _users(s: Session, key_id: int) -> list[str]:
|
||||||
|
# Direkt abfragen statt key.mounts: die Beziehung ist nach dem Löschen von
|
||||||
|
# Anbindungen in derselben Session veraltet (passive_deletes).
|
||||||
|
return sorted(s.scalars(select(Mount.name).where(Mount.cred_id == key_id)))
|
||||||
|
|
||||||
|
|
||||||
|
# -- Passwörter -------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class PasswordInfo:
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
username: str
|
||||||
|
domain: str | None
|
||||||
|
created_at: datetime
|
||||||
|
used_by: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_login(username: str, password: str | None, domain: str) -> None:
|
||||||
|
from netmover.validation import ValidationError, validate_secret, validate_username
|
||||||
|
|
||||||
|
try:
|
||||||
|
validate_username(username)
|
||||||
|
if domain:
|
||||||
|
validate_username(domain)
|
||||||
|
if password is not None:
|
||||||
|
validate_secret(password)
|
||||||
|
except ValidationError as exc:
|
||||||
|
raise AdminError(str(exc)) from None
|
||||||
|
|
||||||
|
|
||||||
|
def create_password(
|
||||||
|
s: Session, secrets: SecretBox, name: str, username: str, password: str, domain: str = ""
|
||||||
|
) -> Credential:
|
||||||
|
name = _validate_name(s, name)
|
||||||
|
username, domain = username.strip(), domain.strip()
|
||||||
|
if not password:
|
||||||
|
raise AdminError("Passwort fehlt.")
|
||||||
|
_validate_login(username, password, domain)
|
||||||
|
cred = Credential(
|
||||||
|
name=name,
|
||||||
|
kind=CredentialKind.PASSWORD,
|
||||||
|
username=username,
|
||||||
|
domain=domain or None,
|
||||||
|
secret_enc=secrets.encrypt(password),
|
||||||
|
managed=True,
|
||||||
|
)
|
||||||
|
s.add(cred)
|
||||||
|
s.flush()
|
||||||
|
return cred
|
||||||
|
|
||||||
|
|
||||||
|
def update_password(
|
||||||
|
s: Session, secrets: SecretBox, cred_id: int, username: str, password: str, domain: str = ""
|
||||||
|
) -> Credential:
|
||||||
|
"""Benutzer/Domäne ändern; leeres Passwort = unverändert."""
|
||||||
|
cred = managed_password(s, cred_id)
|
||||||
|
username, domain = username.strip(), domain.strip()
|
||||||
|
_validate_login(username, password or None, domain)
|
||||||
|
cred.username = username
|
||||||
|
cred.domain = domain or None
|
||||||
|
if password:
|
||||||
|
cred.secret_enc = secrets.encrypt(password)
|
||||||
|
return cred
|
||||||
|
|
||||||
|
|
||||||
|
def managed_password(s: Session, cred_id: int) -> Credential:
|
||||||
|
cred = s.get(Credential, cred_id)
|
||||||
|
if cred is None or not cred.managed or cred.kind is not CredentialKind.PASSWORD:
|
||||||
|
raise AdminError("Zugangsdaten existieren nicht.")
|
||||||
|
return cred
|
||||||
|
|
||||||
|
|
||||||
|
def list_passwords(s: Session) -> list[PasswordInfo]:
|
||||||
|
rows = s.scalars(
|
||||||
|
select(Credential)
|
||||||
|
.where(Credential.managed.is_(True), Credential.kind == CredentialKind.PASSWORD)
|
||||||
|
.order_by(Credential.name)
|
||||||
|
).all()
|
||||||
|
return [
|
||||||
|
PasswordInfo(c.id, c.name, c.username, c.domain, c.created_at, _users(s, c.id))
|
||||||
|
for c in rows
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def delete_credential(s: Session, cred_id: int) -> str:
|
||||||
|
"""Gespeicherte Zugangsdaten (Passwort oder Schlüssel) löschen – nur unbenutzt."""
|
||||||
|
cred = s.get(Credential, cred_id)
|
||||||
|
if cred is None or not cred.managed:
|
||||||
|
raise AdminError("Zugangsdaten existieren nicht.")
|
||||||
|
users = _users(s, cred_id)
|
||||||
|
if users:
|
||||||
|
raise AdminError(f"„{cred.name}“ wird noch benutzt von: {', '.join(users)}")
|
||||||
|
name = cred.name
|
||||||
|
s.delete(cred)
|
||||||
|
s.flush()
|
||||||
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
def rename_credential(s: Session, cred_id: int, name: str) -> None:
|
||||||
|
cred = s.get(Credential, cred_id)
|
||||||
|
if cred is None or not cred.managed:
|
||||||
|
raise AdminError("Zugangsdaten existieren nicht.")
|
||||||
|
cred.name = _validate_name(s, name, exclude_id=cred_id)
|
||||||
|
|
||||||
|
|
||||||
|
# -- SSH-Schlüssel ------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def list_keys(s: Session) -> list[KeyInfo]:
|
||||||
|
keys = s.scalars(
|
||||||
|
select(Credential)
|
||||||
|
.where(Credential.managed.is_(True), Credential.kind == CredentialKind.SSH_KEY)
|
||||||
|
.order_by(Credential.name)
|
||||||
|
).all()
|
||||||
|
result = []
|
||||||
|
for k in keys:
|
||||||
|
pub = k.public_key or ""
|
||||||
|
result.append(
|
||||||
|
KeyInfo(
|
||||||
|
id=k.id,
|
||||||
|
name=k.name,
|
||||||
|
key_type=pub.split()[0] if pub else "?",
|
||||||
|
fingerprint=fingerprint(pub) if pub else "?",
|
||||||
|
public_key=pub,
|
||||||
|
created_at=k.created_at,
|
||||||
|
used_by=_users(s, k.id),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def managed_key(s: Session, key_id: int) -> Credential:
|
||||||
|
key = s.get(Credential, key_id)
|
||||||
|
if key is None or not key.managed or key.kind is not CredentialKind.SSH_KEY:
|
||||||
|
raise AdminError("Schlüssel existiert nicht.")
|
||||||
|
return key
|
||||||
|
|
||||||
|
|
||||||
|
def rename_key(s: Session, key_id: int, name: str) -> None:
|
||||||
|
key = managed_key(s, key_id)
|
||||||
|
key.name = _validate_name(s, name, exclude_id=key_id)
|
||||||
|
|
||||||
|
|
||||||
|
def delete_key(s: Session, key_id: int) -> str:
|
||||||
|
key = managed_key(s, key_id)
|
||||||
|
users = _users(s, key_id)
|
||||||
|
if users:
|
||||||
|
raise AdminError(f"Schlüssel „{key.name}“ wird noch benutzt von: {', '.join(users)}")
|
||||||
|
name = key.name
|
||||||
|
s.delete(key)
|
||||||
|
s.flush()
|
||||||
|
return name
|
||||||
@@ -33,6 +33,35 @@ def make_engine(url: str) -> Engine:
|
|||||||
return engine
|
return engine
|
||||||
|
|
||||||
|
|
||||||
|
def make_migration_engine(url: str) -> Engine:
|
||||||
|
"""Verbindung nur für Migrationen.
|
||||||
|
|
||||||
|
* Fremdschlüssel AUS: SQLite baut Tabellen bei Änderungen neu auf (Alembic
|
||||||
|
„batch mode“) und löscht dabei die alte Tabelle. Mit eingeschalteter Prüfung
|
||||||
|
scheitert das, sobald andere Tabellen darauf verweisen (Praxisbefund beim
|
||||||
|
Update auf 0.2.0). Die Integrität prüft ``PRAGMA foreign_key_check`` danach.
|
||||||
|
* Echte Transaktion: pysqlite eigene Transaktionssteuerung abnehmen und selbst
|
||||||
|
``BEGIN`` senden. So wird eine abgebrochene Migration vollständig
|
||||||
|
zurückgerollt, statt einen halben Zustand zu hinterlassen.
|
||||||
|
"""
|
||||||
|
engine = create_engine(url, connect_args={"check_same_thread": False})
|
||||||
|
if url.startswith("sqlite"):
|
||||||
|
|
||||||
|
def on_connect(dbapi_conn: Any, _record: Any) -> None:
|
||||||
|
dbapi_conn.isolation_level = None
|
||||||
|
cur = dbapi_conn.cursor()
|
||||||
|
cur.execute("PRAGMA foreign_keys=OFF") # außerhalb einer Transaktion setzen
|
||||||
|
cur.execute("PRAGMA busy_timeout=5000")
|
||||||
|
cur.close()
|
||||||
|
|
||||||
|
def on_begin(conn: Any) -> None:
|
||||||
|
conn.exec_driver_sql("BEGIN IMMEDIATE")
|
||||||
|
|
||||||
|
event.listen(engine, "connect", on_connect)
|
||||||
|
event.listen(engine, "begin", on_begin)
|
||||||
|
return engine
|
||||||
|
|
||||||
|
|
||||||
def make_session_factory(engine: Engine) -> sessionmaker[Session]:
|
def make_session_factory(engine: Engine) -> sessionmaker[Session]:
|
||||||
return sessionmaker(bind=engine, expire_on_commit=False)
|
return sessionmaker(bind=engine, expire_on_commit=False)
|
||||||
|
|
||||||
|
|||||||
+200
-26
@@ -1,39 +1,77 @@
|
|||||||
"""Gesundheitszustand für ``/healthz`` und die Statusseite (Fallstrick 9).
|
"""Gesundheitszustand für ``/healthz``, Uptime Kuma und die Statusseite (Fallstrick 9).
|
||||||
|
|
||||||
Ein Werkzeug, das nichts mehr tut, darf nicht wie eines aussehen, das nichts zu
|
Ein Werkzeug, das nichts mehr tut, darf nicht wie eines aussehen, das nichts zu
|
||||||
tun hat. Deshalb:
|
tun hat. Geprüft wird:
|
||||||
|
|
||||||
* der Scheduler muss leben (Herzschlag nicht älter als 3 Minuten),
|
* der Scheduler lebt (Herzschlag nicht älter als 3 Minuten),
|
||||||
* keine aktive Anbindung im Fehlerzustand,
|
* jede aktive Anbindung ist nicht im Fehlerzustand,
|
||||||
* keine aktive Regel, deren letzter erfolgreicher Lauf länger als das Dreifache
|
* jede aktive Regel hatte vor höchstens dem Dreifachen ihres Intervalls einen
|
||||||
ihres Intervalls zurückliegt. Bezugspunkt ist der späteste von letztem Erfolg,
|
erfolgreichen Lauf (Bezug: spätester von letztem Erfolg, Anlage der Regel und
|
||||||
Anlage der Regel und Dienststart – sonst wäre direkt nach einem Neustart alles rot.
|
Dienststart – sonst wäre direkt nach einem Neustart alles rot) und ist nicht
|
||||||
|
mehrmals hintereinander gescheitert.
|
||||||
|
|
||||||
|
Jeder Punkt ist ein eigener :class:`Check` – so kann Uptime Kuma auch einzelne
|
||||||
|
Anbindungen und Regeln überwachen (``/healthz/mount/<slug>``, ``/healthz/rule/<id>``).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import asdict, dataclass, field
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
from sqlalchemy import select
|
from sqlalchemy import func, select
|
||||||
from sqlalchemy.orm import Session, sessionmaker
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
from netmover.models import Mount, MountState, Rule, utcnow
|
from netmover.models import (
|
||||||
|
FileFailure,
|
||||||
|
ItemStatus,
|
||||||
|
Mount,
|
||||||
|
MountState,
|
||||||
|
Rule,
|
||||||
|
Run,
|
||||||
|
RunItem,
|
||||||
|
RunStatus,
|
||||||
|
utcnow,
|
||||||
|
)
|
||||||
|
|
||||||
STALE_FACTOR = 3
|
STALE_FACTOR = 3
|
||||||
|
FAILURES_ALERT = 3
|
||||||
HEARTBEAT_MAX_AGE = timedelta(minutes=3)
|
HEARTBEAT_MAX_AGE = timedelta(minutes=3)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Check:
|
||||||
|
kind: str # "scheduler" | "mount" | "rule" | "backup"
|
||||||
|
name: str
|
||||||
|
ok: bool
|
||||||
|
message: str
|
||||||
|
id: int | None = None
|
||||||
|
slug: str | None = None
|
||||||
|
|
||||||
|
def as_dict(self) -> dict[str, object]:
|
||||||
|
return {k: v for k, v in asdict(self).items() if v is not None}
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class Health:
|
class Health:
|
||||||
ok: bool = True
|
ok: bool = True
|
||||||
problems: list[str] = field(default_factory=list)
|
problems: list[str] = field(default_factory=list)
|
||||||
stale_rules: set[int] = field(default_factory=set)
|
stale_rules: set[int] = field(default_factory=set)
|
||||||
|
checks: list[Check] = field(default_factory=list)
|
||||||
|
maintenance: bool = False
|
||||||
|
|
||||||
def fail(self, problem: str) -> None:
|
def add(self, check: Check) -> None:
|
||||||
self.ok = False
|
self.checks.append(check)
|
||||||
self.problems.append(problem)
|
if not check.ok:
|
||||||
|
self.ok = False
|
||||||
|
self.problems.append(check.message)
|
||||||
|
|
||||||
|
def find(self, kind: str, *, slug: str | None = None, id: int | None = None) -> Check | None:
|
||||||
|
for c in self.checks:
|
||||||
|
if c.kind == kind and (slug is None or c.slug == slug) and (id is None or c.id == id):
|
||||||
|
return c
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def rule_is_stale(rule: Rule, service_started: datetime, now: datetime) -> bool:
|
def rule_is_stale(rule: Rule, service_started: datetime, now: datetime) -> bool:
|
||||||
@@ -45,6 +83,50 @@ def rule_is_stale(rule: Rule, service_started: datetime, now: datetime) -> bool:
|
|||||||
return now - reference > timedelta(seconds=STALE_FACTOR * rule.interval_seconds)
|
return now - reference > timedelta(seconds=STALE_FACTOR * rule.interval_seconds)
|
||||||
|
|
||||||
|
|
||||||
|
def _mount_check(m: Mount) -> Check:
|
||||||
|
def mk(ok: bool, message: str) -> Check:
|
||||||
|
return Check("mount", m.name, ok, message, id=m.id, slug=m.slug)
|
||||||
|
|
||||||
|
if not m.enabled:
|
||||||
|
return mk(True, f"Anbindung „{m.name}“ deaktiviert")
|
||||||
|
if m.state is MountState.ERROR:
|
||||||
|
detail = (m.last_error or "").strip()
|
||||||
|
return mk(False, f"Anbindung „{m.name}“ im Fehlerzustand: {detail}".strip())
|
||||||
|
if m.state is MountState.DEGRADED:
|
||||||
|
return mk(True, f"Anbindung „{m.name}“ gestört, wird neu eingehängt")
|
||||||
|
return mk(True, f"Anbindung „{m.name}“ in Ordnung")
|
||||||
|
|
||||||
|
|
||||||
|
def _rule_check(
|
||||||
|
r: Rule, service_started: datetime, now: datetime, maintenance: bool = False
|
||||||
|
) -> Check:
|
||||||
|
def mk(ok: bool, message: str) -> Check:
|
||||||
|
return Check("rule", r.name, ok, message, id=r.id)
|
||||||
|
|
||||||
|
if not r.enabled:
|
||||||
|
return mk(True, f"Regel „{r.name}“ deaktiviert")
|
||||||
|
if maintenance:
|
||||||
|
return mk(True, f"Regel „{r.name}“ pausiert (Wartungsmodus)")
|
||||||
|
if r.waiting_since is not None:
|
||||||
|
return mk(
|
||||||
|
False,
|
||||||
|
f"Regel „{r.name}“ wartet seit {r.waiting_since:%Y-%m-%d %H:%M} UTC auf eine "
|
||||||
|
f"Anbindung: {r.waiting_reason or ''}".strip(),
|
||||||
|
)
|
||||||
|
if rule_is_stale(r, service_started, now):
|
||||||
|
ago = (
|
||||||
|
"noch nie"
|
||||||
|
if r.last_success_at is None
|
||||||
|
else f"seit {r.last_success_at:%Y-%m-%d %H:%M} UTC"
|
||||||
|
)
|
||||||
|
return mk(False, f"Regel „{r.name}“: kein erfolgreicher Lauf {ago}")
|
||||||
|
if r.consecutive_failures >= FAILURES_ALERT:
|
||||||
|
return mk(
|
||||||
|
False, f"Regel „{r.name}“: {r.consecutive_failures} fehlgeschlagene Läufe in Folge"
|
||||||
|
)
|
||||||
|
return mk(True, f"Regel „{r.name}“ in Ordnung")
|
||||||
|
|
||||||
|
|
||||||
def check_health(
|
def check_health(
|
||||||
session_factory: sessionmaker[Session],
|
session_factory: sessionmaker[Session],
|
||||||
*,
|
*,
|
||||||
@@ -52,24 +134,116 @@ def check_health(
|
|||||||
last_heartbeat: datetime | None,
|
last_heartbeat: datetime | None,
|
||||||
service_started: datetime,
|
service_started: datetime,
|
||||||
now: datetime | None = None,
|
now: datetime | None = None,
|
||||||
|
maintenance: bool = False,
|
||||||
) -> Health:
|
) -> Health:
|
||||||
now = now or utcnow()
|
now = now or utcnow()
|
||||||
health = Health()
|
health = Health()
|
||||||
|
health.maintenance = maintenance
|
||||||
if not scheduler_running:
|
if not scheduler_running:
|
||||||
health.fail("Scheduler läuft nicht")
|
health.add(Check("scheduler", "Scheduler", False, "Scheduler läuft nicht"))
|
||||||
elif last_heartbeat is None or now - last_heartbeat > HEARTBEAT_MAX_AGE:
|
elif last_heartbeat is None or now - last_heartbeat > HEARTBEAT_MAX_AGE:
|
||||||
health.fail("Scheduler reagiert nicht (kein Herzschlag)")
|
health.add(
|
||||||
|
Check("scheduler", "Scheduler", False, "Scheduler reagiert nicht (kein Herzschlag)")
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
health.add(Check("scheduler", "Scheduler", True, "Scheduler läuft"))
|
||||||
with session_scope(session_factory) as s:
|
with session_scope(session_factory) as s:
|
||||||
for m in s.scalars(select(Mount).where(Mount.enabled.is_(True))):
|
for m in s.scalars(select(Mount).order_by(Mount.name)):
|
||||||
if m.state is MountState.ERROR:
|
health.add(_mount_check(m))
|
||||||
health.fail(f"Anbindung „{m.name}“ im Fehlerzustand: {m.last_error or ''}".strip())
|
for r in s.scalars(select(Rule).order_by(Rule.name)):
|
||||||
for r in s.scalars(select(Rule).where(Rule.enabled.is_(True))):
|
check = _rule_check(r, service_started, now, maintenance)
|
||||||
if rule_is_stale(r, service_started, now):
|
if r.enabled and not maintenance and rule_is_stale(r, service_started, now):
|
||||||
health.stale_rules.add(r.id)
|
health.stale_rules.add(r.id)
|
||||||
ago = (
|
health.add(check)
|
||||||
"noch nie"
|
|
||||||
if r.last_success_at is None
|
|
||||||
else f"seit {r.last_success_at:%Y-%m-%d %H:%M} UTC"
|
|
||||||
)
|
|
||||||
health.fail(f"Regel „{r.name}“: kein erfolgreicher Lauf {ago}")
|
|
||||||
return health
|
return health
|
||||||
|
|
||||||
|
|
||||||
|
def backup_check(
|
||||||
|
*,
|
||||||
|
enabled: bool,
|
||||||
|
last_success: datetime | None,
|
||||||
|
last_error: str,
|
||||||
|
stale_after: timedelta,
|
||||||
|
service_started: datetime,
|
||||||
|
now: datetime | None = None,
|
||||||
|
) -> Check:
|
||||||
|
"""Datensicherung: überfällig, wenn länger als ``stale_after`` keine gelungen ist
|
||||||
|
(Bezug wie bei Regeln: spätester von letztem Erfolg und Dienststart)."""
|
||||||
|
now = now or utcnow()
|
||||||
|
|
||||||
|
def mk(ok: bool, message: str) -> Check:
|
||||||
|
return Check("backup", "Sicherung", ok, message)
|
||||||
|
|
||||||
|
if not enabled:
|
||||||
|
return mk(True, "Sicherung ausgeschaltet")
|
||||||
|
reference = max(d for d in (last_success, service_started) if d is not None)
|
||||||
|
if now - reference > stale_after:
|
||||||
|
when = "noch nie" if last_success is None else f"seit {last_success:%Y-%m-%d %H:%M} UTC"
|
||||||
|
detail = f": {last_error}" if last_error else ""
|
||||||
|
return mk(False, f"Sicherung überfällig – keine erfolgreiche {when}{detail}")
|
||||||
|
if last_success is None:
|
||||||
|
return mk(True, "Sicherung eingerichtet, noch keine gelaufen")
|
||||||
|
note = f" (letzter Versuch fehlgeschlagen: {last_error})" if last_error else ""
|
||||||
|
return mk(True, f"Letzte Sicherung {last_success:%Y-%m-%d %H:%M} UTC{note}")
|
||||||
|
|
||||||
|
|
||||||
|
METRICS_WINDOW = timedelta(hours=24)
|
||||||
|
COUNTERS = ("files_24h", "failed_24h", "quarantined_24h", "errors_24h", "pending_failures")
|
||||||
|
|
||||||
|
|
||||||
|
def metrics(
|
||||||
|
session_factory: sessionmaker[Session], now: datetime | None = None
|
||||||
|
) -> dict[str, object]:
|
||||||
|
"""Kennzahlen für Uptime Kuma („HTTP(s) – JSON-Abfrage“) und andere Monitore:
|
||||||
|
gesamt und je Regel (Schlüssel = Regel-ID als Text). Probeläufe zählen nicht."""
|
||||||
|
since = (now or utcnow()) - METRICS_WINDOW
|
||||||
|
counts: dict[int, dict[str, int]] = {}
|
||||||
|
last_moved: dict[int, datetime] = {}
|
||||||
|
|
||||||
|
def entry(rule_id: int) -> dict[str, int]:
|
||||||
|
return counts.setdefault(rule_id, dict.fromkeys(COUNTERS, 0))
|
||||||
|
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
for (rid,) in s.execute(select(Rule.id)):
|
||||||
|
entry(rid)
|
||||||
|
real = (Run.dry_run.is_(False), Run.started_at >= since)
|
||||||
|
for rid, moved, failed, errors in s.execute(
|
||||||
|
select(
|
||||||
|
Run.rule_id,
|
||||||
|
func.coalesce(func.sum(Run.moved), 0),
|
||||||
|
func.coalesce(func.sum(Run.failed), 0),
|
||||||
|
func.count().filter(Run.status == RunStatus.ERROR),
|
||||||
|
)
|
||||||
|
.where(*real)
|
||||||
|
.group_by(Run.rule_id)
|
||||||
|
):
|
||||||
|
entry(rid).update(files_24h=moved, failed_24h=failed, errors_24h=errors)
|
||||||
|
for rid, n in s.execute(
|
||||||
|
select(Run.rule_id, func.count())
|
||||||
|
.join(RunItem, RunItem.run_id == Run.id)
|
||||||
|
.where(*real, RunItem.status == ItemStatus.QUARANTINED)
|
||||||
|
.group_by(Run.rule_id)
|
||||||
|
):
|
||||||
|
entry(rid)["quarantined_24h"] = n
|
||||||
|
for rid, n in s.execute(
|
||||||
|
select(FileFailure.rule_id, func.count()).group_by(FileFailure.rule_id)
|
||||||
|
):
|
||||||
|
entry(rid)["pending_failures"] = n
|
||||||
|
for rid, latest in s.execute(
|
||||||
|
select(Run.rule_id, func.max(Run.finished_at))
|
||||||
|
.where(Run.dry_run.is_(False), Run.moved > 0)
|
||||||
|
.group_by(Run.rule_id)
|
||||||
|
):
|
||||||
|
if latest is not None:
|
||||||
|
last_moved[rid] = latest
|
||||||
|
|
||||||
|
def iso(d: datetime | None) -> str | None:
|
||||||
|
return d.isoformat() if d else None
|
||||||
|
|
||||||
|
total: dict[str, object] = {k: sum(c[k] for c in counts.values()) for k in COUNTERS}
|
||||||
|
total["last_moved_at"] = iso(max(last_moved.values(), default=None))
|
||||||
|
total["rules"] = {
|
||||||
|
str(rid): {**c, "last_moved_at": iso(last_moved.get(rid))}
|
||||||
|
for rid, c in sorted(counts.items())
|
||||||
|
}
|
||||||
|
return total
|
||||||
|
|||||||
@@ -0,0 +1,66 @@
|
|||||||
|
"""Uptime Kuma: NetMover meldet sich regelmäßig bei einem „Push“-Monitor.
|
||||||
|
|
||||||
|
Vorteil gegenüber dem Abfragen von ``/healthz``: Es funktioniert auch, wenn Kuma
|
||||||
|
NetMover nicht erreichen kann (z. B. Bindung an 127.0.0.1 hinter einem Proxy). Und
|
||||||
|
steht der Dienst oder die ganze VM, bleibt die Meldung aus – dann schlägt Kuma
|
||||||
|
von selbst Alarm.
|
||||||
|
|
||||||
|
Die Push-URL kopiert man aus Kuma (``https://kuma/api/push/<token>?status=up&msg=OK&ping=``);
|
||||||
|
NetMover ersetzt ``status`` und ``msg`` bei jeder Meldung.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from netmover.health import Health
|
||||||
|
from netmover.logging_setup import event, get_logger
|
||||||
|
|
||||||
|
log = get_logger(__name__)
|
||||||
|
|
||||||
|
MAX_MESSAGE = 250
|
||||||
|
|
||||||
|
Getter = Callable[..., Any]
|
||||||
|
|
||||||
|
|
||||||
|
def ok_message(metrics: dict[str, object] | None) -> str:
|
||||||
|
"""Kurzer Text für die Kuma-Ansicht, z. B. „OK · 24 h: 12 Dateien“."""
|
||||||
|
if not metrics:
|
||||||
|
return "OK"
|
||||||
|
parts = [f"24 h: {metrics.get('files_24h', 0)} Dateien"]
|
||||||
|
failed = metrics.get("failed_24h") or 0
|
||||||
|
if failed:
|
||||||
|
parts.append(f"{failed} Fehler")
|
||||||
|
return "OK · " + ", ".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def push_url(url: str, health: Health, metrics: dict[str, object] | None = None) -> str:
|
||||||
|
"""Push-URL mit aktuellem Zustand (vorhandene Parameter werden ersetzt)."""
|
||||||
|
msg = ok_message(metrics) if health.ok else "; ".join(health.problems)
|
||||||
|
if len(msg) > MAX_MESSAGE:
|
||||||
|
msg = msg[: MAX_MESSAGE - 1] + "…"
|
||||||
|
target = httpx.URL(url)
|
||||||
|
for key, value in (("status", "up" if health.ok else "down"), ("msg", msg), ("ping", "")):
|
||||||
|
target = target.copy_set_param(key, value)
|
||||||
|
return str(target)
|
||||||
|
|
||||||
|
|
||||||
|
def push(
|
||||||
|
url: str,
|
||||||
|
health: Health,
|
||||||
|
getter: Getter | None = None,
|
||||||
|
metrics: dict[str, object] | None = None,
|
||||||
|
) -> bool:
|
||||||
|
"""Meldet den Zustand an Kuma. Fehler werden protokolliert, nie weitergereicht –
|
||||||
|
die URL nicht, weil sie das Token enthält."""
|
||||||
|
try:
|
||||||
|
resp = (getter or httpx.get)(push_url(url, health, metrics), timeout=10)
|
||||||
|
resp.raise_for_status()
|
||||||
|
except Exception as exc:
|
||||||
|
event(log, "kuma.push_failed", level=30, error=type(exc).__name__)
|
||||||
|
return False
|
||||||
|
event(log, "kuma.pushed", level=10, status="up" if health.ok else "down")
|
||||||
|
return True
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
"""E-Mail-Benachrichtigung per SMTP.
|
||||||
|
|
||||||
|
* Verschlüsselung: ``starttls`` (Port 587, Standard), ``ssl`` (Port 465) oder ``none``
|
||||||
|
(nur für interne Relays ohne TLS). Zertifikate werden immer geprüft.
|
||||||
|
* Das Passwort liegt Fernet-verschlüsselt in den Einstellungen.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import smtplib
|
||||||
|
import socket
|
||||||
|
import ssl
|
||||||
|
from datetime import datetime
|
||||||
|
from email.message import EmailMessage
|
||||||
|
from email.utils import formatdate, make_msgid
|
||||||
|
|
||||||
|
from netmover.settings_store import AppSettings
|
||||||
|
|
||||||
|
SECURITY_MODES = ("starttls", "ssl", "none")
|
||||||
|
_ADDRESS = re.compile(r"^[^@\s,;<>]+@[^@\s,;<>]+\.[^@\s,;<>]+$")
|
||||||
|
|
||||||
|
|
||||||
|
class MailConfigError(ValueError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def parse_recipients(raw: str) -> list[str]:
|
||||||
|
recipients = [r.strip() for r in re.split(r"[,;\n]", raw) if r.strip()]
|
||||||
|
bad = [r for r in recipients if not _ADDRESS.fullmatch(r)]
|
||||||
|
if bad:
|
||||||
|
raise MailConfigError(f"Ungültige Adresse: {', '.join(bad)}")
|
||||||
|
return recipients
|
||||||
|
|
||||||
|
|
||||||
|
def valid_address(value: str) -> bool:
|
||||||
|
return bool(_ADDRESS.fullmatch(value.strip()))
|
||||||
|
|
||||||
|
|
||||||
|
def mail_configured(cfg: AppSettings) -> bool:
|
||||||
|
return bool(cfg.email_enabled and cfg.smtp_host and cfg.smtp_from and cfg.smtp_to)
|
||||||
|
|
||||||
|
|
||||||
|
def build_message(cfg: AppSettings, subject: str, body: str) -> EmailMessage:
|
||||||
|
msg = EmailMessage()
|
||||||
|
msg["Subject"] = subject
|
||||||
|
msg["From"] = cfg.smtp_from
|
||||||
|
msg["To"] = ", ".join(parse_recipients(cfg.smtp_to))
|
||||||
|
msg["Date"] = formatdate(localtime=True)
|
||||||
|
msg["Message-ID"] = make_msgid(domain=cfg.smtp_from.rsplit("@", 1)[-1])
|
||||||
|
msg["X-Mailer"] = "NetMover"
|
||||||
|
footer = (
|
||||||
|
f"\n\n-- \nNetMover auf {socket.gethostname()}, "
|
||||||
|
f"{datetime.now().astimezone():%d.%m.%Y %H:%M:%S %Z}"
|
||||||
|
)
|
||||||
|
msg.set_content(body + footer)
|
||||||
|
return msg
|
||||||
|
|
||||||
|
|
||||||
|
def tls_context() -> ssl.SSLContext:
|
||||||
|
"""Zertifikatsprüfung gegen die System-CAs (eigene Funktion, damit Tests ihr
|
||||||
|
Test-Zertifikat einsetzen können, ohne die Prüfung abzuschalten)."""
|
||||||
|
return ssl.create_default_context()
|
||||||
|
|
||||||
|
|
||||||
|
def smtp_send(
|
||||||
|
cfg: AppSettings, password: str | None, subject: str, body: str, timeout: float = 20
|
||||||
|
) -> None:
|
||||||
|
"""Verschickt eine Nachricht; Fehler werden als Ausnahme weitergereicht."""
|
||||||
|
if cfg.smtp_security not in SECURITY_MODES:
|
||||||
|
raise MailConfigError(f"Unbekannte Verschlüsselung: {cfg.smtp_security}")
|
||||||
|
msg = build_message(cfg, subject, body)
|
||||||
|
context = tls_context()
|
||||||
|
smtp: smtplib.SMTP
|
||||||
|
if cfg.smtp_security == "ssl":
|
||||||
|
smtp = smtplib.SMTP_SSL(cfg.smtp_host, cfg.smtp_port, timeout=timeout, context=context)
|
||||||
|
else:
|
||||||
|
smtp = smtplib.SMTP(cfg.smtp_host, cfg.smtp_port, timeout=timeout)
|
||||||
|
try:
|
||||||
|
smtp.ehlo()
|
||||||
|
if cfg.smtp_security == "starttls":
|
||||||
|
smtp.starttls(context=context)
|
||||||
|
smtp.ehlo()
|
||||||
|
if cfg.smtp_user:
|
||||||
|
smtp.login(cfg.smtp_user, password or "")
|
||||||
|
smtp.send_message(msg)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
smtp.quit()
|
||||||
|
except smtplib.SMTPException:
|
||||||
|
smtp.close()
|
||||||
@@ -7,7 +7,7 @@ from logging.config import fileConfig
|
|||||||
from alembic import context
|
from alembic import context
|
||||||
|
|
||||||
from netmover.config import get_settings
|
from netmover.config import get_settings
|
||||||
from netmover.db import make_engine
|
from netmover.db import make_migration_engine
|
||||||
from netmover.models import Base
|
from netmover.models import Base
|
||||||
|
|
||||||
config = context.config
|
config = context.config
|
||||||
@@ -33,16 +33,27 @@ def run_migrations_offline() -> None:
|
|||||||
|
|
||||||
|
|
||||||
def run_migrations_online() -> None:
|
def run_migrations_online() -> None:
|
||||||
engine = make_engine(_url())
|
url = _url()
|
||||||
|
engine = make_migration_engine(url)
|
||||||
with engine.connect() as connection:
|
with engine.connect() as connection:
|
||||||
context.configure(
|
context.configure(
|
||||||
connection=connection,
|
connection=connection,
|
||||||
target_metadata=target_metadata,
|
target_metadata=target_metadata,
|
||||||
render_as_batch=True,
|
render_as_batch=True,
|
||||||
compare_type=True,
|
compare_type=True,
|
||||||
|
# alles oder nichts: auch SQLite kann DDL in einer Transaktion
|
||||||
|
transactional_ddl=True,
|
||||||
)
|
)
|
||||||
with context.begin_transaction():
|
with context.begin_transaction():
|
||||||
context.run_migrations()
|
context.run_migrations()
|
||||||
|
if url.startswith("sqlite"):
|
||||||
|
# Fremdschlüssel waren während der Migration aus – vor dem Commit prüfen.
|
||||||
|
bad = connection.exec_driver_sql("PRAGMA foreign_key_check").fetchall()
|
||||||
|
if bad:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Migration verletzt Fremdschlüssel ({len(bad)} Zeilen), "
|
||||||
|
f"zurückgerollt: {bad[:5]}"
|
||||||
|
)
|
||||||
engine.dispose()
|
engine.dispose()
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,34 @@
|
|||||||
|
"""key management
|
||||||
|
|
||||||
|
Revision ID: 0002
|
||||||
|
Revises: 0001
|
||||||
|
Create Date: 2026-10-10
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision: str = "0002"
|
||||||
|
down_revision: str | None = "0001"
|
||||||
|
branch_labels: str | Sequence[str] | None = None
|
||||||
|
depends_on: str | Sequence[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("credentials", schema=None) as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("managed", sa.Boolean(), server_default=sa.text("0"), nullable=False)
|
||||||
|
)
|
||||||
|
|
||||||
|
with op.batch_alter_table("mounts", schema=None) as batch_op:
|
||||||
|
batch_op.add_column(sa.Column("username", sa.String(length=255), nullable=True))
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("mounts", schema=None) as batch_op:
|
||||||
|
batch_op.drop_column("username")
|
||||||
|
|
||||||
|
with op.batch_alter_table("credentials", schema=None) as batch_op:
|
||||||
|
batch_op.drop_column("managed")
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
"""rule waiting state
|
||||||
|
|
||||||
|
Revision ID: 0003
|
||||||
|
Revises: 0002
|
||||||
|
Create Date: 2026-10-11
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
import netmover.models
|
||||||
|
|
||||||
|
revision: str = "0003"
|
||||||
|
down_revision: str | None = "0002"
|
||||||
|
branch_labels: str | Sequence[str] | None = None
|
||||||
|
depends_on: str | Sequence[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("rules", schema=None) as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("waiting_since", netmover.models.UTCDateTime(), nullable=True)
|
||||||
|
)
|
||||||
|
batch_op.add_column(sa.Column("waiting_reason", sa.Text(), nullable=True))
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("rules", schema=None) as batch_op:
|
||||||
|
batch_op.drop_column("waiting_reason")
|
||||||
|
batch_op.drop_column("waiting_since")
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
"""archive
|
||||||
|
|
||||||
|
Revision ID: 0004
|
||||||
|
Revises: 0003
|
||||||
|
Create Date: 2026-10-12
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
import netmover.models
|
||||||
|
|
||||||
|
revision: str = "0004"
|
||||||
|
down_revision: str | None = "0003"
|
||||||
|
branch_labels: str | Sequence[str] | None = None
|
||||||
|
depends_on: str | Sequence[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table(
|
||||||
|
"archived_files",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("rule_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("src_relpath", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("size", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("mtime_ns", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("archive_path", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("archived_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["rule_id"],
|
||||||
|
["rules.id"],
|
||||||
|
name=op.f("fk_archived_files_rule_id_rules"),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_archived_files")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"rule_id",
|
||||||
|
"src_relpath",
|
||||||
|
"size",
|
||||||
|
"mtime_ns",
|
||||||
|
name=op.f("uq_archived_files_rule_id_src_relpath_size_mtime_ns"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
with op.batch_alter_table("rules", schema=None) as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"archive_kind",
|
||||||
|
sa.Enum(
|
||||||
|
"none",
|
||||||
|
"mount",
|
||||||
|
"local",
|
||||||
|
name="archive_kind",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
server_default="none",
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(sa.Column("archive_mount_id", sa.Integer(), nullable=True))
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("archive_path", sa.String(length=1024), server_default="", nullable=False)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"archive_layout",
|
||||||
|
sa.Enum(
|
||||||
|
"flat",
|
||||||
|
"date",
|
||||||
|
name="archive_layout",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
server_default="flat",
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.create_foreign_key(
|
||||||
|
batch_op.f("fk_rules_archive_mount_id_mounts"),
|
||||||
|
"mounts",
|
||||||
|
["archive_mount_id"],
|
||||||
|
["id"],
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("rules", schema=None) as batch_op:
|
||||||
|
batch_op.drop_constraint(batch_op.f("fk_rules_archive_mount_id_mounts"), type_="foreignkey")
|
||||||
|
# CHECK-Regeln der Auswahlfelder zuerst entfernen – sonst verweisen sie beim
|
||||||
|
# Neuaufbau der Tabelle auf Spalten, die es nicht mehr gibt.
|
||||||
|
batch_op.drop_constraint(batch_op.f("ck_rules_archive_kind"), type_="check")
|
||||||
|
batch_op.drop_constraint(batch_op.f("ck_rules_archive_layout"), type_="check")
|
||||||
|
batch_op.drop_column("archive_layout")
|
||||||
|
batch_op.drop_column("archive_path")
|
||||||
|
batch_op.drop_column("archive_mount_id")
|
||||||
|
batch_op.drop_column("archive_kind")
|
||||||
|
|
||||||
|
op.drop_table("archived_files")
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
"""rule targets
|
||||||
|
|
||||||
|
Weitere Ziele je Regel (bis zu 9 zusätzlich zum Hauptziel). Ein vorhandenes Archiv
|
||||||
|
(Migration 0004) wird zum ersten weiteren Ziel „Archiv“, seine Buchführung
|
||||||
|
(``archived_files``) wandert nach ``target_files``. Downgrade: das erste weitere
|
||||||
|
Ziel wird wieder zum Archiv, alle weiteren gehen verloren.
|
||||||
|
|
||||||
|
Revision ID: 0005
|
||||||
|
Revises: 0004
|
||||||
|
Create Date: 2026-10-24
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
import netmover.models
|
||||||
|
|
||||||
|
revision: str = "0005"
|
||||||
|
down_revision: str | None = "0004"
|
||||||
|
branch_labels: str | Sequence[str] | None = None
|
||||||
|
depends_on: str | Sequence[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def _enum(name: str, *values: str) -> sa.Enum:
|
||||||
|
return sa.Enum(*values, name=name, native_enum=False, create_constraint=True, length=24)
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table(
|
||||||
|
"rule_targets",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("rule_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("position", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("name", sa.String(length=100), nullable=False),
|
||||||
|
sa.Column("kind", _enum("target_kind", "mount", "local"), nullable=False),
|
||||||
|
sa.Column("mount_id", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("path", sa.String(length=1024), nullable=False),
|
||||||
|
sa.Column("layout", _enum("target_layout", "flat", "date"), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["mount_id"],
|
||||||
|
["mounts.id"],
|
||||||
|
name=op.f("fk_rule_targets_mount_id_mounts"),
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["rule_id"],
|
||||||
|
["rules.id"],
|
||||||
|
name=op.f("fk_rule_targets_rule_id_rules"),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_rule_targets")),
|
||||||
|
)
|
||||||
|
op.create_index(op.f("ix_rule_targets_rule_id"), "rule_targets", ["rule_id"])
|
||||||
|
op.create_table(
|
||||||
|
"target_files",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("target_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("src_relpath", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("size", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("mtime_ns", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("dst_path", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("delivered_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["target_id"],
|
||||||
|
["rule_targets.id"],
|
||||||
|
name=op.f("fk_target_files_target_id_rule_targets"),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_target_files")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"target_id",
|
||||||
|
"src_relpath",
|
||||||
|
"size",
|
||||||
|
"mtime_ns",
|
||||||
|
name=op.f("uq_target_files_target_id_src_relpath_size_mtime_ns"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.create_index(op.f("ix_target_files_target_id"), "target_files", ["target_id"])
|
||||||
|
|
||||||
|
# Archiv → erstes weiteres Ziel, Buchführung mitnehmen
|
||||||
|
op.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO rule_targets (rule_id, position, name, kind, mount_id, path, layout)
|
||||||
|
SELECT id, 0, 'Archiv', archive_kind,
|
||||||
|
CASE WHEN archive_kind = 'mount' THEN archive_mount_id END,
|
||||||
|
archive_path, archive_layout
|
||||||
|
FROM rules WHERE archive_kind != 'none'
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
op.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO target_files (target_id, src_relpath, size, mtime_ns, dst_path, delivered_at)
|
||||||
|
SELECT t.id, a.src_relpath, a.size, a.mtime_ns, a.archive_path, a.archived_at
|
||||||
|
FROM archived_files a JOIN rule_targets t ON t.rule_id = a.rule_id
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
op.drop_table("archived_files")
|
||||||
|
with op.batch_alter_table("rules", schema=None) as batch_op:
|
||||||
|
batch_op.drop_constraint(batch_op.f("fk_rules_archive_mount_id_mounts"), type_="foreignkey")
|
||||||
|
batch_op.drop_constraint(batch_op.f("ck_rules_archive_kind"), type_="check")
|
||||||
|
batch_op.drop_constraint(batch_op.f("ck_rules_archive_layout"), type_="check")
|
||||||
|
batch_op.drop_column("archive_layout")
|
||||||
|
batch_op.drop_column("archive_path")
|
||||||
|
batch_op.drop_column("archive_mount_id")
|
||||||
|
batch_op.drop_column("archive_kind")
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("rules", schema=None) as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"archive_kind",
|
||||||
|
_enum("archive_kind", "none", "mount", "local"),
|
||||||
|
server_default="none",
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.add_column(sa.Column("archive_mount_id", sa.Integer(), nullable=True))
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("archive_path", sa.String(length=1024), server_default="", nullable=False)
|
||||||
|
)
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column(
|
||||||
|
"archive_layout",
|
||||||
|
_enum("archive_layout", "flat", "date"),
|
||||||
|
server_default="flat",
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
batch_op.create_foreign_key(
|
||||||
|
batch_op.f("fk_rules_archive_mount_id_mounts"),
|
||||||
|
"mounts",
|
||||||
|
["archive_mount_id"],
|
||||||
|
["id"],
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
)
|
||||||
|
for col, src in (
|
||||||
|
("archive_kind", "kind"),
|
||||||
|
("archive_mount_id", "mount_id"),
|
||||||
|
("archive_path", "path"),
|
||||||
|
("archive_layout", "layout"),
|
||||||
|
):
|
||||||
|
# Spaltennamen sind feste Literale aus dieser Liste
|
||||||
|
op.execute(
|
||||||
|
f"UPDATE rules SET {col} = (SELECT t.{src} FROM rule_targets t " # noqa: S608
|
||||||
|
"WHERE t.rule_id = rules.id ORDER BY t.position, t.id LIMIT 1) "
|
||||||
|
"WHERE EXISTS (SELECT 1 FROM rule_targets t WHERE t.rule_id = rules.id)"
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"archived_files",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("rule_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("src_relpath", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("size", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("mtime_ns", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("archive_path", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("archived_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["rule_id"],
|
||||||
|
["rules.id"],
|
||||||
|
name=op.f("fk_archived_files_rule_id_rules"),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_archived_files")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"rule_id",
|
||||||
|
"src_relpath",
|
||||||
|
"size",
|
||||||
|
"mtime_ns",
|
||||||
|
name=op.f("uq_archived_files_rule_id_src_relpath_size_mtime_ns"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO archived_files (rule_id, src_relpath, size, mtime_ns, archive_path,
|
||||||
|
archived_at)
|
||||||
|
SELECT t.rule_id, f.src_relpath, f.size, f.mtime_ns, f.dst_path, f.delivered_at
|
||||||
|
FROM target_files f JOIN rule_targets t ON t.id = f.target_id
|
||||||
|
WHERE t.id = (SELECT t2.id FROM rule_targets t2 WHERE t2.rule_id = t.rule_id
|
||||||
|
ORDER BY t2.position, t2.id LIMIT 1)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
op.drop_index(op.f("ix_target_files_target_id"), table_name="target_files")
|
||||||
|
op.drop_table("target_files")
|
||||||
|
op.drop_index(op.f("ix_rule_targets_rule_id"), table_name="rule_targets")
|
||||||
|
op.drop_table("rule_targets")
|
||||||
@@ -7,6 +7,14 @@ Abweichungen bzw. Ergänzungen gegenüber dem Mindestumfang aus der Spezifikatio
|
|||||||
(beim ersten Test festgehaltener Hostschlüssel, danach verbindlich).
|
(beim ersten Test festgehaltener Hostschlüssel, danach verbindlich).
|
||||||
* ``credentials.kind`` (``password`` | ``ssh_key``) und ``credentials.public_key``.
|
* ``credentials.kind`` (``password`` | ``ssh_key``) und ``credentials.public_key``.
|
||||||
* ``credentials.domain`` – Windows-/AD-Freigaben brauchen oft eine Domäne.
|
* ``credentials.domain`` – Windows-/AD-Freigaben brauchen oft eine Domäne.
|
||||||
|
* ``credentials.managed`` + ``mounts.username`` (Migration 0002) –
|
||||||
|
Schlüsselverwaltung: gespeicherte SSH-Schlüssel, die mehrere Anbindungen mit
|
||||||
|
jeweils eigenem Benutzernamen nutzen.
|
||||||
|
* Tabellen ``rule_targets`` + ``target_files`` (Migration 0005) – weitere Ziele je
|
||||||
|
Regel (z. B. Archiv, Buchhaltung) mit eigener Buchführung; lösen das Archiv aus
|
||||||
|
Migration 0004 (``rules.archive_*``, ``archived_files``) ab.
|
||||||
|
* ``rules.waiting_since`` / ``rules.waiting_reason`` (Migration 0003) – Regel wartet
|
||||||
|
auf eine nicht erreichbare Anbindung (automatisch pausiert).
|
||||||
* ``mounts.state_since`` – für „hängt seit …“ / „gestört seit …“ auf der Statusseite.
|
* ``mounts.state_since`` – für „hängt seit …“ / „gestört seit …“ auf der Statusseite.
|
||||||
* ``mounts.write_ok`` / ``mounts.root_squash`` / ``mounts.last_test_*`` – Ergebnis
|
* ``mounts.write_ok`` / ``mounts.root_squash`` / ``mounts.last_test_*`` – Ergebnis
|
||||||
des letzten Mount-Tests; ``root_squash`` entscheidet, ob ``chown`` auf NFS-Zielen
|
des letzten Mount-Tests; ``root_squash`` entscheidet, ob ``chown`` auf NFS-Zielen
|
||||||
@@ -38,6 +46,7 @@ from sqlalchemy import (
|
|||||||
TypeDecorator,
|
TypeDecorator,
|
||||||
UniqueConstraint,
|
UniqueConstraint,
|
||||||
)
|
)
|
||||||
|
from sqlalchemy import false as sa_false
|
||||||
from sqlalchemy.engine import Dialect
|
from sqlalchemy.engine import Dialect
|
||||||
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
|
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
|
||||||
|
|
||||||
@@ -132,6 +141,16 @@ class CollisionPolicy(enum.StrEnum):
|
|||||||
OVERWRITE = "overwrite"
|
OVERWRITE = "overwrite"
|
||||||
|
|
||||||
|
|
||||||
|
class TargetKind(enum.StrEnum):
|
||||||
|
MOUNT = "mount"
|
||||||
|
LOCAL = "local"
|
||||||
|
|
||||||
|
|
||||||
|
class TargetLayout(enum.StrEnum):
|
||||||
|
FLAT = "flat" # wie in der Quelle
|
||||||
|
DATE = "date" # Jahr/Monat/…, nach Tag der Verarbeitung
|
||||||
|
|
||||||
|
|
||||||
class RunStatus(enum.StrEnum):
|
class RunStatus(enum.StrEnum):
|
||||||
RUNNING = "running"
|
RUNNING = "running"
|
||||||
OK = "ok" # keine Dateifehler
|
OK = "ok" # keine Dateifehler
|
||||||
@@ -166,6 +185,9 @@ class Credential(Base):
|
|||||||
domain: Mapped[str | None] = mapped_column(String(255))
|
domain: Mapped[str | None] = mapped_column(String(255))
|
||||||
secret_enc: Mapped[str] = mapped_column(Text) # Passwort bzw. privater Schlüssel (Fernet)
|
secret_enc: Mapped[str] = mapped_column(Text) # Passwort bzw. privater Schlüssel (Fernet)
|
||||||
public_key: Mapped[str | None] = mapped_column(Text) # nur ssh_key, OpenSSH-Format
|
public_key: Mapped[str | None] = mapped_column(Text) # nur ssh_key, OpenSSH-Format
|
||||||
|
# True: in der Schlüsselverwaltung gepflegt, von mehreren Anbindungen nutzbar.
|
||||||
|
# False: gehört genau einer Anbindung und wird mit ihr gelöscht.
|
||||||
|
managed: Mapped[bool] = mapped_column(Boolean, default=False, server_default=sa_false())
|
||||||
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
# passive_deletes="all": das ORM soll cred_id nicht still auf NULL setzen,
|
# passive_deletes="all": das ORM soll cred_id nicht still auf NULL setzen,
|
||||||
@@ -183,6 +205,9 @@ class Mount(Base):
|
|||||||
server: Mapped[str] = mapped_column(String(255))
|
server: Mapped[str] = mapped_column(String(255))
|
||||||
share: Mapped[str] = mapped_column(String(1024))
|
share: Mapped[str] = mapped_column(String(1024))
|
||||||
port: Mapped[int | None] = mapped_column(Integer) # nur sftp
|
port: Mapped[int | None] = mapped_column(Integer) # nur sftp
|
||||||
|
# Benutzer für die Anmeldung; überschreibt credentials.username. Nötig, weil ein
|
||||||
|
# gespeicherter SSH-Schlüssel für verschiedene Server/Benutzer gelten kann.
|
||||||
|
username: Mapped[str | None] = mapped_column(String(255))
|
||||||
host_key: Mapped[str | None] = mapped_column(Text) # nur sftp, "<typ> <base64>"
|
host_key: Mapped[str | None] = mapped_column(Text) # nur sftp, "<typ> <base64>"
|
||||||
mountpoint: Mapped[str | None] = mapped_column(String(1024), unique=True) # NULL bei sftp
|
mountpoint: Mapped[str | None] = mapped_column(String(1024), unique=True) # NULL bei sftp
|
||||||
cred_id: Mapped[int | None] = mapped_column(ForeignKey("credentials.id", ondelete="RESTRICT"))
|
cred_id: Mapped[int | None] = mapped_column(ForeignKey("credentials.id", ondelete="RESTRICT"))
|
||||||
@@ -250,10 +275,19 @@ class Rule(Base):
|
|||||||
last_run_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
last_run_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
last_success_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
last_success_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
consecutive_failures: Mapped[int] = mapped_column(Integer, default=0)
|
consecutive_failures: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
# Automatisch pausiert, solange Quelle oder Ziel nicht erreichbar ist (Migration 0003)
|
||||||
|
waiting_since: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
|
waiting_reason: Mapped[str | None] = mapped_column(Text)
|
||||||
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
src_mount: Mapped[Mount] = relationship(foreign_keys=[src_mount_id])
|
src_mount: Mapped[Mount] = relationship(foreign_keys=[src_mount_id])
|
||||||
dst_mount: Mapped[Mount | None] = relationship(foreign_keys=[dst_mount_id])
|
dst_mount: Mapped[Mount | None] = relationship(foreign_keys=[dst_mount_id])
|
||||||
|
targets: Mapped[list[RuleTarget]] = relationship(
|
||||||
|
back_populates="rule",
|
||||||
|
cascade="all, delete-orphan",
|
||||||
|
passive_deletes=True,
|
||||||
|
order_by="RuleTarget.position",
|
||||||
|
)
|
||||||
runs: Mapped[list[Run]] = relationship(
|
runs: Mapped[list[Run]] = relationship(
|
||||||
back_populates="rule", cascade="all, delete-orphan", passive_deletes=True
|
back_populates="rule", cascade="all, delete-orphan", passive_deletes=True
|
||||||
)
|
)
|
||||||
@@ -315,6 +349,46 @@ class CopiedFile(Base):
|
|||||||
copied_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
copied_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
|
|
||||||
|
class RuleTarget(Base):
|
||||||
|
"""Weiteres Ziel einer Regel (Archiv, Buchhaltung, …). Wird vor dem Hauptziel
|
||||||
|
beliefert; die Quelle wird erst gelöscht, wenn alle Ziele die Datei haben."""
|
||||||
|
|
||||||
|
__tablename__ = "rule_targets"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
rule_id: Mapped[int] = mapped_column(ForeignKey("rules.id", ondelete="CASCADE"), index=True)
|
||||||
|
position: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
name: Mapped[str] = mapped_column(String(100), default="")
|
||||||
|
kind: Mapped[TargetKind] = mapped_column(_enum(TargetKind, "target_kind"))
|
||||||
|
# kind=mount: Unterordner relativ zur Anbindung; kind=local: absoluter Pfad
|
||||||
|
mount_id: Mapped[int | None] = mapped_column(ForeignKey("mounts.id", ondelete="RESTRICT"))
|
||||||
|
path: Mapped[str] = mapped_column(String(1024), default="")
|
||||||
|
layout: Mapped[TargetLayout] = mapped_column(
|
||||||
|
_enum(TargetLayout, "target_layout"), default=TargetLayout.FLAT
|
||||||
|
)
|
||||||
|
|
||||||
|
rule: Mapped[Rule] = relationship(back_populates="targets")
|
||||||
|
mount: Mapped[Mount | None] = relationship()
|
||||||
|
|
||||||
|
|
||||||
|
class TargetFile(Base):
|
||||||
|
"""Welche Dateiversion ein weiteres Ziel schon hat – ein Neuversuch (anderes Ziel
|
||||||
|
gescheitert oder nicht erreichbar) legt so keine zweite Kopie an."""
|
||||||
|
|
||||||
|
__tablename__ = "target_files"
|
||||||
|
__table_args__ = (UniqueConstraint("target_id", "src_relpath", "size", "mtime_ns"),)
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
target_id: Mapped[int] = mapped_column(
|
||||||
|
ForeignKey("rule_targets.id", ondelete="CASCADE"), index=True
|
||||||
|
)
|
||||||
|
src_relpath: Mapped[str] = mapped_column(String(2048))
|
||||||
|
size: Mapped[int] = mapped_column(BigInteger)
|
||||||
|
mtime_ns: Mapped[int] = mapped_column(BigInteger)
|
||||||
|
dst_path: Mapped[str] = mapped_column(String(2048))
|
||||||
|
delivered_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
|
|
||||||
class FileFailure(Base):
|
class FileFailure(Base):
|
||||||
"""Fehlversuche pro Datei, Grundlage für die Quarantäne."""
|
"""Fehlversuche pro Datei, Grundlage für die Quarantäne."""
|
||||||
|
|
||||||
|
|||||||
@@ -58,6 +58,13 @@ def explain_mount_error(fstype: str, message: str) -> str:
|
|||||||
if "helper program" in low or "unknown filesystem type" in low:
|
if "helper program" in low or "unknown filesystem type" in low:
|
||||||
pkg = "cifs-utils" if fstype == "cifs" else "nfs-common"
|
pkg = "cifs-utils" if fstype == "cifs" else "nfs-common"
|
||||||
hint = f"Das Paket „{pkg}“ fehlt: apt install {pkg}"
|
hint = f"Das Paket „{pkg}“ fehlt: apt install {pkg}"
|
||||||
|
elif fstype == "cifs" and ("error(95)" in low or "operation not supported" in low):
|
||||||
|
hint = (
|
||||||
|
"Server und Client finden keine gemeinsame SMB-Version. Am besten am NAS "
|
||||||
|
"SMB2/SMB3 erlauben (Synology: Systemsteuerung → Dateidienste → SMB → "
|
||||||
|
"Erweiterte Einstellungen → Maximales SMB-Protokoll: SMB3). Alternativ "
|
||||||
|
"Zusatzoption vers=2.0 – vers=1.0 nur im Notfall, SMB1 ist unsicher."
|
||||||
|
)
|
||||||
elif "permission denied" in low and fstype == "cifs":
|
elif "permission denied" in low and fstype == "cifs":
|
||||||
hint = "Benutzername/Passwort falsch oder der Benutzer hat keine Rechte auf die Freigabe."
|
hint = "Benutzername/Passwort falsch oder der Benutzer hat keine Rechte auf die Freigabe."
|
||||||
elif "permission denied" in low or "access denied by server" in low:
|
elif "permission denied" in low or "access denied by server" in low:
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ from sqlalchemy.orm import Session, sessionmaker
|
|||||||
from netmover.crypto import SecretBox, redact
|
from netmover.crypto import SecretBox, redact
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
from netmover.logging_setup import event, get_logger
|
from netmover.logging_setup import event, get_logger
|
||||||
from netmover.models import CredentialKind, Mount, MountState, MountType, utcnow
|
from netmover.models import CredentialKind, Mount, MountState, MountType, Rule, utcnow
|
||||||
from netmover.mounting.backend import CifsCredentials, MountBackend, MountError, MountSpec
|
from netmover.mounting.backend import CifsCredentials, MountBackend, MountError, MountSpec
|
||||||
from netmover.mounting.report import CheckReport, assess_write
|
from netmover.mounting.report import CheckReport, assess_write
|
||||||
from netmover.mounting.sftp import SftpConfig, SftpError, check_connection, connect
|
from netmover.mounting.sftp import SftpConfig, SftpError, check_connection, connect
|
||||||
@@ -203,12 +203,31 @@ class MountManager:
|
|||||||
with self._locks_guard:
|
with self._locks_guard:
|
||||||
return self._locks.setdefault(mount_id, threading.RLock())
|
return self._locks.setdefault(mount_id, threading.RLock())
|
||||||
|
|
||||||
def _notify(self, kind: str, name: str, message: str) -> None:
|
def _notify(self, kind: str, name: str, message: str, mount_id: int | None = None) -> None:
|
||||||
if self._notifier is not None:
|
if self._notifier is None:
|
||||||
try:
|
return
|
||||||
self._notifier(kind, name, message)
|
if mount_id is not None:
|
||||||
except Exception:
|
rules = self._affected_rules(mount_id)
|
||||||
log.exception("notifier.failed")
|
if rules and kind != "mount.recovered":
|
||||||
|
waiting = "\nBetroffene Regeln (warten, bis die Anbindung zurück ist): "
|
||||||
|
message += waiting + ", ".join(rules)
|
||||||
|
elif rules:
|
||||||
|
message += f"\nRegeln laufen wieder: {', '.join(rules)}"
|
||||||
|
try:
|
||||||
|
self._notifier(kind, name, message)
|
||||||
|
except Exception:
|
||||||
|
log.exception("notifier.failed")
|
||||||
|
|
||||||
|
def _affected_rules(self, mount_id: int) -> list[str]:
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
return list(
|
||||||
|
s.scalars(
|
||||||
|
select(Rule.name)
|
||||||
|
.where(Rule.enabled.is_(True))
|
||||||
|
.where((Rule.src_mount_id == mount_id) | (Rule.dst_mount_id == mount_id))
|
||||||
|
.order_by(Rule.name)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
def mountpoint_for(self, slug: str) -> Path:
|
def mountpoint_for(self, slug: str) -> Path:
|
||||||
if slugify(slug) != slug:
|
if slugify(slug) != slug:
|
||||||
@@ -228,7 +247,7 @@ class MountManager:
|
|||||||
if m.credential is not None:
|
if m.credential is not None:
|
||||||
c = m.credential
|
c = m.credential
|
||||||
cred = CredentialData(
|
cred = CredentialData(
|
||||||
c.kind, c.username, self.secrets.decrypt(c.secret_enc), c.domain
|
c.kind, m.username or c.username, self.secrets.decrypt(c.secret_enc), c.domain
|
||||||
)
|
)
|
||||||
s.expunge(m)
|
s.expunge(m)
|
||||||
return m, cred
|
return m, cred
|
||||||
@@ -347,7 +366,7 @@ class MountManager:
|
|||||||
# z. B. Stromausfall: die VM bootet schneller als das NAS
|
# z. B. Stromausfall: die VM bootet schneller als das NAS
|
||||||
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
||||||
results.append(RestoreResult(name, False, str(exc)))
|
results.append(RestoreResult(name, False, str(exc)))
|
||||||
self._notify("mount.restore_failed", name, str(exc))
|
self._notify("mount.restore_failed", name, str(exc), mount_id)
|
||||||
for r in results:
|
for r in results:
|
||||||
event(
|
event(
|
||||||
log,
|
log,
|
||||||
@@ -507,6 +526,11 @@ class MountManager:
|
|||||||
if cfg.type is MountType.SFTP and m.host_key is None and report.can_read:
|
if cfg.type is MountType.SFTP and m.host_key is None and report.can_read:
|
||||||
m.host_key = report.host_key
|
m.host_key = report.host_key
|
||||||
event(log, "sftp.host_key_pinned", mount=m.name, fp=report.host_key_fingerprint)
|
event(log, "sftp.host_key_pinned", mount=m.name, fp=report.host_key_fingerprint)
|
||||||
|
# SFTP hat keinen Mount-Zustand – ein erfolgreicher Test heißt „erreichbar“.
|
||||||
|
if cfg.type is MountType.SFTP and report.ok and m.state is not MountState.MOUNTED:
|
||||||
|
m.state = MountState.MOUNTED
|
||||||
|
m.state_since = utcnow()
|
||||||
|
m.last_error = None
|
||||||
|
|
||||||
# -- Lebenszeichen und Wiederherstellung --------------------------------------
|
# -- Lebenszeichen und Wiederherstellung --------------------------------------
|
||||||
|
|
||||||
@@ -526,6 +550,8 @@ class MountManager:
|
|||||||
self._retry_mount(mount_id, m, cred, mp)
|
self._retry_mount(mount_id, m, cred, mp)
|
||||||
return mp
|
return mp
|
||||||
msg = f"„{m.name}“ ist nicht eingehängt – Lauf abgebrochen, nichts angefasst."
|
msg = f"„{m.name}“ ist nicht eingehängt – Lauf abgebrochen, nichts angefasst."
|
||||||
|
if m.state is not MountState.ERROR:
|
||||||
|
self._notify("mount.error", m.name, msg, mount_id)
|
||||||
self._set_state(mount_id, MountState.ERROR, msg)
|
self._set_state(mount_id, MountState.ERROR, msg)
|
||||||
raise MountUnavailable(msg)
|
raise MountUnavailable(msg)
|
||||||
try:
|
try:
|
||||||
@@ -558,8 +584,10 @@ class MountManager:
|
|||||||
msg = f"„{m.name}“ nicht erreichbar: {exc}"
|
msg = f"„{m.name}“ nicht erreichbar: {exc}"
|
||||||
self._set_state(mount_id, MountState.ERROR, msg)
|
self._set_state(mount_id, MountState.ERROR, msg)
|
||||||
if m.state is not MountState.ERROR:
|
if m.state is not MountState.ERROR:
|
||||||
self._notify("mount.error", m.name, msg)
|
self._notify("mount.error", m.name, msg, mount_id)
|
||||||
raise MountUnavailable(msg) from None
|
raise MountUnavailable(msg) from None
|
||||||
|
if m.state is MountState.ERROR:
|
||||||
|
self._notify("mount.recovered", m.name, f"„{m.name}“ ist wieder erreichbar.", mount_id)
|
||||||
if m.state is not MountState.MOUNTED:
|
if m.state is not MountState.MOUNTED:
|
||||||
self._set_state(mount_id, MountState.MOUNTED)
|
self._set_state(mount_id, MountState.MOUNTED)
|
||||||
|
|
||||||
@@ -596,7 +624,7 @@ class MountManager:
|
|||||||
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
||||||
msg += f"\n→ Neuer Versuch automatisch alle {self.retry_interval:g} s."
|
msg += f"\n→ Neuer Versuch automatisch alle {self.retry_interval:g} s."
|
||||||
self._set_state(mount_id, MountState.ERROR, msg)
|
self._set_state(mount_id, MountState.ERROR, msg)
|
||||||
self._notify("mount.error", m.name, msg)
|
self._notify("mount.error", m.name, msg, mount_id)
|
||||||
raise MountUnavailable(msg)
|
raise MountUnavailable(msg)
|
||||||
|
|
||||||
def retry_pending(self) -> list[str]:
|
def retry_pending(self) -> list[str]:
|
||||||
@@ -658,7 +686,7 @@ class MountManager:
|
|||||||
del self._retry_at[mount_id]
|
del self._retry_at[mount_id]
|
||||||
self._set_state(mount_id, MountState.MOUNTED, mounted=True)
|
self._set_state(mount_id, MountState.MOUNTED, mounted=True)
|
||||||
event(log, "mount.recovered", mount=m.name, attempt="retry")
|
event(log, "mount.recovered", mount=m.name, attempt="retry")
|
||||||
self._notify("mount.recovered", m.name, f"„{m.name}“ ist wieder eingehängt.")
|
self._notify("mount.recovered", m.name, f"„{m.name}“ ist wieder eingehängt.", mount_id)
|
||||||
|
|
||||||
def usage(self, mount_id: int, timeout: float | None = None) -> tuple[int, int] | None:
|
def usage(self, mount_id: int, timeout: float | None = None) -> tuple[int, int] | None:
|
||||||
"""(gesamt, frei) in Bytes – nur bei eingehängten Kernel-Mounts."""
|
"""(gesamt, frei) in Bytes – nur bei eingehängten Kernel-Mounts."""
|
||||||
|
|||||||
@@ -30,12 +30,13 @@ from abc import ABC, abstractmethod
|
|||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Literal
|
from typing import Any, Literal, Protocol
|
||||||
|
|
||||||
from netmover.models import CollisionPolicy
|
from netmover.models import CollisionPolicy
|
||||||
from netmover.mounting.sftp import SftpSession
|
from netmover.mounting.sftp import SftpSession
|
||||||
from netmover.mover.fsops import rename_noreplace
|
from netmover.mover.fsops import rename_noreplace
|
||||||
from netmover.mover.naming import collision_candidates
|
from netmover.mover.naming import collision_candidates
|
||||||
|
from netmover.mover.sources import network_errors
|
||||||
from netmover.timeouts import CancelToken
|
from netmover.timeouts import CancelToken
|
||||||
|
|
||||||
CHUNK = 1024 * 1024
|
CHUNK = 1024 * 1024
|
||||||
@@ -81,6 +82,13 @@ class CommitResult:
|
|||||||
Verify = Callable[[], None]
|
Verify = Callable[[], None]
|
||||||
|
|
||||||
|
|
||||||
|
class Readable(Protocol):
|
||||||
|
"""Quelle einer Übertragung: ein lokaler ``Path`` oder eine Datei einer
|
||||||
|
:class:`~netmover.mover.sources.Source` (über einen kleinen Adapter)."""
|
||||||
|
|
||||||
|
def open(self, mode: str = ...) -> Any: ...
|
||||||
|
|
||||||
|
|
||||||
class Destination(ABC):
|
class Destination(ABC):
|
||||||
kind: FsKind
|
kind: FsKind
|
||||||
|
|
||||||
@@ -101,13 +109,20 @@ class Destination(ABC):
|
|||||||
def close(self) -> None: # noqa: B027 – absichtlich optional
|
def close(self) -> None: # noqa: B027 – absichtlich optional
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def list_names(self) -> list[str]:
|
||||||
|
"""Dateinamen direkt im Zielordner (für das Ausdünnen von Sicherungen)."""
|
||||||
|
|
||||||
|
def remove(self, rel: str) -> None:
|
||||||
|
self._remove(rel)
|
||||||
|
|
||||||
# -- Primitive der Unterklassen -------------------------------------------------
|
# -- Primitive der Unterklassen -------------------------------------------------
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def _ensure_parent(self, rel: str) -> None: ...
|
def _ensure_parent(self, rel: str) -> None: ...
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def _upload(self, src: Path, tmp: str, size: int, token: CancelToken) -> list[str]: ...
|
def _upload(self, src: Readable, tmp: str, size: int, token: CancelToken) -> list[str]: ...
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def _apply_meta(self, rel: str, meta: MetaOptions) -> list[str]: ...
|
def _apply_meta(self, rel: str, meta: MetaOptions) -> list[str]: ...
|
||||||
@@ -140,7 +155,7 @@ class Destination(ABC):
|
|||||||
|
|
||||||
def commit(
|
def commit(
|
||||||
self,
|
self,
|
||||||
src: Path,
|
src: Readable,
|
||||||
size: int,
|
size: int,
|
||||||
rel: str,
|
rel: str,
|
||||||
policy: CollisionPolicy,
|
policy: CollisionPolicy,
|
||||||
@@ -227,6 +242,9 @@ class LocalDestination(Destination):
|
|||||||
def exists(self, rel: str) -> bool:
|
def exists(self, rel: str) -> bool:
|
||||||
return os.path.lexists(self._p(rel))
|
return os.path.lexists(self._p(rel))
|
||||||
|
|
||||||
|
def list_names(self) -> list[str]:
|
||||||
|
return [e.name for e in os.scandir(self.root) if e.is_file(follow_symlinks=False)]
|
||||||
|
|
||||||
def free_bytes(self) -> int | None:
|
def free_bytes(self) -> int | None:
|
||||||
st = os.statvfs(self.root)
|
st = os.statvfs(self.root)
|
||||||
return st.f_bavail * st.f_frsize
|
return st.f_bavail * st.f_frsize
|
||||||
@@ -234,7 +252,7 @@ class LocalDestination(Destination):
|
|||||||
def _ensure_parent(self, rel: str) -> None:
|
def _ensure_parent(self, rel: str) -> None:
|
||||||
self._p(rel).parent.mkdir(mode=0o755, parents=True, exist_ok=True)
|
self._p(rel).parent.mkdir(mode=0o755, parents=True, exist_ok=True)
|
||||||
|
|
||||||
def _upload(self, src: Path, tmp: str, size: int, token: CancelToken) -> list[str]:
|
def _upload(self, src: Readable, tmp: str, size: int, token: CancelToken) -> list[str]:
|
||||||
written = 0
|
written = 0
|
||||||
fd = os.open(self._p(tmp), os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
fd = os.open(self._p(tmp), os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||||
try:
|
try:
|
||||||
@@ -413,6 +431,13 @@ class SftpDestination(Destination):
|
|||||||
return False
|
return False
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
def list_names(self) -> list[str]:
|
||||||
|
return [
|
||||||
|
a.filename
|
||||||
|
for a in self.client.listdir_attr(self.base)
|
||||||
|
if a.st_mode is not None and stat.S_ISREG(a.st_mode)
|
||||||
|
]
|
||||||
|
|
||||||
def free_bytes(self) -> int | None:
|
def free_bytes(self) -> int | None:
|
||||||
usage = self.session.statvfs(self.base)
|
usage = self.session.statvfs(self.base)
|
||||||
return usage[1] if usage else None
|
return usage[1] if usage else None
|
||||||
@@ -429,10 +454,10 @@ class SftpDestination(Destination):
|
|||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
self.client.mkdir(current, mode=0o755)
|
self.client.mkdir(current, mode=0o755)
|
||||||
|
|
||||||
def _upload(self, src: Path, tmp: str, size: int, token: CancelToken) -> list[str]:
|
def _upload(self, src: Readable, tmp: str, size: int, token: CancelToken) -> list[str]:
|
||||||
notes: list[str] = []
|
notes: list[str] = []
|
||||||
written = 0
|
written = 0
|
||||||
with self.client.open(self._p(tmp), "wb") as fh, src.open("rb") as fin:
|
with network_errors(), self.client.open(self._p(tmp), "wb") as fh, src.open("rb") as fin:
|
||||||
fh.set_pipelined(True)
|
fh.set_pipelined(True)
|
||||||
while chunk := fin.read(CHUNK // 4):
|
while chunk := fin.read(CHUNK // 4):
|
||||||
token.check()
|
token.check()
|
||||||
|
|||||||
+304
-107
@@ -9,18 +9,25 @@ Reihenfolge pro Lauf:
|
|||||||
unveränderter Größe **und** mtime gehen weiter (Fallstrick 1),
|
unveränderter Größe **und** mtime gehen weiter (Fallstrick 1),
|
||||||
5. pro Datei: ggf. Quarantäne, Kollisionsregel, Platzprüfung (Größe + 5 %,
|
5. pro Datei: ggf. Quarantäne, Kollisionsregel, Platzprüfung (Größe + 5 %,
|
||||||
Fallstrick 14), atomare Übertragung, erst dann Quelle löschen.
|
Fallstrick 14), atomare Übertragung, erst dann Quelle löschen.
|
||||||
|
|
||||||
|
Weitere Ziele (Archiv, Buchhaltung, …) werden **vor** dem Hauptziel beliefert,
|
||||||
|
jedes mit eigener Buchführung. Die Quelle wird erst gelöscht, wenn alle Ziele die
|
||||||
|
Datei haben. Ist ein Ziel nicht erreichbar, bekommen die übrigen die Datei trotzdem
|
||||||
|
(das Hauptziel dann als Kopie); die Quelle bleibt liegen, bis das fehlende Ziel
|
||||||
|
nachbeliefert ist.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import dataclasses
|
import dataclasses
|
||||||
import errno
|
import errno
|
||||||
import os
|
import posixpath
|
||||||
import stat
|
|
||||||
import time
|
import time
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable, Sequence
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
from netmover.logging_setup import event, get_logger
|
from netmover.logging_setup import event, get_logger
|
||||||
from netmover.models import (
|
from netmover.models import (
|
||||||
@@ -40,7 +47,8 @@ from netmover.mover.destinations import (
|
|||||||
)
|
)
|
||||||
from netmover.mover.ledger import FailureInfo, Ledger
|
from netmover.mover.ledger import FailureInfo, Ledger
|
||||||
from netmover.mover.naming import collision_candidates
|
from netmover.mover.naming import collision_candidates
|
||||||
from netmover.mover.scan import BUSY, Candidate, FileState, Matcher, current_state, scan
|
from netmover.mover.scan import BUSY, Candidate, FileState, Matcher
|
||||||
|
from netmover.mover.sources import LocalSource, Source, SourceMissing
|
||||||
from netmover.timeouts import Cancelled, CancelToken
|
from netmover.timeouts import Cancelled, CancelToken
|
||||||
|
|
||||||
log = get_logger(__name__)
|
log = get_logger(__name__)
|
||||||
@@ -73,6 +81,17 @@ class RuleSnapshot:
|
|||||||
dst_root_squash: bool | None = None # nur NFS-Ziele: Ergebnis des Schreibtests
|
dst_root_squash: bool | None = None # nur NFS-Ziele: Ergebnis des Schreibtests
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Target:
|
||||||
|
"""Weiteres Ziel eines Laufs. ``dest`` None = gerade nicht erreichbar (``reason``)."""
|
||||||
|
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
dest: Destination | None
|
||||||
|
layout: str = "flat" # "flat" | "date" (Jahr/Monat/…)
|
||||||
|
reason: str = ""
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class ItemResult:
|
class ItemResult:
|
||||||
src: str
|
src: str
|
||||||
@@ -92,6 +111,10 @@ class RunResult:
|
|||||||
failed: int = 0
|
failed: int = 0
|
||||||
message: str | None = None
|
message: str | None = None
|
||||||
items: list[ItemResult] = field(default_factory=list)
|
items: list[ItemResult] = field(default_factory=list)
|
||||||
|
# Quelle oder Ziel nicht erreichbar → Regel wartet (automatisch pausiert)
|
||||||
|
unavailable: bool = False
|
||||||
|
# Ziele, die in diesem Lauf nicht erreichbar waren (die übrigen wurden beliefert)
|
||||||
|
waiting_on: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
def add(self, item: ItemResult) -> None:
|
def add(self, item: ItemResult) -> None:
|
||||||
self.items.append(item)
|
self.items.append(item)
|
||||||
@@ -112,6 +135,17 @@ def _mb(n: int) -> str:
|
|||||||
return f"{n / 1024 / 1024:.1f} MB"
|
return f"{n / 1024 / 1024:.1f} MB"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class _SourceFile:
|
||||||
|
"""Adapter: eine Datei einer :class:`Source` mit ``open()`` wie ein ``Path``."""
|
||||||
|
|
||||||
|
source: Source
|
||||||
|
rel: str
|
||||||
|
|
||||||
|
def open(self, mode: str = "rb") -> Any:
|
||||||
|
return self.source.open(self.rel)
|
||||||
|
|
||||||
|
|
||||||
class Engine:
|
class Engine:
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -138,20 +172,32 @@ class Engine:
|
|||||||
def execute(
|
def execute(
|
||||||
self,
|
self,
|
||||||
rule: RuleSnapshot,
|
rule: RuleSnapshot,
|
||||||
src_root: Path,
|
source: Source | Path,
|
||||||
dest: Destination,
|
dest: Destination | None,
|
||||||
token: CancelToken,
|
token: CancelToken,
|
||||||
*,
|
*,
|
||||||
dry_run: bool = False,
|
dry_run: bool = False,
|
||||||
|
quarantine_rel: str | None = None,
|
||||||
quarantine_root: Path | None = None,
|
quarantine_root: Path | None = None,
|
||||||
|
targets: Sequence[Target] = (),
|
||||||
|
dest_reason: str = "",
|
||||||
) -> RunResult:
|
) -> RunResult:
|
||||||
|
"""``source``: eine :class:`Source` oder (Kurzform) ein lokales Verzeichnis.
|
||||||
|
Quarantäne relativ zur Wurzel der Anbindung (``quarantine_rel``) oder – bei
|
||||||
|
der Kurzform – als Pfad innerhalb des Verzeichnisses (``quarantine_root``).
|
||||||
|
``dest`` None = Hauptziel gerade nicht erreichbar (``dest_reason``)."""
|
||||||
result = RunResult(dry_run=dry_run)
|
result = RunResult(dry_run=dry_run)
|
||||||
|
src = source if isinstance(source, Source) else LocalSource(source)
|
||||||
|
if quarantine_root is not None and isinstance(src, LocalSource):
|
||||||
|
quarantine_rel = str(quarantine_root.relative_to(src.root))
|
||||||
try:
|
try:
|
||||||
self._execute(rule, src_root, dest, token, result, dry_run, quarantine_root)
|
self._execute(
|
||||||
|
rule, src, dest, token, result, dry_run, quarantine_rel, targets, dest_reason
|
||||||
|
)
|
||||||
except Cancelled:
|
except Cancelled:
|
||||||
result.status = RunStatus.ERROR
|
result.status = RunStatus.ERROR
|
||||||
result.message = "Lauf abgebrochen"
|
result.message = "Lauf abgebrochen"
|
||||||
except RunAborted as exc:
|
except (RunAborted, SourceMissing) as exc:
|
||||||
result.status = RunStatus.ERROR
|
result.status = RunStatus.ERROR
|
||||||
result.message = str(exc)
|
result.message = str(exc)
|
||||||
except TransferError as exc:
|
except TransferError as exc:
|
||||||
@@ -168,35 +214,53 @@ class Engine:
|
|||||||
def _execute(
|
def _execute(
|
||||||
self,
|
self,
|
||||||
rule: RuleSnapshot,
|
rule: RuleSnapshot,
|
||||||
src_root: Path,
|
src: Source,
|
||||||
dest: Destination,
|
dest: Destination | None,
|
||||||
token: CancelToken,
|
token: CancelToken,
|
||||||
result: RunResult,
|
result: RunResult,
|
||||||
dry_run: bool,
|
dry_run: bool,
|
||||||
quarantine_root: Path | None,
|
quarantine_rel: str | None,
|
||||||
|
targets: Sequence[Target] = (),
|
||||||
|
dest_reason: str = "",
|
||||||
) -> None:
|
) -> None:
|
||||||
if not src_root.is_dir():
|
src.check_root()
|
||||||
raise RunAborted(f"Quellordner {src_root} existiert nicht")
|
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
dest.prepare()
|
if dest is not None:
|
||||||
|
dest.prepare()
|
||||||
|
for t in targets:
|
||||||
|
if t.dest is not None:
|
||||||
|
t.dest.prepare()
|
||||||
token.check()
|
token.check()
|
||||||
|
result.waiting_on = (
|
||||||
|
[dest_reason or "Hauptziel nicht erreichbar"] if dest is None else []
|
||||||
|
) + [t.reason or f"„{t.name}“ nicht erreichbar" for t in targets if t.dest is None]
|
||||||
|
|
||||||
exclude: set[Path] = set()
|
exclude = frozenset({quarantine_rel}) if quarantine_rel else frozenset()
|
||||||
if quarantine_root is not None:
|
if isinstance(src, LocalSource):
|
||||||
exclude.add(quarantine_root.resolve())
|
for d in (dest, *(t.dest for t in targets)):
|
||||||
if isinstance(dest, LocalDestination):
|
if isinstance(d, LocalDestination):
|
||||||
exclude.add(dest.root.resolve()) # Ziel innerhalb der Quelle nie erneut scannen
|
src.exclude_paths.add(d.root.resolve()) # Ziel innerhalb der Quelle auslassen
|
||||||
matcher = Matcher(rule.pattern, rule.pattern_type)
|
matcher = Matcher(rule.pattern, rule.pattern_type)
|
||||||
candidates = list(scan(src_root, matcher, rule.recursive, frozenset(exclude)))
|
candidates = src.scan(matcher, rule.recursive, exclude)
|
||||||
result.seen = len(candidates)
|
result.seen = len(candidates)
|
||||||
token.check()
|
token.check()
|
||||||
|
|
||||||
# Kopier-Buchführung (Fallstrick 11). Gilt auch für „verschieben“, falls
|
# Kopier-Buchführung (Fallstrick 11). Gilt auch für „verschieben“, falls
|
||||||
# dort einmal die Quelle nicht gelöscht werden konnte.
|
# dort einmal die Quelle nicht gelöscht werden konnte oder ein anderes Ziel
|
||||||
|
# noch fehlt. Dazu je weiteres Ziel, welche Dateiversion es schon hat.
|
||||||
done = self.ledger.copied_keys(rule.id)
|
done = self.ledger.copied_keys(rule.id)
|
||||||
|
delivered = {t.id: self.ledger.target_keys(t.id) for t in targets}
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
self.ledger.prune_copied(rule.id, {c.relpath for c in candidates})
|
present = {c.relpath for c in candidates}
|
||||||
fresh = [c for c in candidates if (c.relpath, c.state.size, c.state.mtime_ns) not in done]
|
self.ledger.prune_copied(rule.id, present)
|
||||||
|
for t in targets:
|
||||||
|
self.ledger.prune_target(t.id, present)
|
||||||
|
|
||||||
|
def complete(c: Candidate) -> bool:
|
||||||
|
key = (c.relpath, c.state.size, c.state.mtime_ns)
|
||||||
|
return key in done and all(key in delivered[t.id] for t in targets)
|
||||||
|
|
||||||
|
fresh = [c for c in candidates if not complete(c)]
|
||||||
unchanged = len(candidates) - len(fresh)
|
unchanged = len(candidates) - len(fresh)
|
||||||
|
|
||||||
pending: list[Candidate] = []
|
pending: list[Candidate] = []
|
||||||
@@ -225,13 +289,13 @@ class Engine:
|
|||||||
|
|
||||||
# Stabilität (Fallstrick 1): zwei frische Messungen (öffnen + fstat, nicht
|
# Stabilität (Fallstrick 1): zwei frische Messungen (öffnen + fstat, nicht
|
||||||
# der Verzeichnis-Cache) im Abstand des Stabilitätsfensters.
|
# der Verzeichnis-Cache) im Abstand des Stabilitätsfensters.
|
||||||
first: dict[str, FileState | None] = {c.relpath: current_state(c.path) for c in pending}
|
first: dict[str, FileState | None] = {c.relpath: src.state(c.relpath) for c in pending}
|
||||||
if pending and rule.stability_seconds > 0:
|
if pending and rule.stability_seconds > 0:
|
||||||
self._sleep(rule.stability_seconds)
|
self._sleep(rule.stability_seconds)
|
||||||
token.check()
|
token.check()
|
||||||
stable: list[Candidate] = []
|
stable: list[Candidate] = []
|
||||||
for c in pending:
|
for c in pending:
|
||||||
before, st = first[c.relpath], current_state(c.path)
|
before, st = first[c.relpath], src.state(c.relpath)
|
||||||
if before is None or st is None:
|
if before is None or st is None:
|
||||||
continue # inzwischen verschwunden – nichts zu tun
|
continue # inzwischen verschwunden – nichts zu tun
|
||||||
if st != before or st == BUSY:
|
if st != before or st == BUSY:
|
||||||
@@ -253,7 +317,21 @@ class Engine:
|
|||||||
failures = self.ledger.failures(rule.id)
|
failures = self.ledger.failures(rule.id)
|
||||||
for c in stable:
|
for c in stable:
|
||||||
token.check()
|
token.check()
|
||||||
result.add(self._process(rule, c, dest, token, dry_run, quarantine_root, failures))
|
item = self._process(
|
||||||
|
rule,
|
||||||
|
c,
|
||||||
|
src,
|
||||||
|
dest,
|
||||||
|
token,
|
||||||
|
dry_run,
|
||||||
|
quarantine_rel,
|
||||||
|
failures,
|
||||||
|
targets,
|
||||||
|
delivered,
|
||||||
|
done,
|
||||||
|
)
|
||||||
|
if item is not None:
|
||||||
|
result.add(item)
|
||||||
|
|
||||||
if unchanged:
|
if unchanged:
|
||||||
result.message = f"{unchanged} bereits übertragen"
|
result.message = f"{unchanged} bereits übertragen"
|
||||||
@@ -261,9 +339,9 @@ class Engine:
|
|||||||
# -- einzelne Datei ------------------------------------------------------------------
|
# -- einzelne Datei ------------------------------------------------------------------
|
||||||
|
|
||||||
def _meta(
|
def _meta(
|
||||||
self, rule: RuleSnapshot, path: Path, dest: Destination
|
self, rule: RuleSnapshot, src: Source, rel: str, dest: Destination
|
||||||
) -> tuple[MetaOptions, list[str]]:
|
) -> tuple[MetaOptions, list[str]]:
|
||||||
st = os.stat(path)
|
st = src.stat(rel)
|
||||||
notes: list[str] = []
|
notes: list[str] = []
|
||||||
owner = rule.set_owner
|
owner = rule.set_owner
|
||||||
if owner is not None:
|
if owner is not None:
|
||||||
@@ -278,25 +356,39 @@ class Engine:
|
|||||||
if self._warn_once(f"{rule.id}:nfs-chown"):
|
if self._warn_once(f"{rule.id}:nfs-chown"):
|
||||||
reason = "Root-Squash aktiv" if rule.dst_root_squash else "Schreibtest fehlt"
|
reason = "Root-Squash aktiv" if rule.dst_root_squash else "Schreibtest fehlt"
|
||||||
notes.append(f"NFS-Ziel: chown übersprungen ({reason})")
|
notes.append(f"NFS-Ziel: chown übersprungen ({reason})")
|
||||||
mode = rule.set_mode if rule.set_mode is not None else stat.S_IMODE(st.st_mode)
|
mode = rule.set_mode if rule.set_mode is not None else st.mode
|
||||||
return MetaOptions(st.st_atime_ns, st.st_mtime_ns, mode, owner), notes
|
return MetaOptions(st.atime_ns, st.mtime_ns, mode, owner), notes
|
||||||
|
|
||||||
def _process(
|
def _process(
|
||||||
self,
|
self,
|
||||||
rule: RuleSnapshot,
|
rule: RuleSnapshot,
|
||||||
c: Candidate,
|
c: Candidate,
|
||||||
dest: Destination,
|
src: Source,
|
||||||
|
dest: Destination | None,
|
||||||
token: CancelToken,
|
token: CancelToken,
|
||||||
dry_run: bool,
|
dry_run: bool,
|
||||||
quarantine_root: Path | None,
|
quarantine_rel: str | None,
|
||||||
failures: dict[str, FailureInfo],
|
failures: dict[str, FailureInfo],
|
||||||
) -> ItemResult:
|
targets: Sequence[Target] = (),
|
||||||
|
delivered: dict[int, set[tuple[str, int, int]]] | None = None,
|
||||||
|
done: set[tuple[str, int, int]] | None = None,
|
||||||
|
) -> ItemResult | None:
|
||||||
|
"""Eine Datei an alle Ziele, die sie noch nicht haben. None = nichts zu tun,
|
||||||
|
weil nur noch nicht erreichbare Ziele fehlen (kein Protokolleintrag)."""
|
||||||
size = c.state.size
|
size = c.state.size
|
||||||
|
key = (c.relpath, size, c.state.mtime_ns)
|
||||||
|
delivered = delivered or {}
|
||||||
|
need = [t for t in targets if key not in delivered.get(t.id, set())]
|
||||||
|
reachable = [(t, t.dest) for t in need if t.dest is not None]
|
||||||
|
waiting = [t.name for t in need if t.dest is None]
|
||||||
|
main_needed = key not in (done or set())
|
||||||
|
if main_needed and dest is None:
|
||||||
|
waiting.append("Hauptziel")
|
||||||
failure = failures.get(c.relpath)
|
failure = failures.get(c.relpath)
|
||||||
|
|
||||||
# Quarantäne für wiederholt scheiternde Dateien
|
# Quarantäne für wiederholt scheiternde Dateien
|
||||||
if (
|
if (
|
||||||
quarantine_root is not None
|
quarantine_rel is not None
|
||||||
and failure is not None
|
and failure is not None
|
||||||
and failure.attempts >= rule.quarantine_after
|
and failure.attempts >= rule.quarantine_after
|
||||||
and (failure.size, failure.mtime_ns) == (c.state.size, c.state.mtime_ns)
|
and (failure.size, failure.mtime_ns) == (c.state.size, c.state.mtime_ns)
|
||||||
@@ -305,88 +397,131 @@ class Engine:
|
|||||||
return ItemResult(
|
return ItemResult(
|
||||||
c.relpath,
|
c.relpath,
|
||||||
ItemStatus.PLANNED,
|
ItemStatus.PLANNED,
|
||||||
str(quarantine_root / c.relpath),
|
src.describe_root_rel(posixpath.join(quarantine_rel, c.relpath)),
|
||||||
size,
|
size,
|
||||||
"würde in Quarantäne verschoben",
|
"würde in Quarantäne verschoben",
|
||||||
)
|
)
|
||||||
return self._quarantine(rule, c, quarantine_root, failure.attempts)
|
return self._quarantine(rule, c, src, quarantine_rel, failure.attempts)
|
||||||
|
|
||||||
try:
|
planned: str | None = None
|
||||||
planned = dest.plan(c.relpath, rule.on_collision)
|
if main_needed and dest is not None:
|
||||||
except OSError as exc:
|
try:
|
||||||
return self._fail(rule, c, exc, dry_run)
|
planned = dest.plan(c.relpath, rule.on_collision)
|
||||||
if planned is None:
|
except OSError as exc:
|
||||||
return ItemResult(
|
return self._fail(rule, c, exc, dry_run)
|
||||||
c.relpath,
|
if planned is None:
|
||||||
ItemStatus.SKIPPED,
|
return ItemResult(
|
||||||
dest.describe(c.relpath),
|
c.relpath,
|
||||||
size,
|
ItemStatus.SKIPPED,
|
||||||
"existiert bereits im Ziel",
|
dest.describe(c.relpath),
|
||||||
)
|
size,
|
||||||
|
"existiert bereits im Ziel",
|
||||||
|
)
|
||||||
if dry_run:
|
if dry_run:
|
||||||
msg = (
|
return self._plan_item(rule, c, dest, planned, targets, delivered, key)
|
||||||
"würde überschreiben"
|
if not reachable and planned is None:
|
||||||
if (rule.on_collision is CollisionPolicy.OVERWRITE and dest.exists(planned))
|
return None # es fehlen nur Ziele, die gerade nicht erreichbar sind
|
||||||
else None
|
|
||||||
)
|
|
||||||
verb = "kopieren" if rule.action is RuleAction.COPY else "verschieben"
|
|
||||||
return ItemResult(
|
|
||||||
c.relpath, ItemStatus.PLANNED, dest.describe(planned), size, msg or f"würde {verb}"
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
if dest is not None and planned is not None:
|
||||||
free = dest.free_bytes()
|
short = self._no_space(dest, size, "zu wenig Platz am Ziel")
|
||||||
except OSError:
|
if short:
|
||||||
free = None
|
return ItemResult(c.relpath, ItemStatus.FAILED, dest.describe(planned), size, short)
|
||||||
if free is not None and free < size * SPACE_RESERVE:
|
for t, tdest in reachable:
|
||||||
return ItemResult(
|
short = self._no_space(tdest, size, f"{t.name}: zu wenig Platz")
|
||||||
c.relpath,
|
if short:
|
||||||
ItemStatus.FAILED,
|
return ItemResult(
|
||||||
dest.describe(planned),
|
c.relpath, ItemStatus.FAILED, tdest.describe(c.relpath), size, short
|
||||||
size,
|
)
|
||||||
f"zu wenig Platz am Ziel (frei {_mb(free)}, "
|
|
||||||
f"benötigt {_mb(int(size * SPACE_RESERVE))})",
|
|
||||||
)
|
|
||||||
|
|
||||||
def verify() -> None:
|
def verify() -> None:
|
||||||
if current_state(c.path) != c.state:
|
if src.state(c.relpath) != c.state:
|
||||||
raise SourceChanged("Quelle hat sich während der Übertragung geändert")
|
raise SourceChanged("Quelle hat sich während der Übertragung geändert")
|
||||||
|
|
||||||
try:
|
# Weitere Ziele ZUERST: scheitert eines, ist noch nichts ans Hauptziel (z. B.
|
||||||
meta, notes = self._meta(rule, c.path, dest)
|
# Paperless) geliefert. Scheitert später etwas anderes, verhindert die
|
||||||
res = None
|
# Buchführung eine zweite Kopie beim nächsten Versuch.
|
||||||
if rule.action is RuleAction.MOVE:
|
notes: list[str] = []
|
||||||
res = dest.move_same_device(c.path, c.relpath, rule.on_collision, meta)
|
for t, tdest in reachable:
|
||||||
if res is None:
|
rel = self._layout_rel(c.relpath, t.layout)
|
||||||
res = dest.commit(c.path, size, c.relpath, rule.on_collision, meta, verify, token)
|
try:
|
||||||
except CollisionSkip:
|
meta, _ = self._meta(rule, src, c.relpath, tdest)
|
||||||
return ItemResult(
|
tres = tdest.commit(
|
||||||
c.relpath,
|
_SourceFile(src, c.relpath),
|
||||||
ItemStatus.SKIPPED,
|
size,
|
||||||
dest.describe(c.relpath),
|
rel,
|
||||||
size,
|
CollisionPolicy.RENAME, # weitere Ziele überschreiben nie
|
||||||
"existiert bereits im Ziel",
|
MetaOptions(meta.atime_ns, meta.mtime_ns, meta.mode, None),
|
||||||
)
|
verify,
|
||||||
except SourceChanged as exc:
|
token,
|
||||||
return ItemResult(c.relpath, ItemStatus.SKIPPED, None, size, str(exc))
|
)
|
||||||
except (TransferError, OSError) as exc:
|
except CollisionSkip: # kann bei RENAME nicht auftreten
|
||||||
return self._fail(rule, c, exc, dry_run)
|
return ItemResult(c.relpath, ItemStatus.SKIPPED, None, size, f"{t.name}: belegt")
|
||||||
|
except SourceChanged as exc:
|
||||||
|
return ItemResult(c.relpath, ItemStatus.SKIPPED, None, size, str(exc))
|
||||||
|
except (TransferError, OSError) as exc:
|
||||||
|
return self._fail(rule, c, exc, dry_run, prefix=f"{t.name}: ")
|
||||||
|
path = tdest.describe(tres.final_rel)
|
||||||
|
self.ledger.record_target(t.id, c.relpath, size, c.state.mtime_ns, path)
|
||||||
|
notes.append(f"{t.name}: {path}")
|
||||||
|
|
||||||
|
target: str | None = None
|
||||||
|
consumed = False
|
||||||
|
if dest is not None and planned is not None:
|
||||||
|
try:
|
||||||
|
meta, mnotes = self._meta(rule, src, c.relpath, dest)
|
||||||
|
res = None
|
||||||
|
local = src.local_path(c.relpath)
|
||||||
|
# Schneller Weg (umbenennen) nur, wenn danach kein Ziel mehr fehlt
|
||||||
|
if rule.action is RuleAction.MOVE and not waiting and local is not None:
|
||||||
|
res = dest.move_same_device(local, c.relpath, rule.on_collision, meta)
|
||||||
|
if res is None:
|
||||||
|
res = dest.commit(
|
||||||
|
_SourceFile(src, c.relpath),
|
||||||
|
size,
|
||||||
|
c.relpath,
|
||||||
|
rule.on_collision,
|
||||||
|
meta,
|
||||||
|
verify,
|
||||||
|
token,
|
||||||
|
)
|
||||||
|
except CollisionSkip:
|
||||||
|
return ItemResult(
|
||||||
|
c.relpath,
|
||||||
|
ItemStatus.SKIPPED,
|
||||||
|
dest.describe(c.relpath),
|
||||||
|
size,
|
||||||
|
"existiert bereits im Ziel",
|
||||||
|
)
|
||||||
|
except SourceChanged as exc:
|
||||||
|
return ItemResult(c.relpath, ItemStatus.SKIPPED, None, size, str(exc))
|
||||||
|
except (TransferError, OSError) as exc:
|
||||||
|
return self._fail(rule, c, exc, dry_run)
|
||||||
|
notes += mnotes + res.notes
|
||||||
|
target = dest.describe(res.final_rel)
|
||||||
|
consumed = res.source_consumed
|
||||||
|
if waiting or rule.action is RuleAction.COPY:
|
||||||
|
self.ledger.record_copy(rule.id, c.relpath, size, c.state.mtime_ns, target)
|
||||||
|
elif not main_needed:
|
||||||
|
notes.append("Hauptziel hatte die Datei bereits")
|
||||||
|
|
||||||
notes += res.notes
|
|
||||||
target = dest.describe(res.final_rel)
|
|
||||||
if c.relpath in failures:
|
if c.relpath in failures:
|
||||||
self.ledger.clear_failure(rule.id, c.relpath)
|
self.ledger.clear_failure(rule.id, c.relpath)
|
||||||
|
|
||||||
|
if waiting:
|
||||||
|
notes.append(f"wartet auf: {', '.join(waiting)} – Quelle bleibt bis dahin")
|
||||||
|
return ItemResult(c.relpath, ItemStatus.SKIPPED, target, size, "; ".join(notes))
|
||||||
|
|
||||||
if rule.action is RuleAction.COPY:
|
if rule.action is RuleAction.COPY:
|
||||||
self.ledger.record_copy(rule.id, c.relpath, size, c.state.mtime_ns, target)
|
|
||||||
return ItemResult(c.relpath, ItemStatus.COPIED, target, size, "; ".join(notes) or None)
|
return ItemResult(c.relpath, ItemStatus.COPIED, target, size, "; ".join(notes) or None)
|
||||||
|
|
||||||
if not res.source_consumed:
|
if not consumed:
|
||||||
try:
|
try:
|
||||||
os.unlink(c.path)
|
src.delete(c.relpath)
|
||||||
except OSError as exc:
|
except OSError as exc:
|
||||||
# Ziel ist vollständig – Buchführung verhindert Duplikate im nächsten Lauf.
|
# Ziele sind vollständig – Buchführung verhindert Duplikate im nächsten Lauf.
|
||||||
self.ledger.record_copy(rule.id, c.relpath, size, c.state.mtime_ns, target)
|
self.ledger.record_copy(
|
||||||
|
rule.id, c.relpath, size, c.state.mtime_ns, target or "(früher geliefert)"
|
||||||
|
)
|
||||||
return ItemResult(
|
return ItemResult(
|
||||||
c.relpath,
|
c.relpath,
|
||||||
ItemStatus.FAILED,
|
ItemStatus.FAILED,
|
||||||
@@ -397,10 +532,55 @@ class Engine:
|
|||||||
)
|
)
|
||||||
return ItemResult(c.relpath, ItemStatus.MOVED, target, size, "; ".join(notes) or None)
|
return ItemResult(c.relpath, ItemStatus.MOVED, target, size, "; ".join(notes) or None)
|
||||||
|
|
||||||
def _fail(self, rule: RuleSnapshot, c: Candidate, exc: Exception, dry_run: bool) -> ItemResult:
|
def _plan_item(
|
||||||
|
self,
|
||||||
|
rule: RuleSnapshot,
|
||||||
|
c: Candidate,
|
||||||
|
dest: Destination | None,
|
||||||
|
planned: str | None,
|
||||||
|
targets: Sequence[Target],
|
||||||
|
delivered: dict[int, set[tuple[str, int, int]]],
|
||||||
|
key: tuple[str, int, int],
|
||||||
|
) -> ItemResult:
|
||||||
|
"""Probelauf: was mit der Datei an welchem Ziel passieren würde."""
|
||||||
|
verb = "kopieren" if rule.action is RuleAction.COPY else "verschieben"
|
||||||
|
if dest is None:
|
||||||
|
parts = ["Hauptziel nicht erreichbar"]
|
||||||
|
elif planned is None:
|
||||||
|
parts = ["Hauptziel hat die Datei bereits"]
|
||||||
|
elif rule.on_collision is CollisionPolicy.OVERWRITE and dest.exists(planned):
|
||||||
|
parts = ["würde überschreiben"]
|
||||||
|
else:
|
||||||
|
parts = [f"würde {verb}"]
|
||||||
|
for t in targets:
|
||||||
|
if key in delivered.get(t.id, set()):
|
||||||
|
parts.append(f"{t.name}: bereits geliefert")
|
||||||
|
elif t.dest is None:
|
||||||
|
parts.append(f"{t.name}: nicht erreichbar")
|
||||||
|
else:
|
||||||
|
rel = self._layout_rel(c.relpath, t.layout)
|
||||||
|
parts.append(
|
||||||
|
f"{t.name}: {t.dest.describe(t.dest.plan(rel, CollisionPolicy.RENAME) or rel)}"
|
||||||
|
)
|
||||||
|
where = dest.describe(planned) if dest is not None and planned is not None else None
|
||||||
|
return ItemResult(c.relpath, ItemStatus.PLANNED, where, c.state.size, "; ".join(parts))
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _no_space(d: Destination, size: int, label: str) -> str | None:
|
||||||
|
try:
|
||||||
|
free = d.free_bytes()
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
if free is not None and free < size * SPACE_RESERVE:
|
||||||
|
return f"{label} (frei {_mb(free)}, benötigt {_mb(int(size * SPACE_RESERVE))})"
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _fail(
|
||||||
|
self, rule: RuleSnapshot, c: Candidate, exc: Exception, dry_run: bool, prefix: str = ""
|
||||||
|
) -> ItemResult:
|
||||||
if isinstance(exc, OSError) and exc.errno in MOUNT_ERRNOS:
|
if isinstance(exc, OSError) and exc.errno in MOUNT_ERRNOS:
|
||||||
raise RunAborted(f"Anbindung gestört: {_describe_os_error(exc)}") from exc
|
raise RunAborted(f"{prefix}Anbindung gestört: {_describe_os_error(exc)}") from exc
|
||||||
msg = _describe_os_error(exc) if isinstance(exc, OSError) else str(exc)
|
msg = prefix + (_describe_os_error(exc) if isinstance(exc, OSError) else str(exc))
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
attempts = self.ledger.record_failure(
|
attempts = self.ledger.record_failure(
|
||||||
rule.id, c.relpath, c.state.size, c.state.mtime_ns, msg
|
rule.id, c.relpath, c.state.size, c.state.mtime_ns, msg
|
||||||
@@ -409,16 +589,25 @@ class Engine:
|
|||||||
event(log, "file.failed", level=30, rule=rule.name, file=c.relpath, error=msg)
|
event(log, "file.failed", level=30, rule=rule.name, file=c.relpath, error=msg)
|
||||||
return ItemResult(c.relpath, ItemStatus.FAILED, None, c.state.size, msg)
|
return ItemResult(c.relpath, ItemStatus.FAILED, None, c.state.size, msg)
|
||||||
|
|
||||||
|
def _layout_rel(self, relpath: str, layout: str) -> str:
|
||||||
|
if layout == "date":
|
||||||
|
t = datetime.fromtimestamp(self._clock())
|
||||||
|
return f"{t:%Y}/{t:%m}/{relpath}"
|
||||||
|
return relpath
|
||||||
|
|
||||||
def _quarantine(
|
def _quarantine(
|
||||||
self, rule: RuleSnapshot, c: Candidate, root: Path, attempts: int
|
self, rule: RuleSnapshot, c: Candidate, src: Source, q_rel: str, attempts: int
|
||||||
) -> ItemResult:
|
) -> ItemResult:
|
||||||
|
target = ""
|
||||||
try:
|
try:
|
||||||
for cand in collision_candidates(c.relpath):
|
for cand in collision_candidates(c.relpath):
|
||||||
target = root / cand
|
target = posixpath.join(q_rel, cand)
|
||||||
if os.path.lexists(target):
|
if src.exists_root_rel(target):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
src.move_to(c.relpath, target)
|
||||||
|
except FileExistsError:
|
||||||
continue
|
continue
|
||||||
target.parent.mkdir(mode=0o755, parents=True, exist_ok=True)
|
|
||||||
os.rename(c.path, target)
|
|
||||||
break
|
break
|
||||||
else:
|
else:
|
||||||
raise TransferError("kein freier Name im Quarantäneordner")
|
raise TransferError("kein freier Name im Quarantäneordner")
|
||||||
@@ -432,10 +621,18 @@ class Engine:
|
|||||||
return ItemResult(
|
return ItemResult(
|
||||||
c.relpath,
|
c.relpath,
|
||||||
ItemStatus.QUARANTINED,
|
ItemStatus.QUARANTINED,
|
||||||
str(target),
|
src.describe_root_rel(target),
|
||||||
c.state.size,
|
c.state.size,
|
||||||
f"nach {attempts} Fehlversuchen in Quarantäne verschoben",
|
f"nach {attempts} Fehlversuchen in Quarantäne verschoben",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["Engine", "FileState", "ItemResult", "RuleSnapshot", "RunAborted", "RunResult"]
|
__all__ = [
|
||||||
|
"Engine",
|
||||||
|
"FileState",
|
||||||
|
"ItemResult",
|
||||||
|
"RuleSnapshot",
|
||||||
|
"RunAborted",
|
||||||
|
"RunResult",
|
||||||
|
"Target",
|
||||||
|
]
|
||||||
|
|||||||
@@ -3,6 +3,7 @@
|
|||||||
* ``copied_files`` (Fallstrick 11): Welche Dateiversion (Pfad, Größe, mtime) wurde
|
* ``copied_files`` (Fallstrick 11): Welche Dateiversion (Pfad, Größe, mtime) wurde
|
||||||
schon übertragen? Verhindert endloses Neu-Kopieren im Kopiermodus – und im
|
schon übertragen? Verhindert endloses Neu-Kopieren im Kopiermodus – und im
|
||||||
Verschiebemodus Duplikate, wenn die Quelle nicht gelöscht werden konnte.
|
Verschiebemodus Duplikate, wenn die Quelle nicht gelöscht werden konnte.
|
||||||
|
* ``target_files``: welche Dateiversion ein weiteres Ziel schon hat.
|
||||||
* ``file_failures``: Fehlversuche pro Datei als Grundlage der Quarantäne.
|
* ``file_failures``: Fehlversuche pro Datei als Grundlage der Quarantäne.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -14,7 +15,7 @@ from sqlalchemy import delete, select
|
|||||||
from sqlalchemy.orm import Session, sessionmaker
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
from netmover.models import CopiedFile, FileFailure, utcnow
|
from netmover.models import CopiedFile, FileFailure, TargetFile, utcnow
|
||||||
|
|
||||||
CopyKey = tuple[str, int, int] # (relpath, size, mtime_ns)
|
CopyKey = tuple[str, int, int] # (relpath, size, mtime_ns)
|
||||||
|
|
||||||
@@ -72,6 +73,53 @@ class Ledger:
|
|||||||
s.execute(delete(CopiedFile).where(CopiedFile.id.in_(stale)))
|
s.execute(delete(CopiedFile).where(CopiedFile.id.in_(stale)))
|
||||||
return len(stale)
|
return len(stale)
|
||||||
|
|
||||||
|
# -- weitere Ziele ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def target_keys(self, target_id: int) -> set[CopyKey]:
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
rows = s.execute(
|
||||||
|
select(TargetFile.src_relpath, TargetFile.size, TargetFile.mtime_ns).where(
|
||||||
|
TargetFile.target_id == target_id
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
return {(r[0], r[1], r[2]) for r in rows}
|
||||||
|
|
||||||
|
def record_target(
|
||||||
|
self, target_id: int, relpath: str, size: int, mtime_ns: int, dst: str
|
||||||
|
) -> None:
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
exists = s.scalar(
|
||||||
|
select(TargetFile.id).where(
|
||||||
|
TargetFile.target_id == target_id,
|
||||||
|
TargetFile.src_relpath == relpath,
|
||||||
|
TargetFile.size == size,
|
||||||
|
TargetFile.mtime_ns == mtime_ns,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if exists is None:
|
||||||
|
s.add(
|
||||||
|
TargetFile(
|
||||||
|
target_id=target_id,
|
||||||
|
src_relpath=relpath,
|
||||||
|
size=size,
|
||||||
|
mtime_ns=mtime_ns,
|
||||||
|
dst_path=dst,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def prune_target(self, target_id: int, present: set[str]) -> int:
|
||||||
|
"""Wie :meth:`prune_copied`: nur nach einem vollständigen Scan aufrufen."""
|
||||||
|
with session_scope(self.session_factory) as s:
|
||||||
|
rows = s.execute(
|
||||||
|
select(TargetFile.id, TargetFile.src_relpath).where(
|
||||||
|
TargetFile.target_id == target_id
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
stale = [r[0] for r in rows if r[1] not in present]
|
||||||
|
if stale:
|
||||||
|
s.execute(delete(TargetFile).where(TargetFile.id.in_(stale)))
|
||||||
|
return len(stale)
|
||||||
|
|
||||||
def failures(self, rule_id: int) -> dict[str, FailureInfo]:
|
def failures(self, rule_id: int) -> dict[str, FailureInfo]:
|
||||||
with session_scope(self.session_factory) as s:
|
with session_scope(self.session_factory) as s:
|
||||||
rows = s.scalars(select(FileFailure).where(FileFailure.rule_id == rule_id)).all()
|
rows = s.scalars(select(FileFailure).where(FileFailure.rule_id == rule_id)).all()
|
||||||
|
|||||||
+157
-39
@@ -23,7 +23,7 @@ from sqlalchemy.orm import Session, sessionmaker
|
|||||||
|
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
from netmover.logging_setup import get_logger
|
from netmover.logging_setup import get_logger
|
||||||
from netmover.models import MountType, Rule, RunStatus
|
from netmover.models import MountType, Rule, RunStatus, TargetKind
|
||||||
from netmover.mounting.manager import MountManager, MountUnavailable
|
from netmover.mounting.manager import MountManager, MountUnavailable
|
||||||
from netmover.mounting.sftp import SftpError, connect
|
from netmover.mounting.sftp import SftpError, connect
|
||||||
from netmover.mover.destinations import (
|
from netmover.mover.destinations import (
|
||||||
@@ -32,8 +32,9 @@ from netmover.mover.destinations import (
|
|||||||
SftpDestination,
|
SftpDestination,
|
||||||
TransferError,
|
TransferError,
|
||||||
)
|
)
|
||||||
from netmover.mover.engine import Engine, RuleSnapshot, RunResult
|
from netmover.mover.engine import Engine, RuleSnapshot, RunResult, Target
|
||||||
from netmover.mover.ledger import Ledger
|
from netmover.mover.ledger import Ledger
|
||||||
|
from netmover.mover.sources import LocalSource, SftpSource, Source
|
||||||
from netmover.timeouts import CancelToken, OperationTimeout, run_with_watchdog
|
from netmover.timeouts import CancelToken, OperationTimeout, run_with_watchdog
|
||||||
from netmover.validation import ValidationError, safe_abspath, safe_relpath
|
from netmover.validation import ValidationError, safe_abspath, safe_relpath
|
||||||
|
|
||||||
@@ -46,6 +47,7 @@ class RulePlan:
|
|||||||
name: str
|
name: str
|
||||||
src_mount_id: int
|
src_mount_id: int
|
||||||
src_type: MountType
|
src_type: MountType
|
||||||
|
src_name: str
|
||||||
src_subpath: str
|
src_subpath: str
|
||||||
dst_mount_id: int | None
|
dst_mount_id: int | None
|
||||||
dst_type: MountType | None
|
dst_type: MountType | None
|
||||||
@@ -54,6 +56,18 @@ class RulePlan:
|
|||||||
quarantine_path: str | None
|
quarantine_path: str | None
|
||||||
op_timeout: int
|
op_timeout: int
|
||||||
dry_run: bool
|
dry_run: bool
|
||||||
|
targets: tuple[TargetPlan, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TargetPlan:
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
mount_id: int | None # None = lokaler Pfad
|
||||||
|
mount_type: MountType | None
|
||||||
|
mount_name: str | None
|
||||||
|
path: str
|
||||||
|
layout: str
|
||||||
|
|
||||||
|
|
||||||
def parse_owner(value: str | None) -> tuple[int, int] | None:
|
def parse_owner(value: str | None) -> tuple[int, int] | None:
|
||||||
@@ -96,6 +110,7 @@ def load_plan(session: Session, rule_id: int) -> RulePlan:
|
|||||||
name=r.name,
|
name=r.name,
|
||||||
src_mount_id=r.src_mount_id,
|
src_mount_id=r.src_mount_id,
|
||||||
src_type=r.src_mount.type,
|
src_type=r.src_mount.type,
|
||||||
|
src_name=r.src_mount.name,
|
||||||
src_subpath=safe_relpath("src_subpath", r.src_subpath),
|
src_subpath=safe_relpath("src_subpath", r.src_subpath),
|
||||||
dst_mount_id=r.dst_mount_id,
|
dst_mount_id=r.dst_mount_id,
|
||||||
dst_type=dst.type if dst is not None else None,
|
dst_type=dst.type if dst is not None else None,
|
||||||
@@ -106,9 +121,38 @@ def load_plan(session: Session, rule_id: int) -> RulePlan:
|
|||||||
else None,
|
else None,
|
||||||
op_timeout=r.op_timeout_seconds,
|
op_timeout=r.op_timeout_seconds,
|
||||||
dry_run=r.dry_run,
|
dry_run=r.dry_run,
|
||||||
|
targets=tuple(
|
||||||
|
TargetPlan(
|
||||||
|
id=t.id,
|
||||||
|
name=t.name or f"Ziel {i + 2}",
|
||||||
|
mount_id=t.mount_id if t.kind is TargetKind.MOUNT else None,
|
||||||
|
mount_type=t.mount.type if t.mount is not None else None,
|
||||||
|
mount_name=t.mount.name if t.mount is not None else None,
|
||||||
|
path=t.path,
|
||||||
|
layout=t.layout.value,
|
||||||
|
)
|
||||||
|
for i, t in enumerate(r.targets)
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _with_waiting(result: RunResult) -> RunResult:
|
||||||
|
"""Nicht erreichbare Ziele: Ist in diesem Lauf sonst nichts passiert, gilt die
|
||||||
|
Regel als wartend (wie bei einer fehlenden Quelle, ohne Protokolleintrag);
|
||||||
|
sonst wird der Lauf normal protokolliert und nennt die fehlenden Ziele."""
|
||||||
|
if not result.waiting_on or result.status is RunStatus.ERROR:
|
||||||
|
return result
|
||||||
|
reason = "; ".join(result.waiting_on)
|
||||||
|
if not result.items:
|
||||||
|
result.status = RunStatus.ERROR
|
||||||
|
result.unavailable = True
|
||||||
|
result.message = reason
|
||||||
|
else:
|
||||||
|
note = f"wartet auf: {reason}"
|
||||||
|
result.message = f"{result.message}; {note}" if result.message else note
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
class RuleRunner:
|
class RuleRunner:
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -166,42 +210,89 @@ class RuleRunner:
|
|||||||
rule_id: int,
|
rule_id: int,
|
||||||
handed_over: list[bool],
|
handed_over: list[bool],
|
||||||
) -> RunResult:
|
) -> RunResult:
|
||||||
if not plan.src_type.is_kernel_mount:
|
|
||||||
return RunResult(
|
|
||||||
RunStatus.ERROR, dry, message="SFTP-Anbindungen sind nur als Ziel möglich"
|
|
||||||
)
|
|
||||||
try:
|
try:
|
||||||
src_mp = self.mounts.check_alive(plan.src_mount_id)
|
src_mp = self.mounts.check_alive(plan.src_mount_id)
|
||||||
dst_mp = (
|
|
||||||
self.mounts.check_alive(plan.dst_mount_id)
|
|
||||||
if plan.dst_mount_id is not None
|
|
||||||
else None
|
|
||||||
)
|
|
||||||
except MountUnavailable as exc:
|
except MountUnavailable as exc:
|
||||||
return RunResult(RunStatus.ERROR, dry, message=str(exc))
|
return RunResult(RunStatus.ERROR, dry, message=str(exc), unavailable=True)
|
||||||
if src_mp is None: # nur bei SFTP, oben bereits ausgeschlossen
|
|
||||||
return RunResult(RunStatus.ERROR, dry, message="Quelle ist kein Kernel-Mount")
|
# Ziele einzeln prüfen: ein nicht erreichbares Ziel hält die übrigen nicht auf.
|
||||||
src_root = src_mp / plan.src_subpath if plan.src_subpath else src_mp
|
def alive(mount_id: int | None) -> tuple[Path | None, str]:
|
||||||
quarantine_root = src_mp / plan.quarantine_path if plan.quarantine_path else None
|
if mount_id is None:
|
||||||
|
return None, ""
|
||||||
|
try:
|
||||||
|
return self.mounts.check_alive(mount_id), ""
|
||||||
|
except MountUnavailable as exc:
|
||||||
|
return None, str(exc)
|
||||||
|
|
||||||
|
dst_mp, dst_down = alive(plan.dst_mount_id)
|
||||||
|
target_state = [(t, *alive(t.mount_id)) for t in plan.targets]
|
||||||
|
if dst_down and all(down for _t, _mp, down in target_state):
|
||||||
|
return RunResult(RunStatus.ERROR, dry, message=dst_down, unavailable=True)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
make_dest = self._dest_factory(plan, dst_mp, src_root)
|
make_source = self._source_factory(plan, src_mp)
|
||||||
|
src_root = src_mp / plan.src_subpath if src_mp and plan.src_subpath else src_mp
|
||||||
|
make_dest = (
|
||||||
|
None
|
||||||
|
if dst_down
|
||||||
|
else self.destination_factory(
|
||||||
|
"dst_path",
|
||||||
|
plan.dst_mount_id,
|
||||||
|
plan.dst_type,
|
||||||
|
plan.dst_name,
|
||||||
|
plan.dst_path,
|
||||||
|
dst_mp,
|
||||||
|
src_root,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
make_targets = [
|
||||||
|
(
|
||||||
|
t,
|
||||||
|
down,
|
||||||
|
None
|
||||||
|
if down
|
||||||
|
else self.destination_factory(
|
||||||
|
"target_path", t.mount_id, t.mount_type, t.mount_name, t.path, mp, src_root
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for t, mp, down in target_state
|
||||||
|
]
|
||||||
except (ValidationError, TransferError, SftpError) as exc:
|
except (ValidationError, TransferError, SftpError) as exc:
|
||||||
return RunResult(RunStatus.ERROR, dry, message=str(exc))
|
return RunResult(RunStatus.ERROR, dry, message=str(exc))
|
||||||
|
|
||||||
def work(token: CancelToken) -> RunResult:
|
def work(token: CancelToken) -> RunResult:
|
||||||
dest = make_dest()
|
opened: list[Source | Destination] = []
|
||||||
try:
|
try:
|
||||||
return self.engine.execute(
|
source = make_source()
|
||||||
|
opened.append(source)
|
||||||
|
dest = make_dest() if make_dest is not None else None
|
||||||
|
if dest is not None:
|
||||||
|
opened.append(dest)
|
||||||
|
targets: list[Target] = []
|
||||||
|
for t, down, factory in make_targets:
|
||||||
|
tdest = None
|
||||||
|
if factory is not None:
|
||||||
|
try:
|
||||||
|
tdest = factory()
|
||||||
|
except (TransferError, SftpError, OSError) as exc:
|
||||||
|
down = f"„{t.name}“ nicht erreichbar: {exc}"
|
||||||
|
if tdest is not None:
|
||||||
|
opened.append(tdest)
|
||||||
|
targets.append(Target(t.id, t.name, tdest, t.layout, down))
|
||||||
|
result = self.engine.execute(
|
||||||
plan.snapshot,
|
plan.snapshot,
|
||||||
src_root,
|
source,
|
||||||
dest,
|
dest,
|
||||||
token,
|
token,
|
||||||
dry_run=dry,
|
dry_run=dry,
|
||||||
quarantine_root=quarantine_root,
|
quarantine_rel=plan.quarantine_path,
|
||||||
|
targets=targets,
|
||||||
|
dest_reason=dst_down,
|
||||||
)
|
)
|
||||||
finally:
|
finally:
|
||||||
dest.close()
|
for item in reversed(opened):
|
||||||
|
item.close()
|
||||||
|
return _with_waiting(result)
|
||||||
|
|
||||||
def finish() -> None:
|
def finish() -> None:
|
||||||
self._started.pop(rule_id, None)
|
self._started.pop(rule_id, None)
|
||||||
@@ -230,41 +321,68 @@ class RuleRunner:
|
|||||||
paths = {path.absolute(), path.resolve()}
|
paths = {path.absolute(), path.resolve()}
|
||||||
return any(p == b or b in p.parents for p in paths for b in bases)
|
return any(p == b or b in p.parents for p in paths for b in bases)
|
||||||
|
|
||||||
def _dest_factory(
|
def _source_factory(self, plan: RulePlan, src_mp: Path | None) -> Callable[[], Source]:
|
||||||
self, plan: RulePlan, dst_mp: Path | None, src_root: Path
|
if plan.src_type is MountType.SFTP:
|
||||||
|
scfg = self.mounts.sftp_config_for(plan.src_mount_id)
|
||||||
|
label = plan.src_name or "sftp"
|
||||||
|
|
||||||
|
def make_sftp() -> Source:
|
||||||
|
return SftpSource(connect(scfg), scfg.base_path, plan.src_subpath, label=label)
|
||||||
|
|
||||||
|
return make_sftp
|
||||||
|
if src_mp is None:
|
||||||
|
raise TransferError("Quell-Anbindung ist nicht eingehängt")
|
||||||
|
root = src_mp
|
||||||
|
|
||||||
|
def make_local() -> Source:
|
||||||
|
return LocalSource(root, plan.src_subpath)
|
||||||
|
|
||||||
|
return make_local
|
||||||
|
|
||||||
|
def destination_factory(
|
||||||
|
self,
|
||||||
|
field: str,
|
||||||
|
mount_id: int | None,
|
||||||
|
mount_type: MountType | None,
|
||||||
|
mount_name: str | None,
|
||||||
|
path: str,
|
||||||
|
mp: Path | None,
|
||||||
|
src_root: Path | None,
|
||||||
) -> Callable[[], Destination]:
|
) -> Callable[[], Destination]:
|
||||||
if plan.dst_mount_id is None:
|
"""Ziel oder Archiv: lokaler Pfad (``mount_id`` None), SFTP oder Kernel-Mount."""
|
||||||
root = Path(safe_abspath("dst_path", plan.dst_path))
|
what = {"target_path": "weiteres Ziel", "backup_path": "Sicherungsziel"}.get(field, "Ziel")
|
||||||
|
if mount_id is None:
|
||||||
|
root = Path(safe_abspath(field, path))
|
||||||
if self._under_mount_base(root):
|
if self._under_mount_base(root):
|
||||||
base = self.mounts.mount_base
|
base = self.mounts.mount_base
|
||||||
raise ValidationError(
|
raise ValidationError(
|
||||||
"dst_path",
|
field,
|
||||||
f"Lokales Ziel liegt unter {base} – bitte die Anbindung als Ziel wählen, "
|
f"Lokales {what} liegt unter {base} – bitte die Anbindung als {what} wählen, "
|
||||||
"damit geprüft wird, ob sie eingehängt ist.",
|
"damit geprüft wird, ob sie eingehängt ist.",
|
||||||
)
|
)
|
||||||
kind = "local"
|
kind = "local"
|
||||||
local_root = root
|
local_root = root
|
||||||
create = False
|
create = False
|
||||||
elif plan.dst_type is MountType.SFTP:
|
elif mount_type is MountType.SFTP:
|
||||||
scfg = self.mounts.sftp_config_for(plan.dst_mount_id)
|
scfg = self.mounts.sftp_config_for(mount_id)
|
||||||
sub = safe_relpath("dst_path", plan.dst_path)
|
sub = safe_relpath(field, path)
|
||||||
remote = posixpath.join(scfg.base_path, sub) if sub else scfg.base_path
|
remote = posixpath.join(scfg.base_path, sub) if sub else scfg.base_path
|
||||||
label = plan.dst_name or "sftp"
|
label = mount_name or "sftp"
|
||||||
|
|
||||||
def make_sftp() -> Destination:
|
def make_sftp() -> Destination:
|
||||||
return SftpDestination(connect(scfg), remote, label=label)
|
return SftpDestination(connect(scfg), remote, label=label)
|
||||||
|
|
||||||
return make_sftp
|
return make_sftp
|
||||||
else:
|
else:
|
||||||
if dst_mp is None:
|
if mp is None:
|
||||||
raise TransferError("Ziel-Anbindung ist nicht eingehängt")
|
raise TransferError(f"{what}-Anbindung ist nicht eingehängt")
|
||||||
sub = safe_relpath("dst_path", plan.dst_path)
|
sub = safe_relpath(field, path)
|
||||||
local_root = dst_mp / sub if sub else dst_mp
|
local_root = mp / sub if sub else mp
|
||||||
kind = "cifs" if plan.dst_type is MountType.CIFS else "nfs"
|
kind = "cifs" if mount_type is MountType.CIFS else "nfs"
|
||||||
create = True
|
create = True
|
||||||
|
|
||||||
if local_root.resolve() == src_root.resolve():
|
if src_root is not None and local_root.resolve() == src_root.resolve():
|
||||||
raise ValidationError("dst_path", "Quelle und Ziel sind derselbe Ordner")
|
raise ValidationError(field, f"Quelle und {what} sind derselbe Ordner")
|
||||||
|
|
||||||
def make_local() -> Destination:
|
def make_local() -> Destination:
|
||||||
return LocalDestination(local_root, kind, create_root=create) # type: ignore[arg-type]
|
return LocalDestination(local_root, kind, create_root=create) # type: ignore[arg-type]
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ class FileState:
|
|||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class Candidate:
|
class Candidate:
|
||||||
relpath: str # POSIX, relativ zum Quellordner
|
relpath: str # POSIX, relativ zum Quellordner
|
||||||
path: Path
|
path: Path | None # nur bei lokalen Quellen
|
||||||
state: FileState
|
state: FileState
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,311 @@
|
|||||||
|
"""Quellen: eingehängtes Verzeichnis (SMB/NFS) oder SFTP.
|
||||||
|
|
||||||
|
Die Engine greift nur über diese Schnittstelle auf die Quelle zu – scannen,
|
||||||
|
frischen Zustand messen, lesen, löschen, in die Quarantäne verschieben. So gelten
|
||||||
|
für SFTP-Quellen dieselben Garantien wie für eingehängte Freigaben.
|
||||||
|
|
||||||
|
Adressierung: ``root`` ist die Wurzel der Anbindung, ``subdir`` der Quell-
|
||||||
|
Unterordner der Regel. Kandidaten tragen Pfade relativ zu ``subdir`` (so landen sie
|
||||||
|
im Ziel und in der Buchführung); die Quarantäne liegt relativ zu ``root``.
|
||||||
|
|
||||||
|
SFTP-Besonderheit: Änderungszeiten kommen nur sekundengenau. Die Stabilitäts-
|
||||||
|
prüfung stützt sich dort vor allem auf die Größe – mit einem Fenster ab 2 s
|
||||||
|
unproblematisch.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
import errno
|
||||||
|
import os
|
||||||
|
import posixpath
|
||||||
|
import socket
|
||||||
|
import stat
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import IO, Any
|
||||||
|
|
||||||
|
import paramiko
|
||||||
|
|
||||||
|
from netmover.mounting.sftp import SftpSession
|
||||||
|
from netmover.mover.fsops import rename_noreplace
|
||||||
|
from netmover.mover.scan import (
|
||||||
|
IGNORED_DIRS,
|
||||||
|
Candidate,
|
||||||
|
FileState,
|
||||||
|
Matcher,
|
||||||
|
current_state,
|
||||||
|
is_ignored_name,
|
||||||
|
scan,
|
||||||
|
)
|
||||||
|
|
||||||
|
_NET_ERRORS = (paramiko.SSHException, EOFError, socket.timeout, ConnectionError)
|
||||||
|
|
||||||
|
|
||||||
|
class SourceMissing(Exception):
|
||||||
|
"""Quellordner existiert nicht (Lauf abbrechen, nichts anfassen)."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SourceStat:
|
||||||
|
size: int
|
||||||
|
mtime_ns: int
|
||||||
|
atime_ns: int
|
||||||
|
mode: int # Rechte-Bits
|
||||||
|
|
||||||
|
|
||||||
|
def _join(*parts: str) -> str:
|
||||||
|
return posixpath.join(*[p for p in parts if p]) if any(parts) else ""
|
||||||
|
|
||||||
|
|
||||||
|
@contextlib.contextmanager
|
||||||
|
def network_errors() -> Iterator[None]:
|
||||||
|
"""Verbindungsabbrüche bei SFTP als ``ENOTCONN`` melden – die Engine bricht den
|
||||||
|
Lauf dann ab (Anbindung gestört), statt jede Datei einzeln scheitern zu lassen."""
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
except _NET_ERRORS as exc:
|
||||||
|
raise OSError(errno.ENOTCONN, f"SFTP-Verbindung unterbrochen: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
class Source(ABC):
|
||||||
|
kind: str
|
||||||
|
|
||||||
|
def __init__(self, subdir: str = "") -> None:
|
||||||
|
self.subdir = subdir.strip("/")
|
||||||
|
|
||||||
|
def full(self, rel: str) -> str:
|
||||||
|
"""Pfad relativ zur Wurzel der Anbindung."""
|
||||||
|
return _join(self.subdir, rel)
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def describe(self, rel: str = "") -> str: ...
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def check_root(self) -> None:
|
||||||
|
"""Wirft :class:`SourceMissing`, wenn der Quellordner fehlt."""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def scan(
|
||||||
|
self, matcher: Matcher, recursive: bool, exclude: frozenset[str] = frozenset()
|
||||||
|
) -> list[Candidate]:
|
||||||
|
"""``exclude``: Verzeichnisse relativ zur Wurzel der Anbindung."""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def state(self, rel: str) -> FileState | None:
|
||||||
|
"""Frischer Zustand; ``None`` = verschwunden, ``BUSY`` = gerade gesperrt."""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def stat(self, rel: str) -> SourceStat: ...
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def open(self, rel: str) -> contextlib.AbstractContextManager[IO[bytes]]: ...
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def delete(self, rel: str) -> None: ...
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def exists_root_rel(self, root_rel: str) -> bool: ...
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def describe_root_rel(self, root_rel: str) -> str: ...
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def move_to(self, rel: str, root_rel_target: str) -> None:
|
||||||
|
"""Innerhalb der Anbindung verschieben, ohne zu überschreiben
|
||||||
|
(``FileExistsError``, wenn das Ziel existiert)."""
|
||||||
|
|
||||||
|
def local_path(self, rel: str) -> Path | None:
|
||||||
|
"""Lokaler Pfad, falls vorhanden (für den Schnellweg ohne Kopie)."""
|
||||||
|
return None
|
||||||
|
|
||||||
|
def close(self) -> None: # noqa: B027 – absichtlich optional
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class LocalSource(Source):
|
||||||
|
"""Eingehängte Freigabe (oder lokales Verzeichnis)."""
|
||||||
|
|
||||||
|
kind = "local"
|
||||||
|
|
||||||
|
def __init__(self, root: Path, subdir: str = "") -> None:
|
||||||
|
super().__init__(subdir)
|
||||||
|
self.root = root
|
||||||
|
self.base = root / self.subdir if self.subdir else root
|
||||||
|
# zusätzlich auszulassende Verzeichnisse, z. B. ein Ziel innerhalb der Quelle
|
||||||
|
self.exclude_paths: set[Path] = set()
|
||||||
|
|
||||||
|
def _p(self, rel: str) -> Path:
|
||||||
|
return self.base / rel
|
||||||
|
|
||||||
|
def describe(self, rel: str = "") -> str:
|
||||||
|
return str(self._p(rel)) if rel else str(self.base)
|
||||||
|
|
||||||
|
def check_root(self) -> None:
|
||||||
|
if not self.base.is_dir():
|
||||||
|
raise SourceMissing(f"Quellordner {self.base} existiert nicht")
|
||||||
|
|
||||||
|
def scan(
|
||||||
|
self, matcher: Matcher, recursive: bool, exclude: frozenset[str] = frozenset()
|
||||||
|
) -> list[Candidate]:
|
||||||
|
paths = {(self.root / e).resolve() for e in exclude} | set(self.exclude_paths)
|
||||||
|
return list(scan(self.base, matcher, recursive, frozenset(paths)))
|
||||||
|
|
||||||
|
def state(self, rel: str) -> FileState | None:
|
||||||
|
return current_state(self._p(rel))
|
||||||
|
|
||||||
|
def stat(self, rel: str) -> SourceStat:
|
||||||
|
st = os.stat(self._p(rel))
|
||||||
|
return SourceStat(st.st_size, st.st_mtime_ns, st.st_atime_ns, stat.S_IMODE(st.st_mode))
|
||||||
|
|
||||||
|
def open(self, rel: str) -> contextlib.AbstractContextManager[IO[bytes]]:
|
||||||
|
return self._p(rel).open("rb")
|
||||||
|
|
||||||
|
def delete(self, rel: str) -> None:
|
||||||
|
os.unlink(self._p(rel))
|
||||||
|
|
||||||
|
def exists_root_rel(self, root_rel: str) -> bool:
|
||||||
|
return os.path.lexists(self.root / root_rel)
|
||||||
|
|
||||||
|
def describe_root_rel(self, root_rel: str) -> str:
|
||||||
|
return str(self.root / root_rel)
|
||||||
|
|
||||||
|
def move_to(self, rel: str, root_rel_target: str) -> None:
|
||||||
|
target = self.root / root_rel_target
|
||||||
|
target.parent.mkdir(mode=0o755, parents=True, exist_ok=True)
|
||||||
|
if rename_noreplace(self._p(rel), target):
|
||||||
|
return
|
||||||
|
if os.path.lexists(target):
|
||||||
|
raise FileExistsError(errno.EEXIST, "exists", str(target))
|
||||||
|
os.rename(self._p(rel), target)
|
||||||
|
|
||||||
|
def local_path(self, rel: str) -> Path:
|
||||||
|
return self._p(rel)
|
||||||
|
|
||||||
|
|
||||||
|
class SftpSource(Source):
|
||||||
|
"""Quelle auf einem SFTP-Server (z. B. ein Server, der nur per SSH erreichbar ist)."""
|
||||||
|
|
||||||
|
kind = "sftp"
|
||||||
|
|
||||||
|
def __init__(self, session: SftpSession, base: str, subdir: str = "", label: str = ""):
|
||||||
|
super().__init__(subdir)
|
||||||
|
self.session = session
|
||||||
|
self.client = session.client
|
||||||
|
self.root = base.rstrip("/") or "/"
|
||||||
|
self.label = label or "sftp"
|
||||||
|
|
||||||
|
def _p(self, rel: str) -> str:
|
||||||
|
return posixpath.join(self.root, self.full(rel)) if self.full(rel) else self.root
|
||||||
|
|
||||||
|
def describe(self, rel: str = "") -> str:
|
||||||
|
return f"{self.label}:{self._p(rel)}"
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
|
||||||
|
def check_root(self) -> None:
|
||||||
|
with network_errors():
|
||||||
|
try:
|
||||||
|
attrs = self.client.stat(self._p(""))
|
||||||
|
except FileNotFoundError:
|
||||||
|
raise SourceMissing(f"Quellordner {self.describe()} existiert nicht") from None
|
||||||
|
if attrs.st_mode is None or not stat.S_ISDIR(attrs.st_mode):
|
||||||
|
raise SourceMissing(f"{self.describe()} ist kein Verzeichnis")
|
||||||
|
|
||||||
|
def scan(
|
||||||
|
self, matcher: Matcher, recursive: bool, exclude: frozenset[str] = frozenset()
|
||||||
|
) -> list[Candidate]:
|
||||||
|
excluded = {e.strip("/") for e in exclude}
|
||||||
|
found: list[Candidate] = []
|
||||||
|
|
||||||
|
def walk(rel_dir: str) -> None:
|
||||||
|
with network_errors():
|
||||||
|
entries = sorted(
|
||||||
|
self.client.listdir_attr(self._p(rel_dir)), key=lambda a: a.filename
|
||||||
|
)
|
||||||
|
for a in entries:
|
||||||
|
name = a.filename
|
||||||
|
if name.startswith(".") or a.st_mode is None:
|
||||||
|
continue
|
||||||
|
rel = _join(rel_dir, name)
|
||||||
|
if stat.S_ISDIR(a.st_mode):
|
||||||
|
if recursive and name not in IGNORED_DIRS and self.full(rel) not in excluded:
|
||||||
|
walk(rel)
|
||||||
|
continue
|
||||||
|
if not stat.S_ISREG(a.st_mode) or is_ignored_name(name):
|
||||||
|
continue # Symlinks, Geräte … und unfertige Übertragungen
|
||||||
|
if matcher(rel):
|
||||||
|
found.append(Candidate(rel, None, _state(a)))
|
||||||
|
|
||||||
|
walk("")
|
||||||
|
return found
|
||||||
|
|
||||||
|
def state(self, rel: str) -> FileState | None:
|
||||||
|
with network_errors():
|
||||||
|
try:
|
||||||
|
a = self.client.lstat(self._p(rel))
|
||||||
|
except FileNotFoundError:
|
||||||
|
return None
|
||||||
|
if a.st_mode is None or not stat.S_ISREG(a.st_mode):
|
||||||
|
return None
|
||||||
|
return _state(a)
|
||||||
|
|
||||||
|
def stat(self, rel: str) -> SourceStat:
|
||||||
|
with network_errors():
|
||||||
|
a = self.client.lstat(self._p(rel))
|
||||||
|
mtime = int(a.st_mtime or 0) * 1_000_000_000
|
||||||
|
atime = int(a.st_atime or a.st_mtime or 0) * 1_000_000_000
|
||||||
|
return SourceStat(int(a.st_size or 0), mtime, atime, stat.S_IMODE(a.st_mode or 0o644))
|
||||||
|
|
||||||
|
@contextlib.contextmanager
|
||||||
|
def _reader(self, rel: str) -> Iterator[IO[bytes]]:
|
||||||
|
with network_errors():
|
||||||
|
size = self.client.lstat(self._p(rel)).st_size
|
||||||
|
fh: Any = self.client.open(self._p(rel), "rb")
|
||||||
|
try:
|
||||||
|
fh.prefetch(size) # Größe mitgeben: spart ein fstat und liest vorab
|
||||||
|
yield fh
|
||||||
|
finally:
|
||||||
|
with contextlib.suppress(*_NET_ERRORS, OSError):
|
||||||
|
fh.close()
|
||||||
|
|
||||||
|
def open(self, rel: str) -> contextlib.AbstractContextManager[IO[bytes]]:
|
||||||
|
return self._reader(rel)
|
||||||
|
|
||||||
|
def delete(self, rel: str) -> None:
|
||||||
|
with network_errors():
|
||||||
|
self.client.remove(self._p(rel))
|
||||||
|
|
||||||
|
def exists_root_rel(self, root_rel: str) -> bool:
|
||||||
|
with network_errors():
|
||||||
|
try:
|
||||||
|
self.client.lstat(posixpath.join(self.root, root_rel))
|
||||||
|
except FileNotFoundError:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
def describe_root_rel(self, root_rel: str) -> str:
|
||||||
|
return f"{self.label}:{posixpath.join(self.root, root_rel)}"
|
||||||
|
|
||||||
|
def move_to(self, rel: str, root_rel_target: str) -> None:
|
||||||
|
target = posixpath.join(self.root, root_rel_target)
|
||||||
|
with network_errors():
|
||||||
|
current = self.root
|
||||||
|
for part in posixpath.dirname(root_rel_target).split("/"):
|
||||||
|
if not part:
|
||||||
|
continue
|
||||||
|
current = posixpath.join(current, part)
|
||||||
|
try:
|
||||||
|
self.client.stat(current)
|
||||||
|
except FileNotFoundError:
|
||||||
|
self.client.mkdir(current, mode=0o755)
|
||||||
|
if self.exists_root_rel(root_rel_target):
|
||||||
|
raise FileExistsError(errno.EEXIST, "exists", target)
|
||||||
|
self.client.rename(self._p(rel), target) # SFTPv3-rename überschreibt nicht
|
||||||
|
|
||||||
|
|
||||||
|
def _state(a: paramiko.SFTPAttributes) -> FileState:
|
||||||
|
return FileState(int(a.st_size or 0), int(a.st_mtime or 0) * 1_000_000_000)
|
||||||
+47
-8
@@ -22,7 +22,9 @@ from string import Template
|
|||||||
import httpx
|
import httpx
|
||||||
from sqlalchemy.orm import Session, sessionmaker
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
from netmover.logging_setup import event, get_logger
|
from netmover.logging_setup import event, get_logger
|
||||||
|
from netmover.mail import mail_configured, smtp_send
|
||||||
from netmover.settings_store import AppSettings, NotifyMode, load_settings
|
from netmover.settings_store import AppSettings, NotifyMode, load_settings
|
||||||
|
|
||||||
log = get_logger(__name__)
|
log = get_logger(__name__)
|
||||||
@@ -60,34 +62,71 @@ def http_sender(url: str, content_type: str, body: str) -> None:
|
|||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
|
|
||||||
|
|
||||||
|
Mailer = Callable[[AppSettings, "str | None", str, str], None] # (cfg, passwort, betreff, text)
|
||||||
|
|
||||||
|
|
||||||
class Notifier:
|
class Notifier:
|
||||||
|
"""Verteilt Benachrichtigungen auf die eingerichteten Kanäle (Webhook, E-Mail)."""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
session_factory: sessionmaker[Session],
|
session_factory: sessionmaker[Session],
|
||||||
*,
|
*,
|
||||||
sender: Sender = http_sender,
|
sender: Sender = http_sender,
|
||||||
|
mailer: Mailer | None = None,
|
||||||
|
secrets: SecretBox | None = None,
|
||||||
background: bool = True,
|
background: bool = True,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.session_factory = session_factory
|
self.session_factory = session_factory
|
||||||
self._sender = sender
|
self._sender = sender
|
||||||
|
self._mailer: Mailer = mailer or smtp_send
|
||||||
|
self._secrets = secrets
|
||||||
self._background = background
|
self._background = background
|
||||||
|
|
||||||
def settings(self) -> AppSettings:
|
def settings(self) -> AppSettings:
|
||||||
return load_settings(self.session_factory)
|
return load_settings(self.session_factory)
|
||||||
|
|
||||||
|
def smtp_password(self, cfg: AppSettings) -> str | None:
|
||||||
|
if not cfg.smtp_password_enc or self._secrets is None:
|
||||||
|
return None
|
||||||
|
return self._secrets.decrypt(cfg.smtp_password_enc)
|
||||||
|
|
||||||
|
def send_mail_now(self, subject: str, body: str, settings: AppSettings | None = None) -> None:
|
||||||
|
"""Synchron senden (Test-Knopf); Fehler werden weitergereicht."""
|
||||||
|
cfg = settings or self.settings()
|
||||||
|
self._mailer(cfg, self.smtp_password(cfg), subject, body)
|
||||||
|
|
||||||
def send(self, n: Notification, settings: AppSettings | None = None) -> None:
|
def send(self, n: Notification, settings: AppSettings | None = None) -> None:
|
||||||
cfg = settings or self.settings()
|
cfg = settings or self.settings()
|
||||||
if cfg.notify_mode is NotifyMode.OFF or not cfg.webhook_url:
|
if cfg.notify_mode is NotifyMode.OFF:
|
||||||
|
return
|
||||||
|
jobs: list[tuple[str, Callable[[], None]]] = []
|
||||||
|
if cfg.webhook_url:
|
||||||
|
body = render(cfg.webhook_template, cfg.webhook_content_type, n)
|
||||||
|
jobs.append(
|
||||||
|
("webhook", lambda: self._sender(cfg.webhook_url, cfg.webhook_content_type, body))
|
||||||
|
)
|
||||||
|
if mail_configured(cfg):
|
||||||
|
password = self.smtp_password(cfg)
|
||||||
|
jobs.append(("email", lambda: self._mailer(cfg, password, n.title, n.message)))
|
||||||
|
if not jobs:
|
||||||
return
|
return
|
||||||
body = render(cfg.webhook_template, cfg.webhook_content_type, n)
|
|
||||||
|
|
||||||
def deliver() -> None:
|
def deliver() -> None:
|
||||||
try:
|
for channel, job in jobs:
|
||||||
self._sender(cfg.webhook_url, cfg.webhook_content_type, body)
|
try:
|
||||||
event(log, "notify.sent", kind=n.event, rule=n.rule)
|
job()
|
||||||
except Exception as exc: # Benachrichtigung darf nie den Dienst stören
|
event(log, "notify.sent", kind=n.event, rule=n.rule, channel=channel)
|
||||||
# URL nicht protokollieren – sie enthält oft ein Token.
|
except Exception as exc: # Benachrichtigung darf nie den Dienst stören
|
||||||
event(log, "notify.failed", level=30, kind=n.event, error=type(exc).__name__)
|
# Weder URL noch Zugangsdaten protokollieren.
|
||||||
|
event(
|
||||||
|
log,
|
||||||
|
"notify.failed",
|
||||||
|
level=30,
|
||||||
|
kind=n.event,
|
||||||
|
channel=channel,
|
||||||
|
error=type(exc).__name__,
|
||||||
|
)
|
||||||
|
|
||||||
if self._background:
|
if self._background:
|
||||||
threading.Thread(target=deliver, name="nm-notify", daemon=True).start()
|
threading.Thread(target=deliver, name="nm-notify", daemon=True).start()
|
||||||
|
|||||||
@@ -8,6 +8,14 @@ die Einzeldateien in ``run_items``. Danach die Regelzähler:
|
|||||||
* ``consecutive_failures`` – ``error`` und ``partial`` zählen hoch, ``ok`` setzt zurück.
|
* ``consecutive_failures`` – ``error`` und ``partial`` zählen hoch, ``ok`` setzt zurück.
|
||||||
|
|
||||||
Ein manuell gestarteter Probelauf verändert keine Zähler.
|
Ein manuell gestarteter Probelauf verändert keine Zähler.
|
||||||
|
|
||||||
|
Automatisch pausieren (0.4.0): Ist Quelle oder Ziel nicht erreichbar, wechselt die
|
||||||
|
Regel in „wartet auf Anbindung“. Protokolliert wird nur der Beginn, nicht jeder
|
||||||
|
weitere Termin, und „Fehler in Folge“ zählt nicht hoch – die Benachrichtigung
|
||||||
|
kommt einmal pro Anbindung vom MountManager. Sobald ein Lauf wieder durchkommt,
|
||||||
|
endet das Warten von selbst, und der Lauf vermerkt die Wartezeit.
|
||||||
|
|
||||||
|
Wartungsmodus: geplante Läufe werden gar nicht erst ausgeführt.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -51,6 +59,8 @@ class RunService:
|
|||||||
) -> int | None:
|
) -> int | None:
|
||||||
"""Führt die Regel aus, protokolliert den Lauf und liefert die Run-ID
|
"""Führt die Regel aus, protokolliert den Lauf und liefert die Run-ID
|
||||||
(None, wenn die Regel inzwischen gelöscht wurde)."""
|
(None, wenn die Regel inzwischen gelöscht wurde)."""
|
||||||
|
if trigger is RunTrigger.SCHEDULE and load_settings(self.session_factory).maintenance:
|
||||||
|
return None
|
||||||
started = utcnow()
|
started = utcnow()
|
||||||
result = self.runner.run(rule_id, dry_run=dry_run)
|
result = self.runner.run(rule_id, dry_run=dry_run)
|
||||||
finished = utcnow()
|
finished = utcnow()
|
||||||
@@ -73,6 +83,36 @@ class RunService:
|
|||||||
if rule is None:
|
if rule is None:
|
||||||
return None
|
return None
|
||||||
message = result.message
|
message = result.message
|
||||||
|
if update_counters and result.unavailable:
|
||||||
|
if rule.waiting_since is not None:
|
||||||
|
# wartet bereits – nur den Grund aktualisieren, kein neuer Eintrag
|
||||||
|
rule.waiting_reason = result.message
|
||||||
|
event(log, "rule.still_waiting", level=10, rule=rule.name)
|
||||||
|
return None
|
||||||
|
rule.waiting_since = finished
|
||||||
|
rule.waiting_reason = result.message
|
||||||
|
rule.last_run_at = finished
|
||||||
|
message = f"Pausiert, bis die Anbindung zurück ist: {result.message}"
|
||||||
|
update_counters = False # „Fehler in Folge“ nicht hochzählen
|
||||||
|
event(log, "rule.waiting", level=30, rule=rule.name, reason=result.message or "")
|
||||||
|
elif update_counters and result.waiting_on:
|
||||||
|
# Ein Ziel fehlt, die übrigen wurden beliefert: Lauf normal
|
||||||
|
# protokollieren, Regel aber als wartend kennzeichnen (Statusseite, Kuma).
|
||||||
|
if rule.waiting_since is None:
|
||||||
|
rule.waiting_since = finished
|
||||||
|
event(log, "rule.waiting", level=30, rule=rule.name, reason=message or "")
|
||||||
|
rule.waiting_reason = "; ".join(result.waiting_on)
|
||||||
|
elif (
|
||||||
|
update_counters
|
||||||
|
and rule.waiting_since is not None
|
||||||
|
and result.status is not RunStatus.SKIPPED
|
||||||
|
):
|
||||||
|
waited = _human(finished - rule.waiting_since)
|
||||||
|
rule.waiting_since = None
|
||||||
|
rule.waiting_reason = None
|
||||||
|
note = f"Fortgesetzt nach {waited} Wartezeit"
|
||||||
|
message = f"{note}; {message}" if message else note
|
||||||
|
event(log, "rule.resumed", rule=rule.name, waited=waited)
|
||||||
items = result.items
|
items = result.items
|
||||||
if len(items) > MAX_ITEMS_PER_RUN:
|
if len(items) > MAX_ITEMS_PER_RUN:
|
||||||
extra = len(items) - MAX_ITEMS_PER_RUN
|
extra = len(items) - MAX_ITEMS_PER_RUN
|
||||||
@@ -153,6 +193,15 @@ class RunService:
|
|||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def _human(delta: timedelta) -> str:
|
||||||
|
seconds = int(delta.total_seconds())
|
||||||
|
if seconds < 90:
|
||||||
|
return f"{seconds} s"
|
||||||
|
if seconds < 5400:
|
||||||
|
return f"{round(seconds / 60)} min"
|
||||||
|
return f"{seconds / 3600:.1f} h".replace(".", ",")
|
||||||
|
|
||||||
|
|
||||||
def cleanup(session_factory: sessionmaker[Session], now: datetime | None = None) -> dict[str, int]:
|
def cleanup(session_factory: sessionmaker[Session], now: datetime | None = None) -> dict[str, int]:
|
||||||
"""Protokolle älter als die Aufbewahrungsfrist löschen (Items per CASCADE),
|
"""Protokolle älter als die Aufbewahrungsfrist löschen (Items per CASCADE),
|
||||||
dazu Fehlerzähler, die seitdem nicht mehr angefasst wurden."""
|
dazu Fehlerzähler, die seitdem nicht mehr angefasst wurden."""
|
||||||
|
|||||||
@@ -6,7 +6,8 @@
|
|||||||
* Die Läufe selbst sind blockierend und laufen in einem eigenen Thread-Pool,
|
* Die Läufe selbst sind blockierend und laufen in einem eigenen Thread-Pool,
|
||||||
nie im Event-Loop. Überlappung regelt der Lock im :class:`RuleRunner`, der den
|
nie im Event-Loop. Überlappung regelt der Lock im :class:`RuleRunner`, der den
|
||||||
übersprungenen Termin auch protokolliert – deshalb ``max_instances`` > 1.
|
übersprungenen Termin auch protokolliert – deshalb ``max_instances`` > 1.
|
||||||
* Herzschlag-Job für ``/healthz`` und stündliches Aufräumen der Protokolle.
|
* Herzschlag-Job für ``/healthz``, stündliches Aufräumen der Protokolle und alle
|
||||||
|
5 Minuten die Frage, ob die tägliche Datensicherung fällig ist.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -33,6 +34,8 @@ MAX_START_OFFSET = 10.0
|
|||||||
HEARTBEAT_SECONDS = 30
|
HEARTBEAT_SECONDS = 30
|
||||||
CLEANUP_SECONDS = 3600
|
CLEANUP_SECONDS = 3600
|
||||||
RETRY_SECONDS = 60
|
RETRY_SECONDS = 60
|
||||||
|
PUSH_SECONDS = 60
|
||||||
|
BACKUP_CHECK_SECONDS = 300
|
||||||
|
|
||||||
|
|
||||||
def _job_id(rule_id: int) -> str:
|
def _job_id(rule_id: int) -> str:
|
||||||
@@ -46,12 +49,16 @@ class RuleScheduler:
|
|||||||
service: RunService,
|
service: RunService,
|
||||||
*,
|
*,
|
||||||
retry_mounts: Callable[[], object] | None = None,
|
retry_mounts: Callable[[], object] | None = None,
|
||||||
|
push_health: Callable[[], object] | None = None,
|
||||||
|
backup_tick: Callable[[], object] | None = None,
|
||||||
rng: random.Random | None = None,
|
rng: random.Random | None = None,
|
||||||
workers: int = 8,
|
workers: int = 8,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.session_factory = session_factory
|
self.session_factory = session_factory
|
||||||
self.service = service
|
self.service = service
|
||||||
self._retry_mounts = retry_mounts
|
self._retry_mounts = retry_mounts
|
||||||
|
self._push_health = push_health
|
||||||
|
self._backup_tick = backup_tick
|
||||||
self._rng = rng or random.Random() # noqa: S311 – nur Startversatz
|
self._rng = rng or random.Random() # noqa: S311 – nur Startversatz
|
||||||
self._executor = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="nm-run")
|
self._executor = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="nm-run")
|
||||||
self.scheduler = AsyncIOScheduler(
|
self.scheduler = AsyncIOScheduler(
|
||||||
@@ -78,6 +85,18 @@ class RuleScheduler:
|
|||||||
self.scheduler.add_job(
|
self.scheduler.add_job(
|
||||||
self._retry, IntervalTrigger(seconds=RETRY_SECONDS), id="mount-retry"
|
self._retry, IntervalTrigger(seconds=RETRY_SECONDS), id="mount-retry"
|
||||||
)
|
)
|
||||||
|
if self._push_health is not None:
|
||||||
|
self.scheduler.add_job(
|
||||||
|
self._push, IntervalTrigger(seconds=PUSH_SECONDS), id="kuma-push"
|
||||||
|
)
|
||||||
|
if self._backup_tick is not None:
|
||||||
|
self.scheduler.add_job(
|
||||||
|
self._backup,
|
||||||
|
IntervalTrigger(
|
||||||
|
seconds=BACKUP_CHECK_SECONDS, start_date=utcnow() + timedelta(minutes=2)
|
||||||
|
),
|
||||||
|
id="backup",
|
||||||
|
)
|
||||||
self.sync()
|
self.sync()
|
||||||
event(log, "scheduler.started", rules=len(self._intervals))
|
event(log, "scheduler.started", rules=len(self._intervals))
|
||||||
|
|
||||||
@@ -154,6 +173,16 @@ class RuleScheduler:
|
|||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
await loop.run_in_executor(self._executor, self._retry_mounts)
|
await loop.run_in_executor(self._executor, self._retry_mounts)
|
||||||
|
|
||||||
|
async def _push(self) -> None:
|
||||||
|
if self._push_health is not None:
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
await loop.run_in_executor(self._executor, self._push_health)
|
||||||
|
|
||||||
|
async def _backup(self) -> None:
|
||||||
|
if self._backup_tick is not None:
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
await loop.run_in_executor(self._executor, self._backup_tick)
|
||||||
|
|
||||||
async def _cleanup(self) -> None:
|
async def _cleanup(self) -> None:
|
||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
await loop.run_in_executor(self._executor, cleanup, self.session_factory)
|
await loop.run_in_executor(self._executor, cleanup, self.session_factory)
|
||||||
|
|||||||
@@ -34,6 +34,31 @@ class AppSettings:
|
|||||||
notify_recovery: bool = True
|
notify_recovery: bool = True
|
||||||
webhook_template: str = DEFAULT_TEMPLATE
|
webhook_template: str = DEFAULT_TEMPLATE
|
||||||
webhook_content_type: str = "application/json"
|
webhook_content_type: str = "application/json"
|
||||||
|
kuma_push_url: str = ""
|
||||||
|
# Wartungsmodus: geplante Läufe pausiert, kein Alarm für Regeln
|
||||||
|
maintenance: bool = False
|
||||||
|
# E-Mail (SMTP); Passwort Fernet-verschlüsselt
|
||||||
|
email_enabled: bool = False
|
||||||
|
smtp_host: str = ""
|
||||||
|
smtp_port: int = 587
|
||||||
|
smtp_security: str = "starttls"
|
||||||
|
smtp_user: str = ""
|
||||||
|
smtp_password_enc: str = ""
|
||||||
|
smtp_from: str = ""
|
||||||
|
smtp_to: str = ""
|
||||||
|
# Anmeldung: Netze ohne Passwort (trusted_networks), Netze mit Passwort
|
||||||
|
# (allowed_networks, leer = alle übrigen); alle anderen Netze werden abgewiesen.
|
||||||
|
# auth_mode "off" = ohne Anmeldung von überall (aus 0.5.0, weiterhin gültig).
|
||||||
|
auth_mode: str = "always"
|
||||||
|
trusted_networks: str = ""
|
||||||
|
allowed_networks: str = ""
|
||||||
|
# Datensicherung (backup.py); Passphrase Fernet-verschlüsselt
|
||||||
|
backup_enabled: bool = False
|
||||||
|
backup_mount_id: int = 0 # 0 = kein Ziel
|
||||||
|
backup_path: str = ""
|
||||||
|
backup_time: str = "03:30"
|
||||||
|
backup_keep: int = 14
|
||||||
|
backup_passphrase_enc: str = ""
|
||||||
|
|
||||||
|
|
||||||
_PREFIX = "app."
|
_PREFIX = "app."
|
||||||
|
|||||||
@@ -23,6 +23,9 @@ from netmover.models import (
|
|||||||
PatternType,
|
PatternType,
|
||||||
Rule,
|
Rule,
|
||||||
RuleAction,
|
RuleAction,
|
||||||
|
RuleTarget,
|
||||||
|
TargetKind,
|
||||||
|
TargetLayout,
|
||||||
)
|
)
|
||||||
from netmover.mounting.manager import MountManager
|
from netmover.mounting.manager import MountManager
|
||||||
from netmover.settings_store import AppSettings, NotifyMode
|
from netmover.settings_store import AppSettings, NotifyMode
|
||||||
@@ -99,6 +102,16 @@ def export_config(s: Session, settings: AppSettings) -> dict[str, Any]:
|
|||||||
**{f: _plain(getattr(r, f)) for f in RULE_FIELDS},
|
**{f: _plain(getattr(r, f)) for f in RULE_FIELDS},
|
||||||
"src_mount": names[r.src_mount_id],
|
"src_mount": names[r.src_mount_id],
|
||||||
"dst_mount": names.get(r.dst_mount_id) if r.dst_mount_id else None,
|
"dst_mount": names.get(r.dst_mount_id) if r.dst_mount_id else None,
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"name": t.name,
|
||||||
|
"kind": t.kind.value,
|
||||||
|
"mount": names.get(t.mount_id) if t.mount_id else None,
|
||||||
|
"path": t.path,
|
||||||
|
"layout": t.layout.value,
|
||||||
|
}
|
||||||
|
for t in r.targets
|
||||||
|
],
|
||||||
}
|
}
|
||||||
for r in s.scalars(select(Rule).order_by(Rule.name))
|
for r in s.scalars(select(Rule).order_by(Rule.name))
|
||||||
],
|
],
|
||||||
@@ -172,6 +185,7 @@ def import_config(s: Session, mounts: MountManager, data: dict[str, Any]) -> Imp
|
|||||||
r.pattern_type = PatternType(r.pattern_type)
|
r.pattern_type = PatternType(r.pattern_type)
|
||||||
r.action = RuleAction(r.action)
|
r.action = RuleAction(r.action)
|
||||||
r.on_collision = CollisionPolicy(r.on_collision)
|
r.on_collision = CollisionPolicy(r.on_collision)
|
||||||
|
r.targets = _import_targets(raw, ids)
|
||||||
r.src_subpath = safe_relpath("src_subpath", r.src_subpath or "")
|
r.src_subpath = safe_relpath("src_subpath", r.src_subpath or "")
|
||||||
if int(r.interval_seconds) < 10:
|
if int(r.interval_seconds) < 10:
|
||||||
raise ValueError("Intervall < 10 s")
|
raise ValueError("Intervall < 10 s")
|
||||||
@@ -189,6 +203,35 @@ def import_config(s: Session, mounts: MountManager, data: dict[str, Any]) -> Imp
|
|||||||
return report
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def _import_targets(raw: dict[str, Any], ids: dict[str, int]) -> list[RuleTarget]:
|
||||||
|
"""Weitere Ziele; Exporte vor 0.11.0 kennen nur ein Archiv (``archive_*``)."""
|
||||||
|
items = raw.get("targets")
|
||||||
|
if items is None and raw.get("archive_kind") not in (None, "none"):
|
||||||
|
items = [
|
||||||
|
{
|
||||||
|
"name": "Archiv",
|
||||||
|
"kind": raw["archive_kind"],
|
||||||
|
"mount": raw.get("archive_mount"),
|
||||||
|
"path": raw.get("archive_path") or "",
|
||||||
|
"layout": raw.get("archive_layout") or "flat",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
targets = []
|
||||||
|
for pos, t in enumerate(items or []):
|
||||||
|
kind = TargetKind(t["kind"])
|
||||||
|
targets.append(
|
||||||
|
RuleTarget(
|
||||||
|
position=pos,
|
||||||
|
name=str(t.get("name") or f"Ziel {pos + 2}")[:100],
|
||||||
|
kind=kind,
|
||||||
|
mount_id=ids[t["mount"]] if kind is TargetKind.MOUNT else None,
|
||||||
|
path=str(t.get("path") or ""),
|
||||||
|
layout=TargetLayout(t.get("layout") or "flat"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return targets
|
||||||
|
|
||||||
|
|
||||||
def apply_imported_settings(current: AppSettings, values: dict[str, Any]) -> AppSettings:
|
def apply_imported_settings(current: AppSettings, values: dict[str, Any]) -> AppSettings:
|
||||||
"""Nach dem Commit des Imports aufrufen (eigene Transaktion)."""
|
"""Nach dem Commit des Imports aufrufen (eigene Transaktion)."""
|
||||||
for key, value in values.items():
|
for key, value in values.items():
|
||||||
|
|||||||
+709
-28
@@ -12,17 +12,22 @@ import asyncio
|
|||||||
import dataclasses
|
import dataclasses
|
||||||
import json
|
import json
|
||||||
import math
|
import math
|
||||||
|
import re
|
||||||
from collections.abc import AsyncIterator, Awaitable, Callable
|
from collections.abc import AsyncIterator, Awaitable, Callable
|
||||||
from contextlib import asynccontextmanager
|
from contextlib import asynccontextmanager
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
from functools import lru_cache
|
||||||
from importlib.resources import files
|
from importlib.resources import files
|
||||||
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
import markdown
|
||||||
from fastapi import APIRouter, Depends, FastAPI, Request
|
from fastapi import APIRouter, Depends, FastAPI, Request
|
||||||
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse, Response
|
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse, RedirectResponse, Response
|
||||||
from fastapi.staticfiles import StaticFiles
|
from fastapi.staticfiles import StaticFiles
|
||||||
from fastapi.templating import Jinja2Templates
|
from fastapi.templating import Jinja2Templates
|
||||||
|
from markupsafe import escape
|
||||||
from sqlalchemy import func, or_, select
|
from sqlalchemy import func, or_, select
|
||||||
from starlette.concurrency import run_in_threadpool
|
from starlette.concurrency import run_in_threadpool
|
||||||
from starlette.datastructures import UploadFile
|
from starlette.datastructures import UploadFile
|
||||||
@@ -30,24 +35,44 @@ from starlette.middleware.sessions import SessionMiddleware
|
|||||||
|
|
||||||
from netmover import __version__
|
from netmover import __version__
|
||||||
from netmover.admin import (
|
from netmover.admin import (
|
||||||
|
MAX_TARGETS,
|
||||||
AdminError,
|
AdminError,
|
||||||
MountInput,
|
MountInput,
|
||||||
RuleInput,
|
RuleInput,
|
||||||
|
copy_name,
|
||||||
credential_for_test,
|
credential_for_test,
|
||||||
delete_mount,
|
delete_mount,
|
||||||
mount_input_from_model,
|
mount_input_from_model,
|
||||||
parse_mount_form,
|
parse_mount_form,
|
||||||
parse_rule_form,
|
parse_rule_form,
|
||||||
|
rule_copy_name,
|
||||||
rule_input_from_model,
|
rule_input_from_model,
|
||||||
save_mount,
|
save_mount,
|
||||||
save_rule,
|
save_rule,
|
||||||
)
|
)
|
||||||
from netmover.app_state import NetMover
|
from netmover.app_state import NetMover
|
||||||
|
from netmover.backup import load_state as load_backup_state
|
||||||
|
from netmover.backup import parse_time as parse_backup_time
|
||||||
|
from netmover.backup import passphrase_problem
|
||||||
from netmover.config import Settings, get_settings
|
from netmover.config import Settings, get_settings
|
||||||
|
from netmover.credentials import (
|
||||||
|
create_password,
|
||||||
|
delete_credential,
|
||||||
|
generate_key,
|
||||||
|
import_key,
|
||||||
|
list_keys,
|
||||||
|
list_passwords,
|
||||||
|
rename_credential,
|
||||||
|
update_password,
|
||||||
|
)
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
|
from netmover.health import metrics as health_metrics
|
||||||
|
from netmover.kuma import push as kuma_push
|
||||||
from netmover.logging_setup import event, get_logger
|
from netmover.logging_setup import event, get_logger
|
||||||
|
from netmover.mail import SECURITY_MODES, MailConfigError, parse_recipients, valid_address
|
||||||
from netmover.models import (
|
from netmover.models import (
|
||||||
CollisionPolicy,
|
CollisionPolicy,
|
||||||
|
Credential,
|
||||||
CredentialKind,
|
CredentialKind,
|
||||||
Mount,
|
Mount,
|
||||||
MountState,
|
MountState,
|
||||||
@@ -55,6 +80,7 @@ from netmover.models import (
|
|||||||
PatternType,
|
PatternType,
|
||||||
Rule,
|
Rule,
|
||||||
RuleAction,
|
RuleAction,
|
||||||
|
RuleTarget,
|
||||||
Run,
|
Run,
|
||||||
RunItem,
|
RunItem,
|
||||||
RunStatus,
|
RunStatus,
|
||||||
@@ -65,6 +91,7 @@ from netmover.mounting.report import CheckReport
|
|||||||
from netmover.notify import Notification
|
from netmover.notify import Notification
|
||||||
from netmover.settings_store import AppSettings, NotifyMode, load_settings, save_settings
|
from netmover.settings_store import AppSettings, NotifyMode, load_settings, save_settings
|
||||||
from netmover.transfer import apply_imported_settings, export_config, import_config
|
from netmover.transfer import apply_imported_settings, export_config, import_config
|
||||||
|
from netmover.validation import ValidationError, safe_relpath
|
||||||
from netmover.web import auth
|
from netmover.web import auth
|
||||||
|
|
||||||
log = get_logger(__name__)
|
log = get_logger(__name__)
|
||||||
@@ -136,6 +163,7 @@ def _duration(start: datetime | None, end: datetime | None) -> str:
|
|||||||
|
|
||||||
TEMPLATES = Jinja2Templates(directory=str(files("netmover.web") / "templates"))
|
TEMPLATES = Jinja2Templates(directory=str(files("netmover.web") / "templates"))
|
||||||
TEMPLATES.env.filters.update(ago=_ago, localdt=_localdt, bytes=_bytes)
|
TEMPLATES.env.filters.update(ago=_ago, localdt=_localdt, bytes=_bytes)
|
||||||
|
TEMPLATES.env.globals["max_targets"] = MAX_TARGETS
|
||||||
TEMPLATES.env.globals.update(
|
TEMPLATES.env.globals.update(
|
||||||
version=__version__,
|
version=__version__,
|
||||||
duration=_duration,
|
duration=_duration,
|
||||||
@@ -158,15 +186,73 @@ def flash(request: Request, message: str, kind: str = "ok") -> None:
|
|||||||
request.session["flash"] = [*request.session.get("flash", []), [kind, message]]
|
request.session["flash"] = [*request.session.get("flash", []), [kind, message]]
|
||||||
|
|
||||||
|
|
||||||
|
# Hilfe-Abschnitt passend zur Seite (Link „Hilfe zu dieser Seite“)
|
||||||
|
HELP_ANCHORS = {
|
||||||
|
"status": "status",
|
||||||
|
"mounts": "anbindungen",
|
||||||
|
"keys": "zugangsdaten",
|
||||||
|
"rules": "regeln",
|
||||||
|
"log": "protokoll",
|
||||||
|
"settings": "einstellungen",
|
||||||
|
"backup": "sicherung",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class NavState:
|
||||||
|
healthy: bool
|
||||||
|
problems: int
|
||||||
|
mounts: int
|
||||||
|
mounts_bad: int
|
||||||
|
keys: int
|
||||||
|
rules: int
|
||||||
|
rules_active: int
|
||||||
|
maintenance: bool = False
|
||||||
|
backup_ok: bool = True
|
||||||
|
|
||||||
|
|
||||||
|
def _nav_state(request: Request) -> NavState | None:
|
||||||
|
if not _authorized(request):
|
||||||
|
return None
|
||||||
|
state = nm(request)
|
||||||
|
health = state.health()
|
||||||
|
with state.session_factory() as s:
|
||||||
|
mounts = s.execute(select(Mount.state, Mount.enabled)).all()
|
||||||
|
rules = s.execute(select(Rule.enabled)).all()
|
||||||
|
keys = s.scalar(
|
||||||
|
select(func.count()).select_from(Credential).where(Credential.managed.is_(True))
|
||||||
|
)
|
||||||
|
return NavState(
|
||||||
|
healthy=health.ok,
|
||||||
|
problems=len(health.problems),
|
||||||
|
mounts=len(mounts),
|
||||||
|
mounts_bad=sum(
|
||||||
|
1 for st, en in mounts if en and st in (MountState.ERROR, MountState.DEGRADED)
|
||||||
|
),
|
||||||
|
keys=keys or 0,
|
||||||
|
rules=len(rules),
|
||||||
|
rules_active=sum(1 for (en,) in rules if en),
|
||||||
|
maintenance=health.maintenance,
|
||||||
|
backup_ok=(c := health.find("backup")) is None or c.ok,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def render(request: Request, name: str, status_code: int = 200, **ctx: Any) -> HTMLResponse:
|
def render(request: Request, name: str, status_code: int = 200, **ctx: Any) -> HTMLResponse:
|
||||||
flashes = request.session.pop("flash", []) if not name.startswith("partials/") else []
|
partial = name.startswith("partials/")
|
||||||
|
flashes = request.session.pop("flash", []) if not partial else []
|
||||||
|
active = ctx.get("active", "")
|
||||||
|
if not partial:
|
||||||
|
ctx.setdefault("nav", _nav_state(request))
|
||||||
|
ctx.setdefault("help_anchor", HELP_ANCHORS.get(active))
|
||||||
return TEMPLATES.TemplateResponse(
|
return TEMPLATES.TemplateResponse(
|
||||||
request,
|
request,
|
||||||
name,
|
name,
|
||||||
{
|
{
|
||||||
"csrf": auth.csrf_token(request),
|
"csrf": auth.csrf_token(request),
|
||||||
"flashes": flashes,
|
"flashes": flashes,
|
||||||
"logged_in": auth.is_logged_in(request),
|
"logged_in": _authorized(request),
|
||||||
|
"session_login": auth.is_logged_in(request),
|
||||||
|
"auth_bypass": getattr(request.state, "auth_bypass", None),
|
||||||
"active": ctx.pop("active", ""),
|
"active": ctx.pop("active", ""),
|
||||||
**ctx,
|
**ctx,
|
||||||
},
|
},
|
||||||
@@ -192,13 +278,22 @@ def _form_dict(form: Any) -> dict[str, str]:
|
|||||||
|
|
||||||
async def guard(request: Request) -> None:
|
async def guard(request: Request) -> None:
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
|
# Ein Passwort ist immer Pflicht – sonst ließe sich die Anmeldung nie wieder einschalten.
|
||||||
if not auth.password_is_set(state.session_factory):
|
if not auth.password_is_set(state.session_factory):
|
||||||
raise auth.LoginRequired(setup=True)
|
raise auth.LoginRequired(setup=True)
|
||||||
if not auth.is_logged_in(request):
|
cfg = load_settings(state.session_factory)
|
||||||
|
bypass = auth.login_bypass(request, cfg.auth_mode, cfg.trusted_networks, cfg.allowed_networks)
|
||||||
|
if bypass:
|
||||||
|
request.state.auth_bypass = bypass
|
||||||
|
elif not auth.is_logged_in(request):
|
||||||
raise auth.LoginRequired()
|
raise auth.LoginRequired()
|
||||||
await auth.check_csrf(request)
|
await auth.check_csrf(request)
|
||||||
|
|
||||||
|
|
||||||
|
def _authorized(request: Request) -> bool:
|
||||||
|
return auth.is_logged_in(request) or bool(getattr(request.state, "auth_bypass", None))
|
||||||
|
|
||||||
|
|
||||||
protected = APIRouter(dependencies=[Depends(guard)])
|
protected = APIRouter(dependencies=[Depends(guard)])
|
||||||
public = APIRouter()
|
public = APIRouter()
|
||||||
|
|
||||||
@@ -208,11 +303,65 @@ public = APIRouter()
|
|||||||
|
|
||||||
@public.get("/healthz")
|
@public.get("/healthz")
|
||||||
async def healthz(request: Request) -> JSONResponse:
|
async def healthz(request: Request) -> JSONResponse:
|
||||||
health = await run_in_threadpool(nm(request).health)
|
"""Für Uptime Kuma & Co.: 200 = alles in Ordnung, 503 = mindestens ein Problem.
|
||||||
body = {"status": "ok" if health.ok else "error", "problems": health.problems}
|
Ohne Anmeldung erreichbar; enthält Namen und Fehlermeldungen, keine Geheimnisse."""
|
||||||
|
state = nm(request)
|
||||||
|
health = await run_in_threadpool(state.health)
|
||||||
|
beat = state.scheduler.last_heartbeat
|
||||||
|
body = {
|
||||||
|
"status": "ok" if health.ok else "error",
|
||||||
|
"version": __version__,
|
||||||
|
"problems": health.problems,
|
||||||
|
"scheduler": {
|
||||||
|
"running": state.scheduler.running,
|
||||||
|
"last_heartbeat": beat.isoformat() if beat else None,
|
||||||
|
},
|
||||||
|
"checks": [c.as_dict() for c in health.checks],
|
||||||
|
"metrics": await run_in_threadpool(health_metrics, state.session_factory),
|
||||||
|
}
|
||||||
return JSONResponse(body, status_code=200 if health.ok else 503)
|
return JSONResponse(body, status_code=200 if health.ok else 503)
|
||||||
|
|
||||||
|
|
||||||
|
async def _single_check(
|
||||||
|
request: Request, kind: str, extra: dict[str, Any] | None = None, **key: Any
|
||||||
|
) -> JSONResponse:
|
||||||
|
health = await run_in_threadpool(nm(request).health)
|
||||||
|
check = health.find(kind, **key)
|
||||||
|
if check is None:
|
||||||
|
return JSONResponse({"status": "unknown", "message": "nicht gefunden"}, status_code=404)
|
||||||
|
body = {"status": "ok" if check.ok else "error", "name": check.name, "message": check.message}
|
||||||
|
body.update(extra or {})
|
||||||
|
return JSONResponse(body, status_code=200 if check.ok else 503)
|
||||||
|
|
||||||
|
|
||||||
|
@public.get("/healthz/mount/{slug}")
|
||||||
|
async def healthz_mount(request: Request, slug: str) -> JSONResponse:
|
||||||
|
"""Eigener Kuma-Monitor pro Anbindung."""
|
||||||
|
return await _single_check(request, "mount", slug=slug)
|
||||||
|
|
||||||
|
|
||||||
|
@public.get("/healthz/rule/{rule_id}")
|
||||||
|
async def healthz_rule(request: Request, rule_id: int) -> JSONResponse:
|
||||||
|
"""Eigener Kuma-Monitor pro Regel – mit Kennzahlen der letzten 24 Stunden."""
|
||||||
|
all_metrics = await run_in_threadpool(health_metrics, nm(request).session_factory)
|
||||||
|
rules = all_metrics["rules"]
|
||||||
|
rule_metrics = rules.get(str(rule_id)) if isinstance(rules, dict) else None
|
||||||
|
return await _single_check(request, "rule", {"metrics": rule_metrics}, id=rule_id)
|
||||||
|
|
||||||
|
|
||||||
|
@public.get("/healthz/backup")
|
||||||
|
async def healthz_backup(request: Request) -> JSONResponse:
|
||||||
|
"""Eigener Kuma-Monitor für die Datensicherung."""
|
||||||
|
st = await run_in_threadpool(load_backup_state, nm(request).session_factory)
|
||||||
|
extra = {
|
||||||
|
"last_success": st.last_success.isoformat() if st.last_success else None,
|
||||||
|
"last_file": st.last_file or None,
|
||||||
|
"last_size": st.last_size,
|
||||||
|
"failures": st.failures,
|
||||||
|
}
|
||||||
|
return await _single_check(request, "backup", extra)
|
||||||
|
|
||||||
|
|
||||||
@public.get("/setup", response_class=HTMLResponse)
|
@public.get("/setup", response_class=HTMLResponse)
|
||||||
async def setup_form(request: Request) -> Response:
|
async def setup_form(request: Request) -> Response:
|
||||||
if auth.password_is_set(nm(request).session_factory):
|
if auth.password_is_set(nm(request).session_factory):
|
||||||
@@ -244,8 +393,12 @@ async def setup_submit(request: Request) -> Response:
|
|||||||
|
|
||||||
@public.get("/login", response_class=HTMLResponse)
|
@public.get("/login", response_class=HTMLResponse)
|
||||||
async def login_form(request: Request) -> Response:
|
async def login_form(request: Request) -> Response:
|
||||||
if not auth.password_is_set(nm(request).session_factory):
|
state = nm(request)
|
||||||
|
if not auth.password_is_set(state.session_factory):
|
||||||
return redirect("/setup")
|
return redirect("/setup")
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
if auth.login_bypass(request, cfg.auth_mode, cfg.trusted_networks, cfg.allowed_networks):
|
||||||
|
return redirect("/")
|
||||||
return render(request, "login.html")
|
return render(request, "login.html")
|
||||||
|
|
||||||
|
|
||||||
@@ -302,13 +455,18 @@ class RuleView:
|
|||||||
target: str
|
target: str
|
||||||
|
|
||||||
|
|
||||||
|
def _extra_target(t: RuleTarget) -> str:
|
||||||
|
place = t.path if t.kind.value == "local" or t.mount is None else f"{t.mount.name}:/{t.path}"
|
||||||
|
return f"{t.name} {place}" + (" (nach Datum)" if t.layout.value == "date" else "")
|
||||||
|
|
||||||
|
|
||||||
def _rule_target(r: Rule) -> str:
|
def _rule_target(r: Rule) -> str:
|
||||||
src = f"{r.src_mount.name}:/{r.src_subpath}" if r.src_subpath else f"{r.src_mount.name}:/"
|
src = f"{r.src_mount.name}:/{r.src_subpath}" if r.src_subpath else f"{r.src_mount.name}:/"
|
||||||
if r.dst_mount is None:
|
if r.dst_mount is None:
|
||||||
dst = r.dst_path
|
dst = r.dst_path
|
||||||
else:
|
else:
|
||||||
dst = f"{r.dst_mount.name}:/{r.dst_path}" if r.dst_path else f"{r.dst_mount.name}:/"
|
dst = f"{r.dst_mount.name}:/{r.dst_path}" if r.dst_path else f"{r.dst_mount.name}:/"
|
||||||
return f"{src} → {dst}"
|
return f"{src} → {dst}" + "".join(f" + {_extra_target(t)}" for t in r.targets)
|
||||||
|
|
||||||
|
|
||||||
async def _status_context(request: Request) -> dict[str, Any]:
|
async def _status_context(request: Request) -> dict[str, Any]:
|
||||||
@@ -367,6 +525,23 @@ async def status_partial(request: Request) -> Response:
|
|||||||
return render(request, "partials/status_body.html", **await _status_context(request))
|
return render(request, "partials/status_body.html", **await _status_context(request))
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/maintenance")
|
||||||
|
async def toggle_maintenance(request: Request) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
cfg.maintenance = not cfg.maintenance
|
||||||
|
save_settings(state.session_factory, cfg)
|
||||||
|
event(log, "maintenance", enabled=cfg.maintenance)
|
||||||
|
flash(
|
||||||
|
request,
|
||||||
|
"Wartungsmodus eingeschaltet – geplante Läufe sind pausiert."
|
||||||
|
if cfg.maintenance
|
||||||
|
else "Wartungsmodus beendet – Regeln laufen wieder nach Plan.",
|
||||||
|
"info",
|
||||||
|
)
|
||||||
|
return redirect("/")
|
||||||
|
|
||||||
|
|
||||||
@protected.post("/rules/{rule_id}/run")
|
@protected.post("/rules/{rule_id}/run")
|
||||||
async def rule_run_now(request: Request, rule_id: int) -> Response:
|
async def rule_run_now(request: Request, rule_id: int) -> Response:
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
@@ -392,10 +567,21 @@ async def mounts_page(request: Request) -> Response:
|
|||||||
pubkeys = {
|
pubkeys = {
|
||||||
m.id: m.credential.public_key
|
m.id: m.credential.public_key
|
||||||
for m in mounts
|
for m in mounts
|
||||||
if m.credential is not None and m.credential.public_key
|
if m.credential is not None and m.credential.public_key and not m.credential.managed
|
||||||
}
|
}
|
||||||
|
key_names = {
|
||||||
|
m.id: m.credential.name
|
||||||
|
for m in mounts
|
||||||
|
if m.credential is not None and m.credential.managed
|
||||||
|
} # Name gespeicherter Zugangsdaten (Passwort oder Schlüssel)
|
||||||
return render(
|
return render(
|
||||||
request, "mounts.html", active="mounts", mounts=mounts, reports=reports, pubkeys=pubkeys
|
request,
|
||||||
|
"mounts.html",
|
||||||
|
active="mounts",
|
||||||
|
mounts=mounts,
|
||||||
|
reports=reports,
|
||||||
|
pubkeys=pubkeys,
|
||||||
|
key_names=key_names,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -409,14 +595,37 @@ def _report_data(raw: str | None) -> dict[str, Any]:
|
|||||||
return data
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def _password_options(request: Request) -> list[tuple[str, str]]:
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
stored = list_passwords(s)
|
||||||
|
return [("own", "eigene Angaben für diese Anbindung")] + [
|
||||||
|
(str(p.id), f"{p.name} ({p.domain + chr(92) if p.domain else ''}{p.username})")
|
||||||
|
for p in stored
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _key_options(request: Request) -> list[tuple[str, str]]:
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
keys = list_keys(s)
|
||||||
|
return [("new", "neuer Schlüssel nur für diese Anbindung")] + [
|
||||||
|
(str(k.id), f"{k.name} ({k.key_type.removeprefix('ssh-')})") for k in keys
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def _mount_form(
|
def _mount_form(
|
||||||
request: Request,
|
request: Request,
|
||||||
data: MountInput,
|
data: MountInput,
|
||||||
errors: dict[str, str],
|
errors: dict[str, str],
|
||||||
mount: Mount | None = None,
|
mount: Mount | None = None,
|
||||||
status_code: int = 200,
|
status_code: int = 200,
|
||||||
|
copy_from: Mount | None = None,
|
||||||
) -> Response:
|
) -> Response:
|
||||||
pub = mount.credential.public_key if mount is not None and mount.credential else None
|
owner = mount or copy_from
|
||||||
|
pub = (
|
||||||
|
owner.credential.public_key
|
||||||
|
if owner is not None and owner.credential and not owner.credential.managed
|
||||||
|
else None
|
||||||
|
)
|
||||||
return render(
|
return render(
|
||||||
request,
|
request,
|
||||||
"mount_form.html",
|
"mount_form.html",
|
||||||
@@ -426,7 +635,12 @@ def _mount_form(
|
|||||||
errors=errors,
|
errors=errors,
|
||||||
mount=mount,
|
mount=mount,
|
||||||
public_key=pub,
|
public_key=pub,
|
||||||
has_secret=bool(mount is not None and mount.credential is not None),
|
has_secret=bool(
|
||||||
|
owner is not None and owner.credential is not None and not owner.credential.managed
|
||||||
|
),
|
||||||
|
copy_from=copy_from,
|
||||||
|
key_options=_key_options(request),
|
||||||
|
password_options=_password_options(request),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -444,16 +658,33 @@ async def mount_edit(request: Request, mount_id: int) -> Response:
|
|||||||
return _mount_form(request, mount_input_from_model(m), {}, m)
|
return _mount_form(request, mount_input_from_model(m), {}, m)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/mounts/{source_id}/copy", response_class=HTMLResponse)
|
||||||
|
async def mount_copy(request: Request, source_id: int) -> Response:
|
||||||
|
"""Formular für eine neue Anbindung, vorausgefüllt aus einer bestehenden."""
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
src = s.get(Mount, source_id)
|
||||||
|
if src is None:
|
||||||
|
return redirect("/mounts")
|
||||||
|
data = mount_input_from_model(src)
|
||||||
|
data.name = copy_name(s, src.name)
|
||||||
|
return _mount_form(request, data, {}, copy_from=src)
|
||||||
|
|
||||||
|
|
||||||
@protected.post("/mounts/new", response_class=HTMLResponse)
|
@protected.post("/mounts/new", response_class=HTMLResponse)
|
||||||
@protected.post("/mounts/{mount_id}/edit", response_class=HTMLResponse)
|
@protected.post("/mounts/{mount_id}/edit", response_class=HTMLResponse)
|
||||||
async def mount_save(request: Request, mount_id: int | None = None) -> Response:
|
@protected.post("/mounts/{source_id}/copy", response_class=HTMLResponse)
|
||||||
|
async def mount_save(
|
||||||
|
request: Request, mount_id: int | None = None, source_id: int | None = None
|
||||||
|
) -> Response:
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
data, errors = parse_mount_form(_form_dict(await request.form()))
|
data, errors = parse_mount_form(_form_dict(await request.form()))
|
||||||
with session_scope(state.session_factory) as s:
|
with session_scope(state.session_factory) as s:
|
||||||
existing = s.get(Mount, mount_id) if mount_id is not None else None
|
copy_from = s.get(Mount, source_id) if source_id is not None else None
|
||||||
if not errors:
|
if not errors:
|
||||||
try:
|
try:
|
||||||
m, errors = save_mount(s, state.secrets, state.mounts, data, mount_id)
|
m, errors = save_mount(
|
||||||
|
s, state.secrets, state.mounts, data, mount_id, copy_from=copy_from
|
||||||
|
)
|
||||||
except AdminError as exc:
|
except AdminError as exc:
|
||||||
errors = {"_": str(exc)}
|
errors = {"_": str(exc)}
|
||||||
m = None
|
m = None
|
||||||
@@ -462,7 +693,10 @@ async def mount_save(request: Request, mount_id: int | None = None) -> Response:
|
|||||||
if errors or m is None:
|
if errors or m is None:
|
||||||
s.rollback()
|
s.rollback()
|
||||||
existing = s.get(Mount, mount_id) if mount_id is not None else None
|
existing = s.get(Mount, mount_id) if mount_id is not None else None
|
||||||
return _mount_form(request, data, errors, existing, status_code=400)
|
copy_from = s.get(Mount, source_id) if source_id is not None else None
|
||||||
|
return _mount_form(
|
||||||
|
request, data, errors, existing, status_code=400, copy_from=copy_from
|
||||||
|
)
|
||||||
saved_id, name = m.id, m.name
|
saved_id, name = m.id, m.name
|
||||||
is_sftp_key = m.type is MountType.SFTP and data.auth is CredentialKind.SSH_KEY
|
is_sftp_key = m.type is MountType.SFTP and data.auth is CredentialKind.SSH_KEY
|
||||||
flash(request, f"Anbindung „{name}“ gespeichert.")
|
flash(request, f"Anbindung „{name}“ gespeichert.")
|
||||||
@@ -484,12 +718,17 @@ async def mount_test_form(request: Request) -> Response:
|
|||||||
data, errors = parse_mount_form(form)
|
data, errors = parse_mount_form(form)
|
||||||
if errors:
|
if errors:
|
||||||
return render(request, "partials/check_result.html", errors=errors, report=None)
|
return render(request, "partials/check_result.html", errors=errors, report=None)
|
||||||
existing_id = int(form["mount_id"]) if form.get("mount_id", "").isdigit() else None
|
ref = form.get("mount_id") or form.get("copy_from") or ""
|
||||||
|
existing_id = int(ref) if ref.isdigit() else None
|
||||||
with state.session_factory() as s:
|
with state.session_factory() as s:
|
||||||
existing = s.get(Mount, existing_id) if existing_id else None
|
existing = s.get(Mount, existing_id) if existing_id else None
|
||||||
cred = credential_for_test(data, existing, state.secrets)
|
cred = credential_for_test(data, existing, state.secrets, s)
|
||||||
cfg = data.config()
|
cfg = data.config()
|
||||||
if data.type is MountType.SFTP and existing is not None:
|
if (
|
||||||
|
data.type is MountType.SFTP
|
||||||
|
and existing is not None
|
||||||
|
and (existing.server, existing.port) == (data.server, data.port)
|
||||||
|
):
|
||||||
cfg = dataclasses.replace(cfg, host_key=existing.host_key)
|
cfg = dataclasses.replace(cfg, host_key=existing.host_key)
|
||||||
report = await run_in_threadpool(state.mounts.check, cfg, cred, None)
|
report = await run_in_threadpool(state.mounts.check, cfg, cred, None)
|
||||||
return render(request, "partials/check_result.html", report=report, errors={})
|
return render(request, "partials/check_result.html", report=report, errors={})
|
||||||
@@ -503,7 +742,7 @@ async def mount_test(request: Request, mount_id: int) -> Response:
|
|||||||
if m is None:
|
if m is None:
|
||||||
return HTMLResponse("Anbindung existiert nicht.", status_code=404)
|
return HTMLResponse("Anbindung existiert nicht.", status_code=404)
|
||||||
data = mount_input_from_model(m)
|
data = mount_input_from_model(m)
|
||||||
cred = credential_for_test(data, m, state.secrets)
|
cred = credential_for_test(data, m, state.secrets, s)
|
||||||
from netmover.mounting.manager import MountConfig
|
from netmover.mounting.manager import MountConfig
|
||||||
|
|
||||||
cfg = MountConfig.from_model(m)
|
cfg = MountConfig.from_model(m)
|
||||||
@@ -563,6 +802,123 @@ async def mount_reset_hostkey(request: Request, mount_id: int) -> Response:
|
|||||||
return redirect(f"/mounts#mount-{mount_id}")
|
return redirect(f"/mounts#mount-{mount_id}")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Zugangsdaten ------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/credentials", response_class=HTMLResponse)
|
||||||
|
async def credentials_page(request: Request) -> Response:
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
return render(
|
||||||
|
request,
|
||||||
|
"credentials.html",
|
||||||
|
active="keys",
|
||||||
|
keys=list_keys(s),
|
||||||
|
passwords=list_passwords(s),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/keys")
|
||||||
|
async def keys_redirect(request: Request) -> Response:
|
||||||
|
return RedirectResponse("/credentials", status_code=301)
|
||||||
|
|
||||||
|
|
||||||
|
async def _cred_action(request: Request, work: Callable[[Any], str]) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
try:
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
flash(request, work(s))
|
||||||
|
except AdminError as exc:
|
||||||
|
flash(request, str(exc), "error")
|
||||||
|
return redirect("/credentials")
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/credentials/passwords")
|
||||||
|
async def password_create(request: Request) -> Response:
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
secrets = nm(request).secrets
|
||||||
|
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
cred = create_password(
|
||||||
|
s,
|
||||||
|
secrets,
|
||||||
|
form.get("name", ""),
|
||||||
|
form.get("username", ""),
|
||||||
|
form.get("password", ""),
|
||||||
|
form.get("domain", ""),
|
||||||
|
)
|
||||||
|
return f"Zugangsdaten „{cred.name}“ gespeichert."
|
||||||
|
|
||||||
|
return await _cred_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/credentials/{cred_id}/password")
|
||||||
|
async def password_update(request: Request, cred_id: int) -> Response:
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
secrets = nm(request).secrets
|
||||||
|
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
cred = update_password(
|
||||||
|
s,
|
||||||
|
secrets,
|
||||||
|
cred_id,
|
||||||
|
form.get("username", ""),
|
||||||
|
form.get("password", ""),
|
||||||
|
form.get("domain", ""),
|
||||||
|
)
|
||||||
|
return f"Zugangsdaten „{cred.name}“ geändert – gilt ab sofort für alle Anbindungen."
|
||||||
|
|
||||||
|
return await _cred_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/credentials/keys/generate")
|
||||||
|
async def key_generate(request: Request) -> Response:
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
secrets = nm(request).secrets
|
||||||
|
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
key = generate_key(s, secrets, form.get("name", ""))
|
||||||
|
return f"Schlüssel „{key.name}“ erzeugt."
|
||||||
|
|
||||||
|
return await _cred_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/credentials/keys/import")
|
||||||
|
async def key_import(request: Request) -> Response:
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
secrets = nm(request).secrets
|
||||||
|
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
key = import_key(
|
||||||
|
s,
|
||||||
|
secrets,
|
||||||
|
form.get("name", ""),
|
||||||
|
form.get("private_key", ""),
|
||||||
|
form.get("passphrase", ""),
|
||||||
|
)
|
||||||
|
return f"Schlüssel „{key.name}“ importiert."
|
||||||
|
|
||||||
|
return await _cred_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/credentials/{cred_id}/rename")
|
||||||
|
async def credential_rename(request: Request, cred_id: int) -> Response:
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
rename_credential(s, cred_id, form.get("name", ""))
|
||||||
|
return "Umbenannt."
|
||||||
|
|
||||||
|
return await _cred_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/credentials/{cred_id}/delete")
|
||||||
|
async def credential_delete(request: Request, cred_id: int) -> Response:
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
return f"„{delete_credential(s, cred_id)}“ gelöscht."
|
||||||
|
|
||||||
|
return await _cred_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
# -- Regeln -----------------------------------------------------------------------------------
|
# -- Regeln -----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
@@ -572,6 +928,7 @@ def _rule_form(
|
|||||||
errors: dict[str, str],
|
errors: dict[str, str],
|
||||||
rule_id: int | None,
|
rule_id: int | None,
|
||||||
status_code: int = 200,
|
status_code: int = 200,
|
||||||
|
copy_of: str | None = None,
|
||||||
) -> Response:
|
) -> Response:
|
||||||
with nm(request).session_factory() as s:
|
with nm(request).session_factory() as s:
|
||||||
mounts = s.scalars(select(Mount).order_by(Mount.name)).all()
|
mounts = s.scalars(select(Mount).order_by(Mount.name)).all()
|
||||||
@@ -585,9 +942,9 @@ def _rule_form(
|
|||||||
d=d,
|
d=d,
|
||||||
errors=errors,
|
errors=errors,
|
||||||
rule_id=rule_id,
|
rule_id=rule_id,
|
||||||
src_mounts=[m for m in mounts if m.type.is_kernel_mount],
|
copy_of=copy_of,
|
||||||
src_options=[("", "– wählen –")]
|
src_mounts=mounts,
|
||||||
+ [(m.id, m.name) for m in mounts if m.type.is_kernel_mount],
|
src_options=[("", "– wählen –")] + [(m.id, f"{m.name} ({m.type.value})") for m in mounts],
|
||||||
dst_options=[("", "– wählen –")] + [(m.id, f"{m.name} ({m.type.value})") for m in mounts],
|
dst_options=[("", "– wählen –")] + [(m.id, f"{m.name} ({m.type.value})") for m in mounts],
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -613,25 +970,47 @@ async def rule_edit(request: Request, rule_id: int) -> Response:
|
|||||||
return _rule_form(request, d, {}, rule_id)
|
return _rule_form(request, d, {}, rule_id)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/rules/{source_id}/copy", response_class=HTMLResponse)
|
||||||
|
async def rule_copy(request: Request, source_id: int) -> Response:
|
||||||
|
"""Neue Regel, vorausgefüllt aus einer bestehenden – zunächst deaktiviert, damit sie
|
||||||
|
nicht sofort mit dem Original um dieselben Dateien konkurriert."""
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
src = s.get(Rule, source_id)
|
||||||
|
if src is None:
|
||||||
|
return redirect("/rules")
|
||||||
|
d = rule_input_from_model(src)
|
||||||
|
d.name = rule_copy_name(s, src.name)
|
||||||
|
d.enabled = False
|
||||||
|
source_name = src.name
|
||||||
|
return _rule_form(request, d, {}, None, copy_of=source_name)
|
||||||
|
|
||||||
|
|
||||||
@protected.post("/rules/new", response_class=HTMLResponse)
|
@protected.post("/rules/new", response_class=HTMLResponse)
|
||||||
@protected.post("/rules/{rule_id}/edit", response_class=HTMLResponse)
|
@protected.post("/rules/{rule_id}/edit", response_class=HTMLResponse)
|
||||||
async def rule_save(request: Request, rule_id: int | None = None) -> Response:
|
@protected.post("/rules/{source_id}/copy", response_class=HTMLResponse)
|
||||||
|
async def rule_save(
|
||||||
|
request: Request, rule_id: int | None = None, source_id: int | None = None
|
||||||
|
) -> Response:
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
form = _form_dict(await request.form())
|
form = _form_dict(await request.form())
|
||||||
with session_scope(state.session_factory) as s:
|
with session_scope(state.session_factory) as s:
|
||||||
|
source = s.get(Rule, source_id) if source_id is not None else None
|
||||||
|
copy_of = source.name if source is not None else None
|
||||||
d, errors = parse_rule_form(s, form)
|
d, errors = parse_rule_form(s, form)
|
||||||
r = None
|
r = None
|
||||||
if not errors:
|
if not errors:
|
||||||
try:
|
try:
|
||||||
r, errors = save_rule(s, d, rule_id)
|
r, errors = save_rule(s, d, rule_id) # Kopie: rule_id ist None → neue Regel
|
||||||
except AdminError as exc:
|
except AdminError as exc:
|
||||||
errors = {"_": str(exc)}
|
errors = {"_": str(exc)}
|
||||||
if errors or r is None:
|
if errors or r is None:
|
||||||
s.rollback()
|
s.rollback()
|
||||||
return _rule_form(request, d, errors, rule_id, status_code=400)
|
return _rule_form(request, d, errors, rule_id, status_code=400, copy_of=copy_of)
|
||||||
name = r.name
|
name, enabled = r.name, r.enabled
|
||||||
state.scheduler.sync()
|
state.scheduler.sync()
|
||||||
flash(request, f"Regel „{name}“ gespeichert.")
|
flash(request, f"Regel „{name}“ gespeichert.")
|
||||||
|
if copy_of and not enabled:
|
||||||
|
flash(request, "Die Kopie ist deaktiviert. Probelauf machen, dann aktivieren.", "info")
|
||||||
return redirect("/rules")
|
return redirect("/rules")
|
||||||
|
|
||||||
|
|
||||||
@@ -721,13 +1100,76 @@ async def log_items(request: Request, run_id: int) -> Response:
|
|||||||
return render(request, "partials/run_items.html", items=items, run=run)
|
return render(request, "partials/run_items.html", items=items, run=run)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Hilfe --------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
REPO_URL = "https://git.wilde-server.de/florian.wilde/netmover"
|
||||||
|
_IMG_NAME = re.compile(r"^[a-z0-9_-]+\.png$")
|
||||||
|
|
||||||
|
|
||||||
|
def _help_dir() -> Path:
|
||||||
|
"""Im installierten Paket mitgeliefert (siehe pyproject), sonst docs/ im Repo."""
|
||||||
|
packaged = Path(str(files("netmover.web") / "help"))
|
||||||
|
if (packaged / "ANLEITUNG.md").is_file():
|
||||||
|
return packaged
|
||||||
|
return Path(__file__).resolve().parents[3] / "docs"
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def _help_html() -> str:
|
||||||
|
text = (_help_dir() / "ANLEITUNG.md").read_text(encoding="utf-8")
|
||||||
|
html = markdown.markdown(text, extensions=["tables", "attr_list", "fenced_code"])
|
||||||
|
html = html.replace('src="img/', 'src="/help/img/')
|
||||||
|
return html.replace(
|
||||||
|
'href="../README.md"', f'href="{REPO_URL}#readme" target="_blank" rel="noopener"'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/help", response_class=HTMLResponse)
|
||||||
|
async def help_page(request: Request) -> Response:
|
||||||
|
return render(request, "help.html", active="help", content=_help_html())
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/help/img/{name}")
|
||||||
|
async def help_image(request: Request, name: str) -> Response:
|
||||||
|
path = _help_dir() / "img" / name
|
||||||
|
if not _IMG_NAME.fullmatch(name) or not path.is_file():
|
||||||
|
return Response(status_code=404)
|
||||||
|
return FileResponse(path, media_type="image/png")
|
||||||
|
|
||||||
|
|
||||||
# -- Einstellungen ------------------------------------------------------------------------------
|
# -- Einstellungen ------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _monitor_urls(request: Request) -> list[tuple[str, str]]:
|
||||||
|
"""Adressen für Uptime-Kuma-HTTP-Monitore (Gesamtzustand und Einzel-Endpunkte)."""
|
||||||
|
base = str(request.base_url).rstrip("/")
|
||||||
|
urls = [("Gesamtzustand", f"{base}/healthz")]
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
urls += [
|
||||||
|
(f"Anbindung „{n}“", f"{base}/healthz/mount/{slug}")
|
||||||
|
for n, slug in s.execute(select(Mount.name, Mount.slug).order_by(Mount.name))
|
||||||
|
]
|
||||||
|
urls += [
|
||||||
|
(f"Regel „{n}“", f"{base}/healthz/rule/{rid}")
|
||||||
|
for rid, n in s.execute(select(Rule.id, Rule.name).order_by(Rule.name))
|
||||||
|
]
|
||||||
|
urls.append(("Sicherung", f"{base}/healthz/backup"))
|
||||||
|
return urls
|
||||||
|
|
||||||
|
|
||||||
@protected.get("/settings", response_class=HTMLResponse)
|
@protected.get("/settings", response_class=HTMLResponse)
|
||||||
async def settings_page(request: Request) -> Response:
|
async def settings_page(request: Request) -> Response:
|
||||||
cfg = load_settings(nm(request).session_factory)
|
cfg = load_settings(nm(request).session_factory)
|
||||||
return render(request, "settings.html", active="settings", cfg=cfg, errors={})
|
return render(
|
||||||
|
request,
|
||||||
|
"settings.html",
|
||||||
|
active="settings",
|
||||||
|
cfg=cfg,
|
||||||
|
errors={},
|
||||||
|
monitor_urls=_monitor_urls(request),
|
||||||
|
client_ip=request.client.host if request.client else "",
|
||||||
|
loopback_trusted=auth.includes_loopback(cfg.trusted_networks),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@protected.post("/settings", response_class=HTMLResponse)
|
@protected.post("/settings", response_class=HTMLResponse)
|
||||||
@@ -754,6 +1196,37 @@ async def settings_save(request: Request) -> Response:
|
|||||||
if url and not url.startswith(("http://", "https://")):
|
if url and not url.startswith(("http://", "https://")):
|
||||||
errors["webhook_url"] = "URL muss mit http:// oder https:// beginnen."
|
errors["webhook_url"] = "URL muss mit http:// oder https:// beginnen."
|
||||||
cfg.webhook_url = url
|
cfg.webhook_url = url
|
||||||
|
cfg.email_enabled = form.get("email_enabled") == "on"
|
||||||
|
cfg.smtp_host = (form.get("smtp_host") or "").strip()
|
||||||
|
cfg.smtp_user = (form.get("smtp_user") or "").strip()
|
||||||
|
cfg.smtp_from = (form.get("smtp_from") or "").strip()
|
||||||
|
cfg.smtp_to = (form.get("smtp_to") or "").strip()
|
||||||
|
cfg.smtp_security = form.get("smtp_security") or "starttls"
|
||||||
|
try:
|
||||||
|
cfg.smtp_port = int(form.get("smtp_port") or 587)
|
||||||
|
if not 1 <= cfg.smtp_port <= 65535:
|
||||||
|
raise ValueError
|
||||||
|
except ValueError:
|
||||||
|
errors["smtp_port"] = "Port zwischen 1 und 65535."
|
||||||
|
if cfg.smtp_security not in SECURITY_MODES:
|
||||||
|
errors["smtp_security"] = "Ungültig."
|
||||||
|
if cfg.smtp_host and not re.fullmatch(r"[A-Za-z0-9.-]+", cfg.smtp_host):
|
||||||
|
errors["smtp_host"] = "Hostname oder IP-Adresse."
|
||||||
|
if cfg.smtp_from and not valid_address(cfg.smtp_from):
|
||||||
|
errors["smtp_from"] = "Keine gültige E-Mail-Adresse."
|
||||||
|
try:
|
||||||
|
parse_recipients(cfg.smtp_to)
|
||||||
|
except MailConfigError as exc:
|
||||||
|
errors["smtp_to"] = str(exc)
|
||||||
|
if cfg.email_enabled and not (cfg.smtp_host and cfg.smtp_from and cfg.smtp_to):
|
||||||
|
errors["smtp_host"] = "Für E-Mail werden Server, Absender und Empfänger benötigt."
|
||||||
|
if form.get("smtp_password"):
|
||||||
|
cfg.smtp_password_enc = state.secrets.encrypt(form["smtp_password"])
|
||||||
|
kuma = (form.get("kuma_push_url") or "").strip()
|
||||||
|
if kuma != auth_mask(cfg.kuma_push_url):
|
||||||
|
if kuma and not kuma.startswith(("http://", "https://")):
|
||||||
|
errors["kuma_push_url"] = "URL muss mit http:// oder https:// beginnen."
|
||||||
|
cfg.kuma_push_url = kuma
|
||||||
cfg.notify_recovery = form.get("notify_recovery") == "on"
|
cfg.notify_recovery = form.get("notify_recovery") == "on"
|
||||||
cfg.webhook_template = form.get("webhook_template") or cfg.webhook_template
|
cfg.webhook_template = form.get("webhook_template") or cfg.webhook_template
|
||||||
cfg.webhook_content_type = (form.get("webhook_content_type") or "application/json").strip()
|
cfg.webhook_content_type = (form.get("webhook_content_type") or "application/json").strip()
|
||||||
@@ -766,6 +1239,100 @@ async def settings_save(request: Request) -> Response:
|
|||||||
return redirect("/settings")
|
return redirect("/settings")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Datensicherung -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _backup_page(
|
||||||
|
request: Request, cfg: AppSettings, errors: dict[str, str], status_code: int = 200
|
||||||
|
) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
with state.session_factory() as s:
|
||||||
|
mounts = [
|
||||||
|
(m.id, f"{m.name} ({m.type.value})")
|
||||||
|
for m in s.scalars(select(Mount).order_by(Mount.name))
|
||||||
|
]
|
||||||
|
health = state.health()
|
||||||
|
return render(
|
||||||
|
request,
|
||||||
|
"backup.html",
|
||||||
|
status_code=status_code,
|
||||||
|
active="backup",
|
||||||
|
cfg=cfg,
|
||||||
|
errors=errors,
|
||||||
|
mounts=mounts,
|
||||||
|
st=load_backup_state(state.session_factory),
|
||||||
|
check=health.find("backup"),
|
||||||
|
running=state.backups.running,
|
||||||
|
monitor_url=f"{str(request.base_url).rstrip('/')}/healthz/backup",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/backup", response_class=HTMLResponse)
|
||||||
|
async def backup_page(request: Request) -> Response:
|
||||||
|
return _backup_page(request, load_settings(nm(request).session_factory), {})
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/backup", response_class=HTMLResponse)
|
||||||
|
async def backup_save(request: Request) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
errors: dict[str, str] = {}
|
||||||
|
cfg.backup_enabled = form.get("backup_enabled") == "on"
|
||||||
|
try:
|
||||||
|
cfg.backup_mount_id = int(form.get("backup_mount_id") or 0)
|
||||||
|
except ValueError:
|
||||||
|
cfg.backup_mount_id = 0
|
||||||
|
with state.session_factory() as s:
|
||||||
|
if cfg.backup_mount_id and s.get(Mount, cfg.backup_mount_id) is None:
|
||||||
|
errors["backup_mount_id"] = "Anbindung existiert nicht."
|
||||||
|
try:
|
||||||
|
cfg.backup_path = safe_relpath("backup_path", form.get("backup_path") or "")
|
||||||
|
except ValidationError as exc:
|
||||||
|
errors["backup_path"] = str(exc)
|
||||||
|
cfg.backup_time = (form.get("backup_time") or "03:30").strip()
|
||||||
|
try:
|
||||||
|
parse_backup_time(cfg.backup_time)
|
||||||
|
except ValueError as exc:
|
||||||
|
errors["backup_time"] = str(exc)
|
||||||
|
try:
|
||||||
|
cfg.backup_keep = int(form.get("backup_keep") or 14)
|
||||||
|
if not 1 <= cfg.backup_keep <= 365:
|
||||||
|
raise ValueError
|
||||||
|
except ValueError:
|
||||||
|
errors["backup_keep"] = "Zwischen 1 und 365."
|
||||||
|
passphrase = form.get("passphrase") or ""
|
||||||
|
if passphrase or form.get("passphrase_confirm"):
|
||||||
|
problem = passphrase_problem(passphrase, form.get("passphrase_confirm") or "")
|
||||||
|
if problem:
|
||||||
|
errors["passphrase"] = problem
|
||||||
|
else:
|
||||||
|
cfg.backup_passphrase_enc = state.secrets.encrypt(passphrase)
|
||||||
|
if cfg.backup_enabled and not cfg.backup_mount_id:
|
||||||
|
errors["backup_mount_id"] = "Für die Sicherung eine Anbindung wählen."
|
||||||
|
if cfg.backup_enabled and not cfg.backup_passphrase_enc and "passphrase" not in errors:
|
||||||
|
errors["passphrase"] = "Für die Sicherung eine Passphrase festlegen." # noqa: S105
|
||||||
|
if errors:
|
||||||
|
return _backup_page(request, cfg, errors, status_code=400)
|
||||||
|
save_settings(state.session_factory, cfg)
|
||||||
|
flash(request, "Sicherung gespeichert.")
|
||||||
|
if passphrase:
|
||||||
|
flash(
|
||||||
|
request,
|
||||||
|
"Passphrase jetzt im Passwortmanager ablegen – ohne sie lässt sich keine "
|
||||||
|
"Sicherung wiederherstellen.",
|
||||||
|
"warn",
|
||||||
|
)
|
||||||
|
return redirect("/backup")
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/backup/run")
|
||||||
|
async def backup_run(request: Request) -> Response:
|
||||||
|
result = await run_in_threadpool(nm(request).backups.run)
|
||||||
|
flash(request, result.message, "ok" if result.ok else "error")
|
||||||
|
return redirect("/backup")
|
||||||
|
|
||||||
|
|
||||||
def auth_mask(url: str) -> str:
|
def auth_mask(url: str) -> str:
|
||||||
"""Webhook-URL nur gekürzt anzeigen – sie enthält oft ein Token."""
|
"""Webhook-URL nur gekürzt anzeigen – sie enthält oft ein Token."""
|
||||||
if not url:
|
if not url:
|
||||||
@@ -799,6 +1366,39 @@ async def settings_webhook_test(request: Request) -> Response:
|
|||||||
return HTMLResponse('<p class="msg ok">Testnachricht gesendet.</p>')
|
return HTMLResponse('<p class="msg ok">Testnachricht gesendet.</p>')
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/settings/mail-test", response_class=HTMLResponse)
|
||||||
|
async def settings_mail_test(request: Request) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
if not (cfg.smtp_host and cfg.smtp_from and cfg.smtp_to):
|
||||||
|
return HTMLResponse('<p class="msg error">Server, Absender und Empfänger speichern.</p>')
|
||||||
|
try:
|
||||||
|
await run_in_threadpool(
|
||||||
|
state.notifier.send_mail_now,
|
||||||
|
"NetMover: Testnachricht",
|
||||||
|
"Die E-Mail-Benachrichtigung funktioniert.",
|
||||||
|
cfg,
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
detail = escape(str(exc))[:300] or type(exc).__name__
|
||||||
|
return HTMLResponse(f'<p class="msg error">Fehlgeschlagen: {detail}</p>')
|
||||||
|
return HTMLResponse(f'<p class="msg ok">Gesendet an {escape(cfg.smtp_to)}.</p>')
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/settings/kuma-test", response_class=HTMLResponse)
|
||||||
|
async def settings_kuma_test(request: Request) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
url = load_settings(state.session_factory).kuma_push_url
|
||||||
|
if not url:
|
||||||
|
return HTMLResponse('<p class="msg error">Keine Push-URL gespeichert.</p>')
|
||||||
|
ok = await run_in_threadpool(kuma_push, url, await run_in_threadpool(state.health))
|
||||||
|
if not ok:
|
||||||
|
return HTMLResponse('<p class="msg error">Kuma nicht erreichbar oder URL falsch.</p>')
|
||||||
|
return HTMLResponse(
|
||||||
|
'<p class="msg ok">Gemeldet – in Kuma sollte der Monitor jetzt grün/rot sein.</p>'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@protected.post("/settings/password", response_class=HTMLResponse)
|
@protected.post("/settings/password", response_class=HTMLResponse)
|
||||||
async def settings_password(request: Request) -> Response:
|
async def settings_password(request: Request) -> Response:
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
@@ -818,6 +1418,58 @@ async def settings_password(request: Request) -> Response:
|
|||||||
return redirect("/settings")
|
return redirect("/settings")
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/settings/auth", response_class=HTMLResponse)
|
||||||
|
async def settings_auth(request: Request) -> Response:
|
||||||
|
"""Netze ohne / mit Passwort festlegen. Nur mit dem aktuellen Passwort und nie so,
|
||||||
|
dass man sich mit der eigenen Adresse aussperrt."""
|
||||||
|
state = nm(request)
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
back = redirect("/settings#anmeldung")
|
||||||
|
if not await run_in_threadpool(
|
||||||
|
auth.verify_password, state.session_factory, form.get("current", "")
|
||||||
|
):
|
||||||
|
flash(request, "Aktuelles Passwort falsch – Anmeldung unverändert.", "error")
|
||||||
|
return back
|
||||||
|
trusted = (form.get("trusted_networks") or "").strip()
|
||||||
|
allowed = (form.get("allowed_networks") or "").strip()
|
||||||
|
try:
|
||||||
|
auth.parse_networks(trusted)
|
||||||
|
auth.parse_networks(allowed)
|
||||||
|
except ValueError as exc:
|
||||||
|
flash(request, str(exc), "error")
|
||||||
|
return back
|
||||||
|
me = auth.client_host(request)
|
||||||
|
if auth.access_decision(me, auth.AUTH_ALWAYS, trusted, allowed) == auth.ACCESS_DENY:
|
||||||
|
flash(
|
||||||
|
request,
|
||||||
|
f"Nicht gespeichert: Damit wärst du selbst ausgesperrt (deine Adresse {me}). "
|
||||||
|
"Trage dein Netz bei „mit“ oder „ohne Passwort“ ein.",
|
||||||
|
"error",
|
||||||
|
)
|
||||||
|
return back
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
cfg.trusted_networks, cfg.allowed_networks = trusted, allowed
|
||||||
|
cfg.auth_mode = auth.AUTH_TRUSTED if trusted else auth.AUTH_ALWAYS
|
||||||
|
save_settings(state.session_factory, cfg)
|
||||||
|
event(
|
||||||
|
log,
|
||||||
|
"auth.access_changed",
|
||||||
|
level=30,
|
||||||
|
without_password=trusted,
|
||||||
|
with_password=allowed or "alle übrigen",
|
||||||
|
)
|
||||||
|
parts = [f"ohne Passwort aus: {trusted}" if trusted else "überall mit Passwort"]
|
||||||
|
parts.append(
|
||||||
|
f"mit Passwort nur aus: {allowed} – alle anderen Netze abgewiesen"
|
||||||
|
if allowed
|
||||||
|
else "alle übrigen Netze mit Passwort"
|
||||||
|
)
|
||||||
|
flash(
|
||||||
|
request, "Anmeldung gespeichert: " + "; ".join(parts) + ".", "warn" if trusted else "info"
|
||||||
|
)
|
||||||
|
return back
|
||||||
|
|
||||||
|
|
||||||
@protected.get("/settings/export")
|
@protected.get("/settings/export")
|
||||||
async def settings_export(request: Request) -> Response:
|
async def settings_export(request: Request) -> Response:
|
||||||
state = nm(request)
|
state = nm(request)
|
||||||
@@ -919,6 +1571,35 @@ def create_app(settings: Settings | None = None, state: NetMover | None = None)
|
|||||||
openapi_url=None,
|
openapi_url=None,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@app.middleware("http")
|
||||||
|
async def network_access(
|
||||||
|
request: Request, call_next: Callable[[Request], Awaitable[Response]]
|
||||||
|
) -> Response:
|
||||||
|
path = request.url.path
|
||||||
|
if not (path.startswith("/healthz") or path.startswith("/static/")):
|
||||||
|
app_state = getattr(request.app.state, "nm", None)
|
||||||
|
if app_state is not None:
|
||||||
|
cfg = load_settings(app_state.session_factory)
|
||||||
|
decision = auth.access_decision(
|
||||||
|
auth.client_host(request),
|
||||||
|
cfg.auth_mode,
|
||||||
|
cfg.trusted_networks,
|
||||||
|
cfg.allowed_networks,
|
||||||
|
)
|
||||||
|
if decision == auth.ACCESS_DENY:
|
||||||
|
event(
|
||||||
|
log,
|
||||||
|
"auth.denied_network",
|
||||||
|
level=30,
|
||||||
|
client=auth.client_host(request) or "?",
|
||||||
|
)
|
||||||
|
return HTMLResponse(
|
||||||
|
"<!doctype html><meta charset=utf-8><title>NetMover</title>"
|
||||||
|
"<p>Zugriff aus diesem Netz ist nicht erlaubt.</p>",
|
||||||
|
status_code=403,
|
||||||
|
)
|
||||||
|
return await call_next(request)
|
||||||
|
|
||||||
@app.middleware("http")
|
@app.middleware("http")
|
||||||
async def security_headers(
|
async def security_headers(
|
||||||
request: Request, call_next: Callable[[Request], Awaitable[Response]]
|
request: Request, call_next: Callable[[Request], Awaitable[Response]]
|
||||||
|
|||||||
@@ -12,6 +12,8 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import hmac
|
import hmac
|
||||||
|
import ipaddress
|
||||||
|
import re
|
||||||
import secrets
|
import secrets
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
@@ -32,6 +34,88 @@ SAFE_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
|
|||||||
_hasher = PasswordHasher()
|
_hasher = PasswordHasher()
|
||||||
|
|
||||||
|
|
||||||
|
AUTH_ALWAYS, AUTH_TRUSTED, AUTH_OFF = "always", "trusted", "off"
|
||||||
|
AUTH_MODES = (AUTH_ALWAYS, AUTH_TRUSTED, AUTH_OFF)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_networks(raw: str) -> list[ipaddress.IPv4Network | ipaddress.IPv6Network]:
|
||||||
|
"""„10.0.10.0/24, 192.168.1.5“ → Netze; ``ValueError`` mit verständlicher Meldung."""
|
||||||
|
nets = []
|
||||||
|
for part in re.split(r"[,;\s]+", raw.strip()):
|
||||||
|
if not part:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
nets.append(ipaddress.ip_network(part, strict=False))
|
||||||
|
except ValueError:
|
||||||
|
raise ValueError(f"„{part}“ ist kein gültiges Netz (z. B. 10.0.10.0/24)") from None
|
||||||
|
return nets
|
||||||
|
|
||||||
|
|
||||||
|
ACCESS_NOPW, ACCESS_PW, ACCESS_DENY = "nopw", "pw", "deny"
|
||||||
|
|
||||||
|
|
||||||
|
def _address(host: str | None) -> ipaddress.IPv4Address | ipaddress.IPv6Address | None:
|
||||||
|
if not host:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return ipaddress.ip_address(host)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _networks_or_empty(raw: str) -> list[ipaddress.IPv4Network | ipaddress.IPv6Network]:
|
||||||
|
try:
|
||||||
|
return parse_networks(raw)
|
||||||
|
except ValueError:
|
||||||
|
return [] # gespeicherte Werte sind beim Speichern geprüft
|
||||||
|
|
||||||
|
|
||||||
|
def access_decision(host: str | None, mode: str, trusted: str, allowed: str) -> str:
|
||||||
|
"""Wie darf diese Adresse auf die Oberfläche zugreifen?
|
||||||
|
|
||||||
|
1. aus ``trusted`` (oder Modus „off“): **ohne Passwort**,
|
||||||
|
2. ``allowed`` leer oder Adresse darin: **mit Passwort**,
|
||||||
|
3. sonst: **abgewiesen**.
|
||||||
|
|
||||||
|
Hinter einem Reverse Proxy ist die Client-Adresse nur dann echt, wenn
|
||||||
|
``NETMOVER_FORWARDED_ALLOW_IPS`` den Proxy kennt – sonst sähe jede Anfrage aus,
|
||||||
|
als käme sie vom Proxy selbst."""
|
||||||
|
if mode == AUTH_OFF:
|
||||||
|
return ACCESS_NOPW
|
||||||
|
addr = _address(host)
|
||||||
|
if addr is not None and any(addr in n for n in _networks_or_empty(trusted)):
|
||||||
|
return ACCESS_NOPW
|
||||||
|
allowed_nets = _networks_or_empty(allowed)
|
||||||
|
if allowed_nets and (addr is None or not any(addr in n for n in allowed_nets)):
|
||||||
|
return ACCESS_DENY
|
||||||
|
return ACCESS_PW
|
||||||
|
|
||||||
|
|
||||||
|
def client_host(request: Request) -> str | None:
|
||||||
|
return request.client.host if request.client else None
|
||||||
|
|
||||||
|
|
||||||
|
def login_bypass(request: Request, mode: str, trusted: str, allowed: str = "") -> str | None:
|
||||||
|
"""Grund, warum keine Anmeldung nötig ist (``"off"``/``"trusted"``), sonst ``None``."""
|
||||||
|
if access_decision(client_host(request), mode, trusted, allowed) != ACCESS_NOPW:
|
||||||
|
return None
|
||||||
|
return AUTH_OFF if mode == AUTH_OFF or _covers_everything(trusted) else AUTH_TRUSTED
|
||||||
|
|
||||||
|
|
||||||
|
def _covers_everything(raw: str) -> bool:
|
||||||
|
return any(n.prefixlen == 0 for n in _networks_or_empty(raw))
|
||||||
|
|
||||||
|
|
||||||
|
def includes_loopback(raw: str) -> bool:
|
||||||
|
try:
|
||||||
|
nets = parse_networks(raw)
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
return any(
|
||||||
|
ipaddress.ip_address("127.0.0.1") in n or ipaddress.ip_address("::1") in n for n in nets
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class LoginRequired(Exception):
|
class LoginRequired(Exception):
|
||||||
def __init__(self, setup: bool = False) -> None:
|
def __init__(self, setup: bool = False) -> None:
|
||||||
self.setup = setup
|
self.setup = setup
|
||||||
|
|||||||
@@ -19,14 +19,68 @@ a { color: var(--accent); }
|
|||||||
code, pre, .mono { font-family: ui-monospace, "SFMono-Regular", Menlo, Consolas, monospace; font-size: .9em; }
|
code, pre, .mono { font-family: ui-monospace, "SFMono-Regular", Menlo, Consolas, monospace; font-size: .9em; }
|
||||||
pre { white-space: pre-wrap; word-break: break-word; margin: 0; }
|
pre { white-space: pre-wrap; word-break: break-word; margin: 0; }
|
||||||
|
|
||||||
header.top { background: var(--card); border-bottom: 1px solid var(--line); }
|
/* --- Layout: Kopfzeile + Seitenleiste (Desktop), aufklappbares Menü (Handy) --- */
|
||||||
header.top .inner { max-width: 1200px; margin: 0 auto; display: flex; align-items: center;
|
:root { --sidebar: 228px; --topbar: 52px; }
|
||||||
gap: var(--gap); padding: 8px 16px; flex-wrap: wrap; }
|
.topbar { position: sticky; top: 0; z-index: 20; height: var(--topbar); display: flex; align-items: center;
|
||||||
header.top .brand { font-weight: 700; text-decoration: none; color: var(--fg); margin-right: 8px; }
|
gap: 12px; padding: 0 16px; background: var(--card); border-bottom: 1px solid var(--line); }
|
||||||
nav.main { display: flex; gap: 4px; flex-wrap: wrap; flex: 1; }
|
.topbar .brand { font-weight: 700; font-size: 1.05rem; text-decoration: none; color: var(--fg); }
|
||||||
nav.main a { padding: 6px 10px; border-radius: var(--radius); text-decoration: none; color: var(--fg); }
|
.topbar .spacer { flex: 1; }
|
||||||
nav.main a.active, nav.main a:hover { background: var(--bg); }
|
.health { display: inline-flex; align-items: center; gap: 7px; font-size: .85em; font-weight: 600;
|
||||||
main { max-width: 1200px; margin: 0 auto; padding: 16px; }
|
padding: 4px 10px; border-radius: 99px; text-decoration: none; }
|
||||||
|
.health .dot { width: 9px; height: 9px; border-radius: 50%; background: currentColor; }
|
||||||
|
.health.ok { color: var(--ok); background: var(--ok-bg); }
|
||||||
|
.health.bad { color: var(--err); background: var(--err-bg); }
|
||||||
|
.health.warn { color: var(--warn); background: var(--warn-bg); }
|
||||||
|
.help-link { display: inline-flex; align-items: center; gap: 6px; font-size: .9em; text-decoration: none; }
|
||||||
|
.menu-toggle { display: none; align-items: center; gap: 6px; }
|
||||||
|
.ico { width: 18px; height: 18px; flex: none; fill: none; stroke: currentColor; stroke-width: 2;
|
||||||
|
stroke-linecap: round; stroke-linejoin: round; }
|
||||||
|
|
||||||
|
.sidebar { position: fixed; top: var(--topbar); left: 0; bottom: 0; width: var(--sidebar); overflow-y: auto;
|
||||||
|
padding: 14px 10px; background: var(--card); border-right: 1px solid var(--line);
|
||||||
|
display: flex; flex-direction: column; gap: 14px; }
|
||||||
|
.nav-title { font-size: .72em; font-weight: 700; letter-spacing: .06em; text-transform: uppercase;
|
||||||
|
color: var(--muted); padding: 0 10px 4px; }
|
||||||
|
.sidebar a, .nav-button { display: flex; align-items: center; gap: 10px; width: 100%; padding: 7px 10px;
|
||||||
|
border-radius: var(--radius); color: var(--fg); text-decoration: none; font: inherit;
|
||||||
|
background: none; border: 0; cursor: pointer; text-align: left; }
|
||||||
|
.sidebar a:hover, .nav-button:hover { background: var(--bg); }
|
||||||
|
.sidebar a.active { background: var(--info-bg); color: var(--accent); font-weight: 600; }
|
||||||
|
.sidebar .count { margin-left: auto; font-size: .75em; color: var(--muted); background: var(--bg);
|
||||||
|
border-radius: 99px; padding: 1px 7px; white-space: nowrap; }
|
||||||
|
.sidebar .count.bad { color: var(--err); background: var(--err-bg); }
|
||||||
|
.nav-foot { margin-top: auto; border-top: 1px solid var(--line); padding-top: 10px; }
|
||||||
|
.nav-foot form { margin: 0; }
|
||||||
|
.nav-foot .version { font-size: .75em; color: var(--muted); padding: 6px 10px 0; }
|
||||||
|
|
||||||
|
main { max-width: 1200px; margin: 0 auto; padding: 18px 20px 40px; }
|
||||||
|
body.with-nav main { margin-left: var(--sidebar); }
|
||||||
|
@media (min-width: 1430px) { body.with-nav main { margin-left: calc(var(--sidebar) + (100vw - var(--sidebar) - 1200px) / 2); } }
|
||||||
|
|
||||||
|
@media (max-width: 900px) {
|
||||||
|
.menu-toggle { display: inline-flex; }
|
||||||
|
.help-link span { display: none; }
|
||||||
|
body.with-nav main { margin-left: 0; }
|
||||||
|
/* Ohne JavaScript bleibt das Menü sichtbar (oben, nicht fixiert). */
|
||||||
|
.sidebar { position: static; width: auto; border-right: 0; border-bottom: 1px solid var(--line); }
|
||||||
|
.js .sidebar { position: fixed; width: min(280px, 85vw); transform: translateX(-105%);
|
||||||
|
transition: transform .18s ease; border-right: 1px solid var(--line); box-shadow: 0 8px 24px rgba(0,0,0,.18); z-index: 30; }
|
||||||
|
.js .sidebar.open { transform: none; }
|
||||||
|
}
|
||||||
|
@media (max-width: 480px) { .health { font-size: 0; padding: 6px; } .health .dot { width: 11px; height: 11px; } }
|
||||||
|
|
||||||
|
/* --- Anleitung --- */
|
||||||
|
.doc { max-width: 860px; line-height: 1.6; padding: 18px 26px 26px; }
|
||||||
|
.doc h1 { font-size: 1.6rem; margin-top: 4px; }
|
||||||
|
.doc h2 { font-size: 1.25rem; margin-top: 34px; padding-top: 12px; border-top: 1px solid var(--line); }
|
||||||
|
.doc h3 { font-size: 1.05rem; margin-top: 22px; }
|
||||||
|
.doc table { margin: 10px 0 14px; font-size: .93em; }
|
||||||
|
.doc img { max-width: 100%; border: 1px solid var(--line); border-radius: var(--radius); margin: 8px 0; }
|
||||||
|
.doc pre { background: var(--bg); border: 1px solid var(--line); border-radius: 6px; padding: 10px 12px; overflow-x: auto; }
|
||||||
|
.doc code { background: var(--bg); border-radius: 4px; padding: 0 4px; }
|
||||||
|
.doc pre code { background: none; padding: 0; }
|
||||||
|
.doc :target { scroll-margin-top: calc(var(--topbar) + 12px); }
|
||||||
|
|
||||||
h1 { font-size: 1.4rem; margin: 4px 0 16px; }
|
h1 { font-size: 1.4rem; margin: 4px 0 16px; }
|
||||||
h2 { font-size: 1.1rem; margin: 24px 0 8px; }
|
h2 { font-size: 1.1rem; margin: 24px 0 8px; }
|
||||||
.muted { color: var(--muted); }
|
.muted { color: var(--muted); }
|
||||||
@@ -95,7 +149,7 @@ details > summary { cursor: pointer; }
|
|||||||
[hidden] { display: none !important; }
|
[hidden] { display: none !important; }
|
||||||
|
|
||||||
@media (max-width: 640px) {
|
@media (max-width: 640px) {
|
||||||
main { padding: 10px; }
|
main { padding: 12px 12px 32px; }
|
||||||
th, td { padding: 6px; }
|
th, td { padding: 6px; }
|
||||||
table.responsive thead { display: none; }
|
table.responsive thead { display: none; }
|
||||||
table.responsive tr { display: block; border-bottom: 1px solid var(--line); padding: 6px 0; }
|
table.responsive tr { display: block; border-bottom: 1px solid var(--line); padding: 6px 0; }
|
||||||
@@ -109,3 +163,19 @@ pre.text-err { margin-top: 6px; }
|
|||||||
ul.tight { margin: 4px 0 0; padding-left: 20px; }
|
ul.tight { margin: 4px 0 0; padding-left: 20px; }
|
||||||
.foot { text-align: center; padding: 16px; }
|
.foot { text-align: center; padding: 16px; }
|
||||||
meter.fill { width: 100%; height: 8px; margin: 6px 0 2px; display: block; }
|
meter.fill { width: 100%; height: 8px; margin: 6px 0 2px; display: block; }
|
||||||
|
|
||||||
|
.doc table { display: block; overflow-x: auto; max-width: 100%; }
|
||||||
|
.legend-help { display: inline-grid; place-items: center; width: 18px; height: 18px; margin-left: 4px;
|
||||||
|
border-radius: 50%; font-size: .75em; font-weight: 700; text-decoration: none; color: var(--muted);
|
||||||
|
border: 1px solid var(--line); vertical-align: 1px; }
|
||||||
|
.legend-help:hover { color: var(--accent); border-color: var(--accent); }
|
||||||
|
details.rename { display: inline-block; position: relative; }
|
||||||
|
details.rename > summary { list-style: none; }
|
||||||
|
details.rename > summary::-webkit-details-marker { display: none; }
|
||||||
|
details.rename[open] > form { position: absolute; right: 0; top: calc(100% + 4px); z-index: 5; width: 300px;
|
||||||
|
padding: 8px; background: var(--card); border: 1px solid var(--line); border-radius: var(--radius);
|
||||||
|
box-shadow: 0 6px 18px rgba(0,0,0,.12); flex-wrap: nowrap; }
|
||||||
|
input.w-auto { width: auto; flex: 1; min-width: 200px; max-width: 360px; }
|
||||||
|
/* Weitere Ziele im Regelformular: je Platz eine eigene Zeile */
|
||||||
|
.target-slot { padding-top: 10px; margin-top: 10px; border-top: 1px dashed var(--line); }
|
||||||
|
.target-slot:first-of-type { border-top: 0; margin-top: 4px; padding-top: 0; }
|
||||||
|
|||||||
@@ -1,6 +1,27 @@
|
|||||||
// NetMover – kleine Helfer. Keine Inline-Skripte (CSP: script-src 'self').
|
// NetMover – kleine Helfer. Keine Inline-Skripte (CSP: script-src 'self').
|
||||||
(function () {
|
(function () {
|
||||||
"use strict";
|
"use strict";
|
||||||
|
document.documentElement.classList.add("js");
|
||||||
|
|
||||||
|
// Menü auf dem Handy auf-/zuklappen
|
||||||
|
var toggle = document.querySelector(".menu-toggle");
|
||||||
|
var sidebar = document.getElementById("sidebar");
|
||||||
|
if (toggle && sidebar) {
|
||||||
|
toggle.addEventListener("click", function (e) {
|
||||||
|
e.stopPropagation();
|
||||||
|
var open = sidebar.classList.toggle("open");
|
||||||
|
toggle.setAttribute("aria-expanded", open ? "true" : "false");
|
||||||
|
});
|
||||||
|
document.addEventListener("click", function (e) {
|
||||||
|
if (sidebar.classList.contains("open") && !sidebar.contains(e.target)) {
|
||||||
|
sidebar.classList.remove("open");
|
||||||
|
toggle.setAttribute("aria-expanded", "false");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
document.addEventListener("keydown", function (e) {
|
||||||
|
if (e.key === "Escape") { sidebar.classList.remove("open"); toggle.setAttribute("aria-expanded", "false"); }
|
||||||
|
});
|
||||||
|
}
|
||||||
var meta = document.querySelector('meta[name="csrf-token"]');
|
var meta = document.querySelector('meta[name="csrf-token"]');
|
||||||
var csrf = meta ? meta.getAttribute("content") : "";
|
var csrf = meta ? meta.getAttribute("content") : "";
|
||||||
|
|
||||||
@@ -17,13 +38,14 @@
|
|||||||
|
|
||||||
// Felder je nach Auswahl ein-/ausblenden: data-show-for="select-name:wert1 wert2"
|
// Felder je nach Auswahl ein-/ausblenden: data-show-for="select-name:wert1 wert2"
|
||||||
function refresh(form) {
|
function refresh(form) {
|
||||||
|
// data-show-for="feld:wert1 wert2" – mehrere Bedingungen mit „|“ (ODER)
|
||||||
form.querySelectorAll("[data-show-for]").forEach(function (el) {
|
form.querySelectorAll("[data-show-for]").forEach(function (el) {
|
||||||
var spec = el.getAttribute("data-show-for").split(":");
|
el.hidden = !el.getAttribute("data-show-for").split("|").some(function (cond) {
|
||||||
var ctl = form.querySelector('[name="' + spec[0] + '"]:checked') ||
|
var spec = cond.split(":");
|
||||||
form.querySelector('select[name="' + spec[0] + '"]');
|
var ctl = form.querySelector('[name="' + spec[0] + '"]:checked') ||
|
||||||
var value = ctl ? ctl.value : "";
|
form.querySelector('select[name="' + spec[0] + '"]');
|
||||||
var show = spec[1].split(" ").indexOf(value) !== -1;
|
return spec[1].split(" ").indexOf(ctl ? ctl.value : "") !== -1;
|
||||||
el.hidden = !show;
|
});
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
document.querySelectorAll("form[data-dynamic]").forEach(function (form) {
|
document.querySelectorAll("form[data-dynamic]").forEach(function (form) {
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
{% from "macros.html" import field, select, check, csrf_field %}
|
||||||
|
{% block title %}Sicherung{% endblock %}
|
||||||
|
{% block content %}
|
||||||
|
<h1>Sicherung</h1>
|
||||||
|
<p class="muted">Täglich Datenbank und Schlüssel – also alle Anbindungen, Regeln, Zugangsdaten und Einstellungen –
|
||||||
|
verschlüsselt auf eine Anbindung, z. B. eine eigene Freigabe auf dem NAS.</p>
|
||||||
|
|
||||||
|
<div class="card stack">
|
||||||
|
<h2>Zustand</h2>
|
||||||
|
{% if check %}<p class="msg {{ 'ok' if check.ok else 'error' }}">{{ check.message }}</p>{% endif %}
|
||||||
|
<table class="small"><tbody>
|
||||||
|
<tr><td>Letzte erfolgreiche</td><td>{{ st.last_success|localdt if st.last_success else '–' }}</td></tr>
|
||||||
|
<tr><td>Datei</td><td class="mono">{{ st.last_file or '–' }}{% if st.last_size %} ({{ st.last_size|bytes }}){% endif %}</td></tr>
|
||||||
|
<tr><td>Letzter Versuch</td><td>{{ st.last_attempt|localdt if st.last_attempt else '–' }}</td></tr>
|
||||||
|
{% if st.last_error %}<tr><td>Letzter Fehler</td><td>{{ st.last_error }}{% if st.failures > 1 %} ({{ st.failures }}× in Folge){% endif %}</td></tr>{% endif %}
|
||||||
|
</tbody></table>
|
||||||
|
<form class="row" method="post" action="/backup/run">
|
||||||
|
{{ csrf_field(csrf) }}
|
||||||
|
<button {% if running or not cfg.backup_mount_id or not cfg.backup_passphrase_enc %}disabled{% endif %}>{{ 'Sicherung läuft …' if running else 'Jetzt sichern' }}</button>
|
||||||
|
<span class="hint">Kuma-Monitor: <code>{{ monitor_url }}</code></span>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2>Einstellungen <a class="legend-help" href="/help#sicherung" title="Hilfe">?</a></h2>
|
||||||
|
<form class="form card stack" method="post" action="/backup">
|
||||||
|
{{ csrf_field(csrf) }}
|
||||||
|
<div class="fields">
|
||||||
|
{{ check('backup_enabled', 'Tägliche Sicherung einschalten', cfg.backup_enabled, wide=true) }}
|
||||||
|
{{ select('backup_mount_id', 'Ziel-Anbindung', [(0, '– bitte wählen –')] + mounts, cfg.backup_mount_id, errors=errors, hint='SMB, NFS oder SFTP – am besten eine eigene Freigabe, getrennt von Scans und Archiv.') }}
|
||||||
|
{{ field('backup_path', 'Unterordner', cfg.backup_path, errors=errors, placeholder='z. B. netmover', hint='Wird angelegt, falls nötig. Leer = direkt in der Anbindung.') }}
|
||||||
|
{{ field('backup_time', 'Uhrzeit', cfg.backup_time, type='time', errors=errors, hint='Ortszeit des Servers. Verpasste Sicherungen werden nachgeholt, nach einem Fehler stündlich neuer Versuch.') }}
|
||||||
|
{{ field('backup_keep', 'Anzahl behalten', cfg.backup_keep, type='number', errors=errors, attrs='min="1" max="365"', hint='Ältere NetMover-Sicherungen im Ordner werden gelöscht – andere Dateien nie.') }}
|
||||||
|
<div class="field{{ ' has-error' if errors.get('passphrase') }}"><label for="f-passphrase">Passphrase</label>
|
||||||
|
<input id="f-passphrase" type="password" name="passphrase" autocomplete="new-password" minlength="12" placeholder="{{ '••••••' if cfg.backup_passphrase_enc }}">
|
||||||
|
{% if errors.get('passphrase') %}<span class="error">{{ errors['passphrase'] }}</span>{% endif %}
|
||||||
|
<span class="hint">{{ 'Leer lassen = unverändert. ' if cfg.backup_passphrase_enc }}Mindestens 12 Zeichen.</span></div>
|
||||||
|
<div class="field"><label for="f-passphrase2">Passphrase wiederholen</label>
|
||||||
|
<input id="f-passphrase2" type="password" name="passphrase_confirm" autocomplete="new-password"></div>
|
||||||
|
</div>
|
||||||
|
<p class="msg warn small">Die Passphrase gehört in deinen Passwortmanager. Fällt der Server aus, ist sie der einzige Weg an die Sicherung –
|
||||||
|
NetMover kann sie dann nicht mehr verraten. Eine neue Passphrase gilt für künftige Sicherungen; ältere Dateien brauchen weiter die alte.</p>
|
||||||
|
<div class="row"><button class="primary">Speichern</button></div>
|
||||||
|
</form>
|
||||||
|
|
||||||
|
<h2>Wiederherstellen</h2>
|
||||||
|
<div class="card stack small">
|
||||||
|
<p>Auf dem (neu installierten) Server, gleiche oder neuere NetMover-Version:</p>
|
||||||
|
<pre class="mono">systemctl stop netmover
|
||||||
|
netmover restore /pfad/zur/netmover-….nmbak
|
||||||
|
systemctl start netmover</pre>
|
||||||
|
<p class="muted">Die bisherigen Dateien bleiben in <code>/var/lib/netmover/vor-wiederherstellung-…</code> erhalten.
|
||||||
|
Ausführlich: <a href="/help#sicherung">Hilfe → Sicherung</a>.</p>
|
||||||
|
</div>
|
||||||
|
{% endblock %}
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
{% from "icons.html" import icon %}
|
||||||
<!doctype html>
|
<!doctype html>
|
||||||
<html lang="de">
|
<html lang="de">
|
||||||
<head>
|
<head>
|
||||||
@@ -10,27 +11,59 @@
|
|||||||
<script src="/static/htmx.min.js?v=2.0.4" defer></script>
|
<script src="/static/htmx.min.js?v=2.0.4" defer></script>
|
||||||
<script src="/static/app.js?v={{ version }}" defer></script>
|
<script src="/static/app.js?v={{ version }}" defer></script>
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body class="{{ 'with-nav' if logged_in else 'no-nav' }}">
|
||||||
{% if logged_in %}
|
{% if logged_in %}
|
||||||
<header class="top"><div class="inner">
|
{% macro item(key, href, label, badge='', badge_cls='') -%}
|
||||||
|
<a href="{{ href }}" class="{{ 'active' if active == key }}"{% if active == key %} aria-current="page"{% endif %}>
|
||||||
|
{{ icon(key) }}<span>{{ label }}</span>{% if badge %}<span class="count {{ badge_cls }}">{{ badge }}</span>{% endif %}</a>
|
||||||
|
{%- endmacro %}
|
||||||
|
<header class="topbar">
|
||||||
|
<button type="button" class="menu-toggle" aria-controls="sidebar" aria-expanded="false">{{ icon('menu') }}<span>Menü</span></button>
|
||||||
<a class="brand" href="/">NetMover</a>
|
<a class="brand" href="/">NetMover</a>
|
||||||
<nav class="main">
|
{% if nav %}
|
||||||
<a href="/" class="{{ 'active' if active == 'status' }}">Status</a>
|
<a class="health {{ 'ok' if nav.healthy else 'bad' }}" href="/" title="Gesundheitszustand">
|
||||||
<a href="/mounts" class="{{ 'active' if active == 'mounts' }}">Anbindungen</a>
|
<span class="dot"></span>{{ 'Alles in Ordnung' if nav.healthy else (nav.problems ~ (' Problem' if nav.problems == 1 else ' Probleme')) }}</a>
|
||||||
<a href="/rules" class="{{ 'active' if active == 'rules' }}">Regeln</a>
|
{% if auth_bypass %}<a class="health warn" href="/settings#anmeldung" title="Ohne Passwort bedienbar">{{ 'ohne Anmeldung' if auth_bypass == 'off' else 'ohne Anmeldung (vertrautes Netz)' }}</a>{% endif %}
|
||||||
<a href="/log" class="{{ 'active' if active == 'log' }}">Protokoll</a>
|
{% if nav.maintenance %}<a class="health warn" href="/" title="Geplante Läufe sind pausiert"><span class="dot"></span>Wartungsmodus</a>{% endif %}
|
||||||
<a href="/settings" class="{{ 'active' if active == 'settings' }}">Einstellungen</a>
|
{% endif %}
|
||||||
</nav>
|
<span class="spacer"></span>
|
||||||
<form class="inline" method="post" action="/logout">
|
{% if help_anchor %}<a class="help-link" href="/help#{{ help_anchor }}">{{ icon('help') }}<span>Hilfe zu dieser Seite</span></a>{% endif %}
|
||||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
</header>
|
||||||
<button class="small">Abmelden</button>
|
<nav id="sidebar" class="sidebar" aria-label="Hauptmenü">
|
||||||
</form>
|
<div class="nav-group">
|
||||||
</div></header>
|
<div class="nav-title">Übersicht</div>
|
||||||
|
{{ item('status', '/', 'Status') }}
|
||||||
|
</div>
|
||||||
|
<div class="nav-group">
|
||||||
|
<div class="nav-title">Verbindungen</div>
|
||||||
|
{{ item('mounts', '/mounts', 'Anbindungen', (nav.mounts_bad ~ ' gestört') if nav and nav.mounts_bad else (nav.mounts if nav and nav.mounts else ''), 'bad' if nav and nav.mounts_bad else '') }}
|
||||||
|
{{ item('keys', '/credentials', 'Zugangsdaten', nav.keys if nav and nav.keys else '') }}
|
||||||
|
</div>
|
||||||
|
<div class="nav-group">
|
||||||
|
<div class="nav-title">Automatisierung</div>
|
||||||
|
{{ item('rules', '/rules', 'Regeln', (nav.rules_active ~ '/' ~ nav.rules ~ ' aktiv') if nav and nav.rules else '') }}
|
||||||
|
{{ item('log', '/log', 'Protokoll') }}
|
||||||
|
</div>
|
||||||
|
<div class="nav-group">
|
||||||
|
<div class="nav-title">System</div>
|
||||||
|
{{ item('settings', '/settings', 'Einstellungen') }}
|
||||||
|
{{ item('backup', '/backup', 'Sicherung', '' if not nav or nav.backup_ok else 'überfällig', '' if not nav or nav.backup_ok else 'bad') }}
|
||||||
|
{{ item('help', '/help', 'Hilfe') }}
|
||||||
|
</div>
|
||||||
|
<div class="nav-foot">
|
||||||
|
{% if session_login %}
|
||||||
|
<form method="post" action="/logout">
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||||
|
<button class="nav-button">{{ icon('logout') }}<span>Abmelden</span></button>
|
||||||
|
</form>
|
||||||
|
{% endif %}
|
||||||
|
<div class="version">NetMover {{ version }}</div>
|
||||||
|
</div>
|
||||||
|
</nav>
|
||||||
{% endif %}
|
{% endif %}
|
||||||
<main>
|
<main>
|
||||||
{% for kind, text in flashes %}<p class="msg {{ kind }}">{{ text }}</p>{% endfor %}
|
{% for kind, text in flashes %}<p class="msg {{ kind }}">{{ text }}</p>{% endfor %}
|
||||||
{% block content %}{% endblock %}
|
{% block content %}{% endblock %}
|
||||||
</main>
|
</main>
|
||||||
<footer class="foot muted small">NetMover {{ version }}</footer>
|
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -0,0 +1,97 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
{% from "macros.html" import csrf_field %}
|
||||||
|
{% block title %}Zugangsdaten{% endblock %}
|
||||||
|
{% block content %}
|
||||||
|
<h1>Zugangsdaten</h1>
|
||||||
|
<p class="muted small">Einmal hinterlegen, in beliebig vielen Anbindungen auswählen. Passwörter und private Schlüssel
|
||||||
|
liegen verschlüsselt in der Datenbank und werden nie angezeigt.</p>
|
||||||
|
|
||||||
|
<h2>Passwörter <a class="legend-help" href="/help#zugangsdaten" title="Hilfe">?</a></h2>
|
||||||
|
<p class="muted small">Für SMB-Freigaben und SFTP-Server mit Passwort. Ein neues Passwort gilt sofort für alle Anbindungen, die es nutzen.</p>
|
||||||
|
{% if not passwords %}<p class="muted">Noch keine Passwörter gespeichert.</p>{% endif %}
|
||||||
|
<div class="stack">
|
||||||
|
{% for p in passwords %}
|
||||||
|
<div class="card" id="cred-{{ p.id }}">
|
||||||
|
<div class="row spread">
|
||||||
|
<div><strong>{{ p.name }}</strong> <span class="badge">{% if p.domain %}{{ p.domain }}\{% endif %}{{ p.username }}</span>
|
||||||
|
{% if p.used_by %}<span class="badge ok">benutzt von {{ p.used_by|join(', ') }}</span>{% else %}<span class="badge">unbenutzt</span>{% endif %}</div>
|
||||||
|
<div class="row">
|
||||||
|
<details class="rename"><summary class="btn small">Ändern</summary>
|
||||||
|
<form class="stack" method="post" action="/credentials/{{ p.id }}/password">{{ csrf_field(csrf) }}
|
||||||
|
<input type="text" name="username" value="{{ p.username }}" aria-label="Benutzer" required>
|
||||||
|
<input type="text" name="domain" value="{{ p.domain or '' }}" placeholder="Domäne (optional)" aria-label="Domäne">
|
||||||
|
<input type="password" name="password" placeholder="neues Passwort (leer = unverändert)" aria-label="Passwort" autocomplete="new-password">
|
||||||
|
<button class="small primary">Speichern</button></form></details>
|
||||||
|
<details class="rename"><summary class="btn small">Umbenennen</summary>
|
||||||
|
<form class="row" method="post" action="/credentials/{{ p.id }}/rename">{{ csrf_field(csrf) }}
|
||||||
|
<input type="text" name="name" value="{{ p.name }}" aria-label="Neuer Name" required maxlength="100">
|
||||||
|
<button class="small primary">Speichern</button></form></details>
|
||||||
|
<form class="inline" method="post" action="/credentials/{{ p.id }}/delete" data-confirm="„{{ p.name }}“ löschen?">{{ csrf_field(csrf) }}
|
||||||
|
<button class="small danger"{% if p.used_by %} disabled title="Wird noch benutzt"{% endif %}>Löschen</button></form>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="small muted">angelegt {{ p.created_at|localdt }}</div>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
</div>
|
||||||
|
<form class="form card stack" method="post" action="/credentials/passwords">{{ csrf_field(csrf) }}
|
||||||
|
<div class="fields">
|
||||||
|
<div class="field"><label for="p-name">Name</label><input id="p-name" type="text" name="name" placeholder="z. B. NAS-Scanner" required maxlength="100"></div>
|
||||||
|
<div class="field"><label for="p-user">Benutzer</label><input id="p-user" type="text" name="username" required autocomplete="off"></div>
|
||||||
|
<div class="field"><label for="p-pass">Passwort</label><input id="p-pass" type="password" name="password" required autocomplete="new-password"></div>
|
||||||
|
<div class="field"><label for="p-dom">Domäne (optional)</label><input id="p-dom" type="text" name="domain"></div>
|
||||||
|
</div>
|
||||||
|
<button class="primary">Passwort speichern</button>
|
||||||
|
</form>
|
||||||
|
|
||||||
|
<h2>SSH-Schlüssel</h2>
|
||||||
|
<p class="muted small">Für SFTP-Anbindungen. Ein Schlüssel kann von mehreren Anbindungen benutzt werden; den Benutzernamen legt jede Anbindung selbst fest.</p>
|
||||||
|
|
||||||
|
{% if not keys %}<p class="muted">Noch keine Schlüssel gespeichert.</p>{% endif %}
|
||||||
|
<div class="stack">
|
||||||
|
{% for k in keys %}
|
||||||
|
<div class="card" id="key-{{ k.id }}">
|
||||||
|
<div class="row spread">
|
||||||
|
<div><strong>{{ k.name }}</strong> <span class="badge">{{ k.key_type.removeprefix('ssh-') }}</span>
|
||||||
|
{% if k.used_by %}<span class="badge ok">benutzt von {{ k.used_by|join(', ') }}</span>{% else %}<span class="badge">unbenutzt</span>{% endif %}</div>
|
||||||
|
<div class="row">
|
||||||
|
<details class="rename"><summary class="btn small">Umbenennen</summary>
|
||||||
|
<form class="row" method="post" action="/credentials/{{ k.id }}/rename">{{ csrf_field(csrf) }}
|
||||||
|
<input type="text" name="name" value="{{ k.name }}" aria-label="Neuer Name" required maxlength="100">
|
||||||
|
<button class="small primary">Speichern</button></form></details>
|
||||||
|
<form class="inline" method="post" action="/credentials/{{ k.id }}/delete" data-confirm="Schlüssel „{{ k.name }}“ löschen?">{{ csrf_field(csrf) }}
|
||||||
|
<button class="small danger"{% if k.used_by %} disabled title="Wird noch benutzt"{% endif %}>Löschen</button></form>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="small muted">Fingerabdruck <span class="mono">{{ k.fingerprint }}</span> · angelegt {{ k.created_at|localdt }}</div>
|
||||||
|
<details class="small"><summary>Öffentlicher Schlüssel / authorized_keys</summary>
|
||||||
|
<div class="field"><label>Zeile für <code>~/.ssh/authorized_keys</code> (nur SFTP erlaubt, empfohlen)</label>
|
||||||
|
<pre class="mono card" id="auth-{{ k.id }}">{{ k.authorized_keys_line }}</pre>
|
||||||
|
<button class="small" type="button" data-copy="auth-{{ k.id }}">kopieren</button></div>
|
||||||
|
<div class="field"><label>Öffentlicher Schlüssel</label>
|
||||||
|
<pre class="mono card" id="pub-{{ k.id }}">{{ k.public_key }}</pre>
|
||||||
|
<button class="small" type="button" data-copy="pub-{{ k.id }}">kopieren</button></div>
|
||||||
|
</details>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h3>Neuen Schlüssel erzeugen</h3>
|
||||||
|
<form class="form card row" method="post" action="/credentials/keys/generate">{{ csrf_field(csrf) }}
|
||||||
|
<input class="w-auto" type="text" name="name" placeholder="Name, z. B. paperless-lxc" aria-label="Name" required maxlength="100">
|
||||||
|
<button class="primary">Ed25519-Schlüssel erzeugen</button>
|
||||||
|
</form>
|
||||||
|
|
||||||
|
<h3>Vorhandenen Schlüssel importieren</h3>
|
||||||
|
<form class="form card stack" method="post" action="/credentials/keys/import">{{ csrf_field(csrf) }}
|
||||||
|
<div class="fields">
|
||||||
|
<div class="field"><label for="k-name">Name</label><input id="k-name" type="text" name="name" required maxlength="100"></div>
|
||||||
|
<div class="field"><label for="k-pass">Passphrase (falls vorhanden)</label><input id="k-pass" type="password" name="passphrase" autocomplete="off"></div>
|
||||||
|
<div class="field wide"><label for="k-priv">Privater Schlüssel</label>
|
||||||
|
<textarea id="k-priv" name="private_key" class="mono" rows="8" autocomplete="off" spellcheck="false"
|
||||||
|
placeholder="-----BEGIN OPENSSH PRIVATE KEY----- … -----END OPENSSH PRIVATE KEY-----" required></textarea>
|
||||||
|
<span class="hint">Ed25519, ECDSA oder RSA (mind. 2048 Bit), OpenSSH- oder PEM-Format. Eine Passphrase wird nur zum Entschlüsseln benutzt.</span></div>
|
||||||
|
</div>
|
||||||
|
<button>Importieren</button>
|
||||||
|
</form>
|
||||||
|
{% endblock %}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
{% block title %}Hilfe{% endblock %}
|
||||||
|
{% block content %}
|
||||||
|
<article class="doc card">
|
||||||
|
{{ content|safe }}
|
||||||
|
</article>
|
||||||
|
{% endblock %}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
{# Symbole als Inline-SVG (keine Fremd-Ressourcen, CSP-verträglich). Pfade im Stil von Feather/Lucide. #}
|
||||||
|
{% macro icon(name) -%}
|
||||||
|
<svg class="ico" viewBox="0 0 24 24" aria-hidden="true" focusable="false">
|
||||||
|
{%- if name == 'status' -%}<path d="M3 12h4l3-8 4 16 3-8h4"/>
|
||||||
|
{%- elif name == 'mounts' -%}<rect x="3" y="4" width="18" height="7" rx="1.5"/><rect x="3" y="13" width="18" height="7" rx="1.5"/><path d="M7 7.5h.01M7 16.5h.01"/>
|
||||||
|
{%- elif name == 'keys' -%}<circle cx="8" cy="15" r="4"/><path d="M11 12l9-9M16 7l3 3M14 9l2 2"/>
|
||||||
|
{%- elif name == 'rules' -%}<path d="M4 7h13l-3-3M20 17H7l3 3"/>
|
||||||
|
{%- elif name == 'log' -%}<path d="M8 6h13M8 12h13M8 18h13M3 6h.01M3 12h.01M3 18h.01"/>
|
||||||
|
{%- elif name == 'settings' -%}<circle cx="12" cy="12" r="3"/><path d="M12 2v3M12 19v3M4.2 4.2l2.1 2.1M17.7 17.7l2.1 2.1M2 12h3M19 12h3M4.2 19.8l2.1-2.1M17.7 6.3l2.1-2.1"/>
|
||||||
|
{%- elif name == 'backup' -%}<path d="M4 7c0-1.7 3.6-3 8-3s8 1.3 8 3-3.6 3-8 3-8-1.3-8-3zM4 7v10c0 1.7 3.6 3 8 3s8-1.3 8-3V7M4 12c0 1.7 3.6 3 8 3s8-1.3 8-3"/>
|
||||||
|
{%- elif name == 'help' -%}<circle cx="12" cy="12" r="9"/><path d="M9.5 9a2.5 2.5 0 1 1 3.5 2.3c-.7.3-1 .9-1 1.7M12 17h.01"/>
|
||||||
|
{%- elif name == 'logout' -%}<path d="M15 4h4v16h-4M10 8l-4 4 4 4M6 12h11"/>
|
||||||
|
{%- elif name == 'menu' -%}<path d="M4 6h16M4 12h16M4 18h16"/>
|
||||||
|
{%- endif -%}
|
||||||
|
</svg>
|
||||||
|
{%- endmacro %}
|
||||||
@@ -21,8 +21,8 @@
|
|||||||
</div>
|
</div>
|
||||||
{%- endmacro %}
|
{%- endmacro %}
|
||||||
|
|
||||||
{% macro check(name, label, checked, hint='') -%}
|
{% macro check(name, label, checked, hint='', wide=false) -%}
|
||||||
<div class="field check">
|
<div class="field check{{ ' wide' if wide }}">
|
||||||
<input id="f-{{ name }}" type="checkbox" name="{{ name }}" {{ 'checked' if checked }}>
|
<input id="f-{{ name }}" type="checkbox" name="{{ name }}" {{ 'checked' if checked }}>
|
||||||
<label for="f-{{ name }}">{{ label }}</label>
|
<label for="f-{{ name }}">{{ label }}</label>
|
||||||
{% if hint %}<span class="hint">{{ hint }}</span>{% endif %}
|
{% if hint %}<span class="hint">{{ hint }}</span>{% endif %}
|
||||||
|
|||||||
@@ -1,15 +1,17 @@
|
|||||||
{% extends "base.html" %}
|
{% extends "base.html" %}
|
||||||
{% from "macros.html" import field, select, check %}
|
{% from "macros.html" import field, select, check %}
|
||||||
{% block title %}{{ 'Anbindung bearbeiten' if mount else 'Neue Anbindung' }}{% endblock %}
|
{% block title %}{{ 'Anbindung bearbeiten' if mount else ('Anbindung kopieren' if copy_from else 'Neue Anbindung') }}{% endblock %}
|
||||||
{% block content %}
|
{% block content %}
|
||||||
<h1>{{ 'Anbindung „' ~ mount.name ~ '“ bearbeiten' if mount else 'Neue Anbindung' }}</h1>
|
<h1>{% if mount %}Anbindung „{{ mount.name }}“ bearbeiten{% elif copy_from %}Kopie von „{{ copy_from.name }}“{% else %}Neue Anbindung{% endif %}</h1>
|
||||||
|
{% if copy_from %}<p class="msg info small">Alle Werte stammen aus „{{ copy_from.name }}“. Zugangsdaten werden übernommen, solange du das Passwort leer lässt – die Kopie bekommt ihr eigenes Exemplar.{% if copy_from.credential and copy_from.credential.public_key and not copy_from.credential.managed %} Der SSH-Schlüssel bleibt derselbe, ein neuer Eintrag in <code>authorized_keys</code> ist nicht nötig.{% endif %}</p>{% endif %}
|
||||||
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
||||||
<form class="form" method="post" data-dynamic>
|
<form class="form" method="post" data-dynamic>
|
||||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||||
{% if mount %}<input type="hidden" name="mount_id" value="{{ mount.id }}">{% endif %}
|
{% if mount %}<input type="hidden" name="mount_id" value="{{ mount.id }}">{% endif %}
|
||||||
<fieldset><legend>Verbindung</legend><div class="fields">
|
{% if copy_from %}<input type="hidden" name="copy_from" value="{{ copy_from.id }}">{% endif %}
|
||||||
{{ field('name', 'Name', d.name, errors=errors, attrs='required maxlength="100"', hint=('Verzeichnis: ' ~ mount.mountpoint if mount and mount.mountpoint else 'Bestimmt das Verzeichnis unter /mnt/netmover')) }}
|
<fieldset><legend>Verbindung <a class="legend-help" href="/help#anbindung-felder" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ select('type', 'Typ', [('cifs', 'SMB/CIFS'), ('nfs4', 'NFS 4'), ('nfs3', 'NFS 3'), ('sftp', 'SFTP (nur Ziel)')], d.type.value, errors=errors) }}
|
{{ field('name', 'Name', d.name, errors=errors, attrs='required maxlength="100"', hint=('Verzeichnis: ' ~ mount.mountpoint if mount and mount.mountpoint else '')) }}
|
||||||
|
{{ select('type', 'Typ', [('cifs', 'SMB/CIFS'), ('nfs4', 'NFS 4'), ('nfs3', 'NFS 3'), ('sftp', 'SFTP')], d.type.value, errors=errors) }}
|
||||||
{{ field('server', 'Server', d.server, errors=errors, placeholder='192.168.1.10 oder nas.lan', attrs='required') }}
|
{{ field('server', 'Server', d.server, errors=errors, placeholder='192.168.1.10 oder nas.lan', attrs='required') }}
|
||||||
<div class="field{{ ' has-error' if errors.get('share') }}">
|
<div class="field{{ ' has-error' if errors.get('share') }}">
|
||||||
<label for="f-share">Freigabe / Pfad</label>
|
<label for="f-share">Freigabe / Pfad</label>
|
||||||
@@ -22,29 +24,37 @@
|
|||||||
<div data-show-for="type:sftp">{{ field('port', 'SSH-Port', d.port or '', type='number', errors=errors, placeholder='22') }}</div>
|
<div data-show-for="type:sftp">{{ field('port', 'SSH-Port', d.port or '', type='number', errors=errors, placeholder='22') }}</div>
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
|
|
||||||
<fieldset data-show-for="type:cifs sftp"><legend>Anmeldung</legend><div class="fields">
|
<fieldset data-show-for="type:cifs sftp"><legend>Anmeldung <a class="legend-help" href="/help#anbindung-felder" title="Hilfe">?</a></legend><div class="fields">
|
||||||
<div data-show-for="type:sftp">{{ select('auth', 'Anmeldeart', [('ssh_key', 'SSH-Schlüssel (NetMover erzeugt ihn)'), ('password', 'Passwort')], d.auth.value, errors=errors) }}</div>
|
<div data-show-for="type:sftp">{{ select('auth', 'Anmeldeart', [('ssh_key', 'SSH-Schlüssel'), ('password', 'Passwort')], d.auth.value, errors=errors) }}</div>
|
||||||
{{ field('username', 'Benutzername', d.username, errors=errors, attrs='autocomplete="off"', hint='Bei CIFS leer lassen für Gastzugang') }}
|
<div data-show-for="type:sftp"><div data-show-for="auth:ssh_key">
|
||||||
<div data-show-for="type:cifs">{{ field('domain', 'Domäne (optional)', d.domain, errors=errors) }}</div>
|
{{ select('key_id', 'Schlüssel', key_options, d.key_id, errors=errors, hint='Verwalten unter „Zugangsdaten“.') }}
|
||||||
<div class="field{{ ' has-error' if errors.get('password') }}" id="pw-field">
|
</div></div>
|
||||||
|
<div data-show-for="type:cifs|auth:password">
|
||||||
|
{{ select('cred_id', 'Zugangsdaten', password_options, d.cred_id, errors=errors, hint='Gespeicherte Passwörter verwalten unter „Zugangsdaten“.') }}
|
||||||
|
</div>
|
||||||
|
<div data-show-for="auth:ssh_key|cred_id:own">
|
||||||
|
{{ field('username', 'Benutzername', d.username, errors=errors, attrs='autocomplete="off"') }}
|
||||||
|
</div>
|
||||||
|
<div data-show-for="cred_id:own"><div data-show-for="type:cifs">{{ field('domain', 'Domäne (optional)', d.domain, errors=errors) }}</div></div>
|
||||||
|
<div data-show-for="cred_id:own"><div class="field{{ ' has-error' if errors.get('password') }}" id="pw-field" data-show-for="type:cifs|auth:password">
|
||||||
<label for="f-password">Passwort</label>
|
<label for="f-password">Passwort</label>
|
||||||
<input id="f-password" type="password" name="password" autocomplete="new-password" placeholder="{{ '••••••' if has_secret }}">
|
<input id="f-password" type="password" name="password" autocomplete="new-password" placeholder="{{ '••••••' if has_secret }}">
|
||||||
{% if errors.get('password') %}<span class="error">{{ errors['password'] }}</span>{% endif %}
|
{% if errors.get('password') %}<span class="error">{{ errors['password'] }}</span>{% endif %}
|
||||||
<span class="hint">{{ 'Leer lassen = unverändert. ' if has_secret }}Bei SFTP mit Schlüssel leer lassen.</span>
|
<span class="hint">{{ 'Leer lassen = unverändert. ' if has_secret }}<span data-show-for="type:cifs">Benutzer und Passwort leer = Gastzugang.</span></span>
|
||||||
</div>
|
</div></div>
|
||||||
</div>
|
</div>
|
||||||
{% if public_key %}<div class="field wide"><label>Öffentlicher Schlüssel (in ~/.ssh/authorized_keys des Zielbenutzers eintragen)</label>
|
{% if public_key %}<div class="field wide"><label>Öffentlicher Schlüssel dieser Anbindung (in ~/.ssh/authorized_keys des Zielbenutzers eintragen)</label>
|
||||||
<pre class="mono card" id="pubkey">{{ public_key }}</pre><button class="small" type="button" data-copy="pubkey">kopieren</button></div>{% endif %}
|
<pre class="mono card" id="pubkey">{{ public_key }}</pre><button class="small" type="button" data-copy="pubkey">kopieren</button></div>{% endif %}
|
||||||
</fieldset>
|
</fieldset>
|
||||||
|
|
||||||
<fieldset data-show-for="type:cifs"><legend>Eigentümer und Rechte (CIFS)</legend><div class="fields">
|
<fieldset data-show-for="type:cifs"><legend>Eigentümer und Rechte (CIFS) <a class="legend-help" href="/help#anbindung-felder" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ field('uid', 'uid', d.uid, type='number', errors=errors, hint='Eigentümer aller Dateien im Mount') }}
|
{{ field('uid', 'uid', d.uid, type='number', errors=errors, hint='Eigentümer aller Dateien im Mount') }}
|
||||||
{{ field('gid', 'gid', d.gid, type='number', errors=errors) }}
|
{{ field('gid', 'gid', d.gid, type='number', errors=errors) }}
|
||||||
{{ field('file_mode', 'file_mode', d.file_mode, errors=errors, placeholder='0644') }}
|
{{ field('file_mode', 'file_mode', d.file_mode, errors=errors, placeholder='0644') }}
|
||||||
{{ field('dir_mode', 'dir_mode', d.dir_mode, errors=errors, placeholder='0755') }}
|
{{ field('dir_mode', 'dir_mode', d.dir_mode, errors=errors, placeholder='0755') }}
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
|
|
||||||
<fieldset data-show-for="type:cifs nfs3 nfs4"><legend>Erweitert</legend><div class="fields">
|
<fieldset data-show-for="type:cifs nfs3 nfs4"><legend>Erweitert <a class="legend-help" href="/help#anbindung-felder" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ field('extra_options', 'Zusatzoptionen', d.extra_options, errors=errors, wide=true, placeholder='vers=3.0,noserverino', hint='Kommagetrennt; nur erlaubte Optionen. Zugangsdaten setzt NetMover selbst.') }}
|
{{ field('extra_options', 'Zusatzoptionen', d.extra_options, errors=errors, wide=true, placeholder='vers=3.0,noserverino', hint='Kommagetrennt; nur erlaubte Optionen. Zugangsdaten setzt NetMover selbst.') }}
|
||||||
{{ check('auto_mount', 'Beim Start automatisch einhängen', d.auto_mount) }}
|
{{ check('auto_mount', 'Beim Start automatisch einhängen', d.auto_mount) }}
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
@@ -52,7 +62,7 @@
|
|||||||
|
|
||||||
<div class="row">
|
<div class="row">
|
||||||
<button class="primary">Speichern</button>
|
<button class="primary">Speichern</button>
|
||||||
<button type="button" hx-post="/mounts/test" hx-include="closest form" hx-target="#form-check" hx-swap="innerHTML">
|
<button type="button" title="Einhängen, lesen, schreiben – ohne zu speichern" hx-post="/mounts/test" hx-include="closest form" hx-target="#form-check" hx-swap="innerHTML">
|
||||||
Testen <span class="htmx-indicator">(läuft …)</span></button>
|
Testen <span class="htmx-indicator">(läuft …)</span></button>
|
||||||
<a class="btn" href="/mounts">Abbrechen</a>
|
<a class="btn" href="/mounts">Abbrechen</a>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -22,12 +22,14 @@
|
|||||||
{% endif %}
|
{% endif %}
|
||||||
{% endif %}
|
{% endif %}
|
||||||
<a class="btn small" href="/mounts/{{ m.id }}/edit">Bearbeiten</a>
|
<a class="btn small" href="/mounts/{{ m.id }}/edit">Bearbeiten</a>
|
||||||
|
<a class="btn small" href="/mounts/{{ m.id }}/copy" title="Neue Anbindung mit denselben Werten und Zugangsdaten">Kopieren</a>
|
||||||
<form class="inline" method="post" action="/mounts/{{ m.id }}/delete" data-confirm="„{{ m.name }}“ wirklich löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
<form class="inline" method="post" action="/mounts/{{ m.id }}/delete" data-confirm="„{{ m.name }}“ wirklich löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="muted small mono">{{ m.type.value }} · {% if m.type == MountType.CIFS %}//{{ m.server }}/{{ m.share }}{% elif m.type == MountType.SFTP %}{{ m.server }}:{{ m.port or 22 }}{{ m.share }}{% else %}{{ m.server }}:{{ m.share }}{% endif %}
|
<div class="muted small mono">{{ m.type.value }} · {% if m.type == MountType.CIFS %}//{{ m.server }}/{{ m.share }}{% elif m.type == MountType.SFTP %}{{ m.server }}:{{ m.port or 22 }}{{ m.share }}{% else %}{{ m.server }}:{{ m.share }}{% endif %}
|
||||||
{% if m.mountpoint %} → {{ m.mountpoint }}{% endif %}</div>
|
{% if m.mountpoint %} → {{ m.mountpoint }}{% endif %}</div>
|
||||||
{% if m.last_error %}<pre class="small text-err">{{ m.last_error }}</pre>{% endif %}
|
{% if m.last_error %}<pre class="small text-err">{{ m.last_error }}</pre>{% endif %}
|
||||||
|
{% if key_names.get(m.id) %}<div class="small">Anmeldung mit gespeicherten Zugangsdaten <a href="/credentials">„{{ key_names[m.id] }}“</a></div>{% endif %}
|
||||||
{% if m.type == MountType.SFTP %}
|
{% if m.type == MountType.SFTP %}
|
||||||
<div class="small">Hostschlüssel: {% if m.host_key %}<span class="mono">{{ m.host_key.split()[0] }}</span> festgehalten
|
<div class="small">Hostschlüssel: {% if m.host_key %}<span class="mono">{{ m.host_key.split()[0] }}</span> festgehalten
|
||||||
<form class="inline" method="post" action="/mounts/{{ m.id }}/reset-hostkey" data-confirm="Hostschlüssel zurücksetzen? Nur tun, wenn der Server neu aufgesetzt wurde.">{{ csrf_field(csrf) }}<button class="small">zurücksetzen</button></form>
|
<form class="inline" method="post" action="/mounts/{{ m.id }}/reset-hostkey" data-confirm="Hostschlüssel zurücksetzen? Nur tun, wenn der Server neu aufgesetzt wurde.">{{ csrf_field(csrf) }}<button class="small">zurücksetzen</button></form>
|
||||||
|
|||||||
@@ -33,17 +33,19 @@
|
|||||||
<thead><tr><th>Regel</th><th>Letztes Ergebnis</th><th>Letzter Erfolg</th><th>Nächster Lauf</th><th></th></tr></thead>
|
<thead><tr><th>Regel</th><th>Letztes Ergebnis</th><th>Letzter Erfolg</th><th>Nächster Lauf</th><th></th></tr></thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
{% for v in rules %}{% set r = v.r %}
|
{% for v in rules %}{% set r = v.r %}
|
||||||
<tr class="{{ 'warn' if v.stale }}">
|
<tr class="{{ 'warn' if v.stale or r.waiting_since }}">
|
||||||
<td data-label="Regel"><strong>{{ r.name }}</strong>{% if not r.enabled %} <span class="badge">aus</span>{% endif %}{% if r.dry_run %} <span class="badge warn">Probelauf</span>{% endif %}
|
<td data-label="Regel"><strong>{{ r.name }}</strong>{% if not r.enabled %} <span class="badge">aus</span>{% endif %}{% if r.dry_run %} <span class="badge warn">Probelauf</span>{% endif %}
|
||||||
<div class="muted small mono">{{ v.target }}</div></td>
|
<div class="muted small mono">{{ v.target }}</div></td>
|
||||||
<td data-label="Letztes Ergebnis">
|
<td data-label="Letztes Ergebnis">
|
||||||
{% if v.running_since %}<span class="badge warn">läuft seit {{ v.running_since|ago|replace('vor ', '') }}</span>
|
{% if r.waiting_since %}<span class="badge warn">wartet seit {{ r.waiting_since|ago|replace('vor ', '') }}</span>
|
||||||
|
<div class="small text-warn">{{ r.waiting_reason }}</div>
|
||||||
|
{% elif v.running_since %}<span class="badge warn">läuft seit {{ v.running_since|ago|replace('vor ', '') }}</span>
|
||||||
{% elif v.last_run %}{{ run_badge(v.last_run.status) }}
|
{% elif v.last_run %}{{ run_badge(v.last_run.status) }}
|
||||||
<span class="small muted">{{ v.last_run.finished_at|ago }}{% if v.last_run.seen %} · {{ v.last_run.moved }}/{{ v.last_run.seen }}{% endif %}</span>
|
<span class="small muted">{{ v.last_run.finished_at|ago }}{% if v.last_run.seen %} · {{ v.last_run.moved }}/{{ v.last_run.seen }}{% endif %}</span>
|
||||||
{% if v.last_run.status in (RunStatus.ERROR, RunStatus.PARTIAL) and v.last_run.message %}<div class="small text-err">{{ v.last_run.message }}</div>{% endif %}
|
{% if v.last_run.status in (RunStatus.ERROR, RunStatus.PARTIAL) and v.last_run.message %}<div class="small text-err">{{ v.last_run.message }}</div>{% endif %}
|
||||||
{% else %}<span class="muted">–</span>{% endif %}</td>
|
{% else %}<span class="muted">–</span>{% endif %}</td>
|
||||||
<td data-label="Letzter Erfolg">{{ r.last_success_at|ago }}{% if v.stale %}<div class="small text-warn">länger als 3 × Intervall!</div>{% endif %}
|
<td data-label="Letzter Erfolg">{{ r.last_success_at|ago }}{% if v.stale %}<div class="small text-warn">länger als 3 × Intervall!</div>{% endif %}
|
||||||
{% if r.consecutive_failures %}<div class="small text-err">{{ r.consecutive_failures }} Fehler in Folge</div>{% endif %}</td>
|
{% if r.consecutive_failures and not r.waiting_since %}<div class="small text-err">{{ r.consecutive_failures }} Fehler in Folge</div>{% endif %}</td>
|
||||||
<td data-label="Nächster Lauf">{{ v.next_run|ago if v.next_run else '–' }}</td>
|
<td data-label="Nächster Lauf">{{ v.next_run|ago if v.next_run else '–' }}</td>
|
||||||
<td><form class="inline" method="post" action="/rules/{{ r.id }}/run">
|
<td><form class="inline" method="post" action="/rules/{{ r.id }}/run">
|
||||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||||
|
|||||||
@@ -1,20 +1,23 @@
|
|||||||
{% extends "base.html" %}
|
{% extends "base.html" %}
|
||||||
{% from "macros.html" import field, select, check %}
|
{% from "macros.html" import field, select, check %}
|
||||||
{% block title %}{{ 'Regel bearbeiten' if rule_id else 'Neue Regel' }}{% endblock %}
|
{% block title %}{{ 'Regel bearbeiten' if rule_id else ('Regel kopieren' if copy_of else 'Neue Regel') }}{% endblock %}
|
||||||
{% block content %}
|
{% block content %}
|
||||||
<h1>{{ 'Regel bearbeiten' if rule_id else 'Neue Regel' }}</h1>
|
<h1>{% if rule_id %}Regel bearbeiten{% elif copy_of %}Kopie von „{{ copy_of }}“{% else %}Neue Regel{% endif %}</h1>
|
||||||
|
{% if copy_of %}<p class="msg info small">Alle Einstellungen stammen aus „{{ copy_of }}“. Die Kopie ist zunächst
|
||||||
|
<strong>deaktiviert</strong>, damit sie nicht mit dem Original um dieselben Dateien konkurriert – Quelle bzw. Ziel
|
||||||
|
anpassen, speichern, Probelauf machen, dann aktivieren. Protokoll und Buchführung beginnen bei der Kopie neu.</p>{% endif %}
|
||||||
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
||||||
{% if not src_mounts %}<p class="msg info">Zuerst eine SMB- oder NFS-Anbindung als Quelle <a href="/mounts/new">anlegen</a>.</p>{% endif %}
|
{% if not src_mounts %}<p class="msg info">Zuerst eine Anbindung als Quelle <a href="/mounts/new">anlegen</a>.</p>{% endif %}
|
||||||
<form class="form" method="post" data-dynamic>
|
<form class="form" method="post" data-dynamic>
|
||||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||||
<fieldset><legend>Allgemein</legend><div class="fields">
|
<fieldset><legend>Allgemein <a class="legend-help" href="/help#regeln" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ field('name', 'Name', d.name, errors=errors, attrs='required maxlength="100"') }}
|
{{ field('name', 'Name', d.name, errors=errors, attrs='required maxlength="100"') }}
|
||||||
{{ field('interval_seconds', 'Intervall (Sekunden)', d.interval_seconds, type='number', errors=errors, attrs='min="10" required', hint='mindestens 10') }}
|
{{ field('interval_seconds', 'Intervall (Sekunden)', d.interval_seconds, type='number', errors=errors, attrs='min="10" required', hint='mindestens 10') }}
|
||||||
{{ check('enabled', 'Aktiv', d.enabled) }}
|
{{ check('enabled', 'Aktiv', d.enabled) }}
|
||||||
{{ check('dry_run', 'Nur Probelauf (nichts verändern)', d.dry_run) }}
|
{{ check('dry_run', 'Nur Probelauf (nichts verändern)', d.dry_run) }}
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
|
|
||||||
<fieldset><legend>Quelle</legend><div class="fields">
|
<fieldset><legend>Quelle <a class="legend-help" href="/help#regel-quelle-ziel" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ select('src_mount_id', 'Anbindung', src_options, d.src_mount_id or '', errors=errors) }}
|
{{ select('src_mount_id', 'Anbindung', src_options, d.src_mount_id or '', errors=errors) }}
|
||||||
{{ field('src_subpath', 'Unterordner', d.src_subpath, errors=errors, placeholder='z. B. Scans/Max', hint='relativ zur Freigabe; leer = Wurzel') }}
|
{{ field('src_subpath', 'Unterordner', d.src_subpath, errors=errors, placeholder='z. B. Scans/Max', hint='relativ zur Freigabe; leer = Wurzel') }}
|
||||||
{{ select('pattern_type', 'Filterart', [('glob', 'Glob'), ('regex', 'Regulärer Ausdruck')], d.pattern_type.value, errors=errors) }}
|
{{ select('pattern_type', 'Filterart', [('glob', 'Glob'), ('regex', 'Regulärer Ausdruck')], d.pattern_type.value, errors=errors) }}
|
||||||
@@ -22,7 +25,7 @@
|
|||||||
{{ check('recursive', 'Unterordner einbeziehen (Struktur bleibt erhalten)', d.recursive) }}
|
{{ check('recursive', 'Unterordner einbeziehen (Struktur bleibt erhalten)', d.recursive) }}
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
|
|
||||||
<fieldset><legend>Ziel</legend><div class="fields">
|
<fieldset><legend>Hauptziel <a class="legend-help" href="/help#regel-quelle-ziel" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ select('dst_kind', 'Zielart', [('mount', 'Anbindung'), ('local', 'Lokaler Pfad')], d.dst_kind, errors=errors) }}
|
{{ select('dst_kind', 'Zielart', [('mount', 'Anbindung'), ('local', 'Lokaler Pfad')], d.dst_kind, errors=errors) }}
|
||||||
<div data-show-for="dst_kind:mount">{{ select('dst_mount_id', 'Anbindung', dst_options, d.dst_mount_id or '', errors=errors) }}</div>
|
<div data-show-for="dst_kind:mount">{{ select('dst_mount_id', 'Anbindung', dst_options, d.dst_mount_id or '', errors=errors) }}</div>
|
||||||
{{ field('dst_path', 'Zielordner', d.dst_path, errors=errors, hint='Anbindung: relativer Unterordner (wird angelegt). Lokal: absoluter Pfad, muss existieren.') }}
|
{{ field('dst_path', 'Zielordner', d.dst_path, errors=errors, hint='Anbindung: relativer Unterordner (wird angelegt). Lokal: absoluter Pfad, muss existieren.') }}
|
||||||
@@ -30,14 +33,31 @@
|
|||||||
{{ select('on_collision', 'Wenn Datei existiert', [('rename', 'umbenennen (scan-1.pdf)'), ('skip', 'überspringen'), ('overwrite', 'überschreiben')], d.on_collision.value, errors=errors) }}
|
{{ select('on_collision', 'Wenn Datei existiert', [('rename', 'umbenennen (scan-1.pdf)'), ('skip', 'überspringen'), ('overwrite', 'überschreiben')], d.on_collision.value, errors=errors) }}
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
|
|
||||||
<fieldset><legend>Sicherheit und Grenzen</legend><div class="fields">
|
<fieldset><legend>Weitere Ziele <a class="legend-help" href="/help#weitere-ziele" title="Hilfe">?</a></legend>
|
||||||
|
<p class="hint">Zusätzliche Kopien, z. B. Archiv oder Buchhaltung – bis zu {{ max_targets }}. Sie werden vor dem Hauptziel beliefert;
|
||||||
|
die Quelle wird erst gelöscht, wenn <strong>alle</strong> Ziele die Datei haben. Weitere Ziele überschreiben nie, gleiche Namen werden umbenannt.</p>
|
||||||
|
{% for i in range(max_targets) %}
|
||||||
|
{% set t = d.targets[i] if i < d.targets|length else none %}
|
||||||
|
<div class="target-slot"{% if i > 0 %} data-show-for="target_{{ i - 1 }}_kind:mount local|target_{{ i }}_kind:mount local"{% endif %}>
|
||||||
|
<div class="fields">
|
||||||
|
{{ select('target_' ~ i ~ '_kind', 'Ziel ' ~ (i + 2), [('', '– kein weiteres Ziel –'), ('mount', 'Anbindung'), ('local', 'Lokaler Pfad')], t.kind if t else '', errors=errors) }}
|
||||||
|
<div data-show-for="target_{{ i }}_kind:mount local">{{ field('target_' ~ i ~ '_name', 'Bezeichnung', t.name if t else '', errors=errors, placeholder='z. B. Archiv, Buchhaltung', attrs='maxlength="100"', hint='erscheint im Protokoll; leer = Name der Anbindung') }}</div>
|
||||||
|
<div data-show-for="target_{{ i }}_kind:mount">{{ select('target_' ~ i ~ '_mount_id', 'Anbindung', dst_options, (t.mount_id if t and t.mount_id else ''), errors=errors) }}</div>
|
||||||
|
<div data-show-for="target_{{ i }}_kind:mount local">{{ field('target_' ~ i ~ '_path', 'Ordner', t.path if t else '', errors=errors, hint='Anbindung: relativer Unterordner (wird angelegt). Lokal: absoluter Pfad, muss existieren.') }}</div>
|
||||||
|
<div data-show-for="target_{{ i }}_kind:mount local">{{ select('target_' ~ i ~ '_layout', 'Ablage', [('flat', 'wie in der Quelle'), ('date', 'nach Datum (Jahr/Monat)')], t.layout.value if t else 'flat', errors=errors) }}</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
</fieldset>
|
||||||
|
|
||||||
|
<fieldset><legend>Sicherheit und Grenzen <a class="legend-help" href="/help#regel-sicherheit" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ field('stability_seconds', 'Stabilitätsfenster (s)', d.stability_seconds, type='number', errors=errors, attrs='min="0"', hint='Größe und Zeit müssen so lange gleich bleiben') }}
|
{{ field('stability_seconds', 'Stabilitätsfenster (s)', d.stability_seconds, type='number', errors=errors, attrs='min="0"', hint='Größe und Zeit müssen so lange gleich bleiben') }}
|
||||||
{{ field('min_age_seconds', 'Mindestalter (s)', d.min_age_seconds, type='number', errors=errors, attrs='min="0"') }}
|
{{ field('min_age_seconds', 'Mindestalter (s)', d.min_age_seconds, type='number', errors=errors, attrs='min="0"') }}
|
||||||
{{ field('max_size_mb', 'Max. Größe (MB)', d.max_size_mb or '', type='number', errors=errors, hint='leer = unbegrenzt') }}
|
{{ field('max_size_mb', 'Max. Größe (MB)', d.max_size_mb or '', type='number', errors=errors, hint='leer = unbegrenzt') }}
|
||||||
{{ field('op_timeout_seconds', 'Zeitlimit ohne Fortschritt (s)', d.op_timeout_seconds, type='number', errors=errors, attrs='min="10"') }}
|
{{ field('op_timeout_seconds', 'Zeitlimit ohne Fortschritt (s)', d.op_timeout_seconds, type='number', errors=errors, attrs='min="10"') }}
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
|
|
||||||
<fieldset><legend>Rechte und Quarantäne</legend><div class="fields">
|
<fieldset><legend>Rechte und Quarantäne <a class="legend-help" href="/help#regel-rechte" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ field('set_owner', 'Eigentümer setzen', d.set_owner, errors=errors, placeholder='1000:1000', hint='leer = unverändert; auf CIFS wirkungslos') }}
|
{{ field('set_owner', 'Eigentümer setzen', d.set_owner, errors=errors, placeholder='1000:1000', hint='leer = unverändert; auf CIFS wirkungslos') }}
|
||||||
{{ field('set_mode', 'Modus setzen', d.set_mode, errors=errors, placeholder='0644', hint='leer = wie Quelle') }}
|
{{ field('set_mode', 'Modus setzen', d.set_mode, errors=errors, placeholder='0644', hint='leer = wie Quelle') }}
|
||||||
{{ field('quarantine_path', 'Quarantäneordner', d.quarantine_path, errors=errors, placeholder='_quarantaene', hint='relativ zur Quell-Anbindung; leer = keine Quarantäne') }}
|
{{ field('quarantine_path', 'Quarantäneordner', d.quarantine_path, errors=errors, placeholder='_quarantaene', hint='relativ zur Quell-Anbindung; leer = keine Quarantäne') }}
|
||||||
|
|||||||
@@ -18,10 +18,13 @@
|
|||||||
<button class="small" hx-post="/rules/{{ r.id }}/dryrun" hx-target="#dry-{{ r.id }}" hx-swap="innerHTML">Probelauf <span class="htmx-indicator">…</span></button>
|
<button class="small" hx-post="/rules/{{ r.id }}/dryrun" hx-target="#dry-{{ r.id }}" hx-swap="innerHTML">Probelauf <span class="htmx-indicator">…</span></button>
|
||||||
<form class="inline" method="post" action="/rules/{{ r.id }}/toggle">{{ csrf_field(csrf) }}<button class="small">{{ 'Deaktivieren' if r.enabled else 'Aktivieren' }}</button></form>
|
<form class="inline" method="post" action="/rules/{{ r.id }}/toggle">{{ csrf_field(csrf) }}<button class="small">{{ 'Deaktivieren' if r.enabled else 'Aktivieren' }}</button></form>
|
||||||
<a class="btn small" href="/rules/{{ r.id }}/edit">Bearbeiten</a>
|
<a class="btn small" href="/rules/{{ r.id }}/edit">Bearbeiten</a>
|
||||||
|
<a class="btn small" href="/rules/{{ r.id }}/copy" title="Neue Regel mit denselben Einstellungen">Kopieren</a>
|
||||||
<form class="inline" method="post" action="/rules/{{ r.id }}/delete" data-confirm="Regel „{{ r.name }}“ und ihr Protokoll löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
<form class="inline" method="post" action="/rules/{{ r.id }}/delete" data-confirm="Regel „{{ r.name }}“ und ihr Protokoll löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="muted small mono">{{ v.target }} · Filter {{ r.pattern }}{{ ' (rekursiv)' if r.recursive }}</div>
|
<div class="muted small mono">{{ v.target }} · Filter {{ r.pattern }}{{ ' (rekursiv)' if r.recursive }}</div>
|
||||||
|
{% if r.waiting_since %}<p class="msg warn small">Wartet seit {{ r.waiting_since|localdt }} ({{ r.waiting_since|ago }}) – {{ r.waiting_reason }}
|
||||||
|
Die Regel läuft von selbst weiter, sobald die Anbindung zurück ist.</p>{% endif %}
|
||||||
<div class="small">Letzter Lauf: {% if v.running_since %}<span class="badge warn">läuft seit {{ v.running_since|ago|replace('vor ', '') }}</span>{% elif v.last_run %}{{ run_badge(v.last_run.status) }} {{ v.last_run.finished_at|ago }}{% else %}–{% endif %}
|
<div class="small">Letzter Lauf: {% if v.running_since %}<span class="badge warn">läuft seit {{ v.running_since|ago|replace('vor ', '') }}</span>{% elif v.last_run %}{{ run_badge(v.last_run.status) }} {{ v.last_run.finished_at|ago }}{% else %}–{% endif %}
|
||||||
· letzter Erfolg: <span class="{{ 'text-warn' if v.stale }}">{{ r.last_success_at|ago }}</span>
|
· letzter Erfolg: <span class="{{ 'text-warn' if v.stale }}">{{ r.last_success_at|ago }}</span>
|
||||||
· nächster: {{ v.next_run|ago if v.next_run else '–' }}</div>
|
· nächster: {{ v.next_run|ago if v.next_run else '–' }}</div>
|
||||||
|
|||||||
@@ -8,18 +8,49 @@
|
|||||||
<fieldset><legend>Protokoll</legend><div class="fields">
|
<fieldset><legend>Protokoll</legend><div class="fields">
|
||||||
{{ field('retention_days', 'Aufbewahrung (Tage)', cfg.retention_days, type='number', errors=errors, attrs='min="1" max="3650"') }}
|
{{ field('retention_days', 'Aufbewahrung (Tage)', cfg.retention_days, type='number', errors=errors, attrs='min="1" max="3650"') }}
|
||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
<fieldset><legend>Benachrichtigung (Webhook)</legend><div class="fields">
|
<fieldset><legend>Benachrichtigungen <a class="legend-help" href="/help#benachrichtigungen" title="Hilfe">?</a></legend><div class="fields">
|
||||||
{{ select('notify_mode', 'Benachrichtigen', [('off', 'nie'), ('every', 'bei jedem fehlgeschlagenen Lauf'), ('after_n', 'nach N Fehlschlägen in Folge')], cfg.notify_mode.value, errors=errors) }}
|
{{ select('notify_mode', 'Benachrichtigen', [('off', 'nie'), ('every', 'bei jedem fehlgeschlagenen Lauf'), ('after_n', 'nach N Fehlschlägen in Folge')], cfg.notify_mode.value, errors=errors, hint='Gilt für Webhook und E-Mail. Ausfälle von Anbindungen werden immer sofort gemeldet (außer bei „nie“).') }}
|
||||||
<div data-show-for="notify_mode:after_n">{{ field('notify_after', 'N', cfg.notify_after, type='number', errors=errors, attrs='min="1"') }}</div>
|
<div data-show-for="notify_mode:after_n">{{ field('notify_after', 'N', cfg.notify_after, type='number', errors=errors, attrs='min="1"') }}</div>
|
||||||
{{ field('webhook_url', 'Webhook-URL', mask_url(cfg.webhook_url), errors=errors, wide=true, placeholder='https://ntfy.sh/mein-topic', hint='Wird nur gekürzt angezeigt; unverändert lassen = behalten, leeren = entfernen.') }}
|
{{ check('notify_recovery', 'Entwarnung senden, wenn es wieder läuft', cfg.notify_recovery, wide=true) }}
|
||||||
|
</div></fieldset>
|
||||||
|
|
||||||
|
<fieldset><legend>Webhook</legend><div class="fields">
|
||||||
|
{{ field('webhook_url', 'Webhook-URL', mask_url(cfg.webhook_url), errors=errors, wide=true, placeholder='https://ntfy.sh/mein-topic', hint='Leer = kein Webhook. Wird nur gekürzt angezeigt; unverändert lassen = behalten.') }}
|
||||||
{{ field('webhook_content_type', 'Content-Type', cfg.webhook_content_type, errors=errors) }}
|
{{ field('webhook_content_type', 'Content-Type', cfg.webhook_content_type, errors=errors) }}
|
||||||
{{ check('notify_recovery', 'Entwarnung senden, wenn es wieder läuft', cfg.notify_recovery) }}
|
|
||||||
<div class="field wide"><label for="f-tpl">Vorlage</label>
|
<div class="field wide"><label for="f-tpl">Vorlage</label>
|
||||||
<textarea id="f-tpl" name="webhook_template" class="mono">{{ cfg.webhook_template }}</textarea>
|
<textarea id="f-tpl" name="webhook_template" class="mono">{{ cfg.webhook_template }}</textarea>
|
||||||
<span class="hint">Platzhalter: $event $title $message $rule $time $host – bei JSON automatisch maskiert.</span></div>
|
<span class="hint">Platzhalter: $event $title $message $rule $time $host – bei JSON automatisch maskiert.</span></div>
|
||||||
</div>
|
</div>
|
||||||
<div class="row"><button type="button" hx-post="/settings/webhook-test" hx-target="#hook-result">Test senden (gespeicherte Einstellungen)</button><span id="hook-result"></span></div>
|
<div class="row"><button type="button" hx-post="/settings/webhook-test" hx-target="#hook-result">Test senden (gespeicherte Einstellungen)</button><span id="hook-result"></span></div>
|
||||||
</fieldset>
|
</fieldset>
|
||||||
|
|
||||||
|
<fieldset><legend>E-Mail</legend><div class="fields">
|
||||||
|
{{ check('email_enabled', 'E-Mail-Benachrichtigung einschalten', cfg.email_enabled, wide=true) }}
|
||||||
|
{{ field('smtp_host', 'SMTP-Server', cfg.smtp_host, errors=errors, placeholder='smtp.example.org') }}
|
||||||
|
{{ field('smtp_port', 'Port', cfg.smtp_port, type='number', errors=errors, attrs='min="1" max="65535"', hint='587 (STARTTLS) oder 465 (SSL)') }}
|
||||||
|
{{ select('smtp_security', 'Verschlüsselung', [('starttls', 'STARTTLS'), ('ssl', 'SSL/TLS'), ('none', 'keine (nur interne Relays)')], cfg.smtp_security, errors=errors) }}
|
||||||
|
{{ field('smtp_user', 'Benutzer', cfg.smtp_user, errors=errors, attrs='autocomplete="off"', hint='leer = ohne Anmeldung') }}
|
||||||
|
<div class="field{{ ' has-error' if errors.get('smtp_password') }}"><label for="f-smtp_password">Passwort</label>
|
||||||
|
<input id="f-smtp_password" type="password" name="smtp_password" autocomplete="new-password" placeholder="{{ '••••••' if cfg.smtp_password_enc }}">
|
||||||
|
<span class="hint">{{ 'Leer lassen = unverändert.' if cfg.smtp_password_enc else '' }} Wird verschlüsselt gespeichert.</span></div>
|
||||||
|
{{ field('smtp_from', 'Absender', cfg.smtp_from, errors=errors, placeholder='netmover@example.org') }}
|
||||||
|
{{ field('smtp_to', 'Empfänger', cfg.smtp_to, errors=errors, wide=true, placeholder='ich@example.org, admin@example.org', hint='mehrere mit Komma getrennt') }}
|
||||||
|
</div>
|
||||||
|
<div class="row"><button type="button" hx-post="/settings/mail-test" hx-target="#mail-result">Test-E-Mail senden (gespeicherte Einstellungen)</button><span id="mail-result"></span></div>
|
||||||
|
</fieldset>
|
||||||
|
|
||||||
|
<fieldset><legend>Überwachung (Uptime Kuma) <a class="legend-help" href="/help#uptime-kuma" title="Hilfe">?</a></legend>
|
||||||
|
<div class="fields">
|
||||||
|
{{ field('kuma_push_url', 'Push-URL (Kuma-Monitor vom Typ „Push“)', mask_url(cfg.kuma_push_url), errors=errors, wide=true, placeholder='https://kuma.lan/api/push/AbCdEf?status=up&msg=OK&ping=', hint='NetMover meldet jede Minute den Zustand. Leer = aus. Wird nur gekürzt angezeigt.') }}
|
||||||
|
</div>
|
||||||
|
<div class="row"><button type="button" hx-post="/settings/kuma-test" hx-target="#kuma-result">Push jetzt testen (gespeicherte URL)</button><span id="kuma-result"></span></div>
|
||||||
|
<details class="small"><summary>Adressen für HTTP-Monitore (Kuma fragt NetMover ab)</summary>
|
||||||
|
<table class="small"><tbody>
|
||||||
|
{% for label, url in monitor_urls %}<tr><td>{{ label }}</td><td class="mono">{{ url }}</td></tr>{% endfor %}
|
||||||
|
</tbody></table>
|
||||||
|
<p class="muted small">200 = in Ordnung, 503 = Problem. Ohne Anmeldung erreichbar.</p>
|
||||||
|
</details>
|
||||||
|
</fieldset>
|
||||||
<button class="primary">Speichern</button>
|
<button class="primary">Speichern</button>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
@@ -34,6 +65,27 @@
|
|||||||
<button>Passwort ändern</button>
|
<button>Passwort ändern</button>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
|
<h2 id="anmeldung">Anmeldung <a class="legend-help" href="/help#anmeldung" title="Hilfe">?</a></h2>
|
||||||
|
<form class="form card stack" method="post" action="/settings/auth">
|
||||||
|
{{ csrf_field(csrf) }}
|
||||||
|
<p class="small muted">Deine Adresse gerade: <code>{{ client_ip }}</code></p>
|
||||||
|
<div class="fields">
|
||||||
|
<div class="field wide"><label for="f-nets">Ohne Passwort aus diesen Netzen</label>
|
||||||
|
<input id="f-nets" type="text" name="trusted_networks" value="{{ cfg.trusted_networks }}" placeholder="z. B. 10.0.10.0/24, 10.1.5.0/24">
|
||||||
|
<span class="hint">Mit Komma getrennt; einzelne Adressen gehen auch. Leer = niemand ohne Passwort. <code>0.0.0.0/0, ::/0</code> = Anmeldung ganz aus.</span></div>
|
||||||
|
<div class="field wide"><label for="f-allowed">Mit Passwort aus diesen Netzen</label>
|
||||||
|
<input id="f-allowed" type="text" name="allowed_networks" value="{{ cfg.allowed_networks }}" placeholder="leer = alle übrigen Netze">
|
||||||
|
<span class="hint">Leer = von überall sonst mit Passwort. Ausgefüllt = <strong>alle anderen Netze werden abgewiesen</strong> (auch die Anmeldeseite). <code>/healthz</code> bleibt für die Überwachung erreichbar.</span></div>
|
||||||
|
<div class="field"><label for="f-cur-auth">Aktuelles Passwort (zur Bestätigung)</label>
|
||||||
|
<input id="f-cur-auth" type="password" name="current" autocomplete="current-password" required></div>
|
||||||
|
</div>
|
||||||
|
<p class="msg warn small">NetMover läuft als root und kennt die Zugangsdaten deiner Freigaben. Wer die Oberfläche ohne Passwort erreicht,
|
||||||
|
kann alles ändern – Netze „ohne Passwort“ daher so eng wie möglich wählen. Hinter einem Reverse Proxy <code>NETMOVER_FORWARDED_ALLOW_IPS</code> setzen,
|
||||||
|
sonst sieht jede Anfrage aus, als käme sie vom Proxy. Eine Einstellung, mit der du dich selbst aussperren würdest, wird nicht gespeichert.</p>
|
||||||
|
{% if loopback_trusted %}<p class="msg warn small">„Ohne Passwort“ enthält 127.0.0.1/::1. Läuft ein Reverse Proxy auf diesem Server, gilt dann <strong>jede</strong> Anfrage über den Proxy als vertrauenswürdig.</p>{% endif %}
|
||||||
|
<div class="row"><button class="primary">Anmeldung speichern</button></div>
|
||||||
|
</form>
|
||||||
|
|
||||||
<h2>Export / Import</h2>
|
<h2>Export / Import</h2>
|
||||||
<div class="card stack">
|
<div class="card stack">
|
||||||
<p class="small muted">Export enthält Anbindungen, Regeln und Einstellungen – <strong>ohne</strong> Zugangsdaten und Webhook-URL.
|
<p class="small muted">Export enthält Anbindungen, Regeln und Einstellungen – <strong>ohne</strong> Zugangsdaten und Webhook-URL.
|
||||||
|
|||||||
@@ -2,7 +2,14 @@
|
|||||||
{% block title %}Status{% endblock %}
|
{% block title %}Status{% endblock %}
|
||||||
{% block content %}
|
{% block content %}
|
||||||
<div class="row spread"><h1>Status</h1>
|
<div class="row spread"><h1>Status</h1>
|
||||||
<span class="muted small">aktualisiert sich alle 15 s</span></div>
|
<div class="row"><span class="muted small">aktualisiert sich alle 15 s</span>
|
||||||
|
<form class="inline" method="post" action="/maintenance"
|
||||||
|
{% if not (nav and nav.maintenance) %}data-confirm="Alle geplanten Läufe pausieren?"{% endif %}>
|
||||||
|
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||||
|
<button class="small">{{ 'Wartungsmodus beenden' if nav and nav.maintenance else 'Wartungsmodus' }}</button>
|
||||||
|
</form></div></div>
|
||||||
|
{% if nav and nav.maintenance %}<p class="msg warn"><strong>Wartungsmodus:</strong> Geplante Läufe sind pausiert.
|
||||||
|
„Jetzt ausführen“ geht weiterhin. Für Regeln schlägt <code>/healthz</code> keinen Alarm.</p>{% endif %}
|
||||||
<div id="status-body" hx-get="/partials/status" hx-trigger="every 15s" hx-swap="innerHTML">
|
<div id="status-body" hx-get="/partials/status" hx-trigger="every 15s" hx-swap="innerHTML">
|
||||||
{% include "partials/status_body.html" %}
|
{% include "partials/status_body.html" %}
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -5,13 +5,18 @@ from pathlib import Path
|
|||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from cryptography.fernet import Fernet
|
from cryptography.fernet import Fernet
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
from sqlalchemy.orm import Session, sessionmaker
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.app_state import NetMover
|
||||||
|
from netmover.config import Settings
|
||||||
from netmover.crypto import SecretBox
|
from netmover.crypto import SecretBox
|
||||||
from netmover.db import make_engine, make_session_factory, session_scope, upgrade
|
from netmover.db import make_engine, make_session_factory, session_scope, upgrade
|
||||||
from netmover.models import Credential, CredentialKind, Mount, MountType
|
from netmover.models import Credential, CredentialKind, Mount, MountType
|
||||||
from netmover.mounting.fake import FakeMountBackend
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
from netmover.mounting.manager import MountManager
|
from netmover.mounting.manager import MountManager
|
||||||
|
from netmover.web.app import create_app
|
||||||
|
from tests.webutil import do_setup
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
@@ -124,3 +129,40 @@ def make_mount(
|
|||||||
return m.id
|
return m.id
|
||||||
|
|
||||||
return factory
|
return factory
|
||||||
|
|
||||||
|
|
||||||
|
# -- Weboberfläche -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def settings(tmp_path: Path) -> Settings:
|
||||||
|
return Settings(
|
||||||
|
data_dir=tmp_path / "lib",
|
||||||
|
run_dir=tmp_path / "run",
|
||||||
|
mount_base=tmp_path / "mnt",
|
||||||
|
mount_backend="fake",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def backend(tmp_path: Path) -> Iterator[FakeMountBackend]:
|
||||||
|
b = FakeMountBackend(tmp_path / "fake")
|
||||||
|
yield b
|
||||||
|
b.release_hangs()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def state(settings: Settings, backend: FakeMountBackend) -> NetMover:
|
||||||
|
return NetMover(settings, backend=backend)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def raw_client(settings: Settings, state: NetMover) -> Iterator[TestClient]:
|
||||||
|
with TestClient(create_app(settings, state), follow_redirects=False) as c:
|
||||||
|
yield c
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client(raw_client: TestClient) -> TestClient:
|
||||||
|
raw_client.headers["X-Test-CSRF"] = do_setup(raw_client)
|
||||||
|
return raw_client
|
||||||
|
|||||||
+5
-1
@@ -48,7 +48,11 @@ class _Server(paramiko.ServerInterface):
|
|||||||
|
|
||||||
|
|
||||||
class _Handle(SFTPHandle):
|
class _Handle(SFTPHandle):
|
||||||
pass
|
def stat(self) -> Any: # fstat wie bei OpenSSH
|
||||||
|
try:
|
||||||
|
return SFTPAttributes.from_stat(os.fstat(self.readfile.fileno()))
|
||||||
|
except OSError as e:
|
||||||
|
return SFTPServer.convert_errno(e.errno)
|
||||||
|
|
||||||
|
|
||||||
class _RootedSFTP(SFTPServerInterface):
|
class _RootedSFTP(SFTPServerInterface):
|
||||||
|
|||||||
@@ -0,0 +1,167 @@
|
|||||||
|
"""Anmeldung je Netz: ohne Passwort / mit Passwort / abgewiesen."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from netmover.settings_store import load_settings, save_settings
|
||||||
|
from netmover.web.auth import (
|
||||||
|
ACCESS_DENY,
|
||||||
|
ACCESS_NOPW,
|
||||||
|
ACCESS_PW,
|
||||||
|
access_decision,
|
||||||
|
includes_loopback,
|
||||||
|
parse_networks,
|
||||||
|
)
|
||||||
|
from tests.webutil import PASSWORD, csrf_of
|
||||||
|
|
||||||
|
|
||||||
|
def _other(client: TestClient, host: str) -> TestClient:
|
||||||
|
"""Weiterer Client für dieselbe App (ohne eigenen Lebenszyklus) mit fester Adresse."""
|
||||||
|
return TestClient(client.app, follow_redirects=False, client=(host, 51000))
|
||||||
|
|
||||||
|
|
||||||
|
def _save(c: TestClient, trusted: str = "", allowed: str = "", current: str = PASSWORD) -> Any:
|
||||||
|
token = (
|
||||||
|
csrf_of(c.get("/settings").text)
|
||||||
|
if "X-Test-CSRF" not in c.headers
|
||||||
|
else c.headers["X-Test-CSRF"]
|
||||||
|
)
|
||||||
|
return c.post(
|
||||||
|
"/settings/auth",
|
||||||
|
data={
|
||||||
|
"csrf_token": token,
|
||||||
|
"trusted_networks": trusted,
|
||||||
|
"allowed_networks": allowed,
|
||||||
|
"current": current,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _set(state: Any, **kw: str) -> None:
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
for k, v in kw.items():
|
||||||
|
setattr(cfg, k, v)
|
||||||
|
save_settings(state.session_factory, cfg)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Entscheidung ------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("host", "trusted", "allowed", "expected"),
|
||||||
|
[
|
||||||
|
("10.0.10.50", "", "", ACCESS_PW), # Standard: überall mit Passwort
|
||||||
|
("10.0.10.50", "10.0.10.0/24", "", ACCESS_NOPW),
|
||||||
|
("10.1.5.9", "10.0.10.0/24", "", ACCESS_PW), # übrige Netze mit Passwort
|
||||||
|
("10.1.5.9", "10.0.10.0/24", "10.1.5.0/24", ACCESS_PW),
|
||||||
|
("192.168.5.5", "10.0.10.0/24", "10.1.5.0/24", ACCESS_DENY),
|
||||||
|
("10.0.10.50", "10.0.10.0/24", "10.1.5.0/24", ACCESS_NOPW), # ohne PW schlägt Liste
|
||||||
|
("2001:db8::5", "0.0.0.0/0, ::/0", "", ACCESS_NOPW), # Anmeldung ganz aus
|
||||||
|
("testclient", "", "10.0.0.0/8", ACCESS_DENY), # keine IP-Adresse
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_access_decision(host: str, trusted: str, allowed: str, expected: str) -> None:
|
||||||
|
assert access_decision(host, "always", trusted, allowed) == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_off_mode_still_works() -> None:
|
||||||
|
assert access_decision("192.168.5.5", "off", "", "10.1.5.0/24") == ACCESS_NOPW
|
||||||
|
|
||||||
|
|
||||||
|
# -- Oberfläche --------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_requires_login(client: TestClient) -> None:
|
||||||
|
assert _other(client, "10.0.10.50").get("/").status_code == 303
|
||||||
|
|
||||||
|
|
||||||
|
def test_change_needs_current_password(client: TestClient, state: Any) -> None:
|
||||||
|
_save(client, trusted="10.0.10.0/24", current="falsch")
|
||||||
|
assert load_settings(state.session_factory).trusted_networks == ""
|
||||||
|
assert "Aktuelles Passwort falsch" in client.get("/settings").text
|
||||||
|
|
||||||
|
|
||||||
|
def test_without_password_from_trusted_network(client: TestClient, state: Any) -> None:
|
||||||
|
_save(client, trusted="10.0.10.0/24")
|
||||||
|
lan = _other(client, "10.0.10.50")
|
||||||
|
r = lan.get("/")
|
||||||
|
assert r.status_code == 200 and "ohne Anmeldung (vertrautes Netz)" in r.text
|
||||||
|
assert "Abmelden" not in r.text and lan.get("/login").headers["location"] == "/"
|
||||||
|
assert lan.post("/maintenance").status_code == 403 # CSRF gilt weiterhin
|
||||||
|
assert (
|
||||||
|
lan.post("/maintenance", data={"csrf_token": csrf_of(lan.get("/").text)}).status_code == 303
|
||||||
|
)
|
||||||
|
assert _other(client, "10.1.5.9").get("/").status_code == 303 # übrige mit Passwort
|
||||||
|
|
||||||
|
|
||||||
|
def test_other_networks_are_rejected_when_password_list_is_set(
|
||||||
|
client: TestClient, state: Any
|
||||||
|
) -> None:
|
||||||
|
_save(client, trusted="0.0.0.0/0") # damit der Test-Client (keine IP) weiterkommt …
|
||||||
|
_set(state, trusted_networks="10.0.10.0/24", allowed_networks="10.1.5.0/24") # … dann scharf
|
||||||
|
assert _other(client, "10.0.10.50").get("/").status_code == 200
|
||||||
|
assert _other(client, "10.1.5.9").get("/").headers["location"].startswith("/login")
|
||||||
|
blocked = _other(client, "192.168.5.5")
|
||||||
|
for url in ("/", "/login", "/setup", "/mounts"):
|
||||||
|
assert blocked.get(url).status_code == 403, url
|
||||||
|
assert blocked.get("/healthz").status_code == 200 # Überwachung bleibt erreichbar
|
||||||
|
assert blocked.get("/static/app.css").status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_cannot_lock_yourself_out(client: TestClient, state: Any) -> None:
|
||||||
|
lan = _other(client, "10.0.10.50")
|
||||||
|
_save(client, trusted="10.0.10.0/24")
|
||||||
|
# aus 10.0.10.50 eine Liste speichern, in der das eigene Netz fehlt
|
||||||
|
r = _save(lan, trusted="", allowed="10.1.5.0/24")
|
||||||
|
assert r.status_code == 303
|
||||||
|
assert "ausgesperrt" in lan.get("/settings").text
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
assert cfg.trusted_networks == "10.0.10.0/24" and cfg.allowed_networks == ""
|
||||||
|
# eigenes Netz bei „mit Passwort“ – erlaubt
|
||||||
|
_save(lan, trusted="10.0.10.0/24", allowed="10.1.5.0/24")
|
||||||
|
assert load_settings(state.session_factory).allowed_networks == "10.1.5.0/24"
|
||||||
|
|
||||||
|
|
||||||
|
def test_everything_without_password(client: TestClient) -> None:
|
||||||
|
_save(client, trusted="0.0.0.0/0, ::/0")
|
||||||
|
r = _other(client, "192.168.77.1").get("/rules")
|
||||||
|
assert r.status_code == 200 and ">ohne Anmeldung<" in r.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_setup_still_required_without_password(raw_client: TestClient, state: Any) -> None:
|
||||||
|
_set(state, trusted_networks="0.0.0.0/0")
|
||||||
|
assert raw_client.get("/").headers["location"] == "/setup"
|
||||||
|
|
||||||
|
|
||||||
|
def test_validation_and_loopback_hint(client: TestClient, state: Any) -> None:
|
||||||
|
_save(client, trusted="10.0.10.0/24, quatsch")
|
||||||
|
assert "kein gültiges Netz" in client.get("/settings").text
|
||||||
|
assert load_settings(state.session_factory).trusted_networks == ""
|
||||||
|
_save(client, trusted="127.0.0.1")
|
||||||
|
assert "jede</strong> Anfrage über den Proxy" in client.get("/settings").text
|
||||||
|
|
||||||
|
|
||||||
|
def test_network_parsing() -> None:
|
||||||
|
nets = parse_networks("10.0.10.0/24; 192.168.1.20 fe80::/10")
|
||||||
|
assert [str(n) for n in nets] == ["10.0.10.0/24", "192.168.1.20/32", "fe80::/10"]
|
||||||
|
assert includes_loopback("10.0.0.0/8, 127.0.0.0/8") and not includes_loopback("10.0.0.0/8")
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
parse_networks("10.0.0.0/33")
|
||||||
|
|
||||||
|
|
||||||
|
def test_reset_password_turns_login_back_on(
|
||||||
|
state: Any, settings: Any, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
from netmover import cli
|
||||||
|
from netmover.web.auth import set_password
|
||||||
|
|
||||||
|
set_password(state.session_factory, PASSWORD)
|
||||||
|
_set(state, auth_mode="off", trusted_networks="0.0.0.0/0", allowed_networks="10.1.5.0/24")
|
||||||
|
monkeypatch.setattr(cli, "get_settings", lambda: settings)
|
||||||
|
assert cli.main(["reset-password"]) == 0
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
assert cfg.auth_mode == "always" and cfg.trusted_networks == "" and cfg.allowed_networks == ""
|
||||||
@@ -0,0 +1,307 @@
|
|||||||
|
"""0.10.0: Datensicherung – verschlüsselt auf eine Anbindung, Wiederherstellen per CLI."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import sqlite3
|
||||||
|
import tarfile
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from netmover import __version__, backup
|
||||||
|
from netmover.app_state import NetMover
|
||||||
|
from netmover.backup import (
|
||||||
|
BackupError,
|
||||||
|
is_due,
|
||||||
|
load_state,
|
||||||
|
make_backup,
|
||||||
|
open_archive,
|
||||||
|
prune_names,
|
||||||
|
)
|
||||||
|
from netmover.cli import cmd_backup, cmd_restore
|
||||||
|
from netmover.config import Settings
|
||||||
|
from netmover.models import Mount, MountType
|
||||||
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
|
from netmover.settings_store import AppSettings, load_settings, save_settings
|
||||||
|
from tests.webutil import post
|
||||||
|
|
||||||
|
PASS = "korrekt-pferd-batterie"
|
||||||
|
NOW = datetime(2026, 10, 23, 12, 0, tzinfo=UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def _nas(state: NetMover, backend: FakeMountBackend) -> tuple[int, Path]:
|
||||||
|
"""CIFS-Anbindung „Backup“ einhängen; liefert (id, Freigabe auf dem „NAS“)."""
|
||||||
|
share = backend.export("nas", "backup")
|
||||||
|
with state.session_factory() as s:
|
||||||
|
m = Mount(
|
||||||
|
name="Backup",
|
||||||
|
slug="backup",
|
||||||
|
type=MountType.CIFS,
|
||||||
|
server="nas",
|
||||||
|
share="backup",
|
||||||
|
mountpoint=str(state.settings.mount_base.resolve() / "backup"),
|
||||||
|
)
|
||||||
|
s.add(m)
|
||||||
|
s.commit()
|
||||||
|
mid = m.id
|
||||||
|
state.mounts.mount(mid)
|
||||||
|
return mid, share
|
||||||
|
|
||||||
|
|
||||||
|
def _configure(state: NetMover, mount_id: int, **kw: Any) -> AppSettings:
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
cfg.backup_enabled = True
|
||||||
|
cfg.backup_mount_id = mount_id
|
||||||
|
cfg.backup_path = "netmover"
|
||||||
|
cfg.backup_passphrase_enc = state.secrets.encrypt(PASS)
|
||||||
|
for k, v in kw.items():
|
||||||
|
setattr(cfg, k, v)
|
||||||
|
save_settings(state.session_factory, cfg)
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
# -- Format ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_encrypted_roundtrip_and_wrong_passphrase(state: NetMover) -> None:
|
||||||
|
blob, manifest = make_backup(state.settings, PASS)
|
||||||
|
assert backup.is_encrypted(blob)
|
||||||
|
# Weder Schlüssel noch Datenbank im Klartext
|
||||||
|
key = state.settings.secret_key_path.read_bytes().strip()
|
||||||
|
assert key not in blob and b"SQLite format 3" not in blob
|
||||||
|
archive = open_archive(blob, PASS)
|
||||||
|
assert archive.manifest["version"] == __version__
|
||||||
|
assert archive.manifest["revision"]
|
||||||
|
assert archive.files["secret.key"].strip() == key
|
||||||
|
assert archive.files["netmover.db"].startswith(b"SQLite format 3")
|
||||||
|
assert manifest["files"] == sorted(archive.files)
|
||||||
|
with pytest.raises(BackupError, match="Passphrase falsch"):
|
||||||
|
open_archive(blob, "falsche-passphrase")
|
||||||
|
with pytest.raises(BackupError, match="verschlüsselt"):
|
||||||
|
open_archive(blob, None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_tampered_backup_is_rejected(state: NetMover) -> None:
|
||||||
|
blob = bytearray(make_backup(state.settings, PASS)[0])
|
||||||
|
blob[-10] ^= 0x01
|
||||||
|
with pytest.raises(BackupError):
|
||||||
|
open_archive(bytes(blob), PASS)
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_file_is_included(state: NetMover, tmp_path: Path) -> None:
|
||||||
|
env = tmp_path / "netmover.env"
|
||||||
|
env.write_text("NETMOVER_PORT=8080\n")
|
||||||
|
settings = state.settings.model_copy(update={"config_file": env})
|
||||||
|
archive = open_archive(make_backup(settings, None)[0], None)
|
||||||
|
assert archive.files["netmover.env"] == b"NETMOVER_PORT=8080\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_archive_without_secret_key_is_rejected() -> None:
|
||||||
|
buf = io.BytesIO()
|
||||||
|
with tarfile.open(fileobj=buf, mode="w:gz") as tar:
|
||||||
|
data = b'{"format": 1}'
|
||||||
|
info = tarfile.TarInfo("manifest.json")
|
||||||
|
info.size = len(data)
|
||||||
|
tar.addfile(info, io.BytesIO(data))
|
||||||
|
with pytest.raises(BackupError, match="unvollständig"):
|
||||||
|
open_archive(buf.getvalue(), None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_keeps_newest_and_ignores_foreign_files() -> None:
|
||||||
|
ours = [f"netmover-h-202610{d:02d}-033000.nmbak" for d in range(1, 8)]
|
||||||
|
names = [*ours, "urlaub.jpg", "netmover-notizen.txt"]
|
||||||
|
assert sorted(prune_names(names, 3)) == ours[:4]
|
||||||
|
assert prune_names(names, 10) == []
|
||||||
|
|
||||||
|
|
||||||
|
# -- Termin ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(time: str = "03:30") -> AppSettings:
|
||||||
|
return AppSettings(backup_enabled=True, backup_mount_id=1, backup_time=time)
|
||||||
|
|
||||||
|
|
||||||
|
def test_due_after_slot_once_per_day_and_hourly_retry() -> None:
|
||||||
|
local = datetime(2026, 10, 23, 2, 0).astimezone() # 02:00 Ortszeit
|
||||||
|
st = backup.BackupState()
|
||||||
|
yesterday_ok = backup.BackupState(last_success=local - timedelta(hours=3))
|
||||||
|
assert is_due(_cfg(), st, local) # noch nie gesichert → nachholen
|
||||||
|
assert not is_due(_cfg(), yesterday_ok, local) # 23:00 gestern nach 03:30 gestern
|
||||||
|
later = local + timedelta(hours=2) # 04:00 → heutiger Termin ist fällig
|
||||||
|
assert is_due(_cfg(), yesterday_ok, later)
|
||||||
|
failed = backup.BackupState(
|
||||||
|
last_success=yesterday_ok.last_success, last_attempt=later - timedelta(minutes=10)
|
||||||
|
)
|
||||||
|
assert not is_due(_cfg(), failed, later) # gerade gescheitert → nicht jeden Takt
|
||||||
|
assert is_due(_cfg(), failed, later + timedelta(hours=1))
|
||||||
|
assert not is_due(AppSettings(backup_enabled=False, backup_mount_id=1), st, later)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
backup.parse_time("25:00")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Dienst ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_service_writes_encrypted_backup_to_mount_and_prunes(
|
||||||
|
state: NetMover, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
mid, share = _nas(state, backend)
|
||||||
|
_configure(state, mid, backup_keep=3)
|
||||||
|
(share / "netmover").mkdir()
|
||||||
|
(share / "netmover" / "fremd.txt").write_text("bleibt")
|
||||||
|
for i in range(5):
|
||||||
|
result = state.backups.run(NOW + timedelta(days=i))
|
||||||
|
assert result.ok, result.message
|
||||||
|
files = sorted(p.name for p in (share / "netmover").iterdir())
|
||||||
|
assert "fremd.txt" in files
|
||||||
|
backups = [f for f in files if f.endswith(".nmbak")]
|
||||||
|
assert len(backups) == 3
|
||||||
|
assert not [f for f in files if f.endswith(".part")]
|
||||||
|
newest = share / "netmover" / backups[-1]
|
||||||
|
assert open_archive(newest.read_bytes(), PASS).manifest["version"] == __version__
|
||||||
|
st = load_state(state.session_factory)
|
||||||
|
assert st.last_success == NOW + timedelta(days=4)
|
||||||
|
assert st.last_file == backups[-1] and st.last_size == newest.stat().st_size
|
||||||
|
check = state.health().find("backup")
|
||||||
|
assert check is not None and check.ok
|
||||||
|
|
||||||
|
|
||||||
|
def test_failure_is_recorded_notified_once_and_becomes_problem(
|
||||||
|
settings: Settings, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
sent: list[tuple[str, str, str]] = []
|
||||||
|
state = NetMover(settings, backend=backend)
|
||||||
|
mid, _share = _nas(state, backend)
|
||||||
|
_configure(state, mid)
|
||||||
|
state.backups._notify = lambda *a: sent.append(a)
|
||||||
|
backend.vanish(state.mounts.mountpoint_for("backup"))
|
||||||
|
backend.fail_next_mount(times=10)
|
||||||
|
for i in range(3):
|
||||||
|
result = state.backups.run(NOW + timedelta(hours=i))
|
||||||
|
assert not result.ok
|
||||||
|
st = load_state(state.session_factory)
|
||||||
|
assert st.failures == 3 and "eingehängt" in st.last_error
|
||||||
|
assert [k for k, _t, _m in sent] == ["backup.failed"]
|
||||||
|
# Überfällig erst nach STALE_AFTER – dann rot in /healthz und Kuma
|
||||||
|
state.started_at = datetime.now(UTC) - backup.STALE_AFTER * 2
|
||||||
|
check = state.health().find("backup")
|
||||||
|
assert check is not None and not check.ok and "überfällig" in check.message
|
||||||
|
|
||||||
|
|
||||||
|
def test_tick_runs_only_when_due(state: NetMover, backend: FakeMountBackend) -> None:
|
||||||
|
mid, _ = _nas(state, backend)
|
||||||
|
_configure(state, mid)
|
||||||
|
assert state.backups.tick(NOW) is not None # noch nie gesichert
|
||||||
|
assert state.backups.tick(NOW + timedelta(minutes=5)) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_passphrase_refuses(state: NetMover, backend: FakeMountBackend) -> None:
|
||||||
|
mid, _ = _nas(state, backend)
|
||||||
|
_configure(state, mid, backup_passphrase_enc="")
|
||||||
|
result = state.backups.run(NOW)
|
||||||
|
assert not result.ok and "Passphrase" in result.message
|
||||||
|
|
||||||
|
|
||||||
|
# -- Wiederherstellen ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _mount_names(db: Path) -> list[str]:
|
||||||
|
con = sqlite3.connect(db)
|
||||||
|
try:
|
||||||
|
return [r[0] for r in con.execute("select name from mounts order by name")]
|
||||||
|
finally:
|
||||||
|
con.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_backup_and_restore_roundtrip(
|
||||||
|
state: NetMover, backend: FakeMountBackend, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
settings = state.settings
|
||||||
|
mid, _ = _nas(state, backend)
|
||||||
|
_configure(state, mid)
|
||||||
|
out = tmp_path / "out"
|
||||||
|
out.mkdir()
|
||||||
|
assert cmd_backup(settings, out, encrypted=True) == 0
|
||||||
|
(file,) = out.iterdir()
|
||||||
|
assert file.suffix == ".nmbak" and file.stat().st_mode & 0o077 == 0
|
||||||
|
old_key = settings.secret_key_path.read_bytes()
|
||||||
|
state.stop()
|
||||||
|
|
||||||
|
# Nach der Sicherung geht etwas schief: Anbindung gelöscht, Schlüssel weg
|
||||||
|
con = sqlite3.connect(settings.db_path)
|
||||||
|
con.execute("delete from mounts")
|
||||||
|
con.commit()
|
||||||
|
con.close()
|
||||||
|
settings.secret_key_path.write_bytes(b"kaputt")
|
||||||
|
phrase = tmp_path / "phrase"
|
||||||
|
phrase.write_text(PASS + "\n")
|
||||||
|
|
||||||
|
assert cmd_restore(settings, file, phrase, assume_yes=True, active=lambda: True) == 1
|
||||||
|
assert "Dienst läuft" in capsys.readouterr().out
|
||||||
|
assert cmd_restore(settings, file, phrase, assume_yes=True, active=lambda: False) == 0
|
||||||
|
|
||||||
|
assert _mount_names(settings.db_path) == ["Backup"]
|
||||||
|
assert settings.secret_key_path.read_bytes() == old_key
|
||||||
|
assert settings.secret_key_path.stat().st_mode & 0o077 == 0
|
||||||
|
(kept,) = settings.data_dir.glob("vor-wiederherstellung-*")
|
||||||
|
assert (kept / "secret.key").read_bytes() == b"kaputt"
|
||||||
|
assert _mount_names(kept / "netmover.db") == []
|
||||||
|
# Gespeicherte Zugangsdaten sind mit dem zurückgeholten Schlüssel wieder lesbar
|
||||||
|
restarted = NetMover(settings, backend=backend)
|
||||||
|
cfg = load_settings(restarted.session_factory)
|
||||||
|
assert restarted.backups.passphrase(cfg) == PASS
|
||||||
|
restarted.stop()
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_refuses_newer_version(state: NetMover, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
archive = open_archive(make_backup(state.settings, None)[0], None)
|
||||||
|
archive.manifest["version"] = "99.0.0"
|
||||||
|
with pytest.raises(BackupError, match="erst aktualisieren"):
|
||||||
|
backup.restore(state.settings, archive)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Oberfläche ------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_backup_page_save_run_and_health(
|
||||||
|
client: TestClient, state: NetMover, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
mid, share = _nas(state, backend)
|
||||||
|
page = client.get("/backup")
|
||||||
|
assert page.status_code == 200 and 'href="/backup"' in page.text
|
||||||
|
form = {
|
||||||
|
"backup_enabled": "on",
|
||||||
|
"backup_mount_id": str(mid),
|
||||||
|
"backup_path": "netmover",
|
||||||
|
"backup_time": "03:30",
|
||||||
|
"backup_keep": "14",
|
||||||
|
}
|
||||||
|
r = post(client, "/backup", form)
|
||||||
|
assert r.status_code == 400 and "Passphrase festlegen" in r.text
|
||||||
|
r = post(client, "/backup", {**form, "passphrase": PASS, "passphrase_confirm": "anders"})
|
||||||
|
assert r.status_code == 400 and "stimmen nicht" in r.text
|
||||||
|
r = post(client, "/backup", {**form, "backup_path": "../raus", "passphrase": PASS})
|
||||||
|
assert r.status_code == 400
|
||||||
|
r = post(client, "/backup", {**form, "passphrase": PASS, "passphrase_confirm": PASS})
|
||||||
|
assert r.status_code == 303
|
||||||
|
with state.session_factory() as s:
|
||||||
|
from netmover.models import Setting
|
||||||
|
|
||||||
|
raw = " ".join(v for (v,) in s.execute(select(Setting.value)))
|
||||||
|
assert PASS not in raw # Passphrase nur verschlüsselt gespeichert
|
||||||
|
# Passphrase leer lassen = unverändert
|
||||||
|
assert post(client, "/backup", {**form, "backup_keep": "7"}).status_code == 303
|
||||||
|
assert state.backups.passphrase(load_settings(state.session_factory)) == PASS
|
||||||
|
|
||||||
|
assert post(client, "/backup/run").status_code == 303
|
||||||
|
assert "geschrieben" in client.get("/backup").text
|
||||||
|
assert len(list((share / "netmover").glob("*.nmbak"))) == 1
|
||||||
|
r = client.get("/healthz/backup")
|
||||||
|
assert r.status_code == 200 and r.json()["status"] == "ok"
|
||||||
|
assert "/healthz/backup" in client.get("/settings").text
|
||||||
|
assert any(c["kind"] == "backup" for c in client.get("/healthz").json()["checks"])
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
"""0.8.0: Anbindungen kopieren – inklusive Zugangsdaten."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from netmover.admin import MountInput, copy_name, delete_mount, save_mount
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
|
from netmover.models import Credential, CredentialKind, Mount, MountType
|
||||||
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
|
from netmover.mounting.manager import MountManager
|
||||||
|
from tests.webutil import post
|
||||||
|
|
||||||
|
|
||||||
|
def _cifs(**kw: object) -> MountInput:
|
||||||
|
data: dict[str, object] = {
|
||||||
|
"name": "NAS",
|
||||||
|
"type": MountType.CIFS,
|
||||||
|
"server": "10.0.10.6",
|
||||||
|
"share": "Scans",
|
||||||
|
"username": "scanner",
|
||||||
|
"password": "Pa$$wort",
|
||||||
|
}
|
||||||
|
data.update(kw)
|
||||||
|
return MountInput(**data) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_reuses_own_password_as_independent_copy(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
src, _ = save_mount(session, secretbox, manager, _cifs())
|
||||||
|
assert src is not None and src.credential is not None
|
||||||
|
copy, err = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_cifs(name="Archiv", share="Archiv", password=""),
|
||||||
|
copy_from=src,
|
||||||
|
)
|
||||||
|
assert not err and copy is not None and copy.credential is not None
|
||||||
|
assert copy.credential.id != src.credential.id and not copy.credential.managed
|
||||||
|
assert secretbox.decrypt(copy.credential.secret_enc) == "Pa$$wort"
|
||||||
|
assert copy.credential.username == "scanner"
|
||||||
|
# Vorlage löschen – die Kopie behält ihre Zugangsdaten
|
||||||
|
delete_mount(session, manager, src.id)
|
||||||
|
session.flush()
|
||||||
|
assert secretbox.decrypt(copy.credential.secret_enc) == "Pa$$wort"
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_with_new_password_uses_new_one(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
src, _ = save_mount(session, secretbox, manager, _cifs())
|
||||||
|
copy, _ = save_mount(
|
||||||
|
session, secretbox, manager, _cifs(name="B", share="B", password="anders"), copy_from=src
|
||||||
|
)
|
||||||
|
assert copy is not None and copy.credential is not None
|
||||||
|
assert secretbox.decrypt(copy.credential.secret_enc) == "anders"
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_with_stored_credentials_shares_them(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
from netmover.credentials import create_password
|
||||||
|
|
||||||
|
cred = create_password(session, secretbox, "NAS-Scanner", "scanner", "x")
|
||||||
|
src, _ = save_mount(
|
||||||
|
session, secretbox, manager, _cifs(username="", password="", cred_id=str(cred.id))
|
||||||
|
)
|
||||||
|
copy, _ = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_cifs(name="B", share="B", username="", password="", cred_id=str(cred.id)),
|
||||||
|
copy_from=src,
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
copy is not None and copy.credential is cred and src is not None and src.credential is cred
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_sftp_keeps_keypair_and_host_key_only_for_same_server(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
sftp = {
|
||||||
|
"type": MountType.SFTP,
|
||||||
|
"server": "10.0.10.17",
|
||||||
|
"share": "/consume",
|
||||||
|
"port": 22,
|
||||||
|
"username": "paperless",
|
||||||
|
"auth": CredentialKind.SSH_KEY,
|
||||||
|
"password": "",
|
||||||
|
}
|
||||||
|
src, _ = save_mount(session, secretbox, manager, _cifs(name="Paperless", **sftp))
|
||||||
|
assert src is not None and src.credential is not None
|
||||||
|
src.host_key = "ssh-ed25519 AAAAhost"
|
||||||
|
same, _ = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_cifs(name="Paperless 2", **{**sftp, "share": "/consume2"}),
|
||||||
|
copy_from=src,
|
||||||
|
)
|
||||||
|
other, _ = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_cifs(name="Anderer", **{**sftp, "server": "10.0.10.99"}),
|
||||||
|
copy_from=src,
|
||||||
|
)
|
||||||
|
assert same is not None and other is not None
|
||||||
|
for m in (same, other):
|
||||||
|
assert m.credential is not None and m.credential.id != src.credential.id
|
||||||
|
assert m.credential.public_key == src.credential.public_key # gleiches Schlüsselpaar
|
||||||
|
assert m.credential.secret_enc == src.credential.secret_enc
|
||||||
|
assert same.host_key == "ssh-ed25519 AAAAhost" and other.host_key is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_name(session: Session, secretbox: SecretBox, manager: MountManager) -> None:
|
||||||
|
save_mount(session, secretbox, manager, _cifs())
|
||||||
|
assert copy_name(session, "NAS") == "NAS (Kopie)"
|
||||||
|
save_mount(session, secretbox, manager, _cifs(name="NAS (Kopie)", share="x"))
|
||||||
|
assert copy_name(session, "NAS") == "NAS (Kopie 2)"
|
||||||
|
|
||||||
|
|
||||||
|
# -- Oberfläche --------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
MOUNT = {
|
||||||
|
"name": "NAS",
|
||||||
|
"type": "cifs",
|
||||||
|
"server": "nas",
|
||||||
|
"share": "scans",
|
||||||
|
"username": "scanner",
|
||||||
|
"password": "Geheim,123",
|
||||||
|
"auto_mount": "on",
|
||||||
|
"enabled": "on",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _mid(state: Any, name: str) -> int:
|
||||||
|
with state.session_factory() as s:
|
||||||
|
mid = s.scalar(select(Mount.id).where(Mount.name == name))
|
||||||
|
assert mid is not None
|
||||||
|
return mid
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_button_form_test_and_save(
|
||||||
|
client: TestClient, state: Any, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
backend.export("nas", "scans")
|
||||||
|
backend.export("nas", "archiv")
|
||||||
|
post(client, "/mounts/new", MOUNT)
|
||||||
|
src_id = _mid(state, "NAS")
|
||||||
|
assert f'href="/mounts/{src_id}/copy"' in client.get("/mounts").text
|
||||||
|
|
||||||
|
form = client.get(f"/mounts/{src_id}/copy").text
|
||||||
|
assert "Kopie von „NAS“" in form and 'value="NAS (Kopie)"' in form
|
||||||
|
assert 'value="scanner"' in form and "Geheim,123" not in form
|
||||||
|
assert f'name="copy_from" value="{src_id}"' in form
|
||||||
|
|
||||||
|
# Testen mit übernommenem Passwort (Feld leer)
|
||||||
|
copy_form = {
|
||||||
|
**MOUNT,
|
||||||
|
"name": "NAS (Kopie)",
|
||||||
|
"share": "archiv",
|
||||||
|
"password": "",
|
||||||
|
"copy_from": str(src_id),
|
||||||
|
}
|
||||||
|
r = post(client, "/mounts/test", copy_form)
|
||||||
|
assert "Einhängen erfolgreich" in r.text
|
||||||
|
spec = [c for c in backend.calls if c.action == "mount"][-1].spec
|
||||||
|
assert spec is not None and spec.credentials is not None
|
||||||
|
assert spec.credentials.password == "Geheim,123"
|
||||||
|
|
||||||
|
r = post(client, f"/mounts/{src_id}/copy", copy_form)
|
||||||
|
assert r.status_code == 303, r.text
|
||||||
|
with state.session_factory() as s:
|
||||||
|
copy = s.get(Mount, _mid(state, "NAS (Kopie)"))
|
||||||
|
assert copy is not None and copy.share == "archiv" and copy.slug == "nas-kopie"
|
||||||
|
assert copy.credential is not None
|
||||||
|
assert state.secrets.decrypt(copy.credential.secret_enc) == "Geheim,123"
|
||||||
|
assert len(s.scalars(select(Credential)).all()) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_validation_error_keeps_copy_context(client: TestClient, state: Any) -> None:
|
||||||
|
post(client, "/mounts/new", MOUNT)
|
||||||
|
src_id = _mid(state, "NAS")
|
||||||
|
r = post(client, f"/mounts/{src_id}/copy", {**MOUNT, "name": "NAS", "password": ""})
|
||||||
|
assert r.status_code == 400 and "bereits vergeben" in r.text and "Kopie von „NAS“" in r.text
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
"""0.9.0: Regeln kopieren – vorausgefüllt, zunächst deaktiviert."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.models import (
|
||||||
|
CopiedFile,
|
||||||
|
Mount,
|
||||||
|
MountType,
|
||||||
|
Rule,
|
||||||
|
RuleAction,
|
||||||
|
RuleTarget,
|
||||||
|
Run,
|
||||||
|
TargetKind,
|
||||||
|
TargetLayout,
|
||||||
|
)
|
||||||
|
from tests.webutil import post
|
||||||
|
|
||||||
|
|
||||||
|
def _setup(state: Any, tmp_path: Path) -> tuple[int, int]:
|
||||||
|
consume = tmp_path / "consume"
|
||||||
|
consume.mkdir()
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
m = Mount(
|
||||||
|
name="NAS",
|
||||||
|
slug="nas",
|
||||||
|
type=MountType.CIFS,
|
||||||
|
server="nas",
|
||||||
|
share="scans",
|
||||||
|
mountpoint=str(tmp_path / "mnt" / "nas"),
|
||||||
|
)
|
||||||
|
a = Mount(
|
||||||
|
name="Archiv",
|
||||||
|
slug="archiv",
|
||||||
|
type=MountType.CIFS,
|
||||||
|
server="nas",
|
||||||
|
share="archiv",
|
||||||
|
mountpoint=str(tmp_path / "mnt" / "archiv"),
|
||||||
|
)
|
||||||
|
s.add_all([m, a])
|
||||||
|
s.flush()
|
||||||
|
r = Rule(
|
||||||
|
name="Scans → Paperless",
|
||||||
|
src_mount_id=m.id,
|
||||||
|
src_subpath="max",
|
||||||
|
dst_path=str(consume),
|
||||||
|
pattern="*.pdf;*.jpg",
|
||||||
|
recursive=True,
|
||||||
|
interval_seconds=45,
|
||||||
|
stability_seconds=7,
|
||||||
|
action=RuleAction.COPY,
|
||||||
|
set_mode="0640",
|
||||||
|
quarantine_path="_q",
|
||||||
|
targets=[
|
||||||
|
RuleTarget(
|
||||||
|
position=0,
|
||||||
|
name="Archiv",
|
||||||
|
kind=TargetKind.MOUNT,
|
||||||
|
mount_id=a.id,
|
||||||
|
path="scans",
|
||||||
|
layout=TargetLayout.DATE,
|
||||||
|
),
|
||||||
|
RuleTarget(position=1, name="Buchhaltung", kind=TargetKind.LOCAL, path="/srv/buha"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
s.add(Run(rule_id=r.id))
|
||||||
|
s.add(CopiedFile(rule_id=r.id, src_relpath="a.pdf", size=1, mtime_ns=1, dst_path="x"))
|
||||||
|
return r.id, m.id
|
||||||
|
|
||||||
|
|
||||||
|
def _rule(state: Any, name: str) -> Rule:
|
||||||
|
with state.session_factory() as s:
|
||||||
|
r = s.scalars(select(Rule).where(Rule.name == name)).one()
|
||||||
|
_ = list(r.targets) # vor dem Lösen aus der Session laden
|
||||||
|
s.expunge_all()
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
def _form_from_page(html: str) -> dict[str, str]:
|
||||||
|
"""Wie ein Browser: alle Felder des Formulars mit ihren vorausgefüllten Werten."""
|
||||||
|
import re
|
||||||
|
|
||||||
|
form: dict[str, str] = {}
|
||||||
|
for m in re.finditer(r'<input[^>]*name="([^"]+)"[^>]*>', html):
|
||||||
|
tag, name = m.group(0), m.group(1)
|
||||||
|
if 'type="checkbox"' in tag:
|
||||||
|
if " checked" in tag:
|
||||||
|
form[name] = "on"
|
||||||
|
else:
|
||||||
|
v = re.search(r'value="([^"]*)"', tag)
|
||||||
|
form[name] = v.group(1) if v else ""
|
||||||
|
for m in re.finditer(r'<select[^>]*name="([^"]+)"[^>]*>(.*?)</select>', html, re.S):
|
||||||
|
sel = re.search(r'<option value="([^"]*)" selected>', m.group(2))
|
||||||
|
form[m.group(1)] = sel.group(1) if sel else ""
|
||||||
|
return form
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_form_is_prefilled_and_disabled(
|
||||||
|
client: TestClient, state: Any, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
rid, _ = _setup(state, tmp_path)
|
||||||
|
assert f'href="/rules/{rid}/copy"' in client.get("/rules").text
|
||||||
|
html = client.get(f"/rules/{rid}/copy").text
|
||||||
|
assert "Kopie von „Scans → Paperless“" in html and "deaktiviert" in html
|
||||||
|
form = _form_from_page(html)
|
||||||
|
assert form["name"] == "Scans → Paperless (Kopie)"
|
||||||
|
assert "enabled" not in form # Häkchen „Aktiv“ nicht gesetzt
|
||||||
|
assert form["pattern"] == "*.pdf;*.jpg" and form["src_subpath"] == "max"
|
||||||
|
assert form["target_0_kind"] == "mount" and form["target_0_layout"] == "date"
|
||||||
|
assert form["target_0_name"] == "Archiv" and form["target_1_path"] == "/srv/buha"
|
||||||
|
assert form["target_2_kind"] == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_saves_new_rule_with_all_settings(
|
||||||
|
client: TestClient, state: Any, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
rid, _ = _setup(state, tmp_path)
|
||||||
|
form = _form_from_page(client.get(f"/rules/{rid}/copy").text)
|
||||||
|
form["src_subpath"] = "erika" # typische Anpassung
|
||||||
|
r = post(client, f"/rules/{rid}/copy", form)
|
||||||
|
assert r.status_code == 303, r.text
|
||||||
|
assert "Die Kopie ist deaktiviert" in client.get("/rules").text
|
||||||
|
|
||||||
|
orig, copy = _rule(state, "Scans → Paperless"), _rule(state, "Scans → Paperless (Kopie)")
|
||||||
|
assert copy.id != orig.id and copy.enabled is False and orig.enabled is True
|
||||||
|
assert copy.src_subpath == "erika" and orig.src_subpath == "max"
|
||||||
|
for f in (
|
||||||
|
"src_mount_id",
|
||||||
|
"dst_path",
|
||||||
|
"pattern",
|
||||||
|
"recursive",
|
||||||
|
"interval_seconds",
|
||||||
|
"stability_seconds",
|
||||||
|
"action",
|
||||||
|
"set_mode",
|
||||||
|
"quarantine_path",
|
||||||
|
):
|
||||||
|
assert getattr(copy, f) == getattr(orig, f), f
|
||||||
|
|
||||||
|
def targets(r: Rule) -> list[tuple[object, ...]]:
|
||||||
|
return [(t.position, t.name, t.kind, t.mount_id, t.path, t.layout) for t in r.targets]
|
||||||
|
|
||||||
|
assert len(targets(copy)) == 2 and targets(copy) == targets(orig)
|
||||||
|
assert {t.id for t in copy.targets}.isdisjoint(t.id for t in orig.targets)
|
||||||
|
# Protokoll und Buchführung beginnen neu
|
||||||
|
with state.session_factory() as s:
|
||||||
|
assert s.scalars(select(Run).where(Run.rule_id == copy.id)).all() == []
|
||||||
|
assert s.scalars(select(CopiedFile).where(CopiedFile.rule_id == copy.id)).all() == []
|
||||||
|
assert state.scheduler.next_run(copy.id) is None # deaktiviert → kein Termin
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_names_count_up(client: TestClient, state: Any, tmp_path: Path) -> None:
|
||||||
|
rid, _ = _setup(state, tmp_path)
|
||||||
|
for expected in ("Scans → Paperless (Kopie)", "Scans → Paperless (Kopie 2)"):
|
||||||
|
form = _form_from_page(client.get(f"/rules/{rid}/copy").text)
|
||||||
|
assert form["name"] == expected
|
||||||
|
post(client, f"/rules/{rid}/copy", form)
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_validation_error_keeps_context(
|
||||||
|
client: TestClient, state: Any, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
rid, _ = _setup(state, tmp_path)
|
||||||
|
form = _form_from_page(client.get(f"/rules/{rid}/copy").text)
|
||||||
|
form["name"] = "Scans → Paperless" # schon vergeben
|
||||||
|
r = post(client, f"/rules/{rid}/copy", form)
|
||||||
|
assert r.status_code == 400 and "bereits vergeben" in r.text and "Kopie von" in r.text
|
||||||
|
with state.session_factory() as s:
|
||||||
|
assert len(s.scalars(select(Rule)).all()) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_of_missing_rule_redirects(client: TestClient) -> None:
|
||||||
|
assert client.get("/rules/4711/copy").headers["location"] == "/rules"
|
||||||
@@ -0,0 +1,383 @@
|
|||||||
|
"""Schlüsselverwaltung: erzeugen, importieren, auswählen, schützen."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from cryptography.hazmat.primitives import serialization
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ec, rsa
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.admin import AdminError, MountInput, delete_mount, save_mount
|
||||||
|
from netmover.credentials import (
|
||||||
|
delete_key,
|
||||||
|
fingerprint,
|
||||||
|
generate_key,
|
||||||
|
import_key,
|
||||||
|
list_keys,
|
||||||
|
rename_key,
|
||||||
|
)
|
||||||
|
from netmover.crypto import SecretBox, generate_ssh_keypair
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.models import Credential, CredentialKind, Mount, MountType
|
||||||
|
from netmover.mounting.manager import MountConfig, MountManager
|
||||||
|
from tests.sftp_stub import StubSFTPServer
|
||||||
|
|
||||||
|
|
||||||
|
def _sftp_input(**kw: object) -> MountInput:
|
||||||
|
data: dict[str, object] = {
|
||||||
|
"name": "paperless",
|
||||||
|
"type": MountType.SFTP,
|
||||||
|
"server": "127.0.0.1",
|
||||||
|
"share": "/consume",
|
||||||
|
"port": 22,
|
||||||
|
"username": "paperless",
|
||||||
|
"auth": CredentialKind.SSH_KEY,
|
||||||
|
}
|
||||||
|
data.update(kw)
|
||||||
|
return MountInput(**data) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
# -- Modul --------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_key(session: Session, secretbox: SecretBox) -> None:
|
||||||
|
key = generate_key(session, secretbox, "paperless-lxc")
|
||||||
|
assert key.managed and key.kind is CredentialKind.SSH_KEY
|
||||||
|
assert key.public_key and key.public_key.startswith("ssh-ed25519 ")
|
||||||
|
assert key.public_key.endswith(" netmover:paperless-lxc")
|
||||||
|
assert "PRIVATE" not in key.public_key
|
||||||
|
assert "OPENSSH PRIVATE KEY" in secretbox.decrypt(key.secret_enc)
|
||||||
|
info = list_keys(session)[0]
|
||||||
|
assert info.fingerprint.startswith("SHA256:") and info.used_by == []
|
||||||
|
assert info.authorized_keys_line.startswith(
|
||||||
|
'restrict,command="/usr/lib/openssh/sftp-server" ssh-ed25519 '
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fingerprint_matches_ssh_keygen(tmp_path: Path) -> None:
|
||||||
|
_, pub = generate_ssh_keypair("x")
|
||||||
|
f = tmp_path / "k.pub"
|
||||||
|
f.write_text(pub + "\n")
|
||||||
|
try:
|
||||||
|
out = subprocess.run(
|
||||||
|
["ssh-keygen", "-lf", str(f)], # noqa: S607 – Abgleich mit OpenSSH
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True,
|
||||||
|
).stdout
|
||||||
|
except FileNotFoundError:
|
||||||
|
pytest.skip("ssh-keygen nicht installiert")
|
||||||
|
assert fingerprint(pub) == out.split()[1]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"kind", ["ed25519-openssh-pass", "rsa-pem", "ecdsa-pem-pass", "rsa-openssh"]
|
||||||
|
)
|
||||||
|
def test_import_formats(session: Session, secretbox: SecretBox, kind: str) -> None:
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ed25519
|
||||||
|
|
||||||
|
if kind.startswith("ed25519"):
|
||||||
|
key = ed25519.Ed25519PrivateKey.generate()
|
||||||
|
elif kind.startswith("rsa"):
|
||||||
|
key = rsa.generate_private_key(public_exponent=65537, key_size=2048) # type: ignore[assignment]
|
||||||
|
else:
|
||||||
|
key = ec.generate_private_key(ec.SECP256R1()) # type: ignore[assignment]
|
||||||
|
enc = (
|
||||||
|
serialization.BestAvailableEncryption(b"geheim")
|
||||||
|
if kind.endswith("pass")
|
||||||
|
else serialization.NoEncryption()
|
||||||
|
)
|
||||||
|
fmt = (
|
||||||
|
serialization.PrivateFormat.OpenSSH
|
||||||
|
if "openssh" in kind
|
||||||
|
else serialization.PrivateFormat.PKCS8
|
||||||
|
)
|
||||||
|
pem = key.private_bytes(serialization.Encoding.PEM, fmt, enc).decode()
|
||||||
|
cred = import_key(session, secretbox, kind, pem, "geheim" if kind.endswith("pass") else "")
|
||||||
|
expected_pub = (
|
||||||
|
key.public_key()
|
||||||
|
.public_bytes(serialization.Encoding.OpenSSH, serialization.PublicFormat.OpenSSH)
|
||||||
|
.decode()
|
||||||
|
)
|
||||||
|
assert cred.public_key is not None and cred.public_key.startswith(expected_pub)
|
||||||
|
stored = secretbox.decrypt(cred.secret_enc)
|
||||||
|
assert "OPENSSH PRIVATE KEY" in stored
|
||||||
|
# ohne Passphrase gespeichert → von paramiko direkt nutzbar
|
||||||
|
from netmover.mounting.sftp import load_private_key
|
||||||
|
|
||||||
|
assert load_private_key(stored).get_base64() == expected_pub.split()[1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_errors(session: Session, secretbox: SecretBox) -> None:
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ed25519
|
||||||
|
|
||||||
|
key = ed25519.Ed25519PrivateKey.generate()
|
||||||
|
protected = key.private_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PrivateFormat.OpenSSH,
|
||||||
|
serialization.BestAvailableEncryption(b"richtig"),
|
||||||
|
).decode()
|
||||||
|
with pytest.raises(AdminError, match="Passphrase angeben"):
|
||||||
|
import_key(session, secretbox, "a", protected, "")
|
||||||
|
with pytest.raises(AdminError, match="Passphrase falsch"):
|
||||||
|
import_key(session, secretbox, "a", protected, "falsch")
|
||||||
|
_, pub = generate_ssh_keypair()
|
||||||
|
with pytest.raises(AdminError, match="öffentlicher Schlüssel"):
|
||||||
|
import_key(session, secretbox, "a", pub)
|
||||||
|
with pytest.raises(AdminError, match="Kein lesbarer"):
|
||||||
|
import_key(
|
||||||
|
session, secretbox, "a", "-----BEGIN PRIVATE KEY-----\nMüll\n-----END PRIVATE KEY-----"
|
||||||
|
)
|
||||||
|
weak = (
|
||||||
|
rsa.generate_private_key(public_exponent=65537, key_size=1024) # noqa: S505 – absichtlich schwach
|
||||||
|
.private_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PrivateFormat.PKCS8,
|
||||||
|
serialization.NoEncryption(),
|
||||||
|
)
|
||||||
|
.decode()
|
||||||
|
)
|
||||||
|
with pytest.raises(AdminError, match="zu schwach"):
|
||||||
|
import_key(session, secretbox, "a", weak)
|
||||||
|
assert list_keys(session) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_names(session: Session, secretbox: SecretBox) -> None:
|
||||||
|
generate_key(session, secretbox, "eins")
|
||||||
|
with pytest.raises(AdminError, match="existieren bereits"):
|
||||||
|
generate_key(session, secretbox, "eins")
|
||||||
|
with pytest.raises(AdminError, match="reserviert"):
|
||||||
|
generate_key(session, secretbox, "mount:x")
|
||||||
|
zwei = generate_key(session, secretbox, "zwei")
|
||||||
|
rename_key(session, zwei.id, "drei")
|
||||||
|
assert [k.name for k in list_keys(session)] == ["drei", "eins"]
|
||||||
|
with pytest.raises(AdminError):
|
||||||
|
rename_key(session, zwei.id, "eins")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Zusammenspiel mit Anbindungen ----------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_mount_uses_shared_key_with_own_username(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
key = generate_key(session, secretbox, "lxc")
|
||||||
|
a, err = save_mount(session, secretbox, manager, _sftp_input(key_id=str(key.id)))
|
||||||
|
assert not err and a is not None
|
||||||
|
b, err = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_sftp_input(name="anderer", server="10.0.0.9", username="root", key_id=str(key.id)),
|
||||||
|
)
|
||||||
|
assert not err and b is not None
|
||||||
|
assert a.credential is key and b.credential is key
|
||||||
|
assert (a.username, b.username) == ("paperless", "root")
|
||||||
|
assert key.username == "" # der Schlüssel selbst bleibt unverändert
|
||||||
|
assert list_keys(session)[0].used_by == ["anderer", "paperless"]
|
||||||
|
|
||||||
|
# Schlüssel in Benutzung → nicht löschbar
|
||||||
|
with pytest.raises(AdminError, match="noch benutzt"):
|
||||||
|
delete_key(session, key.id)
|
||||||
|
# Anbindung löschen lässt den gespeicherten Schlüssel stehen
|
||||||
|
delete_mount(session, manager, a.id)
|
||||||
|
delete_mount(session, manager, b.id)
|
||||||
|
assert session.get(Credential, key.id) is not None
|
||||||
|
delete_key(session, key.id)
|
||||||
|
assert session.get(Credential, key.id) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_switching_between_own_and_shared_key(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
m, _ = save_mount(session, secretbox, manager, _sftp_input()) # eigener Schlüssel
|
||||||
|
assert m is not None and m.credential is not None and not m.credential.managed
|
||||||
|
own_id = m.credential.id
|
||||||
|
key = generate_key(session, secretbox, "geteilt")
|
||||||
|
m, _ = save_mount(session, secretbox, manager, _sftp_input(key_id=str(key.id)), m.id)
|
||||||
|
assert m is not None and m.credential is key
|
||||||
|
assert session.get(Credential, own_id) is None # eigener Schlüssel aufgeräumt
|
||||||
|
# Wechsel auf Passwort darf den gespeicherten Schlüssel nicht verändern
|
||||||
|
secret_before = key.secret_enc
|
||||||
|
m, _ = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_sftp_input(auth=CredentialKind.PASSWORD, password="pw123"),
|
||||||
|
m.id,
|
||||||
|
)
|
||||||
|
assert m is not None and m.credential is not None and m.credential is not key
|
||||||
|
assert not m.credential.managed and m.credential.username == "paperless"
|
||||||
|
assert m.username is None
|
||||||
|
assert key.secret_enc == secret_before and key.kind is CredentialKind.SSH_KEY
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_or_foreign_key_id_is_rejected(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
own, _ = save_mount(session, secretbox, manager, _sftp_input(name="a"))
|
||||||
|
assert own is not None and own.credential is not None
|
||||||
|
for bad in ("4711", str(own.credential.id)): # nicht vorhanden / kein gespeicherter Schlüssel
|
||||||
|
m, err = save_mount(session, secretbox, manager, _sftp_input(name="b", key_id=bad))
|
||||||
|
assert m is None and "key_id" in err
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def server(tmp_path: Path) -> Iterator[StubSFTPServer]:
|
||||||
|
root = tmp_path / "sftp-root"
|
||||||
|
(root / "consume").mkdir(parents=True)
|
||||||
|
srv = StubSFTPServer(root, username="paperless").start()
|
||||||
|
yield srv
|
||||||
|
srv.stop()
|
||||||
|
|
||||||
|
|
||||||
|
def test_manager_connects_with_shared_key_and_mount_username(
|
||||||
|
server: StubSFTPServer,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
secretbox: SecretBox,
|
||||||
|
manager: MountManager,
|
||||||
|
) -> None:
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
key = import_key(s, secretbox, "lxc", server.client_private_key)
|
||||||
|
m, err = save_mount(
|
||||||
|
s, secretbox, manager, _sftp_input(port=server.port, key_id=str(key.id))
|
||||||
|
)
|
||||||
|
assert not err and m is not None
|
||||||
|
mid = m.id
|
||||||
|
with session_factory() as s:
|
||||||
|
mm = s.get(Mount, mid)
|
||||||
|
assert mm is not None
|
||||||
|
cfg = MountConfig.from_model(mm)
|
||||||
|
from netmover.admin import credential_for_test, mount_input_from_model
|
||||||
|
|
||||||
|
with session_factory() as s:
|
||||||
|
mm = s.get(Mount, mid)
|
||||||
|
assert mm is not None
|
||||||
|
data = mount_input_from_model(mm)
|
||||||
|
assert data.key_id == str(mm.cred_id) and data.username == "paperless"
|
||||||
|
cred = credential_for_test(data, mm, secretbox, s)
|
||||||
|
assert cred is not None and cred.username == "paperless"
|
||||||
|
assert manager.check(cfg, cred, mount_id=mid).ok
|
||||||
|
assert manager.check_alive(mid) is None # Regellauf-Weg: Benutzer der Anbindung
|
||||||
|
|
||||||
|
# Gegenprobe: falscher Benutzer an der Anbindung → Anmeldung abgelehnt
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
mm = s.get(Mount, mid)
|
||||||
|
assert mm is not None
|
||||||
|
mm.username = "jemand"
|
||||||
|
from netmover.mounting.manager import MountUnavailable
|
||||||
|
|
||||||
|
with pytest.raises(MountUnavailable, match="Anmeldung abgelehnt"):
|
||||||
|
manager.check_alive(mid)
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_keys_ignores_per_mount_credentials(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
save_mount(session, secretbox, manager, _sftp_input())
|
||||||
|
assert list_keys(session) == []
|
||||||
|
assert session.scalars(select(Credential)).one().managed is False
|
||||||
|
|
||||||
|
|
||||||
|
# -- Gespeicherte Passwörter (0.5.0) ----------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _cifs(**kw: object) -> MountInput:
|
||||||
|
data: dict[str, object] = {
|
||||||
|
"name": "nas",
|
||||||
|
"type": MountType.CIFS,
|
||||||
|
"server": "10.0.10.6",
|
||||||
|
"share": "scans",
|
||||||
|
}
|
||||||
|
data.update(kw)
|
||||||
|
return MountInput(**data) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_stored_password_shared_by_mounts(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
from netmover.credentials import create_password, list_passwords, update_password
|
||||||
|
|
||||||
|
cred = create_password(session, secretbox, "NAS-Scanner", "scanner", "Pa$$,wort", "WG")
|
||||||
|
a, err = save_mount(session, secretbox, manager, _cifs(cred_id=str(cred.id)))
|
||||||
|
assert not err and a is not None
|
||||||
|
b, err = save_mount(
|
||||||
|
session, secretbox, manager, _cifs(name="archiv", share="archiv", cred_id=str(cred.id))
|
||||||
|
)
|
||||||
|
assert not err and b is not None
|
||||||
|
assert a.credential is cred and b.credential is cred and a.username is None
|
||||||
|
assert list_passwords(session)[0].used_by == ["archiv", "nas"]
|
||||||
|
|
||||||
|
# ein neues Passwort gilt für beide
|
||||||
|
update_password(session, secretbox, cred.id, "scanner", "neu!", "WG")
|
||||||
|
from netmover.admin import credential_for_test, mount_input_from_model
|
||||||
|
|
||||||
|
for m in (a, b):
|
||||||
|
data = mount_input_from_model(m)
|
||||||
|
assert data.cred_id == str(cred.id) and data.username == "scanner"
|
||||||
|
c = credential_for_test(data, m, secretbox, session)
|
||||||
|
assert c is not None and c.secret == "neu!" and c.domain == "WG"
|
||||||
|
|
||||||
|
# Anbindung löschen lässt die gespeicherten Zugangsdaten stehen
|
||||||
|
delete_mount(session, manager, a.id)
|
||||||
|
from netmover.credentials import delete_credential
|
||||||
|
|
||||||
|
with pytest.raises(AdminError, match="noch benutzt"):
|
||||||
|
delete_credential(session, cred.id)
|
||||||
|
delete_mount(session, manager, b.id)
|
||||||
|
assert delete_credential(session, cred.id) == "NAS-Scanner"
|
||||||
|
|
||||||
|
|
||||||
|
def test_switch_from_own_to_stored_password_cleans_up(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
from netmover.credentials import create_password
|
||||||
|
|
||||||
|
m, _ = save_mount(session, secretbox, manager, _cifs(username="alt", password="pw"))
|
||||||
|
assert m is not None and m.credential is not None
|
||||||
|
own_id = m.credential.id
|
||||||
|
cred = create_password(session, secretbox, "geteilt", "scanner", "x")
|
||||||
|
m, err = save_mount(session, secretbox, manager, _cifs(cred_id=str(cred.id)), m.id)
|
||||||
|
assert not err and m is not None and m.credential is cred
|
||||||
|
assert session.get(Credential, own_id) is None
|
||||||
|
# zurück auf eigene Angaben: gespeicherter Eintrag bleibt unverändert
|
||||||
|
m, err = save_mount(session, secretbox, manager, _cifs(username="eigen", password="pw2"), m.id)
|
||||||
|
assert not err and m is not None and not m.credential.managed # type: ignore[union-attr]
|
||||||
|
assert cred.username == "scanner" and secretbox.decrypt(cred.secret_enc) == "x"
|
||||||
|
|
||||||
|
|
||||||
|
def test_stored_password_validation(session: Session, secretbox: SecretBox) -> None:
|
||||||
|
from netmover.credentials import create_password, update_password
|
||||||
|
|
||||||
|
with pytest.raises(AdminError, match="Passwort fehlt"):
|
||||||
|
create_password(session, secretbox, "a", "u", "")
|
||||||
|
with pytest.raises(AdminError):
|
||||||
|
create_password(session, secretbox, "a", "u", "zeile\numbruch")
|
||||||
|
cred = create_password(session, secretbox, "a", "u", "pw")
|
||||||
|
update_password(session, secretbox, cred.id, "u2", "") # leer = unverändert
|
||||||
|
assert cred.username == "u2" and secretbox.decrypt(cred.secret_enc) == "pw"
|
||||||
|
|
||||||
|
|
||||||
|
def test_stored_password_mount_actually_mounts(
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
secretbox: SecretBox,
|
||||||
|
manager: MountManager,
|
||||||
|
fake_backend: object,
|
||||||
|
) -> None:
|
||||||
|
from netmover.credentials import create_password
|
||||||
|
|
||||||
|
fake_backend.export("10.0.10.6", "scans") # type: ignore[attr-defined]
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
cred = create_password(s, secretbox, "NAS", "scanner", "Pa$$wort", "WG")
|
||||||
|
m, _ = save_mount(s, secretbox, manager, _cifs(cred_id=str(cred.id)))
|
||||||
|
assert m is not None
|
||||||
|
mid = m.id
|
||||||
|
manager.mount(mid)
|
||||||
|
spec = fake_backend.calls[-1].spec # type: ignore[attr-defined]
|
||||||
|
assert spec.credentials.username == "scanner" and spec.credentials.password == "Pa$$wort"
|
||||||
|
assert spec.credentials.domain == "WG"
|
||||||
@@ -251,7 +251,7 @@ def test_source_not_deletable_keeps_target_and_prevents_duplicate(
|
|||||||
raise PermissionError(errno.EACCES, "Permission denied", str(path))
|
raise PermissionError(errno.EACCES, "Permission denied", str(path))
|
||||||
real_unlink(path) # type: ignore[arg-type]
|
real_unlink(path) # type: ignore[arg-type]
|
||||||
|
|
||||||
monkeypatch.setattr("netmover.mover.engine.os.unlink", deny)
|
monkeypatch.setattr("netmover.mover.sources.os.unlink", deny)
|
||||||
first = run(engine, snap(rule_id), src, dst)
|
first = run(engine, snap(rule_id), src, dst)
|
||||||
assert first.items[0].status is ItemStatus.FAILED
|
assert first.items[0].status is ItemStatus.FAILED
|
||||||
assert "nicht löschbar" in (first.items[0].message or "")
|
assert "nicht löschbar" in (first.items[0].message or "")
|
||||||
|
|||||||
@@ -110,6 +110,12 @@ def test_is_mounted_reads_mountinfo(tmp_path: Path) -> None:
|
|||||||
),
|
),
|
||||||
("nfs4", "mount.nfs4: access denied by server while mounting nas:/x", "NFS-Berechtigungen"),
|
("nfs4", "mount.nfs4: access denied by server while mounting nas:/x", "NFS-Berechtigungen"),
|
||||||
("cifs", "mount error(112): Host is down", "nicht erreichbar"),
|
("cifs", "mount error(112): Host is down", "nicht erreichbar"),
|
||||||
|
# Praxisfall: Synology mit „Maximales SMB-Protokoll: SMB2“
|
||||||
|
(
|
||||||
|
"cifs",
|
||||||
|
"mount error(95): Operation not supported\nRefer to the mount.cifs(8) manual page",
|
||||||
|
"vers=2.0",
|
||||||
|
),
|
||||||
("cifs", "mount: only root can do that", "CAP_SYS_ADMIN"),
|
("cifs", "mount: only root can do that", "CAP_SYS_ADMIN"),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
@@ -121,7 +127,7 @@ def _alive(pid: int) -> bool:
|
|||||||
try:
|
try:
|
||||||
with open(f"/proc/{pid}/stat") as fh:
|
with open(f"/proc/{pid}/stat") as fh:
|
||||||
return fh.read().split(")")[-1].split()[0] != "Z"
|
return fh.read().split(")")[-1].split()[0] != "Z"
|
||||||
except FileNotFoundError:
|
except (FileNotFoundError, ProcessLookupError): # verschwindet beim Lesen → ESRCH
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,304 @@
|
|||||||
|
"""E-Mail-Benachrichtigung gegen einen echten SMTP-Server im Testprozess (aiosmtpd)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import datetime
|
||||||
|
import socket
|
||||||
|
import ssl
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from email import message_from_bytes
|
||||||
|
from email.policy import default
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from aiosmtpd.controller import Controller
|
||||||
|
from aiosmtpd.smtp import AuthResult, LoginPassword
|
||||||
|
from cryptography import x509
|
||||||
|
from cryptography.hazmat.primitives import hashes, serialization
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ec
|
||||||
|
from cryptography.x509.oid import NameOID
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
|
from netmover.mail import MailConfigError, parse_recipients, smtp_send
|
||||||
|
from netmover.notify import Notification, Notifier
|
||||||
|
from netmover.settings_store import AppSettings, NotifyMode, load_settings, save_settings
|
||||||
|
from tests.webutil import post
|
||||||
|
|
||||||
|
|
||||||
|
class Inbox:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.messages: list[tuple[list[str], Any]] = []
|
||||||
|
|
||||||
|
async def handle_DATA(self, server: Any, session: Any, envelope: Any) -> str:
|
||||||
|
self.messages.append(
|
||||||
|
(envelope.rcpt_tos, message_from_bytes(envelope.content, policy=default))
|
||||||
|
)
|
||||||
|
return "250 OK"
|
||||||
|
|
||||||
|
|
||||||
|
def _free_port() -> int:
|
||||||
|
s = socket.socket()
|
||||||
|
s.bind(("127.0.0.1", 0))
|
||||||
|
port: int = s.getsockname()[1]
|
||||||
|
s.close()
|
||||||
|
return port
|
||||||
|
|
||||||
|
|
||||||
|
def _auth(server: Any, session: Any, envelope: Any, mechanism: str, data: Any) -> AuthResult:
|
||||||
|
ok = (
|
||||||
|
isinstance(data, LoginPassword)
|
||||||
|
and data.login == b"netmover"
|
||||||
|
and data.password == b"Pa$$,wort"
|
||||||
|
)
|
||||||
|
return AuthResult(success=ok, handled=False)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def tls_files(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, Path]:
|
||||||
|
d = tmp_path_factory.mktemp("tls")
|
||||||
|
key = ec.generate_private_key(ec.SECP256R1())
|
||||||
|
name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "localhost")])
|
||||||
|
now = datetime.datetime.now(datetime.UTC)
|
||||||
|
cert = (
|
||||||
|
x509.CertificateBuilder()
|
||||||
|
.subject_name(name)
|
||||||
|
.issuer_name(name)
|
||||||
|
.public_key(key.public_key())
|
||||||
|
.serial_number(x509.random_serial_number())
|
||||||
|
.not_valid_before(now - datetime.timedelta(days=1))
|
||||||
|
.not_valid_after(now + datetime.timedelta(days=1))
|
||||||
|
.add_extension(x509.SubjectAlternativeName([x509.DNSName("localhost")]), critical=False)
|
||||||
|
.add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
|
||||||
|
.sign(key, hashes.SHA256())
|
||||||
|
)
|
||||||
|
(d / "cert.pem").write_bytes(cert.public_bytes(serialization.Encoding.PEM))
|
||||||
|
(d / "key.pem").write_bytes(
|
||||||
|
key.private_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PrivateFormat.PKCS8,
|
||||||
|
serialization.NoEncryption(),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return d / "cert.pem", d / "key.pem"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def trust(tls_files: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""Client vertraut unserem Test-Zertifikat (Prüfung bleibt eingeschaltet)."""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"netmover.mail.tls_context", lambda: ssl.create_default_context(cafile=str(tls_files[0]))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _server(mode: str, tls_files: tuple[Path, Path]) -> tuple[Controller, Inbox, int]:
|
||||||
|
inbox, port = Inbox(), _free_port()
|
||||||
|
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
|
||||||
|
ctx.load_cert_chain(*map(str, tls_files))
|
||||||
|
kw: dict[str, Any] = {
|
||||||
|
"hostname": "localhost",
|
||||||
|
"port": port,
|
||||||
|
"authenticator": _auth,
|
||||||
|
"auth_exclude_mechanism": ["LOGIN"],
|
||||||
|
}
|
||||||
|
if mode == "starttls":
|
||||||
|
kw.update(tls_context=ctx, require_starttls=True)
|
||||||
|
elif mode == "ssl":
|
||||||
|
# implizites TLS erkennt aiosmtpd nicht als „verschlüsselt“ – AUTH trotzdem anbieten
|
||||||
|
kw.update(ssl_context=ctx, auth_require_tls=False)
|
||||||
|
else:
|
||||||
|
kw.update(auth_require_tls=False)
|
||||||
|
controller = Controller(inbox, **kw)
|
||||||
|
controller.start()
|
||||||
|
return controller, inbox, port
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(params=["starttls", "ssl", "none"])
|
||||||
|
def smtp_server(
|
||||||
|
request: pytest.FixtureRequest, tls_files: tuple[Path, Path], trust: None
|
||||||
|
) -> Iterator[tuple[str, Inbox, int]]:
|
||||||
|
controller, inbox, port = _server(request.param, tls_files)
|
||||||
|
yield request.param, inbox, port
|
||||||
|
controller.stop()
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(mode: str, port: int, **kw: Any) -> AppSettings:
|
||||||
|
data: dict[str, Any] = {
|
||||||
|
"email_enabled": True,
|
||||||
|
"smtp_host": "localhost",
|
||||||
|
"smtp_port": port,
|
||||||
|
"smtp_security": mode,
|
||||||
|
"smtp_user": "netmover",
|
||||||
|
"smtp_from": "netmover@example.org",
|
||||||
|
"smtp_to": "ich@example.org; admin@example.org",
|
||||||
|
"notify_mode": NotifyMode.EVERY,
|
||||||
|
}
|
||||||
|
data.update(kw)
|
||||||
|
return AppSettings(**data)
|
||||||
|
|
||||||
|
|
||||||
|
def test_smtp_send_all_security_modes(smtp_server: tuple[str, Inbox, int]) -> None:
|
||||||
|
mode, inbox, port = smtp_server
|
||||||
|
smtp_send(_cfg(mode, port), "Pa$$,wort", "NetMover: Anbindung „NAS“ gestört", "Host is down")
|
||||||
|
((rcpts, msg),) = inbox.messages
|
||||||
|
assert rcpts == ["ich@example.org", "admin@example.org"]
|
||||||
|
assert msg["Subject"] == "NetMover: Anbindung „NAS“ gestört"
|
||||||
|
assert msg["From"] == "netmover@example.org"
|
||||||
|
body = msg.get_content()
|
||||||
|
assert "Host is down" in body and "NetMover auf" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_wrong_password_raises(smtp_server: tuple[str, Inbox, int]) -> None:
|
||||||
|
import smtplib
|
||||||
|
|
||||||
|
mode, inbox, port = smtp_server
|
||||||
|
with pytest.raises(smtplib.SMTPAuthenticationError):
|
||||||
|
smtp_send(_cfg(mode, port), "falsch", "x", "y")
|
||||||
|
assert inbox.messages == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_certificate_is_verified(tls_files: tuple[Path, Path]) -> None:
|
||||||
|
"""Ohne Vertrauen in das Zertifikat wird abgelehnt – keine stille Herabstufung."""
|
||||||
|
controller, inbox, port = _server("starttls", tls_files)
|
||||||
|
try:
|
||||||
|
with pytest.raises(ssl.SSLCertVerificationError):
|
||||||
|
smtp_send(_cfg("starttls", port), "Pa$$,wort", "x", "y")
|
||||||
|
finally:
|
||||||
|
controller.stop()
|
||||||
|
assert inbox.messages == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_recipients() -> None:
|
||||||
|
assert parse_recipients("a@b.de, c@d.org;e@f.net\n") == ["a@b.de", "c@d.org", "e@f.net"]
|
||||||
|
with pytest.raises(MailConfigError, match="kein-mail"):
|
||||||
|
parse_recipients("a@b.de, kein-mail")
|
||||||
|
|
||||||
|
|
||||||
|
def test_notifier_sends_webhook_and_mail(
|
||||||
|
session_factory: sessionmaker[Session], secretbox: SecretBox
|
||||||
|
) -> None:
|
||||||
|
hooks: list[Any] = []
|
||||||
|
mails: list[Any] = []
|
||||||
|
cfg = _cfg(
|
||||||
|
"starttls",
|
||||||
|
587,
|
||||||
|
smtp_password_enc=secretbox.encrypt("Pa$$,wort"),
|
||||||
|
webhook_url="http://hook.invalid/x",
|
||||||
|
)
|
||||||
|
save_settings(session_factory, cfg)
|
||||||
|
n = Notifier(
|
||||||
|
session_factory,
|
||||||
|
sender=lambda *a: hooks.append(a),
|
||||||
|
mailer=lambda *a: mails.append(a),
|
||||||
|
secrets=secretbox,
|
||||||
|
background=False,
|
||||||
|
)
|
||||||
|
n.send(Notification("mount.error", "NetMover: Anbindung „NAS“ gestört", "weg"))
|
||||||
|
assert len(hooks) == 1 and len(mails) == 1
|
||||||
|
_cfg_used, password, subject, body = mails[0]
|
||||||
|
assert password == "Pa$$,wort" and subject.endswith("gestört") and body == "weg"
|
||||||
|
|
||||||
|
|
||||||
|
def test_mail_only_and_mode_off(
|
||||||
|
session_factory: sessionmaker[Session], secretbox: SecretBox
|
||||||
|
) -> None:
|
||||||
|
mails: list[Any] = []
|
||||||
|
n = Notifier(
|
||||||
|
session_factory, mailer=lambda *a: mails.append(a), secrets=secretbox, background=False
|
||||||
|
)
|
||||||
|
save_settings(session_factory, _cfg("starttls", 587, notify_mode=NotifyMode.OFF))
|
||||||
|
n.send(Notification("x", "t", "m"))
|
||||||
|
save_settings(session_factory, _cfg("starttls", 587, email_enabled=False))
|
||||||
|
n.send(Notification("x", "t", "m"))
|
||||||
|
assert mails == []
|
||||||
|
save_settings(session_factory, _cfg("starttls", 587)) # nur E-Mail, kein Webhook
|
||||||
|
n.send(Notification("x", "t", "m"))
|
||||||
|
assert len(mails) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_mail_failure_is_swallowed(
|
||||||
|
session_factory: sessionmaker[Session], secretbox: SecretBox
|
||||||
|
) -> None:
|
||||||
|
def broken(*a: Any) -> None:
|
||||||
|
raise ConnectionRefusedError("weg")
|
||||||
|
|
||||||
|
save_settings(session_factory, _cfg("starttls", 587))
|
||||||
|
Notifier(session_factory, mailer=broken, secrets=secretbox, background=False).send(
|
||||||
|
Notification("x", "t", "m")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Einstellungsseite ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _form(**kw: str) -> dict[str, str]:
|
||||||
|
data = {
|
||||||
|
"retention_days": "30",
|
||||||
|
"notify_mode": "every",
|
||||||
|
"notify_after": "3",
|
||||||
|
"webhook_url": "",
|
||||||
|
"webhook_content_type": "application/json",
|
||||||
|
"webhook_template": "$message",
|
||||||
|
"email_enabled": "on",
|
||||||
|
"smtp_host": "smtp.example.org",
|
||||||
|
"smtp_port": "587",
|
||||||
|
"smtp_security": "starttls",
|
||||||
|
"smtp_user": "netmover",
|
||||||
|
"smtp_password": "Geheim,123!",
|
||||||
|
"smtp_from": "netmover@example.org",
|
||||||
|
"smtp_to": "ich@example.org",
|
||||||
|
}
|
||||||
|
data.update(kw)
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def test_settings_store_encrypted_password(client: TestClient, state: Any) -> None:
|
||||||
|
assert post(client, "/settings", _form()).status_code == 303
|
||||||
|
cfg = load_settings(state.session_factory)
|
||||||
|
assert cfg.email_enabled and "Geheim" not in cfg.smtp_password_enc
|
||||||
|
assert state.secrets.decrypt(cfg.smtp_password_enc) == "Geheim,123!"
|
||||||
|
page = client.get("/settings").text
|
||||||
|
assert "Geheim,123!" not in page and 'placeholder="••••••"' in page
|
||||||
|
# leer lassen = unverändert
|
||||||
|
post(client, "/settings", _form(smtp_password=""))
|
||||||
|
assert (
|
||||||
|
state.secrets.decrypt(load_settings(state.session_factory).smtp_password_enc)
|
||||||
|
== "Geheim,123!"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_settings_validation(client: TestClient) -> None:
|
||||||
|
r = post(
|
||||||
|
client,
|
||||||
|
"/settings",
|
||||||
|
_form(smtp_to="ich@example.org, quatsch", smtp_port="99999", smtp_from="kein-absender"),
|
||||||
|
)
|
||||||
|
assert r.status_code == 400
|
||||||
|
for msg in ("Ungültige Adresse: quatsch", "Port zwischen", "Keine gültige E-Mail-Adresse"):
|
||||||
|
assert msg in r.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_mail_test_button(
|
||||||
|
client: TestClient, state: Any, tls_files: tuple[Path, Path], trust: None
|
||||||
|
) -> None:
|
||||||
|
controller, inbox, port = _server("starttls", tls_files)
|
||||||
|
try:
|
||||||
|
post(
|
||||||
|
client,
|
||||||
|
"/settings",
|
||||||
|
_form(smtp_host="localhost", smtp_port=str(port), smtp_password="Pa$$,wort"),
|
||||||
|
)
|
||||||
|
r = post(client, "/settings/mail-test")
|
||||||
|
assert "Gesendet an ich@example.org" in r.text
|
||||||
|
post(
|
||||||
|
client,
|
||||||
|
"/settings",
|
||||||
|
_form(smtp_host="localhost", smtp_port=str(port), smtp_password="falsch"),
|
||||||
|
)
|
||||||
|
r = post(client, "/settings/mail-test")
|
||||||
|
assert "Fehlgeschlagen" in r.text and "falsch" not in r.text
|
||||||
|
finally:
|
||||||
|
controller.stop()
|
||||||
|
assert len(inbox.messages) == 1
|
||||||
@@ -0,0 +1,274 @@
|
|||||||
|
"""Migrationen auf Datenbanken MIT Daten.
|
||||||
|
|
||||||
|
Praxisbefund beim Update 0.1.1 → 0.2.0: Migration 0002 baut auf SQLite Tabellen neu
|
||||||
|
auf (batch mode) und löscht dafür die alte Tabelle. Mit eingeschalteter
|
||||||
|
Fremdschlüsselprüfung scheiterte das, sobald Anbindungen auf Zugangsdaten
|
||||||
|
verwiesen – die Tests liefen bis dahin nur auf leeren Datenbanken. Und weil die
|
||||||
|
Migration nicht in einer Transaktion lief, blieb ein halber Zustand zurück.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlite3
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from alembic import command
|
||||||
|
|
||||||
|
from netmover.db import alembic_config, upgrade
|
||||||
|
|
||||||
|
|
||||||
|
def _head() -> str:
|
||||||
|
from alembic.script import ScriptDirectory
|
||||||
|
|
||||||
|
head = ScriptDirectory.from_config(alembic_config("sqlite://")).get_current_head()
|
||||||
|
assert head is not None
|
||||||
|
return head
|
||||||
|
|
||||||
|
|
||||||
|
def _seed(db: Path) -> None:
|
||||||
|
"""Datenbank auf Stand 0001 mit verknüpften Daten, wie auf einem echten System."""
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
c.execute("PRAGMA foreign_keys=ON")
|
||||||
|
c.executescript(
|
||||||
|
"""
|
||||||
|
INSERT INTO credentials (id, name, kind, username, secret_enc, created_at)
|
||||||
|
VALUES (1, 'mount:nas', 'password', 'scanner', 'gAAAA-x', '2026-10-10 20:00:00'),
|
||||||
|
(2, 'mount:paperless', 'ssh_key', 'root', 'gAAAA-y', '2026-10-10 20:00:00');
|
||||||
|
INSERT INTO mounts (id, name, slug, type, server, share, mountpoint, cred_id,
|
||||||
|
extra_options, auto_mount, enabled, state)
|
||||||
|
VALUES (1, 'NAS', 'nas', 'cifs', '10.0.10.6', 'Scans', '/mnt/netmover/nas', 1,
|
||||||
|
'', 1, 1, 'mounted'),
|
||||||
|
(2, 'paperless', 'paperless', 'sftp', '10.0.10.17', '/consume', NULL, 2,
|
||||||
|
'', 0, 1, 'mounted');
|
||||||
|
INSERT INTO rules (id, name, enabled, src_mount_id, src_subpath, dst_mount_id, dst_path,
|
||||||
|
pattern, pattern_type, recursive, interval_seconds, stability_seconds,
|
||||||
|
min_age_seconds, op_timeout_seconds, action, on_collision, dry_run,
|
||||||
|
quarantine_after, consecutive_failures, created_at)
|
||||||
|
VALUES (1, 'Scans → Paperless', 0, 1, '', 2, '', '*.pdf', 'glob', 0, 30, 5, 0, 120,
|
||||||
|
'move', 'rename', 0, 3, 0, '2026-10-10 20:00:00');
|
||||||
|
INSERT INTO runs (id, rule_id, started_at, status, "trigger", dry_run, seen, moved,
|
||||||
|
skipped, failed)
|
||||||
|
VALUES (1, 1, '2026-10-10 20:00:00', 'ok', 'manual', 1, 3, 3, 0, 0);
|
||||||
|
INSERT INTO run_items (run_id, src, status) VALUES (1, 'a.pdf', 'planned');
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
c.commit()
|
||||||
|
c.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def db_0001(tmp_path: Path) -> tuple[str, Path]:
|
||||||
|
db = tmp_path / "alt.db"
|
||||||
|
url = f"sqlite:///{db}"
|
||||||
|
upgrade(url, "0001")
|
||||||
|
_seed(db)
|
||||||
|
return url, db
|
||||||
|
|
||||||
|
|
||||||
|
def _state(db: Path) -> dict[str, object]:
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
try:
|
||||||
|
return {
|
||||||
|
"version": c.execute("SELECT version_num FROM alembic_version").fetchone()[0],
|
||||||
|
"tmp_tables": c.execute(
|
||||||
|
"SELECT name FROM sqlite_master WHERE name LIKE '_alembic_tmp%'"
|
||||||
|
).fetchall(),
|
||||||
|
"cred_columns": [r[1] for r in c.execute("PRAGMA table_info(credentials)")],
|
||||||
|
"mounts": c.execute("SELECT name, cred_id FROM mounts ORDER BY id").fetchall(),
|
||||||
|
"rules": c.execute("SELECT name, src_mount_id, dst_mount_id FROM rules").fetchall(),
|
||||||
|
"run_items": c.execute("SELECT COUNT(*) FROM run_items").fetchone()[0],
|
||||||
|
"fk_violations": c.execute("PRAGMA foreign_key_check").fetchall(),
|
||||||
|
}
|
||||||
|
finally:
|
||||||
|
c.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_with_linked_data(db_0001: tuple[str, Path]) -> None:
|
||||||
|
url, db = db_0001
|
||||||
|
upgrade(url)
|
||||||
|
st = _state(db)
|
||||||
|
assert st["version"] == _head()
|
||||||
|
assert st["tmp_tables"] == [] and st["fk_violations"] == []
|
||||||
|
assert "managed" in st["cred_columns"] # type: ignore[operator]
|
||||||
|
assert st["mounts"] == [("NAS", 1), ("paperless", 2)]
|
||||||
|
assert st["rules"] == [("Scans → Paperless", 1, 2)]
|
||||||
|
assert st["run_items"] == 1
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
assert c.execute("SELECT managed FROM credentials ORDER BY id").fetchall() == [(0,), (0,)]
|
||||||
|
assert c.execute("SELECT username FROM mounts ORDER BY id").fetchall() == [(None,), (None,)]
|
||||||
|
c.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_downgrade_with_linked_data(db_0001: tuple[str, Path]) -> None:
|
||||||
|
url, db = db_0001
|
||||||
|
upgrade(url)
|
||||||
|
command.downgrade(alembic_config(url), "0001")
|
||||||
|
st = _state(db)
|
||||||
|
assert st["version"] == "0001" and st["fk_violations"] == []
|
||||||
|
assert st["mounts"] == [("NAS", 1), ("paperless", 2)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_failed_migration_leaves_database_untouched(
|
||||||
|
db_0001: tuple[str, Path], monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""Bricht eine Migration mittendrin ab, muss alles beim Alten bleiben – keine
|
||||||
|
halb umgebauten Tabellen, kein _alembic_tmp-Rest, Version unverändert."""
|
||||||
|
from alembic.ddl.impl import DefaultImpl
|
||||||
|
|
||||||
|
url, db = db_0001
|
||||||
|
before = _state(db)
|
||||||
|
real_drop = DefaultImpl.drop_table
|
||||||
|
|
||||||
|
def boom(self: DefaultImpl, *a: object, **kw: object) -> None:
|
||||||
|
raise RuntimeError("Absturz mitten in der Migration")
|
||||||
|
|
||||||
|
monkeypatch.setattr(DefaultImpl, "drop_table", boom)
|
||||||
|
with pytest.raises(RuntimeError, match="Absturz"):
|
||||||
|
upgrade(url)
|
||||||
|
assert _state(db) == before
|
||||||
|
|
||||||
|
monkeypatch.setattr(DefaultImpl, "drop_table", real_drop)
|
||||||
|
upgrade(url) # danach klappt es ohne Handarbeit
|
||||||
|
assert _state(db)["version"] == _head()
|
||||||
|
|
||||||
|
|
||||||
|
def test_app_engine_still_enforces_foreign_keys(db_url: str) -> None:
|
||||||
|
from sqlalchemy import text
|
||||||
|
|
||||||
|
from netmover.db import make_engine
|
||||||
|
|
||||||
|
engine = make_engine(db_url)
|
||||||
|
with engine.connect() as conn:
|
||||||
|
assert conn.execute(text("PRAGMA foreign_keys")).scalar() == 1
|
||||||
|
engine.dispose()
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_from_0002_keeps_everything_linked_to_rules(tmp_path: Path) -> None:
|
||||||
|
"""0003 baut ``rules`` neu auf – Läufe, Buchführung und Fehlerzähler verweisen darauf."""
|
||||||
|
db = tmp_path / "v02.db"
|
||||||
|
url = f"sqlite:///{db}"
|
||||||
|
upgrade(url, "0001")
|
||||||
|
_seed(db)
|
||||||
|
upgrade(url, "0002")
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
c.execute("PRAGMA foreign_keys=ON")
|
||||||
|
c.executescript(
|
||||||
|
"""
|
||||||
|
UPDATE credentials SET managed = 1 WHERE id = 2;
|
||||||
|
UPDATE mounts SET username = 'paperless' WHERE id = 2;
|
||||||
|
INSERT INTO copied_files (rule_id, src_relpath, size, mtime_ns, dst_path, copied_at)
|
||||||
|
VALUES (1, 'a.pdf', 1, 1, '/x/a.pdf', '2026-10-10 20:00:00');
|
||||||
|
INSERT INTO file_failures (rule_id, src_relpath, size, mtime_ns, attempts, last_attempt_at)
|
||||||
|
VALUES (1, 'b.pdf', 1, 1, 2, '2026-10-10 20:00:00');
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
c.commit()
|
||||||
|
c.close()
|
||||||
|
upgrade(url)
|
||||||
|
st = _state(db)
|
||||||
|
assert st["version"] == _head() and st["fk_violations"] == [] and st["tmp_tables"] == []
|
||||||
|
assert st["rules"] == [("Scans → Paperless", 1, 2)] and st["run_items"] == 1
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
assert c.execute("SELECT COUNT(*) FROM copied_files").fetchone()[0] == 1
|
||||||
|
assert c.execute("SELECT attempts FROM file_failures").fetchone()[0] == 2
|
||||||
|
assert c.execute("SELECT waiting_since FROM rules").fetchone()[0] is None
|
||||||
|
assert c.execute("SELECT username FROM mounts WHERE id = 2").fetchone()[0] == "paperless"
|
||||||
|
c.close()
|
||||||
|
|
||||||
|
|
||||||
|
# -- Pflicht für jede (auch künftige) Migration -------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _revisions() -> list[str]:
|
||||||
|
from alembic.script import ScriptDirectory
|
||||||
|
|
||||||
|
script = ScriptDirectory.from_config(alembic_config("sqlite://"))
|
||||||
|
return [r.revision for r in reversed(list(script.walk_revisions()))]
|
||||||
|
|
||||||
|
|
||||||
|
def _invariants(db: Path) -> tuple[object, ...]:
|
||||||
|
st = _state(db)
|
||||||
|
assert st["tmp_tables"] == [], "Reste einer Batch-Migration"
|
||||||
|
assert st["fk_violations"] == [], "Fremdschlüssel verletzt"
|
||||||
|
return st["mounts"], st["rules"], st["run_items"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("rev", _revisions()[1:])
|
||||||
|
def test_every_migration_up_and_down_on_populated_db(tmp_path: Path, rev: str) -> None:
|
||||||
|
"""Jede Migration einzeln: hoch und wieder runter auf einer Datenbank mit Daten –
|
||||||
|
ohne Fremdschlüsselfehler, ohne Reste, ohne Datenverlust. Neue Migrationen werden
|
||||||
|
hier automatisch erfasst."""
|
||||||
|
revs = _revisions()
|
||||||
|
prev = revs[revs.index(rev) - 1]
|
||||||
|
db = tmp_path / "pflicht.db"
|
||||||
|
url = f"sqlite:///{db}"
|
||||||
|
upgrade(url, revs[0])
|
||||||
|
_seed(db)
|
||||||
|
upgrade(url, prev)
|
||||||
|
before = _invariants(db)
|
||||||
|
upgrade(url, rev)
|
||||||
|
assert _state(db)["version"] == rev
|
||||||
|
assert _invariants(db) == before
|
||||||
|
command.downgrade(alembic_config(url), prev)
|
||||||
|
assert _state(db)["version"] == prev
|
||||||
|
assert _invariants(db) == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_migration_files_are_named_after_their_revision() -> None:
|
||||||
|
versions = Path(__file__).parent.parent / "src/netmover/migrations/versions"
|
||||||
|
names = sorted(p.name.split("_", 1)[0] for p in versions.glob("[0-9]*.py"))
|
||||||
|
assert names == _revisions()
|
||||||
|
|
||||||
|
|
||||||
|
def test_0005_turns_archive_into_first_target_and_back(tmp_path: Path) -> None:
|
||||||
|
"""Archiv (0004) samt Buchführung wird zum weiteren Ziel „Archiv“ – und zurück."""
|
||||||
|
db = tmp_path / "v09.db"
|
||||||
|
url = f"sqlite:///{db}"
|
||||||
|
upgrade(url, "0001")
|
||||||
|
_seed(db)
|
||||||
|
upgrade(url, "0004")
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
c.executescript(
|
||||||
|
"""
|
||||||
|
UPDATE rules SET archive_kind = 'mount', archive_mount_id = 1,
|
||||||
|
archive_path = 'archiv', archive_layout = 'date';
|
||||||
|
INSERT INTO archived_files (rule_id, src_relpath, size, mtime_ns, archive_path,
|
||||||
|
archived_at)
|
||||||
|
VALUES (1, 'a.pdf', 3, 7, '/mnt/netmover/nas/archiv/2026/10/a.pdf',
|
||||||
|
'2026-10-12 09:00:00');
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
c.commit()
|
||||||
|
c.close()
|
||||||
|
|
||||||
|
upgrade(url, "0005")
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
assert c.execute(
|
||||||
|
"SELECT rule_id, position, name, kind, mount_id, path, layout FROM rule_targets"
|
||||||
|
).fetchall() == [(1, 0, "Archiv", "mount", 1, "archiv", "date")]
|
||||||
|
assert c.execute("SELECT src_relpath, size, mtime_ns FROM target_files").fetchall() == [
|
||||||
|
("a.pdf", 3, 7)
|
||||||
|
]
|
||||||
|
cols = [r[1] for r in c.execute("PRAGMA table_info(rules)")]
|
||||||
|
assert not [col for col in cols if col.startswith("archive_")]
|
||||||
|
assert (
|
||||||
|
c.execute("SELECT name FROM sqlite_master WHERE name = 'archived_files'").fetchall() == []
|
||||||
|
)
|
||||||
|
# zweites Ziel – geht beim Downgrade verloren, das erste bleibt
|
||||||
|
c.execute(
|
||||||
|
"INSERT INTO rule_targets (rule_id, position, name, kind, mount_id, path, layout) "
|
||||||
|
"VALUES (1, 1, 'Buchhaltung', 'local', NULL, '/srv/buha', 'flat')"
|
||||||
|
)
|
||||||
|
c.commit()
|
||||||
|
c.close()
|
||||||
|
assert _state(db)["fk_violations"] == []
|
||||||
|
|
||||||
|
command.downgrade(alembic_config(url), "0004")
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
assert c.execute(
|
||||||
|
"SELECT archive_kind, archive_mount_id, archive_path, archive_layout FROM rules"
|
||||||
|
).fetchall() == [("mount", 1, "archiv", "date")]
|
||||||
|
assert c.execute("SELECT rule_id, src_relpath FROM archived_files").fetchall() == [(1, "a.pdf")]
|
||||||
|
c.close()
|
||||||
|
assert _state(db)["fk_violations"] == []
|
||||||
@@ -0,0 +1,262 @@
|
|||||||
|
"""Uptime Kuma: /healthz, Einzel-Endpunkte, Push; Hilfe-Seite und Menü."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import timedelta
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.health import Check, Health
|
||||||
|
from netmover.kuma import push, push_url
|
||||||
|
from netmover.models import Mount, MountState, MountType, Rule, utcnow
|
||||||
|
from netmover.settings_store import AppSettings, load_settings, save_settings
|
||||||
|
from tests.webutil import post
|
||||||
|
|
||||||
|
# -- push_url / push ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _health(ok: bool, *problems: str) -> Health:
|
||||||
|
h = Health()
|
||||||
|
h.add(Check("scheduler", "Scheduler", True, "läuft"))
|
||||||
|
for p in problems:
|
||||||
|
h.add(Check("mount", "x", False, p))
|
||||||
|
assert h.ok is ok
|
||||||
|
return h
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_url_replaces_kuma_defaults() -> None:
|
||||||
|
url = push_url(
|
||||||
|
"https://kuma.lan/api/push/AbC?status=up&msg=OK&ping=", _health(False, "NAS weg")
|
||||||
|
)
|
||||||
|
parsed = httpx.URL(url)
|
||||||
|
assert parsed.path == "/api/push/AbC"
|
||||||
|
assert parsed.params["status"] == "down" and parsed.params["msg"] == "NAS weg"
|
||||||
|
assert push_url("https://kuma.lan/api/push/AbC", _health(True)).endswith(
|
||||||
|
"status=up&msg=OK&ping="
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_message_is_truncated() -> None:
|
||||||
|
url = push_url("https://k/api/push/x", _health(False, "x" * 1000))
|
||||||
|
assert len(httpx.URL(url).params["msg"]) == 250
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_swallows_errors_and_reports_result() -> None:
|
||||||
|
calls: list[str] = []
|
||||||
|
|
||||||
|
def ok_get(url: str, **kw: Any) -> httpx.Response:
|
||||||
|
calls.append(url)
|
||||||
|
return httpx.Response(200, request=httpx.Request("GET", url))
|
||||||
|
|
||||||
|
def bad_get(url: str, **kw: Any) -> httpx.Response:
|
||||||
|
raise httpx.ConnectError("weg")
|
||||||
|
|
||||||
|
assert push("https://k/api/push/x", _health(True), getter=ok_get) is True
|
||||||
|
assert calls and "status=up" in calls[0]
|
||||||
|
assert push("https://k/api/push/x", _health(True), getter=bad_get) is False
|
||||||
|
|
||||||
|
|
||||||
|
# -- Endpunkte ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _add_mount(
|
||||||
|
state: Any, name: str, slug: str, mstate: MountState, error: str | None = None
|
||||||
|
) -> int:
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
m = Mount(
|
||||||
|
name=name,
|
||||||
|
slug=slug,
|
||||||
|
type=MountType.CIFS,
|
||||||
|
server="nas",
|
||||||
|
share=slug,
|
||||||
|
mountpoint=f"/mnt/netmover/{slug}",
|
||||||
|
state=mstate,
|
||||||
|
last_error=error,
|
||||||
|
)
|
||||||
|
s.add(m)
|
||||||
|
s.flush()
|
||||||
|
return m.id
|
||||||
|
|
||||||
|
|
||||||
|
def test_healthz_details_and_single_endpoints(client: TestClient, state: Any) -> None:
|
||||||
|
ok_id = _add_mount(state, "NAS", "nas", MountState.MOUNTED)
|
||||||
|
_add_mount(state, "Archiv", "archiv", MountState.ERROR, "Host is down")
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
r = Rule(name="Scans", src_mount_id=ok_id, dst_path="/tmp", consecutive_failures=3)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
rid = r.id
|
||||||
|
|
||||||
|
r = client.get("/healthz")
|
||||||
|
assert r.status_code == 503
|
||||||
|
body = r.json()
|
||||||
|
assert body["status"] == "error" and body["version"]
|
||||||
|
assert body["scheduler"]["running"] is True
|
||||||
|
kinds = {(c["kind"], c["name"], c["ok"]) for c in body["checks"]}
|
||||||
|
assert ("mount", "NAS", True) in kinds and ("mount", "Archiv", False) in kinds
|
||||||
|
assert ("rule", "Scans", False) in kinds # drei Fehlschläge in Folge
|
||||||
|
|
||||||
|
assert client.get("/healthz/mount/nas").status_code == 200
|
||||||
|
r = client.get("/healthz/mount/archiv")
|
||||||
|
assert r.status_code == 503 and "Host is down" in r.json()["message"]
|
||||||
|
r = client.get(f"/healthz/rule/{rid}")
|
||||||
|
assert r.status_code == 503 and "3 fehlgeschlagene Läufe" in r.json()["message"]
|
||||||
|
assert client.get("/healthz/mount/gibtsnicht").status_code == 404
|
||||||
|
assert client.get("/healthz/rule/4711").status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_healthz_needs_no_login(raw_client: TestClient) -> None:
|
||||||
|
assert raw_client.get("/healthz").status_code == 200
|
||||||
|
assert raw_client.get("/healthz/mount/x").status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_disabled_rule_and_degraded_mount_are_not_alarms(client: TestClient, state: Any) -> None:
|
||||||
|
mid = _add_mount(state, "NAS", "nas", MountState.DEGRADED)
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
s.add(
|
||||||
|
Rule(
|
||||||
|
name="aus",
|
||||||
|
src_mount_id=mid,
|
||||||
|
dst_path="/tmp",
|
||||||
|
enabled=False,
|
||||||
|
created_at=utcnow() - timedelta(days=3),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert client.get("/healthz").status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_kuma_push_job_uses_saved_url(state: Any, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
sent: list[str] = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"netmover.kuma.httpx.get",
|
||||||
|
lambda url, **kw: (
|
||||||
|
sent.append(url),
|
||||||
|
httpx.Response(200, request=httpx.Request("GET", url)),
|
||||||
|
)[1],
|
||||||
|
)
|
||||||
|
state.push_health() # ohne URL: nichts
|
||||||
|
assert sent == []
|
||||||
|
save_settings(
|
||||||
|
state.session_factory, AppSettings(kuma_push_url="https://kuma.lan/api/push/T0k3n")
|
||||||
|
)
|
||||||
|
state.push_health()
|
||||||
|
assert len(sent) == 1 and sent[0].startswith("https://kuma.lan/api/push/T0k3n?status=")
|
||||||
|
|
||||||
|
|
||||||
|
def test_settings_kuma_url_masked_and_monitor_urls_listed(client: TestClient, state: Any) -> None:
|
||||||
|
_add_mount(state, "NAS", "nas", MountState.MOUNTED)
|
||||||
|
post(
|
||||||
|
client,
|
||||||
|
"/settings",
|
||||||
|
{
|
||||||
|
"retention_days": "30",
|
||||||
|
"notify_mode": "off",
|
||||||
|
"notify_after": "3",
|
||||||
|
"webhook_url": "",
|
||||||
|
"webhook_content_type": "application/json",
|
||||||
|
"webhook_template": "$message",
|
||||||
|
"kuma_push_url": "https://kuma.lan/api/push/GeHeIm?status=up",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert load_settings(state.session_factory).kuma_push_url.endswith("GeHeIm?status=up")
|
||||||
|
page = client.get("/settings").text
|
||||||
|
assert "GeHeIm" not in page and "kuma.lan" in page
|
||||||
|
assert "/healthz/mount/nas" in page and "/healthz" in page
|
||||||
|
|
||||||
|
|
||||||
|
# -- Hilfe und Menü ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_help_page_renders_guide_with_anchors(client: TestClient) -> None:
|
||||||
|
html = client.get("/help").text
|
||||||
|
for anchor in (
|
||||||
|
"begriffe",
|
||||||
|
"anbindungen",
|
||||||
|
"schluessel",
|
||||||
|
"regeln",
|
||||||
|
"regel-sicherheit",
|
||||||
|
"probelauf",
|
||||||
|
"status",
|
||||||
|
"protokoll",
|
||||||
|
"uptime-kuma",
|
||||||
|
"fragen",
|
||||||
|
):
|
||||||
|
assert f'id="{anchor}"' in html, anchor
|
||||||
|
assert "<table>" in html and "{#" not in html
|
||||||
|
assert 'src="/help/img/' in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_help_images_and_path_safety(client: TestClient) -> None:
|
||||||
|
from netmover.web.app import _help_dir
|
||||||
|
|
||||||
|
if (_help_dir() / "img" / "status.png").is_file():
|
||||||
|
r = client.get("/help/img/status.png")
|
||||||
|
assert r.status_code == 200 and r.headers["content-type"] == "image/png"
|
||||||
|
for bad in ("../ANLEITUNG.md", "..%2F..%2Fpyproject.toml", "x.svg"):
|
||||||
|
assert client.get(f"/help/img/{bad}").status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_navigation_groups_health_and_help_links(client: TestClient, state: Any) -> None:
|
||||||
|
_add_mount(state, "Archiv", "archiv", MountState.ERROR, "weg")
|
||||||
|
html = client.get("/mounts").text
|
||||||
|
for group in ("Übersicht", "Verbindungen", "Automatisierung", "System"):
|
||||||
|
assert group in html
|
||||||
|
assert 'class="health bad"' in html and "1 Problem" in html
|
||||||
|
assert "1 gestört" in html
|
||||||
|
assert 'href="/help#anbindungen"' in html # Hilfe zu dieser Seite
|
||||||
|
assert 'aria-current="page"' in html and 'class="menu-toggle"' in html
|
||||||
|
form = client.get("/rules/new").text
|
||||||
|
assert 'href="/help#regel-sicherheit"' in form
|
||||||
|
|
||||||
|
|
||||||
|
def test_logged_out_pages_have_no_menu(raw_client: TestClient) -> None:
|
||||||
|
html = raw_client.get("/setup").text
|
||||||
|
assert 'id="sidebar"' not in html
|
||||||
|
|
||||||
|
|
||||||
|
# -- Kennzahlen (0.10.0) ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_metrics_in_healthz_and_rule_endpoint(client: TestClient, state: Any) -> None:
|
||||||
|
from netmover.models import FileFailure, ItemStatus, Run, RunItem, RunStatus
|
||||||
|
|
||||||
|
mid = _add_mount(state, "NAS", "nas", MountState.MOUNTED)
|
||||||
|
now = utcnow()
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
a = Rule(name="A", src_mount_id=mid, dst_path="/tmp", last_success_at=now)
|
||||||
|
b = Rule(name="B", src_mount_id=mid, dst_path="/tmp", last_success_at=now)
|
||||||
|
s.add_all([a, b])
|
||||||
|
s.flush()
|
||||||
|
|
||||||
|
def run(rule: Rule, ago: timedelta, **kw: Any) -> Run:
|
||||||
|
r = Run(rule_id=rule.id, started_at=now - ago, finished_at=now - ago, **kw)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
return r
|
||||||
|
|
||||||
|
run(a, timedelta(hours=1), status=RunStatus.OK, moved=3)
|
||||||
|
q = run(a, timedelta(hours=2), status=RunStatus.PARTIAL, moved=1, failed=2)
|
||||||
|
s.add(RunItem(run_id=q.id, src="x.pdf", status=ItemStatus.QUARANTINED))
|
||||||
|
run(a, timedelta(hours=3), status=RunStatus.OK, moved=50, dry_run=True) # Probelauf
|
||||||
|
run(a, timedelta(hours=30), status=RunStatus.OK, moved=7) # älter als 24 h
|
||||||
|
run(b, timedelta(hours=4), status=RunStatus.ERROR)
|
||||||
|
s.add(FileFailure(rule_id=b.id, src_relpath="y.pdf", size=1, mtime_ns=1, attempts=1))
|
||||||
|
aid, bid = a.id, b.id
|
||||||
|
|
||||||
|
m = client.get("/healthz").json()["metrics"]
|
||||||
|
assert (m["files_24h"], m["failed_24h"], m["quarantined_24h"]) == (4, 2, 1)
|
||||||
|
assert m["errors_24h"] == 1 and m["pending_failures"] == 1
|
||||||
|
assert m["last_moved_at"].startswith(str((now - timedelta(hours=1)).date()))
|
||||||
|
ra = client.get(f"/healthz/rule/{aid}").json()["metrics"]
|
||||||
|
assert ra["files_24h"] == 4 and ra["quarantined_24h"] == 1
|
||||||
|
rb = client.get(f"/healthz/rule/{bid}").json()["metrics"]
|
||||||
|
assert rb["files_24h"] == 0 and rb["errors_24h"] == 1 and rb["last_moved_at"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_ok_message_shows_files() -> None:
|
||||||
|
url = push_url("https://k/api/push/x", _health(True), {"files_24h": 12, "failed_24h": 1})
|
||||||
|
assert httpx.URL(url).params["msg"] == "OK · 24 h: 12 Dateien, 1 Fehler"
|
||||||
@@ -147,6 +147,8 @@ def test_manager_pins_host_key_and_checks_alive(
|
|||||||
with session_factory() as s:
|
with session_factory() as s:
|
||||||
mm = s.get(Mount, mid)
|
mm = s.get(Mount, mid)
|
||||||
assert mm is not None and mm.host_key == server.host_key_string
|
assert mm is not None and mm.host_key == server.host_key_string
|
||||||
|
# Praxisbefund: nach erfolgreichem Test „erreichbar“ statt „nicht geprüft“
|
||||||
|
assert mm.state is MountState.MOUNTED
|
||||||
|
|
||||||
assert manager.check_alive(mid) is None
|
assert manager.check_alive(mid) is None
|
||||||
with session_factory() as s:
|
with session_factory() as s:
|
||||||
|
|||||||
@@ -0,0 +1,398 @@
|
|||||||
|
"""0.5.0: SFTP als Quelle – dieselben Garantien wie bei eingehängten Freigaben."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import errno
|
||||||
|
import hashlib
|
||||||
|
import os
|
||||||
|
from collections.abc import Callable, Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.models import (
|
||||||
|
CopiedFile,
|
||||||
|
Credential,
|
||||||
|
CredentialKind,
|
||||||
|
ItemStatus,
|
||||||
|
Mount,
|
||||||
|
MountType,
|
||||||
|
PatternType,
|
||||||
|
Rule,
|
||||||
|
RuleAction,
|
||||||
|
RunStatus,
|
||||||
|
)
|
||||||
|
from netmover.mounting.manager import MountManager
|
||||||
|
from netmover.mounting.sftp import SftpConfig, connect
|
||||||
|
from netmover.mover.destinations import LocalDestination, SftpDestination
|
||||||
|
from netmover.mover.engine import Engine, RuleSnapshot, RunResult
|
||||||
|
from netmover.mover.ledger import Ledger
|
||||||
|
from netmover.mover.runner import RuleRunner
|
||||||
|
from netmover.mover.scan import Matcher
|
||||||
|
from netmover.mover.sources import SftpSource
|
||||||
|
from netmover.timeouts import CancelToken
|
||||||
|
from tests.conftest import MountFactory
|
||||||
|
from tests.sftp_stub import StubSFTPServer
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def server(tmp_path: Path) -> Iterator[StubSFTPServer]:
|
||||||
|
root = tmp_path / "sftp-root"
|
||||||
|
for d in ("eingang", "ziel", "consume"):
|
||||||
|
(root / d).mkdir(parents=True)
|
||||||
|
srv = StubSFTPServer(root).start()
|
||||||
|
yield srv
|
||||||
|
srv.stop()
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(server: StubSFTPServer) -> SftpConfig:
|
||||||
|
return SftpConfig(
|
||||||
|
"127.0.0.1",
|
||||||
|
server.port,
|
||||||
|
server.username,
|
||||||
|
"/",
|
||||||
|
private_key=server.client_private_key,
|
||||||
|
host_key=server.host_key_string,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def source(server: StubSFTPServer) -> Iterator[SftpSource]:
|
||||||
|
s = SftpSource(connect(_cfg(server)), "/", "eingang", label="nas2")
|
||||||
|
yield s
|
||||||
|
s.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def rule_id(make_mount: MountFactory, session_factory: sessionmaker[Session]) -> int:
|
||||||
|
mid = make_mount()
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
r = Rule(name="r", src_mount_id=mid, dst_path="/x")
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
return r.id
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def hooks() -> list[Callable[[], None]]:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def engine(session_factory: sessionmaker[Session], hooks: list[Callable[[], None]]) -> Engine:
|
||||||
|
def sleep(_: float) -> None:
|
||||||
|
for h in hooks:
|
||||||
|
h()
|
||||||
|
|
||||||
|
return Engine(Ledger(session_factory), sleep=sleep)
|
||||||
|
|
||||||
|
|
||||||
|
def snap(rule_id: int, **kw: object) -> RuleSnapshot:
|
||||||
|
return RuleSnapshot(id=rule_id, name="r", stability_seconds=2, **kw) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def write(path: Path, data: bytes = b"inhalt") -> Path:
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_bytes(data)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def run(
|
||||||
|
engine: Engine, rule: RuleSnapshot, src: SftpSource, dest: object, **kw: object
|
||||||
|
) -> RunResult:
|
||||||
|
return engine.execute(rule, src, dest, CancelToken(), **kw) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def sha(p: Path) -> str:
|
||||||
|
return hashlib.sha256(p.read_bytes()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
# -- Scan ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_scan_filters_like_local_sources(source: SftpSource, server: StubSFTPServer) -> None:
|
||||||
|
root = server.root / "eingang"
|
||||||
|
for name in ("a.pdf", "b.PDF", "notiz.txt", ".versteckt.pdf", "halb.pdf.part", "x.tmp"):
|
||||||
|
write(root / name)
|
||||||
|
write(root / "max" / "c.pdf")
|
||||||
|
write(root / "@eaDir" / "a.pdf" / "thumb.pdf")
|
||||||
|
write(root / "quarantaene" / "d.pdf")
|
||||||
|
m = Matcher("*.pdf", PatternType.GLOB)
|
||||||
|
assert [c.relpath for c in source.scan(m, recursive=False)] == ["a.pdf", "b.PDF"]
|
||||||
|
rec = [
|
||||||
|
c.relpath
|
||||||
|
for c in source.scan(m, recursive=True, exclude=frozenset({"eingang/quarantaene"}))
|
||||||
|
]
|
||||||
|
assert rec == ["a.pdf", "b.PDF", "max/c.pdf"]
|
||||||
|
c = source.scan(m, recursive=False)[0]
|
||||||
|
assert c.path is None and c.state.size == len(b"inhalt")
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_source_dir_aborts(
|
||||||
|
engine: Engine, rule_id: int, server: StubSFTPServer, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
src = SftpSource(connect(_cfg(server)), "/", "gibtsnicht")
|
||||||
|
try:
|
||||||
|
result = run(
|
||||||
|
engine, snap(rule_id), src, LocalDestination(tmp_path / "dst", create_root=True)
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
src.close()
|
||||||
|
assert result.status is RunStatus.ERROR and "existiert nicht" in (result.message or "")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Verschieben / Kopieren ----------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_move_sftp_to_local(
|
||||||
|
engine: Engine, rule_id: int, source: SftpSource, server: StubSFTPServer, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
f = write(server.root / "eingang" / "scan001.pdf", os.urandom(300_000))
|
||||||
|
digest = sha(f)
|
||||||
|
dst = tmp_path / "dst"
|
||||||
|
dst.mkdir()
|
||||||
|
write(dst / "scan001.pdf", b"alt")
|
||||||
|
result = run(engine, snap(rule_id), source, LocalDestination(dst))
|
||||||
|
assert result.status is RunStatus.OK, result.message
|
||||||
|
assert [(i.src, i.status) for i in result.items] == [("scan001.pdf", ItemStatus.MOVED)]
|
||||||
|
assert sha(dst / "scan001-1.pdf") == digest and (dst / "scan001.pdf").read_bytes() == b"alt"
|
||||||
|
assert not f.exists()
|
||||||
|
assert sorted(p.name for p in dst.iterdir()) == ["scan001-1.pdf", "scan001.pdf"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_move_sftp_to_sftp(
|
||||||
|
engine: Engine, rule_id: int, source: SftpSource, server: StubSFTPServer
|
||||||
|
) -> None:
|
||||||
|
f = write(server.root / "eingang" / "a.pdf", os.urandom(50_000))
|
||||||
|
digest = sha(f)
|
||||||
|
dest = SftpDestination(connect(_cfg(server)), "/ziel", label="nas3")
|
||||||
|
try:
|
||||||
|
result = run(engine, snap(rule_id), source, dest)
|
||||||
|
finally:
|
||||||
|
dest.close()
|
||||||
|
assert result.items[0].status is ItemStatus.MOVED and result.items[0].dst == "nas3:/ziel/a.pdf"
|
||||||
|
assert sha(server.root / "ziel" / "a.pdf") == digest and not f.exists()
|
||||||
|
assert list((server.root / "ziel").glob(".netmover-*")) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_mode_remembers_versions(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
source: SftpSource,
|
||||||
|
server: StubSFTPServer,
|
||||||
|
tmp_path: Path,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
f = write(server.root / "eingang" / "a.pdf", b"v1")
|
||||||
|
dst = tmp_path / "dst"
|
||||||
|
dst.mkdir()
|
||||||
|
rule = snap(rule_id, action=RuleAction.COPY)
|
||||||
|
for _ in range(3):
|
||||||
|
run(engine, rule, source, LocalDestination(dst))
|
||||||
|
assert sorted(p.name for p in dst.iterdir()) == ["a.pdf"] and f.exists()
|
||||||
|
f.write_bytes(b"version 2") # andere Größe → neue Version
|
||||||
|
run(engine, rule, source, LocalDestination(dst))
|
||||||
|
assert sorted(p.name for p in dst.iterdir()) == ["a-1.pdf", "a.pdf"]
|
||||||
|
with session_factory() as s:
|
||||||
|
assert len(s.scalars(select(CopiedFile)).all()) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_mtime_is_carried_over(
|
||||||
|
engine: Engine, rule_id: int, source: SftpSource, server: StubSFTPServer, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
f = write(server.root / "eingang" / "a.pdf")
|
||||||
|
os.utime(f, (1_600_000_000, 1_600_000_123))
|
||||||
|
dst = tmp_path / "dst"
|
||||||
|
dst.mkdir()
|
||||||
|
run(engine, snap(rule_id, action=RuleAction.COPY), source, LocalDestination(dst))
|
||||||
|
assert int((dst / "a.pdf").stat().st_mtime) == 1_600_000_123
|
||||||
|
|
||||||
|
|
||||||
|
# -- Fallstricke ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_growing_file_is_not_touched(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
source: SftpSource,
|
||||||
|
server: StubSFTPServer,
|
||||||
|
tmp_path: Path,
|
||||||
|
hooks: list[Callable[[], None]],
|
||||||
|
) -> None:
|
||||||
|
growing = write(server.root / "eingang" / "wachsend.pdf", b"x" * 10)
|
||||||
|
hooks.append(lambda: growing.write_bytes(b"x" * 20))
|
||||||
|
dst = tmp_path / "dst"
|
||||||
|
dst.mkdir()
|
||||||
|
result = run(engine, snap(rule_id), source, LocalDestination(dst))
|
||||||
|
assert result.items[0].status is ItemStatus.SKIPPED and growing.exists()
|
||||||
|
assert list(dst.iterdir()) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_changed_during_transfer_keeps_source(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
source: SftpSource,
|
||||||
|
server: StubSFTPServer,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
f = write(server.root / "eingang" / "a.pdf", b"original")
|
||||||
|
real = LocalDestination._apply_meta
|
||||||
|
|
||||||
|
def change(self: LocalDestination, rel: str, meta: object) -> list[str]:
|
||||||
|
f.write_bytes(b"geaendert!!")
|
||||||
|
return real(self, rel, meta) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_apply_meta", change)
|
||||||
|
dst = tmp_path / "dst"
|
||||||
|
dst.mkdir()
|
||||||
|
result = run(engine, snap(rule_id), source, LocalDestination(dst))
|
||||||
|
assert result.items[0].status is ItemStatus.SKIPPED and f.exists()
|
||||||
|
assert list(dst.iterdir()) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_not_deletable_prevents_duplicates(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
source: SftpSource,
|
||||||
|
server: StubSFTPServer,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
write(server.root / "eingang" / "a.pdf")
|
||||||
|
|
||||||
|
def deny(*a: object, **kw: object) -> None:
|
||||||
|
raise PermissionError(errno.EACCES, "Permission denied")
|
||||||
|
|
||||||
|
monkeypatch.setattr(source.client, "remove", deny)
|
||||||
|
dst = tmp_path / "dst"
|
||||||
|
dst.mkdir()
|
||||||
|
first = run(engine, snap(rule_id), source, LocalDestination(dst))
|
||||||
|
assert first.items[0].status is ItemStatus.FAILED and "nicht löschbar" in (
|
||||||
|
first.items[0].message or ""
|
||||||
|
)
|
||||||
|
second = run(engine, snap(rule_id), source, LocalDestination(dst))
|
||||||
|
assert second.items == [] and sorted(p.name for p in dst.iterdir()) == ["a.pdf"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_quarantine_on_sftp_source(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
source: SftpSource,
|
||||||
|
server: StubSFTPServer,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
from netmover.mover.destinations import TransferError
|
||||||
|
|
||||||
|
def broken(*a: object, **kw: object) -> None:
|
||||||
|
raise TransferError("kaputt")
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_link_noreplace", broken)
|
||||||
|
write(server.root / "eingang" / "defekt.pdf")
|
||||||
|
dst = tmp_path / "dst"
|
||||||
|
dst.mkdir()
|
||||||
|
rule = snap(rule_id, quarantine_after=2)
|
||||||
|
for _ in range(2):
|
||||||
|
run(engine, rule, source, LocalDestination(dst), quarantine_rel="eingang/_q")
|
||||||
|
r = run(engine, rule, source, LocalDestination(dst), quarantine_rel="eingang/_q")
|
||||||
|
assert r.items[0].status is ItemStatus.QUARANTINED
|
||||||
|
assert r.items[0].dst == "nas2:/eingang/_q/defekt.pdf"
|
||||||
|
assert (server.root / "eingang" / "_q" / "defekt.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_connection_loss_aborts_run(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
source: SftpSource,
|
||||||
|
server: StubSFTPServer,
|
||||||
|
tmp_path: Path,
|
||||||
|
hooks: list[Callable[[], None]],
|
||||||
|
) -> None:
|
||||||
|
write(server.root / "eingang" / "a.pdf")
|
||||||
|
write(server.root / "eingang" / "b.pdf")
|
||||||
|
hooks.append(lambda: source.session.transport.close()) # Verbindung bricht weg
|
||||||
|
dst = tmp_path / "dst"
|
||||||
|
dst.mkdir()
|
||||||
|
result = run(engine, snap(rule_id), source, LocalDestination(dst))
|
||||||
|
assert result.status is RunStatus.ERROR
|
||||||
|
assert (server.root / "eingang" / "a.pdf").exists() and list(dst.iterdir()) == []
|
||||||
|
|
||||||
|
|
||||||
|
# -- Regellauf mit SFTP-Quelle ------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_with_sftp_source(
|
||||||
|
server: StubSFTPServer,
|
||||||
|
manager: MountManager,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
secretbox: SecretBox,
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
write(server.root / "eingang" / "scan.pdf", b"%PDF")
|
||||||
|
consume = tmp_path / "consume"
|
||||||
|
consume.mkdir()
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
c = Credential(
|
||||||
|
name="k",
|
||||||
|
kind=CredentialKind.SSH_KEY,
|
||||||
|
username=server.username,
|
||||||
|
secret_enc=secretbox.encrypt(server.client_private_key),
|
||||||
|
)
|
||||||
|
s.add(c)
|
||||||
|
s.flush()
|
||||||
|
m = Mount(
|
||||||
|
name="Scanner-Server",
|
||||||
|
slug="scanner",
|
||||||
|
type=MountType.SFTP,
|
||||||
|
server="127.0.0.1",
|
||||||
|
share="/",
|
||||||
|
port=server.port,
|
||||||
|
cred_id=c.id,
|
||||||
|
host_key=server.host_key_string,
|
||||||
|
)
|
||||||
|
s.add(m)
|
||||||
|
s.flush()
|
||||||
|
r = Rule(
|
||||||
|
name="SFTP → lokal",
|
||||||
|
src_mount_id=m.id,
|
||||||
|
src_subpath="eingang",
|
||||||
|
dst_path=str(consume),
|
||||||
|
stability_seconds=2,
|
||||||
|
)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
rid = r.id
|
||||||
|
result = RuleRunner(session_factory, manager, sleep=lambda _: None).run(rid)
|
||||||
|
assert result.status is RunStatus.OK, result.message
|
||||||
|
assert result.items[0].dst == str(consume / "scan.pdf")
|
||||||
|
assert (consume / "scan.pdf").read_bytes() == b"%PDF"
|
||||||
|
assert not (server.root / "eingang" / "scan.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_rule_form_accepts_sftp_source_with_min_stability(
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
from netmover.admin import parse_rule_form
|
||||||
|
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
m = Mount(name="sftp", slug="sftp", type=MountType.SFTP, server="10.0.0.1", share="/in")
|
||||||
|
s.add(m)
|
||||||
|
s.flush()
|
||||||
|
form = {
|
||||||
|
"name": "x",
|
||||||
|
"src_mount_id": str(m.id),
|
||||||
|
"dst_kind": "local",
|
||||||
|
"dst_path": "/tmp",
|
||||||
|
"interval_seconds": "30",
|
||||||
|
"stability_seconds": "1",
|
||||||
|
"op_timeout_seconds": "120",
|
||||||
|
}
|
||||||
|
_, errors = parse_rule_form(s, form)
|
||||||
|
assert "src_mount_id" not in errors and "mindestens 2 s" in errors["stability_seconds"]
|
||||||
|
_, errors = parse_rule_form(s, {**form, "stability_seconds": "3"})
|
||||||
|
assert errors == {}
|
||||||
@@ -0,0 +1,683 @@
|
|||||||
|
"""Weitere Ziele (0.11.0, vorher Archiv aus 0.7.0): Kopie an alle Ziele, Quelle erst
|
||||||
|
danach löschen; nicht erreichbare Ziele werden nachbeliefert."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.models import (
|
||||||
|
CollisionPolicy,
|
||||||
|
ItemStatus,
|
||||||
|
Rule,
|
||||||
|
RuleAction,
|
||||||
|
RuleTarget,
|
||||||
|
RunStatus,
|
||||||
|
TargetFile,
|
||||||
|
TargetKind,
|
||||||
|
)
|
||||||
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
|
from netmover.mounting.manager import MountManager
|
||||||
|
from netmover.mover.destinations import LocalDestination, TransferError
|
||||||
|
from netmover.mover.engine import Engine, RuleSnapshot, RunResult, Target
|
||||||
|
from netmover.mover.ledger import Ledger
|
||||||
|
from netmover.mover.runner import RuleRunner
|
||||||
|
from netmover.timeouts import CancelToken
|
||||||
|
from tests.conftest import MountFactory
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def rule_id(make_mount: MountFactory, session_factory: sessionmaker[Session]) -> int:
|
||||||
|
mid = make_mount()
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
r = Rule(
|
||||||
|
name="r",
|
||||||
|
src_mount_id=mid,
|
||||||
|
dst_path="/x",
|
||||||
|
targets=[
|
||||||
|
RuleTarget(position=i, name=n, kind=TargetKind.LOCAL, path=f"/{n}")
|
||||||
|
for i, n in enumerate(("Archiv", "Buchhaltung", "Extra"))
|
||||||
|
],
|
||||||
|
)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
return r.id
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def tids(rule_id: int, session_factory: sessionmaker[Session]) -> dict[str, int]:
|
||||||
|
with session_factory() as s:
|
||||||
|
return {t.name: t.id for t in s.scalars(select(RuleTarget))}
|
||||||
|
|
||||||
|
|
||||||
|
FIXED = datetime(2026, 10, 12, 9, 30).timestamp()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def engine(session_factory: sessionmaker[Session]) -> Engine:
|
||||||
|
return Engine(Ledger(session_factory), sleep=lambda _: None, clock=lambda: FIXED)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def dirs(tmp_path: Path) -> tuple[Path, Path, Path]:
|
||||||
|
src, dst, arch = tmp_path / "src", tmp_path / "dst", tmp_path / "archiv"
|
||||||
|
for d in (src, dst, arch):
|
||||||
|
d.mkdir()
|
||||||
|
return src, dst, arch
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def cross_device(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""tmp_path liegt auf einem Gerät – realistischer Weg über Kopie erzwingen."""
|
||||||
|
monkeypatch.setattr(LocalDestination, "move_same_device", lambda *a, **k: None)
|
||||||
|
|
||||||
|
|
||||||
|
def snap(rule_id: int, **kw: object) -> RuleSnapshot:
|
||||||
|
return RuleSnapshot(id=rule_id, name="r", stability_seconds=0, **kw) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
LAYOUT: dict[str, str] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def run(
|
||||||
|
engine: Engine,
|
||||||
|
rule: RuleSnapshot,
|
||||||
|
src: Path,
|
||||||
|
dst: Path | None,
|
||||||
|
arch: Path | None,
|
||||||
|
*,
|
||||||
|
layout: str = "flat",
|
||||||
|
extra: list[Target] | None = None,
|
||||||
|
**kw: object,
|
||||||
|
) -> RunResult:
|
||||||
|
"""Hauptziel ``dst`` (None = nicht erreichbar) und Archiv ``arch`` (Ziel „Archiv“)."""
|
||||||
|
targets = list(extra or [])
|
||||||
|
if arch is not None:
|
||||||
|
targets.insert(0, Target(_TIDS["Archiv"], "Archiv", LocalDestination(arch), layout))
|
||||||
|
dest = LocalDestination(dst) if dst is not None else None
|
||||||
|
return engine.execute(rule, src, dest, CancelToken(), targets=targets, **kw) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
_TIDS: dict[str, int] = {}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _register_tids(request: pytest.FixtureRequest) -> None:
|
||||||
|
if "rule_id" in request.fixturenames:
|
||||||
|
_TIDS.clear()
|
||||||
|
_TIDS.update(request.getfixturevalue("tids"))
|
||||||
|
|
||||||
|
|
||||||
|
def write(p: Path, data: bytes = b"inhalt") -> Path:
|
||||||
|
p.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
p.write_bytes(data)
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def names(d: Path) -> list[str]:
|
||||||
|
return sorted(str(p.relative_to(d)) for p in d.rglob("*") if p.is_file())
|
||||||
|
|
||||||
|
|
||||||
|
def test_move_delivers_to_target_and_archive_then_deletes_source(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
data = os.urandom(200_000)
|
||||||
|
write(src / "scan001.pdf", data)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert result.status is RunStatus.OK
|
||||||
|
item = result.items[0]
|
||||||
|
assert item.status is ItemStatus.MOVED and item.message and "Archiv: " in item.message
|
||||||
|
assert (dst / "scan001.pdf").read_bytes() == data == (arch / "scan001.pdf").read_bytes()
|
||||||
|
assert not (src / "scan001.pdf").exists()
|
||||||
|
assert names(dst) == ["scan001.pdf"] and names(arch) == ["scan001.pdf"] # keine .part-Reste
|
||||||
|
|
||||||
|
|
||||||
|
def test_date_layout(engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "max" / "a.pdf")
|
||||||
|
run(engine, snap(rule_id, recursive=True), src, dst, arch, layout="date")
|
||||||
|
assert names(arch) == ["2026/10/max/a.pdf"] and names(dst) == ["max/a.pdf"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_archive_failure_delivers_nothing_and_keeps_source(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path], monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf")
|
||||||
|
real = LocalDestination._link_noreplace
|
||||||
|
|
||||||
|
def fail_in_archive(self: LocalDestination, tmp: str, final: str) -> None:
|
||||||
|
if self.root == arch:
|
||||||
|
raise TransferError("Archiv voll")
|
||||||
|
real(self, tmp, final)
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_link_noreplace", fail_in_archive)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
item = result.items[0]
|
||||||
|
assert item.status is ItemStatus.FAILED and item.message == "Archiv: Archiv voll (Versuch 1)"
|
||||||
|
assert (src / "a.pdf").exists() and names(dst) == [] and names(arch) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_target_failure_after_archive_never_archives_twice(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
dirs: tuple[Path, Path, Path],
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf")
|
||||||
|
real = LocalDestination._link_noreplace
|
||||||
|
|
||||||
|
def fail_in_target(self: LocalDestination, tmp: str, final: str) -> None:
|
||||||
|
if self.root == dst:
|
||||||
|
raise TransferError("Ziel gesperrt")
|
||||||
|
real(self, tmp, final)
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_link_noreplace", fail_in_target)
|
||||||
|
first = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert first.items[0].status is ItemStatus.FAILED
|
||||||
|
assert (src / "a.pdf").exists() and names(arch) == ["a.pdf"] and names(dst) == []
|
||||||
|
with session_factory() as s:
|
||||||
|
assert len(s.scalars(select(TargetFile)).all()) == 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_link_noreplace", real) # Ziel wieder frei
|
||||||
|
second = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert second.items[0].status is ItemStatus.MOVED
|
||||||
|
assert names(arch) == ["a.pdf"] # keine zweite Archivkopie (a-1.pdf)
|
||||||
|
assert names(dst) == ["a.pdf"] and not (src / "a.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_archive_never_overwrites(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(arch / "scan.pdf", b"alt im Archiv")
|
||||||
|
write(src / "scan.pdf", b"neu")
|
||||||
|
run(engine, snap(rule_id, on_collision=CollisionPolicy.OVERWRITE), src, dst, arch)
|
||||||
|
assert (arch / "scan.pdf").read_bytes() == b"alt im Archiv"
|
||||||
|
assert (arch / "scan-1.pdf").read_bytes() == b"neu"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_space_in_archive_writes_nothing(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path], monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf", b"x" * 1000)
|
||||||
|
real = LocalDestination.free_bytes
|
||||||
|
monkeypatch.setattr(
|
||||||
|
LocalDestination, "free_bytes", lambda self: 10 if self.root == arch else real(self)
|
||||||
|
)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert "Archiv: zu wenig Platz" in (result.items[0].message or "")
|
||||||
|
assert names(dst) == [] and names(arch) == [] and (src / "a.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_changed_during_archive_copy(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path], monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
f = write(src / "a.pdf", b"original")
|
||||||
|
real = LocalDestination._apply_meta
|
||||||
|
|
||||||
|
def change(self: LocalDestination, rel: str, meta: object) -> list[str]:
|
||||||
|
if self.root == arch:
|
||||||
|
f.write_bytes(b"geaendert!!")
|
||||||
|
return real(self, rel, meta) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
monkeypatch.setattr(LocalDestination, "_apply_meta", change)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch)
|
||||||
|
assert result.items[0].status is ItemStatus.SKIPPED
|
||||||
|
assert names(dst) == [] and names(arch) == [] and f.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_mode_with_archive(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
f = write(src / "a.pdf", b"v1")
|
||||||
|
rule = snap(rule_id, action=RuleAction.COPY)
|
||||||
|
for _ in range(3):
|
||||||
|
run(engine, rule, src, dst, arch)
|
||||||
|
assert f.exists() and names(dst) == ["a.pdf"] and names(arch) == ["a.pdf"]
|
||||||
|
f.write_bytes(b"version 2")
|
||||||
|
run(engine, rule, src, dst, arch)
|
||||||
|
assert names(dst) == ["a-1.pdf", "a.pdf"] and names(arch) == ["a-1.pdf", "a.pdf"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dry_run_shows_archive_and_writes_nothing(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf")
|
||||||
|
result = run(engine, snap(rule_id), src, dst, arch, layout="date", dry_run=True)
|
||||||
|
assert result.items[0].status is ItemStatus.PLANNED
|
||||||
|
assert f"Archiv: {arch / '2026/10/a.pdf'}" in (result.items[0].message or "")
|
||||||
|
assert names(dst) == [] and names(arch) == [] and (src / "a.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_without_archive_unchanged(
|
||||||
|
engine: Engine, rule_id: int, dirs: tuple[Path, Path, Path]
|
||||||
|
) -> None:
|
||||||
|
src, dst, arch = dirs
|
||||||
|
write(src / "a.pdf")
|
||||||
|
result = run(engine, snap(rule_id), src, dst, None)
|
||||||
|
assert result.items[0].message is None and names(arch) == []
|
||||||
|
|
||||||
|
|
||||||
|
# -- mehrere Ziele ------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _three(tmp_path: Path, rule_id: int, tids: dict[str, int], down: str = "") -> list[Target]:
|
||||||
|
"""Archiv, Buchhaltung, Extra als lokale Ordner; ``down`` = nicht erreichbar."""
|
||||||
|
out = []
|
||||||
|
for name in ("Archiv", "Buchhaltung", "Extra"):
|
||||||
|
d = tmp_path / name.lower()
|
||||||
|
d.mkdir(exist_ok=True)
|
||||||
|
dest = None if name == down else LocalDestination(d)
|
||||||
|
out.append(Target(tids[name], name, dest, reason=f"„{name}“ nicht erreichbar"))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def test_four_targets_all_get_the_file_then_source_is_deleted(
|
||||||
|
engine: Engine, rule_id: int, tids: dict[str, int], tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
src, dst = tmp_path / "src", tmp_path / "dst"
|
||||||
|
src.mkdir()
|
||||||
|
dst.mkdir()
|
||||||
|
data = os.urandom(50_000)
|
||||||
|
write(src / "rechnung.pdf", data)
|
||||||
|
targets = _three(tmp_path, rule_id, tids)
|
||||||
|
result = run(engine, snap(rule_id), src, dst, None, extra=targets)
|
||||||
|
assert result.status is RunStatus.OK and result.waiting_on == []
|
||||||
|
item = result.items[0]
|
||||||
|
assert item.status is ItemStatus.MOVED
|
||||||
|
for name in ("Archiv", "Buchhaltung", "Extra"):
|
||||||
|
assert f"{name}: " in (item.message or "")
|
||||||
|
assert (tmp_path / name.lower() / "rechnung.pdf").read_bytes() == data
|
||||||
|
assert (dst / "rechnung.pdf").read_bytes() == data
|
||||||
|
assert not (src / "rechnung.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_unreachable_target_others_are_served_and_caught_up_later(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
tids: dict[str, int],
|
||||||
|
tmp_path: Path,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
src, dst = tmp_path / "src", tmp_path / "dst"
|
||||||
|
src.mkdir()
|
||||||
|
dst.mkdir()
|
||||||
|
write(src / "r1.pdf", b"%PDF-1")
|
||||||
|
down = _three(tmp_path, rule_id, tids, down="Buchhaltung")
|
||||||
|
|
||||||
|
first = run(engine, snap(rule_id), src, dst, None, extra=down)
|
||||||
|
assert first.waiting_on == ["„Buchhaltung“ nicht erreichbar"]
|
||||||
|
item = first.items[0]
|
||||||
|
assert item.status is ItemStatus.SKIPPED and "wartet auf: Buchhaltung" in (item.message or "")
|
||||||
|
# Hauptziel und erreichbare Ziele haben die Datei, die Quelle bleibt
|
||||||
|
assert (dst / "r1.pdf").exists() and (tmp_path / "archiv" / "r1.pdf").exists()
|
||||||
|
assert (tmp_path / "extra" / "r1.pdf").exists() and (src / "r1.pdf").exists()
|
||||||
|
assert not (tmp_path / "buchhaltung" / "r1.pdf").exists()
|
||||||
|
|
||||||
|
# Weitere Läufe während des Ausfalls: nichts doppelt, kein Protokolleintrag
|
||||||
|
again = run(engine, snap(rule_id), src, dst, None, extra=down)
|
||||||
|
assert again.items == [] and names(dst) == ["r1.pdf"]
|
||||||
|
assert names(tmp_path / "archiv") == ["r1.pdf"]
|
||||||
|
|
||||||
|
# Buchhaltung wieder da → nachbeliefert, erst jetzt Quelle gelöscht
|
||||||
|
back = run(engine, snap(rule_id), src, dst, None, extra=_three(tmp_path, rule_id, tids))
|
||||||
|
item = back.items[0]
|
||||||
|
assert item.status is ItemStatus.MOVED, item.message
|
||||||
|
assert "Hauptziel hatte die Datei bereits" in (item.message or "")
|
||||||
|
assert names(tmp_path / "buchhaltung") == ["r1.pdf"]
|
||||||
|
assert names(dst) == ["r1.pdf"] and names(tmp_path / "archiv") == ["r1.pdf"]
|
||||||
|
assert not (src / "r1.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_unreachable_main_target_other_targets_still_served(
|
||||||
|
engine: Engine, rule_id: int, tids: dict[str, int], tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
src, dst = tmp_path / "src", tmp_path / "dst"
|
||||||
|
src.mkdir()
|
||||||
|
dst.mkdir()
|
||||||
|
write(src / "r.pdf")
|
||||||
|
targets = _three(tmp_path, rule_id, tids)
|
||||||
|
first = run(engine, snap(rule_id), src, None, None, extra=targets, dest_reason="DMS weg")
|
||||||
|
assert first.waiting_on == ["DMS weg"]
|
||||||
|
assert "wartet auf: Hauptziel" in (first.items[0].message or "")
|
||||||
|
assert names(tmp_path / "archiv") == ["r.pdf"] and (src / "r.pdf").exists()
|
||||||
|
second = run(engine, snap(rule_id), src, dst, None, extra=targets)
|
||||||
|
assert second.items[0].status is ItemStatus.MOVED
|
||||||
|
assert names(dst) == ["r.pdf"] and names(tmp_path / "extra") == ["r.pdf"]
|
||||||
|
assert not (src / "r.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_copy_mode_new_target_gets_backfilled(
|
||||||
|
engine: Engine, rule_id: int, tids: dict[str, int], tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
src, dst = tmp_path / "src", tmp_path / "dst"
|
||||||
|
src.mkdir()
|
||||||
|
dst.mkdir()
|
||||||
|
write(src / "a.pdf")
|
||||||
|
rule = snap(rule_id, action=RuleAction.COPY)
|
||||||
|
targets = _three(tmp_path, rule_id, tids)
|
||||||
|
run(engine, rule, src, dst, None, extra=targets[:1])
|
||||||
|
run(engine, rule, src, dst, None, extra=targets[:1])
|
||||||
|
assert names(tmp_path / "archiv") == ["a.pdf"] and names(dst) == ["a.pdf"]
|
||||||
|
# Ziel hinzugefügt: nur das neue Ziel bekommt die Datei
|
||||||
|
result = run(engine, rule, src, dst, None, extra=targets[:2])
|
||||||
|
assert result.items[0].status is ItemStatus.COPIED
|
||||||
|
assert names(tmp_path / "buchhaltung") == ["a.pdf"]
|
||||||
|
assert names(dst) == ["a.pdf"] and names(tmp_path / "archiv") == ["a.pdf"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dry_run_lists_every_target(
|
||||||
|
engine: Engine, rule_id: int, tids: dict[str, int], tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
src, dst = tmp_path / "src", tmp_path / "dst"
|
||||||
|
src.mkdir()
|
||||||
|
dst.mkdir()
|
||||||
|
write(src / "a.pdf")
|
||||||
|
targets = _three(tmp_path, rule_id, tids, down="Extra")
|
||||||
|
result = run(engine, snap(rule_id), src, dst, None, extra=targets, dry_run=True)
|
||||||
|
msg = result.items[0].message or ""
|
||||||
|
assert (
|
||||||
|
"würde verschieben" in msg and "Buchhaltung: " in msg and "Extra: nicht erreichbar" in msg
|
||||||
|
)
|
||||||
|
assert (src / "a.pdf").exists() and names(dst) == []
|
||||||
|
|
||||||
|
|
||||||
|
# -- Regellauf ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def setup(
|
||||||
|
manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
make_mount: MountFactory,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> Iterator[tuple[int, Path, Path, Path, Path]]:
|
||||||
|
scans = fake_backend.export("nas", "scans")
|
||||||
|
arch_remote = fake_backend.export("nas", "archiv")
|
||||||
|
buha_remote = fake_backend.export("nas", "buha")
|
||||||
|
src_id = make_mount("scans", share="scans")
|
||||||
|
arch_id = make_mount("archiv", share="archiv")
|
||||||
|
buha_id = make_mount("buha", share="buha")
|
||||||
|
for mid in (src_id, arch_id, buha_id):
|
||||||
|
manager.mount(mid)
|
||||||
|
consume = tmp_path / "consume"
|
||||||
|
consume.mkdir()
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
r = Rule(
|
||||||
|
name="Rechnungen → DMS + Buchhaltung + Archiv",
|
||||||
|
src_mount_id=src_id,
|
||||||
|
dst_path=str(consume),
|
||||||
|
stability_seconds=0,
|
||||||
|
targets=[
|
||||||
|
RuleTarget(position=0, name="Buchhaltung", kind=TargetKind.MOUNT, mount_id=buha_id),
|
||||||
|
RuleTarget(
|
||||||
|
position=1,
|
||||||
|
name="Archiv",
|
||||||
|
kind=TargetKind.MOUNT,
|
||||||
|
mount_id=arch_id,
|
||||||
|
path="rechnungen",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
rid = r.id
|
||||||
|
yield rid, scans, consume, arch_remote, buha_remote
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_with_three_targets(
|
||||||
|
setup: tuple[int, Path, Path, Path, Path],
|
||||||
|
manager: MountManager,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
rid, scans, consume, arch, buha = setup
|
||||||
|
write(scans / "re.pdf", b"%PDF")
|
||||||
|
result = RuleRunner(session_factory, manager, sleep=lambda _: None).run(rid)
|
||||||
|
assert result.status is RunStatus.OK, result.message
|
||||||
|
for path in (consume / "re.pdf", arch / "rechnungen" / "re.pdf", buha / "re.pdf"):
|
||||||
|
assert path.read_bytes() == b"%PDF"
|
||||||
|
assert not (scans / "re.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_unavailable_target_waits_without_blocking_others(
|
||||||
|
setup: tuple[int, Path, Path, Path, Path],
|
||||||
|
manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
from netmover.runs import RunService
|
||||||
|
|
||||||
|
rid, scans, consume, arch, buha = setup
|
||||||
|
write(scans / "re.pdf")
|
||||||
|
fake_backend.vanish(manager.mountpoint_for("buha"))
|
||||||
|
fake_backend.fail_next_mount(times=50)
|
||||||
|
runner = RuleRunner(session_factory, manager, sleep=lambda _: None)
|
||||||
|
service = RunService(session_factory, runner)
|
||||||
|
service.execute(rid)
|
||||||
|
assert (consume / "re.pdf").exists() and (arch / "rechnungen" / "re.pdf").exists()
|
||||||
|
assert (scans / "re.pdf").exists() and not (buha / "re.pdf").exists()
|
||||||
|
with session_factory() as s:
|
||||||
|
rule = s.get(Rule, rid)
|
||||||
|
assert rule is not None and rule.waiting_since is not None
|
||||||
|
assert "buha" in (rule.waiting_reason or "")
|
||||||
|
runs_before = len(rule.runs)
|
||||||
|
# weiterer Lauf während des Ausfalls: kein neuer Protokolleintrag
|
||||||
|
assert service.execute(rid) is None
|
||||||
|
with session_factory() as s:
|
||||||
|
rule = s.get(Rule, rid)
|
||||||
|
assert rule is not None and len(rule.runs) == runs_before
|
||||||
|
|
||||||
|
fake_backend.mount_failures.clear()
|
||||||
|
with session_factory() as s:
|
||||||
|
buha_id = s.scalar(select(RuleTarget.mount_id).where(RuleTarget.name == "Buchhaltung"))
|
||||||
|
assert buha_id is not None
|
||||||
|
manager.mount(buha_id) # Anbindung wieder da
|
||||||
|
service.execute(rid)
|
||||||
|
assert (buha / "re.pdf").exists() and not (scans / "re.pdf").exists()
|
||||||
|
assert len(list(consume.iterdir())) == 1 # nicht doppelt geliefert
|
||||||
|
with session_factory() as s:
|
||||||
|
rule = s.get(Rule, rid)
|
||||||
|
assert rule is not None and rule.waiting_since is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_targets_unavailable_makes_rule_wait(
|
||||||
|
setup: tuple[int, Path, Path, Path, Path],
|
||||||
|
manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
) -> None:
|
||||||
|
rid, scans, _consume, _arch, _buha = setup
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
r = s.get(Rule, rid)
|
||||||
|
assert r is not None
|
||||||
|
r.dst_mount_id = r.targets[0].mount_id # Hauptziel ebenfalls auf „buha“
|
||||||
|
r.dst_path = "dms"
|
||||||
|
r.targets = r.targets[:1]
|
||||||
|
write(scans / "doc.pdf")
|
||||||
|
fake_backend.vanish(manager.mountpoint_for("buha"))
|
||||||
|
fake_backend.fail_next_mount(times=50)
|
||||||
|
result = RuleRunner(session_factory, manager, sleep=lambda _: None).run(rid)
|
||||||
|
assert result.unavailable and result.status is RunStatus.ERROR
|
||||||
|
assert (scans / "doc.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_mount_used_as_target_cannot_be_deleted(
|
||||||
|
setup: tuple[int, Path, Path, Path, Path],
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
manager: MountManager,
|
||||||
|
) -> None:
|
||||||
|
from netmover.admin import AdminError, delete_mount
|
||||||
|
from netmover.models import Mount
|
||||||
|
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
arch_id = s.scalar(select(Mount.id).where(Mount.name == "archiv"))
|
||||||
|
assert arch_id is not None
|
||||||
|
with pytest.raises(AdminError, match="Wird noch von Regeln benutzt"):
|
||||||
|
delete_mount(s, manager, arch_id)
|
||||||
|
|
||||||
|
|
||||||
|
# -- Formular ---------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _base(mid: int) -> dict[str, str]:
|
||||||
|
return {
|
||||||
|
"name": "x",
|
||||||
|
"src_mount_id": str(mid),
|
||||||
|
"dst_kind": "local",
|
||||||
|
"dst_path": "/srv/consume",
|
||||||
|
"interval_seconds": "30",
|
||||||
|
"stability_seconds": "5",
|
||||||
|
"op_timeout_seconds": "120",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_rule_form_target_validation(
|
||||||
|
session_factory: sessionmaker[Session], make_mount: MountFactory
|
||||||
|
) -> None:
|
||||||
|
from netmover.admin import parse_rule_form
|
||||||
|
|
||||||
|
mid = make_mount()
|
||||||
|
base = _base(mid)
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
_, err = parse_rule_form(
|
||||||
|
s, {**base, "target_0_kind": "local", "target_0_path": "/srv/consume"}
|
||||||
|
)
|
||||||
|
assert "Hauptziel" in err["target_0_path"]
|
||||||
|
_, err = parse_rule_form(s, {**base, "target_0_kind": "mount", "target_0_path": "a"})
|
||||||
|
assert "target_0_mount_id" in err
|
||||||
|
_, err = parse_rule_form(s, {**base, "target_0_kind": "local", "target_0_path": "rel"})
|
||||||
|
assert "absoluten Pfad" in err["target_0_path"]
|
||||||
|
two = {
|
||||||
|
"target_0_kind": "local",
|
||||||
|
"target_0_path": "/srv/a",
|
||||||
|
"target_1_kind": "local",
|
||||||
|
"target_1_path": "/srv/a",
|
||||||
|
}
|
||||||
|
_, err = parse_rule_form(s, {**base, **two})
|
||||||
|
assert "Derselbe Ordner" in err["target_1_path"]
|
||||||
|
_, err = parse_rule_form(
|
||||||
|
s,
|
||||||
|
{
|
||||||
|
**base,
|
||||||
|
**two,
|
||||||
|
"target_1_path": "/srv/b",
|
||||||
|
"target_0_name": "X",
|
||||||
|
"target_1_name": "X",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert "schon vor" in err["target_1_name"]
|
||||||
|
_, err = parse_rule_form(
|
||||||
|
s, {**base, "target_0_kind": "mount", "target_0_mount_id": str(mid)}
|
||||||
|
)
|
||||||
|
assert "derselbe Ordner" in err["target_0_path"] # = Quelle
|
||||||
|
|
||||||
|
|
||||||
|
def test_rule_form_saves_compacts_and_keeps_ledger(
|
||||||
|
session_factory: sessionmaker[Session], make_mount: MountFactory
|
||||||
|
) -> None:
|
||||||
|
from netmover.admin import parse_rule_form, save_rule
|
||||||
|
|
||||||
|
mid = make_mount()
|
||||||
|
arch = make_mount("archiv", share="archiv")
|
||||||
|
form = {
|
||||||
|
**_base(mid),
|
||||||
|
# Platz 0 leer gelassen, Plätze 1 und 3 belegt → kompakt 0, 1
|
||||||
|
"target_1_kind": "mount",
|
||||||
|
"target_1_mount_id": str(arch),
|
||||||
|
"target_1_path": "rechnungen",
|
||||||
|
"target_1_layout": "date",
|
||||||
|
"target_3_kind": "local",
|
||||||
|
"target_3_name": "Buchhaltung",
|
||||||
|
"target_3_path": "/srv/buha",
|
||||||
|
}
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
d, err = parse_rule_form(s, form)
|
||||||
|
assert err == {}
|
||||||
|
r, _ = save_rule(s, d)
|
||||||
|
assert r is not None
|
||||||
|
rid = r.id
|
||||||
|
got = [(t.position, t.name, t.kind.value, t.path, t.layout.value) for t in r.targets]
|
||||||
|
assert got == [
|
||||||
|
(0, "archiv", "mount", "rechnungen", "date"), # Name = Anbindung
|
||||||
|
(1, "Buchhaltung", "local", "/srv/buha", "flat"),
|
||||||
|
]
|
||||||
|
first, second = r.targets[0].id, r.targets[1].id
|
||||||
|
for tid in (first, second):
|
||||||
|
s.add(TargetFile(target_id=tid, src_relpath="a", size=1, mtime_ns=1, dst_path="x"))
|
||||||
|
# erneut speichern: Archiv unverändert, Buchhaltung mit neuem Ordner
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
d, err = parse_rule_form(s, {**form, "target_3_path": "/srv/buha2"})
|
||||||
|
assert err == {}
|
||||||
|
save_rule(s, d, rid)
|
||||||
|
with session_factory() as s:
|
||||||
|
r = s.get(Rule, rid)
|
||||||
|
assert r is not None and [t.id for t in r.targets] == [first, second]
|
||||||
|
kept = {f.target_id for f in s.scalars(select(TargetFile))}
|
||||||
|
assert kept == {first} # Buchführung nur bei geändertem Ordner verworfen
|
||||||
|
# Ziel entfernen
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
d, _ = parse_rule_form(s, {**form, "target_3_kind": ""})
|
||||||
|
save_rule(s, d, rid)
|
||||||
|
with session_factory() as s:
|
||||||
|
r = s.get(Rule, rid)
|
||||||
|
assert r is not None and [t.name for t in r.targets] == ["archiv"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_rule_form_page_has_nine_target_slots(client: Any, state: Any) -> None:
|
||||||
|
html = client.get("/rules/new").text
|
||||||
|
assert "Weitere Ziele" in html and 'name="target_8_kind"' in html
|
||||||
|
assert 'name="target_9_kind"' not in html
|
||||||
|
assert 'data-show-for="target_0_kind:mount local|target_1_kind:mount local"' in html
|
||||||
|
|
||||||
|
|
||||||
|
# -- Export / Import --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_import_targets_and_legacy_archive(
|
||||||
|
setup: tuple[int, Path, Path, Path, Path],
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
manager: MountManager,
|
||||||
|
) -> None:
|
||||||
|
from netmover.settings_store import AppSettings
|
||||||
|
from netmover.transfer import export_config, import_config
|
||||||
|
|
||||||
|
with session_factory() as s:
|
||||||
|
data = export_config(s, AppSettings())
|
||||||
|
(rule,) = data["rules"]
|
||||||
|
assert [(t["name"], t["mount"], t["path"]) for t in rule["targets"]] == [
|
||||||
|
("Buchhaltung", "buha", ""),
|
||||||
|
("Archiv", "archiv", "rechnungen"),
|
||||||
|
]
|
||||||
|
new = {**rule, "name": "Kopie"}
|
||||||
|
legacy = {k: v for k, v in rule.items() if k != "targets"} | {
|
||||||
|
"name": "Alt",
|
||||||
|
"archive_kind": "mount",
|
||||||
|
"archive_mount": "archiv",
|
||||||
|
"archive_path": "alt",
|
||||||
|
"archive_layout": "date",
|
||||||
|
}
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
report = import_config(s, manager, {**data, "mounts": [], "rules": [new, legacy]})
|
||||||
|
assert report.errors == [] and len(report.created) == 2
|
||||||
|
with session_factory() as s:
|
||||||
|
by_name = {r.name: r for r in s.scalars(select(Rule))}
|
||||||
|
assert [t.name for t in by_name["Kopie"].targets] == ["Buchhaltung", "Archiv"]
|
||||||
|
(t,) = by_name["Alt"].targets
|
||||||
|
assert (t.name, t.path, t.layout.value) == ("Archiv", "alt", "date")
|
||||||
@@ -0,0 +1,184 @@
|
|||||||
|
"""0.4.0: automatisch pausieren/fortsetzen, Wartungsmodus, eine Meldung pro Anbindung."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.health import check_health
|
||||||
|
from netmover.models import Rule, Run, RunStatus, RunTrigger, utcnow
|
||||||
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
|
from netmover.mounting.manager import MountManager
|
||||||
|
from netmover.mover.engine import RunResult
|
||||||
|
from netmover.mover.runner import RuleRunner
|
||||||
|
from netmover.runs import RunService
|
||||||
|
from netmover.settings_store import AppSettings, save_settings
|
||||||
|
from tests.conftest import MountFactory
|
||||||
|
|
||||||
|
|
||||||
|
class FakeRunner:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.results: list[RunResult] = []
|
||||||
|
self.calls = 0
|
||||||
|
|
||||||
|
def run(self, rule_id: int, *, dry_run: bool | None = None) -> RunResult:
|
||||||
|
self.calls += 1
|
||||||
|
return self.results.pop(0)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def rule_id(make_mount: MountFactory, session_factory: sessionmaker[Session]) -> int:
|
||||||
|
mid = make_mount()
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
r = Rule(name="Scans", src_mount_id=mid, dst_path="/x", interval_seconds=30)
|
||||||
|
s.add(r)
|
||||||
|
s.flush()
|
||||||
|
return r.id
|
||||||
|
|
||||||
|
|
||||||
|
def _rule(sf: sessionmaker[Session], rid: int) -> Rule:
|
||||||
|
with sf() as s:
|
||||||
|
r = s.get(Rule, rid)
|
||||||
|
assert r is not None
|
||||||
|
s.expunge(r)
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
def _runs(sf: sessionmaker[Session]) -> list[Run]:
|
||||||
|
with sf() as s:
|
||||||
|
return list(s.scalars(select(Run).order_by(Run.id)))
|
||||||
|
|
||||||
|
|
||||||
|
def unavailable() -> RunResult:
|
||||||
|
return RunResult(
|
||||||
|
RunStatus.ERROR,
|
||||||
|
message="„NAS“ ist nicht eingehängt – Lauf abgebrochen, nichts angefasst.",
|
||||||
|
unavailable=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_waiting_logs_once_and_resumes_by_itself(
|
||||||
|
session_factory: sessionmaker[Session], rule_id: int
|
||||||
|
) -> None:
|
||||||
|
runner = FakeRunner()
|
||||||
|
service = RunService(session_factory, runner) # type: ignore[arg-type]
|
||||||
|
runner.results += [unavailable(), unavailable(), unavailable(), RunResult(RunStatus.OK)]
|
||||||
|
|
||||||
|
first = service.execute(rule_id)
|
||||||
|
assert first is not None
|
||||||
|
r = _rule(session_factory, rule_id)
|
||||||
|
assert r.waiting_since is not None and "nicht eingehängt" in (r.waiting_reason or "")
|
||||||
|
assert r.consecutive_failures == 0 # Warten ist kein Fehlschlag
|
||||||
|
|
||||||
|
assert service.execute(rule_id) is None # kein neuer Eintrag alle 30 s
|
||||||
|
assert service.execute(rule_id) is None
|
||||||
|
assert len(_runs(session_factory)) == 1
|
||||||
|
assert _runs(session_factory)[0].message.startswith("Pausiert, bis die Anbindung zurück ist")
|
||||||
|
|
||||||
|
service.execute(rule_id) # Anbindung wieder da
|
||||||
|
r = _rule(session_factory, rule_id)
|
||||||
|
assert r.waiting_since is None and r.waiting_reason is None and r.last_success_at is not None
|
||||||
|
runs = _runs(session_factory)
|
||||||
|
assert len(runs) == 2 and runs[1].message and runs[1].message.startswith("Fortgesetzt nach")
|
||||||
|
assert r.consecutive_failures == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_real_errors_still_count(session_factory: sessionmaker[Session], rule_id: int) -> None:
|
||||||
|
runner = FakeRunner()
|
||||||
|
service = RunService(session_factory, runner) # type: ignore[arg-type]
|
||||||
|
runner.results += [RunResult(RunStatus.ERROR, message="kaputt")] * 2
|
||||||
|
service.execute(rule_id)
|
||||||
|
service.execute(rule_id)
|
||||||
|
r = _rule(session_factory, rule_id)
|
||||||
|
assert r.consecutive_failures == 2 and r.waiting_since is None
|
||||||
|
assert len(_runs(session_factory)) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_manual_dry_run_does_not_start_waiting(
|
||||||
|
session_factory: sessionmaker[Session], rule_id: int
|
||||||
|
) -> None:
|
||||||
|
runner = FakeRunner()
|
||||||
|
service = RunService(session_factory, runner) # type: ignore[arg-type]
|
||||||
|
runner.results.append(unavailable())
|
||||||
|
service.execute(rule_id, RunTrigger.MANUAL, dry_run=True)
|
||||||
|
assert _rule(session_factory, rule_id).waiting_since is None
|
||||||
|
assert len(_runs(session_factory)) == 1 # Probelauf sieht man trotzdem
|
||||||
|
|
||||||
|
|
||||||
|
def test_maintenance_skips_scheduled_runs_but_not_manual(
|
||||||
|
session_factory: sessionmaker[Session], rule_id: int
|
||||||
|
) -> None:
|
||||||
|
runner = FakeRunner()
|
||||||
|
service = RunService(session_factory, runner) # type: ignore[arg-type]
|
||||||
|
save_settings(session_factory, AppSettings(maintenance=True))
|
||||||
|
assert service.execute(rule_id) is None
|
||||||
|
assert runner.calls == 0 and _runs(session_factory) == []
|
||||||
|
runner.results.append(RunResult(RunStatus.OK))
|
||||||
|
assert service.execute(rule_id, RunTrigger.MANUAL) is not None
|
||||||
|
assert runner.calls == 1
|
||||||
|
|
||||||
|
|
||||||
|
def _health(sf: sessionmaker[Session], **kw: object): # type: ignore[no-untyped-def]
|
||||||
|
now = utcnow()
|
||||||
|
return check_health(
|
||||||
|
sf,
|
||||||
|
scheduler_running=True,
|
||||||
|
last_heartbeat=now,
|
||||||
|
service_started=now - timedelta(hours=1),
|
||||||
|
now=now,
|
||||||
|
**kw,
|
||||||
|
) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_health_waiting_and_maintenance(
|
||||||
|
session_factory: sessionmaker[Session], rule_id: int
|
||||||
|
) -> None:
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
r = s.get(Rule, rule_id)
|
||||||
|
assert r is not None
|
||||||
|
r.created_at = utcnow() - timedelta(days=1)
|
||||||
|
r.last_success_at = utcnow() - timedelta(hours=2)
|
||||||
|
r.waiting_since = utcnow() - timedelta(minutes=12)
|
||||||
|
r.waiting_reason = "„NAS“ nicht verfügbar"
|
||||||
|
h = _health(session_factory)
|
||||||
|
assert not h.ok and "wartet seit" in h.problems[0] and "NAS" in h.problems[0]
|
||||||
|
h = _health(session_factory, maintenance=True)
|
||||||
|
assert h.ok and h.maintenance and h.stale_rules == set()
|
||||||
|
check = h.find("rule", id=rule_id)
|
||||||
|
assert check is not None and "Wartungsmodus" in check.message
|
||||||
|
|
||||||
|
|
||||||
|
# -- Integration mit MountManager und Runner ------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_vanished_mount_notifies_once_with_affected_rules(
|
||||||
|
manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
make_mount: MountFactory,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
notifications: list[tuple[str, str, str]],
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
fake_backend.export("nas", "scans")
|
||||||
|
mid = make_mount("nas", share="scans")
|
||||||
|
manager.mount(mid)
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
s.add(Rule(name="Scans → Paperless", src_mount_id=mid, dst_path=str(tmp_path)))
|
||||||
|
s.add(Rule(name="Archiv", src_mount_id=mid, dst_path=str(tmp_path), enabled=False))
|
||||||
|
fake_backend.vanish(manager.mountpoint_for("nas"))
|
||||||
|
|
||||||
|
runner = RuleRunner(session_factory, manager, sleep=lambda _: None)
|
||||||
|
with session_factory() as s:
|
||||||
|
rid = s.scalars(select(Rule.id).where(Rule.name == "Scans → Paperless")).one()
|
||||||
|
for _ in range(3):
|
||||||
|
result = runner.run(rid)
|
||||||
|
assert result.unavailable and result.status is RunStatus.ERROR
|
||||||
|
errors = [n for n in notifications if n[0] == "mount.error"]
|
||||||
|
assert len(errors) == 1
|
||||||
|
assert "Betroffene Regeln" in errors[0][2] and "Scans → Paperless" in errors[0][2]
|
||||||
|
assert "Archiv" not in errors[0][2] # deaktivierte Regel nicht erwähnt
|
||||||
+128
-62
@@ -4,7 +4,6 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
import re
|
import re
|
||||||
from collections.abc import Iterator
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
@@ -17,66 +16,7 @@ from netmover.db import session_scope
|
|||||||
from netmover.models import Credential, Mount, MountState, Rule, Run, RunStatus
|
from netmover.models import Credential, Mount, MountState, Rule, Run, RunStatus
|
||||||
from netmover.mounting.fake import FakeMountBackend
|
from netmover.mounting.fake import FakeMountBackend
|
||||||
from netmover.web.app import create_app
|
from netmover.web.app import create_app
|
||||||
|
from tests.webutil import PASSWORD, csrf_of, do_setup, post
|
||||||
PASSWORD = "sehr-geheim-123"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def settings(tmp_path: Path) -> Settings:
|
|
||||||
return Settings(
|
|
||||||
data_dir=tmp_path / "lib",
|
|
||||||
run_dir=tmp_path / "run",
|
|
||||||
mount_base=tmp_path / "mnt",
|
|
||||||
mount_backend="fake",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def backend(tmp_path: Path) -> Iterator[FakeMountBackend]:
|
|
||||||
b = FakeMountBackend(tmp_path / "fake")
|
|
||||||
yield b
|
|
||||||
b.release_hangs()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def state(settings: Settings, backend: FakeMountBackend) -> NetMover:
|
|
||||||
return NetMover(settings, backend=backend)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def raw_client(settings: Settings, state: NetMover) -> Iterator[TestClient]:
|
|
||||||
with TestClient(create_app(settings, state), follow_redirects=False) as c:
|
|
||||||
yield c
|
|
||||||
|
|
||||||
|
|
||||||
def csrf_of(html: str) -> str:
|
|
||||||
m = re.search(r'name="csrf-token" content="([^"]+)"', html)
|
|
||||||
assert m, "kein CSRF-Token in der Seite"
|
|
||||||
return m.group(1)
|
|
||||||
|
|
||||||
|
|
||||||
def do_setup(c: TestClient) -> str:
|
|
||||||
page = c.get("/setup")
|
|
||||||
token = csrf_of(page.text)
|
|
||||||
code = c.app.state.setup_code.code # type: ignore[attr-defined]
|
|
||||||
r = c.post(
|
|
||||||
"/setup",
|
|
||||||
data={"csrf_token": token, "code": code, "password": PASSWORD, "confirm": PASSWORD},
|
|
||||||
)
|
|
||||||
assert r.status_code == 303, r.text
|
|
||||||
return csrf_of(c.get("/").text)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client(raw_client: TestClient) -> TestClient:
|
|
||||||
raw_client.headers["X-Test-CSRF"] = do_setup(raw_client)
|
|
||||||
return raw_client
|
|
||||||
|
|
||||||
|
|
||||||
def post(c: TestClient, url: str, data: dict[str, str] | None = None, **kw: object): # type: ignore[no-untyped-def]
|
|
||||||
payload = {"csrf_token": c.headers["X-Test-CSRF"], **(data or {})}
|
|
||||||
return c.post(url, data=payload, **kw) # type: ignore[arg-type]
|
|
||||||
|
|
||||||
|
|
||||||
# -- Einrichtung und Anmeldung --------------------------------------------------------------
|
# -- Einrichtung und Anmeldung --------------------------------------------------------------
|
||||||
|
|
||||||
@@ -154,7 +94,16 @@ def test_security_headers(client: TestClient) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def test_no_inline_scripts_or_styles(client: TestClient) -> None:
|
def test_no_inline_scripts_or_styles(client: TestClient) -> None:
|
||||||
for url in ("/", "/mounts", "/mounts/new", "/rules", "/rules/new", "/log", "/settings"):
|
for url in (
|
||||||
|
"/",
|
||||||
|
"/mounts",
|
||||||
|
"/mounts/new",
|
||||||
|
"/keys",
|
||||||
|
"/rules",
|
||||||
|
"/rules/new",
|
||||||
|
"/log",
|
||||||
|
"/settings",
|
||||||
|
):
|
||||||
html = client.get(url).text
|
html = client.get(url).text
|
||||||
assert "style=" not in html, url
|
assert "style=" not in html, url
|
||||||
assert not re.search(r"<script(?![^>]*\bsrc=)", html), url
|
assert not re.search(r"<script(?![^>]*\bsrc=)", html), url
|
||||||
@@ -555,3 +504,120 @@ def test_export_excludes_secrets_and_import_roundtrip(
|
|||||||
files={"file": ("export.json", text, "application/json")},
|
files={"file": ("export.json", text, "application/json")},
|
||||||
)
|
)
|
||||||
assert "Übersprungen" in c2.get("/settings").text
|
assert "Übersprungen" in c2.get("/settings").text
|
||||||
|
|
||||||
|
|
||||||
|
# -- Schlüsselverwaltung -------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_keys_page_generate_import_select_delete(client: TestClient, state: NetMover) -> None:
|
||||||
|
from netmover.crypto import generate_ssh_keypair
|
||||||
|
|
||||||
|
r = post(client, "/credentials/keys/generate", {"name": "paperless-lxc"})
|
||||||
|
assert r.status_code == 303
|
||||||
|
private, public = generate_ssh_keypair()
|
||||||
|
r = post(
|
||||||
|
client,
|
||||||
|
"/credentials/keys/import",
|
||||||
|
{"name": "alt", "private_key": private, "passphrase": ""},
|
||||||
|
)
|
||||||
|
assert r.status_code == 303
|
||||||
|
page = client.get("/credentials").text
|
||||||
|
assert "paperless-lxc" in page and "alt" in page
|
||||||
|
assert public.split()[1] in page # öffentlicher Teil sichtbar
|
||||||
|
for line in private.splitlines()[1:-1]: # keine Zeile des privaten Schlüssels
|
||||||
|
assert line not in page
|
||||||
|
assert "restrict,command="/usr/lib/openssh/sftp-server"" in page
|
||||||
|
|
||||||
|
with state.session_factory() as s:
|
||||||
|
key_id = s.scalar(select(Credential.id).where(Credential.name == "alt"))
|
||||||
|
form = client.get("/mounts/new").text
|
||||||
|
assert f'value="{key_id}"' in form and "neuer Schlüssel nur für diese Anbindung" in form
|
||||||
|
|
||||||
|
r = post(
|
||||||
|
client,
|
||||||
|
"/mounts/new",
|
||||||
|
{
|
||||||
|
"name": "paperless",
|
||||||
|
"type": "sftp",
|
||||||
|
"server": "10.0.0.5",
|
||||||
|
"share": "/consume",
|
||||||
|
"port": "22",
|
||||||
|
"auth": "ssh_key",
|
||||||
|
"key_id": str(key_id),
|
||||||
|
"username": "root",
|
||||||
|
"enabled": "on",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert r.status_code == 303, r.text
|
||||||
|
mounts = client.get("/mounts").text
|
||||||
|
assert "gespeicherten Zugangsdaten" in mounts and "„alt“" in mounts
|
||||||
|
edit = client.get(f"/mounts/{_mount_id(state, 'paperless')}/edit").text
|
||||||
|
assert f'<option value="{key_id}" selected>' in edit and 'value="root"' in edit
|
||||||
|
|
||||||
|
post(client, f"/credentials/{key_id}/delete")
|
||||||
|
assert "noch benutzt" in client.get("/credentials").text
|
||||||
|
post(client, f"/mounts/{_mount_id(state, 'paperless')}/delete")
|
||||||
|
post(client, f"/credentials/{key_id}/delete")
|
||||||
|
page = client.get("/credentials").text
|
||||||
|
assert "gelöscht" in page and ">alt<" not in page
|
||||||
|
|
||||||
|
|
||||||
|
def test_key_import_error_is_shown(client: TestClient) -> None:
|
||||||
|
post(
|
||||||
|
client,
|
||||||
|
"/credentials/keys/import",
|
||||||
|
{"name": "x", "private_key": "kein schlüssel", "passphrase": ""},
|
||||||
|
)
|
||||||
|
assert "Kein lesbarer privater Schlüssel" in client.get("/credentials").text
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_escaped_attributes_in_pages(client: TestClient, state: NetMover) -> None:
|
||||||
|
"""Attribute in {{ … }} werden von Jinja maskiert – dann landet " im HTML."""
|
||||||
|
|
||||||
|
post(client, "/credentials/keys/generate", {"name": "k"})
|
||||||
|
for url in (
|
||||||
|
"/",
|
||||||
|
"/mounts",
|
||||||
|
"/mounts/new",
|
||||||
|
"/keys",
|
||||||
|
"/rules",
|
||||||
|
"/rules/new",
|
||||||
|
"/log",
|
||||||
|
"/settings",
|
||||||
|
"/help",
|
||||||
|
):
|
||||||
|
html = client.get(url).text
|
||||||
|
broken = re.findall(r"<[a-z][^<>]*="[^<>]*>", html) # nur in Tags, nicht im Text
|
||||||
|
assert not broken, (url, broken[:2])
|
||||||
|
|
||||||
|
|
||||||
|
def test_credentials_page_passwords(
|
||||||
|
client: TestClient, state: NetMover, backend: FakeMountBackend
|
||||||
|
) -> None:
|
||||||
|
r = post(
|
||||||
|
client,
|
||||||
|
"/credentials/passwords",
|
||||||
|
{"name": "NAS-Scanner", "username": "scanner", "password": "Geheim,123", "domain": ""},
|
||||||
|
)
|
||||||
|
assert r.status_code == 303
|
||||||
|
page = client.get("/credentials").text
|
||||||
|
assert "NAS-Scanner" in page and "scanner" in page and "Geheim,123" not in page
|
||||||
|
with state.session_factory() as s:
|
||||||
|
cid = s.scalar(select(Credential.id).where(Credential.name == "NAS-Scanner"))
|
||||||
|
form = client.get("/mounts/new").text
|
||||||
|
assert f'<option value="{cid}"' in form and "eigene Angaben für diese Anbindung" in form
|
||||||
|
backend.export("nas", "scans")
|
||||||
|
r = post(
|
||||||
|
client, "/mounts/new", {**MOUNT_FORM, "username": "", "password": "", "cred_id": str(cid)}
|
||||||
|
)
|
||||||
|
assert r.status_code == 303, r.text
|
||||||
|
assert "gespeicherten Zugangsdaten" in client.get("/mounts").text
|
||||||
|
mid = _mount_id(state, "Synology Scans")
|
||||||
|
assert "Quelle mit Löschen" in post(client, f"/mounts/{mid}/test").text
|
||||||
|
post(
|
||||||
|
client,
|
||||||
|
f"/credentials/{cid}/password",
|
||||||
|
{"username": "scanner2", "domain": "", "password": ""},
|
||||||
|
)
|
||||||
|
assert "gilt ab sofort für alle" in client.get("/credentials").text
|
||||||
|
assert client.get("/keys").headers["location"] == "/credentials"
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
"""Hilfsfunktionen für Tests der Weboberfläche."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
PASSWORD = "sehr-geheim-123"
|
||||||
|
|
||||||
|
|
||||||
|
def csrf_of(html: str) -> str:
|
||||||
|
m = re.search(r'name="csrf-token" content="([^"]+)"', html)
|
||||||
|
assert m, "kein CSRF-Token in der Seite"
|
||||||
|
return m.group(1)
|
||||||
|
|
||||||
|
|
||||||
|
def do_setup(c: TestClient) -> str:
|
||||||
|
page = c.get("/setup")
|
||||||
|
token = csrf_of(page.text)
|
||||||
|
code = c.app.state.setup_code.code # type: ignore[attr-defined]
|
||||||
|
r = c.post(
|
||||||
|
"/setup",
|
||||||
|
data={"csrf_token": token, "code": code, "password": PASSWORD, "confirm": PASSWORD},
|
||||||
|
)
|
||||||
|
assert r.status_code == 303, r.text
|
||||||
|
return csrf_of(c.get("/").text)
|
||||||
|
|
||||||
|
|
||||||
|
def post(c: TestClient, url: str, data: dict[str, str] | None = None, **kw: Any) -> Any:
|
||||||
|
payload = {"csrf_token": c.headers["X-Test-CSRF"], **(data or {})}
|
||||||
|
return c.post(url, data=payload, **kw)
|
||||||
Reference in New Issue
Block a user