Compare commits
25
Commits
642def66bd
..
v0.2.1
@@ -0,0 +1,8 @@
|
||||
.venv
|
||||
.git
|
||||
data
|
||||
**/__pycache__
|
||||
.pytest_cache
|
||||
.mypy_cache
|
||||
.ruff_cache
|
||||
tests
|
||||
+112
@@ -8,6 +8,76 @@ Die Version steht an genau einer Stelle: `src/netmover/__init__.py`.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.2.1] – 2026-10-11
|
||||
|
||||
### Behoben
|
||||
- **Update von 0.1.x auf 0.2.0 scheiterte**: Migration 0002 brach auf einer
|
||||
Datenbank mit Anbindungen ab (`FOREIGN KEY constraint failed` beim Neuaufbau der
|
||||
Tabelle `credentials`). Weil SQLite-Migrationen nicht in einer Transaktion liefen,
|
||||
blieb dabei ein halber Zustand zurück. Migrationen laufen jetzt ohne
|
||||
Fremdschlüsselprüfung, aber mit Integritätsprüfung vor dem Commit und vollständig
|
||||
in einer Transaktion: Ein Abbruch hinterlässt die Datenbank unverändert.
|
||||
Geprüft mit Tests auf verknüpften Daten und an einer Kopie einer echten Datenbank.
|
||||
- `netmover-update` zeigt beim Zurückstufen keinen irreführenden Changelog-Auszug mehr.
|
||||
|
||||
## [0.2.0] – 2026-10-11
|
||||
|
||||
> ⚠️ **Nicht verwenden** – das Update von 0.1.x scheitert an Migration 0002.
|
||||
> Stattdessen 0.2.1 installieren. Wer 0.2.0 schon versucht hat: Die Datenbank aus der
|
||||
> Sicherung von `netmover-update` zurückspielen und auf 0.2.1 aktualisieren.
|
||||
|
||||
### Hinzugefügt
|
||||
- **Schlüsselverwaltung** (Seite „Schlüssel“): SSH-Schlüssel erzeugen (Ed25519) oder
|
||||
importieren (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH/PEM, auch mit Passphrase),
|
||||
Fingerabdruck und `authorized_keys`-Zeile mit SFTP-Beschränkung zum Kopieren,
|
||||
Umbenennen, Löschen nur wenn unbenutzt. Private Schlüssel werden nie angezeigt.
|
||||
- SFTP-Anbindungen können einen gespeicherten Schlüssel auswählen; der Benutzername
|
||||
gehört zur Anbindung (Migration 0002: `credentials.managed`, `mounts.username`).
|
||||
- **`netmover-update`**: Update aus dem Git-Repo mit Sicherung von Datenbank,
|
||||
Schlüsseln und Einstellungen, Changelog-Auszug, Neustart und Healthcheck;
|
||||
Versionswahl, Warnung beim Zurückstufen.
|
||||
- Installation als Git-Klon unter `/opt/netmover/src`; `install.sh` installiert
|
||||
`git` und das Update-Skript.
|
||||
|
||||
### Geändert
|
||||
- Klartext-Hinweis bei `mount error(95)` (keine gemeinsame SMB-Version) mit
|
||||
Synology-Einstellung; aufgefallen bei der Anbindung des echten NAS.
|
||||
- SFTP-Anbindungen gelten nach erfolgreichem Test als „erreichbar“ (vorher bis
|
||||
zum ersten Regellauf „nicht geprüft“).
|
||||
|
||||
## [0.1.1] – 2026-10-10
|
||||
|
||||
Korrekturen aus dem ersten Praxistest auf einer Debian-13-VM mit echten
|
||||
SMB-, NFS- und SFTP-Servern.
|
||||
|
||||
### Behoben
|
||||
- **systemd-Unit:** `ProtectSystem`, `PrivateTmp` und `ProtectControlGroups`
|
||||
gaben dem Dienst einen eigenen Mount-Namespace. Die Mounts waren für das System
|
||||
und andere Programme unsichtbar und verschwanden bei jedem Neustart des Dienstes
|
||||
(`MountFlags=shared` half nicht). Jetzt nur noch Härtungen ohne Namespace.
|
||||
- **Stabilitätsprüfung:** CIFS lieferte per `stat()` sekundenlang eine veraltete
|
||||
Größe einer noch wachsenden Datei. Gemessen wird jetzt per `open()` + `fstat()`,
|
||||
das erzwingt frische Attribute. Gesperrte Dateien gelten als „in Benutzung“.
|
||||
(Die zweite Sicherung, die Byte-Zählung beim Kopieren, hatte den Fall bereits
|
||||
abgefangen; es kam keine unvollständige Datei an.)
|
||||
- **Ausfall länger als die Wiederherstellung (z. B. NAS-Neustart, Stromausfall):**
|
||||
Von NetMover selbst ausgehängte Anbindungen mit „automatisch einhängen“ werden
|
||||
jetzt jede Minute erneut eingehängt, auch wenn das Einhängen schon beim
|
||||
Dienststart scheiterte. Entwarnung per Webhook. Von außen ausgehängte Anbindungen
|
||||
bleiben weiterhin unangetastet.
|
||||
- Bei einer Zeitüberschreitung von `mount` wird die ganze Prozessgruppe beendet, damit
|
||||
`mount.nfs` nicht später doch noch einhängt.
|
||||
- Hinweistexte zu `mount`-Fehlern werden nicht mehr mitgeschwärzt.
|
||||
- `install.sh` ohne `debconf`-Warnungen, wenn kein Terminal angeschlossen ist.
|
||||
|
||||
### Dokumentation
|
||||
- Fehlersuche: verzögert sichtbare Dateien auf CIFS → `nohandlecache`.
|
||||
- README erklärt, warum die Unit keine Namespace-Härtung nutzt.
|
||||
|
||||
## [0.1.0] – 2026-10-10
|
||||
|
||||
Erste Version.
|
||||
|
||||
### Hinzugefügt
|
||||
- Projektgerüst (Python ≥ 3.12, FastAPI-Stack, src-Layout, ruff, mypy strict, pytest).
|
||||
- Datenmodell mit SQLAlchemy 2.x: `credentials`, `mounts`, `rules`, `runs`,
|
||||
@@ -54,7 +124,49 @@ Die Version steht an genau einer Stelle: `src/netmover/__init__.py`.
|
||||
Inaktivitäts-Zeitlimit; ein hängender Worker blockiert nur seine eigene Regel.
|
||||
- Lokale Ziele unter dem Mount-Basisverzeichnis werden abgelehnt (dort muss die
|
||||
Anbindung als Ziel gewählt werden, damit „eingehängt?“ geprüft wird).
|
||||
- Scheduler (APScheduler): ein Job pro aktiver Regel mit zufälligem
|
||||
Startversatz und Jitter; Läufe in eigenem Thread-Pool; Abgleich nach Änderungen.
|
||||
- Protokoll: pro Lauf eine Zeile (auch übersprungene Termine), Einzeldateien
|
||||
in `run_items` (max. 2000 pro Lauf); Regelzähler letzter Lauf, letzter Erfolg,
|
||||
Fehler in Folge.
|
||||
- Aufräumen der Protokolle nach konfigurierbarer Frist (Standard 30 Tage).
|
||||
- Gesundheitsprüfung: Scheduler-Herzschlag, Anbindungen im Fehlerzustand,
|
||||
Regeln ohne Erfolg seit mehr als dem Dreifachen ihres Intervalls.
|
||||
- Webhook-Benachrichtigung mit frei wählbarer Vorlage (JSON-sicher), Modus
|
||||
„bei jedem Fehler“ oder „nach N Fehlern in Folge“, optional Entwarnung;
|
||||
auch für Anbindungen, deren Wiederherstellung scheitert.
|
||||
- Dienststart: Migration → Schlüssel → Anbindungen wiederherstellen → Scheduler;
|
||||
beim Beenden werden Mounts nicht ausgehängt.
|
||||
- Endname lokal per `renameat2(RENAME_NOREPLACE)`: echtes `rename` (Watcher sehen
|
||||
`IN_MOVED_TO`), trotzdem nie überschreibend; Rückfall auf `link` + `unlink`.
|
||||
- Deployment: systemd-Unit (`ProtectSystem=full`, `MountFlags=shared`),
|
||||
Installationsskript für Debian 13, `Dockerfile` und `compose.yml` mit
|
||||
`rshared`-Propagation.
|
||||
- README mit Installation, Synology→Paperless-Beispiel (SMB-Quelle, SFTP-Ziel),
|
||||
Docker, Reverse Proxy, Konfiguration, Fehlersuche und Sicherheitshinweisen.
|
||||
- Weboberfläche (FastAPI, Jinja2, HTMX ohne Build-Schritt, handyfähig, Hell/Dunkel):
|
||||
Status, Anbindungen, Regeln, Protokoll, Einstellungen.
|
||||
- Ersteinrichtung mit Pflicht-Passwort (≥ 10 Zeichen) und einmaligem
|
||||
Einrichtungscode aus dem Log; Anmeldung mit Argon2, Sperre nach 5 Fehlversuchen.
|
||||
- Session-Cookie `HttpOnly`, `SameSite=Lax` (optional `Secure`), CSRF-Token bei
|
||||
jedem POST, strikte Content-Security-Policy ohne Inline-Skripte/-Styles.
|
||||
- Mount-Formular mit „Testen“ vor dem Speichern; Passwörter werden nie
|
||||
angezeigt (leer = unverändert); SFTP-Schlüsselpaar wird erzeugt und der
|
||||
öffentliche Schlüssel zum Kopieren angezeigt.
|
||||
- Probelauf zeigt in einer Tabelle, welche Datei wohin ginge.
|
||||
- Protokoll filterbar nach Regel und Status, Einzeldateien aufklappbar.
|
||||
- Export/Import der Konfiguration als JSON (ohne Zugangsdaten und Webhook-URL).
|
||||
- `GET /healthz`, `netmover serve`, `netmover reset-password`.
|
||||
|
||||
### Behoben
|
||||
- Import-Upload wurde nie erkannt (falsche `UploadFile`-Klasse geprüft).
|
||||
- Hinweise nach Aktionen gingen verloren, wenn schon ein Hinweis in der
|
||||
Session stand (Session erkennt nur Neuzuweisungen als Änderung).
|
||||
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
||||
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
||||
|
||||
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.2.1...HEAD
|
||||
[0.2.1]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.2.0...v0.2.1
|
||||
[0.2.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.1...v0.2.0
|
||||
[0.1.1]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.0...v0.1.1
|
||||
[0.1.0]: https://git.wilde-server.de/florian.wilde/netmover/releases/tag/v0.1.0
|
||||
|
||||
@@ -10,10 +10,178 @@ Zeiten in Ortszeit (Europe/Berlin).
|
||||
| 2026-10-10 | 17:21 | 17:27 | 0:06 | 1 | Gerüst, Datenmodell, Migrationen |
|
||||
| 2026-10-10 | 18:35 | 18:48 | 0:13 | 2 | Mount-Schicht, Validierung, SFTP |
|
||||
| 2026-10-10 | 18:53 | 19:07 | 0:14 | 3 | Mover-Logik |
|
||||
| **Summe** | | | **0:33** | | |
|
||||
| 2026-10-10 | 19:10 | 19:17 | 0:07 | 4 | Scheduler, Protokoll, Benachrichtigung |
|
||||
| 2026-10-10 | 19:18 | 19:34 | 0:16 | 5 | Weboberfläche und Anmeldung |
|
||||
| 2026-10-10 | 19:40 | 19:50 | 0:10 | 6 | Deployment, README, Release 0.1.0 |
|
||||
| 2026-10-10 | 21:36 | 22:05 | 0:29 | – | Praxistest auf VM, Release 0.1.1 |
|
||||
| 2026-10-10 | 22:45 | 23:00 | 0:15 | – | Echtes NAS/Paperless anbinden |
|
||||
| 2026-10-10 | 23:31 | 23:58 | 0:27 | – | Schlüsselverwaltung, Update-Skript, 0.2.x |
|
||||
| **Summe** | | | **2:17** | | |
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-10 · 23:31–23:58 · Schlüsselverwaltung, Update-Skript, Releases 0.2.0/0.2.1
|
||||
|
||||
**Erledigt**
|
||||
- Schlüsselverwaltung (`keys.py`, Seite „Schlüssel“, Auswahl in SFTP-Anbindungen),
|
||||
Migration 0002 (`credentials.managed`, `mounts.username`).
|
||||
- `deploy/update.sh` → `netmover-update`: Tags holen, Sicherung (SQLite-Online-
|
||||
Backup + Schlüssel + env), Checkout, `install.sh`, Healthcheck, Rückweg-Anleitung.
|
||||
- VM auf Git-Klon `/opt/netmover/src` umgestellt (Repo ist öffentlich lesbar).
|
||||
|
||||
**Zwischenfall beim Update der VM auf 0.2.0**
|
||||
- Migration 0002 scheiterte: `FOREIGN KEY constraint failed` bei `DROP TABLE
|
||||
credentials` (Alembic-Batch baut die Tabelle neu; `foreign_keys=ON`, und die
|
||||
Anbindungen verweisen darauf). Danach crashte jeder Neustart an der
|
||||
übriggebliebenen `_alembic_tmp_credentials` (DDL lief ohne Transaktion).
|
||||
- `netmover-update` hat den Fehler erkannt und die Rückweg-Befehle genannt.
|
||||
Sicherung zurückgespielt, mit `netmover-update v0.1.1` zurückgestuft – in
|
||||
~1 Minute wieder lauffähig, NAS-Mount blieb durchgehend eingehängt.
|
||||
- Ursache, warum die Tests es nicht fanden: Migrationen wurden nur auf **leeren**
|
||||
Datenbanken getestet. Neu: `tests/test_migrations.py` mit verknüpften Daten und
|
||||
mit Abbruch mitten in der Migration (Datenbank muss unverändert bleiben).
|
||||
- Fix: eigene Migrationsverbindung (FK aus, `BEGIN IMMEDIATE`, `transactional_ddl`,
|
||||
`foreign_key_check` vor dem Commit). Vor dem Release an einer Kopie der echten
|
||||
Datenbank auf der VM geprüft.
|
||||
- Tag v0.2.0 bleibt (veröffentlicht), im Changelog als „nicht verwenden“ markiert.
|
||||
|
||||
**Lehre**
|
||||
- Jede künftige Migration bekommt einen Test auf einer befüllten Datenbank.
|
||||
|
||||
## 2026-10-10 · 21:36–22:05 · Praxistest auf der Debian-13-VM, Release 0.1.1
|
||||
|
||||
**Umgebung**
|
||||
- KVM-VM `srv-rz-netmover` (10.0.10.223), Debian 13, Python 3.13, 2 GB RAM.
|
||||
- Installation per `deploy/install.sh` aus einem `git archive` (VM ohne Gitea-Zugang).
|
||||
- Vorübergehende Testserver auf der VM: Samba (nur `lo`), NFS-Exporte für
|
||||
`127.0.0.1` (mit/ohne Root-Squash), SFTP-Benutzer `paperless` mit Schlüssel
|
||||
`restrict,command="/usr/lib/openssh/sftp-server"`.
|
||||
- Auf Wunsch `NETMOVER_HOST=0.0.0.0`, damit die Oberfläche im LAN erreichbar ist.
|
||||
|
||||
**Bestanden**
|
||||
- Ersteinrichtung mit Code aus dem Log, Cookie-Flags.
|
||||
- CIFS (Passwort mit Komma/Ausrufezeichen, Credentials-Datei danach weg), NFS 4,
|
||||
NFS 3, Root-Squash-Erkennung mit Synology-Hinweis, SFTP mit TOFU (Fingerabdruck
|
||||
= `ssh-keygen -lf`), echtes OpenSSH liefert `statvfs`.
|
||||
- Probelauf, Kollision `scan001-1.pdf`, `.txt`/`.part` ignoriert, keine `.part`-Reste,
|
||||
Eigentümer `paperless` im SFTP-Ziel.
|
||||
- Langsam wachsende Datei (20 × 50 KB/s): erst nach Ende verschoben, Prüfsumme gleich.
|
||||
- Kopiermodus rekursiv mit Eigentümer/Modus; zweiter Lauf „2 bereits übertragen“.
|
||||
- Samba gestoppt: Lebenszeichen 15 s → gestört → 3 Versuche → Fehler, `/healthz` 503,
|
||||
nichts angefasst. NFS-Server gestoppt (hängender `hard`-Mount): Oberfläche und
|
||||
andere Regel liefen weiter, Rückkehr automatisch.
|
||||
- Mounts überstehen Stopp/Start des Dienstes (gleiche Mount-IDs).
|
||||
- Neustart der VM: NFS sofort, SMB per Neuversuch nach ~1 min, Regel läuft weiter.
|
||||
|
||||
**Gefunden und behoben (v0.1.1)**
|
||||
1. **systemd-Unit mit eigenem Mount-Namespace** (ProtectSystem/PrivateTmp/…):
|
||||
Mounts nur im Dienst sichtbar, `shared,slave`. Härtung ohne Namespace.
|
||||
2. **CIFS-Attribut-Cache**: `stat()` meldete 3 s lang eine veraltete Größe einer
|
||||
wachsenden Datei; die Byte-Zählung beim Kopieren fing das ab. Jetzt
|
||||
`open()` + `fstat()` für beide Messungen.
|
||||
3. **Nach drei gescheiterten Versuchen endgültig aufgegeben** – zu kurz für einen
|
||||
NAS-Neustart, und beim Booten vor dem NAS kam die Freigabe nie zurück.
|
||||
Jetzt minütlicher Neuversuch (nur für von NetMover selbst ausgehängte
|
||||
Anbindungen mit `auto_mount`) plus Scheduler-Job.
|
||||
4. `mount`-Zeitlimit beendete nur `mount`, nicht `mount.nfs` → später doch
|
||||
eingehängt. Jetzt ganze Prozessgruppe.
|
||||
5. Hinweistext wurde mitgeschwärzt; `debconf`-Warnungen in `install.sh`.
|
||||
|
||||
**Nicht nachstellbar**
|
||||
- Einmalig 46 s Verzögerung direkt nach dem Booten (CIFS-Verzeichnis-Cache bei
|
||||
lokal statt per SMB angelegter Datei). Messungen danach 9–11 s, mit und ohne
|
||||
`nohandlecache`. Als Hinweis in die Fehlersuche aufgenommen.
|
||||
|
||||
## 2026-10-10 · 19:40–19:50 · Schritt 6: Deployment, README, Release 0.1.0
|
||||
|
||||
**Erledigt**
|
||||
- `deploy/netmover.service`, `deploy/netmover.env`, `deploy/install.sh`,
|
||||
`Dockerfile` (python:3.12-slim-trixie, mehrstufig), `compose.yml`, `.dockerignore`.
|
||||
- README vollständig; `LICENSE` (MIT, wie seit Schritt 1 in `pyproject.toml`).
|
||||
- Version 0.1.0, Changelog-Abschnitt, Tag `v0.1.0`.
|
||||
|
||||
**Geprüft**
|
||||
- Wheel enthält Templates, statische Dateien und Migrationen.
|
||||
- Docker-Image gebaut; Container startet, `/healthz` = 200, `mount.cifs` 7.4 und
|
||||
`mount.nfs` vorhanden. **Propagation getestet:** tmpfs im Container unter
|
||||
`/mnt/netmover` eingehängt → mit `rshared` auf dem Host sichtbar, ohne nicht.
|
||||
- `systemd-analyze verify` der Unit ohne Beanstandung. Ein echter Start der Unit
|
||||
als root (insb. `MountFlags=shared` mit `ProtectSystem`) war hier nicht möglich →
|
||||
beim ersten Einsatz mit `findmnt -R /mnt/netmover` prüfen.
|
||||
|
||||
**Entscheidungen**
|
||||
- Paperless-ngx-Consumer geprüft: nutzt `watchfiles` (`Change.added`) plus
|
||||
5-Minuten-Rescan – erkennt also auch per `link()` erzeugte Dateien. Für Watcher,
|
||||
die nur `IN_MOVED_TO`/`IN_CLOSE_WRITE` kennen, trotzdem auf
|
||||
`renameat2(RENAME_NOREPLACE)` umgestellt (entspricht der Vorgabe `os.replace`,
|
||||
ohne die Garantie „nie überschreiben“ aufzugeben).
|
||||
- Härtung der Unit bewusst zurückhaltend (keine `CapabilityBoundingSet`, kein
|
||||
`ProtectKernelModules`): ungetestete Einschränkungen könnten Mounten brechen
|
||||
(z. B. NFS mit privilegiertem Quellport, Modul-Autoload für cifs).
|
||||
- SFTP-Schlüssel im README per `restrict,command="…/sftp-server"` auf SFTP beschränkt.
|
||||
|
||||
**Offen für später**
|
||||
- Erster Praxistest auf der Ziel-VM mit echtem Synology-NAS und Paperless-LXC.
|
||||
|
||||
## 2026-10-10 · 19:18–19:34 · Schritt 5: Weboberfläche und Anmeldung
|
||||
|
||||
**Erledigt**
|
||||
- `admin.py` (Formulare prüfen/speichern, frameworkunabhängig), `transfer.py`
|
||||
(Export/Import), `web/auth.py`, `web/app.py`, 14 Templates, `app.css`, `app.js`.
|
||||
- htmx 2.0.4 lokal eingebunden (SHA-256
|
||||
`e209dda5c8235479f3166defc7750e1dbcd5a5c1808b7792fc2e6733768fb447`).
|
||||
- 25 Web-Tests (232 gesamt); Rauchtest mit echtem `netmover serve`.
|
||||
|
||||
**Entscheidungen**
|
||||
- Einmaliger Einrichtungscode im Log, damit nicht irgendwer im LAN beim ersten
|
||||
Aufruf das Admin-Passwort setzt.
|
||||
- Strikte CSP (`script-src 'self'; style-src 'self'`) – deshalb keine
|
||||
Inline-Styles; Füllstand als `<meter>`; htmx-Indikator-Styles abgeschaltet.
|
||||
- Jede Anbindung pflegt eigene Zugangsdaten (Datenmodell erlaubt Teilen, die
|
||||
Oberfläche bleibt dadurch einfacher).
|
||||
- Verbindungsdaten nur im ausgehängten Zustand änderbar; SFTP-Hostschlüssel wird
|
||||
bei Server-/Portwechsel verworfen.
|
||||
- „Jetzt ausführen“ startet im Hintergrund (Ergebnis im Protokoll); „Probelauf“
|
||||
wartet und zeigt die Tabelle.
|
||||
- Webhook-URL wird nur gekürzt angezeigt; unverändert zurückgeschickte Maske = behalten.
|
||||
- Import: vorhandene Namen überspringen, importierte Regeln deaktiviert,
|
||||
CIFS-Anbindungen ohne `auto_mount` (Zugangsdaten fehlen).
|
||||
- APScheduler-Logs auf WARNING (sonst alle 30 s eine Zeile pro Job).
|
||||
|
||||
**Stolpersteine**
|
||||
- Starlette-Session erkennt `setdefault(...).append()` nicht als Änderung →
|
||||
zweite Flash-Meldung ging verloren. Jetzt Neuzuweisung.
|
||||
- `request.form()` liefert `starlette.datastructures.UploadFile`, nicht die
|
||||
FastAPI-Unterklasse → Import erkannte nie eine Datei.
|
||||
- Settings-Import öffnete eine zweite DB-Verbindung in offener Schreib-
|
||||
transaktion (SQLite: „database is locked“) → Einstellungen nach dem Commit.
|
||||
|
||||
## 2026-10-10 · 19:10–19:17 · Schritt 4: Scheduler und Protokollierung
|
||||
|
||||
**Erledigt**
|
||||
- `settings_store.py` (Laufzeit-Einstellungen), `notify.py` (Webhook),
|
||||
`runs.py` (Lauf ausführen + protokollieren, Zähler, Aufräumen),
|
||||
`health.py`, `scheduler.py`, `app_state.py` (Verdrahtung, Start/Stop).
|
||||
- 24 neue Tests (207 gesamt). Mutationsprobe: 6/6 erkannt.
|
||||
|
||||
**Entscheidungen**
|
||||
- Erfolg = `ok` oder `partial` (Werkzeug arbeitet); Fehlschlag = `error` oder
|
||||
`partial` (für Benachrichtigungen). Übersprungen zählt nicht; manueller
|
||||
Probelauf ändert keine Zähler.
|
||||
- Bezug für „zu lange kein Erfolg“: spätester von letztem Erfolg, Anlage der
|
||||
Regel und Dienststart – sonst wäre nach einem Neustart sofort alles rot.
|
||||
- „Nach N Fehlern“ meldet genau einmal beim Erreichen der Schwelle, danach
|
||||
Entwarnung beim ersten Erfolg (abschaltbar).
|
||||
- `max_instances=3` im Scheduler, damit überlappende Termine beim eigenen
|
||||
Regel-Lock ankommen und protokolliert werden, statt dass APScheduler sie still verwirft.
|
||||
- Leere Läufe (alle 30 s) loggen nur auf DEBUG; in `runs` stehen sie trotzdem
|
||||
(die Oberfläche filtert standardmäßig auf Läufe mit Aktivität).
|
||||
- Webhook-URL wird nie geloggt (enthält oft ein Token).
|
||||
|
||||
**Stolpersteine**
|
||||
- Zwei Testfehler lagen in den Tests selbst (Fake-Symlink bei `resolve()`,
|
||||
Jitter in der Zeitschranke vergessen).
|
||||
|
||||
## 2026-10-10 · 18:53–19:07 · Schritt 3: Mover-Logik
|
||||
|
||||
**Erledigt**
|
||||
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
# NetMover im Container (Alternative zum systemd-Dienst, siehe README).
|
||||
# Braucht zur Laufzeit CAP_SYS_ADMIN und ein Mount-Verzeichnis mit
|
||||
# Propagation „rshared“ – sonst sieht der Host die Mounts nicht.
|
||||
|
||||
FROM python:3.12-slim-trixie AS build
|
||||
WORKDIR /src
|
||||
COPY pyproject.toml README.md ./
|
||||
COPY src ./src
|
||||
RUN pip wheel --no-cache-dir --wheel-dir /wheels .
|
||||
|
||||
FROM python:3.12-slim-trixie
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends cifs-utils nfs-common \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
COPY --from=build /wheels /wheels
|
||||
RUN pip install --no-cache-dir /wheels/*.whl && rm -rf /wheels
|
||||
|
||||
ENV NETMOVER_HOST=0.0.0.0 \
|
||||
NETMOVER_PORT=8080 \
|
||||
NETMOVER_DATA_DIR=/var/lib/netmover \
|
||||
NETMOVER_RUN_DIR=/run/netmover \
|
||||
NETMOVER_MOUNT_BASE=/mnt/netmover \
|
||||
PYTHONUNBUFFERED=1
|
||||
|
||||
VOLUME ["/var/lib/netmover"]
|
||||
EXPOSE 8080
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s \
|
||||
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/healthz', timeout=4).status == 200 else 1)"
|
||||
CMD ["netmover", "serve"]
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Florian Wilde
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -1,9 +1,400 @@
|
||||
# NetMover
|
||||
|
||||
Selbst gehosteter Dienst mit Web-Oberfläche, der SMB/CIFS- und NFS-Freigaben
|
||||
verwaltet und Dateien zeitgesteuert zwischen ihnen verschiebt.
|
||||
Selbst gehosteter Dienst mit Weboberfläche, der **SMB/CIFS- und NFS-Freigaben
|
||||
verwaltet** und **Dateien zeitgesteuert verschiebt** – „alle N Sekunden Ordner 1
|
||||
prüfen, passende Dateien nach Ordner 2 bringen“. Quelle und Ziel sind
|
||||
unabhängig voneinander per SMB, NFS, lokal oder (nur als Ziel) per SFTP angebunden.
|
||||
|
||||
> Status: in Entwicklung (Schritt 3 von 6). Vollständige Doku folgt mit Schritt 6.
|
||||
Typischer Einsatz: Ein Synology-NAS nimmt Scans entgegen, NetMover schiebt sie alle
|
||||
30 Sekunden in den Eingangsordner von Paperless-ngx.
|
||||
|
||||
- [Wie NetMover Datenverlust verhindert](#wie-netmover-datenverlust-verhindert)
|
||||
- [Installation](#installation) · [Erster Start](#erster-start)
|
||||
- [Beispiel: Synology → Paperless](#beispiel-synology--paperless)
|
||||
- [Docker](#alternative-docker) · [Reverse Proxy](#reverse-proxy-mit-tls)
|
||||
- [Konfiguration](#konfiguration) · [Fehlersuche](#fehlersuche) · [Sicherheit](#sicherheit)
|
||||
- [Entwicklung](#entwicklung)
|
||||
|
||||
## Wie NetMover Datenverlust verhindert
|
||||
|
||||
Quelle und Ziel liegen fast immer auf verschiedenen Dateisystemen. „Verschieben“
|
||||
heißt deshalb immer **kopieren, prüfen, dann erst die Quelle löschen**:
|
||||
|
||||
| Gefahr | Was NetMover tut |
|
||||
|---|---|
|
||||
| Halb geschriebene Scans | Datei wird nur angefasst, wenn Größe **und** Änderungszeit über das Stabilitätsfenster (Standard 5 s) gleich bleiben. `.part`, `.tmp`, `.crdownload`, `.filepart`, `.!ut` und versteckte Dateien werden ignoriert. |
|
||||
| Freigabe nicht eingehängt | Vor jedem Lauf prüft NetMover über `/proc/self/mountinfo`, ob Quelle und Ziel wirklich eingehängt sind – nie daran, ob Dateien da sind. Sonst: Lauf abbrechen, nichts anfassen. |
|
||||
| Wachsende Datei im Zielordner | Geschrieben wird in `.netmover-<uuid>.part`, dann `fsync`, Größenprüfung, erneute Prüfung der Quelle und atomares Umbenennen ohne Überschreiben. Erst danach wird die Quelle gelöscht. |
|
||||
| `scan001.pdf` existiert schon | `scan001-1.pdf`, `scan001-2.pdf`, … – jedes Mal neu gegen das Dateisystem geprüft. Überschreiben nur, wenn ausdrücklich eingestellt. |
|
||||
| Kopiermodus kopiert endlos | Übertragene Dateiversionen (Pfad, Größe, mtime) werden gemerkt. |
|
||||
| Hängender NFS-Mount, „Stale file handle“ | Jeder Lauf arbeitet in einem eigenen Worker mit Zeitlimit; gestörte Mounts werden per `umount -l` neu eingehängt (max. 3 Versuche). |
|
||||
| Ziel voll | Vor jeder Kopie: freier Platz ≥ Dateigröße + 5 %. |
|
||||
| Stiller Ausfall | Statusseite und `/healthz` werden rot, wenn eine Regel länger als das Dreifache ihres Intervalls keinen erfolgreichen Lauf hatte. |
|
||||
|
||||
## Installation
|
||||
|
||||
### Wo installieren?
|
||||
|
||||
| Variante | Empfehlung |
|
||||
|---|---|
|
||||
| **VM mit Debian 13** | ✅ Empfohlen. Einhängen funktioniert ohne Sonderkonfiguration. |
|
||||
| Proxmox-Host direkt | Funktioniert, aber zusätzliche Software auf dem Hypervisor ist meist unerwünscht. |
|
||||
| **Privilegierter** LXC | Möglich: `pct set <id> -features mount=cifs;nfs`. |
|
||||
| Unprivilegierter LXC | ❌ Der Kernel erlaubt dort kein CIFS/NFS-Mount. |
|
||||
| Docker | Möglich, siehe [Alternative: Docker](#alternative-docker). |
|
||||
|
||||
### systemd-Dienst (empfohlen)
|
||||
|
||||
```bash
|
||||
apt install -y git
|
||||
git clone https://git.wilde-server.de/florian.wilde/netmover.git /opt/netmover/src
|
||||
cd /opt/netmover/src && git checkout "$(git tag -l 'v*' --sort=-v:refname | head -n 1)"
|
||||
./deploy/install.sh
|
||||
```
|
||||
|
||||
Das Skript installiert `python3-venv`, `cifs-utils`, `nfs-common` und `git`. Es
|
||||
richtet NetMover in `/opt/netmover/venv` ein, installiert
|
||||
[`deploy/netmover.service`](deploy/netmover.service) und das Update-Skript
|
||||
`netmover-update` und startet den Dienst.
|
||||
|
||||
### Update
|
||||
|
||||
```bash
|
||||
netmover-update # auf die neueste Version
|
||||
netmover-update --list # verfügbare Versionen
|
||||
netmover-update v0.2.0 # bestimmte Version
|
||||
```
|
||||
|
||||
`netmover-update` geht so vor:
|
||||
1. Neue Versions-Tags aus Gitea holen und die Änderungen laut Changelog anzeigen.
|
||||
2. **Datenbank, `secret.key` und Einstellungen nach `/var/backups/netmover/` sichern**
|
||||
(die letzten 10 bleiben).
|
||||
3. Die Version auschecken, installieren und den Dienst neu starten. Neue
|
||||
Datenbank-Migrationen laufen dabei automatisch.
|
||||
4. `/healthz` prüfen.
|
||||
|
||||
Die Mounts bleiben die ganze Zeit eingehängt. Geht etwas schief, nennt das Skript
|
||||
die Befehle, um die Sicherung zurückzuspielen und zur vorherigen Version zu wechseln.
|
||||
**Zurückstufen** auf eine ältere Version geht nur mit der Datenbank-Sicherung von
|
||||
vor dem Update, wenn das Update eine Migration enthielt; das Skript fragt deshalb nach.
|
||||
|
||||
Was wo liegt:
|
||||
|
||||
| Pfad | Inhalt |
|
||||
|---|---|
|
||||
| `/var/lib/netmover/netmover.db` | Datenbank (SQLite) |
|
||||
| `/var/lib/netmover/secret.key` | Schlüssel für die Zugangsdaten (0600) – **mitsichern!** |
|
||||
| `/etc/netmover/netmover.env` | Einstellungen (Port, Logs, …) |
|
||||
| `/mnt/netmover/<name>` | Mountpoints – nur NetMover legt hier etwas an |
|
||||
| `/run/netmover` | kurzlebige Credentials-Dateien für `mount.cifs` |
|
||||
|
||||
**Warum als root?** Ein- und Aushängen braucht `CAP_SYS_ADMIN`. Außerdem soll
|
||||
NetMover Dateien auf beliebigen Freigaben lesen, löschen und mit den richtigen
|
||||
Eigentümern anlegen können.
|
||||
|
||||
**Warum so wenig systemd-Härtung?** Optionen wie `ProtectSystem=`, `PrivateTmp=`
|
||||
oder `ProtectHome=` geben dem Dienst einen eigenen Mount-Namespace. Die Mounts
|
||||
landeten dann *nur dort*: Das System und Paperless sähen sie nicht, und jeder
|
||||
Neustart des Dienstes würde sie verwerfen. `MountFlags=shared` hilft dagegen nicht.
|
||||
Auf der Test-VM nachgewiesen. Die Unit nutzt deshalb nur Härtungen ohne eigenen
|
||||
Namespace (`NoNewPrivileges`, `RestrictSUIDSGID`, `LockPersonality`,
|
||||
`RestrictRealtime`); Datenbank und Schlüssel sind nur für root lesbar. Prüfen:
|
||||
|
||||
```bash
|
||||
findmnt -l | grep /mnt/netmover # die eingehängten Freigaben müssen hier stehen
|
||||
```
|
||||
|
||||
**Beim Beenden bleiben die Mounts eingehängt**, weil andere Programme sie benutzen
|
||||
könnten. Beim Start werden alle Anbindungen mit „automatisch einhängen“
|
||||
wiederhergestellt. Erst danach startet der Scheduler.
|
||||
|
||||
## Erster Start
|
||||
|
||||
NetMover lauscht standardmäßig nur auf `127.0.0.1:8080`. Vom Arbeitsplatz aus
|
||||
erreichst du die Oberfläche per SSH-Tunnel:
|
||||
|
||||
```bash
|
||||
ssh -L 8080:127.0.0.1:8080 root@netmover-vm
|
||||
# dann im Browser: http://localhost:8080
|
||||
```
|
||||
|
||||
Beim ersten Aufruf musst du ein Administratorpasswort setzen (mindestens 10 Zeichen).
|
||||
Dafür brauchst du den **Einrichtungscode** aus dem Log. Damit kann nicht irgendwer
|
||||
im Netz, der die Seite zuerst öffnet, das Passwort setzen:
|
||||
|
||||
```bash
|
||||
journalctl -u netmover | grep setup.required
|
||||
# … event=setup.required code=4f2a-91c0-7d3e …
|
||||
```
|
||||
|
||||
Passwort vergessen? `sudo /opt/netmover/venv/bin/netmover reset-password`, dann
|
||||
`systemctl restart netmover` und die Einrichtung neu durchlaufen.
|
||||
|
||||
## Beispiel: Synology → Paperless
|
||||
|
||||
Ziel: Scans landen auf dem NAS in `scans`. NetMover schiebt PDFs alle 30 Sekunden
|
||||
in den Consume-Ordner von Paperless, das in einem LXC per SSH erreichbar ist.
|
||||
|
||||
### 1. Synology vorbereiten
|
||||
|
||||
1. **Systemsteuerung → Benutzer** → Benutzer `netmover` anlegen, Lese-/Schreibrecht
|
||||
auf den gemeinsamen Ordner `scans`.
|
||||
2. **Systemsteuerung → Dateidienste → SMB**: aktiviert. Unter *Erweitert* als
|
||||
minimales Protokoll SMB2 oder höher.
|
||||
|
||||
### 2. Anbindung „Synology Scans“ anlegen
|
||||
|
||||
*Anbindungen → Neue Anbindung*:
|
||||
|
||||
| Feld | Wert |
|
||||
|---|---|
|
||||
| Name | `Synology Scans` |
|
||||
| Typ | SMB/CIFS |
|
||||
| Server | `192.168.1.10` (IP des NAS) |
|
||||
| Freigabe | `scans` |
|
||||
| Benutzername / Passwort | `netmover` / … |
|
||||
| Zusatzoptionen | `vers=3.0` |
|
||||
| Beim Start automatisch einhängen | ✓ |
|
||||
|
||||
Klicke auf **Testen**. Alle drei Schritte (Einhängen, Lesen, Schreiben) sollten ✓
|
||||
zeigen, und die Rolle sollte „Quelle mit Löschen“ lauten. Danach speichern und
|
||||
**Einhängen**.
|
||||
|
||||
### 3. Paperless-LXC vorbereiten (SFTP-Ziel)
|
||||
|
||||
Lege in NetMover die *Anbindung → Neue Anbindung* an:
|
||||
|
||||
| Feld | Wert |
|
||||
|---|---|
|
||||
| Name | `Paperless` |
|
||||
| Typ | SFTP (nur Ziel) |
|
||||
| Server / SSH-Port | IP des LXC / `22` |
|
||||
| Freigabe / Pfad | `/opt/paperless_data/consume` |
|
||||
| Anmeldeart | SSH-Schlüssel |
|
||||
| Schlüssel | einen gespeicherten Schlüssel wählen (siehe unten) oder „neuer Schlüssel nur für diese Anbindung“ |
|
||||
| Benutzername | `paperless` (der Benutzer, unter dem Paperless läuft) |
|
||||
|
||||
**Schlüsselverwaltung:** Unter *Schlüssel* erzeugst du Ed25519-Schlüssel oder
|
||||
importierst vorhandene (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH- oder PEM-Format,
|
||||
auch mit Passphrase). Ein Schlüssel kann von mehreren Anbindungen genutzt werden; den
|
||||
Benutzernamen legt jede Anbindung selbst fest. Private Schlüssel liegen verschlüsselt
|
||||
in der Datenbank und werden nie angezeigt. Die Seite zeigt den Fingerabdruck und eine
|
||||
fertige `authorized_keys`-Zeile zum Kopieren.
|
||||
|
||||
Trage den öffentlichen Schlüssel im LXC so ein, dass er **nur SFTP** darf:
|
||||
|
||||
```bash
|
||||
# im Paperless-LXC als root
|
||||
install -d -m 700 -o paperless -g paperless ~paperless/.ssh
|
||||
echo 'restrict,command="/usr/lib/openssh/sftp-server" ssh-ed25519 AAAA… netmover@paperless' \
|
||||
>> ~paperless/.ssh/authorized_keys
|
||||
chown paperless:paperless ~paperless/.ssh/authorized_keys
|
||||
chmod 600 ~paperless/.ssh/authorized_keys
|
||||
```
|
||||
|
||||
Dann in NetMover bei der Anbindung **Testen**. Prüfe den angezeigten Fingerabdruck
|
||||
des Hostschlüssels. Ermitteln kannst du ihn im LXC mit
|
||||
`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`. NetMover hält ihn beim ersten
|
||||
Test fest und verweigert später jede Verbindung, wenn er sich ändert.
|
||||
|
||||
> Weil die Dateien als Benutzer `paperless` ankommen, darf Paperless sie nach dem
|
||||
> Einlesen auch löschen. Paperless-ngx (aktuelle Versionen mit `watchfiles`)
|
||||
> bemerkt neue Dateien sofort. Ältere Versionen, die nur auf
|
||||
> `IN_CLOSE_WRITE`/`IN_MOVED_TO` lauschen, siehe [Fehlersuche](#fehlersuche).
|
||||
|
||||
### 4. Regel anlegen
|
||||
|
||||
*Regeln → Neue Regel*:
|
||||
|
||||
| Feld | Wert |
|
||||
|---|---|
|
||||
| Name | `Scans → Paperless` |
|
||||
| Intervall | `30` |
|
||||
| Quelle | `Synology Scans`, Unterordner leer |
|
||||
| Filter | Glob `*.pdf` |
|
||||
| Ziel | Anbindung `Paperless`, Zielordner leer |
|
||||
| Aktion | verschieben |
|
||||
| Wenn Datei existiert | umbenennen |
|
||||
| Stabilitätsfenster | `5` |
|
||||
| Quarantäneordner | `_quarantaene` (optional) |
|
||||
|
||||
Klicke auf **Probelauf**. Die Tabelle zeigt, welche Datei wohin ginge, ohne etwas
|
||||
zu verändern. Danach aktivieren und auf der Statusseite verfolgen.
|
||||
|
||||
**Getrennt nach Unterordnern?** Scannt z. B. jede Person in einen eigenen
|
||||
Unterordner (`scans/max`, `scans/erika`), legst du pro Unterordner eine Regel mit
|
||||
eigenem Quell-Unterordner und eigenem Ziel an.
|
||||
|
||||
**Paperless lokal statt per SSH?** Läuft Paperless auf demselben Host, wähle als Ziel
|
||||
„Lokaler Pfad“ (z. B. `/opt/paperless_data/consume`). Der Ordner muss existieren und
|
||||
darf nicht unter `/mnt/netmover` liegen. Mit „Eigentümer setzen“ (z. B. `1000:1000`)
|
||||
gehören die Dateien danach dem Paperless-Benutzer.
|
||||
|
||||
### NFS statt SMB
|
||||
|
||||
Am NAS: *Dateidienste → NFS* aktivieren. Dann unter *Gemeinsamer Ordner →
|
||||
Bearbeiten → NFS-Berechtigungen* eine Regel mit der IP des NetMover-Hosts anlegen.
|
||||
In NetMover Typ „NFS 4“, Freigabe `/volume1/scans`.
|
||||
|
||||
**Root-Squash:** NetMover läuft als root. Synology bildet root standardmäßig auf
|
||||
`nobody` ab, dann scheitert das Schreiben oder `chown`. Der **Testen**-Knopf erkennt
|
||||
das und sagt es dir. Abhilfe in der NFS-Regel: *Squash → „Keine Zuordnung“* (No
|
||||
mapping) oder „Alle Benutzer zu admin zuordnen“.
|
||||
|
||||
## Alternative: Docker
|
||||
|
||||
```bash
|
||||
mkdir -p /mnt/netmover
|
||||
docker compose up -d --build
|
||||
docker compose logs | grep setup.required
|
||||
```
|
||||
|
||||
[`compose.yml`](compose.yml) gibt dem Container die nötigen Rechte:
|
||||
`cap_add: [SYS_ADMIN, DAC_READ_SEARCH]` und `security_opt: [apparmor:unconfined]`.
|
||||
|
||||
> **Wichtig:** `/mnt/netmover` muss als `bind` mit **`propagation: rshared`**
|
||||
> eingebunden sein. Ohne das bleiben die Mounts im Container gefangen: NetMover
|
||||
> selbst sieht sie, aber weder der Host noch andere Container (z. B. Paperless).
|
||||
> Getestet: Mit `rshared` erscheint ein im Container eingehängtes Dateisystem
|
||||
> sofort unter `/mnt/netmover` auf dem Host, ohne `rshared` nicht.
|
||||
|
||||
Lokale Ziele musst du zusätzlich als Volume einbinden. Für NFSv3 im Container die
|
||||
Option `nolock` setzen oder besser NFSv4 nehmen; NFSv3-Locking braucht
|
||||
`rpc.statd`, und den gibt es im Container nicht.
|
||||
|
||||
**Unterschied zum systemd-Dienst:** Der Dienst direkt auf dem Host braucht keine
|
||||
Propagation-Tricks, keine AppArmor-Ausnahme und keine zusätzlichen Volumes. Docker
|
||||
kapselt die Software, hat aber trotz Container `CAP_SYS_ADMIN` und ist damit kaum
|
||||
weniger mächtig als root auf dem Host.
|
||||
|
||||
## Reverse Proxy mit TLS
|
||||
|
||||
NetMover gehört **nicht ins Internet**. Es hat root-Rechte, hängt beliebige
|
||||
Freigaben ein und speichert Zugangsdaten zu deinem NAS. Ein Fehler oder ein
|
||||
schwaches Passwort wäre ein direkter Weg in dein Netz. Für den Zugriff im LAN gehört
|
||||
ein Reverse Proxy mit TLS davor; für unterwegs nimm VPN (z. B. WireGuard).
|
||||
|
||||
Beispiel Caddy auf demselben Host:
|
||||
|
||||
```caddyfile
|
||||
netmover.lan {
|
||||
tls internal
|
||||
reverse_proxy 127.0.0.1:8080
|
||||
}
|
||||
```
|
||||
|
||||
Beispiel nginx:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name netmover.lan;
|
||||
ssl_certificate /etc/ssl/netmover.crt;
|
||||
ssl_certificate_key /etc/ssl/netmover.key;
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_read_timeout 300s; # Mount-Tests können dauern
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Dann in `/etc/netmover/netmover.env`:
|
||||
|
||||
```bash
|
||||
NETMOVER_COOKIE_SECURE=true # Session-Cookie nur noch per HTTPS
|
||||
NETMOVER_FORWARDED_ALLOW_IPS=127.0.0.1
|
||||
```
|
||||
|
||||
## Konfiguration
|
||||
|
||||
Umgebungsvariablen (in `/etc/netmover/netmover.env`); alles Übrige stellst du in
|
||||
der Oberfläche unter *Einstellungen* ein:
|
||||
|
||||
| Variable | Standard | Bedeutung |
|
||||
|---|---|---|
|
||||
| `NETMOVER_HOST` | `127.0.0.1` | Adresse, auf der NetMover lauscht |
|
||||
| `NETMOVER_PORT` | `8080` | Port |
|
||||
| `NETMOVER_DATA_DIR` | `/var/lib/netmover` | Datenbank und Schlüssel |
|
||||
| `NETMOVER_MOUNT_BASE` | `/mnt/netmover` | Basisverzeichnis der Mountpoints |
|
||||
| `NETMOVER_RUN_DIR` | `/run/netmover` | kurzlebige Credentials-Dateien |
|
||||
| `NETMOVER_COOKIE_SECURE` | `false` | Cookie nur per HTTPS (hinter TLS-Proxy: `true`) |
|
||||
| `NETMOVER_FORWARDED_ALLOW_IPS` | `127.0.0.1` | Proxy-IPs, deren `X-Forwarded-*` gilt |
|
||||
| `NETMOVER_SESSION_HOURS` | `12` | Gültigkeit der Anmeldung |
|
||||
| `NETMOVER_LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR` |
|
||||
| `NETMOVER_LOG_FORMAT` | `logfmt` | `logfmt` oder `json` (ein Ereignis pro Zeile) |
|
||||
|
||||
In der Oberfläche unter *Einstellungen* stellst du ein:
|
||||
- Aufbewahrung der Protokolle (Standard 30 Tage).
|
||||
- Webhook-Benachrichtigung: „bei jedem Fehler“ oder „nach N Fehlern in Folge“,
|
||||
jeweils mit Entwarnung.
|
||||
- Export und Import der Konfiguration, **ohne** Zugangsdaten und Webhook-URL.
|
||||
|
||||
Webhook-Vorlagen (Platzhalter `$event $title $message $rule $time $host`):
|
||||
|
||||
| Dienst | URL | Content-Type | Vorlage |
|
||||
|---|---|---|---|
|
||||
| ntfy | `https://ntfy.sh/<topic>` | `text/plain` | `$title: $message` |
|
||||
| Gotify | `https://gotify.lan/message?token=…` | `application/json` | `{"title": "$title", "message": "$message", "priority": 5}` |
|
||||
| eigener Empfänger | beliebig | `application/json` | Standardvorlage |
|
||||
|
||||
`GET /healthz` liefert `200` oder `503` mit der Liste der Probleme (ohne Anmeldung),
|
||||
z. B. für Uptime Kuma.
|
||||
|
||||
## Fehlersuche
|
||||
|
||||
```bash
|
||||
journalctl -u netmover -f # Log live
|
||||
journalctl -u netmover | grep level=error
|
||||
findmnt -R /mnt/netmover # was ist wirklich eingehängt?
|
||||
```
|
||||
|
||||
| Meldung / Symptom | Ursache und Abhilfe |
|
||||
|---|---|
|
||||
| `mount error(13): Permission denied` (CIFS) | Benutzer oder Passwort falsch, oder keine Rechte auf die Freigabe. |
|
||||
| `mount error(2): No such file or directory` | Freigabename falsch geschrieben (Groß/klein!). |
|
||||
| `mount error(95): Operation not supported` | Keine gemeinsame SMB-Version (im Kernel-Log: „Dialect not supported by server“). Am NAS das maximale SMB-Protokoll auf SMB3 stellen oder Zusatzoption `vers=2.0`. |
|
||||
| `mount error(112): Host is down` | NAS nicht erreichbar, oder SMB1 erwartet: `vers=3.0` bzw. `vers=2.1` probieren. |
|
||||
| `… helper program` / `unknown filesystem type` | `apt install cifs-utils nfs-common` |
|
||||
| `Operation not permitted` / `only root can do that` | Kein `CAP_SYS_ADMIN`: unprivilegierter LXC oder Docker ohne `cap_add`. |
|
||||
| `access denied by server` (NFS) | NFS-Regel am NAS fehlt für die IP dieses Hosts (oder falsches Subnetz in der Regel). |
|
||||
| Test: „Root-Squash“ | Siehe [NFS statt SMB](#nfs-statt-smb); `chown` wird bis dahin übersprungen. |
|
||||
| „… ist nicht leer“ beim Einhängen | Im Mountpoint liegen Dateien, die geschrieben wurden, als nichts eingehängt war. Inhalt prüfen und sichern, Verzeichnis leeren, erneut einhängen. |
|
||||
| „nicht eingehängt – Lauf abgebrochen“ | Die Freigabe ist weg (z. B. von Hand ausgehängt). Auf der Anbindungsseite **Einhängen**. NetMover hängt verschwundene Mounts absichtlich nicht selbst neu ein. |
|
||||
| „gestört (ESTALE …)“, danach wieder OK | NAS-Neustart; NetMover hat neu eingehängt. Bleibt der Fehler: Netzwerk/NAS prüfen. |
|
||||
| „weiterhin nicht verfügbar – neuer Versuch in 60 s“ | Das NAS ist (noch) weg, z. B. während eines DSM-Updates. Bei Anbindungen mit „automatisch einhängen“ versucht NetMover es jede Minute erneut und meldet per Webhook, wenn es wieder klappt. |
|
||||
| „Keine Reaktion seit 120 s“ | Ein Dateizugriff hängt (meist NFS mit `hard`). Die Regel bleibt blockiert („läuft seit …“), bis der Zugriff zurückkehrt; die Anbindung wird vor dem nächsten Lauf geprüft. |
|
||||
| „Hostschlüssel … hat sich geändert“ (SFTP) | Wurde der Zielserver neu aufgesetzt? Dann bei der Anbindung „Hostschlüssel zurücksetzen“ und neu testen. Sonst **nicht** verbinden. |
|
||||
| SFTP: „Anmeldung abgelehnt“ | Öffentlicher Schlüssel fehlt in `authorized_keys`, oder Rechte von `~/.ssh` stimmen nicht (700/600, Eigentümer). |
|
||||
| Paperless bemerkt neue Dateien nicht | Ältere Paperless-Versionen: `PAPERLESS_CONSUMER_POLLING=10` setzen. |
|
||||
| Neue Dateien erscheinen erst nach bis zu 30 s | Der CIFS-Client speichert Verzeichnislisten zwischen. Gelangen Dateien nicht per SMB aufs NAS (z. B. per FTP vom Scanner), erfährt er nichts davon. Abhilfe: Zusatzoption `nohandlecache`. |
|
||||
| Regel gelb „länger als 3 × Intervall“ | Seit dem Dreifachen des Intervalls kein erfolgreicher Lauf; Protokoll der Regel ansehen. |
|
||||
| Einrichtungscode verloren | `systemctl restart netmover` erzeugt einen neuen. |
|
||||
|
||||
## Sicherheit
|
||||
|
||||
- **Nicht ins Internet stellen** (siehe [Reverse Proxy](#reverse-proxy-mit-tls)).
|
||||
- Zugangsdaten liegen nur verschlüsselt in der Datenbank (Fernet, Schlüssel
|
||||
`secret.key`, 0600). Die Oberfläche zeigt sie nie an; in der Prozessliste stehen
|
||||
sie ebenfalls nie, weil `mount.cifs` eine kurzlebige Credentials-Datei bekommt.
|
||||
Ausgaben von `mount` werden geschwärzt, bevor sie protokolliert werden.
|
||||
- **Sichere `secret.key` zusammen mit der Datenbank.** Ohne den Schlüssel sind die
|
||||
gespeicherten Passwörter verloren.
|
||||
- Anmeldung:
|
||||
- Ein Administratorkonto mit Argon2-Passworthash.
|
||||
- Sperre nach 5 Fehlversuchen.
|
||||
- Session-Cookie `HttpOnly`, `SameSite=Lax`.
|
||||
- CSRF-Token bei jedem POST.
|
||||
- Strikte Content-Security-Policy ohne Inline-Skripte.
|
||||
- Eingaben, die in `mount` landen, werden streng geprüft. Zusatzoptionen
|
||||
sind nur aus einer Allowlist erlaubt, `credentials`, `username`, `password` und
|
||||
`pass` sind gesperrt, `mount` wird nie über eine Shell aufgerufen.
|
||||
- Mountpoints entstehen ausschließlich unter `/mnt/netmover`; NetMover ändert
|
||||
`/etc/fstab` nie.
|
||||
- SFTP-Ziele: Schlüssel per `restrict,command="/usr/lib/openssh/sftp-server"` auf
|
||||
SFTP beschränken; der Hostschlüssel wird festgehalten.
|
||||
|
||||
## Entwicklung
|
||||
|
||||
@@ -12,10 +403,26 @@ python3 -m venv .venv # oder: --without-pip + get-pip.py
|
||||
.venv/bin/pip install -e '.[dev]'
|
||||
.venv/bin/pytest
|
||||
.venv/bin/ruff check . && .venv/bin/ruff format --check . && .venv/bin/mypy
|
||||
NETMOVER_DATA_DIR=$PWD/data .venv/bin/netmover migrate
|
||||
|
||||
# Oberfläche ohne root und ohne NAS ausprobieren (simulierte Mounts):
|
||||
NETMOVER_DATA_DIR=/tmp/nm NETMOVER_RUN_DIR=/tmp/nm/run NETMOVER_MOUNT_BASE=/tmp/nm/mnt \
|
||||
NETMOVER_MOUNT_BACKEND=fake .venv/bin/netmover serve
|
||||
# Freigaben simulieren: mkdir -p /tmp/nm/fake/remote/<server>/<freigabe>
|
||||
```
|
||||
|
||||
Neue Migration nach Modelländerung:
|
||||
Aufbau:
|
||||
|
||||
| Modul | Aufgabe |
|
||||
|---|---|
|
||||
| `mounting/` | `MountBackend` (Linux/Fake), Mount-Verwaltung, Test, Wiederherstellung, SFTP |
|
||||
| `mover/` | Scan, Stabilität, atomares Schreiben, Kollisionen, Buchführung, Regellauf |
|
||||
| `runs.py`, `scheduler.py`, `health.py`, `notify.py` | Protokoll, Zeitsteuerung, Gesundheit, Webhook |
|
||||
| `admin.py`, `keys.py`, `transfer.py`, `web/` | Formulare, Schlüsselverwaltung, Export/Import, Oberfläche |
|
||||
|
||||
Die gesamte Mover-Logik ist mit dem `FakeMountBackend` ohne root testbar. SFTP
|
||||
wird gegen einen echten SFTP-Server im Testprozess getestet.
|
||||
|
||||
Neue Migration nach einer Modelländerung:
|
||||
|
||||
```bash
|
||||
NETMOVER_DATA_DIR=$PWD/data .venv/bin/alembic revision --autogenerate -m "…"
|
||||
@@ -23,11 +430,19 @@ NETMOVER_DATA_DIR=$PWD/data .venv/bin/alembic revision --autogenerate -m "…"
|
||||
|
||||
Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt.
|
||||
|
||||
## Arbeitsweise
|
||||
### Arbeitsweise
|
||||
|
||||
- Commits nach [Conventional Commits](https://www.conventionalcommits.org/de/)
|
||||
(`feat:`, `fix:`, `docs:`, `test:`, `chore:` …).
|
||||
- Jede nutzerrelevante Änderung bekommt einen Eintrag unter `[Unreleased]` in
|
||||
[CHANGELOG.md](CHANGELOG.md). Beim Release wird der Abschnitt versioniert,
|
||||
die Version in `src/netmover/__init__.py` angehoben und ein Tag `vX.Y.Z` gesetzt.
|
||||
- Arbeitszeit und Entscheidungen: [DEVLOG.md](DEVLOG.md).
|
||||
- Jede nutzerrelevante Änderung kommt unter `[Unreleased]` in
|
||||
[CHANGELOG.md](CHANGELOG.md). Bei einem Release:
|
||||
1. Den Abschnitt `[Unreleased]` mit Versionsnummer versehen.
|
||||
2. Die Version in `src/netmover/__init__.py` anheben.
|
||||
3. Einen Tag `vX.Y.Z` setzen.
|
||||
- Arbeitszeit und Entscheidungen stehen in [DEVLOG.md](DEVLOG.md).
|
||||
- htmx 2.0.4 liegt unter `src/netmover/web/static/htmx.min.js`
|
||||
(SHA-256 `e209dda5c8235479f3166defc7750e1dbcd5a5c1808b7792fc2e6733768fb447`).
|
||||
|
||||
## Lizenz
|
||||
|
||||
MIT
|
||||
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
# NetMover per Docker Compose. Vorher auf dem Host:
|
||||
# mkdir -p /mnt/netmover
|
||||
# Die Weboberfläche ist nur auf dem Host selbst erreichbar (127.0.0.1:8080).
|
||||
services:
|
||||
netmover:
|
||||
build: .
|
||||
image: netmover:0.2.1
|
||||
container_name: netmover
|
||||
restart: unless-stopped
|
||||
cap_add:
|
||||
- SYS_ADMIN # mount/umount
|
||||
- DAC_READ_SEARCH # mount.cifs liest die Credentials-Datei
|
||||
security_opt:
|
||||
- apparmor:unconfined # Dockers AppArmor-Profil verbietet mount
|
||||
ports:
|
||||
- "127.0.0.1:8080:8080"
|
||||
environment:
|
||||
NETMOVER_LOG_LEVEL: INFO
|
||||
# Hinter einem Reverse Proxy mit HTTPS:
|
||||
# NETMOVER_COOKIE_SECURE: "true"
|
||||
# NETMOVER_FORWARDED_ALLOW_IPS: "172.16.0.0/12"
|
||||
volumes:
|
||||
- netmover-data:/var/lib/netmover
|
||||
# WICHTIG: rshared – sonst bleiben die Mounts im Container gefangen und
|
||||
# weder der Host noch andere Container sehen sie.
|
||||
- type: bind
|
||||
source: /mnt/netmover
|
||||
target: /mnt/netmover
|
||||
bind:
|
||||
propagation: rshared
|
||||
# Lokale Ziele (z. B. Paperless-Consume-Ordner) zusätzlich einbinden:
|
||||
# - /opt/paperless/consume:/opt/paperless/consume
|
||||
tmpfs:
|
||||
- /run/netmover:mode=0700
|
||||
|
||||
volumes:
|
||||
netmover-data:
|
||||
Executable
+52
@@ -0,0 +1,52 @@
|
||||
#!/bin/sh
|
||||
# NetMover auf Debian 13 installieren bzw. aktualisieren (als root ausführen).
|
||||
#
|
||||
# git clone https://git.wilde-server.de/florian.wilde/netmover.git /opt/netmover/src
|
||||
# /opt/netmover/src/deploy/install.sh
|
||||
#
|
||||
# Spätere Updates: netmover-update (siehe deploy/update.sh).
|
||||
set -eu
|
||||
|
||||
SOURCE="${1:-$(cd "$(dirname "$0")/.." && pwd)}"
|
||||
PREFIX=/opt/netmover
|
||||
|
||||
if [ "$(id -u)" -ne 0 ]; then
|
||||
echo "Bitte als root ausführen." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "==> Pakete"
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq python3 python3-venv cifs-utils nfs-common git >/dev/null
|
||||
|
||||
echo "==> Virtuelle Umgebung in $PREFIX/venv"
|
||||
install -d -m 0755 "$PREFIX"
|
||||
[ -x "$PREFIX/venv/bin/python" ] || python3 -m venv "$PREFIX/venv"
|
||||
"$PREFIX/venv/bin/pip" install -q --upgrade pip
|
||||
"$PREFIX/venv/bin/pip" install -q --upgrade "$SOURCE"
|
||||
|
||||
echo "==> Konfiguration"
|
||||
install -d -m 0755 /etc/netmover
|
||||
if [ ! -f /etc/netmover/netmover.env ]; then
|
||||
install -m 0640 "$(dirname "$0")/netmover.env" /etc/netmover/netmover.env
|
||||
fi
|
||||
install -d -m 0755 /mnt/netmover
|
||||
|
||||
echo "==> Update-Skript"
|
||||
install -m 0755 "$(dirname "$0")/update.sh" /usr/local/sbin/netmover-update
|
||||
|
||||
echo "==> systemd"
|
||||
install -m 0644 "$(dirname "$0")/netmover.service" /etc/systemd/system/netmover.service
|
||||
systemctl daemon-reload
|
||||
systemctl enable netmover >/dev/null 2>&1
|
||||
systemctl restart netmover
|
||||
|
||||
sleep 2
|
||||
systemctl --no-pager --lines=0 status netmover || true
|
||||
echo
|
||||
if journalctl -u netmover -n 200 --no-pager | grep -q "setup.required"; then
|
||||
echo "Ersteinrichtung: Einrichtungscode aus dem Log:"
|
||||
journalctl -u netmover -n 200 --no-pager | grep "setup.required" | tail -1
|
||||
fi
|
||||
echo "Fertig. Weboberfläche: http://127.0.0.1:8080 (README: SSH-Tunnel / Reverse Proxy)"
|
||||
@@ -0,0 +1,17 @@
|
||||
# Umgebungsvariablen für NetMover (/etc/netmover/netmover.env).
|
||||
# Nach Änderungen: systemctl restart netmover
|
||||
|
||||
# Nur lokal erreichbar – für Zugriff aus dem LAN einen Reverse Proxy mit TLS
|
||||
# davorsetzen (README, „Reverse Proxy“) oder per SSH-Tunnel arbeiten.
|
||||
NETMOVER_HOST=127.0.0.1
|
||||
NETMOVER_PORT=8080
|
||||
|
||||
# Hinter einem Reverse Proxy mit HTTPS:
|
||||
# NETMOVER_COOKIE_SECURE=true
|
||||
# NETMOVER_FORWARDED_ALLOW_IPS=127.0.0.1
|
||||
|
||||
# NETMOVER_MOUNT_BASE=/mnt/netmover
|
||||
# NETMOVER_SESSION_HOURS=12
|
||||
NETMOVER_LOG_LEVEL=INFO
|
||||
# logfmt (gut lesbar im Journal) oder json
|
||||
NETMOVER_LOG_FORMAT=logfmt
|
||||
@@ -0,0 +1,44 @@
|
||||
# NetMover als systemd-Dienst direkt auf dem Host (empfohlen).
|
||||
# Installation: siehe README, Abschnitt „Installation“.
|
||||
[Unit]
|
||||
Description=NetMover – NFS/SMB-Mounts und zeitgesteuertes Verschieben
|
||||
Documentation=https://git.wilde-server.de/florian.wilde/netmover
|
||||
Wants=network-online.target
|
||||
After=network-online.target remote-fs-pre.target
|
||||
|
||||
[Service]
|
||||
Type=exec
|
||||
# root ist nötig: mount/umount brauchen CAP_SYS_ADMIN, und Dateien auf beliebigen
|
||||
# Freigaben sollen mit den richtigen Eigentümern geschrieben werden (siehe README).
|
||||
User=root
|
||||
EnvironmentFile=-/etc/netmover/netmover.env
|
||||
ExecStart=/opt/netmover/venv/bin/netmover serve
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
# Beim Stoppen werden Mounts bewusst NICHT ausgehängt.
|
||||
KillMode=mixed
|
||||
TimeoutStopSec=20
|
||||
|
||||
# /var/lib/netmover (Datenbank, Schlüssel) und /run/netmover (kurzlebige
|
||||
# Credentials-Dateien für mount.cifs) – beide nur für root lesbar.
|
||||
StateDirectory=netmover
|
||||
StateDirectoryMode=0700
|
||||
RuntimeDirectory=netmover
|
||||
RuntimeDirectoryMode=0700
|
||||
|
||||
# Härtung, soweit mit Mounten vereinbar.
|
||||
# WICHTIG: Keine Optionen, die einen eigenen Mount-Namespace erzeugen
|
||||
# (ProtectSystem=, ProtectHome=, PrivateTmp=, ProtectControlGroups=,
|
||||
# ReadOnlyPaths=, BindPaths= …). Sonst landen die Mounts im Namespace des
|
||||
# Dienstes: das System und andere Programme (Paperless!) sähen sie nicht, und
|
||||
# jeder Neustart des Dienstes würde sie verwerfen. MountFlags=shared hilft dagegen
|
||||
# nicht – systemd setzt den Namespace als „slave“ auf, Mounts fließen nur hinein.
|
||||
# Auf dem Testsystem nachgewiesen; prüfen mit: findmnt -l | grep /mnt/netmover
|
||||
NoNewPrivileges=yes
|
||||
LockPersonality=yes
|
||||
RestrictRealtime=yes
|
||||
RestrictSUIDSGID=yes
|
||||
UMask=0022
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Executable
+153
@@ -0,0 +1,153 @@
|
||||
#!/bin/sh
|
||||
# NetMover aktualisieren (als root). Wird von install.sh als
|
||||
# /usr/local/sbin/netmover-update installiert.
|
||||
#
|
||||
# netmover-update # auf die neueste Version (neuester Tag vX.Y.Z)
|
||||
# netmover-update v0.2.0 # auf eine bestimmte Version (auch zurück)
|
||||
# netmover-update --list # verfügbare Versionen anzeigen
|
||||
#
|
||||
# Ablauf: Tags holen → Datenbank und Schlüssel sichern → Version auschecken →
|
||||
# install.sh (installiert, migriert beim Start, startet den Dienst neu) →
|
||||
# Gesundheit prüfen. Die Mounts bleiben dabei eingehängt.
|
||||
set -eu
|
||||
|
||||
SRC="${NETMOVER_SRC:-/opt/netmover/src}"
|
||||
VENV=/opt/netmover/venv
|
||||
ENV_FILE=/etc/netmover/netmover.env
|
||||
BACKUP_ROOT=/var/backups/netmover
|
||||
KEEP_BACKUPS=10
|
||||
|
||||
if [ "$(id -u)" -ne 0 ]; then
|
||||
echo "Bitte als root ausführen." >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -d "$SRC/.git" ]; then
|
||||
echo "Kein Git-Klon unter $SRC. Einmalig einrichten:" >&2
|
||||
echo " git clone https://git.wilde-server.de/florian.wilde/netmover.git $SRC" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Einstellungen des Dienstes (Port, Datenverzeichnis)
|
||||
NETMOVER_PORT=8080
|
||||
NETMOVER_DATA_DIR=/var/lib/netmover
|
||||
# shellcheck disable=SC1090
|
||||
[ -f "$ENV_FILE" ] && . "$ENV_FILE"
|
||||
|
||||
cd "$SRC"
|
||||
git fetch --tags --quiet origin
|
||||
|
||||
if [ "${1:-}" = "--list" ]; then
|
||||
git tag -l 'v*' --sort=-v:refname
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TARGET="${1:-$(git tag -l 'v*' --sort=-v:refname | head -n 1)}"
|
||||
if ! git rev-parse -q --verify "refs/tags/$TARGET" >/dev/null; then
|
||||
echo "Version $TARGET gibt es nicht. Verfügbar:" >&2
|
||||
git tag -l 'v*' --sort=-v:refname | head -n 10 >&2
|
||||
exit 1
|
||||
fi
|
||||
CURRENT="v$("$VENV/bin/netmover" version 2>/dev/null || echo "?")"
|
||||
echo "Installiert: $CURRENT"
|
||||
echo "Ziel: $TARGET"
|
||||
if [ "$CURRENT" = "$TARGET" ] && [ "${FORCE:-0}" != "1" ]; then
|
||||
echo "Bereits auf dem gewünschten Stand (FORCE=1 erzwingt eine Neuinstallation)."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
NEWEST="$(printf '%s\n%s\n' "$CURRENT" "$TARGET" | sort -V | tail -n 1)"
|
||||
if [ "$NEWEST" = "$CURRENT" ] && [ "$CURRENT" != "$TARGET" ]; then
|
||||
echo
|
||||
echo "!! Zurückstufen auf eine ältere Version: Wurde die Datenbank inzwischen migriert,"
|
||||
echo " startet die alte Version nicht. Dann vorher die Sicherung von *vor* dem Update"
|
||||
echo " zurückspielen (siehe $BACKUP_ROOT, Anleitung im README)."
|
||||
printf " Trotzdem fortfahren? [j/N] "
|
||||
read -r answer
|
||||
[ "$answer" = "j" ] || [ "$answer" = "J" ] || exit 1
|
||||
fi
|
||||
|
||||
if [ "$NEWEST" = "$TARGET" ]; then
|
||||
echo
|
||||
echo "==> Änderungen laut Changelog"
|
||||
git show "$TARGET:CHANGELOG.md" | awk -v cur="${CURRENT#v}" '
|
||||
/^## \[/ { if (index($0, "[" cur "]")) exit; show = ($0 !~ /Unreleased/) }
|
||||
show' | head -n 60
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "==> Sicherung"
|
||||
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||
BACKUP="$BACKUP_ROOT/$STAMP-$CURRENT"
|
||||
install -d -m 0700 "$BACKUP_ROOT" "$BACKUP"
|
||||
# Konsistente Kopie auch bei laufendem Dienst (SQLite-Online-Backup)
|
||||
"$VENV/bin/python" - "$NETMOVER_DATA_DIR/netmover.db" "$BACKUP/netmover.db" <<'EOF'
|
||||
import sqlite3, sys
|
||||
src, dst = sqlite3.connect(sys.argv[1]), sqlite3.connect(sys.argv[2])
|
||||
with dst:
|
||||
src.backup(dst)
|
||||
src.close(); dst.close()
|
||||
EOF
|
||||
for f in secret.key session.key; do
|
||||
[ -f "$NETMOVER_DATA_DIR/$f" ] && cp -p "$NETMOVER_DATA_DIR/$f" "$BACKUP/"
|
||||
done
|
||||
[ -f "$ENV_FILE" ] && cp -p "$ENV_FILE" "$BACKUP/"
|
||||
chmod 0600 "$BACKUP"/*
|
||||
echo " $BACKUP"
|
||||
# alte Sicherungen aufräumen
|
||||
find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -print | sort \
|
||||
| head -n "-$KEEP_BACKUPS" | xargs -r rm -rf --
|
||||
|
||||
echo
|
||||
echo "==> Version $TARGET auschecken und installieren"
|
||||
git -c advice.detachedHead=false checkout --quiet "$TARGET"
|
||||
"$SRC/deploy/install.sh" "$SRC" >/tmp/netmover-update.log 2>&1 || {
|
||||
echo "install.sh ist fehlgeschlagen – Ausgabe: /tmp/netmover-update.log" >&2
|
||||
tail -n 20 /tmp/netmover-update.log >&2
|
||||
FAILED=1
|
||||
}
|
||||
|
||||
echo
|
||||
echo "==> Prüfung"
|
||||
OK=0
|
||||
if [ "${FAILED:-0}" != "1" ]; then
|
||||
i=0
|
||||
while [ $i -lt 30 ]; do
|
||||
if "$VENV/bin/python" - "$NETMOVER_PORT" <<'EOF'
|
||||
import json, sys, urllib.error, urllib.request
|
||||
url = f"http://127.0.0.1:{sys.argv[1]}/healthz"
|
||||
try:
|
||||
body = urllib.request.urlopen(url, timeout=3).read()
|
||||
except urllib.error.HTTPError as exc: # 503 = läuft, meldet aber Probleme
|
||||
body = exc.read()
|
||||
except OSError:
|
||||
sys.exit(1)
|
||||
data = json.loads(body)
|
||||
print(f" Gesundheit: {data['status']}")
|
||||
for p in data.get("problems", []):
|
||||
print(f" – {p}")
|
||||
EOF
|
||||
then
|
||||
OK=1
|
||||
break
|
||||
fi
|
||||
i=$((i + 1))
|
||||
sleep 1
|
||||
done
|
||||
fi
|
||||
|
||||
if [ "$OK" = "1" ] && systemctl is-active --quiet netmover; then
|
||||
echo " Dienst läuft: $("$VENV/bin/netmover" version)"
|
||||
echo
|
||||
echo "Fertig. Sicherung: $BACKUP"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "!! Der Dienst läuft nach dem Update nicht richtig." >&2
|
||||
echo " Log: journalctl -u netmover -n 50" >&2
|
||||
echo " Zurück auf $CURRENT (Datenbank aus der Sicherung, weil sie migriert sein kann):" >&2
|
||||
echo " systemctl stop netmover" >&2
|
||||
echo " cp $BACKUP/netmover.db $NETMOVER_DATA_DIR/netmover.db" >&2
|
||||
echo " rm -f $NETMOVER_DATA_DIR/netmover.db-wal $NETMOVER_DATA_DIR/netmover.db-shm" >&2
|
||||
echo " netmover-update $CURRENT" >&2
|
||||
exit 1
|
||||
@@ -33,6 +33,7 @@ dev = [
|
||||
"ruff>=0.6",
|
||||
"mypy>=1.11",
|
||||
"types-paramiko",
|
||||
"shellcheck-py",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
||||
|
||||
__version__ = "0.1.0.dev0"
|
||||
__version__ = "0.2.1"
|
||||
|
||||
@@ -0,0 +1,582 @@
|
||||
"""Verwaltung von Anbindungen und Regeln – Prüfen und Speichern von Formulardaten.
|
||||
|
||||
Unabhängig vom Web-Framework: nimmt ein ``Mapping[str, str]`` (Formular) und
|
||||
liefert entweder gespeicherte Objekte oder ein Fehler-Dict ``{feld: meldung}``.
|
||||
|
||||
Passwörter: leer lassen heißt „unverändert“; sie werden nie zurückgegeben.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Callable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from netmover.crypto import SecretBox, generate_ssh_keypair
|
||||
from netmover.models import (
|
||||
CollisionPolicy,
|
||||
Credential,
|
||||
CredentialKind,
|
||||
Mount,
|
||||
MountState,
|
||||
MountType,
|
||||
PatternType,
|
||||
Rule,
|
||||
RuleAction,
|
||||
)
|
||||
from netmover.mounting.manager import CredentialData, MountConfig, MountManager
|
||||
from netmover.validation import (
|
||||
MountOption,
|
||||
ValidationError,
|
||||
parse_extra_options,
|
||||
safe_abspath,
|
||||
safe_relpath,
|
||||
slugify,
|
||||
validate_id,
|
||||
validate_mode,
|
||||
validate_port,
|
||||
validate_secret,
|
||||
validate_server,
|
||||
validate_share,
|
||||
validate_username,
|
||||
)
|
||||
|
||||
Errors = dict[str, str]
|
||||
|
||||
MIN_INTERVAL = 10
|
||||
_OWNER_RE = re.compile(r"^\d{1,10}(:\d{1,10})?$")
|
||||
|
||||
|
||||
class AdminError(Exception):
|
||||
"""Aktion nicht möglich – Meldung für die Oberfläche."""
|
||||
|
||||
|
||||
def _bool(form: Mapping[str, str], key: str) -> bool:
|
||||
return form.get(key, "") in ("on", "true", "1", "yes")
|
||||
|
||||
|
||||
def _int(
|
||||
form: Mapping[str, str],
|
||||
key: str,
|
||||
errors: Errors,
|
||||
*,
|
||||
default: int | None = None,
|
||||
minimum: int | None = None,
|
||||
label: str = "",
|
||||
) -> int | None:
|
||||
raw = (form.get(key) or "").strip()
|
||||
if raw == "":
|
||||
return default
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError:
|
||||
errors[key] = "Bitte eine ganze Zahl angeben."
|
||||
return default
|
||||
if minimum is not None and value < minimum:
|
||||
errors[key] = f"{label or 'Wert'} muss mindestens {minimum} sein."
|
||||
return value
|
||||
|
||||
|
||||
# -- Anbindungen ------------------------------------------------------------------------
|
||||
|
||||
|
||||
NEW_KEY = "new"
|
||||
|
||||
|
||||
@dataclass
|
||||
class MountInput:
|
||||
name: str = ""
|
||||
type: MountType = MountType.CIFS
|
||||
server: str = ""
|
||||
share: str = ""
|
||||
port: int | None = None
|
||||
username: str = ""
|
||||
password: str = "" # leer = unverändert
|
||||
domain: str = ""
|
||||
auth: CredentialKind = CredentialKind.PASSWORD # nur sftp: Schlüssel oder Passwort
|
||||
# nur sftp + Schlüssel: "new" = eigener Schlüssel nur für diese Anbindung,
|
||||
# sonst ID eines Schlüssels aus der Schlüsselverwaltung
|
||||
key_id: str = NEW_KEY
|
||||
uid: int | None = None
|
||||
gid: int | None = None
|
||||
file_mode: str | None = None
|
||||
dir_mode: str | None = None
|
||||
extra_options: str = ""
|
||||
auto_mount: bool = True
|
||||
enabled: bool = True
|
||||
|
||||
def config(self) -> MountConfig:
|
||||
return MountConfig(
|
||||
name=self.name or "neu",
|
||||
type=self.type,
|
||||
server=self.server,
|
||||
share=self.share,
|
||||
port=self.port,
|
||||
uid=self.uid,
|
||||
gid=self.gid,
|
||||
file_mode=self.file_mode,
|
||||
dir_mode=self.dir_mode,
|
||||
extra_options=self.extra_options,
|
||||
)
|
||||
|
||||
|
||||
def mount_input_from_model(m: Mount) -> MountInput:
|
||||
c = m.credential
|
||||
return MountInput(
|
||||
name=m.name,
|
||||
type=m.type,
|
||||
server=m.server,
|
||||
share=m.share,
|
||||
port=m.port,
|
||||
username=m.username or (c.username if c else ""),
|
||||
domain=(c.domain or "") if c else "",
|
||||
auth=c.kind if c else CredentialKind.PASSWORD,
|
||||
key_id=str(c.id) if c is not None and c.managed else NEW_KEY,
|
||||
uid=m.uid,
|
||||
gid=m.gid,
|
||||
file_mode=m.file_mode,
|
||||
dir_mode=m.dir_mode,
|
||||
extra_options=m.extra_options,
|
||||
auto_mount=m.auto_mount,
|
||||
enabled=m.enabled,
|
||||
)
|
||||
|
||||
|
||||
def parse_mount_form(form: Mapping[str, str]) -> tuple[MountInput, Errors]:
|
||||
errors: Errors = {}
|
||||
data = MountInput(
|
||||
name=(form.get("name") or "").strip(),
|
||||
server=(form.get("server") or "").strip(),
|
||||
share=(form.get("share") or "").strip(),
|
||||
username=(form.get("username") or "").strip(),
|
||||
password=form.get("password") or "",
|
||||
domain=(form.get("domain") or "").strip(),
|
||||
extra_options=(form.get("extra_options") or "").strip(),
|
||||
auto_mount=_bool(form, "auto_mount"),
|
||||
enabled=_bool(form, "enabled"),
|
||||
key_id=(form.get("key_id") or NEW_KEY).strip(),
|
||||
)
|
||||
if data.key_id != NEW_KEY and not data.key_id.isdigit():
|
||||
errors["key_id"] = "Ungültige Schlüsselauswahl."
|
||||
data.key_id = NEW_KEY
|
||||
try:
|
||||
data.type = MountType(form.get("type", "cifs"))
|
||||
except ValueError:
|
||||
errors["type"] = "Unbekannter Typ."
|
||||
return data, errors
|
||||
try:
|
||||
data.auth = CredentialKind(form.get("auth") or "password")
|
||||
except ValueError:
|
||||
errors["auth"] = "Unbekannte Anmeldeart."
|
||||
|
||||
if not data.name:
|
||||
errors["name"] = "Name fehlt."
|
||||
elif len(data.name) > 100:
|
||||
errors["name"] = "Höchstens 100 Zeichen."
|
||||
else:
|
||||
try:
|
||||
slugify(data.name)
|
||||
except ValidationError as exc:
|
||||
errors["name"] = str(exc)
|
||||
|
||||
def check[T](key: str, fn: Callable[[], T], fallback: T) -> T:
|
||||
try:
|
||||
return fn()
|
||||
except ValidationError as exc:
|
||||
errors[key] = str(exc)
|
||||
return fallback
|
||||
|
||||
data.server = check("server", lambda: validate_server(data.server), data.server)
|
||||
data.share = check("share", lambda: validate_share(data.type, data.share), data.share)
|
||||
no_options: list[MountOption] = []
|
||||
check("extra_options", lambda: parse_extra_options(data.type, data.extra_options), no_options)
|
||||
check("password", lambda: validate_secret(data.password), "")
|
||||
if data.type is MountType.CIFS:
|
||||
data.uid = check("uid", lambda: validate_id("uid", form.get("uid")), None)
|
||||
data.gid = check("gid", lambda: validate_id("gid", form.get("gid")), None)
|
||||
data.file_mode = check(
|
||||
"file_mode", lambda: validate_mode("file_mode", form.get("file_mode")), None
|
||||
)
|
||||
data.dir_mode = check(
|
||||
"dir_mode", lambda: validate_mode("dir_mode", form.get("dir_mode")), None
|
||||
)
|
||||
if data.type is MountType.SFTP:
|
||||
data.port = check("port", lambda: validate_port(form.get("port")), None)
|
||||
if data.username:
|
||||
data.username = check("username", lambda: validate_username(data.username), data.username)
|
||||
if data.domain:
|
||||
check("domain", lambda: validate_username(data.domain), "")
|
||||
if data.type is MountType.SFTP and not data.username:
|
||||
errors["username"] = "SFTP braucht einen Benutzernamen."
|
||||
if data.type is not MountType.SFTP:
|
||||
data.auth = CredentialKind.PASSWORD
|
||||
if data.auth is not CredentialKind.SSH_KEY:
|
||||
data.key_id = NEW_KEY
|
||||
return data, errors
|
||||
|
||||
|
||||
def credential_for_test(
|
||||
data: MountInput, existing: Mount | None, secrets: SecretBox, s: Session | None = None
|
||||
) -> CredentialData | None:
|
||||
"""Zugangsdaten für „Testen“ aus dem Formular – leeres Passwort = gespeichertes,
|
||||
gewählter Schlüssel aus der Schlüsselverwaltung."""
|
||||
if data.type in (MountType.NFS3, MountType.NFS4) or not data.username:
|
||||
return None
|
||||
if data.auth is CredentialKind.SSH_KEY and data.key_id != NEW_KEY and s is not None:
|
||||
key = s.get(Credential, int(data.key_id))
|
||||
if key is None or not key.managed:
|
||||
return None
|
||||
return CredentialData(data.auth, data.username, secrets.decrypt(key.secret_enc))
|
||||
secret = data.password
|
||||
stored = existing.credential if existing is not None else None
|
||||
if stored is not None and stored.managed:
|
||||
stored = None # eigener Schlüssel gewählt, gespeicherter ist nicht gemeint
|
||||
if not secret and stored is not None and stored.kind is data.auth:
|
||||
secret = secrets.decrypt(stored.secret_enc)
|
||||
if not secret:
|
||||
return None
|
||||
return CredentialData(data.auth, data.username, secret, data.domain or None)
|
||||
|
||||
|
||||
_CONNECTION_FIELDS = (
|
||||
"type",
|
||||
"server",
|
||||
"share",
|
||||
"port",
|
||||
"uid",
|
||||
"gid",
|
||||
"file_mode",
|
||||
"dir_mode",
|
||||
"extra_options",
|
||||
)
|
||||
|
||||
|
||||
def save_mount(
|
||||
s: Session,
|
||||
secrets: SecretBox,
|
||||
mounts: MountManager,
|
||||
data: MountInput,
|
||||
mount_id: int | None = None,
|
||||
) -> tuple[Mount | None, Errors]:
|
||||
errors: Errors = {}
|
||||
m = s.get(Mount, mount_id) if mount_id is not None else None
|
||||
if mount_id is not None and m is None:
|
||||
raise AdminError("Anbindung existiert nicht.")
|
||||
dup = s.scalar(select(Mount.id).where(Mount.name == data.name))
|
||||
if dup is not None and dup != mount_id:
|
||||
errors["name"] = "Name ist bereits vergeben."
|
||||
return None, errors
|
||||
|
||||
if m is None:
|
||||
base = slugify(data.name)
|
||||
slug, n = base, 1
|
||||
while s.scalar(select(Mount.id).where(Mount.slug == slug)) is not None:
|
||||
n += 1
|
||||
slug = f"{base[:44]}-{n}"
|
||||
m = Mount(slug=slug, state=MountState.UNMOUNTED, extra_options="")
|
||||
if data.type.is_kernel_mount:
|
||||
m.mountpoint = str(mounts.mountpoint_for(slug))
|
||||
s.add(m)
|
||||
else:
|
||||
changed = [f for f in _CONNECTION_FIELDS if getattr(m, f) != getattr(data, f)]
|
||||
if (
|
||||
changed
|
||||
and m.type.is_kernel_mount
|
||||
and m.state in (MountState.MOUNTED, MountState.DEGRADED)
|
||||
):
|
||||
errors["_"] = (
|
||||
"Verbindungsdaten lassen sich nur ändern, wenn die Anbindung ausgehängt ist."
|
||||
)
|
||||
return None, errors
|
||||
if m.type is MountType.SFTP and (m.server != data.server or m.port != data.port):
|
||||
m.host_key = None # anderer Server → Hostschlüssel neu bestätigen
|
||||
if data.type.is_kernel_mount and not m.mountpoint:
|
||||
m.mountpoint = str(mounts.mountpoint_for(m.slug))
|
||||
if not data.type.is_kernel_mount:
|
||||
m.mountpoint = None
|
||||
|
||||
for f in (
|
||||
"name",
|
||||
"type",
|
||||
"server",
|
||||
"share",
|
||||
"port",
|
||||
"uid",
|
||||
"gid",
|
||||
"file_mode",
|
||||
"dir_mode",
|
||||
"extra_options",
|
||||
"auto_mount",
|
||||
"enabled",
|
||||
):
|
||||
setattr(m, f, getattr(data, f))
|
||||
if not data.type.is_kernel_mount:
|
||||
m.auto_mount = False
|
||||
|
||||
# Zugangsdaten: entweder ein gespeicherter Schlüssel aus der Schlüsselverwaltung
|
||||
# (geteilt, wird hier nie verändert) oder eigene Zugangsdaten nur dieser Anbindung.
|
||||
current = m.credential
|
||||
own = current if current is not None and not current.managed else None
|
||||
needs_cred = data.type is MountType.SFTP or (
|
||||
data.type is MountType.CIFS and bool(data.username)
|
||||
)
|
||||
use_managed = (
|
||||
data.type is MountType.SFTP
|
||||
and data.auth is CredentialKind.SSH_KEY
|
||||
and data.key_id != NEW_KEY
|
||||
)
|
||||
if not needs_cred:
|
||||
m.credential = None
|
||||
m.username = None
|
||||
elif use_managed:
|
||||
key = s.get(Credential, int(data.key_id))
|
||||
if key is None or not key.managed or key.kind is not CredentialKind.SSH_KEY:
|
||||
s.rollback()
|
||||
return None, {"key_id": "Schlüssel existiert nicht."}
|
||||
m.credential = key
|
||||
m.username = data.username
|
||||
else:
|
||||
m.username = None
|
||||
cred = own
|
||||
if cred is None or cred.kind is not data.auth:
|
||||
if data.auth is CredentialKind.SSH_KEY:
|
||||
private, public = generate_ssh_keypair(f"netmover@{m.slug}")
|
||||
secret, pub = private, public
|
||||
else:
|
||||
if not data.password:
|
||||
errors["password"] = "Passwort fehlt." # noqa: S105
|
||||
s.rollback()
|
||||
return None, errors
|
||||
secret, pub = data.password, None
|
||||
if cred is None:
|
||||
cred = Credential(name=f"mount:{m.slug}", managed=False)
|
||||
s.add(cred)
|
||||
cred.kind = data.auth
|
||||
cred.secret_enc = secrets.encrypt(secret)
|
||||
cred.public_key = pub
|
||||
elif data.password and data.auth is CredentialKind.PASSWORD:
|
||||
cred.secret_enc = secrets.encrypt(data.password)
|
||||
cred.username = data.username
|
||||
cred.domain = data.domain or None
|
||||
m.credential = cred
|
||||
# Eigene Zugangsdaten, die nicht mehr gebraucht werden, entfernen.
|
||||
if own is not None and m.credential is not own:
|
||||
s.flush()
|
||||
if not own.mounts:
|
||||
s.delete(own)
|
||||
try:
|
||||
s.flush()
|
||||
except IntegrityError:
|
||||
s.rollback()
|
||||
errors["_"] = "Speichern fehlgeschlagen (Name oder Mountpoint doppelt)."
|
||||
return None, errors
|
||||
return m, errors
|
||||
|
||||
|
||||
def delete_mount(s: Session, mounts: MountManager, mount_id: int) -> None:
|
||||
m = s.get(Mount, mount_id)
|
||||
if m is None:
|
||||
return
|
||||
used = s.scalars(
|
||||
select(Rule.name).where((Rule.src_mount_id == mount_id) | (Rule.dst_mount_id == mount_id))
|
||||
).all()
|
||||
if used:
|
||||
raise AdminError(f"Wird noch von Regeln benutzt: {', '.join(used)}")
|
||||
if m.type.is_kernel_mount and m.state is not MountState.UNMOUNTED:
|
||||
mounts.unmount(mount_id)
|
||||
cred = m.credential
|
||||
s.delete(m)
|
||||
s.flush()
|
||||
# Eigene Zugangsdaten gehen mit; Schlüssel aus der Schlüsselverwaltung bleiben.
|
||||
if cred is not None and not cred.managed and not cred.mounts:
|
||||
s.delete(cred)
|
||||
|
||||
|
||||
# -- Regeln ----------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class RuleInput:
|
||||
name: str = ""
|
||||
enabled: bool = True
|
||||
src_mount_id: int | None = None
|
||||
src_subpath: str = ""
|
||||
dst_kind: str = "mount" # "mount" | "local"
|
||||
dst_mount_id: int | None = None
|
||||
dst_path: str = ""
|
||||
pattern: str = "*"
|
||||
pattern_type: PatternType = PatternType.GLOB
|
||||
recursive: bool = False
|
||||
interval_seconds: int = 30
|
||||
stability_seconds: int = 5
|
||||
min_age_seconds: int = 0
|
||||
max_size_mb: int | None = None
|
||||
op_timeout_seconds: int = 120
|
||||
action: RuleAction = RuleAction.MOVE
|
||||
on_collision: CollisionPolicy = CollisionPolicy.RENAME
|
||||
set_owner: str = ""
|
||||
set_mode: str = ""
|
||||
dry_run: bool = False
|
||||
quarantine_path: str = ""
|
||||
quarantine_after: int = 3
|
||||
extra: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
def rule_input_from_model(r: Rule) -> RuleInput:
|
||||
return RuleInput(
|
||||
name=r.name,
|
||||
enabled=r.enabled,
|
||||
src_mount_id=r.src_mount_id,
|
||||
src_subpath=r.src_subpath,
|
||||
dst_kind="mount" if r.dst_mount_id else "local",
|
||||
dst_mount_id=r.dst_mount_id,
|
||||
dst_path=r.dst_path,
|
||||
pattern=r.pattern,
|
||||
pattern_type=r.pattern_type,
|
||||
recursive=r.recursive,
|
||||
interval_seconds=r.interval_seconds,
|
||||
stability_seconds=r.stability_seconds,
|
||||
min_age_seconds=r.min_age_seconds,
|
||||
max_size_mb=r.max_size_mb,
|
||||
op_timeout_seconds=r.op_timeout_seconds,
|
||||
action=r.action,
|
||||
on_collision=r.on_collision,
|
||||
set_owner=r.set_owner or "",
|
||||
set_mode=r.set_mode or "",
|
||||
dry_run=r.dry_run,
|
||||
quarantine_path=r.quarantine_path or "",
|
||||
quarantine_after=r.quarantine_after,
|
||||
)
|
||||
|
||||
|
||||
def parse_rule_form(s: Session, form: Mapping[str, str]) -> tuple[RuleInput, Errors]:
|
||||
errors: Errors = {}
|
||||
d = RuleInput(
|
||||
name=(form.get("name") or "").strip(),
|
||||
enabled=_bool(form, "enabled"),
|
||||
src_subpath=form.get("src_subpath") or "",
|
||||
dst_kind=form.get("dst_kind") or "mount",
|
||||
dst_path=(form.get("dst_path") or "").strip(),
|
||||
pattern=(form.get("pattern") or "").strip() or "*",
|
||||
recursive=_bool(form, "recursive"),
|
||||
set_owner=(form.get("set_owner") or "").strip(),
|
||||
set_mode=(form.get("set_mode") or "").strip(),
|
||||
dry_run=_bool(form, "dry_run"),
|
||||
quarantine_path=form.get("quarantine_path") or "",
|
||||
)
|
||||
if not d.name:
|
||||
errors["name"] = "Name fehlt."
|
||||
for key, enum_cls in (
|
||||
("pattern_type", PatternType),
|
||||
("action", RuleAction),
|
||||
("on_collision", CollisionPolicy),
|
||||
):
|
||||
try:
|
||||
setattr(d, key, enum_cls(form.get(key) or getattr(d, key).value))
|
||||
except ValueError:
|
||||
errors[key] = "Ungültiger Wert."
|
||||
d.src_mount_id = _int(form, "src_mount_id", errors)
|
||||
d.dst_mount_id = _int(form, "dst_mount_id", errors) if d.dst_kind == "mount" else None
|
||||
d.interval_seconds = (
|
||||
_int(form, "interval_seconds", errors, default=30, minimum=MIN_INTERVAL, label="Intervall")
|
||||
or 30
|
||||
)
|
||||
d.stability_seconds = _int(form, "stability_seconds", errors, default=5, minimum=0) or 0
|
||||
d.min_age_seconds = _int(form, "min_age_seconds", errors, default=0, minimum=0) or 0
|
||||
d.max_size_mb = _int(form, "max_size_mb", errors, minimum=1)
|
||||
d.op_timeout_seconds = (
|
||||
_int(form, "op_timeout_seconds", errors, default=120, minimum=10, label="Zeitlimit") or 120
|
||||
)
|
||||
d.quarantine_after = _int(form, "quarantine_after", errors, default=3, minimum=1) or 3
|
||||
|
||||
src = s.get(Mount, d.src_mount_id) if d.src_mount_id else None
|
||||
if src is None:
|
||||
errors["src_mount_id"] = "Quelle wählen."
|
||||
elif not src.type.is_kernel_mount:
|
||||
errors["src_mount_id"] = "SFTP-Anbindungen sind nur als Ziel möglich."
|
||||
try:
|
||||
d.src_subpath = safe_relpath("src_subpath", d.src_subpath)
|
||||
except ValidationError as exc:
|
||||
errors["src_subpath"] = str(exc)
|
||||
if d.dst_kind == "mount":
|
||||
if d.dst_mount_id is None or s.get(Mount, d.dst_mount_id) is None:
|
||||
errors["dst_mount_id"] = "Ziel wählen."
|
||||
try:
|
||||
d.dst_path = safe_relpath("dst_path", d.dst_path)
|
||||
except ValidationError as exc:
|
||||
errors["dst_path"] = str(exc)
|
||||
if (
|
||||
d.dst_mount_id == d.src_mount_id
|
||||
and "src_subpath" not in errors
|
||||
and d.dst_path == d.src_subpath
|
||||
):
|
||||
errors["dst_path"] = "Quelle und Ziel sind derselbe Ordner."
|
||||
elif d.dst_kind == "local":
|
||||
try:
|
||||
d.dst_path = safe_abspath("dst_path", d.dst_path)
|
||||
except ValidationError as exc:
|
||||
errors["dst_path"] = str(exc)
|
||||
else:
|
||||
errors["dst_kind"] = "Ungültige Zielart."
|
||||
|
||||
if d.pattern_type is PatternType.REGEX:
|
||||
try:
|
||||
re.compile(d.pattern)
|
||||
except re.error as exc:
|
||||
errors["pattern"] = f"Ungültiger regulärer Ausdruck: {exc}"
|
||||
if d.stability_seconds >= d.op_timeout_seconds:
|
||||
errors["stability_seconds"] = "Muss kleiner als das Zeitlimit sein."
|
||||
if d.set_owner and not _OWNER_RE.fullmatch(d.set_owner):
|
||||
errors["set_owner"] = "Als „uid:gid“ angeben, z. B. „1000:1000“."
|
||||
try:
|
||||
d.set_mode = validate_mode("set_mode", d.set_mode) or ""
|
||||
except ValidationError as exc:
|
||||
errors["set_mode"] = str(exc)
|
||||
if d.quarantine_path:
|
||||
try:
|
||||
d.quarantine_path = safe_relpath("quarantine_path", d.quarantine_path)
|
||||
except ValidationError as exc:
|
||||
errors["quarantine_path"] = str(exc)
|
||||
return d, errors
|
||||
|
||||
|
||||
def save_rule(s: Session, d: RuleInput, rule_id: int | None = None) -> tuple[Rule | None, Errors]:
|
||||
dup = s.scalar(select(Rule.id).where(Rule.name == d.name))
|
||||
if dup is not None and dup != rule_id:
|
||||
return None, {"name": "Name ist bereits vergeben."}
|
||||
r = s.get(Rule, rule_id) if rule_id is not None else None
|
||||
if rule_id is not None and r is None:
|
||||
raise AdminError("Regel existiert nicht.")
|
||||
if r is None:
|
||||
r = Rule()
|
||||
s.add(r)
|
||||
for f in (
|
||||
"name",
|
||||
"enabled",
|
||||
"src_mount_id",
|
||||
"src_subpath",
|
||||
"dst_mount_id",
|
||||
"dst_path",
|
||||
"pattern",
|
||||
"pattern_type",
|
||||
"recursive",
|
||||
"interval_seconds",
|
||||
"stability_seconds",
|
||||
"min_age_seconds",
|
||||
"max_size_mb",
|
||||
"op_timeout_seconds",
|
||||
"action",
|
||||
"on_collision",
|
||||
"dry_run",
|
||||
"quarantine_after",
|
||||
):
|
||||
setattr(r, f, getattr(d, f))
|
||||
r.set_owner = d.set_owner or None
|
||||
r.set_mode = d.set_mode or None
|
||||
r.quarantine_path = d.quarantine_path or None
|
||||
s.flush()
|
||||
return r, {}
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Verdrahtung aller Bausteine – einmal pro Prozess.
|
||||
|
||||
Start: Datenbank migrieren → Schlüssel laden → Anbindungen mit ``auto_mount``
|
||||
wiederherstellen (Ergebnis protokollieren) → erst dann Scheduler starten.
|
||||
Ende: Scheduler stoppen, Mounts **nicht** aushängen (andere könnten sie nutzen).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from netmover.config import Settings
|
||||
from netmover.crypto import SecretBox, load_or_create_key
|
||||
from netmover.db import make_engine, make_session_factory, upgrade
|
||||
from netmover.health import Health, check_health
|
||||
from netmover.logging_setup import event, get_logger
|
||||
from netmover.models import utcnow
|
||||
from netmover.mounting.backend import MountBackend
|
||||
from netmover.mounting.fake import FakeMountBackend
|
||||
from netmover.mounting.linux import LinuxMountBackend
|
||||
from netmover.mounting.manager import MountManager, RestoreResult
|
||||
from netmover.mover.runner import RuleRunner
|
||||
from netmover.notify import Notifier
|
||||
from netmover.runs import RunService
|
||||
from netmover.scheduler import RuleScheduler
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
def make_backend(settings: Settings) -> MountBackend:
|
||||
if settings.mount_backend == "fake":
|
||||
return FakeMountBackend(settings.data_dir / "fake")
|
||||
return LinuxMountBackend(settings.run_dir)
|
||||
|
||||
|
||||
class NetMover:
|
||||
def __init__(
|
||||
self,
|
||||
settings: Settings,
|
||||
backend: MountBackend | None = None,
|
||||
notifier: Notifier | None = None,
|
||||
) -> None:
|
||||
self.settings = settings
|
||||
settings.data_dir.mkdir(parents=True, exist_ok=True)
|
||||
url = settings.effective_database_url
|
||||
upgrade(url)
|
||||
self.db_engine = make_engine(url)
|
||||
self.session_factory = make_session_factory(self.db_engine)
|
||||
self.secrets = SecretBox(load_or_create_key(settings.secret_key_path))
|
||||
self.backend = backend or make_backend(settings)
|
||||
self.notifier = notifier or Notifier(self.session_factory)
|
||||
self.mounts = MountManager(
|
||||
self.session_factory,
|
||||
self.backend,
|
||||
self.secrets,
|
||||
settings.mount_base,
|
||||
notifier=self.notifier.mount_event,
|
||||
)
|
||||
self.runner = RuleRunner(self.session_factory, self.mounts)
|
||||
self.runs = RunService(self.session_factory, self.runner, self.notifier)
|
||||
self.scheduler = RuleScheduler(
|
||||
self.session_factory, self.runs, retry_mounts=self.mounts.retry_pending
|
||||
)
|
||||
self.started_at: datetime = utcnow()
|
||||
self.restore_results: list[RestoreResult] = []
|
||||
|
||||
def start(self) -> None:
|
||||
"""Muss im laufenden Event-Loop aufgerufen werden (AsyncIOScheduler)."""
|
||||
self.started_at = utcnow()
|
||||
self.restore_results = self.mounts.restore_all()
|
||||
ok = sum(r.ok for r in self.restore_results)
|
||||
event(log, "startup.mounts_restored", ok=ok, failed=len(self.restore_results) - ok)
|
||||
self.scheduler.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
self.scheduler.shutdown()
|
||||
self.db_engine.dispose()
|
||||
|
||||
def health(self) -> Health:
|
||||
return check_health(
|
||||
self.session_factory,
|
||||
scheduler_running=self.scheduler.running,
|
||||
last_heartbeat=self.scheduler.last_heartbeat,
|
||||
service_started=self.started_at,
|
||||
)
|
||||
+37
-4
@@ -1,7 +1,5 @@
|
||||
"""Kommandozeile: ``netmover migrate`` / ``netmover version``.
|
||||
|
||||
``netmover serve`` folgt mit dem Web-Teil (Schritt 5).
|
||||
"""
|
||||
"""Kommandozeile: ``netmover serve`` / ``netmover migrate`` / ``netmover version``
|
||||
/ ``netmover reset-password``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -22,6 +20,11 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
sub.add_parser("version", help="Version ausgeben")
|
||||
sub.add_parser("migrate", help="Datenbank auf den neuesten Stand bringen")
|
||||
sub.add_parser("serve", help="Dienst mit Weboberfläche starten")
|
||||
sub.add_parser(
|
||||
"reset-password",
|
||||
help="Administratorpasswort löschen – beim nächsten Aufruf startet die Einrichtung",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
settings = get_settings()
|
||||
@@ -35,6 +38,36 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
upgrade(settings.effective_database_url)
|
||||
event(log, "db.migrated", url=settings.effective_database_url)
|
||||
return 0
|
||||
if args.cmd == "serve":
|
||||
import uvicorn
|
||||
|
||||
from netmover.web.app import create_app
|
||||
|
||||
uvicorn.run(
|
||||
create_app(settings),
|
||||
host=settings.host,
|
||||
port=settings.port,
|
||||
log_config=None, # unser strukturiertes Logging
|
||||
proxy_headers=True,
|
||||
forwarded_allow_ips=settings.forwarded_allow_ips,
|
||||
server_header=False,
|
||||
)
|
||||
return 0
|
||||
if args.cmd == "reset-password":
|
||||
from netmover.db import make_engine, make_session_factory
|
||||
from netmover.models import Setting
|
||||
from netmover.web.auth import PASSWORD_KEY
|
||||
|
||||
upgrade(settings.effective_database_url)
|
||||
engine = make_engine(settings.effective_database_url)
|
||||
with make_session_factory(engine)() as s:
|
||||
row = s.get(Setting, PASSWORD_KEY)
|
||||
if row is not None:
|
||||
s.delete(row)
|
||||
s.commit()
|
||||
engine.dispose()
|
||||
print("Passwort gelöscht. Dienst neu starten; der Einrichtungscode steht im Log.")
|
||||
return 0
|
||||
return 2
|
||||
|
||||
|
||||
|
||||
@@ -25,6 +25,12 @@ class Settings(BaseSettings):
|
||||
host: str = "127.0.0.1"
|
||||
port: int = Field(default=8080, ge=1, le=65535)
|
||||
|
||||
# Hinter einem Reverse Proxy mit TLS auf true setzen (Cookie nur per HTTPS).
|
||||
cookie_secure: bool = False
|
||||
# IPs, deren X-Forwarded-For/-Proto übernommen wird (Reverse Proxy).
|
||||
forwarded_allow_ips: str = "127.0.0.1"
|
||||
session_hours: int = Field(default=12, ge=1, le=24 * 30)
|
||||
|
||||
log_level: Literal["DEBUG", "INFO", "WARNING", "ERROR"] = "INFO"
|
||||
log_format: Literal["logfmt", "json"] = "logfmt"
|
||||
|
||||
@@ -37,6 +43,10 @@ class Settings(BaseSettings):
|
||||
def db_path(self) -> Path:
|
||||
return self.data_dir / "netmover.db"
|
||||
|
||||
@property
|
||||
def session_key_path(self) -> Path:
|
||||
return self.data_dir / "session.key"
|
||||
|
||||
@property
|
||||
def secret_key_path(self) -> Path:
|
||||
return self.data_dir / "secret.key"
|
||||
|
||||
@@ -33,6 +33,35 @@ def make_engine(url: str) -> Engine:
|
||||
return engine
|
||||
|
||||
|
||||
def make_migration_engine(url: str) -> Engine:
|
||||
"""Verbindung nur für Migrationen.
|
||||
|
||||
* Fremdschlüssel AUS: SQLite baut Tabellen bei Änderungen neu auf (Alembic
|
||||
„batch mode“) und löscht dabei die alte Tabelle. Mit eingeschalteter Prüfung
|
||||
scheitert das, sobald andere Tabellen darauf verweisen (Praxisbefund beim
|
||||
Update auf 0.2.0). Die Integrität prüft ``PRAGMA foreign_key_check`` danach.
|
||||
* Echte Transaktion: pysqlite eigene Transaktionssteuerung abnehmen und selbst
|
||||
``BEGIN`` senden. So wird eine abgebrochene Migration vollständig
|
||||
zurückgerollt, statt einen halben Zustand zu hinterlassen.
|
||||
"""
|
||||
engine = create_engine(url, connect_args={"check_same_thread": False})
|
||||
if url.startswith("sqlite"):
|
||||
|
||||
def on_connect(dbapi_conn: Any, _record: Any) -> None:
|
||||
dbapi_conn.isolation_level = None
|
||||
cur = dbapi_conn.cursor()
|
||||
cur.execute("PRAGMA foreign_keys=OFF") # außerhalb einer Transaktion setzen
|
||||
cur.execute("PRAGMA busy_timeout=5000")
|
||||
cur.close()
|
||||
|
||||
def on_begin(conn: Any) -> None:
|
||||
conn.exec_driver_sql("BEGIN IMMEDIATE")
|
||||
|
||||
event.listen(engine, "connect", on_connect)
|
||||
event.listen(engine, "begin", on_begin)
|
||||
return engine
|
||||
|
||||
|
||||
def make_session_factory(engine: Engine) -> sessionmaker[Session]:
|
||||
return sessionmaker(bind=engine, expire_on_commit=False)
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Gesundheitszustand für ``/healthz`` und die Statusseite (Fallstrick 9).
|
||||
|
||||
Ein Werkzeug, das nichts mehr tut, darf nicht wie eines aussehen, das nichts zu
|
||||
tun hat. Deshalb:
|
||||
|
||||
* der Scheduler muss leben (Herzschlag nicht älter als 3 Minuten),
|
||||
* keine aktive Anbindung im Fehlerzustand,
|
||||
* keine aktive Regel, deren letzter erfolgreicher Lauf länger als das Dreifache
|
||||
ihres Intervalls zurückliegt. Bezugspunkt ist der späteste von letztem Erfolg,
|
||||
Anlage der Regel und Dienststart – sonst wäre direkt nach einem Neustart alles rot.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.db import session_scope
|
||||
from netmover.models import Mount, MountState, Rule, utcnow
|
||||
|
||||
STALE_FACTOR = 3
|
||||
HEARTBEAT_MAX_AGE = timedelta(minutes=3)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Health:
|
||||
ok: bool = True
|
||||
problems: list[str] = field(default_factory=list)
|
||||
stale_rules: set[int] = field(default_factory=set)
|
||||
|
||||
def fail(self, problem: str) -> None:
|
||||
self.ok = False
|
||||
self.problems.append(problem)
|
||||
|
||||
|
||||
def rule_is_stale(rule: Rule, service_started: datetime, now: datetime) -> bool:
|
||||
if not rule.enabled:
|
||||
return False
|
||||
reference = max(
|
||||
d for d in (rule.last_success_at, rule.created_at, service_started) if d is not None
|
||||
)
|
||||
return now - reference > timedelta(seconds=STALE_FACTOR * rule.interval_seconds)
|
||||
|
||||
|
||||
def check_health(
|
||||
session_factory: sessionmaker[Session],
|
||||
*,
|
||||
scheduler_running: bool,
|
||||
last_heartbeat: datetime | None,
|
||||
service_started: datetime,
|
||||
now: datetime | None = None,
|
||||
) -> Health:
|
||||
now = now or utcnow()
|
||||
health = Health()
|
||||
if not scheduler_running:
|
||||
health.fail("Scheduler läuft nicht")
|
||||
elif last_heartbeat is None or now - last_heartbeat > HEARTBEAT_MAX_AGE:
|
||||
health.fail("Scheduler reagiert nicht (kein Herzschlag)")
|
||||
with session_scope(session_factory) as s:
|
||||
for m in s.scalars(select(Mount).where(Mount.enabled.is_(True))):
|
||||
if m.state is MountState.ERROR:
|
||||
health.fail(f"Anbindung „{m.name}“ im Fehlerzustand: {m.last_error or ''}".strip())
|
||||
for r in s.scalars(select(Rule).where(Rule.enabled.is_(True))):
|
||||
if rule_is_stale(r, service_started, now):
|
||||
health.stale_rules.add(r.id)
|
||||
ago = (
|
||||
"noch nie"
|
||||
if r.last_success_at is None
|
||||
else f"seit {r.last_success_at:%Y-%m-%d %H:%M} UTC"
|
||||
)
|
||||
health.fail(f"Regel „{r.name}“: kein erfolgreicher Lauf {ago}")
|
||||
return health
|
||||
@@ -0,0 +1,197 @@
|
||||
"""Schlüsselverwaltung: gespeicherte SSH-Schlüssel für SFTP-Anbindungen.
|
||||
|
||||
* Erzeugen (Ed25519) oder Importieren (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH-
|
||||
oder PEM-Format, optional mit Passphrase). Eine Passphrase wird beim Import
|
||||
einmal benutzt; gespeichert wird der Schlüssel ohne Passphrase, aber mit
|
||||
Fernet verschlüsselt – wie alle Geheimnisse.
|
||||
* Der private Schlüssel verlässt NetMover nie: Die Oberfläche zeigt nur den
|
||||
öffentlichen Schlüssel und den Fingerabdruck.
|
||||
* Ein Schlüssel kann von mehreren Anbindungen genutzt werden; den Benutzernamen
|
||||
legt jede Anbindung selbst fest. Löschen geht nur, solange ihn keine nutzt.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import ec, ed25519, rsa
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from netmover.admin import AdminError
|
||||
from netmover.crypto import SecretBox
|
||||
from netmover.models import Credential, CredentialKind, Mount
|
||||
|
||||
MAX_KEY_BYTES = 16 * 1024
|
||||
MIN_RSA_BITS = 2048
|
||||
SFTP_ONLY_PREFIX = 'restrict,command="/usr/lib/openssh/sftp-server" '
|
||||
|
||||
|
||||
@dataclass
|
||||
class KeyInfo:
|
||||
id: int
|
||||
name: str
|
||||
key_type: str
|
||||
fingerprint: str
|
||||
public_key: str
|
||||
created_at: datetime
|
||||
used_by: list[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def authorized_keys_line(self) -> str:
|
||||
"""Zeile für ~/.ssh/authorized_keys, auf SFTP beschränkt."""
|
||||
return SFTP_ONLY_PREFIX + self.public_key
|
||||
|
||||
|
||||
def fingerprint(public_key: str) -> str:
|
||||
"""SHA256-Fingerabdruck wie ``ssh-keygen -lf``."""
|
||||
blob = base64.b64decode(public_key.split()[1])
|
||||
return "SHA256:" + base64.b64encode(hashlib.sha256(blob).digest()).decode().rstrip("=")
|
||||
|
||||
|
||||
def _validate_name(s: Session, name: str, exclude_id: int | None = None) -> str:
|
||||
name = name.strip()
|
||||
if not name:
|
||||
raise AdminError("Name fehlt.")
|
||||
if len(name) > 100:
|
||||
raise AdminError("Name höchstens 100 Zeichen.")
|
||||
if name.startswith("mount:"):
|
||||
raise AdminError("Namen mit „mount:“ sind für NetMover reserviert.")
|
||||
dup = s.scalar(select(Credential.id).where(Credential.name == name))
|
||||
if dup is not None and dup != exclude_id:
|
||||
raise AdminError(f"Ein Schlüssel „{name}“ existiert bereits.")
|
||||
return name
|
||||
|
||||
|
||||
def _comment(name: str) -> str:
|
||||
return "netmover:" + "".join(c if c.isalnum() or c in "-_." else "-" for c in name)
|
||||
|
||||
|
||||
PrivateKey = ed25519.Ed25519PrivateKey | ec.EllipticCurvePrivateKey | rsa.RSAPrivateKey
|
||||
|
||||
|
||||
def _store(s: Session, secrets: SecretBox, name: str, key: PrivateKey) -> Credential:
|
||||
private = key.private_bytes(
|
||||
serialization.Encoding.PEM,
|
||||
serialization.PrivateFormat.OpenSSH,
|
||||
serialization.NoEncryption(),
|
||||
).decode()
|
||||
public = (
|
||||
key.public_key()
|
||||
.public_bytes(serialization.Encoding.OpenSSH, serialization.PublicFormat.OpenSSH)
|
||||
.decode()
|
||||
)
|
||||
cred = Credential(
|
||||
name=name,
|
||||
kind=CredentialKind.SSH_KEY,
|
||||
username="",
|
||||
secret_enc=secrets.encrypt(private),
|
||||
public_key=f"{public} {_comment(name)}",
|
||||
managed=True,
|
||||
)
|
||||
s.add(cred)
|
||||
s.flush()
|
||||
return cred
|
||||
|
||||
|
||||
def generate_key(s: Session, secrets: SecretBox, name: str) -> Credential:
|
||||
return _store(s, secrets, _validate_name(s, name), ed25519.Ed25519PrivateKey.generate())
|
||||
|
||||
|
||||
def import_key(
|
||||
s: Session, secrets: SecretBox, name: str, private_key: str, passphrase: str = ""
|
||||
) -> Credential:
|
||||
name = _validate_name(s, name)
|
||||
data = private_key.strip().encode()
|
||||
if not data:
|
||||
raise AdminError("Privaten Schlüssel einfügen.")
|
||||
if len(data) > MAX_KEY_BYTES:
|
||||
raise AdminError("Schlüssel ist zu groß.")
|
||||
if b"PUBLIC KEY" in data or data.startswith((b"ssh-", b"ecdsa-")):
|
||||
raise AdminError("Das ist ein öffentlicher Schlüssel – benötigt wird der private.")
|
||||
password = passphrase.encode() if passphrase else None
|
||||
loaders = (
|
||||
[serialization.load_ssh_private_key]
|
||||
if b"OPENSSH PRIVATE KEY" in data
|
||||
else [serialization.load_pem_private_key]
|
||||
)
|
||||
try:
|
||||
key = loaders[0](data, password)
|
||||
except TypeError:
|
||||
raise AdminError(
|
||||
"Der Schlüssel ist mit einer Passphrase geschützt – bitte Passphrase angeben."
|
||||
if password is None
|
||||
else "Der Schlüssel hat keine Passphrase – Feld bitte leer lassen."
|
||||
) from None
|
||||
except ValueError as exc:
|
||||
msg = str(exc).lower()
|
||||
if password is not None or "password" in msg or "decrypt" in msg:
|
||||
# Bei OpenSSH-Schlüsseln meldet cryptography eine falsche Passphrase
|
||||
# als „Corrupt data“ – beides lässt sich nicht unterscheiden.
|
||||
raise AdminError("Passphrase falsch (oder Schlüssel beschädigt).") from None
|
||||
raise AdminError(
|
||||
"Kein lesbarer privater Schlüssel (erwartet OpenSSH- oder PEM-Format)."
|
||||
) from None
|
||||
except Exception: # unbekannte Formate/Algorithmen sauber melden
|
||||
raise AdminError("Schlüsselformat wird nicht unterstützt.") from None
|
||||
if not isinstance(
|
||||
key, ed25519.Ed25519PrivateKey | ec.EllipticCurvePrivateKey | rsa.RSAPrivateKey
|
||||
):
|
||||
raise AdminError("Unterstützt werden Ed25519, ECDSA und RSA.")
|
||||
if isinstance(key, rsa.RSAPrivateKey) and key.key_size < MIN_RSA_BITS:
|
||||
raise AdminError(f"RSA-Schlüssel mit {key.key_size} Bit ist zu schwach (mind. 2048).")
|
||||
return _store(s, secrets, name, key)
|
||||
|
||||
|
||||
def _users(s: Session, key_id: int) -> list[str]:
|
||||
# Direkt abfragen statt key.mounts: die Beziehung ist nach dem Löschen von
|
||||
# Anbindungen in derselben Session veraltet (passive_deletes).
|
||||
return sorted(s.scalars(select(Mount.name).where(Mount.cred_id == key_id)))
|
||||
|
||||
|
||||
def list_keys(s: Session) -> list[KeyInfo]:
|
||||
keys = s.scalars(
|
||||
select(Credential).where(Credential.managed.is_(True)).order_by(Credential.name)
|
||||
).all()
|
||||
result = []
|
||||
for k in keys:
|
||||
pub = k.public_key or ""
|
||||
result.append(
|
||||
KeyInfo(
|
||||
id=k.id,
|
||||
name=k.name,
|
||||
key_type=pub.split()[0] if pub else "?",
|
||||
fingerprint=fingerprint(pub) if pub else "?",
|
||||
public_key=pub,
|
||||
created_at=k.created_at,
|
||||
used_by=_users(s, k.id),
|
||||
)
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def managed_key(s: Session, key_id: int) -> Credential:
|
||||
key = s.get(Credential, key_id)
|
||||
if key is None or not key.managed or key.kind is not CredentialKind.SSH_KEY:
|
||||
raise AdminError("Schlüssel existiert nicht.")
|
||||
return key
|
||||
|
||||
|
||||
def rename_key(s: Session, key_id: int, name: str) -> None:
|
||||
key = managed_key(s, key_id)
|
||||
key.name = _validate_name(s, name, exclude_id=key_id)
|
||||
|
||||
|
||||
def delete_key(s: Session, key_id: int) -> str:
|
||||
key = managed_key(s, key_id)
|
||||
users = _users(s, key_id)
|
||||
if users:
|
||||
raise AdminError(f"Schlüssel „{key.name}“ wird noch benutzt von: {', '.join(users)}")
|
||||
name = key.name
|
||||
s.delete(key)
|
||||
s.flush()
|
||||
return name
|
||||
@@ -62,6 +62,9 @@ def setup_logging(level: str = "INFO", fmt: str = "logfmt") -> None:
|
||||
lg = logging.getLogger(name)
|
||||
lg.handlers[:] = []
|
||||
lg.propagate = True
|
||||
# APScheduler meldet jeden Job-Start auf INFO – bei 30-s-Intervallen nur Rauschen.
|
||||
if level != "DEBUG":
|
||||
logging.getLogger("apscheduler").setLevel(logging.WARNING)
|
||||
|
||||
|
||||
def get_logger(name: str) -> logging.Logger:
|
||||
|
||||
@@ -7,7 +7,7 @@ from logging.config import fileConfig
|
||||
from alembic import context
|
||||
|
||||
from netmover.config import get_settings
|
||||
from netmover.db import make_engine
|
||||
from netmover.db import make_migration_engine
|
||||
from netmover.models import Base
|
||||
|
||||
config = context.config
|
||||
@@ -33,16 +33,27 @@ def run_migrations_offline() -> None:
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
engine = make_engine(_url())
|
||||
url = _url()
|
||||
engine = make_migration_engine(url)
|
||||
with engine.connect() as connection:
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
render_as_batch=True,
|
||||
compare_type=True,
|
||||
# alles oder nichts: auch SQLite kann DDL in einer Transaktion
|
||||
transactional_ddl=True,
|
||||
)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
if url.startswith("sqlite"):
|
||||
# Fremdschlüssel waren während der Migration aus – vor dem Commit prüfen.
|
||||
bad = connection.exec_driver_sql("PRAGMA foreign_key_check").fetchall()
|
||||
if bad:
|
||||
raise RuntimeError(
|
||||
f"Migration verletzt Fremdschlüssel ({len(bad)} Zeilen), "
|
||||
f"zurückgerollt: {bad[:5]}"
|
||||
)
|
||||
engine.dispose()
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
"""key management
|
||||
|
||||
Revision ID: 0002
|
||||
Revises: 0001
|
||||
Create Date: 2026-10-10
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0002"
|
||||
down_revision: str | None = "0001"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("credentials", schema=None) as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("managed", sa.Boolean(), server_default=sa.text("0"), nullable=False)
|
||||
)
|
||||
|
||||
with op.batch_alter_table("mounts", schema=None) as batch_op:
|
||||
batch_op.add_column(sa.Column("username", sa.String(length=255), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("mounts", schema=None) as batch_op:
|
||||
batch_op.drop_column("username")
|
||||
|
||||
with op.batch_alter_table("credentials", schema=None) as batch_op:
|
||||
batch_op.drop_column("managed")
|
||||
@@ -7,6 +7,9 @@ Abweichungen bzw. Ergänzungen gegenüber dem Mindestumfang aus der Spezifikatio
|
||||
(beim ersten Test festgehaltener Hostschlüssel, danach verbindlich).
|
||||
* ``credentials.kind`` (``password`` | ``ssh_key``) und ``credentials.public_key``.
|
||||
* ``credentials.domain`` – Windows-/AD-Freigaben brauchen oft eine Domäne.
|
||||
* ``credentials.managed`` + ``mounts.username`` (Migration 0002) –
|
||||
Schlüsselverwaltung: gespeicherte SSH-Schlüssel, die mehrere Anbindungen mit
|
||||
jeweils eigenem Benutzernamen nutzen.
|
||||
* ``mounts.state_since`` – für „hängt seit …“ / „gestört seit …“ auf der Statusseite.
|
||||
* ``mounts.write_ok`` / ``mounts.root_squash`` / ``mounts.last_test_*`` – Ergebnis
|
||||
des letzten Mount-Tests; ``root_squash`` entscheidet, ob ``chown`` auf NFS-Zielen
|
||||
@@ -38,6 +41,7 @@ from sqlalchemy import (
|
||||
TypeDecorator,
|
||||
UniqueConstraint,
|
||||
)
|
||||
from sqlalchemy import false as sa_false
|
||||
from sqlalchemy.engine import Dialect
|
||||
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
|
||||
|
||||
@@ -166,6 +170,9 @@ class Credential(Base):
|
||||
domain: Mapped[str | None] = mapped_column(String(255))
|
||||
secret_enc: Mapped[str] = mapped_column(Text) # Passwort bzw. privater Schlüssel (Fernet)
|
||||
public_key: Mapped[str | None] = mapped_column(Text) # nur ssh_key, OpenSSH-Format
|
||||
# True: in der Schlüsselverwaltung gepflegt, von mehreren Anbindungen nutzbar.
|
||||
# False: gehört genau einer Anbindung und wird mit ihr gelöscht.
|
||||
managed: Mapped[bool] = mapped_column(Boolean, default=False, server_default=sa_false())
|
||||
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||
|
||||
# passive_deletes="all": das ORM soll cred_id nicht still auf NULL setzen,
|
||||
@@ -183,6 +190,9 @@ class Mount(Base):
|
||||
server: Mapped[str] = mapped_column(String(255))
|
||||
share: Mapped[str] = mapped_column(String(1024))
|
||||
port: Mapped[int | None] = mapped_column(Integer) # nur sftp
|
||||
# Benutzer für die Anmeldung; überschreibt credentials.username. Nötig, weil ein
|
||||
# gespeicherter SSH-Schlüssel für verschiedene Server/Benutzer gelten kann.
|
||||
username: Mapped[str | None] = mapped_column(String(255))
|
||||
host_key: Mapped[str | None] = mapped_column(Text) # nur sftp, "<typ> <base64>"
|
||||
mountpoint: Mapped[str | None] = mapped_column(String(1024), unique=True) # NULL bei sftp
|
||||
cred_id: Mapped[int | None] = mapped_column(ForeignKey("credentials.id", ondelete="RESTRICT"))
|
||||
|
||||
@@ -8,7 +8,9 @@ Optionszeile – die stünde sonst in der Prozessliste.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import uuid
|
||||
from collections.abc import Callable, Sequence
|
||||
@@ -24,6 +26,28 @@ log = get_logger(__name__)
|
||||
|
||||
Runner = Callable[..., "subprocess.CompletedProcess[str]"]
|
||||
|
||||
|
||||
def run_process_group(argv: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
"""Wie ``subprocess.run``, aber in eigener Prozessgruppe: Bei Zeitüberschreitung
|
||||
wird die ganze Gruppe beendet. ``mount`` startet Hilfsprogramme wie
|
||||
``mount.nfs``; würde nur ``mount`` beendet, liefe ``mount.nfs`` weiter und
|
||||
hängte womöglich Minuten später doch noch ein (auf der Test-VM beobachtet)."""
|
||||
timeout = kwargs.pop("timeout", None)
|
||||
kwargs.pop("check", None)
|
||||
capture = kwargs.pop("capture_output", False)
|
||||
if capture:
|
||||
kwargs["stdout"] = kwargs["stderr"] = subprocess.PIPE
|
||||
with subprocess.Popen(argv, start_new_session=True, **kwargs) as proc:
|
||||
try:
|
||||
out, err = proc.communicate(timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
os.killpg(proc.pid, signal.SIGKILL)
|
||||
proc.communicate()
|
||||
raise
|
||||
return subprocess.CompletedProcess(argv, proc.returncode, out, err)
|
||||
|
||||
|
||||
_ENV = {"LC_ALL": "C", "PATH": "/usr/sbin:/usr/bin:/sbin:/bin"}
|
||||
|
||||
|
||||
@@ -34,6 +58,13 @@ def explain_mount_error(fstype: str, message: str) -> str:
|
||||
if "helper program" in low or "unknown filesystem type" in low:
|
||||
pkg = "cifs-utils" if fstype == "cifs" else "nfs-common"
|
||||
hint = f"Das Paket „{pkg}“ fehlt: apt install {pkg}"
|
||||
elif fstype == "cifs" and ("error(95)" in low or "operation not supported" in low):
|
||||
hint = (
|
||||
"Server und Client finden keine gemeinsame SMB-Version. Am besten am NAS "
|
||||
"SMB2/SMB3 erlauben (Synology: Systemsteuerung → Dateidienste → SMB → "
|
||||
"Erweiterte Einstellungen → Maximales SMB-Protokoll: SMB3). Alternativ "
|
||||
"Zusatzoption vers=2.0 – vers=1.0 nur im Notfall, SMB1 ist unsicher."
|
||||
)
|
||||
elif "permission denied" in low and fstype == "cifs":
|
||||
hint = "Benutzername/Passwort falsch oder der Benutzer hat keine Rechte auf die Freigabe."
|
||||
elif "permission denied" in low or "access denied by server" in low:
|
||||
@@ -60,7 +91,7 @@ class LinuxMountBackend(MountBackend):
|
||||
self,
|
||||
run_dir: Path,
|
||||
mountinfo_path: Path = MOUNTINFO,
|
||||
runner: Runner = subprocess.run,
|
||||
runner: Runner = run_process_group,
|
||||
) -> None:
|
||||
self.run_dir = run_dir
|
||||
self.mountinfo_path = mountinfo_path
|
||||
|
||||
@@ -15,7 +15,9 @@ Fallstricke:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import errno
|
||||
import math
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
@@ -104,6 +106,13 @@ class RestoreResult:
|
||||
message: str
|
||||
|
||||
|
||||
def _redact_output(message: str, secrets: Sequence[str]) -> str:
|
||||
"""Schwärzt nur die Ausgabe von ``mount``, nicht den angehängten Hinweistext
|
||||
(sonst würde z. B. das Passwort „falsch“ auch im Hinweis geschwärzt)."""
|
||||
output, sep, hint = message.partition("\n→ ")
|
||||
return redact(output, secrets) + sep + hint
|
||||
|
||||
|
||||
def build_spec(cfg: MountConfig, mountpoint: Path, cred: CredentialData | None) -> MountSpec:
|
||||
"""Baut den ``mount``-Aufruf. Validiert erneut (Verteidigung in der Tiefe) –
|
||||
auch Daten aus der Datenbank laufen hier noch einmal durch die Prüfung."""
|
||||
@@ -166,6 +175,8 @@ class MountManager:
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
euid: Callable[[], int] = os.geteuid,
|
||||
notifier: Notifier | None = None,
|
||||
retry_interval: float = 60,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
) -> None:
|
||||
self.session_factory = session_factory
|
||||
self.backend = backend
|
||||
@@ -177,6 +188,12 @@ class MountManager:
|
||||
self._sleep = sleep
|
||||
self._euid = euid
|
||||
self._notifier = notifier
|
||||
self.retry_interval = retry_interval
|
||||
self._clock = clock
|
||||
# Mounts, die NetMover nach gescheiterter Wiederherstellung selbst ausgehängt
|
||||
# hat: nächster erlaubter Versuch (monotone Zeit). Nur diese werden
|
||||
# automatisch neu eingehängt – von außen ausgehängte nie (Fallstrick 2).
|
||||
self._retry_at: dict[int, float] = {}
|
||||
self._locks: dict[int, threading.RLock] = {}
|
||||
self._locks_guard = threading.Lock()
|
||||
|
||||
@@ -211,7 +228,7 @@ class MountManager:
|
||||
if m.credential is not None:
|
||||
c = m.credential
|
||||
cred = CredentialData(
|
||||
c.kind, c.username, self.secrets.decrypt(c.secret_enc), c.domain
|
||||
c.kind, m.username or c.username, self.secrets.decrypt(c.secret_enc), c.domain
|
||||
)
|
||||
s.expunge(m)
|
||||
return m, cred
|
||||
@@ -267,6 +284,7 @@ class MountManager:
|
||||
|
||||
def mount(self, mount_id: int) -> None:
|
||||
with self._lock(mount_id):
|
||||
self._retry_at.pop(mount_id, None)
|
||||
m, cred = self._load(mount_id)
|
||||
if not m.type.is_kernel_mount:
|
||||
raise MountError("SFTP-Anbindungen werden nicht eingehängt, nur getestet")
|
||||
@@ -292,6 +310,7 @@ class MountManager:
|
||||
|
||||
def unmount(self, mount_id: int) -> None:
|
||||
with self._lock(mount_id):
|
||||
self._retry_at.pop(mount_id, None)
|
||||
m, _ = self._load(mount_id)
|
||||
if not m.type.is_kernel_mount:
|
||||
self._set_state(mount_id, MountState.UNMOUNTED)
|
||||
@@ -325,6 +344,8 @@ class MountManager:
|
||||
self.mount(mount_id)
|
||||
results.append(RestoreResult(name, True, "eingehängt"))
|
||||
except (MountError, ValidationError, LookupError) as exc:
|
||||
# z. B. Stromausfall: die VM bootet schneller als das NAS
|
||||
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
||||
results.append(RestoreResult(name, False, str(exc)))
|
||||
self._notify("mount.restore_failed", name, str(exc))
|
||||
for r in results:
|
||||
@@ -391,7 +412,7 @@ class MountManager:
|
||||
self.backend.mount(build_spec(cfg, temp, cred), timeout=self.mount_timeout)
|
||||
except (MountError, ValidationError) as exc:
|
||||
self._remove_mountpoint(temp)
|
||||
report.step("mount", False, redact(str(exc), secrets))
|
||||
report.step("mount", False, _redact_output(str(exc), secrets))
|
||||
report.step("read", None)
|
||||
report.step("write", None)
|
||||
return report
|
||||
@@ -486,6 +507,11 @@ class MountManager:
|
||||
if cfg.type is MountType.SFTP and m.host_key is None and report.can_read:
|
||||
m.host_key = report.host_key
|
||||
event(log, "sftp.host_key_pinned", mount=m.name, fp=report.host_key_fingerprint)
|
||||
# SFTP hat keinen Mount-Zustand – ein erfolgreicher Test heißt „erreichbar“.
|
||||
if cfg.type is MountType.SFTP and report.ok and m.state is not MountState.MOUNTED:
|
||||
m.state = MountState.MOUNTED
|
||||
m.state_since = utcnow()
|
||||
m.last_error = None
|
||||
|
||||
# -- Lebenszeichen und Wiederherstellung --------------------------------------
|
||||
|
||||
@@ -501,6 +527,9 @@ class MountManager:
|
||||
return None
|
||||
mp = self.mountpoint_for(m.slug)
|
||||
if not self.backend.is_mounted(mp):
|
||||
if mount_id in self._retry_at:
|
||||
self._retry_mount(mount_id, m, cred, mp)
|
||||
return mp
|
||||
msg = f"„{m.name}“ ist nicht eingehängt – Lauf abgebrochen, nichts angefasst."
|
||||
self._set_state(mount_id, MountState.ERROR, msg)
|
||||
raise MountUnavailable(msg)
|
||||
@@ -568,11 +597,75 @@ class MountManager:
|
||||
f"„{m.name}“ gestört ({reason}); Neu-Einhängen nach "
|
||||
f"{len(self.recover_backoff)} Versuchen gescheitert: {last}"
|
||||
)
|
||||
if m.auto_mount:
|
||||
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
||||
msg += f"\n→ Neuer Versuch automatisch alle {self.retry_interval:g} s."
|
||||
self._set_state(mount_id, MountState.ERROR, msg)
|
||||
self._notify("mount.error", m.name, msg)
|
||||
raise MountUnavailable(msg)
|
||||
|
||||
def usage(self, mount_id: int) -> tuple[int, int] | None:
|
||||
def retry_pending(self) -> list[str]:
|
||||
"""Fällige Neuversuche ausführen (Scheduler-Job, jede Minute) – auch für
|
||||
Anbindungen, die gerade keine Regel benutzt. Liefert die zurückgeholten Namen."""
|
||||
recovered: list[str] = []
|
||||
for mount_id in list(self._retry_at):
|
||||
if self._retry_at.get(mount_id, float("inf")) > self._clock():
|
||||
continue
|
||||
with self._lock(mount_id):
|
||||
if mount_id not in self._retry_at:
|
||||
continue
|
||||
try:
|
||||
m, cred = self._load(mount_id)
|
||||
except LookupError:
|
||||
self._retry_at.pop(mount_id, None)
|
||||
continue
|
||||
if not (m.enabled and m.auto_mount and m.type.is_kernel_mount):
|
||||
self._retry_at.pop(mount_id, None)
|
||||
continue
|
||||
mp = self.mountpoint_for(m.slug)
|
||||
if self.backend.is_mounted(mp):
|
||||
self._retry_at.pop(mount_id, None)
|
||||
continue
|
||||
try:
|
||||
self._retry_mount(mount_id, m, cred, mp)
|
||||
recovered.append(m.name)
|
||||
except MountUnavailable:
|
||||
pass
|
||||
return recovered
|
||||
|
||||
def _retry_mount(self, mount_id: int, m: Mount, cred: CredentialData | None, mp: Path) -> None:
|
||||
"""Ein Versuch, einen von NetMover selbst ausgehängten Mount zurückzuholen –
|
||||
höchstens einmal pro ``retry_interval``, damit kein Lauf alle 10 s hängt."""
|
||||
wait = self._retry_at[mount_id] - self._clock()
|
||||
if wait > 0:
|
||||
raise MountUnavailable(
|
||||
f"„{m.name}“ nicht verfügbar – nächster Versuch, neu einzuhängen, "
|
||||
f"in {math.ceil(wait)} s. Lauf abgebrochen, nichts angefasst."
|
||||
)
|
||||
try:
|
||||
self._prepare_mountpoint(mp)
|
||||
self.backend.mount(
|
||||
build_spec(MountConfig.from_model(m), mp, cred), timeout=self.mount_timeout
|
||||
)
|
||||
self._probe(m.name, mp)
|
||||
except (MountError, OSError, OperationTimeout, ValidationError) as exc:
|
||||
if self.backend.is_mounted(mp):
|
||||
with contextlib.suppress(MountError):
|
||||
self.backend.unmount(mp, lazy=True, timeout=self.mount_timeout)
|
||||
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
||||
msg = (
|
||||
f"„{m.name}“ weiterhin nicht verfügbar: {exc}\n"
|
||||
f"→ Neuer Versuch in {self.retry_interval:g} s."
|
||||
)
|
||||
self._set_state(mount_id, MountState.ERROR, msg)
|
||||
event(log, "mount.retry_failed", level=30, mount=m.name, error=str(exc))
|
||||
raise MountUnavailable(msg) from None
|
||||
del self._retry_at[mount_id]
|
||||
self._set_state(mount_id, MountState.MOUNTED, mounted=True)
|
||||
event(log, "mount.recovered", mount=m.name, attempt="retry")
|
||||
self._notify("mount.recovered", m.name, f"„{m.name}“ ist wieder eingehängt.")
|
||||
|
||||
def usage(self, mount_id: int, timeout: float | None = None) -> tuple[int, int] | None:
|
||||
"""(gesamt, frei) in Bytes – nur bei eingehängten Kernel-Mounts."""
|
||||
m, _ = self._load(mount_id)
|
||||
if not m.type.is_kernel_mount:
|
||||
@@ -581,7 +674,9 @@ class MountManager:
|
||||
if not self.backend.is_mounted(mp):
|
||||
return None
|
||||
try:
|
||||
st = run_with_timeout(lambda: os.statvfs(mp), self.probe_timeout, f"statvfs:{m.name}")
|
||||
st = run_with_timeout(
|
||||
lambda: os.statvfs(mp), timeout or self.probe_timeout, f"statvfs:{m.name}"
|
||||
)
|
||||
except (OSError, OperationTimeout):
|
||||
return None
|
||||
return st.f_blocks * st.f_frsize, st.f_bavail * st.f_frsize
|
||||
|
||||
@@ -7,7 +7,8 @@ verlieren. Ablauf für jede Datei:
|
||||
Größe prüfen,
|
||||
2. Metadaten setzen (mtime, Modus, Eigentümer – Fehler sind nur Hinweise),
|
||||
3. prüfen, dass sich die Quelle seit der Stabilitätsmessung nicht verändert hat,
|
||||
4. atomar auf den Endnamen umbenennen – ohne zu überschreiben (``link`` bzw.
|
||||
4. atomar auf den Endnamen umbenennen – ohne zu überschreiben
|
||||
(``renameat2(RENAME_NOREPLACE)``, sonst ``link`` + ``unlink`` bzw.
|
||||
SFTP-``rename``), außer „überschreiben“ ist ausdrücklich eingestellt,
|
||||
5. erst danach löscht der Aufrufer die Quelle.
|
||||
|
||||
@@ -33,6 +34,7 @@ from typing import Literal
|
||||
|
||||
from netmover.models import CollisionPolicy
|
||||
from netmover.mounting.sftp import SftpSession
|
||||
from netmover.mover.fsops import rename_noreplace
|
||||
from netmover.mover.naming import collision_candidates
|
||||
from netmover.timeouts import CancelToken
|
||||
|
||||
@@ -283,6 +285,9 @@ class LocalDestination(Destination):
|
||||
|
||||
def _link_noreplace(self, tmp: str, final: str) -> None:
|
||||
src, dst = self._p(tmp), self._p(final)
|
||||
# Bevorzugt ein echtes rename ohne Überschreiben (Watcher sehen IN_MOVED_TO).
|
||||
if rename_noreplace(src, dst):
|
||||
return
|
||||
try:
|
||||
os.link(src, dst)
|
||||
except FileExistsError:
|
||||
@@ -340,6 +345,9 @@ class LocalDestination(Destination):
|
||||
raise CollisionSkip
|
||||
continue
|
||||
try:
|
||||
if rename_noreplace(src, self._p(cand)):
|
||||
final = cand
|
||||
break
|
||||
os.link(src, self._p(cand))
|
||||
except FileExistsError:
|
||||
if policy is CollisionPolicy.SKIP:
|
||||
|
||||
@@ -13,6 +13,7 @@ Reihenfolge pro Lauf:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import errno
|
||||
import os
|
||||
import stat
|
||||
@@ -39,7 +40,7 @@ from netmover.mover.destinations import (
|
||||
)
|
||||
from netmover.mover.ledger import FailureInfo, Ledger
|
||||
from netmover.mover.naming import collision_candidates
|
||||
from netmover.mover.scan import Candidate, FileState, Matcher, current_state, scan
|
||||
from netmover.mover.scan import BUSY, Candidate, FileState, Matcher, current_state, scan
|
||||
from netmover.timeouts import Cancelled, CancelToken
|
||||
|
||||
log = get_logger(__name__)
|
||||
@@ -222,26 +223,32 @@ class Engine:
|
||||
else:
|
||||
pending.append(c)
|
||||
|
||||
# Stabilität (Fallstrick 1): zweite Messung nach dem Stabilitätsfenster.
|
||||
# Stabilität (Fallstrick 1): zwei frische Messungen (öffnen + fstat, nicht
|
||||
# der Verzeichnis-Cache) im Abstand des Stabilitätsfensters.
|
||||
first: dict[str, FileState | None] = {c.relpath: current_state(c.path) for c in pending}
|
||||
if pending and rule.stability_seconds > 0:
|
||||
self._sleep(rule.stability_seconds)
|
||||
token.check()
|
||||
stable: list[Candidate] = []
|
||||
for c in pending:
|
||||
st = current_state(c.path)
|
||||
if st is None:
|
||||
before, st = first[c.relpath], current_state(c.path)
|
||||
if before is None or st is None:
|
||||
continue # inzwischen verschwunden – nichts zu tun
|
||||
if st != c.state:
|
||||
if st != before or st == BUSY:
|
||||
result.add(
|
||||
ItemResult(
|
||||
c.relpath,
|
||||
ItemStatus.SKIPPED,
|
||||
size=st.size,
|
||||
message="wird noch geschrieben (Größe/Zeit geändert)",
|
||||
size=max(st.size, 0),
|
||||
message="wird noch geschrieben (Größe/Zeit geändert)"
|
||||
if st != BUSY
|
||||
else "Datei ist gerade in Benutzung",
|
||||
)
|
||||
)
|
||||
continue
|
||||
stable.append(c)
|
||||
# ab hier gilt der frisch gemessene Zustand (für Prüfung vor dem Umbenennen
|
||||
# und für die Kopier-Buchführung)
|
||||
stable.append(dataclasses.replace(c, state=st))
|
||||
|
||||
failures = self.ledger.failures(rule.id)
|
||||
for c in stable:
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Atomares Umbenennen ohne Überschreiben.
|
||||
|
||||
``renameat2(..., RENAME_NOREPLACE)`` ist ein echtes ``rename`` (Watcher sehen
|
||||
``IN_MOVED_TO`` wie bei ``os.replace``), scheitert aber mit ``EEXIST``, statt
|
||||
eine vorhandene Datei zu ersetzen. ext4, xfs, btrfs und tmpfs können das; NFS
|
||||
und CIFS meist nicht – dann liefert :func:`rename_noreplace` ``False`` und der
|
||||
Aufrufer weicht auf ``link()`` + ``unlink()`` aus.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ctypes
|
||||
import ctypes.util
|
||||
import errno
|
||||
import os
|
||||
from collections.abc import Callable
|
||||
from functools import cache
|
||||
|
||||
AT_FDCWD = -100
|
||||
RENAME_NOREPLACE = 1
|
||||
_UNSUPPORTED = frozenset({errno.EINVAL, errno.ENOSYS, errno.ENOTSUP, errno.EOPNOTSUPP, errno.EPERM})
|
||||
|
||||
|
||||
@cache
|
||||
def _renameat2() -> Callable[..., int] | None:
|
||||
name = ctypes.util.find_library("c")
|
||||
if name is None:
|
||||
return None
|
||||
libc = ctypes.CDLL(name, use_errno=True)
|
||||
fn = getattr(libc, "renameat2", None)
|
||||
if fn is None:
|
||||
return None
|
||||
fn.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_uint]
|
||||
fn.restype = ctypes.c_int
|
||||
renameat2: Callable[..., int] = fn
|
||||
return renameat2
|
||||
|
||||
|
||||
def rename_noreplace(src: str | os.PathLike[str], dst: str | os.PathLike[str]) -> bool:
|
||||
"""True = umbenannt. False = vom Dateisystem nicht unterstützt (nichts passiert).
|
||||
``FileExistsError``, wenn ``dst`` existiert; andere Fehler als ``OSError``."""
|
||||
fn = _renameat2()
|
||||
if fn is None:
|
||||
return False
|
||||
res = fn(AT_FDCWD, os.fsencode(src), AT_FDCWD, os.fsencode(dst), RENAME_NOREPLACE)
|
||||
if res == 0:
|
||||
return True
|
||||
err = ctypes.get_errno()
|
||||
if err == errno.EEXIST:
|
||||
raise FileExistsError(err, os.strerror(err), os.fsdecode(dst))
|
||||
if err in _UNSUPPORTED:
|
||||
return False
|
||||
raise OSError(err, os.strerror(err), os.fsdecode(src))
|
||||
@@ -10,6 +10,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
import fnmatch
|
||||
import os
|
||||
import re
|
||||
@@ -100,11 +101,31 @@ def scan(
|
||||
yield from walk(root, "")
|
||||
|
||||
|
||||
# Datei ist geöffnet, aber gerade nicht lesbar (z. B. exklusiv vom Schreiber
|
||||
# gesperrt): gilt als „noch nicht fertig“ – gleicht nie einem echten Zustand.
|
||||
BUSY = FileState(-1, -1)
|
||||
|
||||
|
||||
def current_state(path: Path) -> FileState | None:
|
||||
"""Aktuelle Größe und mtime – **frisch vom Server**.
|
||||
|
||||
``stat()`` beantworten CIFS- und NFS-Clients aus ihrem Attribut-Cache; im
|
||||
Praxistest lieferte CIFS so sekundenlang eine veraltete Größe, während die
|
||||
Datei noch wuchs. Erst ``open()`` erzwingt frische Attribute (NFS:
|
||||
Close-to-open-Konsistenz, SMB: Antwort auf CREATE). Darum öffnen + ``fstat``.
|
||||
"""
|
||||
try:
|
||||
st = os.stat(path, follow_symlinks=False)
|
||||
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
except OSError as exc:
|
||||
if exc.errno in (errno.ELOOP, errno.ENOTDIR):
|
||||
return None # Symlink oder kein Pfad mehr
|
||||
return BUSY
|
||||
try:
|
||||
st = os.fstat(fd)
|
||||
finally:
|
||||
os.close(fd)
|
||||
if not stat.S_ISREG(st.st_mode):
|
||||
return None
|
||||
return FileState(st.st_size, st.st_mtime_ns)
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
"""Benachrichtigung per generischem Webhook.
|
||||
|
||||
Der Body ist eine Vorlage mit ``$platzhaltern`` (``string.Template``):
|
||||
``$event``, ``$title``, ``$message``, ``$rule``, ``$time``, ``$host``.
|
||||
Ist der Content-Type JSON, werden die Werte JSON-maskiert eingesetzt – so
|
||||
funktionieren auch eigene JSON-Formate (z. B. Gotify ``{"title": "$title", …}``).
|
||||
Für ntfy reicht ``text/plain`` mit ``$message`` und der Topic-URL.
|
||||
|
||||
Versand im Hintergrund-Thread, damit ein langsamer Empfänger nie einen Lauf bremst.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import socket
|
||||
import threading
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from string import Template
|
||||
|
||||
import httpx
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.logging_setup import event, get_logger
|
||||
from netmover.settings_store import AppSettings, NotifyMode, load_settings
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Notification:
|
||||
event: str # z. B. "rule.failed", "rule.recovered", "mount.error"
|
||||
title: str
|
||||
message: str
|
||||
rule: str = ""
|
||||
|
||||
|
||||
def render(template: str, content_type: str, n: Notification, now: datetime | None = None) -> str:
|
||||
values = {
|
||||
"event": n.event,
|
||||
"title": n.title,
|
||||
"message": n.message,
|
||||
"rule": n.rule,
|
||||
"time": (now or datetime.now(UTC)).isoformat(timespec="seconds"),
|
||||
"host": socket.gethostname(),
|
||||
}
|
||||
if "json" in content_type.lower():
|
||||
values = {k: json.dumps(v, ensure_ascii=False)[1:-1] for k, v in values.items()}
|
||||
return Template(template).safe_substitute(values)
|
||||
|
||||
|
||||
Sender = Callable[[str, str, str], None] # (url, content_type, body)
|
||||
|
||||
|
||||
def http_sender(url: str, content_type: str, body: str) -> None:
|
||||
resp = httpx.post(
|
||||
url, content=body.encode(), headers={"Content-Type": content_type}, timeout=10
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
|
||||
class Notifier:
|
||||
def __init__(
|
||||
self,
|
||||
session_factory: sessionmaker[Session],
|
||||
*,
|
||||
sender: Sender = http_sender,
|
||||
background: bool = True,
|
||||
) -> None:
|
||||
self.session_factory = session_factory
|
||||
self._sender = sender
|
||||
self._background = background
|
||||
|
||||
def settings(self) -> AppSettings:
|
||||
return load_settings(self.session_factory)
|
||||
|
||||
def send(self, n: Notification, settings: AppSettings | None = None) -> None:
|
||||
cfg = settings or self.settings()
|
||||
if cfg.notify_mode is NotifyMode.OFF or not cfg.webhook_url:
|
||||
return
|
||||
body = render(cfg.webhook_template, cfg.webhook_content_type, n)
|
||||
|
||||
def deliver() -> None:
|
||||
try:
|
||||
self._sender(cfg.webhook_url, cfg.webhook_content_type, body)
|
||||
event(log, "notify.sent", kind=n.event, rule=n.rule)
|
||||
except Exception as exc: # Benachrichtigung darf nie den Dienst stören
|
||||
# URL nicht protokollieren – sie enthält oft ein Token.
|
||||
event(log, "notify.failed", level=30, kind=n.event, error=type(exc).__name__)
|
||||
|
||||
if self._background:
|
||||
threading.Thread(target=deliver, name="nm-notify", daemon=True).start()
|
||||
else:
|
||||
deliver()
|
||||
|
||||
def rule_result(self, rule: str, failed: bool, consecutive: int, message: str) -> None:
|
||||
"""Entscheidet nach Modus, ob ein Regelergebnis gemeldet wird."""
|
||||
cfg = self.settings()
|
||||
if cfg.notify_mode is NotifyMode.OFF:
|
||||
return
|
||||
threshold = 1 if cfg.notify_mode is NotifyMode.EVERY else max(1, cfg.notify_after)
|
||||
if failed:
|
||||
if cfg.notify_mode is NotifyMode.EVERY or consecutive == threshold:
|
||||
suffix = f" ({consecutive}× in Folge)" if consecutive > 1 else ""
|
||||
self.send(
|
||||
Notification(
|
||||
"rule.failed",
|
||||
f"NetMover: Regel „{rule}“ fehlgeschlagen{suffix}",
|
||||
message,
|
||||
rule,
|
||||
),
|
||||
cfg,
|
||||
)
|
||||
elif cfg.notify_recovery and consecutive >= threshold:
|
||||
self.send(
|
||||
Notification(
|
||||
"rule.recovered",
|
||||
f"NetMover: Regel „{rule}“ läuft wieder",
|
||||
f"Nach {consecutive} fehlgeschlagenen Läufen wieder erfolgreich.",
|
||||
rule,
|
||||
),
|
||||
cfg,
|
||||
)
|
||||
|
||||
def mount_event(self, kind: str, mount: str, message: str) -> None:
|
||||
"""Passt zur ``Notifier``-Signatur des :class:`MountManager`."""
|
||||
title = (
|
||||
f"NetMover: Anbindung „{mount}“ wieder verfügbar"
|
||||
if kind == "mount.recovered"
|
||||
else f"NetMover: Anbindung „{mount}“ gestört"
|
||||
)
|
||||
self.send(Notification(kind, title, message))
|
||||
@@ -0,0 +1,171 @@
|
||||
"""Läufe ausführen und protokollieren.
|
||||
|
||||
Pro Lauf eine Zeile in ``runs`` (auch für übersprungene Termine, Fallstrick 8),
|
||||
die Einzeldateien in ``run_items``. Danach die Regelzähler:
|
||||
|
||||
* ``last_run_at`` – jeder Lauf außer übersprungenen,
|
||||
* ``last_success_at`` – Status ``ok`` oder ``partial`` (das Werkzeug hat gearbeitet),
|
||||
* ``consecutive_failures`` – ``error`` und ``partial`` zählen hoch, ``ok`` setzt zurück.
|
||||
|
||||
Ein manuell gestarteter Probelauf verändert keine Zähler.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.db import session_scope
|
||||
from netmover.logging_setup import event, get_logger
|
||||
from netmover.models import FileFailure, Rule, Run, RunItem, RunStatus, RunTrigger, utcnow
|
||||
from netmover.mover.engine import RunResult
|
||||
from netmover.mover.runner import RuleRunner
|
||||
from netmover.notify import Notifier
|
||||
from netmover.settings_store import load_settings
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
MAX_ITEMS_PER_RUN = 2000
|
||||
SUCCESS = frozenset({RunStatus.OK, RunStatus.PARTIAL})
|
||||
FAILURE = frozenset({RunStatus.ERROR, RunStatus.PARTIAL})
|
||||
|
||||
|
||||
class RunService:
|
||||
def __init__(
|
||||
self,
|
||||
session_factory: sessionmaker[Session],
|
||||
runner: RuleRunner,
|
||||
notifier: Notifier | None = None,
|
||||
) -> None:
|
||||
self.session_factory = session_factory
|
||||
self.runner = runner
|
||||
self.notifier = notifier
|
||||
|
||||
def execute(
|
||||
self,
|
||||
rule_id: int,
|
||||
trigger: RunTrigger = RunTrigger.SCHEDULE,
|
||||
dry_run: bool | None = None,
|
||||
) -> int | None:
|
||||
"""Führt die Regel aus, protokolliert den Lauf und liefert die Run-ID
|
||||
(None, wenn die Regel inzwischen gelöscht wurde)."""
|
||||
started = utcnow()
|
||||
result = self.runner.run(rule_id, dry_run=dry_run)
|
||||
finished = utcnow()
|
||||
count_it = not (trigger is RunTrigger.MANUAL and dry_run)
|
||||
return self.record(rule_id, trigger, result, started, finished, update_counters=count_it)
|
||||
|
||||
def record(
|
||||
self,
|
||||
rule_id: int,
|
||||
trigger: RunTrigger,
|
||||
result: RunResult,
|
||||
started: datetime,
|
||||
finished: datetime,
|
||||
*,
|
||||
update_counters: bool = True,
|
||||
) -> int | None:
|
||||
notify: tuple[str, bool, int, str] | None = None
|
||||
with session_scope(self.session_factory) as s:
|
||||
rule = s.get(Rule, rule_id)
|
||||
if rule is None:
|
||||
return None
|
||||
message = result.message
|
||||
items = result.items
|
||||
if len(items) > MAX_ITEMS_PER_RUN:
|
||||
extra = len(items) - MAX_ITEMS_PER_RUN
|
||||
items = items[:MAX_ITEMS_PER_RUN]
|
||||
note = f"{extra} weitere Dateien nicht einzeln protokolliert"
|
||||
message = f"{message}; {note}" if message else note
|
||||
run = Run(
|
||||
rule_id=rule_id,
|
||||
started_at=started,
|
||||
finished_at=finished,
|
||||
status=result.status,
|
||||
trigger=trigger,
|
||||
dry_run=result.dry_run,
|
||||
seen=result.seen,
|
||||
moved=result.moved,
|
||||
skipped=result.skipped,
|
||||
failed=result.failed,
|
||||
message=message,
|
||||
)
|
||||
s.add(run)
|
||||
s.flush()
|
||||
s.add_all(
|
||||
RunItem(
|
||||
run_id=run.id,
|
||||
src=i.src,
|
||||
dst=i.dst,
|
||||
status=i.status,
|
||||
size=i.size,
|
||||
message=i.message,
|
||||
)
|
||||
for i in items
|
||||
)
|
||||
if update_counters and result.status is not RunStatus.SKIPPED:
|
||||
previous = rule.consecutive_failures
|
||||
rule.last_run_at = finished
|
||||
if result.status in SUCCESS:
|
||||
rule.last_success_at = finished
|
||||
if result.status in FAILURE:
|
||||
rule.consecutive_failures += 1
|
||||
notify = (rule.name, True, rule.consecutive_failures, self._summary(result))
|
||||
else:
|
||||
rule.consecutive_failures = 0
|
||||
if previous:
|
||||
notify = (rule.name, False, previous, "")
|
||||
run_id = run.id
|
||||
rule_name = rule.name
|
||||
|
||||
level = 20 if result.status in (RunStatus.OK, RunStatus.SKIPPED) else 30
|
||||
if result.status is RunStatus.OK and not result.items:
|
||||
level = 10 # leere Läufe alle 30 s nur auf DEBUG
|
||||
event(
|
||||
log,
|
||||
"run.finished",
|
||||
level=level,
|
||||
rule=rule_name,
|
||||
run=run_id,
|
||||
status=result.status.value,
|
||||
trigger=trigger.value,
|
||||
dry_run=result.dry_run,
|
||||
seen=result.seen,
|
||||
moved=result.moved,
|
||||
skipped=result.skipped,
|
||||
failed=result.failed,
|
||||
took_ms=int((finished - started).total_seconds() * 1000),
|
||||
message=result.message or "",
|
||||
)
|
||||
if notify is not None and self.notifier is not None:
|
||||
self.notifier.rule_result(*notify)
|
||||
return run_id
|
||||
|
||||
@staticmethod
|
||||
def _summary(result: RunResult) -> str:
|
||||
if result.status is RunStatus.ERROR:
|
||||
return result.message or "Lauf fehlgeschlagen"
|
||||
failed = [i for i in result.items if i.status.value == "failed"][:5]
|
||||
lines = [f"{result.failed} Datei(en) fehlgeschlagen:"]
|
||||
lines += [f"• {i.src}: {i.message}" for i in failed]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def cleanup(session_factory: sessionmaker[Session], now: datetime | None = None) -> dict[str, int]:
|
||||
"""Protokolle älter als die Aufbewahrungsfrist löschen (Items per CASCADE),
|
||||
dazu Fehlerzähler, die seitdem nicht mehr angefasst wurden."""
|
||||
days = load_settings(session_factory).retention_days
|
||||
cutoff = (now or utcnow()) - timedelta(days=days)
|
||||
with session_scope(session_factory) as s:
|
||||
old_ids = list(s.scalars(select(Run.id).where(Run.started_at < cutoff)))
|
||||
for i in range(0, len(old_ids), 500):
|
||||
chunk = old_ids[i : i + 500]
|
||||
s.execute(delete(RunItem).where(RunItem.run_id.in_(chunk)))
|
||||
s.execute(delete(Run).where(Run.id.in_(chunk)))
|
||||
failures = s.execute(delete(FileFailure).where(FileFailure.last_attempt_at < cutoff))
|
||||
removed_failures = getattr(failures, "rowcount", 0) or 0
|
||||
stats = {"runs": len(old_ids), "file_failures": removed_failures}
|
||||
event(log, "cleanup.done", retention_days=days, **stats)
|
||||
return stats
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Zeitsteuerung mit APScheduler (``AsyncIOScheduler``).
|
||||
|
||||
* Ein Job pro aktiver Regel im jeweiligen Intervall. Erster Start mit kleinem
|
||||
zufälligem Versatz, dazu Jitter pro Termin – damit nicht alle Regeln
|
||||
gleichzeitig starten (Fallstrick 8).
|
||||
* Die Läufe selbst sind blockierend und laufen in einem eigenen Thread-Pool,
|
||||
nie im Event-Loop. Überlappung regelt der Lock im :class:`RuleRunner`, der den
|
||||
übersprungenen Termin auch protokolliert – deshalb ``max_instances`` > 1.
|
||||
* Herzschlag-Job für ``/healthz`` und stündliches Aufräumen der Protokolle.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import random
|
||||
from collections.abc import Callable
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.db import session_scope
|
||||
from netmover.logging_setup import event, get_logger
|
||||
from netmover.models import Rule, RunTrigger, utcnow
|
||||
from netmover.runs import RunService, cleanup
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
MAX_START_OFFSET = 10.0
|
||||
HEARTBEAT_SECONDS = 30
|
||||
CLEANUP_SECONDS = 3600
|
||||
RETRY_SECONDS = 60
|
||||
|
||||
|
||||
def _job_id(rule_id: int) -> str:
|
||||
return f"rule:{rule_id}"
|
||||
|
||||
|
||||
class RuleScheduler:
|
||||
def __init__(
|
||||
self,
|
||||
session_factory: sessionmaker[Session],
|
||||
service: RunService,
|
||||
*,
|
||||
retry_mounts: Callable[[], object] | None = None,
|
||||
rng: random.Random | None = None,
|
||||
workers: int = 8,
|
||||
) -> None:
|
||||
self.session_factory = session_factory
|
||||
self.service = service
|
||||
self._retry_mounts = retry_mounts
|
||||
self._rng = rng or random.Random() # noqa: S311 – nur Startversatz
|
||||
self._executor = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="nm-run")
|
||||
self.scheduler = AsyncIOScheduler(
|
||||
timezone=UTC,
|
||||
job_defaults={"coalesce": True, "max_instances": 3},
|
||||
)
|
||||
self._intervals: dict[int, int] = {}
|
||||
self.last_heartbeat: datetime | None = None
|
||||
|
||||
# -- Lebenszyklus ---------------------------------------------------------------
|
||||
|
||||
def start(self) -> None:
|
||||
self.scheduler.start()
|
||||
self.heartbeat()
|
||||
self.scheduler.add_job(
|
||||
self.heartbeat, IntervalTrigger(seconds=HEARTBEAT_SECONDS), id="heartbeat"
|
||||
)
|
||||
self.scheduler.add_job(
|
||||
self._cleanup,
|
||||
IntervalTrigger(seconds=CLEANUP_SECONDS, start_date=utcnow() + timedelta(minutes=1)),
|
||||
id="cleanup",
|
||||
)
|
||||
if self._retry_mounts is not None:
|
||||
self.scheduler.add_job(
|
||||
self._retry, IntervalTrigger(seconds=RETRY_SECONDS), id="mount-retry"
|
||||
)
|
||||
self.sync()
|
||||
event(log, "scheduler.started", rules=len(self._intervals))
|
||||
|
||||
def shutdown(self) -> None:
|
||||
if self.scheduler.running:
|
||||
self.scheduler.shutdown(wait=False)
|
||||
# Laufende Worker nicht abwarten: abgebrochene Übertragungen hinterlassen
|
||||
# höchstens .part-Dateien, die Quelle bleibt unangetastet.
|
||||
self._executor.shutdown(wait=False, cancel_futures=True)
|
||||
event(log, "scheduler.stopped")
|
||||
|
||||
@property
|
||||
def running(self) -> bool:
|
||||
return bool(self.scheduler.running)
|
||||
|
||||
def heartbeat(self) -> None:
|
||||
self.last_heartbeat = utcnow()
|
||||
|
||||
# -- Regeln -------------------------------------------------------------------------
|
||||
|
||||
def sync(self) -> None:
|
||||
"""Jobs an die Regeln in der Datenbank angleichen (nach jeder Änderung)."""
|
||||
with session_scope(self.session_factory) as s:
|
||||
wanted = {
|
||||
rid: interval
|
||||
for rid, interval in s.execute(
|
||||
select(Rule.id, Rule.interval_seconds).where(Rule.enabled.is_(True))
|
||||
).all()
|
||||
}
|
||||
for rid in list(self._intervals):
|
||||
if rid not in wanted:
|
||||
if self.scheduler.get_job(_job_id(rid)):
|
||||
self.scheduler.remove_job(_job_id(rid))
|
||||
del self._intervals[rid]
|
||||
for rid, interval in wanted.items():
|
||||
if self._intervals.get(rid) == interval and self.scheduler.get_job(_job_id(rid)):
|
||||
continue
|
||||
offset = self._rng.uniform(0, min(float(interval), MAX_START_OFFSET))
|
||||
trigger = IntervalTrigger(
|
||||
seconds=interval,
|
||||
start_date=utcnow() + timedelta(seconds=offset),
|
||||
jitter=max(1, min(5, interval // 10)),
|
||||
)
|
||||
self.scheduler.add_job(
|
||||
self._run_rule,
|
||||
trigger,
|
||||
args=[rid],
|
||||
id=_job_id(rid),
|
||||
name=f"Regel {rid}",
|
||||
replace_existing=True,
|
||||
misfire_grace_time=interval,
|
||||
)
|
||||
self._intervals[rid] = interval
|
||||
|
||||
def next_run(self, rule_id: int) -> datetime | None:
|
||||
job = self.scheduler.get_job(_job_id(rule_id))
|
||||
return job.next_run_time if job is not None else None
|
||||
|
||||
async def _run_rule(self, rule_id: int) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(
|
||||
self._executor, self.service.execute, rule_id, RunTrigger.SCHEDULE
|
||||
)
|
||||
|
||||
async def run_now(self, rule_id: int, dry_run: bool | None = None) -> int | None:
|
||||
"""„Jetzt ausführen“ / „Probelauf“ aus der Oberfläche."""
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(
|
||||
self._executor, self.service.execute, rule_id, RunTrigger.MANUAL, dry_run
|
||||
)
|
||||
|
||||
async def _retry(self) -> None:
|
||||
if self._retry_mounts is not None:
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(self._executor, self._retry_mounts)
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(self._executor, cleanup, self.session_factory)
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Laufzeit-Einstellungen in der Tabelle ``settings`` (in der Oberfläche änderbar)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import asdict, dataclass, fields
|
||||
from enum import StrEnum
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.db import session_scope
|
||||
from netmover.models import Setting
|
||||
|
||||
|
||||
class NotifyMode(StrEnum):
|
||||
OFF = "off"
|
||||
EVERY = "every" # bei jedem fehlgeschlagenen Lauf
|
||||
AFTER_N = "after_n" # erst nach N fehlgeschlagenen Läufen in Folge
|
||||
|
||||
|
||||
DEFAULT_TEMPLATE = (
|
||||
'{"event": "$event", "title": "$title", "message": "$message", '
|
||||
'"rule": "$rule", "time": "$time", "host": "$host"}'
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AppSettings:
|
||||
retention_days: int = 30
|
||||
webhook_url: str = ""
|
||||
notify_mode: NotifyMode = NotifyMode.OFF
|
||||
notify_after: int = 3
|
||||
notify_recovery: bool = True
|
||||
webhook_template: str = DEFAULT_TEMPLATE
|
||||
webhook_content_type: str = "application/json"
|
||||
|
||||
|
||||
_PREFIX = "app."
|
||||
|
||||
|
||||
def load_settings(factory: sessionmaker[Session]) -> AppSettings:
|
||||
with session_scope(factory) as s:
|
||||
rows = {
|
||||
r.key[len(_PREFIX) :]: r.value
|
||||
for r in s.scalars(select(Setting).where(Setting.key.like(f"{_PREFIX}%")))
|
||||
}
|
||||
result = AppSettings()
|
||||
for f in fields(AppSettings):
|
||||
if f.name in rows:
|
||||
raw = json.loads(rows[f.name])
|
||||
setattr(result, f.name, NotifyMode(raw) if f.name == "notify_mode" else raw)
|
||||
return result
|
||||
|
||||
|
||||
def save_settings(factory: sessionmaker[Session], settings: AppSettings) -> None:
|
||||
with session_scope(factory) as s:
|
||||
for key, value in asdict(settings).items():
|
||||
row = s.get(Setting, _PREFIX + key)
|
||||
encoded = json.dumps(value.value if isinstance(value, StrEnum) else value)
|
||||
if row is None:
|
||||
s.add(Setting(key=_PREFIX + key, value=encoded))
|
||||
else:
|
||||
row.value = encoded
|
||||
|
||||
|
||||
def get_value(factory: sessionmaker[Session], key: str) -> str | None:
|
||||
with session_scope(factory) as s:
|
||||
row = s.get(Setting, key)
|
||||
return row.value if row else None
|
||||
|
||||
|
||||
def set_value(factory: sessionmaker[Session], key: str, value: str) -> None:
|
||||
with session_scope(factory) as s:
|
||||
row = s.get(Setting, key)
|
||||
if row is None:
|
||||
s.add(Setting(key=key, value=value))
|
||||
else:
|
||||
row.value = value
|
||||
@@ -0,0 +1,196 @@
|
||||
"""Export/Import der Konfiguration als JSON – ohne Zugangsdaten.
|
||||
|
||||
Ausgenommen sind Passwörter, private Schlüssel, Benutzernamen und die
|
||||
Webhook-URL (enthält meist ein Token). Nach einem Import müssen Zugangsdaten
|
||||
neu eingegeben werden. Anbindungen und Regeln werden über ihren Namen
|
||||
zugeordnet; vorhandene Namen werden übersprungen, nie überschrieben.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from netmover import __version__
|
||||
from netmover.models import (
|
||||
CollisionPolicy,
|
||||
Mount,
|
||||
MountState,
|
||||
MountType,
|
||||
PatternType,
|
||||
Rule,
|
||||
RuleAction,
|
||||
)
|
||||
from netmover.mounting.manager import MountManager
|
||||
from netmover.settings_store import AppSettings, NotifyMode
|
||||
from netmover.validation import (
|
||||
ValidationError,
|
||||
parse_extra_options,
|
||||
safe_relpath,
|
||||
slugify,
|
||||
validate_server,
|
||||
validate_share,
|
||||
)
|
||||
|
||||
FORMAT_VERSION = 1
|
||||
|
||||
MOUNT_FIELDS = (
|
||||
"name",
|
||||
"type",
|
||||
"server",
|
||||
"share",
|
||||
"port",
|
||||
"host_key",
|
||||
"uid",
|
||||
"gid",
|
||||
"file_mode",
|
||||
"dir_mode",
|
||||
"extra_options",
|
||||
"auto_mount",
|
||||
"enabled",
|
||||
)
|
||||
RULE_FIELDS = (
|
||||
"name",
|
||||
"enabled",
|
||||
"src_subpath",
|
||||
"dst_path",
|
||||
"pattern",
|
||||
"pattern_type",
|
||||
"recursive",
|
||||
"interval_seconds",
|
||||
"stability_seconds",
|
||||
"min_age_seconds",
|
||||
"max_size_mb",
|
||||
"op_timeout_seconds",
|
||||
"action",
|
||||
"on_collision",
|
||||
"set_owner",
|
||||
"set_mode",
|
||||
"dry_run",
|
||||
"quarantine_path",
|
||||
"quarantine_after",
|
||||
)
|
||||
SETTING_FIELDS = (
|
||||
"retention_days",
|
||||
"notify_mode",
|
||||
"notify_after",
|
||||
"notify_recovery",
|
||||
"webhook_template",
|
||||
"webhook_content_type",
|
||||
)
|
||||
|
||||
|
||||
def _plain(value: Any) -> Any:
|
||||
return value.value if hasattr(value, "value") else value
|
||||
|
||||
|
||||
def export_config(s: Session, settings: AppSettings) -> dict[str, Any]:
|
||||
mounts = s.scalars(select(Mount).order_by(Mount.name)).all()
|
||||
names = {m.id: m.name for m in mounts}
|
||||
return {
|
||||
"format": FORMAT_VERSION,
|
||||
"netmover": __version__,
|
||||
"mounts": [{f: _plain(getattr(m, f)) for f in MOUNT_FIELDS} for m in mounts],
|
||||
"rules": [
|
||||
{
|
||||
**{f: _plain(getattr(r, f)) for f in RULE_FIELDS},
|
||||
"src_mount": names[r.src_mount_id],
|
||||
"dst_mount": names.get(r.dst_mount_id) if r.dst_mount_id else None,
|
||||
}
|
||||
for r in s.scalars(select(Rule).order_by(Rule.name))
|
||||
],
|
||||
"settings": {f: _plain(getattr(settings, f)) for f in SETTING_FIELDS},
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ImportReport:
|
||||
created: list[str] = field(default_factory=list)
|
||||
skipped: list[str] = field(default_factory=list)
|
||||
errors: list[str] = field(default_factory=list)
|
||||
settings: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
def import_config(s: Session, mounts: MountManager, data: dict[str, Any]) -> ImportReport:
|
||||
report = ImportReport()
|
||||
if not isinstance(data, dict) or data.get("format") != FORMAT_VERSION:
|
||||
report.errors.append("Unbekanntes Format – erwartet wird ein NetMover-Export (format 1).")
|
||||
return report
|
||||
|
||||
for raw in data.get("mounts") or []:
|
||||
name = str(raw.get("name", ""))
|
||||
if s.scalar(select(Mount.id).where(Mount.name == name)) is not None:
|
||||
report.skipped.append(f"Anbindung „{name}“ (existiert)")
|
||||
continue
|
||||
try:
|
||||
mtype = MountType(raw["type"])
|
||||
slug = slugify(name)
|
||||
if s.scalar(select(Mount.id).where(Mount.slug == slug)) is not None:
|
||||
raise ValidationError("name", "Slug bereits vergeben")
|
||||
m = Mount(
|
||||
name=name,
|
||||
slug=slug,
|
||||
type=mtype,
|
||||
server=validate_server(str(raw["server"])),
|
||||
share=validate_share(mtype, str(raw["share"])),
|
||||
port=raw.get("port"),
|
||||
host_key=raw.get("host_key"),
|
||||
uid=raw.get("uid"),
|
||||
gid=raw.get("gid"),
|
||||
file_mode=raw.get("file_mode"),
|
||||
dir_mode=raw.get("dir_mode"),
|
||||
extra_options=str(raw.get("extra_options") or ""),
|
||||
# ohne Zugangsdaten nicht automatisch einhängen
|
||||
auto_mount=bool(raw.get("auto_mount")) and mtype is not MountType.CIFS,
|
||||
enabled=bool(raw.get("enabled", True)),
|
||||
state=MountState.UNMOUNTED,
|
||||
mountpoint=str(mounts.mountpoint_for(slug)) if mtype.is_kernel_mount else None,
|
||||
)
|
||||
parse_extra_options(mtype, m.extra_options)
|
||||
s.add(m)
|
||||
s.flush()
|
||||
report.created.append(f"Anbindung „{name}“")
|
||||
except (KeyError, ValueError, ValidationError) as exc:
|
||||
report.errors.append(f"Anbindung „{name}“: {exc}")
|
||||
|
||||
ids = {name: mid for mid, name in s.execute(select(Mount.id, Mount.name)).all()}
|
||||
for raw in data.get("rules") or []:
|
||||
name = str(raw.get("name", ""))
|
||||
if s.scalar(select(Rule.id).where(Rule.name == name)) is not None:
|
||||
report.skipped.append(f"Regel „{name}“ (existiert)")
|
||||
continue
|
||||
try:
|
||||
src = ids[raw["src_mount"]]
|
||||
dst = ids[raw["dst_mount"]] if raw.get("dst_mount") else None
|
||||
r = Rule(src_mount_id=src, dst_mount_id=dst)
|
||||
for f in RULE_FIELDS:
|
||||
if f in raw:
|
||||
setattr(r, f, raw[f])
|
||||
r.pattern_type = PatternType(r.pattern_type)
|
||||
r.action = RuleAction(r.action)
|
||||
r.on_collision = CollisionPolicy(r.on_collision)
|
||||
r.src_subpath = safe_relpath("src_subpath", r.src_subpath or "")
|
||||
if int(r.interval_seconds) < 10:
|
||||
raise ValueError("Intervall < 10 s")
|
||||
r.enabled = False # erst nach Eingabe der Zugangsdaten aktivieren
|
||||
s.add(r)
|
||||
s.flush()
|
||||
report.created.append(f"Regel „{name}“ (deaktiviert)")
|
||||
except KeyError as exc:
|
||||
report.errors.append(f"Regel „{name}“: Anbindung {exc} fehlt")
|
||||
except (ValueError, ValidationError) as exc:
|
||||
report.errors.append(f"Regel „{name}“: {exc}")
|
||||
|
||||
if isinstance(data.get("settings"), dict):
|
||||
report.settings = {f: data["settings"][f] for f in SETTING_FIELDS if f in data["settings"]}
|
||||
return report
|
||||
|
||||
|
||||
def apply_imported_settings(current: AppSettings, values: dict[str, Any]) -> AppSettings:
|
||||
"""Nach dem Commit des Imports aufrufen (eigene Transaktion)."""
|
||||
for key, value in values.items():
|
||||
setattr(current, key, NotifyMode(value) if key == "notify_mode" else value)
|
||||
return current
|
||||
@@ -0,0 +1 @@
|
||||
"""Weboberfläche: FastAPI + Jinja2 + HTMX, serverseitig gerendert."""
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,163 @@
|
||||
"""Anmeldung, Einrichtung beim ersten Start, CSRF und Sperre bei Fehlversuchen.
|
||||
|
||||
* Ein einziges Administratorkonto; Passwort-Hash (Argon2) in ``settings``.
|
||||
* Erster Start: Ohne Passwort führt jede Seite zur Einrichtung. Dafür wird ein
|
||||
einmaliger Einrichtungscode ins Log geschrieben – so kann nicht irgendwer im
|
||||
LAN, der die Seite zuerst öffnet, das Passwort festlegen.
|
||||
* Session-Cookie ``HttpOnly``, ``SameSite=Lax`` (Starlette-SessionMiddleware,
|
||||
signiert). CSRF-Token pro Session, Pflicht bei jedem POST – als Formularfeld
|
||||
``csrf_token`` oder Header ``X-CSRF-Token`` (HTMX).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hmac
|
||||
import secrets
|
||||
import threading
|
||||
import time
|
||||
from collections import defaultdict, deque
|
||||
from pathlib import Path
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerificationError
|
||||
from fastapi import HTTPException, Request
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.settings_store import get_value, set_value
|
||||
|
||||
PASSWORD_KEY = "auth.admin_password_hash" # noqa: S105 – Schlüsselname
|
||||
MIN_PASSWORD_LENGTH = 10
|
||||
SAFE_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
|
||||
|
||||
_hasher = PasswordHasher()
|
||||
|
||||
|
||||
class LoginRequired(Exception):
|
||||
def __init__(self, setup: bool = False) -> None:
|
||||
self.setup = setup
|
||||
|
||||
|
||||
def load_or_create_session_key(path: Path) -> str:
|
||||
import os
|
||||
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
except FileExistsError:
|
||||
return path.read_text().strip()
|
||||
key = secrets.token_urlsafe(48)
|
||||
with os.fdopen(fd, "w") as fh:
|
||||
fh.write(key + "\n")
|
||||
return key
|
||||
|
||||
|
||||
def password_is_set(factory: sessionmaker[Session]) -> bool:
|
||||
return bool(get_value(factory, PASSWORD_KEY))
|
||||
|
||||
|
||||
def set_password(factory: sessionmaker[Session], password: str) -> None:
|
||||
set_value(factory, PASSWORD_KEY, _hasher.hash(password))
|
||||
|
||||
|
||||
def verify_password(factory: sessionmaker[Session], password: str) -> bool:
|
||||
stored = get_value(factory, PASSWORD_KEY)
|
||||
if not stored:
|
||||
return False
|
||||
try:
|
||||
_hasher.verify(stored, password)
|
||||
except (VerificationError, InvalidHashError):
|
||||
return False
|
||||
if _hasher.check_needs_rehash(stored):
|
||||
set_value(factory, PASSWORD_KEY, _hasher.hash(password))
|
||||
return True
|
||||
|
||||
|
||||
def password_problem(password: str, confirm: str) -> str | None:
|
||||
if len(password) < MIN_PASSWORD_LENGTH:
|
||||
return f"Mindestens {MIN_PASSWORD_LENGTH} Zeichen."
|
||||
if password != confirm:
|
||||
return "Die Passwörter stimmen nicht überein."
|
||||
return None
|
||||
|
||||
|
||||
# -- Session / CSRF -------------------------------------------------------------------
|
||||
|
||||
|
||||
def csrf_token(request: Request) -> str:
|
||||
token = request.session.get("csrf")
|
||||
if not isinstance(token, str):
|
||||
token = secrets.token_urlsafe(32)
|
||||
request.session["csrf"] = token
|
||||
return token
|
||||
|
||||
|
||||
def login_session(request: Request) -> None:
|
||||
"""Neue Session nach erfolgreicher Anmeldung (Schutz vor Session-Fixation)."""
|
||||
request.session.clear()
|
||||
request.session["user"] = "admin"
|
||||
request.session["csrf"] = secrets.token_urlsafe(32)
|
||||
request.session["since"] = int(time.time())
|
||||
|
||||
|
||||
def is_logged_in(request: Request) -> bool:
|
||||
return request.session.get("user") == "admin"
|
||||
|
||||
|
||||
async def check_csrf(request: Request) -> None:
|
||||
if request.method in SAFE_METHODS:
|
||||
return
|
||||
expected = request.session.get("csrf")
|
||||
sent = request.headers.get("x-csrf-token")
|
||||
if sent is None:
|
||||
form = await request.form()
|
||||
value = form.get("csrf_token")
|
||||
sent = value if isinstance(value, str) else None
|
||||
if not expected or not sent or not hmac.compare_digest(str(expected), sent):
|
||||
raise HTTPException(status_code=403, detail="CSRF-Token fehlt oder ist ungültig.")
|
||||
|
||||
|
||||
# -- Sperre bei Fehlversuchen --------------------------------------------------------------
|
||||
|
||||
|
||||
class LoginThrottle:
|
||||
"""Nach ``limit`` Fehlversuchen innerhalb von ``window`` Sekunden wird die
|
||||
Client-IP für ``lockout`` Sekunden gesperrt."""
|
||||
|
||||
def __init__(self, limit: int = 5, window: float = 900, lockout: float = 300) -> None:
|
||||
self.limit = limit
|
||||
self.window = window
|
||||
self.lockout = lockout
|
||||
self._fails: dict[str, deque[float]] = defaultdict(deque)
|
||||
self._locked: dict[str, float] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def locked_for(self, client: str) -> float:
|
||||
with self._lock:
|
||||
until = self._locked.get(client, 0)
|
||||
return max(0.0, until - time.monotonic())
|
||||
|
||||
def failure(self, client: str) -> None:
|
||||
now = time.monotonic()
|
||||
with self._lock:
|
||||
q = self._fails[client]
|
||||
q.append(now)
|
||||
while q and now - q[0] > self.window:
|
||||
q.popleft()
|
||||
if len(q) >= self.limit:
|
||||
self._locked[client] = now + self.lockout
|
||||
q.clear()
|
||||
|
||||
def success(self, client: str) -> None:
|
||||
with self._lock:
|
||||
self._fails.pop(client, None)
|
||||
self._locked.pop(client, None)
|
||||
|
||||
|
||||
class SetupCode:
|
||||
"""Einmaliger Code für die Ersteinrichtung (steht im Log)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.code = "-".join(secrets.token_hex(2) for _ in range(3))
|
||||
|
||||
def matches(self, value: str) -> bool:
|
||||
return hmac.compare_digest(self.code, value.strip().lower())
|
||||
@@ -0,0 +1,111 @@
|
||||
/* NetMover – handgeschrieben, ohne Build-Schritt. Hell/Dunkel nach Systemeinstellung. */
|
||||
:root {
|
||||
--bg: #f6f7f9; --card: #fff; --fg: #1d2330; --muted: #5d6676; --line: #dfe3ea;
|
||||
--accent: #2f6fde; --accent-fg: #fff; --ok: #1e8e3e; --warn: #b26a00; --err: #c62828;
|
||||
--ok-bg: #e6f4ea; --warn-bg: #fff4e0; --err-bg: #fdecea; --info-bg: #e8f0fe;
|
||||
--radius: 8px; --gap: 12px;
|
||||
font: 15px/1.45 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #12151b; --card: #1b2028; --fg: #e6e9ef; --muted: #9aa3b2; --line: #2c333e;
|
||||
--accent: #6b9bff; --accent-fg: #0b1020; --ok: #57c27a; --warn: #f0a63a; --err: #ff6b6b;
|
||||
--ok-bg: #16301f; --warn-bg: #36280f; --err-bg: #3a1717; --info-bg: #182540;
|
||||
}
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; background: var(--bg); color: var(--fg); }
|
||||
a { color: var(--accent); }
|
||||
code, pre, .mono { font-family: ui-monospace, "SFMono-Regular", Menlo, Consolas, monospace; font-size: .9em; }
|
||||
pre { white-space: pre-wrap; word-break: break-word; margin: 0; }
|
||||
|
||||
header.top { background: var(--card); border-bottom: 1px solid var(--line); }
|
||||
header.top .inner { max-width: 1200px; margin: 0 auto; display: flex; align-items: center;
|
||||
gap: var(--gap); padding: 8px 16px; flex-wrap: wrap; }
|
||||
header.top .brand { font-weight: 700; text-decoration: none; color: var(--fg); margin-right: 8px; }
|
||||
nav.main { display: flex; gap: 4px; flex-wrap: wrap; flex: 1; }
|
||||
nav.main a { padding: 6px 10px; border-radius: var(--radius); text-decoration: none; color: var(--fg); }
|
||||
nav.main a.active, nav.main a:hover { background: var(--bg); }
|
||||
main { max-width: 1200px; margin: 0 auto; padding: 16px; }
|
||||
h1 { font-size: 1.4rem; margin: 4px 0 16px; }
|
||||
h2 { font-size: 1.1rem; margin: 24px 0 8px; }
|
||||
.muted { color: var(--muted); }
|
||||
.small { font-size: .85em; }
|
||||
|
||||
.card { background: var(--card); border: 1px solid var(--line); border-radius: var(--radius); padding: 14px; }
|
||||
.grid { display: grid; gap: var(--gap); grid-template-columns: repeat(auto-fill, minmax(260px, 1fr)); }
|
||||
.row { display: flex; gap: 8px; align-items: center; flex-wrap: wrap; }
|
||||
.spread { justify-content: space-between; }
|
||||
.stack > * + * { margin-top: 8px; }
|
||||
|
||||
.badge { display: inline-block; padding: 1px 8px; border-radius: 99px; font-size: .8em; font-weight: 600;
|
||||
background: var(--bg); border: 1px solid var(--line); white-space: nowrap; }
|
||||
.badge.ok { background: var(--ok-bg); color: var(--ok); border-color: transparent; }
|
||||
.badge.warn { background: var(--warn-bg); color: var(--warn); border-color: transparent; }
|
||||
.badge.err { background: var(--err-bg); color: var(--err); border-color: transparent; }
|
||||
|
||||
.msg { padding: 8px 12px; border-radius: var(--radius); margin: 0 0 10px; }
|
||||
.msg.ok { background: var(--ok-bg); } .msg.error { background: var(--err-bg); }
|
||||
.msg.info { background: var(--info-bg); } .msg.warn { background: var(--warn-bg); }
|
||||
|
||||
|
||||
table { width: 100%; border-collapse: collapse; background: var(--card); border: 1px solid var(--line);
|
||||
border-radius: var(--radius); overflow: hidden; }
|
||||
th, td { text-align: left; padding: 7px 10px; border-bottom: 1px solid var(--line); vertical-align: top; }
|
||||
th { font-size: .8em; text-transform: uppercase; letter-spacing: .03em; color: var(--muted); background: var(--bg); }
|
||||
tr:last-child td { border-bottom: 0; }
|
||||
tr.warn td { background: var(--warn-bg); }
|
||||
tr.err td { background: var(--err-bg); }
|
||||
.table-wrap { overflow-x: auto; }
|
||||
td.num, th.num { text-align: right; font-variant-numeric: tabular-nums; }
|
||||
|
||||
button, .btn { font: inherit; padding: 6px 12px; border-radius: var(--radius); border: 1px solid var(--line);
|
||||
background: var(--card); color: var(--fg); cursor: pointer; text-decoration: none; display: inline-block; }
|
||||
button.primary, .btn.primary { background: var(--accent); color: var(--accent-fg); border-color: var(--accent); }
|
||||
button.danger { color: var(--err); }
|
||||
button.small, .btn.small { padding: 3px 9px; font-size: .85em; }
|
||||
button:disabled { opacity: .5; }
|
||||
form.inline { display: inline; margin: 0; }
|
||||
|
||||
form.form { max-width: 760px; }
|
||||
fieldset { border: 1px solid var(--line); border-radius: var(--radius); margin: 0 0 14px; padding: 10px 14px 14px; background: var(--card); }
|
||||
legend { font-weight: 600; padding: 0 6px; }
|
||||
.fields { display: grid; gap: 10px 14px; grid-template-columns: repeat(auto-fill, minmax(200px, 1fr)); }
|
||||
.field { display: flex; flex-direction: column; gap: 3px; }
|
||||
.field.wide { grid-column: 1 / -1; }
|
||||
.field label { font-size: .85em; font-weight: 600; }
|
||||
.field .hint { font-size: .8em; color: var(--muted); }
|
||||
.field .error { font-size: .85em; color: var(--err); }
|
||||
input[type=text], input[type=password], input[type=number], input[type=url], select, textarea {
|
||||
font: inherit; padding: 6px 8px; border-radius: 6px; border: 1px solid var(--line);
|
||||
background: var(--bg); color: var(--fg); width: 100%; }
|
||||
textarea { min-height: 70px; }
|
||||
.check { flex-direction: row; align-items: center; gap: 6px; }
|
||||
.has-error input, .has-error select, .has-error textarea { border-color: var(--err); }
|
||||
|
||||
.steps { list-style: none; padding: 0; margin: 0; }
|
||||
.steps li { padding: 3px 0; }
|
||||
.steps .ico { display: inline-block; width: 1.4em; font-weight: 700; }
|
||||
.steps .ok { color: var(--ok); } .steps .fail { color: var(--err); } .steps .skip { color: var(--muted); }
|
||||
|
||||
details > summary { cursor: pointer; }
|
||||
.htmx-indicator { display: none; }
|
||||
.htmx-request .htmx-indicator, .htmx-request.htmx-indicator { display: inline; }
|
||||
.login { max-width: 380px; margin: 10vh auto; }
|
||||
[hidden] { display: none !important; }
|
||||
|
||||
@media (max-width: 640px) {
|
||||
main { padding: 10px; }
|
||||
th, td { padding: 6px; }
|
||||
table.responsive thead { display: none; }
|
||||
table.responsive tr { display: block; border-bottom: 1px solid var(--line); padding: 6px 0; }
|
||||
table.responsive td { display: flex; justify-content: space-between; gap: 10px; border: 0; padding: 3px 8px; }
|
||||
table.responsive td::before { content: attr(data-label); color: var(--muted); font-size: .85em; }
|
||||
table.responsive td.num { text-align: left; }
|
||||
}
|
||||
|
||||
.text-err { color: var(--err); } .text-warn { color: var(--warn); } .text-ok { color: var(--ok); }
|
||||
pre.text-err { margin-top: 6px; }
|
||||
ul.tight { margin: 4px 0 0; padding-left: 20px; }
|
||||
.foot { text-align: center; padding: 16px; }
|
||||
meter.fill { width: 100%; height: 8px; margin: 6px 0 2px; display: block; }
|
||||
@@ -0,0 +1,45 @@
|
||||
// NetMover – kleine Helfer. Keine Inline-Skripte (CSP: script-src 'self').
|
||||
(function () {
|
||||
"use strict";
|
||||
var meta = document.querySelector('meta[name="csrf-token"]');
|
||||
var csrf = meta ? meta.getAttribute("content") : "";
|
||||
|
||||
// CSRF-Token für alle HTMX-Anfragen
|
||||
document.addEventListener("htmx:configRequest", function (e) {
|
||||
e.detail.headers["X-CSRF-Token"] = csrf;
|
||||
});
|
||||
|
||||
// Rückfrage vor gefährlichen Aktionen: <form data-confirm="…">
|
||||
document.addEventListener("submit", function (e) {
|
||||
var msg = e.target.getAttribute && e.target.getAttribute("data-confirm");
|
||||
if (msg && !window.confirm(msg)) { e.preventDefault(); }
|
||||
}, true);
|
||||
|
||||
// Felder je nach Auswahl ein-/ausblenden: data-show-for="select-name:wert1 wert2"
|
||||
function refresh(form) {
|
||||
form.querySelectorAll("[data-show-for]").forEach(function (el) {
|
||||
var spec = el.getAttribute("data-show-for").split(":");
|
||||
var ctl = form.querySelector('[name="' + spec[0] + '"]:checked') ||
|
||||
form.querySelector('select[name="' + spec[0] + '"]');
|
||||
var value = ctl ? ctl.value : "";
|
||||
var show = spec[1].split(" ").indexOf(value) !== -1;
|
||||
el.hidden = !show;
|
||||
});
|
||||
}
|
||||
document.querySelectorAll("form[data-dynamic]").forEach(function (form) {
|
||||
form.addEventListener("change", function () { refresh(form); });
|
||||
refresh(form);
|
||||
});
|
||||
|
||||
// Öffentlichen Schlüssel kopieren
|
||||
document.addEventListener("click", function (e) {
|
||||
var btn = e.target.closest && e.target.closest("[data-copy]");
|
||||
if (!btn) return;
|
||||
var src = document.getElementById(btn.getAttribute("data-copy"));
|
||||
if (src && navigator.clipboard) {
|
||||
navigator.clipboard.writeText(src.textContent.trim()).then(function () {
|
||||
btn.textContent = "kopiert ✓";
|
||||
});
|
||||
}
|
||||
});
|
||||
})();
|
||||
Vendored
+1
File diff suppressed because one or more lines are too long
@@ -0,0 +1,37 @@
|
||||
<!doctype html>
|
||||
<html lang="de">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="csrf-token" content="{{ csrf }}">
|
||||
<meta name="htmx-config" content='{"includeIndicatorStyles": false, "allowEval": false}'>
|
||||
<title>{% block title %}NetMover{% endblock %} · NetMover</title>
|
||||
<link rel="stylesheet" href="/static/app.css?v={{ version }}">
|
||||
<script src="/static/htmx.min.js?v=2.0.4" defer></script>
|
||||
<script src="/static/app.js?v={{ version }}" defer></script>
|
||||
</head>
|
||||
<body>
|
||||
{% if logged_in %}
|
||||
<header class="top"><div class="inner">
|
||||
<a class="brand" href="/">NetMover</a>
|
||||
<nav class="main">
|
||||
<a href="/" class="{{ 'active' if active == 'status' }}">Status</a>
|
||||
<a href="/mounts" class="{{ 'active' if active == 'mounts' }}">Anbindungen</a>
|
||||
<a href="/keys" class="{{ 'active' if active == 'keys' }}">Schlüssel</a>
|
||||
<a href="/rules" class="{{ 'active' if active == 'rules' }}">Regeln</a>
|
||||
<a href="/log" class="{{ 'active' if active == 'log' }}">Protokoll</a>
|
||||
<a href="/settings" class="{{ 'active' if active == 'settings' }}">Einstellungen</a>
|
||||
</nav>
|
||||
<form class="inline" method="post" action="/logout">
|
||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||
<button class="small">Abmelden</button>
|
||||
</form>
|
||||
</div></header>
|
||||
{% endif %}
|
||||
<main>
|
||||
{% for kind, text in flashes %}<p class="msg {{ kind }}">{{ text }}</p>{% endfor %}
|
||||
{% block content %}{% endblock %}
|
||||
</main>
|
||||
<footer class="foot muted small">NetMover {{ version }}</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,56 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import csrf_field %}
|
||||
{% block title %}Schlüssel{% endblock %}
|
||||
{% block content %}
|
||||
<h1>SSH-Schlüssel</h1>
|
||||
<p class="muted small">Gespeicherte Schlüssel für SFTP-Anbindungen. Ein Schlüssel kann von mehreren
|
||||
Anbindungen benutzt werden; den Benutzernamen legt jede Anbindung selbst fest. Private Schlüssel
|
||||
liegen verschlüsselt in der Datenbank und werden nie angezeigt.</p>
|
||||
|
||||
{% if not keys %}<p class="muted">Noch keine Schlüssel gespeichert.</p>{% endif %}
|
||||
<div class="stack">
|
||||
{% for k in keys %}
|
||||
<div class="card" id="key-{{ k.id }}">
|
||||
<div class="row spread">
|
||||
<div><strong>{{ k.name }}</strong> <span class="badge">{{ k.key_type.removeprefix('ssh-') }}</span>
|
||||
{% if k.used_by %}<span class="badge ok">benutzt von {{ k.used_by|join(', ') }}</span>{% else %}<span class="badge">unbenutzt</span>{% endif %}</div>
|
||||
<div class="row">
|
||||
<form class="inline row" method="post" action="/keys/{{ k.id }}/rename">{{ csrf_field(csrf) }}
|
||||
<input type="text" name="name" value="{{ k.name }}" aria-label="Neuer Name" required maxlength="100">
|
||||
<button class="small">Umbenennen</button></form>
|
||||
<form class="inline" method="post" action="/keys/{{ k.id }}/delete" data-confirm="Schlüssel „{{ k.name }}“ löschen?">{{ csrf_field(csrf) }}
|
||||
<button class="small danger" {{ 'disabled' if k.used_by }}>Löschen</button></form>
|
||||
</div>
|
||||
</div>
|
||||
<div class="small muted">Fingerabdruck <span class="mono">{{ k.fingerprint }}</span> · angelegt {{ k.created_at|localdt }}</div>
|
||||
<details class="small"><summary>Öffentlicher Schlüssel / authorized_keys</summary>
|
||||
<div class="field"><label>Zeile für <code>~/.ssh/authorized_keys</code> (nur SFTP erlaubt, empfohlen)</label>
|
||||
<pre class="mono card" id="auth-{{ k.id }}">{{ k.authorized_keys_line }}</pre>
|
||||
<button class="small" type="button" data-copy="auth-{{ k.id }}">kopieren</button></div>
|
||||
<div class="field"><label>Öffentlicher Schlüssel</label>
|
||||
<pre class="mono card" id="pub-{{ k.id }}">{{ k.public_key }}</pre>
|
||||
<button class="small" type="button" data-copy="pub-{{ k.id }}">kopieren</button></div>
|
||||
</details>
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
|
||||
<h2>Neuen Schlüssel erzeugen</h2>
|
||||
<form class="form card row" method="post" action="/keys/generate">{{ csrf_field(csrf) }}
|
||||
<input type="text" name="name" placeholder="Name, z. B. paperless-lxc" aria-label="Name" required maxlength="100">
|
||||
<button class="primary">Ed25519-Schlüssel erzeugen</button>
|
||||
</form>
|
||||
|
||||
<h2>Vorhandenen Schlüssel importieren</h2>
|
||||
<form class="form card stack" method="post" action="/keys/import">{{ csrf_field(csrf) }}
|
||||
<div class="fields">
|
||||
<div class="field"><label for="k-name">Name</label><input id="k-name" type="text" name="name" required maxlength="100"></div>
|
||||
<div class="field"><label for="k-pass">Passphrase (falls vorhanden)</label><input id="k-pass" type="password" name="passphrase" autocomplete="off"></div>
|
||||
<div class="field wide"><label for="k-priv">Privater Schlüssel</label>
|
||||
<textarea id="k-priv" name="private_key" class="mono" rows="8" autocomplete="off" spellcheck="false"
|
||||
placeholder="-----BEGIN OPENSSH PRIVATE KEY----- … -----END OPENSSH PRIVATE KEY-----" required></textarea>
|
||||
<span class="hint">Ed25519, ECDSA oder RSA (mind. 2048 Bit), OpenSSH- oder PEM-Format. Eine Passphrase wird nur zum Entschlüsseln benutzt.</span></div>
|
||||
</div>
|
||||
<button>Importieren</button>
|
||||
</form>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,51 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import run_badge %}
|
||||
{% block title %}Protokoll{% endblock %}
|
||||
{% block content %}
|
||||
<h1>Protokoll</h1>
|
||||
<form class="row card" method="get">
|
||||
<select name="rule"><option value="">alle Regeln</option>
|
||||
{% for id, name in rules %}<option value="{{ id }}" {{ 'selected' if id == rule_id }}>{{ name }}</option>{% endfor %}</select>
|
||||
<select name="status"><option value="">jeder Status</option>
|
||||
{% for st in [RunStatus.OK, RunStatus.PARTIAL, RunStatus.ERROR, RunStatus.SKIPPED] %}
|
||||
<option value="{{ st.value }}" {{ 'selected' if st.value == status }}>{{ RUN_LABELS[st] }}</option>{% endfor %}</select>
|
||||
<label class="row small"><input type="checkbox" name="all" value="1" {{ 'checked' if not active_only }}> auch leere Läufe</label>
|
||||
<button>Filtern</button>
|
||||
<span class="muted small">{{ total }} Läufe</span>
|
||||
</form>
|
||||
<div class="table-wrap"><table class="responsive">
|
||||
<thead><tr><th>Zeitpunkt</th><th>Regel</th><th>Status</th><th class="num">Dauer</th><th class="num">gesehen</th><th class="num">übertragen</th><th class="num">übersprungen</th><th class="num">Fehler</th></tr></thead>
|
||||
<tbody>
|
||||
{% for run, rule_name in rows %}
|
||||
<tr class="{{ 'err' if run.status == RunStatus.ERROR }}">
|
||||
<td data-label="Zeitpunkt">{{ run.started_at|localdt }}{% if run.dry_run %} <span class="badge">Probe</span>{% endif %}{% if run.trigger.value == 'manual' %} <span class="badge">manuell</span>{% endif %}</td>
|
||||
<td data-label="Regel">{{ rule_name }}</td>
|
||||
<td data-label="Status">{{ run_badge(run.status) }}</td>
|
||||
<td data-label="Dauer" class="num">{{ duration(run.started_at, run.finished_at) }}</td>
|
||||
<td data-label="gesehen" class="num">{{ run.seen }}</td>
|
||||
<td data-label="übertragen" class="num">{{ run.moved }}</td>
|
||||
<td data-label="übersprungen" class="num">{{ run.skipped }}</td>
|
||||
<td data-label="Fehler" class="num">{{ run.failed }}</td>
|
||||
</tr>
|
||||
{% if run.message or run.moved or run.skipped or run.failed %}
|
||||
<tr><td colspan="8">
|
||||
{% if run.message %}<div class="small {{ 'text-err' if run.status == RunStatus.ERROR }}">{{ run.message }}</div>{% endif %}
|
||||
{% if run.moved or run.skipped or run.failed %}
|
||||
<details hx-get="/log/{{ run.id }}/items" hx-trigger="toggle once" hx-target="find .items" hx-swap="innerHTML">
|
||||
<summary class="small">Einzeldateien</summary><div class="items muted small">lädt …</div></details>
|
||||
{% endif %}
|
||||
</td></tr>
|
||||
{% endif %}
|
||||
{% else %}
|
||||
<tr><td colspan="8" class="muted">Keine Läufe.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody></table></div>
|
||||
{% if pages > 1 %}
|
||||
<div class="row">
|
||||
{% set base = '?rule=' ~ (rule_id or '') ~ '&status=' ~ status ~ ('&all=1' if not active_only else '') %}
|
||||
{% if page > 1 %}<a class="btn small" href="{{ base }}&page={{ page - 1 }}">← neuer</a>{% endif %}
|
||||
<span class="small muted">Seite {{ page }} / {{ pages }}</span>
|
||||
{% if page < pages %}<a class="btn small" href="{{ base }}&page={{ page + 1 }}">älter →</a>{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,16 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Anmelden{% endblock %}
|
||||
{% block content %}
|
||||
<div class="login card">
|
||||
<h1>NetMover</h1>
|
||||
{% if error %}<p class="msg error">{{ error }}</p>{% endif %}
|
||||
<form method="post" class="stack">
|
||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||
<div class="field">
|
||||
<label for="pw">Passwort</label>
|
||||
<input id="pw" type="password" name="password" autocomplete="current-password" autofocus required>
|
||||
</div>
|
||||
<button class="primary">Anmelden</button>
|
||||
</form>
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,43 @@
|
||||
{% macro field(name, label, value='', type='text', errors={}, hint='', wide=false, placeholder='', attrs='') -%}
|
||||
<div class="field{{ ' wide' if wide }}{{ ' has-error' if errors.get(name) }}">
|
||||
<label for="f-{{ name }}">{{ label }}</label>
|
||||
<input id="f-{{ name }}" type="{{ type }}" name="{{ name }}" value="{{ '' if value is none else value }}"
|
||||
placeholder="{{ placeholder }}" {{ attrs|safe }}>
|
||||
{% if errors.get(name) %}<span class="error">{{ errors[name] }}</span>{% endif %}
|
||||
{% if hint %}<span class="hint">{{ hint }}</span>{% endif %}
|
||||
</div>
|
||||
{%- endmacro %}
|
||||
|
||||
{% macro select(name, label, options, current, errors={}, hint='') -%}
|
||||
<div class="field{{ ' has-error' if errors.get(name) }}">
|
||||
<label for="f-{{ name }}">{{ label }}</label>
|
||||
<select id="f-{{ name }}" name="{{ name }}">
|
||||
{% for value, text in options %}
|
||||
<option value="{{ value }}" {{ 'selected' if value|string == current|string }}>{{ text }}</option>
|
||||
{% endfor %}
|
||||
</select>
|
||||
{% if errors.get(name) %}<span class="error">{{ errors[name] }}</span>{% endif %}
|
||||
{% if hint %}<span class="hint">{{ hint }}</span>{% endif %}
|
||||
</div>
|
||||
{%- endmacro %}
|
||||
|
||||
{% macro check(name, label, checked, hint='') -%}
|
||||
<div class="field check">
|
||||
<input id="f-{{ name }}" type="checkbox" name="{{ name }}" {{ 'checked' if checked }}>
|
||||
<label for="f-{{ name }}">{{ label }}</label>
|
||||
{% if hint %}<span class="hint">{{ hint }}</span>{% endif %}
|
||||
</div>
|
||||
{%- endmacro %}
|
||||
|
||||
{% macro csrf_field(csrf) %}<input type="hidden" name="csrf_token" value="{{ csrf }}">{% endmacro %}
|
||||
|
||||
{% macro state_badge(m) -%}
|
||||
{% set labels = SFTP_STATE_LABELS if m.type == MountType.SFTP else STATE_LABELS %}
|
||||
{% set cls = {'mounted': 'ok', 'degraded': 'warn', 'error': 'err'}.get(m.state.value, '') %}
|
||||
<span class="badge {{ cls }}">{{ labels[m.state] }}</span>
|
||||
{%- endmacro %}
|
||||
|
||||
{% macro run_badge(status) -%}
|
||||
{% set cls = {'ok': 'ok', 'partial': 'warn', 'error': 'err'}.get(status.value, '') %}
|
||||
<span class="badge {{ cls }}">{{ RUN_LABELS[status] }}</span>
|
||||
{%- endmacro %}
|
||||
@@ -0,0 +1,64 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import field, select, check %}
|
||||
{% block title %}{{ 'Anbindung bearbeiten' if mount else 'Neue Anbindung' }}{% endblock %}
|
||||
{% block content %}
|
||||
<h1>{{ 'Anbindung „' ~ mount.name ~ '“ bearbeiten' if mount else 'Neue Anbindung' }}</h1>
|
||||
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
||||
<form class="form" method="post" data-dynamic>
|
||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||
{% if mount %}<input type="hidden" name="mount_id" value="{{ mount.id }}">{% endif %}
|
||||
<fieldset><legend>Verbindung</legend><div class="fields">
|
||||
{{ field('name', 'Name', d.name, errors=errors, attrs='required maxlength="100"', hint=('Verzeichnis: ' ~ mount.mountpoint if mount and mount.mountpoint else 'Bestimmt das Verzeichnis unter /mnt/netmover')) }}
|
||||
{{ select('type', 'Typ', [('cifs', 'SMB/CIFS'), ('nfs4', 'NFS 4'), ('nfs3', 'NFS 3'), ('sftp', 'SFTP (nur Ziel)')], d.type.value, errors=errors) }}
|
||||
{{ field('server', 'Server', d.server, errors=errors, placeholder='192.168.1.10 oder nas.lan', attrs='required') }}
|
||||
<div class="field{{ ' has-error' if errors.get('share') }}">
|
||||
<label for="f-share">Freigabe / Pfad</label>
|
||||
<input id="f-share" type="text" name="share" value="{{ d.share }}" required>
|
||||
{% if errors.get('share') %}<span class="error">{{ errors['share'] }}</span>{% endif %}
|
||||
<span class="hint" data-show-for="type:cifs">Freigabename, z. B. <code>scans</code></span>
|
||||
<span class="hint" data-show-for="type:nfs3 nfs4">Exportpfad, z. B. <code>/volume1/scans</code></span>
|
||||
<span class="hint" data-show-for="type:sftp">Absoluter Basisordner, z. B. <code>/opt/paperless/consume</code></span>
|
||||
</div>
|
||||
<div data-show-for="type:sftp">{{ field('port', 'SSH-Port', d.port or '', type='number', errors=errors, placeholder='22') }}</div>
|
||||
</div></fieldset>
|
||||
|
||||
<fieldset data-show-for="type:cifs sftp"><legend>Anmeldung</legend><div class="fields">
|
||||
<div data-show-for="type:sftp">{{ select('auth', 'Anmeldeart', [('ssh_key', 'SSH-Schlüssel'), ('password', 'Passwort')], d.auth.value, errors=errors) }}</div>
|
||||
<div data-show-for="type:sftp"><div data-show-for="auth:ssh_key">
|
||||
{{ select('key_id', 'Schlüssel', key_options, d.key_id, errors=errors, hint='Gespeicherte Schlüssel verwalten unter „Schlüssel“.') }}
|
||||
</div></div>
|
||||
{{ field('username', 'Benutzername', d.username, errors=errors, attrs='autocomplete="off"', hint='Bei CIFS leer lassen für Gastzugang') }}
|
||||
<div data-show-for="type:cifs">{{ field('domain', 'Domäne (optional)', d.domain, errors=errors) }}</div>
|
||||
<div class="field{{ ' has-error' if errors.get('password') }}" id="pw-field">
|
||||
<label for="f-password">Passwort</label>
|
||||
<input id="f-password" type="password" name="password" autocomplete="new-password" placeholder="{{ '••••••' if has_secret }}">
|
||||
{% if errors.get('password') %}<span class="error">{{ errors['password'] }}</span>{% endif %}
|
||||
<span class="hint">{{ 'Leer lassen = unverändert. ' if has_secret }}Bei SFTP mit Schlüssel leer lassen.</span>
|
||||
</div>
|
||||
</div>
|
||||
{% if public_key %}<div class="field wide"><label>Öffentlicher Schlüssel dieser Anbindung (in ~/.ssh/authorized_keys des Zielbenutzers eintragen)</label>
|
||||
<pre class="mono card" id="pubkey">{{ public_key }}</pre><button class="small" type="button" data-copy="pubkey">kopieren</button></div>{% endif %}
|
||||
</fieldset>
|
||||
|
||||
<fieldset data-show-for="type:cifs"><legend>Eigentümer und Rechte (CIFS)</legend><div class="fields">
|
||||
{{ field('uid', 'uid', d.uid, type='number', errors=errors, hint='Eigentümer aller Dateien im Mount') }}
|
||||
{{ field('gid', 'gid', d.gid, type='number', errors=errors) }}
|
||||
{{ field('file_mode', 'file_mode', d.file_mode, errors=errors, placeholder='0644') }}
|
||||
{{ field('dir_mode', 'dir_mode', d.dir_mode, errors=errors, placeholder='0755') }}
|
||||
</div></fieldset>
|
||||
|
||||
<fieldset data-show-for="type:cifs nfs3 nfs4"><legend>Erweitert</legend><div class="fields">
|
||||
{{ field('extra_options', 'Zusatzoptionen', d.extra_options, errors=errors, wide=true, placeholder='vers=3.0,noserverino', hint='Kommagetrennt; nur erlaubte Optionen. Zugangsdaten setzt NetMover selbst.') }}
|
||||
{{ check('auto_mount', 'Beim Start automatisch einhängen', d.auto_mount) }}
|
||||
</div></fieldset>
|
||||
{{ check('enabled', 'Aktiv', d.enabled) }}
|
||||
|
||||
<div class="row">
|
||||
<button class="primary">Speichern</button>
|
||||
<button type="button" hx-post="/mounts/test" hx-include="closest form" hx-target="#form-check" hx-swap="innerHTML">
|
||||
Testen <span class="htmx-indicator">(läuft …)</span></button>
|
||||
<a class="btn" href="/mounts">Abbrechen</a>
|
||||
</div>
|
||||
<div id="form-check"></div>
|
||||
</form>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,51 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import state_badge, csrf_field %}
|
||||
{% block title %}Anbindungen{% endblock %}
|
||||
{% block content %}
|
||||
<div class="row spread"><h1>Anbindungen</h1><a class="btn primary" href="/mounts/new">+ Neue Anbindung</a></div>
|
||||
{% if not mounts %}<p class="muted">Noch keine Anbindungen.</p>{% endif %}
|
||||
<div class="stack">
|
||||
{% for m in mounts %}
|
||||
<div class="card" id="mount-{{ m.id }}">
|
||||
<div class="row spread">
|
||||
<div><strong>{{ m.name }}</strong> {{ state_badge(m) }}
|
||||
{% if not m.enabled %}<span class="badge">deaktiviert</span>{% endif %}
|
||||
{% if m.auto_mount %}<span class="badge">auto</span>{% endif %}</div>
|
||||
<div class="row">
|
||||
<button class="small" hx-post="/mounts/{{ m.id }}/test" hx-target="#check-{{ m.id }}" hx-swap="innerHTML">
|
||||
Testen <span class="htmx-indicator">…</span></button>
|
||||
{% if m.type != MountType.SFTP %}
|
||||
{% if m.state == MountState.MOUNTED or m.state == MountState.DEGRADED %}
|
||||
<form class="inline" method="post" action="/mounts/{{ m.id }}/unmount" data-confirm="„{{ m.name }}“ aushängen?">{{ csrf_field(csrf) }}<button class="small">Aushängen</button></form>
|
||||
{% else %}
|
||||
<form class="inline" method="post" action="/mounts/{{ m.id }}/mount">{{ csrf_field(csrf) }}<button class="small">Einhängen</button></form>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
<a class="btn small" href="/mounts/{{ m.id }}/edit">Bearbeiten</a>
|
||||
<form class="inline" method="post" action="/mounts/{{ m.id }}/delete" data-confirm="„{{ m.name }}“ wirklich löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
||||
</div>
|
||||
</div>
|
||||
<div class="muted small mono">{{ m.type.value }} · {% if m.type == MountType.CIFS %}//{{ m.server }}/{{ m.share }}{% elif m.type == MountType.SFTP %}{{ m.server }}:{{ m.port or 22 }}{{ m.share }}{% else %}{{ m.server }}:{{ m.share }}{% endif %}
|
||||
{% if m.mountpoint %} → {{ m.mountpoint }}{% endif %}</div>
|
||||
{% if m.last_error %}<pre class="small text-err">{{ m.last_error }}</pre>{% endif %}
|
||||
{% if m.type == MountType.SFTP %}
|
||||
<div class="small">Hostschlüssel: {% if m.host_key %}<span class="mono">{{ m.host_key.split()[0] }}</span> festgehalten
|
||||
<form class="inline" method="post" action="/mounts/{{ m.id }}/reset-hostkey" data-confirm="Hostschlüssel zurücksetzen? Nur tun, wenn der Server neu aufgesetzt wurde.">{{ csrf_field(csrf) }}<button class="small">zurücksetzen</button></form>
|
||||
{% else %}<span class="text-warn">noch nicht bestätigt – bitte „Testen“</span>{% endif %}</div>
|
||||
{% if key_names.get(m.id) %}
|
||||
<div class="small">Anmeldung mit gespeichertem Schlüssel <a href="/keys">„{{ key_names[m.id] }}“</a></div>
|
||||
{% endif %}
|
||||
{% if pubkeys.get(m.id) %}
|
||||
<details class="small"><summary>Öffentlicher Schlüssel für authorized_keys</summary>
|
||||
<pre class="mono" id="pub-{{ m.id }}">{{ pubkeys[m.id] }}</pre>
|
||||
<button class="small" type="button" data-copy="pub-{{ m.id }}">kopieren</button></details>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
<div id="check-{{ m.id }}">
|
||||
{% if reports.get(m.id) %}<details class="small"><summary class="muted">Letzter Test {{ m.last_test_at|ago }}</summary>
|
||||
{% with report = reports[m.id], errors = {} %}{% include "partials/check_result.html" %}{% endwith %}</details>{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,20 @@
|
||||
{% set step_names = {'mount': 'Einhängen', 'read': 'Lesen', 'write': 'Schreiben', 'connect': 'Verbinden'} %}
|
||||
<div class="card stack">
|
||||
{% if errors %}
|
||||
<p class="msg error">Eingaben unvollständig:</p>
|
||||
<ul class="tight">{% for k, v in errors.items() %}<li><strong>{{ k }}</strong>: {{ v }}</li>{% endfor %}</ul>
|
||||
{% elif report %}
|
||||
<ul class="steps">
|
||||
{% for st in report.steps %}
|
||||
<li><span class="ico {{ 'ok' if st.ok else 'skip' if st.ok is none else 'fail' }}">{{ '✓' if st.ok else '–' if st.ok is none else '✗' }}</span>
|
||||
<strong>{{ step_names.get(st.name, st.name) }}</strong>{% if st.message %}: <span class="{{ 'text-err' if st.ok == false }}">{{ st.message }}</span>{% endif %}</li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
<div class="small">Geeignet als:
|
||||
{% for role in report.roles %}<span class="badge ok">{{ role.value }}</span> {% else %}<span class="badge err">nichts</span>{% endfor %}</div>
|
||||
{% if report.owner %}<div class="small muted">Testdatei gehörte {{ report.owner }}{% if report.root_squash %} (Root-Squash){% endif %}</div>{% endif %}
|
||||
{% if report.free_bytes is not none %}<div class="small muted">{{ report.free_bytes|bytes }} frei von {{ report.total_bytes|bytes }}</div>{% endif %}
|
||||
{% if report.host_key_fingerprint %}<div class="small">Hostschlüssel: <span class="mono">{{ report.host_key_fingerprint }}</span></div>{% endif %}
|
||||
{% for h in report.hints %}<p class="msg info small">{{ h }}</p>{% endfor %}
|
||||
{% endif %}
|
||||
</div>
|
||||
@@ -0,0 +1,18 @@
|
||||
{% from "macros.html" import run_badge %}
|
||||
<div class="card stack">
|
||||
{% if not run %}<p class="msg error">Probelauf nicht möglich (Regel gelöscht?).</p>
|
||||
{% else %}
|
||||
<div class="row"><strong>Probelauf</strong> {{ run_badge(run.status) }}
|
||||
<span class="muted small">{{ run.seen }} gefunden · {{ run.moved }} würden übertragen · {{ run.skipped }} übersprungen</span></div>
|
||||
{% if run.message %}<p class="msg {{ 'error' if run.status == RunStatus.ERROR else 'info' }} small">{{ run.message }}</p>{% endif %}
|
||||
{% if items %}
|
||||
<div class="table-wrap"><table class="responsive small">
|
||||
<thead><tr><th>Datei</th><th>Ziel</th><th>Ergebnis</th></tr></thead>
|
||||
<tbody>{% for i in items %}
|
||||
<tr><td data-label="Datei" class="mono">{{ i.src }}</td><td data-label="Ziel" class="mono">{{ i.dst or '–' }}</td>
|
||||
<td data-label="Ergebnis">{{ i.message or i.status.value }}</td></tr>
|
||||
{% endfor %}</tbody></table></div>
|
||||
{% elif run.status != RunStatus.ERROR %}<p class="muted small">Keine passenden Dateien.</p>{% endif %}
|
||||
<p class="muted small">Es wurde nichts verändert.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
@@ -0,0 +1,11 @@
|
||||
{% set cls = {'moved': 'ok', 'copied': 'ok', 'planned': 'ok', 'failed': 'err', 'quarantined': 'warn'} %}
|
||||
{% set labels = {'moved': 'verschoben', 'copied': 'kopiert', 'skipped': 'übersprungen', 'failed': 'Fehler', 'quarantined': 'Quarantäne', 'planned': 'geplant'} %}
|
||||
<table class="responsive small">
|
||||
<thead><tr><th>Datei</th><th>Ziel</th><th class="num">Größe</th><th>Ergebnis</th></tr></thead>
|
||||
<tbody>{% for i in items %}
|
||||
<tr><td data-label="Datei" class="mono">{{ i.src }}</td><td data-label="Ziel" class="mono">{{ i.dst or '–' }}</td>
|
||||
<td data-label="Größe" class="num">{{ i.size|bytes }}</td>
|
||||
<td data-label="Ergebnis"><span class="badge {{ cls.get(i.status.value, '') }}">{{ labels[i.status.value] }}</span>
|
||||
{% if i.message %}<span class="{{ 'text-err' if i.status.value == 'failed' }}">{{ i.message }}</span>{% endif %}</td></tr>
|
||||
{% else %}<tr><td colspan="4" class="muted">Keine Einzeldateien protokolliert.</td></tr>{% endfor %}
|
||||
</tbody></table>
|
||||
@@ -0,0 +1,54 @@
|
||||
{% from "macros.html" import state_badge, run_badge %}
|
||||
{% if not health.ok %}
|
||||
<div class="msg error"><strong>Achtung:</strong>
|
||||
<ul class="tight">{% for p in health.problems %}<li>{{ p }}</li>{% endfor %}</ul></div>
|
||||
{% else %}
|
||||
<p class="msg ok">Alles in Ordnung.</p>
|
||||
{% endif %}
|
||||
|
||||
<h2>Anbindungen</h2>
|
||||
{% if not mounts %}<p class="muted">Noch keine Anbindungen – <a href="/mounts/new">jetzt anlegen</a>.</p>{% endif %}
|
||||
<div class="grid">
|
||||
{% for v in mounts %}{% set m = v.m %}
|
||||
<div class="card">
|
||||
<div class="row spread"><strong><a href="/mounts#mount-{{ m.id }}">{{ m.name }}</a></strong>{{ state_badge(m) }}</div>
|
||||
<div class="muted small mono">{{ m.type.value }} · {{ m.server }}{{ ':' if m.type != MountType.CIFS else '/' }}{{ m.share }}</div>
|
||||
{% if m.state == MountState.MOUNTED and m.last_mounted_at and m.type != MountType.SFTP %}
|
||||
<div class="small">eingehängt seit {{ m.last_mounted_at|localdt }}</div>
|
||||
{% elif m.state in (MountState.DEGRADED, MountState.ERROR) %}
|
||||
<div class="small">{{ 'gestört' if m.state == MountState.DEGRADED else 'Fehler' }} seit {{ m.state_since|localdt }} ({{ m.state_since|ago }})</div>
|
||||
{% endif %}
|
||||
{% if v.usage %}{% set used = (v.usage[0] - v.usage[1]) %}{% set pct = (100 * used / v.usage[0]) if v.usage[0] else 0 %}
|
||||
<meter class="fill" min="0" max="100" low="85" high="95" optimum="0" value="{{ pct|round(1) }}">{{ pct|round(0)|int }} %</meter>
|
||||
<div class="small muted">{{ v.usage[1]|bytes }} frei von {{ v.usage[0]|bytes }} ({{ pct|round(0)|int }} % belegt)</div>
|
||||
{% endif %}
|
||||
{% if m.last_error %}<pre class="small text-err">{{ m.last_error }}</pre>{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
|
||||
<h2>Regeln</h2>
|
||||
{% if not rules %}<p class="muted">Noch keine Regeln – <a href="/rules/new">jetzt anlegen</a>.</p>{% else %}
|
||||
<div class="table-wrap"><table class="responsive">
|
||||
<thead><tr><th>Regel</th><th>Letztes Ergebnis</th><th>Letzter Erfolg</th><th>Nächster Lauf</th><th></th></tr></thead>
|
||||
<tbody>
|
||||
{% for v in rules %}{% set r = v.r %}
|
||||
<tr class="{{ 'warn' if v.stale }}">
|
||||
<td data-label="Regel"><strong>{{ r.name }}</strong>{% if not r.enabled %} <span class="badge">aus</span>{% endif %}{% if r.dry_run %} <span class="badge warn">Probelauf</span>{% endif %}
|
||||
<div class="muted small mono">{{ v.target }}</div></td>
|
||||
<td data-label="Letztes Ergebnis">
|
||||
{% if v.running_since %}<span class="badge warn">läuft seit {{ v.running_since|ago|replace('vor ', '') }}</span>
|
||||
{% elif v.last_run %}{{ run_badge(v.last_run.status) }}
|
||||
<span class="small muted">{{ v.last_run.finished_at|ago }}{% if v.last_run.seen %} · {{ v.last_run.moved }}/{{ v.last_run.seen }}{% endif %}</span>
|
||||
{% if v.last_run.status in (RunStatus.ERROR, RunStatus.PARTIAL) and v.last_run.message %}<div class="small text-err">{{ v.last_run.message }}</div>{% endif %}
|
||||
{% else %}<span class="muted">–</span>{% endif %}</td>
|
||||
<td data-label="Letzter Erfolg">{{ r.last_success_at|ago }}{% if v.stale %}<div class="small text-warn">länger als 3 × Intervall!</div>{% endif %}
|
||||
{% if r.consecutive_failures %}<div class="small text-err">{{ r.consecutive_failures }} Fehler in Folge</div>{% endif %}</td>
|
||||
<td data-label="Nächster Lauf">{{ v.next_run|ago if v.next_run else '–' }}</td>
|
||||
<td><form class="inline" method="post" action="/rules/{{ r.id }}/run">
|
||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||
<button class="small" {{ 'disabled' if v.running_since }}>Jetzt ausführen</button></form></td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody></table></div>
|
||||
{% endif %}
|
||||
@@ -0,0 +1,49 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import field, select, check %}
|
||||
{% block title %}{{ 'Regel bearbeiten' if rule_id else 'Neue Regel' }}{% endblock %}
|
||||
{% block content %}
|
||||
<h1>{{ 'Regel bearbeiten' if rule_id else 'Neue Regel' }}</h1>
|
||||
{% if errors.get('_') %}<p class="msg error">{{ errors['_'] }}</p>{% endif %}
|
||||
{% if not src_mounts %}<p class="msg info">Zuerst eine SMB- oder NFS-Anbindung als Quelle <a href="/mounts/new">anlegen</a>.</p>{% endif %}
|
||||
<form class="form" method="post" data-dynamic>
|
||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||
<fieldset><legend>Allgemein</legend><div class="fields">
|
||||
{{ field('name', 'Name', d.name, errors=errors, attrs='required maxlength="100"') }}
|
||||
{{ field('interval_seconds', 'Intervall (Sekunden)', d.interval_seconds, type='number', errors=errors, attrs='min="10" required', hint='mindestens 10') }}
|
||||
{{ check('enabled', 'Aktiv', d.enabled) }}
|
||||
{{ check('dry_run', 'Nur Probelauf (nichts verändern)', d.dry_run) }}
|
||||
</div></fieldset>
|
||||
|
||||
<fieldset><legend>Quelle</legend><div class="fields">
|
||||
{{ select('src_mount_id', 'Anbindung', src_options, d.src_mount_id or '', errors=errors) }}
|
||||
{{ field('src_subpath', 'Unterordner', d.src_subpath, errors=errors, placeholder='z. B. Scans/Max', hint='relativ zur Freigabe; leer = Wurzel') }}
|
||||
{{ select('pattern_type', 'Filterart', [('glob', 'Glob'), ('regex', 'Regulärer Ausdruck')], d.pattern_type.value, errors=errors) }}
|
||||
{{ field('pattern', 'Filter', d.pattern, errors=errors, hint='Glob: *.pdf;*.jpg (Dateiname, ohne Groß/klein). Regex: gegen relativen Pfad.') }}
|
||||
{{ check('recursive', 'Unterordner einbeziehen (Struktur bleibt erhalten)', d.recursive) }}
|
||||
</div></fieldset>
|
||||
|
||||
<fieldset><legend>Ziel</legend><div class="fields">
|
||||
{{ select('dst_kind', 'Zielart', [('mount', 'Anbindung'), ('local', 'Lokaler Pfad')], d.dst_kind, errors=errors) }}
|
||||
<div data-show-for="dst_kind:mount">{{ select('dst_mount_id', 'Anbindung', dst_options, d.dst_mount_id or '', errors=errors) }}</div>
|
||||
{{ field('dst_path', 'Zielordner', d.dst_path, errors=errors, hint='Anbindung: relativer Unterordner (wird angelegt). Lokal: absoluter Pfad, muss existieren.') }}
|
||||
{{ select('action', 'Aktion', [('move', 'verschieben'), ('copy', 'kopieren (Quelle bleibt)')], d.action.value, errors=errors) }}
|
||||
{{ select('on_collision', 'Wenn Datei existiert', [('rename', 'umbenennen (scan-1.pdf)'), ('skip', 'überspringen'), ('overwrite', 'überschreiben')], d.on_collision.value, errors=errors) }}
|
||||
</div></fieldset>
|
||||
|
||||
<fieldset><legend>Sicherheit und Grenzen</legend><div class="fields">
|
||||
{{ field('stability_seconds', 'Stabilitätsfenster (s)', d.stability_seconds, type='number', errors=errors, attrs='min="0"', hint='Größe und Zeit müssen so lange gleich bleiben') }}
|
||||
{{ field('min_age_seconds', 'Mindestalter (s)', d.min_age_seconds, type='number', errors=errors, attrs='min="0"') }}
|
||||
{{ field('max_size_mb', 'Max. Größe (MB)', d.max_size_mb or '', type='number', errors=errors, hint='leer = unbegrenzt') }}
|
||||
{{ field('op_timeout_seconds', 'Zeitlimit ohne Fortschritt (s)', d.op_timeout_seconds, type='number', errors=errors, attrs='min="10"') }}
|
||||
</div></fieldset>
|
||||
|
||||
<fieldset><legend>Rechte und Quarantäne</legend><div class="fields">
|
||||
{{ field('set_owner', 'Eigentümer setzen', d.set_owner, errors=errors, placeholder='1000:1000', hint='leer = unverändert; auf CIFS wirkungslos') }}
|
||||
{{ field('set_mode', 'Modus setzen', d.set_mode, errors=errors, placeholder='0644', hint='leer = wie Quelle') }}
|
||||
{{ field('quarantine_path', 'Quarantäneordner', d.quarantine_path, errors=errors, placeholder='_quarantaene', hint='relativ zur Quell-Anbindung; leer = keine Quarantäne') }}
|
||||
{{ field('quarantine_after', 'nach Fehlversuchen', d.quarantine_after, type='number', errors=errors, attrs='min="1"') }}
|
||||
</div></fieldset>
|
||||
|
||||
<div class="row"><button class="primary">Speichern</button><a class="btn" href="/rules">Abbrechen</a></div>
|
||||
</form>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,32 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import run_badge, csrf_field %}
|
||||
{% block title %}Regeln{% endblock %}
|
||||
{% block content %}
|
||||
<div class="row spread"><h1>Regeln</h1><a class="btn primary" href="/rules/new">+ Neue Regel</a></div>
|
||||
{% if not rules %}<p class="muted">Noch keine Regeln.</p>{% endif %}
|
||||
<div class="stack">
|
||||
{% for v in rules %}{% set r = v.r %}
|
||||
<div class="card" id="rule-{{ r.id }}">
|
||||
<div class="row spread">
|
||||
<div><strong>{{ r.name }}</strong>
|
||||
{% if r.enabled %}<span class="badge ok">aktiv</span>{% else %}<span class="badge">aus</span>{% endif %}
|
||||
{% if r.dry_run %}<span class="badge warn">nur Probelauf</span>{% endif %}
|
||||
<span class="badge">{{ 'kopieren' if r.action == RuleAction.COPY else 'verschieben' }}</span>
|
||||
<span class="badge">alle {{ r.interval_seconds }} s</span></div>
|
||||
<div class="row">
|
||||
<form class="inline" method="post" action="/rules/{{ r.id }}/run">{{ csrf_field(csrf) }}<button class="small" {{ 'disabled' if v.running_since }}>Jetzt ausführen</button></form>
|
||||
<button class="small" hx-post="/rules/{{ r.id }}/dryrun" hx-target="#dry-{{ r.id }}" hx-swap="innerHTML">Probelauf <span class="htmx-indicator">…</span></button>
|
||||
<form class="inline" method="post" action="/rules/{{ r.id }}/toggle">{{ csrf_field(csrf) }}<button class="small">{{ 'Deaktivieren' if r.enabled else 'Aktivieren' }}</button></form>
|
||||
<a class="btn small" href="/rules/{{ r.id }}/edit">Bearbeiten</a>
|
||||
<form class="inline" method="post" action="/rules/{{ r.id }}/delete" data-confirm="Regel „{{ r.name }}“ und ihr Protokoll löschen?">{{ csrf_field(csrf) }}<button class="small danger">Löschen</button></form>
|
||||
</div>
|
||||
</div>
|
||||
<div class="muted small mono">{{ v.target }} · Filter {{ r.pattern }}{{ ' (rekursiv)' if r.recursive }}</div>
|
||||
<div class="small">Letzter Lauf: {% if v.running_since %}<span class="badge warn">läuft seit {{ v.running_since|ago|replace('vor ', '') }}</span>{% elif v.last_run %}{{ run_badge(v.last_run.status) }} {{ v.last_run.finished_at|ago }}{% else %}–{% endif %}
|
||||
· letzter Erfolg: <span class="{{ 'text-warn' if v.stale }}">{{ r.last_success_at|ago }}</span>
|
||||
· nächster: {{ v.next_run|ago if v.next_run else '–' }}</div>
|
||||
<div id="dry-{{ r.id }}"></div>
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,46 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import field, select, check, csrf_field %}
|
||||
{% block title %}Einstellungen{% endblock %}
|
||||
{% block content %}
|
||||
<h1>Einstellungen</h1>
|
||||
<form class="form" method="post" action="/settings" data-dynamic>
|
||||
{{ csrf_field(csrf) }}
|
||||
<fieldset><legend>Protokoll</legend><div class="fields">
|
||||
{{ field('retention_days', 'Aufbewahrung (Tage)', cfg.retention_days, type='number', errors=errors, attrs='min="1" max="3650"') }}
|
||||
</div></fieldset>
|
||||
<fieldset><legend>Benachrichtigung (Webhook)</legend><div class="fields">
|
||||
{{ select('notify_mode', 'Benachrichtigen', [('off', 'nie'), ('every', 'bei jedem fehlgeschlagenen Lauf'), ('after_n', 'nach N Fehlschlägen in Folge')], cfg.notify_mode.value, errors=errors) }}
|
||||
<div data-show-for="notify_mode:after_n">{{ field('notify_after', 'N', cfg.notify_after, type='number', errors=errors, attrs='min="1"') }}</div>
|
||||
{{ field('webhook_url', 'Webhook-URL', mask_url(cfg.webhook_url), errors=errors, wide=true, placeholder='https://ntfy.sh/mein-topic', hint='Wird nur gekürzt angezeigt; unverändert lassen = behalten, leeren = entfernen.') }}
|
||||
{{ field('webhook_content_type', 'Content-Type', cfg.webhook_content_type, errors=errors) }}
|
||||
{{ check('notify_recovery', 'Entwarnung senden, wenn es wieder läuft', cfg.notify_recovery) }}
|
||||
<div class="field wide"><label for="f-tpl">Vorlage</label>
|
||||
<textarea id="f-tpl" name="webhook_template" class="mono">{{ cfg.webhook_template }}</textarea>
|
||||
<span class="hint">Platzhalter: $event $title $message $rule $time $host – bei JSON automatisch maskiert.</span></div>
|
||||
</div>
|
||||
<div class="row"><button type="button" hx-post="/settings/webhook-test" hx-target="#hook-result">Test senden (gespeicherte Einstellungen)</button><span id="hook-result"></span></div>
|
||||
</fieldset>
|
||||
<button class="primary">Speichern</button>
|
||||
</form>
|
||||
|
||||
<h2>Passwort ändern</h2>
|
||||
<form class="form card stack" method="post" action="/settings/password">
|
||||
{{ csrf_field(csrf) }}
|
||||
<div class="fields">
|
||||
<div class="field"><label for="cur">Aktuelles Passwort</label><input id="cur" type="password" name="current" autocomplete="current-password" required></div>
|
||||
<div class="field"><label for="np">Neues Passwort</label><input id="np" type="password" name="password" autocomplete="new-password" minlength="10" required></div>
|
||||
<div class="field"><label for="np2">Wiederholen</label><input id="np2" type="password" name="confirm" autocomplete="new-password" minlength="10" required></div>
|
||||
</div>
|
||||
<button>Passwort ändern</button>
|
||||
</form>
|
||||
|
||||
<h2>Export / Import</h2>
|
||||
<div class="card stack">
|
||||
<p class="small muted">Export enthält Anbindungen, Regeln und Einstellungen – <strong>ohne</strong> Zugangsdaten und Webhook-URL.
|
||||
Beim Import werden vorhandene Namen übersprungen; importierte Regeln sind zunächst deaktiviert.</p>
|
||||
<div class="row"><a class="btn" href="/settings/export">Konfiguration exportieren</a></div>
|
||||
<form class="row" method="post" action="/settings/import" enctype="multipart/form-data">
|
||||
{{ csrf_field(csrf) }}<input type="file" name="file" accept="application/json,.json" required><button>Importieren</button>
|
||||
</form>
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,21 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Einrichtung{% endblock %}
|
||||
{% block content %}
|
||||
<div class="login card">
|
||||
<h1>Ersteinrichtung</h1>
|
||||
<p class="muted">Bitte ein Administratorpasswort festlegen (mindestens 10 Zeichen).
|
||||
Den Einrichtungscode schreibt NetMover beim Start ins Log:
|
||||
<code>journalctl -u netmover | grep setup</code></p>
|
||||
{% if error %}<p class="msg error">{{ error }}</p>{% endif %}
|
||||
<form method="post" class="stack">
|
||||
<input type="hidden" name="csrf_token" value="{{ csrf }}">
|
||||
<div class="field"><label for="code">Einrichtungscode</label>
|
||||
<input id="code" type="text" name="code" autocomplete="off" placeholder="xxxx-xxxx-xxxx" required></div>
|
||||
<div class="field"><label for="pw">Passwort</label>
|
||||
<input id="pw" type="password" name="password" autocomplete="new-password" minlength="10" required></div>
|
||||
<div class="field"><label for="pw2">Passwort wiederholen</label>
|
||||
<input id="pw2" type="password" name="confirm" autocomplete="new-password" minlength="10" required></div>
|
||||
<button class="primary">Passwort setzen</button>
|
||||
</form>
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,9 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Status{% endblock %}
|
||||
{% block content %}
|
||||
<div class="row spread"><h1>Status</h1>
|
||||
<span class="muted small">aktualisiert sich alle 15 s</span></div>
|
||||
<div id="status-body" hx-get="/partials/status" hx-trigger="every 15s" hx-swap="innerHTML">
|
||||
{% include "partials/status_body.html" %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
+107
-6
@@ -240,6 +240,9 @@ def test_source_not_deletable_keeps_target_and_prevents_duplicate(
|
||||
) -> None:
|
||||
if path == "copy":
|
||||
monkeypatch.setattr(LocalDestination, "move_same_device", lambda *a, **k: None)
|
||||
else:
|
||||
# Dateisystem ohne renameat2 → link + unlink, unlink scheitert
|
||||
monkeypatch.setattr("netmover.mover.destinations.rename_noreplace", lambda *a: False)
|
||||
write(src / "scan.pdf")
|
||||
real_unlink = os.unlink
|
||||
|
||||
@@ -637,22 +640,24 @@ def test_local_destination_must_exist(
|
||||
def test_fsync_before_final_rename(
|
||||
engine: Engine, rule_id: int, src: Path, dst: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
from netmover.mover import fsops
|
||||
|
||||
events: list[str] = []
|
||||
real_fsync, real_link = os.fsync, os.link
|
||||
real_fsync, real_rename = os.fsync, fsops.rename_noreplace
|
||||
|
||||
def fsync(fd: int) -> None:
|
||||
events.append("fsync")
|
||||
real_fsync(fd)
|
||||
|
||||
def link(a: object, b: object, *args: object, **kw: object) -> None:
|
||||
events.append(f"link:{Path(str(b)).name}")
|
||||
real_link(a, b) # type: ignore[arg-type]
|
||||
def rename(a: object, b: object) -> bool:
|
||||
events.append(f"rename:{Path(str(b)).name}")
|
||||
return real_rename(a, b) # type: ignore[arg-type]
|
||||
|
||||
monkeypatch.setattr("netmover.mover.destinations.os.fsync", fsync)
|
||||
monkeypatch.setattr("netmover.mover.destinations.os.link", link)
|
||||
monkeypatch.setattr("netmover.mover.destinations.rename_noreplace", rename)
|
||||
write(src / "a.pdf")
|
||||
run(engine, snap(rule_id), src, dst)
|
||||
assert events.index("fsync") < events.index("link:a.pdf")
|
||||
assert events.index("fsync") < events.index("rename:a.pdf")
|
||||
|
||||
|
||||
@pytest.mark.usefixtures("cross_device")
|
||||
@@ -678,3 +683,99 @@ def test_name_taken_between_check_and_link_is_never_overwritten(
|
||||
assert result.items[0].dst == str(dst / "scan-1.pdf")
|
||||
assert (dst / "scan.pdf").read_text() == "fremd"
|
||||
assert (dst / "scan-1.pdf").read_text() == "neu"
|
||||
|
||||
|
||||
# -- renameat2(RENAME_NOREPLACE) ----------------------------------------------------------
|
||||
|
||||
|
||||
def test_rename_noreplace(tmp_path: Path) -> None:
|
||||
from netmover.mover.fsops import rename_noreplace
|
||||
|
||||
a, b = write(tmp_path / "a", "neu"), write(tmp_path / "b", "alt")
|
||||
ino = a.stat().st_ino
|
||||
if not rename_noreplace(a, tmp_path / "c"):
|
||||
pytest.skip("Dateisystem des Testrechners kann kein renameat2")
|
||||
assert (tmp_path / "c").stat().st_ino == ino and not a.exists() # echtes rename
|
||||
with pytest.raises(FileExistsError):
|
||||
rename_noreplace(tmp_path / "c", b)
|
||||
assert b.read_text() == "alt" and (tmp_path / "c").exists()
|
||||
|
||||
|
||||
@pytest.mark.usefixtures("cross_device")
|
||||
@pytest.mark.parametrize("supported", [True, False])
|
||||
def test_final_step_is_a_real_rename_when_possible(
|
||||
engine: Engine,
|
||||
rule_id: int,
|
||||
src: Path,
|
||||
dst: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
supported: bool,
|
||||
) -> None:
|
||||
"""Watcher wie Paperless sehen ein rename (IN_MOVED_TO); ohne Unterstützung
|
||||
bleibt es beim link-Rückfallweg – überschrieben wird in keinem Fall."""
|
||||
links: list[object] = []
|
||||
real_link = os.link
|
||||
|
||||
def spy_link(a: object, b: object, *args: object, **kw: object) -> None:
|
||||
links.append(b)
|
||||
real_link(a, b) # type: ignore[arg-type]
|
||||
|
||||
monkeypatch.setattr("netmover.mover.destinations.os.link", spy_link)
|
||||
if not supported:
|
||||
monkeypatch.setattr("netmover.mover.destinations.rename_noreplace", lambda *a: False)
|
||||
write(dst / "a.pdf", "alt")
|
||||
write(src / "a.pdf", "neu")
|
||||
result = run(engine, snap(rule_id), src, dst)
|
||||
assert result.items[0].dst == str(dst / "a-1.pdf")
|
||||
assert (dst / "a.pdf").read_text() == "alt" and (dst / "a-1.pdf").read_text() == "neu"
|
||||
assert bool(links) is not supported
|
||||
assert [p.name for p in dst.iterdir() if p.name.endswith(".part")] == []
|
||||
|
||||
|
||||
# -- Praxisbefund: veralteter Attribut-Cache (CIFS/NFS) -------------------------------------
|
||||
|
||||
|
||||
def test_stability_uses_fresh_attributes_not_stat_cache(
|
||||
engine: Engine,
|
||||
rule_id: int,
|
||||
src: Path,
|
||||
dst: Path,
|
||||
hooks: list[Callable[[], None]],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Auf dem Testhost lieferte CIFS per stat() sekundenlang eine veraltete Größe,
|
||||
während die Datei wuchs. Simuliert: os.stat liefert für die Datei immer den
|
||||
alten Stand (Cache), nur öffnen + fstat sieht die Wahrheit."""
|
||||
growing = write(src / "langsam.pdf", "a" * 10)
|
||||
stale = os.stat(growing)
|
||||
real_stat = os.stat
|
||||
|
||||
def cached_stat(path: object, *a: object, **kw: object) -> os.stat_result:
|
||||
if str(path) == str(growing):
|
||||
return stale
|
||||
return real_stat(path, *a, **kw) # type: ignore[arg-type]
|
||||
|
||||
monkeypatch.setattr(os, "stat", cached_stat)
|
||||
hooks.append(lambda: growing.write_text("a" * 20))
|
||||
result = run(engine, snap(rule_id), src, dst)
|
||||
assert statuses(result) == {"langsam.pdf": ItemStatus.SKIPPED}
|
||||
assert "wird noch geschrieben" in (result.items[0].message or "")
|
||||
assert growing.exists() and not (dst / "langsam.pdf").exists()
|
||||
|
||||
|
||||
def test_locked_file_counts_as_in_use(
|
||||
engine: Engine, rule_id: int, src: Path, dst: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
f = write(src / "gesperrt.pdf")
|
||||
real_open = os.open
|
||||
|
||||
def locked(path: object, flags: int, *a: object, **kw: object) -> int:
|
||||
if str(path) == str(f):
|
||||
raise PermissionError(errno.EACCES, "Permission denied", str(path))
|
||||
return real_open(path, flags, *a, **kw) # type: ignore[arg-type]
|
||||
|
||||
monkeypatch.setattr("netmover.mover.scan.os.open", locked)
|
||||
result = run(engine, snap(rule_id), src, dst)
|
||||
assert statuses(result) == {"gesperrt.pdf": ItemStatus.SKIPPED}
|
||||
assert result.items[0].message == "Datei ist gerade in Benutzung"
|
||||
assert f.exists()
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
"""Schlüsselverwaltung: erzeugen, importieren, auswählen, schützen."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import ec, rsa
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.admin import AdminError, MountInput, delete_mount, save_mount
|
||||
from netmover.crypto import SecretBox, generate_ssh_keypair
|
||||
from netmover.db import session_scope
|
||||
from netmover.keys import delete_key, fingerprint, generate_key, import_key, list_keys, rename_key
|
||||
from netmover.models import Credential, CredentialKind, Mount, MountType
|
||||
from netmover.mounting.manager import MountConfig, MountManager
|
||||
from tests.sftp_stub import StubSFTPServer
|
||||
|
||||
|
||||
def _sftp_input(**kw: object) -> MountInput:
|
||||
data: dict[str, object] = {
|
||||
"name": "paperless",
|
||||
"type": MountType.SFTP,
|
||||
"server": "127.0.0.1",
|
||||
"share": "/consume",
|
||||
"port": 22,
|
||||
"username": "paperless",
|
||||
"auth": CredentialKind.SSH_KEY,
|
||||
}
|
||||
data.update(kw)
|
||||
return MountInput(**data) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -- Modul --------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_generate_key(session: Session, secretbox: SecretBox) -> None:
|
||||
key = generate_key(session, secretbox, "paperless-lxc")
|
||||
assert key.managed and key.kind is CredentialKind.SSH_KEY
|
||||
assert key.public_key and key.public_key.startswith("ssh-ed25519 ")
|
||||
assert key.public_key.endswith(" netmover:paperless-lxc")
|
||||
assert "PRIVATE" not in key.public_key
|
||||
assert "OPENSSH PRIVATE KEY" in secretbox.decrypt(key.secret_enc)
|
||||
info = list_keys(session)[0]
|
||||
assert info.fingerprint.startswith("SHA256:") and info.used_by == []
|
||||
assert info.authorized_keys_line.startswith(
|
||||
'restrict,command="/usr/lib/openssh/sftp-server" ssh-ed25519 '
|
||||
)
|
||||
|
||||
|
||||
def test_fingerprint_matches_ssh_keygen(tmp_path: Path) -> None:
|
||||
_, pub = generate_ssh_keypair("x")
|
||||
f = tmp_path / "k.pub"
|
||||
f.write_text(pub + "\n")
|
||||
try:
|
||||
out = subprocess.run(
|
||||
["ssh-keygen", "-lf", str(f)], # noqa: S607 – Abgleich mit OpenSSH
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout
|
||||
except FileNotFoundError:
|
||||
pytest.skip("ssh-keygen nicht installiert")
|
||||
assert fingerprint(pub) == out.split()[1]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"kind", ["ed25519-openssh-pass", "rsa-pem", "ecdsa-pem-pass", "rsa-openssh"]
|
||||
)
|
||||
def test_import_formats(session: Session, secretbox: SecretBox, kind: str) -> None:
|
||||
from cryptography.hazmat.primitives.asymmetric import ed25519
|
||||
|
||||
if kind.startswith("ed25519"):
|
||||
key = ed25519.Ed25519PrivateKey.generate()
|
||||
elif kind.startswith("rsa"):
|
||||
key = rsa.generate_private_key(public_exponent=65537, key_size=2048) # type: ignore[assignment]
|
||||
else:
|
||||
key = ec.generate_private_key(ec.SECP256R1()) # type: ignore[assignment]
|
||||
enc = (
|
||||
serialization.BestAvailableEncryption(b"geheim")
|
||||
if kind.endswith("pass")
|
||||
else serialization.NoEncryption()
|
||||
)
|
||||
fmt = (
|
||||
serialization.PrivateFormat.OpenSSH
|
||||
if "openssh" in kind
|
||||
else serialization.PrivateFormat.PKCS8
|
||||
)
|
||||
pem = key.private_bytes(serialization.Encoding.PEM, fmt, enc).decode()
|
||||
cred = import_key(session, secretbox, kind, pem, "geheim" if kind.endswith("pass") else "")
|
||||
expected_pub = (
|
||||
key.public_key()
|
||||
.public_bytes(serialization.Encoding.OpenSSH, serialization.PublicFormat.OpenSSH)
|
||||
.decode()
|
||||
)
|
||||
assert cred.public_key is not None and cred.public_key.startswith(expected_pub)
|
||||
stored = secretbox.decrypt(cred.secret_enc)
|
||||
assert "OPENSSH PRIVATE KEY" in stored
|
||||
# ohne Passphrase gespeichert → von paramiko direkt nutzbar
|
||||
from netmover.mounting.sftp import load_private_key
|
||||
|
||||
assert load_private_key(stored).get_base64() == expected_pub.split()[1]
|
||||
|
||||
|
||||
def test_import_errors(session: Session, secretbox: SecretBox) -> None:
|
||||
from cryptography.hazmat.primitives.asymmetric import ed25519
|
||||
|
||||
key = ed25519.Ed25519PrivateKey.generate()
|
||||
protected = key.private_bytes(
|
||||
serialization.Encoding.PEM,
|
||||
serialization.PrivateFormat.OpenSSH,
|
||||
serialization.BestAvailableEncryption(b"richtig"),
|
||||
).decode()
|
||||
with pytest.raises(AdminError, match="Passphrase angeben"):
|
||||
import_key(session, secretbox, "a", protected, "")
|
||||
with pytest.raises(AdminError, match="Passphrase falsch"):
|
||||
import_key(session, secretbox, "a", protected, "falsch")
|
||||
_, pub = generate_ssh_keypair()
|
||||
with pytest.raises(AdminError, match="öffentlicher Schlüssel"):
|
||||
import_key(session, secretbox, "a", pub)
|
||||
with pytest.raises(AdminError, match="Kein lesbarer"):
|
||||
import_key(
|
||||
session, secretbox, "a", "-----BEGIN PRIVATE KEY-----\nMüll\n-----END PRIVATE KEY-----"
|
||||
)
|
||||
weak = (
|
||||
rsa.generate_private_key(public_exponent=65537, key_size=1024) # noqa: S505 – absichtlich schwach
|
||||
.private_bytes(
|
||||
serialization.Encoding.PEM,
|
||||
serialization.PrivateFormat.PKCS8,
|
||||
serialization.NoEncryption(),
|
||||
)
|
||||
.decode()
|
||||
)
|
||||
with pytest.raises(AdminError, match="zu schwach"):
|
||||
import_key(session, secretbox, "a", weak)
|
||||
assert list_keys(session) == []
|
||||
|
||||
|
||||
def test_names(session: Session, secretbox: SecretBox) -> None:
|
||||
generate_key(session, secretbox, "eins")
|
||||
with pytest.raises(AdminError, match="existiert bereits"):
|
||||
generate_key(session, secretbox, "eins")
|
||||
with pytest.raises(AdminError, match="reserviert"):
|
||||
generate_key(session, secretbox, "mount:x")
|
||||
zwei = generate_key(session, secretbox, "zwei")
|
||||
rename_key(session, zwei.id, "drei")
|
||||
assert [k.name for k in list_keys(session)] == ["drei", "eins"]
|
||||
with pytest.raises(AdminError):
|
||||
rename_key(session, zwei.id, "eins")
|
||||
|
||||
|
||||
# -- Zusammenspiel mit Anbindungen ----------------------------------------------------------
|
||||
|
||||
|
||||
def test_mount_uses_shared_key_with_own_username(
|
||||
session: Session, secretbox: SecretBox, manager: MountManager
|
||||
) -> None:
|
||||
key = generate_key(session, secretbox, "lxc")
|
||||
a, err = save_mount(session, secretbox, manager, _sftp_input(key_id=str(key.id)))
|
||||
assert not err and a is not None
|
||||
b, err = save_mount(
|
||||
session,
|
||||
secretbox,
|
||||
manager,
|
||||
_sftp_input(name="anderer", server="10.0.0.9", username="root", key_id=str(key.id)),
|
||||
)
|
||||
assert not err and b is not None
|
||||
assert a.credential is key and b.credential is key
|
||||
assert (a.username, b.username) == ("paperless", "root")
|
||||
assert key.username == "" # der Schlüssel selbst bleibt unverändert
|
||||
assert list_keys(session)[0].used_by == ["anderer", "paperless"]
|
||||
|
||||
# Schlüssel in Benutzung → nicht löschbar
|
||||
with pytest.raises(AdminError, match="noch benutzt"):
|
||||
delete_key(session, key.id)
|
||||
# Anbindung löschen lässt den gespeicherten Schlüssel stehen
|
||||
delete_mount(session, manager, a.id)
|
||||
delete_mount(session, manager, b.id)
|
||||
assert session.get(Credential, key.id) is not None
|
||||
delete_key(session, key.id)
|
||||
assert session.get(Credential, key.id) is None
|
||||
|
||||
|
||||
def test_switching_between_own_and_shared_key(
|
||||
session: Session, secretbox: SecretBox, manager: MountManager
|
||||
) -> None:
|
||||
m, _ = save_mount(session, secretbox, manager, _sftp_input()) # eigener Schlüssel
|
||||
assert m is not None and m.credential is not None and not m.credential.managed
|
||||
own_id = m.credential.id
|
||||
key = generate_key(session, secretbox, "geteilt")
|
||||
m, _ = save_mount(session, secretbox, manager, _sftp_input(key_id=str(key.id)), m.id)
|
||||
assert m is not None and m.credential is key
|
||||
assert session.get(Credential, own_id) is None # eigener Schlüssel aufgeräumt
|
||||
# Wechsel auf Passwort darf den gespeicherten Schlüssel nicht verändern
|
||||
secret_before = key.secret_enc
|
||||
m, _ = save_mount(
|
||||
session,
|
||||
secretbox,
|
||||
manager,
|
||||
_sftp_input(auth=CredentialKind.PASSWORD, password="pw123"),
|
||||
m.id,
|
||||
)
|
||||
assert m is not None and m.credential is not None and m.credential is not key
|
||||
assert not m.credential.managed and m.credential.username == "paperless"
|
||||
assert m.username is None
|
||||
assert key.secret_enc == secret_before and key.kind is CredentialKind.SSH_KEY
|
||||
|
||||
|
||||
def test_unknown_or_foreign_key_id_is_rejected(
|
||||
session: Session, secretbox: SecretBox, manager: MountManager
|
||||
) -> None:
|
||||
own, _ = save_mount(session, secretbox, manager, _sftp_input(name="a"))
|
||||
assert own is not None and own.credential is not None
|
||||
for bad in ("4711", str(own.credential.id)): # nicht vorhanden / kein gespeicherter Schlüssel
|
||||
m, err = save_mount(session, secretbox, manager, _sftp_input(name="b", key_id=bad))
|
||||
assert m is None and "key_id" in err
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def server(tmp_path: Path) -> Iterator[StubSFTPServer]:
|
||||
root = tmp_path / "sftp-root"
|
||||
(root / "consume").mkdir(parents=True)
|
||||
srv = StubSFTPServer(root, username="paperless").start()
|
||||
yield srv
|
||||
srv.stop()
|
||||
|
||||
|
||||
def test_manager_connects_with_shared_key_and_mount_username(
|
||||
server: StubSFTPServer,
|
||||
session_factory: sessionmaker[Session],
|
||||
secretbox: SecretBox,
|
||||
manager: MountManager,
|
||||
) -> None:
|
||||
with session_scope(session_factory) as s:
|
||||
key = import_key(s, secretbox, "lxc", server.client_private_key)
|
||||
m, err = save_mount(
|
||||
s, secretbox, manager, _sftp_input(port=server.port, key_id=str(key.id))
|
||||
)
|
||||
assert not err and m is not None
|
||||
mid = m.id
|
||||
with session_factory() as s:
|
||||
mm = s.get(Mount, mid)
|
||||
assert mm is not None
|
||||
cfg = MountConfig.from_model(mm)
|
||||
from netmover.admin import credential_for_test, mount_input_from_model
|
||||
|
||||
with session_factory() as s:
|
||||
mm = s.get(Mount, mid)
|
||||
assert mm is not None
|
||||
data = mount_input_from_model(mm)
|
||||
assert data.key_id == str(mm.cred_id) and data.username == "paperless"
|
||||
cred = credential_for_test(data, mm, secretbox, s)
|
||||
assert cred is not None and cred.username == "paperless"
|
||||
assert manager.check(cfg, cred, mount_id=mid).ok
|
||||
assert manager.check_alive(mid) is None # Regellauf-Weg: Benutzer der Anbindung
|
||||
|
||||
# Gegenprobe: falscher Benutzer an der Anbindung → Anmeldung abgelehnt
|
||||
with session_scope(session_factory) as s:
|
||||
mm = s.get(Mount, mid)
|
||||
assert mm is not None
|
||||
mm.username = "jemand"
|
||||
from netmover.mounting.manager import MountUnavailable
|
||||
|
||||
with pytest.raises(MountUnavailable, match="Anmeldung abgelehnt"):
|
||||
manager.check_alive(mid)
|
||||
|
||||
|
||||
def test_list_keys_ignores_per_mount_credentials(
|
||||
session: Session, secretbox: SecretBox, manager: MountManager
|
||||
) -> None:
|
||||
save_mount(session, secretbox, manager, _sftp_input())
|
||||
assert list_keys(session) == []
|
||||
assert session.scalars(select(Credential)).one().managed is False
|
||||
@@ -110,8 +110,49 @@ def test_is_mounted_reads_mountinfo(tmp_path: Path) -> None:
|
||||
),
|
||||
("nfs4", "mount.nfs4: access denied by server while mounting nas:/x", "NFS-Berechtigungen"),
|
||||
("cifs", "mount error(112): Host is down", "nicht erreichbar"),
|
||||
# Praxisfall: Synology mit „Maximales SMB-Protokoll: SMB2“
|
||||
(
|
||||
"cifs",
|
||||
"mount error(95): Operation not supported\nRefer to the mount.cifs(8) manual page",
|
||||
"vers=2.0",
|
||||
),
|
||||
("cifs", "mount: only root can do that", "CAP_SYS_ADMIN"),
|
||||
],
|
||||
)
|
||||
def test_explain(fstype: str, msg: str, expect: str) -> None:
|
||||
assert expect in explain_mount_error(fstype, msg)
|
||||
|
||||
|
||||
def _alive(pid: int) -> bool:
|
||||
try:
|
||||
with open(f"/proc/{pid}/stat") as fh:
|
||||
return fh.read().split(")")[-1].split()[0] != "Z"
|
||||
except (FileNotFoundError, ProcessLookupError): # verschwindet beim Lesen → ESRCH
|
||||
return False
|
||||
|
||||
|
||||
def test_timeout_kills_whole_process_group(tmp_path: Path) -> None:
|
||||
"""mount startet Helfer (mount.nfs) – die dürfen nach dem Zeitlimit nicht weiterlaufen."""
|
||||
from netmover.mounting.linux import run_process_group
|
||||
|
||||
pidfile = tmp_path / "helper.pid"
|
||||
script = f"sleep 30 & echo $! > {pidfile}; wait"
|
||||
with pytest.raises(subprocess.TimeoutExpired):
|
||||
run_process_group(["sh", "-c", script], capture_output=True, text=True, timeout=0.5)
|
||||
helper = int(pidfile.read_text())
|
||||
for _ in range(20):
|
||||
if not _alive(helper):
|
||||
break
|
||||
import time
|
||||
|
||||
time.sleep(0.05)
|
||||
assert not _alive(helper)
|
||||
|
||||
|
||||
def test_process_group_runner_returns_output() -> None:
|
||||
from netmover.mounting.linux import run_process_group
|
||||
|
||||
r = run_process_group(
|
||||
["sh", "-c", "echo out; echo err >&2; exit 3"], capture_output=True, text=True, timeout=5
|
||||
)
|
||||
assert (r.returncode, r.stdout, r.stderr) == (3, "out\n", "err\n")
|
||||
|
||||
@@ -367,3 +367,159 @@ def test_usage(
|
||||
manager.mount(mid)
|
||||
usage = manager.usage(mid)
|
||||
assert usage is not None and usage[0] >= usage[1] > 0
|
||||
|
||||
|
||||
# -- Praxisbefund: NAS-Neustart dauert länger als die Wiederherstellung ----------------------
|
||||
|
||||
|
||||
class Clock:
|
||||
def __init__(self) -> None:
|
||||
self.now = 1000.0
|
||||
|
||||
def __call__(self) -> float:
|
||||
return self.now
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clock() -> Clock:
|
||||
return Clock()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def retry_manager(
|
||||
session_factory: sessionmaker[Session],
|
||||
fake_backend: FakeMountBackend,
|
||||
secretbox: object,
|
||||
mount_base: Path,
|
||||
notifications: list[tuple[str, str, str]],
|
||||
clock: Clock,
|
||||
) -> MountManager:
|
||||
return MountManager(
|
||||
session_factory,
|
||||
fake_backend,
|
||||
secretbox,
|
||||
mount_base, # type: ignore[arg-type]
|
||||
probe_timeout=0.5,
|
||||
recover_backoff=(0, 0, 0),
|
||||
sleep=lambda _: None,
|
||||
notifier=lambda *a: notifications.append(a),
|
||||
retry_interval=60,
|
||||
clock=clock,
|
||||
)
|
||||
|
||||
|
||||
def test_failed_recovery_retries_every_interval_until_nas_is_back(
|
||||
retry_manager: MountManager,
|
||||
fake_backend: FakeMountBackend,
|
||||
make_mount: MountFactory,
|
||||
session_factory: sessionmaker[Session],
|
||||
notifications: list[tuple[str, str, str]],
|
||||
clock: Clock,
|
||||
) -> None:
|
||||
fake_backend.export("nas", "scans")
|
||||
mid = make_mount()
|
||||
retry_manager.mount(mid)
|
||||
mp = retry_manager.mountpoint_for("scans")
|
||||
fake_backend.inject_fault(mp, errno.ESTALE)
|
||||
fake_backend.fail_next_mount(times=3) # NAS startet neu
|
||||
with pytest.raises(MountUnavailable, match="alle 60 s"):
|
||||
retry_manager.check_alive(mid)
|
||||
assert not fake_backend.is_mounted(mp)
|
||||
|
||||
mounts_before = sum(1 for c in fake_backend.calls if c.action == "mount")
|
||||
clock.now += 30 # zu früh: kein Versuch, Lauf abgebrochen
|
||||
with pytest.raises(MountUnavailable, match="in 30 s"):
|
||||
retry_manager.check_alive(mid)
|
||||
assert sum(1 for c in fake_backend.calls if c.action == "mount") == mounts_before
|
||||
|
||||
clock.now += 30 # NAS noch nicht zurück
|
||||
fake_backend.fail_next_mount()
|
||||
with pytest.raises(MountUnavailable, match="weiterhin nicht verfügbar"):
|
||||
retry_manager.check_alive(mid)
|
||||
|
||||
clock.now += 60 # NAS wieder da
|
||||
assert retry_manager.check_alive(mid) == mp
|
||||
assert fake_backend.is_mounted(mp)
|
||||
assert _state(session_factory, mid).state is MountState.MOUNTED
|
||||
assert notifications[-1][0] == "mount.recovered"
|
||||
|
||||
|
||||
def test_no_retry_without_auto_mount(
|
||||
retry_manager: MountManager,
|
||||
fake_backend: FakeMountBackend,
|
||||
make_mount: MountFactory,
|
||||
clock: Clock,
|
||||
) -> None:
|
||||
fake_backend.export("nas", "scans")
|
||||
mid = make_mount(auto_mount=False)
|
||||
retry_manager.mount(mid)
|
||||
fake_backend.inject_fault(retry_manager.mountpoint_for("scans"), errno.ESTALE)
|
||||
fake_backend.fail_next_mount(times=3)
|
||||
with pytest.raises(MountUnavailable):
|
||||
retry_manager.check_alive(mid)
|
||||
clock.now += 600
|
||||
with pytest.raises(MountUnavailable, match="nicht eingehängt"):
|
||||
retry_manager.check_alive(mid)
|
||||
|
||||
|
||||
def test_externally_unmounted_is_still_never_remounted(
|
||||
retry_manager: MountManager,
|
||||
fake_backend: FakeMountBackend,
|
||||
make_mount: MountFactory,
|
||||
clock: Clock,
|
||||
) -> None:
|
||||
fake_backend.export("nas", "scans")
|
||||
mid = make_mount()
|
||||
retry_manager.mount(mid)
|
||||
fake_backend.vanish(retry_manager.mountpoint_for("scans"))
|
||||
clock.now += 600
|
||||
with pytest.raises(MountUnavailable, match="nicht eingehängt"):
|
||||
retry_manager.check_alive(mid)
|
||||
assert [c.action for c in fake_backend.calls] == ["mount"]
|
||||
|
||||
|
||||
def test_hint_text_is_not_redacted() -> None:
|
||||
from netmover.mounting.manager import _redact_output
|
||||
|
||||
msg = "mount error(13): Permission denied falsch\n→ Benutzername/Passwort falsch oder …"
|
||||
out = _redact_output(msg, ["falsch"])
|
||||
assert out.startswith("mount error(13): Permission denied ••••••")
|
||||
assert out.endswith("→ Benutzername/Passwort falsch oder …")
|
||||
|
||||
|
||||
def test_restore_failure_at_boot_is_retried_by_background_job(
|
||||
retry_manager: MountManager,
|
||||
fake_backend: FakeMountBackend,
|
||||
make_mount: MountFactory,
|
||||
session_factory: sessionmaker[Session],
|
||||
notifications: list[tuple[str, str, str]],
|
||||
clock: Clock,
|
||||
) -> None:
|
||||
"""Stromausfall: die VM bootet schneller als das NAS."""
|
||||
fake_backend.export("nas", "scans")
|
||||
mid = make_mount() # keine Regel benutzt die Anbindung
|
||||
fake_backend.fail_next_mount()
|
||||
assert [r.ok for r in retry_manager.restore_all()] == [False]
|
||||
assert retry_manager.retry_pending() == [] # noch nicht fällig
|
||||
clock.now += 60
|
||||
assert retry_manager.retry_pending() == ["scans"]
|
||||
assert fake_backend.is_mounted(retry_manager.mountpoint_for("scans"))
|
||||
assert _state(session_factory, mid).state is MountState.MOUNTED
|
||||
assert notifications[-1][0] == "mount.recovered"
|
||||
clock.now += 60
|
||||
assert retry_manager.retry_pending() == [] # erledigt
|
||||
|
||||
|
||||
def test_manual_unmount_stops_retrying(
|
||||
retry_manager: MountManager,
|
||||
fake_backend: FakeMountBackend,
|
||||
make_mount: MountFactory,
|
||||
clock: Clock,
|
||||
) -> None:
|
||||
mid = make_mount() # nicht exportiert → Wiederherstellung scheitert
|
||||
retry_manager.restore_all()
|
||||
retry_manager.unmount(mid)
|
||||
fake_backend.export("nas", "scans")
|
||||
clock.now += 120
|
||||
assert retry_manager.retry_pending() == []
|
||||
assert not fake_backend.mounted
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
"""Migrationen auf Datenbanken MIT Daten.
|
||||
|
||||
Praxisbefund beim Update 0.1.1 → 0.2.0: Migration 0002 baut auf SQLite Tabellen neu
|
||||
auf (batch mode) und löscht dafür die alte Tabelle. Mit eingeschalteter
|
||||
Fremdschlüsselprüfung scheiterte das, sobald Anbindungen auf Zugangsdaten
|
||||
verwiesen – die Tests liefen bis dahin nur auf leeren Datenbanken. Und weil die
|
||||
Migration nicht in einer Transaktion lief, blieb ein halber Zustand zurück.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from alembic import command
|
||||
|
||||
from netmover.db import alembic_config, upgrade
|
||||
|
||||
|
||||
def _seed(db: Path) -> None:
|
||||
"""Datenbank auf Stand 0001 mit verknüpften Daten, wie auf einem echten System."""
|
||||
c = sqlite3.connect(db)
|
||||
c.execute("PRAGMA foreign_keys=ON")
|
||||
c.executescript(
|
||||
"""
|
||||
INSERT INTO credentials (id, name, kind, username, secret_enc, created_at)
|
||||
VALUES (1, 'mount:nas', 'password', 'scanner', 'gAAAA-x', '2026-10-10 20:00:00'),
|
||||
(2, 'mount:paperless', 'ssh_key', 'root', 'gAAAA-y', '2026-10-10 20:00:00');
|
||||
INSERT INTO mounts (id, name, slug, type, server, share, mountpoint, cred_id,
|
||||
extra_options, auto_mount, enabled, state)
|
||||
VALUES (1, 'NAS', 'nas', 'cifs', '10.0.10.6', 'Scans', '/mnt/netmover/nas', 1,
|
||||
'', 1, 1, 'mounted'),
|
||||
(2, 'paperless', 'paperless', 'sftp', '10.0.10.17', '/consume', NULL, 2,
|
||||
'', 0, 1, 'mounted');
|
||||
INSERT INTO rules (id, name, enabled, src_mount_id, src_subpath, dst_mount_id, dst_path,
|
||||
pattern, pattern_type, recursive, interval_seconds, stability_seconds,
|
||||
min_age_seconds, op_timeout_seconds, action, on_collision, dry_run,
|
||||
quarantine_after, consecutive_failures, created_at)
|
||||
VALUES (1, 'Scans → Paperless', 0, 1, '', 2, '', '*.pdf', 'glob', 0, 30, 5, 0, 120,
|
||||
'move', 'rename', 0, 3, 0, '2026-10-10 20:00:00');
|
||||
INSERT INTO runs (id, rule_id, started_at, status, "trigger", dry_run, seen, moved,
|
||||
skipped, failed)
|
||||
VALUES (1, 1, '2026-10-10 20:00:00', 'ok', 'manual', 1, 3, 3, 0, 0);
|
||||
INSERT INTO run_items (run_id, src, status) VALUES (1, 'a.pdf', 'planned');
|
||||
"""
|
||||
)
|
||||
c.commit()
|
||||
c.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db_0001(tmp_path: Path) -> tuple[str, Path]:
|
||||
db = tmp_path / "alt.db"
|
||||
url = f"sqlite:///{db}"
|
||||
upgrade(url, "0001")
|
||||
_seed(db)
|
||||
return url, db
|
||||
|
||||
|
||||
def _state(db: Path) -> dict[str, object]:
|
||||
c = sqlite3.connect(db)
|
||||
try:
|
||||
return {
|
||||
"version": c.execute("SELECT version_num FROM alembic_version").fetchone()[0],
|
||||
"tmp_tables": c.execute(
|
||||
"SELECT name FROM sqlite_master WHERE name LIKE '_alembic_tmp%'"
|
||||
).fetchall(),
|
||||
"cred_columns": [r[1] for r in c.execute("PRAGMA table_info(credentials)")],
|
||||
"mounts": c.execute("SELECT name, cred_id FROM mounts ORDER BY id").fetchall(),
|
||||
"rules": c.execute("SELECT name, src_mount_id, dst_mount_id FROM rules").fetchall(),
|
||||
"run_items": c.execute("SELECT COUNT(*) FROM run_items").fetchone()[0],
|
||||
"fk_violations": c.execute("PRAGMA foreign_key_check").fetchall(),
|
||||
}
|
||||
finally:
|
||||
c.close()
|
||||
|
||||
|
||||
def test_upgrade_with_linked_data(db_0001: tuple[str, Path]) -> None:
|
||||
url, db = db_0001
|
||||
upgrade(url)
|
||||
st = _state(db)
|
||||
assert st["version"] == "0002"
|
||||
assert st["tmp_tables"] == [] and st["fk_violations"] == []
|
||||
assert "managed" in st["cred_columns"] # type: ignore[operator]
|
||||
assert st["mounts"] == [("NAS", 1), ("paperless", 2)]
|
||||
assert st["rules"] == [("Scans → Paperless", 1, 2)]
|
||||
assert st["run_items"] == 1
|
||||
c = sqlite3.connect(db)
|
||||
assert c.execute("SELECT managed FROM credentials ORDER BY id").fetchall() == [(0,), (0,)]
|
||||
assert c.execute("SELECT username FROM mounts ORDER BY id").fetchall() == [(None,), (None,)]
|
||||
c.close()
|
||||
|
||||
|
||||
def test_downgrade_with_linked_data(db_0001: tuple[str, Path]) -> None:
|
||||
url, db = db_0001
|
||||
upgrade(url)
|
||||
command.downgrade(alembic_config(url), "0001")
|
||||
st = _state(db)
|
||||
assert st["version"] == "0001" and st["fk_violations"] == []
|
||||
assert st["mounts"] == [("NAS", 1), ("paperless", 2)]
|
||||
|
||||
|
||||
def test_failed_migration_leaves_database_untouched(
|
||||
db_0001: tuple[str, Path], monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Bricht eine Migration mittendrin ab, muss alles beim Alten bleiben – keine
|
||||
halb umgebauten Tabellen, kein _alembic_tmp-Rest, Version unverändert."""
|
||||
from alembic.ddl.impl import DefaultImpl
|
||||
|
||||
url, db = db_0001
|
||||
before = _state(db)
|
||||
real_drop = DefaultImpl.drop_table
|
||||
|
||||
def boom(self: DefaultImpl, *a: object, **kw: object) -> None:
|
||||
raise RuntimeError("Absturz mitten in der Migration")
|
||||
|
||||
monkeypatch.setattr(DefaultImpl, "drop_table", boom)
|
||||
with pytest.raises(RuntimeError, match="Absturz"):
|
||||
upgrade(url)
|
||||
assert _state(db) == before
|
||||
|
||||
monkeypatch.setattr(DefaultImpl, "drop_table", real_drop)
|
||||
upgrade(url) # danach klappt es ohne Handarbeit
|
||||
assert _state(db)["version"] == "0002"
|
||||
|
||||
|
||||
def test_app_engine_still_enforces_foreign_keys(db_url: str) -> None:
|
||||
from sqlalchemy import text
|
||||
|
||||
from netmover.db import make_engine
|
||||
|
||||
engine = make_engine(db_url)
|
||||
with engine.connect() as conn:
|
||||
assert conn.execute(text("PRAGMA foreign_keys")).scalar() == 1
|
||||
engine.dispose()
|
||||
@@ -0,0 +1,395 @@
|
||||
"""Protokollierung, Regelzähler, Benachrichtigung, Aufräumen, Health (Fallstrick 9)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.db import session_scope
|
||||
from netmover.health import check_health
|
||||
from netmover.models import (
|
||||
FileFailure,
|
||||
ItemStatus,
|
||||
Mount,
|
||||
MountState,
|
||||
Rule,
|
||||
Run,
|
||||
RunItem,
|
||||
RunStatus,
|
||||
RunTrigger,
|
||||
utcnow,
|
||||
)
|
||||
from netmover.mover.engine import ItemResult, RunResult
|
||||
from netmover.mover.runner import RuleRunner
|
||||
from netmover.notify import Notification, Notifier, render
|
||||
from netmover.runs import RunService, cleanup
|
||||
from netmover.settings_store import AppSettings, NotifyMode, load_settings, save_settings
|
||||
from tests.conftest import MountFactory
|
||||
|
||||
|
||||
class FakeRunner:
|
||||
def __init__(self) -> None:
|
||||
self.results: list[RunResult] = []
|
||||
|
||||
def run(self, rule_id: int, *, dry_run: bool | None = None) -> RunResult:
|
||||
return self.results.pop(0)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def rule_id(make_mount: MountFactory, session_factory: sessionmaker[Session]) -> int:
|
||||
mid = make_mount()
|
||||
with session_scope(session_factory) as s:
|
||||
r = Rule(name="scans", src_mount_id=mid, dst_path="/x", interval_seconds=30)
|
||||
s.add(r)
|
||||
s.flush()
|
||||
return r.id
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sent() -> list[tuple[str, str, str]]:
|
||||
return []
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def notifier(session_factory: sessionmaker[Session], sent: list[tuple[str, str, str]]) -> Notifier:
|
||||
return Notifier(session_factory, sender=lambda *a: sent.append(a), background=False)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_runner() -> FakeRunner:
|
||||
return FakeRunner()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def service(
|
||||
session_factory: sessionmaker[Session], fake_runner: FakeRunner, notifier: Notifier
|
||||
) -> RunService:
|
||||
return RunService(session_factory, fake_runner, notifier) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _rule(sf: sessionmaker[Session], rid: int) -> Rule:
|
||||
with sf() as s:
|
||||
r = s.get(Rule, rid)
|
||||
assert r is not None
|
||||
s.expunge(r)
|
||||
return r
|
||||
|
||||
|
||||
def _enable_webhook(sf: sessionmaker[Session], mode: NotifyMode, after: int = 3) -> None:
|
||||
save_settings(
|
||||
sf, AppSettings(webhook_url="http://hook.invalid/x", notify_mode=mode, notify_after=after)
|
||||
)
|
||||
|
||||
|
||||
def ok() -> RunResult:
|
||||
return RunResult(
|
||||
RunStatus.OK, seen=1, moved=1, items=[ItemResult("a.pdf", ItemStatus.MOVED, "/x/a.pdf", 3)]
|
||||
)
|
||||
|
||||
|
||||
def err() -> RunResult:
|
||||
return RunResult(RunStatus.ERROR, message="„scans“ ist nicht eingehängt")
|
||||
|
||||
|
||||
def partial() -> RunResult:
|
||||
return RunResult(
|
||||
RunStatus.PARTIAL,
|
||||
seen=2,
|
||||
moved=1,
|
||||
failed=1,
|
||||
items=[ItemResult("b.pdf", ItemStatus.FAILED, None, 1, "kaputt (Versuch 1)")],
|
||||
)
|
||||
|
||||
|
||||
def skipped() -> RunResult:
|
||||
return RunResult(RunStatus.SKIPPED, message="läuft noch")
|
||||
|
||||
|
||||
def test_run_and_items_are_recorded(
|
||||
service: RunService,
|
||||
fake_runner: FakeRunner,
|
||||
rule_id: int,
|
||||
session_factory: sessionmaker[Session],
|
||||
) -> None:
|
||||
fake_runner.results.append(ok())
|
||||
run_id = service.execute(rule_id)
|
||||
with session_factory() as s:
|
||||
run = s.get(Run, run_id)
|
||||
assert run is not None
|
||||
assert (run.status, run.seen, run.moved, run.trigger) == (
|
||||
RunStatus.OK,
|
||||
1,
|
||||
1,
|
||||
RunTrigger.SCHEDULE,
|
||||
)
|
||||
assert run.finished_at is not None and run.finished_at >= run.started_at
|
||||
items = s.scalars(select(RunItem)).all()
|
||||
assert [(i.src, i.dst, i.status) for i in items] == [
|
||||
("a.pdf", "/x/a.pdf", ItemStatus.MOVED)
|
||||
]
|
||||
r = _rule(session_factory, rule_id)
|
||||
assert r.last_success_at is not None and r.consecutive_failures == 0
|
||||
|
||||
|
||||
def test_counters(
|
||||
service: RunService,
|
||||
fake_runner: FakeRunner,
|
||||
rule_id: int,
|
||||
session_factory: sessionmaker[Session],
|
||||
) -> None:
|
||||
fake_runner.results += [err(), partial(), skipped()]
|
||||
service.execute(rule_id)
|
||||
r = _rule(session_factory, rule_id)
|
||||
assert r.consecutive_failures == 1 and r.last_success_at is None and r.last_run_at is not None
|
||||
service.execute(rule_id)
|
||||
r = _rule(session_factory, rule_id)
|
||||
assert r.consecutive_failures == 2 and r.last_success_at is not None # partial: Werkzeug lebt
|
||||
last_run = r.last_run_at
|
||||
service.execute(rule_id) # übersprungen: protokolliert, Zähler unverändert
|
||||
r = _rule(session_factory, rule_id)
|
||||
assert r.consecutive_failures == 2 and r.last_run_at == last_run
|
||||
with session_factory() as s:
|
||||
assert [x.status for x in s.scalars(select(Run).order_by(Run.id))] == [
|
||||
RunStatus.ERROR,
|
||||
RunStatus.PARTIAL,
|
||||
RunStatus.SKIPPED,
|
||||
]
|
||||
|
||||
|
||||
def test_manual_dry_run_does_not_touch_counters(
|
||||
service: RunService,
|
||||
fake_runner: FakeRunner,
|
||||
rule_id: int,
|
||||
session_factory: sessionmaker[Session],
|
||||
) -> None:
|
||||
fake_runner.results.append(RunResult(RunStatus.ERROR, dry_run=True, message="x"))
|
||||
service.execute(rule_id, RunTrigger.MANUAL, dry_run=True)
|
||||
r = _rule(session_factory, rule_id)
|
||||
assert r.consecutive_failures == 0 and r.last_run_at is None
|
||||
|
||||
|
||||
def test_deleted_rule_is_not_recorded(service: RunService, fake_runner: FakeRunner) -> None:
|
||||
fake_runner.results.append(err())
|
||||
assert service.execute(4711) is None
|
||||
|
||||
|
||||
def test_items_are_capped(
|
||||
service: RunService,
|
||||
fake_runner: FakeRunner,
|
||||
rule_id: int,
|
||||
session_factory: sessionmaker[Session],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr("netmover.runs.MAX_ITEMS_PER_RUN", 3)
|
||||
fake_runner.results.append(
|
||||
RunResult(RunStatus.OK, items=[ItemResult(f"{i}.pdf", ItemStatus.MOVED) for i in range(5)])
|
||||
)
|
||||
run_id = service.execute(rule_id)
|
||||
with session_factory() as s:
|
||||
assert len(s.scalars(select(RunItem)).all()) == 3
|
||||
run = s.get(Run, run_id)
|
||||
assert run is not None and "2 weitere" in (run.message or "")
|
||||
|
||||
|
||||
# -- Benachrichtigung -------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_notify_every_failure(
|
||||
service: RunService,
|
||||
fake_runner: FakeRunner,
|
||||
rule_id: int,
|
||||
session_factory: sessionmaker[Session],
|
||||
sent: list[tuple[str, str, str]],
|
||||
) -> None:
|
||||
_enable_webhook(session_factory, NotifyMode.EVERY)
|
||||
fake_runner.results += [err(), err(), ok()]
|
||||
for _ in range(3):
|
||||
service.execute(rule_id)
|
||||
bodies = [json.loads(b) for _, _, b in sent]
|
||||
assert [b["event"] for b in bodies] == ["rule.failed", "rule.failed", "rule.recovered"]
|
||||
assert bodies[0]["message"] == "„scans“ ist nicht eingehängt"
|
||||
assert "2× in Folge" in bodies[1]["title"]
|
||||
|
||||
|
||||
def test_notify_after_n_only_once_at_threshold(
|
||||
service: RunService,
|
||||
fake_runner: FakeRunner,
|
||||
rule_id: int,
|
||||
session_factory: sessionmaker[Session],
|
||||
sent: list[tuple[str, str, str]],
|
||||
) -> None:
|
||||
_enable_webhook(session_factory, NotifyMode.AFTER_N, after=3)
|
||||
fake_runner.results += [err(), err(), err(), err(), partial(), ok()]
|
||||
events = []
|
||||
for _ in range(6):
|
||||
service.execute(rule_id)
|
||||
events.append(len(sent))
|
||||
assert events == [0, 0, 1, 1, 1, 2]
|
||||
assert json.loads(sent[-1][2])["event"] == "rule.recovered"
|
||||
|
||||
|
||||
def test_no_recovery_message_below_threshold(
|
||||
service: RunService,
|
||||
fake_runner: FakeRunner,
|
||||
rule_id: int,
|
||||
session_factory: sessionmaker[Session],
|
||||
sent: list[tuple[str, str, str]],
|
||||
) -> None:
|
||||
_enable_webhook(session_factory, NotifyMode.AFTER_N, after=3)
|
||||
fake_runner.results += [err(), ok()]
|
||||
service.execute(rule_id)
|
||||
service.execute(rule_id)
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_notify_off_by_default(
|
||||
service: RunService, fake_runner: FakeRunner, rule_id: int, sent: list[tuple[str, str, str]]
|
||||
) -> None:
|
||||
fake_runner.results.append(err())
|
||||
service.execute(rule_id)
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_render_json_escapes_values() -> None:
|
||||
n = Notification("rule.failed", 'Titel mit "Anführung"', "Zeile 1\nZeile 2 \\ $x", "r")
|
||||
body = render('{"t": "$title", "m": "$message", "x": "$unbekannt"}', "application/json", n)
|
||||
data = json.loads(body)
|
||||
assert data == {"t": 'Titel mit "Anführung"', "m": "Zeile 1\nZeile 2 \\ $x", "x": "$unbekannt"}
|
||||
assert render("$message", "text/plain", n) == "Zeile 1\nZeile 2 \\ $x"
|
||||
|
||||
|
||||
def test_sender_failure_is_swallowed(session_factory: sessionmaker[Session]) -> None:
|
||||
def broken(*a: object) -> None:
|
||||
raise ConnectionError("weg")
|
||||
|
||||
_enable_webhook(session_factory, NotifyMode.EVERY)
|
||||
Notifier(session_factory, sender=broken, background=False).send(Notification("x", "t", "m"))
|
||||
|
||||
|
||||
def test_settings_roundtrip(session_factory: sessionmaker[Session]) -> None:
|
||||
assert load_settings(session_factory) == AppSettings()
|
||||
save_settings(session_factory, AppSettings(retention_days=7, notify_mode=NotifyMode.EVERY))
|
||||
loaded = load_settings(session_factory)
|
||||
assert loaded.retention_days == 7 and loaded.notify_mode is NotifyMode.EVERY
|
||||
|
||||
|
||||
# -- Aufräumen ---------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cleanup_removes_old_runs_with_items(
|
||||
service: RunService,
|
||||
fake_runner: FakeRunner,
|
||||
rule_id: int,
|
||||
session_factory: sessionmaker[Session],
|
||||
) -> None:
|
||||
fake_runner.results += [ok(), ok()]
|
||||
old = service.execute(rule_id)
|
||||
new = service.execute(rule_id)
|
||||
with session_scope(session_factory) as s:
|
||||
run = s.get(Run, old)
|
||||
assert run is not None
|
||||
run.started_at = utcnow() - timedelta(days=31)
|
||||
s.add(
|
||||
FileFailure(
|
||||
rule_id=rule_id,
|
||||
src_relpath="x",
|
||||
size=1,
|
||||
mtime_ns=1,
|
||||
attempts=1,
|
||||
last_attempt_at=utcnow() - timedelta(days=40),
|
||||
)
|
||||
)
|
||||
stats = cleanup(session_factory)
|
||||
assert stats == {"runs": 1, "file_failures": 1}
|
||||
with session_factory() as s:
|
||||
assert [r.id for r in s.scalars(select(Run))] == [new]
|
||||
assert len(s.scalars(select(RunItem)).all()) == 1
|
||||
|
||||
|
||||
# -- Health (Fallstrick 9) -------------------------------------------------------------------
|
||||
|
||||
|
||||
def _health(sf: sessionmaker[Session], started_ago: float = 3600, **kw: object): # type: ignore[no-untyped-def]
|
||||
now = utcnow()
|
||||
args: dict[str, object] = {
|
||||
"scheduler_running": True,
|
||||
"last_heartbeat": now,
|
||||
"service_started": now - timedelta(seconds=started_ago),
|
||||
"now": now,
|
||||
}
|
||||
args.update(kw)
|
||||
return check_health(sf, **args) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _age_rule(sf: sessionmaker[Session], rid: int, seconds: float | None) -> None:
|
||||
with session_scope(sf) as s:
|
||||
r = s.get(Rule, rid)
|
||||
assert r is not None
|
||||
r.created_at = utcnow() - timedelta(days=1)
|
||||
r.last_success_at = None if seconds is None else utcnow() - timedelta(seconds=seconds)
|
||||
|
||||
|
||||
def test_health_ok(session_factory: sessionmaker[Session], rule_id: int) -> None:
|
||||
_age_rule(session_factory, rule_id, 60) # 2× Intervall
|
||||
h = _health(session_factory)
|
||||
assert h.ok, h.problems
|
||||
|
||||
|
||||
def test_health_stale_rule_after_three_intervals(
|
||||
session_factory: sessionmaker[Session], rule_id: int
|
||||
) -> None:
|
||||
_age_rule(session_factory, rule_id, 91)
|
||||
h = _health(session_factory)
|
||||
assert not h.ok and h.stale_rules == {rule_id}
|
||||
assert "kein erfolgreicher Lauf" in h.problems[0]
|
||||
|
||||
|
||||
def test_health_never_succeeded_but_service_just_started(
|
||||
session_factory: sessionmaker[Session], rule_id: int
|
||||
) -> None:
|
||||
_age_rule(session_factory, rule_id, None)
|
||||
assert _health(session_factory, started_ago=30).ok
|
||||
assert not _health(session_factory, started_ago=120).ok
|
||||
|
||||
|
||||
def test_health_disabled_rule_is_ignored(
|
||||
session_factory: sessionmaker[Session], rule_id: int
|
||||
) -> None:
|
||||
_age_rule(session_factory, rule_id, None)
|
||||
with session_scope(session_factory) as s:
|
||||
r = s.get(Rule, rule_id)
|
||||
assert r is not None
|
||||
r.enabled = False
|
||||
assert _health(session_factory).ok
|
||||
|
||||
|
||||
def test_health_mount_error_only_for_enabled_mounts(
|
||||
session_factory: sessionmaker[Session], make_mount: MountFactory
|
||||
) -> None:
|
||||
mid = make_mount("b", share="b", enabled=False)
|
||||
with session_scope(session_factory) as s:
|
||||
m = s.get(Mount, mid)
|
||||
assert m is not None
|
||||
m.state = MountState.ERROR
|
||||
assert _health(session_factory).ok
|
||||
with session_scope(session_factory) as s:
|
||||
m = s.get(Mount, mid)
|
||||
assert m is not None
|
||||
m.enabled = True
|
||||
h = _health(session_factory)
|
||||
assert not h.ok and "Fehlerzustand" in h.problems[0]
|
||||
|
||||
|
||||
def test_health_scheduler(session_factory: sessionmaker[Session]) -> None:
|
||||
assert not _health(session_factory, scheduler_running=False).ok
|
||||
assert not _health(session_factory, last_heartbeat=utcnow() - timedelta(minutes=5)).ok
|
||||
|
||||
|
||||
def test_runner_type_is_compatible(session_factory: sessionmaker[Session], tmp_path: Path) -> None:
|
||||
# FakeRunner oben muss dieselbe Schnittstelle bieten wie RuleRunner.run
|
||||
assert RuleRunner.run.__code__.co_kwonlyargcount == 1
|
||||
@@ -0,0 +1,177 @@
|
||||
"""Scheduler: Jobs je Regel, Versatz, Neuabgleich, manuelle Läufe, Dienststart."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import random
|
||||
from collections.abc import Callable, Coroutine
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from netmover.app_state import NetMover
|
||||
from netmover.config import Settings
|
||||
from netmover.db import session_scope
|
||||
from netmover.models import Mount, MountState, MountType, Rule, Run, RunStatus, RunTrigger, utcnow
|
||||
from netmover.mounting.fake import FakeMountBackend
|
||||
from netmover.mounting.manager import MountManager
|
||||
from netmover.mover.runner import RuleRunner
|
||||
from netmover.runs import RunService
|
||||
from netmover.scheduler import MAX_START_OFFSET, RuleScheduler
|
||||
from tests.conftest import MountFactory
|
||||
|
||||
|
||||
def arun(fn: Callable[[], Coroutine[Any, Any, None]]) -> None:
|
||||
asyncio.run(fn())
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def rules(
|
||||
make_mount: MountFactory,
|
||||
session_factory: sessionmaker[Session],
|
||||
fake_backend: FakeMountBackend,
|
||||
manager: MountManager,
|
||||
) -> list[int]:
|
||||
fake_backend.export("nas", "scans")
|
||||
mid = make_mount()
|
||||
manager.mount(mid)
|
||||
ids = []
|
||||
with session_scope(session_factory) as s:
|
||||
for name, interval, enabled in [("a", 30, True), ("b", 60, True), ("c", 30, False)]:
|
||||
r = Rule(
|
||||
name=name,
|
||||
src_mount_id=mid,
|
||||
dst_path=str(fake_backend.remote_root),
|
||||
interval_seconds=interval,
|
||||
enabled=enabled,
|
||||
stability_seconds=0,
|
||||
)
|
||||
s.add(r)
|
||||
s.flush()
|
||||
ids.append(r.id)
|
||||
return ids
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def scheduler(session_factory: sessionmaker[Session], manager: MountManager) -> RuleScheduler:
|
||||
runner = RuleRunner(session_factory, manager, sleep=lambda _: None)
|
||||
return RuleScheduler(session_factory, RunService(session_factory, runner), rng=random.Random(1)) # noqa: S311
|
||||
|
||||
|
||||
def test_jobs_follow_rules(
|
||||
scheduler: RuleScheduler, rules: list[int], session_factory: sessionmaker[Session]
|
||||
) -> None:
|
||||
a, b, c = rules
|
||||
|
||||
async def main() -> None:
|
||||
scheduler.start()
|
||||
try:
|
||||
assert scheduler.next_run(a) is not None
|
||||
assert scheduler.next_run(b) is not None
|
||||
assert scheduler.next_run(c) is None
|
||||
# erster Termin mit kleinem Versatz
|
||||
for rid in (a, b):
|
||||
nxt = scheduler.next_run(rid)
|
||||
assert nxt is not None
|
||||
# Versatz ≤ 10 s plus Jitter ≤ 5 s
|
||||
assert nxt - utcnow() <= timedelta(seconds=MAX_START_OFFSET + 5 + 1)
|
||||
|
||||
with session_scope(session_factory) as s:
|
||||
ra, rc = s.get(Rule, a), s.get(Rule, c)
|
||||
assert ra is not None and rc is not None
|
||||
ra.enabled = False
|
||||
rc.enabled = True
|
||||
rb = s.get(Rule, b)
|
||||
assert rb is not None
|
||||
rb.interval_seconds = 120
|
||||
scheduler.sync()
|
||||
assert scheduler.next_run(a) is None
|
||||
assert scheduler.next_run(c) is not None
|
||||
job = scheduler.scheduler.get_job(f"rule:{b}")
|
||||
assert job is not None and job.trigger.interval == timedelta(seconds=120)
|
||||
finally:
|
||||
scheduler.shutdown()
|
||||
|
||||
arun(main)
|
||||
|
||||
|
||||
def test_run_now_records_manual_run(
|
||||
scheduler: RuleScheduler, rules: list[int], session_factory: sessionmaker[Session]
|
||||
) -> None:
|
||||
a = rules[0]
|
||||
|
||||
async def main() -> None:
|
||||
scheduler.start()
|
||||
try:
|
||||
run_id = await scheduler.run_now(a, dry_run=True)
|
||||
finally:
|
||||
scheduler.shutdown()
|
||||
with session_factory() as s:
|
||||
run = s.get(Run, run_id)
|
||||
assert run is not None
|
||||
assert run.trigger is RunTrigger.MANUAL and run.dry_run and run.status is RunStatus.OK
|
||||
|
||||
arun(main)
|
||||
|
||||
|
||||
def test_scheduled_job_runs_in_thread_pool(
|
||||
scheduler: RuleScheduler, rules: list[int], session_factory: sessionmaker[Session]
|
||||
) -> None:
|
||||
a = rules[0]
|
||||
|
||||
async def main() -> None:
|
||||
scheduler.start()
|
||||
try:
|
||||
await scheduler._run_rule(a)
|
||||
assert scheduler.running and scheduler.last_heartbeat is not None
|
||||
finally:
|
||||
scheduler.shutdown()
|
||||
with session_factory() as s:
|
||||
runs = s.scalars(select(Run)).all()
|
||||
assert [(r.rule_id, r.trigger) for r in runs] == [(a, RunTrigger.SCHEDULE)]
|
||||
|
||||
arun(main)
|
||||
|
||||
|
||||
def test_app_startup_restores_mounts_before_scheduler(tmp_path: Path) -> None:
|
||||
settings = Settings(
|
||||
data_dir=tmp_path / "lib",
|
||||
run_dir=tmp_path / "run",
|
||||
mount_base=tmp_path / "mnt",
|
||||
mount_backend="fake",
|
||||
)
|
||||
backend = FakeMountBackend(tmp_path / "fake")
|
||||
backend.export("nas", "scans")
|
||||
app = NetMover(settings, backend=backend)
|
||||
with session_scope(app.session_factory) as s:
|
||||
s.add(
|
||||
Mount(
|
||||
name="scans",
|
||||
slug="scans",
|
||||
type=MountType.CIFS,
|
||||
server="nas",
|
||||
share="scans",
|
||||
mountpoint=str((tmp_path / "mnt" / "scans").resolve()),
|
||||
)
|
||||
)
|
||||
assert (tmp_path / "lib" / "secret.key").stat().st_mode & 0o777 == 0o600
|
||||
|
||||
async def main() -> None:
|
||||
app.start()
|
||||
try:
|
||||
assert [r.ok for r in app.restore_results] == [True]
|
||||
assert app.scheduler.running
|
||||
assert app.health().ok
|
||||
finally:
|
||||
app.stop()
|
||||
|
||||
arun(main)
|
||||
# Beim Beenden NICHT aushängen
|
||||
assert backend.is_mounted((tmp_path / "mnt").resolve() / "scans")
|
||||
with app.session_factory() as s:
|
||||
m = s.scalars(select(Mount)).one()
|
||||
assert m.state is MountState.MOUNTED
|
||||
@@ -147,6 +147,8 @@ def test_manager_pins_host_key_and_checks_alive(
|
||||
with session_factory() as s:
|
||||
mm = s.get(Mount, mid)
|
||||
assert mm is not None and mm.host_key == server.host_key_string
|
||||
# Praxisbefund: nach erfolgreichem Test „erreichbar“ statt „nicht geprüft“
|
||||
assert mm.state is MountState.MOUNTED
|
||||
|
||||
assert manager.check_alive(mid) is None
|
||||
with session_factory() as s:
|
||||
|
||||
@@ -0,0 +1,623 @@
|
||||
"""Weboberfläche: Einrichtung, Anmeldung, CSRF, Seiten und Aktionen (FakeMountBackend)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from netmover.app_state import NetMover
|
||||
from netmover.config import Settings
|
||||
from netmover.db import session_scope
|
||||
from netmover.models import Credential, Mount, MountState, Rule, Run, RunStatus
|
||||
from netmover.mounting.fake import FakeMountBackend
|
||||
from netmover.web.app import create_app
|
||||
|
||||
PASSWORD = "sehr-geheim-123"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def settings(tmp_path: Path) -> Settings:
|
||||
return Settings(
|
||||
data_dir=tmp_path / "lib",
|
||||
run_dir=tmp_path / "run",
|
||||
mount_base=tmp_path / "mnt",
|
||||
mount_backend="fake",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def backend(tmp_path: Path) -> Iterator[FakeMountBackend]:
|
||||
b = FakeMountBackend(tmp_path / "fake")
|
||||
yield b
|
||||
b.release_hangs()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def state(settings: Settings, backend: FakeMountBackend) -> NetMover:
|
||||
return NetMover(settings, backend=backend)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def raw_client(settings: Settings, state: NetMover) -> Iterator[TestClient]:
|
||||
with TestClient(create_app(settings, state), follow_redirects=False) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def csrf_of(html: str) -> str:
|
||||
m = re.search(r'name="csrf-token" content="([^"]+)"', html)
|
||||
assert m, "kein CSRF-Token in der Seite"
|
||||
return m.group(1)
|
||||
|
||||
|
||||
def do_setup(c: TestClient) -> str:
|
||||
page = c.get("/setup")
|
||||
token = csrf_of(page.text)
|
||||
code = c.app.state.setup_code.code # type: ignore[attr-defined]
|
||||
r = c.post(
|
||||
"/setup",
|
||||
data={"csrf_token": token, "code": code, "password": PASSWORD, "confirm": PASSWORD},
|
||||
)
|
||||
assert r.status_code == 303, r.text
|
||||
return csrf_of(c.get("/").text)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(raw_client: TestClient) -> TestClient:
|
||||
raw_client.headers["X-Test-CSRF"] = do_setup(raw_client)
|
||||
return raw_client
|
||||
|
||||
|
||||
def post(c: TestClient, url: str, data: dict[str, str] | None = None, **kw: object): # type: ignore[no-untyped-def]
|
||||
payload = {"csrf_token": c.headers["X-Test-CSRF"], **(data or {})}
|
||||
return c.post(url, data=payload, **kw) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -- Einrichtung und Anmeldung --------------------------------------------------------------
|
||||
|
||||
|
||||
def test_first_start_forces_setup(raw_client: TestClient) -> None:
|
||||
for url in ("/", "/mounts", "/rules", "/log", "/settings", "/login"):
|
||||
r = raw_client.get(url)
|
||||
assert r.status_code == 303 and r.headers["location"] == "/setup", url
|
||||
|
||||
|
||||
def test_setup_requires_code_and_strong_password(raw_client: TestClient) -> None:
|
||||
token = csrf_of(raw_client.get("/setup").text)
|
||||
code = raw_client.app.state.setup_code.code # type: ignore[attr-defined]
|
||||
bad = [
|
||||
{"code": "falsch", "password": PASSWORD, "confirm": PASSWORD},
|
||||
{"code": code, "password": "kurz", "confirm": "kurz"},
|
||||
{"code": code, "password": PASSWORD, "confirm": PASSWORD + "x"},
|
||||
]
|
||||
for data in bad:
|
||||
r = raw_client.post("/setup", data={"csrf_token": token, **data})
|
||||
assert r.status_code == 400
|
||||
assert raw_client.get("/").status_code == 303
|
||||
|
||||
|
||||
def test_setup_only_once(client: TestClient) -> None:
|
||||
assert client.get("/setup").headers["location"] == "/login"
|
||||
|
||||
|
||||
def test_login_logout_and_cookie_flags(raw_client: TestClient) -> None:
|
||||
do_setup(raw_client)
|
||||
token = csrf_of(raw_client.get("/").text)
|
||||
raw_client.post("/logout", data={"csrf_token": token})
|
||||
assert raw_client.get("/").status_code == 303
|
||||
page = raw_client.get("/login")
|
||||
r = raw_client.post("/login", data={"csrf_token": csrf_of(page.text), "password": "falsch"})
|
||||
assert r.status_code == 401
|
||||
r = raw_client.post("/login", data={"csrf_token": csrf_of(page.text), "password": PASSWORD})
|
||||
assert r.status_code == 303
|
||||
cookie = r.headers["set-cookie"].lower()
|
||||
assert "httponly" in cookie and "samesite=lax" in cookie
|
||||
assert raw_client.get("/").status_code == 200
|
||||
|
||||
|
||||
def test_login_lockout(raw_client: TestClient) -> None:
|
||||
do_setup(raw_client)
|
||||
raw_client.cookies.clear()
|
||||
token = csrf_of(raw_client.get("/login").text)
|
||||
codes = [
|
||||
raw_client.post("/login", data={"csrf_token": token, "password": "x"}).status_code
|
||||
for _ in range(6)
|
||||
]
|
||||
assert codes[:5] == [401] * 5 and codes[5] == 429
|
||||
|
||||
|
||||
def test_csrf_required_for_post(client: TestClient) -> None:
|
||||
assert client.post("/rules/1/toggle").status_code == 403
|
||||
assert client.post("/rules/1/toggle", data={"csrf_token": "falsch"}).status_code == 403
|
||||
assert post(client, "/rules/1/toggle").status_code == 303
|
||||
r = client.post("/rules/1/toggle", headers={"X-CSRF-Token": client.headers["X-Test-CSRF"]})
|
||||
assert r.status_code == 303
|
||||
|
||||
|
||||
def test_htmx_request_without_login_gets_hx_redirect(raw_client: TestClient) -> None:
|
||||
do_setup(raw_client)
|
||||
raw_client.cookies.clear()
|
||||
r = raw_client.get("/partials/status", headers={"HX-Request": "true"})
|
||||
assert r.status_code == 204 and r.headers["HX-Redirect"].startswith("/login")
|
||||
|
||||
|
||||
def test_security_headers(client: TestClient) -> None:
|
||||
r = client.get("/")
|
||||
assert "script-src 'self'" in r.headers["content-security-policy"]
|
||||
assert r.headers["x-frame-options"] == "DENY"
|
||||
assert r.headers["cache-control"] == "no-store"
|
||||
|
||||
|
||||
def test_no_inline_scripts_or_styles(client: TestClient) -> None:
|
||||
for url in (
|
||||
"/",
|
||||
"/mounts",
|
||||
"/mounts/new",
|
||||
"/keys",
|
||||
"/rules",
|
||||
"/rules/new",
|
||||
"/log",
|
||||
"/settings",
|
||||
):
|
||||
html = client.get(url).text
|
||||
assert "style=" not in html, url
|
||||
assert not re.search(r"<script(?![^>]*\bsrc=)", html), url
|
||||
|
||||
|
||||
def test_healthz(client: TestClient, state: NetMover) -> None:
|
||||
r = client.get("/healthz")
|
||||
assert r.status_code == 200 and r.json()["status"] == "ok"
|
||||
with session_scope(state.session_factory) as s:
|
||||
s.add(
|
||||
Mount(
|
||||
name="kaputt",
|
||||
slug="kaputt",
|
||||
type="cifs",
|
||||
server="nas",
|
||||
share="x",
|
||||
mountpoint="/mnt/netmover/kaputt",
|
||||
state=MountState.ERROR,
|
||||
last_error="weg",
|
||||
)
|
||||
)
|
||||
r = client.get("/healthz")
|
||||
assert r.status_code == 503 and "kaputt" in r.json()["problems"][0]
|
||||
|
||||
|
||||
# -- Anbindungen ------------------------------------------------------------------------------
|
||||
|
||||
|
||||
MOUNT_FORM = {
|
||||
"name": "Synology Scans",
|
||||
"type": "cifs",
|
||||
"server": "nas",
|
||||
"share": "scans",
|
||||
"username": "scanner",
|
||||
"password": "Pa$$w0rd,mit=Zeichen",
|
||||
"uid": "1000",
|
||||
"gid": "1000",
|
||||
"file_mode": "0644",
|
||||
"dir_mode": "0755",
|
||||
"extra_options": "vers=3.0",
|
||||
"auto_mount": "on",
|
||||
"enabled": "on",
|
||||
}
|
||||
|
||||
|
||||
def _mount_id(state: NetMover, name: str) -> int:
|
||||
with state.session_factory() as s:
|
||||
mid = s.scalar(select(Mount.id).where(Mount.name == name))
|
||||
assert mid is not None
|
||||
return mid
|
||||
|
||||
|
||||
def test_create_mount_test_mount_unmount_delete(
|
||||
client: TestClient, state: NetMover, backend: FakeMountBackend
|
||||
) -> None:
|
||||
backend.export("nas", "scans")
|
||||
r = post(client, "/mounts/new", MOUNT_FORM)
|
||||
assert r.status_code == 303, r.text
|
||||
mid = _mount_id(state, "Synology Scans")
|
||||
with state.session_factory() as s:
|
||||
m = s.get(Mount, mid)
|
||||
assert m is not None and m.slug == "synology-scans" and m.credential is not None
|
||||
assert "Pa$$" not in m.credential.secret_enc
|
||||
assert state.secrets.decrypt(m.credential.secret_enc) == MOUNT_FORM["password"]
|
||||
|
||||
# Passwort taucht nirgends auf
|
||||
for url in ("/mounts", f"/mounts/{mid}/edit"):
|
||||
assert MOUNT_FORM["password"] not in client.get(url).text
|
||||
|
||||
r = post(client, f"/mounts/{mid}/test")
|
||||
assert r.status_code == 200 and "Quelle mit Löschen" in r.text and "✓" in r.text
|
||||
|
||||
assert post(client, f"/mounts/{mid}/mount").status_code == 303
|
||||
assert backend.is_mounted(state.mounts.mountpoint_for("synology-scans"))
|
||||
spec = [c for c in backend.calls if c.action == "mount"][-1].spec
|
||||
assert spec is not None and spec.credentials is not None
|
||||
assert spec.credentials.password == MOUNT_FORM["password"]
|
||||
assert "Fehler" not in client.get("/").text
|
||||
|
||||
assert post(client, f"/mounts/{mid}/unmount").status_code == 303
|
||||
assert not backend.mounted
|
||||
assert post(client, f"/mounts/{mid}/delete").status_code == 303
|
||||
with state.session_factory() as s:
|
||||
assert s.scalars(select(Mount)).all() == []
|
||||
assert s.scalars(select(Credential)).all() == []
|
||||
|
||||
|
||||
def test_empty_password_keeps_secret(client: TestClient, state: NetMover) -> None:
|
||||
post(client, "/mounts/new", MOUNT_FORM)
|
||||
mid = _mount_id(state, "Synology Scans")
|
||||
r = post(client, f"/mounts/{mid}/edit", {**MOUNT_FORM, "password": "", "username": "neu"})
|
||||
assert r.status_code == 303
|
||||
with state.session_factory() as s:
|
||||
m = s.get(Mount, mid)
|
||||
assert m is not None and m.credential is not None
|
||||
assert m.credential.username == "neu"
|
||||
assert state.secrets.decrypt(m.credential.secret_enc) == MOUNT_FORM["password"]
|
||||
|
||||
|
||||
def test_mount_form_validation_shows_errors(client: TestClient, state: NetMover) -> None:
|
||||
bad = {**MOUNT_FORM, "server": "nas;reboot", "extra_options": "password=x"}
|
||||
r = post(client, "/mounts/new", bad)
|
||||
assert r.status_code == 400
|
||||
assert "Ungültiger Servername" in r.text and "gesperrt" in r.text
|
||||
with state.session_factory() as s:
|
||||
assert s.scalars(select(Mount)).all() == []
|
||||
|
||||
|
||||
def test_connection_change_requires_unmount(
|
||||
client: TestClient, state: NetMover, backend: FakeMountBackend
|
||||
) -> None:
|
||||
backend.export("nas", "scans")
|
||||
post(client, "/mounts/new", MOUNT_FORM)
|
||||
mid = _mount_id(state, "Synology Scans")
|
||||
post(client, f"/mounts/{mid}/mount")
|
||||
r = post(client, f"/mounts/{mid}/edit", {**MOUNT_FORM, "share": "anders", "password": ""})
|
||||
assert r.status_code == 400 and "ausgehängt" in r.text
|
||||
# Nicht-Verbindungsfelder gehen trotzdem
|
||||
r = post(client, f"/mounts/{mid}/edit", {**MOUNT_FORM, "password": "", "auto_mount": ""})
|
||||
assert r.status_code == 303
|
||||
|
||||
|
||||
def test_form_test_with_unsaved_data(client: TestClient, backend: FakeMountBackend) -> None:
|
||||
backend.export("nas", "scans")
|
||||
r = post(client, "/mounts/test", MOUNT_FORM)
|
||||
assert r.status_code == 200 and "Einhängen" in r.text and "✓" in r.text
|
||||
r = post(client, "/mounts/test", {**MOUNT_FORM, "share": "fehlt"})
|
||||
assert ("✗" in r.text and "existiert auf dem Server nicht" not in r.text) or "✗" in r.text
|
||||
|
||||
|
||||
def test_sftp_mount_generates_key(client: TestClient, state: NetMover) -> None:
|
||||
r = post(
|
||||
client,
|
||||
"/mounts/new",
|
||||
{
|
||||
"name": "paperless",
|
||||
"type": "sftp",
|
||||
"server": "10.0.0.5",
|
||||
"share": "/opt/consume",
|
||||
"port": "22",
|
||||
"auth": "ssh_key",
|
||||
"username": "paperless",
|
||||
"enabled": "on",
|
||||
},
|
||||
)
|
||||
assert r.status_code == 303, r.text
|
||||
page = client.get("/mounts").text
|
||||
assert "ssh-ed25519 " in page and "BEGIN OPENSSH PRIVATE KEY" not in page
|
||||
with state.session_factory() as s:
|
||||
m = s.scalars(select(Mount)).one()
|
||||
assert m.mountpoint is None and m.auto_mount is False
|
||||
|
||||
|
||||
def test_delete_mount_used_by_rule_is_refused(client: TestClient, state: NetMover) -> None:
|
||||
post(client, "/mounts/new", MOUNT_FORM)
|
||||
mid = _mount_id(state, "Synology Scans")
|
||||
with session_scope(state.session_factory) as s:
|
||||
s.add(Rule(name="r", src_mount_id=mid, dst_path="/tmp"))
|
||||
post(client, f"/mounts/{mid}/delete")
|
||||
assert "Wird noch von Regeln benutzt" in client.get("/mounts").text
|
||||
assert _mount_id(state, "Synology Scans") == mid
|
||||
|
||||
|
||||
# -- Regeln -----------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mounted(
|
||||
client: TestClient, state: NetMover, backend: FakeMountBackend, tmp_path: Path
|
||||
) -> tuple[int, Path, Path]:
|
||||
remote = backend.export("nas", "scans")
|
||||
post(client, "/mounts/new", MOUNT_FORM)
|
||||
mid = _mount_id(state, "Synology Scans")
|
||||
post(client, f"/mounts/{mid}/mount")
|
||||
consume = tmp_path / "consume"
|
||||
consume.mkdir()
|
||||
return mid, remote, consume
|
||||
|
||||
|
||||
def rule_form(mid: int, consume: Path, **kw: str) -> dict[str, str]:
|
||||
data = {
|
||||
"name": "Scans → Paperless",
|
||||
"enabled": "on",
|
||||
"src_mount_id": str(mid),
|
||||
"src_subpath": "",
|
||||
"dst_kind": "local",
|
||||
"dst_path": str(consume),
|
||||
"pattern": "*.pdf",
|
||||
"pattern_type": "glob",
|
||||
"interval_seconds": "30",
|
||||
"stability_seconds": "0",
|
||||
"min_age_seconds": "0",
|
||||
"op_timeout_seconds": "120",
|
||||
"action": "move",
|
||||
"on_collision": "rename",
|
||||
"quarantine_after": "3",
|
||||
}
|
||||
data.update(kw)
|
||||
return data
|
||||
|
||||
|
||||
def test_rule_validation(client: TestClient, mounted: tuple[int, Path, Path]) -> None:
|
||||
mid, _, consume = mounted
|
||||
r = post(
|
||||
client,
|
||||
"/rules/new",
|
||||
rule_form(
|
||||
mid,
|
||||
consume,
|
||||
interval_seconds="5",
|
||||
pattern_type="regex",
|
||||
pattern="([",
|
||||
set_owner="root",
|
||||
dst_path="relativ",
|
||||
),
|
||||
)
|
||||
assert r.status_code == 400
|
||||
for msg in ("mindestens 10", "regulärer Ausdruck", "uid:gid", "absoluten Pfad"):
|
||||
assert msg in r.text
|
||||
|
||||
|
||||
def test_rule_create_dryrun_run(
|
||||
client: TestClient, state: NetMover, mounted: tuple[int, Path, Path]
|
||||
) -> None:
|
||||
mid, remote, consume = mounted
|
||||
(remote / "scan001.pdf").write_text("pdf")
|
||||
(remote / "notiz.txt").write_text("txt")
|
||||
r = post(client, "/rules/new", rule_form(mid, consume))
|
||||
assert r.status_code == 303, r.text
|
||||
with state.session_factory() as s:
|
||||
rid = s.scalars(select(Rule.id)).one()
|
||||
assert state.scheduler.next_run(rid) is not None
|
||||
|
||||
r = post(client, f"/rules/{rid}/dryrun")
|
||||
assert r.status_code == 200
|
||||
assert "scan001.pdf" in r.text and str(consume / "scan001.pdf") in r.text
|
||||
assert "notiz.txt" not in r.text and "nichts verändert" in r.text
|
||||
assert (remote / "scan001.pdf").exists()
|
||||
|
||||
result = state.runs.execute(rid) # „Jetzt ausführen“ synchron für den Test
|
||||
assert result is not None
|
||||
assert (consume / "scan001.pdf").exists() and not (remote / "scan001.pdf").exists()
|
||||
|
||||
log = client.get("/log").text
|
||||
assert "Scans → Paperless" in log
|
||||
with state.session_factory() as s:
|
||||
run = s.scalars(select(Run).where(Run.dry_run.is_(False))).one()
|
||||
assert run.status is RunStatus.OK
|
||||
items = client.get(f"/log/{run.id}/items").text
|
||||
assert "scan001.pdf" in items and "verschoben" in items
|
||||
|
||||
|
||||
def test_rule_run_now_button(
|
||||
client: TestClient, state: NetMover, mounted: tuple[int, Path, Path]
|
||||
) -> None:
|
||||
mid, _, consume = mounted
|
||||
post(client, "/rules/new", rule_form(mid, consume))
|
||||
with state.session_factory() as s:
|
||||
rid = s.scalars(select(Rule.id)).one()
|
||||
r = post(client, f"/rules/{rid}/run")
|
||||
assert r.status_code == 303
|
||||
assert "Lauf gestartet" in client.get("/").text
|
||||
|
||||
|
||||
def test_rule_toggle_and_delete_sync_scheduler(
|
||||
client: TestClient, state: NetMover, mounted: tuple[int, Path, Path]
|
||||
) -> None:
|
||||
mid, _, consume = mounted
|
||||
post(client, "/rules/new", rule_form(mid, consume))
|
||||
with state.session_factory() as s:
|
||||
rid = s.scalars(select(Rule.id)).one()
|
||||
post(client, f"/rules/{rid}/toggle")
|
||||
assert state.scheduler.next_run(rid) is None
|
||||
post(client, f"/rules/{rid}/toggle")
|
||||
assert state.scheduler.next_run(rid) is not None
|
||||
post(client, f"/rules/{rid}/delete")
|
||||
assert state.scheduler.next_run(rid) is None
|
||||
|
||||
|
||||
def test_status_page_shows_stale_warning(
|
||||
client: TestClient, state: NetMover, mounted: tuple[int, Path, Path]
|
||||
) -> None:
|
||||
from datetime import timedelta
|
||||
|
||||
from netmover.models import utcnow
|
||||
|
||||
mid, _, consume = mounted
|
||||
post(client, "/rules/new", rule_form(mid, consume))
|
||||
with session_scope(state.session_factory) as s:
|
||||
r = s.scalars(select(Rule)).one()
|
||||
r.created_at = utcnow() - timedelta(hours=1)
|
||||
r.last_success_at = utcnow() - timedelta(minutes=10)
|
||||
state.started_at = utcnow() - timedelta(hours=1)
|
||||
page = client.get("/").text
|
||||
assert "länger als 3 × Intervall" in page and 'class="warn"' in page
|
||||
assert client.get("/healthz").status_code == 503
|
||||
|
||||
|
||||
# -- Einstellungen -------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_settings_webhook_url_is_masked(client: TestClient, state: NetMover) -> None:
|
||||
from netmover.settings_store import load_settings
|
||||
|
||||
url = "https://ntfy.example/geheimes-topic-token"
|
||||
r = post(
|
||||
client,
|
||||
"/settings",
|
||||
{
|
||||
"retention_days": "14",
|
||||
"notify_mode": "every",
|
||||
"notify_after": "3",
|
||||
"webhook_url": url,
|
||||
"webhook_content_type": "text/plain",
|
||||
"webhook_template": "$message",
|
||||
},
|
||||
)
|
||||
assert r.status_code == 303
|
||||
page = client.get("/settings").text
|
||||
assert "geheimes-topic-token" not in page and "ntfy.example" in page
|
||||
# Maske unverändert zurückschicken → URL bleibt
|
||||
masked = re.search(r'name="webhook_url" value="([^"]*)"', page)
|
||||
assert masked
|
||||
post(
|
||||
client,
|
||||
"/settings",
|
||||
{
|
||||
"retention_days": "14",
|
||||
"notify_mode": "every",
|
||||
"notify_after": "3",
|
||||
"webhook_url": masked.group(1),
|
||||
"webhook_content_type": "text/plain",
|
||||
"webhook_template": "$message",
|
||||
},
|
||||
)
|
||||
cfg = load_settings(state.session_factory)
|
||||
assert cfg.webhook_url == url and cfg.retention_days == 14
|
||||
|
||||
|
||||
def test_change_password(client: TestClient) -> None:
|
||||
r = post(
|
||||
client,
|
||||
"/settings/password",
|
||||
{"current": "falsch", "password": "neues-passwort-1", "confirm": "neues-passwort-1"},
|
||||
)
|
||||
assert "falsch" in client.get("/settings").text
|
||||
r = post(
|
||||
client,
|
||||
"/settings/password",
|
||||
{"current": PASSWORD, "password": "neues-passwort-1", "confirm": "neues-passwort-1"},
|
||||
)
|
||||
assert r.status_code == 303
|
||||
|
||||
|
||||
def test_export_excludes_secrets_and_import_roundtrip(
|
||||
client: TestClient,
|
||||
state: NetMover,
|
||||
mounted: tuple[int, Path, Path],
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
mid, _, consume = mounted
|
||||
post(client, "/rules/new", rule_form(mid, consume))
|
||||
r = client.get("/settings/export")
|
||||
assert r.status_code == 200 and "attachment" in r.headers["content-disposition"]
|
||||
text = r.text
|
||||
assert MOUNT_FORM["password"] not in text and "scanner" not in text and "secret_enc" not in text
|
||||
data = json.loads(text)
|
||||
assert data["mounts"][0]["name"] == "Synology Scans"
|
||||
assert data["rules"][0]["src_mount"] == "Synology Scans"
|
||||
|
||||
# in eine frische Instanz importieren
|
||||
settings2 = Settings(
|
||||
data_dir=tmp_path / "lib2",
|
||||
run_dir=tmp_path / "run2",
|
||||
mount_base=tmp_path / "mnt2",
|
||||
mount_backend="fake",
|
||||
)
|
||||
state2 = NetMover(settings2, backend=FakeMountBackend(tmp_path / "fake2"))
|
||||
with TestClient(create_app(settings2, state2), follow_redirects=False) as c2:
|
||||
c2.headers["X-Test-CSRF"] = do_setup(c2)
|
||||
r = c2.post(
|
||||
"/settings/import",
|
||||
data={"csrf_token": c2.headers["X-Test-CSRF"]},
|
||||
files={"file": ("export.json", text, "application/json")},
|
||||
)
|
||||
assert r.status_code == 303 and r.headers["location"] == "/settings", r.headers
|
||||
page = c2.get("/settings").text
|
||||
assert "Importiert" in page and "neu eingeben" in page
|
||||
with state2.session_factory() as s:
|
||||
m = s.scalars(select(Mount)).one()
|
||||
assert m.credential is None and m.auto_mount is False
|
||||
rule = s.scalars(select(Rule)).one()
|
||||
assert rule.enabled is False and rule.src_mount_id == m.id
|
||||
# zweiter Import überspringt vorhandene Namen
|
||||
c2.post(
|
||||
"/settings/import",
|
||||
data={"csrf_token": c2.headers["X-Test-CSRF"]},
|
||||
files={"file": ("export.json", text, "application/json")},
|
||||
)
|
||||
assert "Übersprungen" in c2.get("/settings").text
|
||||
|
||||
|
||||
# -- Schlüsselverwaltung -------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_keys_page_generate_import_select_delete(client: TestClient, state: NetMover) -> None:
|
||||
from netmover.crypto import generate_ssh_keypair
|
||||
|
||||
r = post(client, "/keys/generate", {"name": "paperless-lxc"})
|
||||
assert r.status_code == 303
|
||||
private, public = generate_ssh_keypair()
|
||||
r = post(client, "/keys/import", {"name": "alt", "private_key": private, "passphrase": ""})
|
||||
assert r.status_code == 303
|
||||
page = client.get("/keys").text
|
||||
assert "paperless-lxc" in page and "alt" in page
|
||||
assert public.split()[1] in page # öffentlicher Teil sichtbar
|
||||
for line in private.splitlines()[1:-1]: # keine Zeile des privaten Schlüssels
|
||||
assert line not in page
|
||||
assert "restrict,command="/usr/lib/openssh/sftp-server"" in page
|
||||
|
||||
with state.session_factory() as s:
|
||||
key_id = s.scalar(select(Credential.id).where(Credential.name == "alt"))
|
||||
form = client.get("/mounts/new").text
|
||||
assert f'value="{key_id}"' in form and "neuer Schlüssel nur für diese Anbindung" in form
|
||||
|
||||
r = post(
|
||||
client,
|
||||
"/mounts/new",
|
||||
{
|
||||
"name": "paperless",
|
||||
"type": "sftp",
|
||||
"server": "10.0.0.5",
|
||||
"share": "/consume",
|
||||
"port": "22",
|
||||
"auth": "ssh_key",
|
||||
"key_id": str(key_id),
|
||||
"username": "root",
|
||||
"enabled": "on",
|
||||
},
|
||||
)
|
||||
assert r.status_code == 303, r.text
|
||||
mounts = client.get("/mounts").text
|
||||
assert "gespeichertem Schlüssel" in mounts and "„alt“" in mounts
|
||||
edit = client.get(f"/mounts/{_mount_id(state, 'paperless')}/edit").text
|
||||
assert f'<option value="{key_id}" selected>' in edit and 'value="root"' in edit
|
||||
|
||||
post(client, f"/keys/{key_id}/delete")
|
||||
assert "noch benutzt" in client.get("/keys").text
|
||||
post(client, f"/mounts/{_mount_id(state, 'paperless')}/delete")
|
||||
post(client, f"/keys/{key_id}/delete")
|
||||
page = client.get("/keys").text
|
||||
assert "gelöscht" in page and ">alt<" not in page
|
||||
|
||||
|
||||
def test_key_import_error_is_shown(client: TestClient) -> None:
|
||||
post(client, "/keys/import", {"name": "x", "private_key": "kein schlüssel", "passphrase": ""})
|
||||
assert "Kein lesbarer privater Schlüssel" in client.get("/keys").text
|
||||
Reference in New Issue
Block a user