Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fef90fc059 | ||
|
|
d46b1ba5c5 | ||
|
|
ce5bef8306 | ||
|
|
fb975f1eae | ||
|
|
686542ff6f | ||
|
|
4369faf144 | ||
|
|
0a317e2aa9 | ||
|
|
2d2238ce4e | ||
|
|
4f9c5bdc1d | ||
|
|
1bf0fae2fc | ||
|
|
ea50eacc9d | ||
|
|
c836f58189 | ||
|
|
8276ce2afb | ||
|
|
48a52c5290 |
+53
-1
@@ -8,6 +8,56 @@ Die Version steht an genau einer Stelle: `src/netmover/__init__.py`.
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.2.0] – 2026-10-11
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- **Schlüsselverwaltung** (Seite „Schlüssel“): SSH-Schlüssel erzeugen (Ed25519) oder
|
||||||
|
importieren (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH/PEM, auch mit Passphrase),
|
||||||
|
Fingerabdruck und `authorized_keys`-Zeile mit SFTP-Beschränkung zum Kopieren,
|
||||||
|
Umbenennen, Löschen nur wenn unbenutzt. Private Schlüssel werden nie angezeigt.
|
||||||
|
- SFTP-Anbindungen können einen gespeicherten Schlüssel auswählen; der Benutzername
|
||||||
|
gehört zur Anbindung (Migration 0002: `credentials.managed`, `mounts.username`).
|
||||||
|
- **`netmover-update`**: Update aus dem Git-Repo mit Sicherung von Datenbank,
|
||||||
|
Schlüsseln und Einstellungen, Changelog-Auszug, Neustart und Healthcheck;
|
||||||
|
Versionswahl, Warnung beim Zurückstufen.
|
||||||
|
- Installation als Git-Klon unter `/opt/netmover/src`; `install.sh` installiert
|
||||||
|
`git` und das Update-Skript.
|
||||||
|
|
||||||
|
### Geändert
|
||||||
|
- Klartext-Hinweis bei `mount error(95)` (keine gemeinsame SMB-Version) mit
|
||||||
|
Synology-Einstellung; aufgefallen bei der Anbindung des echten NAS.
|
||||||
|
- SFTP-Anbindungen gelten nach erfolgreichem Test als „erreichbar“ (vorher bis
|
||||||
|
zum ersten Regellauf „nicht geprüft“).
|
||||||
|
|
||||||
|
## [0.1.1] – 2026-10-10
|
||||||
|
|
||||||
|
Korrekturen aus dem ersten Praxistest auf einer Debian-13-VM mit echten
|
||||||
|
SMB-, NFS- und SFTP-Servern.
|
||||||
|
|
||||||
|
### Behoben
|
||||||
|
- **systemd-Unit:** `ProtectSystem`, `PrivateTmp` und `ProtectControlGroups`
|
||||||
|
gaben dem Dienst einen eigenen Mount-Namespace. Die Mounts waren für das System
|
||||||
|
und andere Programme unsichtbar und verschwanden bei jedem Neustart des Dienstes
|
||||||
|
(`MountFlags=shared` half nicht). Jetzt nur noch Härtungen ohne Namespace.
|
||||||
|
- **Stabilitätsprüfung:** CIFS lieferte per `stat()` sekundenlang eine veraltete
|
||||||
|
Größe einer noch wachsenden Datei. Gemessen wird jetzt per `open()` + `fstat()`,
|
||||||
|
das erzwingt frische Attribute. Gesperrte Dateien gelten als „in Benutzung“.
|
||||||
|
(Die zweite Sicherung, die Byte-Zählung beim Kopieren, hatte den Fall bereits
|
||||||
|
abgefangen; es kam keine unvollständige Datei an.)
|
||||||
|
- **Ausfall länger als die Wiederherstellung (z. B. NAS-Neustart, Stromausfall):**
|
||||||
|
Von NetMover selbst ausgehängte Anbindungen mit „automatisch einhängen“ werden
|
||||||
|
jetzt jede Minute erneut eingehängt, auch wenn das Einhängen schon beim
|
||||||
|
Dienststart scheiterte. Entwarnung per Webhook. Von außen ausgehängte Anbindungen
|
||||||
|
bleiben weiterhin unangetastet.
|
||||||
|
- Bei einer Zeitüberschreitung von `mount` wird die ganze Prozessgruppe beendet, damit
|
||||||
|
`mount.nfs` nicht später doch noch einhängt.
|
||||||
|
- Hinweistexte zu `mount`-Fehlern werden nicht mehr mitgeschwärzt.
|
||||||
|
- `install.sh` ohne `debconf`-Warnungen, wenn kein Terminal angeschlossen ist.
|
||||||
|
|
||||||
|
### Dokumentation
|
||||||
|
- Fehlersuche: verzögert sichtbare Dateien auf CIFS → `nohandlecache`.
|
||||||
|
- README erklärt, warum die Unit keine Namespace-Härtung nutzt.
|
||||||
|
|
||||||
## [0.1.0] – 2026-10-10
|
## [0.1.0] – 2026-10-10
|
||||||
|
|
||||||
Erste Version.
|
Erste Version.
|
||||||
@@ -99,5 +149,7 @@ Erste Version.
|
|||||||
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
||||||
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
||||||
|
|
||||||
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.0...HEAD
|
[Unreleased]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.2.0...HEAD
|
||||||
|
[0.2.0]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.1...v0.2.0
|
||||||
|
[0.1.1]: https://git.wilde-server.de/florian.wilde/netmover/compare/v0.1.0...v0.1.1
|
||||||
[0.1.0]: https://git.wilde-server.de/florian.wilde/netmover/releases/tag/v0.1.0
|
[0.1.0]: https://git.wilde-server.de/florian.wilde/netmover/releases/tag/v0.1.0
|
||||||
|
|||||||
@@ -13,10 +13,55 @@ Zeiten in Ortszeit (Europe/Berlin).
|
|||||||
| 2026-10-10 | 19:10 | 19:17 | 0:07 | 4 | Scheduler, Protokoll, Benachrichtigung |
|
| 2026-10-10 | 19:10 | 19:17 | 0:07 | 4 | Scheduler, Protokoll, Benachrichtigung |
|
||||||
| 2026-10-10 | 19:18 | 19:34 | 0:16 | 5 | Weboberfläche und Anmeldung |
|
| 2026-10-10 | 19:18 | 19:34 | 0:16 | 5 | Weboberfläche und Anmeldung |
|
||||||
| 2026-10-10 | 19:40 | 19:50 | 0:10 | 6 | Deployment, README, Release 0.1.0 |
|
| 2026-10-10 | 19:40 | 19:50 | 0:10 | 6 | Deployment, README, Release 0.1.0 |
|
||||||
| **Summe** | | | **1:06** | | |
|
| 2026-10-10 | 21:36 | 22:05 | 0:29 | – | Praxistest auf VM, Release 0.1.1 |
|
||||||
|
| **Summe** | | | **1:35** | | |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 2026-10-10 · 21:36–22:05 · Praxistest auf der Debian-13-VM, Release 0.1.1
|
||||||
|
|
||||||
|
**Umgebung**
|
||||||
|
- KVM-VM `srv-rz-netmover` (10.0.10.223), Debian 13, Python 3.13, 2 GB RAM.
|
||||||
|
- Installation per `deploy/install.sh` aus einem `git archive` (VM ohne Gitea-Zugang).
|
||||||
|
- Vorübergehende Testserver auf der VM: Samba (nur `lo`), NFS-Exporte für
|
||||||
|
`127.0.0.1` (mit/ohne Root-Squash), SFTP-Benutzer `paperless` mit Schlüssel
|
||||||
|
`restrict,command="/usr/lib/openssh/sftp-server"`.
|
||||||
|
- Auf Wunsch `NETMOVER_HOST=0.0.0.0`, damit die Oberfläche im LAN erreichbar ist.
|
||||||
|
|
||||||
|
**Bestanden**
|
||||||
|
- Ersteinrichtung mit Code aus dem Log, Cookie-Flags.
|
||||||
|
- CIFS (Passwort mit Komma/Ausrufezeichen, Credentials-Datei danach weg), NFS 4,
|
||||||
|
NFS 3, Root-Squash-Erkennung mit Synology-Hinweis, SFTP mit TOFU (Fingerabdruck
|
||||||
|
= `ssh-keygen -lf`), echtes OpenSSH liefert `statvfs`.
|
||||||
|
- Probelauf, Kollision `scan001-1.pdf`, `.txt`/`.part` ignoriert, keine `.part`-Reste,
|
||||||
|
Eigentümer `paperless` im SFTP-Ziel.
|
||||||
|
- Langsam wachsende Datei (20 × 50 KB/s): erst nach Ende verschoben, Prüfsumme gleich.
|
||||||
|
- Kopiermodus rekursiv mit Eigentümer/Modus; zweiter Lauf „2 bereits übertragen“.
|
||||||
|
- Samba gestoppt: Lebenszeichen 15 s → gestört → 3 Versuche → Fehler, `/healthz` 503,
|
||||||
|
nichts angefasst. NFS-Server gestoppt (hängender `hard`-Mount): Oberfläche und
|
||||||
|
andere Regel liefen weiter, Rückkehr automatisch.
|
||||||
|
- Mounts überstehen Stopp/Start des Dienstes (gleiche Mount-IDs).
|
||||||
|
- Neustart der VM: NFS sofort, SMB per Neuversuch nach ~1 min, Regel läuft weiter.
|
||||||
|
|
||||||
|
**Gefunden und behoben (v0.1.1)**
|
||||||
|
1. **systemd-Unit mit eigenem Mount-Namespace** (ProtectSystem/PrivateTmp/…):
|
||||||
|
Mounts nur im Dienst sichtbar, `shared,slave`. Härtung ohne Namespace.
|
||||||
|
2. **CIFS-Attribut-Cache**: `stat()` meldete 3 s lang eine veraltete Größe einer
|
||||||
|
wachsenden Datei; die Byte-Zählung beim Kopieren fing das ab. Jetzt
|
||||||
|
`open()` + `fstat()` für beide Messungen.
|
||||||
|
3. **Nach drei gescheiterten Versuchen endgültig aufgegeben** – zu kurz für einen
|
||||||
|
NAS-Neustart, und beim Booten vor dem NAS kam die Freigabe nie zurück.
|
||||||
|
Jetzt minütlicher Neuversuch (nur für von NetMover selbst ausgehängte
|
||||||
|
Anbindungen mit `auto_mount`) plus Scheduler-Job.
|
||||||
|
4. `mount`-Zeitlimit beendete nur `mount`, nicht `mount.nfs` → später doch
|
||||||
|
eingehängt. Jetzt ganze Prozessgruppe.
|
||||||
|
5. Hinweistext wurde mitgeschwärzt; `debconf`-Warnungen in `install.sh`.
|
||||||
|
|
||||||
|
**Nicht nachstellbar**
|
||||||
|
- Einmalig 46 s Verzögerung direkt nach dem Booten (CIFS-Verzeichnis-Cache bei
|
||||||
|
lokal statt per SMB angelegter Datei). Messungen danach 9–11 s, mit und ohne
|
||||||
|
`nohandlecache`. Als Hinweis in die Fehlersuche aufgenommen.
|
||||||
|
|
||||||
## 2026-10-10 · 19:40–19:50 · Schritt 6: Deployment, README, Release 0.1.0
|
## 2026-10-10 · 19:40–19:50 · Schritt 6: Deployment, README, Release 0.1.0
|
||||||
|
|
||||||
**Erledigt**
|
**Erledigt**
|
||||||
|
|||||||
@@ -47,15 +47,36 @@ heißt deshalb immer **kopieren, prüfen, dann erst die Quelle löschen**:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
apt install -y git
|
apt install -y git
|
||||||
git clone https://git.wilde-server.de/florian.wilde/netmover.git
|
git clone https://git.wilde-server.de/florian.wilde/netmover.git /opt/netmover/src
|
||||||
cd netmover
|
cd /opt/netmover/src && git checkout "$(git tag -l 'v*' --sort=-v:refname | head -n 1)"
|
||||||
sudo ./deploy/install.sh
|
./deploy/install.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
Das Skript installiert `python3-venv`, `cifs-utils` und `nfs-common` und richtet
|
Das Skript installiert `python3-venv`, `cifs-utils`, `nfs-common` und `git`. Es
|
||||||
NetMover in `/opt/netmover/venv` ein. Danach installiert es
|
richtet NetMover in `/opt/netmover/venv` ein, installiert
|
||||||
[`deploy/netmover.service`](deploy/netmover.service) und startet den Dienst.
|
[`deploy/netmover.service`](deploy/netmover.service) und das Update-Skript
|
||||||
Ein Update ist derselbe Aufruf nach einem `git pull`.
|
`netmover-update` und startet den Dienst.
|
||||||
|
|
||||||
|
### Update
|
||||||
|
|
||||||
|
```bash
|
||||||
|
netmover-update # auf die neueste Version
|
||||||
|
netmover-update --list # verfügbare Versionen
|
||||||
|
netmover-update v0.2.0 # bestimmte Version
|
||||||
|
```
|
||||||
|
|
||||||
|
`netmover-update` geht so vor:
|
||||||
|
1. Neue Versions-Tags aus Gitea holen und die Änderungen laut Changelog anzeigen.
|
||||||
|
2. **Datenbank, `secret.key` und Einstellungen nach `/var/backups/netmover/` sichern**
|
||||||
|
(die letzten 10 bleiben).
|
||||||
|
3. Die Version auschecken, installieren und den Dienst neu starten. Neue
|
||||||
|
Datenbank-Migrationen laufen dabei automatisch.
|
||||||
|
4. `/healthz` prüfen.
|
||||||
|
|
||||||
|
Die Mounts bleiben die ganze Zeit eingehängt. Geht etwas schief, nennt das Skript
|
||||||
|
die Befehle, um die Sicherung zurückzuspielen und zur vorherigen Version zu wechseln.
|
||||||
|
**Zurückstufen** auf eine ältere Version geht nur mit der Datenbank-Sicherung von
|
||||||
|
vor dem Update, wenn das Update eine Migration enthielt; das Skript fragt deshalb nach.
|
||||||
|
|
||||||
Was wo liegt:
|
Was wo liegt:
|
||||||
|
|
||||||
@@ -69,13 +90,18 @@ Was wo liegt:
|
|||||||
|
|
||||||
**Warum als root?** Ein- und Aushängen braucht `CAP_SYS_ADMIN`. Außerdem soll
|
**Warum als root?** Ein- und Aushängen braucht `CAP_SYS_ADMIN`. Außerdem soll
|
||||||
NetMover Dateien auf beliebigen Freigaben lesen, löschen und mit den richtigen
|
NetMover Dateien auf beliebigen Freigaben lesen, löschen und mit den richtigen
|
||||||
Eigentümern anlegen können. Die Unit härtet den Dienst so weit, wie es sich mit
|
Eigentümern anlegen können.
|
||||||
Mounten verträgt: `ProtectSystem=full`, `PrivateTmp`, `NoNewPrivileges`, Zustand nur
|
|
||||||
für root lesbar. Weil diese Optionen einen eigenen Mount-Namespace erzeugen, steht
|
**Warum so wenig systemd-Härtung?** Optionen wie `ProtectSystem=`, `PrivateTmp=`
|
||||||
`MountFlags=shared` in der Unit. Nur so sieht das System die Mounts:
|
oder `ProtectHome=` geben dem Dienst einen eigenen Mount-Namespace. Die Mounts
|
||||||
|
landeten dann *nur dort*: Das System und Paperless sähen sie nicht, und jeder
|
||||||
|
Neustart des Dienstes würde sie verwerfen. `MountFlags=shared` hilft dagegen nicht.
|
||||||
|
Auf der Test-VM nachgewiesen. Die Unit nutzt deshalb nur Härtungen ohne eigenen
|
||||||
|
Namespace (`NoNewPrivileges`, `RestrictSUIDSGID`, `LockPersonality`,
|
||||||
|
`RestrictRealtime`); Datenbank und Schlüssel sind nur für root lesbar. Prüfen:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
findmnt -R /mnt/netmover # sollte die eingehängten Freigaben zeigen
|
findmnt -l | grep /mnt/netmover # die eingehängten Freigaben müssen hier stehen
|
||||||
```
|
```
|
||||||
|
|
||||||
**Beim Beenden bleiben die Mounts eingehängt**, weil andere Programme sie benutzen
|
**Beim Beenden bleiben die Mounts eingehängt**, weil andere Programme sie benutzen
|
||||||
@@ -144,11 +170,18 @@ Lege in NetMover die *Anbindung → Neue Anbindung* an:
|
|||||||
| Typ | SFTP (nur Ziel) |
|
| Typ | SFTP (nur Ziel) |
|
||||||
| Server / SSH-Port | IP des LXC / `22` |
|
| Server / SSH-Port | IP des LXC / `22` |
|
||||||
| Freigabe / Pfad | `/opt/paperless_data/consume` |
|
| Freigabe / Pfad | `/opt/paperless_data/consume` |
|
||||||
| Anmeldeart | SSH-Schlüssel (NetMover erzeugt ihn) |
|
| Anmeldeart | SSH-Schlüssel |
|
||||||
|
| Schlüssel | einen gespeicherten Schlüssel wählen (siehe unten) oder „neuer Schlüssel nur für diese Anbindung“ |
|
||||||
| Benutzername | `paperless` (der Benutzer, unter dem Paperless läuft) |
|
| Benutzername | `paperless` (der Benutzer, unter dem Paperless läuft) |
|
||||||
|
|
||||||
Nach dem Speichern zeigt NetMover den **öffentlichen Schlüssel**. Trage ihn im LXC
|
**Schlüsselverwaltung:** Unter *Schlüssel* erzeugst du Ed25519-Schlüssel oder
|
||||||
ein, und zwar so, dass der Schlüssel **nur SFTP** darf:
|
importierst vorhandene (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH- oder PEM-Format,
|
||||||
|
auch mit Passphrase). Ein Schlüssel kann von mehreren Anbindungen genutzt werden; den
|
||||||
|
Benutzernamen legt jede Anbindung selbst fest. Private Schlüssel liegen verschlüsselt
|
||||||
|
in der Datenbank und werden nie angezeigt. Die Seite zeigt den Fingerabdruck und eine
|
||||||
|
fertige `authorized_keys`-Zeile zum Kopieren.
|
||||||
|
|
||||||
|
Trage den öffentlichen Schlüssel im LXC so ein, dass er **nur SFTP** darf:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# im Paperless-LXC als root
|
# im Paperless-LXC als root
|
||||||
@@ -322,6 +355,7 @@ findmnt -R /mnt/netmover # was ist wirklich eingehängt?
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `mount error(13): Permission denied` (CIFS) | Benutzer oder Passwort falsch, oder keine Rechte auf die Freigabe. |
|
| `mount error(13): Permission denied` (CIFS) | Benutzer oder Passwort falsch, oder keine Rechte auf die Freigabe. |
|
||||||
| `mount error(2): No such file or directory` | Freigabename falsch geschrieben (Groß/klein!). |
|
| `mount error(2): No such file or directory` | Freigabename falsch geschrieben (Groß/klein!). |
|
||||||
|
| `mount error(95): Operation not supported` | Keine gemeinsame SMB-Version (im Kernel-Log: „Dialect not supported by server“). Am NAS das maximale SMB-Protokoll auf SMB3 stellen oder Zusatzoption `vers=2.0`. |
|
||||||
| `mount error(112): Host is down` | NAS nicht erreichbar, oder SMB1 erwartet: `vers=3.0` bzw. `vers=2.1` probieren. |
|
| `mount error(112): Host is down` | NAS nicht erreichbar, oder SMB1 erwartet: `vers=3.0` bzw. `vers=2.1` probieren. |
|
||||||
| `… helper program` / `unknown filesystem type` | `apt install cifs-utils nfs-common` |
|
| `… helper program` / `unknown filesystem type` | `apt install cifs-utils nfs-common` |
|
||||||
| `Operation not permitted` / `only root can do that` | Kein `CAP_SYS_ADMIN`: unprivilegierter LXC oder Docker ohne `cap_add`. |
|
| `Operation not permitted` / `only root can do that` | Kein `CAP_SYS_ADMIN`: unprivilegierter LXC oder Docker ohne `cap_add`. |
|
||||||
@@ -330,10 +364,12 @@ findmnt -R /mnt/netmover # was ist wirklich eingehängt?
|
|||||||
| „… ist nicht leer“ beim Einhängen | Im Mountpoint liegen Dateien, die geschrieben wurden, als nichts eingehängt war. Inhalt prüfen und sichern, Verzeichnis leeren, erneut einhängen. |
|
| „… ist nicht leer“ beim Einhängen | Im Mountpoint liegen Dateien, die geschrieben wurden, als nichts eingehängt war. Inhalt prüfen und sichern, Verzeichnis leeren, erneut einhängen. |
|
||||||
| „nicht eingehängt – Lauf abgebrochen“ | Die Freigabe ist weg (z. B. von Hand ausgehängt). Auf der Anbindungsseite **Einhängen**. NetMover hängt verschwundene Mounts absichtlich nicht selbst neu ein. |
|
| „nicht eingehängt – Lauf abgebrochen“ | Die Freigabe ist weg (z. B. von Hand ausgehängt). Auf der Anbindungsseite **Einhängen**. NetMover hängt verschwundene Mounts absichtlich nicht selbst neu ein. |
|
||||||
| „gestört (ESTALE …)“, danach wieder OK | NAS-Neustart; NetMover hat neu eingehängt. Bleibt der Fehler: Netzwerk/NAS prüfen. |
|
| „gestört (ESTALE …)“, danach wieder OK | NAS-Neustart; NetMover hat neu eingehängt. Bleibt der Fehler: Netzwerk/NAS prüfen. |
|
||||||
|
| „weiterhin nicht verfügbar – neuer Versuch in 60 s“ | Das NAS ist (noch) weg, z. B. während eines DSM-Updates. Bei Anbindungen mit „automatisch einhängen“ versucht NetMover es jede Minute erneut und meldet per Webhook, wenn es wieder klappt. |
|
||||||
| „Keine Reaktion seit 120 s“ | Ein Dateizugriff hängt (meist NFS mit `hard`). Die Regel bleibt blockiert („läuft seit …“), bis der Zugriff zurückkehrt; die Anbindung wird vor dem nächsten Lauf geprüft. |
|
| „Keine Reaktion seit 120 s“ | Ein Dateizugriff hängt (meist NFS mit `hard`). Die Regel bleibt blockiert („läuft seit …“), bis der Zugriff zurückkehrt; die Anbindung wird vor dem nächsten Lauf geprüft. |
|
||||||
| „Hostschlüssel … hat sich geändert“ (SFTP) | Wurde der Zielserver neu aufgesetzt? Dann bei der Anbindung „Hostschlüssel zurücksetzen“ und neu testen. Sonst **nicht** verbinden. |
|
| „Hostschlüssel … hat sich geändert“ (SFTP) | Wurde der Zielserver neu aufgesetzt? Dann bei der Anbindung „Hostschlüssel zurücksetzen“ und neu testen. Sonst **nicht** verbinden. |
|
||||||
| SFTP: „Anmeldung abgelehnt“ | Öffentlicher Schlüssel fehlt in `authorized_keys`, oder Rechte von `~/.ssh` stimmen nicht (700/600, Eigentümer). |
|
| SFTP: „Anmeldung abgelehnt“ | Öffentlicher Schlüssel fehlt in `authorized_keys`, oder Rechte von `~/.ssh` stimmen nicht (700/600, Eigentümer). |
|
||||||
| Paperless bemerkt neue Dateien nicht | Ältere Paperless-Versionen: `PAPERLESS_CONSUMER_POLLING=10` setzen. |
|
| Paperless bemerkt neue Dateien nicht | Ältere Paperless-Versionen: `PAPERLESS_CONSUMER_POLLING=10` setzen. |
|
||||||
|
| Neue Dateien erscheinen erst nach bis zu 30 s | Der CIFS-Client speichert Verzeichnislisten zwischen. Gelangen Dateien nicht per SMB aufs NAS (z. B. per FTP vom Scanner), erfährt er nichts davon. Abhilfe: Zusatzoption `nohandlecache`. |
|
||||||
| Regel gelb „länger als 3 × Intervall“ | Seit dem Dreifachen des Intervalls kein erfolgreicher Lauf; Protokoll der Regel ansehen. |
|
| Regel gelb „länger als 3 × Intervall“ | Seit dem Dreifachen des Intervalls kein erfolgreicher Lauf; Protokoll der Regel ansehen. |
|
||||||
| Einrichtungscode verloren | `systemctl restart netmover` erzeugt einen neuen. |
|
| Einrichtungscode verloren | `systemctl restart netmover` erzeugt einen neuen. |
|
||||||
|
|
||||||
@@ -381,7 +417,7 @@ Aufbau:
|
|||||||
| `mounting/` | `MountBackend` (Linux/Fake), Mount-Verwaltung, Test, Wiederherstellung, SFTP |
|
| `mounting/` | `MountBackend` (Linux/Fake), Mount-Verwaltung, Test, Wiederherstellung, SFTP |
|
||||||
| `mover/` | Scan, Stabilität, atomares Schreiben, Kollisionen, Buchführung, Regellauf |
|
| `mover/` | Scan, Stabilität, atomares Schreiben, Kollisionen, Buchführung, Regellauf |
|
||||||
| `runs.py`, `scheduler.py`, `health.py`, `notify.py` | Protokoll, Zeitsteuerung, Gesundheit, Webhook |
|
| `runs.py`, `scheduler.py`, `health.py`, `notify.py` | Protokoll, Zeitsteuerung, Gesundheit, Webhook |
|
||||||
| `admin.py`, `transfer.py`, `web/` | Formulare, Export/Import, Oberfläche |
|
| `admin.py`, `keys.py`, `transfer.py`, `web/` | Formulare, Schlüsselverwaltung, Export/Import, Oberfläche |
|
||||||
|
|
||||||
Die gesamte Mover-Logik ist mit dem `FakeMountBackend` ohne root testbar. SFTP
|
Die gesamte Mover-Logik ist mit dem `FakeMountBackend` ohne root testbar. SFTP
|
||||||
wird gegen einen echten SFTP-Server im Testprozess getestet.
|
wird gegen einen echten SFTP-Server im Testprozess getestet.
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@
|
|||||||
services:
|
services:
|
||||||
netmover:
|
netmover:
|
||||||
build: .
|
build: .
|
||||||
image: netmover:0.1.0
|
image: netmover:0.2.0
|
||||||
container_name: netmover
|
container_name: netmover
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
cap_add:
|
cap_add:
|
||||||
|
|||||||
+9
-3
@@ -1,8 +1,10 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# NetMover auf Debian 13 installieren bzw. aktualisieren (als root ausführen).
|
# NetMover auf Debian 13 installieren bzw. aktualisieren (als root ausführen).
|
||||||
#
|
#
|
||||||
# ./deploy/install.sh # aus diesem Verzeichnis installieren
|
# git clone https://git.wilde-server.de/florian.wilde/netmover.git /opt/netmover/src
|
||||||
# ./deploy/install.sh 'git+https://git.wilde-server.de/florian.wilde/netmover.git@v0.1.0'
|
# /opt/netmover/src/deploy/install.sh
|
||||||
|
#
|
||||||
|
# Spätere Updates: netmover-update (siehe deploy/update.sh).
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SOURCE="${1:-$(cd "$(dirname "$0")/.." && pwd)}"
|
SOURCE="${1:-$(cd "$(dirname "$0")/.." && pwd)}"
|
||||||
@@ -14,8 +16,9 @@ if [ "$(id -u)" -ne 0 ]; then
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
echo "==> Pakete"
|
echo "==> Pakete"
|
||||||
|
export DEBIAN_FRONTEND=noninteractive
|
||||||
apt-get update -qq
|
apt-get update -qq
|
||||||
apt-get install -y -qq python3 python3-venv cifs-utils nfs-common >/dev/null
|
apt-get install -y -qq python3 python3-venv cifs-utils nfs-common git >/dev/null
|
||||||
|
|
||||||
echo "==> Virtuelle Umgebung in $PREFIX/venv"
|
echo "==> Virtuelle Umgebung in $PREFIX/venv"
|
||||||
install -d -m 0755 "$PREFIX"
|
install -d -m 0755 "$PREFIX"
|
||||||
@@ -30,6 +33,9 @@ if [ ! -f /etc/netmover/netmover.env ]; then
|
|||||||
fi
|
fi
|
||||||
install -d -m 0755 /mnt/netmover
|
install -d -m 0755 /mnt/netmover
|
||||||
|
|
||||||
|
echo "==> Update-Skript"
|
||||||
|
install -m 0755 "$(dirname "$0")/update.sh" /usr/local/sbin/netmover-update
|
||||||
|
|
||||||
echo "==> systemd"
|
echo "==> systemd"
|
||||||
install -m 0644 "$(dirname "$0")/netmover.service" /etc/systemd/system/netmover.service
|
install -m 0644 "$(dirname "$0")/netmover.service" /etc/systemd/system/netmover.service
|
||||||
systemctl daemon-reload
|
systemctl daemon-reload
|
||||||
|
|||||||
@@ -27,18 +27,17 @@ RuntimeDirectory=netmover
|
|||||||
RuntimeDirectoryMode=0700
|
RuntimeDirectoryMode=0700
|
||||||
|
|
||||||
# Härtung, soweit mit Mounten vereinbar.
|
# Härtung, soweit mit Mounten vereinbar.
|
||||||
# Die Optionen unten erzeugen einen eigenen Mount-Namespace für den Dienst.
|
# WICHTIG: Keine Optionen, die einen eigenen Mount-Namespace erzeugen
|
||||||
# MountFlags=shared sorgt dafür, dass Mounts unter /mnt/netmover trotzdem im
|
# (ProtectSystem=, ProtectHome=, PrivateTmp=, ProtectControlGroups=,
|
||||||
# System sichtbar sind (prüfen mit: findmnt -R /mnt/netmover).
|
# ReadOnlyPaths=, BindPaths= …). Sonst landen die Mounts im Namespace des
|
||||||
MountFlags=shared
|
# Dienstes: das System und andere Programme (Paperless!) sähen sie nicht, und
|
||||||
# /usr, /boot und /etc schreibgeschützt; /mnt, /opt, /srv, /var bleiben schreibbar
|
# jeder Neustart des Dienstes würde sie verwerfen. MountFlags=shared hilft dagegen
|
||||||
# (lokale Ziele wie /opt/paperless/consume funktionieren also).
|
# nicht – systemd setzt den Namespace als „slave“ auf, Mounts fließen nur hinein.
|
||||||
ProtectSystem=full
|
# Auf dem Testsystem nachgewiesen; prüfen mit: findmnt -l | grep /mnt/netmover
|
||||||
PrivateTmp=yes
|
|
||||||
NoNewPrivileges=yes
|
NoNewPrivileges=yes
|
||||||
ProtectControlGroups=yes
|
|
||||||
LockPersonality=yes
|
LockPersonality=yes
|
||||||
RestrictRealtime=yes
|
RestrictRealtime=yes
|
||||||
|
RestrictSUIDSGID=yes
|
||||||
UMask=0022
|
UMask=0022
|
||||||
|
|
||||||
[Install]
|
[Install]
|
||||||
|
|||||||
Executable
+151
@@ -0,0 +1,151 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# NetMover aktualisieren (als root). Wird von install.sh als
|
||||||
|
# /usr/local/sbin/netmover-update installiert.
|
||||||
|
#
|
||||||
|
# netmover-update # auf die neueste Version (neuester Tag vX.Y.Z)
|
||||||
|
# netmover-update v0.2.0 # auf eine bestimmte Version (auch zurück)
|
||||||
|
# netmover-update --list # verfügbare Versionen anzeigen
|
||||||
|
#
|
||||||
|
# Ablauf: Tags holen → Datenbank und Schlüssel sichern → Version auschecken →
|
||||||
|
# install.sh (installiert, migriert beim Start, startet den Dienst neu) →
|
||||||
|
# Gesundheit prüfen. Die Mounts bleiben dabei eingehängt.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
SRC="${NETMOVER_SRC:-/opt/netmover/src}"
|
||||||
|
VENV=/opt/netmover/venv
|
||||||
|
ENV_FILE=/etc/netmover/netmover.env
|
||||||
|
BACKUP_ROOT=/var/backups/netmover
|
||||||
|
KEEP_BACKUPS=10
|
||||||
|
|
||||||
|
if [ "$(id -u)" -ne 0 ]; then
|
||||||
|
echo "Bitte als root ausführen." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if [ ! -d "$SRC/.git" ]; then
|
||||||
|
echo "Kein Git-Klon unter $SRC. Einmalig einrichten:" >&2
|
||||||
|
echo " git clone https://git.wilde-server.de/florian.wilde/netmover.git $SRC" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Einstellungen des Dienstes (Port, Datenverzeichnis)
|
||||||
|
NETMOVER_PORT=8080
|
||||||
|
NETMOVER_DATA_DIR=/var/lib/netmover
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
[ -f "$ENV_FILE" ] && . "$ENV_FILE"
|
||||||
|
|
||||||
|
cd "$SRC"
|
||||||
|
git fetch --tags --quiet origin
|
||||||
|
|
||||||
|
if [ "${1:-}" = "--list" ]; then
|
||||||
|
git tag -l 'v*' --sort=-v:refname
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
TARGET="${1:-$(git tag -l 'v*' --sort=-v:refname | head -n 1)}"
|
||||||
|
if ! git rev-parse -q --verify "refs/tags/$TARGET" >/dev/null; then
|
||||||
|
echo "Version $TARGET gibt es nicht. Verfügbar:" >&2
|
||||||
|
git tag -l 'v*' --sort=-v:refname | head -n 10 >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
CURRENT="v$("$VENV/bin/netmover" version 2>/dev/null || echo "?")"
|
||||||
|
echo "Installiert: $CURRENT"
|
||||||
|
echo "Ziel: $TARGET"
|
||||||
|
if [ "$CURRENT" = "$TARGET" ] && [ "${FORCE:-0}" != "1" ]; then
|
||||||
|
echo "Bereits auf dem gewünschten Stand (FORCE=1 erzwingt eine Neuinstallation)."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
NEWEST="$(printf '%s\n%s\n' "$CURRENT" "$TARGET" | sort -V | tail -n 1)"
|
||||||
|
if [ "$NEWEST" = "$CURRENT" ] && [ "$CURRENT" != "$TARGET" ]; then
|
||||||
|
echo
|
||||||
|
echo "!! Zurückstufen auf eine ältere Version: Wurde die Datenbank inzwischen migriert,"
|
||||||
|
echo " startet die alte Version nicht. Dann vorher die Sicherung von *vor* dem Update"
|
||||||
|
echo " zurückspielen (siehe $BACKUP_ROOT, Anleitung im README)."
|
||||||
|
printf " Trotzdem fortfahren? [j/N] "
|
||||||
|
read -r answer
|
||||||
|
[ "$answer" = "j" ] || [ "$answer" = "J" ] || exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "==> Änderungen laut Changelog"
|
||||||
|
git show "$TARGET:CHANGELOG.md" | awk -v cur="${CURRENT#v}" '
|
||||||
|
/^## \[/ { if (index($0, "[" cur "]")) exit; show = ($0 !~ /Unreleased/) }
|
||||||
|
show' | head -n 60
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "==> Sicherung"
|
||||||
|
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||||
|
BACKUP="$BACKUP_ROOT/$STAMP-$CURRENT"
|
||||||
|
install -d -m 0700 "$BACKUP_ROOT" "$BACKUP"
|
||||||
|
# Konsistente Kopie auch bei laufendem Dienst (SQLite-Online-Backup)
|
||||||
|
"$VENV/bin/python" - "$NETMOVER_DATA_DIR/netmover.db" "$BACKUP/netmover.db" <<'EOF'
|
||||||
|
import sqlite3, sys
|
||||||
|
src, dst = sqlite3.connect(sys.argv[1]), sqlite3.connect(sys.argv[2])
|
||||||
|
with dst:
|
||||||
|
src.backup(dst)
|
||||||
|
src.close(); dst.close()
|
||||||
|
EOF
|
||||||
|
for f in secret.key session.key; do
|
||||||
|
[ -f "$NETMOVER_DATA_DIR/$f" ] && cp -p "$NETMOVER_DATA_DIR/$f" "$BACKUP/"
|
||||||
|
done
|
||||||
|
[ -f "$ENV_FILE" ] && cp -p "$ENV_FILE" "$BACKUP/"
|
||||||
|
chmod 0600 "$BACKUP"/*
|
||||||
|
echo " $BACKUP"
|
||||||
|
# alte Sicherungen aufräumen
|
||||||
|
find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -print | sort \
|
||||||
|
| head -n "-$KEEP_BACKUPS" | xargs -r rm -rf --
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "==> Version $TARGET auschecken und installieren"
|
||||||
|
git -c advice.detachedHead=false checkout --quiet "$TARGET"
|
||||||
|
"$SRC/deploy/install.sh" "$SRC" >/tmp/netmover-update.log 2>&1 || {
|
||||||
|
echo "install.sh ist fehlgeschlagen – Ausgabe: /tmp/netmover-update.log" >&2
|
||||||
|
tail -n 20 /tmp/netmover-update.log >&2
|
||||||
|
FAILED=1
|
||||||
|
}
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "==> Prüfung"
|
||||||
|
OK=0
|
||||||
|
if [ "${FAILED:-0}" != "1" ]; then
|
||||||
|
i=0
|
||||||
|
while [ $i -lt 30 ]; do
|
||||||
|
if "$VENV/bin/python" - "$NETMOVER_PORT" <<'EOF'
|
||||||
|
import json, sys, urllib.error, urllib.request
|
||||||
|
url = f"http://127.0.0.1:{sys.argv[1]}/healthz"
|
||||||
|
try:
|
||||||
|
body = urllib.request.urlopen(url, timeout=3).read()
|
||||||
|
except urllib.error.HTTPError as exc: # 503 = läuft, meldet aber Probleme
|
||||||
|
body = exc.read()
|
||||||
|
except OSError:
|
||||||
|
sys.exit(1)
|
||||||
|
data = json.loads(body)
|
||||||
|
print(f" Gesundheit: {data['status']}")
|
||||||
|
for p in data.get("problems", []):
|
||||||
|
print(f" – {p}")
|
||||||
|
EOF
|
||||||
|
then
|
||||||
|
OK=1
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
i=$((i + 1))
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$OK" = "1" ] && systemctl is-active --quiet netmover; then
|
||||||
|
echo " Dienst läuft: $("$VENV/bin/netmover" version)"
|
||||||
|
echo
|
||||||
|
echo "Fertig. Sicherung: $BACKUP"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "!! Der Dienst läuft nach dem Update nicht richtig." >&2
|
||||||
|
echo " Log: journalctl -u netmover -n 50" >&2
|
||||||
|
echo " Zurück auf $CURRENT (Datenbank aus der Sicherung, weil sie migriert sein kann):" >&2
|
||||||
|
echo " systemctl stop netmover" >&2
|
||||||
|
echo " cp $BACKUP/netmover.db $NETMOVER_DATA_DIR/netmover.db" >&2
|
||||||
|
echo " rm -f $NETMOVER_DATA_DIR/netmover.db-wal $NETMOVER_DATA_DIR/netmover.db-shm" >&2
|
||||||
|
echo " netmover-update $CURRENT" >&2
|
||||||
|
exit 1
|
||||||
@@ -33,6 +33,7 @@ dev = [
|
|||||||
"ruff>=0.6",
|
"ruff>=0.6",
|
||||||
"mypy>=1.11",
|
"mypy>=1.11",
|
||||||
"types-paramiko",
|
"types-paramiko",
|
||||||
|
"shellcheck-py",
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
||||||
|
|
||||||
__version__ = "0.1.0"
|
__version__ = "0.2.0"
|
||||||
|
|||||||
+53
-14
@@ -84,6 +84,9 @@ def _int(
|
|||||||
# -- Anbindungen ------------------------------------------------------------------------
|
# -- Anbindungen ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
NEW_KEY = "new"
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class MountInput:
|
class MountInput:
|
||||||
name: str = ""
|
name: str = ""
|
||||||
@@ -95,6 +98,9 @@ class MountInput:
|
|||||||
password: str = "" # leer = unverändert
|
password: str = "" # leer = unverändert
|
||||||
domain: str = ""
|
domain: str = ""
|
||||||
auth: CredentialKind = CredentialKind.PASSWORD # nur sftp: Schlüssel oder Passwort
|
auth: CredentialKind = CredentialKind.PASSWORD # nur sftp: Schlüssel oder Passwort
|
||||||
|
# nur sftp + Schlüssel: "new" = eigener Schlüssel nur für diese Anbindung,
|
||||||
|
# sonst ID eines Schlüssels aus der Schlüsselverwaltung
|
||||||
|
key_id: str = NEW_KEY
|
||||||
uid: int | None = None
|
uid: int | None = None
|
||||||
gid: int | None = None
|
gid: int | None = None
|
||||||
file_mode: str | None = None
|
file_mode: str | None = None
|
||||||
@@ -126,9 +132,10 @@ def mount_input_from_model(m: Mount) -> MountInput:
|
|||||||
server=m.server,
|
server=m.server,
|
||||||
share=m.share,
|
share=m.share,
|
||||||
port=m.port,
|
port=m.port,
|
||||||
username=c.username if c else "",
|
username=m.username or (c.username if c else ""),
|
||||||
domain=(c.domain or "") if c else "",
|
domain=(c.domain or "") if c else "",
|
||||||
auth=c.kind if c else CredentialKind.PASSWORD,
|
auth=c.kind if c else CredentialKind.PASSWORD,
|
||||||
|
key_id=str(c.id) if c is not None and c.managed else NEW_KEY,
|
||||||
uid=m.uid,
|
uid=m.uid,
|
||||||
gid=m.gid,
|
gid=m.gid,
|
||||||
file_mode=m.file_mode,
|
file_mode=m.file_mode,
|
||||||
@@ -151,7 +158,11 @@ def parse_mount_form(form: Mapping[str, str]) -> tuple[MountInput, Errors]:
|
|||||||
extra_options=(form.get("extra_options") or "").strip(),
|
extra_options=(form.get("extra_options") or "").strip(),
|
||||||
auto_mount=_bool(form, "auto_mount"),
|
auto_mount=_bool(form, "auto_mount"),
|
||||||
enabled=_bool(form, "enabled"),
|
enabled=_bool(form, "enabled"),
|
||||||
|
key_id=(form.get("key_id") or NEW_KEY).strip(),
|
||||||
)
|
)
|
||||||
|
if data.key_id != NEW_KEY and not data.key_id.isdigit():
|
||||||
|
errors["key_id"] = "Ungültige Schlüsselauswahl."
|
||||||
|
data.key_id = NEW_KEY
|
||||||
try:
|
try:
|
||||||
data.type = MountType(form.get("type", "cifs"))
|
data.type = MountType(form.get("type", "cifs"))
|
||||||
except ValueError:
|
except ValueError:
|
||||||
@@ -203,17 +214,27 @@ def parse_mount_form(form: Mapping[str, str]) -> tuple[MountInput, Errors]:
|
|||||||
errors["username"] = "SFTP braucht einen Benutzernamen."
|
errors["username"] = "SFTP braucht einen Benutzernamen."
|
||||||
if data.type is not MountType.SFTP:
|
if data.type is not MountType.SFTP:
|
||||||
data.auth = CredentialKind.PASSWORD
|
data.auth = CredentialKind.PASSWORD
|
||||||
|
if data.auth is not CredentialKind.SSH_KEY:
|
||||||
|
data.key_id = NEW_KEY
|
||||||
return data, errors
|
return data, errors
|
||||||
|
|
||||||
|
|
||||||
def credential_for_test(
|
def credential_for_test(
|
||||||
data: MountInput, existing: Mount | None, secrets: SecretBox
|
data: MountInput, existing: Mount | None, secrets: SecretBox, s: Session | None = None
|
||||||
) -> CredentialData | None:
|
) -> CredentialData | None:
|
||||||
"""Zugangsdaten für „Testen“ aus dem Formular – leeres Passwort = gespeichertes."""
|
"""Zugangsdaten für „Testen“ aus dem Formular – leeres Passwort = gespeichertes,
|
||||||
|
gewählter Schlüssel aus der Schlüsselverwaltung."""
|
||||||
if data.type in (MountType.NFS3, MountType.NFS4) or not data.username:
|
if data.type in (MountType.NFS3, MountType.NFS4) or not data.username:
|
||||||
return None
|
return None
|
||||||
|
if data.auth is CredentialKind.SSH_KEY and data.key_id != NEW_KEY and s is not None:
|
||||||
|
key = s.get(Credential, int(data.key_id))
|
||||||
|
if key is None or not key.managed:
|
||||||
|
return None
|
||||||
|
return CredentialData(data.auth, data.username, secrets.decrypt(key.secret_enc))
|
||||||
secret = data.password
|
secret = data.password
|
||||||
stored = existing.credential if existing is not None else None
|
stored = existing.credential if existing is not None else None
|
||||||
|
if stored is not None and stored.managed:
|
||||||
|
stored = None # eigener Schlüssel gewählt, gespeicherter ist nicht gemeint
|
||||||
if not secret and stored is not None and stored.kind is data.auth:
|
if not secret and stored is not None and stored.kind is data.auth:
|
||||||
secret = secrets.decrypt(stored.secret_enc)
|
secret = secrets.decrypt(stored.secret_enc)
|
||||||
if not secret:
|
if not secret:
|
||||||
@@ -296,20 +317,32 @@ def save_mount(
|
|||||||
if not data.type.is_kernel_mount:
|
if not data.type.is_kernel_mount:
|
||||||
m.auto_mount = False
|
m.auto_mount = False
|
||||||
|
|
||||||
# Zugangsdaten: jede Anbindung pflegt ihre eigenen.
|
# Zugangsdaten: entweder ein gespeicherter Schlüssel aus der Schlüsselverwaltung
|
||||||
cred = m.credential
|
# (geteilt, wird hier nie verändert) oder eigene Zugangsdaten nur dieser Anbindung.
|
||||||
|
current = m.credential
|
||||||
|
own = current if current is not None and not current.managed else None
|
||||||
needs_cred = data.type is MountType.SFTP or (
|
needs_cred = data.type is MountType.SFTP or (
|
||||||
data.type is MountType.CIFS and bool(data.username)
|
data.type is MountType.CIFS and bool(data.username)
|
||||||
)
|
)
|
||||||
|
use_managed = (
|
||||||
|
data.type is MountType.SFTP
|
||||||
|
and data.auth is CredentialKind.SSH_KEY
|
||||||
|
and data.key_id != NEW_KEY
|
||||||
|
)
|
||||||
if not needs_cred:
|
if not needs_cred:
|
||||||
if cred is not None:
|
|
||||||
m.credential = None
|
m.credential = None
|
||||||
s.flush()
|
m.username = None
|
||||||
if not cred.mounts:
|
elif use_managed:
|
||||||
s.delete(cred)
|
key = s.get(Credential, int(data.key_id))
|
||||||
|
if key is None or not key.managed or key.kind is not CredentialKind.SSH_KEY:
|
||||||
|
s.rollback()
|
||||||
|
return None, {"key_id": "Schlüssel existiert nicht."}
|
||||||
|
m.credential = key
|
||||||
|
m.username = data.username
|
||||||
else:
|
else:
|
||||||
kind_changed = cred is not None and cred.kind is not data.auth
|
m.username = None
|
||||||
if cred is None or kind_changed:
|
cred = own
|
||||||
|
if cred is None or cred.kind is not data.auth:
|
||||||
if data.auth is CredentialKind.SSH_KEY:
|
if data.auth is CredentialKind.SSH_KEY:
|
||||||
private, public = generate_ssh_keypair(f"netmover@{m.slug}")
|
private, public = generate_ssh_keypair(f"netmover@{m.slug}")
|
||||||
secret, pub = private, public
|
secret, pub = private, public
|
||||||
@@ -320,9 +353,8 @@ def save_mount(
|
|||||||
return None, errors
|
return None, errors
|
||||||
secret, pub = data.password, None
|
secret, pub = data.password, None
|
||||||
if cred is None:
|
if cred is None:
|
||||||
cred = Credential(name=f"mount:{m.slug}")
|
cred = Credential(name=f"mount:{m.slug}", managed=False)
|
||||||
s.add(cred)
|
s.add(cred)
|
||||||
m.credential = cred
|
|
||||||
cred.kind = data.auth
|
cred.kind = data.auth
|
||||||
cred.secret_enc = secrets.encrypt(secret)
|
cred.secret_enc = secrets.encrypt(secret)
|
||||||
cred.public_key = pub
|
cred.public_key = pub
|
||||||
@@ -330,6 +362,12 @@ def save_mount(
|
|||||||
cred.secret_enc = secrets.encrypt(data.password)
|
cred.secret_enc = secrets.encrypt(data.password)
|
||||||
cred.username = data.username
|
cred.username = data.username
|
||||||
cred.domain = data.domain or None
|
cred.domain = data.domain or None
|
||||||
|
m.credential = cred
|
||||||
|
# Eigene Zugangsdaten, die nicht mehr gebraucht werden, entfernen.
|
||||||
|
if own is not None and m.credential is not own:
|
||||||
|
s.flush()
|
||||||
|
if not own.mounts:
|
||||||
|
s.delete(own)
|
||||||
try:
|
try:
|
||||||
s.flush()
|
s.flush()
|
||||||
except IntegrityError:
|
except IntegrityError:
|
||||||
@@ -353,7 +391,8 @@ def delete_mount(s: Session, mounts: MountManager, mount_id: int) -> None:
|
|||||||
cred = m.credential
|
cred = m.credential
|
||||||
s.delete(m)
|
s.delete(m)
|
||||||
s.flush()
|
s.flush()
|
||||||
if cred is not None and not cred.mounts:
|
# Eigene Zugangsdaten gehen mit; Schlüssel aus der Schlüsselverwaltung bleiben.
|
||||||
|
if cred is not None and not cred.managed and not cred.mounts:
|
||||||
s.delete(cred)
|
s.delete(cred)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -58,7 +58,9 @@ class NetMover:
|
|||||||
)
|
)
|
||||||
self.runner = RuleRunner(self.session_factory, self.mounts)
|
self.runner = RuleRunner(self.session_factory, self.mounts)
|
||||||
self.runs = RunService(self.session_factory, self.runner, self.notifier)
|
self.runs = RunService(self.session_factory, self.runner, self.notifier)
|
||||||
self.scheduler = RuleScheduler(self.session_factory, self.runs)
|
self.scheduler = RuleScheduler(
|
||||||
|
self.session_factory, self.runs, retry_mounts=self.mounts.retry_pending
|
||||||
|
)
|
||||||
self.started_at: datetime = utcnow()
|
self.started_at: datetime = utcnow()
|
||||||
self.restore_results: list[RestoreResult] = []
|
self.restore_results: list[RestoreResult] = []
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,197 @@
|
|||||||
|
"""Schlüsselverwaltung: gespeicherte SSH-Schlüssel für SFTP-Anbindungen.
|
||||||
|
|
||||||
|
* Erzeugen (Ed25519) oder Importieren (Ed25519, ECDSA, RSA ≥ 2048 Bit; OpenSSH-
|
||||||
|
oder PEM-Format, optional mit Passphrase). Eine Passphrase wird beim Import
|
||||||
|
einmal benutzt; gespeichert wird der Schlüssel ohne Passphrase, aber mit
|
||||||
|
Fernet verschlüsselt – wie alle Geheimnisse.
|
||||||
|
* Der private Schlüssel verlässt NetMover nie: Die Oberfläche zeigt nur den
|
||||||
|
öffentlichen Schlüssel und den Fingerabdruck.
|
||||||
|
* Ein Schlüssel kann von mehreren Anbindungen genutzt werden; den Benutzernamen
|
||||||
|
legt jede Anbindung selbst fest. Löschen geht nur, solange ihn keine nutzt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import hashlib
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from cryptography.hazmat.primitives import serialization
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ec, ed25519, rsa
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from netmover.admin import AdminError
|
||||||
|
from netmover.crypto import SecretBox
|
||||||
|
from netmover.models import Credential, CredentialKind, Mount
|
||||||
|
|
||||||
|
MAX_KEY_BYTES = 16 * 1024
|
||||||
|
MIN_RSA_BITS = 2048
|
||||||
|
SFTP_ONLY_PREFIX = 'restrict,command="/usr/lib/openssh/sftp-server" '
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class KeyInfo:
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
key_type: str
|
||||||
|
fingerprint: str
|
||||||
|
public_key: str
|
||||||
|
created_at: datetime
|
||||||
|
used_by: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def authorized_keys_line(self) -> str:
|
||||||
|
"""Zeile für ~/.ssh/authorized_keys, auf SFTP beschränkt."""
|
||||||
|
return SFTP_ONLY_PREFIX + self.public_key
|
||||||
|
|
||||||
|
|
||||||
|
def fingerprint(public_key: str) -> str:
|
||||||
|
"""SHA256-Fingerabdruck wie ``ssh-keygen -lf``."""
|
||||||
|
blob = base64.b64decode(public_key.split()[1])
|
||||||
|
return "SHA256:" + base64.b64encode(hashlib.sha256(blob).digest()).decode().rstrip("=")
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_name(s: Session, name: str, exclude_id: int | None = None) -> str:
|
||||||
|
name = name.strip()
|
||||||
|
if not name:
|
||||||
|
raise AdminError("Name fehlt.")
|
||||||
|
if len(name) > 100:
|
||||||
|
raise AdminError("Name höchstens 100 Zeichen.")
|
||||||
|
if name.startswith("mount:"):
|
||||||
|
raise AdminError("Namen mit „mount:“ sind für NetMover reserviert.")
|
||||||
|
dup = s.scalar(select(Credential.id).where(Credential.name == name))
|
||||||
|
if dup is not None and dup != exclude_id:
|
||||||
|
raise AdminError(f"Ein Schlüssel „{name}“ existiert bereits.")
|
||||||
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
def _comment(name: str) -> str:
|
||||||
|
return "netmover:" + "".join(c if c.isalnum() or c in "-_." else "-" for c in name)
|
||||||
|
|
||||||
|
|
||||||
|
PrivateKey = ed25519.Ed25519PrivateKey | ec.EllipticCurvePrivateKey | rsa.RSAPrivateKey
|
||||||
|
|
||||||
|
|
||||||
|
def _store(s: Session, secrets: SecretBox, name: str, key: PrivateKey) -> Credential:
|
||||||
|
private = key.private_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PrivateFormat.OpenSSH,
|
||||||
|
serialization.NoEncryption(),
|
||||||
|
).decode()
|
||||||
|
public = (
|
||||||
|
key.public_key()
|
||||||
|
.public_bytes(serialization.Encoding.OpenSSH, serialization.PublicFormat.OpenSSH)
|
||||||
|
.decode()
|
||||||
|
)
|
||||||
|
cred = Credential(
|
||||||
|
name=name,
|
||||||
|
kind=CredentialKind.SSH_KEY,
|
||||||
|
username="",
|
||||||
|
secret_enc=secrets.encrypt(private),
|
||||||
|
public_key=f"{public} {_comment(name)}",
|
||||||
|
managed=True,
|
||||||
|
)
|
||||||
|
s.add(cred)
|
||||||
|
s.flush()
|
||||||
|
return cred
|
||||||
|
|
||||||
|
|
||||||
|
def generate_key(s: Session, secrets: SecretBox, name: str) -> Credential:
|
||||||
|
return _store(s, secrets, _validate_name(s, name), ed25519.Ed25519PrivateKey.generate())
|
||||||
|
|
||||||
|
|
||||||
|
def import_key(
|
||||||
|
s: Session, secrets: SecretBox, name: str, private_key: str, passphrase: str = ""
|
||||||
|
) -> Credential:
|
||||||
|
name = _validate_name(s, name)
|
||||||
|
data = private_key.strip().encode()
|
||||||
|
if not data:
|
||||||
|
raise AdminError("Privaten Schlüssel einfügen.")
|
||||||
|
if len(data) > MAX_KEY_BYTES:
|
||||||
|
raise AdminError("Schlüssel ist zu groß.")
|
||||||
|
if b"PUBLIC KEY" in data or data.startswith((b"ssh-", b"ecdsa-")):
|
||||||
|
raise AdminError("Das ist ein öffentlicher Schlüssel – benötigt wird der private.")
|
||||||
|
password = passphrase.encode() if passphrase else None
|
||||||
|
loaders = (
|
||||||
|
[serialization.load_ssh_private_key]
|
||||||
|
if b"OPENSSH PRIVATE KEY" in data
|
||||||
|
else [serialization.load_pem_private_key]
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
key = loaders[0](data, password)
|
||||||
|
except TypeError:
|
||||||
|
raise AdminError(
|
||||||
|
"Der Schlüssel ist mit einer Passphrase geschützt – bitte Passphrase angeben."
|
||||||
|
if password is None
|
||||||
|
else "Der Schlüssel hat keine Passphrase – Feld bitte leer lassen."
|
||||||
|
) from None
|
||||||
|
except ValueError as exc:
|
||||||
|
msg = str(exc).lower()
|
||||||
|
if password is not None or "password" in msg or "decrypt" in msg:
|
||||||
|
# Bei OpenSSH-Schlüsseln meldet cryptography eine falsche Passphrase
|
||||||
|
# als „Corrupt data“ – beides lässt sich nicht unterscheiden.
|
||||||
|
raise AdminError("Passphrase falsch (oder Schlüssel beschädigt).") from None
|
||||||
|
raise AdminError(
|
||||||
|
"Kein lesbarer privater Schlüssel (erwartet OpenSSH- oder PEM-Format)."
|
||||||
|
) from None
|
||||||
|
except Exception: # unbekannte Formate/Algorithmen sauber melden
|
||||||
|
raise AdminError("Schlüsselformat wird nicht unterstützt.") from None
|
||||||
|
if not isinstance(
|
||||||
|
key, ed25519.Ed25519PrivateKey | ec.EllipticCurvePrivateKey | rsa.RSAPrivateKey
|
||||||
|
):
|
||||||
|
raise AdminError("Unterstützt werden Ed25519, ECDSA und RSA.")
|
||||||
|
if isinstance(key, rsa.RSAPrivateKey) and key.key_size < MIN_RSA_BITS:
|
||||||
|
raise AdminError(f"RSA-Schlüssel mit {key.key_size} Bit ist zu schwach (mind. 2048).")
|
||||||
|
return _store(s, secrets, name, key)
|
||||||
|
|
||||||
|
|
||||||
|
def _users(s: Session, key_id: int) -> list[str]:
|
||||||
|
# Direkt abfragen statt key.mounts: die Beziehung ist nach dem Löschen von
|
||||||
|
# Anbindungen in derselben Session veraltet (passive_deletes).
|
||||||
|
return sorted(s.scalars(select(Mount.name).where(Mount.cred_id == key_id)))
|
||||||
|
|
||||||
|
|
||||||
|
def list_keys(s: Session) -> list[KeyInfo]:
|
||||||
|
keys = s.scalars(
|
||||||
|
select(Credential).where(Credential.managed.is_(True)).order_by(Credential.name)
|
||||||
|
).all()
|
||||||
|
result = []
|
||||||
|
for k in keys:
|
||||||
|
pub = k.public_key or ""
|
||||||
|
result.append(
|
||||||
|
KeyInfo(
|
||||||
|
id=k.id,
|
||||||
|
name=k.name,
|
||||||
|
key_type=pub.split()[0] if pub else "?",
|
||||||
|
fingerprint=fingerprint(pub) if pub else "?",
|
||||||
|
public_key=pub,
|
||||||
|
created_at=k.created_at,
|
||||||
|
used_by=_users(s, k.id),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def managed_key(s: Session, key_id: int) -> Credential:
|
||||||
|
key = s.get(Credential, key_id)
|
||||||
|
if key is None or not key.managed or key.kind is not CredentialKind.SSH_KEY:
|
||||||
|
raise AdminError("Schlüssel existiert nicht.")
|
||||||
|
return key
|
||||||
|
|
||||||
|
|
||||||
|
def rename_key(s: Session, key_id: int, name: str) -> None:
|
||||||
|
key = managed_key(s, key_id)
|
||||||
|
key.name = _validate_name(s, name, exclude_id=key_id)
|
||||||
|
|
||||||
|
|
||||||
|
def delete_key(s: Session, key_id: int) -> str:
|
||||||
|
key = managed_key(s, key_id)
|
||||||
|
users = _users(s, key_id)
|
||||||
|
if users:
|
||||||
|
raise AdminError(f"Schlüssel „{key.name}“ wird noch benutzt von: {', '.join(users)}")
|
||||||
|
name = key.name
|
||||||
|
s.delete(key)
|
||||||
|
s.flush()
|
||||||
|
return name
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
"""key management
|
||||||
|
|
||||||
|
Revision ID: 0002
|
||||||
|
Revises: 0001
|
||||||
|
Create Date: 2026-10-10
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision: str = "0002"
|
||||||
|
down_revision: str | None = "0001"
|
||||||
|
branch_labels: str | Sequence[str] | None = None
|
||||||
|
depends_on: str | Sequence[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
with op.batch_alter_table("credentials", schema=None) as batch_op:
|
||||||
|
batch_op.add_column(
|
||||||
|
sa.Column("managed", sa.Boolean(), server_default=sa.text("0"), nullable=False)
|
||||||
|
)
|
||||||
|
|
||||||
|
with op.batch_alter_table("mounts", schema=None) as batch_op:
|
||||||
|
batch_op.add_column(sa.Column("username", sa.String(length=255), nullable=True))
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("mounts", schema=None) as batch_op:
|
||||||
|
batch_op.drop_column("username")
|
||||||
|
|
||||||
|
with op.batch_alter_table("credentials", schema=None) as batch_op:
|
||||||
|
batch_op.drop_column("managed")
|
||||||
@@ -7,6 +7,9 @@ Abweichungen bzw. Ergänzungen gegenüber dem Mindestumfang aus der Spezifikatio
|
|||||||
(beim ersten Test festgehaltener Hostschlüssel, danach verbindlich).
|
(beim ersten Test festgehaltener Hostschlüssel, danach verbindlich).
|
||||||
* ``credentials.kind`` (``password`` | ``ssh_key``) und ``credentials.public_key``.
|
* ``credentials.kind`` (``password`` | ``ssh_key``) und ``credentials.public_key``.
|
||||||
* ``credentials.domain`` – Windows-/AD-Freigaben brauchen oft eine Domäne.
|
* ``credentials.domain`` – Windows-/AD-Freigaben brauchen oft eine Domäne.
|
||||||
|
* ``credentials.managed`` + ``mounts.username`` (Migration 0002) –
|
||||||
|
Schlüsselverwaltung: gespeicherte SSH-Schlüssel, die mehrere Anbindungen mit
|
||||||
|
jeweils eigenem Benutzernamen nutzen.
|
||||||
* ``mounts.state_since`` – für „hängt seit …“ / „gestört seit …“ auf der Statusseite.
|
* ``mounts.state_since`` – für „hängt seit …“ / „gestört seit …“ auf der Statusseite.
|
||||||
* ``mounts.write_ok`` / ``mounts.root_squash`` / ``mounts.last_test_*`` – Ergebnis
|
* ``mounts.write_ok`` / ``mounts.root_squash`` / ``mounts.last_test_*`` – Ergebnis
|
||||||
des letzten Mount-Tests; ``root_squash`` entscheidet, ob ``chown`` auf NFS-Zielen
|
des letzten Mount-Tests; ``root_squash`` entscheidet, ob ``chown`` auf NFS-Zielen
|
||||||
@@ -38,6 +41,7 @@ from sqlalchemy import (
|
|||||||
TypeDecorator,
|
TypeDecorator,
|
||||||
UniqueConstraint,
|
UniqueConstraint,
|
||||||
)
|
)
|
||||||
|
from sqlalchemy import false as sa_false
|
||||||
from sqlalchemy.engine import Dialect
|
from sqlalchemy.engine import Dialect
|
||||||
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
|
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
|
||||||
|
|
||||||
@@ -166,6 +170,9 @@ class Credential(Base):
|
|||||||
domain: Mapped[str | None] = mapped_column(String(255))
|
domain: Mapped[str | None] = mapped_column(String(255))
|
||||||
secret_enc: Mapped[str] = mapped_column(Text) # Passwort bzw. privater Schlüssel (Fernet)
|
secret_enc: Mapped[str] = mapped_column(Text) # Passwort bzw. privater Schlüssel (Fernet)
|
||||||
public_key: Mapped[str | None] = mapped_column(Text) # nur ssh_key, OpenSSH-Format
|
public_key: Mapped[str | None] = mapped_column(Text) # nur ssh_key, OpenSSH-Format
|
||||||
|
# True: in der Schlüsselverwaltung gepflegt, von mehreren Anbindungen nutzbar.
|
||||||
|
# False: gehört genau einer Anbindung und wird mit ihr gelöscht.
|
||||||
|
managed: Mapped[bool] = mapped_column(Boolean, default=False, server_default=sa_false())
|
||||||
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
# passive_deletes="all": das ORM soll cred_id nicht still auf NULL setzen,
|
# passive_deletes="all": das ORM soll cred_id nicht still auf NULL setzen,
|
||||||
@@ -183,6 +190,9 @@ class Mount(Base):
|
|||||||
server: Mapped[str] = mapped_column(String(255))
|
server: Mapped[str] = mapped_column(String(255))
|
||||||
share: Mapped[str] = mapped_column(String(1024))
|
share: Mapped[str] = mapped_column(String(1024))
|
||||||
port: Mapped[int | None] = mapped_column(Integer) # nur sftp
|
port: Mapped[int | None] = mapped_column(Integer) # nur sftp
|
||||||
|
# Benutzer für die Anmeldung; überschreibt credentials.username. Nötig, weil ein
|
||||||
|
# gespeicherter SSH-Schlüssel für verschiedene Server/Benutzer gelten kann.
|
||||||
|
username: Mapped[str | None] = mapped_column(String(255))
|
||||||
host_key: Mapped[str | None] = mapped_column(Text) # nur sftp, "<typ> <base64>"
|
host_key: Mapped[str | None] = mapped_column(Text) # nur sftp, "<typ> <base64>"
|
||||||
mountpoint: Mapped[str | None] = mapped_column(String(1024), unique=True) # NULL bei sftp
|
mountpoint: Mapped[str | None] = mapped_column(String(1024), unique=True) # NULL bei sftp
|
||||||
cred_id: Mapped[int | None] = mapped_column(ForeignKey("credentials.id", ondelete="RESTRICT"))
|
cred_id: Mapped[int | None] = mapped_column(ForeignKey("credentials.id", ondelete="RESTRICT"))
|
||||||
|
|||||||
@@ -8,7 +8,9 @@ Optionszeile – die stünde sonst in der Prozessliste.
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
import os
|
import os
|
||||||
|
import signal
|
||||||
import subprocess
|
import subprocess
|
||||||
import uuid
|
import uuid
|
||||||
from collections.abc import Callable, Sequence
|
from collections.abc import Callable, Sequence
|
||||||
@@ -24,6 +26,28 @@ log = get_logger(__name__)
|
|||||||
|
|
||||||
Runner = Callable[..., "subprocess.CompletedProcess[str]"]
|
Runner = Callable[..., "subprocess.CompletedProcess[str]"]
|
||||||
|
|
||||||
|
|
||||||
|
def run_process_group(argv: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||||
|
"""Wie ``subprocess.run``, aber in eigener Prozessgruppe: Bei Zeitüberschreitung
|
||||||
|
wird die ganze Gruppe beendet. ``mount`` startet Hilfsprogramme wie
|
||||||
|
``mount.nfs``; würde nur ``mount`` beendet, liefe ``mount.nfs`` weiter und
|
||||||
|
hängte womöglich Minuten später doch noch ein (auf der Test-VM beobachtet)."""
|
||||||
|
timeout = kwargs.pop("timeout", None)
|
||||||
|
kwargs.pop("check", None)
|
||||||
|
capture = kwargs.pop("capture_output", False)
|
||||||
|
if capture:
|
||||||
|
kwargs["stdout"] = kwargs["stderr"] = subprocess.PIPE
|
||||||
|
with subprocess.Popen(argv, start_new_session=True, **kwargs) as proc:
|
||||||
|
try:
|
||||||
|
out, err = proc.communicate(timeout=timeout)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
with contextlib.suppress(ProcessLookupError):
|
||||||
|
os.killpg(proc.pid, signal.SIGKILL)
|
||||||
|
proc.communicate()
|
||||||
|
raise
|
||||||
|
return subprocess.CompletedProcess(argv, proc.returncode, out, err)
|
||||||
|
|
||||||
|
|
||||||
_ENV = {"LC_ALL": "C", "PATH": "/usr/sbin:/usr/bin:/sbin:/bin"}
|
_ENV = {"LC_ALL": "C", "PATH": "/usr/sbin:/usr/bin:/sbin:/bin"}
|
||||||
|
|
||||||
|
|
||||||
@@ -34,6 +58,13 @@ def explain_mount_error(fstype: str, message: str) -> str:
|
|||||||
if "helper program" in low or "unknown filesystem type" in low:
|
if "helper program" in low or "unknown filesystem type" in low:
|
||||||
pkg = "cifs-utils" if fstype == "cifs" else "nfs-common"
|
pkg = "cifs-utils" if fstype == "cifs" else "nfs-common"
|
||||||
hint = f"Das Paket „{pkg}“ fehlt: apt install {pkg}"
|
hint = f"Das Paket „{pkg}“ fehlt: apt install {pkg}"
|
||||||
|
elif fstype == "cifs" and ("error(95)" in low or "operation not supported" in low):
|
||||||
|
hint = (
|
||||||
|
"Server und Client finden keine gemeinsame SMB-Version. Am besten am NAS "
|
||||||
|
"SMB2/SMB3 erlauben (Synology: Systemsteuerung → Dateidienste → SMB → "
|
||||||
|
"Erweiterte Einstellungen → Maximales SMB-Protokoll: SMB3). Alternativ "
|
||||||
|
"Zusatzoption vers=2.0 – vers=1.0 nur im Notfall, SMB1 ist unsicher."
|
||||||
|
)
|
||||||
elif "permission denied" in low and fstype == "cifs":
|
elif "permission denied" in low and fstype == "cifs":
|
||||||
hint = "Benutzername/Passwort falsch oder der Benutzer hat keine Rechte auf die Freigabe."
|
hint = "Benutzername/Passwort falsch oder der Benutzer hat keine Rechte auf die Freigabe."
|
||||||
elif "permission denied" in low or "access denied by server" in low:
|
elif "permission denied" in low or "access denied by server" in low:
|
||||||
@@ -60,7 +91,7 @@ class LinuxMountBackend(MountBackend):
|
|||||||
self,
|
self,
|
||||||
run_dir: Path,
|
run_dir: Path,
|
||||||
mountinfo_path: Path = MOUNTINFO,
|
mountinfo_path: Path = MOUNTINFO,
|
||||||
runner: Runner = subprocess.run,
|
runner: Runner = run_process_group,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.run_dir = run_dir
|
self.run_dir = run_dir
|
||||||
self.mountinfo_path = mountinfo_path
|
self.mountinfo_path = mountinfo_path
|
||||||
|
|||||||
@@ -15,7 +15,9 @@ Fallstricke:
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
import errno
|
import errno
|
||||||
|
import math
|
||||||
import os
|
import os
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
@@ -104,6 +106,13 @@ class RestoreResult:
|
|||||||
message: str
|
message: str
|
||||||
|
|
||||||
|
|
||||||
|
def _redact_output(message: str, secrets: Sequence[str]) -> str:
|
||||||
|
"""Schwärzt nur die Ausgabe von ``mount``, nicht den angehängten Hinweistext
|
||||||
|
(sonst würde z. B. das Passwort „falsch“ auch im Hinweis geschwärzt)."""
|
||||||
|
output, sep, hint = message.partition("\n→ ")
|
||||||
|
return redact(output, secrets) + sep + hint
|
||||||
|
|
||||||
|
|
||||||
def build_spec(cfg: MountConfig, mountpoint: Path, cred: CredentialData | None) -> MountSpec:
|
def build_spec(cfg: MountConfig, mountpoint: Path, cred: CredentialData | None) -> MountSpec:
|
||||||
"""Baut den ``mount``-Aufruf. Validiert erneut (Verteidigung in der Tiefe) –
|
"""Baut den ``mount``-Aufruf. Validiert erneut (Verteidigung in der Tiefe) –
|
||||||
auch Daten aus der Datenbank laufen hier noch einmal durch die Prüfung."""
|
auch Daten aus der Datenbank laufen hier noch einmal durch die Prüfung."""
|
||||||
@@ -166,6 +175,8 @@ class MountManager:
|
|||||||
sleep: Callable[[float], None] = time.sleep,
|
sleep: Callable[[float], None] = time.sleep,
|
||||||
euid: Callable[[], int] = os.geteuid,
|
euid: Callable[[], int] = os.geteuid,
|
||||||
notifier: Notifier | None = None,
|
notifier: Notifier | None = None,
|
||||||
|
retry_interval: float = 60,
|
||||||
|
clock: Callable[[], float] = time.monotonic,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.session_factory = session_factory
|
self.session_factory = session_factory
|
||||||
self.backend = backend
|
self.backend = backend
|
||||||
@@ -177,6 +188,12 @@ class MountManager:
|
|||||||
self._sleep = sleep
|
self._sleep = sleep
|
||||||
self._euid = euid
|
self._euid = euid
|
||||||
self._notifier = notifier
|
self._notifier = notifier
|
||||||
|
self.retry_interval = retry_interval
|
||||||
|
self._clock = clock
|
||||||
|
# Mounts, die NetMover nach gescheiterter Wiederherstellung selbst ausgehängt
|
||||||
|
# hat: nächster erlaubter Versuch (monotone Zeit). Nur diese werden
|
||||||
|
# automatisch neu eingehängt – von außen ausgehängte nie (Fallstrick 2).
|
||||||
|
self._retry_at: dict[int, float] = {}
|
||||||
self._locks: dict[int, threading.RLock] = {}
|
self._locks: dict[int, threading.RLock] = {}
|
||||||
self._locks_guard = threading.Lock()
|
self._locks_guard = threading.Lock()
|
||||||
|
|
||||||
@@ -211,7 +228,7 @@ class MountManager:
|
|||||||
if m.credential is not None:
|
if m.credential is not None:
|
||||||
c = m.credential
|
c = m.credential
|
||||||
cred = CredentialData(
|
cred = CredentialData(
|
||||||
c.kind, c.username, self.secrets.decrypt(c.secret_enc), c.domain
|
c.kind, m.username or c.username, self.secrets.decrypt(c.secret_enc), c.domain
|
||||||
)
|
)
|
||||||
s.expunge(m)
|
s.expunge(m)
|
||||||
return m, cred
|
return m, cred
|
||||||
@@ -267,6 +284,7 @@ class MountManager:
|
|||||||
|
|
||||||
def mount(self, mount_id: int) -> None:
|
def mount(self, mount_id: int) -> None:
|
||||||
with self._lock(mount_id):
|
with self._lock(mount_id):
|
||||||
|
self._retry_at.pop(mount_id, None)
|
||||||
m, cred = self._load(mount_id)
|
m, cred = self._load(mount_id)
|
||||||
if not m.type.is_kernel_mount:
|
if not m.type.is_kernel_mount:
|
||||||
raise MountError("SFTP-Anbindungen werden nicht eingehängt, nur getestet")
|
raise MountError("SFTP-Anbindungen werden nicht eingehängt, nur getestet")
|
||||||
@@ -292,6 +310,7 @@ class MountManager:
|
|||||||
|
|
||||||
def unmount(self, mount_id: int) -> None:
|
def unmount(self, mount_id: int) -> None:
|
||||||
with self._lock(mount_id):
|
with self._lock(mount_id):
|
||||||
|
self._retry_at.pop(mount_id, None)
|
||||||
m, _ = self._load(mount_id)
|
m, _ = self._load(mount_id)
|
||||||
if not m.type.is_kernel_mount:
|
if not m.type.is_kernel_mount:
|
||||||
self._set_state(mount_id, MountState.UNMOUNTED)
|
self._set_state(mount_id, MountState.UNMOUNTED)
|
||||||
@@ -325,6 +344,8 @@ class MountManager:
|
|||||||
self.mount(mount_id)
|
self.mount(mount_id)
|
||||||
results.append(RestoreResult(name, True, "eingehängt"))
|
results.append(RestoreResult(name, True, "eingehängt"))
|
||||||
except (MountError, ValidationError, LookupError) as exc:
|
except (MountError, ValidationError, LookupError) as exc:
|
||||||
|
# z. B. Stromausfall: die VM bootet schneller als das NAS
|
||||||
|
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
||||||
results.append(RestoreResult(name, False, str(exc)))
|
results.append(RestoreResult(name, False, str(exc)))
|
||||||
self._notify("mount.restore_failed", name, str(exc))
|
self._notify("mount.restore_failed", name, str(exc))
|
||||||
for r in results:
|
for r in results:
|
||||||
@@ -391,7 +412,7 @@ class MountManager:
|
|||||||
self.backend.mount(build_spec(cfg, temp, cred), timeout=self.mount_timeout)
|
self.backend.mount(build_spec(cfg, temp, cred), timeout=self.mount_timeout)
|
||||||
except (MountError, ValidationError) as exc:
|
except (MountError, ValidationError) as exc:
|
||||||
self._remove_mountpoint(temp)
|
self._remove_mountpoint(temp)
|
||||||
report.step("mount", False, redact(str(exc), secrets))
|
report.step("mount", False, _redact_output(str(exc), secrets))
|
||||||
report.step("read", None)
|
report.step("read", None)
|
||||||
report.step("write", None)
|
report.step("write", None)
|
||||||
return report
|
return report
|
||||||
@@ -486,6 +507,11 @@ class MountManager:
|
|||||||
if cfg.type is MountType.SFTP and m.host_key is None and report.can_read:
|
if cfg.type is MountType.SFTP and m.host_key is None and report.can_read:
|
||||||
m.host_key = report.host_key
|
m.host_key = report.host_key
|
||||||
event(log, "sftp.host_key_pinned", mount=m.name, fp=report.host_key_fingerprint)
|
event(log, "sftp.host_key_pinned", mount=m.name, fp=report.host_key_fingerprint)
|
||||||
|
# SFTP hat keinen Mount-Zustand – ein erfolgreicher Test heißt „erreichbar“.
|
||||||
|
if cfg.type is MountType.SFTP and report.ok and m.state is not MountState.MOUNTED:
|
||||||
|
m.state = MountState.MOUNTED
|
||||||
|
m.state_since = utcnow()
|
||||||
|
m.last_error = None
|
||||||
|
|
||||||
# -- Lebenszeichen und Wiederherstellung --------------------------------------
|
# -- Lebenszeichen und Wiederherstellung --------------------------------------
|
||||||
|
|
||||||
@@ -501,6 +527,9 @@ class MountManager:
|
|||||||
return None
|
return None
|
||||||
mp = self.mountpoint_for(m.slug)
|
mp = self.mountpoint_for(m.slug)
|
||||||
if not self.backend.is_mounted(mp):
|
if not self.backend.is_mounted(mp):
|
||||||
|
if mount_id in self._retry_at:
|
||||||
|
self._retry_mount(mount_id, m, cred, mp)
|
||||||
|
return mp
|
||||||
msg = f"„{m.name}“ ist nicht eingehängt – Lauf abgebrochen, nichts angefasst."
|
msg = f"„{m.name}“ ist nicht eingehängt – Lauf abgebrochen, nichts angefasst."
|
||||||
self._set_state(mount_id, MountState.ERROR, msg)
|
self._set_state(mount_id, MountState.ERROR, msg)
|
||||||
raise MountUnavailable(msg)
|
raise MountUnavailable(msg)
|
||||||
@@ -568,10 +597,74 @@ class MountManager:
|
|||||||
f"„{m.name}“ gestört ({reason}); Neu-Einhängen nach "
|
f"„{m.name}“ gestört ({reason}); Neu-Einhängen nach "
|
||||||
f"{len(self.recover_backoff)} Versuchen gescheitert: {last}"
|
f"{len(self.recover_backoff)} Versuchen gescheitert: {last}"
|
||||||
)
|
)
|
||||||
|
if m.auto_mount:
|
||||||
|
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
||||||
|
msg += f"\n→ Neuer Versuch automatisch alle {self.retry_interval:g} s."
|
||||||
self._set_state(mount_id, MountState.ERROR, msg)
|
self._set_state(mount_id, MountState.ERROR, msg)
|
||||||
self._notify("mount.error", m.name, msg)
|
self._notify("mount.error", m.name, msg)
|
||||||
raise MountUnavailable(msg)
|
raise MountUnavailable(msg)
|
||||||
|
|
||||||
|
def retry_pending(self) -> list[str]:
|
||||||
|
"""Fällige Neuversuche ausführen (Scheduler-Job, jede Minute) – auch für
|
||||||
|
Anbindungen, die gerade keine Regel benutzt. Liefert die zurückgeholten Namen."""
|
||||||
|
recovered: list[str] = []
|
||||||
|
for mount_id in list(self._retry_at):
|
||||||
|
if self._retry_at.get(mount_id, float("inf")) > self._clock():
|
||||||
|
continue
|
||||||
|
with self._lock(mount_id):
|
||||||
|
if mount_id not in self._retry_at:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
m, cred = self._load(mount_id)
|
||||||
|
except LookupError:
|
||||||
|
self._retry_at.pop(mount_id, None)
|
||||||
|
continue
|
||||||
|
if not (m.enabled and m.auto_mount and m.type.is_kernel_mount):
|
||||||
|
self._retry_at.pop(mount_id, None)
|
||||||
|
continue
|
||||||
|
mp = self.mountpoint_for(m.slug)
|
||||||
|
if self.backend.is_mounted(mp):
|
||||||
|
self._retry_at.pop(mount_id, None)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
self._retry_mount(mount_id, m, cred, mp)
|
||||||
|
recovered.append(m.name)
|
||||||
|
except MountUnavailable:
|
||||||
|
pass
|
||||||
|
return recovered
|
||||||
|
|
||||||
|
def _retry_mount(self, mount_id: int, m: Mount, cred: CredentialData | None, mp: Path) -> None:
|
||||||
|
"""Ein Versuch, einen von NetMover selbst ausgehängten Mount zurückzuholen –
|
||||||
|
höchstens einmal pro ``retry_interval``, damit kein Lauf alle 10 s hängt."""
|
||||||
|
wait = self._retry_at[mount_id] - self._clock()
|
||||||
|
if wait > 0:
|
||||||
|
raise MountUnavailable(
|
||||||
|
f"„{m.name}“ nicht verfügbar – nächster Versuch, neu einzuhängen, "
|
||||||
|
f"in {math.ceil(wait)} s. Lauf abgebrochen, nichts angefasst."
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
self._prepare_mountpoint(mp)
|
||||||
|
self.backend.mount(
|
||||||
|
build_spec(MountConfig.from_model(m), mp, cred), timeout=self.mount_timeout
|
||||||
|
)
|
||||||
|
self._probe(m.name, mp)
|
||||||
|
except (MountError, OSError, OperationTimeout, ValidationError) as exc:
|
||||||
|
if self.backend.is_mounted(mp):
|
||||||
|
with contextlib.suppress(MountError):
|
||||||
|
self.backend.unmount(mp, lazy=True, timeout=self.mount_timeout)
|
||||||
|
self._retry_at[mount_id] = self._clock() + self.retry_interval
|
||||||
|
msg = (
|
||||||
|
f"„{m.name}“ weiterhin nicht verfügbar: {exc}\n"
|
||||||
|
f"→ Neuer Versuch in {self.retry_interval:g} s."
|
||||||
|
)
|
||||||
|
self._set_state(mount_id, MountState.ERROR, msg)
|
||||||
|
event(log, "mount.retry_failed", level=30, mount=m.name, error=str(exc))
|
||||||
|
raise MountUnavailable(msg) from None
|
||||||
|
del self._retry_at[mount_id]
|
||||||
|
self._set_state(mount_id, MountState.MOUNTED, mounted=True)
|
||||||
|
event(log, "mount.recovered", mount=m.name, attempt="retry")
|
||||||
|
self._notify("mount.recovered", m.name, f"„{m.name}“ ist wieder eingehängt.")
|
||||||
|
|
||||||
def usage(self, mount_id: int, timeout: float | None = None) -> tuple[int, int] | None:
|
def usage(self, mount_id: int, timeout: float | None = None) -> tuple[int, int] | None:
|
||||||
"""(gesamt, frei) in Bytes – nur bei eingehängten Kernel-Mounts."""
|
"""(gesamt, frei) in Bytes – nur bei eingehängten Kernel-Mounts."""
|
||||||
m, _ = self._load(mount_id)
|
m, _ = self._load(mount_id)
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ Reihenfolge pro Lauf:
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import dataclasses
|
||||||
import errno
|
import errno
|
||||||
import os
|
import os
|
||||||
import stat
|
import stat
|
||||||
@@ -39,7 +40,7 @@ from netmover.mover.destinations import (
|
|||||||
)
|
)
|
||||||
from netmover.mover.ledger import FailureInfo, Ledger
|
from netmover.mover.ledger import FailureInfo, Ledger
|
||||||
from netmover.mover.naming import collision_candidates
|
from netmover.mover.naming import collision_candidates
|
||||||
from netmover.mover.scan import Candidate, FileState, Matcher, current_state, scan
|
from netmover.mover.scan import BUSY, Candidate, FileState, Matcher, current_state, scan
|
||||||
from netmover.timeouts import Cancelled, CancelToken
|
from netmover.timeouts import Cancelled, CancelToken
|
||||||
|
|
||||||
log = get_logger(__name__)
|
log = get_logger(__name__)
|
||||||
@@ -222,26 +223,32 @@ class Engine:
|
|||||||
else:
|
else:
|
||||||
pending.append(c)
|
pending.append(c)
|
||||||
|
|
||||||
# Stabilität (Fallstrick 1): zweite Messung nach dem Stabilitätsfenster.
|
# Stabilität (Fallstrick 1): zwei frische Messungen (öffnen + fstat, nicht
|
||||||
|
# der Verzeichnis-Cache) im Abstand des Stabilitätsfensters.
|
||||||
|
first: dict[str, FileState | None] = {c.relpath: current_state(c.path) for c in pending}
|
||||||
if pending and rule.stability_seconds > 0:
|
if pending and rule.stability_seconds > 0:
|
||||||
self._sleep(rule.stability_seconds)
|
self._sleep(rule.stability_seconds)
|
||||||
token.check()
|
token.check()
|
||||||
stable: list[Candidate] = []
|
stable: list[Candidate] = []
|
||||||
for c in pending:
|
for c in pending:
|
||||||
st = current_state(c.path)
|
before, st = first[c.relpath], current_state(c.path)
|
||||||
if st is None:
|
if before is None or st is None:
|
||||||
continue # inzwischen verschwunden – nichts zu tun
|
continue # inzwischen verschwunden – nichts zu tun
|
||||||
if st != c.state:
|
if st != before or st == BUSY:
|
||||||
result.add(
|
result.add(
|
||||||
ItemResult(
|
ItemResult(
|
||||||
c.relpath,
|
c.relpath,
|
||||||
ItemStatus.SKIPPED,
|
ItemStatus.SKIPPED,
|
||||||
size=st.size,
|
size=max(st.size, 0),
|
||||||
message="wird noch geschrieben (Größe/Zeit geändert)",
|
message="wird noch geschrieben (Größe/Zeit geändert)"
|
||||||
|
if st != BUSY
|
||||||
|
else "Datei ist gerade in Benutzung",
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
continue
|
continue
|
||||||
stable.append(c)
|
# ab hier gilt der frisch gemessene Zustand (für Prüfung vor dem Umbenennen
|
||||||
|
# und für die Kopier-Buchführung)
|
||||||
|
stable.append(dataclasses.replace(c, state=st))
|
||||||
|
|
||||||
failures = self.ledger.failures(rule.id)
|
failures = self.ledger.failures(rule.id)
|
||||||
for c in stable:
|
for c in stable:
|
||||||
|
|||||||
@@ -10,6 +10,7 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import errno
|
||||||
import fnmatch
|
import fnmatch
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
@@ -100,11 +101,31 @@ def scan(
|
|||||||
yield from walk(root, "")
|
yield from walk(root, "")
|
||||||
|
|
||||||
|
|
||||||
|
# Datei ist geöffnet, aber gerade nicht lesbar (z. B. exklusiv vom Schreiber
|
||||||
|
# gesperrt): gilt als „noch nicht fertig“ – gleicht nie einem echten Zustand.
|
||||||
|
BUSY = FileState(-1, -1)
|
||||||
|
|
||||||
|
|
||||||
def current_state(path: Path) -> FileState | None:
|
def current_state(path: Path) -> FileState | None:
|
||||||
|
"""Aktuelle Größe und mtime – **frisch vom Server**.
|
||||||
|
|
||||||
|
``stat()`` beantworten CIFS- und NFS-Clients aus ihrem Attribut-Cache; im
|
||||||
|
Praxistest lieferte CIFS so sekundenlang eine veraltete Größe, während die
|
||||||
|
Datei noch wuchs. Erst ``open()`` erzwingt frische Attribute (NFS:
|
||||||
|
Close-to-open-Konsistenz, SMB: Antwort auf CREATE). Darum öffnen + ``fstat``.
|
||||||
|
"""
|
||||||
try:
|
try:
|
||||||
st = os.stat(path, follow_symlinks=False)
|
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
return None
|
return None
|
||||||
|
except OSError as exc:
|
||||||
|
if exc.errno in (errno.ELOOP, errno.ENOTDIR):
|
||||||
|
return None # Symlink oder kein Pfad mehr
|
||||||
|
return BUSY
|
||||||
|
try:
|
||||||
|
st = os.fstat(fd)
|
||||||
|
finally:
|
||||||
|
os.close(fd)
|
||||||
if not stat.S_ISREG(st.st_mode):
|
if not stat.S_ISREG(st.st_mode):
|
||||||
return None
|
return None
|
||||||
return FileState(st.st_size, st.st_mtime_ns)
|
return FileState(st.st_size, st.st_mtime_ns)
|
||||||
|
|||||||
@@ -125,4 +125,9 @@ class Notifier:
|
|||||||
|
|
||||||
def mount_event(self, kind: str, mount: str, message: str) -> None:
|
def mount_event(self, kind: str, mount: str, message: str) -> None:
|
||||||
"""Passt zur ``Notifier``-Signatur des :class:`MountManager`."""
|
"""Passt zur ``Notifier``-Signatur des :class:`MountManager`."""
|
||||||
self.send(Notification(kind, f"NetMover: Anbindung „{mount}“ gestört", message))
|
title = (
|
||||||
|
f"NetMover: Anbindung „{mount}“ wieder verfügbar"
|
||||||
|
if kind == "mount.recovered"
|
||||||
|
else f"NetMover: Anbindung „{mount}“ gestört"
|
||||||
|
)
|
||||||
|
self.send(Notification(kind, title, message))
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import random
|
import random
|
||||||
|
from collections.abc import Callable
|
||||||
from concurrent.futures import ThreadPoolExecutor
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
from datetime import UTC, datetime, timedelta
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
@@ -31,6 +32,7 @@ log = get_logger(__name__)
|
|||||||
MAX_START_OFFSET = 10.0
|
MAX_START_OFFSET = 10.0
|
||||||
HEARTBEAT_SECONDS = 30
|
HEARTBEAT_SECONDS = 30
|
||||||
CLEANUP_SECONDS = 3600
|
CLEANUP_SECONDS = 3600
|
||||||
|
RETRY_SECONDS = 60
|
||||||
|
|
||||||
|
|
||||||
def _job_id(rule_id: int) -> str:
|
def _job_id(rule_id: int) -> str:
|
||||||
@@ -43,11 +45,13 @@ class RuleScheduler:
|
|||||||
session_factory: sessionmaker[Session],
|
session_factory: sessionmaker[Session],
|
||||||
service: RunService,
|
service: RunService,
|
||||||
*,
|
*,
|
||||||
|
retry_mounts: Callable[[], object] | None = None,
|
||||||
rng: random.Random | None = None,
|
rng: random.Random | None = None,
|
||||||
workers: int = 8,
|
workers: int = 8,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.session_factory = session_factory
|
self.session_factory = session_factory
|
||||||
self.service = service
|
self.service = service
|
||||||
|
self._retry_mounts = retry_mounts
|
||||||
self._rng = rng or random.Random() # noqa: S311 – nur Startversatz
|
self._rng = rng or random.Random() # noqa: S311 – nur Startversatz
|
||||||
self._executor = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="nm-run")
|
self._executor = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="nm-run")
|
||||||
self.scheduler = AsyncIOScheduler(
|
self.scheduler = AsyncIOScheduler(
|
||||||
@@ -70,6 +74,10 @@ class RuleScheduler:
|
|||||||
IntervalTrigger(seconds=CLEANUP_SECONDS, start_date=utcnow() + timedelta(minutes=1)),
|
IntervalTrigger(seconds=CLEANUP_SECONDS, start_date=utcnow() + timedelta(minutes=1)),
|
||||||
id="cleanup",
|
id="cleanup",
|
||||||
)
|
)
|
||||||
|
if self._retry_mounts is not None:
|
||||||
|
self.scheduler.add_job(
|
||||||
|
self._retry, IntervalTrigger(seconds=RETRY_SECONDS), id="mount-retry"
|
||||||
|
)
|
||||||
self.sync()
|
self.sync()
|
||||||
event(log, "scheduler.started", rules=len(self._intervals))
|
event(log, "scheduler.started", rules=len(self._intervals))
|
||||||
|
|
||||||
@@ -141,6 +149,11 @@ class RuleScheduler:
|
|||||||
self._executor, self.service.execute, rule_id, RunTrigger.MANUAL, dry_run
|
self._executor, self.service.execute, rule_id, RunTrigger.MANUAL, dry_run
|
||||||
)
|
)
|
||||||
|
|
||||||
|
async def _retry(self) -> None:
|
||||||
|
if self._retry_mounts is not None:
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
await loop.run_in_executor(self._executor, self._retry_mounts)
|
||||||
|
|
||||||
async def _cleanup(self) -> None:
|
async def _cleanup(self) -> None:
|
||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
await loop.run_in_executor(self._executor, cleanup, self.session_factory)
|
await loop.run_in_executor(self._executor, cleanup, self.session_factory)
|
||||||
|
|||||||
+102
-6
@@ -45,6 +45,7 @@ from netmover.admin import (
|
|||||||
from netmover.app_state import NetMover
|
from netmover.app_state import NetMover
|
||||||
from netmover.config import Settings, get_settings
|
from netmover.config import Settings, get_settings
|
||||||
from netmover.db import session_scope
|
from netmover.db import session_scope
|
||||||
|
from netmover.keys import delete_key, generate_key, import_key, list_keys, rename_key
|
||||||
from netmover.logging_setup import event, get_logger
|
from netmover.logging_setup import event, get_logger
|
||||||
from netmover.models import (
|
from netmover.models import (
|
||||||
CollisionPolicy,
|
CollisionPolicy,
|
||||||
@@ -392,10 +393,21 @@ async def mounts_page(request: Request) -> Response:
|
|||||||
pubkeys = {
|
pubkeys = {
|
||||||
m.id: m.credential.public_key
|
m.id: m.credential.public_key
|
||||||
for m in mounts
|
for m in mounts
|
||||||
if m.credential is not None and m.credential.public_key
|
if m.credential is not None and m.credential.public_key and not m.credential.managed
|
||||||
|
}
|
||||||
|
key_names = {
|
||||||
|
m.id: m.credential.name
|
||||||
|
for m in mounts
|
||||||
|
if m.credential is not None and m.credential.managed
|
||||||
}
|
}
|
||||||
return render(
|
return render(
|
||||||
request, "mounts.html", active="mounts", mounts=mounts, reports=reports, pubkeys=pubkeys
|
request,
|
||||||
|
"mounts.html",
|
||||||
|
active="mounts",
|
||||||
|
mounts=mounts,
|
||||||
|
reports=reports,
|
||||||
|
pubkeys=pubkeys,
|
||||||
|
key_names=key_names,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -409,6 +421,15 @@ def _report_data(raw: str | None) -> dict[str, Any]:
|
|||||||
return data
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def _key_options(request: Request) -> list[tuple[str, str]]:
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
keys = list_keys(s)
|
||||||
|
return [("new", "neuer Schlüssel nur für diese Anbindung")] + [
|
||||||
|
(str(k.id), f"{k.name} ({k.key_type.removeprefix('ssh-')}, {k.fingerprint[7:19]}…)")
|
||||||
|
for k in keys
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def _mount_form(
|
def _mount_form(
|
||||||
request: Request,
|
request: Request,
|
||||||
data: MountInput,
|
data: MountInput,
|
||||||
@@ -416,7 +437,11 @@ def _mount_form(
|
|||||||
mount: Mount | None = None,
|
mount: Mount | None = None,
|
||||||
status_code: int = 200,
|
status_code: int = 200,
|
||||||
) -> Response:
|
) -> Response:
|
||||||
pub = mount.credential.public_key if mount is not None and mount.credential else None
|
pub = (
|
||||||
|
mount.credential.public_key
|
||||||
|
if mount is not None and mount.credential and not mount.credential.managed
|
||||||
|
else None
|
||||||
|
)
|
||||||
return render(
|
return render(
|
||||||
request,
|
request,
|
||||||
"mount_form.html",
|
"mount_form.html",
|
||||||
@@ -426,7 +451,10 @@ def _mount_form(
|
|||||||
errors=errors,
|
errors=errors,
|
||||||
mount=mount,
|
mount=mount,
|
||||||
public_key=pub,
|
public_key=pub,
|
||||||
has_secret=bool(mount is not None and mount.credential is not None),
|
has_secret=bool(
|
||||||
|
mount is not None and mount.credential is not None and not mount.credential.managed
|
||||||
|
),
|
||||||
|
key_options=_key_options(request),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -487,7 +515,7 @@ async def mount_test_form(request: Request) -> Response:
|
|||||||
existing_id = int(form["mount_id"]) if form.get("mount_id", "").isdigit() else None
|
existing_id = int(form["mount_id"]) if form.get("mount_id", "").isdigit() else None
|
||||||
with state.session_factory() as s:
|
with state.session_factory() as s:
|
||||||
existing = s.get(Mount, existing_id) if existing_id else None
|
existing = s.get(Mount, existing_id) if existing_id else None
|
||||||
cred = credential_for_test(data, existing, state.secrets)
|
cred = credential_for_test(data, existing, state.secrets, s)
|
||||||
cfg = data.config()
|
cfg = data.config()
|
||||||
if data.type is MountType.SFTP and existing is not None:
|
if data.type is MountType.SFTP and existing is not None:
|
||||||
cfg = dataclasses.replace(cfg, host_key=existing.host_key)
|
cfg = dataclasses.replace(cfg, host_key=existing.host_key)
|
||||||
@@ -503,7 +531,7 @@ async def mount_test(request: Request, mount_id: int) -> Response:
|
|||||||
if m is None:
|
if m is None:
|
||||||
return HTMLResponse("Anbindung existiert nicht.", status_code=404)
|
return HTMLResponse("Anbindung existiert nicht.", status_code=404)
|
||||||
data = mount_input_from_model(m)
|
data = mount_input_from_model(m)
|
||||||
cred = credential_for_test(data, m, state.secrets)
|
cred = credential_for_test(data, m, state.secrets, s)
|
||||||
from netmover.mounting.manager import MountConfig
|
from netmover.mounting.manager import MountConfig
|
||||||
|
|
||||||
cfg = MountConfig.from_model(m)
|
cfg = MountConfig.from_model(m)
|
||||||
@@ -563,6 +591,74 @@ async def mount_reset_hostkey(request: Request, mount_id: int) -> Response:
|
|||||||
return redirect(f"/mounts#mount-{mount_id}")
|
return redirect(f"/mounts#mount-{mount_id}")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Schlüsselverwaltung ----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@protected.get("/keys", response_class=HTMLResponse)
|
||||||
|
async def keys_page(request: Request) -> Response:
|
||||||
|
with nm(request).session_factory() as s:
|
||||||
|
return render(request, "keys.html", active="keys", keys=list_keys(s))
|
||||||
|
|
||||||
|
|
||||||
|
async def _key_action(request: Request, work: Callable[[Any], str]) -> Response:
|
||||||
|
state = nm(request)
|
||||||
|
try:
|
||||||
|
with session_scope(state.session_factory) as s:
|
||||||
|
flash(request, work(s))
|
||||||
|
except AdminError as exc:
|
||||||
|
flash(request, str(exc), "error")
|
||||||
|
return redirect("/keys")
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/keys/generate")
|
||||||
|
async def key_generate(request: Request) -> Response:
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
secrets = nm(request).secrets
|
||||||
|
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
key = generate_key(s, secrets, form.get("name", ""))
|
||||||
|
return f"Schlüssel „{key.name}“ erzeugt."
|
||||||
|
|
||||||
|
return await _key_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/keys/import")
|
||||||
|
async def key_import(request: Request) -> Response:
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
secrets = nm(request).secrets
|
||||||
|
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
key = import_key(
|
||||||
|
s,
|
||||||
|
secrets,
|
||||||
|
form.get("name", ""),
|
||||||
|
form.get("private_key", ""),
|
||||||
|
form.get("passphrase", ""),
|
||||||
|
)
|
||||||
|
return f"Schlüssel „{key.name}“ importiert."
|
||||||
|
|
||||||
|
return await _key_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/keys/{key_id}/rename")
|
||||||
|
async def key_rename(request: Request, key_id: int) -> Response:
|
||||||
|
form = _form_dict(await request.form())
|
||||||
|
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
rename_key(s, key_id, form.get("name", ""))
|
||||||
|
return "Schlüssel umbenannt."
|
||||||
|
|
||||||
|
return await _key_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
|
@protected.post("/keys/{key_id}/delete")
|
||||||
|
async def key_delete(request: Request, key_id: int) -> Response:
|
||||||
|
def work(s: Any) -> str:
|
||||||
|
return f"Schlüssel „{delete_key(s, key_id)}“ gelöscht."
|
||||||
|
|
||||||
|
return await _key_action(request, work)
|
||||||
|
|
||||||
|
|
||||||
# -- Regeln -----------------------------------------------------------------------------------
|
# -- Regeln -----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -17,6 +17,7 @@
|
|||||||
<nav class="main">
|
<nav class="main">
|
||||||
<a href="/" class="{{ 'active' if active == 'status' }}">Status</a>
|
<a href="/" class="{{ 'active' if active == 'status' }}">Status</a>
|
||||||
<a href="/mounts" class="{{ 'active' if active == 'mounts' }}">Anbindungen</a>
|
<a href="/mounts" class="{{ 'active' if active == 'mounts' }}">Anbindungen</a>
|
||||||
|
<a href="/keys" class="{{ 'active' if active == 'keys' }}">Schlüssel</a>
|
||||||
<a href="/rules" class="{{ 'active' if active == 'rules' }}">Regeln</a>
|
<a href="/rules" class="{{ 'active' if active == 'rules' }}">Regeln</a>
|
||||||
<a href="/log" class="{{ 'active' if active == 'log' }}">Protokoll</a>
|
<a href="/log" class="{{ 'active' if active == 'log' }}">Protokoll</a>
|
||||||
<a href="/settings" class="{{ 'active' if active == 'settings' }}">Einstellungen</a>
|
<a href="/settings" class="{{ 'active' if active == 'settings' }}">Einstellungen</a>
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
{% from "macros.html" import csrf_field %}
|
||||||
|
{% block title %}Schlüssel{% endblock %}
|
||||||
|
{% block content %}
|
||||||
|
<h1>SSH-Schlüssel</h1>
|
||||||
|
<p class="muted small">Gespeicherte Schlüssel für SFTP-Anbindungen. Ein Schlüssel kann von mehreren
|
||||||
|
Anbindungen benutzt werden; den Benutzernamen legt jede Anbindung selbst fest. Private Schlüssel
|
||||||
|
liegen verschlüsselt in der Datenbank und werden nie angezeigt.</p>
|
||||||
|
|
||||||
|
{% if not keys %}<p class="muted">Noch keine Schlüssel gespeichert.</p>{% endif %}
|
||||||
|
<div class="stack">
|
||||||
|
{% for k in keys %}
|
||||||
|
<div class="card" id="key-{{ k.id }}">
|
||||||
|
<div class="row spread">
|
||||||
|
<div><strong>{{ k.name }}</strong> <span class="badge">{{ k.key_type.removeprefix('ssh-') }}</span>
|
||||||
|
{% if k.used_by %}<span class="badge ok">benutzt von {{ k.used_by|join(', ') }}</span>{% else %}<span class="badge">unbenutzt</span>{% endif %}</div>
|
||||||
|
<div class="row">
|
||||||
|
<form class="inline row" method="post" action="/keys/{{ k.id }}/rename">{{ csrf_field(csrf) }}
|
||||||
|
<input type="text" name="name" value="{{ k.name }}" aria-label="Neuer Name" required maxlength="100">
|
||||||
|
<button class="small">Umbenennen</button></form>
|
||||||
|
<form class="inline" method="post" action="/keys/{{ k.id }}/delete" data-confirm="Schlüssel „{{ k.name }}“ löschen?">{{ csrf_field(csrf) }}
|
||||||
|
<button class="small danger" {{ 'disabled' if k.used_by }}>Löschen</button></form>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="small muted">Fingerabdruck <span class="mono">{{ k.fingerprint }}</span> · angelegt {{ k.created_at|localdt }}</div>
|
||||||
|
<details class="small"><summary>Öffentlicher Schlüssel / authorized_keys</summary>
|
||||||
|
<div class="field"><label>Zeile für <code>~/.ssh/authorized_keys</code> (nur SFTP erlaubt, empfohlen)</label>
|
||||||
|
<pre class="mono card" id="auth-{{ k.id }}">{{ k.authorized_keys_line }}</pre>
|
||||||
|
<button class="small" type="button" data-copy="auth-{{ k.id }}">kopieren</button></div>
|
||||||
|
<div class="field"><label>Öffentlicher Schlüssel</label>
|
||||||
|
<pre class="mono card" id="pub-{{ k.id }}">{{ k.public_key }}</pre>
|
||||||
|
<button class="small" type="button" data-copy="pub-{{ k.id }}">kopieren</button></div>
|
||||||
|
</details>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2>Neuen Schlüssel erzeugen</h2>
|
||||||
|
<form class="form card row" method="post" action="/keys/generate">{{ csrf_field(csrf) }}
|
||||||
|
<input type="text" name="name" placeholder="Name, z. B. paperless-lxc" aria-label="Name" required maxlength="100">
|
||||||
|
<button class="primary">Ed25519-Schlüssel erzeugen</button>
|
||||||
|
</form>
|
||||||
|
|
||||||
|
<h2>Vorhandenen Schlüssel importieren</h2>
|
||||||
|
<form class="form card stack" method="post" action="/keys/import">{{ csrf_field(csrf) }}
|
||||||
|
<div class="fields">
|
||||||
|
<div class="field"><label for="k-name">Name</label><input id="k-name" type="text" name="name" required maxlength="100"></div>
|
||||||
|
<div class="field"><label for="k-pass">Passphrase (falls vorhanden)</label><input id="k-pass" type="password" name="passphrase" autocomplete="off"></div>
|
||||||
|
<div class="field wide"><label for="k-priv">Privater Schlüssel</label>
|
||||||
|
<textarea id="k-priv" name="private_key" class="mono" rows="8" autocomplete="off" spellcheck="false"
|
||||||
|
placeholder="-----BEGIN OPENSSH PRIVATE KEY----- … -----END OPENSSH PRIVATE KEY-----" required></textarea>
|
||||||
|
<span class="hint">Ed25519, ECDSA oder RSA (mind. 2048 Bit), OpenSSH- oder PEM-Format. Eine Passphrase wird nur zum Entschlüsseln benutzt.</span></div>
|
||||||
|
</div>
|
||||||
|
<button>Importieren</button>
|
||||||
|
</form>
|
||||||
|
{% endblock %}
|
||||||
@@ -23,7 +23,10 @@
|
|||||||
</div></fieldset>
|
</div></fieldset>
|
||||||
|
|
||||||
<fieldset data-show-for="type:cifs sftp"><legend>Anmeldung</legend><div class="fields">
|
<fieldset data-show-for="type:cifs sftp"><legend>Anmeldung</legend><div class="fields">
|
||||||
<div data-show-for="type:sftp">{{ select('auth', 'Anmeldeart', [('ssh_key', 'SSH-Schlüssel (NetMover erzeugt ihn)'), ('password', 'Passwort')], d.auth.value, errors=errors) }}</div>
|
<div data-show-for="type:sftp">{{ select('auth', 'Anmeldeart', [('ssh_key', 'SSH-Schlüssel'), ('password', 'Passwort')], d.auth.value, errors=errors) }}</div>
|
||||||
|
<div data-show-for="type:sftp"><div data-show-for="auth:ssh_key">
|
||||||
|
{{ select('key_id', 'Schlüssel', key_options, d.key_id, errors=errors, hint='Gespeicherte Schlüssel verwalten unter „Schlüssel“.') }}
|
||||||
|
</div></div>
|
||||||
{{ field('username', 'Benutzername', d.username, errors=errors, attrs='autocomplete="off"', hint='Bei CIFS leer lassen für Gastzugang') }}
|
{{ field('username', 'Benutzername', d.username, errors=errors, attrs='autocomplete="off"', hint='Bei CIFS leer lassen für Gastzugang') }}
|
||||||
<div data-show-for="type:cifs">{{ field('domain', 'Domäne (optional)', d.domain, errors=errors) }}</div>
|
<div data-show-for="type:cifs">{{ field('domain', 'Domäne (optional)', d.domain, errors=errors) }}</div>
|
||||||
<div class="field{{ ' has-error' if errors.get('password') }}" id="pw-field">
|
<div class="field{{ ' has-error' if errors.get('password') }}" id="pw-field">
|
||||||
@@ -33,7 +36,7 @@
|
|||||||
<span class="hint">{{ 'Leer lassen = unverändert. ' if has_secret }}Bei SFTP mit Schlüssel leer lassen.</span>
|
<span class="hint">{{ 'Leer lassen = unverändert. ' if has_secret }}Bei SFTP mit Schlüssel leer lassen.</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
{% if public_key %}<div class="field wide"><label>Öffentlicher Schlüssel (in ~/.ssh/authorized_keys des Zielbenutzers eintragen)</label>
|
{% if public_key %}<div class="field wide"><label>Öffentlicher Schlüssel dieser Anbindung (in ~/.ssh/authorized_keys des Zielbenutzers eintragen)</label>
|
||||||
<pre class="mono card" id="pubkey">{{ public_key }}</pre><button class="small" type="button" data-copy="pubkey">kopieren</button></div>{% endif %}
|
<pre class="mono card" id="pubkey">{{ public_key }}</pre><button class="small" type="button" data-copy="pubkey">kopieren</button></div>{% endif %}
|
||||||
</fieldset>
|
</fieldset>
|
||||||
|
|
||||||
|
|||||||
@@ -32,6 +32,9 @@
|
|||||||
<div class="small">Hostschlüssel: {% if m.host_key %}<span class="mono">{{ m.host_key.split()[0] }}</span> festgehalten
|
<div class="small">Hostschlüssel: {% if m.host_key %}<span class="mono">{{ m.host_key.split()[0] }}</span> festgehalten
|
||||||
<form class="inline" method="post" action="/mounts/{{ m.id }}/reset-hostkey" data-confirm="Hostschlüssel zurücksetzen? Nur tun, wenn der Server neu aufgesetzt wurde.">{{ csrf_field(csrf) }}<button class="small">zurücksetzen</button></form>
|
<form class="inline" method="post" action="/mounts/{{ m.id }}/reset-hostkey" data-confirm="Hostschlüssel zurücksetzen? Nur tun, wenn der Server neu aufgesetzt wurde.">{{ csrf_field(csrf) }}<button class="small">zurücksetzen</button></form>
|
||||||
{% else %}<span class="text-warn">noch nicht bestätigt – bitte „Testen“</span>{% endif %}</div>
|
{% else %}<span class="text-warn">noch nicht bestätigt – bitte „Testen“</span>{% endif %}</div>
|
||||||
|
{% if key_names.get(m.id) %}
|
||||||
|
<div class="small">Anmeldung mit gespeichertem Schlüssel <a href="/keys">„{{ key_names[m.id] }}“</a></div>
|
||||||
|
{% endif %}
|
||||||
{% if pubkeys.get(m.id) %}
|
{% if pubkeys.get(m.id) %}
|
||||||
<details class="small"><summary>Öffentlicher Schlüssel für authorized_keys</summary>
|
<details class="small"><summary>Öffentlicher Schlüssel für authorized_keys</summary>
|
||||||
<pre class="mono" id="pub-{{ m.id }}">{{ pubkeys[m.id] }}</pre>
|
<pre class="mono" id="pub-{{ m.id }}">{{ pubkeys[m.id] }}</pre>
|
||||||
|
|||||||
@@ -730,3 +730,52 @@ def test_final_step_is_a_real_rename_when_possible(
|
|||||||
assert (dst / "a.pdf").read_text() == "alt" and (dst / "a-1.pdf").read_text() == "neu"
|
assert (dst / "a.pdf").read_text() == "alt" and (dst / "a-1.pdf").read_text() == "neu"
|
||||||
assert bool(links) is not supported
|
assert bool(links) is not supported
|
||||||
assert [p.name for p in dst.iterdir() if p.name.endswith(".part")] == []
|
assert [p.name for p in dst.iterdir() if p.name.endswith(".part")] == []
|
||||||
|
|
||||||
|
|
||||||
|
# -- Praxisbefund: veralteter Attribut-Cache (CIFS/NFS) -------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_stability_uses_fresh_attributes_not_stat_cache(
|
||||||
|
engine: Engine,
|
||||||
|
rule_id: int,
|
||||||
|
src: Path,
|
||||||
|
dst: Path,
|
||||||
|
hooks: list[Callable[[], None]],
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
"""Auf dem Testhost lieferte CIFS per stat() sekundenlang eine veraltete Größe,
|
||||||
|
während die Datei wuchs. Simuliert: os.stat liefert für die Datei immer den
|
||||||
|
alten Stand (Cache), nur öffnen + fstat sieht die Wahrheit."""
|
||||||
|
growing = write(src / "langsam.pdf", "a" * 10)
|
||||||
|
stale = os.stat(growing)
|
||||||
|
real_stat = os.stat
|
||||||
|
|
||||||
|
def cached_stat(path: object, *a: object, **kw: object) -> os.stat_result:
|
||||||
|
if str(path) == str(growing):
|
||||||
|
return stale
|
||||||
|
return real_stat(path, *a, **kw) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
monkeypatch.setattr(os, "stat", cached_stat)
|
||||||
|
hooks.append(lambda: growing.write_text("a" * 20))
|
||||||
|
result = run(engine, snap(rule_id), src, dst)
|
||||||
|
assert statuses(result) == {"langsam.pdf": ItemStatus.SKIPPED}
|
||||||
|
assert "wird noch geschrieben" in (result.items[0].message or "")
|
||||||
|
assert growing.exists() and not (dst / "langsam.pdf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_locked_file_counts_as_in_use(
|
||||||
|
engine: Engine, rule_id: int, src: Path, dst: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
f = write(src / "gesperrt.pdf")
|
||||||
|
real_open = os.open
|
||||||
|
|
||||||
|
def locked(path: object, flags: int, *a: object, **kw: object) -> int:
|
||||||
|
if str(path) == str(f):
|
||||||
|
raise PermissionError(errno.EACCES, "Permission denied", str(path))
|
||||||
|
return real_open(path, flags, *a, **kw) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
monkeypatch.setattr("netmover.mover.scan.os.open", locked)
|
||||||
|
result = run(engine, snap(rule_id), src, dst)
|
||||||
|
assert statuses(result) == {"gesperrt.pdf": ItemStatus.SKIPPED}
|
||||||
|
assert result.items[0].message == "Datei ist gerade in Benutzung"
|
||||||
|
assert f.exists()
|
||||||
|
|||||||
@@ -0,0 +1,277 @@
|
|||||||
|
"""Schlüsselverwaltung: erzeugen, importieren, auswählen, schützen."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from cryptography.hazmat.primitives import serialization
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ec, rsa
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from netmover.admin import AdminError, MountInput, delete_mount, save_mount
|
||||||
|
from netmover.crypto import SecretBox, generate_ssh_keypair
|
||||||
|
from netmover.db import session_scope
|
||||||
|
from netmover.keys import delete_key, fingerprint, generate_key, import_key, list_keys, rename_key
|
||||||
|
from netmover.models import Credential, CredentialKind, Mount, MountType
|
||||||
|
from netmover.mounting.manager import MountConfig, MountManager
|
||||||
|
from tests.sftp_stub import StubSFTPServer
|
||||||
|
|
||||||
|
|
||||||
|
def _sftp_input(**kw: object) -> MountInput:
|
||||||
|
data: dict[str, object] = {
|
||||||
|
"name": "paperless",
|
||||||
|
"type": MountType.SFTP,
|
||||||
|
"server": "127.0.0.1",
|
||||||
|
"share": "/consume",
|
||||||
|
"port": 22,
|
||||||
|
"username": "paperless",
|
||||||
|
"auth": CredentialKind.SSH_KEY,
|
||||||
|
}
|
||||||
|
data.update(kw)
|
||||||
|
return MountInput(**data) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
# -- Modul --------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_key(session: Session, secretbox: SecretBox) -> None:
|
||||||
|
key = generate_key(session, secretbox, "paperless-lxc")
|
||||||
|
assert key.managed and key.kind is CredentialKind.SSH_KEY
|
||||||
|
assert key.public_key and key.public_key.startswith("ssh-ed25519 ")
|
||||||
|
assert key.public_key.endswith(" netmover:paperless-lxc")
|
||||||
|
assert "PRIVATE" not in key.public_key
|
||||||
|
assert "OPENSSH PRIVATE KEY" in secretbox.decrypt(key.secret_enc)
|
||||||
|
info = list_keys(session)[0]
|
||||||
|
assert info.fingerprint.startswith("SHA256:") and info.used_by == []
|
||||||
|
assert info.authorized_keys_line.startswith(
|
||||||
|
'restrict,command="/usr/lib/openssh/sftp-server" ssh-ed25519 '
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fingerprint_matches_ssh_keygen(tmp_path: Path) -> None:
|
||||||
|
_, pub = generate_ssh_keypair("x")
|
||||||
|
f = tmp_path / "k.pub"
|
||||||
|
f.write_text(pub + "\n")
|
||||||
|
try:
|
||||||
|
out = subprocess.run(
|
||||||
|
["ssh-keygen", "-lf", str(f)], # noqa: S607 – Abgleich mit OpenSSH
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True,
|
||||||
|
).stdout
|
||||||
|
except FileNotFoundError:
|
||||||
|
pytest.skip("ssh-keygen nicht installiert")
|
||||||
|
assert fingerprint(pub) == out.split()[1]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"kind", ["ed25519-openssh-pass", "rsa-pem", "ecdsa-pem-pass", "rsa-openssh"]
|
||||||
|
)
|
||||||
|
def test_import_formats(session: Session, secretbox: SecretBox, kind: str) -> None:
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ed25519
|
||||||
|
|
||||||
|
if kind.startswith("ed25519"):
|
||||||
|
key = ed25519.Ed25519PrivateKey.generate()
|
||||||
|
elif kind.startswith("rsa"):
|
||||||
|
key = rsa.generate_private_key(public_exponent=65537, key_size=2048) # type: ignore[assignment]
|
||||||
|
else:
|
||||||
|
key = ec.generate_private_key(ec.SECP256R1()) # type: ignore[assignment]
|
||||||
|
enc = (
|
||||||
|
serialization.BestAvailableEncryption(b"geheim")
|
||||||
|
if kind.endswith("pass")
|
||||||
|
else serialization.NoEncryption()
|
||||||
|
)
|
||||||
|
fmt = (
|
||||||
|
serialization.PrivateFormat.OpenSSH
|
||||||
|
if "openssh" in kind
|
||||||
|
else serialization.PrivateFormat.PKCS8
|
||||||
|
)
|
||||||
|
pem = key.private_bytes(serialization.Encoding.PEM, fmt, enc).decode()
|
||||||
|
cred = import_key(session, secretbox, kind, pem, "geheim" if kind.endswith("pass") else "")
|
||||||
|
expected_pub = (
|
||||||
|
key.public_key()
|
||||||
|
.public_bytes(serialization.Encoding.OpenSSH, serialization.PublicFormat.OpenSSH)
|
||||||
|
.decode()
|
||||||
|
)
|
||||||
|
assert cred.public_key is not None and cred.public_key.startswith(expected_pub)
|
||||||
|
stored = secretbox.decrypt(cred.secret_enc)
|
||||||
|
assert "OPENSSH PRIVATE KEY" in stored
|
||||||
|
# ohne Passphrase gespeichert → von paramiko direkt nutzbar
|
||||||
|
from netmover.mounting.sftp import load_private_key
|
||||||
|
|
||||||
|
assert load_private_key(stored).get_base64() == expected_pub.split()[1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_errors(session: Session, secretbox: SecretBox) -> None:
|
||||||
|
from cryptography.hazmat.primitives.asymmetric import ed25519
|
||||||
|
|
||||||
|
key = ed25519.Ed25519PrivateKey.generate()
|
||||||
|
protected = key.private_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PrivateFormat.OpenSSH,
|
||||||
|
serialization.BestAvailableEncryption(b"richtig"),
|
||||||
|
).decode()
|
||||||
|
with pytest.raises(AdminError, match="Passphrase angeben"):
|
||||||
|
import_key(session, secretbox, "a", protected, "")
|
||||||
|
with pytest.raises(AdminError, match="Passphrase falsch"):
|
||||||
|
import_key(session, secretbox, "a", protected, "falsch")
|
||||||
|
_, pub = generate_ssh_keypair()
|
||||||
|
with pytest.raises(AdminError, match="öffentlicher Schlüssel"):
|
||||||
|
import_key(session, secretbox, "a", pub)
|
||||||
|
with pytest.raises(AdminError, match="Kein lesbarer"):
|
||||||
|
import_key(
|
||||||
|
session, secretbox, "a", "-----BEGIN PRIVATE KEY-----\nMüll\n-----END PRIVATE KEY-----"
|
||||||
|
)
|
||||||
|
weak = (
|
||||||
|
rsa.generate_private_key(public_exponent=65537, key_size=1024) # noqa: S505 – absichtlich schwach
|
||||||
|
.private_bytes(
|
||||||
|
serialization.Encoding.PEM,
|
||||||
|
serialization.PrivateFormat.PKCS8,
|
||||||
|
serialization.NoEncryption(),
|
||||||
|
)
|
||||||
|
.decode()
|
||||||
|
)
|
||||||
|
with pytest.raises(AdminError, match="zu schwach"):
|
||||||
|
import_key(session, secretbox, "a", weak)
|
||||||
|
assert list_keys(session) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_names(session: Session, secretbox: SecretBox) -> None:
|
||||||
|
generate_key(session, secretbox, "eins")
|
||||||
|
with pytest.raises(AdminError, match="existiert bereits"):
|
||||||
|
generate_key(session, secretbox, "eins")
|
||||||
|
with pytest.raises(AdminError, match="reserviert"):
|
||||||
|
generate_key(session, secretbox, "mount:x")
|
||||||
|
zwei = generate_key(session, secretbox, "zwei")
|
||||||
|
rename_key(session, zwei.id, "drei")
|
||||||
|
assert [k.name for k in list_keys(session)] == ["drei", "eins"]
|
||||||
|
with pytest.raises(AdminError):
|
||||||
|
rename_key(session, zwei.id, "eins")
|
||||||
|
|
||||||
|
|
||||||
|
# -- Zusammenspiel mit Anbindungen ----------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_mount_uses_shared_key_with_own_username(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
key = generate_key(session, secretbox, "lxc")
|
||||||
|
a, err = save_mount(session, secretbox, manager, _sftp_input(key_id=str(key.id)))
|
||||||
|
assert not err and a is not None
|
||||||
|
b, err = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_sftp_input(name="anderer", server="10.0.0.9", username="root", key_id=str(key.id)),
|
||||||
|
)
|
||||||
|
assert not err and b is not None
|
||||||
|
assert a.credential is key and b.credential is key
|
||||||
|
assert (a.username, b.username) == ("paperless", "root")
|
||||||
|
assert key.username == "" # der Schlüssel selbst bleibt unverändert
|
||||||
|
assert list_keys(session)[0].used_by == ["anderer", "paperless"]
|
||||||
|
|
||||||
|
# Schlüssel in Benutzung → nicht löschbar
|
||||||
|
with pytest.raises(AdminError, match="noch benutzt"):
|
||||||
|
delete_key(session, key.id)
|
||||||
|
# Anbindung löschen lässt den gespeicherten Schlüssel stehen
|
||||||
|
delete_mount(session, manager, a.id)
|
||||||
|
delete_mount(session, manager, b.id)
|
||||||
|
assert session.get(Credential, key.id) is not None
|
||||||
|
delete_key(session, key.id)
|
||||||
|
assert session.get(Credential, key.id) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_switching_between_own_and_shared_key(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
m, _ = save_mount(session, secretbox, manager, _sftp_input()) # eigener Schlüssel
|
||||||
|
assert m is not None and m.credential is not None and not m.credential.managed
|
||||||
|
own_id = m.credential.id
|
||||||
|
key = generate_key(session, secretbox, "geteilt")
|
||||||
|
m, _ = save_mount(session, secretbox, manager, _sftp_input(key_id=str(key.id)), m.id)
|
||||||
|
assert m is not None and m.credential is key
|
||||||
|
assert session.get(Credential, own_id) is None # eigener Schlüssel aufgeräumt
|
||||||
|
# Wechsel auf Passwort darf den gespeicherten Schlüssel nicht verändern
|
||||||
|
secret_before = key.secret_enc
|
||||||
|
m, _ = save_mount(
|
||||||
|
session,
|
||||||
|
secretbox,
|
||||||
|
manager,
|
||||||
|
_sftp_input(auth=CredentialKind.PASSWORD, password="pw123"),
|
||||||
|
m.id,
|
||||||
|
)
|
||||||
|
assert m is not None and m.credential is not None and m.credential is not key
|
||||||
|
assert not m.credential.managed and m.credential.username == "paperless"
|
||||||
|
assert m.username is None
|
||||||
|
assert key.secret_enc == secret_before and key.kind is CredentialKind.SSH_KEY
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_or_foreign_key_id_is_rejected(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
own, _ = save_mount(session, secretbox, manager, _sftp_input(name="a"))
|
||||||
|
assert own is not None and own.credential is not None
|
||||||
|
for bad in ("4711", str(own.credential.id)): # nicht vorhanden / kein gespeicherter Schlüssel
|
||||||
|
m, err = save_mount(session, secretbox, manager, _sftp_input(name="b", key_id=bad))
|
||||||
|
assert m is None and "key_id" in err
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def server(tmp_path: Path) -> Iterator[StubSFTPServer]:
|
||||||
|
root = tmp_path / "sftp-root"
|
||||||
|
(root / "consume").mkdir(parents=True)
|
||||||
|
srv = StubSFTPServer(root, username="paperless").start()
|
||||||
|
yield srv
|
||||||
|
srv.stop()
|
||||||
|
|
||||||
|
|
||||||
|
def test_manager_connects_with_shared_key_and_mount_username(
|
||||||
|
server: StubSFTPServer,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
secretbox: SecretBox,
|
||||||
|
manager: MountManager,
|
||||||
|
) -> None:
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
key = import_key(s, secretbox, "lxc", server.client_private_key)
|
||||||
|
m, err = save_mount(
|
||||||
|
s, secretbox, manager, _sftp_input(port=server.port, key_id=str(key.id))
|
||||||
|
)
|
||||||
|
assert not err and m is not None
|
||||||
|
mid = m.id
|
||||||
|
with session_factory() as s:
|
||||||
|
mm = s.get(Mount, mid)
|
||||||
|
assert mm is not None
|
||||||
|
cfg = MountConfig.from_model(mm)
|
||||||
|
from netmover.admin import credential_for_test, mount_input_from_model
|
||||||
|
|
||||||
|
with session_factory() as s:
|
||||||
|
mm = s.get(Mount, mid)
|
||||||
|
assert mm is not None
|
||||||
|
data = mount_input_from_model(mm)
|
||||||
|
assert data.key_id == str(mm.cred_id) and data.username == "paperless"
|
||||||
|
cred = credential_for_test(data, mm, secretbox, s)
|
||||||
|
assert cred is not None and cred.username == "paperless"
|
||||||
|
assert manager.check(cfg, cred, mount_id=mid).ok
|
||||||
|
assert manager.check_alive(mid) is None # Regellauf-Weg: Benutzer der Anbindung
|
||||||
|
|
||||||
|
# Gegenprobe: falscher Benutzer an der Anbindung → Anmeldung abgelehnt
|
||||||
|
with session_scope(session_factory) as s:
|
||||||
|
mm = s.get(Mount, mid)
|
||||||
|
assert mm is not None
|
||||||
|
mm.username = "jemand"
|
||||||
|
from netmover.mounting.manager import MountUnavailable
|
||||||
|
|
||||||
|
with pytest.raises(MountUnavailable, match="Anmeldung abgelehnt"):
|
||||||
|
manager.check_alive(mid)
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_keys_ignores_per_mount_credentials(
|
||||||
|
session: Session, secretbox: SecretBox, manager: MountManager
|
||||||
|
) -> None:
|
||||||
|
save_mount(session, secretbox, manager, _sftp_input())
|
||||||
|
assert list_keys(session) == []
|
||||||
|
assert session.scalars(select(Credential)).one().managed is False
|
||||||
@@ -110,8 +110,49 @@ def test_is_mounted_reads_mountinfo(tmp_path: Path) -> None:
|
|||||||
),
|
),
|
||||||
("nfs4", "mount.nfs4: access denied by server while mounting nas:/x", "NFS-Berechtigungen"),
|
("nfs4", "mount.nfs4: access denied by server while mounting nas:/x", "NFS-Berechtigungen"),
|
||||||
("cifs", "mount error(112): Host is down", "nicht erreichbar"),
|
("cifs", "mount error(112): Host is down", "nicht erreichbar"),
|
||||||
|
# Praxisfall: Synology mit „Maximales SMB-Protokoll: SMB2“
|
||||||
|
(
|
||||||
|
"cifs",
|
||||||
|
"mount error(95): Operation not supported\nRefer to the mount.cifs(8) manual page",
|
||||||
|
"vers=2.0",
|
||||||
|
),
|
||||||
("cifs", "mount: only root can do that", "CAP_SYS_ADMIN"),
|
("cifs", "mount: only root can do that", "CAP_SYS_ADMIN"),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
def test_explain(fstype: str, msg: str, expect: str) -> None:
|
def test_explain(fstype: str, msg: str, expect: str) -> None:
|
||||||
assert expect in explain_mount_error(fstype, msg)
|
assert expect in explain_mount_error(fstype, msg)
|
||||||
|
|
||||||
|
|
||||||
|
def _alive(pid: int) -> bool:
|
||||||
|
try:
|
||||||
|
with open(f"/proc/{pid}/stat") as fh:
|
||||||
|
return fh.read().split(")")[-1].split()[0] != "Z"
|
||||||
|
except (FileNotFoundError, ProcessLookupError): # verschwindet beim Lesen → ESRCH
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def test_timeout_kills_whole_process_group(tmp_path: Path) -> None:
|
||||||
|
"""mount startet Helfer (mount.nfs) – die dürfen nach dem Zeitlimit nicht weiterlaufen."""
|
||||||
|
from netmover.mounting.linux import run_process_group
|
||||||
|
|
||||||
|
pidfile = tmp_path / "helper.pid"
|
||||||
|
script = f"sleep 30 & echo $! > {pidfile}; wait"
|
||||||
|
with pytest.raises(subprocess.TimeoutExpired):
|
||||||
|
run_process_group(["sh", "-c", script], capture_output=True, text=True, timeout=0.5)
|
||||||
|
helper = int(pidfile.read_text())
|
||||||
|
for _ in range(20):
|
||||||
|
if not _alive(helper):
|
||||||
|
break
|
||||||
|
import time
|
||||||
|
|
||||||
|
time.sleep(0.05)
|
||||||
|
assert not _alive(helper)
|
||||||
|
|
||||||
|
|
||||||
|
def test_process_group_runner_returns_output() -> None:
|
||||||
|
from netmover.mounting.linux import run_process_group
|
||||||
|
|
||||||
|
r = run_process_group(
|
||||||
|
["sh", "-c", "echo out; echo err >&2; exit 3"], capture_output=True, text=True, timeout=5
|
||||||
|
)
|
||||||
|
assert (r.returncode, r.stdout, r.stderr) == (3, "out\n", "err\n")
|
||||||
|
|||||||
@@ -367,3 +367,159 @@ def test_usage(
|
|||||||
manager.mount(mid)
|
manager.mount(mid)
|
||||||
usage = manager.usage(mid)
|
usage = manager.usage(mid)
|
||||||
assert usage is not None and usage[0] >= usage[1] > 0
|
assert usage is not None and usage[0] >= usage[1] > 0
|
||||||
|
|
||||||
|
|
||||||
|
# -- Praxisbefund: NAS-Neustart dauert länger als die Wiederherstellung ----------------------
|
||||||
|
|
||||||
|
|
||||||
|
class Clock:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.now = 1000.0
|
||||||
|
|
||||||
|
def __call__(self) -> float:
|
||||||
|
return self.now
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def clock() -> Clock:
|
||||||
|
return Clock()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def retry_manager(
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
secretbox: object,
|
||||||
|
mount_base: Path,
|
||||||
|
notifications: list[tuple[str, str, str]],
|
||||||
|
clock: Clock,
|
||||||
|
) -> MountManager:
|
||||||
|
return MountManager(
|
||||||
|
session_factory,
|
||||||
|
fake_backend,
|
||||||
|
secretbox,
|
||||||
|
mount_base, # type: ignore[arg-type]
|
||||||
|
probe_timeout=0.5,
|
||||||
|
recover_backoff=(0, 0, 0),
|
||||||
|
sleep=lambda _: None,
|
||||||
|
notifier=lambda *a: notifications.append(a),
|
||||||
|
retry_interval=60,
|
||||||
|
clock=clock,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_failed_recovery_retries_every_interval_until_nas_is_back(
|
||||||
|
retry_manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
make_mount: MountFactory,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
notifications: list[tuple[str, str, str]],
|
||||||
|
clock: Clock,
|
||||||
|
) -> None:
|
||||||
|
fake_backend.export("nas", "scans")
|
||||||
|
mid = make_mount()
|
||||||
|
retry_manager.mount(mid)
|
||||||
|
mp = retry_manager.mountpoint_for("scans")
|
||||||
|
fake_backend.inject_fault(mp, errno.ESTALE)
|
||||||
|
fake_backend.fail_next_mount(times=3) # NAS startet neu
|
||||||
|
with pytest.raises(MountUnavailable, match="alle 60 s"):
|
||||||
|
retry_manager.check_alive(mid)
|
||||||
|
assert not fake_backend.is_mounted(mp)
|
||||||
|
|
||||||
|
mounts_before = sum(1 for c in fake_backend.calls if c.action == "mount")
|
||||||
|
clock.now += 30 # zu früh: kein Versuch, Lauf abgebrochen
|
||||||
|
with pytest.raises(MountUnavailable, match="in 30 s"):
|
||||||
|
retry_manager.check_alive(mid)
|
||||||
|
assert sum(1 for c in fake_backend.calls if c.action == "mount") == mounts_before
|
||||||
|
|
||||||
|
clock.now += 30 # NAS noch nicht zurück
|
||||||
|
fake_backend.fail_next_mount()
|
||||||
|
with pytest.raises(MountUnavailable, match="weiterhin nicht verfügbar"):
|
||||||
|
retry_manager.check_alive(mid)
|
||||||
|
|
||||||
|
clock.now += 60 # NAS wieder da
|
||||||
|
assert retry_manager.check_alive(mid) == mp
|
||||||
|
assert fake_backend.is_mounted(mp)
|
||||||
|
assert _state(session_factory, mid).state is MountState.MOUNTED
|
||||||
|
assert notifications[-1][0] == "mount.recovered"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_retry_without_auto_mount(
|
||||||
|
retry_manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
make_mount: MountFactory,
|
||||||
|
clock: Clock,
|
||||||
|
) -> None:
|
||||||
|
fake_backend.export("nas", "scans")
|
||||||
|
mid = make_mount(auto_mount=False)
|
||||||
|
retry_manager.mount(mid)
|
||||||
|
fake_backend.inject_fault(retry_manager.mountpoint_for("scans"), errno.ESTALE)
|
||||||
|
fake_backend.fail_next_mount(times=3)
|
||||||
|
with pytest.raises(MountUnavailable):
|
||||||
|
retry_manager.check_alive(mid)
|
||||||
|
clock.now += 600
|
||||||
|
with pytest.raises(MountUnavailable, match="nicht eingehängt"):
|
||||||
|
retry_manager.check_alive(mid)
|
||||||
|
|
||||||
|
|
||||||
|
def test_externally_unmounted_is_still_never_remounted(
|
||||||
|
retry_manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
make_mount: MountFactory,
|
||||||
|
clock: Clock,
|
||||||
|
) -> None:
|
||||||
|
fake_backend.export("nas", "scans")
|
||||||
|
mid = make_mount()
|
||||||
|
retry_manager.mount(mid)
|
||||||
|
fake_backend.vanish(retry_manager.mountpoint_for("scans"))
|
||||||
|
clock.now += 600
|
||||||
|
with pytest.raises(MountUnavailable, match="nicht eingehängt"):
|
||||||
|
retry_manager.check_alive(mid)
|
||||||
|
assert [c.action for c in fake_backend.calls] == ["mount"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_hint_text_is_not_redacted() -> None:
|
||||||
|
from netmover.mounting.manager import _redact_output
|
||||||
|
|
||||||
|
msg = "mount error(13): Permission denied falsch\n→ Benutzername/Passwort falsch oder …"
|
||||||
|
out = _redact_output(msg, ["falsch"])
|
||||||
|
assert out.startswith("mount error(13): Permission denied ••••••")
|
||||||
|
assert out.endswith("→ Benutzername/Passwort falsch oder …")
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_failure_at_boot_is_retried_by_background_job(
|
||||||
|
retry_manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
make_mount: MountFactory,
|
||||||
|
session_factory: sessionmaker[Session],
|
||||||
|
notifications: list[tuple[str, str, str]],
|
||||||
|
clock: Clock,
|
||||||
|
) -> None:
|
||||||
|
"""Stromausfall: die VM bootet schneller als das NAS."""
|
||||||
|
fake_backend.export("nas", "scans")
|
||||||
|
mid = make_mount() # keine Regel benutzt die Anbindung
|
||||||
|
fake_backend.fail_next_mount()
|
||||||
|
assert [r.ok for r in retry_manager.restore_all()] == [False]
|
||||||
|
assert retry_manager.retry_pending() == [] # noch nicht fällig
|
||||||
|
clock.now += 60
|
||||||
|
assert retry_manager.retry_pending() == ["scans"]
|
||||||
|
assert fake_backend.is_mounted(retry_manager.mountpoint_for("scans"))
|
||||||
|
assert _state(session_factory, mid).state is MountState.MOUNTED
|
||||||
|
assert notifications[-1][0] == "mount.recovered"
|
||||||
|
clock.now += 60
|
||||||
|
assert retry_manager.retry_pending() == [] # erledigt
|
||||||
|
|
||||||
|
|
||||||
|
def test_manual_unmount_stops_retrying(
|
||||||
|
retry_manager: MountManager,
|
||||||
|
fake_backend: FakeMountBackend,
|
||||||
|
make_mount: MountFactory,
|
||||||
|
clock: Clock,
|
||||||
|
) -> None:
|
||||||
|
mid = make_mount() # nicht exportiert → Wiederherstellung scheitert
|
||||||
|
retry_manager.restore_all()
|
||||||
|
retry_manager.unmount(mid)
|
||||||
|
fake_backend.export("nas", "scans")
|
||||||
|
clock.now += 120
|
||||||
|
assert retry_manager.retry_pending() == []
|
||||||
|
assert not fake_backend.mounted
|
||||||
|
|||||||
@@ -147,6 +147,8 @@ def test_manager_pins_host_key_and_checks_alive(
|
|||||||
with session_factory() as s:
|
with session_factory() as s:
|
||||||
mm = s.get(Mount, mid)
|
mm = s.get(Mount, mid)
|
||||||
assert mm is not None and mm.host_key == server.host_key_string
|
assert mm is not None and mm.host_key == server.host_key_string
|
||||||
|
# Praxisbefund: nach erfolgreichem Test „erreichbar“ statt „nicht geprüft“
|
||||||
|
assert mm.state is MountState.MOUNTED
|
||||||
|
|
||||||
assert manager.check_alive(mid) is None
|
assert manager.check_alive(mid) is None
|
||||||
with session_factory() as s:
|
with session_factory() as s:
|
||||||
|
|||||||
+67
-1
@@ -154,7 +154,16 @@ def test_security_headers(client: TestClient) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def test_no_inline_scripts_or_styles(client: TestClient) -> None:
|
def test_no_inline_scripts_or_styles(client: TestClient) -> None:
|
||||||
for url in ("/", "/mounts", "/mounts/new", "/rules", "/rules/new", "/log", "/settings"):
|
for url in (
|
||||||
|
"/",
|
||||||
|
"/mounts",
|
||||||
|
"/mounts/new",
|
||||||
|
"/keys",
|
||||||
|
"/rules",
|
||||||
|
"/rules/new",
|
||||||
|
"/log",
|
||||||
|
"/settings",
|
||||||
|
):
|
||||||
html = client.get(url).text
|
html = client.get(url).text
|
||||||
assert "style=" not in html, url
|
assert "style=" not in html, url
|
||||||
assert not re.search(r"<script(?![^>]*\bsrc=)", html), url
|
assert not re.search(r"<script(?![^>]*\bsrc=)", html), url
|
||||||
@@ -555,3 +564,60 @@ def test_export_excludes_secrets_and_import_roundtrip(
|
|||||||
files={"file": ("export.json", text, "application/json")},
|
files={"file": ("export.json", text, "application/json")},
|
||||||
)
|
)
|
||||||
assert "Übersprungen" in c2.get("/settings").text
|
assert "Übersprungen" in c2.get("/settings").text
|
||||||
|
|
||||||
|
|
||||||
|
# -- Schlüsselverwaltung -------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_keys_page_generate_import_select_delete(client: TestClient, state: NetMover) -> None:
|
||||||
|
from netmover.crypto import generate_ssh_keypair
|
||||||
|
|
||||||
|
r = post(client, "/keys/generate", {"name": "paperless-lxc"})
|
||||||
|
assert r.status_code == 303
|
||||||
|
private, public = generate_ssh_keypair()
|
||||||
|
r = post(client, "/keys/import", {"name": "alt", "private_key": private, "passphrase": ""})
|
||||||
|
assert r.status_code == 303
|
||||||
|
page = client.get("/keys").text
|
||||||
|
assert "paperless-lxc" in page and "alt" in page
|
||||||
|
assert public.split()[1] in page # öffentlicher Teil sichtbar
|
||||||
|
for line in private.splitlines()[1:-1]: # keine Zeile des privaten Schlüssels
|
||||||
|
assert line not in page
|
||||||
|
assert "restrict,command="/usr/lib/openssh/sftp-server"" in page
|
||||||
|
|
||||||
|
with state.session_factory() as s:
|
||||||
|
key_id = s.scalar(select(Credential.id).where(Credential.name == "alt"))
|
||||||
|
form = client.get("/mounts/new").text
|
||||||
|
assert f'value="{key_id}"' in form and "neuer Schlüssel nur für diese Anbindung" in form
|
||||||
|
|
||||||
|
r = post(
|
||||||
|
client,
|
||||||
|
"/mounts/new",
|
||||||
|
{
|
||||||
|
"name": "paperless",
|
||||||
|
"type": "sftp",
|
||||||
|
"server": "10.0.0.5",
|
||||||
|
"share": "/consume",
|
||||||
|
"port": "22",
|
||||||
|
"auth": "ssh_key",
|
||||||
|
"key_id": str(key_id),
|
||||||
|
"username": "root",
|
||||||
|
"enabled": "on",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert r.status_code == 303, r.text
|
||||||
|
mounts = client.get("/mounts").text
|
||||||
|
assert "gespeichertem Schlüssel" in mounts and "„alt“" in mounts
|
||||||
|
edit = client.get(f"/mounts/{_mount_id(state, 'paperless')}/edit").text
|
||||||
|
assert f'<option value="{key_id}" selected>' in edit and 'value="root"' in edit
|
||||||
|
|
||||||
|
post(client, f"/keys/{key_id}/delete")
|
||||||
|
assert "noch benutzt" in client.get("/keys").text
|
||||||
|
post(client, f"/mounts/{_mount_id(state, 'paperless')}/delete")
|
||||||
|
post(client, f"/keys/{key_id}/delete")
|
||||||
|
page = client.get("/keys").text
|
||||||
|
assert "gelöscht" in page and ">alt<" not in page
|
||||||
|
|
||||||
|
|
||||||
|
def test_key_import_error_is_shown(client: TestClient) -> None:
|
||||||
|
post(client, "/keys/import", {"name": "x", "private_key": "kein schlüssel", "passphrase": ""})
|
||||||
|
assert "Kein lesbarer privater Schlüssel" in client.get("/keys").text
|
||||||
|
|||||||
Reference in New Issue
Block a user