feat: Mount-Schicht mit Linux-/Fake-Backend, Test und Wiederherstellung

MountBackend-Interface mit LinuxMountBackend (Argumentliste, kurzlebige
Credentials-Datei) und FakeMountBackend; MountManager für Einhängen,
Aushängen, Start-Wiederherstellung, Drei-Schritt-Test mit Root-Squash-
Erkennung, Lebenszeichen und umount -l/Neu-Einhängen; SFTP-Anbindung mit
festgehaltenem Hostschlüssel. Tests inkl. echtem SFTP-Server im Prozess.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Florian Wilde
2026-10-10 18:46:50 +02:00
co-authored by Claude Opus 5.5
parent 689ddf0d94
commit e5ed8c4d88
18 changed files with 2478 additions and 9 deletions
+18
View File
@@ -16,6 +16,24 @@ Die Version steht an genau einer Stelle: `src/netmover/__init__.py`.
- Zeitzonenbewusste Zeitstempel (UTC) auch auf SQLite. - Zeitzonenbewusste Zeitstempel (UTC) auch auf SQLite.
- Strukturierte Logs (logfmt oder JSON, ein Ereignis pro Zeile). - Strukturierte Logs (logfmt oder JSON, ein Ereignis pro Zeile).
- CLI `netmover migrate` / `netmover version`. - CLI `netmover migrate` / `netmover version`.
- SFTP als Zieltyp (`sftp`) für Ziele, die nur per SSH erreichbar sind; Anmeldung
per selbst erzeugtem Ed25519-Schlüssel oder Passwort, Hostschlüssel wird beim
ersten Test festgehalten und danach erzwungen.
- Eingabeprüfung für Server, Freigaben, Pfade und Mount-Optionen (Allowlist pro
Typ; `credentials`, `username`, `password`, `pass`, `domain` gesperrt).
- Fernet-Verschlüsselung der Zugangsdaten (`secret.key`, 0600, beim ersten Start
erzeugt) und Schwärzung bekannter Geheimnisse in `mount`-Ausgaben.
- Mount-Schicht hinter `MountBackend` mit `LinuxMountBackend` (`mount`/`umount`
nur mit Argumentliste, CIFS-Zugangsdaten über kurzlebige Datei unter
`/run/netmover`) und `FakeMountBackend` für Tests ohne root.
- Mount-Test in drei Schritten (einhängen, lesen, schreiben) mit Rollen
(„nur Quelle, kopieren“ / „Quelle mit Löschen“ / „Ziel“) und Root-Squash-Erkennung
inklusive Synology-Hinweis.
- Lebenszeichen mit Zeitlimit vor jedem Lauf; gestörte Mounts (ESTALE, EIO,
ENOTCONN, Zeitüberschreitung) werden per `umount -l` bis zu dreimal mit
wachsendem Abstand neu eingehängt.
- Mountpoints ausschließlich unter `/mnt/netmover/<slug>`, müssen vor dem
Einhängen leer sein und werden nach dem Aushängen entfernt.
### Behoben ### Behoben
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den - Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
+45 -1
View File
@@ -8,10 +8,54 @@ Zeiten in Ortszeit (Europe/Berlin).
| Datum | Von | Bis | Dauer | Schritt | Thema | | Datum | Von | Bis | Dauer | Schritt | Thema |
|------------|-------|-------|-------|---------|----------------------------------------| |------------|-------|-------|-------|---------|----------------------------------------|
| 2026-10-10 | 17:21 | 17:27 | 0:06 | 1 | Gerüst, Datenmodell, Migrationen | | 2026-10-10 | 17:21 | 17:27 | 0:06 | 1 | Gerüst, Datenmodell, Migrationen |
| **Summe** | | | **0:06** | | | | 2026-10-10 | 18:35 | 18:48 | 0:13 | 2 | Mount-Schicht, Validierung, SFTP |
| **Summe** | | | **0:19** | | |
--- ---
## 2026-10-10 · 18:35–18:48 · Schritt 2: Mount-Schicht, Validierung, SFTP
**Anforderungsänderung (Rücksprache)**
- Ziel ist voraussichtlich ein LXC, der nur per SSH erreichbar ist → neuer
Anbindungstyp `sftp` (nur als Ziel). Host: VM mit Debian 13 empfohlen.
- `/healthz` zählt nur aktive Anbindungen; Python ≥ 3.12 (Debian 13: 3.13).
**Erledigt**
- `validation.py`: strikte Muster für Server/Freigabe/Pfade, Options-Allowlist
pro Typ, gesperrte Schlüssel, Werte gegen `^[A-Za-z0-9._:/@-]+$`.
- `crypto.py`: Fernet-Schlüssel (0600, O_EXCL), Ed25519-Schlüsselerzeugung,
Schwärzung.
- `mounting/`: `MountBackend`, `LinuxMountBackend`, `FakeMountBackend`,
`mountinfo`-Parser (Oktal-Escapes, übereinanderliegende Mounts),
`MountManager` (einhängen, aushängen, Wiederherstellung beim Start, Test,
Lebenszeichen, Wiederherstellung), SFTP-Anbindung mit TOFU-Hostschlüssel.
- `timeouts.py`: Zugriffe in Daemon-Threads mit Zeitlimit (kein Pool, damit
hängende Threads nichts blockieren).
- 113 neue Tests (123 gesamt), u. a. gegen einen echten SFTP-Server im Prozess.
- Mutationsprobe: 5 absichtlich eingebaute Fehler (Credentials-Datei bleibt liegen,
fehlende ismount-Prüfung, kein `umount -l`, nicht-leerer Mountpoint, gesperrte
Optionen) werden alle von Tests erkannt.
**Entscheidungen**
- Migration 0001 angepasst statt 0002 – noch kein Release, keine Datenbank im
Einsatz. Ab v0.1.0 sind veröffentlichte Migrationen unveränderlich.
- Mount verschwunden (nicht eingehängt) → Lauf abbrechen, **kein** automatisches
Neu-Einhängen (Fallstrick 2). Gestört (ESTALE & Co.) → Neu-Einhängen (Fallstrick 13).
- Root-Squash-Auswertung als reine Funktion (`assess_write`), damit testbar ohne
root und ohne NAS.
- `statvfs`/`fsync` per SFTP über die OpenSSH-Erweiterungen; fehlen sie, gibt es
einen Hinweis statt eines Fehlers.
- Ruff-Regel `PTH` abgeschaltet: die bewussten `os.open`/`os.fsync`-Aufrufe
sind hier richtig.
**Stolpersteine**
- Funktionen und Klassen mit dem Präfix `test_`/`Test` in Produktionscode würden
von pytest beim Import gesammelt → `check_connection`, `CheckReport`.
**Offen / nächster Schritt**
- Schritt 3: Mover (Stabilität, Atomarität, Kollisionen, Kopier-Buchführung,
Worker mit Zeitlimit, Quarantäne) mit Zielen lokal/Kernel-Mount und SFTP.
## 2026-10-10 · 17:21–17:27 · Schritt 1: Gerüst, Datenmodell, Migrationen ## 2026-10-10 · 17:21–17:27 · Schritt 1: Gerüst, Datenmodell, Migrationen
**Erledigt** **Erledigt**
+3 -1
View File
@@ -3,7 +3,7 @@
Selbst gehosteter Dienst mit Web-Oberfläche, der SMB/CIFS- und NFS-Freigaben Selbst gehosteter Dienst mit Web-Oberfläche, der SMB/CIFS- und NFS-Freigaben
verwaltet und Dateien zeitgesteuert zwischen ihnen verschiebt. verwaltet und Dateien zeitgesteuert zwischen ihnen verschiebt.
> Status: in Entwicklung (Schritt 1 von 6). Vollständige Doku folgt mit Schritt 6. > Status: in Entwicklung (Schritt 2 von 6). Vollständige Doku folgt mit Schritt 6.
## Entwicklung ## Entwicklung
@@ -21,6 +21,8 @@ Neue Migration nach Modelländerung:
NETMOVER_DATA_DIR=$PWD/data .venv/bin/alembic revision --autogenerate -m "…" NETMOVER_DATA_DIR=$PWD/data .venv/bin/alembic revision --autogenerate -m "…"
``` ```
Veröffentlichte Migrationen (ab v0.1.0) werden nie mehr geändert, nur ergänzt.
## Arbeitsweise ## Arbeitsweise
- Commits nach [Conventional Commits](https://www.conventionalcommits.org/de/) - Commits nach [Conventional Commits](https://www.conventionalcommits.org/de/)
+1
View File
@@ -0,0 +1 @@
"""Mount-Schicht: Backends (Linux/Fake), Verwaltung, Tests, SFTP-Anbindung."""
+53
View File
@@ -0,0 +1,53 @@
"""Interface der Mount-Schicht.
Alles, was den Kernel betrifft (einhängen, aushängen, „ist eingehängt?“, Lebens-
zeichen), läuft über :class:`MountBackend`. Dadurch ist die gesamte Mover-Logik
mit :class:`~netmover.mounting.fake.FakeMountBackend` ohne root testbar.
"""
from __future__ import annotations
import os
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from pathlib import Path
@dataclass(frozen=True)
class CifsCredentials:
username: str
password: str
domain: str | None = None
def __repr__(self) -> str: # nie das Passwort in Logs/Tracebacks
return f"CifsCredentials(username={self.username!r}, password='***')"
@dataclass(frozen=True)
class MountSpec:
fstype: str # Kernel-Dateisystemtyp: "cifs", "nfs", "nfs4"
source: str # "//server/share" bzw. "server:/export"
mountpoint: Path
options: tuple[str, ...] = ()
credentials: CifsCredentials | None = field(default=None, repr=True)
class MountError(RuntimeError):
"""``mount``/``umount`` gescheitert. Die Meldung ist bereits geschwärzt."""
class MountBackend(ABC):
@abstractmethod
def mount(self, spec: MountSpec, timeout: float = 60) -> None: ...
@abstractmethod
def unmount(self, mountpoint: Path, lazy: bool = False, timeout: float = 60) -> None: ...
@abstractmethod
def is_mounted(self, mountpoint: Path) -> bool: ...
def probe(self, path: Path) -> None:
"""Lebenszeichen: ``stat`` + ``listdir``. Kann bei hängenden Mounts blockieren
– Aufrufer verwenden :func:`netmover.timeouts.run_with_timeout`."""
os.stat(path)
os.listdir(path)
+126
View File
@@ -0,0 +1,126 @@
"""Test-Implementierung: simuliert Mounts auf einem temporären Verzeichnis.
* „Entfernte“ Freigaben sind Verzeichnisse unter ``<root>/remote/<server>/<share>``
(anlegen mit :meth:`FakeMountBackend.export`).
* Einhängen ersetzt das leere Mountpoint-Verzeichnis durch einen Symlink auf die
Freigabe; Aushängen stellt das leere Verzeichnis wieder her. Ein ausgehängter
Mountpoint ist also – genau wie in echt – ein ganz normales leeres Verzeichnis.
* :meth:`inject_fault` lässt das Lebenszeichen eines Mounts mit ``ESTALE``,
``EIO``, ``ENOTCONN`` scheitern oder hängen (Fallstrick 13).
"""
from __future__ import annotations
import errno
import os
import threading
from collections import deque
from dataclasses import dataclass
from pathlib import Path
from typing import Literal
from netmover.mounting.backend import MountBackend, MountError, MountSpec
Fault = int | Literal["hang"]
@dataclass
class FakeCall:
action: str
mountpoint: Path
spec: MountSpec | None = None
lazy: bool = False
class FakeMountBackend(MountBackend):
def __init__(self, root: Path) -> None:
self.remote_root = root / "remote"
self.remote_root.mkdir(parents=True, exist_ok=True)
self.mounted: dict[Path, Path] = {}
self.faults: dict[Path, Fault] = {}
self.mount_failures: deque[str] = deque()
self.heal_on_remount = True
self.busy: set[Path] = set()
self.calls: list[FakeCall] = []
self._release = threading.Event()
# -- Steuerung für Tests -----------------------------------------------------
def export(self, server: str, share: str) -> Path:
path = self.remote_root / server / share.strip("/")
path.mkdir(parents=True, exist_ok=True)
return path
def fail_next_mount(
self, message: str = "mount error(112): Host is down", times: int = 1
) -> None:
self.mount_failures.extend([message] * times)
def inject_fault(self, mountpoint: Path, fault: Fault) -> None:
self.faults[Path(mountpoint)] = fault
def clear_fault(self, mountpoint: Path) -> None:
self.faults.pop(Path(mountpoint), None)
def vanish(self, mountpoint: Path) -> None:
"""Mount verschwindet von außen (z. B. ``umount`` durch Dritte)."""
mp = Path(mountpoint)
if mp in self.mounted:
mp.unlink()
mp.mkdir()
del self.mounted[mp]
def release_hangs(self) -> None:
self._release.set()
def _remote_for(self, spec: MountSpec) -> Path:
if spec.source.startswith("//"):
server, _, share = spec.source[2:].partition("/")
else:
server, _, share = spec.source.partition(":")
return self.remote_root / server / share.strip("/")
# -- Interface -----------------------------------------------------------------
def mount(self, spec: MountSpec, timeout: float = 60) -> None:
mp = Path(spec.mountpoint)
self.calls.append(FakeCall("mount", mp, spec=spec))
if self.mount_failures:
raise MountError(self.mount_failures.popleft())
if mp in self.mounted:
raise MountError(f"mount: {mp}: already mounted")
remote = self._remote_for(spec)
if not remote.is_dir():
raise MountError("mount error(2): No such file or directory")
if not mp.is_dir() or mp.is_symlink():
raise MountError(f"mount: {mp}: mount point does not exist")
mp.rmdir() # scheitert, wenn nicht leer – wie ein echter Mount, der verdeckt
mp.symlink_to(remote, target_is_directory=True)
self.mounted[mp] = remote
if self.heal_on_remount:
self.faults.pop(mp, None)
def unmount(self, mountpoint: Path, lazy: bool = False, timeout: float = 60) -> None:
mp = Path(mountpoint)
self.calls.append(FakeCall("unmount", mp, lazy=lazy))
if mp not in self.mounted:
raise MountError(f"umount: {mp}: not mounted.")
if mp in self.busy and not lazy:
raise MountError(f"umount: {mp}: target is busy.")
mp.unlink()
mp.mkdir()
del self.mounted[mp]
self.busy.discard(mp)
def is_mounted(self, mountpoint: Path) -> bool:
return Path(mountpoint) in self.mounted
def probe(self, path: Path) -> None:
p = Path(path)
for mp, fault in self.faults.items():
if p == mp or mp in p.parents:
if fault == "hang":
self._release.wait()
raise OSError(errno.EIO, os.strerror(errno.EIO), str(p))
raise OSError(fault, os.strerror(fault), str(p))
super().probe(p)
+132
View File
@@ -0,0 +1,132 @@
"""Echte Implementierung: ruft ``mount``/``umount`` auf.
Fallstrick 5: ``subprocess`` nur mit Argumentliste, nie ``shell=True``.
Fallstrick 7: Zugangsdaten für ``mount.cifs`` nur über eine kurzlebige
Credentials-Datei (0600) unter ``/run/netmover``, nie als ``password=`` in der
Optionszeile – die stünde sonst in der Prozessliste.
"""
from __future__ import annotations
import os
import subprocess
import uuid
from collections.abc import Callable, Sequence
from pathlib import Path
from typing import Any
from netmover.crypto import redact
from netmover.logging_setup import event, get_logger
from netmover.mounting.backend import CifsCredentials, MountBackend, MountError, MountSpec
from netmover.mounting.mountinfo import MOUNTINFO, find_mount, read_mountinfo
log = get_logger(__name__)
Runner = Callable[..., "subprocess.CompletedProcess[str]"]
_ENV = {"LC_ALL": "C", "PATH": "/usr/sbin:/usr/bin:/sbin:/bin"}
def explain_mount_error(fstype: str, message: str) -> str:
"""Hängt an typische ``mount``-Meldungen einen Hinweis im Klartext an."""
low = message.lower()
hint = ""
if "helper program" in low or "unknown filesystem type" in low:
pkg = "cifs-utils" if fstype == "cifs" else "nfs-common"
hint = f"Das Paket „{pkg}“ fehlt: apt install {pkg}"
elif "permission denied" in low and fstype == "cifs":
hint = "Benutzername/Passwort falsch oder der Benutzer hat keine Rechte auf die Freigabe."
elif "permission denied" in low or "access denied by server" in low:
hint = (
"Der NFS-Server erlaubt diesem Host den Zugriff nicht. Synology: Gemeinsamer "
"Ordner → Bearbeiten → NFS-Berechtigungen → Regel mit der IP dieses Hosts anlegen."
)
elif "operation not permitted" in low or "only root can" in low:
hint = (
"Keine Berechtigung zum Einhängen. NetMover muss als root mit CAP_SYS_ADMIN laufen "
"(in unprivilegierten LXC-Containern ist Einhängen nicht möglich)."
)
elif "no such file or directory" in low or "mount error(2)" in low:
hint = "Freigabe bzw. Exportpfad existiert auf dem Server nicht – Schreibweise prüfen."
elif "host is down" in low or "no route to host" in low or "timed out" in low:
hint = "Server nicht erreichbar – Adresse, Netzwerk und Firewall prüfen."
elif "connection refused" in low:
hint = "Server lehnt die Verbindung ab – ist SMB bzw. NFS am NAS aktiviert?"
return f"{message}\n→ {hint}" if hint else message
class LinuxMountBackend(MountBackend):
def __init__(
self,
run_dir: Path,
mountinfo_path: Path = MOUNTINFO,
runner: Runner = subprocess.run,
) -> None:
self.run_dir = run_dir
self.mountinfo_path = mountinfo_path
self._runner = runner
# -- Hilfen ------------------------------------------------------------------
def _write_credentials(self, creds: CifsCredentials) -> Path:
self.run_dir.mkdir(mode=0o700, parents=True, exist_ok=True)
path = self.run_dir / f"cred-{uuid.uuid4().hex}"
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(f"username={creds.username}\n")
fh.write(f"password={creds.password}\n")
if creds.domain:
fh.write(f"domain={creds.domain}\n")
return path
def _run(self, argv: Sequence[str], timeout: float, secrets: Sequence[str]) -> None:
kwargs: dict[str, Any] = {
"capture_output": True,
"text": True,
"timeout": timeout,
"env": _ENV,
"check": False,
"stdin": subprocess.DEVNULL,
}
try:
proc = self._runner(list(argv), **kwargs)
except subprocess.TimeoutExpired:
raise MountError(f"{argv[0]}: keine Antwort nach {timeout:g} s") from None
except FileNotFoundError:
raise MountError(f"{argv[0]}: Programm nicht gefunden") from None
if proc.returncode != 0:
output = (proc.stderr or "").strip() or (proc.stdout or "").strip()
output = redact(output or f"Exit-Code {proc.returncode}", secrets)
raise MountError(output)
# -- Interface -----------------------------------------------------------------
def mount(self, spec: MountSpec, timeout: float = 60) -> None:
options = list(spec.options)
secrets: list[str] = []
cred_file: Path | None = None
try:
if spec.credentials is not None:
cred_file = self._write_credentials(spec.credentials)
options.append(f"credentials={cred_file}")
secrets.append(spec.credentials.password)
argv = ["mount", "-t", spec.fstype]
if options:
argv += ["-o", ",".join(options)]
argv += [spec.source, str(spec.mountpoint)]
try:
self._run(argv, timeout, secrets)
except MountError as exc:
raise MountError(explain_mount_error(spec.fstype, str(exc))) from None
finally:
if cred_file is not None:
cred_file.unlink(missing_ok=True)
event(log, "mount.mounted", source=spec.source, mountpoint=str(spec.mountpoint))
def unmount(self, mountpoint: Path, lazy: bool = False, timeout: float = 60) -> None:
argv = ["umount"] + (["-l"] if lazy else []) + [str(mountpoint)]
self._run(argv, timeout, [])
event(log, "mount.unmounted", mountpoint=str(mountpoint), lazy=lazy)
def is_mounted(self, mountpoint: Path) -> bool:
return find_mount(mountpoint, read_mountinfo(self.mountinfo_path)) is not None
+587
View File
@@ -0,0 +1,587 @@
"""Verwaltung der Anbindungen: einhängen, aushängen, testen, wiederherstellen.
Verbindet Datenbank, Geheimnisse und :class:`MountBackend`. Alle Methoden sind
synchron und blockierend; der Web-Teil ruft sie über ``asyncio.to_thread`` auf,
die Regel-Worker direkt.
Fallstricke:
* 2 – :meth:`MountManager.check_alive` prüft „eingehängt“ über das Backend
(``/proc/self/mountinfo``) und bricht ab, statt in ein leeres Verzeichnis zu schreiben.
* 6 – Mountpoints entstehen nur als ``<mount_base>/<slug>``; vor dem Einhängen
existiert das Verzeichnis und ist leer, nach dem Aushängen wird es entfernt.
* 13 – Lebenszeichen mit Zeitlimit; bei ESTALE/EIO/ENOTCONN/Zeitüberschreitung
``umount -l`` und neu einhängen, höchstens N Versuche mit wachsendem Abstand.
"""
from __future__ import annotations
import errno
import os
import threading
import time
import uuid
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session, sessionmaker
from netmover.crypto import SecretBox, redact
from netmover.db import session_scope
from netmover.logging_setup import event, get_logger
from netmover.models import CredentialKind, Mount, MountState, MountType, utcnow
from netmover.mounting.backend import CifsCredentials, MountBackend, MountError, MountSpec
from netmover.mounting.report import CheckReport, assess_write
from netmover.mounting.sftp import SftpConfig, SftpError, check_connection, connect
from netmover.timeouts import OperationTimeout, run_with_timeout
from netmover.validation import (
ValidationError,
parse_extra_options,
slugify,
validate_server,
validate_share,
)
log = get_logger(__name__)
# Fehler, bei denen ein Neu-Einhängen hilft (Fallstrick 13).
RECOVERABLE_ERRNOS = frozenset({errno.ESTALE, errno.EIO, errno.ENOTCONN, errno.EHOSTDOWN})
Notifier = Callable[[str, str, str], None] # (ereignis, anbindung, meldung)
class MountUnavailable(RuntimeError):
"""Anbindung für einen Lauf nicht nutzbar – Lauf abbrechen, nichts anfassen."""
@dataclass(frozen=True)
class CredentialData:
kind: CredentialKind
username: str
secret: str
domain: str | None = None
def __repr__(self) -> str:
return f"CredentialData(kind={self.kind}, username={self.username!r}, secret='***')"
@dataclass(frozen=True)
class MountConfig:
name: str
type: MountType
server: str
share: str
port: int | None = None
host_key: str | None = None
uid: int | None = None
gid: int | None = None
file_mode: str | None = None
dir_mode: str | None = None
extra_options: str = ""
@classmethod
def from_model(cls, m: Mount) -> MountConfig:
return cls(
name=m.name,
type=m.type,
server=m.server,
share=m.share,
port=m.port,
host_key=m.host_key,
uid=m.uid,
gid=m.gid,
file_mode=m.file_mode,
dir_mode=m.dir_mode,
extra_options=m.extra_options,
)
@dataclass(frozen=True)
class RestoreResult:
name: str
ok: bool
message: str
def build_spec(cfg: MountConfig, mountpoint: Path, cred: CredentialData | None) -> MountSpec:
"""Baut den ``mount``-Aufruf. Validiert erneut (Verteidigung in der Tiefe) –
auch Daten aus der Datenbank laufen hier noch einmal durch die Prüfung."""
if not cfg.type.is_kernel_mount:
raise MountError("SFTP-Anbindungen werden nicht eingehängt")
server = validate_server(cfg.server)
share = validate_share(cfg.type, cfg.share)
options = [o.render() for o in parse_extra_options(cfg.type, cfg.extra_options)]
credentials: CifsCredentials | None = None
if cfg.type is MountType.CIFS:
fstype, source = "cifs", f"//{server}/{share}"
for key, value in (
("uid", cfg.uid),
("gid", cfg.gid),
("file_mode", cfg.file_mode),
("dir_mode", cfg.dir_mode),
):
if value is not None:
options.append(f"{key}={value}")
if cred is None:
options.append("guest")
else:
if cred.kind is not CredentialKind.PASSWORD:
raise MountError("CIFS braucht Zugangsdaten mit Passwort, keinen SSH-Schlüssel")
credentials = CifsCredentials(cred.username, cred.secret, cred.domain)
elif cfg.type is MountType.NFS4:
fstype, source = "nfs4", f"{server}:{share}"
else:
fstype, source = "nfs", f"{server}:{share}"
options.insert(0, "vers=3")
return MountSpec(fstype, source, mountpoint, tuple(options), credentials)
def sftp_config(cfg: MountConfig, cred: CredentialData | None) -> SftpConfig:
if cred is None:
raise SftpError("SFTP braucht Zugangsdaten (SSH-Schlüssel oder Passwort)")
return SftpConfig(
host=validate_server(cfg.server),
port=cfg.port or 22,
username=cred.username,
base_path=validate_share(MountType.SFTP, cfg.share),
password=cred.secret if cred.kind is CredentialKind.PASSWORD else None,
private_key=cred.secret if cred.kind is CredentialKind.SSH_KEY else None,
host_key=cfg.host_key,
)
class MountManager:
def __init__(
self,
session_factory: sessionmaker[Session],
backend: MountBackend,
secrets: SecretBox,
mount_base: Path,
*,
mount_timeout: float = 60,
probe_timeout: float = 15,
recover_backoff: Sequence[float] = (0, 5, 15),
sleep: Callable[[float], None] = time.sleep,
euid: Callable[[], int] = os.geteuid,
notifier: Notifier | None = None,
) -> None:
self.session_factory = session_factory
self.backend = backend
self.secrets = secrets
self.mount_base = mount_base
self.mount_timeout = mount_timeout
self.probe_timeout = probe_timeout
self.recover_backoff = tuple(recover_backoff)
self._sleep = sleep
self._euid = euid
self._notifier = notifier
self._locks: dict[int, threading.RLock] = {}
self._locks_guard = threading.Lock()
# -- Hilfen --------------------------------------------------------------------
def _lock(self, mount_id: int) -> threading.RLock:
with self._locks_guard:
return self._locks.setdefault(mount_id, threading.RLock())
def _notify(self, kind: str, name: str, message: str) -> None:
if self._notifier is not None:
try:
self._notifier(kind, name, message)
except Exception:
log.exception("notifier.failed")
def mountpoint_for(self, slug: str) -> Path:
if slugify(slug) != slug:
raise ValidationError("slug", f"Ungültiger Slug: {slug!r}")
base = self.mount_base.resolve()
mp = base / slug
if mp.parent != base:
raise ValidationError("slug", "Mountpoint außerhalb des Basisverzeichnisses")
return mp
def _load(self, mount_id: int) -> tuple[Mount, CredentialData | None]:
with session_scope(self.session_factory) as s:
m = s.get(Mount, mount_id)
if m is None:
raise LookupError(f"Anbindung {mount_id} existiert nicht")
cred: CredentialData | None = None
if m.credential is not None:
c = m.credential
cred = CredentialData(
c.kind, c.username, self.secrets.decrypt(c.secret_enc), c.domain
)
s.expunge(m)
return m, cred
def _set_state(
self,
mount_id: int,
state: MountState,
error: str | None = None,
*,
mounted: bool = False,
) -> None:
with session_scope(self.session_factory) as s:
m = s.get(Mount, mount_id)
if m is None:
return
if m.state is not state:
m.state_since = utcnow()
event(log, "mount.state", mount=m.name, old=m.state.value, new=state.value)
m.state = state
m.last_error = error
if mounted:
m.last_mounted_at = utcnow()
def _prepare_mountpoint(self, mp: Path) -> bool:
"""Legt den Mountpoint an bzw. prüft, dass er leer ist. True = neu angelegt."""
self.mount_base.mkdir(mode=0o755, parents=True, exist_ok=True)
if mp.is_symlink() or (mp.exists() and not mp.is_dir()):
raise MountError(f"{mp} ist kein Verzeichnis – bitte von Hand prüfen")
if mp.is_dir():
if any(mp.iterdir()):
raise MountError(
f"{mp} ist nicht leer. Vermutlich wurde dort geschrieben, während nichts "
"eingehängt war. Inhalt prüfen und sichern, dann Verzeichnis leeren."
)
return False
mp.mkdir(mode=0o755)
return True
@staticmethod
def _remove_mountpoint(mp: Path) -> None:
try:
mp.rmdir()
except FileNotFoundError:
pass
except OSError as exc:
event(log, "mount.rmdir_failed", level=30, mountpoint=str(mp), error=str(exc))
def _probe(self, name: str, mp: Path) -> None:
run_with_timeout(lambda: self.backend.probe(mp), self.probe_timeout, f"probe:{name}")
# -- Einhängen / Aushängen -----------------------------------------------------
def mount(self, mount_id: int) -> None:
with self._lock(mount_id):
m, cred = self._load(mount_id)
if not m.type.is_kernel_mount:
raise MountError("SFTP-Anbindungen werden nicht eingehängt, nur getestet")
mp = self.mountpoint_for(m.slug)
if self.backend.is_mounted(mp):
self._set_state(mount_id, MountState.MOUNTED)
return
try:
spec = build_spec(MountConfig.from_model(m), mp, cred)
created = self._prepare_mountpoint(mp)
except (MountError, ValidationError) as exc:
self._set_state(mount_id, MountState.ERROR, str(exc))
raise MountError(str(exc)) from None
try:
self.backend.mount(spec, timeout=self.mount_timeout)
except MountError as exc:
if created:
self._remove_mountpoint(mp)
self._set_state(mount_id, MountState.ERROR, str(exc))
event(log, "mount.failed", level=40, mount=m.name, error=str(exc))
raise
self._set_state(mount_id, MountState.MOUNTED, mounted=True)
def unmount(self, mount_id: int) -> None:
with self._lock(mount_id):
m, _ = self._load(mount_id)
if not m.type.is_kernel_mount:
self._set_state(mount_id, MountState.UNMOUNTED)
return
mp = self.mountpoint_for(m.slug)
if self.backend.is_mounted(mp):
try:
self.backend.unmount(mp, timeout=self.mount_timeout)
except MountError as exc:
msg = str(exc)
if "busy" in msg.lower():
msg += "\n→ Ein Prozess benutzt den Mount noch (z. B. ein Regellauf)."
raise MountError(msg) from None
if not self.backend.is_mounted(mp):
self._remove_mountpoint(mp)
self._set_state(mount_id, MountState.UNMOUNTED)
def restore_all(self) -> list[RestoreResult]:
"""Beim Dienststart: alle Anbindungen mit ``auto_mount`` wiederherstellen."""
with session_scope(self.session_factory) as s:
rows = s.execute(
select(Mount.id, Mount.name, Mount.type).where(
Mount.auto_mount.is_(True), Mount.enabled.is_(True)
)
).all()
results: list[RestoreResult] = []
for mount_id, name, mtype in rows:
if not MountType(mtype).is_kernel_mount:
continue
try:
self.mount(mount_id)
results.append(RestoreResult(name, True, "eingehängt"))
except (MountError, ValidationError, LookupError) as exc:
results.append(RestoreResult(name, False, str(exc)))
self._notify("mount.restore_failed", name, str(exc))
for r in results:
event(
log,
"mount.restore",
level=20 if r.ok else 40,
mount=r.name,
ok=r.ok,
message=r.message,
)
return results
# -- Test -------------------------------------------------------------------------
def check(
self,
cfg: MountConfig,
cred: CredentialData | None,
mount_id: int | None = None,
) -> CheckReport:
"""Testet eine (ggf. noch nicht gespeicherte) Anbindung in drei Schritten."""
if cfg.type is MountType.SFTP:
report = self._check_sftp(cfg, cred)
else:
report = self._check_kernel(cfg, cred, mount_id)
if mount_id is not None:
self._store_check(mount_id, report, cfg)
return report
def _check_sftp(self, cfg: MountConfig, cred: CredentialData | None) -> CheckReport:
try:
scfg = sftp_config(cfg, cred)
except (SftpError, ValidationError) as exc:
report = CheckReport()
report.step("connect", False, str(exc))
return report
return run_with_timeout(
lambda: check_connection(scfg, timeout=self.probe_timeout),
self.probe_timeout * 4,
f"check:{cfg.name}",
)
def _check_kernel(
self, cfg: MountConfig, cred: CredentialData | None, mount_id: int | None
) -> CheckReport:
report = CheckReport()
secrets = [cred.secret] if cred else []
target: Path | None = None
temp: Path | None = None
if mount_id is not None:
m, _ = self._load(mount_id)
mp = self.mountpoint_for(m.slug)
if self.backend.is_mounted(mp):
target = mp
report.step("mount", True, "bereits eingehängt")
if target is None:
self.mount_base.mkdir(mode=0o755, parents=True, exist_ok=True)
temp = self.mount_base / f".test-{uuid.uuid4().hex[:12]}"
temp.mkdir(mode=0o700)
try:
self.backend.mount(build_spec(cfg, temp, cred), timeout=self.mount_timeout)
except (MountError, ValidationError) as exc:
self._remove_mountpoint(temp)
report.step("mount", False, redact(str(exc), secrets))
report.step("read", None)
report.step("write", None)
return report
target = temp
report.step("mount", True, "Einhängen erfolgreich")
try:
self._check_read_write(cfg, target, report)
finally:
if temp is not None:
try:
self.backend.unmount(temp, timeout=self.mount_timeout)
except MountError:
try:
self.backend.unmount(temp, lazy=True, timeout=self.mount_timeout)
except MountError as exc:
event(log, "check.unmount_failed", level=40, path=str(temp), error=str(exc))
if not self.backend.is_mounted(temp):
self._remove_mountpoint(temp)
return report
def _check_read_write(self, cfg: MountConfig, target: Path, report: CheckReport) -> None:
name = cfg.name
try:
count = run_with_timeout(
lambda: len(os.listdir(target)), self.probe_timeout, f"read:{name}"
)
report.can_read = True
report.step("read", True, f"{count} Einträge")
except (OSError, OperationTimeout) as exc:
report.step("read", False, str(exc))
report.step("write", None)
return
def write_probe() -> tuple[OSError | None, int | None, int | None, OSError | None]:
# Punkt am Anfang + .tmp: ein parallel laufender Mover fasst sie nie an.
path = target / f".netmover-test-{uuid.uuid4().hex}.tmp"
try:
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o644)
try:
os.write(fd, b"netmover")
os.fsync(fd)
finally:
os.close(fd)
except OSError as exc:
return exc, None, None, None
st = os.stat(path)
try:
path.unlink()
except OSError as exc:
return None, st.st_uid, st.st_gid, exc
return None, st.st_uid, st.st_gid, None
try:
werr, uid, gid, derr = run_with_timeout(
write_probe, self.probe_timeout, f"write:{name}"
)
except OperationTimeout as exc:
report.step("write", False, str(exc))
return
result = assess_write(cfg.type, self._euid(), werr, uid, derr)
report.can_write = result.write_ok
report.can_delete = result.delete_ok
report.root_squash = result.root_squash
report.hints.extend(result.hints)
if uid is not None:
report.owner = f"{uid}:{gid}"
report.step("write", result.write_ok, result.message)
if cfg.type is MountType.CIFS and result.write_ok:
report.hints.append(
"CIFS: Eigentümer und Rechte bestimmen die Mount-Optionen uid/gid/file_mode – "
"chown/chmod nach dem Verschieben werden übersprungen."
)
try:
st = run_with_timeout(lambda: os.statvfs(target), self.probe_timeout, f"statvfs:{name}")
report.total_bytes = st.f_blocks * st.f_frsize
report.free_bytes = st.f_bavail * st.f_frsize
except (OSError, OperationTimeout):
pass
def _store_check(self, mount_id: int, report: CheckReport, cfg: MountConfig) -> None:
with session_scope(self.session_factory) as s:
m = s.get(Mount, mount_id)
if m is None:
return
m.last_test_at = utcnow()
m.last_test_result = report.to_json()
m.write_ok = report.can_write
if report.root_squash is not None:
m.root_squash = report.root_squash
# SFTP: Hostschlüssel beim ersten erfolgreichen Test festhalten (TOFU).
if cfg.type is MountType.SFTP and m.host_key is None and report.can_read:
m.host_key = report.host_key
event(log, "sftp.host_key_pinned", mount=m.name, fp=report.host_key_fingerprint)
# -- Lebenszeichen und Wiederherstellung --------------------------------------
def check_alive(self, mount_id: int) -> Path | None:
"""Vor jedem Lauf. Liefert den Mountpoint (None bei SFTP) oder wirft
:class:`MountUnavailable`. Stellt gestörte Mounts selbst wieder her."""
with self._lock(mount_id):
m, cred = self._load(mount_id)
if not m.enabled:
raise MountUnavailable(f"Anbindung „{m.name}“ ist deaktiviert")
if m.type is MountType.SFTP:
self._check_alive_sftp(mount_id, m, cred)
return None
mp = self.mountpoint_for(m.slug)
if not self.backend.is_mounted(mp):
msg = f"„{m.name}“ ist nicht eingehängt – Lauf abgebrochen, nichts angefasst."
self._set_state(mount_id, MountState.ERROR, msg)
raise MountUnavailable(msg)
try:
self._probe(m.name, mp)
except OperationTimeout as exc:
self._recover(mount_id, m, cred, f"hängt: {exc}")
except OSError as exc:
if exc.errno not in RECOVERABLE_ERRNOS:
msg = f"„{m.name}“: {exc.strerror} ({exc.filename})"
self._set_state(mount_id, MountState.ERROR, msg)
raise MountUnavailable(msg) from None
self._recover(
mount_id, m, cred, f"{errno.errorcode.get(exc.errno or 0, '?')}: {exc.strerror}"
)
else:
if m.state is not MountState.MOUNTED:
self._set_state(mount_id, MountState.MOUNTED)
return mp
def _check_alive_sftp(self, mount_id: int, m: Mount, cred: CredentialData | None) -> None:
try:
scfg = sftp_config(MountConfig.from_model(m), cred)
def probe() -> None:
with connect(scfg, timeout=self.probe_timeout) as sess:
sess.client.stat(scfg.base_path)
run_with_timeout(probe, self.probe_timeout * 2, f"probe:{m.name}")
except (SftpError, ValidationError, OSError, OperationTimeout) as exc:
msg = f"„{m.name}“ nicht erreichbar: {exc}"
self._set_state(mount_id, MountState.ERROR, msg)
if m.state is not MountState.ERROR:
self._notify("mount.error", m.name, msg)
raise MountUnavailable(msg) from None
if m.state is not MountState.MOUNTED:
self._set_state(mount_id, MountState.MOUNTED)
def _recover(self, mount_id: int, m: Mount, cred: CredentialData | None, reason: str) -> None:
mp = self.mountpoint_for(m.slug)
self._set_state(mount_id, MountState.DEGRADED, f"gestört ({reason}) – wird neu eingehängt")
event(log, "mount.degraded", level=30, mount=m.name, reason=reason)
last: str = reason
for attempt, delay in enumerate(self.recover_backoff, start=1):
if delay:
self._sleep(delay)
try:
if self.backend.is_mounted(mp):
self.backend.unmount(mp, lazy=True, timeout=self.mount_timeout)
self._prepare_mountpoint(mp)
self.backend.mount(
build_spec(MountConfig.from_model(m), mp, cred), timeout=self.mount_timeout
)
self._probe(m.name, mp)
except (MountError, OSError, OperationTimeout, ValidationError) as exc:
last = str(exc)
event(
log, "mount.recover_failed", level=30, mount=m.name, attempt=attempt, error=last
)
continue
self._set_state(mount_id, MountState.MOUNTED, mounted=True)
event(log, "mount.recovered", mount=m.name, attempt=attempt)
return
msg = (
f"„{m.name}“ gestört ({reason}); Neu-Einhängen nach "
f"{len(self.recover_backoff)} Versuchen gescheitert: {last}"
)
self._set_state(mount_id, MountState.ERROR, msg)
self._notify("mount.error", m.name, msg)
raise MountUnavailable(msg)
def usage(self, mount_id: int) -> tuple[int, int] | None:
"""(gesamt, frei) in Bytes – nur bei eingehängten Kernel-Mounts."""
m, _ = self._load(mount_id)
if not m.type.is_kernel_mount:
return None
mp = self.mountpoint_for(m.slug)
if not self.backend.is_mounted(mp):
return None
try:
st = run_with_timeout(lambda: os.statvfs(mp), self.probe_timeout, f"statvfs:{m.name}")
except (OSError, OperationTimeout):
return None
return st.f_blocks * st.f_frsize, st.f_bavail * st.f_frsize
+68
View File
@@ -0,0 +1,68 @@
"""``/proc/self/mountinfo`` lesen (Fallstrick 2).
Ob ein Verzeichnis eingehängt ist, entscheidet ausschließlich der Kernel – nie
das Vorhandensein von Dateien. Ein leerer Mountpoint sieht sonst genauso aus wie
eine leere Freigabe.
"""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from pathlib import Path
MOUNTINFO = Path("/proc/self/mountinfo")
_OCTAL_RE = re.compile(r"\\([0-7]{3})")
def _unescape(field: str) -> str:
# Der Kernel kodiert Leerzeichen, Tab, Zeilenumbruch und Backslash oktal.
return _OCTAL_RE.sub(lambda m: chr(int(m.group(1), 8)), field)
@dataclass(frozen=True)
class MountEntry:
mountpoint: str
fstype: str
source: str
options: str
def parse_mountinfo(text: str) -> list[MountEntry]:
entries: list[MountEntry] = []
for line in text.splitlines():
if not line.strip():
continue
left, sep, right = line.partition(" - ")
if not sep:
continue
lf = left.split(" ")
rf = right.split(" ")
if len(lf) < 6 or len(rf) < 3:
continue
entries.append(
MountEntry(
mountpoint=_unescape(lf[4]),
fstype=rf[0],
source=_unescape(rf[1]),
options=rf[2],
)
)
return entries
def read_mountinfo(path: Path = MOUNTINFO) -> list[MountEntry]:
return parse_mountinfo(path.read_text(encoding="utf-8", errors="surrogateescape"))
def find_mount(mountpoint: Path, entries: list[MountEntry]) -> MountEntry | None:
"""Liefert den obersten Eintrag für genau diesen Mountpoint (bei Übereinander-
Mounts gewinnt der zuletzt eingehängte)."""
target = os.path.normpath(str(mountpoint))
found: MountEntry | None = None
for entry in entries:
if entry.mountpoint == target:
found = entry
return found
+134
View File
@@ -0,0 +1,134 @@
"""Ergebnis eines Mount-Tests: drei getrennte Schritte plus abgeleitete Rollen.
Fallstrick 12: Der Schreibtest erkennt Root-Squash auf NFS und nennt Ursache
und Abhilfe im Klartext.
"""
from __future__ import annotations
import errno
import json
from dataclasses import asdict, dataclass, field
from enum import StrEnum
from netmover.models import MountType
class Role(StrEnum):
SOURCE_COPY = "nur Quelle, kopieren"
SOURCE_MOVE = "Quelle mit Löschen"
TARGET = "Ziel"
SYNOLOGY_SQUASH_HINT = (
"Synology: Systemsteuerung → Gemeinsamer Ordner → Ordner wählen → Bearbeiten → "
"NFS-Berechtigungen → Regel bearbeiten → Squash: „Keine Zuordnung“ "
"(No mapping) oder „Alle Benutzer zu admin zuordnen“."
)
@dataclass
class CheckStep:
name: str # "mount" | "read" | "write" (bei sftp: "connect")
ok: bool | None # None = nicht ausgeführt
message: str = ""
@dataclass
class CheckReport:
steps: list[CheckStep] = field(default_factory=list)
can_read: bool = False
can_write: bool = False
can_delete: bool = False
root_squash: bool | None = None
owner: str | None = None
total_bytes: int | None = None
free_bytes: int | None = None
hints: list[str] = field(default_factory=list)
host_key: str | None = None # sftp: festgestellter Hostschlüssel
host_key_fingerprint: str | None = None
@property
def ok(self) -> bool:
return all(s.ok is not False for s in self.steps) and self.can_read
@property
def roles(self) -> list[Role]:
roles: list[Role] = []
if self.can_read:
roles.append(Role.SOURCE_MOVE if self.can_delete else Role.SOURCE_COPY)
if self.can_write:
roles.append(Role.TARGET)
return roles
def step(self, name: str, ok: bool | None, message: str = "") -> CheckStep:
s = CheckStep(name, ok, message)
self.steps.append(s)
return s
def to_json(self) -> str:
data = asdict(self)
data["roles"] = [r.value for r in self.roles]
data["ok"] = self.ok
return json.dumps(data, ensure_ascii=False)
@dataclass
class WriteAssessment:
write_ok: bool
delete_ok: bool
root_squash: bool | None
message: str
hints: list[str]
def assess_write(
mount_type: MountType,
euid: int,
write_error: OSError | None,
created_uid: int | None,
delete_error: OSError | None,
) -> WriteAssessment:
"""Reine Auswertung des Schreibtests – getrennt von den Dateizugriffen, damit
sich Root-Squash ohne root und ohne NAS testen lässt."""
is_nfs = mount_type in (MountType.NFS3, MountType.NFS4)
hints: list[str] = []
if write_error is not None:
denied = write_error.errno in (errno.EACCES, errno.EPERM)
if is_nfs and euid == 0 and denied:
hints.append(
"Schreiben als root wurde abgelehnt – vermutlich Root-Squash: Der NFS-Server "
"bildet root auf „nobody“ ab, und „nobody“ darf hier nicht schreiben. "
+ SYNOLOGY_SQUASH_HINT
)
return WriteAssessment(False, False, True, "Schreiben verweigert (Root-Squash?)", hints)
if write_error.errno == errno.EROFS:
return WriteAssessment(False, False, None, "Freigabe ist schreibgeschützt", hints)
if denied:
hints.append("Keine Schreibrechte – als Quelle nur zum Kopieren geeignet.")
return WriteAssessment(
False, False, None, f"Schreiben fehlgeschlagen: {write_error.strerror}", hints
)
root_squash: bool | None = None
message = "Datei angelegt und gelöscht"
if is_nfs and euid == 0 and created_uid is not None:
root_squash = created_uid != 0
if root_squash:
message = f"Schreiben möglich, Datei gehört aber uid {created_uid} (Root-Squash)"
hints.append(
"Root-Squash aktiv: Dateien gehören nicht root, „chown“ nach dem Verschieben "
"ist nicht möglich und wird übersprungen. Wer Eigentümer setzen will: "
+ SYNOLOGY_SQUASH_HINT
)
if delete_error is not None:
return WriteAssessment(
True,
False,
root_squash,
f"Angelegt, aber Löschen fehlgeschlagen: {delete_error.strerror}",
[*hints, "Testdatei konnte nicht gelöscht werden – bitte von Hand entfernen."],
)
return WriteAssessment(True, True, root_squash, message, hints)
+260
View File
@@ -0,0 +1,260 @@
"""SFTP-Anbindung für Ziele, die nur per SSH erreichbar sind (z. B. ein LXC).
Kein Kernel-Mount – daher kein Hängen im Kernel, aber auch kein ``ismount``.
Der Hostschlüssel wird beim ersten Test festgehalten (Trust on first use) und ist
danach verbindlich; eine Abweichung bricht jede Verbindung ab.
"""
from __future__ import annotations
import base64
import contextlib
import hashlib
import io
import posixpath
import socket
import stat
import uuid
from collections.abc import Iterator
from dataclasses import dataclass
from types import TracebackType
import paramiko
from paramiko.sftp import CMD_EXTENDED, CMD_EXTENDED_REPLY
from netmover.crypto import redact
from netmover.mounting.report import CheckReport
class SftpError(RuntimeError):
pass
class HostKeyMismatch(SftpError):
pass
class HostKeyUnknown(SftpError):
pass
@dataclass(frozen=True)
class SftpConfig:
host: str
port: int
username: str
base_path: str
password: str | None = None
private_key: str | None = None
host_key: str | None = None # "<typ> <base64>"
def __repr__(self) -> str:
return (
f"SftpConfig(host={self.host!r}, port={self.port}, username={self.username!r}, "
f"base_path={self.base_path!r})"
)
@property
def secrets(self) -> list[str | None]:
return [self.password, self.private_key]
def fingerprint(key: paramiko.PKey) -> str:
digest = hashlib.sha256(key.asbytes()).digest()
return "SHA256:" + base64.b64encode(digest).decode().rstrip("=")
def host_key_string(key: paramiko.PKey) -> str:
return f"{key.get_name()} {key.get_base64()}"
def load_private_key(pem: str) -> paramiko.PKey:
errors: list[str] = []
key_classes: list[type[paramiko.PKey]] = [
paramiko.Ed25519Key,
paramiko.ECDSAKey,
paramiko.RSAKey,
]
for cls in key_classes:
try:
return cls.from_private_key(io.StringIO(pem))
except (paramiko.SSHException, ValueError) as exc:
errors.append(f"{cls.__name__}: {exc}")
raise SftpError("Privater Schlüssel nicht lesbar (Ed25519, ECDSA oder RSA ohne Passphrase)")
class SftpSession:
def __init__(self, transport: paramiko.Transport, client: paramiko.SFTPClient) -> None:
self.transport = transport
self.client = client
key = transport.get_remote_server_key()
self.host_key = host_key_string(key)
self.fingerprint = fingerprint(key)
def close(self) -> None:
with contextlib.suppress(Exception):
self.client.close()
with contextlib.suppress(Exception):
self.transport.close()
def __enter__(self) -> SftpSession:
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc: BaseException | None,
tb: TracebackType | None,
) -> None:
self.close()
# -- Erweiterungen von OpenSSH (paramiko bietet sie nicht direkt an) -----------
def statvfs(self, path: str) -> tuple[int, int] | None:
"""(gesamt, frei für Nutzer) in Bytes über ``statvfs@openssh.com``, sonst None."""
try:
t, msg = self.client._request( # type: ignore[attr-defined]
CMD_EXTENDED, "statvfs@openssh.com", path
)
except (OSError, paramiko.SSHException):
return None
if t != CMD_EXTENDED_REPLY or msg is None:
return None
_bsize, frsize, blocks, _bfree, bavail = (msg.get_int64() for _ in range(5))
return blocks * frsize, bavail * frsize
def fsync(self, handle: paramiko.SFTPFile) -> bool:
"""``fsync@openssh.com`` auf eine offene Datei; False, wenn nicht unterstützt."""
try:
self.client._request( # type: ignore[attr-defined]
CMD_EXTENDED, "fsync@openssh.com", handle.handle
)
except (OSError, paramiko.SSHException):
return False
return True
def connect(
cfg: SftpConfig, timeout: float = 20, allow_unknown_host_key: bool = False
) -> SftpSession:
try:
sock = socket.create_connection((cfg.host, cfg.port), timeout=timeout)
except OSError as exc:
raise SftpError(f"Verbindung zu {cfg.host}:{cfg.port} fehlgeschlagen: {exc}") from None
transport = paramiko.Transport(sock)
transport.banner_timeout = timeout
transport.auth_timeout = timeout
try:
transport.start_client(timeout=timeout)
server_key = transport.get_remote_server_key()
seen = host_key_string(server_key)
if cfg.host_key is None:
if not allow_unknown_host_key:
raise HostKeyUnknown(
"Hostschlüssel noch nicht bestätigt – bitte die Anbindung zuerst testen."
)
elif cfg.host_key.split()[:2] != seen.split()[:2]:
raise HostKeyMismatch(
f"Hostschlüssel von {cfg.host} hat sich geändert "
f"(jetzt {fingerprint(server_key)}). "
"Wurde der Server neu aufgesetzt? Dann Hostschlüssel in der Anbindung "
"zurücksetzen und neu testen – sonst nicht verbinden!"
)
if cfg.private_key:
transport.auth_publickey(cfg.username, load_private_key(cfg.private_key))
elif cfg.password is not None:
transport.auth_password(cfg.username, cfg.password)
else:
raise SftpError("Keine Zugangsdaten hinterlegt")
client = paramiko.SFTPClient.from_transport(transport)
if client is None:
raise SftpError("SFTP-Subsystem nicht verfügbar")
client.get_channel().settimeout(timeout) # type: ignore[union-attr]
return SftpSession(transport, client)
except paramiko.AuthenticationException:
transport.close()
raise SftpError(
"Anmeldung abgelehnt – Benutzername prüfen und den öffentlichen Schlüssel in "
"~/.ssh/authorized_keys des Zielbenutzers eintragen."
) from None
except (paramiko.SSHException, OSError, EOFError) as exc:
transport.close()
raise SftpError(redact(f"SSH-Fehler: {exc}", cfg.secrets)) from None
except SftpError:
transport.close()
raise
@contextlib.contextmanager
def session(cfg: SftpConfig, timeout: float = 20) -> Iterator[SftpSession]:
s = connect(cfg, timeout=timeout)
try:
yield s
finally:
s.close()
def check_connection(cfg: SftpConfig, timeout: float = 20) -> CheckReport:
"""Verbinden, Basisordner lesen, Testdatei schreiben/umbenennen/löschen."""
report = CheckReport()
try:
sess = connect(cfg, timeout=timeout, allow_unknown_host_key=True)
except SftpError as exc:
report.step("connect", False, str(exc))
report.step("read", None)
report.step("write", None)
return report
with sess:
report.host_key = sess.host_key
report.host_key_fingerprint = sess.fingerprint
new = " (neu – bitte Fingerabdruck prüfen)" if cfg.host_key is None else ""
report.step("connect", True, f"Hostschlüssel {sess.fingerprint}{new}")
try:
attrs = sess.client.stat(cfg.base_path)
if attrs.st_mode is None or not stat.S_ISDIR(attrs.st_mode):
raise SftpError(f"{cfg.base_path} ist kein Verzeichnis")
count = len(sess.client.listdir(cfg.base_path))
report.can_read = True
report.step("read", True, f"{count} Einträge")
except (OSError, SftpError) as exc:
report.step("read", False, f"{cfg.base_path}: {exc}")
report.step("write", None)
return report
token = uuid.uuid4().hex
part = posixpath.join(cfg.base_path, f".netmover-test-{token}.part")
final = posixpath.join(cfg.base_path, f".netmover-test-{token}.tmp")
try:
with sess.client.open(part, "wb") as fh:
fh.write(b"netmover")
fh.flush()
synced = sess.fsync(fh)
sess.client.posix_rename(part, final)
st = sess.client.stat(final)
report.owner = f"{st.st_uid}:{st.st_gid}"
report.can_write = True
except OSError as exc:
report.step("write", False, f"Schreiben fehlgeschlagen: {exc}")
with contextlib.suppress(OSError):
sess.client.remove(part)
return report
try:
sess.client.remove(final)
report.can_delete = True
report.step(
"write",
True,
"Datei angelegt, atomar umbenannt und gelöscht"
+ ("" if synced else " (Server unterstützt kein fsync – Hinweis)"),
)
except OSError as exc:
report.step("write", True, f"Angelegt, aber Löschen fehlgeschlagen: {exc}")
usage = sess.statvfs(cfg.base_path)
if usage:
report.total_bytes, report.free_bytes = usage
else:
report.hints.append(
"Server meldet keinen freien Speicher (statvfs) – Platzprüfung wird übersprungen."
)
return report
+108 -7
View File
@@ -1,12 +1,17 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Iterator from collections.abc import Callable, Iterator
from pathlib import Path from pathlib import Path
import pytest import pytest
from sqlalchemy.orm import Session from cryptography.fernet import Fernet
from sqlalchemy.orm import Session, sessionmaker
from netmover.db import make_engine, make_session_factory, upgrade from netmover.crypto import SecretBox
from netmover.db import make_engine, make_session_factory, session_scope, upgrade
from netmover.models import Credential, CredentialKind, Mount, MountType
from netmover.mounting.fake import FakeMountBackend
from netmover.mounting.manager import MountManager
@pytest.fixture @pytest.fixture
@@ -17,9 +22,105 @@ def db_url(tmp_path: Path) -> str:
@pytest.fixture @pytest.fixture
def session(db_url: str) -> Iterator[Session]: def session_factory(db_url: str) -> Iterator[sessionmaker[Session]]:
engine = make_engine(db_url) engine = make_engine(db_url)
factory = make_session_factory(engine) yield make_session_factory(engine)
with factory() as s:
yield s
engine.dispose() engine.dispose()
@pytest.fixture
def session(session_factory: sessionmaker[Session]) -> Iterator[Session]:
with session_factory() as s:
yield s
@pytest.fixture
def secretbox() -> SecretBox:
return SecretBox(Fernet.generate_key())
@pytest.fixture
def fake_backend(tmp_path: Path) -> Iterator[FakeMountBackend]:
backend = FakeMountBackend(tmp_path / "fake")
yield backend
backend.release_hangs()
@pytest.fixture
def mount_base(tmp_path: Path) -> Path:
return tmp_path / "mnt"
@pytest.fixture
def sleeps() -> list[float]:
return []
@pytest.fixture
def notifications() -> list[tuple[str, str, str]]:
return []
@pytest.fixture
def manager(
session_factory: sessionmaker[Session],
fake_backend: FakeMountBackend,
secretbox: SecretBox,
mount_base: Path,
sleeps: list[float],
notifications: list[tuple[str, str, str]],
) -> MountManager:
return MountManager(
session_factory,
fake_backend,
secretbox,
mount_base,
probe_timeout=0.5,
recover_backoff=(0, 1, 2),
sleep=sleeps.append,
notifier=lambda *a: notifications.append(a),
)
MountFactory = Callable[..., int]
@pytest.fixture
def make_mount(
session_factory: sessionmaker[Session], secretbox: SecretBox, mount_base: Path
) -> MountFactory:
def factory(
name: str = "scans",
type: MountType = MountType.CIFS,
server: str = "nas",
share: str = "scans",
password: str | None = "geheim123",
**kw: object,
) -> int:
with session_scope(session_factory) as s:
cred_id = None
if password is not None:
c = Credential(
name=f"cred-{name}",
kind=CredentialKind.PASSWORD,
username="scanner",
secret_enc=secretbox.encrypt(password),
)
s.add(c)
s.flush()
cred_id = c.id
m = Mount(
name=name,
slug=name,
type=type,
server=server,
share=share,
mountpoint=None if type is MountType.SFTP else str(mount_base.resolve() / name),
cred_id=cred_id,
**kw,
)
s.add(m)
s.flush()
return m.id
return factory
+202
View File
@@ -0,0 +1,202 @@
"""Minimaler SFTP-Server für Tests (paramiko-Serverseite, auf ein Verzeichnis beschränkt).
Unterstützt ``posix-rename@openssh.com`` (paramiko leitet es an ``posix_rename``
weiter), aber bewusst nicht ``statvfs@openssh.com`` – damit ist auch der
Rückfallweg ohne Platzprüfung abgedeckt.
"""
from __future__ import annotations
import contextlib
import io
import os
import socket
import threading
from pathlib import Path
from typing import Any
import paramiko
from paramiko import SFTPAttributes, SFTPHandle, SFTPServer, SFTPServerInterface
from paramiko.sftp import SFTP_FAILURE, SFTP_OK
from netmover.crypto import generate_ssh_keypair
class _Server(paramiko.ServerInterface):
def __init__(self, username: str, password: str | None, pubkey: paramiko.PKey | None) -> None:
self.username = username
self.password = password
self.pubkey = pubkey
def check_auth_password(self, username: str, password: str) -> int:
if username == self.username and self.password is not None and password == self.password:
return paramiko.AUTH_SUCCESSFUL
return paramiko.AUTH_FAILED
def check_auth_publickey(self, username: str, key: paramiko.PKey) -> int:
if username == self.username and self.pubkey is not None and key == self.pubkey:
return paramiko.AUTH_SUCCESSFUL
return paramiko.AUTH_FAILED
def get_allowed_auths(self, username: str) -> str:
return "password,publickey"
def check_channel_request(self, kind: str, chanid: int) -> int:
if kind == "session":
return paramiko.OPEN_SUCCEEDED
return paramiko.OPEN_FAILED_ADMINISTRATIVELY_PROHIBITED
class _Handle(SFTPHandle):
pass
class _RootedSFTP(SFTPServerInterface):
def __init__(self, server: Any, *args: Any, root: Path, **kwargs: Any) -> None:
super().__init__(server, *args, **kwargs)
self.root = root
def _real(self, path: str) -> str:
return str(self.root / self.canonicalize(path).lstrip("/"))
def list_folder(self, path: str) -> Any:
try:
real = self._real(path)
out = []
for name in os.listdir(real):
attr = SFTPAttributes.from_stat(os.stat(os.path.join(real, name)))
attr.filename = name
out.append(attr)
return out
except OSError as e:
return SFTPServer.convert_errno(e.errno)
def stat(self, path: str) -> Any:
try:
return SFTPAttributes.from_stat(os.stat(self._real(path)))
except OSError as e:
return SFTPServer.convert_errno(e.errno)
lstat = stat
def open(self, path: str, flags: int, attr: SFTPAttributes) -> Any:
real = self._real(path)
try:
fd = os.open(real, flags | getattr(os, "O_BINARY", 0), 0o644)
except OSError as e:
return SFTPServer.convert_errno(e.errno)
if flags & os.O_WRONLY:
mode = "ab" if flags & os.O_APPEND else "wb"
elif flags & os.O_RDWR:
mode = "a+b" if flags & os.O_APPEND else "r+b"
else:
mode = "rb"
f = os.fdopen(fd, mode)
h = _Handle(flags)
h.filename = real
h.readfile = f
h.writefile = f
return h
def remove(self, path: str) -> int:
try:
os.remove(self._real(path))
except OSError as e:
return SFTPServer.convert_errno(e.errno)
return SFTP_OK
def rename(self, oldpath: str, newpath: str) -> int:
new = self._real(newpath)
if os.path.exists(new):
return SFTP_FAILURE # SFTPv3-rename überschreibt nicht
try:
os.rename(self._real(oldpath), new)
except OSError as e:
return SFTPServer.convert_errno(e.errno)
return SFTP_OK
def posix_rename(self, oldpath: str, newpath: str) -> int:
try:
os.replace(self._real(oldpath), self._real(newpath))
except OSError as e:
return SFTPServer.convert_errno(e.errno)
return SFTP_OK
def mkdir(self, path: str, attr: SFTPAttributes) -> int:
try:
os.mkdir(self._real(path))
except OSError as e:
return SFTPServer.convert_errno(e.errno)
return SFTP_OK
def rmdir(self, path: str) -> int:
try:
os.rmdir(self._real(path))
except OSError as e:
return SFTPServer.convert_errno(e.errno)
return SFTP_OK
def chattr(self, path: str, attr: SFTPAttributes) -> int:
try:
if attr.st_mode is not None:
os.chmod(self._real(path), attr.st_mode & 0o7777)
if attr.st_atime is not None and attr.st_mtime is not None:
os.utime(self._real(path), (attr.st_atime, attr.st_mtime))
except OSError as e:
return SFTPServer.convert_errno(e.errno)
return SFTP_OK
def _load_key(pem: str) -> paramiko.PKey:
return paramiko.Ed25519Key.from_private_key(io.StringIO(pem))
class StubSFTPServer:
def __init__(
self, root: Path, username: str = "paperless", password: str | None = None
) -> None:
self.root = root
self.username = username
self.password = password
host_pem, _ = generate_ssh_keypair("host")
self.host_key = _load_key(host_pem)
self.client_private_key, self.client_public_key = generate_ssh_keypair("netmover")
self.authorized = _load_key(self.client_private_key)
self._sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._sock.bind(("127.0.0.1", 0))
self._sock.listen(8)
self.port: int = self._sock.getsockname()[1]
self._stop = threading.Event()
self._transports: list[paramiko.Transport] = []
self._thread = threading.Thread(target=self._serve, daemon=True)
@property
def host_key_string(self) -> str:
return f"{self.host_key.get_name()} {self.host_key.get_base64()}"
def start(self) -> StubSFTPServer:
self._thread.start()
return self
def _serve(self) -> None:
self._sock.settimeout(0.2)
while not self._stop.is_set():
try:
conn, _ = self._sock.accept()
except (TimeoutError, OSError):
continue
t = paramiko.Transport(conn)
t.add_server_key(self.host_key)
t.set_subsystem_handler("sftp", SFTPServer, _RootedSFTP, root=self.root)
self._transports.append(t)
with contextlib.suppress(Exception):
t.start_server(server=_Server(self.username, self.password, self.authorized))
def stop(self) -> None:
self._stop.set()
for t in self._transports:
with contextlib.suppress(Exception):
t.close()
self._sock.close()
self._thread.join(timeout=2)
+117
View File
@@ -0,0 +1,117 @@
"""Fallstricke 5 und 7 am echten Backend – ``subprocess`` wird nur abgefangen."""
from __future__ import annotations
import stat
import subprocess
from pathlib import Path
from typing import Any
import pytest
from netmover.mounting.backend import CifsCredentials, MountError, MountSpec
from netmover.mounting.linux import LinuxMountBackend, explain_mount_error
class Recorder:
def __init__(self, returncode: int = 0, stderr: str = "") -> None:
self.returncode = returncode
self.stderr = stderr
self.calls: list[tuple[list[str], dict[str, Any]]] = []
self.cred_snapshot: dict[str, Any] = {}
def __call__(self, argv: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
self.calls.append((argv, kwargs))
for opt in ",".join(argv).split(","):
if opt.startswith("credentials="):
p = Path(opt.removeprefix("credentials="))
self.cred_snapshot = {
"path": p,
"mode": stat.S_IMODE(p.stat().st_mode),
"content": p.read_text(),
}
return subprocess.CompletedProcess(argv, self.returncode, "", self.stderr)
def _spec(tmp_path: Path, password: str = "S3cr3t,pw=x") -> MountSpec:
return MountSpec(
fstype="cifs",
source="//nas/scans",
mountpoint=tmp_path / "mnt" / "scans",
options=("vers=3.0", "uid=1000"),
credentials=CifsCredentials("scanner", password, "WORKGROUP"),
)
def test_mount_uses_argv_and_credentials_file(tmp_path: Path) -> None:
rec = Recorder()
backend = LinuxMountBackend(tmp_path / "run", runner=rec)
backend.mount(_spec(tmp_path))
argv, kwargs = rec.calls[0]
assert "shell" not in kwargs
assert argv[:4] == ["mount", "-t", "cifs", "-o"]
assert argv[-2:] == ["//nas/scans", str(tmp_path / "mnt" / "scans")]
assert argv[4].startswith("vers=3.0,uid=1000,credentials=")
# Passwort nie in der Prozessliste
assert not any("S3cr3t" in a for a in argv)
assert "password" not in argv[4]
# Credentials-Datei: 0600, Inhalt korrekt, danach gelöscht
snap = rec.cred_snapshot
assert snap["mode"] == 0o600
assert snap["content"] == "username=scanner\npassword=S3cr3t,pw=x\ndomain=WORKGROUP\n"
assert snap["path"].parent == tmp_path / "run"
assert not snap["path"].exists()
assert kwargs["env"]["LC_ALL"] == "C"
def test_credentials_file_removed_on_failure_and_output_redacted(tmp_path: Path) -> None:
rec = Recorder(returncode=32, stderr="mount error(13): Permission denied for S3cr3t,pw=x")
backend = LinuxMountBackend(tmp_path / "run", runner=rec)
with pytest.raises(MountError) as exc:
backend.mount(_spec(tmp_path))
assert "S3cr3t" not in str(exc.value)
assert "Benutzername/Passwort" in str(exc.value)
assert list((tmp_path / "run").iterdir()) == []
def test_mount_timeout(tmp_path: Path) -> None:
def runner(argv: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
raise subprocess.TimeoutExpired(argv, kwargs["timeout"])
backend = LinuxMountBackend(tmp_path / "run", runner=runner)
with pytest.raises(MountError, match="keine Antwort nach 5 s"):
backend.mount(_spec(tmp_path), timeout=5)
assert list((tmp_path / "run").iterdir()) == []
def test_unmount_lazy_argv(tmp_path: Path) -> None:
rec = Recorder()
backend = LinuxMountBackend(tmp_path / "run", runner=rec)
backend.unmount(Path("/mnt/netmover/scans"), lazy=True)
assert rec.calls[0][0] == ["umount", "-l", "/mnt/netmover/scans"]
def test_is_mounted_reads_mountinfo(tmp_path: Path) -> None:
info = tmp_path / "mountinfo"
info.write_text("98 22 0:52 / /mnt/netmover/scans rw - cifs //nas/scans rw\n")
backend = LinuxMountBackend(tmp_path / "run", mountinfo_path=info, runner=Recorder())
assert backend.is_mounted(Path("/mnt/netmover/scans"))
assert not backend.is_mounted(Path("/mnt/netmover/other"))
@pytest.mark.parametrize(
("fstype", "msg", "expect"),
[
(
"cifs",
"mount: wrong fs type, bad option; you might need a /sbin/mount.<type> helper program",
"cifs-utils",
),
("nfs4", "mount.nfs4: access denied by server while mounting nas:/x", "NFS-Berechtigungen"),
("cifs", "mount error(112): Host is down", "nicht erreichbar"),
("cifs", "mount: only root can do that", "CAP_SYS_ADMIN"),
],
)
def test_explain(fstype: str, msg: str, expect: str) -> None:
assert expect in explain_mount_error(fstype, msg)
+369
View File
@@ -0,0 +1,369 @@
"""Mount-Verwaltung mit FakeMountBackend – Fallstricke 2, 6, 7 und 13."""
from __future__ import annotations
import errno
from pathlib import Path
import pytest
from sqlalchemy.orm import Session, sessionmaker
from netmover.models import CredentialKind, Mount, MountState, MountType
from netmover.mounting.backend import MountError
from netmover.mounting.fake import FakeMountBackend
from netmover.mounting.manager import (
CredentialData,
MountConfig,
MountManager,
MountUnavailable,
build_spec,
)
from netmover.validation import ValidationError
from tests.conftest import MountFactory
def _state(sf: sessionmaker[Session], mount_id: int) -> Mount:
with sf() as s:
m = s.get(Mount, mount_id)
assert m is not None
s.expunge(m)
return m
# -- build_spec -------------------------------------------------------------------
def test_build_spec_cifs(tmp_path: Path) -> None:
cfg = MountConfig(
"scans",
MountType.CIFS,
"nas",
"scans",
uid=1000,
gid=100,
file_mode="0644",
extra_options="vers=3.0",
)
cred = CredentialData(CredentialKind.PASSWORD, "scanner", "pw", "WG")
spec = build_spec(cfg, tmp_path, cred)
assert spec.fstype == "cifs"
assert spec.source == "//nas/scans"
assert spec.options == ("vers=3.0", "uid=1000", "gid=100", "file_mode=0644")
assert spec.credentials is not None and spec.credentials.password == "pw"
assert "pw" not in repr(spec)
def test_build_spec_nfs() -> None:
nfs3 = build_spec(MountConfig("a", MountType.NFS3, "nas", "/volume1/a"), Path("/x"), None)
assert (nfs3.fstype, nfs3.source, nfs3.options) == ("nfs", "nas:/volume1/a", ("vers=3",))
nfs4 = build_spec(MountConfig("a", MountType.NFS4, "nas", "/volume1/a"), Path("/x"), None)
assert (nfs4.fstype, nfs4.options) == ("nfs4", ())
def test_build_spec_guest_and_revalidation() -> None:
spec = build_spec(MountConfig("a", MountType.CIFS, "nas", "pub"), Path("/x"), None)
assert spec.options == ("guest",)
# Auch Werte „aus der Datenbank“ laufen erneut durch die Validierung
with pytest.raises(ValidationError):
build_spec(MountConfig("a", MountType.CIFS, "nas;id", "pub"), Path("/x"), None)
with pytest.raises(ValidationError):
build_spec(
MountConfig("a", MountType.CIFS, "nas", "pub", extra_options="password=x"),
Path("/x"),
None,
)
# -- Einhängen / Aushängen (Fallstrick 6) --------------------------------------------
def test_mount_creates_mountpoint_under_base_and_unmount_removes_it(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
mount_base: Path,
session_factory: sessionmaker[Session],
) -> None:
remote = fake_backend.export("nas", "scans")
(remote / "a.pdf").write_text("x")
mid = make_mount()
manager.mount(mid)
mp = mount_base.resolve() / "scans"
assert fake_backend.is_mounted(mp)
assert (mp / "a.pdf").read_text() == "x"
m = _state(session_factory, mid)
assert m.state is MountState.MOUNTED and m.last_mounted_at is not None
# Zugangsdaten wurden entschlüsselt übergeben
spec = fake_backend.calls[0].spec
assert spec is not None and spec.credentials is not None
assert spec.credentials.password == "geheim123"
manager.unmount(mid)
assert not mp.exists()
assert _state(session_factory, mid).state is MountState.UNMOUNTED
def test_mount_refuses_non_empty_mountpoint(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
mount_base: Path,
session_factory: sessionmaker[Session],
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
mp = mount_base.resolve() / "scans"
mp.mkdir(parents=True)
(mp / "verirrt.pdf").write_text("lokal geschrieben")
with pytest.raises(MountError, match="nicht leer"):
manager.mount(mid)
assert not fake_backend.calls
assert (mp / "verirrt.pdf").exists()
assert _state(session_factory, mid).state is MountState.ERROR
def test_failed_mount_cleans_up_and_records_error(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
mount_base: Path,
session_factory: sessionmaker[Session],
) -> None:
mid = make_mount() # nicht exportiert → „No such file or directory“
with pytest.raises(MountError):
manager.mount(mid)
assert not (mount_base.resolve() / "scans").exists()
m = _state(session_factory, mid)
assert m.state is MountState.ERROR
assert m.last_error and "No such file" in m.last_error
def test_mountpoint_for_rejects_escape(manager: MountManager) -> None:
for bad in ["../etc", "a/b", "", "Scans"]:
with pytest.raises(ValidationError):
manager.mountpoint_for(bad)
def test_sftp_is_never_mounted(manager: MountManager, make_mount: MountFactory) -> None:
mid = make_mount("paperless", type=MountType.SFTP, share="/consume")
with pytest.raises(MountError, match="SFTP"):
manager.mount(mid)
def test_restore_all_only_auto_mount(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
notifications: list[tuple[str, str, str]],
) -> None:
fake_backend.export("nas", "a")
fake_backend.export("nas", "b")
a = make_mount("a", share="a")
make_mount("b", share="b", auto_mount=False)
make_mount("c", share="fehlt")
make_mount("d", share="a", enabled=False)
results = {r.name: r.ok for r in manager.restore_all()}
assert results == {"a": True, "c": False}
assert fake_backend.is_mounted(manager.mountpoint_for("a"))
assert notifications and notifications[0][:2] == ("mount.restore_failed", "c")
# zweiter Aufruf ist idempotent
manager.mount(a)
assert (
sum(1 for c in fake_backend.calls if c.action == "mount" and c.mountpoint.name == "a") == 1
)
# -- Test (drei Schritte) -----------------------------------------------------------
def test_check_unsaved_config_mounts_temporarily(
manager: MountManager, fake_backend: FakeMountBackend, mount_base: Path
) -> None:
fake_backend.export("nas", "scans")
cfg = MountConfig("neu", MountType.CIFS, "nas", "scans")
cred = CredentialData(CredentialKind.PASSWORD, "u", "pw")
report = manager.check(cfg, cred)
assert [(s.name, s.ok) for s in report.steps] == [
("mount", True),
("read", True),
("write", True),
]
assert report.can_write and report.can_delete
assert report.free_bytes is not None
assert [r.value for r in report.roles] == ["Quelle mit Löschen", "Ziel"]
# temporärer Mountpoint wieder weg, Testdatei gelöscht
assert list(mount_base.iterdir()) == []
assert list(fake_backend.remote_root.joinpath("nas", "scans").iterdir()) == []
assert not fake_backend.mounted
def test_check_mount_failure_reports_steps(manager: MountManager, mount_base: Path) -> None:
report = manager.check(MountConfig("x", MountType.CIFS, "nas", "fehlt"), None)
assert [(s.name, s.ok) for s in report.steps] == [
("mount", False),
("read", None),
("write", None),
]
assert not report.ok
assert list(mount_base.iterdir()) == []
def test_check_read_only_share_is_source_copy_only(
manager: MountManager, fake_backend: FakeMountBackend
) -> None:
remote = fake_backend.export("nas", "ro")
remote.chmod(0o555)
try:
report = manager.check(MountConfig("ro", MountType.CIFS, "nas", "ro"), None)
finally:
remote.chmod(0o755)
assert report.can_read and not report.can_write
assert [r.value for r in report.roles] == ["nur Quelle, kopieren"]
assert report.steps[2].ok is False
def test_check_stores_result_on_saved_mount(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
session_factory: sessionmaker[Session],
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
manager.mount(mid)
m = _state(session_factory, mid)
report = manager.check(MountConfig.from_model(m), None, mount_id=mid)
assert report.steps[0].message == "bereits eingehängt"
m = _state(session_factory, mid)
assert m.write_ok is True and m.last_test_result and '"ok": true' in m.last_test_result
assert fake_backend.is_mounted(manager.mountpoint_for("scans")) # bleibt eingehängt
# -- Fallstrick 2: nicht eingehängt ---------------------------------------------------
def test_check_alive_refuses_unmounted_mountpoint(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
session_factory: sessionmaker[Session],
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
manager.mount(mid)
mp = manager.check_alive(mid)
assert mp is not None
fake_backend.vanish(mp)
assert mp.is_dir() and list(mp.iterdir()) == [] # sieht aus wie eine leere Freigabe
with pytest.raises(MountUnavailable, match="nicht eingehängt"):
manager.check_alive(mid)
m = _state(session_factory, mid)
assert m.state is MountState.ERROR
# kein Wiederherstellungsversuch, kein Schreiben
assert [c.action for c in fake_backend.calls] == ["mount"]
assert list(mp.iterdir()) == []
# -- Fallstrick 13: gestörte und hängende Mounts ---------------------------------------
@pytest.mark.parametrize("code", [errno.ESTALE, errno.EIO, errno.ENOTCONN])
def test_stale_mount_is_lazily_unmounted_and_remounted(
code: int,
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
session_factory: sessionmaker[Session],
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
manager.mount(mid)
mp = manager.mountpoint_for("scans")
fake_backend.inject_fault(mp, code)
assert manager.check_alive(mid) == mp
actions = [(c.action, c.lazy) for c in fake_backend.calls]
assert actions == [("mount", False), ("unmount", True), ("mount", False)]
m = _state(session_factory, mid)
assert m.state is MountState.MOUNTED and m.last_error is None
def test_recovery_gives_up_after_three_attempts_with_growing_delay(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
session_factory: sessionmaker[Session],
sleeps: list[float],
notifications: list[tuple[str, str, str]],
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
manager.mount(mid)
mp = manager.mountpoint_for("scans")
fake_backend.inject_fault(mp, errno.ESTALE)
fake_backend.fail_next_mount(times=3)
with pytest.raises(MountUnavailable, match="3 Versuchen"):
manager.check_alive(mid)
assert sleeps == [1, 2] # 1. Versuch sofort, dann wachsender Abstand
assert sum(1 for c in fake_backend.calls if c.action == "mount") == 4
m = _state(session_factory, mid)
assert m.state is MountState.ERROR
assert notifications[-1][0] == "mount.error"
def test_hanging_mount_times_out_and_is_recovered(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
session_factory: sessionmaker[Session],
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
manager.mount(mid)
mp = manager.mountpoint_for("scans")
fake_backend.inject_fault(mp, "hang")
assert manager.check_alive(mid) == mp # blockiert nicht, Zeitlimit 0.5 s
assert ("unmount", True) in [(c.action, c.lazy) for c in fake_backend.calls]
assert _state(session_factory, mid).state is MountState.MOUNTED
def test_hanging_mount_without_recovery_shows_since(
manager: MountManager,
fake_backend: FakeMountBackend,
make_mount: MountFactory,
session_factory: sessionmaker[Session],
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
manager.mount(mid)
mp = manager.mountpoint_for("scans")
fake_backend.heal_on_remount = False
fake_backend.inject_fault(mp, "hang")
with pytest.raises(MountUnavailable):
manager.check_alive(mid)
m = _state(session_factory, mid)
assert m.state is MountState.ERROR
assert m.state_since is not None
assert m.last_error and "keine Antwort" in m.last_error
def test_non_recoverable_error_does_not_remount(
manager: MountManager, fake_backend: FakeMountBackend, make_mount: MountFactory
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
manager.mount(mid)
fake_backend.inject_fault(manager.mountpoint_for("scans"), errno.EACCES)
with pytest.raises(MountUnavailable):
manager.check_alive(mid)
assert [c.action for c in fake_backend.calls] == ["mount"]
def test_usage(
manager: MountManager, fake_backend: FakeMountBackend, make_mount: MountFactory
) -> None:
fake_backend.export("nas", "scans")
mid = make_mount()
assert manager.usage(mid) is None
manager.mount(mid)
usage = manager.usage(mid)
assert usage is not None and usage[0] >= usage[1] > 0
+40
View File
@@ -0,0 +1,40 @@
"""Fallstrick 2: „eingehängt?“ entscheidet /proc/self/mountinfo."""
from __future__ import annotations
from pathlib import Path
from netmover.mounting.mountinfo import find_mount, parse_mountinfo
SAMPLE = """\
22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw
98 22 0:52 / /mnt/netmover/scans rw,relatime shared:50 - cifs //nas/scans rw,vers=3.1.1
99 22 0:53 / /mnt/netmover/mit\\040leer rw,relatime - nfs4 nas:/volume1/a\\040b rw
100 22 0:54 / /mnt/netmover/scans rw,relatime - cifs //nas2/scans rw
"""
def test_parse_with_octal_escapes() -> None:
entries = parse_mountinfo(SAMPLE)
assert len(entries) == 4
assert entries[2].mountpoint == "/mnt/netmover/mit leer"
assert entries[2].source == "nas:/volume1/a b"
assert entries[2].fstype == "nfs4"
def test_find_mount_uses_topmost_entry() -> None:
entries = parse_mountinfo(SAMPLE)
found = find_mount(Path("/mnt/netmover/scans/"), entries)
assert found is not None
assert found.source == "//nas2/scans"
def test_unmounted_directory_is_not_mounted() -> None:
entries = parse_mountinfo(SAMPLE)
assert find_mount(Path("/mnt/netmover/archiv"), entries) is None
# Unterverzeichnis eines Mounts ist selbst kein Mountpoint
assert find_mount(Path("/mnt/netmover/scans/eingang"), entries) is None
def test_garbage_lines_are_ignored() -> None:
assert parse_mountinfo("kaputt\n\n1 2 3\n") == []
+61
View File
@@ -0,0 +1,61 @@
"""Fallstrick 12: Schreibtest erkennt Root-Squash und leitet Rollen ab."""
from __future__ import annotations
import errno
from netmover.models import MountType
from netmover.mounting.report import CheckReport, Role, assess_write
def _err(code: int) -> OSError:
return OSError(code, "x")
def test_nfs_root_write_denied_is_root_squash_with_synology_hint() -> None:
r = assess_write(MountType.NFS4, 0, _err(errno.EACCES), None, None)
assert not r.write_ok
assert r.root_squash is True
assert any("Synology" in h and "Squash" in h for h in r.hints)
def test_nfs_file_owned_by_nobody_is_root_squash() -> None:
r = assess_write(MountType.NFS3, 0, None, 65534, None)
assert r.write_ok and r.delete_ok
assert r.root_squash is True
assert "65534" in r.message
assert any("chown" in h for h in r.hints)
def test_nfs_no_root_squash() -> None:
r = assess_write(MountType.NFS4, 0, None, 0, None)
assert r.root_squash is False
assert r.hints == []
def test_cifs_owner_never_counts_as_squash() -> None:
r = assess_write(MountType.CIFS, 0, None, 1000, None)
assert r.root_squash is None
def test_not_root_cannot_judge_squash() -> None:
assert assess_write(MountType.NFS4, 1000, None, 1000, None).root_squash is None
def test_read_only() -> None:
r = assess_write(MountType.CIFS, 0, _err(errno.EROFS), None, None)
assert not r.write_ok
assert "schreibgeschützt" in r.message
def test_delete_failed() -> None:
r = assess_write(MountType.CIFS, 0, None, 0, _err(errno.EACCES))
assert r.write_ok and not r.delete_ok
def test_roles() -> None:
rep = CheckReport(can_read=True)
assert rep.roles == [Role.SOURCE_COPY]
rep.can_write = rep.can_delete = True
assert rep.roles == [Role.SOURCE_MOVE, Role.TARGET]
assert '"Quelle mit Löschen"' in rep.to_json()
+154
View File
@@ -0,0 +1,154 @@
"""SFTP-Ziel gegen einen echten (im Prozess laufenden) SFTP-Server."""
from __future__ import annotations
from collections.abc import Iterator
from pathlib import Path
import pytest
from sqlalchemy.orm import Session, sessionmaker
from netmover.crypto import SecretBox
from netmover.db import session_scope
from netmover.models import Credential, CredentialKind, Mount, MountState, MountType
from netmover.mounting.manager import MountManager, MountUnavailable
from netmover.mounting.sftp import (
HostKeyMismatch,
HostKeyUnknown,
SftpConfig,
SftpError,
check_connection,
connect,
)
from tests.sftp_stub import StubSFTPServer
@pytest.fixture
def server(tmp_path: Path) -> Iterator[StubSFTPServer]:
root = tmp_path / "sftp-root"
(root / "consume").mkdir(parents=True)
srv = StubSFTPServer(root, password="pw").start()
yield srv
srv.stop()
def _cfg(server: StubSFTPServer, **kw: object) -> SftpConfig:
data: dict[str, object] = {
"host": "127.0.0.1",
"port": server.port,
"username": server.username,
"base_path": "/consume",
"private_key": server.client_private_key,
}
data.update(kw)
return SftpConfig(**data) # type: ignore[arg-type]
def test_check_connection_with_key_tofu(server: StubSFTPServer) -> None:
report = check_connection(_cfg(server))
assert [(s.name, s.ok) for s in report.steps] == [
("connect", True),
("read", True),
("write", True),
]
assert report.host_key == server.host_key_string
assert report.host_key_fingerprint and report.host_key_fingerprint.startswith("SHA256:")
assert "neu" in report.steps[0].message
assert report.can_write and report.can_delete
# Stub kennt kein statvfs → Hinweis statt Fehler
assert report.free_bytes is None
assert any("statvfs" in h for h in report.hints)
assert list((server.root / "consume").iterdir()) == []
def test_password_auth(server: StubSFTPServer) -> None:
report = check_connection(
_cfg(server, private_key=None, password="pw", host_key=server.host_key_string)
)
assert report.ok
def test_wrong_credentials(server: StubSFTPServer) -> None:
report = check_connection(_cfg(server, private_key=None, password="falsch"))
assert report.steps[0].ok is False
assert "authorized_keys" in report.steps[0].message
assert "falsch" not in report.steps[0].message
def test_unknown_host_key_refused_outside_test(server: StubSFTPServer) -> None:
with pytest.raises(HostKeyUnknown):
connect(_cfg(server))
def test_host_key_mismatch(server: StubSFTPServer) -> None:
other = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOmvG9t0hJ7oC2Y3Fq6m0eKZ1x2y3z4A5B6C7D8E9F0G"
with pytest.raises(HostKeyMismatch, match="geändert"):
connect(_cfg(server, host_key=other))
report = check_connection(_cfg(server, host_key=other))
assert report.steps[0].ok is False
def test_missing_base_path(server: StubSFTPServer) -> None:
report = check_connection(_cfg(server, base_path="/fehlt"))
assert [(s.name, s.ok) for s in report.steps] == [
("connect", True),
("read", False),
("write", None),
]
def test_unreachable() -> None:
with pytest.raises(SftpError, match="fehlgeschlagen"):
connect(SftpConfig("127.0.0.1", 1, "x", "/", password="x"), timeout=2)
def test_manager_pins_host_key_and_checks_alive(
server: StubSFTPServer,
manager: MountManager,
session_factory: sessionmaker[Session],
secretbox: SecretBox,
) -> None:
with session_scope(session_factory) as s:
c = Credential(
name="paperless-key",
kind=CredentialKind.SSH_KEY,
username=server.username,
secret_enc=secretbox.encrypt(server.client_private_key),
public_key=server.client_public_key,
)
s.add(c)
s.flush()
m = Mount(
name="paperless",
slug="paperless",
type=MountType.SFTP,
server="127.0.0.1",
share="/consume",
port=server.port,
cred_id=c.id,
)
s.add(m)
s.flush()
mid = m.id
# vor dem ersten Test: Hostschlüssel unbekannt → kein Lauf möglich
with pytest.raises(MountUnavailable, match="Hostschlüssel"):
manager.check_alive(mid)
from netmover.mounting.manager import CredentialData, MountConfig
with session_factory() as s:
mm = s.get(Mount, mid)
assert mm is not None
cfg = MountConfig.from_model(mm)
cred = CredentialData(CredentialKind.SSH_KEY, server.username, server.client_private_key)
assert manager.check(cfg, cred, mount_id=mid).ok
with session_factory() as s:
mm = s.get(Mount, mid)
assert mm is not None and mm.host_key == server.host_key_string
assert manager.check_alive(mid) is None
with session_factory() as s:
mm = s.get(Mount, mid)
assert mm is not None and mm.state is MountState.MOUNTED