feat: Projektgerüst, Datenmodell und initiale Migration
Schritt 1: src-Layout mit FastAPI-Stack, SQLAlchemy-2.x-Modell inkl. Buchführung für Kopiermodus und Quarantäne, Alembic-Migration 0001, strukturierte Logs, CLI (migrate/version), Changelog und Devlog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+14
@@ -0,0 +1,14 @@
|
|||||||
|
.venv/
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*.egg-info/
|
||||||
|
build/
|
||||||
|
dist/
|
||||||
|
.pytest_cache/
|
||||||
|
.mypy_cache/
|
||||||
|
.ruff_cache/
|
||||||
|
*.db
|
||||||
|
*.db-journal
|
||||||
|
*.db-wal
|
||||||
|
*.db-shm
|
||||||
|
/data/
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
Alle nennenswerten Änderungen an NetMover stehen hier.
|
||||||
|
|
||||||
|
Format nach [Keep a Changelog](https://keepachangelog.com/de/1.1.0/),
|
||||||
|
Versionierung nach [Semantic Versioning](https://semver.org/lang/de/).
|
||||||
|
Die Version steht an genau einer Stelle: `src/netmover/__init__.py`.
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Hinzugefügt
|
||||||
|
- Projektgerüst (Python ≥ 3.12, FastAPI-Stack, src-Layout, ruff, mypy strict, pytest).
|
||||||
|
- Datenmodell mit SQLAlchemy 2.x: `credentials`, `mounts`, `rules`, `runs`,
|
||||||
|
`run_items`, `copied_files`, `file_failures`, `settings`.
|
||||||
|
- Alembic-Migration `0001` (initiales Schema); NetMover migriert beim Start selbst.
|
||||||
|
- Zeitzonenbewusste Zeitstempel (UTC) auch auf SQLite.
|
||||||
|
- Strukturierte Logs (logfmt oder JSON, ein Ereignis pro Zeile).
|
||||||
|
- CLI `netmover migrate` / `netmover version`.
|
||||||
|
|
||||||
|
### Behoben
|
||||||
|
- Löschen von Zugangsdaten, die noch von einem Mount benutzt werden, trennt den
|
||||||
|
Mount nicht mehr stillschweigend ab, sondern wird von der Datenbank abgelehnt.
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Devlog
|
||||||
|
|
||||||
|
Arbeitsprotokoll mit Zeiterfassung. Ein Eintrag pro Arbeitsblock, neueste oben.
|
||||||
|
Zeiten in Ortszeit (Europe/Berlin).
|
||||||
|
|
||||||
|
## Zeitübersicht
|
||||||
|
|
||||||
|
| Datum | Von | Bis | Dauer | Schritt | Thema |
|
||||||
|
|------------|-------|-------|-------|---------|----------------------------------------|
|
||||||
|
| 2026-10-10 | 17:21 | 17:27 | 0:06 | 1 | Gerüst, Datenmodell, Migrationen |
|
||||||
|
| **Summe** | | | **0:06** | | |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-10-10 · 17:21–17:27 · Schritt 1: Gerüst, Datenmodell, Migrationen
|
||||||
|
|
||||||
|
**Erledigt**
|
||||||
|
- Repo angelegt, venv (lokal Python 3.14; Code zielt auf ≥ 3.12, Debian 13 liefert 3.13).
|
||||||
|
`python3-venv` fehlt auf der Entwicklungsmaschine → venv mit `--without-pip` + `get-pip`.
|
||||||
|
- `pyproject.toml` (hatchling, src-Layout), ruff + mypy strict.
|
||||||
|
- Datenmodell nach Spezifikation plus Ergänzungen (siehe Docstring `models.py`):
|
||||||
|
`credentials.domain`, `mounts.state_since/write_ok/root_squash/last_test_*`,
|
||||||
|
`rules.op_timeout_seconds/quarantine_after`, `runs.status/trigger/dry_run`,
|
||||||
|
Tabelle `file_failures`, `copied_files.mtime_ns` statt Float-`mtime`.
|
||||||
|
- Alembic-Migration 0001; Migrationen liegen im Paket, damit sie mit installiert werden.
|
||||||
|
- 10 Tests: Schema = Modell (kein Drift), Down-/Upgrade, Defaults, UTC-Roundtrip,
|
||||||
|
RESTRICT bei benutzten Mounts/Zugangsdaten, CASCADE bei Regeln, Eindeutigkeit
|
||||||
|
in `copied_files`, CHECK-Constraints der Enums.
|
||||||
|
|
||||||
|
**Entscheidungen**
|
||||||
|
- Enums als `VARCHAR` + `CHECK` (kein natives Enum; SQLite kann das ohnehin nicht).
|
||||||
|
- `UTCDateTime` lehnt naive `datetime` ab, statt sie still als UTC zu interpretieren.
|
||||||
|
- SQLite mit WAL, `foreign_keys=ON`, `busy_timeout=5000` (Scheduler + Web + Worker-Threads).
|
||||||
|
|
||||||
|
**Stolpersteine**
|
||||||
|
- Alembic-Autogenerate erzeugt CHECK-Constraints bei `Enum(create_constraint=True)`
|
||||||
|
dreifach → explizite `CheckConstraint`-Zeilen aus der Migration entfernt.
|
||||||
|
- Test deckte auf: ORM setzt ohne `passive_deletes` `mounts.cred_id` auf NULL,
|
||||||
|
statt das `RESTRICT` greifen zu lassen. Behoben.
|
||||||
|
|
||||||
|
**Offen / nächster Schritt**
|
||||||
|
- Schritt 2: Mount-Schicht, Validierung, `FakeMountBackend`, Fernet-Verschlüsselung.
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# NetMover
|
||||||
|
|
||||||
|
Selbst gehosteter Dienst mit Web-Oberfläche, der SMB/CIFS- und NFS-Freigaben
|
||||||
|
verwaltet und Dateien zeitgesteuert zwischen ihnen verschiebt.
|
||||||
|
|
||||||
|
> Status: in Entwicklung (Schritt 1 von 6). Vollständige Doku folgt mit Schritt 6.
|
||||||
|
|
||||||
|
## Entwicklung
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv # oder: --without-pip + get-pip.py
|
||||||
|
.venv/bin/pip install -e '.[dev]'
|
||||||
|
.venv/bin/pytest
|
||||||
|
.venv/bin/ruff check . && .venv/bin/ruff format --check . && .venv/bin/mypy
|
||||||
|
NETMOVER_DATA_DIR=$PWD/data .venv/bin/netmover migrate
|
||||||
|
```
|
||||||
|
|
||||||
|
Neue Migration nach Modelländerung:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
NETMOVER_DATA_DIR=$PWD/data .venv/bin/alembic revision --autogenerate -m "…"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Arbeitsweise
|
||||||
|
|
||||||
|
- Commits nach [Conventional Commits](https://www.conventionalcommits.org/de/)
|
||||||
|
(`feat:`, `fix:`, `docs:`, `test:`, `chore:` …).
|
||||||
|
- Jede nutzerrelevante Änderung bekommt einen Eintrag unter `[Unreleased]` in
|
||||||
|
[CHANGELOG.md](CHANGELOG.md). Beim Release wird der Abschnitt versioniert,
|
||||||
|
die Version in `src/netmover/__init__.py` angehoben und ein Tag `vX.Y.Z` gesetzt.
|
||||||
|
- Arbeitszeit und Entscheidungen: [DEVLOG.md](DEVLOG.md).
|
||||||
+37
@@ -0,0 +1,37 @@
|
|||||||
|
# Nur für die Entwicklung (alembic revision --autogenerate).
|
||||||
|
# Im Betrieb migriert NetMover beim Start selbst (netmover.db.upgrade).
|
||||||
|
[alembic]
|
||||||
|
script_location = src/netmover/migrations
|
||||||
|
# sqlalchemy.url bleibt leer → env.py nimmt NETMOVER_DATABASE_URL / NETMOVER_DATA_DIR
|
||||||
|
|
||||||
|
[loggers]
|
||||||
|
keys = root,sqlalchemy,alembic
|
||||||
|
|
||||||
|
[handlers]
|
||||||
|
keys = console
|
||||||
|
|
||||||
|
[formatters]
|
||||||
|
keys = generic
|
||||||
|
|
||||||
|
[logger_root]
|
||||||
|
level = WARNING
|
||||||
|
handlers = console
|
||||||
|
|
||||||
|
[logger_sqlalchemy]
|
||||||
|
level = WARNING
|
||||||
|
handlers =
|
||||||
|
qualname = sqlalchemy.engine
|
||||||
|
|
||||||
|
[logger_alembic]
|
||||||
|
level = INFO
|
||||||
|
handlers =
|
||||||
|
qualname = alembic
|
||||||
|
|
||||||
|
[handler_console]
|
||||||
|
class = StreamHandler
|
||||||
|
args = (sys.stderr,)
|
||||||
|
level = NOTSET
|
||||||
|
formatter = generic
|
||||||
|
|
||||||
|
[formatter_generic]
|
||||||
|
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
[build-system]
|
||||||
|
requires = ["hatchling>=1.25"]
|
||||||
|
build-backend = "hatchling.build"
|
||||||
|
|
||||||
|
[project]
|
||||||
|
name = "netmover"
|
||||||
|
dynamic = ["version"]
|
||||||
|
description = "Web-GUI für NFS/SMB-Mounts und zeitgesteuertes Verschieben von Dateien"
|
||||||
|
readme = "README.md"
|
||||||
|
requires-python = ">=3.12"
|
||||||
|
license = { text = "MIT" }
|
||||||
|
authors = [{ name = "Florian Wilde", email = "fw@wildeflorian.de" }]
|
||||||
|
dependencies = [
|
||||||
|
"fastapi>=0.115",
|
||||||
|
"uvicorn[standard]>=0.30",
|
||||||
|
"sqlalchemy>=2.0.30",
|
||||||
|
"alembic>=1.13",
|
||||||
|
"jinja2>=3.1",
|
||||||
|
"python-multipart>=0.0.9",
|
||||||
|
"apscheduler>=3.10,<4",
|
||||||
|
"cryptography>=43",
|
||||||
|
"argon2-cffi>=23.1",
|
||||||
|
"pydantic>=2.7",
|
||||||
|
"pydantic-settings>=2.3",
|
||||||
|
"itsdangerous>=2.2",
|
||||||
|
"httpx>=0.27",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
dev = [
|
||||||
|
"pytest>=8.2",
|
||||||
|
"ruff>=0.6",
|
||||||
|
"mypy>=1.11",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.scripts]
|
||||||
|
netmover = "netmover.cli:main"
|
||||||
|
|
||||||
|
[tool.hatch.version]
|
||||||
|
path = "src/netmover/__init__.py"
|
||||||
|
|
||||||
|
[tool.hatch.build.targets.wheel]
|
||||||
|
packages = ["src/netmover"]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
testpaths = ["tests"]
|
||||||
|
addopts = "-ra"
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 100
|
||||||
|
target-version = "py312"
|
||||||
|
src = ["src", "tests"]
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
|
select = ["E", "F", "W", "I", "B", "UP", "S", "SIM", "RUF", "PTH"]
|
||||||
|
ignore = [
|
||||||
|
"S603", # subprocess ohne shell=True ist genau das gewünschte Muster
|
||||||
|
"RUF001", "RUF002", "RUF003", # deutsche Texte mit Typografie
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.ruff.lint.per-file-ignores]
|
||||||
|
"tests/**" = ["S101", "S105", "S106", "S108"]
|
||||||
|
"src/netmover/migrations/**" = ["E501"]
|
||||||
|
|
||||||
|
[tool.mypy]
|
||||||
|
python_version = "3.12"
|
||||||
|
strict = true
|
||||||
|
mypy_path = "src"
|
||||||
|
packages = ["netmover"]
|
||||||
|
plugins = ["pydantic.mypy"]
|
||||||
|
|
||||||
|
[[tool.mypy.overrides]]
|
||||||
|
module = ["apscheduler.*"]
|
||||||
|
ignore_missing_imports = true
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
"""NetMover – NFS/SMB-Mounts verwalten und Dateien zeitgesteuert verschieben."""
|
||||||
|
|
||||||
|
__version__ = "0.1.0.dev0"
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
"""Kommandozeile: ``netmover migrate`` / ``netmover version``.
|
||||||
|
|
||||||
|
``netmover serve`` folgt mit dem Web-Teil (Schritt 5).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
from netmover import __version__
|
||||||
|
from netmover.config import get_settings
|
||||||
|
from netmover.db import upgrade
|
||||||
|
from netmover.logging_setup import event, get_logger, setup_logging
|
||||||
|
|
||||||
|
log = get_logger("netmover.cli")
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: Sequence[str] | None = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(prog="netmover")
|
||||||
|
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||||
|
sub.add_parser("version", help="Version ausgeben")
|
||||||
|
sub.add_parser("migrate", help="Datenbank auf den neuesten Stand bringen")
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
setup_logging(settings.log_level, settings.log_format)
|
||||||
|
|
||||||
|
if args.cmd == "version":
|
||||||
|
print(__version__)
|
||||||
|
return 0
|
||||||
|
if args.cmd == "migrate":
|
||||||
|
settings.data_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
upgrade(settings.effective_database_url)
|
||||||
|
event(log, "db.migrated", url=settings.effective_database_url)
|
||||||
|
return 0
|
||||||
|
return 2
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
"""Start-Konfiguration aus Umgebungsvariablen (Präfix ``NETMOVER_``).
|
||||||
|
|
||||||
|
Hier steht nur, was vor dem Öffnen der Datenbank feststehen muss. Alles, was der
|
||||||
|
Nutzer zur Laufzeit ändert (Aufbewahrung, Webhook, …), liegt in der Tabelle
|
||||||
|
``settings``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from functools import lru_cache
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
from pydantic import Field
|
||||||
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||||
|
|
||||||
|
|
||||||
|
class Settings(BaseSettings):
|
||||||
|
model_config = SettingsConfigDict(env_prefix="NETMOVER_", env_file=None, extra="ignore")
|
||||||
|
|
||||||
|
data_dir: Path = Path("/var/lib/netmover")
|
||||||
|
run_dir: Path = Path("/run/netmover")
|
||||||
|
mount_base: Path = Path("/mnt/netmover")
|
||||||
|
|
||||||
|
host: str = "127.0.0.1"
|
||||||
|
port: int = Field(default=8080, ge=1, le=65535)
|
||||||
|
|
||||||
|
log_level: Literal["DEBUG", "INFO", "WARNING", "ERROR"] = "INFO"
|
||||||
|
log_format: Literal["logfmt", "json"] = "logfmt"
|
||||||
|
|
||||||
|
# "fake" nur für Tests und Entwicklung ohne root
|
||||||
|
mount_backend: Literal["linux", "fake"] = "linux"
|
||||||
|
|
||||||
|
database_url: str | None = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def db_path(self) -> Path:
|
||||||
|
return self.data_dir / "netmover.db"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def secret_key_path(self) -> Path:
|
||||||
|
return self.data_dir / "secret.key"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def effective_database_url(self) -> str:
|
||||||
|
return self.database_url or f"sqlite:///{self.db_path}"
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def get_settings() -> Settings:
|
||||||
|
return Settings()
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
"""Datenbank-Engine, Sessions und Migrationen."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from importlib.resources import files
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from alembic import command
|
||||||
|
from alembic.config import Config
|
||||||
|
from sqlalchemy import Engine, create_engine, event
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
|
||||||
|
def _sqlite_pragmas(dbapi_conn: Any, _record: Any) -> None:
|
||||||
|
cur = dbapi_conn.cursor()
|
||||||
|
cur.execute("PRAGMA foreign_keys=ON")
|
||||||
|
cur.execute("PRAGMA journal_mode=WAL")
|
||||||
|
cur.execute("PRAGMA synchronous=NORMAL")
|
||||||
|
cur.execute("PRAGMA busy_timeout=5000")
|
||||||
|
cur.close()
|
||||||
|
|
||||||
|
|
||||||
|
def make_engine(url: str) -> Engine:
|
||||||
|
connect_args: dict[str, Any] = {}
|
||||||
|
if url.startswith("sqlite"):
|
||||||
|
# Worker-Threads der Regelläufe greifen ebenfalls zu.
|
||||||
|
connect_args["check_same_thread"] = False
|
||||||
|
engine = create_engine(url, connect_args=connect_args)
|
||||||
|
if url.startswith("sqlite"):
|
||||||
|
event.listen(engine, "connect", _sqlite_pragmas)
|
||||||
|
return engine
|
||||||
|
|
||||||
|
|
||||||
|
def make_session_factory(engine: Engine) -> sessionmaker[Session]:
|
||||||
|
return sessionmaker(bind=engine, expire_on_commit=False)
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def session_scope(factory: sessionmaker[Session]) -> Iterator[Session]:
|
||||||
|
session = factory()
|
||||||
|
try:
|
||||||
|
yield session
|
||||||
|
session.commit()
|
||||||
|
except Exception:
|
||||||
|
session.rollback()
|
||||||
|
raise
|
||||||
|
finally:
|
||||||
|
session.close()
|
||||||
|
|
||||||
|
|
||||||
|
def alembic_config(url: str) -> Config:
|
||||||
|
cfg = Config()
|
||||||
|
cfg.set_main_option("script_location", str(files("netmover") / "migrations"))
|
||||||
|
cfg.set_main_option("sqlalchemy.url", url)
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade(url: str, revision: str = "head") -> None:
|
||||||
|
command.upgrade(alembic_config(url), revision)
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
"""Strukturierte Logs: ein Ereignis pro Zeile, wahlweise logfmt oder JSON.
|
||||||
|
|
||||||
|
Benutzung::
|
||||||
|
|
||||||
|
log = get_logger(__name__)
|
||||||
|
log.info("mount.ok", extra={"fields": {"mount": "scans", "took_ms": 120}})
|
||||||
|
|
||||||
|
oder kürzer über :func:`event`::
|
||||||
|
|
||||||
|
event(log, "mount.ok", mount="scans", took_ms=120)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import sys
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
_RESERVED = {"ts", "level", "logger", "event"}
|
||||||
|
|
||||||
|
|
||||||
|
def _logfmt_value(value: Any) -> str:
|
||||||
|
text = str(value)
|
||||||
|
if text == "" or any(c in text for c in ' ="\n\t'):
|
||||||
|
text = '"' + text.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + '"'
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
class StructuredFormatter(logging.Formatter):
|
||||||
|
def __init__(self, fmt: str = "logfmt") -> None:
|
||||||
|
super().__init__()
|
||||||
|
self.fmt = fmt
|
||||||
|
|
||||||
|
def format(self, record: logging.LogRecord) -> str:
|
||||||
|
data: dict[str, Any] = {
|
||||||
|
"ts": datetime.fromtimestamp(record.created, tz=UTC).isoformat(timespec="milliseconds"),
|
||||||
|
"level": record.levelname.lower(),
|
||||||
|
"logger": record.name,
|
||||||
|
"event": record.getMessage(),
|
||||||
|
}
|
||||||
|
fields = getattr(record, "fields", None)
|
||||||
|
if isinstance(fields, dict):
|
||||||
|
for key, value in fields.items():
|
||||||
|
data[f"f_{key}" if key in _RESERVED else key] = value
|
||||||
|
if record.exc_info:
|
||||||
|
data["exc"] = self.formatException(record.exc_info)
|
||||||
|
if self.fmt == "json":
|
||||||
|
return json.dumps(data, default=str, ensure_ascii=False)
|
||||||
|
return " ".join(f"{k}={_logfmt_value(v)}" for k, v in data.items())
|
||||||
|
|
||||||
|
|
||||||
|
def setup_logging(level: str = "INFO", fmt: str = "logfmt") -> None:
|
||||||
|
handler = logging.StreamHandler(sys.stderr)
|
||||||
|
handler.setFormatter(StructuredFormatter(fmt))
|
||||||
|
root = logging.getLogger()
|
||||||
|
root.handlers[:] = [handler]
|
||||||
|
root.setLevel(level)
|
||||||
|
# Uvicorn bringt eigene Handler mit; wir wollen ein einheitliches Format.
|
||||||
|
for name in ("uvicorn", "uvicorn.error", "uvicorn.access", "apscheduler"):
|
||||||
|
lg = logging.getLogger(name)
|
||||||
|
lg.handlers[:] = []
|
||||||
|
lg.propagate = True
|
||||||
|
|
||||||
|
|
||||||
|
def get_logger(name: str) -> logging.Logger:
|
||||||
|
return logging.getLogger(name)
|
||||||
|
|
||||||
|
|
||||||
|
def event(logger: logging.Logger, name: str, level: int = logging.INFO, **fields: Any) -> None:
|
||||||
|
logger.log(level, name, extra={"fields": fields})
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
"""Alembic-Umgebung für NetMover."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from logging.config import fileConfig
|
||||||
|
|
||||||
|
from alembic import context
|
||||||
|
|
||||||
|
from netmover.config import get_settings
|
||||||
|
from netmover.db import make_engine
|
||||||
|
from netmover.models import Base
|
||||||
|
|
||||||
|
config = context.config
|
||||||
|
if config.config_file_name is not None:
|
||||||
|
fileConfig(config.config_file_name, disable_existing_loggers=False)
|
||||||
|
|
||||||
|
target_metadata = Base.metadata
|
||||||
|
|
||||||
|
|
||||||
|
def _url() -> str:
|
||||||
|
return config.get_main_option("sqlalchemy.url") or get_settings().effective_database_url
|
||||||
|
|
||||||
|
|
||||||
|
def run_migrations_offline() -> None:
|
||||||
|
context.configure(
|
||||||
|
url=_url(),
|
||||||
|
target_metadata=target_metadata,
|
||||||
|
literal_binds=True,
|
||||||
|
render_as_batch=True,
|
||||||
|
)
|
||||||
|
with context.begin_transaction():
|
||||||
|
context.run_migrations()
|
||||||
|
|
||||||
|
|
||||||
|
def run_migrations_online() -> None:
|
||||||
|
engine = make_engine(_url())
|
||||||
|
with engine.connect() as connection:
|
||||||
|
context.configure(
|
||||||
|
connection=connection,
|
||||||
|
target_metadata=target_metadata,
|
||||||
|
render_as_batch=True,
|
||||||
|
compare_type=True,
|
||||||
|
)
|
||||||
|
with context.begin_transaction():
|
||||||
|
context.run_migrations()
|
||||||
|
engine.dispose()
|
||||||
|
|
||||||
|
|
||||||
|
if context.is_offline_mode():
|
||||||
|
run_migrations_offline()
|
||||||
|
else:
|
||||||
|
run_migrations_online()
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
"""${message}
|
||||||
|
|
||||||
|
Revision ID: ${up_revision}
|
||||||
|
Revises: ${down_revision | comma,n}
|
||||||
|
Create Date: ${create_date}
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
import netmover.models
|
||||||
|
${imports if imports else ""}
|
||||||
|
|
||||||
|
revision: str = ${repr(up_revision)}
|
||||||
|
down_revision: str | None = ${repr(down_revision)}
|
||||||
|
branch_labels: str | Sequence[str] | None = ${repr(branch_labels)}
|
||||||
|
depends_on: str | Sequence[str] | None = ${repr(depends_on)}
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
${upgrades if upgrades else "pass"}
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
${downgrades if downgrades else "pass"}
|
||||||
@@ -0,0 +1,318 @@
|
|||||||
|
"""initial schema
|
||||||
|
|
||||||
|
Revision ID: 0001
|
||||||
|
Revises:
|
||||||
|
Create Date: 2026-10-10
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
import netmover.models
|
||||||
|
|
||||||
|
revision: str = "0001"
|
||||||
|
down_revision: str | None = None
|
||||||
|
branch_labels: str | Sequence[str] | None = None
|
||||||
|
depends_on: str | Sequence[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table(
|
||||||
|
"credentials",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("name", sa.String(length=100), nullable=False),
|
||||||
|
sa.Column("username", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("domain", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("secret_enc", sa.Text(), nullable=False),
|
||||||
|
sa.Column("created_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_credentials")),
|
||||||
|
sa.UniqueConstraint("name", name=op.f("uq_credentials_name")),
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"settings",
|
||||||
|
sa.Column("key", sa.String(length=100), nullable=False),
|
||||||
|
sa.Column("value", sa.Text(), nullable=False),
|
||||||
|
sa.PrimaryKeyConstraint("key", name=op.f("pk_settings")),
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"mounts",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("name", sa.String(length=100), nullable=False),
|
||||||
|
sa.Column("slug", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column(
|
||||||
|
"type",
|
||||||
|
sa.Enum(
|
||||||
|
"cifs",
|
||||||
|
"nfs4",
|
||||||
|
"nfs3",
|
||||||
|
name="mount_type",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("server", sa.String(length=255), nullable=False),
|
||||||
|
sa.Column("share", sa.String(length=1024), nullable=False),
|
||||||
|
sa.Column("mountpoint", sa.String(length=1024), nullable=False),
|
||||||
|
sa.Column("cred_id", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("uid", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("gid", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("file_mode", sa.String(length=5), nullable=True),
|
||||||
|
sa.Column("dir_mode", sa.String(length=5), nullable=True),
|
||||||
|
sa.Column("extra_options", sa.Text(), nullable=False),
|
||||||
|
sa.Column("auto_mount", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("enabled", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column(
|
||||||
|
"state",
|
||||||
|
sa.Enum(
|
||||||
|
"unmounted",
|
||||||
|
"mounted",
|
||||||
|
"degraded",
|
||||||
|
"error",
|
||||||
|
name="mount_state",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("state_since", netmover.models.UTCDateTime(), nullable=True),
|
||||||
|
sa.Column("last_error", sa.Text(), nullable=True),
|
||||||
|
sa.Column("last_mounted_at", netmover.models.UTCDateTime(), nullable=True),
|
||||||
|
sa.Column("last_test_at", netmover.models.UTCDateTime(), nullable=True),
|
||||||
|
sa.Column("last_test_result", sa.Text(), nullable=True),
|
||||||
|
sa.Column("write_ok", sa.Boolean(), nullable=True),
|
||||||
|
sa.Column("root_squash", sa.Boolean(), nullable=True),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["cred_id"],
|
||||||
|
["credentials.id"],
|
||||||
|
name=op.f("fk_mounts_cred_id_credentials"),
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_mounts")),
|
||||||
|
sa.UniqueConstraint("mountpoint", name=op.f("uq_mounts_mountpoint")),
|
||||||
|
sa.UniqueConstraint("name", name=op.f("uq_mounts_name")),
|
||||||
|
sa.UniqueConstraint("slug", name=op.f("uq_mounts_slug")),
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"rules",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("name", sa.String(length=100), nullable=False),
|
||||||
|
sa.Column("enabled", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("src_mount_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("src_subpath", sa.String(length=1024), nullable=False),
|
||||||
|
sa.Column("dst_mount_id", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("dst_path", sa.String(length=1024), nullable=False),
|
||||||
|
sa.Column("pattern", sa.String(length=512), nullable=False),
|
||||||
|
sa.Column(
|
||||||
|
"pattern_type",
|
||||||
|
sa.Enum(
|
||||||
|
"glob",
|
||||||
|
"regex",
|
||||||
|
name="pattern_type",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("recursive", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("interval_seconds", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("stability_seconds", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("min_age_seconds", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("max_size_mb", sa.Integer(), nullable=True),
|
||||||
|
sa.Column("op_timeout_seconds", sa.Integer(), nullable=False),
|
||||||
|
sa.Column(
|
||||||
|
"action",
|
||||||
|
sa.Enum(
|
||||||
|
"move",
|
||||||
|
"copy",
|
||||||
|
name="rule_action",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column(
|
||||||
|
"on_collision",
|
||||||
|
sa.Enum(
|
||||||
|
"rename",
|
||||||
|
"skip",
|
||||||
|
"overwrite",
|
||||||
|
name="collision_policy",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("set_owner", sa.String(length=64), nullable=True),
|
||||||
|
sa.Column("set_mode", sa.String(length=5), nullable=True),
|
||||||
|
sa.Column("dry_run", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("quarantine_path", sa.String(length=1024), nullable=True),
|
||||||
|
sa.Column("quarantine_after", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("last_run_at", netmover.models.UTCDateTime(), nullable=True),
|
||||||
|
sa.Column("last_success_at", netmover.models.UTCDateTime(), nullable=True),
|
||||||
|
sa.Column("consecutive_failures", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("created_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["dst_mount_id"],
|
||||||
|
["mounts.id"],
|
||||||
|
name=op.f("fk_rules_dst_mount_id_mounts"),
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["src_mount_id"],
|
||||||
|
["mounts.id"],
|
||||||
|
name=op.f("fk_rules_src_mount_id_mounts"),
|
||||||
|
ondelete="RESTRICT",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_rules")),
|
||||||
|
sa.UniqueConstraint("name", name=op.f("uq_rules_name")),
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"copied_files",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("rule_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("src_relpath", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("size", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("mtime_ns", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("dst_path", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("copied_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["rule_id"],
|
||||||
|
["rules.id"],
|
||||||
|
name=op.f("fk_copied_files_rule_id_rules"),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_copied_files")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"rule_id",
|
||||||
|
"src_relpath",
|
||||||
|
"size",
|
||||||
|
"mtime_ns",
|
||||||
|
name=op.f("uq_copied_files_rule_id_src_relpath_size_mtime_ns"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"file_failures",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("rule_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("src_relpath", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("size", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("mtime_ns", sa.BigInteger(), nullable=False),
|
||||||
|
sa.Column("attempts", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("last_error", sa.Text(), nullable=True),
|
||||||
|
sa.Column("last_attempt_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["rule_id"],
|
||||||
|
["rules.id"],
|
||||||
|
name=op.f("fk_file_failures_rule_id_rules"),
|
||||||
|
ondelete="CASCADE",
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_file_failures")),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"rule_id", "src_relpath", name=op.f("uq_file_failures_rule_id_src_relpath")
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"runs",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("rule_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("started_at", netmover.models.UTCDateTime(), nullable=False),
|
||||||
|
sa.Column("finished_at", netmover.models.UTCDateTime(), nullable=True),
|
||||||
|
sa.Column(
|
||||||
|
"status",
|
||||||
|
sa.Enum(
|
||||||
|
"running",
|
||||||
|
"ok",
|
||||||
|
"partial",
|
||||||
|
"error",
|
||||||
|
"skipped",
|
||||||
|
name="run_status",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column(
|
||||||
|
"trigger",
|
||||||
|
sa.Enum(
|
||||||
|
"schedule",
|
||||||
|
"manual",
|
||||||
|
name="run_trigger",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("dry_run", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column("seen", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("moved", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("skipped", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("failed", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("message", sa.Text(), nullable=True),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["rule_id"], ["rules.id"], name=op.f("fk_runs_rule_id_rules"), ondelete="CASCADE"
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_runs")),
|
||||||
|
)
|
||||||
|
with op.batch_alter_table("runs", schema=None) as batch_op:
|
||||||
|
batch_op.create_index("ix_runs_rule_started", ["rule_id", "started_at"], unique=False)
|
||||||
|
batch_op.create_index(batch_op.f("ix_runs_started_at"), ["started_at"], unique=False)
|
||||||
|
|
||||||
|
op.create_table(
|
||||||
|
"run_items",
|
||||||
|
sa.Column("id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("run_id", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("src", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("dst", sa.String(length=2048), nullable=True),
|
||||||
|
sa.Column(
|
||||||
|
"status",
|
||||||
|
sa.Enum(
|
||||||
|
"moved",
|
||||||
|
"copied",
|
||||||
|
"skipped",
|
||||||
|
"failed",
|
||||||
|
"quarantined",
|
||||||
|
"planned",
|
||||||
|
name="item_status",
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
length=24,
|
||||||
|
),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("size", sa.BigInteger(), nullable=True),
|
||||||
|
sa.Column("message", sa.Text(), nullable=True),
|
||||||
|
sa.ForeignKeyConstraint(
|
||||||
|
["run_id"], ["runs.id"], name=op.f("fk_run_items_run_id_runs"), ondelete="CASCADE"
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id", name=op.f("pk_run_items")),
|
||||||
|
)
|
||||||
|
with op.batch_alter_table("run_items", schema=None) as batch_op:
|
||||||
|
batch_op.create_index(batch_op.f("ix_run_items_run_id"), ["run_id"], unique=False)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
with op.batch_alter_table("run_items", schema=None) as batch_op:
|
||||||
|
batch_op.drop_index(batch_op.f("ix_run_items_run_id"))
|
||||||
|
|
||||||
|
op.drop_table("run_items")
|
||||||
|
with op.batch_alter_table("runs", schema=None) as batch_op:
|
||||||
|
batch_op.drop_index(batch_op.f("ix_runs_started_at"))
|
||||||
|
batch_op.drop_index("ix_runs_rule_started")
|
||||||
|
|
||||||
|
op.drop_table("runs")
|
||||||
|
op.drop_table("file_failures")
|
||||||
|
op.drop_table("copied_files")
|
||||||
|
op.drop_table("rules")
|
||||||
|
op.drop_table("mounts")
|
||||||
|
op.drop_table("settings")
|
||||||
|
op.drop_table("credentials")
|
||||||
@@ -0,0 +1,318 @@
|
|||||||
|
"""SQLAlchemy-Datenmodell.
|
||||||
|
|
||||||
|
Abweichungen bzw. Ergänzungen gegenüber dem Mindestumfang aus der Spezifikation:
|
||||||
|
|
||||||
|
* ``credentials.domain`` – Windows-/AD-Freigaben brauchen oft eine Domäne.
|
||||||
|
* ``mounts.state_since`` – für „hängt seit …“ / „gestört seit …“ auf der Statusseite.
|
||||||
|
* ``mounts.write_ok`` / ``mounts.root_squash`` / ``mounts.last_test_*`` – Ergebnis
|
||||||
|
des letzten Mount-Tests; ``root_squash`` entscheidet, ob ``chown`` auf NFS-Zielen
|
||||||
|
versucht wird (Fallstrick 12).
|
||||||
|
* ``rules.op_timeout_seconds`` – Zeitlimit des Worker-Laufs (Fallstrick 13).
|
||||||
|
* ``rules.quarantine_after`` + Tabelle ``file_failures`` – Zählung wiederholt
|
||||||
|
scheiternder Dateien für den Quarantäneordner.
|
||||||
|
* ``runs.status`` / ``runs.trigger`` / ``runs.dry_run`` – Filter im Protokoll.
|
||||||
|
* ``copied_files.mtime_ns`` – ``mtime`` als ganzzahlige Nanosekunden statt Float,
|
||||||
|
damit der Gleichheitsvergleich exakt ist (Fallstrick 11).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import enum
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
from sqlalchemy import (
|
||||||
|
BigInteger,
|
||||||
|
Boolean,
|
||||||
|
DateTime,
|
||||||
|
Enum,
|
||||||
|
ForeignKey,
|
||||||
|
Index,
|
||||||
|
Integer,
|
||||||
|
MetaData,
|
||||||
|
String,
|
||||||
|
Text,
|
||||||
|
TypeDecorator,
|
||||||
|
UniqueConstraint,
|
||||||
|
)
|
||||||
|
from sqlalchemy.engine import Dialect
|
||||||
|
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
|
||||||
|
|
||||||
|
|
||||||
|
def utcnow() -> datetime:
|
||||||
|
return datetime.now(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
class UTCDateTime(TypeDecorator[datetime]):
|
||||||
|
"""Speichert naive UTC-Zeitstempel (SQLite kennt keine Zeitzonen) und liefert
|
||||||
|
immer zeitzonenbewusste ``datetime``-Objekte zurück."""
|
||||||
|
|
||||||
|
impl = DateTime
|
||||||
|
cache_ok = True
|
||||||
|
|
||||||
|
def process_bind_param(self, value: datetime | None, dialect: Dialect) -> datetime | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
if value.tzinfo is None:
|
||||||
|
raise ValueError("naive datetime – bitte zeitzonenbewusst (UTC) übergeben")
|
||||||
|
return value.astimezone(UTC).replace(tzinfo=None)
|
||||||
|
|
||||||
|
def process_result_value(self, value: datetime | None, dialect: Dialect) -> datetime | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
return value.replace(tzinfo=UTC)
|
||||||
|
|
||||||
|
|
||||||
|
# Einheitliche Constraint-Namen, damit Alembic-Batch-Migrationen auf SQLite
|
||||||
|
# Constraints wiederfinden.
|
||||||
|
NAMING_CONVENTION = {
|
||||||
|
"ix": "ix_%(column_0_label)s",
|
||||||
|
"uq": "uq_%(table_name)s_%(column_0_N_name)s",
|
||||||
|
"ck": "ck_%(table_name)s_%(constraint_name)s",
|
||||||
|
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
|
||||||
|
"pk": "pk_%(table_name)s",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class Base(DeclarativeBase):
|
||||||
|
metadata = MetaData(naming_convention=NAMING_CONVENTION)
|
||||||
|
|
||||||
|
|
||||||
|
def _enum(cls: type[enum.Enum], name: str) -> Enum:
|
||||||
|
return Enum(
|
||||||
|
cls,
|
||||||
|
name=name,
|
||||||
|
native_enum=False,
|
||||||
|
create_constraint=True,
|
||||||
|
validate_strings=True,
|
||||||
|
length=24,
|
||||||
|
values_callable=lambda e: [m.value for m in e],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class MountType(enum.StrEnum):
|
||||||
|
CIFS = "cifs"
|
||||||
|
NFS4 = "nfs4"
|
||||||
|
NFS3 = "nfs3"
|
||||||
|
|
||||||
|
|
||||||
|
class MountState(enum.StrEnum):
|
||||||
|
UNMOUNTED = "unmounted"
|
||||||
|
MOUNTED = "mounted"
|
||||||
|
DEGRADED = "degraded" # gestört: Wiederherstellung läuft
|
||||||
|
ERROR = "error" # Wiederherstellung gescheitert
|
||||||
|
|
||||||
|
|
||||||
|
class PatternType(enum.StrEnum):
|
||||||
|
GLOB = "glob"
|
||||||
|
REGEX = "regex"
|
||||||
|
|
||||||
|
|
||||||
|
class RuleAction(enum.StrEnum):
|
||||||
|
MOVE = "move"
|
||||||
|
COPY = "copy"
|
||||||
|
|
||||||
|
|
||||||
|
class CollisionPolicy(enum.StrEnum):
|
||||||
|
RENAME = "rename"
|
||||||
|
SKIP = "skip"
|
||||||
|
OVERWRITE = "overwrite"
|
||||||
|
|
||||||
|
|
||||||
|
class RunStatus(enum.StrEnum):
|
||||||
|
RUNNING = "running"
|
||||||
|
OK = "ok" # keine Dateifehler
|
||||||
|
PARTIAL = "partial" # einzelne Dateien gescheitert
|
||||||
|
ERROR = "error" # Lauf abgebrochen (Mount fehlt, Zeitlimit, …)
|
||||||
|
SKIPPED = "skipped" # Termin übersprungen, voriger Lauf noch aktiv
|
||||||
|
|
||||||
|
|
||||||
|
class RunTrigger(enum.StrEnum):
|
||||||
|
SCHEDULE = "schedule"
|
||||||
|
MANUAL = "manual"
|
||||||
|
|
||||||
|
|
||||||
|
class ItemStatus(enum.StrEnum):
|
||||||
|
MOVED = "moved"
|
||||||
|
COPIED = "copied"
|
||||||
|
SKIPPED = "skipped"
|
||||||
|
FAILED = "failed"
|
||||||
|
QUARANTINED = "quarantined"
|
||||||
|
PLANNED = "planned" # Probelauf
|
||||||
|
|
||||||
|
|
||||||
|
class Credential(Base):
|
||||||
|
__tablename__ = "credentials"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||||||
|
username: Mapped[str] = mapped_column(String(255))
|
||||||
|
domain: Mapped[str | None] = mapped_column(String(255))
|
||||||
|
secret_enc: Mapped[str] = mapped_column(Text)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
|
# passive_deletes="all": das ORM soll cred_id nicht still auf NULL setzen,
|
||||||
|
# sondern das RESTRICT der Datenbank greifen lassen.
|
||||||
|
mounts: Mapped[list[Mount]] = relationship(back_populates="credential", passive_deletes="all")
|
||||||
|
|
||||||
|
|
||||||
|
class Mount(Base):
|
||||||
|
__tablename__ = "mounts"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||||||
|
slug: Mapped[str] = mapped_column(String(64), unique=True)
|
||||||
|
type: Mapped[MountType] = mapped_column(_enum(MountType, "mount_type"))
|
||||||
|
server: Mapped[str] = mapped_column(String(255))
|
||||||
|
share: Mapped[str] = mapped_column(String(1024))
|
||||||
|
mountpoint: Mapped[str] = mapped_column(String(1024), unique=True)
|
||||||
|
cred_id: Mapped[int | None] = mapped_column(ForeignKey("credentials.id", ondelete="RESTRICT"))
|
||||||
|
uid: Mapped[int | None] = mapped_column(Integer)
|
||||||
|
gid: Mapped[int | None] = mapped_column(Integer)
|
||||||
|
file_mode: Mapped[str | None] = mapped_column(String(5))
|
||||||
|
dir_mode: Mapped[str | None] = mapped_column(String(5))
|
||||||
|
extra_options: Mapped[str] = mapped_column(Text, default="")
|
||||||
|
auto_mount: Mapped[bool] = mapped_column(Boolean, default=True)
|
||||||
|
enabled: Mapped[bool] = mapped_column(Boolean, default=True)
|
||||||
|
|
||||||
|
state: Mapped[MountState] = mapped_column(
|
||||||
|
_enum(MountState, "mount_state"), default=MountState.UNMOUNTED
|
||||||
|
)
|
||||||
|
state_since: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
|
last_error: Mapped[str | None] = mapped_column(Text)
|
||||||
|
last_mounted_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
|
|
||||||
|
last_test_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
|
last_test_result: Mapped[str | None] = mapped_column(Text) # JSON
|
||||||
|
write_ok: Mapped[bool | None] = mapped_column(Boolean)
|
||||||
|
root_squash: Mapped[bool | None] = mapped_column(Boolean)
|
||||||
|
|
||||||
|
credential: Mapped[Credential | None] = relationship(back_populates="mounts")
|
||||||
|
|
||||||
|
|
||||||
|
class Rule(Base):
|
||||||
|
__tablename__ = "rules"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
name: Mapped[str] = mapped_column(String(100), unique=True)
|
||||||
|
enabled: Mapped[bool] = mapped_column(Boolean, default=True)
|
||||||
|
|
||||||
|
src_mount_id: Mapped[int] = mapped_column(ForeignKey("mounts.id", ondelete="RESTRICT"))
|
||||||
|
src_subpath: Mapped[str] = mapped_column(String(1024), default="")
|
||||||
|
# dst_mount_id NULL → dst_path ist ein absoluter lokaler Pfad
|
||||||
|
dst_mount_id: Mapped[int | None] = mapped_column(ForeignKey("mounts.id", ondelete="RESTRICT"))
|
||||||
|
dst_path: Mapped[str] = mapped_column(String(1024), default="")
|
||||||
|
|
||||||
|
pattern: Mapped[str] = mapped_column(String(512), default="*")
|
||||||
|
pattern_type: Mapped[PatternType] = mapped_column(
|
||||||
|
_enum(PatternType, "pattern_type"), default=PatternType.GLOB
|
||||||
|
)
|
||||||
|
recursive: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
|
||||||
|
interval_seconds: Mapped[int] = mapped_column(Integer, default=30)
|
||||||
|
stability_seconds: Mapped[int] = mapped_column(Integer, default=5)
|
||||||
|
min_age_seconds: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
max_size_mb: Mapped[int | None] = mapped_column(Integer)
|
||||||
|
op_timeout_seconds: Mapped[int] = mapped_column(Integer, default=120)
|
||||||
|
|
||||||
|
action: Mapped[RuleAction] = mapped_column(
|
||||||
|
_enum(RuleAction, "rule_action"), default=RuleAction.MOVE
|
||||||
|
)
|
||||||
|
on_collision: Mapped[CollisionPolicy] = mapped_column(
|
||||||
|
_enum(CollisionPolicy, "collision_policy"), default=CollisionPolicy.RENAME
|
||||||
|
)
|
||||||
|
set_owner: Mapped[str | None] = mapped_column(String(64)) # "uid:gid", NULL = unverändert
|
||||||
|
set_mode: Mapped[str | None] = mapped_column(String(5)) # "0644", NULL = unverändert
|
||||||
|
dry_run: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
|
||||||
|
quarantine_path: Mapped[str | None] = mapped_column(String(1024))
|
||||||
|
quarantine_after: Mapped[int] = mapped_column(Integer, default=3)
|
||||||
|
|
||||||
|
last_run_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
|
last_success_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
|
consecutive_failures: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
|
src_mount: Mapped[Mount] = relationship(foreign_keys=[src_mount_id])
|
||||||
|
dst_mount: Mapped[Mount | None] = relationship(foreign_keys=[dst_mount_id])
|
||||||
|
runs: Mapped[list[Run]] = relationship(
|
||||||
|
back_populates="rule", cascade="all, delete-orphan", passive_deletes=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class Run(Base):
|
||||||
|
__tablename__ = "runs"
|
||||||
|
__table_args__ = (Index("ix_runs_rule_started", "rule_id", "started_at"),)
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
rule_id: Mapped[int] = mapped_column(ForeignKey("rules.id", ondelete="CASCADE"))
|
||||||
|
started_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow, index=True)
|
||||||
|
finished_at: Mapped[datetime | None] = mapped_column(UTCDateTime)
|
||||||
|
status: Mapped[RunStatus] = mapped_column(
|
||||||
|
_enum(RunStatus, "run_status"), default=RunStatus.RUNNING
|
||||||
|
)
|
||||||
|
trigger: Mapped[RunTrigger] = mapped_column(
|
||||||
|
_enum(RunTrigger, "run_trigger"), default=RunTrigger.SCHEDULE
|
||||||
|
)
|
||||||
|
dry_run: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
seen: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
moved: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
skipped: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
failed: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
message: Mapped[str | None] = mapped_column(Text)
|
||||||
|
|
||||||
|
rule: Mapped[Rule] = relationship(back_populates="runs")
|
||||||
|
items: Mapped[list[RunItem]] = relationship(
|
||||||
|
back_populates="run", cascade="all, delete-orphan", passive_deletes=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class RunItem(Base):
|
||||||
|
__tablename__ = "run_items"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
run_id: Mapped[int] = mapped_column(ForeignKey("runs.id", ondelete="CASCADE"), index=True)
|
||||||
|
src: Mapped[str] = mapped_column(String(2048))
|
||||||
|
dst: Mapped[str | None] = mapped_column(String(2048))
|
||||||
|
status: Mapped[ItemStatus] = mapped_column(_enum(ItemStatus, "item_status"))
|
||||||
|
size: Mapped[int | None] = mapped_column(BigInteger)
|
||||||
|
message: Mapped[str | None] = mapped_column(Text)
|
||||||
|
|
||||||
|
run: Mapped[Run] = relationship(back_populates="items")
|
||||||
|
|
||||||
|
|
||||||
|
class CopiedFile(Base):
|
||||||
|
"""Buchführung für Aktion ``copy`` (Fallstrick 11)."""
|
||||||
|
|
||||||
|
__tablename__ = "copied_files"
|
||||||
|
__table_args__ = (UniqueConstraint("rule_id", "src_relpath", "size", "mtime_ns"),)
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
rule_id: Mapped[int] = mapped_column(ForeignKey("rules.id", ondelete="CASCADE"))
|
||||||
|
src_relpath: Mapped[str] = mapped_column(String(2048))
|
||||||
|
size: Mapped[int] = mapped_column(BigInteger)
|
||||||
|
mtime_ns: Mapped[int] = mapped_column(BigInteger)
|
||||||
|
dst_path: Mapped[str] = mapped_column(String(2048))
|
||||||
|
copied_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
|
|
||||||
|
class FileFailure(Base):
|
||||||
|
"""Fehlversuche pro Datei, Grundlage für die Quarantäne."""
|
||||||
|
|
||||||
|
__tablename__ = "file_failures"
|
||||||
|
__table_args__ = (UniqueConstraint("rule_id", "src_relpath"),)
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
rule_id: Mapped[int] = mapped_column(ForeignKey("rules.id", ondelete="CASCADE"))
|
||||||
|
src_relpath: Mapped[str] = mapped_column(String(2048))
|
||||||
|
size: Mapped[int] = mapped_column(BigInteger)
|
||||||
|
mtime_ns: Mapped[int] = mapped_column(BigInteger)
|
||||||
|
attempts: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
last_error: Mapped[str | None] = mapped_column(Text)
|
||||||
|
last_attempt_at: Mapped[datetime] = mapped_column(UTCDateTime, default=utcnow)
|
||||||
|
|
||||||
|
|
||||||
|
class Setting(Base):
|
||||||
|
__tablename__ = "settings"
|
||||||
|
|
||||||
|
key: Mapped[str] = mapped_column(String(100), primary_key=True)
|
||||||
|
value: Mapped[str] = mapped_column(Text)
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from netmover.db import make_engine, make_session_factory, upgrade
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def db_url(tmp_path: Path) -> str:
|
||||||
|
url = f"sqlite:///{tmp_path / 'test.db'}"
|
||||||
|
upgrade(url)
|
||||||
|
return url
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def session(db_url: str) -> Iterator[Session]:
|
||||||
|
engine = make_engine(db_url)
|
||||||
|
factory = make_session_factory(engine)
|
||||||
|
with factory() as s:
|
||||||
|
yield s
|
||||||
|
engine.dispose()
|
||||||
@@ -0,0 +1,167 @@
|
|||||||
|
"""Schritt 1: Schema, Migrationen und Integritätsregeln."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import UTC, datetime, timedelta, timezone
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from alembic.autogenerate import compare_metadata
|
||||||
|
from alembic.migration import MigrationContext
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.exc import IntegrityError, StatementError
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from netmover.db import make_engine
|
||||||
|
from netmover.models import (
|
||||||
|
Base,
|
||||||
|
CopiedFile,
|
||||||
|
Credential,
|
||||||
|
ItemStatus,
|
||||||
|
Mount,
|
||||||
|
MountState,
|
||||||
|
MountType,
|
||||||
|
Rule,
|
||||||
|
Run,
|
||||||
|
RunItem,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _mount(name: str = "scans", **kw: object) -> Mount:
|
||||||
|
defaults: dict[str, object] = {
|
||||||
|
"name": name,
|
||||||
|
"slug": name,
|
||||||
|
"type": MountType.CIFS,
|
||||||
|
"server": "192.168.1.10",
|
||||||
|
"share": "scans",
|
||||||
|
"mountpoint": f"/mnt/netmover/{name}",
|
||||||
|
}
|
||||||
|
defaults.update(kw)
|
||||||
|
return Mount(**defaults)
|
||||||
|
|
||||||
|
|
||||||
|
def test_migration_matches_models(db_url: str) -> None:
|
||||||
|
engine = make_engine(db_url)
|
||||||
|
with engine.connect() as conn:
|
||||||
|
ctx = MigrationContext.configure(conn, opts={"compare_type": True})
|
||||||
|
diff = compare_metadata(ctx, Base.metadata)
|
||||||
|
engine.dispose()
|
||||||
|
assert diff == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_downgrade_and_upgrade_roundtrip(db_url: str) -> None:
|
||||||
|
from alembic import command
|
||||||
|
|
||||||
|
from netmover.db import alembic_config, upgrade
|
||||||
|
|
||||||
|
command.downgrade(alembic_config(db_url), "base")
|
||||||
|
upgrade(db_url)
|
||||||
|
|
||||||
|
|
||||||
|
def test_defaults(session: Session) -> None:
|
||||||
|
m = _mount()
|
||||||
|
session.add(m)
|
||||||
|
session.flush()
|
||||||
|
r = Rule(name="r", src_mount_id=m.id, dst_path="/opt/consume")
|
||||||
|
session.add(r)
|
||||||
|
session.commit()
|
||||||
|
assert m.state is MountState.UNMOUNTED
|
||||||
|
assert m.auto_mount is True
|
||||||
|
assert r.interval_seconds == 30
|
||||||
|
assert r.stability_seconds == 5
|
||||||
|
assert r.op_timeout_seconds == 120
|
||||||
|
assert r.dst_mount_id is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_datetime_roundtrip_is_utc_aware(session: Session) -> None:
|
||||||
|
berlin = timezone(timedelta(hours=2))
|
||||||
|
t = datetime(2026, 10, 10, 17, 0, tzinfo=berlin)
|
||||||
|
m = _mount(last_mounted_at=t)
|
||||||
|
session.add(m)
|
||||||
|
session.commit()
|
||||||
|
session.expire_all()
|
||||||
|
loaded = session.scalars(select(Mount)).one()
|
||||||
|
assert loaded.last_mounted_at == t
|
||||||
|
assert loaded.last_mounted_at is not None
|
||||||
|
assert loaded.last_mounted_at.tzinfo is UTC
|
||||||
|
|
||||||
|
|
||||||
|
def test_naive_datetime_rejected(session: Session) -> None:
|
||||||
|
session.add(_mount(last_mounted_at=datetime(2026, 1, 1)))
|
||||||
|
with pytest.raises(StatementError):
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_mount_in_use_cannot_be_deleted(session: Session) -> None:
|
||||||
|
m = _mount()
|
||||||
|
session.add(m)
|
||||||
|
session.flush()
|
||||||
|
session.add(Rule(name="r", src_mount_id=m.id, dst_path="/x"))
|
||||||
|
session.commit()
|
||||||
|
session.delete(m)
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_credential_in_use_cannot_be_deleted(session: Session) -> None:
|
||||||
|
c = Credential(name="nas", username="scan", secret_enc="x")
|
||||||
|
session.add(c)
|
||||||
|
session.flush()
|
||||||
|
session.add(_mount(cred_id=c.id))
|
||||||
|
session.commit()
|
||||||
|
session.delete(c)
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_rule_delete_cascades_runs_items_and_bookkeeping(session: Session) -> None:
|
||||||
|
m = _mount()
|
||||||
|
session.add(m)
|
||||||
|
session.flush()
|
||||||
|
r = Rule(name="r", src_mount_id=m.id, dst_path="/x")
|
||||||
|
session.add(r)
|
||||||
|
session.flush()
|
||||||
|
run = Run(rule_id=r.id)
|
||||||
|
session.add(run)
|
||||||
|
session.flush()
|
||||||
|
session.add(RunItem(run_id=run.id, src="a.pdf", status=ItemStatus.MOVED))
|
||||||
|
session.add(
|
||||||
|
CopiedFile(rule_id=r.id, src_relpath="a.pdf", size=1, mtime_ns=1, dst_path="/x/a.pdf")
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
session.delete(r)
|
||||||
|
session.commit()
|
||||||
|
assert session.scalars(select(Run)).all() == []
|
||||||
|
assert session.scalars(select(RunItem)).all() == []
|
||||||
|
assert session.scalars(select(CopiedFile)).all() == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_copied_files_unique_per_version(session: Session) -> None:
|
||||||
|
m = _mount()
|
||||||
|
session.add(m)
|
||||||
|
session.flush()
|
||||||
|
r = Rule(name="r", src_mount_id=m.id, dst_path="/x")
|
||||||
|
session.add(r)
|
||||||
|
session.flush()
|
||||||
|
session.add(
|
||||||
|
CopiedFile(rule_id=r.id, src_relpath="a.pdf", size=1, mtime_ns=1, dst_path="/x/a.pdf")
|
||||||
|
)
|
||||||
|
# neue Version derselben Datei ist erlaubt
|
||||||
|
session.add(
|
||||||
|
CopiedFile(rule_id=r.id, src_relpath="a.pdf", size=2, mtime_ns=2, dst_path="/x/a-1.pdf")
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
session.add(
|
||||||
|
CopiedFile(rule_id=r.id, src_relpath="a.pdf", size=1, mtime_ns=1, dst_path="/x/a-2.pdf")
|
||||||
|
)
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_enum_check_constraint_enforced_in_db(session: Session) -> None:
|
||||||
|
from sqlalchemy import text
|
||||||
|
|
||||||
|
session.add(_mount())
|
||||||
|
session.commit()
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
session.execute(text("UPDATE mounts SET type = 'ftp'"))
|
||||||
Reference in New Issue
Block a user