Development¶

This page is for people changing duck soup itself. To use it, see Getting started.

Setup¶

git clone https://github.com/henrik716/duck-soup.git
cd duck-soup
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[test,docs]"
pytest -v

Code layout¶

duck_soup/
  config.py      Pydantic schema for the YAML (the source of truth for every option)
  sources/       one reader per format → a SQL table expression (mostly ST_Read)
  engine.py      builds the src_<id> → step_N → mapped view chain
  steps.py       the SQL of each step type
  mapping.py     the mapping compiled to a layer's SELECT
  output.py      writes the layers: GeoPackage (GDAL) or GeoParquet (DuckDB)
  derive.py      DuckDB bootstrap (extensions) + Python UDFs (to_mgrs, to_geohash)
  sql_util.py    identifier / literal quoting
  worker.py      the editor's engine process: preview / counts / run, crash-isolated
  cli.py         check / run / serve / tutorial
  tutorial.py    copies the bundled tutorial (tutorial/) into a project folder
  web/
    app.py       FastAPI backend (/api/inspect, /api/preview, /api/run, …)
    ui/          TypeScript + Vite + MapLibre editor source
    static/      built editor (committed)
pipelines/       example pipelines
data/            test fixtures
tests/           pytest suite
docs/            this site (MkDocs Material)

Engine in brief¶

Each source becomes a src_<id> view, reprojected to the working CRS (sources/).
Each step becomes step_N on top of the previous view (engine.py). Spatial joins, clip, erase and nearest-neighbour use a plain JOIN on the spatial predicate, which DuckDB plans as an R-tree SPATIAL_JOIN, then aggregate per base row. Avoid LATERAL subqueries for spatial predicates: they compare every pair.
The mapping compiles to a final SELECT, written with COPY … (FORMAT GDAL, DRIVER 'GPKG'), or COPY … (FORMAT PARQUET) for a .parquet output (GeoParquet, with the geometry cast to GEOMETRY('<crs>') so the CRS is embedded).

The engine worker¶

The editor's server never opens a DuckDB connection itself: all DuckDB work happens in child processes (worker.py), so a native crash in DuckDB or its spatial extension kills only that process. The request answers with crashed: true and an error naming the exit code and the last log lines, and the next request starts a new worker.

ENGINE takes the interactive work: inspecting sources and listing their layers, previews, row counts. One call at a time. Each preview carries its editor tab's id (client); a newer preview from the same tab interrupts the running one (EngineWorker.interrupt, which stops the DuckDB query through derive.interrupt_all and keeps the worker) and skips any still waiting (call(..., stale=...)).
RUNNER takes runs, so a long run doesn't hold up previews. cancel (/api/run/cancel) kills it; the next run starts a new one.
Converting a large CSV/Excel file to Parquet gets a one-off worker per file, since it can take a minute and would otherwise hold up ENGINE.

While a call runs, the worker reads how far its query is (derive.query_progress, DuckDB's own progress bar) every half second and sends it to the server whenever it moves. EngineWorker.progress(tag) returns the latest value, and /api/progress?client=… hands the editor that tab's preview and the run in progress.

The server keeps everything that reports progress: it downloads remote files (for a run too, before handing it over) and tracks conversions, and the workers read the results from the shared cache folders. A worker exits by itself when the server process goes away. The CLI runs the engine in its own process, where a crash simply ends the run with a non-zero exit code.

Frontend¶

cd duck_soup/web/ui
npm install
npm run dev     # Vite on :5173, proxies /api to uvicorn on :8000
npm run build   # tsc + vite build → ../static/ (commit the output)

Run uvicorn duck_soup.web.app:app --reload alongside it. src/types.ts mirrors the Pydantic models in config.py, so keep them in sync when adding fields.

Docs¶

pip install -e ".[docs]"
mkdocs serve           # live preview at http://127.0.0.1:8000
mkdocs build --strict  # what CI runs; fails on broken links

Pages live in docs/, and navigation is in mkdocs.yml. When you add or change an option in config.py, update the matching reference page. The site deploys to GitHub Pages automatically on every push to main that touches docs/ or mkdocs.yml.

The tutorial pages include tables, YAML and diagrams from docs/tutorials/generated/, which scripts/build_tutorial.py produces by running every example through the engine. It also writes the sample data and pondsworth.yaml into duck_soup/tutorial/, which ships inside the package so duck-soup tutorial (duck_soup/tutorial.py) can copy it into a user's project folder. Re-run it after changing the engine or the examples, and commit the output:

python scripts/build_tutorial.py

The editor screenshots in docs/assets/screenshots/ (a light and a dark version of each) come from scripts/build_screenshots.py. It sets up a throwaway project from the tutorial data, starts the editor from inside it and captures every shot with Playwright. Retake them when the UI they show changes:

pip install -e ".[screenshots]"
python scripts/build_screenshots.py                     # every shot, ~2 minutes
python scripts/build_screenshots.py overview yaml-tab   # just these

It drives your installed Microsoft Edge (--browser chrome for Chrome), so there's no playwright install step. Playwright's bundled Chromium draws the map with software WebGL so slowly that the capture times out.

CI¶

.github/workflows/ci.yml runs pytest and the frontend build on every push and PR. On v* tags it also publishes the package to PyPI (duck-soup-etl) and the Docker image to ghcr.io/henrik716/duck-soup.

To release: bump version in pyproject.toml and add the release to the changelog in the same commit, push, then push a matching tag (git tag v0.1.15 && git push origin v0.1.15). PyPI rejects a version that's already been published. .github/workflows/docs.yml builds this site and deploys it.