A practical guide to packaging your algorithms and running them over federated data on Ocean Protocol / Pontus-X — without the data ever leaving its source.
How to navigate
Scroll horizontally, use the ← → arrow keys or the buttons below. There are 13 slides.
/data directoryWhen the algorithm starts, the orchestrator mounts this folder structure. It is the only point of contact with the outside world.
Execution parameters (algoCustomData.json) and the files of each dataset, in subfolders named after the dataset identifier (DID).
Metadata (DDOs) of the contracted datasets. They describe the files available for the computation.
Secrets and credentials for the algorithm (e.g. username/password for an external service).
The only directory you really write to. Here you leave the result that will be delivered to the user.
Log output for debugging the execution.
Read from the input folders (read-only), work in memory or in /tmp, and write only to outputs.
Accessing the dataset files
The files to process are mounted under /data/inputs/<did>/ — one subfolder per dataset, named with its identifier (DID). No need to walk the disk by hand: job_details.files gives you the files for each DID and job_details.inputs() iterates (did, path) pairs ready to use. In clip-raster only input parameters are used —the data comes from external services— but this is how a generic algorithm would consume the contracted dataset.
Keep a clear separation between the algorithm code, the packaging and the test data.
Key idea
This same template works for any algorithm in your organization: only the contents of src/ change.
clip-raster/
├── algorithm/
│ ├── src/
│ │ ├── algorithm.py # orchestrator (run/save)
│ │ ├── data.py # input model (Pydantic)
│ │ ├── auth.py # external credentials
│ │ ├── download.py # data retrieval
│ │ ├── raster.py # domain logic
│ │ └── indices.py # computations
│ ├── pyproject.toml # dependencies
│ └── uv.lock
├── _data/ # local test data
│ ├── inputs/algoCustomData.json
│ ├── transformations/algorithm
│ └── outputs/
├── Dockerfile
└── docker-compose.yaml
You don't write the execution loop by hand. You register functions per phase with decorators — FastAPI style — and the runner orchestrates them.
/data/outputs.Why it matters
Splitting into phases makes your algorithm readable, testable and predictable. The runner injects the job context (job_details) and manages the lifecycle for you.
Every algorithm starts the same way: you create an Algorithm instance with its Config, and hang the phases on it.
src.algorithm — that is what ocean-execute looks for.Convention
Always start here and fill in each phase. The default run raises an error until you register your own.
from pathlib import Path
from ocean_runner import Algorithm, Config
from .data import InputParameters
# Result type your algorithm produces
type ResultsT = dict[str, ...]
algorithm = Algorithm[InputParameters, ResultsT].create(
Config(custom_input=InputParameters)
)
@algorithm.run
def run(_) -> ResultsT:
params = algorithm.job_details.input_parameters
... # your logic
return result
@algorithm.save_results
def save(_, result: ResultsT, base: Path):
... # write to `base` (= /data/outputs)
The user sends parameters as JSON. Turn them into a typed model: that gives you automatic validation and clear error messages before running anything.
In clip-raster
The only parameter is refcat, a cadastral reference. The validator requires it to be alphanumeric.
from typing import Any
from pydantic import BaseModel, field_validator
class InputParameters(BaseModel):
refcat: str
@field_validator("refcat", mode="before")
@classmethod
def is_valid_refcat(cls, v: Any) -> str | None:
if v is None:
return v
stripped = str(v).strip()
if stripped and not stripped.isalnum():
raise ValueError(f"'{stripped}' must be alpha-numeric")
return stripped
{ "refcat": "25168A010000060000IZ" }
Before running the heavy logic, the runner calls the validation phase. By default it checks that the dataset metadata and files exist.
You can override it with @algorithm.validate to add your own preconditions: are the credentials present? is there a connection to the external service? does the dataset have the expected format?
Good practice
Every validation you add here is a failure you save mid-computation — and a clear message for the user.
@algorithm.validate
def validate(_) -> None:
jd = algorithm.job_details
# Did the dataset metadata arrive?
assert jd.metadata, "Missing the dataset DDOs"
# Are there files to process?
assert jd.files, "The dataset contains no files"
# Your own business preconditions...
algorithm.logger.info("Input validated successfully")
If you don't register this phase, the runner applies a reasonable default validation. In clip-raster the data comes from external services, so the key validation lives inside run itself.
run orchestrates; the modules do the workThe run function should read like a summary of the algorithm: a few lines that tell the story. The detail lives in specialized modules.
The 6 stages of clip-raster
① resolve geometry → ② find product → ③ download bands → ④ clip → ⑤ compute indices → ⑥ return result.
import geopandas as gpd
from .download import download_product
@algorithm.run
def run(_) -> ResultsT:
params = algorithm.job_details.input_parameters
refcat = params.refcat
# ① parcel geometry (Cadastre service)
url = f"{CATASTRO_URL_PREFIX}{refcat}"
gdf = gpd.read_file(url)
geometry = gdf.geometry.item()
# ②③ search and download the Sentinel-2 product
download_product(geometry)
# ④⑤⑥ clip bands and compute indices
return compute_indices(geometry)
Never embed credentials in the code or the image. The runner exposes them in /data/transformations/algorithm as key/value pairs you read at runtime.
In clip-raster
The Copernicus username and password are read from that file to request a token and temporary S3 credentials.
Raise Algorithm.Error for expected failures. The runner catches it in the on_error phase and ends the job cleanly, with a useful message in the logs.
from dotenv import get_key
from ocean_runner import Algorithm
def get_copernicus_token() -> str:
user = get_key('/data/transformations/algorithm', 'username')
pwd = get_key('/data/transformations/algorithm', 'password')
if not user or not pwd:
raise Algorithm.Error('Missing credentials')
return request_copernicus_token(user, pwd)
/data/outputsThe save_results phase receives the result from run and the output base path. It is the only place where you write files the user will receive.
base already points to /data/outputs.In clip-raster
The four indices are drawn in a 2×2 grid and saved as indices.png.
from pathlib import Path
from .raster import save_as_img
@algorithm.save_results
def save(_, result: ResultsT, base: Path):
out_path = base / 'indices.png' # /data/outputs/indices.png
save_as_img(result, out_path)
Why separate run and save? Because it lets you change the output format without touching the logic, and test the computation without writing to disk.
The algorithm is delivered as a Docker image. Use a multistage build: one stage installs dependencies and another, lightweight, runs.
uv.lock (reproducible).ocean-execute --base-dir /data.Important
The ocean-execute command locates your algorithm object in src.algorithm and triggers the pipeline. You don't need a main().
# Stage 1 — build the environment with dependencies
FROM python:3.12 AS builder
COPY --from=ghcr.io/astral-sh/uv:0.5.11 /uv /uvx /bin/
ENV PATH="/algorithm/.venv/bin:$PATH"
WORKDIR /algorithm/
RUN --mount=type=bind,source=algorithm/uv.lock,target=uv.lock \
--mount=type=bind,source=algorithm/pyproject.toml,target=pyproject.toml \
uv sync --frozen --no-install-project
# Stage 2 — final, lightweight image
FROM python:3.12-slim
RUN apt-get update && apt-get install -y \
libgdal-dev gdal-bin libspatialindex-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /algorithm
COPY --from=builder /algorithm/.venv /algorithm/.venv
COPY algorithm/src /algorithm/src
ENV PATH="/algorithm/.venv/bin:$PATH"
ENV PYTHONPATH="/algorithm"
CMD ["ocean-execute", "--base-dir", "/data"]
Before publishing, test the image exactly as it will run on the portal. The trick is to mount your _data/ folder at the paths the algorithm expects.
:ro).:rw).src/ lets you iterate without rebuilding the image.Result
After docker compose up, the file appears in _data/outputs/ — just as the end user would receive it.
services:
algorithm:
build: .
entrypoint: 'ocean-execute --base-dir /data'
volumes:
- ./_data/inputs:/data/inputs:ro
- ./_data/ddos:/data/ddos:ro
- ./_data/transformations:/data/transformations:ro
- ./_data/outputs:/data/outputs:rw
- ./_data/logs:/data/logs:rw
- ./algorithm/src:/algorithm/src/ # iterate without rebuild
docker compose up --build # first time
docker compose up # changes only in src/
algorithm/src + Dockerfile + _data.run as a readable orchestrator./data/transformations.Algorithm.Error on expected failures./data/outputs.ocean-execute.Resources
Reference code clip-raster: github.com/AgrospAI/ocean-algo
Questions about the framework or publishing? The AgrospAI team supports you through the integration.
AgrospAI · Universitat de Lleida · Compute-to-Data data space