Features#

LaminDB#

Access data & metadata across storage (files, arrays) & database (SQL) backends.

Track data lineage across notebooks, pipelines & UI: track(), Transform & Run.

Manage registries for experimental metadata & in-house ontologies, import public ontologies.

Validate, standardize & annotate data using registries: validate & standardize.

  • Inspect validation failures: inspect

  • Annotate with untyped or typed labels: add

  • Save data & metadata ACID: save

Organize and share data across a mesh of LaminDB instances.

  • Create & load instances like git repos: lamin init & lamin load

  • Zero-copy transfer data across instances

Zero lock-in, scalable, auditable, access management, and more.

  • Zero lock-in: LaminDB runs on generic backends server-side and is not a client for “Lamin Cloud”

    • Flexible storage backends (local, S3, GCP, anything fsspec supports)

    • Currently two SQL backends for managing metadata: SQLite & Postgres

  • Scalable: metadata tables support 100s of millions of entries

  • Auditable: data & metadata records are hashed, timestamped, and attributed to users (soon to come: LaminDB Log)

  • Access management:

    • High-level access management through Lamin’s collaborator roles

    • Fine-grained access management via storage & SQL roles (soon to come: Lamin Vault)

  • Secure: embedded in your infrastructure (Lamin has no access to your data & metadata)

  • Tested & typed (up to Django Model fields)

  • Idempotent & ACID operations

LaminHub#

Public demo instances to explore in the UI or load using the CLI via lamin load owner/instance:

LaminHub neither hosts data nor metadata, but connects to distributed storage locations & databases through LaminDB.

See validated data artifacts in context of ontologies & experimental metadata.

Query & search.

See scripts, notebooks & pipelines with their inputs & outputs.

Track pipelines, notebooks & UI transforms in one registry.

See parents and children of transforms.