Tutorial: Features & labels#

In the previous tutorial (Tutorial: Artifacts), we learned about about how to leverage basic metadata for artifacts to access data (query, search, cache & load).

Here, we walk through annotating & validating data with features & labels to improve:

  1. Finding data: Which collections measured expression of cell marker CD14? Which characterized cell line K562? Which collections have a test & train split? Etc.

  2. Using data: Are there typos in feature names? Are there typos in sampled labels? Are units of features consistent? Etc.

What was LaminDB’s most basic inspiration?

The pydata family of objects is at the heart of most data science, ML & comp bio workflows: DataFrame, AnnData, pytorch.DataLoader, zarr.Array, pyarrow.Table, xarray.Collection, …

And still, we couldn’t find a tool to link these objects to context so that they could be analyzed in context!

Context relevant for analyses includes anything that’s needed to interpret & model data.

So, lamindb.Artifact and lamindb.Collection track:

  • data sources, data transformations, models, users & pipelines that performed transformations (provenance)

  • any entity of the domain in which data is generated and modeled (features & labels)

import lamindb as ln
import pandas as pd
💡 connected lamindb: anonymous/lamin-tutorial
ln.settings.verbosity = "hint"

Register metadata#

Features and labels are the primary ways of registering domain-knowledge related metadata in LaminDB.

Features represent measurement dimensions (e.g. organism) and labels represent measurement values (e.g. iris setosa, iris versicolor, iris virginica).

Register labels#

We study 3 organism of the Iris plant: setosa, versicolor & virginica.

Let’s populate the universal label registry (ULabel) for them:

organisms = [ln.ULabel(name=name) for name in ["setosa", "versicolor", "virginica"]]
ln.save(organisms)
organisms
[ULabel(uid='I1k8dnmd', name='setosa', updated_at=2024-04-19 17:40:17 UTC, created_by_id=1),
 ULabel(uid='01RYRmFX', name='versicolor', updated_at=2024-04-19 17:40:17 UTC, created_by_id=1),
 ULabel(uid='zanEc0S7', name='virginica', updated_at=2024-04-19 17:40:17 UTC, created_by_id=1)]

Anticipating that we’ll have many different labels when working with more data, we’d like to express that all 3 labels are organism labels:

is_organism = ln.ULabel(name="is_organism")
is_organism.save()
is_organism.children.set(organisms)
is_organism.view_parents(with_children=True)
_images/0e491eef81b44033c2ce6d8c0e9db82e587110ed2596529a49aae28898e41e7f.svg

ULabel enables you to manage an in-house ontology to manage all kinds of untyped labels.

If you’d like to leverage pre-built typed ontologies for basic biological entities in the same way, see: Manage biological registries.

In addition to organism, we’d like to track the studies that produced the data:

studies = [ln.ULabel(name=name) for name in ["study0", "study1", "study2"]]
ln.save(studies)
is_study = ln.ULabel(name="is_study")
is_study.save()
is_study.children.set(studies)
is_study.view_parents(with_children=True)
_images/5eb18b9245470fbad84a5ed1a716ab93a8bbe514d6624747033c2c3160fe85cb.svg
Why label a dataset by study?

We can then

  1. query all artifacts link to this experiment

  2. model it as a confounder when we’ll analyze similar data from a follow-up experiment, and concatenate data using the label as a feature in a data matrix

Register features#

For every set of studied labels (measured values), we typically also want an identifier for the corresponding measurement dimension: the feature.

When we integrate datasets, feature names will label columns that store data.

Let’s create and save two Feature records to identify measurements of the iris organism label and the study:

ln.Feature(name="iris_organism_name", type="category").save()
ln.Feature(name="study_name", type="category").save()
# create a lookup object so that we can access features with auto-complete
features = ln.Feature.lookup()

Run an ML model#

Let’s now run a ML model that transforms the images into 4 high-level features.

def run_ml_model() -> pd.DataFrame:
    transform = ln.Transform(name="Petal & sepal regressor", type="pipeline")
    ln.track(transform=transform)
    input_data = ln.Collection.filter(name__startswith="Iris collection", version="1").one()
    input_paths = [
        path.download_to(path.name) for path in input_data.artifacts[0].path.glob("*")
    ]
    # apply ML model
    output_data = ln.core.datasets.df_iris_in_meter_study1()
    return output_data


df = run_ml_model()
Hide code cell output
💡 saved: Transform(uid='kN9FbAH51CVdR6ZL', name='Petal & sepal regressor', type='pipeline', updated_at=2024-04-19 17:40:18 UTC, created_by_id=1)
💡 saved: Run(uid='Fofu8defmraG42OJ21di', transform_id=2, created_by_id=1)
💡 tracked pip freeze > /home/runner/.cache/lamindb/run_env_pip_Fofu8defmraG42OJ21di.txt

The output is a dataframe:

df.head()
sepal_length sepal_width petal_length petal_width iris_organism_name
0 0.051 0.035 0.014 0.002 setosa
1 0.049 0.030 0.014 0.002 setosa
2 0.047 0.032 0.013 0.002 setosa
3 0.046 0.031 0.015 0.002 setosa
4 0.050 0.036 0.014 0.002 setosa

And this is the ML pipeline that produced the dataframe:

ln.core.run_context.transform.view_parents()
_images/493f708c2f4557456ea89fd0976a744efb6462b08bcc3b8c481642f741fd92fd.svg

Register the output data#

Let’s first register the features of the transformed data:

new_features = ln.Feature.from_df(df)
ln.save(new_features)
How to track units of features?

Use the unit field of Feature. In the above example, you’d do:

for feature in features:
    if feature.type == "number":
        feature.unit = "m"  # SI unit for meters
        feature.save()

We can now validate & register the dataframe in one line:

artifact = ln.Artifact.from_df(
    df,
    description="Iris study 1 - after measuring sepal & petal metrics",
)
artifact.save()
💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/toLh2R5lM3YouybbvGfe.parquet')
✅ storing artifact 'toLh2R5lM3YouybbvGfe' at '/home/runner/work/lamindb/lamindb/docs/lamin-tutorial/.lamindb/toLh2R5lM3YouybbvGfe.parquet'
artifact.features.add(new_features)

Feature sets#

Get an overview of linked features:

artifact.features
Features:
  columns: FeatureSet(uid='kL8ZpHrbmqfSh8GTCYoc', n=5, registry='core.Feature')
    🔗 iris_organism_name (0, core.ULabel): 
    sepal_length (number)
    sepal_width (number)
    petal_length (number)
    petal_width (number)

You’ll see that they’re always grouped in sets that correspond to records of FeatureSet.

Why does LaminDB model feature sets, not just features?
  1. Performance: Imagine you measure the same panel of 20k transcripts in 1M samples. By modeling the panel as a feature set, you’ll only need to store 1M instead of 1M x 20k = 20B links.

  2. Interpretation: Model protein panels, gene panels, etc.

  3. Data integration: Feature sets provide the currency that determines whether two collections can be easily concatenated.

These reasons do not hold for label sets. Hence, LaminDB does not model label sets.

A slot provides a string key to access feature sets. It’s typically the accessor within the registered data object, here pd.DataFrame.columns.

Let’s use it to access all linked features:

artifact.features["columns"].df()
uid name type unit description registries synonyms created_at updated_at created_by_id
id
1 KmGDrpOiHtC6 iris_organism_name category None None core.ULabel None 2024-04-19 17:40:17.352144+00:00 2024-04-19 17:40:18.275205+00:00 1
3 SO34iUteV8s5 sepal_length number None None None None 2024-04-19 17:40:23.498306+00:00 2024-04-19 17:40:23.498332+00:00 1
4 ryGUPpvH0pH4 sepal_width number None None None None 2024-04-19 17:40:23.498452+00:00 2024-04-19 17:40:23.498468+00:00 1
5 LtYvZb5cCLve petal_length number None None None None 2024-04-19 17:40:23.498578+00:00 2024-04-19 17:40:23.498593+00:00 1
6 LAceSnVNghQN petal_width number None None None None 2024-04-19 17:40:23.498723+00:00 2024-04-19 17:40:23.498739+00:00 1

There is one categorical feature, let’s add the organism labels:

organism_labels = ln.ULabel.filter(parents__name="is_organism").all()
artifact.labels.add(organism_labels, feature=features.iris_organism_name)

Let’s now add study labels:

artifact.labels.add(ulabels.study0, feature=features.study_name)
✅ linked new feature 'study_name' together with new feature set FeatureSet(uid='Zlyo3DTvxXoDyzuxZd43', n=1, registry='core.Feature', hash='_h0npb2Ssng_mepitxwZ', updated_at=2024-04-19 17:40:23 UTC, created_by_id=1)

In addition to the columns feature set, we now have an external feature set:

artifact.features
Features:
  columns: FeatureSet(uid='kL8ZpHrbmqfSh8GTCYoc', n=5, registry='core.Feature')
    🔗 iris_organism_name (3, core.ULabel): 'versicolor', 'setosa', 'virginica'
    sepal_length (number)
    sepal_width (number)
    petal_length (number)
    petal_width (number)
  external: FeatureSet(uid='Zlyo3DTvxXoDyzuxZd43', n=1, registry='core.Feature')
    🔗 study_name (1, core.ULabel): 'study0'

This is the context for our artifact:

artifact.describe()
Artifact(uid='toLh2R5lM3YouybbvGfe', suffix='.parquet', accessor='DataFrame', description='Iris study 1 - after measuring sepal & petal metrics', size=5347, hash='lLXVNjXWccXaDorKIth_kg', hash_type='md5', visibility=1, key_is_virtual=True, updated_at=2024-04-19 17:40:23 UTC)

Provenance:
  📎 storage: Storage(uid='NIXrqfee', root='/home/runner/work/lamindb/lamindb/docs/lamin-tutorial', type='local')
  📎 transform: Transform(uid='kN9FbAH51CVdR6ZL', name='Petal & sepal regressor', type='pipeline')
  📎 run: Run(uid='Fofu8defmraG42OJ21di', started_at=2024-04-19 17:40:18 UTC, is_consecutive=True)
  📎 created_by: User(uid='00000000', handle='anonymous')
Features:
  columns: FeatureSet(uid='kL8ZpHrbmqfSh8GTCYoc', n=5, registry='core.Feature')
    🔗 iris_organism_name (3, core.ULabel): 'versicolor', 'setosa', 'virginica'
    sepal_length (number)
    sepal_width (number)
    petal_length (number)
    petal_width (number)
  external: FeatureSet(uid='Zlyo3DTvxXoDyzuxZd43', n=1, registry='core.Feature')
    🔗 study_name (1, core.ULabel): 'study0'
Labels:
  📎 ulabels (4, core.ULabel): 'versicolor', 'setosa', 'virginica', 'study0'
artifact.view_lineage()
_images/9c4a7b273d763e914ecaaea59310f2d4c8ba023844201b0116ff5885864dd342.svg

See the database content:

ln.view(registries=["Feature", "FeatureSet", "ULabel"])
Hide code cell output
Feature
uid name type unit description registries synonyms created_at updated_at created_by_id
id
6 LAceSnVNghQN petal_width number None None None None 2024-04-19 17:40:23.498723+00:00 2024-04-19 17:40:23.498739+00:00 1
5 LtYvZb5cCLve petal_length number None None None None 2024-04-19 17:40:23.498578+00:00 2024-04-19 17:40:23.498593+00:00 1
4 ryGUPpvH0pH4 sepal_width number None None None None 2024-04-19 17:40:23.498452+00:00 2024-04-19 17:40:23.498468+00:00 1
3 SO34iUteV8s5 sepal_length number None None None None 2024-04-19 17:40:23.498306+00:00 2024-04-19 17:40:23.498332+00:00 1
1 KmGDrpOiHtC6 iris_organism_name category None None core.ULabel None 2024-04-19 17:40:17.352144+00:00 2024-04-19 17:40:18.275205+00:00 1
2 bV1WHecoJz6B study_name category None None core.ULabel None 2024-04-19 17:40:17.369993+00:00 2024-04-19 17:40:17.916065+00:00 1
FeatureSet
uid name n type registry hash created_at updated_at created_by_id
id
5 Zlyo3DTvxXoDyzuxZd43 None 1 None core.Feature _h0npb2Ssng_mepitxwZ 2024-04-19 17:40:23.600736+00:00 2024-04-19 17:40:23.606630+00:00 1
4 kL8ZpHrbmqfSh8GTCYoc None 5 None core.Feature Ms7NFNWzsU4hAz_rCsG9 2024-04-19 17:40:23.526982+00:00 2024-04-19 17:40:23.527006+00:00 1
2 UrsDV7dAJjOxIZMBe6pB None 2 None core.Feature BickiRh7YQcPu60k6m5n 2024-04-19 17:40:18.282847+00:00 2024-04-19 17:40:18.778262+00:00 1
ULabel
uid name description reference reference_type created_at updated_at created_by_id
id
8 qFSGm1Q6 is_study None None None 2024-04-19 17:40:17.289348+00:00 2024-04-19 17:40:17.289377+00:00 1
7 Ne7rmW0w study2 None None None 2024-04-19 17:40:17.279257+00:00 2024-04-19 17:40:17.279271+00:00 1
6 ps3hM9vQ study1 None None None 2024-04-19 17:40:17.279155+00:00 2024-04-19 17:40:17.279170+00:00 1
5 f5lLSSRn study0 None None None 2024-04-19 17:40:17.279021+00:00 2024-04-19 17:40:17.279054+00:00 1
4 IMoMRNcj is_organism None None None 2024-04-19 17:40:17.175096+00:00 2024-04-19 17:40:17.175132+00:00 1
3 zanEc0S7 virginica None None None 2024-04-19 17:40:17.132548+00:00 2024-04-19 17:40:17.132562+00:00 1
2 01RYRmFX versicolor None None None 2024-04-19 17:40:17.132435+00:00 2024-04-19 17:40:17.132453+00:00 1

Manage follow-up data#

Assume that a couple of weeks later, we receive a new dataset in a follow-up study 2.

Let’s track a new analysis:

ln.settings.transform.stem_uid = "dMtrt8YMSdl6"
ln.settings.transform.version = "1"

ln.track()
💡 Assuming editor is Jupyter Lab.
💡 Attaching notebook metadata
💡 notebook imports: lamindb==0.70.2 pandas==1.5.3
💡 saved: Transform(uid='dMtrt8YMSdl65zKv', name='Tutorial: Features & labels', key='tutorial2', version='1', type='notebook', updated_at=2024-04-19 17:40:25 UTC, created_by_id=1)
💡 saved: Run(uid='xGGMtqX4l5duM2MuQFQy', transform_id=3, created_by_id=1)
💡 tracked pip freeze > /home/runner/.cache/lamindb/run_env_pip_xGGMtqX4l5duM2MuQFQy.txt

Register a joint collection#

Assume we already ran all preprocessing including the ML model.

We get a DataFrame and store it as an artifact:

df = ln.core.datasets.df_iris_in_meter_study2()
ln.Artifact.from_df(df, description="Iris study 2 - transformed").save()
Hide code cell output
💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/lebLiz6TAYIxbVtcPxMi.parquet')
✅ storing artifact 'lebLiz6TAYIxbVtcPxMi' at '/home/runner/work/lamindb/lamindb/docs/lamin-tutorial/.lamindb/lebLiz6TAYIxbVtcPxMi.parquet'

Let’s load it:

artifact2 = ln.Artifact.filter(description="Iris study 2 - transformed").one()

We can now store the joint collection:

collection = ln.Collection(
    [artifact, artifact2], name="Iris flower study 1 & 2 - transformed"
)
collection.save()
✅ loaded: FeatureSet(uid='kL8ZpHrbmqfSh8GTCYoc', n=5, registry='core.Feature', hash='Ms7NFNWzsU4hAz_rCsG9', updated_at=2024-04-19 17:40:23 UTC, created_by_id=1)
✅ loaded: FeatureSet(uid='Zlyo3DTvxXoDyzuxZd43', n=1, registry='core.Feature', hash='_h0npb2Ssng_mepitxwZ', updated_at=2024-04-19 17:40:23 UTC, created_by_id=1)
💡 adding artifact [5] as input for run 3, adding parent transform 2

Auto-concatenate datasets#

Because both datasets measured the same validated feature set, we can auto-concatenate the collection:

collection.load().tail()
sepal_length sepal_width petal_length petal_width iris_organism_name
145 0.067 0.030 0.052 0.023 virginica
146 0.063 0.025 0.050 0.019 virginica
147 0.065 0.030 0.052 0.020 virginica
148 0.062 0.034 0.054 0.023 virginica
149 0.059 0.030 0.051 0.018 virginica

We can also access & query the underlying two artifact objects:

collection.artifacts.df()
uid storage_id key suffix accessor description version size hash hash_type n_objects n_observations transform_id run_id visibility key_is_virtual created_at updated_at created_by_id
id
5 toLh2R5lM3YouybbvGfe 1 None .parquet DataFrame Iris study 1 - after measuring sepal & petal m... None 5347 lLXVNjXWccXaDorKIth_kg md5 None None 2 2 1 True 2024-04-19 17:40:23.514768+00:00 2024-04-19 17:40:23.514794+00:00 1
6 lebLiz6TAYIxbVtcPxMi 1 None .parquet DataFrame Iris study 2 - transformed None 5397 LyfSIYPwVLRO_vGm3N7c2w md5 None None 3 3 1 True 2024-04-19 17:40:26.219701+00:00 2024-04-19 17:40:26.219730+00:00 1

Or look at their data lineage:

collection.view_lineage()
_images/ed6e469af9c076c86bac6da15ea34827730a12081899ee92d9aefa697166bc67.svg

Or look at the database:

ln.view()
Hide code cell output
Artifact
uid storage_id key suffix accessor description version size hash hash_type n_objects n_observations transform_id run_id visibility key_is_virtual created_at updated_at created_by_id
id
6 lebLiz6TAYIxbVtcPxMi 1 None .parquet DataFrame Iris study 2 - transformed None 5397 LyfSIYPwVLRO_vGm3N7c2w md5 NaN None 3 3 1 True 2024-04-19 17:40:26.219701+00:00 2024-04-19 17:40:26.219730+00:00 1
5 toLh2R5lM3YouybbvGfe 1 None .parquet DataFrame Iris study 1 - after measuring sepal & petal m... None 5347 lLXVNjXWccXaDorKIth_kg md5 NaN None 2 2 1 True 2024-04-19 17:40:23.514768+00:00 2024-04-19 17:40:23.514794+00:00 1
4 fmI4CzVG0FgA9xPcrA5a 2 iris_studies/study2_raw_images None None None 665518 PX8Vt9T28y-uCEJO1tKm7A md5-d 51.0 None 1 1 1 False 2024-04-19 17:40:14.096765+00:00 2024-04-19 17:40:14.096795+00:00 1
3 gPpSlQysec2GLvbVQjUy 2 iris_studies/study1_raw_images None None None 640617 j61W__GgImA18CKrIf7FVg md5-d 49.0 None 1 1 1 False 2024-04-19 17:40:13.520866+00:00 2024-04-19 17:40:13.520900+00:00 1
2 CDanhHQaYdJVTlQw2cAj 2 iris_studies/study0_raw_images None None None 656692 wVYKPpEsmmrqSpAZIRXCFg md5-d 51.0 None 1 1 1 False 2024-04-19 17:40:12.699383+00:00 2024-04-19 17:40:12.699413+00:00 1
1 Jcs76l0nQW6Oi1go0aeP 2 iris_studies/study0_raw_images/meta.csv .csv None None None 4355 ZpAEpN0iFYH6vjZNigic7g md5 NaN None 1 1 1 False 2024-04-19 17:40:11.795128+00:00 2024-04-19 17:40:11.795156+00:00 1
Collection
uid name description version hash reference reference_type transform_id run_id artifact_id visibility created_at updated_at created_by_id
id
4 kSeqwBltNa05IRcwyKTL Iris flower study 1 & 2 - transformed None None WiCRU6rEyPz4bxmvMt8t None None 3 3 None 1 2024-04-19 17:40:26.262892+00:00 2024-04-19 17:40:26.262916+00:00 1
3 KeVx4xgsVQWQHhlVQCAZ Iris collection Another 50 images 3 T-U8z2Zi5rFYdAD9pzmS None None 1 1 None 1 2024-04-19 17:40:14.110651+00:00 2024-04-19 17:40:14.110697+00:00 1
2 KeVx4xgsVQWQHhlVt74c Iris collection Another 50 images 2 5cCK6ZLOPB0cV3tyeZup None None 1 1 None 1 2024-04-19 17:40:13.534888+00:00 2024-04-19 17:40:13.534914+00:00 1
1 KeVx4xgsVQWQHhlVssV0 Iris collection 50 image files and metadata 1 WwFLpNFmK8GMC2dSGj1W None None 1 1 None 1 2024-04-19 17:40:12.894475+00:00 2024-04-19 17:40:12.894503+00:00 1
Feature
uid name type unit description registries synonyms created_at updated_at created_by_id
id
6 LAceSnVNghQN petal_width number None None None None 2024-04-19 17:40:23.498723+00:00 2024-04-19 17:40:23.498739+00:00 1
5 LtYvZb5cCLve petal_length number None None None None 2024-04-19 17:40:23.498578+00:00 2024-04-19 17:40:23.498593+00:00 1
4 ryGUPpvH0pH4 sepal_width number None None None None 2024-04-19 17:40:23.498452+00:00 2024-04-19 17:40:23.498468+00:00 1
3 SO34iUteV8s5 sepal_length number None None None None 2024-04-19 17:40:23.498306+00:00 2024-04-19 17:40:23.498332+00:00 1
1 KmGDrpOiHtC6 iris_organism_name category None None core.ULabel None 2024-04-19 17:40:17.352144+00:00 2024-04-19 17:40:18.275205+00:00 1
2 bV1WHecoJz6B study_name category None None core.ULabel None 2024-04-19 17:40:17.369993+00:00 2024-04-19 17:40:17.916065+00:00 1
FeatureSet
uid name n type registry hash created_at updated_at created_by_id
id
5 Zlyo3DTvxXoDyzuxZd43 None 1 None core.Feature _h0npb2Ssng_mepitxwZ 2024-04-19 17:40:23.600736+00:00 2024-04-19 17:40:23.606630+00:00 1
4 kL8ZpHrbmqfSh8GTCYoc None 5 None core.Feature Ms7NFNWzsU4hAz_rCsG9 2024-04-19 17:40:23.526982+00:00 2024-04-19 17:40:23.527006+00:00 1
2 UrsDV7dAJjOxIZMBe6pB None 2 None core.Feature BickiRh7YQcPu60k6m5n 2024-04-19 17:40:18.282847+00:00 2024-04-19 17:40:18.778262+00:00 1
Run
uid transform_id started_at finished_at created_by_id json report_id environment_id is_consecutive reference reference_type created_at
id
1 Un6kaGLyWaVTuoRq275x 1 2024-04-19 17:40:09.302707+00:00 None 1 None None None True None None 2024-04-19 17:40:09.302838+00:00
2 Fofu8defmraG42OJ21di 2 2024-04-19 17:40:18.828235+00:00 None 1 None None None True None None 2024-04-19 17:40:18.828373+00:00
3 xGGMtqX4l5duM2MuQFQy 3 2024-04-19 17:40:25.111240+00:00 None 1 None None None True None None 2024-04-19 17:40:25.111371+00:00
Storage
uid root description type region created_at updated_at created_by_id
id
2 kobXuZ1eiiLn s3://lamindb-dev-datasets None s3 us-east-1 2024-04-19 17:40:11.732916+00:00 2024-04-19 17:40:11.732955+00:00 1
1 NIXrqfee /home/runner/work/lamindb/lamindb/docs/lamin-t... None local None 2024-04-19 17:40:07.762074+00:00 2024-04-19 17:40:07.762097+00:00 1
Transform
uid name key version description type latest_report_id source_code_id reference reference_type created_at updated_at created_by_id
id
3 dMtrt8YMSdl65zKv Tutorial: Features & labels tutorial2 1 None notebook None None None None 2024-04-19 17:40:25.105254+00:00 2024-04-19 17:40:25.105288+00:00 1
2 kN9FbAH51CVdR6ZL Petal & sepal regressor None None None pipeline None None None None 2024-04-19 17:40:18.823665+00:00 2024-04-19 17:40:18.823694+00:00 1
1 NJvdsWWbJlZS6K79 Tutorial: Artifacts tutorial 0 None notebook None None None None 2024-04-19 17:40:09.295516+00:00 2024-04-19 17:40:09.295547+00:00 1
ULabel
uid name description reference reference_type created_at updated_at created_by_id
id
8 qFSGm1Q6 is_study None None None 2024-04-19 17:40:17.289348+00:00 2024-04-19 17:40:17.289377+00:00 1
7 Ne7rmW0w study2 None None None 2024-04-19 17:40:17.279257+00:00 2024-04-19 17:40:17.279271+00:00 1
6 ps3hM9vQ study1 None None None 2024-04-19 17:40:17.279155+00:00 2024-04-19 17:40:17.279170+00:00 1
5 f5lLSSRn study0 None None None 2024-04-19 17:40:17.279021+00:00 2024-04-19 17:40:17.279054+00:00 1
4 IMoMRNcj is_organism None None None 2024-04-19 17:40:17.175096+00:00 2024-04-19 17:40:17.175132+00:00 1
3 zanEc0S7 virginica None None None 2024-04-19 17:40:17.132548+00:00 2024-04-19 17:40:17.132562+00:00 1
2 01RYRmFX versicolor None None None 2024-04-19 17:40:17.132435+00:00 2024-04-19 17:40:17.132453+00:00 1
User
uid handle name created_at updated_at
id
1 00000000 anonymous None 2024-04-19 17:40:07.757709+00:00 2024-04-19 17:40:07.757734+00:00

This is it! 😅

If you’re interested, please check out guides & use cases or make an issue on GitHub to discuss.

Appendix#

Manage metadata#

Avoid duplicates#

Let’s create a label “project1”:

ln.ULabel(name="project1").save()

We already created a project1 label before, let’s see what happens if we try to create it again:

label = ln.ULabel(name="project1")
label.save()
❗ loaded ULabel record with same name: 'project1' (disable via ln.settings.upon_create_search_names)

Instead of creating a new record, LaminDB loads and returns the existing record from the database.

If there is no exact match, LaminDB will warn you upon creating a record about potential duplicates.

Say, we spell “project 1” with a white space:

ln.ULabel(name="project 1")
❗ record with similar name exist! did you mean to load it?
uid score
name
project1 lUAo32vg 94.1
ULabel(uid='sgve57OV', name='project 1', created_by_id=1)

To avoid inserting duplicates when creating new records, a search compares whether a similar record already exists.

You can switch it off for performance gains via upon_create_search_names.

Update & delete records#

label = ln.ULabel.filter(name="project1").first()
label
ULabel(uid='lUAo32vg', name='project1', updated_at=2024-04-19 17:40:26 UTC, created_by_id=1)
label.name = "project1a"
label.save()
label
ULabel(uid='lUAo32vg', name='project1a', updated_at=2024-04-19 17:40:26 UTC, created_by_id=1)
label.delete()
(1, {'lnschema_core.ULabel': 1})

Manage storage#

Change default storage#

The default storage location is:

ln.settings.storage
PosixUPath('/home/runner/work/lamindb/lamindb/docs/lamin-tutorial')

You can change it by setting ln.settings.storage = "s3://my-bucket".

See all storage locations#

ln.Storage.df()
uid root description type region created_at updated_at created_by_id
id
2 kobXuZ1eiiLn s3://lamindb-dev-datasets None s3 us-east-1 2024-04-19 17:40:11.732916+00:00 2024-04-19 17:40:11.732955+00:00 1
1 NIXrqfee /home/runner/work/lamindb/lamindb/docs/lamin-t... None local None 2024-04-19 17:40:07.762074+00:00 2024-04-19 17:40:07.762097+00:00 1

Set verbosity#

To reduce the number of logging messages, set verbosity:

ln.settings.verbosity = 3  # only show info, no hints
# clean up what we wrote in this notebook
!lamin delete --force lamin-tutorial
!rm -r lamin-tutorial
💡 deleting instance anonymous/lamin-tutorial
❗ manually delete your stored data: /home/runner/work/lamindb/lamindb/docs/lamin-tutorial