Annotate data for developers#

While data is the primary information or raw facts that are collected and stored, metadata is the supporting information that provides context and meaning to that data.

LaminDB let’s you annotate data with metadata in two ways: features and labels. (Also see tutorial.)

This guide extends Quickstart to explain the details of annotating data.

Setup#

Let us create an instance that has bionty mounted:

!lamin init --storage ./test-annotate --schema bionty

import lamindb as ln
import bionty as bt
import pandas as pd
import anndata as ad

💡 connected lamindb: testuser1/test-annotate

bt.settings.auto_save_parents = False  # ignores ontological hierarchy
ln.settings.verbosity = "info"
ln.settings.transform.stem_uid = "sU0y1kF3igep"
ln.settings.transform.version = "0"

Register a artifact#

Let’s use the same example data as in the Quickstart:

df = pd.DataFrame(
    {"CD8A": [1, 2, 3], "CD4": [3, 4, 5], "CD14": [5, 6, 7]},
    index=["sample1", "sample2", "sample3"],
)

In addition to the data, we also have two types of metadata as follows:

# observational metadata (1:1 correspondence with samples)
obs_meta = pd.DataFrame(
    {
        "cell_type": ["T cell", "T cell", "Monocyte"],
        "tissue": ["capillary blood", "arterial blood", "capillary blood"],
    },
    index=["sample1", "sample2", "sample3"],
)

# external metadata (describes the entire artifact)
external_meta = {
    "organism": "human",
    "assay": "scRNA-seq",
    "experiment": "EXP0001",
    "project": "PRJ0001",
}

To store both data and observational metadata, we use an AnnData object:

# note that we didn't add external metadata to adata.uns, because we will use LaminDB to store it
adata = ad.AnnData(df, obs=obs_meta)
adata

AnnData object with n_obs × n_vars = 3 × 3
    obs: 'cell_type', 'tissue'

Now let’s register the AnnData object without annotating with any metadata:

ln.track()

artifact = ln.Artifact.from_anndata(adata, description="my RNA-seq")
artifact.save()

We don’t see any metadata in the registered artifact yet:

artifact.describe()

Artifact(uid='WMi37whpkywfG2WZARwK', suffix='.h5ad', accessor='AnnData', description='my RNA-seq', size=21224, hash='jBNzT3fmTNEcfJ19FK2euw', hash_type='md5', visibility=1, key_is_virtual=True, updated_at=2024-05-01 18:50:18 UTC)

Provenance:
  📎 storage: Storage(uid='Pca4eGu9pKMS', root='/home/runner/work/lamindb/lamindb/docs/test-annotate', type='local', instance_uid='3kW5y8h7c8wG')
  📎 transform: Transform(uid='sU0y1kF3igep6K79', name='Annotate data for developers', key='annotate-for-developers', version='0', type='notebook')
  📎 run: Run(uid='ir7GHMTgACV1mS2jxd8W', started_at=2024-05-01 18:50:17 UTC, is_consecutive=True)
  📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1')

Define features and labels#

Features and labels are records from their respective registries.

You can define them schema-less using Feature and ULabel registries, or schema-full using dedicated registries.

Define data features#

Data features refer to individual measurable properties or characteristics of a phenomenon being observed. In data analysis and machine learning, features are the input variables used to predict or classify an outcome.

Data features are often numeric, but can also be categorical. For example, in the case of gene expression data, the features are the expression levels of individual genes. They are often stored as columns in a data table (adata.var_names for AnnData objects).

Here we define them using the Gene registry:

data_features = bt.Gene.from_values(
    adata.var_names, 
    organism="human", # or set globally: bt.settings.organism = "human"
    )
ln.save(data_features)
data_features

✅ loaded 2 Gene records matching symbol: 'CD4', 'CD8A'

✅ created 1 Gene record from Bionty matching symbol: 'CD14'

[Gene(uid='1j4At3x7akJU', symbol='CD4', ensembl_gene_id='ENSG00000010610', ncbi_gene_ids='920', biotype='protein_coding', description='CD4 molecule ', synonyms='', updated_at=2024-05-01 18:49:42 UTC, organism_id=1, public_source_id=9, created_by_id=1),
 Gene(uid='6Aqvc8ckDYeN', symbol='CD8A', ensembl_gene_id='ENSG00000153563', ncbi_gene_ids='925', biotype='protein_coding', description='CD8 subunit alpha ', synonyms='CD8|P32|CD8ALPHA', updated_at=2024-05-01 18:49:42 UTC, organism_id=1, public_source_id=9, created_by_id=1),
 Gene(uid='3bhNYquOnA4s', symbol='CD14', ensembl_gene_id='ENSG00000170458', ncbi_gene_ids='929', biotype='protein_coding', description='CD14 molecule ', synonyms='', updated_at=2024-05-01 18:50:21 UTC, organism_id=1, public_source_id=9, created_by_id=1)]

Define metadata features#

Metadata features refer to descriptive or contextual information about the data. They don’t directly describe the content of the data but rather its characteristics.

In this example, the metadata features are “cell_type”, “tissue” that describe observations (stored in adata.obs.columns) and “organism”, “assay”, “experiment” that describe the entire artifact.

Here we define them using the Feature registry:

# obs metadata features
obs_meta_features = ln.Feature.from_df(adata.obs)
ln.save(obs_meta_features)
obs_meta_features

RecordsList([Feature(uid='clnGmFZmC2Ro', name='cell_type', type='category', registries='bionty.CellType', updated_at=2024-05-01 18:49:47 UTC, created_by_id=1),
             Feature(uid='G1dQozF1dlHg', name='tissue', type='category', updated_at=2024-05-01 18:50:21 UTC, created_by_id=1)])

# external metadata features
external_meta_features = [
    ln.Feature(name=name, type="category") for name in external_meta.keys()
]
ln.save(external_meta_features)
external_meta_features

❗ record with similar name exist! did you mean to load it?

	uid	synonyms	score
name
assay_ontology_id	BMRQl603m5Sg		90.0

[Feature(uid='r4XCHsu3y7jO', name='organism', type='category', updated_at=2024-05-01 18:50:21 UTC, created_by_id=1),
 Feature(uid='0xwR6Yh04sul', name='assay', type='category', updated_at=2024-05-01 18:50:21 UTC, created_by_id=1),
 Feature(uid='RDKBzU655VBy', name='experiment', type='category', updated_at=2024-05-01 18:50:21 UTC, created_by_id=1),
 Feature(uid='Obu9YZtR2nFM', name='project', type='category', updated_at=2024-05-01 18:50:21 UTC, created_by_id=1)]

Define metadata labels#

Metadata labels are the categorical values of metadata features. They are more specific than features and are often used in classification.

In this example, the metadata labels of feature “cell_type” are “T cell” and “Monocyte”; the metadata labels of feature “tissue” are “capillary blood”, “arterial blood”; the metadata labels of feature “organism” is “human”; and so on.

Let’s define them with their respective registries:

cell_types = bt.CellType.from_values(adata.obs["cell_type"])
ln.save(cell_types)
cell_types

✅ created 1 CellType record from Bionty matching name: 'T cell'

✅ created 1 CellType record from Bionty matching synonyms: 'Monocyte'

[CellType(uid='22LvKd01', name='T cell', ontology_id='CL:0000084', synonyms='T-cell|T-lymphocyte|T lymphocyte', description='A Type Of Lymphocyte Whose Defining Characteristic Is The Expression Of A T Cell Receptor Complex.', updated_at=2024-05-01 18:50:22 UTC, public_source_id=21, created_by_id=1),
 CellType(uid='28V22coI', name='monocyte', ontology_id='CL:0000576', description='Myeloid Mononuclear Recirculating Leukocyte That Can Act As A Precursor Of Tissue Macrophages, Osteoclasts And Some Populations Of Tissue Dendritic Cells.', updated_at=2024-05-01 18:50:22 UTC, public_source_id=21, created_by_id=1)]

tissues = bt.Tissue.from_values(adata.obs["tissue"])
ln.save(tissues)
tissues

✅ created 2 Tissue records from Bionty matching name: 'arterial blood', 'capillary blood'

[Tissue(uid='7gWJkhPG', name='capillary blood', ontology_id='UBERON:0013757', synonyms='portion of capillary blood|blood in capillary|portion of blood in capillary', description='A Blood That Is Part Of A Capillary.', updated_at=2024-05-01 18:50:22 UTC, public_source_id=25, created_by_id=1),
 Tissue(uid='3O0QD2cL', name='arterial blood', ontology_id='UBERON:0013755', synonyms='portion of arterial blood|arterial blood|blood in artery', description='A Blood That Is Part Of A Artery.', updated_at=2024-05-01 18:50:22 UTC, public_source_id=25, created_by_id=1)]

organism = bt.Organism.from_public(name=external_meta["organism"])
organism.save()
organism

Organism(uid='1dpCL6Td', name='human', ontology_id='NCBITaxon:9606', scientific_name='homo_sapiens', updated_at=2024-05-01 18:50:22 UTC, public_source_id=1, created_by_id=1)

assay = bt.ExperimentalFactor.from_public(name=external_meta["assay"])
assay.save()
assay

✅ loaded 1 ExperimentalFactor record matching synonyms: 'scRNA-seq'

ExperimentalFactor(uid='4WYv9kl0', name='single-cell RNA sequencing', ontology_id='EFO:0008913', synonyms='scRNA-seq|single cell RNA sequencing|single-cell transcriptome sequencing|single-cell RNA-seq', description='A Protocol That Provides The Expression Profiles Of Single Cells Via The Isolation And Barcoding Of Single Cells And Their Rna, Reverse Transcription, Amplification, Library Generation And Sequencing.', molecule='RNA assay', instrument='single cell sequencing', updated_at=2024-05-01 18:50:22 UTC, public_source_id=39, created_by_id=1)

experiment = ln.ULabel(name=external_meta["experiment"], description="An experiment")
experiment.save()
experiment

ULabel(uid='aAfWrw1i', name='EXP0001', description='An experiment', updated_at=2024-05-01 18:50:22 UTC, created_by_id=1)

project = ln.ULabel(name=external_meta["project"], description="A project")
project.save()
project

ULabel(uid='e7GPlNlY', name='PRJ0001', description='A project', updated_at=2024-05-01 18:50:22 UTC, created_by_id=1)

Annotate with features#

Non-external features are annotated when registering artifacts using .from_df or .from_anndata methods:

(See the below “Annotate with labels stratified by metadata features” session for adding external features.)

artifact = ln.Artifact.from_anndata(adata, description="my RNA-seq")
artifact.save()

artifact.features.add_from_anndata(var_field=bt.Gene.symbol, organism="human")

💡 parsing feature names of X stored in slot 'var'

✅    3 terms (100.00%) are validated for symbol

✅    linked: FeatureSet(uid='KyHezRiMy6ymU3JwsYON', n=3, type='number', registry='bionty.Gene', hash='f2UVeHefaZxXFjmUwo9O', created_by_id=1)

💡 parsing feature names of slot 'obs'

✅    2 terms (100.00%) are validated for name

✅    linked: FeatureSet(uid='AoOZPq8bTog5A40Oo0Hb', n=2, registry='core.Feature', hash='nyP0vwjSTTPufYunlX5h', created_by_id=1)

✅ saved 2 feature sets for slots: 'var','obs'

This artifact is now annotated with features:

artifact.describe()

Artifact(uid='WMi37whpkywfG2WZARwK', suffix='.h5ad', accessor='AnnData', description='my RNA-seq', size=21224, hash='jBNzT3fmTNEcfJ19FK2euw', hash_type='md5', visibility=1, key_is_virtual=True, updated_at=2024-05-01 18:50:22 UTC)

Provenance:
  📎 storage: Storage(uid='Pca4eGu9pKMS', root='/home/runner/work/lamindb/lamindb/docs/test-annotate', type='local', instance_uid='3kW5y8h7c8wG')
  📎 transform: Transform(uid='sU0y1kF3igep6K79', name='Annotate data for developers', key='annotate-for-developers', version='0', type='notebook')
  📎 run: Run(uid='ir7GHMTgACV1mS2jxd8W', started_at=2024-05-01 18:50:17 UTC, is_consecutive=True)
  📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1')
Features:
  var: FeatureSet(uid='KyHezRiMy6ymU3JwsYON', n=3, type='number', registry='bionty.Gene')
    'CD14', 'CD4', 'CD8A'
  obs: FeatureSet(uid='AoOZPq8bTog5A40Oo0Hb', n=2, registry='core.Feature')
    🔗 cell_type (0, bionty.CellType): 
    tissue (category)

You see two types of features are annotated and organized as featuresets by slot:

“var”: data features
“obs”: observational metadata features

artifact.features

Features:
  var: FeatureSet(uid='KyHezRiMy6ymU3JwsYON', n=3, type='number', registry='bionty.Gene')
    'CD14', 'CD4', 'CD8A'
  obs: FeatureSet(uid='AoOZPq8bTog5A40Oo0Hb', n=2, registry='core.Feature')
    🔗 cell_type (0, bionty.CellType): 
    tissue (category)

Use slots to retrieve corresponding annotated features:

artifact.features["var"].df()

	uid	symbol	stable_id	ensembl_gene_id	ncbi_gene_ids	biotype	description	synonyms	organism_id	public_source_id	created_at	updated_at	created_by_id
id
5	1j4At3x7akJU	CD4	None	ENSG00000010610	920	protein_coding	CD4 molecule		1	9	2024-05-01 18:49:42.627934+00:00	2024-05-01 18:49:42.627952+00:00	1
6	6Aqvc8ckDYeN	CD8A	None	ENSG00000153563	925	protein_coding	CD8 subunit alpha	CD8\|P32\|CD8ALPHA	1	9	2024-05-01 18:49:42.628108+00:00	2024-05-01 18:49:42.628123+00:00	1
7	3bhNYquOnA4s	CD14	None	ENSG00000170458	929	protein_coding	CD14 molecule		1	9	2024-05-01 18:50:21.291130+00:00	2024-05-01 18:50:21.291165+00:00	1

artifact.features["obs"].df()

	uid	name	type	unit	description	registries	synonyms	created_at	updated_at	created_by_id
id
1	clnGmFZmC2Ro	cell_type	category	None	None	bionty.CellType	None	2024-05-01 18:49:24.106806+00:00	2024-05-01 18:49:47.588733+00:00	1
4	G1dQozF1dlHg	tissue	category	None	None	None	None	2024-05-01 18:50:21.348238+00:00	2024-05-01 18:50:21.348268+00:00	1

Annotate with labels#

If you simply want to tag a artifact with some descriptive labels, you can pass them to .labels.add. For example, let’s add the experiment label “EXP0001” and project label “PRJ0001” to the artifact:

artifact.labels.add(experiment)
artifact.labels.add(project)

Now you see the artifact is annotated with ‘EXP0001’, ‘PRJ0001’ labels:

artifact.describe()

Artifact(uid='WMi37whpkywfG2WZARwK', suffix='.h5ad', accessor='AnnData', description='my RNA-seq', size=21224, hash='jBNzT3fmTNEcfJ19FK2euw', hash_type='md5', visibility=1, key_is_virtual=True, updated_at=2024-05-01 18:50:22 UTC)

Provenance:
  📎 storage: Storage(uid='Pca4eGu9pKMS', root='/home/runner/work/lamindb/lamindb/docs/test-annotate', type='local', instance_uid='3kW5y8h7c8wG')
  📎 transform: Transform(uid='sU0y1kF3igep6K79', name='Annotate data for developers', key='annotate-for-developers', version='0', type='notebook')
  📎 run: Run(uid='ir7GHMTgACV1mS2jxd8W', started_at=2024-05-01 18:50:17 UTC, is_consecutive=True)
  📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1')
Features:
  var: FeatureSet(uid='KyHezRiMy6ymU3JwsYON', n=3, type='number', registry='bionty.Gene')
    'CD14', 'CD4', 'CD8A'
  obs: FeatureSet(uid='AoOZPq8bTog5A40Oo0Hb', n=2, registry='core.Feature')
    🔗 cell_type (0, bionty.CellType): 
    tissue (category)
Labels:
  📎 ulabels (2, core.ULabel): 'PRJ0001', 'EXP0001'

To view all annotated labels:

artifact.labels

Labels:
  📎 ulabels (2, core.ULabel): 'PRJ0001', 'EXP0001'

Since we didn’t specify which features the labels belongs to, they are accessible only through the default accessor “.ulabels” for ULabel Registry.

You may already notice that it could be difficult to interpret labels without features if they belong to the same registry.

artifact.ulabels.df()

	uid	name	description	reference	reference_type	created_at	updated_at	created_by_id
id
5	aAfWrw1i	EXP0001	An experiment	None	None	2024-05-01 18:50:22.610085+00:00	2024-05-01 18:50:22.610110+00:00	1
6	e7GPlNlY	PRJ0001	A project	None	None	2024-05-01 18:50:22.627720+00:00	2024-05-01 18:50:22.627744+00:00	1

Annotate with labels stratified by metadata features#

For labels associated with metadata features, you can pass “feature” to .labels.add to stratify them by feature. (Another way to stratify labels is through ontological hierarchy, which is covered in the Quickstart)

Let’s add the experiment label “EXP0001” and project label “PRJ0001” to the artifact again, this time specifying their features:

# an auto-complete object of registered features
features = ln.Feature.lookup()

artifact.labels.add(experiment, feature=features.experiment)
artifact.labels.add(project, feature=features.project)

You now see a 3rd featureset is added to the artifact at slot “external”, and the labels are stratified by two features:

artifact.describe()

Artifact(uid='WMi37whpkywfG2WZARwK', suffix='.h5ad', accessor='AnnData', description='my RNA-seq', size=21224, hash='jBNzT3fmTNEcfJ19FK2euw', hash_type='md5', visibility=1, key_is_virtual=True, updated_at=2024-05-01 18:50:22 UTC)

Provenance:
  📎 storage: Storage(uid='Pca4eGu9pKMS', root='/home/runner/work/lamindb/lamindb/docs/test-annotate', type='local', instance_uid='3kW5y8h7c8wG')
  📎 transform: Transform(uid='sU0y1kF3igep6K79', name='Annotate data for developers', key='annotate-for-developers', version='0', type='notebook')
  📎 run: Run(uid='ir7GHMTgACV1mS2jxd8W', started_at=2024-05-01 18:50:17 UTC, is_consecutive=True)
  📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1')
Features:
  var: FeatureSet(uid='KyHezRiMy6ymU3JwsYON', n=3, type='number', registry='bionty.Gene')
    'CD14', 'CD4', 'CD8A'
  obs: FeatureSet(uid='AoOZPq8bTog5A40Oo0Hb', n=2, registry='core.Feature')
    🔗 cell_type (0, bionty.CellType): 
    tissue (category)
  external: FeatureSet(uid='RrZwjxu2WaW1CSjQ8GSM', n=2, registry='core.Feature')
    🔗 project (1, core.ULabel): 'PRJ0001'
    🔗 experiment (1, core.ULabel): 'EXP0001'
Labels:
  📎 ulabels (2, core.ULabel): 'PRJ0001', 'EXP0001'

With feature-stratified labels, you can retrieve labels by feature:

artifact.labels.get(features.experiment).df()

	uid	name	description	reference	reference_type	created_at	updated_at	created_by_id
id
5	aAfWrw1i	EXP0001	An experiment	None	None	2024-05-01 18:50:22.610085+00:00	2024-05-01 18:50:22.610110+00:00	1

Note that adding feature-stratified labels will also allow you to retrieve labels with the default accessor of respective registries:

artifact.labels.add(assay, feature=features.assay)

# access labels directly via default accessor "experimental_factors"
artifact.experimental_factors.df()

	uid	name	ontology_id	abbr	synonyms	description	molecule	instrument	measurement	public_source_id	created_at	updated_at	created_by_id
id
1	4WYv9kl0	single-cell RNA sequencing	EFO:0008913	None	scRNA-seq\|single cell RNA sequencing\|single-ce...	A Protocol That Provides The Expression Profil...	RNA assay	single cell sequencing	None	39	2024-05-01 18:49:45.118737+00:00	2024-05-01 18:50:22.591461+00:00	1

# access labels via feature
artifact.labels.get(features.assay).df()

	uid	name	ontology_id	abbr	synonyms	description	molecule	instrument	measurement	public_source_id	created_at	updated_at	created_by_id
id
1	4WYv9kl0	single-cell RNA sequencing	EFO:0008913	None	scRNA-seq\|single cell RNA sequencing\|single-ce...	A Protocol That Provides The Expression Profil...	RNA assay	single cell sequencing	None	39	2024-05-01 18:49:45.118737+00:00	2024-05-01 18:50:22.591461+00:00	1

Let’s finish the rest annotation of labels:

# labels of obs metadata features
artifact.labels.add(cell_types, feature=features.cell_type)
artifact.labels.add(tissues, feature=features.tissue)

# labels of external metadata features
artifact.labels.add(organism, feature=features.organism)

Now you’ve annotated your artifact with all features and labels:

artifact.describe()

Artifact(uid='WMi37whpkywfG2WZARwK', suffix='.h5ad', accessor='AnnData', description='my RNA-seq', size=21224, hash='jBNzT3fmTNEcfJ19FK2euw', hash_type='md5', visibility=1, key_is_virtual=True, updated_at=2024-05-01 18:50:22 UTC)

Provenance:
  📎 storage: Storage(uid='Pca4eGu9pKMS', root='/home/runner/work/lamindb/lamindb/docs/test-annotate', type='local', instance_uid='3kW5y8h7c8wG')
  📎 transform: Transform(uid='sU0y1kF3igep6K79', name='Annotate data for developers', key='annotate-for-developers', version='0', type='notebook')
  📎 run: Run(uid='ir7GHMTgACV1mS2jxd8W', started_at=2024-05-01 18:50:17 UTC, is_consecutive=True)
  📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1')
Features:
  var: FeatureSet(uid='KyHezRiMy6ymU3JwsYON', n=3, type='number', registry='bionty.Gene')
    'CD14', 'CD4', 'CD8A'
  obs: FeatureSet(uid='AoOZPq8bTog5A40Oo0Hb', n=2, registry='core.Feature')
    🔗 cell_type (2, bionty.CellType): 'monocyte', 'T cell'
    🔗 tissue (2, bionty.Tissue): 'arterial blood', 'capillary blood'
  external: FeatureSet(uid='0Xehgs2wrPYfr8jdHHuD', n=4, registry='core.Feature')
    🔗 project (1, core.ULabel): 'PRJ0001'
    🔗 organism (1, bionty.Organism): 'human'
    🔗 assay (1, bionty.ExperimentalFactor): 'single-cell RNA sequencing'
    🔗 experiment (1, core.ULabel): 'EXP0001'
Labels:
  📎 organism (1, bionty.Organism): 'human'
  📎 tissues (2, bionty.Tissue): 'arterial blood', 'capillary blood'
  📎 cell_types (2, bionty.CellType): 'monocyte', 'T cell'
  📎 experimental_factors (1, bionty.ExperimentalFactor): 'single-cell RNA sequencing'
  📎 ulabels (2, core.ULabel): 'PRJ0001', 'EXP0001'