Project flow#
LaminDB allows tracking data lineage on the entire project level.
Here, we walk through exemplified app uploads, pipelines & notebooks following Schmidt et al., 2022.
A CRISPR screen reading out a phenotypic endpoint on T cells is paired with scRNA-seq to generate insights into IFN-Ξ³ production.
These insights get linked back to the original data through the steps taken in the project to provide context for interpretation & future decision making.
More specifically: Why should I care about data flow?
Data flow tracks data sources & transformations to trace biological insights, verify experimental outcomes, meet regulatory standards, increase the robustness of research and optimize the feedback loop of team-wide learning iterations.
While tracking data flow is easier when itβs governed by deterministic pipelines, it becomes hard when itβs governed by interactive human-driven analyses.
LaminDB interfaces workflow mangers for the former and embraces the latter.
Setup#
Init a test instance:
!lamin init --storage ./mydata
Show code cell output
π‘ connected lamindb: testuser1/mydata
Import lamindb:
import lamindb as ln
from IPython.display import Image, display
π‘ connected lamindb: testuser1/mydata
Steps#
In the following, we walk through exemplified steps covering different types of transforms (Transform).
Note
The full notebooks are in this repository.
App upload of phenotypic data 
#
Register data through app upload from wetlab by testuser1:
# This function mimics the upload of artifacts via the UI
# In reality, you simply drag and drop files into the UI
def mock_upload_crispra_result_app():
ln.setup.login("testuser1")
transform = ln.Transform(name="Upload GWS CRISPRa result", type="upload")
ln.track(transform=transform)
output_path = ln.core.datasets.schmidt22_crispra_gws_IFNG(ln.settings.storage)
output_file = ln.Artifact(
output_path, description="Raw data of schmidt22 crispra GWS"
)
output_file.save()
mock_upload_crispra_result_app()
Show code cell output
π‘ saved: Transform(uid='QQwqsrLyvIPKaMgW', name='Upload GWS CRISPRa result', type='upload', updated_at=2024-05-01 18:55:20 UTC, created_by_id=1)
π‘ saved: Run(uid='oAUl0urGAdiY04DpBs3Q', transform_id=1, created_by_id=1)
Hit identification in notebook 
#
Access, transform & register data in drylab by testuser2 in notebook hit-identification.
Show code cell content
# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
import nbproject_test
from pathlib import Path
cwd = Path.cwd()
nbproject_test.execute_notebooks(cwd / "project-flow-scripts/hit-identification.ipynb", write=True)
Executing notebooks in /home/runner/work/lamin-usecases/lamin-usecases/docs/project-flow-scripts/hit-identification.ipynb
Scheduled: ['hit-identification']
hit-identification
β (4.979s)
Total time: 4.980s
Inspect data flow:
artifact = ln.Artifact.filter(description="hits from schmidt22 crispra GWS").one()
artifact.view_lineage()
Sequencer upload 
#
Upload files from sequencer via script chromium_10x_upload.py:
!python project-flow-scripts/chromium_10x_upload.py
Show code cell output
π‘ connected lamindb: testuser1/mydata
π‘ saved: Transform(uid='qCJPkOuZAi9q5zKv', name='chromium_10x_upload.py', key='chromium_10x_upload.py', version='1', type='script', updated_at=2024-05-01 18:55:27 UTC, created_by_id=1)
π‘ saved: Run(uid='lim4XMAjU8oHM4Q6WQBk', transform_id=3, created_by_id=1)
β
saved transform.source_code: Artifact(uid='EreEm4QgT1VuTtuL3xY4', suffix='.py', description='Source of transform qCJPkOuZAi9q5zKv', version='1', size=474, hash='o-QoKgEZGxbk5oBtcAKoWw', hash_type='md5', visibility=0, key_is_virtual=True, updated_at=2024-05-01 18:55:28 UTC, storage_id=1, created_by_id=1)
β
saved run.environment: Artifact(uid='o6N9l7ITuBeDLQGKKwV8', suffix='.txt', description='requirements.txt', size=3428, hash='Mlvasak9fYb4NHvNU4nz5w', hash_type='md5', visibility=0, key_is_virtual=True, updated_at=2024-05-01 18:55:28 UTC, storage_id=1, created_by_id=1)
scRNA-seq bioinformatics pipeline 
#
Process uploaded files using a script or workflow manager: Pipelines and obtain 3 output files in a directory filtered_feature_bc_matrix/:
!python project-flow-scripts/cellranger.py
Show code cell output
π‘ connected lamindb: testuser1/mydata
π‘ saved: Transform(uid='cbj0CmT3BGZKOxaQ', name='Cell Ranger', version='7.2.0', type='pipeline', reference='https://www.10xgenomics.com/support/software/cell-ranger/7.2', updated_at=2024-05-01 18:55:30 UTC, created_by_id=2)
π‘ saved: Run(uid='Kc70R0wBFn0nFKGR1Ewc', transform_id=4, created_by_id=2)
β this creates one artifact per file in the directory - you might simply call ln.Artifact(dir) to get one artifact for the entire directory
!python project-flow-scripts/postprocess_cellranger.py
Show code cell output
π‘ connected lamindb: testuser1/mydata
π‘ saved: Transform(uid='YqmbO6oMXjRj65cN', name='postprocess_cellranger.py', key='postprocess_cellranger.py', version='2', type='script', updated_at=2024-05-01 18:55:32 UTC, created_by_id=2)
π‘ saved: Run(uid='hUYl4UZgtbdDWhbi7QKY', transform_id=5, created_by_id=2)
β
saved transform.source_code: Artifact(uid='Rf5crseJcyuN9b3gpEIz', suffix='.py', description='Source of transform YqmbO6oMXjRj65cN', version='2', size=495, hash='iLSbWXZ-j7pkIgzO0i6c0w', hash_type='md5', visibility=0, key_is_virtual=True, updated_at=2024-05-01 18:55:32 UTC, storage_id=1, created_by_id=2)
β returning existing artifact with same hash: Artifact(uid='o6N9l7ITuBeDLQGKKwV8', suffix='.txt', description='requirements.txt', size=3428, hash='Mlvasak9fYb4NHvNU4nz5w', hash_type='md5', visibility=0, key_is_virtual=True, updated_at=2024-05-01 18:55:28 UTC, storage_id=1, created_by_id=1)
β
saved run.environment: Artifact(uid='o6N9l7ITuBeDLQGKKwV8', suffix='.txt', description='requirements.txt', size=3428, hash='Mlvasak9fYb4NHvNU4nz5w', hash_type='md5', visibility=0, key_is_virtual=True, updated_at=2024-05-01 18:55:28 UTC, storage_id=1, created_by_id=1)
Inspect data flow:
output_file = ln.Artifact.filter(description="perturbseq counts").one()
output_file.view_lineage()
Integrate scRNA-seq & phenotypic data #
Integrate data in notebook integrated-analysis.
Show code cell content
# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
nbproject_test.execute_notebooks(cwd / "project-flow-scripts/integrated-analysis.ipynb", write=True)
Executing notebooks in /home/runner/work/lamin-usecases/lamin-usecases/docs/project-flow-scripts/integrated-analysis.ipynb
Scheduled: ['integrated-analysis']
integrated-analysis
β (5.412s)
Total time: 5.413s
Review results#
Letβs load one of the plots:
# track the current notebook as transform
ln.settings.transform.stem_uid = "1LCd8kco9lZU"
ln.settings.transform.version = "0"
ln.track()
π‘ notebook imports: ipython==8.24.0 lamindb==0.71.0 nbproject_test==0.5.1
π‘ saved: Transform(uid='1LCd8kco9lZU6K79', name='Project flow', key='project-flow', version='0', type='notebook', updated_at=2024-05-01 18:55:39 UTC, created_by_id=1)
π‘ saved: Run(uid='hy1nWqAiSflv6ABhYsiD', transform_id=7, created_by_id=1)
artifact = ln.Artifact.filter(key__contains="figures/matrixplot").one()
artifact.cache()
Show code cell output
PosixUPath('/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/aqquE8ClNXYk3ICAeTnI.png')
display(Image(filename=artifact.path))
We see that the image artifact is tracked as an input of the current notebook. The input is highlighted, the notebook follows at the bottom:
artifact.view_lineage()
Alternatively, we can also look at the sequence of transforms:
transform = ln.Transform.search("Project flow", return_queryset=True).first()
transform.parents.df()
| uid | name | key | version | description | type | latest_report_id | source_code_id | reference | reference_type | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||
| 6 | lB3IyPLQSmvt5zKv | Perform single cell analysis, integrate with C... | integrated-analysis | 1 | None | notebook | None | None | None | None | 2024-05-01 18:55:37.414064+00:00 | 2024-05-01 18:55:37.414092+00:00 | 2 |
transform.view_parents()
Understand runs#
We tracked pipeline and notebook runs through run_context, which stores a Transform and a Run record as a global context.
Artifact objects are the inputs and outputs of runs.
What if I donβt want a global context?
Sometimes, we donβt want to create a global run context but manually pass a run when creating an artifact:
run = ln.Run(transform=transform)
ln.Artifact(filepath, run=run)
When does an artifact appear as a run input?
When accessing an artifact via cache(), load() or backed(), two things happen:
The current run gets added to
artifact.input_ofThe transform of that artifact gets added as a parent of the current transform
You can then switch off auto-tracking of run inputs if you set ln.settings.track_run_inputs = False: Can I disable tracking run inputs?
You can also track run inputs on a case by case basis via is_run_input=True, e.g., here:
artifact.load(is_run_input=True)
Query by provenance#
We can query or search for the notebook that created the artifact:
transform = ln.Transform.search("GWS CRIPSRa analysis", return_queryset=True).first()
And then find all the artifacts created by that notebook:
ln.Artifact.filter(transform=transform).df()
| uid | storage_id | key | suffix | accessor | description | version | size | hash | hash_type | n_objects | n_observations | transform_id | run_id | visibility | key_is_virtual | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||||||||
| 2 | Mvjkzw1yyx2iMbIOQkK7 | 1 | None | .parquet | DataFrame | hits from schmidt22 crispra GWS | None | 18368 | PihzyuN-FWc-ld6ioxAuPg | md5 | None | None | 2 | 2 | 1 | True | 2024-05-01 18:55:25.432309+00:00 | 2024-05-01 18:55:25.432337+00:00 | 1 |
Which transform ingested a given artifact?
artifact = ln.Artifact.filter().first()
artifact.transform
Transform(uid='QQwqsrLyvIPKaMgW', name='Upload GWS CRISPRa result', type='upload', updated_at=2024-05-01 18:55:20 UTC, created_by_id=1)
And which user?
artifact.created_by
User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2024-05-01 18:55:27 UTC)
Which transforms were created by a given user?
users = ln.User.lookup()
ln.Transform.filter(created_by=users.testuser1).df()
| uid | name | key | version | description | type | latest_report_id | source_code_id | reference | reference_type | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||
| 1 | QQwqsrLyvIPKaMgW | Upload GWS CRISPRa result | None | None | None | upload | None | NaN | None | None | 2024-05-01 18:55:20.177429+00:00 | 2024-05-01 18:55:20.177451+00:00 | 1 |
| 2 | T0T28btuB0PG5zKv | GWS CRIPSRa analysis | hit-identification | 1 | None | notebook | None | NaN | None | None | 2024-05-01 18:55:24.910170+00:00 | 2024-05-01 18:55:24.910202+00:00 | 1 |
| 3 | qCJPkOuZAi9q5zKv | chromium_10x_upload.py | chromium_10x_upload.py | 1 | None | script | None | 3.0 | None | None | 2024-05-01 18:55:27.710569+00:00 | 2024-05-01 18:55:28.173337+00:00 | 1 |
| 7 | 1LCd8kco9lZU6K79 | Project flow | project-flow | 0 | None | notebook | None | NaN | None | None | 2024-05-01 18:55:39.290726+00:00 | 2024-05-01 18:55:39.290770+00:00 | 1 |
Which notebooks were created by a given user?
ln.Transform.filter(created_by=users.testuser1, type="notebook").df()
| uid | name | key | version | description | type | latest_report_id | source_code_id | reference | reference_type | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||
| 2 | T0T28btuB0PG5zKv | GWS CRIPSRa analysis | hit-identification | 1 | None | notebook | None | None | None | None | 2024-05-01 18:55:24.910170+00:00 | 2024-05-01 18:55:24.910202+00:00 | 1 |
| 7 | 1LCd8kco9lZU6K79 | Project flow | project-flow | 0 | None | notebook | None | None | None | None | 2024-05-01 18:55:39.290726+00:00 | 2024-05-01 18:55:39.290770+00:00 | 1 |
We can also view all recent additions to the entire database:
ln.view()
Show code cell output
Artifact
| uid | storage_id | key | suffix | accessor | description | version | size | hash | hash_type | n_objects | n_observations | transform_id | run_id | visibility | key_is_virtual | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||||||||
| 13 | aqquE8ClNXYk3ICAeTnI | 1 | figures/matrixplot_fig2_score-wgs-hits-per-clu... | .png | None | None | None | 28814 | 8zXF_cVwaZnfhmrLbt_0kA | md5 | None | None | 6 | 6 | 1 | True | 2024-05-01 18:55:38.410627+00:00 | 2024-05-01 18:55:38.410653+00:00 | 2 |
| 12 | FCS0mPjsIsGZdXZC5btl | 1 | figures/umap_fig1_score-wgs-hits.png | .png | None | None | None | 118999 | DCFDLUMF-UohaBvkThn0mA | md5 | None | None | 6 | 6 | 1 | True | 2024-05-01 18:55:38.174421+00:00 | 2024-05-01 18:55:38.174449+00:00 | 2 |
| 11 | jekVLajdkX8fE594cQj7 | 1 | schmidt22_perturbseq.h5ad | .h5ad | AnnData | perturbseq counts | None | 20659936 | la7EvqEUMDlug9-rpw-udA | md5 | None | None | 5 | 5 | 1 | False | 2024-05-01 18:55:33.176349+00:00 | 2024-05-01 18:55:33.176380+00:00 | 2 |
| 9 | IuX7bTGS7feLI8DG4hm5 | 1 | perturbseq/filtered_feature_bc_matrix/matrix.m... | .mtx.gz | None | None | None | 6 | I4p4WTDogZhyvrZvuJFl_g | md5 | None | None | 4 | 4 | 1 | False | 2024-05-01 18:55:30.631726+00:00 | 2024-05-01 18:55:30.631743+00:00 | 2 |
| 8 | thmd93aIemMTC3CcdH6w | 1 | perturbseq/filtered_feature_bc_matrix/features... | .tsv.gz | None | None | None | 6 | nIumkWjG_tmPNVal1oynlw | md5 | None | None | 4 | 4 | 1 | False | 2024-05-01 18:55:30.631135+00:00 | 2024-05-01 18:55:30.631153+00:00 | 2 |
| 7 | 1eBgoQWiXXzJxB5WX7Fh | 1 | perturbseq/filtered_feature_bc_matrix/barcodes... | .tsv.gz | None | None | None | 6 | Y9jyUja6_Awk50ePVSB65g | md5 | None | None | 4 | 4 | 1 | False | 2024-05-01 18:55:30.630340+00:00 | 2024-05-01 18:55:30.630364+00:00 | 2 |
| 6 | ohUn4yuyAdjbNR4wXlcD | 1 | fastq/perturbseq_R2_001.fastq.gz | .fastq.gz | None | None | None | 6 | lAUlZE3Hit7HxPK_ur28Sg | md5 | None | None | 3 | 3 | 1 | False | 2024-05-01 18:55:28.182489+00:00 | 2024-05-01 18:55:28.182507+00:00 | 1 |
Run
| uid | transform_id | started_at | finished_at | created_by_id | json | report_id | environment_id | is_consecutive | reference | reference_type | created_at | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||
| 1 | oAUl0urGAdiY04DpBs3Q | 1 | 2024-05-01 18:55:20.180828+00:00 | NaT | 1 | None | None | NaN | True | None | None | 2024-05-01 18:55:20.181016+00:00 |
| 2 | 1xI675djNYYNXHFxTvqF | 2 | 2024-05-01 18:55:24.915907+00:00 | NaT | 1 | None | None | NaN | True | None | None | 2024-05-01 18:55:24.916005+00:00 |
| 3 | lim4XMAjU8oHM4Q6WQBk | 3 | 2024-05-01 18:55:27.712943+00:00 | 2024-05-01 18:55:28.184144+00:00 | 1 | None | None | 4.0 | None | None | None | 2024-05-01 18:55:27.713033+00:00 |
| 4 | Kc70R0wBFn0nFKGR1Ewc | 4 | 2024-05-01 18:55:30.163115+00:00 | NaT | 2 | None | None | NaN | None | None | None | 2024-05-01 18:55:30.163214+00:00 |
| 5 | hUYl4UZgtbdDWhbi7QKY | 5 | 2024-05-01 18:55:32.275884+00:00 | NaT | 2 | None | None | 4.0 | None | None | None | 2024-05-01 18:55:32.275975+00:00 |
| 6 | iyQnUnQsuohKP40CVPym | 6 | 2024-05-01 18:55:37.419993+00:00 | NaT | 2 | None | None | NaN | True | None | None | 2024-05-01 18:55:37.420092+00:00 |
| 7 | hy1nWqAiSflv6ABhYsiD | 7 | 2024-05-01 18:55:39.296535+00:00 | NaT | 1 | None | None | NaN | True | None | None | 2024-05-01 18:55:39.296713+00:00 |
Storage
| uid | root | description | type | region | instance_uid | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||
| 1 | tcDlgs2T6ydS | /home/runner/work/lamin-usecases/lamin-usecase... | None | local | None | 54ZGqgkROOFf | 2024-05-01 18:55:18.451437+00:00 | 2024-05-01 18:55:18.451457+00:00 | 1 |
Transform
| uid | name | key | version | description | type | latest_report_id | source_code_id | reference | reference_type | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||
| 7 | 1LCd8kco9lZU6K79 | Project flow | project-flow | 0 | None | notebook | None | NaN | None | None | 2024-05-01 18:55:39.290726+00:00 | 2024-05-01 18:55:39.290770+00:00 | 1 |
| 6 | lB3IyPLQSmvt5zKv | Perform single cell analysis, integrate with C... | integrated-analysis | 1 | None | notebook | None | NaN | None | None | 2024-05-01 18:55:37.414064+00:00 | 2024-05-01 18:55:37.414092+00:00 | 2 |
| 5 | YqmbO6oMXjRj65cN | postprocess_cellranger.py | postprocess_cellranger.py | 2 | None | script | None | 10.0 | None | None | 2024-05-01 18:55:32.273251+00:00 | 2024-05-01 18:55:32.736121+00:00 | 2 |
| 4 | cbj0CmT3BGZKOxaQ | Cell Ranger | None | 7.2.0 | None | pipeline | None | NaN | https://www.10xgenomics.com/support/software/c... | None | 2024-05-01 18:55:30.160078+00:00 | 2024-05-01 18:55:30.160100+00:00 | 2 |
| 3 | qCJPkOuZAi9q5zKv | chromium_10x_upload.py | chromium_10x_upload.py | 1 | None | script | None | 3.0 | None | None | 2024-05-01 18:55:27.710569+00:00 | 2024-05-01 18:55:28.173337+00:00 | 1 |
| 2 | T0T28btuB0PG5zKv | GWS CRIPSRa analysis | hit-identification | 1 | None | notebook | None | NaN | None | None | 2024-05-01 18:55:24.910170+00:00 | 2024-05-01 18:55:24.910202+00:00 | 1 |
| 1 | QQwqsrLyvIPKaMgW | Upload GWS CRISPRa result | None | None | None | upload | None | NaN | None | None | 2024-05-01 18:55:20.177429+00:00 | 2024-05-01 18:55:20.177451+00:00 | 1 |
User
| uid | handle | name | created_at | updated_at | |
|---|---|---|---|---|---|
| id | |||||
| 2 | bKeW4T6E | testuser2 | Test User2 | 2024-05-01 18:55:30.150208+00:00 | 2024-05-01 18:55:30.150247+00:00 |
| 1 | DzTjkKse | testuser1 | Test User1 | 2024-05-01 18:55:18.448058+00:00 | 2024-05-01 18:55:27.579359+00:00 |
Show code cell content
!lamin login testuser1
!lamin delete --force mydata
!rm -r ./mydata
β
logged in with email [email protected] (uid: DzTjkKse)
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.10.14/x64/bin/lamin", line 8, in <module>
sys.exit(main())
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 360, in __call__
return super().__call__(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 152, in main
rv = self.invoke(ctx)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamin_cli/__main__.py", line 103, in delete
return delete(instance, force=force)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/_delete.py", line 140, in delete
n_objects = check_storage_is_empty(
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/core/upath.py", line 814, in check_storage_is_empty
raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb contains 7 objects ('./lamindb/_is_initialized' ignored) - delete them prior to deleting the instance
['/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/EreEm4QgT1VuTtuL3xY4.py', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/FCS0mPjsIsGZdXZC5btl.png', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/Mvjkzw1yyx2iMbIOQkK7.parquet', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/Rf5crseJcyuN9b3gpEIz.py', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/_is_initialized', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/aqquE8ClNXYk3ICAeTnI.png', '/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/o6N9l7ITuBeDLQGKKwV8.txt']