Project flow#
LaminDB allows tracking data flow on the entire project level.
Here, we walk through exemplified app uploads, pipelines & notebooks following Schmidt et al., 2022.
A CRISPR screen reading out a phenotypic endpoint on T cells is paired with scRNA-seq to generate insights into IFN-γ production.
These insights get linked back to the original data through the steps taken in the project to provide context for interpretation & future decision making.
More specifically: Why should I care about data flow?
Data flow tracks data sources & transformations to trace biological insights, verify experimental outcomes, meet regulatory standards, increase the robustness of research and optimize the feedback loop of team-wide learning iterations.
While tracking data flow is easier when it’s governed by deterministic pipelines, it becomes hard when it’s governed by interactive human-driven analyses.
LaminDB interfaces workflow mangers for the former and embraces the latter.
Setup#
Init a test instance:
!lamin init --storage ./mydata
Show code cell output
✅ saved: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-09-26 15:24:06)
✅ saved: Storage(id='t60LfwYY', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata', type='local', updated_at=2023-09-26 15:24:06, created_by_id='DzTjkKse')
💡 loaded instance: testuser1/mydata
💡 did not register local instance on hub (if you want, call `lamin register`)
Import lamindb:
import lamindb as ln
from IPython.display import Image, display
💡 loaded instance: testuser1/mydata (lamindb 0.54.2)
Steps#
In the following, we walk through exemplified steps covering different types of transforms (Transform).
Note
The full notebooks are in this repository.
App upload of phenotypic data 
#
Register data through app upload from wetlab by testuser1:
ln.setup.login("testuser1")
transform = ln.Transform(name="Upload GWS CRISPRa result", type="app")
ln.track(transform)
output_path = ln.dev.datasets.schmidt22_crispra_gws_IFNG(ln.settings.storage)
output_file = ln.File(output_path, description="Raw data of schmidt22 crispra GWS")
output_file.save()
Show code cell output
💡 Transform(id='WzNj574LRsfOGx', name='Upload GWS CRISPRa result', type='app', updated_at=2023-09-26 15:24:08, created_by_id='DzTjkKse')
💡 Run(id='Qc3O79udiQn1XzWrVXbZ', run_at=2023-09-26 15:24:08, transform_id='WzNj574LRsfOGx', created_by_id='DzTjkKse')
Hit identification in notebook 
#
Access, transform & register data in drylab by testuser2:
ln.setup.login("testuser2")
transform = ln.Transform(name="GWS CRIPSRa analysis", type="notebook")
ln.track(transform)
# access
input_file = ln.File.filter(key="schmidt22-crispra-gws-IFNG.csv").one()
# identify hits
input_df = input_file.load().set_index("id")
output_df = input_df[input_df["pos|fdr"] < 0.01].copy()
# register hits in output file
ln.File(output_df, description="hits from schmidt22 crispra GWS").save()
Show code cell output
💡 Transform(id='KyvY5NSnkfyd3K', name='GWS CRIPSRa analysis', type='notebook', updated_at=2023-09-26 15:24:10, created_by_id='bKeW4T6E')
💡 Run(id='wMK63rvko4na7aE6wIRA', run_at=2023-09-26 15:24:10, transform_id='KyvY5NSnkfyd3K', created_by_id='bKeW4T6E')
Inspect data flow:
file = ln.File.filter(description="hits from schmidt22 crispra GWS").one()
file.view_flow()
Sequencer upload 
#
Upload files from sequencer:
ln.setup.login("testuser1")
ln.track(ln.Transform(name="Chromium 10x upload", type="pipeline"))
# register output files of upload
upload_dir = ln.dev.datasets.dir_scrnaseq_cellranger(
"perturbseq", basedir=ln.settings.storage, output_only=False
)
ln.File(upload_dir.parent / "fastq/perturbseq_R1_001.fastq.gz").save()
ln.File(upload_dir.parent / "fastq/perturbseq_R2_001.fastq.gz").save()
ln.setup.login("testuser2")
Show code cell output
💡 Transform(id='2kKeIFVw0V5Iyr', name='Chromium 10x upload', type='pipeline', updated_at=2023-09-26 15:24:11, created_by_id='DzTjkKse')
💡 Run(id='SiCGpZkoi12cI28H5aam', run_at=2023-09-26 15:24:11, transform_id='2kKeIFVw0V5Iyr', created_by_id='DzTjkKse')
❗ file has more than one suffix (path.suffixes), inferring: '.fastq.gz'
❗ file has more than one suffix (path.suffixes), inferring: '.fastq.gz'
scRNA-seq bioinformatics pipeline #
Process uploaded files using a script or workflow manager: Pipelines and obtain 3 output files in a directory filtered_feature_bc_matrix/:
transform = ln.Transform(name="Cell Ranger", version="7.2.0", type="pipeline")
ln.track(transform)
# access uploaded files as inputs for the pipeline
input_files = ln.File.filter(key__startswith="fastq/perturbseq").all()
input_paths = [file.stage() for file in input_files]
# register output files
output_files = ln.File.from_dir("./mydata/perturbseq/filtered_feature_bc_matrix/")
ln.save(output_files)
Show code cell output
💡 Transform(id='ZrAUOqqQjds9fk', name='Cell Ranger', version='7.2.0', type='pipeline', updated_at=2023-09-26 15:24:12, created_by_id='bKeW4T6E')
💡 Run(id='gjDRYLDekUY4jUtwkRVY', run_at=2023-09-26 15:24:12, transform_id='ZrAUOqqQjds9fk', created_by_id='bKeW4T6E')
❗ file has more than one suffix (path.suffixes), inferring: '.mtx.gz'
❗ file has more than one suffix (path.suffixes), inferring: '.tsv.gz'
❗ file has more than one suffix (path.suffixes), inferring: '.tsv.gz'
Post-process these 3 files:
transform = ln.Transform(name="Postprocess Cell Ranger", version="2.0", type="pipeline")
ln.track(transform)
input_files = [f.stage() for f in output_files]
output_path = ln.dev.datasets.schmidt22_perturbseq(basedir=ln.settings.storage)
output_file = ln.File(output_path, description="perturbseq counts")
output_file.save()
Show code cell output
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| Cell Ranger | ZrAUOqqQjds9fk | 90.0 |
💡 Transform(id='78JHRtYKQPBQIF', name='Postprocess Cell Ranger', version='2.0', type='pipeline', updated_at=2023-09-26 15:24:12, created_by_id='bKeW4T6E')
💡 Run(id='FSUn8cepdKzqWOhF8cIp', run_at=2023-09-26 15:24:12, transform_id='78JHRtYKQPBQIF', created_by_id='bKeW4T6E')
Inspect data flow:
output_files[0].view_flow()
Integrate scRNA-seq & phenotypic data #
Integrate data in a notebook:
transform = ln.Transform(
name="Perform single cell analysis, integrate with CRISPRa screen",
type="notebook",
)
ln.track(transform)
file_ps = ln.File.filter(description__icontains="perturbseq").one()
adata = file_ps.load()
file_hits = ln.File.filter(description="hits from schmidt22 crispra GWS").one()
screen_hits = file_hits.load()
import scanpy as sc
sc.tl.score_genes(adata, adata.var_names.intersection(screen_hits.index).tolist())
filesuffix = "_fig1_score-wgs-hits.png"
sc.pl.umap(adata, color="score", show=False, save=filesuffix)
filepath = f"figures/umap{filesuffix}"
file = ln.File(filepath, key=filepath)
file.save()
filesuffix = "fig2_score-wgs-hits-per-cluster.png"
sc.pl.matrixplot(
adata, groupby="cluster_name", var_names=["score"], show=False, save=filesuffix
)
filepath = f"figures/matrixplot_{filesuffix}"
file = ln.File(filepath, key=filepath)
file.save()
Show code cell output
💡 Transform(id='CmNh72dqiCaQyr', name='Perform single cell analysis, integrate with CRISPRa screen', type='notebook', updated_at=2023-09-26 15:24:13, created_by_id='bKeW4T6E')
💡 Run(id='LhHJaAg03NQ6jzLkJpag', run_at=2023-09-26 15:24:13, transform_id='CmNh72dqiCaQyr', created_by_id='bKeW4T6E')
WARNING: saving figure to file figures/umap_fig1_score-wgs-hits.png
WARNING: saving figure to file figures/matrixplot_fig2_score-wgs-hits-per-cluster.png
Review results#
Let’s load one of the plots:
ln.track()
file = ln.File.filter(key__contains="figures/matrixplot").one()
file.stage()
Show code cell output
💡 notebook imports: ipython==8.15.0 lamindb==0.54.2 scanpy==1.9.5
💡 Transform(id='1LCd8kco9lZUz8', name='Project flow', short_name='project-flow', version='0', type=notebook, updated_at=2023-09-26 15:24:15, created_by_id='bKeW4T6E')
💡 Run(id='YhkmWqDDDoXbNl4M0Wxl', run_at=2023-09-26 15:24:15, transform_id='1LCd8kco9lZUz8', created_by_id='bKeW4T6E')
PosixUPath('/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/figures/matrixplot_fig2_score-wgs-hits-per-cluster.png')
display(Image(filename=file.path))
We see that the image file is tracked as an input of the current notebook. The input is highlighted, the notebook follows at the bottom:
file.view_flow()
Alternatively, we can also look at the sequence of transforms:
transform = ln.Transform.search("Bird's eye view", return_queryset=True).first()
transform.parents.df()
| name | short_name | version | type | reference | reference_type | initial_version_id | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||
| 78JHRtYKQPBQIF | Postprocess Cell Ranger | None | 2.0 | pipeline | None | None | None | 2023-09-26 15:24:13 | bKeW4T6E |
| KyvY5NSnkfyd3K | GWS CRIPSRa analysis | None | None | notebook | None | None | None | 2023-09-26 15:24:10 | bKeW4T6E |
transform.view_parents()
Understand runs#
We tracked pipeline and notebook runs through run_context, which stores a Transform and a Run record as a global context.
File objects are the inputs and outputs of runs.
What if I don’t want a global context?
Sometimes, we don’t want to create a global run context but manually pass a run when creating a file:
run = ln.Run(transform=transform)
ln.File(filepath, run=run)
When does a file appear as a run input?
When accessing a file via stage(), load() or backed(), two things happen:
The current run gets added to
file.input_ofThe transform of that file gets added as a parent of the current transform
You can then switch off auto-tracking of run inputs if you set ln.settings.track_run_inputs = False: Can I disable tracking run inputs?
You can also track run inputs on a case by case basis via is_run_input=True, e.g., here:
file.load(is_run_input=True)
Query by provenance#
We can query or search for the notebook that created the file:
transform = ln.Transform.search("GWS CRIPSRa analysis", return_queryset=True).first()
And then find all the files created by that notebook:
ln.File.filter(transform=transform).df()
| storage_id | key | suffix | accessor | description | version | size | hash | hash_type | transform_id | run_id | initial_version_id | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| 319OvPJPoUGHOnDtk08O | t60LfwYY | None | .parquet | DataFrame | hits from schmidt22 crispra GWS | None | 18368 | O2Owo0_QlM9JBS2zAZD4Lw | md5 | KyvY5NSnkfyd3K | wMK63rvko4na7aE6wIRA | None | 2023-09-26 15:24:10 | bKeW4T6E |
Which transform ingested a given file?
file = ln.File.filter().first()
file.transform
Transform(id='WzNj574LRsfOGx', name='Upload GWS CRISPRa result', type='app', updated_at=2023-09-26 15:24:09, created_by_id='DzTjkKse')
And which user?
file.created_by
User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-09-26 15:24:11)
Which transforms were created by a given user?
users = ln.User.lookup()
ln.Transform.filter(created_by=users.testuser2).df()
| name | short_name | version | type | reference | reference_type | initial_version_id | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||
| KyvY5NSnkfyd3K | GWS CRIPSRa analysis | None | None | notebook | None | None | None | 2023-09-26 15:24:10 | bKeW4T6E |
| ZrAUOqqQjds9fk | Cell Ranger | None | 7.2.0 | pipeline | None | None | None | 2023-09-26 15:24:12 | bKeW4T6E |
| 78JHRtYKQPBQIF | Postprocess Cell Ranger | None | 2.0 | pipeline | None | None | None | 2023-09-26 15:24:13 | bKeW4T6E |
| CmNh72dqiCaQyr | Perform single cell analysis, integrate with C... | None | None | notebook | None | None | None | 2023-09-26 15:24:15 | bKeW4T6E |
| 1LCd8kco9lZUz8 | Project flow | project-flow | 0 | notebook | None | None | None | 2023-09-26 15:24:15 | bKeW4T6E |
Which notebooks were created by a given user?
ln.Transform.filter(created_by=users.testuser2, type="notebook").df()
| name | short_name | version | type | reference | reference_type | initial_version_id | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||
| KyvY5NSnkfyd3K | GWS CRIPSRa analysis | None | None | notebook | None | None | None | 2023-09-26 15:24:10 | bKeW4T6E |
| CmNh72dqiCaQyr | Perform single cell analysis, integrate with C... | None | None | notebook | None | None | None | 2023-09-26 15:24:15 | bKeW4T6E |
| 1LCd8kco9lZUz8 | Project flow | project-flow | 0 | notebook | None | None | None | 2023-09-26 15:24:15 | bKeW4T6E |
We can also view all recent additions to the entire database:
ln.view()
Show code cell output
File
| storage_id | key | suffix | accessor | description | version | size | hash | hash_type | transform_id | run_id | initial_version_id | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| 9qTiOBFiDUIhBxv3pENf | t60LfwYY | figures/matrixplot_fig2_score-wgs-hits-per-clu... | .png | None | None | None | 28814 | H0Pxpa-fZOvigo74eXHZsQ | md5 | CmNh72dqiCaQyr | LhHJaAg03NQ6jzLkJpag | None | 2023-09-26 15:24:15 | bKeW4T6E |
| zT5djSoVJGWdiIP29S0k | t60LfwYY | figures/umap_fig1_score-wgs-hits.png | .png | None | None | None | 118999 | 1-WtAvRL1d_SSjZvMMOMkg | md5 | CmNh72dqiCaQyr | LhHJaAg03NQ6jzLkJpag | None | 2023-09-26 15:24:15 | bKeW4T6E |
| dm8SQVRG8umTMt81T1at | t60LfwYY | schmidt22_perturbseq.h5ad | .h5ad | AnnData | perturbseq counts | None | 20659936 | la7EvqEUMDlug9-rpw-udA | md5 | 78JHRtYKQPBQIF | FSUn8cepdKzqWOhF8cIp | None | 2023-09-26 15:24:13 | bKeW4T6E |
| tZx4t3mJL9msKuCnJFK3 | t60LfwYY | perturbseq/filtered_feature_bc_matrix/features... | .tsv.gz | None | None | None | 6 | PwzgqtHw8Bx5iOSHXkZiZA | md5 | ZrAUOqqQjds9fk | gjDRYLDekUY4jUtwkRVY | None | 2023-09-26 15:24:12 | bKeW4T6E |
| WpyiFu0Y42fJaUdCOUJT | t60LfwYY | perturbseq/filtered_feature_bc_matrix/matrix.m... | .mtx.gz | None | None | None | 6 | 1Ew4LqYXSfq5qxD0_xRSmw | md5 | ZrAUOqqQjds9fk | gjDRYLDekUY4jUtwkRVY | None | 2023-09-26 15:24:12 | bKeW4T6E |
| Q9Q8Aclu2PZhMXKiXuoR | t60LfwYY | perturbseq/filtered_feature_bc_matrix/barcodes... | .tsv.gz | None | None | None | 6 | OOwZXeC6vWB-H-rR9AEDiw | md5 | ZrAUOqqQjds9fk | gjDRYLDekUY4jUtwkRVY | None | 2023-09-26 15:24:12 | bKeW4T6E |
| 6YCZrgtqSWG5toMdDTt8 | t60LfwYY | fastq/perturbseq_R2_001.fastq.gz | .fastq.gz | None | None | None | 6 | fI0-Kt-Bo1z56V2dhLc1tg | md5 | 2kKeIFVw0V5Iyr | SiCGpZkoi12cI28H5aam | None | 2023-09-26 15:24:11 | DzTjkKse |
Run
| transform_id | run_at | created_by_id | reference | reference_type | |
|---|---|---|---|---|---|
| id | |||||
| Qc3O79udiQn1XzWrVXbZ | WzNj574LRsfOGx | 2023-09-26 15:24:08 | DzTjkKse | None | None |
| wMK63rvko4na7aE6wIRA | KyvY5NSnkfyd3K | 2023-09-26 15:24:10 | bKeW4T6E | None | None |
| SiCGpZkoi12cI28H5aam | 2kKeIFVw0V5Iyr | 2023-09-26 15:24:11 | DzTjkKse | None | None |
| gjDRYLDekUY4jUtwkRVY | ZrAUOqqQjds9fk | 2023-09-26 15:24:12 | bKeW4T6E | None | None |
| FSUn8cepdKzqWOhF8cIp | 78JHRtYKQPBQIF | 2023-09-26 15:24:12 | bKeW4T6E | None | None |
| LhHJaAg03NQ6jzLkJpag | CmNh72dqiCaQyr | 2023-09-26 15:24:13 | bKeW4T6E | None | None |
| YhkmWqDDDoXbNl4M0Wxl | 1LCd8kco9lZUz8 | 2023-09-26 15:24:15 | bKeW4T6E | None | None |
Storage
| root | type | region | updated_at | created_by_id | |
|---|---|---|---|---|---|
| id | |||||
| t60LfwYY | /home/runner/work/lamin-usecases/lamin-usecase... | local | None | 2023-09-26 15:24:06 | DzTjkKse |
Transform
| name | short_name | version | type | reference | reference_type | initial_version_id | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||
| 1LCd8kco9lZUz8 | Project flow | project-flow | 0 | notebook | None | None | None | 2023-09-26 15:24:15 | bKeW4T6E |
| CmNh72dqiCaQyr | Perform single cell analysis, integrate with C... | None | None | notebook | None | None | None | 2023-09-26 15:24:15 | bKeW4T6E |
| 78JHRtYKQPBQIF | Postprocess Cell Ranger | None | 2.0 | pipeline | None | None | None | 2023-09-26 15:24:13 | bKeW4T6E |
| ZrAUOqqQjds9fk | Cell Ranger | None | 7.2.0 | pipeline | None | None | None | 2023-09-26 15:24:12 | bKeW4T6E |
| 2kKeIFVw0V5Iyr | Chromium 10x upload | None | None | pipeline | None | None | None | 2023-09-26 15:24:11 | DzTjkKse |
| KyvY5NSnkfyd3K | GWS CRIPSRa analysis | None | None | notebook | None | None | None | 2023-09-26 15:24:10 | bKeW4T6E |
| WzNj574LRsfOGx | Upload GWS CRISPRa result | None | None | app | None | None | None | 2023-09-26 15:24:09 | DzTjkKse |
User
| handle | name | updated_at | ||
|---|---|---|---|---|
| id | ||||
| bKeW4T6E | testuser2 | testuser2@lamin.ai | Test User2 | 2023-09-26 15:24:12 |
| DzTjkKse | testuser1 | testuser1@lamin.ai | Test User1 | 2023-09-26 15:24:11 |
Show code cell content
!lamin login testuser1
!lamin delete --force mydata
!rm -r ./mydata
✅ logged in with email testuser1@lamin.ai and id DzTjkKse
💡 deleting instance testuser1/mydata
✅ deleted instance settings file: /home/runner/.lamin/instance--testuser1--mydata.env
✅ instance cache deleted
✅ deleted '.lndb' sqlite file
❗ consider manually deleting your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/mydata