Are LaminDB’s operations ACID?#

Yes, they are ACID.

Here, we walk through different errors that can occur while saving files & metadata records, and show that the LaminDB instance does not get corrupted by dangling metadata or files.

from laminci.db import setup_local_test_postgres

pgurl = setup_local_test_postgres()
!lamin init --db {pgurl} --storage ./test-acid

💬 Created Postgres test instance: 'postgresql://postgres:pwd@0.0.0.0:5432/pgtest'
It runs in docker container 'pgtest'

✅ saved: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-09-26 15:20:33)
✅ saved: Storage(id='kAipYZcw', root='/home/runner/work/lamindb/lamindb/docs/faq/test-acid', type='local', updated_at=2023-09-26 15:20:33, created_by_id='DzTjkKse')
💡 loaded instance: testuser1/pgtest
💡 did not register local instance on hub (if you want, call `lamin register`)

import pytest
import lamindb as ln
from upath import UPath
from django.db.utils import IntegrityError

💡 loaded instance: testuser1/pgtest (lamindb 0.54.2)

Save error due to failed upload#

Let’s try to save a file to a storage location without permission.

file = ln.File(
    ln.dev.datasets.anndata_mouse_sc_lymph_node(),
    description="Mouse Lymph Node scRNA-seq",
)

❗ no run & transform get linked, consider passing a `run` or calling ln.track()

Because the public API only allows you to set a default storage for which you have permission, we need to hack it:

ln.setup.settings.storage._root = UPath("s3://nf-core-awsmegatests")

ln.settings.storage

S3Path('s3://nf-core-awsmegatests/')

This raises a RuntimeError:

with pytest.raises(RuntimeError) as error:
    file.save()
print(error.exconly())

❗ could not upload file: File(id='SZqzZvZVEZXfhg5uzmAY', suffix='.h5ad', accessor='AnnData', description='Mouse Lymph Node scRNA-seq', size=17177479, hash='7ni-11ZJqq0h3LKvS2NcyQ', hash_type='md5', updated_at=2023-09-26 15:20:36, storage_id='kAipYZcw', created_by_id='DzTjkKse')

RuntimeError: Unable to locate credentials

Let’s now check that no metadata records were added to the database:

assert len(ln.File.filter().all()) == 0

Save error during bulk creation#

Error during metadata save#

filepath = ln.dev.datasets.file_jpg_paradisi05()
file = ln.File(filepath, description="My image")
files = [file, "this is not an Registry"]

❗ no run & transform get linked, consider passing a `run` or calling ln.track()

This raises an exception:

with pytest.raises(Exception) as error:
    ln.save(files)
print(error.exconly())

AttributeError: type object 'str' has no attribute 'objects'

Nothing got saved:

files = ln.File.filter().all()
assert len(files) == 0

Error in one of the uploads#

If a list of data objects is passed to ln.save() and the upload of one of these data objects fails, the successful uploads are maintained and a RuntimeError is raised, listing the successfully uploaded data objects up until that point.

Need a proper test here!