Keep artifacts local in a cloud instance¶

If you want to default to keeping artifacts local in a cloud instance, enable keep_artifacts_local.

import lamindb as ln

ln.settings.transform.stem_uid = "l9lFf83aPwRc"
ln.settings.transform.version = "1"
ln.track()

💡 connected lamindb: testuser1/keep-artifacts-local

💡 notebook imports: lamindb==0.71.3

💡 saved: Transform(version='1', uid='l9lFf83aPwRc5zKv', name='Keep artifacts local in a cloud instance', key='keep-artifacts-local', type='notebook', updated_at=2024-05-14 15:59:25 UTC, created_by_id=1)

💡 saved: Run(uid='QE78OgN38DJ8yYR9Ji12', transform_id=1, created_by_id=1)

You can register a managed local storage location as follows:

ln.settings.storage_local = "./my_storage_local"

💡 defaulting to local storage: /home/runner/work/lamindb/lamindb/docs/faq/my_storage_local

Now, you have two storage locations: one in the S3 bucket, and the other locally.

ln.Storage.df()

Show code cell output Hide code cell output

	uid	root	description	type	region	instance_uid	created_at	updated_at	created_by_id
id
2	kBFUXkGKnOE0	/home/runner/work/lamindb/lamindb/docs/faq/my_...	None	local	None	6uGWmLpZlNoJ	2024-05-14 15:59:26.545238+00:00	2024-05-14 15:59:26.545271+00:00	1
1	TnhnUxvh21Pj	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ	2024-05-14 15:59:21.859227+00:00	2024-05-14 15:59:21.859250+00:00	1

Update storage description¶

You can add a description to the storage by using the description parameter:

storage_record = ln.Storage.filter(root=ln.settings.storage_local).one()
storage_record.description = "Files stored locally in site X on server Y for reason ABC"
storage_record.save()
ln.Storage.df()

	uid	root	description	type	region	instance_uid	created_at	updated_at	created_by_id
id
2	kBFUXkGKnOE0	/home/runner/work/lamindb/lamindb/docs/faq/my_...	Files stored locally in site X on server Y for...	local	None	6uGWmLpZlNoJ	2024-05-14 15:59:26.545238+00:00	2024-05-14 15:59:26.580815+00:00	1
1	TnhnUxvh21Pj	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ	2024-05-14 15:59:21.859227+00:00	2024-05-14 15:59:21.859250+00:00	1

Use local storage¶

If you save an artifact, by default, it’s stored in local storage.

original_filepath = ln.core.datasets.file_fcs()
artifact = ln.Artifact(original_filepath, description="My fcs file").save()
local_path = artifact.path
local_path

You’ll see the .fcs file named by the uid in your .lamindb/ directory under ./my_storage_local/:

ln.settings.storage_local.view_tree()

If you’d like to upload an artifact, you pass upload=True to the save() method.

artifact.save(upload=True)

You now see the artifact in the S3 bucket:

ln.setup.settings.storage.root.view_tree()

And it’s no longer present in local storage:

ln.setup.settings.instance.storage_local.root.view_tree()

Direct upload¶

You can also directly upload a file by passing upload=True:

filepath = ln.core.datasets.file_mini_csv()
artifact2 = ln.Artifact(filepath, description="My csv file").save(upload=True)
artifact2.path

Now we have two files on S3:

ln.Artifact.df(include="storage__root")

Show code cell output Hide code cell output

	storage__root	version	uid	storage_id	key	suffix	accessor	description	size	hash	hash_type	n_objects	n_observations	transform_id	run_id	visibility	key_is_virtual	created_at	updated_at	created_by_id
id
2	s3://lamindb-ci/keep-artifacts-local	None	bjTVAqADXSXmj0OdHDJR	1	None	.csv	None	My csv file	11	z1LdF2qN4cN0M2sXrcW8aw	md5	None	None	1	1	1	True	2024-05-14 15:59:27.915949+00:00	2024-05-14 15:59:27.915978+00:00	1
1	s3://lamindb-ci/keep-artifacts-local	None	bzUoQlrHpwWsMCah9qgd	1	None	.fcs	None	My fcs file	6785467	KCEXRahJ-Ui9Y6nksQ8z1A	md5	None	None	1	1	1	True	2024-05-14 15:59:27.386549+00:00	2024-05-14 15:59:27.426408+00:00	1

Pre-existing artifacts¶

Assume we already have a file in our registered local storage location:

1 sub-directory & 2 files with suffixes '', '.bam'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
├── output.bam
└── .lamindb
    └── _is_initialized

If we create an artifact from it, it remains where it is during saving:

my_existing_file = ln.Artifact("./my_storage_local/output.bam", description="my existing file").save()
ln.UPath("my_storage_local/").view_tree()

The storage path of the artifact is constructed using key because key_is_virtual=False:

my_existing_file

However, if we decide to upload the artifact, we’ll use the uid for constructing the storage path and switch key_is_virtual=True:

my_existing_file.save(upload=True)

Here is the remote path of the artifact:

my_existing_file.path

And here are the contents of the storage locations:

# the path on S3
ln.setup.settings.storage.root.view_tree()
# the local path
ln.setup.settings.instance.storage_local.root.view_tree()

Delete the test instance¶

Delete the artifacts:

artifact.delete(permanent=True)
artifact2.delete(permanent=True)
my_existing_file.delete(permanent=True)

Delete the instance:

ln.setup.delete("keep-artifacts-local", force=True)