run_loading_benchmark_on_collection.py
Reference
CLI: run_loading_benchmark_on_collection.py
Arrayloader benchmarks v2
falexwolf

Runs

    ZSuaqX3B

    48 inputs
    1 output

    Ran last month

    4m 40.25s

    tool annbatch
    project Arrayloader benchmarks v2
    n_samples 2000000
    batch_size 4096
    block_size 4
    chunk_size 512
    collection Tahoe100M
    n_datasets -1
    include_obs true
    num_workers 4
    fetch_factor 16
    preload_nchunks 8
    use_torch_loader false
Input
Output
create_tahoe100M_zarr_shards.py

dataloader_v2/tahoe100M_2025-09-20/chunk_19.zarr

1xSHIdfB (7.2 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_42.zarr

KLTLfiSE (7.28 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_45.zarr

lrv7nir4 (7.26 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_8.zarr

E13OBw56 (7.17 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_6.zarr

Kt06UZom (7.27 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_30.zarr

KLshhMCX (7.24 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_20.zarr

cXH5f0GJ (7.26 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_10.zarr

Q6u0oDds (7.29 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_25.zarr

5Qs1NwSt (7.25 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_29.zarr

kGu902sT (7.26 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_46.zarr

xxJABVIA (7.27 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_2.zarr

aAcYYVgr (7.26 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_1.zarr

GAidCvJt (7.25 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_27.zarr

W8cN3yon (7.23 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_34.zarr

Cqfv2pch (7.3 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_28.zarr

zBYfqv0U (7.26 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_16.zarr

SMoxv9A7 (7.27 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_38.zarr

FIowmOdH (7.36 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_26.zarr

xGS98Ne1 (7.31 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_5.zarr

9utCt8wk (7.22 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_23.zarr

oLHG51K5 (7.25 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_17.zarr

qmK9caSI (7.28 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_22.zarr

UmVcigyn (7.24 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_18.zarr

UbE4XUFD (7.28 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_32.zarr

g906DC23 (7.26 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_36.zarr

Xch28LYn (7.34 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_31.zarr

NYBPmXbR (7.29 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_40.zarr

K2eayrhl (7.31 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_0.zarr

Xw5vR5LW (7.32 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_39.zarr

qXwO88JV (7.23 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_12.zarr

SvsSCpz0 (7.28 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_14.zarr

9pXvRdi5 (7.19 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_41.zarr

U282mkIv (7.29 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_21.zarr

j7pZ9wpt (7.29 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_13.zarr

2Bm8Dbeq (7.27 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_15.zarr

QyXRZNnB (7.27 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_43.zarr

0o4FKGK8 (7.34 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_4.zarr

Nc9jrdhL (7.22 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_37.zarr

gK1OzTRB (7.24 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_44.zarr

KaTzrtln (7.18 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_24.zarr

E5SzFOLg (7.31 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_35.zarr

JfvNNvUi (7.25 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_47.zarr

0o24NtMU (7.25 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_33.zarr

9whMHngw (7.31 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_9.zarr

tGskg24E (7.3 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_7.zarr

uufhs5xZ (7.31 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_11.zarr

LdpaMENO (7.35 GB)

dataloader_v2/tahoe100M_2025-09-20/chunk_3.zarr

lByuPO4c (7.26 GB)
record export

run_loading_benchmark_on_collection

j1G4iHA2