Optimise hdf5 #36

hllelli2 · 2024-10-01T13:13:48Z

MapDataset, ArrayDataset and MapDataloader to handle inputs of .mrc or .map data.
Transformations and Augmentations build upon CCPEM_Utils.
Hdf5 intermediate data store
Added accompanying tests
Added basic pip install

…d arrays into the hdf5 without a file.

…the dataloader, augment them and saves them to array. For a specific usecase in ml-toolkit

…atible with ArrayDataset

…ectory storage

…t sure if needed for multi-class porblems

…lasses

…engths

…aching mechanism

…ing mechanism

src/caked/Transforms/augments.py

tests/test_map_io.py

tests/conftest.py

src/caked/Transforms/transforms.py

src/caked/Transforms/utils.py

aj26git · 2024-10-03T19:21:21Z

@hllelli2 : Overall it looks good. I have added a few comments/suggestions. It is good to go once these are addressed.

…ling

…te/CAKED into OptimiseHdf5

…unction. This allows any keywords to be applied to all MapDatasets.

…c in MapObjectVoxelNormalisation

…te/caked into OptimiseHdf5

hllelli2 added 30 commits June 19, 2024 16:08

Updates to gitignore

c64fbff

basic setup config for pip install

2f4d58f

Added MapDataset and MapDataLoader

6afd99e

Added custom augments and transfroms as well as tiling

ef73dd2

Added tests for new code

2ab4260

feat: Add HDF5DataStore class for handling hdf5 files

8b9d06d

Add pytest configuration file and conftest.py for test fixtures

d8729a1

Added a dataset config class to have all the defaults in one place

1a6ed24

Create a class for lazy-loading a hdf5 file

c8486b8

Moved multi-processing to utils, Added an array dataset which can loa…

3779ee4

…d arrays into the hdf5 without a file.

Moved Multi-processing code here, code here to duplciate arrays from …

b9a91cb

…the dataloader, augment them and saves them to array. For a specific usecase in ml-toolkit

Changed augment to take arrays instead of map-objects to make it comp…

ab2ce2f

…atible with ArrayDataset

Removed mapObject transform base

d6eda3a

chore: Refactor transforms module and update transform classes

a86c910

first attempt at reducing if label/weight... is not none repeated code

41d40b3

chore: Add test fixtures and update conftest.py for test setup

6f73aae

set_gpu is not in the current ccpem-utils so check added JIC

07e2318

chore: Refactor HDF5DataStore class and add support for temporary dir…

a6cc9b1

…ectory storage

Removed "_" id logic in MapDataLoader, might reimplement later but no…

05f792b

…t sure if needed for multi-class porblems

Refactor process_datasets function and remove unnecessary code

5a770f7

Refactor transforms module and update transform classes

0a99fd7

re-added __del__ method to HDF5Store

8772fa6

Refactor MapDataset and ArrayDataset classes to handle weight tensors

93c8424

Refactor

9128e9a

Refactor DecomposeToSlices, MapObjectMaskCrop, and MapObjectPadding c…

b2a49c1

…lasses

Refactor test_map_io.py to handle weight tensors and update dataset l…

ce47030

…engths

Refactor HDF5DataStore class to handle weight tensors and implement c…

5d34da0

…aching mechanism

pre-commit fixes

d049499

Refactor MapDataLoader to add caching mechanism for HDF5DataStore

1e65dd8

Refactor HDF5DataStore class to handle weight tensors and update cach…

f31870b

…ing mechanism

hllelli2 requested a review from aj26git October 1, 2024 13:13

hllelli2 self-assigned this Oct 1, 2024

hllelli2 added 11 commits October 1, 2024 13:28

Add ccpem-utils to project dependencies

9331356

Add h5py and psutil to project dependencies

be1a80b

Fix deprecation warnings in pytest.ini

6b7f0c7

Fix deprecation warnings in pytest.ini

3b4196f

Fix deprecation warnings in pytest.ini

28dc529

Fix deprecation warnings in pytest.ini and update pytest configuration

2358a76

Fix deprecation warnings

a9af9c3

Update pytest configuration and package installation

7cd0e06

Update pytest configuration and package installation

cb6a8c9

Update pytest configuration and package installation

b95eaa6

Fix: __getitem__ logic

a8b367c

aj26git reviewed Oct 2, 2024

View reviewed changes

src/caked/Transforms/augments.py Show resolved Hide resolved

aj26git reviewed Oct 3, 2024

View reviewed changes

tests/test_map_io.py Outdated Show resolved Hide resolved

aj26git reviewed Oct 3, 2024

View reviewed changes

tests/conftest.py Outdated Show resolved Hide resolved

aj26git reviewed Oct 3, 2024

View reviewed changes

src/caked/Transforms/transforms.py Outdated Show resolved Hide resolved

aj26git reviewed Oct 3, 2024

View reviewed changes

src/caked/Transforms/utils.py Outdated Show resolved Hide resolved

hllelli2 and others added 11 commits October 9, 2024 13:53

Add iterator method to HDF5DataStore and simplify key checks in tests

0696806

Refactor test fixtures to use default scope and improve readability

7fbefdf

Refactor: Rename tiles to slice_indicies and update related logic

36a5883

Refactor: Simplify iterator method in HDF5DataStore

b24d299

Refactor: update default voxel parameter type and improve kwargs hand…

7d12ece

…ling

Merge branch 'OptimiseHdf5' of https://github.com/alan-turing-institu…

f4555ae

…te/CAKED into OptimiseHdf5

Updated MapDataLoader to accepted the Mapdataset Kwargs in the load f…

2f32536

…unction. This allows any keywords to be applied to all MapDatasets.

Fix: Change default value of 'vox' to a float and update related logi…

1844ec7

…c in MapObjectVoxelNormalisation

Merge branch 'OptimiseHdf5' of https://github.com/alan-turing-institu…

96772ae

…te/caked into OptimiseHdf5

Cleanup: remove debug print statements from MapDataset class

b8d9c65

update pyproject.toml with dependency version

d975d44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimise hdf5 #36

Optimise hdf5 #36

hllelli2 commented Oct 1, 2024

aj26git commented Oct 3, 2024

Optimise hdf5 #36

Are you sure you want to change the base?

Optimise hdf5 #36

Conversation

hllelli2 commented Oct 1, 2024

aj26git commented Oct 3, 2024