acoupipe.loader#

Provides classes to load the datasets stored with derived classes.

See BaseWriteDataset.

Module Contents#

class acoupipe.loader.DataGenerator#

Bases: traits.api.ABCHasStrictTraits

Abstract base class that serves as a data generator.

This class should not be used directly. It provides a common interface for all classes that generate data via the result() method in a block-wise or sample-wise manner.

abstractmethod get_data()#

Python generator that iteratively yields data set samples as a dictionary.

This method needs to be implemented by derived classes.

Returns:

dict: Dictionary containing a sample of the data set {feature_name[key] : feature[values]}.

class acoupipe.loader.BaseLoadDataset#

Bases: acoupipe.base.DataGenerator

Base class for all derived classes intended to load data.

Loads data stored by BaseWriteDataset.

This class has no functionality and should not be used.

load_data()#: Open a dataset file and set attributes.

class acoupipe.loader.LoadH5Dataset#

Bases: BaseLoadDataset

Loads data sets stored into *.h5 file format.

This class loads data from *.h5 files and provides information like the number of samples (numsamples).

load_data()#: Open the .h5 file and set attributes.

load_metadata()#: Load metadata from .h5 file. Only for internal use.

get_dataset_generator(features=None)#

Create a callable that returns a generator object.

This object can be used in conjunction with the Tensorflow tf.data.Dataset API to create a data generator with the from_generator() method of the Tensorflow Dataset API to feed machine learning models.

Example to create a repeatable data set with the tf.data.Dataset API is given below:

h5data = LoadH5Dataset(name='some_dataset.h5')
generator = h5data.get_dataset_generator(features=['loc'])
sig = {
    'loc': tf.TensorSpec(shape=(3, None), dtype=tf.float32),
}
dataset = tf.data.Dataset.from_generator(generator, output_signature=sig).repeat()
loc = next(iter(dataset))  # return locations

Parameters:

featureslist, optional: a list with names of the features to be yielded by the generator, by default None, meaning that all features will be considered.

Returns:

callable: A callable that returns a generator object

get_data()#

Yield the dataset samples in ascending sample index order (e.g. 1,2,…,N).

Returns:

dict: Dictionary containing a sample of the dataset {feature_name[key] : feature[values]}