acoupipe.writer#

Provides classes to store the data extracted by BasePipeline.

Purpose of the Writer Module#

The writer.py module provides classes to store the data extracted by the pipeline. The current implementation includes classes to save data in a container-like file format (.h5 file with the WriteH5Dataset class) or binary format (.tfrecord file with the WriteTFRecord class). The latter can be efficiently consumed by the Tensorflow framework for machine learning.

file_writer = acoupipe.writer.WriteH5Dataset(
    source=pipeline,
)

file_writer.save()

Module Contents#

class acoupipe.writer.DataGenerator#

Bases: traits.api.ABCHasStrictTraits

Abstract base class that serves as a data generator.

This class should not be used directly. It provides a common interface for all classes that generate data via the result() method in a block-wise or sample-wise manner.

abstractmethod get_data()#

Python generator that iteratively yields data set samples as a dictionary.

This method needs to be implemented by derived classes.

Returns:

dict: Dictionary containing a sample of the data set {feature_name[key] : feature[values]}.

class acoupipe.writer.BaseWriteDataset#

Bases: acoupipe.base.DataGenerator

Base class intended to write data to a specific file format.

Writes data from BasePipeline instances.

This class has no functionality and should not be used.

save()#: Save data from the source instance to file.

get_data(progress_bar=True, start_idx=1)#

Save source output data to file and pass the data to the next object.

Parameters:

progress_barbool: If True, a progress bar is shown.
start_idxint: Index of the first sample which is written to file.

Returns:

dict: Dictionary containing a sample of the data set {feature_name[key] : feature[values]}.

class acoupipe.writer.WriteH5Dataset#

Bases: BaseWriteDataset

Class intended to write data to a .h5 file.

save(progress_bar=True, start_idx=1)#: Save the get_data() output of the source to .h5 file format.

get_data(progress_bar=True, start_idx=1)#

Save the source data to a *.h5 file and yield it to the next object.

Returns:

dict: Dictionary containing a sample of the data set {feature_name[key] : feature[values]}.