acoupipe.datasets.base#

Base classes for generating microphone array datasets.

Module Contents#

class acoupipe.datasets.base.BaseFeatureCatalog#

Bases: traits.api.HasPrivateTraits

BaseFeatureCatalog base class for handling feature funcs.

Attributes:

namestr: Name of the feature.
dtypecallable: Numpy dtype of the feature.

get_feature_func()#: Will return a method depending on the class parameters.

class acoupipe.datasets.base.BaseFeatureCollectionBuilder#

Bases: traits.api.HasPrivateTraits

BaseFeatureCollectionBuilder base class for building a BaseFeatureCollection.

Attributes:

feature_collectionBaseFeatureCollection: BaseFeatureCollection object.

add_custom(feature_func)#

Add a custom feature to the BaseFeatureCollection.

The custom feature_func should be a callable that takes a sampler as input and returns a dictionary of feature name and feature data.

Parameters:

feature_funccallable: Feature to be added.

build()#

Build a BaseFeatureCollection.

Returns:

BaseFeatureCollection: BaseFeatureCollection object.

acoupipe.datasets.base.set_pipeline_seeds(pipeline, start_idx, size, dataset='training')#

Create the random seed list for each sampler object held by the pipeline.

Parameters:

pipelineinstance of class BasePipeline: the pipeline object holding the sampler classes
start_idxint: start index to be calculated by the pipeline
sizeint: number of samples to be yielded by the pipeline
datasetstr, optional: the data set type, by default “training”. Choose from [“training”,”validation”]

class acoupipe.datasets.base.BasePipeline#

Bases: acoupipe.base.DataGenerator

Control the random process and iteratively extract and pass a specified amount of data.

This class can be used to calculate data (extract features) by assigning a name and a callable function to features. Furthermore this class automatically controles the sampling of instances of type BaseSampler specified to the sampler list. Re-seeding is performed at each iteration if random_seeds are given.

validate_random_seeds()#: Validate specified random seeds.

get_data(progress_bar=True, start_idx=1)#

Provide the extracted features, sampler seeds and indices.

Parameters:

progress_barbool, optional: if True, a progress bar is displayed, by default True
start_idxint, optional: the index of the first data sample to be calculated, by default 1

Yields:

dict: a sample of the dataset containing the extracted feature data, seeds, and index

class acoupipe.datasets.base.DistributedPipeline#

Bases: BasePipeline

Class to calculate data (extract features) in parallel to build large datasets.

This class can be used to calculate data (extract various features) by assigning a name and a callable function to features. Furthermore this class automatically controles the sampling of instances of type BaseSampler specified to the sampler list. Re-seeding is performed at each iteration if random_seeds are given.

get_data(progress_bar=True, start_idx=1)#

Provide the extracted features, sampler seeds and indices.

The calculation of all data samples is performed in parallel and asynchronously. In case of specifying more than one worker in the numworker attribute, the output of this generator yields non-ordered features/data samples. However, the exact order can be recovered via the “idx” item (or “seeds” item) provided in the output dictionary.

Parameters:

progress_barbool, optional: if True, a progress bar is displayed, by default True
start_idxint, optional: the index of the first data sample to be calculated, by default 1

Yields:

dict: A sample of the dataset containing the extracted feature data, seeds, and index

class acoupipe.datasets.base.WriteH5Dataset#

Bases: BaseWriteDataset

Class intended to write data to a .h5 file.

save(progress_bar=True, start_idx=1)#: Save the get_data() output of the source to .h5 file format.

get_data(progress_bar=True, start_idx=1)#

Save the source data to a *.h5 file and yield it to the next object.

Returns:

dict: Dictionary containing a sample of the data set {feature_name[key] : feature[values]}.

class acoupipe.datasets.base.DatasetBase(config=None, tasks=1, remote_args=None, logger=None)#

Bases: traits.api.HasPrivateTraits

Base class for generating microphone array datasets with specified features and labels.

Attributes:

configConfigBase: Configuration object for dataset generation.
tasksint: Number of parallel tasks for data generation. Defaults to 1 (sequential calculation).

get_feature_collection(features, f, num)#

Get the feature collection of the dataset.

Returns:

BaseFeatureCollection: BaseFeatureCollection object.

generate(features, size, split='training', f=None, num=0, start_idx=0, progress_bar=True)#

Generate dataset samples iteratively.

Parameters:

featureslist

List of features included in the dataset. The features “seeds” and “idx” are always included.

splitstr

Split name for the dataset (‘training’, ‘validation’ or ‘test’). Defaults to ‘training’.

sizeint

Size of the dataset (number of source cases).

ffloat

The center frequency or list of frequencies of the dataset. If None, all frequencies are included.

numinteger

Controls the width of the frequency bands considered; defaults to 0 (single frequency line).

num	frequency band width
0	single frequency line
1	octave band
3	third-octave band
n	1/n-octave band

start_idxint, optional

Starting sample index (default is 0).

progress_barbool, optional

Whether to show a progress bar (default is True).

Yields:

datadict: Generator that yields dataset samples as dictionaries containing the feature names as keys.

Examples

Generate features iteratively (example below requires a dataset configuration).

from acoupipe.datasets.synthetic import DatasetSynthetic

# define the features
features = ['csm', 'source_strength_analytic', 'loc']
f = 1000
num = 3

# generate the dataset
generator = DatasetSynthetic().generate(
    f=f,
    num=num,
    split='training',
    size=2,
    features=features,
)

# iterate over the dataset
for data in generator:
    print(data)

save_h5(features, size, name, split='training', f=None, num=0, start_idx=0, progress_bar=True)#

Save dataset to a HDF5 file.

Parameters:

featureslist

List of features included in the dataset. The features “seeds” and “idx” are always included.

sizeint

Size of the dataset (number of source cases).

namestr

Name of the HDF5 file.

splitstr

Split name for the dataset (‘training’, ‘validation’ or ‘test’). Defaults to ‘training’.

ffloat

The center frequency or list of frequencies of the dataset. If None, all frequencies are included.

numinteger

Controls the width of the frequency bands considered; defaults to 0 (single frequency line).

num	frequency band width
0	single frequency line
1	octave band
3	third-octave band
n	1/n-octave band

start_idxint, optional

Starting sample index (default is 0).

progress_barbool, optional

Whether to show a progress bar (default is True).

Returns:

None

Examples

Save features to a HDF5 file (example requires proper file path).

from acoupipe.datasets.synthetic import DatasetSynthetic

# define the features
features = ['csm', 'source_strength_analytic', 'loc']
f = 1000
num = 3

# save the dataset
dataset = DatasetSynthetic().save_h5(
    f=f,
    num=num,
    split='training',
    size=10,
    features=features,
    name='/tmp/example.h5',
)

acoupipe.datasets.base.save_tfrecord(self, features, size, name, split='training', f=None, num=0, start_idx=0, progress_bar=True)#

Save dataset to a .tfrecord file.

Parameters:

featureslist

List of features included in the dataset. The features “seeds” and “idx” are always included.

sizeint

Size of the dataset (number of source cases).

namestr

Name of the TFRecord file.

splitstr

Split name for the dataset (‘training’, ‘validation’ or ‘test’). Defaults to ‘training’.

ffloat

The center frequency or list of frequencies of the dataset. If None, all frequencies are included.

numinteger

Controls the width of the frequency bands considered; defaults to 0 (single frequency line).

num	frequency band width
0	single frequency line
1	octave band
3	third-octave band
n	1/n-octave band

start_idxint, optional

Starting sample index (default is 0).

progress_barbool, optional

Whether to show a progress bar (default is True).

Returns:

None

Examples

Save features to a TFRecord file (example requires proper file path).

from acoupipe.datasets.synthetic import DatasetSynthetic

# define the features
features = ['csm', 'source_strength_analytic', 'loc']
f = 1000
num = 3

# save the dataset
dataset = DatasetSynthetic().save_tfrecord(
    f=f,
    num=num,
    split='training',
    size=10,
    features=features,
    name='/tmp/example.tfrecord',
)