paint.data.dataset_splits

Attributes

log

A logger for the dataset splitter.

Classes

DatasetSplitter

Initialize the dataset splitter.

Module Contents

paint.data.dataset_splits.log: A logger for the dataset splitter.

class paint.data.dataset_splits.DatasetSplitter(input_file: pathlib.Path | str, output_dir: pathlib.Path | str, remove_unused_data: bool = True)

Initialize the dataset splitter.

Parameters

input_filePath | str: Path to the input file containing the metadata required to generate the dataset splits.
output_dirPath | str: Path to the output directory where the dataset splits will be saved.
remove_unused_databool: Whether extra metadata should be removed from the resulting dataset splits (Default: True).

metadata

output_dir

remove_unused_data = True

static _get_azimuth_splits(heliostat_data: pandas.DataFrame, training_size: int, validation_size: int) → pandas.DataFrame

Get the splits for the azimuth-based splitting method.

For a single heliostat, the training_size indices with the smallest azimuth values are selected for the training split, while the validation_size indices with the largest values are selected for the validation split. The remaining indices are assigned to the test split.

This ensures that indices with very different azimuth values are considered in the train and validation samples, i.e., the train and validation splits should contain very different samples. This difference leads to a high level of difficulty and should guarantee that the trained calibration method is robust.

Parameters

heliostat_datapd.DataFrame: Data for a single heliostat.
training_sizeint: Size of the training split.
validation_sizeint: Size of the validation split.

Returns

pd.DataFrame: Heliostat data including the associated split.

static _get_nearest_solstice_distance(timestamp: pandas.Timestamp, season: str) → float

Calculate the distances to the nearest December 21 and June 21.

Parameters

timestamppd.Timestamp: The current time stamp considered.
seasonstr: Whether to consider summer or winter solstice. Must be either “summer” or “winter”.

Returns

float: The time distance to the nearest solstice in seconds.

_get_solstice_splits(heliostat_data: pandas.DataFrame, training_size: int, validation_size: int) → pandas.DataFrame

Get the splits for the solstice-based splitting method.

For a single heliostat, the training_size indices closest to the winter solstice are selected for the training split, while the validation_size indices closest to the summer solstice are selected for the validation split. The remaining indices are assigned to the test split.

This ensures that indices from very different seasons, i.e. different conditions, are considered in training and validation, i.e., the train and validation splits should contain very different samples. This difference leads to a high level of difficulty and should guarantee that the trained calibration method is robust.

Parameters

heliostat_datapd.DataFrame: Data for a single heliostat.
training_sizeint: Size of the training split.
validation_sizeint: Size of the validation split.

Returns

pd.DataFrame: Heliostat data including the associated split.

static _get_balanced_splits(heliostat_data: pandas.DataFrame, training_size: int, validation_size: int) → pandas.DataFrame

Get splits using the “balanced” method, based on KMeans clustering.

The clustering is used for stratification. For validation, one candidate is drawn from each cluster. For test, we attempt to select one candidate per cluster that is different from the validation candidate; if a cluster has only one sample, then the missing test candidate is filled from the overall pool. All remaining data points (i.e. those not used for validation or test) are candidates for training. As a result, the size of the validation and test set is identical.

Parameters

heliostat_datapd.DataFrame: Data for a single heliostat.
training_sizeint: Desired number of training samples.
validation_sizeint: Desired number of validation samples (and also test samples).

Returns

pd.DataFrame: DataFrame with splits assigned in the column specified by mappings.SPLIT_KEY. The returned DataFrame contains exactly training_size + 2*validation_size rows.

Raises

ValueError: If the overall number of data points is insufficient.

static _get_high_variance_splits(heliostat_data: pandas.DataFrame, validation_size: int, training_size: int, n_neighbors: int = 3) → pandas.DataFrame

Get splits using the “High-Variance” method, bust on a k-nearest neighbors (KNN) quality metric.

For each data point in the given heliostat’s data, the average distance to its n_neighbors closest points (excluding itself) is computed. Then, the splits are assigned as follows:

Validation: The first validation_size data points from the sorted order (by descending average distance) are assigned to validation.
Test: The next validation_size data points in the sorted order are assigned to test.
Training: The last training_size data points (with the smallest average distances) are assigned to training.
All other data points are discarded.

This requires that the total number of data points is at least:

total_required = training_size + 2 * validation_size

Parameters

heliostat_datapd.DataFrame: Data for a single heliostat.
validation_sizeint: Number of data points to assign to the validation set.
training_sizeint: Number of data points to assign to the training set.
n_neighborsint, optional: Number of nearest neighbors to consider when computing the average distance (Default: 3).

Returns

pd.DataFrame: A dataframe containing only the selected data points with their assigned split in the column specified by mappings.SPLIT_KEY.

Raises

ValueError: If there are not enough data points to compute the KNN metric or to assign the requested number of samples.

get_dataset_splits(split_type: str, training_size: int, validation_size: int) → pandas.DataFrame

Get dataset splits and save splits as a CSV file.

This function determines the dataset splits and creates a dataframe containing information about these splits. This data frame is returned, optionally with extra metadata. Additionally, the splits without metadata are saved as a CSV file.

This function supports the following split types: - azimuth: The azimuth of the sun is used to calculate the splits. Specifically, the images with the smallest azimuth values are used for training, the images with the largest azimuth values are used for validation, and the remaining images for testing. - solstice: The distance from the winter and summer solstice is used to calculate the splits. Specifically, the images closest to the winter solstice are used for training, the images closest to the summer solstice for validation, and the remaining images for testing.

Parameters

split_typestr: Type of split to be performed. Currently, azimuth and solstice split types are available.
training_sizeint: Size of the training split.
validation_sizeint: Size of the validation split.

Returns

pd.DataFrame: Dataframe containing information on the dataset splits.