paint.preprocessing

Submodules

• paint.preprocessing.binary_extractor
• paint.preprocessing.calibration_stac
• paint.preprocessing.catalog_stac
• paint.preprocessing.deflectometry_stac
• paint.preprocessing.dwd_mappings
• paint.preprocessing.dwd_stac_item
• paint.preprocessing.dwd_weather
• paint.preprocessing.focal_spot_extractor
• paint.preprocessing.heliostat_catalog_stac
• paint.preprocessing.juelich_weather_converter
• paint.preprocessing.juelich_weather_mappings
• paint.preprocessing.juelich_weather_stac_item
• paint.preprocessing.properties_stac
• paint.preprocessing.target_cropper
• paint.preprocessing.tower_measurements
• paint.preprocessing.tower_stac
• paint.preprocessing.weather_collection_stac

Attributes

dwd_parameter_mapping	Dictionary to map DWD names to simpler names for saving the data.
juelich_metadata_description	Dictionary to include descriptions for the weather parameters for the attributes in the Jülich weather HDF5 files.
juelich_metadata_units	Dictionary to include units for the weather parameters in the Jülich weather HDF5 files.
juelich_weather_parameter_mapping	Dictionary to map parameter names from the original name to the saved name for the Jülich weather data.

Classes

BinaryExtractor	Initialize the extractor.
DWDWeatherData	Initialize the DWD weather data object.
JuelichWeatherConverter	Initialize the weather converter.

Functions

make_calibration_collection(→ dict[str, Any])	Generate the STAC collection.
make_calibration_item(→ dict[str, Any])	Generate a STAC item for an image.
make_catalog(→ dict[str, Any])	Generate the catalog STAC.
make_deflectometry_collection(→ dict[str, Any])	Generate a deflectometry STAC collection.
make_deflectometry_item(→ tuple[tuple[float, float], ...)	Generate a STAC item for a deflectometry measurement.
make_dwd_item(→ dict[str, Any])	Generate a STAC item for the DWD weather data.
make_heliostat_catalog(→ dict[str, Any])	Generate a catalog for each heliostat STAC.
make_juelich_weather_item(→ dict[str, Any])	Generate a STAC item for the Juelich weather data.
find_min_max_coordinate(→ dict[str, numpy.ndarray])	Extract the min and max values of coordinates (latitude, longitude, elevation) from a nested dictionary.
get_tower_measurements(→ tuple[dict[str, ...)	Generate the tower measurement data.
make_tower_item(→ dict[str, Any])	Generate a STAC item for the tower metadata JSON.
make_weather_collection(→ Dict[str, Any])	Generate a weather STAC collection.

Package Contents

class paint.preprocessing.BinaryExtractor(input_path: str | pathlib.Path, output_path: str | pathlib.Path, deflectometry_created_at_file_name: str, surface_header_name: str, facet_header_name: str, points_on_facet_struct_name: str)

Initialize the extractor.

Parameters

input_pathstr | Path

The file path to the binary data file that will be converted.

output_pathstr | Path

The file path to save the converted h5 deflectometry file.

deflectometry_created_at_file_namestr

The time stamp in the file name format for when the deflectometry data was created.

surface_header_namestr

The name for the surface header in the binary file.

facet_header_namestr

The name for the facet header in the binary file.

points_on_facet_struct_namestr

The name of the point on facet structure in the binary file.

input_path

output_path

heliostat_id

raw_data

file_name

json_handle

surface_header_name

facet_header_name

points_on_facet_struct_name

convert_to_h5() → None

Extract data from a binary file and save the deflectometry measurements.

paint.preprocessing.make_calibration_collection(heliostat_id: str, data: pandas.DataFrame) → dict[str, Any]

Generate the STAC collection.

Parameters

heliostat_idstr

The heliostat ID of the heliostat being considered.

datapd.DataFrame

The dataframe containing all image data.

Returns

dict[str, Any]

The STAC collection as dictionary.

paint.preprocessing.make_calibration_item(image: int, heliostat_data: pandas.Series, processed_available: bool) → dict[str, Any]

Generate a STAC item for an image.

Parameters

imageint

The image ID.

heliostat_datapd.Series

The data belonging to the heliostat.

processed_availablebool

Whether processed images are available or not.

Returns

dict[str, Any]

The STAC item data as dictionary.

paint.preprocessing.make_catalog(data: list) → dict[str, Any]

Generate the catalog STAC.

Parameters

datalist

A list of heliostats.

Returns

dict[str, Any]

The STAC catalog as dictionary

paint.preprocessing.make_deflectometry_collection(heliostat_id: str, data: pandas.DataFrame) → dict[str, Any]

Generate a deflectometry STAC collection.

Parameters

heliostat_id: str

The heliostat ID of the heliostat containing the collection.

data: pd.DataFrame

The dataframe containing all deflectometry metadata.

Returns

dict[str, Any]

The STAC collection as dictionary.

paint.preprocessing.make_deflectometry_item(heliostat_key: str, heliostat_data: pandas.Series, results_exist: bool) → tuple[tuple[float, float], dict[str, Any]]

Generate a STAC item for a deflectometry measurement.

Parameters

heliostat_key: str

The ID of the heliostat which was measured.

heliostat_datapd.Series

The metadata for the heliostat.

results_existbool

Whether the results PDF exists.

Returns

tuple[float, float]

The latitude and longitude coordinates of the heliostat that being measured.

dict[str, Any]

The STAC item data as dictionary.

paint.preprocessing.dwd_parameter_mapping

Dictionary to map DWD names to simpler names for saving the data.

paint.preprocessing.make_dwd_item(data: pandas.Series) → dict[str, Any]

Generate a STAC item for the DWD weather data.

Parameters

datapd.Series

The metadata for the DWD weather data file.

Returns

dict[str, Any]

The STAC item data as dictionary.

class paint.preprocessing.DWDWeatherData(parameters_10min: list[str], parameters_1h: list[str], station_ids: list[str], start_date: str, end_date: str, output_path: str, file_name: str = 'dwd-weather.h5', ts_shape: str = 'long', ts_humanize: bool = True, ts_si_units: bool = False, compression_method: str = 'gzip', compression_level: int = 5)

Initialize the DWD weather data object.

Parameters

parameters_10minlist[str]

The parameters to be downloaded in a 10min temporal resolution.

parameters_1hlist[str]

The parameters to be downloaded in a 1h temporal resolution.

station_idslist[str]

The station IDs to be considered when downloading data.

start_datestr

The start date of the downloaded data.

end_datestr

The end date of the downloaded data.

output_pathstr

The path to save the downloaded data.

file_namestr

The name of the downloaded data (Default: “dwd_weather”).

ts_shapestr

A string indicating how the time series shape should be handled in the wetterdienst package (Default: long).

ts_humanizebool

A boolean indicating whether the time series should be humanized or not within the wetterdienst package (Default:True).

ts_si_unitsbool

A boolean indicating whether the time series units should be converted to SI units within the

wetterdienst package (Default:False).

compression_methodstr

The method used to compress the HDF5 file.

compression_levelint

The compression level.

parameters_10min

parameters_1h

station_ids

start_date

end_date

output_path

file_name = 'dwd-weather.h5'

settings

compression_opts

_get_raw_data() → tuple[pandas.DataFrame, pandas.DataFrame, pandas.DataFrame, pandas.DataFrame]

Download the raw data using the DWD Wetterdienst pacakge.

Returns

pd.DataFrame

The metadata for each weather station included in the 10min temporal resolution data request.

pd.DataFrame

The metadata for each weather station included in the 1h temporal resolution data request.

pd.DataFrame

The data for the weather variables downloaded in 10min temporal resolution.

pd.DataFrame

The data for the weather variables downloaded in 1h temporal resolution.

download_and_save_data() → pandas.DataFrame

Download the desired DWD weather data and save it to an HDF5 file.

Returns

pd.Dataframe

The metadata used for creating the STAC item.

paint.preprocessing.make_heliostat_catalog(heliostat_id: str, include_deflectometry: bool, include_calibration: bool, include_properties: bool) → dict[str, Any]

Generate a catalog for each heliostat STAC.

Parameters

heliostat_idstr

The heliostat ID for the considered heliostat.

include_deflectometrybool

Whether the deflectometry collection is included for this heliostat.

include_calibrationbool

Whether the calibration collection is included for this heliostat.

include_propertiesbool

Whether the properties collection is included for this heliostat.

Returns

dict[str, Any]

The STAC catalog as dictionary

class paint.preprocessing.JuelichWeatherConverter(input_root_dir: str, output_path: str, compression_method: str = 'gzip', compression_level: int = 5)

Initialize the weather converter.

Parameters

input_root_dirstr

The root directory to search for weather files.

output_pathstr

The output path to save the HDF5 file.

compression_methodstr

The method used to compress the HDF5 file.

compression_levelint

The compression level.

input_root_dir

output_path

files_list

compression_opts

find_weather_files() → list[str]

Recursively find all weather.txt files in the directory.

Returns

list[str]

The list of weather.txt files to be concatenated.

concatenate_weather() → pandas.DataFrame

Load all weather.txt files as a dataframe and return the concatenated dataframe.

Returns

pd.DataFrame

The concatenated dataframe.

merge_and_save_to_hdf5() → pandas.DataFrame

Merge the weather files and save the merged data to HDF5.

Returns

pd.Dataframe

The metadata for the merged dataframe to be used for STAC creation.

paint.preprocessing.juelich_metadata_description

Dictionary to include descriptions for the weather parameters for the attributes in the Jülich weather HDF5 files.

paint.preprocessing.juelich_metadata_units

Dictionary to include units for the weather parameters in the Jülich weather HDF5 files.

paint.preprocessing.juelich_weather_parameter_mapping

Dictionary to map parameter names from the original name to the saved name for the Jülich weather data.

paint.preprocessing.make_juelich_weather_item(data: pandas.Series, month_group: str) → dict[str, Any]

Generate a STAC item for the Juelich weather data.

Parameters

datapd.Series

Metadata for the Juelich weather data file.

month_groupstr

Considered month group.

Returns

dict[str, Any]

The STAC item data as dictionary.

paint.preprocessing.find_min_max_coordinate(coordinate_dictionary: dict[Any, Any]) → dict[str, numpy.ndarray]

Extract the min and max values of coordinates (latitude, longitude, elevation) from a nested dictionary.

Parameters

coordinate_dictionarydict[Any, Any]

A nested dictionary containing coordinate tuples.

Returns

dict[str, np.ndarray]

A dictionary containing min and max coordinates for latitude, longitude, and elevation.

paint.preprocessing.get_tower_measurements() → tuple[dict[str, numpy.ndarray], dict[Any, Any]]

Generate the tower measurement data.

This data takes the measured Gauss-Kruger coordinates for each considered calibration target and receiver, converts them to latitude and longitude coordinates, considers the elevation and returns these values in a dictionary. Additionally, the min and max values for each of the dimensions in the coordinates are saved for the STAC item creation.

Returns

dict[str, np.ndarray]

The saved max and min values for each dimension in the coordinates.

dict[Any, Any]

The measurement dictionary.

paint.preprocessing.make_tower_item(extreme_coordinates: dict[str, numpy.ndarray]) → dict[str, Any]

Generate a STAC item for the tower metadata JSON.

Parameters

extreme_coordinatesdict[str, np.ndarray]

The max and min for each of the latitude, longitude, and elevation coordinates from the tower.

Returns

dict[str, Any]

The STAC item data as dictionary.

paint.preprocessing.make_weather_collection(data: pandas.DataFrame) → Dict[str, Any]

Generate a weather STAC collection.

Parameters

data: pd.DataFrame

The dataframe containing all weather metadata.

Returns

dict[str, Any]

The STAC collection as dictionary.