paint.data

Submodules

• paint.data.dataset
• paint.data.dataset_splits
• paint.data.stac_client

Classes

StacClient

Initialize the STAC client.

Package Contents

class paint.data.StacClient(output_dir: str | pathlib.Path, chunk_size: int = 1024 * 1024)

Initialize the STAC client.

Parameters

output_dirstr | pathlib.Path

Output directory for saving the downloaded data.

chunk_sizeint

Chunk size used for download (Default: 1024 * 1024).

output_dir

chunk_size = 1048576

static load_checkpoint(path: pathlib.Path) → dict[str, Any]

Load checkpoint to resume the download from the point it stopped.

Parameters

pathpathlib.Path

The path to the checkpoint.

Returns

dict[str, Any]

The checkpoint data.

static save_checkpoint(path: pathlib.Path, data: dict[str, Any]) → None

Save data to the download checkpoint.

Parameters

pathpathlib.Path

The path to the checkpoint.

datadict[str, Any]

The data to save in the checkpoint.

static mark_done(checkpoint_data: dict[str, Any], catalog_id: str) → None

Mark specific heliostat catalog as downloaded for the checkpoint.

Parameters

checkpoint_datadict[str, Any]

The checkpoint data.

catalog_idstr

The catalog ID to be marked.

static mark_metadata_done(checkpoint_data: dict[str, Any], heliostat_id: str, collection: str) → None

Mark specific heliostat catalog as downloaded for the checkpoint.

Parameters

checkpoint_datadict[str, Any]

The checkpoint data.

heliostat_idstr

The heliostat ID to be marked.

collectionstr

The collection to be marked.

static get_catalog(href: str) → pystac.catalog.Catalog

Help to load a STAC catalog.

Since PySTAC does not check if the URL exists before attempting to download a catalog, we catch any errors raised and raise a value error indicating that the desired catalog does not exist.

Parameters

hrefstr

URL reference to the catalog that should be accessed.

Returns

pystac.catalog.Catalog

Catalog, if it exists.

Raises

ValueError

If the URL does not exist.

static get_child(parent: pystac.catalog.Catalog | pystac.collection.Collection, child_id: str) → pystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection

Help to get a child from a STAC catalog or collection.

Since PySTAC returns None if a child is not present, it is important we log a warning if this is the case.

Parameters

parentpystac.catalog.Catalog | pystac.collection.Collection

Parent STAC catalog or collection.

child_idstr

ID of the child STAC catalog, collection, or item.

Returns

pystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection

Child STAC catalog, collection, or item.

download_file(url: str, file_name: str | pathlib.Path, timeout: int = 60) → bool

Download a file.

Parameters

urlstr

URL of the file to download.

file_namestr | pathlib.Path

File name to be saved.

timeoutint

Timeout for downloading the file (Default: 60 seconds).

Returns

bool

Whether the file was downloaded.

_process_single_heliostat_item(item: pystac.item.Item, start_date: datetime.datetime | None, end_date: datetime.datetime | None, filtered_calibration_keys: list[str] | None, collection_id: str, heliostat_catalog_id: str, save_folder: str, pbar: tqdm.tqdm, for_dataset: bool = False, timeout: int = 60) → bool

Process a single heliostat item.

Parameters

itempystac.item.Item

Item to be processed.

start_datedatetime, optional

Optional start date to filter the heliostat data. If no start date is provided, data for all time periods is downloaded (Default: None).

end_datedatetime, optional

Optional end date to filter the heliostat data. If no end date is provided, data for all time periods is downloaded (Default: None).

filtered_calibration_keyslist[str]

List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).

collection_idstr

ID of the collection to download.

heliostat_catalog_idstr

ID of the considered heliostat catalog.

save_folderstr

Name of the folder to save the collection items in.

pbartqdm

Progress bar.

for_datasetbool

Whether to save all items in one folder for a dataset (Default: False).

timeoutint

Timeout for downloading the heliostat item (Default: 60 seconds).

Returns

bool

Whether the heliostat item was downloaded successfully.

_process_heliostat_items(items: list[pystac.item.Item], start_date: datetime.datetime | None, end_date: datetime.datetime | None, filtered_calibration_keys: list[str] | None, collection_id: str, heliostat_catalog_id: str, save_folder: str, for_dataset: bool = False, timeout: int = 60, num_parallel_workers: int = 10, results_timeout: int = 300) → bool

Process and download items, either with or without date filtering and calibration key filtering.

Parameters

itemslist[pystac.item.Item]

List of items to be processed.

start_datedatetime, optional

Optional start date to filter the heliostat data. If no start date is provided, data for all time periods is downloaded (Default: None).

end_datedatetime, optional

Optional end date to filter the heliostat data. If no end date is provided, data for all time periods is downloaded (Default: None).

filtered_calibration_keyslist[str]

List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).

collection_idstr

ID of the collection to download.

heliostat_catalog_idstr

ID of the considered heliostat catalog.

save_folderstr

Name of the folder to save the collection items in.

for_datasetbool

Whether to save all items in one folder for a dataset (Default: False).

timeoutint

Timeout for downloading the heliostat item (Default: 60 seconds).

num_parallel_workersint

Number of parallel workers for downloading heliostat data (Default: 10).

results_timeoutint

Timeout for collecting results from multiple threads (Default: 300 seconds).

Returns

bool

Whether all heliostat items were downloaded successfully.

_download_heliostat_asset(asset: pystac.asset.Asset, heliostat_catalog_id: str, save_folder: str, for_dataset: bool = False, timeout: int = 60) → bool

Download the asset from the provided URL and save it to the selected location.

Parameters

assetpystac.asset.Asset

STAC asset to be downloaded and saved.

heliostat_catalog_idstr

ID of the considered heliostat catalog.

save_folderstr

Name of the folder to save the asset in.

for_datasetbool

Whether to save all items in one folder for a dataset (Default: False).

timeoutint

Timeout for downloading assets (Default: 60 seconds).

Returns

bool

Whether the asset was successfully downloaded.

_download_heliostat_data(heliostat_catalog: pystac.catalog.Catalog, collection_id: str, save_folder: str, start_date: datetime.datetime | None = None, end_date: datetime.datetime | None = None, filtered_calibration_keys: list[str] | None = None, for_dataset: bool = False, timeout: int = 60, num_parallel_workers: int = 10, results_timeout: int = 300) → bool

Download items from a specified collection from a heliostat and save them to a designated path.

Parameters

heliostat_catalogpystac.catalog.Catalog

Heliostat catalog STAC.

collection_idstr

ID of the collection to download.

save_folderstr

Name of the folder to save the collection items in.

start_datedatetime, optional

Optional start date to filter the heliostat data. If no start date is provided, data for all time periods is downloaded (Default: None).

end_datedatetime, optional

Optional end date to filter the heliostat data. If no end date is provided, data for all time periods is downloaded (Default: None).

filtered_calibration_keyslist[str]

List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).

for_datasetbool

Whether to save all items in one folder for a dataset (Default: False).

timeoutint

Timeout for downloading heliostat data (Default: 60 seconds).

num_parallel_workersint

Number of parallel workers for downloading heliostat data (Default: 10).

results_timeoutint

Timeout for collecting results from multiple threads (Default: 300 seconds).

Returns

bool

Whether all heliostat items were downloaded successfully.

static _check_collection_keys(collections: list[str]) → None

Check whether provided collection keys are valid.

Parameters

collectionslist[str]

Collection(s) for which (meta)data should be downloaded.

Raises

ValueError

When a listed collection value is invalid.

get_heliostat_data(heliostats: list[str] | None = None, collections: list[str] | None = None, start_date: datetime.datetime | None = None, end_date: datetime.datetime | None = None, filtered_calibration_keys: list[str] | None = None, for_dataset: bool = False, resume_download: bool = True, timeout: int = 60, num_parallel_workers: int = 10, results_timeout: int = 300) → None

Download data for one or more heliostats.

Parameters

heliostatslist[str], optional

Optional list of heliostats whose data should be downloaded, if None data for all heliostats is downloaded (Default: None).

collectionslist[str], optional

List of collections to be downloaded. These collections must be one of: calibration, deflectometry, properties. If no list is provided, all collections are downloaded (Default: None).

start_datedatetime, optional

Optional start date to filter the heliostat data. If no start date is provided, data for all time periods is downloaded (Default: None).

end_datedatetime, optional

Optional end date to filter the heliostat data. If no end date is provided, data for all time periods is downloaded (Default: None).

filtered_calibration_keyslist[str]

List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).

for_datasetbool

Whether to save all items in one folder for a dataset (Default: False).

resume_downloadbool

Whether to use a checkpoint to resume downloading data in case it is interrupted (Default: True).

timeoutint

Timeout for downloading heliostat data (Default: 60 seconds).

num_parallel_workersint

Number of parallel workers for downloading heliostat data (Default: 10).

results_timeoutint

Timeout for collecting results from multiple threads (Default: 300 seconds).

_download_weather_assets(weather_item: pystac.item.Item, include_juelich: bool, include_dwd: bool, timeout: int = 60) → bool

Download weather assets for Jülich and/or DWD depending on flags.

Parameters

weather_itempystac.item.Item

Weather item containing assets.

include_juelichbool

Whether to include Jülich data.

include_dwdbool

Whether to include DWD data.

timeoutint

Timeout for downloading assets (Default: 60 seconds).

Returns

bool

Whether the download was successful.

get_weather_data(data_sources: list[str] | None = None, start_date: datetime.datetime | None = None, end_date: datetime.datetime | None = None, timeout: int = 60, resume_download: bool = True) → None

Get weather data.

Parameters

data_sourceslist[str], optional

List of weather sources to include. Must be from: DWD, Jülich. If no source is provided, data from all sources is downloaded (Default: None).

start_datedatetime, optional

Optional start date to filter the weather data.

end_datedatetime, optional

Optional end date to filter the weather data.

timeoutint

Timeout for downloading data (Default: 60 seconds).

resume_downloadbool

Whether to use a checkpoint to resume downloading data in case it is interrupted (Default: True).

get_tower_measurements(timeout: int = 60) → None

Download tower measurements.

Parameters

timeoutint

Timeout for downloading measurements.

_process_child_metadata(child: pystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection, base_id: str) → list[dict[str, Any]]

Extract metadata from a heliostat catalog child.

Parameters

childpystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection

Child catalog to extract metadata from.

base_idstr

Base ID used to define the heliostat collection being accessed.

Returns

list[dict]

List of extracted metadata dictionaries.

_process_metadata_with_checkpoint(child: pystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection, collection: str, id_base: str, checkpoint: dict[str, Any], checkpoint_path: pathlib.Path, checkpoint_lock: threading.Lock, temp_dir: pathlib.Path, pbar: tqdm.tqdm) → bool

Process metadata with checkpoint capabilities.

Parameters

childpystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection

Child being processed.

collectionstr

Collection being processed.

id_basestr

Base collection ID being processed.

checkpointdict[str, Any]

The checkpoint data.

checkpoint_pathpathlib.Path

The checkpoint path.

checkpoint_lockLock

The checkpoint lock to avoid threading problems.

temp_dirpathlib.Path

The path to a temporary directory used to save metadata before combining at the end.

pbartqdm

Progress bar.

Returns

bool

Whether the metadata was successfully processed.

get_heliostat_metadata(heliostats: list[str] | None = None, collections: list[str] | None = None, resume_download: bool = True) → None

Download metadata for desired heliostats.

Parameters

heliostatslist[str], optional

Heliostats for which the metadata should be downloaded. If no list is provided, the metadata for all heliostats is downloaded (Default: None).

collectionslist[str], optional

Collection for which metadata should be downloaded. If no list is provided, the metadata for all collections is downloaded (Default: None).

resume_downloadbool

Whether to use a checkpoint to resume downloading data in case it is interrupted (Default: True).

static _check_filtered_calibration_keys(filtered_calibration_keys: list[str] | None = None) → list[str]

Check that the filtered calibration keys are acceptable.

Parameters

filtered_calibration_keyslist[str], optional

List of keys to filter the calibration data.

Returns

list[str]

The accepted filtered calibration keys.

Raises

ValueError

When invalid calibration keys are provided.

get_single_calibration_item_by_id(heliostat_id: str, item_id: int, filtered_calibration_keys: list[str] | None = None, benchmark_split: str | None = None, pbar: tqdm.tqdm | None = None, verbose: bool = True, timeout: int = 60) → None

Download a specific calibration item from a specific heliostat given the calibration item ID.

Parameters

heliostat_idstr

ID of the considered heliostat.

item_idint

ID of the item to be downloaded.

filtered_calibration_keyslist[str]

List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).

benchmark_splitstr, optional

Benchmark split that this item is being downloaded for. If None, the heliostat will be downloaded according to the PAINT default structure (Default: None).

pbartqdm, optional

Progress bar, optional (Default: None).

verbosebool

Whether log messages should be printed (Default: True).

timeoutint

Timeout for downloading calibration item (Default: 60 seconds).

get_multiple_calibration_items_by_id(heliostat_items_dict: dict[str, list[int] | None], filtered_calibration_keys: list[str] | None = None, timeout: int = 60, num_parallel_workers: int = 10, results_timeout: int = 300) → None

Download multiple calibration items for multiple heliostats.

Parameters

heliostat_items_dictdict[str, list[int] | None]

A dictionary mapping heliostat IDs to lists of item IDs to be downloaded. If the list of item IDs is None, all items for that heliostat will be downloaded.

filtered_calibration_keyslist[str]

List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).

timeoutint

Timeout for downloading individual items in a split.

num_parallel_workersint

Number of parallel workers for downloading data (Default: 10).

results_timeoutint

Timeout for collecting results from multiple threads (Default: 300 seconds).