paint.data.stac_client

Attributes

log

A logger for the STAC client.

Classes

StacClient

Initialize the STAC client.

Module Contents

paint.data.stac_client.log: A logger for the STAC client.

class paint.data.stac_client.StacClient(output_dir: str | pathlib.Path, chunk_size: int = 1024 * 1024)

Initialize the STAC client.

Parameters

output_dirstr | pathlib.Path: Output directory for saving the downloaded data.
chunk_sizeint: Chunk size used for download (Default: 1024 * 1024).

output_dir

chunk_size = 1048576

static load_checkpoint(path: pathlib.Path) → dict[str, Any]

Load checkpoint to resume the download from the point it stopped.

Parameters

pathpathlib.Path: The path to the checkpoint.

Returns

dict[str, Any]: The checkpoint data.

static save_checkpoint(path: pathlib.Path, data: dict[str, Any]) → None

Save data to the download checkpoint.

Parameters

pathpathlib.Path: The path to the checkpoint.
datadict[str, Any]: The data to save in the checkpoint.

static mark_done(checkpoint_data: dict[str, Any], catalog_id: str) → None

Mark specific heliostat catalog as downloaded for the checkpoint.

Parameters

checkpoint_datadict[str, Any]: The checkpoint data.
catalog_idstr: The catalog ID to be marked.

static mark_metadata_done(checkpoint_data: dict[str, Any], heliostat_id: str, collection: str) → None

Mark specific heliostat catalog as downloaded for the checkpoint.

Parameters

checkpoint_datadict[str, Any]: The checkpoint data.
heliostat_idstr: The heliostat ID to be marked.
collectionstr: The collection to be marked.

static get_catalog(href: str) → pystac.catalog.Catalog

Help to load a STAC catalog.

Since PySTAC does not check if the URL exists before attempting to download a catalog, we catch any errors raised and raise a value error indicating that the desired catalog does not exist.

Parameters

hrefstr: URL reference to the catalog that should be accessed.

Returns

pystac.catalog.Catalog: Catalog, if it exists.

Raises

ValueError: If the URL does not exist.

static get_child(parent: pystac.catalog.Catalog | pystac.collection.Collection, child_id: str) → pystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection

Help to get a child from a STAC catalog or collection.

Since PySTAC returns None if a child is not present, it is important we log a warning if this is the case.

Parameters

parentpystac.catalog.Catalog | pystac.collection.Collection: Parent STAC catalog or collection.
child_idstr: ID of the child STAC catalog, collection, or item.

Returns

pystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection: Child STAC catalog, collection, or item.

download_file(url: str, file_name: str | pathlib.Path, timeout: int = 60) → bool

Download a file.

Parameters

urlstr: URL of the file to download.
file_namestr | pathlib.Path: File name to be saved.
timeoutint: Timeout for downloading the file (Default: 60 seconds).

Returns

bool: Whether the file was downloaded.

_process_single_heliostat_item(item: pystac.item.Item, start_date: datetime.datetime | None, end_date: datetime.datetime | None, filtered_calibration_keys: list[str] | None, collection_id: str, heliostat_catalog_id: str, save_folder: str, pbar: tqdm.tqdm, for_dataset: bool = False, timeout: int = 60) → bool

Process a single heliostat item.

Parameters

itempystac.item.Item: Item to be processed.
start_datedatetime, optional: Optional start date to filter the heliostat data. If no start date is provided, data for all time periods is downloaded (Default: None).
end_datedatetime, optional: Optional end date to filter the heliostat data. If no end date is provided, data for all time periods is downloaded (Default: None).
filtered_calibration_keyslist[str]: List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).
collection_idstr: ID of the collection to download.
heliostat_catalog_idstr: ID of the considered heliostat catalog.
save_folderstr: Name of the folder to save the collection items in.
pbartqdm: Progress bar.
for_datasetbool: Whether to save all items in one folder for a dataset (Default: False).
timeoutint: Timeout for downloading the heliostat item (Default: 60 seconds).

Returns

bool: Whether the heliostat item was downloaded successfully.

_process_heliostat_items(items: list[pystac.item.Item], start_date: datetime.datetime | None, end_date: datetime.datetime | None, filtered_calibration_keys: list[str] | None, collection_id: str, heliostat_catalog_id: str, save_folder: str, for_dataset: bool = False, timeout: int = 60, num_parallel_workers: int = 10, results_timeout: int = 300) → bool

Process and download items, either with or without date filtering and calibration key filtering.

Parameters

itemslist[pystac.item.Item]: List of items to be processed.
start_datedatetime, optional: Optional start date to filter the heliostat data. If no start date is provided, data for all time periods is downloaded (Default: None).
end_datedatetime, optional: Optional end date to filter the heliostat data. If no end date is provided, data for all time periods is downloaded (Default: None).
filtered_calibration_keyslist[str]: List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).
collection_idstr: ID of the collection to download.
heliostat_catalog_idstr: ID of the considered heliostat catalog.
save_folderstr: Name of the folder to save the collection items in.
for_datasetbool: Whether to save all items in one folder for a dataset (Default: False).
timeoutint: Timeout for downloading the heliostat item (Default: 60 seconds).
num_parallel_workersint: Number of parallel workers for downloading heliostat data (Default: 10).
results_timeoutint: Timeout for collecting results from multiple threads (Default: 300 seconds).

Returns

bool: Whether all heliostat items were downloaded successfully.

_download_heliostat_asset(asset: pystac.asset.Asset, heliostat_catalog_id: str, save_folder: str, for_dataset: bool = False, timeout: int = 60) → bool

Download the asset from the provided URL and save it to the selected location.

Parameters

assetpystac.asset.Asset: STAC asset to be downloaded and saved.
heliostat_catalog_idstr: ID of the considered heliostat catalog.
save_folderstr: Name of the folder to save the asset in.
for_datasetbool: Whether to save all items in one folder for a dataset (Default: False).
timeoutint: Timeout for downloading assets (Default: 60 seconds).

Returns

bool: Whether the asset was successfully downloaded.

_download_heliostat_data(heliostat_catalog: pystac.catalog.Catalog, collection_id: str, save_folder: str, start_date: datetime.datetime | None = None, end_date: datetime.datetime | None = None, filtered_calibration_keys: list[str] | None = None, for_dataset: bool = False, timeout: int = 60, num_parallel_workers: int = 10, results_timeout: int = 300) → bool

Download items from a specified collection from a heliostat and save them to a designated path.

Parameters

heliostat_catalogpystac.catalog.Catalog: Heliostat catalog STAC.
collection_idstr: ID of the collection to download.
save_folderstr: Name of the folder to save the collection items in.
start_datedatetime, optional: Optional start date to filter the heliostat data. If no start date is provided, data for all time periods is downloaded (Default: None).
end_datedatetime, optional: Optional end date to filter the heliostat data. If no end date is provided, data for all time periods is downloaded (Default: None).
filtered_calibration_keyslist[str]: List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).
for_datasetbool: Whether to save all items in one folder for a dataset (Default: False).
timeoutint: Timeout for downloading heliostat data (Default: 60 seconds).
num_parallel_workersint: Number of parallel workers for downloading heliostat data (Default: 10).
results_timeoutint: Timeout for collecting results from multiple threads (Default: 300 seconds).

Returns

bool: Whether all heliostat items were downloaded successfully.

static _check_collection_keys(collections: list[str]) → None

Check whether provided collection keys are valid.

Parameters

collectionslist[str]: Collection(s) for which (meta)data should be downloaded.

Raises

ValueError: When a listed collection value is invalid.

get_heliostat_data(heliostats: list[str] | None = None, collections: list[str] | None = None, start_date: datetime.datetime | None = None, end_date: datetime.datetime | None = None, filtered_calibration_keys: list[str] | None = None, for_dataset: bool = False, resume_download: bool = True, timeout: int = 60, num_parallel_workers: int = 10, results_timeout: int = 300) → None

Download data for one or more heliostats.

Parameters

heliostatslist[str], optional: Optional list of heliostats whose data should be downloaded, if None data for all heliostats is downloaded (Default: None).
collectionslist[str], optional: List of collections to be downloaded. These collections must be one of: calibration, deflectometry, properties. If no list is provided, all collections are downloaded (Default: None).
start_datedatetime, optional: Optional start date to filter the heliostat data. If no start date is provided, data for all time periods is downloaded (Default: None).
end_datedatetime, optional: Optional end date to filter the heliostat data. If no end date is provided, data for all time periods is downloaded (Default: None).
filtered_calibration_keyslist[str]: List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).
for_datasetbool: Whether to save all items in one folder for a dataset (Default: False).
resume_downloadbool: Whether to use a checkpoint to resume downloading data in case it is interrupted (Default: True).
timeoutint: Timeout for downloading heliostat data (Default: 60 seconds).
num_parallel_workersint: Number of parallel workers for downloading heliostat data (Default: 10).
results_timeoutint: Timeout for collecting results from multiple threads (Default: 300 seconds).

_download_weather_assets(weather_item: pystac.item.Item, include_juelich: bool, include_dwd: bool, timeout: int = 60) → bool

Download weather assets for Jülich and/or DWD depending on flags.

Parameters

weather_itempystac.item.Item: Weather item containing assets.
include_juelichbool: Whether to include Jülich data.
include_dwdbool: Whether to include DWD data.
timeoutint: Timeout for downloading assets (Default: 60 seconds).

Returns

bool: Whether the download was successful.

get_weather_data(data_sources: list[str] | None = None, start_date: datetime.datetime | None = None, end_date: datetime.datetime | None = None, timeout: int = 60, resume_download: bool = True) → None

Get weather data.

Parameters

data_sourceslist[str], optional: List of weather sources to include. Must be from: DWD, Jülich. If no source is provided, data from all sources is downloaded (Default: None).
start_datedatetime, optional: Optional start date to filter the weather data.
end_datedatetime, optional: Optional end date to filter the weather data.
timeoutint: Timeout for downloading data (Default: 60 seconds).
resume_downloadbool: Whether to use a checkpoint to resume downloading data in case it is interrupted (Default: True).

get_tower_measurements(timeout: int = 60) → None

Download tower measurements.

Parameters

timeoutint: Timeout for downloading measurements.

_process_child_metadata(child: pystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection, base_id: str) → list[dict[str, Any]]

Extract metadata from a heliostat catalog child.

Parameters

childpystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection: Child catalog to extract metadata from.
base_idstr: Base ID used to define the heliostat collection being accessed.

Returns

list[dict]: List of extracted metadata dictionaries.

_process_metadata_with_checkpoint(child: pystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection, collection: str, id_base: str, checkpoint: dict[str, Any], checkpoint_path: pathlib.Path, checkpoint_lock: threading.Lock, temp_dir: pathlib.Path, pbar: tqdm.tqdm) → bool

Process metadata with checkpoint capabilities.

Parameters

childpystac.catalog.Catalog | pystac.item.Item | pystac.collection.Collection: Child being processed.
collectionstr: Collection being processed.
id_basestr: Base collection ID being processed.
checkpointdict[str, Any]: The checkpoint data.
checkpoint_pathpathlib.Path: The checkpoint path.
checkpoint_lockLock: The checkpoint lock to avoid threading problems.
temp_dirpathlib.Path: The path to a temporary directory used to save metadata before combining at the end.
pbartqdm: Progress bar.

Returns

bool: Whether the metadata was successfully processed.

get_heliostat_metadata(heliostats: list[str] | None = None, collections: list[str] | None = None, resume_download: bool = True) → None

Download metadata for desired heliostats.

Parameters

heliostatslist[str], optional: Heliostats for which the metadata should be downloaded. If no list is provided, the metadata for all heliostats is downloaded (Default: None).
collectionslist[str], optional: Collection for which metadata should be downloaded. If no list is provided, the metadata for all collections is downloaded (Default: None).
resume_downloadbool: Whether to use a checkpoint to resume downloading data in case it is interrupted (Default: True).

static _check_filtered_calibration_keys(filtered_calibration_keys: list[str] | None = None) → list[str]

Check that the filtered calibration keys are acceptable.

Parameters

filtered_calibration_keyslist[str], optional: List of keys to filter the calibration data.

Returns

list[str]: The accepted filtered calibration keys.

Raises

ValueError: When invalid calibration keys are provided.

get_single_calibration_item_by_id(heliostat_id: str, item_id: int, filtered_calibration_keys: list[str] | None = None, benchmark_split: str | None = None, pbar: tqdm.tqdm | None = None, verbose: bool = True, timeout: int = 60) → None

Download a specific calibration item from a specific heliostat given the calibration item ID.

Parameters

heliostat_idstr: ID of the considered heliostat.
item_idint: ID of the item to be downloaded.
filtered_calibration_keyslist[str]: List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).
benchmark_splitstr, optional: Benchmark split that this item is being downloaded for. If None, the heliostat will be downloaded according to the PAINT default structure (Default: None).
pbartqdm, optional: Progress bar, optional (Default: None).
verbosebool: Whether log messages should be printed (Default: True).
timeoutint: Timeout for downloading calibration item (Default: 60 seconds).

get_multiple_calibration_items_by_id(heliostat_items_dict: dict[str, list[int] | None], filtered_calibration_keys: list[str] | None = None, timeout: int = 60, num_parallel_workers: int = 10, results_timeout: int = 300) → None

Download multiple calibration items for multiple heliostats.

Parameters

heliostat_items_dictdict[str, list[int] | None]: A dictionary mapping heliostat IDs to lists of item IDs to be downloaded. If the list of item IDs is None, all items for that heliostat will be downloaded.
filtered_calibration_keyslist[str]: List of keys to filter the calibration data. These keys must be one of: raw_image, cropped_image, flux_image, flux_centered_image, calibration_properties. If no list is provided, all calibration data is downloaded (Default: None).
timeoutint: Timeout for downloading individual items in a split.
num_parallel_workersint: Number of parallel workers for downloading data (Default: 10).
results_timeoutint: Timeout for collecting results from multiple threads (Default: 300 seconds).