Handling no received data from WeatherSource by PhilippSchmelter · Pull Request #1145 · ie3-institute/PowerSystemDataModel

PhilippSchmelter · 2024-08-27T21:11:18Z

Resolves #554
Added an explicit Exception for handling no received data from WeatherSource to prevent the pipeline moving on without data.

This reverts commit 529bf23

src/main/java/edu/ie3/datamodel/exceptions/NoWeatherDataException.java

…d changed NoDataException

PhilippSchmelter · 2024-09-30T21:24:55Z

Handling CSV-Source done

all getWeather() which returned optional
//TODO
REST of
SQL
Influx
Couchbase

This reverts commit ef08123.

…nto ps/#554-HandlingNoWeatherData

This reverts commit fd0823b.

danielfeismann

Thank you for including my points. I do have another one or two points which are a bit unclear to me

danielfeismann · 2025-08-21T11:53:05Z

CHANGELOG.md

 ### Added
 - Enhance `TimeSeriesSource` with method to retrieve all time keys after a given key [#543](https://github.com/ie3-institute/PowerSystemDataModel/issues/543)
 - Enhance `WeatherSource` with method to retrieve all time keys after a given key [#572](https://github.com/ie3-institute/PowerSystemDataModel/issues/572)
+- Added explicit handling for cases where no weather data is received from any source [#554](https://github.com/ie3-institute/PowerSystemDataModel/issues/554)


Please move this to the end of this subsection

Reminder to move this to the Unreleased/Snapshot section

src/main/java/edu/ie3/datamodel/exceptions/NoDataException.java

src/test/groovy/edu/ie3/datamodel/io/source/influxdb/InfluxDbWeatherSourceIconIT.groovy

src/test/groovy/edu/ie3/datamodel/io/source/influxdb/InfluxDbWeatherSourceCosmoIT.groovy

danielfeismann · 2025-08-21T12:06:36Z

src/main/java/edu/ie3/datamodel/io/source/influxdb/InfluxDbWeatherSource.java

-          IndividualTimeSeries<WeatherValue> timeSeries =
-              new IndividualTimeSeries<>(timeBasedValues);
-          coordinateToTimeSeries.put(entry.getKey(), timeSeries);
+          if (!timeBasedValues.isEmpty()) {


Should it also lead to an exception in case timeBasedValues is empty, since one could assume we would expect weather data when asking for it and those aren't there? Not sure about this, I might have to rethink on this.

Also, testing this would also be great.

I tested throwing an exception when timeBasedValues is empty, but that broke existing behavior and tests. For now we keep it simple: invalid coordinates throw a NoDataException, but valid coordinates with no data just return an empty series. Or do we really want it that strict? If so we could use my last commit which I reverted again.

It'd be great to have equivalent behavior across types of data sources, at least where it's possible

src/main/java/edu/ie3/datamodel/exceptions/NoDataException.java

src/main/java/edu/ie3/datamodel/io/source/couchbase/CouchbaseWeatherSource.java

This reverts commit bdd0ae8.

sebastian-peter

While looking through this, it came to my mind that some of these errors we could try and heal instead, before throwing an exception.

I realize this requires a bit more effort, but maybe it still makes sense to handle it in this PR? On the other hand, SIMONA has to deal with these new types of exceptions anyway, and at a later step it would crash less because we healed the problem instead.

sebastian-peter · 2025-08-29T13:52:33Z

CHANGELOG.md

 ### Added
 - Enhance `TimeSeriesSource` with method to retrieve all time keys after a given key [#543](https://github.com/ie3-institute/PowerSystemDataModel/issues/543)
 - Enhance `WeatherSource` with method to retrieve all time keys after a given key [#572](https://github.com/ie3-institute/PowerSystemDataModel/issues/572)
+- Added explicit handling for cases where no weather data is received from any source [#554](https://github.com/ie3-institute/PowerSystemDataModel/issues/554)


Reminder to move this to the Unreleased/Snapshot section

sebastian-peter · 2025-08-29T13:58:30Z

src/main/java/edu/ie3/datamodel/io/source/couchbase/CouchbaseWeatherSource.java

+      logger.error("Unable to match coordinate {} to a coordinate ID", coordinate);
+      throw new NoDataException("No coordinate ID found for the given point: " + coordinate);


Logging an error is obsolete when we also throw and exception in the next line, imho

sebastian-peter · 2025-08-29T14:01:07Z

src/main/java/edu/ie3/datamodel/io/source/couchbase/CouchbaseWeatherSource.java

+    if (!invalidCoordinates.isEmpty()) {
+      throw new NoDataException("No data for given coordinates: " + invalidCoordinates);
+    }


Hmm. I think there's a way to heal this instead of breaking right away.
I'd say, if for a coordinate we cannot retrieve weather data, we try to provide the last available data (the available data for the last point in time before now) first. In this case, we also log a warning.
If that also fails though, we should throw an exception.

sebastian-peter · 2025-08-29T14:04:33Z

src/main/java/edu/ie3/datamodel/io/source/csv/CsvWeatherSource.java

+    if (!invalidCoordinates.isEmpty()) {
+      throw new NoDataException("No data for given coordinates: " + invalidCoordinates);
+    }


Same as above

sebastian-peter · 2025-08-29T14:11:58Z

src/main/java/edu/ie3/datamodel/io/source/couchbase/CouchbaseWeatherSource.java

+                  new NoDataException(
+                      "No valid weather data found for the given date and coordinate."));


Same here: Maybe try to take an earlier data point instead?

sebastian-peter · 2025-08-29T14:12:24Z

src/main/java/edu/ie3/datamodel/io/source/csv/CsvWeatherSource.java

+                new NoDataException(
+                    "No weather data found for the given coordinate: " + coordinate));


Same as above

sebastian-peter · 2025-08-29T14:14:34Z

src/main/java/edu/ie3/datamodel/io/source/influxdb/InfluxDbWeatherSource.java

-          IndividualTimeSeries<WeatherValue> timeSeries =
-              new IndividualTimeSeries<>(timeBasedValues);
-          coordinateToTimeSeries.put(entry.getKey(), timeSeries);
+          if (!timeBasedValues.isEmpty()) {


It'd be great to have equivalent behavior across types of data sources, at least where it's possible

Testing method

529bf23

PhilippSchmelter self-assigned this Aug 27, 2024

PhilippSchmelter added 8 commits August 27, 2024 23:14

Fixed format

f72bb75

Fixed format

9447e8b

Fixed Test

e0828af

Revert "Testing method"

6c4c0f8

This reverts commit 529bf23

Merge branch 'dev' into ps/#554-HandlingNoWeatherData

7502d97

Testing NoWeatherDataException

743e99a

Handled NoWeatherData in for CSV Source

74f4816

Handled NoWeatherData in for CSV Source

8e8bed0

sebastian-peter reviewed Sep 11, 2024

View reviewed changes

src/main/java/edu/ie3/datamodel/exceptions/NoWeatherDataException.java Outdated Show resolved Hide resolved

PhilippSchmelter added 14 commits September 11, 2024 23:26

Merge branch 'dev' into ps/#554-HandlingNoWeatherData

21a05a3

Adapted exception throwing for getWeather() which returns optional an…

e0bcfb6

…d changed NoDataException

Started adding NoDataException for SQL source

cee6660

Merge branch 'dev' into ps/#554-HandlingNoWeatherData

8ed3f29

Spotless Applied

f7a21d4

Merge branch 'dev' into ps/#554-HandlingNoWeatherData

646eaa8

Merge branch 'dev' into ps/#554-HandlingNoWeatherData

e666ea0

Removed all occurences of Optional return type for getWeather() methods

171610b

spotless

08a2679

fixed tests

2a4d3b7

Adapted SourceIconIT tests

7d449ee

Fixed tests

c83ca93

Fixed tests

34d78aa

Spotless Apply

19d5c3b

Final

ad8946f

PhilippSchmelter requested a review from t-ober September 30, 2024 22:37

PhilippSchmelter marked this pull request as ready for review September 30, 2024 22:37

PhilippSchmelter requested a review from sebastian-peter September 30, 2024 22:38

PhilippSchmelter added 17 commits August 21, 2025 00:31

some fixes

e8a4fee

some fixes

5cc104b

no more missing data

9f97b00

edited exception

4be6590

bugfix

f20db54

fx

fd0823b

redundant log removed

ef08123

hopefully final

cf8adb3

spotless

701d7e2

more information

b5d2ccf

fix

dfb49df

aa

c75b9fa

Revert "redundant log removed"

dc35434

This reverts commit ef08123.

Merge remote-tracking branch 'origin/ps/#554-HandlingNoWeatherData' i…

24e8d3f

…nto ps/#554-HandlingNoWeatherData

Revert "fx"

0800310

This reverts commit fd0823b.

sql

a2911b0

couchbaase

ce294c5

danielfeismann requested changes Aug 21, 2025

View reviewed changes

PhilippSchmelter added 4 commits August 27, 2025 01:49

hopefully final

c38e7a5

bug

3ebda49

Strict fail-fast on partial results

bdd0ae8

Revert "Strict fail-fast on partial results"

7f6a59c

This reverts commit bdd0ae8.

PhilippSchmelter requested a review from danielfeismann August 27, 2025 01:06

PhilippSchmelter added 4 commits August 27, 2025 03:10

little improvement

183e0c8

removed unnecessary info from exception

bf86b78

spotless...

67000bd

removed unnecessary test

1be0b8e

sebastian-peter requested changes Aug 29, 2025

View reviewed changes

PhilippSchmelter added 2 commits August 30, 2025 00:04

Merge branch 'dev' into ps/#554-HandlingNoWeatherData

4b81cd1

Merge branch 'dev' into ps/#554-HandlingNoWeatherData

b67dbd6

		logger.error("Unable to match coordinate {} to a coordinate ID", coordinate);
		throw new NoDataException("No coordinate ID found for the given point: " + coordinate);

		new NoDataException(
		"No valid weather data found for the given date and coordinate."));

		new NoDataException(
		"No weather data found for the given coordinate: " + coordinate));

Comments

Conversation

PhilippSchmelter commented Aug 27, 2024

Uh oh!

Uh oh!

PhilippSchmelter commented Sep 30, 2024

Uh oh!

danielfeismann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sebastian-peter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants