Release v1.15.0-SNAPSHOT: METAR/TAF Parser Integration & Remark Enhancement#45
Merged
bclasky1539 merged 3 commits intomainfrom Feb 7, 2026
Merged
Conversation
…cement Phase 1: Parser Integration (Complete) - Integrated weather-processing parsers into weather-ingestion module - NoaaMetarParser and NoaaTafParser now parse all ingested METAR/TAF data - Fixed parser validation to accept NOAA's ICAO station format (KCLT 062252Z...) - Added pattern for station-first format - Robust fallback handling preserves raw data when parsing fails - Metadata tracking: parsed (boolean), parser_version (String), parse_errors (List) - Creates correct typed instances (NoaaMetarData/NoaaTafData) in all scenarios Phase 2: Remark Field Enhancement - Added copyRemarksToTopLevel() method to extract remark data - Top-level fields now populated: automatedStation, seaLevelPressure, peakWind - Temperature extremes: sixHourMaxTemp, sixHourMinTemp, 24-hour max/min - Pressure data: threeHourPressureTendency, windShift - Type conversions: enum→String, Pressure→Double, Temperature→Double - Null-safe extraction with guard clauses Results: - S3 now receives fully parsed aviation weather data with 50+ fields - Rich metadata enables advanced querying and analytics - All 198 tests passing (9 integration tests with real NOAA data) - 100% success rate on production METAR/TAF parsing - End-to-end validated: NOAA → Parser → S3 (407ms avg) Technical Details: - Parser validation patterns: date-prefix, type-prefix, station-first (NEW) - Fallback creates correct subclasses with all required fields populated - Remark extraction converts complex types to simple serializable types - Production-ready with comprehensive error handling Files Changed: - weather-ingestion: NoaaAviationWeatherClient (parser integration) - weather-processing: NoaaMetarParser (validation fix + remark extraction) - weather-processing: NoaaMetarParser.canParse() (ICAO format support) - Version: 1.14.0-SNAPSHOT → 1.15.0-SNAPSHOT (all POMs) - CHANGELOG.md updated with complete release notes Testing: - Integration: 9 tests with real NOAA data (KCLT, KJFK, KLAX, KORD, etc.) - Unit: 198 total tests (0 failures, 0 errors) - Coverage: Parser integration paths fully tested - Validated: Production ingestion to S3 with full parsing
📝 WalkthroughWalkthroughBumps project to 1.15.0-SNAPSHOT and integrates NOAA METAR/TAF parsing into ingestion. NoaaAviationWeatherClient now attempts structured parsing via NoaaMetarParser/NoaaTafParser, falls back to unparsed objects on failure. NoaaMetarParser copies remark fields to top-level and accepts ICAO station+time formats; integration tests and changelog added. Changes
Sequence DiagramsequenceDiagram
participant Client as Client
participant NAWClient as NoaaAviationWeatherClient
participant METARParser as NoaaMetarParser
participant TAFParser as NoaaTafParser
participant Metadata as Metadata
Client->>NAWClient: requestMetar(station)
NAWClient->>NAWClient: fetch raw METAR
NAWClient->>METARParser: parse(rawMetarLine)
alt parsed
METARParser->>METARParser: copyRemarksToTopLevel()
METARParser-->>NAWClient: parsed NoaaWeatherData
NAWClient->>Metadata: attach(full_response, fetch_timestamp, parsed=true, parser_version)
NAWClient-->>Client: parsed NoaaWeatherData
else parse failure
METARParser-->>NAWClient: parse error
NAWClient->>NAWClient: build NoaaMetarData (unparsed) with parse_error
NAWClient->>Metadata: attach(parse_error, raw_data)
NAWClient-->>Client: fallback NoaaMetarData
end
Client->>NAWClient: requestTaf(station)
NAWClient->>NAWClient: fetch raw TAF
NAWClient->>TAFParser: parse(rawTafText)
alt parsed
TAFParser-->>NAWClient: parsed NoaaWeatherData
NAWClient->>Metadata: attach(full_response, fetch_timestamp, parsed=true, parser_version)
NAWClient-->>Client: parsed NoaaWeatherData
else parse failure
TAFParser-->>NAWClient: parse error
NAWClient->>NAWClient: build NoaaTafData (unparsed) with parse_error
NAWClient->>Metadata: attach(parse_error, raw_data)
NAWClient-->>Client: fallback NoaaTafData
end
Estimated Code Review Effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
No actionable comments were generated in the recent review. 🎉 Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@CHANGELOG.md`:
- Line 20: Remove the stray empty bullet in CHANGELOG.md by deleting the
isolated "-" between the "Parser Integration" and "Remark Field Enhancement"
sections; ensure the surrounding section headings and bullets remain intact and
no extra blank lines are introduced so the changelog formatting stays
consistent.
In
`@noakweather-platform/weather-ingestion/src/test/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClientParserIntegrationTest.java`:
- Around line 213-230: The test method testParserFallback in
NoaaAviationWeatherClientParserIntegrationTest is misnamed because it calls
client.fetchMetarReport("KATL") which returns normal parsed data; either rename
the test to something like testMetadataPresent/testParserMetadata to reflect it
only asserts metadata presence, or modify the test to exercise the real fallback
path by injecting/mocking an unparseable METAR response into the client (e.g.,
stub the HTTP/response layer used by fetchMetarReport) and then assert metadata
contains parsed=false; update assertions and method name accordingly (refer to
testParserFallback, client.fetchMetarReport, and the test class name to locate
code).
🧹 Nitpick comments (3)
noakweather-platform/weather-ingestion/src/test/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClientParserIntegrationTest.java (2)
42-42: Consider adding@Tag("integration")for selective test execution.Since these tests require network access to NOAA servers (as noted in the Javadoc), adding a JUnit 5 tag would allow CI/CD pipelines to easily skip these tests when running in environments without internet access.
♻️ Proposed enhancement
+import org.junit.jupiter.api.Tag; + /** * Integration tests for METAR/TAF parser integration. ... */ +@Tag("integration") class NoaaAviationWeatherClientParserIntegrationTest {
253-265: Hardcoded parser version creates coupling to implementation detail.The assertion
assertEquals("2.0", data.getMetadata().get("parser_version"))will break if the version constant changes inNoaaAviationWeatherClient. Consider verifying the version is non-null/non-empty instead, or extracting the version constant to a shared location.♻️ Alternative assertion
- assertEquals("2.0", data.getMetadata().get("parser_version"), - "Parser version should be tracked"); + assertNotNull(data.getMetadata().get("parser_version"), + "Parser version should be tracked"); + assertFalse(data.getMetadata().get("parser_version").isEmpty(), + "Parser version should not be empty");noakweather-platform/weather-ingestion/src/main/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClient.java (1)
463-463: Minor: Inconsistent spacing inaddMetadata()calls.There's an extra space before the
FULL_RESPONSEconstant in several calls:
- Line 463:
parsedData.addMetadata( FULL_RESPONSE, rawText);- Line 486:
fallbackData.addMetadata( FULL_RESPONSE, rawText);- Line 534:
parsedData.addMetadata( FULL_RESPONSE, rawText);- Line 557:
fallbackData.addMetadata( FULL_RESPONSE, rawText);While functionally correct, this is inconsistent with the other
addMetadatacalls.📝 Proposed fix (example for line 463)
- parsedData.addMetadata( FULL_RESPONSE, rawText); + parsedData.addMetadata(FULL_RESPONSE, rawText);Also applies to: 486-486, 534-534, 557-557
...va/weather/ingestion/service/source/noaa/NoaaAviationWeatherClientParserIntegrationTest.java
Show resolved
Hide resolved
SonarQube identified regex patterns vulnerable to catastrophic
backtracking (ReDoS attacks). Changed \s+.* to \s+\S.* in all
three validation patterns in canParse() method.
Patterns fixed:
- Date pattern: \d{4}/\d{2}/\d{2}\s+\S.*
- METAR/SPECI pattern: ^\s*(METAR|SPECI)\s+\S.*
- Station-first pattern: ^[A-Z]{4}\s+\d{6}Z\s+\S.*
NoaaAviationWeatherClientParserIntegrationTest.java
Rename misleading test: testParserFallback → testMetadataPresent
CodeRabbit identified that testParserFallback was misleading
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In
`@noakweather-platform/weather-ingestion/src/test/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClientParserIntegrationTest.java`:
- Around line 19-49: The tests in NoaaAviationWeatherClientParserIntegrationTest
make live HTTP calls (e.g., client.fetchMetarReport("KCLT") and
client.fetchTafReport(...)) and should be gated to avoid CI flakiness; annotate
the test class or individual tests with a tag like `@Tag`("integration") and add
an Assumptions.assumeTrue(...) check (based on a system property or environment
variable such as "RUN_LIVE_NOAA_TESTS" or a boolean system property) either in
setUp() or at the start of each test to skip execution when the flag is not set,
so live NOAA calls are only performed when explicitly enabled.
🧹 Nitpick comments (1)
noakweather-platform/weather-ingestion/src/test/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClientParserIntegrationTest.java (1)
260-262: Avoid hard-coding parser version in the integration test.
The version string will change with releases; asserting presence/non-blank (or comparing to a shared constant) makes the test resilient.♻️ Make the assertion robust to version bumps
- assertEquals("2.0", data.getMetadata().get("parser_version"), - "Parser version should be tracked"); + String parserVersion = data.getMetadata().get("parser_version"); + assertNotNull(parserVersion, "Parser version should be tracked"); + assertFalse(parserVersion.isBlank(), "Parser version should be tracked");
Comment on lines
+19
to
+49
| import org.junit.jupiter.api.BeforeEach; | ||
| import org.junit.jupiter.api.Test; | ||
| import org.junit.jupiter.api.DisplayName; | ||
| import weather.exception.WeatherServiceException; | ||
| import weather.model.NoaaMetarData; | ||
| import weather.model.NoaaTafData; | ||
| import weather.model.WeatherData; | ||
|
|
||
| import static org.junit.jupiter.api.Assertions.*; | ||
|
|
||
| /** | ||
| * Integration tests for METAR/TAF parser integration. | ||
| * <p> | ||
| * These tests verify that NoaaAviationWeatherClient correctly uses | ||
| * the comprehensive parsers from weather-processing to create fully | ||
| * structured weather data. | ||
| * <p> | ||
| * NOTE: These tests require network access to NOAA servers. | ||
| * Skip these tests if running in CI/CD without internet access. | ||
| * | ||
| * @author bclasky1539 | ||
| * | ||
| */ | ||
| class NoaaAviationWeatherClientParserIntegrationTest { | ||
|
|
||
| private NoaaAviationWeatherClient client; | ||
|
|
||
| @BeforeEach | ||
| void setUp() { | ||
| client = new NoaaAviationWeatherClient(); | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -type f -name "NoaaAviationWeatherClientParserIntegrationTest.java" 2>/dev/nullRepository: bclasky1539/noakweather-engineering-pipeline
Length of output: 232
🏁 Script executed:
wc -l noakweather-platform/weather-ingestion/src/test/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClientParserIntegrationTest.javaRepository: bclasky1539/noakweather-engineering-pipeline
Length of output: 234
🏁 Script executed:
cat -n noakweather-platform/weather-ingestion/src/test/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClientParserIntegrationTest.javaRepository: bclasky1539/noakweather-engineering-pipeline
Length of output: 12344
🏁 Script executed:
find . -type f -name "NoaaAviationWeatherClient.java" 2>/dev/nullRepository: bclasky1539/noakweather-engineering-pipeline
Length of output: 211
🏁 Script executed:
head -100 ./noakweather-platform/weather-ingestion/src/main/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClient.javaRepository: bclasky1539/noakweather-engineering-pipeline
Length of output: 3818
🏁 Script executed:
rg -t java "@Tag|@Disabled|Assumptions.assumeTrue" --max-count 10Repository: bclasky1539/noakweather-engineering-pipeline
Length of output: 70
🏁 Script executed:
find . -type f -name "pom.xml" -o -name "build.gradle" | head -5Repository: bclasky1539/noakweather-engineering-pipeline
Length of output: 296
🏁 Script executed:
find . -type f \( -name "*.yml" -o -name "*.yaml" -o -name ".github" \) | grep -E "(workflow|ci|pipeline)" | head -10Repository: bclasky1539/noakweather-engineering-pipeline
Length of output: 152
🏁 Script executed:
find . -type d -name ".github" 2>/dev/nullRepository: bclasky1539/noakweather-engineering-pipeline
Length of output: 97
🏁 Script executed:
cat -n ./.github/workflows/maven.ymlRepository: bclasky1539/noakweather-engineering-pipeline
Length of output: 1759
Guard live NOAA calls to prevent CI flakiness.
The test methods make unguarded HTTP requests to external NOAA servers (e.g., client.fetchMetarReport("KCLT") at line 55, client.fetchTafReport() at line 156). While the javadoc documents this requirement, no runtime enforcement exists. External service dependencies cause flaky CI/CD when NOAA servers are unavailable or slow. Wrap tests with a conditional skip using @Tag and Assumptions.assumeTrue().
Suggested gating pattern
-import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Tag;
@@
-class NoaaAviationWeatherClientParserIntegrationTest {
+@Tag("noaa-integration")
+class NoaaAviationWeatherClientParserIntegrationTest {
@@
`@BeforeEach`
void setUp() {
+ Assumptions.assumeTrue(
+ Boolean.parseBoolean(System.getenv("NOAA_INTEGRATION_TESTS")),
+ "Set NOAA_INTEGRATION_TESTS=true to enable live NOAA calls");
client = new NoaaAviationWeatherClient();
}🤖 Prompt for AI Agents
In
`@noakweather-platform/weather-ingestion/src/test/java/weather/ingestion/service/source/noaa/NoaaAviationWeatherClientParserIntegrationTest.java`
around lines 19 - 49, The tests in
NoaaAviationWeatherClientParserIntegrationTest make live HTTP calls (e.g.,
client.fetchMetarReport("KCLT") and client.fetchTafReport(...)) and should be
gated to avoid CI flakiness; annotate the test class or individual tests with a
tag like `@Tag`("integration") and add an Assumptions.assumeTrue(...) check (based
on a system property or environment variable such as "RUN_LIVE_NOAA_TESTS" or a
boolean system property) either in setUp() or at the start of each test to skip
execution when the flag is not set, so live NOAA calls are only performed when
explicitly enabled.
SonarQube detected unnecessary stubbings in testSchedulePeriodicIngestion_MultipleStations. The test verifies scheduling behavior but doesn't wait for full async execution, so stubs may not be called. Used lenient() for fetchMetarReport and processWeatherData stubs to allow them to be unused in timing-sensitive async tests. This is appropriate for tests that verify scheduling rather than full execution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1: Parser Integration (Complete)
Phase 2: Remark Field Enhancement
Results:
Technical Details:
Files Changed:
Testing:
Summary by CodeRabbit
New Features
Enhancements
Bug Fixes
Tests
Chores