Skip to content

BCDA-9089: Add IOTA file pattern to cclf parser #1319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
michaeljvaldes merged 6 commits into from
Feb 24, 2026
Merged

BCDA-9089: Add IOTA file pattern to cclf parser #1319

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4dd04df
BCDA-9089: add IOTA to cclf file patterns
michaeljvaldes Feb 23, 2026
ca5330f
Add iota cclf file tests
michaeljvaldes Feb 23, 2026
a2f2f5e
Merge branch 'main' into mvaldes/BCDA-9089
bhagatparwinder Feb 24, 2026
d234932
Update iota cclf pattern to include PRT identifier
michaeljvaldes Feb 24, 2026
675c587
Add test for valid iota cclf zip archive name
michaeljvaldes Feb 24, 2026
e9210f8
Add tests for iota aco ids
michaeljvaldes Feb 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions bcda/cclf/parser.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,9 +158,11 @@ func getCCLFFileMetadata(cmsID, fileName string) (cclfFileMetadata, error) {
aco = `(?:\.ACO)`
bcd = `(?:BCD\.)`

// CCLF filename convention for SSP with BCD identifier: P.BCD.A****.ZC[0|8][Y|R]**.Dyymmdd.Thhmmsst
// CCLF file name convention for SSP with BCD identifier: P.BCD.A****.ZC[0|8][Y|R]**.Dyymmdd.Thhmmsst
ssp = `A\d{4}`
// CCLF filename convention for NGACO: P.V***.ACO.ZC[0|8][Y|R].Dyymmdd.Thhmmsst
// CCLF file name convention for IOTA with BCD identifier: P.BCD.IOTA***.ZC(Y|R).Dyymmdd.Thhmmsst
iotaPattern = `IOTA\d{3}`
Copy link
Contributor

@laurenkrugen-navapbc laurenkrugen-navapbc Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could we rename the other names to follow this convention or could we just call this variable iota?

Copy link
Contributor

@bhagatparwinder bhagatparwinder Feb 23, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding to this, isn't the suffix the same as others, too? ZC[0|8](Y|R)(\d{2})

Edit: I am referring to the comment.

Copy link
Contributor Author

@michaeljvaldes michaeljvaldes Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"iota" is actually a keyword in golang 😓
I updated it to "iotaPrt" since it contains an additional "PRT" section

// CCLF file name convention for NGACO: P.V***.ACO.ZC[0|8][Y|R].Dyymmdd.Thhmmsst
ngaco = `V\d{3}`
// CCLF file name convention for CEC: P.CEC.ZC[0|8][Y|R].Dyymmdd.Thhmmsst
cec = `CEC`
Expand All @@ -175,9 +177,17 @@ func getCCLFFileMetadata(cmsID, fileName string) (cclfFileMetadata, error) {
// CCLF file name convention for SBX: P.SBX*****.ACO.ZC(Y|R)**.Dyymmdd.Thhmmsst
sandbox = `SBX[A-Z]{2}\d{3}`

pattern = prefix + `(` + bcd + ssp + `|` + ngaco + aco + `|` + cec +
`|` + ckcc + aco + `|` + kcf + aco + `|` + dc + aco +
`|` + test + aco + `|` + sandbox + aco + `)` + suffix
pattern = prefix + `(` +
bcd + ssp + `|` +
bcd + iotaPattern + `|` +
ngaco + aco + `|` +
cec + `|` +
ckcc + aco + `|` +
kcf + aco + `|` +
dc + aco + `|` +
test + aco + `|` +
sandbox + aco +
`)` + suffix
)

filenameRegexp := regexp.MustCompile(pattern)
Expand Down
56 changes: 47 additions & 9 deletions bcda/cclf/parser_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,15 +197,16 @@ func TestValidateCSVFileName(t *testing.T) {

func TestGetCCLFMetadata(t *testing.T) {
const (
sspID, cecID, ngacoID, ckccID, kcfID, dcID, testID, sbxID = "A9999", "E9999", "V999", "C9999", "K9999", "D9999", "TEST999", "SBXBD001"
sspProd, sspTest = "P.BCD." + sspID, "T.BCD." + sspID
cecProd, cecTest = "P.CEC", "T.CEC"
ngacoProd, ngacoTest = "P." + ngacoID + ".ACO", "T." + ngacoID + ".ACO"
ckccProd, ckccTest = "P." + ckccID + ".ACO", "T." + ckccID + ".ACO"
kcfProd, kcfTest = "P." + kcfID + ".ACO", "T." + kcfID + ".ACO"
dcProd, dcTest = "P." + dcID + ".ACO", "T." + dcID + ".ACO"
testProd, testTest = "P." + testID + ".ACO", "T." + testID + ".ACO"
sbxProd, sbxTest = "P." + sbxID + ".ACO", "T." + sbxID + ".ACO"
sspID, iotaID, cecID, ngacoID, ckccID, kcfID, dcID, testID, sbxID = "A9999", "IOTA965", "E9999", "V999", "C9999", "K9999", "D9999", "TEST999", "SBXBD001"
sspProd, sspTest = "P.BCD." + sspID, "T.BCD." + sspID
iotaProd, iotaTest = "P.BCD." + iotaID, "T.BCD." + iotaID
cecProd, cecTest = "P.CEC", "T.CEC"
ngacoProd, ngacoTest = "P." + ngacoID + ".ACO", "T." + ngacoID + ".ACO"
ckccProd, ckccTest = "P." + ckccID + ".ACO", "T." + ckccID + ".ACO"
kcfProd, kcfTest = "P." + kcfID + ".ACO", "T." + kcfID + ".ACO"
dcProd, dcTest = "P." + dcID + ".ACO", "T." + dcID + ".ACO"
testProd, testTest = "P." + testID + ".ACO", "T." + testID + ".ACO"
sbxProd, sbxTest = "P." + sbxID + ".ACO", "T." + sbxID + ".ACO"
)

start := time.Now()
Expand All @@ -228,6 +229,7 @@ func TestGetCCLFMetadata(t *testing.T) {
assert.NoError(t, err)
sspProdFile, sspTestFile, sspRunoutFile := gen(sspProd, validTime), gen(sspTest, validTime),
strings.Replace(gen(sspProd, validTime), "ZC8Y", "ZC8R", 1)
iotaProdFile, iotaTestFile, iotaRunoutFile := gen(iotaProd, validTime), gen(iotaTest, validTime), strings.Replace(gen(iotaProd, validTime), "ZC8Y", "ZC8R", 1)
Copy link
Collaborator

@carlpartridge carlpartridge Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe Im misreading but IOTAs middle section should be ZC(Y|R), as opposed to some of the others which tracks the CCLF file number (eg ZC8Y). If so, then I would expect this to fail?

Copy link
Contributor

@laurenkrugen-navapbc laurenkrugen-navapbc Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, it looks like the naming convention doesn't support a performance year. It looks like it's missing?

P.BCD.IOTA***.ZC(Y|R).Dyymmdd.Thhmmsst

Copy link
Collaborator

@carlpartridge carlpartridge Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I cant load jira right now but it looks like the example was P.BCD.IOTA123.ZCY26.D260206.T1125000. I might have missed that in the regex I put in the ticket.

Copy link
Contributor Author

@michaeljvaldes michaeljvaldes Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the PR and the ticket to match new requirements -- both zip archive name and contained files should track with other cclf file conventions, with an extra "PRT" section in the latter
zip archive: P.BCD.IOTA123.ZCY26.D260206.T1125000
contained file: P.IOTA123.PRT.ZC0Y26.D260201.T0737324

cecProdFile, cecTestFile := gen(cecProd, validTime), gen(cecTest, validTime)
ngacoProdFile, ngacoTestFile := gen(ngacoProd, validTime), gen(ngacoTest, validTime)
ckccProdFile, ckccTestFile := gen(ckccProd, validTime), gen(ckccTest, validTime)
Expand Down Expand Up @@ -285,6 +287,42 @@ func TestGetCCLFMetadata(t *testing.T) {
fileType: models.FileTypeRunout,
},
},
{
"Production IOTA file", iotaID, iotaProdFile, "",
cclfFileMetadata{
env: "production",
name: iotaProdFile,
cclfNum: 8,
acoID: iotaID,
timestamp: validTime,
perfYear: perfYear,
fileType: models.FileTypeDefault,
},
},
{
"Test IOTA file", iotaID, iotaTestFile, "",
cclfFileMetadata{
env: "test",
name: iotaTestFile,
cclfNum: 8,
acoID: iotaID,
timestamp: validTime,
perfYear: perfYear,
fileType: models.FileTypeDefault,
},
},
{
"Runout IOTA file", iotaID, iotaRunoutFile, "",
cclfFileMetadata{
env: "production",
name: iotaRunoutFile,
cclfNum: 8,
acoID: iotaID,
timestamp: validTime,
perfYear: perfYear,
fileType: models.FileTypeRunout,
},
},
{
"Production CEC file", cecID, cecProdFile, "",
cclfFileMetadata{
Expand Down