Issue #TG-488 : MergeScript Implementation#67
Issue #TG-488 : MergeScript Implementation#67RevathiKotla wants to merge 7 commits intoSunbird-Obsrv:release-3.4.0from
Conversation
| case class MergeScriptConfig(`type`: String, id: String, frequency: String, basePath: String, rollup: Integer, rollupAge: Option[String] = None, rollupCol: Option[String] = None, rollupRange: Option[Integer] = None, | ||
| merge: MergeFiles, container: String, postContainer: Option[String] = None, deltaFileAccess: Option[Boolean] = Option(true), reportFileAccess: Option[Boolean] = Option(true)) | ||
| case class MergeFiles(files: List[Map[String, String]], dims: List[String]) | ||
|
|
There was a problem hiding this comment.
Rename as MergeConfig. This is no longer a script
| val deltaDF = downloadAzureFile(filePaths("deltaPath"), | ||
| mergeConfig.deltaFileAccess.getOrElse(true), mergeConfig.container) | ||
|
|
||
| val reportDF = downloadAzureFile(filePaths("reportPath"), mergeConfig.reportFileAccess.getOrElse(true), | ||
| mergeConfig.postContainer.getOrElse("reports")) |
There was a problem hiding this comment.
Why do we need to download file?
| mergeDF.saveToBlobStore(mergeResult._2, "csv", FilenameUtils.removeExtension(path.getName), Option(Map("header" -> "true", "mode" -> "overwrite")), None) | ||
| mergeDF.saveToBlobStore(mergeResult._2, "json", FilenameUtils.removeExtension(path.getName), Option(Map("header" -> "true", "mode" -> "overwrite")), None) |
There was a problem hiding this comment.
Can we rename the older file with date before overwriting?
| if (mergeConfig.rollup > 0) { | ||
| val defaultFormat = "dd-MM-yyyy" | ||
| val reportDfColumns = reportDF.columns |
There was a problem hiding this comment.
We would need an append mode where we just append the delta data to the existing data on few columns for the same on the same rollupCol? For ex: adding data to a cumulative report (yearly or monthly or weekly cumulative files)
|
Kudos, SonarCloud Quality Gate passed!
|
2 participants
No description provided.