add ability to switch off/on generation of Parquet Files

[mozzy11]

Fixes #1073

Add ability to switch of creation of a parquet DWH in case of syncying betwen FHIR servers

Added a flag to the createParquetDwh to the controller to switch off/on creation of parquet DWH

E2E test

Adedd e2e tests for synching from a hapi fhir sever to another using the pipeline controller for both FULL and INCREMENTAL modes while swtching on/off creation of parquet DWH

TESTED:
Testes Locally syncying between FHIR server while the parquet DWH is switched off/on

Checklist: I completed these to help reviewers :)

• I have read and will follow the review process.

• I am familiar with Google Style Guides for the language I have coded in.

  No? Please take some time and review Java and Python style guides.

• My IDE is configured to follow the Google code styles.

  No? Unsure? -> configure your IDE.

• I have added tests to cover my changes. (If you refactored existing code that was well tested you do not have to add tests)

• I ran mvn clean package right before creating this pull request and added all formatting changes to my commit.

• All new and existing tests passed.

• My pull request is based on the latest changes of the master branch.

  No? Unsure? -> execute command git pull --rebase upstream master

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 33.33333% with 6 lines in your changes missing coverage. Please review.

Project coverage is 51.41%. Comparing base (eb96271) to head (e9933e4).

Files with missing lines	Patch %	Lines
...java/com/google/fhir/analytics/DataProperties.java	33.33%	1 Missing and 3 partials ⚠️
...a/com/google/fhir/analytics/ConvertResourceFn.java	0.00%	0 Missing and 1 partial ⚠️
...a/com/google/fhir/analytics/FetchSearchPageFn.java	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #1074      +/-   ##
============================================
- Coverage     51.48%   51.41%   -0.07%     
+ Complexity      716      715       -1     
============================================
  Files            95       95              
  Lines          5909     5916       +7     
  Branches        787      790       +3     
============================================
  Hits           3042     3042              
- Misses         2578     2582       +4     
- Partials        289      292       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mozzy11]

cc @bashir2

[mozzy11]

cc @bashir2

[bashir2]

cc @bashir2

Sorry @mozzy11 this fell off my radar (perhaps due to DevDays); thanks for the reminder. I'll provide some feedback by tomorrow.

[bashir2]

Thanks @mozzy11 for this change. I had a look and made some comments but in general I feel we need to think deeper about the implications of skipping Parquet file generation; I feel there are still scenarios not covered in your change (beyond what I have commented below) but need to think more about this.

[mozzy11]

cc @bashir2 . i have worked on the review comments

[mozzy11]

Hello @bashir2 . I addresed the comments above

[bashir2]

Thanks @mozzy11 for the changes. Most of the comments below are minor. The main issue that I think is still remaining is the merger pipeline comment.

Do you mind attending the dev. call this week and we discuss this in more details there? I think it might be easier to clarify things that way and try to merge this sooner.

[mozzy11]

Thanks @bashir2 , ill attend the dev call this week

[mozzy11]

Hello @bashir2 . sorry i missed the dev call. Let me know if you have any other time to have a quick call

[mozzy11]

Thanks @bashir2