cherry-pick commits from v6.4.0 to v6.4.2 #4
Conversation
PXF's automation test framework includes a helper method (`addPathToPxfClassPath`) that modifies pxf-env.sh to set `PXF_LOADER_PATH`. The problem is that the methods reads the content of pxf-env.sh from $PXF_BASE/conf on the local host which may not be the same as the file on a multinode cluster. Since the remote configuration is copied down from the remote cluster to a temporary working directory, the helper method should load pxf-env.sh from this location. Additionally, this commit increases the size of PXF heap for multinode tests. The original heap size of 512 MB was too small for some of our tests but this wasn't noticed until now because the automation framework modified pxf-env.sh without preserving any changes made to it during the installation and setup of PXF on the multinode cluster. Authored-by: Bradford D. Boyle <[email protected]>
… (#858) This commit fixes an issue that could occur with multi-line JSON files. Previously, it was possible for a fragment to improperly parse a JSON object if the split started in the middle of a string, causing wrong results. This commit now uses the HdfsFileFragmenter for multi-line JSON files.
* docs - misc updates/additions to the orc write docs * reword per bradford * more rewording per bradford
…(#868) In the case that there is no pivnet artifact to pull down, do not error out. Instead, keep whatever artifact was already present in the bucket.
… (#870) Commit 94f8cca added a check that the length of received strings (as determined by strnlen) matched the length of the string as determined by the Java-based PXF service; if they did not, this was treated as a fatal error. Some users have reported that they have ORC/Parquet files with strings that contain ASCCII NUL-bytes. These strings would not have a strnlen calculate length that matches with the Java string lenght. This commit removes the requirements that the lengths be equal and instead logs a debug message when they do not match. Authored-by: Bradford D. Boyle <[email protected]>
|
Cool, thanks @ostinru! |
|
Hi @gfphoenix78 could you please help review this PR? Thanks! |
|
@tuhaihe do you have concourse-ci running on premise? How do you test PXF PRs? |
Added PR with fast ci: #6 And I am working on heavy ci: open-gpdb/pxf#3
By doing this I am expecting that developers will be able to run tests & debug it locally in docker. |
|
@gfphoenix78 - I was trying to pick all commits (so, next cherry-picks will be easier to do). Here is a list of commits I picked: |
can we merge this already? or should we wait for CI setup? |
Finally I have merged our heavy CI pipeline [1]. Currently it checks that we can produce deb packages (this includes: java tests, compilation of go, java and C code). |
IMO, the CI setup needs more time. We can merge the PR first. If @gfphoenix78 doesn't respond for a while, we can proceed further. |
|
Hi @ostinru thanks for your great work! |
6 participants
Hi Cloudberry team!
This is cherry-pick of commits from original
greenplum/pxf. In this PR I picked all commits from v6.4.0 to v6.4.2.Useful parts:
It also includes changes in
concoursecode (do you run concourse tests?) and (test)automationframework. I keep these changes to make it easier to cherry-pick next portions of commits.