Problem with creating external table file

[mschubert-postech]

When we're trying to export a table to an external file, the file that will be created is overwritten with every chunk size

cursor.execute("""create external table 'aaaaa.txt' using ( remotesource 'python' delimiter '|' socketbufsize 20983886080) as select * from aaaaa""")

We set the socketbuffsize as high as possible, but for large tables even this sockerbuffsize is not high enough.
To make it maybe a little clearer: when we're using the standard socketbuffsize (which is I think ~ 8mb) the created file has only the entries of the last chunk and has a maximum size of 8mb (the socketbuffsize). So the problem here is, that the created file is overwritten with every chunks instead of adding the chunk to the entries.

Does anyone has a solution for that?

Thank you in advance,
Max

[pocketjoj]

I found this while searching for a solution myself. It looks like the root cause is in the Core.py file around line 2331. There's this block for handling external file data:

       if status == EXTAB_SOCK_DATA:
            # get number of bytes in block
            numBytes = i_unpack(self._read(4))[0]
            try:
                blockBuffer = str(self._read(numBytes),
                                  self._client_encoding)
                fh = open(fname, "w+")
                fh.write(blockBuffer)
                self.log.info("Successfully written data into file")
            except Exception:
                self.log.warning("Error in writing data to file")
            continue

"fh = open(fname, "w+")" is the problem. The 'w+' designation is for read and write -- it does not do appending. I manually changed my Core.py file so that it showed the following for that line:

fh = open(fname, "a")

I believe this function is just for unloading external files, so it wouldn't break anything else. Either way, this did write the full data set to the file for me (instead of repeatedly writing 8MB worth). Hope this helps!