[Bug]: Java heap Space issue

[vkydhyani]

Apache Hop version?

2.15

Java version?

Tried on both 17 and 25

Operating system

Windows

What happened?

Hi Team,

I am getting a Java heap space issue while trying to merge 4 csv file into one excel file.

System configurations are PROD like, 32 Gbs RAM, 4 core, 8 processor

Error :
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-1"
Exception in thread "HopGuiPipelineGraph: redraw timer" Exception in thread "pool-2-thread-1" java.lang.OutOfMemoryError: Java heap space
2025/10/14 09:50:30 - Generate Additional Sheet.0 - ERROR: UnexpectedError:
Exception in thread "Timer-4" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.HashMap$KeySet.toArray(HashMap.java:1002)
at org.apache.commons.vfs2.impl.DefaultFileMonitor.run(DefaultFileMonitor.java:476)
at java.base/java.lang.Thread.runWith(Thread.java:1487)
at java.base/java.lang.Thread.run(Thread.java:1474)
2025/10/14 09:50:30 - Generate Additional Sheet.0 - ERROR: java.lang.OutOfMemoryError: Java heap space

Can you please suggest what is wrong here?

Issue Priority

Priority: 3

Issue Component

Component: Transforms

[bamaer]

What is the file size of your CSV files?
The blocking transforms require all of your data to be buffered in memory before you can write it out to your Excel sheets. You have 32GB of memory in your machine, but Hop only grabs 2GB by default. Try changing the -Xmx parameter in hop-gui.bat, set it to for example 8GB.
If that doesn't work, splitting up your pipeline to load on CSV file into one Excel sheet at a time will eliminate the need for the blocking transforms. You could do this in a single pipeline with a parameter with CSV file to read, there probably isn't any need to create multiple pipelines. You would need 4 different runs instead of a single one, but won't necessarily had a huge impact on your total throughput time.

[vkydhyani]

What is the file size of your CSV files? It has 4 csv files, and the size is also too small. But the number of columns are too much. can this cause any issues.

The blocking transforms require all of your data to be buffered in memory before you can write it out to your Excel sheets.
I am blocking the transform before i read the CSV file for below streams.

You have 32GB of memory in your machine, but Hop only grabs 2GB by default. Try changing the -Xmx parameter in hop-gui.bat, set it to for example 8GB.

I have already tried it, and using "set HOP_OPTIONS=-Xmx8192m" currently.

If that doesn't work, splitting up your pipeline to load on CSV file into one Excel sheet at a time will eliminate the need for the blocking transforms. You could do this in a single pipeline with a parameter with CSV file to read, there probably isn't any need to create multiple pipelines. You would need 4 different runs instead of a single one, but won't necessarily had a huge impact on your total throughput time.

I want to avoid multiple pipeline creation for this small task. Will metadata injection help in this scenario?

[bamaer]

How many rows do you have in the substream that starts with the Generate Rows transform and feeds into the Cartesian join (should only be one row, or your cartesian join will multiple all row numbers from both streams).
Also: even though your CSV files aren't huge, the 4 Excel sheets will be zipped XML in the background. That 40MB CSV file might become quite large in memory. File locking the same file 4 times to write to different sheets also won't help on Windows in this case.
Metadata injection won't change anything if you try to load these 4 files at once, whether you run that directly or through MDI.

[vkydhyani]

generate rows creates one row, which has just the filename of the excel that we need to create, so that all 4 streams can insert into same excel. and Cartesian join will combine one row from "generate rows" and one row from "get File names". It wont be the issue i guess.

Is there a way to free up the memory once the first sheet written on the Excel?

[vkydhyani]

Number of rows in first csv is 35000.

[bamaer]

like I said earlier: adding a parameter to your pipeline to write your sheets one at a time would eliminate the need for blockers. You would only require a single pipeline (more or less the one you have right now, but with an additional parameter) and without the need for blocking transforms). You would have to read your input files multiple times, but it would make the entire process less resource heavy, less headaches and maybe even faster.

[hansva]

I would also check the field definition of the output fiedls of those text file inputs. It could be that you have string types with maximum length and that might cause trying to create 2GB text fields inside the excel file.

[nadment]

Have you enabled the “Stream XLSX data” option?

Perhaps we should enable it by default.

[vkydhyani]

I have tried this and getting below error :

2025/10/15 06:39:36 - Generate Report 2.0 - ERROR: Error writing line :java.lang.IllegalArgumentException: Attempting to write a row[0] in the range [0,0] that is already written to disk.
org.apache.hop.core.exception.HopException:
java.lang.IllegalArgumentException: Attempting to write a row[0] in the range [0,0] that is already written to disk.
Attempting to write a row[0] in the range [0,0] that is already written to disk.