[Feature Request]: TableInput result in RAM

[GRawhideMart]

What would you like to happen?

I don't know if this feature already exists, but it would be nice if, in the TableInput transform, the user would be able to store the result in the RAM. This is to prevent write permission issues on the file system that, more often than not, exist on servers.

Issue Priority

Priority: 2

Issue Component

Component: Transforms

[hansva]

Not sure what the actual question is, where are you getting the file permission issue? There are only a couple of transforms that will store data on disk. All our data is being kept in memory. The exceptions are: sort rows, blocking transforms and our output transforms.

[GRawhideMart]

I am using Hop inside a Docker container running in the WSL of a Windows 11 machine. The TableInput transform with 7+ fields doesn't work (the preview gets stuck and the pipeline does not run), but with SELECT * it works. I guess my actual question is "why may this happen"?

[hansva]

Is there anything in the logs?
And the query that you are using has no conditions/where clause/aggregations?
It could be that it's just waiting for the database to execute the query.

[GRawhideMart]

The log just mentions that "the Connection object is closed". It's like it doesn't handle more than 16 fields. Problem is, I need to mention them explicitly because I need to alias them, so I cannot do SELECT *.

Just for the sake of completeness, the log says:

Invalid state, the Connection object is closed.

        at org.apache.hop.core.database.Database.getQueryFieldsFallback(Database.java:2263)
        at org.apache.hop.core.database.Database.getQueryFields(Database.java:2090)
        at org.apache.hop.pipeline.transforms.tableinput.TableInputMeta.getFields(TableInputMeta.java:221)
        ... 11 more
Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
        at net.sourceforge.jtds.jdbc.JtdsConnection.checkOpen(JtdsConnection.java:1744)
        at net.sourceforge.jtds.jdbc.JtdsConnection.createStatement(JtdsConnection.java:2370)
        at org.apache.hop.core.database.Database.getQueryFieldsFallback(Database.java:2201)
        ... 13 more

[hansva]

Do you need to use Jtds? if possible I would use the native mssql driver/connection.
Jtds drivers are over 10 years old now.

[GRawhideMart]

Sadly, yeah. I'm migrating a SQLServer 2008 database, so the native connection doesn't work. Already tried that.

[hansva]

I'm guessing you can not provide the full stack? and that's the only lines you are getting?
Which version of Hop are you using? The issue might have been caused by some changes in 2.11 and should be resolved in 2.12.

The issue would only be there when using the preview button, when running the pipeline the query should work.

[GRawhideMart]

I am using 2.9.0 version.
The full stack is 6 different SQLServer 2008 databases, which should be migrated to a single SQLServer 2022 one. The only error I'm getting is that one, but as an additional context, my colleagues running the same Docker container on an Ubuntu machine don't have the same problem.

[hansva]

then it might be a network thing between docker/wsl/windows I fear we won't be able to help on that one. You could try with the full client instead of the hop-web container.