Apache Hop / Azure Storage

[in-da]

Hello All,
I am planning to use Apache Hop to load data to an ADLS and I am wondering if:
1- It is possible. If yes how simple would it be? If there are any materials or examples to share beside what is shared in the official doc (very light)?
2- Someone has done it before can share some insights and/or recommendations?

Thank you all for your help.
BR.

[hansva]

Yes, it is possible, and it acts like any other file system in Hop.
I would recommend using the azure authentication to create your own namespaces, this allows you to add multiple accounts and transfer files between them https://hop.apache.org//manual/latest/metadata-types/azure-authentication.html

Then all transforms that support files and file operations will work using ADLS

[in-da]

Thank you for your help,
I am using the following setup:

At the ADLS Gen 2 level:

• storage account name : landingzone
• container name: ldz1

At Hop level:

1. under plugins:

• set the account name to landingzone
• set the key
• set the alternative Local storage to azure

2. under Azure Authentication:

• set the name to datalake
• set the storage account name to landingzone
• set the key
• left storage Endpoint empty

3. under Get file names action :

• set datalake:///ldz1 to file or directory
• set wildcard to .*
• I am including also subfolders

After hitting the show file names, I got the following error:

• FileInputList - ERROR: org.apache.commons.vfs2.FileSystemException: Could not determine the type of file "datalake:///ldz1".
• FileInputList - at org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:171)
• FileInputList - at org.apache.commons.vfs2.provider.AbstractFileObject.getType(AbstractFileObject.java:1370)
• FileInputList - at org.apache.commons.vfs2.provider.AbstractFileObject.exists(AbstractFileObject.java:965)
• FileInputList - at org.apache.hop.core.fileinput.FileInputList.createFileList(FileInputList.java:209)
• FileInputList - at org.apache.hop.core.fileinput.FileInputList.createFilePathList(FileInputList.java:137)
• FileInputList - at org.apache.hop.pipeline.transforms.getfilenames.GetFileNamesMeta.getFilePaths(GetFileNamesMeta.java:452)
• FileInputList - at org.apache.hop.pipeline.transforms.getfilenames.GetFileNamesDialog$10.widgetSelected(GetFileNamesDialog.java:843)
• FileInputList - at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:265)
• FileInputList - at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:91)
• FileInputList - at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4326)
• FileInputList - at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1174)
• FileInputList - at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4124)
• FileInputList - at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3712)
• FileInputList - at org.apache.hop.ui.core.dialog.BaseDialog.defaultShellHandling(BaseDialog.java:528)
• FileInputList - at org.apache.hop.pipeline.transforms.getfilenames.GetFileNamesDialog.open(GetFileNamesDialog.java:893)
• FileInputList - at org.apache.hop.ui.hopgui.file.pipeline.delegates.HopGuiPipelineTransformDelegate.editTransform(HopGuiPipelineTransformDelegate.java:195)
• FileInputList - at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.editTransform(HopGuiPipelineGraph.java:3368)
• FileInputList - at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.mouseUp(HopGuiPipelineGraph.java:940)
• FileInputList - at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:237)
• FileInputList - at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:91)
• FileInputList - at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4326)
• FileInputList - at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1174)
• FileInputList - at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4124)
• FileInputList - at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3712)
• FileInputList - at org.apache.hop.ui.hopgui.HopGui.open(HopGui.java:476)
• FileInputList - at org.apache.hop.ui.hopgui.HopGui.main(HopGui.java:354)
• FileInputList - Caused by: org.apache.hop.core.exception.HopException:
• FileInputList - Container does not exist: /ldz1
• FileInputList -
• FileInputList - at org.apache.hop.vfs.azure.AzureFileObject.doAttach(AzureFileObject.java:141)
• FileInputList - at org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:162)
• FileInputList - ... 25 more

Am I missing Sth?
BR.

[hansva]

What happens if you try to browse to a file and enter "datalake:///ldz1/" in the top? It should show the content of the container.

[in-da]

I got the following error:

Error browsing to location: 'datalake:///ldz1/'
FileSystemException: Could not determine the type of file "datalake:///ldz1".
Root cause: HopException:
Container does not exist: /ldz1

org.apache.commons.vfs2.FileSystemException: Could not determine the type of file "datalake:///ldz1".
at org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:171)
at org.apache.commons.vfs2.provider.AbstractFileObject.getType(AbstractFileObject.java:1370)
at org.apache.commons.vfs2.provider.AbstractFileObject.isFolder(AbstractFileObject.java:1553)
at org.apache.hop.ui.core.vfs.HopVfsFileDialog.refreshBrowser(HopVfsFileDialog.java:880)
at org.apache.hop.ui.core.vfs.HopVfsFileDialog.refreshAll(HopVfsFileDialog.java:1278)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.apache.hop.ui.core.gui.BaseGuiWidgets.lambda$getListener$1(BaseGuiWidgets.java:213)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:91)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4326)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1174)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4124)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3712)
at org.apache.hop.ui.core.vfs.HopVfsFileDialog.open(HopVfsFileDialog.java:584)
at org.apache.hop.ui.core.dialog.BaseDialog.presentFileDialog(BaseDialog.java:278)
at org.apache.hop.ui.core.dialog.BaseDialog.presentFileDialog(BaseDialog.java:172)
at org.apache.hop.pipeline.transforms.getfilenames.GetFileNamesDialog.lambda$open$4(GetFileNamesDialog.java:870)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:91)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4326)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1174)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4124)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3712)
at org.apache.hop.ui.core.dialog.BaseDialog.defaultShellHandling(BaseDialog.java:528)
at org.apache.hop.pipeline.transforms.getfilenames.GetFileNamesDialog.open(GetFileNamesDialog.java:893)
at org.apache.hop.ui.hopgui.file.pipeline.delegates.HopGuiPipelineTransformDelegate.editTransform(HopGuiPipelineTransformDelegate.java:195)
at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.editTransform(HopGuiPipelineGraph.java:3368)
at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.mouseUp(HopGuiPipelineGraph.java:940)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:237)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:91)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4326)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1174)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4124)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3712)
at org.apache.hop.ui.hopgui.HopGui.open(HopGui.java:476)
at org.apache.hop.ui.hopgui.HopGui.main(HopGui.java:354)
Caused by: org.apache.hop.core.exception.HopException:
Container does not exist: /ldz1

at org.apache.hop.vfs.azure.AzureFileObject.doAttach(AzureFileObject.java:141)
at org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:162)
... 34 more

[bamaer]

I may be asking the obvious here, but anyway: do your containers ldz and ldz1 exist and can you access them through Microsoft Azure Storage Explorer or something similar?

[in-da]

Yes, and I was able to fetch its content using another ETL tool.
Below you will find the structure of the container I am using.
Is there any additional configuration to consider ?

[hansva]

Could you retry with a snapshot version? Some improvements have been made, such as listing the containers.

[hansva]

https://repository.apache.org/content/groups/snapshots/org/apache/hop/hop-client/2.17.0-SNAPSHOT/

[in-da]

Hello @hansva , I confirm that the connection is working and I am able to fetch the structure. (The message was misleading / it was a network-related issue).
At the same time I am not able to copy data using the process file transform with copy operation and I got the following error:

Process files.0 - ERROR: Because of an error, this transform can't continue: Cannot invoke "com.azure.storage.file.datalake.DataLakeFileClient.exists()" because "this.dataLakeFileClient" is null
Process files.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)

Please note that move operation is working.
Is there any specific setup that I am missing here?
Thank you for you help.

[hansva]

That will probably be a bug.

[sramazzina]

@in-da can you file a bug related to your copy issue so that we can look into that and see what's happening? Thank you

[in-da]

done #6494
Thank you all.