[Task]: Custom Transform Plugin Not Recognized in Apache Hop 2.8.0

[Raja10D]

What needs to happen?

Hi Apache Hop Team,

We are developing a custom transform plugin (hop-spark-csv) for Apache Hop 2.8.0 but are facing issues getting it recognized by Hop. Despite placing the JAR file in the correct directory, the transform does not appear in the Hop UI or when running pipelines.

Setup Details
Hop Version: 2.8.0 (2024-02-28)
Plugin Location: $HOP_HOME/plugins/transforms/hop-spark-csv-1.0-SNAPSHOT.jar
Plugin JAR Contains:
META-INF/hop/plugin.xml (confirmed using jar tf)
Custom transform class: com.example.hop.CustomCSVSparkTransform

Expected Behavior

1. The transform should appear in Hop UI under the available transforms list.

2. Hop should be able to detect and register the plugin from $HOP_HOME/plugins/transforms/.

Current Issue
The plugin does not show up in the Hop UI.
I want my plugin to show up in the Hop UI

What should I do so that Hop can recognize my plugin???

Issue Priority

Priority: 2

Issue Component

Component: Other

[hansva]

Did you start from the skeleton we provide?
https://github.com/project-hop/hop-plugin-sample

Usually, the issue is, missing the jandex-maven-plugin or mis-configuration in the annotations.
What features is the text file input missing to start creating your own CSV transform?

[hansva]

Or in a spark context we even have the Beam file input which is a CSV reader for the distributed systems beam supports

it's a wrapper around the Beam CsvIO

[Raja10D]

Or in a spark context we even have the Beam file input which is a CSV reader for the distributed systems beam supports

it's a wrapper around the Beam CsvIO

We are developing a custom CSV input component for Spark to eliminate the additional step of exporting a Beam File Definition (metadata.json) when using the Beam File Input transform.

Current Issue with Beam File Input on Spark
When using the Beam File Input transform in Hop, it generates an .hpl file that references a Beam File Definition. To execute this pipeline on Spark, we must first export the metadata.json file using Hop GUI. If the metadata.json file does not contain the Beam File Definition for our CSV input, the Spark job will fail.

Additionally, every time we create a new pipeline with Beam File Input, we must manually export the metadata.json file and ensure it is available when running spark-submit. This extra step adds complexity when automating pipelines in a distributed Spark environment.

Difference Between .hpl Files
The .hpl file generated by CSV File Input is different from the .hpl file generated by Beam File Input.
The Beam File Input .hpl file requires a reference to the Beam File Definition, which forces us to export the metadata.json manually.

On the other hand, the Excel Input component works seamlessly on Spark without requiring additional metadata files.
Our Goal: CSV Input Should Work Like Excel Input on Spark
To simplify the process, we are building a custom CSV input component for Spark that behaves like the Excel Input component—allowing users to directly read CSV files on Spark without exporting a separate metadata.json file.

Would love to hear your thoughts! Is there an internal way to achieve this, or does our approach make sense for a Hop-based Spark pipeline?