How to handle Workflows/Pipelines via REST API on Windows?

[adityaluffy]

Goal I am setting up Apache Hop Server on a Windows environment. My goal is to trigger Workflows and Pipelines remotely using the REST API.

[marvinhorst]

I recently asked a similar question #6226.

A recap of what set me on the correct path.

• this link for REST call references ... https://hop.apache.org//manual/latest/hop-server/async-web-service.html
• hopserver.sh needs to be started with the -e (environment) and -j (project) options for the webservice to "see" your workflows/pipelines.
• hopserver.sh also needs to see your hop-config.json file, (in addition to the hop-server-config.xml if you're using one) where your projects and environments are defined. I defined the location via the ENVRIONMENT variable HOP_CONFIG_FOLDER, there didn't appear to be a command line option for hopserver.sh

[adityaluffy]

Hi Marvin,

I’m facing the same issue and trying to follow the async REST setup described in the docs.

What I’ve done so far:

• Started Hop Server with -e local and -j default
• Set HOP_CONFIG_FOLDER and verified it’s picked up at runtime
• Defined project and environment in hop-config.json
• Project is detected, but server still fails with
• Unable to find lifecycle environment 'local'

It’s unclear to me where exactly the lifecycle environment must be defined for Hop Server (hop-config.json vs project metadata), especially for async REST usage.

Is there any official doc or minimal working example that shows the correct setup for environments with Hop Server?
If you have any working reference or screenshot of the config, that would really help.

command> hop-server.bat C:\hop\config\hop-server-config.xml -e local -j default

[marvinhorst]

My pipelines and workflows were successfully running via hop-run and in the hop-gui. I would verify the pipeline works in the gui first. The argument for -e should be one of the lifecycleEnvironments defined in your hop-config.json and -p argument should be the project name defined in your hop-config.json. Basically the environment and project used to run the pipeline in the hop-gui

[marvinhorst]

For my pipelines and workflows that i wanted to execute via hop-server I created a new project (remote) and lifecyleEnvironment (remote-PROD) using the hop-gui which saves these setting in hop-config.json. Then in that project I went to Asyncronous Web Service in metadata and created a new service as shown in my image. My hop-server-config.xml is basic ...

<hop-server-config> <hop-server> <name>biapp-hop</name> <hostname>0.0.0.0</hostname> <port>8080</port> <username>biuser</username> <password>apassword</password> </hop-server> </hop-server-config>

I'm running hop-server as a service on Debian linux. Here's the config for that.

`
Description=Apache Hop Server
After=network.target

[Service]
User=root
Group=root

WorkingDirectory=/opt/apache/hop
PrivateTmp=false
ProtectSystem=off
ProtectHome=off

ExecStart=/opt/apache/hop/hop-server.sh -e remote-PROD -j remote /opt/hopdata/config/hop-server-config.xml

Restart=always
RestartSec=10

Environment="HOP_SERVER_PORT=8080" "HOP_PROJECTS_FOLDER=/opt/hopdata/projects" "HOP_AUDIT_FOLDER=/opt/hopdata/audit" "HOP_CONFIG_FOLDER=/opt/hopdata/config"

[Install]
WantedBy=multi-user.target
`
The HOP_CONFIG_FOLDER environment variable is crucial as that is used to find the hop-config.json file. HOP_SERVER_PORT is probably redundant. My paths may look different as I've moved my hop data from the installed paths for ease of updating hop. My hop-config.json points to /opt/hopdata ...

, { "name" : "HOP_HOME_FOLDER", "value" : "/opt/hopdata", "description" : "Directory where all hop metadata is stored. Separate from the hop binaries" }

[adityaluffy]

Hi Marvin,

Thanks a lot for the detailed guidance — the approach with the separate project and environment configuration worked perfectly 👍

What I ended up implementing

Based on your suggestions, I set up the following:

• Created a dedicated project (default) and environment (Development) for remote execution

• Configured an Async Web Service in metadata:

  • Service name mapped to the workflow

• Set up hop-server-config.xml with basic authentication (port 8181, username/password)

• Started Hop Server with the project context:

--> hop-server.bat -e Development -j default config\hop-server-config.xml

Results

Everything is working as expected:

• Triggering workflows via REST API:

  http://localhost:8181/hop/asyncRun/?service=test-service

• Async execution returns execution IDs for tracking

Question about service management patterns

My real project has a fairly complex structure:

• Multiple parent workflows
• Each parent workflow calls several child pipelines
• Some workflows call other workflows
• Nested dependencies like: workflow → workflow → pipeline → pipeline

Given this setup, what’s the recommended way to manage Async Web Services?

--> Create a separate Async Web Service for each parent workflow that should be exposed as an API endpoint
--> Do you add or register the workflow or pipeline through the URL?

Thanks again — really appreciate the help!

[marvinhorst]

Great! Glad it worked for you. It wasn't very clear to me either on how to set this up.

The screenshot is the only workflow I have setup and working so far. It contains 2 pipelines which didn't need to be registered. I would assume the same for child workflows. I believe that you only need to register workflows/pipelines that you plan to directly call via REST. Any child workflows or pipelines will not need to be registered (I'm hoping). I'll be adding more workflows soon. Not sure if if any of them have child workflows though.

[hansva]

I'm also not sure you can combine the xml file and the arguments in the hop-server command. If you are using the XML, I would fill in the metadata folder part to point to the folder.