Send progress/status information from worker to main process

[dimitriylol]

As I understand, there isn't a way for listening/sending messages between worker and executor. I refer to message listeners and sender methods worker.send in worker.js and WorkerHandler.js files. May you give a suggestion how I can establish message communication? Maybe you've already had some thought about it.

P.S. Great package, thank you for the work.

[josdejong]

Thanks. I indeed haven't thought about opening up two way communication between the main process and the workers. workerpool creates a "remote procedure call" on top of the bare-bone message passing mechanism that is the basis of the underlying workers.

We could discuss how allowing 2way message passing could look like, though I'm not sure if this fits in the scope of workerpool (sometimes it's best to have a library do just one thing). Do you have any ideas?

[dimitriylol]

I give some thoughts about my specific task and realized that I want to implement server sent events with message passing mechanism of the underlying workers instead of HTTP. So master process is client and worker is server which may produce messages to a master.

It can be done in two way:

1. Quick and easy. Create new API function (let's call it execWithMsg which calls internally exec) and pass a callback for listening messages from worker. This callback is passed down to a callback in WorkerHandler.js file.

this.worker.on('message', function (response) {
// call it here via task.msgCallback
}

And pass carried function for sending message from worker code:

method.apply(method, [ ...request.params, (msg) => worker.send({ id: request.id, msg }) ]);

I've done rough implementation, so it's easy. This approach violates logic of "one function do just one thing" which is pretty bad. But if it's okay to you, I can create pull request in couple days.

2. Create an API function which implements server sent events pattern in addition to RPC. There can be a function establishConnection:

function establishConnection (method, callbackForWorkerEvents) {
// in the same manner as `exec` creates a worker
// but returns undefined. All logic for processing messages from worker must be implemented in `callbackForWorkerEvents`
}

By the way, if someone would like to have 2way connection, then establishConnection must return send function instead of undefined. The function would send messages from master to client . Channel for listening messages from master can be done by writing following code in dedicated worker script:

process.on('message', (msg) => {})

It should be abstracted in some way, but I didn't come up with any thoughts except passing stringified message listener to a worker script. IMO, it's not acceptable.

Please respond which way you like more (or none of it 😃 ).

[josdejong]

So if I understand you correctly you're looking to send and event from the worker to the main application? We could think about extending the workerpool with an on('event', callback) API.

What may be confusing is that when a pool creates multiple workers, all these workers may start sending events. I guess that's not what you want. I'm not sure how this could work out nicely. Do you have a real-world use case of what you try to achieve?

[dimitriylol]

Sorry, my bad, for writing in abstract manner.

Do you have a real-world use case of what you try to achieve?

I have long running task and I want to get some status feedback from them. I've made a fork of your repo and pushed 2 commit to master:

• f11f6cb - for adding execWithMsg function
• 40c828a - for an example of usage

It's not ready to go solution, feel free to criticize it.

[josdejong]

Ah, now I get it, that's really a great idea! Makes a lot of sense to be able to post status or progress information back to the main application.

Would be great to integrate this feature into workerpool. I would love to think a bit more about the API and the most clear naming for this feature. In first instance the name execWithMsg did confuse me. On the other hand, calling it something like onProgress may be clear but maybe too limiting in what you can do with such a feature. Maybe we can call it onUpdate and pass it as an optional extra argument to exec or something. Just thinking aloud here, but I really like your idea :)

[x-077]

Hello @josdejong , I can across your github, create tool :) .

any progress by any chance. I have similar situation and it could be really useful to integrate with redis (pub/sub) as we could bind the worker directly on the appropriate chan based on the name event.

Last comment, @dimitriylol it could be good to use async generator for each task, it could prevent to wait that everything is processed.

As a example to better understand :

..... some code


//this is the async generator that will take care of each promise. this could be a custom function given as argument too

async function* gen(myPromise) {
    const {data} = ...
    // consume the promise here ... 
    return anOtherPromise
}

// instead of using Promise.all([...]) you could do that 

for await(const myPromise in [arrayOfPromises] ) {
   const data = myPromise.then(...) 
   //do something else on the data that were consumed by the async generator 
   // return a Promise. 
}

The good use case that I see is when you dont want to wait to get all the "promises" to be resolved. Let say I have an array of promises to fetch 100 API endpoint, You would not have to wait to get all of them to start whatever I have to do on that data.

sorry for the long post,

Thanks by advance.

[josdejong]

Thanks for your inputs @matth-c3 👍

[x-077]

Hello @josdejong ,

did you have a chance to think about it ? It's a really cool feature ;)

[josdejong]

@matth-c3 what do you mean exactly? About the progress information in general: that sounds like a good idea to me. I do not have any time soon to implement it myself though, so help would be very welcome. About your redis use case: I don't have enough info to understand your case exactly but if the progress plans fit your use case it's only more reason to implement it.

[x-077]

Hello . @josdejong ,

Yes I'm talking about the possibility to name the workers and talk to specifics one.

Regarding Redis, I will try to come up with an example.

thanks.