Inter-app communication

[phroggyy]

This relates to #42181 and is an expansion on it as a more complete idea.

Background

Microservices are becoming more and more prevalent, and even without them, it is somewhat common to have more than one Laravel app to build out your complete app. I currently work with microservices, but have likewise previously worked with a company providing multiple products that sometimes had to interact.

Asynchronous communication via a message broker is generally more reliable than direct calls. I've previously solved this by building a library that abuses the queue system (https://github.com/decahedronio/laravel-app-events) and later a compatible library in Go (https://github.com/jobilla/go-app-events).

Use case

Asynchronous communication between multiple applications, helps improve system reliability and response time when expensive operations are required. Works effectively like a multi-application queue-system.

Why this should be in laravel/framework

There's an argument to be made for just creating a package of this and I think there are good reasons for it. However, the widespread adoption of Laravel would make it easier for developers to build compatible packages for other languages, and the Laravel umbrella guarantees a level of quality and reliability that may be essential to these systems. Further, it may make sense to extend one of the existing systems (queue or event), in which case adding it to core makes sense.

Requirements

• payload serialisation that can be customised to support generic formats like JSON or protocol buffers (but should likely default to serialised objects like the queue system for ease of use), to ease integration for teams with a microservice architecture using multiple programming languages and/or frameworks. The queue system does not currently fit well for this use case as it is the job that is queued, meaning both the handler and its state. A pub/sub or message broker system should work more like the event system, with events and payloads published, and listeners mapped to the events for consumption.
• distinguishing subscribing and publishing; a system can publish to a particular topic without necessarily subscribing to it. Likewise, a system can subscribe to messages on a topic that they do not publish to. The current queue system makes this challenging as there is only a concept of "the queue" and not distinguishing the consumption/publishing

Proposed solution

Extend the current event system to support multiple topics (to match existing terminology)

The current event system would map to the new sync topic, akin to Laravel's system. Just like the queue system, we may then introduce an artisan command, either mapping to the topic/subscriber terminology, or event/listener terminology: artisan topic:subscribe {topic} {--subscriber=} or events:listen with the same signature. --subscriber would allow overriding the name of the subscriber (used by message brokers to determine delivery). The subscriber would default to either config('app.name') or perhaps it could all be determined by an option in a new config file, config/events.php, which would look something like

<?php

return [
    'default' => env('EVENT_TOPIC', 'sync'),

    'topics' => [
        'sync' => [
            'driver' => 'sync',
        ],

        'kafka' => [
            'driver' => 'kafka',
            'topic' => env('KAFKA_TOPIC', 'laravel'),
            'subscription' => env('APP_NAME', 'laravel'), // this could of course default to a specific option too, e.g `KAFKA_SUBSCRIPTION`
        ],

        // This would be a user-added topic where they could listen to events from some `auth` system, for example
        'auth' => [
            'driver' => 'kafka',
            'topic' => env('KAFKA_TOPIC', 'laravel'),
            'subscription' => env('APP_NAME', 'laravel'),
            // you could override the deserializer and serializer (or both as an option `serialization`?)
            // to support other data formats than PHP serialization, e.g JSON or protocol buffers.
            'deserializer' => fn (KafkaMessage $msg) => ProtobufFactory::fromMessage($msg),
        ],
    ],
];

One thing that is not 100% clear to me is how overriding serialisation and deserialisation should work, and what extensions would be required to Event payloads to support it. Perhaps they can be kept as plain as now and it's up to each serialiser to enforce interface fulfilment (e.g by shipping/relying on a SerializesJson interface, which could be part of Laravel or 3rd party).

[Sophist-UK]

@phroggyy Leo

For readers like me, I fear that you are assuming more knowledge than I have as to what you are looking to achieve and what the current issues are.

1. Assuming that you are only wanting to do an MQ within a single Laravel instance, what are the issues with the Laravel Queue functionality?

2. It feels like you are actually looking for a multi-technology, multi-server MQ solution (e.g. between Laravel and Go running separately). This would (obviously) require a generic API implemented in each technology stack. The Queue itself could be within one of the stacks or it could be provided by a 3rd party technology (like one of the ones listed here). I have no idea whether it would be a good idea to create a multi-technology MQ server in Laravel (my guess is that a specialised stand-alone solution would be better performing and more reliable - and why reinvent the wheel and create yet another, possibly suboptimal, solution).

3. Whatever multi-technology solution you use to communicate and queue, you would also need to ensure that the data can be serialised and deserialised successfully between disparate systems. As you suggest, since the sending and receiving code using the MQ will know what the payload is, this probably means JSON encoded in e.g. UTF-8.

[phroggyy]

Thanks for the thoughts! Is there anything in particular I can add to the description to make it more clear?

1. The purpose is to do it across multiple Laravel instances. As is mentioned in the use case "Asynchronous communication between multiple applications".
2. Not looking for the MQ server itself – I'm saying the Laravel system would support an adapter system (just like for queues, logging, etc).
3. Yes, serialisation is, as mentioned, a problem. At its core, a Laravel-backed system for this would just use PHP's object serialisation as it does with queues. The support for customising those is to satisfy broadcasting events to applications in other languages/frameworks.

As you wrote in #42181 it's effectively server-side broadcasting but with the addition of a consumer-side that I'm after. So perhaps there is a way to build on the broadcasting system. I'm not sure how tied the current broadcasting system is to server-client though? One thing that is immediately clear from the broadcasting system is that it's only meant for emitting information, not for handling inbound events, and I'm after both.

[Sophist-UK]

1. You can have multiple Laravel Apps in a single Laravel instance.
2. This sounds like a good idea in principle - once it is clear what the business requirement is, allowing for several different technological solutions is always a good thing.

You can either have 2 queues - one for each direction. Or if Laravel and other technology solutions implement both broadcast and receive functionality, then you can transfer information in both directions. But the fundamental difference between these two is that Queues are guaranteed delivery - if the receiver is down, the request is not lost. Broadcast isn't - if the receiver is down, the message will be gone.