Proposed Subscription Design Change

Background

In the fall of 2018, we have been modifying HAPI-FHIR Server Subscription functionality to add support for running subscription matching outside of the REST server in a separate process. This has happened in stages:

In-memory matcher added a new class SubscriptionMatcherInMemory that matches incoming resources against subscription criteria by examining the content within the resource directly as opposed to querying the database.

Standalone subscription added a new maven module called hapi-fhir-jpaserver-subscription and moved classes there from hapi-fhir-jpaserver-base so that the SubscriptionMatcherInMemory had everything it needed to match resources against subscription criteria without access to a database. Shared classes needed by both hapi-fhir-jpaserver-subscription and hapi-fhir-jpaserver-base were moved into two new maven modules called hapi-fhir-jpaserver-model and hapi-fhir-jpaserver-searchparam.

One final change remains in order to be able to process subscription outside of the REST server: maintaining an in-memory cache of active subscriptions.

Requirements

Four things need to happen when a new subscription arrives on the REST server:

1. Validate the criteria on the subscription and reject it if the criteria are invalid.
2. Activate the subscription (that is change the status of the subscription from REQUESTED to ACTIVE)
3. Add the subscription to an in-memory cache of active subscriptions that future resources are matched against.
4. Create a new Spring MessageChannel and MessageHandler. Matching resources will be sent to that channel and handled by the MessageHandler. The MessageHandler will perform the action specified in the subscription, e.g. e.g. send an e-mail, make a REST call, etc.

How it works today

Currently, all of four of these functions are managed within HAPI-FHIR Interceptors. Most of the functionality resides within the abstract class BaseSubscriptionInterceptor. There are subclasses of this for each type of subscription: SubscriptionRestHookInterceptor, SubscriptionEmailInterceptor etc... When the HAPI-FHIR JPA Server starts up, subscription processing functionality is added to it by enabling one or more of these interceptors.

Interceptor startup

When the interceptor starts up, it creates the following objects that it stores internally:

1. A Processing Queue for processing incoming resources
2. SubscriptionActivatingSubscriber responsible for extra processing on incoming subscriptions
3. Three subscription caches: active subscriptions, subscription channels, and delivery handlers
4. A SubscriptionCheckingSubscriber responsible for matching incoming resources against cached subscriptions

Then at startup, the Interceptor reads the database and for each subscription of this interceptor type, it performs the "Subscription arrives" operation as described below.

Effectively there are multiple subscription caches--one for each type of interceptor. Further, only subscriptions for which there is a registered interceptor, will be activated. (e.g. if there is an EmailInterceptor and a RestHook subscription arrives, it won't be activated.) This subscription-type based activation, could be useful information to the end-user to know which subscriptions are currently matching and sending and which are inactive.

Subscription arrives

This is the extra processing that the SubscriptionActivatingSubscriber does on incoming resources:

1. Validate the criteria, and reject the CREATE/UPDATE if it's invalid.
2. Ignore subscriptions not of the type of this interceptor (e.g. email subscriptions ignored by rest interceptor)
3. If the subscription is REQUESTED, change it to ACTIVE and send it back the interceptor to be processed a second time.
4. If the subscription is ACTIVE, create a Delivery Queue and a DeliveryHandler (the interceptor knows what kind of handler to create) and populate the three subscription hashes in the interceptor.

Resource arrives

When a resource arrives, it is sent to each type of interceptor. Each interceptor does the following:

1. Submit the resource to the Processing Queue
2. Then in a separate thread, the SubscriptionCheckingSubscriber for this interceptor pulls the resource from the Processing Queue and does the following.
3. Loop through all the subscriptions in that Interceptor's cache and check if the resource matches it.
4. If the resource matches, it sends it to the Delivery Queue belonging to that subscription.
5. The DeliveryHandler then pulls it from the Delivery Queue and sends it out as e-mail, a rest call, etc.