Merge pull request #4738 from telefonicaid/feature/4555_jexlexpression_in_subject_condition

JEXL expression in subject.condition #4555

Author: fgalan

CHANGES_NEXT_RELEASE

@@ -1 +1,2 @@
-- Fix: kafkaCustom.headers support including the ability to override the default Fiware-Service and Fiware-ServicePath headers (#4726)
+- Fix: kafkaCustom.headers support including the ability to override the default Fiware-Service and Fiware-ServicePath headers (#4726)
+- Add: `jexlExpression` field to subscription conditions (`subject.condition`) (#4555)

doc/manuals/admin/database_model.md

@@ -302,6 +302,8 @@ Fields:
 -   **expression**: an expression used to evaluate if notifications has
     to be sent or not when updates come. It may be composed of the following
     fields: q, mq, georel, geometry and/or coords (optional)
+-   **jexlExpression**: an expression used to determine whether a notification must be sent.
+    Notifications are triggered only when the expression evaluates to `true` if this field is used (it is optional).
 -   **count**: the number of notifications sent associated to
     the subscription.   
 -   **format**: the format to use to send notification, possible values are **normalized**, **keyValues**, **simplifiedNormalized**, **simplifiedKeyValues** and **values**.
@@ -376,6 +378,7 @@ Example document:
                 "coords" : "",
                 "georel" : ""
         },
+        "jexlExpression": "temperature > 40"
         "format" : "JSON",
         "status" : "active",
         "statusLastChange" : 1637226173.6940024

doc/manuals/admin/statistics.md

@@ -165,8 +165,8 @@ The particular counters are as follows:
   `last` includes the accumulation for all of them. In the case of mongoReadWait, only the time used
   to get the results cursor is taken into account, but not the time to process cursors results (which
   is time that belongs to mongoBackend counters).
-* `exprJexlCtxBld`: time passed building context for custom notification expression evaluation (see [macro substitution](../orion-api.md#macro-substitution) and [JEXL support](../orion-api.md#jexl-support))
-* `exprJexlEval`: time passed evaluating custom notification expressions (see [macro substitution](../orion-api.md#macro-substitution) and [JEXL support](../orion-api.md#jexl-support))
+* `exprJexlCtxBld`: time passed building context for custom notification expression evaluation (see [macro substitution](../orion-api.md#macro-substitution) and [JEXL support in custom notifications](../orion-api.md#jexl-support-in-custom-notifications)). This also includes the time passed building context for `[`jexlExpressions`](../orion-api.md#subscriptionsubjectcondition).
+* `exprJexlEval`: time passed evaluating custom notification expressions (see [macro substitution](../orion-api.md#macro-substitution) and [JEXL support in custom notifications](../orion-api.md#jexl-support-in-custom-notifications)). This also includes the time passed evaluating expressions for `[`jexlExpressions`](../orion-api.md#subscriptionsubjectcondition).
 
 *NOTE*: if Orion binary is build without using cjexl and only basic replacement is available, then `exprBasicCtxtBld` and `exprBasicEval`
 fields appear instead of `exprJexlCtxBld` and `exprJexlEval`.

doc/manuals/orion-api.md

@@ -73,7 +73,7 @@
       - [NGSI payload patching](#ngsi-payload-patching)
       - [Omitting payload](#omitting-payload)
       - [Additional considerations](#additional-considerations)
-    - [JEXL Support](#jexl-support)
+    - [JEXL support in custom notifications](#jexl-support-in-custom-notifications)
       - [JEXL usage example](#jexl-usage-example)
       - [Metadata support](#metadata-support)
       - [Evaluation priority](#evaluation-priority)
@@ -452,7 +452,7 @@ There are some exception cases in which the above restrictions do not apply. In
 * URL parameter `q` allows the special characters needed by the [Simple Query Language](#simple-query-language)
 * URL parameter `mq` allows the special characters needed by the [Simple Query Language](#simple-query-language)
 * URL parameter `georel` and `coords` allow `;`
-* Within `ngsi` (i.e. `id`, `type` and attribute values) in [NGSI Payload patching](#ngsi-payload-patching) (to support characters used in the [JEXL expression syntax](#jexl-support))
+* Within `ngsi` (i.e. `id`, `type` and attribute values) in [NGSI Payload patching](#ngsi-payload-patching) (to support characters used in the [JEXL support in custom notifications](#jexl-support-in-custom-notifications))
 * Whichever attribute value which uses `TextUnrestricted` as attribute type (see [Special Attribute Types](#special-attribute-types) section)
 
 ## Identifiers syntax restrictions
@@ -478,7 +478,7 @@ In addition, the [General syntax restrictions](#general-syntax-restrictions) als
 
 In case a client attempts to use a field that is invalid from a syntax point of view, the client gets a "Bad Request" error response, explaining the cause.
 
-Note that although `:` and `-` are allowed in identifiers, they are strongly discouraged, as they collide with the [JEXL syntax](#jexl-support). In particular, `-` is used for subtraction operation (e.g. `${A-B}`) and `:` is used in the ternary operator (eg. `A?'A is true':'A is false`). Thus, an attribute name `lower-temperature` in an expression `${lower-temperature}` would be interpreted as the value of `lower` attribute minus `temperature` attribute (and not as the value of an attribute named `lower-temperature`).
+Note that although `:` and `-` are allowed in identifiers, they are strongly discouraged, as they collide with the [JEXL syntax](#jexl-support-in-custom-notifications). In particular, `-` is used for subtraction operation (e.g. `${A-B}`) and `:` is used in the ternary operator (eg. `A?'A is true':'A is false`). Thus, an attribute name `lower-temperature` in an expression `${lower-temperature}` would be interpreted as the value of `lower` attribute minus `temperature` attribute (and not as the value of an attribute named `lower-temperature`).
 
 ## Error Responses
 
@@ -1930,18 +1930,25 @@ be automatically deleted in an unwanted way**.
 Based on the [`condition` subscription field](#subscriptionsubjectcondition), upon
 entity update, the notification triggering rules are as follow:
 
-* If `attrs` and `expression` are used, a notification is sent whenever one of the attributes in
-  the `attrs` list changes (or is deleted) and at the same time `expression` matches.
-* If `attrs` is used and `expression` is not used, a notification is sent whenever any of the
+* If `attrs` and expressions are used, a notification is sent whenever one of the attributes in
+  the `attrs` list changes (or is deleted) and at the same time expressions match.
+* If `attrs` is used and expressions are not used, a notification is sent whenever any of the
   attributes in the `attrs` list changes (or is deleted).
-* If `attrs` is not used and `expression` is used, a notification is sent whenever any of the
-  attributes of the entity changes (or is deleted) and at the same time `expression` matches.
-* If neither `attrs` nor `expression` are used, a notification is sent whenever any of the
+* If `attrs` is not used and expressions are used, a notification is sent whenever any of the
+  attributes of the entity change (or is deleted) and at the same time expressions match.
+* If neither `attrs` nor expressions are used, a notification is sent whenever any of the
   attributes of the entity changes (or is deleted).
 
 Note that changing the metadata of a given attribute is considered a change even though the attribute
 value itself hasn't changed.
 
+By *expressions* we mean at least one of the following
+
+* `expression`
+* `jexlExpression`
+
+If both are used (i.e. `expression` and a `jexlExpresion` at the same time) both must match to consider that expressions match in the above description of triggering rules.
+
 ## Notification Messages
 
 Notifications include two fields:
@@ -2069,7 +2076,7 @@ In case of `mqttCustom`:
 * `topic`
 
 Macro substitution for templates is based on the syntax `${<JEXL expression>}`. The support to JEXL
-is explained in [JEXL Support](#jexl-support) section. The following identifiers are included in
+is explained in [JEXL support in custom notifications](#jexl-support-in-custom-notifications) section. The following identifiers are included in
 the context evaluated by the JEXL expression:
 
 * `id`: for the `id` of the entity
@@ -2341,7 +2348,7 @@ Some considerations to take into account when using custom notifications:
   (i.e. `ngsi` field) then `Ngsiv2-AttrsFormat: normalized` is used, as in a regular
   notification (given that the notification format is actually the same).
 
-## JEXL Support
+## JEXL support in custom notifications
 
 Orion Context Broker supports [JEXL expressions](https://github.com/TomFrost/Jexl) in custom notification [macro replacement](#macro-substitution). Thus, subscriptions like this can be defined:
 
@@ -5031,6 +5038,7 @@ A `condition` contains the following subfields:
 |--------------|----------|-------|-------------------------------------------------------------------------------------------------------------------------------|
 | `attrs`      | ✓        | array | Array of attribute names that will trigger the notification. Empty list is not allowed.                                       |
 | `expression` | ✓        | object| An expression composed of `q`, `mq`, `georel`, `geometry` and `coords` (see [List Entities](#list-entities-get-v2entities) operation above about this field). `expression` and sub elements (i.e. `q`) must have content, i.e. `{}` or `""` is not allowed. `georel`, `geometry` and `coords` have to be used together (i.e. "all or nothing"). Check the example using geoquery as expression [below](#create-subscription-post-v2subscriptions).|
+| `jexlExpression` | ✓ | string | JEXL expression evaluated to determine whether a notification must be sent. If this field is used (note it is optional), the notification is triggered only when the expression evaluates to `true`. It works in the same way as the [JEXL expressions in custom notifications](#jexl-support-in-custom-notifications).|
 | `alterationTypes` | ✓   | array | Specify under which alterations (entity creation, entity modification, etc.) the subscription is triggered (see section [Subscriptions based in alteration type](#subscriptions-based-in-alteration-type)) |
 | `notifyOnMetadataChange` | ✓   | boolean | If `true` then metadata is considered part of the value of the attribute in the context of notification, so if the value doesn't change but the metadata changes, then a notification is triggered. If `false` then the metadata is not considered part of the value of the attribute in the context of notification, so if the value doesn't change but the metadata changes, then a notification is not triggered. Default value is `true`. |
 

src/lib/apiTypesV2/Subscription.cpp

@@ -322,6 +322,11 @@ std::string Condition::toJson()
 
   jh.addRaw("attrs", vectorToJson(this->attributes));
 
+  if (!this->jexlExpression.empty())
+  {
+    jh.addString("jexlExpression", this->jexlExpression);
+  }
+
   JsonObjectHelper jhe;
 
   if (!this->expression.q.empty())        jhe.addString("q",        this->expression.q);

src/lib/apiTypesV2/Subscription.h

@@ -126,6 +126,7 @@ struct Condition
 {
   std::vector<std::string>  attributes;
   Expression                expression;
+  std::string               jexlExpression;
   std::vector<SubAltType>   altTypes;
   bool                      notifyOnMetadataChange;
   std::string               toJson();

src/lib/cache/subCache.cpp

@@ -861,6 +861,7 @@ void subCacheItemInsert
   StringFilter*                           mdStringFilterP,
   const std::string&                      status,
   double                                  statusLastChange,
+  const std::string&                      jexlExpression,
   const std::string&                      q,
   const std::string&                      geometry,
   const std::string&                      coords,
@@ -899,6 +900,7 @@ void subCacheItemInsert
   cSubP->failsCounter          = 0;
   cSubP->status                = status;
   cSubP->statusLastChange      = statusLastChange;
+  cSubP->jexlExpression        = jexlExpression;
   cSubP->expression.q          = q;
   cSubP->expression.geometry   = geometry;
   cSubP->expression.coords     = coords;

src/lib/cache/subCache.h

@@ -111,6 +111,7 @@ struct CachedSubscription
   double                           statusLastChange;
   int64_t                          count;
   RenderFormat                     renderFormat;
+  std::string                      jexlExpression;
   Expression                       expression;
   bool                             blacklist;
   bool                             onlyChanged;
@@ -218,6 +219,7 @@ extern void subCacheItemInsert
   StringFilter*                      mdStringFilterP,
   const std::string&                 status,
   double                             statusLastChange,
+  const std::string&                 jexlExpression,
   const std::string&                 q,
   const std::string&                 geometry,
   const std::string&                 coords,

src/lib/jsonParseV2/parseSubscription.cpp

@@ -1511,6 +1511,17 @@ static std::string parseNotifyConditionVector
     }
   }
 
+  // jexlExpression
+  if (condition.HasMember("jexlExpression"))
+  {
+    const rapidjson::Value& jexlExpression = condition["jexlExpression"];
+    if (!jexlExpression.IsString())
+    {
+      return badInput(ciP, "jexlExpression is not a string");
+    }
+    subsP->subject.condition.jexlExpression = jexlExpression.GetString();
+  }
+
   // Expression
   if (condition.HasMember("expression"))
   {

src/lib/mongoBackend/MongoCommonSubscription.cpp

@@ -676,6 +676,18 @@ void setExpression(const Subscription& sub, orion::BSONObjBuilder* b)
 
 
 
+/* ****************************************************************************
+*
+* setJexlExpression -
+*/
+void setJexlExpression(const Subscription& sub, orion::BSONObjBuilder* b)
+{
+  b->append(CSUB_JEXL_EXPR, sub.subject.condition.jexlExpression);
+  LM_T(LmtMongo, ("Subscription jexlExpression: %s", sub.subject.condition.jexlExpression.c_str()));
+}
+
+
+
 /* ****************************************************************************
 *
 * setFormat -