merge 3.3.1 (#1)

* improved tests for proto fields explicit presence validation

* UI: add custom interval option to charts comparison

* test scope: dependency upgrade a refactor for all POMs in project

* test scope: comments edited on versions for the test scope and blackbox test scope

* dependency org.apache.httpcomponents httpclient moved from blackboxtests to the room POM. httpclient used in edge, but not explicitly added by dependency management. httpclient version with no change.

* rollback testcontainers version to 1.11.4 due to the newest testcontainers does not support "container_name" in docker compose

* UI: add copy user id button in user details

* Revert "rollback testcontainers version to 1.11.4 due to the newest testcontainers does not support "container_name" in docker compose"

This reverts commit 8bdd437.

* dependency org.apache.httpcomponents httpclient moved from blackboxtests to the room POM. httpclient used in edge, but not explicitly added by dependency management. httpclient version with no change.

* UI: name function typo fix

* testcontainers - copy source dir to tmp, remove container_name in tmp, run, cleanup tmp on stop.

* UI: change maps widget bundle template

* UI: Multiple attributes widget, patterns support in widget title

* changed slider thumbnail size

* Version set to 3.3.1-SNAPSHOT

* UI:add input widgets required field option

* Implemented rpc sending sequence

* Update versions to 3.3.1 in package.json

* Change json property order in model WidgetTypeDetails

* UI: Added check for DOM element existence before calling draw function

* added new RPC statuses

* improvement custom translation for alarm type field

* Edge functionality enabled by default

* added sequence for the all RPC

* refactoring

* fix bug no entity label for dachboard state

* used timeout from yml

* send next rpc after removing

* send next rpc after removing

* UI: Delete custom type leaflet-editable, add default @types/leaflet-editable

* UI: Added support for UTF-8 characters in MQTT credential Client ID

* Session Activity reporting is isolated and supports short-lived sessions

* Sequential RPC processing support

* Disabled tooltips on Charts widgets

* Add markdown/HTML widget

* Improve markdown widget css processing

* Add upgrade to 3.3.1

* Update docker pkg upgrade version

* Change enableTooltip to showTooltip and change to  true by default

* chenge befault on true

* UI: reduced minimum time for power mode settings

* Added logs for edge license check. Replaced Object class with JsonNode

* Fix: resources-library page can't be redirected

In SysAdmin HomePage, "resources-library" page can't be redirected

* dependency management: org.springframework:spring-core (version with no change, exclusions replaced with transitive dependency management

* dependency management: com.fasterxml (version unified with general jackson.version (no change), exclusions replaced with transitive dependency management), fasterxml-classmate version defined

* dependency management: io.netty (version unified with general netty.version (no change), exclusions replaced with transitive dependency management)

* grpc: grpc-netty replaced with grpc-netty-shaded to decouple grpc and netty versions in the project (CE)

* grpc: grpc-netty replaced with grpc-netty-shaded (edge-api)

* grpc: replaced imports for grpc-netty-shaded (io.grpc.netty. -> io.grpc.netty.shaded.io.grpc.netty.) (CE)

* dependency management: added commons-codec commons-logging in dep management (only greater available version in the classpath left)

* dependency management: removed unnecessary exclude commons-io from org.owasp.antisamy because commons-io already defined in dependencyManagement (transitive dependency are managed as well)

* dependency management: joda-time unified only newer available version in the classpath left (2.3, 2.4, 2.8.1 -> 2.8.1)

* dependency management: apache httpclient (4.5.2, 4.5.13 -> 4.5.13), httpcore (4.4.5, 4.4.13, 4.4.14 -> 4.4.14)

* dependency management: removed aws.sdk.version (already defined in the root pom.xml with the same version)

* dependency management: proto-google-common-protos (1.16.0, 1.17.0, 2.0.1 -> 2.0.1 /required by common used grpc 1.38.0/ ). Version management moved to the root

* msa-black-box-tests: httpclient dependency version inherited from the root pom.xml

* EdgeGrpcClient: added compression

* Added default timeout for asynchronous requests

* Update locale.constant-de_DE.json

* Update locale.constant-de_DE.json

* Update locale.constant-de_DE.json

* Refactoring: Introduced Edge License service. Remove web dependency from dao

* UI: Fixed toggle to editor full-screen mode in JSON forms

* Validation and Replacement of Queue names in the Device profile

* Improve queue name selector

* fix defaultConfig

* Improved SSL context init - trust manager can be build without custom cert file

* LwM2M improved tracking of sent requests

* Fixed has not been used property SECURITY_USER_LOGIN_CASE_SENSITIVE on password recover

* Change button close to reset add change default changeDetection to onPush

* UI: When clicking the button "Show on widget", the default widget bundle did not always work

* Update rule node UI

* Update UserServiceImpl.java

* UI: Refactoring attributes-table component, add changeDetection for load widget bundle

* Fix HTTP interceptor loading state handling

* Update attribute-table.component.ts

* Update widgets-bundle-select.component.ts

* UI: extend list of rxjs operators available in widget context

* remove reset button form filter panel and add clear filter button to table config

* remove isEqual form imports and remove changeDetection from component

* UI: Fixed load counter - requests were not counted when cancel them

* added netty-tcnative-boringssl-static.version 2.0.41.Final along with netty-all to support SSL

* tests: maven-surefire-plugin upgraded 3.0.0-M1 -> 3.0.0-M5

* dependency: netty latest 4.1.67, netty-tcnative-boringssl 2.0.40 (according to the optional netty-transport dependency)

* executors: names added, shutdownNow for some executors to prevent memory leaks during lifecycle (mostly affects the test runner JVM)

* executors: shutdownNow added for tb-rule-engine-consumer-repartition executor on @PreDestroy

* black-box-test: fixed import ThingsBoardThreadFactory

* tests: call destroy for lwm2m devices to shut down executors inside fwLwM2MDevice and swLwM2MDevice in the test scope

* tests: Coap client destroy on processAfterTest at AbstractCoapIntegrationTest

* executors: named fixed thread pools; tests - added executor shutdown on tearDown

* test: assert not null COAP response instead NullPointerException (to clarify the test failure cause)

* dependency-hotfix-after-merge-ce (httpclient doubled, test dependency outdated)

* Fix imports

* Fix device profile update handling by rule node - preserve old key values to correctly update entity keys snapshot

* Update nashorn sandbox version to 0.2.1

* Rule chain actor message processor: added log onTellNext

* RuleChainActorMessageProcessor: onTellNext null pointer fix

* Improve js executor kafka consumer to exit application in case of not retryable error.

* update mdi.svg icon and remove some icon from app.component

* UI: Added protection for invalid device profile alarm condition data

* Handle Device service transactional methods exceptions (fix exception handling for devices with same name)

* Revert "Handle Device service transactional methods exceptions (fix exception handling for devices with same name)"

This reverts commit b67f454

* Save and flush device during transaction to trigger constraint validation.

* Version set to 3.3.1-SNAPSHOT

* Version set to 3.3.1

Co-authored-by: ShvaykaD <[email protected]>
Co-authored-by: Chantsova Ekaterina <[email protected]>
Co-authored-by: Sergey Matvienko <[email protected]>
Co-authored-by: rusikv <[email protected]>
Co-authored-by: Andrii Shvaika <[email protected]>
Co-authored-by: ArtemDzhereleiko <[email protected]>
Co-authored-by: YevhenBondarenko <[email protected]>
Co-authored-by: Igor Kulikov <[email protected]>
Co-authored-by: Vladyslav_Prykhodko <[email protected]>
Co-authored-by: Volodymyr Babak <[email protected]>
Co-authored-by: Chris <[email protected]>
Co-authored-by: Igor Kulikov <[email protected]>

Author: 12 people

application/pom.xml

@@ -20,7 +20,7 @@
     <modelVersion>4.0.0</modelVersion>
     <parent>
         <groupId>org.thingsboard</groupId>
-        <version>3.3.0</version>
+        <version>3.3.1</version>
         <artifactId>thingsboard</artifactId>
     </parent>
     <artifactId>application</artifactId>
@@ -229,9 +229,17 @@
             <groupId>com.google.protobuf</groupId>
             <artifactId>protobuf-java</artifactId>
         </dependency>
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-all</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-tcnative-boringssl-static</artifactId>
+        </dependency>
         <dependency>
             <groupId>io.grpc</groupId>
-            <artifactId>grpc-netty</artifactId>
+            <artifactId>grpc-netty-shaded</artifactId>
         </dependency>
         <dependency>
             <groupId>io.grpc</groupId>
@@ -262,17 +270,6 @@
             <artifactId>rest-client</artifactId>
             <scope>test</scope>
         </dependency>
-        <dependency>
-            <groupId>org.springframework.boot</groupId>
-            <artifactId>spring-boot-starter-test</artifactId>
-            <scope>test</scope>
-            <exclusions>
-                <exclusion>
-                    <groupId>com.vaadin.external.google</groupId>
-                    <artifactId>android-json</artifactId>
-                </exclusion>
-            </exclusions>
-        </dependency>
         <dependency>
             <groupId>org.springframework.security</groupId>
             <artifactId>spring-security-test</artifactId>
@@ -289,18 +286,18 @@
             <scope>test</scope>
         </dependency>
         <dependency>
-            <groupId>junit</groupId>
-            <artifactId>junit</artifactId>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-test</artifactId>
             <scope>test</scope>
         </dependency>
         <dependency>
-            <groupId>org.awaitility</groupId>
-            <artifactId>awaitility</artifactId>
+            <groupId>org.junit.vintage</groupId>
+            <artifactId>junit-vintage-engine</artifactId>
             <scope>test</scope>
         </dependency>
         <dependency>
-            <groupId>org.mockito</groupId>
-            <artifactId>mockito-core</artifactId>
+            <groupId>org.awaitility</groupId>
+            <artifactId>awaitility</artifactId>
             <scope>test</scope>
         </dependency>
         <dependency>
@@ -316,6 +313,7 @@
         <dependency>
             <groupId>org.hsqldb</groupId>
             <artifactId>hsqldb</artifactId>
+            <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>org.javadelight</groupId>

application/src/main/data/json/system/widget_bundles/cards.json

@@ -400,6 +400,14 @@ public void printStats() {
     @Getter
     private String debugPerTenantLimitsConfiguration;
 
+    @Value("${actors.rpc.sequential:false}")
+    @Getter
+    private boolean rpcSequential;
+
+    @Value("${actors.rpc.max_retries:5}")
+    @Getter
+    private int maxRpcRetries;
+
     @Getter
     @Setter
     private TbActorSystem actorSystem;
@@ -478,7 +486,6 @@ public TopicPartitionInfo resolve(ServiceType serviceType, String queueName, Ten
         return partitionService.resolve(serviceType, queueName, tenantId, entityId);
     }
 
-
     public String getServiceId() {
         return serviceInfoProvider.getServiceId();
     }

application/src/main/data/json/system/widget_bundles/charts.json

@@ -25,4 +25,6 @@
 public class ToDeviceRpcRequestMetadata {
     private final ToDeviceRpcRequestActorMsg msg;
     private final boolean sent;
+    private int retries;
+    private boolean delivered;
 }

application/src/main/data/json/system/widget_bundles/input_widgets.json

@@ -250,10 +250,17 @@ private void onTellNext(TbMsg msg, RuleNodeId originatorNodeId, Set<String> rela
             checkActive(msg);
             EntityId entityId = msg.getOriginator();
             TopicPartitionInfo tpi = systemContext.resolve(ServiceType.TB_RULE_ENGINE, msg.getQueueName(), tenantId, entityId);
-            List<RuleNodeRelation> relations = nodeRoutes.get(originatorNodeId).stream()
+
+            List<RuleNodeRelation> ruleNodeRelations = nodeRoutes.get(originatorNodeId);
+            if (ruleNodeRelations == null) { // When unchecked, this will cause NullPointerException when rule node doesn't exist anymore
+                log.warn("[{}][{}][{}] No outbound relations (null). Probably rule node does not exist. Probably old message.", tenantId, entityId, msg.getId());
+                ruleNodeRelations = Collections.emptyList();
+            }
+
+            List<RuleNodeRelation> relationsByTypes = ruleNodeRelations.stream()
                     .filter(r -> contains(relationTypes, r.getType()))
                     .collect(Collectors.toList());
-            int relationsCount = relations.size();
+            int relationsCount = relationsByTypes.size();
             if (relationsCount == 0) {
                 log.trace("[{}][{}][{}] No outbound relations to process", tenantId, entityId, msg.getId());
                 if (relationTypes.contains(TbRelationTypes.FAILURE)) {
@@ -268,21 +275,22 @@ private void onTellNext(TbMsg msg, RuleNodeId originatorNodeId, Set<String> rela
                     msg.getCallback().onSuccess();
                 }
             } else if (relationsCount == 1) {
-                for (RuleNodeRelation relation : relations) {
+                for (RuleNodeRelation relation : relationsByTypes) {
                     log.trace("[{}][{}][{}] Pushing message to single target: [{}]", tenantId, entityId, msg.getId(), relation.getOut());
                     pushToTarget(tpi, msg, relation.getOut(), relation.getType());
                 }
             } else {
                 MultipleTbQueueTbMsgCallbackWrapper callbackWrapper = new MultipleTbQueueTbMsgCallbackWrapper(relationsCount, msg.getCallback());
-                log.trace("[{}][{}][{}] Pushing message to multiple targets: [{}]", tenantId, entityId, msg.getId(), relations);
-                for (RuleNodeRelation relation : relations) {
+                log.trace("[{}][{}][{}] Pushing message to multiple targets: [{}]", tenantId, entityId, msg.getId(), relationsByTypes);
+                for (RuleNodeRelation relation : relationsByTypes) {
                     EntityId target = relation.getOut();
                     putToQueue(tpi, msg, callbackWrapper, target);
                 }
             }
         } catch (RuleNodeException rne) {
             msg.getCallback().onFailure(rne);
         } catch (Exception e) {
+            log.warn("[" + tenantId + "]" + "[" + entityId + "]" + "[" + msg.getId() + "]" + " onTellNext failure", e);
             msg.getCallback().onFailure(new RuleEngineException("onTellNext - " + e.getMessage()));
         }
     }

application/src/main/data/json/system/widget_bundles/maps.json

@@ -75,11 +75,12 @@ protected DeferredResult<ResponseEntity> handleDeviceRPCRequest(boolean oneWay,
             SecurityUser currentUser = getCurrentUser();
             TenantId tenantId = currentUser.getTenantId();
             final DeferredResult<ResponseEntity> response = new DeferredResult<>();
-            long timeout = rpcRequestBody.has("timeout") ? rpcRequestBody.get("timeout").asLong() : defaultTimeout;
-            long expTime = System.currentTimeMillis() + Math.max(minTimeout, timeout);
+            long timeout = rpcRequestBody.has(DataConstants.TIMEOUT) ? rpcRequestBody.get(DataConstants.TIMEOUT).asLong() : defaultTimeout;
+            long expTime = rpcRequestBody.has(DataConstants.EXPIRATION_TIME) ? rpcRequestBody.get(DataConstants.EXPIRATION_TIME).asLong() : System.currentTimeMillis() + Math.max(minTimeout, timeout);
             UUID rpcRequestUUID = rpcRequestBody.has("requestUUID") ? UUID.fromString(rpcRequestBody.get("requestUUID").asText()) : UUID.randomUUID();
             boolean persisted = rpcRequestBody.has(DataConstants.PERSISTENT) && rpcRequestBody.get(DataConstants.PERSISTENT).asBoolean();
             String additionalInfo =  JacksonUtil.toString(rpcRequestBody.get(DataConstants.ADDITIONAL_INFO));
+            Integer retries = rpcRequestBody.has(DataConstants.RETRIES) ? rpcRequestBody.get(DataConstants.RETRIES).asInt() : null;
             accessValidator.validate(currentUser, Operation.RPC_CALL, deviceId, new HttpValidationCallback(response, new FutureCallback<>() {
                 @Override
                 public void onSuccess(@Nullable DeferredResult<ResponseEntity> result) {
@@ -90,6 +91,7 @@ public void onSuccess(@Nullable DeferredResult<ResponseEntity> result) {
                             expTime,
                             body,
                             persisted,
+                            retries,
                             additionalInfo
                     );
                     deviceRpcService.processRestApiRpcRequest(rpcRequest, fromDeviceRpcResponse -> reply(new LocalRequestMetaData(rpcRequest, currentUser, result), fromDeviceRpcResponse, timeoutStatus, noActiveConnectionStatus), currentUser);