Huly Network

Signed-off-by: Andrey Sobolev <[email protected]>

Author: haiodo

.github/workflows/main.yml

@@ -23,6 +23,7 @@ on:
 env:
   CacheFolders: |
     communication
+    network
     common
     desktop
     desktop-package

README.md

@@ -28,13 +28,25 @@ If you want to interact with Huly programmatically, check out our [API Client](.
 
 You can find API usage examples in the [Huly examples](https://github.com/hcengineering/huly-examples) repository.
 
+## Huly Virtual Network
+
+The platform features a distributed network architecture that enables scalable, fault-tolerant communication between accounts, workspaces, and nodes. The [Huly Virtual Network](./network/README.md) provides:
+
+- **Distributed Load Balancing**: Intelligent routing across multiple nodes using consistent hashing
+- **Multi-Tenant Architecture**: Secure workspace isolation with role-based access control
+- **Fault Tolerance**: Automatic failover and recovery mechanisms
+- **Real-time Communication**: Event-driven architecture with broadcast capabilities
+
+For detailed information about the network architecture, deployment, and API reference, see the [Network Documentation](./network/README.md).
+
 ## Table of Contents
 
 - [Huly Platform](#huly-platform)
   - [About](#about)
   - [Self-Hosting](#self-hosting)
   - [Activity](#activity)
   - [API Client](#api-client)
+  - [Huly Virtual Network](#huly-virtual-network)
   - [Table of Contents](#table-of-contents)
   - [Pre-requisites](#pre-requisites)
   - [Verification](#verification)
@@ -106,20 +118,21 @@ This project uses GitHub Packages for dependency management. To successfully dow
 Follow these steps:
 
 1. Generate a GitHub Token:
+
 - Log in to your GitHub account
 - Go to **Settings** > **Developer settings** > **Personal access tokens** (https://github.com/settings/personal-access-tokens)
 - Click **Generate new token**
 - Select the required scopes (at least `read:packages`)
 - Generate the token and copy it
 
 2. Authenticate with npm:
+
 ```bash
 npm login --registry=https://npm.pkg.github.com
 ```
 
 When prompted, enter your GitHub username, use the generated token as your password
 
-
 ## Fast start
 
 ```bash
@@ -280,6 +293,7 @@ This guide describes the nuances of building and running the application from so
 #### Disk Space Requirements
 
 Ensure you have sufficient disk space available:
+
 - A fully deployed local application in clean Docker will consume slightly more than **35 GB** of WSL virtual disk space
 - The application folder after build (sources + artifacts) will occupy **4.5 GB**
 
@@ -303,6 +317,7 @@ Make sure Docker is accessible from WSL:
 Windows Git often automatically replaces line endings. Since most build scripts are `.sh` files, ensure your Windows checkout doesn't break them.
 
 **Solution options:**
+
 - Checkout from WSL instead of Windows
 - Configure Git on Windows to disable auto-replacement:
   ```bash
@@ -343,6 +358,7 @@ After these preparations, the build instructions should work without issues.
 When starting the application (`rush docker:up`), some network ports in Windows might be occupied. You can fix port mapping in the `\dev\docker-compose.yaml` file.
 
 **Important:** Depending on which port you change, you'll need to:
+
 1. Find what's using that port
 2. Update the new address in the corresponding service configuration
 

common/config/rush/pnpm-lock.yaml

@@ -0,0 +1,118 @@
+# Huly Virtual Network
+
+A distributed, scalable virtual network architecture for the Huly that enables fault-tolerant performance communication.
+
+## 🚀 Overview
+
+The Huly Virtual Network is a sophisticated distributed system designed to handle enterprise-scale workloads with the following key capabilities:
+
+- **Distributed Load Balancing**: Intelligent routing to key components across multiple phytsical nodes.
+- **Multi-Tenant Architecture**: Secure isolation of containers for user sessions/search engines/transaction processors.
+- **Fault Tolerance**: Automatic failover and recovery mechanisms
+- **Scaling**: Horizontal and vertical service scalling.
+- **Real-time Communication**: Event-driven architecture with broadcast capabilities between containers.
+
+## Network Architecture Components
+
+### Agents & Containers
+
+Agent is a top of the rock for containers. Containers are work horses for any application build on top of Huly Network, they could be started, located and used.
+Every container could be located by its {kind + uuid} or using a labeling system. After being found anyone could send a message to container and ask for some activity.
+
+Container's could be communicated two ways:
+
+1. Anyone could send a message to container using network.
+2. A request/response connection could be established to container using network.
+
+Network provide a references to containers, if container is references it will be active until reference is exists. If there is no references to contaienr, it will be keept for some timeout and closed.
+
+Agents provide a list of container kinds they support to be started to Huly Network, manage them and provide live/monitoring and networking capabilities.
+Communication to containers are managed by Huly Network.
+
+```mermaid
+flowchart BT
+  subgraph Agent["Agent"]
+    subgraph _TX["Transactor"]
+      TX1["ws1"]
+      TX2["ws2"]
+    end
+    subgraph _VMM["Virtual Machine"]
+      AG1["Hulia Agent"]
+      AG2["James Agent"]
+    end
+    subgraph _Sessions["Session"]
+      S1["user1"]
+      S2["user2"]
+    end
+    subgraph _Query["Query Engine"]
+      Q1["Europe"]
+      Q3["ws3"]
+    end
+  end
+  subgraph Agent2["Agent2"]
+    subgraph _TX_2["Transactor"]
+      TX3["ws3"]
+      TX4["ws4"]
+    end
+    subgraph _Query_2["Query Engine"]
+      Q1_2["Europe"]
+      Q2_2["America"]
+    end
+  end
+
+  subgraph NET["Huly Network"]
+    Containers["Containers"]
+    Agents["Agents"]
+  end
+
+
+  S1 -.-> TX1 & Q1 & Q2_2 & TX4
+  Agent --> NET
+  Agent2 --> NET
+  AG1@{ shape: stored-data}
+  AG2@{ shape: stored-data}
+  S1@{ shape: h-cyl}
+  S2@{ shape: h-cyl}
+  style Agent stroke:#00C853
+```
+
+## Building Huly on top of Huly Network
+
+Huly could be managed by following set of container kinds, `session`, `query`, `transactor`.
+
+- session -> a map/reduce/find executor for queries and transactions from client.
+- query -> a DB query engine, execute `find` requests from session and pass them to DB, allow to search for all data per region. Should have access to tables of account -> workspace mapping for security.
+- transactor -> modification archestrator for all edit operations, do them one by one.
+
+```mermaid
+flowchart
+  Endpoint -.->|
+  connect
+  session/user1
+              |HulyNetwork[Huly Network]
+
+  Endpoint <-->|find,tx| sesion:user1
+
+  sesion:user1 -..->|list-workspaces| Account
+
+  sesion:user1 -..->|find| query:europe
+  sesion:user1 -..->|find| transactor:ws1
+
+  sesion:user1 -..->|cnt:1| query:europe
+  query:europe -..->|resp:1/1| sesion:user1
+  query:europe -..->|resp:1/2:fin| sesion:user1
+
+  sesion:user1 -..->|response chunks| Endpoint
+
+  sesion:user1 -..->|tx| transactor:ws1
+
+  transactor:ws1 -..->|resp| sesion:user1
+
+  query:europe -..->|"select"| DB[DB]
+  transactor:ws1 -..->|select/update| DB
+
+  DB@{shape: database}
+
+```
+
+## ZeroMQ network implementation

network/README.md

@@ -0,0 +1,7 @@
+module.exports = {
+  extends: ['./node_modules/@hcengineering/platform-rig/profiles/node/eslint.config.json'],
+  parserOptions: {
+    tsconfigRootDir: __dirname,
+    project: './tsconfig.json'
+  }
+}

network/core/.eslintrc.js

@@ -0,0 +1,5 @@
+{  
+  "$schema": "https://developer.microsoft.com/json-schemas/rig-package/rig.schema.json",
+  "rigPackageName": "@hcengineering/platform-rig",
+  "rigProfile": "node"
+}

network/core/config/rig.json

@@ -0,0 +1,7 @@
+module.exports = {
+  preset: 'ts-jest',
+  testEnvironment: 'node',
+  testMatch: ['**/?(*.)+(spec|test).[jt]s?(x)'],
+  roots: ["./src"],
+  coverageReporters: ["text-summary", "html"]
+}