HPCC-35595 Improve ESP session ID security

Migrate ESP session IDs from 32-bit to 128-bit secure strings, addressing session hijacking
vulnerability.
- Change session token type from unsigned to const char* in all security interfaces.
- Update createHTTPSession() to generate 128-bit cryptographically secure random session IDs.
- Implement OpenSSL RAND_bytes for secure random generation.
- Fallback to previous hash-based tokens when no OpenSSL or OpenSSL failures.
- Update all session validation and lookup functions to use string-based XPath queries.
- Remove numeric readCookie function, use string-based cookie reading.
- Update session ID documentation to show new 128-bit hex format.
- Add collision detection retry loop with max 3 attempts, do not re-use sessions.

Session token in SecUser distinct from sessionID- can be used to verify auth
but can't be used to lookup sessionID (wasn't used that way).
- Add sessionIDToToken() helper to hash 128-bit session ID to 32-bit token.
- Update all setSessionToken calls to use hash conversion.
- Maintains backward compatibility with existing sessionToken interfaces.
- Session IDs remain 128-bit for security while tokens remain 32-bit unsigned.

Maintain separate external ID that can't derive session ID from.
- Add SHA256 hashing for PropSessionExternalID (when OpenSSL available).
- Fallback to hashc when OpenSSL not present.
- External ID is now different from internal session ID for admin security.

- Add local testing script with option to adjust expectations when OpenSSL
  is not available.
- Fix config XSLT to allow granting admins WsESPControl access.
- Remove error logging when using fallback scheme. Print one-time message on
  startup when OpenSSL unavailable.
- Replace logging of SessionID with ExternalSessionID.

Signed-off-by: Terrence Asselin <terrence.asselin+copilot@lexisnexisrisk.com>

Author: Copilot

esp/bindings/http/platform/httpservice.cpp

@@ -102,27 +102,26 @@ class CEspHttpServer : implements IHttpServerService, public CInterface
     EspAuthState checkUserAuthPerRequest(EspAuthRequest& authReq);
     EspAuthState checkUserAuthPerSession(EspAuthRequest& authReq, StringBuffer& authorizationHeader);
     EspAuthState authNewSession(EspAuthRequest& authReq, const char* _userName, const char* _password, const char* sessionStartURL, bool unlock);
-    EspAuthState authExistingSession(EspAuthRequest& req, unsigned sessionID);
-    void logoutSession(EspAuthRequest& authReq, unsigned sessionID, IPropertyTree* domainSessions, bool lock);
+    EspAuthState authExistingSession(EspAuthRequest& req, const char* sessionID);
+    void logoutSession(EspAuthRequest& authReq, const char* sessionID, IPropertyTree* domainSessions, bool lock);
     void askUserLogin(EspAuthRequest& authReq, const char* msg);
     bool changeRedirectURL(EspAuthRequest& authReq);
     EspAuthState handleUserNameOnlyMode(EspAuthRequest& authReq);
     EspAuthState handleAuthFailed(bool sessionAuth, EspAuthRequest& authReq, bool unlock, const char* msg);
     EspHttpBinding* getEspHttpBinding(EspAuthRequest& req);
     bool isAuthRequiredForBinding(EspAuthRequest& req);
     void authOptionalGroups(EspAuthRequest& req);
-    unsigned createHTTPSession(IEspContext* ctx, EspHttpBinding* authBinding, const char* userID, const char* loginURL);
+    const char* createHTTPSession(IEspContext* ctx, EspHttpBinding* authBinding, const char* userID, const char* loginURL, StringBuffer& sessionID);
     void timeoutESPSessions(EspHttpBinding* authBinding, IPropertyTree* espSessions);
     void addCookie(const char* cookieName, const char *cookieValue, int maxAgeSec, bool httpOnly);
     void clearCookie(const char* cookieName);
     void clearSessionCookies(EspAuthRequest& authReq);
-    unsigned readCookie(const char* cookieName);
     const char* readCookie(const char* cookieName, StringBuffer& cookieValue);
     void sendLockResponse(bool lock, bool error, const char* msg);
     void sendAuthorizationMsg(EspAuthRequest& authReq);
     void sendGetAuthTypeResponse(EspAuthRequest& authReq, const char* authType);
     void createGetSessionTimeoutResponse(StringBuffer& resp, ESPSerializationFormat format, IPropertyTree* sessionTree);
-    void resetSessionTimeout(EspAuthRequest& authReq, unsigned sessionID, StringBuffer& resp, ESPSerializationFormat format, IPropertyTree* sessionTree);
+    void resetSessionTimeout(EspAuthRequest& authReq, const char* sessionID, StringBuffer& resp, ESPSerializationFormat format, IPropertyTree* sessionTree);
     void sendException(EspAuthRequest& authReq, unsigned code, const char* msg);
     void sendInternalError(bool loggedDetails);
     void sendMessage(const char* msg, const char* msgType);

esp/bindings/http/platform/httpservice.hpp

@@ -81,24 +81,24 @@ static const char* const PropSessionLoginURL = "@loginurl";
 <Sessions>
  <Process name="myesp">
    <Application port="8010">
-    <Session_3831947145 createtime="1497376914"
-             id="3831947145"
+    <Session_a3f5e7b9c2d4f81e0b2c4d6f8a1c3e5f createtime="1497376914"
+             id="a3f5e7b9c2d4f81e0b2c4d6f8a1c3e5f"
              lastaccessed="1497377015"
              timeoutAt="1497477015"
              loginurl="/"
              netaddr="10.176.152.200"
              userid="user1"/>
-    <Session_4106750941 createtime="1497377427"
-             id="4106750941"
+    <Session_b4c6d8e0a2f4b81c3d5e7f9a1c3e5f7b createtime="1497377427"
+             id="b4c6d8e0a2f4b81c3d5e7f9a1c3e5f7b"
              lastaccessed="1497377427"
              timeoutAt="1497477427"
              loginurl="/"
              netaddr="10.176.152.200"
              userid="user2"/>
    </Application>
    <Application port="8002">
-    <Session_3680948651 createtime="1497376989"
-             id="3680948651"
+    <Session_c5d7e9f1b3a5c92d4e6f8a0b2c4d6e8f createtime="1497376989"
+             id="c5d7e9f1b3a5c92d4e6f8a0b2c4d6e8f"
              lastaccessed="1497377003"
              timeoutAt="1497477003"
              loginurl="/"

esp/platform/espcontext.hpp

@@ -572,6 +572,9 @@ int init_main(int argc, const char* argv[])
         openEspLogFile(envpt.get(), procpt.get());
 
         DBGLOG("Esp starting %s", hpccBuildInfo.buildTag);
+#ifndef _USE_OPENSSL
+        UWARNLOG("OpenSSL not in use. ESP Session ID generation is less secure. To use OpenSSL, build ESP with OpenSSL support.");
+#endif
 
         StringBuffer componentfilesDir;
         if(procpt->hasProp("@componentfilesDir"))

esp/platform/espp.cpp

@@ -617,6 +617,9 @@ This is required by its binding with ESP service '<xsl:value-of select="$espServ
                 <xsl:for-each select="../EspProcess[Authentication/@ldapServer=$ldapservername]/EspBinding">
                     <Binding name="{@name}" service="{@service}" port="{@port}" basedn="{@resourcesBasedn}" workunitsBasedn="{@workunitsBasedn}"/>
                 </xsl:for-each>
+                <xsl:for-each select="../EspProcess[Authentication/@ldapServer=$ldapservername]/EspControlBinding">
+                    <Binding name="{@name}" service="{@service}" port="{@port}" basedn="{@resourcesBasedn}" workunitsBasedn="{@workunitsBasedn}"/>
+                </xsl:for-each>
             </Resources>
         </EspService>
 

initfiles/componentfiles/configxml/@temp/esp_service_WsSMC.xsl

@@ -0,0 +1,133 @@
+# Overview
+
+These are tests for the ESP session ID moving to a 128-bit cryptographically secure format.
+
+# Implementation
+
+## Prerequisites and Assumptions
+
+The tests require a running ESP with security enabled, so they are expected to be run manually. These assumptions about the environment are set as variables in the test script but can be overridden on the CLI:
+
+1. HTTP default, but can support HTTPS
+2. Target ESP host is 127.0.0.1 (port 8010)
+3. Regular user ID is `hpcc_user`
+4. Admin user ID is `hpcc_admin`
+
+The user passwords are expected to be set as environmental values but can be specified on the CLI:
+
+1. Regular user password in `$HPCC_TEST_USER_PW`
+2. Admin user password in `$HPCC_TEST_ADMIN_PW`
+
+The ESP must have WSESPControl authorized for the Admin user, and that user must belong to a group of the same name as the LDAP Admin group.
+
+### Python Dependencies
+
+Install required Python packages:
+
+```bash
+pip install requests
+```
+
+## Running Tests
+
+### Basic Usage
+
+Set environment variables and run all tests:
+
+```bash
+export HPCC_TEST_USER_PW=your_user_password
+export HPCC_TEST_ADMIN_PW=your_admin_password
+python3 sessionid_test.py
+```
+
+### Advanced Options
+
+```bash
+# Custom host and port
+python3 sessionid_test.py --host 192.168.1.100 --port 8010
+
+# Use HTTPS
+python3 sessionid_test.py --protocol https
+
+# Custom credentials
+python3 sessionid_test.py --user myuser --user-pw mypass --admin myadmin --admin-pw adminpass
+
+# Run specific test
+python3 sessionid_test.py -t test_new_format_validation
+
+# Verbose output
+python3 sessionid_test.py -v
+
+# Show help
+python3 sessionid_test.py --help
+```
+
+## Tests
+
+The test suite (`sessionid_test.py`) implements the following automated tests using ESP APIs. Tests pull data from response fields, headers and cookies as needed. All tests run as regular user unless otherwise specified.
+
+### Test 1: New Format Validation
+- Login via `/esp/login`
+- Extract `ESPSessionID` cookie
+- Validate session ID is 32-character hex string (128-bit)
+- Make authenticated request to `/WsSMC/Activity`
+- Verify successful authentication with new session ID format
+
+### Test 2: Incorrect Format/Unknown Session ID Rejection
+- Craft multiple malformed session IDs:
+  - Empty string
+  - Too short
+  - Too long (33+ characters)
+  - Non-hex characters
+  - Wrong format (with dashes, etc.)
+- Attempt authenticated requests with each invalid session ID
+- Confirm expected failure (HTTP 401/403 or redirect to login)
+
+### Test 3: ws_espcontrol Session Timeout
+1. Login as regular user
+2. Verify session is active with authenticated request
+3. Login as admin
+4. Use `/WSESPControl/SessionQuery` to find user's external session ID
+5. Call `/WSESPControl/SetSessionTimeout` with `TimeoutMinutes=1`
+6. Wait 90 seconds (timeout + ESP cleanup cycle buffer)
+7. Confirm session has timed out (request fails or redirects)
+
+### Test 4: Logout Session Invalidation
+1. Login as user
+2. Extract session ID and verify it works
+3. Call `/esp/logout`
+4. Attempt authenticated request with old session cookie
+5. Confirm session is invalid (HTTP 401/403 or redirect)
+
+### Test 5: Concurrent Sessions Load Test
+- Use `ThreadPoolExecutor` to create multiple concurrent login sessions
+- All threads login as same user simultaneously
+- Capture session IDs from all successful logins
+- Verify each session is active by making authenticated request to `/WsSMC/Activity`
+- Confirm ESP allows multiple concurrent sessions per user
+- Confirm all captured session IDs are functional
+
+### Test 6: Session ID Uniqueness
+1. Login as admin for session queries
+2. Perform many sequential logins as regular user (configurable count, default 1000)
+3. For each iteration:
+   - Login and capture session ID from cookie
+   - Use `/WSESPControl/SessionQuery` to retrieve external session ID
+   - Store both session ID and external ID
+   - Logout to clean up session
+4. Verify all session IDs are unique (no duplicates in set)
+5. Verify all external IDs are unique (no duplicates in set)
+6. Verify external IDs differ from their corresponding session IDs
+7. Report statistics: unique counts, duplicate counts, failed logins/queries
+
+### Test 7: Active Session Collision Detection
+1. Login as admin for session queries
+2. Create many concurrent active sessions (configurable count, default 1000)
+3. Keep all `ESPSession` objects alive (no logout)
+4. Verify all session IDs collected are unique
+5. Use `/WSESPControl/SessionQuery` to retrieve all external session IDs
+6. Extract all external IDs from admin API response
+7. Verify count of external IDs matches count of active sessions
+8. Verify all external IDs are unique
+9. Verify external IDs differ from their session IDs
+10. Confirm collision detection works under realistic concurrent load