TLS/SSL support?

[ryanschneider]

Does the redis extension support servers where TLS/SSL is required? I didn't see any mention of support in the code so I suspect not.

[ryanschneider]

FWIW here's the issue I ran into, not sure if this is due to TLS not being supported or a different issue:

> CREATE SECRET IF NOT EXISTS redis (
          TYPE redis,
          PROVIDER config,
          host 'some.aws.valkey.instance.use1.cache.amazonaws.com',
          port '6379'
      );
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
> SELECT * FROM redis_hscan_over_scan('foo:*', '*', 100, 'redis');
Invalid Input Error:
Redis secret not found

[lmangani]

First of all thanks for giving this a try and for the report!

Does the redis extension support servers where TLS/SSL is required? I didn't see any mention of support in the code so I suspect not.

I've only tested and developed this with redis cloud wich uses TLS for connecting so yes, it should support it fine. About the secret, same as above but i've been using PERSISTENT secrets to work across multiple executions.
Are you running this in DuckDB CLI? Note the password is a required field although secrets will allow skipping it.

CREATE PERSISTENT SECRET IF NOT EXISTS redis (
        TYPE redis,
        PROVIDER config,
        host 'redis-1234.ec2.redns.redis-cloud.com',
        port '16959',
        password 'xxxxxx'
    );

[ryanschneider]

Thanks for the feedback, I'll keep trying things out. Yes, this has all been in a duckdb CLI session. Here's what I've tried so far:

> duckdb
DuckDB v1.3.0 (Ossivalis) 71c5c07cdd
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D FORCE INSTALL redis from community; LOAD redis;
D CREATE SECRET IF NOT EXISTS metadata_cache (
          TYPE redis,
          PROVIDER config,
          host 'REDACTED.use1.cache.amazonaws.com',
          port '6379'
      );
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D SELECT * FROM redis_hscan_over_scan('billing:column_sizes:*', '*', 100, 'metadata_cache');
Invalid Input Error:
Redis secret not found
D DROP SECRET metadata_cache;
D CREATE SECRET IF NOT EXISTS metadata_cache (
          TYPE redis,
          PROVIDER config,
          host 'REDACTED.use1.cache.amazonaws.com',
          port '6379',
          password ''
  );
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D SELECT * FROM redis_hscan_over_scan('billing:column_sizes:*', '*', 100, 'metadata_cache');
Invalid Input Error:
Redis secret not found
D DROP SECRET metadata_cache;
D
D CREATE PERSISTENT SECRET IF NOT EXISTS metadata_cache (
          TYPE redis,
          PROVIDER config,
          host 'REDACTED.use1.cache.amazonaws.com',
          port '6379',
          password ''
  );
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D SELECT * FROM redis_hscan_over_scan('billing:column_sizes:*', '*', 100, 'metadata_cache');
Invalid Input Error:
Redis secret not found

D DROP PERSISTENT SECRET metadata_cache;
Invalid Input Error:
Secret type 'redis' not found, try removing the secret at path '/Users/ryanschneider/.duckdb/stored_secrets/metadata_cache.duckdb_secret'.

D FROM duckdb_secrets();
Invalid Input Error:
Secret type 'redis' not found, try removing the secret at path '/Users/ryanschneider/.duckdb/stored_secrets/metadata_cache.duckdb_secret'.

Those last two errors are really weird, since the file does exist. It's like the secret-handling portion of the plugin isn't getting installed correctly for me.

I'm on MacOS 15.3.2 if that helps. Here's the extension.info file dump:

> xxd ~/.duckdb/extensions/v1.3.0/osx_arm64/redis.duckdb_extension.info
00000000: 6400 0165 0051 6874 7470 3a2f 2f63 6f6d  d..e.Qhttp://com
00000010: 6d75 6e69 7479 2d65 7874 656e 7369 6f6e  munity-extension
00000020: 732e 6475 636b 6462 2e6f 7267 2f76 312e  s.duckdb.org/v1.
00000030: 332e 302f 6f73 785f 6172 6d36 342f 7265  3.0/osx_arm64/re
00000040: 6469 732e 6475 636b 6462 5f65 7874 656e  dis.duckdb_exten
00000050: 7369 6f6e 2e67 7a66 0026 6874 7470 3a2f  sion.gzf.&http:/
00000060: 2f63 6f6d 6d75 6e69 7479 2d65 7874 656e  /community-exten
00000070: 7369 6f6e 732e 6475 636b 6462 2e6f 7267  sions.duckdb.org
00000080: 6700 0730 3362 6238 3363 6800 2222 6337  g..03bb83ch.""c7
00000090: 6261 3263 6464 3230 6564 6233 6437 6339  ba2cdd20edb3d7c9
000000a0: 3566 6264 3038 3566 3437 3962 3364 22ff  5fbd085f479b3d".
000000b0: ff

[ryanschneider]

FWIW the 03bb83c in the info clearly points to this commit 03bb83c

[ryanschneider]

Oh, some progress:

D CREATE SECRET IF NOT EXISTS metadata_cache (
          TYPE redis,
          PROVIDER config,
          host 'REDACTED.use1.cache.amazonaws.com',
          port '6379',
          password 'password'
  );
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D FROM duckdb_secrets();
┌────────────────┬─────────┬──────────┬────────────┬─────────┬───────────┬────────────────────────────────────────────────────┐
│      name      │  type   │ provider │ persistent │ storage │   scope   │                   secret_string                    │
│    varchar     │ varchar │ varchar  │  boolean   │ varchar │ varchar[] │                      varchar                       │
├────────────────┼─────────┼──────────┼────────────┼─────────┼───────────┼────────────────────────────────────────────────────┤
│ metadata_cache │ redis   │ config   │ false      │ memory  │ []        │ name=metadata_cache;type=redis;provider=config;s…  │
└────────────────┴─────────┴──────────┴────────────┴─────────┴───────────┴────────────────────────────────────────────────────┘
D SELECT * FROM redis_hscan_over_scan('billing:column_sizes:*', '*', 100, 'metadata_cache');
Invalid Input Error:
Redis secret not found

So password must not only exist but we non-empty.

[ryanschneider]

Aha!

D DROP SECRET metadata_cache;
D CREATE SECRET IF NOT EXISTS redis (
          TYPE redis,
          PROVIDER config,
          host 'REDACTED.use1.cache.amazonaws.com',
          port '6379',
          password 'password'
  );
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D SELECT * FROM redis_hscan_over_scan('billing:column_sizes:*', '*', 100, 'redis');
# this now hangs.

The secret must be named redis! I was doing that the first time (but without the password field) and switched when debugging.

[ryanschneider]

So I think my issue is twofold, both related to this function: https://github.com/quackscience/duckdb-extension-redis/blob/main/src/redis_extension.cpp#L151-L174

• I don't see TLS being enabled. It's been years since I've done any C++ but I think this would require adding some sort of TLS option to the secret, then adding a TLS client to the io_context like in this example: https://live.boost.org/doc/libs/master/libs/mqtt5/doc/html/mqtt5/hello_world_over_tls.html

• While this function seems to support empty passwords, I think GetRedisSecret doesn't, perhaps TryGetValue errors on empty values here? https://github.com/quackscience/duckdb-extension-redis/blob/3da5899bbfc2266e351233119ed86e424d0660b1/src/redis_extension.cpp#L245

[lmangani]

Thank you SO much for the deep dive @ryanschneider its really useful! Thanks to your analysis I'll be able to work on it this coming weekend and hopefully resolve the problem or clarify the usage pattern more clearly in the docs. To confirm the testing succeeded with https://redis.io/try-free/