install iceberg extension failed

[benqwu]

My steps are as follows:

git clone https://github.com/duckdb/pg_duckdb && cd pg_duckdb && docker compose up -d

docker compose exec db psql

psql (16.9 (Debian 16.9-1.pgdg120+1))
Type "help" for help.

postgres=# \dx
List of installed extensions
Name | Version | Schema | Description
-----------+---------+------------+------------------------------
pg_duckdb | 1.0.0 | public | DuckDB Embedded in Postgres
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
(2 rows)

postgres=# SELECT duckdb.install_extension('iceberg');
ERROR: (PGDuckDB/install_extension_cpp) IO Error: Input is not a GZIP stream

Can someone help me with this?

[Y--]

Hi @benqwu

I just tried in a fresh PG 16 install, and it seemed to work:

psql (16.3)
Type "help" for help.

y=# \dx
                 List of installed extensions
  Name   | Version |   Schema   |         Description
---------+---------+------------+------------------------------
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
(1 row)

y=# create extension pg_duckdb;
CREATE EXTENSION
y=# \dx
                  List of installed extensions
   Name    | Version |   Schema   |         Description
-----------+---------+------------+------------------------------
 pg_duckdb | 1.0.0   | public     | DuckDB Embedded in Postgres
 plpgsql   | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)

y=# SELECT duckdb.install_extension('iceberg');
 install_extension
-------------------

(1 row)

Some quick question to figure out the problem on your end:

• how did you get or compile the pg_duckdb extension?
• could you remove the <PG data dir>/pg_duckdb/extensions folder and try again?
• could you post the output of select duckdb.raw_query($$ from duckdb_extensions(); $$);?

[benqwu]

@Y-- Thank you for your reply!

how did you get or compile the pg_duckdb extension?
//
This step I haven't prepared yet. I thought the Iceberg extension was already packaged. Where can I obtain the Iceberg extension? Does it need to be manually compiled?

could you remove the /pg_duckdb/extensions folder and try again?
//
/var/lib/postgresql/data/pg_duckdb/extensions/v1.3.0/linux_amd64，I see that this directory is empty.

could you post the output of select duckdb.raw_query($$ from duckdb_extensions(); $$);?
//
postgres=# select duckdb.raw_query($$ from duckdb_extensions(); $$);
NOTICE: result: extension_name loaded installed install_path description aliases extension_version install_mode installed_from
VARCHAR BOOLEAN BOOLEAN VARCHAR VARCHAR VARCHAR[] VARCHAR VARCHAR VARCHAR
[ Rows: 25]
autocomplete false false Adds support for autocomplete in the shell [] NOT_INSTALLED
aws false false Provides features that depend on the AWS SDK [] NOT_INSTALLED
azure false false Adds a filesystem abstraction for Azure blob storage to DuckDB [] NOT_INSTALLED
core_functions true true (BUILT-IN) Core function library [] v1.3.0 STATICALLY_LINKED
delta false false Adds support for Delta Lake [] NOT_INSTALLED
encodings false false All unicode encodings to UTF-8 [] NOT_INSTALLED
excel false false Adds support for Excel-like format strings [] NOT_INSTALLED
fts false false Adds support for Full-Text Search Indexes [] NOT_INSTALLED
httpfs true true (BUILT-IN) Adds support for reading and writing files over a HTTP(S) connection [http, https, s3] 7ce5308 STATICALLY_LINKED
iceberg false false Adds support for Apache Iceberg [] NOT_INSTALLED
icu true true (BUILT-IN) Adds support for time zones and collations using the ICU library [] v1.3.0 STATICALLY_LINKED
inet false false Adds support for IP-related data types and functions [] NOT_INSTALLED
jemalloc true true (BUILT-IN) Overwrites system allocator with JEMalloc [] v1.3.0 STATICALLY_LINKED
json true true (BUILT-IN) Adds support for JSON operations [] v1.3.0 STATICALLY_LINKED
motherduck false false Enables motherduck integration with the system [md] NOT_INSTALLED
mysql_scanner false false Adds support for connecting to a MySQL database [mysql] NOT_INSTALLED
parquet true true (BUILT-IN) Adds support for reading and writing parquet files [] v1.3.0 STATICALLY_LINKED
pgduckdb true false [] UNKNOWN
postgres_scanner false false Adds support for connecting to a Postgres database [postgres] NOT_INSTALLED
spatial false false Geospatial extension that adds support for working with spatial data and functions [] NOT_INSTALLED
sqlite_scanner false false Adds support for reading and writing SQLite database files [sqlite, sqlite3] NOT_INSTALLED
tpcds false false Adds TPC-DS data generation and query support [] NOT_INSTALLED
tpch false false Adds TPC-H data generation and query support [] NOT_INSTALLED
ui false false Adds local UI for DuckDB [] NOT_INSTALLED
vss false false Adds indexing support to accelerate Vector Similarity Search [] NOT_INSTALLED

[Y--]

I thought the Iceberg extension was already packaged.

No, the extension isn't packaged/installed by default.

Where can I obtain the Iceberg extension?

Usually, running INSTALL iceberg in DuckDB does the trick (which is what duckdb.install_extension does under the hood.

Does it need to be manually compiled?

No, it is already pre-compiled for most platforms.

Can you install the latest DuckDB CLI from https://duckdb.org/#quickinstall and try to run INSTALL iceberg to make sure there's no network issue of some kind?

[benqwu]

Thank you for your help. The problem has been solved. It was a network issue.

I first tried using the DuckDB CLI and ran INSTALL iceberg, but got the same error message. Then I checked the DuckDB manual and solved the problem by using set http_proxy.
I have another question. Can I set http_proxy in the psql command line? I noticed that its syntax is different from that of the DuckDB CLI. Is there any relevant documentation that introduces the use of DuckDB syntax in psql?
Finally, I temporarily solved the problem by means of directory mapping. I mapped the directory of the extensions downloaded via the DuckDB CLI on the host to the corresponding directory of pg in the container.

[Y--]

Can I set http_proxy in the psql command line?

You should be able with something like:

select duckdb.raw_query($$ set http_proxy='<your proxy url>' $$);

Note, though, that it will only apply to the current DuckDB connection, not all other (nor any background worker).
Adding proper support for this DuckDB setting makes sense. I've created #837 for that.

Is there any relevant documentation that introduces the use of DuckDB syntax in psql?

Generally speaking, you can issue queries directly to DuckDB with the raw_query function as indicated above. We don't have more documentation on this.