Add support for dictionary encoding

[ianmcook]

Support for reading dictionary-encoded arrays is not yet implemented.

Simple example:

1. Write file with Python:

import pyarrow as pa
from pyarrow import compute as pc

table = pa.Table.from_arrays(
  arrays=[pc.dictionary_encode(pa.array(["a", "b", "c"]))],
  names=["x"]
)

with pa.ipc.new_stream("dict.arrows", table.schema) as writer:
  writer.write_table(table)

2. Try to read file with DuckDB:

INSTALL arrow FROM community;
LOAD arrow;

SELECT * FROM read_arrow('dict.arrows');

3. See error:

IO Error:
ArrowIpcDecoderDecodeSchema(decoder.get(), base_schema.get(), &error) failed with errno 45: Schema message field with DictionaryEncoding not supported

[ianmcook]

This depends on implementation in nanoarrow. The issue for that is apache/arrow-nanoarrow#622.

[paleolimbot]

I started PR for that as well but forgot to link it properly: apache/arrow-nanoarrow#738

[aszenz]

Got the same error on loading a compressed arrow file:

ArrowIpcDecoderDecodeSchema(decoder.get(), base_schema.get(), &error) failed with errno 95: Schema message field with DictionaryEncoding not supported

install arrow from community;
load arrow;
install httpfs;
load httpfs;
select * from 'https://cdn.jsdelivr.net/npm/superstore-arrow/superstore.lz4.arrow' limit 10;