Skip to content

[BUG] S3 style storage needs additional configuration #1289

@dcbark01

Description

@dcbark01

Great work on Cocoindex so far.
I've been trying to integrate it with my Minio S3 storage instance.
#600 introduced support for event notifications for S3-style storage (e.g. Minio).
However, I kept getting 'dispatch errors' when trying to connect.
The issue seems to be the path based addressing style that Minio requires.
In theory, you can set addressing_style in your AWS config profile,
but I could not seem to get that to propagate through to the config in the rust client.

Adding a force_path_style boolean parameter on the AWS S3 config spec appears to fix the problem.
I have a PR ready that I'll submit momentarily.

Before:

Image Image

After:

data_scope["documents"] = flow_builder.add_source(
    cocoindex.sources.AmazonS3(
        bucket_name="mybucket",
        included_patterns=["*.pdf"],
        binary=True,
        redis=RedisNotification(
            redis_url="redis://localhost:6379", redis_channel="bucketevents",
        ),
        force_path_style=True,
    ),
    refresh_interval=timedelta(seconds=60),
)
Image

Now the flow ingests no problem:

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions