Inactionable Failure During API Connector Builder Test

[mike-prittie]

Hi all,
I'm trying to use Airbyte's new Connector Builder to load data from an API into my target destination (using the Community Edition).

I quickly set up three Steams, each targeting three different API endpoints of the same service, and it worked like a charm! The 1st was a relatively basic request with only a few GET params and a small response body, the 2nd required managing pagination, and the 3rdrequired using Incremental Sync to window over date ranges.

The 4th and last endpoint was another simple endpoint (only one optional GET param) but with a larger response object than the others (~20k records in a single response). When I run the Stream test in the Connector Builder, it gives the following error:

ERROR The CDK command `test_read` completed properly but no records nor trace were found. Logs were: 0.

Pulling from this endpoint works perfectly fine when using a simple Python script I wrote before trying to transition to Airbyte for this ingestion.

After scratching my head for a while and coming up empty-handed, searching around and reading the docs, I tried using the one GET param available to restrict the size of the response object. This single param limits the lookback window (i.e., what date-time to grab records since). After setting this to a small window of only a few days... boom... it works, and I get a successful response. I then experimented with increasing this window size until it hit a threshold, and it began throwing the same error again.

docker logs -f airbyte-connector-builder-server doesn't provide much help either, as no other errors are logged there.

Again, I have no such issues when I run the simple Python script I wrote to pull the data. I'm reasonably confident that the issue originates in Airbyte or its docker containers (deployed using docker compose on t2.large EC2 as recommended).

I'm left wondering if this is the result of:

• Airbyte imposed response row count limit (Does the Connector Builder have a 10k row count limit on response objects?).
• Airbyte imposed response size limits
• Airbyte imposed response timeout
• Resource limits (Airbyte, Docker container, etc.)

... but I have yet to find an answer.

I've familiarized myself with the scaling docs, but I can't imagine that I'm running into issues here as the Python script I've used successfully stores the response JSON to file in <50MB.

I know the Connector Builder is in beta, but I would love to hear thoughts on what's causing this error and how to resolve it.

[mike-prittie]

This open issue may be what's causing the problem I'm running into.