Problem integrating Tiny ERP customer data to PostgreSQL using Airbyte Custom Connector (UI)

[lorenzobrasil]

Hello!

I'm trying to create a job using Airbyte custom connector (using UI) to fetch contacts data from Tiny (Olist's ERP) and load into PostgreSQL..

What I want to do is simple but tricky, since the Tiny's API seems not to have any cursor field (like updated_at or something) to support incremental refreshs. I want to build a stream that fetch all contacts and then iterate each id on the contact details endpoint using a second child stream, loading this result in a Postgres destination table

I'm assuming that I don't have any workaround on the incremental sync setup part (since the api doesn't have any cursor field). Given that, all the connector setup seems to be correct (url, query parameters, parental stream) because i've tested the results using Postman and I get the expected data. When it turns to execute Airbyte data stream, it works but doesn't retrieve all the expected data (i.e. it covers all or near all the date range of the request, but not all expected registers), just a small part of it (some days are just skipped or poorly filled). I don't have many ideas of why this is happening (in part because I've spent many days figuring out and testing different configurations), but my strongest hypothesis atm is the api's rate limit causing some data loss, since our quota is very short (about 20 reqs/min).

It drives me to the part of rate limit configuration, which i'm not sure if it's working or not:
When I append the following api_budget snippet on yml manifest top level, it disappears right after I press "publish" button and I really think the snippet is deleted or "ignored", because there aren't any mention for api budget pauses on executions log files, just mentions to backoffs (which means the API is receiving more than 20 reqs per minute and makes me even more convinced that my api_budget isn't actually being applied.

api_budget:
  type: "HTTPAPIBudget"
  status_codes_for_ratelimit_hit: [429, 420]
  policies:
    - type: "FixedWindowCallRatePolicy"
      period: "PT1M"
      call_limit: 20
      matchers: []

Anyone can help me on how to solve this issue?