feat(destination-smoke-test): add smoke test destination connector#72441
feat(destination-smoke-test): add smoke test destination connector#72441Aaron ("AJ") Steers (aaronsteers) wants to merge 7 commits intomasterfrom
Conversation
…ector This registers a new destination connector that uses PyAirbyte's cache system to write data to various backends (DuckDB, Postgres, Snowflake, BigQuery, MotherDuck). The connector implementation lives in the PyAirbyte repository and is imported as a dependency. This connector entry provides the metadata and thin wrapper needed to run it as an Airbyte destination. Co-Authored-By: AJ Steers <aj@airbyte.io>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Co-Authored-By: AJ Steers <aj@airbyte.io>
|
PyAirbyte (airbyte package) already includes airbyte-cdk as a dependency, so specifying it explicitly causes a version conflict during poetry install. Co-Authored-By: AJ Steers <aj@airbyte.io>
Co-Authored-By: AJ Steers <aj@airbyte.io>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| # Copyright (c) 2024 Airbyte, Inc., all rights reserved. | ||
| """Universal destination using PyAirbyte caches as backends.""" | ||
|
|
||
| from airbyte.destinations.universal import DestinationPyAirbyteUniversal |
There was a problem hiding this comment.
🔴 Inconsistent import paths for DestinationPyAirbyteUniversal between __init__.py and run.py
The __init__.py imports DestinationPyAirbyteUniversal from airbyte.destinations.universal, while run.py (the actual entry point) imports it from airbyte.cli.universal_connector. These are two completely different module paths within the airbyte package.
Root Cause and Impact
In __init__.py (line 4):
from airbyte.destinations.universal import DestinationPyAirbyteUniversalIn run.py (line 6):
from airbyte.cli.universal_connector import DestinationPyAirbyteUniversalAt most one of these import paths can be correct. If airbyte.cli.universal_connector is the correct module (as used in the entry point run.py), then importing the package (destination_pyairbyte_universal) will fail with an ImportError because airbyte.destinations.universal doesn't exist (or vice versa).
The convention in other destination connectors (e.g., destination-duckdb) is for run.py to import from the package's own __init__.py rather than directly from the external library. Here, run.py bypasses __init__.py entirely and uses a different import path, so the two files are not even consistent about where the class lives.
Impact: Either importing the package or running the connector entry point will fail with an ImportError at runtime, depending on which path is actually valid in the airbyte library.
Prompt for agents
In file airbyte-integrations/connectors/destination-pyairbyte-universal/destination_pyairbyte_universal/__init__.py line 4 and file airbyte-integrations/connectors/destination-pyairbyte-universal/destination_pyairbyte_universal/run.py line 6: The import paths for DestinationPyAirbyteUniversal are inconsistent. __init__.py uses 'airbyte.destinations.universal' while run.py uses 'airbyte.cli.universal_connector'. Determine which module path is correct in the airbyte (PyAirbyte) package and update both files to use the same correct path. Additionally, following the convention of other destination connectors (like destination-duckdb), run.py should import from the package's own __init__.py (i.e., 'from destination_pyairbyte_universal import DestinationPyAirbyteUniversal') rather than directly from the external library.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in commit 8dc1da3. The import path in __init__.py has been updated from airbyte.destinations.universal to airbyte.cli.universal_connector to match run.py. Both files now use the correct path that matches the actual location in the PyAirbyte package after the directory rename.
…te.cli.universal_connector Co-Authored-By: AJ Steers <aj@airbyte.io>
…-test Co-Authored-By: AJ Steers <aj@airbyte.io>
Co-Authored-By: AJ Steers <aj@airbyte.io>
devin-ai-integration
bot
changed the title
|
2 participants
What
Registers a new
destination-smoke-testconnector that uses PyAirbyte's cache system to write data to various backends (DuckDB, PostgreSQL, Snowflake, BigQuery, MotherDuck). This connector is designed for testing and validation purposes.This is part 2 of 2 - the implementation lives in PyAirbyte: airbytehq/PyAirbyte#969
How
This PR adds a thin wrapper connector that imports the
DestinationPyAirbyteUniversalclass from the PyAirbyte library. The connector:airbyte >= 0.20.0as a dependency to get the destination implementationpython-connector-base:4.0.2Docker imageReview guide
metadata.yaml- Connector registration with definitionId, resource requirements, and registry settingspyproject.toml- Dependencies includingairbyte >= 0.20.0which provides the implementationdestination_smoke_test/run.py- Entry point that imports from PyAirbyteKey items to verify:
airbyte.cli.universal_connector.DestinationPyAirbyteUniversalwill exist after PyAirbyte PR Update connector READMEs to reflect new build plugin commands #969 mergesairbyte >= 0.20.0version constraint is appropriateicon.svgbut metadata referencesairbyte.svg- verify this is correctUser Impact
Users will be able to use a single destination connector to write to multiple backend types (DuckDB, Postgres, Snowflake, BigQuery, MotherDuck) by selecting the
destination_typein configuration.Note: This connector depends on PyAirbyte PR #969 being merged first. CI will fail until the PyAirbyte changes are published.
Can this PR be safely reverted and rolled back?
Requested by: Aaron ("AJ") Steers (@aaronsteers) (AJ)
Devin session: https://app.devin.ai/sessions/46cd19c4793a4e1ea32bb660de9c1fe0