Merge remote-tracking branch 'origin/main' into aj/feat/mini-cat-test-suites

Author: aaronsteers

.github/workflows/pypi_publish.yml

@@ -14,7 +14,12 @@ on:
   workflow_dispatch:
     inputs:
       version:
-        description: "Note that this workflow is intended for prereleases. For public-facing stable releases, please use the GitHub Releases workflow instead: https://github.com/airbytehq/airbyte-python-cdk/blob/main/docs/RELEASES.md. If running this workflow from main or from a dev branch, please enter the desired version number here, for instance 1.2.3dev0 or 1.2.3rc1."
+        description: >
+          Note that this workflow is intended for prereleases. For public-facing stable releases,
+          please use the GitHub Releases workflow instead:
+          https://github.com/airbytehq/airbyte-python-cdk/blob/main/docs/RELEASES.md.
+          For prereleases, please leave the version blank to use the detected version. Alternatively,
+          you can override the dynamic versioning for special use cases.
         required: false
       publish_to_pypi:
         description: "Publish to PyPI. If true, the workflow will publish to PyPI."
@@ -60,7 +65,7 @@ jobs:
           echo "Setting detected version to '$DETECTED_VERSION'"
           echo "DETECTED_VERSION=${DETECTED_VERSION}" >> $GITHUB_ENV
 
-      - name: Validate and set VERSION (git-ref='${{ github.ref_name }}', detected='${{ env.DETECTED_VERSION }}', input='${{ github.event.inputs.version || 'none' }}')
+      - name: Validate and set VERSION (detected='${{ env.DETECTED_VERSION }}', input='${{ github.event.inputs.version || 'none' }}')
         id: set_version
         run: |
           INPUT_VERSION=${{ github.event.inputs.version }}

airbyte_cdk/sources/declarative/interpolation/macros.py

@@ -177,6 +177,8 @@ def format_datetime(
         dt_datetime = (
             datetime.datetime.strptime(dt, input_format) if input_format else str_to_datetime(dt)
         )
+    if dt_datetime.tzinfo is None:
+        dt_datetime = dt_datetime.replace(tzinfo=pytz.utc)
     return DatetimeParser().format(dt=dt_datetime, format=format)
 
 

airbyte_cdk/sources/declarative/schema/default_schema_loader.py

@@ -37,7 +37,7 @@ def get_json_schema(self) -> Mapping[str, Any]:
 
         try:
             return self.default_loader.get_json_schema()
-        except OSError:
+        except (OSError, ValueError):
             # A slight hack since we don't directly have the stream name. However, when building the default filepath we assume the
             # runtime options stores stream name 'name' so we'll do the same here
             stream_name = self._parameters.get("name", "")

docs/RELEASES.md

@@ -21,11 +21,13 @@ _Note:_
 
 This process is slightly different from the above, since we don't necessarily want public release notes to be published for internal testing releases. The same underlying workflow will be run, but we'll kick it off directly:
 
-1. Navigate to the "Packaging and Publishing" workflow in GitHub Actions.
-2. Type the version number - including a valid pre-release suffix. Examples: `1.2.3dev0`, `1.2.3rc1`, `1.2.3b0`, etc.
-3. Select `main` or your dev branch from the "Use workflow from" dropdown.
-4. Select your options and click "Run workflow".
-5. Monitor the workflow to ensure the process has succeeded.
+1. Navigate to the "Publish CDK" workflow in GitHub Actions.
+2. Select your dev branch (or `main`) from the "Use workflow from" dropdown.
+3. Leave the version number blank, allowing the CI workflow to pick a version number
+   using [Dunamai](https://dunamai.readthedocs.io).
+4. Select from the other options and click "Run workflow".
+5. Monitor the workflow to ensure the process has succeeded. You will see the
+   version number in the GitHub Actions job output and in GitHub Environments view.
 
 ## Understanding and Debugging Builder and SDM Releases
 
@@ -59,15 +61,15 @@ To manually test changes against a dev image of SDM before committing to a relea
 
 Once the publish pipeline has completed, choose a connector to test. Set the base_image in the connector's metadata to your pre-release version in Dockerhub (make sure to update the SHA as well).
 Next, build the pre-release image locally using `airbyte-ci connectors —name=<source> build`.
-You can now run connector interfaces against the built image using the pattern
`docker run airbyte/<source-name>:dev <spec/check/discover/read>`.
+You can now run connector interfaces against the built image using the pattern`docker run airbyte/<source-name>:dev <spec/check/discover/read>`.
 The connector's README should include a list of these commands, which can be copy/pasted and run from the connector's directory for quick testing against a local config.
 You can also run `airbyte-ci connectors —name=<source> test` to run the CI test suite against the dev image.
 
 #### Pretesting Low-Code Python connectors
 
 Once the publish pipeline has completed, set the version of `airbyte-cdk` in the connector's pyproject.toml file to the pre-release version in PyPI.
-Update the lockfile and run connector interfaces via poetry:
`poetry run source-<name> spec/check/discover/read`.
-You can also run `airbyte-ci connectors —name=<source> test` to run the CI test suite against the dev image.


+Update the lockfile and run connector interfaces via poetry:`poetry run source-<name> spec/check/discover/read`.
+You can also run `airbyte-ci connectors —name=<source> test` to run the CI test suite against the dev image.
 
 #### Pretesting in Cloud
 

unit_tests/sources/declarative/interpolation/test_macros.py

@@ -101,6 +101,18 @@ def test_macros_export(test_name, fn_name, found_in_macros):
             "%ms",
             "2022-01-01T01:01:01Z",
         ),
+        (
+            "2022-01-01T01:01:01+0100",
+            "%Y-%m-%dT%H:%M:%S.%f%z",
+            None,
+            "2022-01-01T00:01:01.000000+0000",
+        ),
+        (
+            "2022-01-01T01:01:01",
+            "%Y-%m-%dT%H:%M:%S.%f%z",
+            None,
+            "2022-01-01T01:01:01.000000+0000",
+        ),
     ],
     ids=[
         "test_datetime_string_to_date",
@@ -117,6 +129,8 @@ def test_macros_export(test_name, fn_name, found_in_macros):
         "test_timestamp_to_format_string",
         "test_timestamp_epoch_microseconds_to_format_string",
         "test_timestamp_ms_to_format_string",
+        "test_datetime_with_timezone",
+        "test_datetime_without_timezone_then_utc_is_inferred",
     ],
 )
 def test_format_datetime(input_value, format, input_format, expected_output):