S3 Source connector nested fields

[helanto]

Hello 👋

I am trying to create a source connector that uses JSONl files from S3. The JSONl have nested fields. I believe that the schema inference part does not infers schema that is deeply nested. The schema stays {"type":"object"}. There is also a relevant discussion in #23133.

When tried to specify the schema on the connector creation manually, I get back an error. For example when I specify:
{"username": "string", "friends": "array", "information": {"type":"object", "properties":{"nested":"string"}}}, I get back:

airbyte-worker-5558cf94b5-kknrs airbyte-worker-container 2023-09-28 12:44:44 ERROR i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - The provided schema could not be transformed into valid JSON Schema. Contact Support if you need assistance.
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container details=Invalid input schema; nested schemas are not supported.
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container Traceback (most recent call last):
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/airbyte/integration_code/main.py", line 42, in <module>
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     launch(source, _args)
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 200, in launch
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     for message in source_entrypoint.run(parsed_args):
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 107, in run
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.check(source_spec, config))
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in check
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     check_result = self.source.check(self.logger, config)
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 82, in check
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     check_succeeded, error = self.check_connection(logger, config)
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/file_based/file_based_source.py", line 61, in check_connection
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     streams = self.streams(config)
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/file_based/file_based_source.py", line 92, in streams
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     parsed_config = self.spec_class(**config)
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/main.py", line 1076, in pydantic.main.validate_model
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/fields.py", line 895, in pydantic.fields.ModelField.validate
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/fields.py", line 928, in pydantic.fields.ModelField._validate_sequence_like
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/fields.py", line 1094, in pydantic.fields.ModelField._validate_singleton
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/fields.py", line 884, in pydantic.fields.ModelField.validate
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/fields.py", line 1101, in pydantic.fields.ModelField._validate_singleton
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/fields.py", line 1157, in pydantic.fields.ModelField._apply_validators
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/class_validators.py", line 337, in pydantic.class_validators._generic_validator_basic.lambda13
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/main.py", line 711, in pydantic.main.BaseModel.validate
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/main.py", line 1076, in pydantic.main.validate_model
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/fields.py", line 867, in pydantic.fields.ModelField.validate
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/fields.py", line 1157, in pydantic.fields.ModelField._apply_validators
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "pydantic/class_validators.py", line 304, in pydantic.class_validators._generic_validator_cls.lambda4
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/file_based/config/file_based_stream_config.py", line 66, in validate_input_schema
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     if type_mapping_to_jsonschema(v):
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/file_based/schema_helpers.py", line 225, in type_mapping_to_jsonschema
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     json_mapping = _parse_json_input(input_schema) or {}
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/file_based/schema_helpers.py", line 202, in _parse_json_input
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container     raise ConfigValidationError(
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container airbyte_cdk.sources.file_based.exceptions.ConfigValidationError: The provided schema could not be transformed into valid JSON Schema. Contact Support if you need assistance.
airbyte-worker-5558cf94b5-kknrs airbyte-worker-container details=Invalid input schema; nested schemas are not supported.

The only way to workaround the issue is to update the schema in the connection details, using the update connection API. However this feels a bit hacky.

I wanted to ask what is the recommended approach to correctly specify nested schemas on S3 source connector ?
Do you consider existing behaviour as a bug or it's how it supposed to work ?

Deployment information

• Helm on K8s.
• S3 source connector version 4.0.4
• Airbyte version: 0.50.30