Correct syntax to control file and folder naming

[ShahBinoy]

As per #111 (comment) I appreciate that you added the change, but I am mis-configuring my setup.

What will the correct setting be if I want the folder based path with auto-generated .parquet files ?

Currently I have

{
  "source": "ACADIA",
  "target": "S3_LAKE_ZONE_A",
  "defaults": {
    "mode": "incremental",
    "object": "path/data_landing/{source_name}/{stream_table}/date={YYYY}-{MM}-{DD}/*.parquet",
    "update_key": "updated_on",
    "target_options": {
      "file_max_rows": 100000
    }
  },
  "streams": {
    "acadia.container_item": {
      "update_key": "id",
      "disabled": false
    }
  },
  "env": {
    "SLING_LOADED_AT_COLUMN": true,
    "SLING_STREAM_URL_COLUMN": true,
    "path_prefix": "${FRUGAL_DATA_LANDING_PREFIX}",
    "SLING_SHOW_PROGRESS": "false",
    "SLING_DISABLE_TELEMETRY": "false",
    "SLING_HOME_DIR": "/User/path/sling/../../sling",
    "SLING_LOGGING": "NO_COLOR",
    "SLING_ALLOW_EMPTY_TABLES": "true"
  }

But it is still creating file with *.parquet in its prefix name
s3://my-bucket/path/data_landing/acadia/container_item/date=2024-01-12/*.parquet/part.01.0001.parquet

[flarco]

🤔 weird, you're using latest version, yes?

[ShahBinoy]

Just FYI,

"source": "ACADIA",
  "target": "S3_LAKE_ZONE_A",
  "defaults": {
    "mode": "incremental",
    "object": "dev-binoy/elt/data_landing/{source_name}/{stream_table}/date={YYYY}-{MM}-{DD}/*",
    "update_key": "updated_on",
    "target_options": {
      "file_extension": ".parquet"
    }
  },
  "streams": {

Above generates * based file path and also creates csv files

[flarco]

yes, you should just provide format: parquet, not file_extension. file_extension is more of a private field, not really for config.

[ShahBinoy]

Gotcha,
yeah works with

"defaults": {
    "mode": "incremental",
    "object": "dev-binoy/elt/data_landing/{source_name}/{stream_table}/date={YYYY}-{MM}-{DD}/*",
    "update_key": "updated_on",
    "target_options": {
      "format": "parquet",
    }
  }

[flarco]

So:

• /* requires format: parquet
• /*.parquet doesn't require format (infers from path)