fix for #48

[parisholley]

Description of change

see #48
the problem is that the mongo extractor is outputting a SCHEMA for every record, which may be a partial object. when this is consumed by the postgres loader, it fails because the schema keeps getting overwritten and it can't find the key_properties. i generate a catalog json with my own python script vs relying on this tap to generate it on the fly. i can't say for sure that this is the right solution (perhaps we should write the key_properties every time), but it is the root cause

Command

mongo meltano elt --job_id mongo --catalog transformed.json tap-mongodb target-postgres

Catalog

{
  "streams": [
    {
      "metadata": [
        {
          "breadcrumb": {},
          "metadata": {
            "database-name": "db",
            "replication-method": "FULL_TABLE",
            "selected": true,
            "tap-mongodb.projection": "{\"_id\": 1, \"name\": 1}"
          }
        }
      ],
      "schema": {
        "key_properties": [
          "_id"
        ],
        "properties": {
          "_id": {
            "type": "string"
          },
          "name": {
            "type": "string"
          }
        }
      }
    }
  ]
}

For those looking for a fix before this is merged, you can use this as your pip_url instead of the standard tap-mongodb package:

git+https://github.com/parisholley/tap-mongodb.git@5e5f95649d4705205b654a1b09a533f006e0126e

[cmerrick]

Hi @parisholley, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

[cmerrick]

You did it @parisholley!

Thank you for signing the Singer Contribution License Agreement.

[je-kr]

Hi,

This fix would be very useful, any idea on when it could be merged ?

Thank you !