[source-salesforce] enable changing cursor column

[LucasSegersFabro]

Case

I have a case where I sync some objects (Account) but there is a batch process that updates one column (and the default "SystemModstamp")
Thing is: not all the columns that get updated need to be fetched again for my batch

---

Problem/Possible solution

On salesforce, I can simply configure another modstamp column that only gets updated if the set of columns I chose changes.
If I could use this custom column as a cursor, the syncs would pull MUCH less data and be really quick

BUT
The source-salesforce connector uses the SystemModstamp column as a "source defined crusor", making it impossible for me to configure another column as a cursor other than "SystemModStamp"

It would be nice if I could change the default cursor

---

Some questions

If I were to attempt to contribute myself, I have some questions before understanding/digging into the airbyte source interfaces:

1. any pointers for "the cursor defines a default/suggested cursor, but make it not mandatory"?
2. if I "hack" into the actor definition and remove the source defined schema (or maybe set the column I need myself), would things fall apart?

It seems that the source-salesforce connector tries to infer the possible cursors

[csalmeida]

We have the same issue, but we can only create a connection via the API with selectFields if the cursorField is SystemModstamp and it has to be present in selectFields.

If we attempt to use LastModifiedDate as the cursorField the request responds with 400 Bad Request.
It seems that LastModifiedDate is supported as a `cursorField at

airbyte/airbyte-integrations/connectors/source-salesforce/source_salesforce/api.py

Line 373 in dbf4834

def get_pk_and_replication_key(json_schema: Mapping[str, Any]) -> Tuple[Optional[str], Optional[str]]:

A valid request to /connections is similar to:

{
    "name": "import_job_id",
    "sourceId": "zf78b8e0-314f-4dea-bz14-fe1365z7b5z0",
    "destinationId": "1zd4df1z-bzeb-493a-a08e-d00755507db1",
    "dataResidency": "eu",
    "namespaceDefinition": "custom_format",
    "namespaceFormat": "1234_job_id",
    "prefix": "contacts_",
    "schedule": {
        "scheduleType": "manual" 
    },
    "configurations": {
        "streams": [
            {
                "name": "Contact",
                "syncMode": "incremental_append",
                // "primaryKey": [
                //     [
                //         "Id"
                //     ]
                // ],
                "cursorField": [
                    "SystemModstamp"
                ],
                "selectedFields": [
                    {
                        "fieldPath": [
                            "Id"
                        ]
                    },
                    {
                        "fieldPath": [
                            "Email"
                        ]
                    },
                    {
                        "fieldPath": [
                            "Fax"
                        ]
                    },
                    {
                        "fieldPath": [
                            "Phone"
                        ]
                    },
                    {
                        "fieldPath": [
                            "Title"
                        ]
                    },
                    {
                        "fieldPath": [
                            "FirstName"
                        ]
                    },
                    {
                        "fieldPath": [
                            "LastName"
                        ]
                    },
                    {
                        "fieldPath": [
                            "MobilePhone"
                        ]
                    },
                                        {
                        "fieldPath": [
                            "SystemModstamp"
                        ]
                    }
                ]
            }
        ]
    }
}

Attempting to make a request with LastModifiedDate returns a 400:

{
    "name": "import_job_id",
    "sourceId": "zf78b8e0-314f-4dea-bz14-fe1365z7b5e0",
    "destinationId": "1zd4df1z-bceb-493a-a08e-d00755507db1",
    "dataResidency": "eu",
    "namespaceDefinition": "custom_format",
    "namespaceFormat": "1234_job_id",
    "prefix": "contacts_",
    "schedule": {
        "scheduleType": "manual" 
    },
    "configurations": {
        "streams": [
            {
                "name": "Contact",
                "syncMode": "incremental_append",
                // "primaryKey": [
                //     [
                //         "Id"
                //     ]
                // ],
                "cursorField": [
                    "LastModifiedDate"
                ],
                "selectedFields": [
                    {
                        "fieldPath": [
                            "Id"
                        ]
                    },
                    {
                        "fieldPath": [
                            "Email"
                        ]
                    },
                    {
                        "fieldPath": [
                            "Fax"
                        ]
                    },
                    {
                        "fieldPath": [
                            "Phone"
                        ]
                    },
                    {
                        "fieldPath": [
                            "Title"
                        ]
                    },
                    {
                        "fieldPath": [
                            "FirstName"
                        ]
                    },
                    {
                        "fieldPath": [
                            "LastName"
                        ]
                    },
                    {
                        "fieldPath": [
                            "MobilePhone"
                        ]
                    },
                                        {
                        "fieldPath": [
                            "LastModifiedDate"
                        ]
                    }
                ]
            }
        ]
    }
}

The response body as follows:

{
    "status": 400,
    "type": "https://reference.airbyte.com/reference/errors#bad-request",
    "title": "bad-request",
    "detail": "The request could not be understood by the server due to malformed syntax.",
    "documentationUrl": null,
    "data": {
        "message": "Cursor field is not selected properly for stream: Contact. Please include the cursor field in selected fields for this stream."
    }
}