feat(file-based): Add logging for observable checkpoints during check and discovery (do not merge)

[devin-ai-integration]

Add logging to file-based module for observable checkpoints

This PR adds logging to the file-based module in the Python CDK to provide observable checkpoints during check and discovery operations. This will help users diagnose failures that otherwise feel like the connector hanging indefinitely.

Changes

• Added INFO level logging at the beginning and end of key operations
• Added DEBUG level logging for detailed progress tracking
• Added logging to track file discovery, schema inference, and cursor operations
• Improved visibility into concurrent file processing

Requested by

Aaron

Link to Devin run: https://app.devin.ai/sessions/080d4edb69294014ab96ffa7659986cf

[devin-ai-integration]

🤖 Devin AI Engineer

Original prompt from Aaron:

@Devin - The file-based sources hang for a long time during "check", sometimes failing. In one PR, add logging to the file CDK (aka the file_based module in the python cdk) such that observable checkpoints during check and/or discovery are logged. This should help users diagnose failures that otherwise feel like the connector hanging indefinitely.

PR #2 is to bump the SFTP source to the CDK branch you created in PR #1. Let me know if you need help with this.

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[devin-ai-integration]

Closing due to inactivity for more than 7 days.