Add better logging functionality

[SebastianArcq]

When running Sling in production you would typically want to monitor if Sling runs as expected. As such it would be great to have better logging options.

Right now Sling(**config).run() does not return anything. it would be good if it returned, e.g., a JSON containing something like

• source / target / mode / envs
• Status = success / failed
• number of rows inserted
• time passed
• speed r/s

I know that I can use the env "SLING_LOGGING": "JSON" to format the output as JSON, but that's not very helpful as there are no identifiers for the data points mentioned above.

[flarco]

Hey @SebastianArcq, interesting. So you're looking for the inputted configuration and result. Could you share, how do you envision using this information?

[SebastianArcq]

When running any ELT in a production environment I want to be sure that everything is running as expected. When an error occurs I want to be able to react as soon as possible.

What I would want to do is process the return value, I would 1) send error messages straight to Slack, an 2) I would write the logs to a database table. This allows me to later look at rows inserted, etc. and trace errors and problems.

I am actually building a solution in Python myself now by rerouting the stdout to a variable, processing it using regex and writing the result to a csv file, which I then write to a database table.