Create a short SparkJob for use in the test framework

Requested by : Dejan Pejchev

We should look into using a custom SparkJob which lasts shorter for most of the tests, we don't a Spark job which runs for 2-3 minutes for all tests currently.
If we wrote a Spark job which runs for 15-30s, the e2e test time would fall to below 8 minutes.

https://gr-oss.slack.com/archives/C085QNB6WQY/p1765380915220639
