Spark 3.4 upgrade

[kshitij-microsoft]

Description

Checklist

• I have read the contribution guidelines.
  • General
  • SDK
  • CLI
• I have coordinated with the docs team (mldocs@microsoft.com) if this PR deletes files or changes any file names or file extensions.
• Pull request includes test coverage for the included changes.
• This notebook or file is added to the CODEOWNERS file, pointing to the author or the author's team.

[jayesh-tanna]

Could you consider using a global variable or constant to manage version upgrades? I have noticed that the version was recently updated from 3.2 to 3.3, and now it is being upgraded to 3.4. Implementing a global variable or constant for the version number could simplify future modifications by reducing the number of files that need to be updated.

[kshitij-microsoft]

@Azure/ai-platform-docs can you please approve?

[Copilot]

Pull Request Overview

This PR upgrades the Spark runtime version from 3.3.x to 3.4.x across SDK examples, pipelines, feature store samples, and CLI templates.

• Bumps runtime_version in Jupyter notebooks and YAML configurations from 3.3 to 3.4
• Removes the deprecated 3.3 version from documentation tables
• Clears notebook execution_count to nil for fresh runs

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
sdk/python/jobs/spark/submit_spark_standalone_jobs_managed_vnet.ipynb	Bumped Spark runtime to 3.4.0
sdk/python/jobs/spark/submit_spark_standalone_jobs.ipynb	Bumped Spark runtime to 3.4.0
sdk/python/jobs/spark/submit_spark_pipeline_jobs.ipynb	Bumped Spark runtime to 3.4.0
sdk/python/jobs/spark/automation/run_interactive_session_notebook.ipynb	Bumped Spark runtime to 3.4.0
sdk/python/jobs/pipelines/1k_demand_forecast_pipeline/aml-demand-forecast-mm-pipeline.ipynb	Updated supported Spark versions to only include 3.4.0
sdk/python/jobs/pipelines/1i_pipeline_with_spark_nodes/pipeline_with_spark_nodes.ipynb	Bumped Spark runtime to 3.4.0
sdk/python/featurestore_sample/project/fraud_model/pipelines/training_pipeline.yaml	Bumped Spark runtime to 3.4
sdk/python/featurestore_sample/project/fraud_model/pipelines/batch_inference_pipeline.yaml	Bumped Spark runtime to 3.4
sdk/python/featurestore_sample/automation-test/test_featurestore_vnet_samples.ipynb	Bumped Spark runtime to 3.4.0
sdk/python/featurestore_sample/automation-test/test_featurestore_sdk_samples.ipynb	Bumped Spark runtime to 3.4.0
sdk/python/featurestore_sample/automation-test/test_featurestore_cli_samples.ipynb	Bumped Spark runtime to 3.4.0
cli/monitoring/out-of-box-monitoring.yaml	Bumped Spark runtime to 3.4
cli/monitoring/model-monitoring-with-collected-data.yaml	Bumped Spark runtime to 3.4
cli/monitoring/generation-safety-quality-monitoring.yaml	Bumped Spark runtime to 3.4
cli/monitoring/custom-monitoring.yaml	Bumped Spark runtime to 3.4
cli/monitoring/azureml-e2e-model-monitoring/notebooks/model-monitoring-e2e.ipynb	Cleared execution counts and bumped runtime to 3.4
cli/monitoring/advanced-model-monitoring.yaml	Bumped Spark runtime to 3.4
cli/jobs/spark/serverless-spark-standalone-user-identity.yml	Bumped Spark runtime to 3.4
cli/jobs/spark/serverless-spark-standalone-managed-identity.yml	Bumped Spark runtime to 3.4
cli/jobs/spark/serverless-spark-standalone-default-identity.yml	Bumped Spark runtime to 3.4
cli/jobs/spark/serverless-spark-pipeline-user-identity.yml	Bumped Spark runtime to 3.4
cli/jobs/spark/serverless-spark-pipeline-managed-identity.yml	Bumped Spark runtime to 3.4
cli/jobs/spark/serverless-spark-pipeline-default-identity.yml	Bumped Spark runtime to 3.4
cli/jobs/pipelines/add-column-and-word-count-using-spark/pipeline.yml	Bumped Spark runtime to 3.4.0
cli/jobs/pipelines-with-components/shakespear_sample_and_word_count_using_spark/pipeline.yml	Bumped Spark runtime to 3.4.0

Comments suppressed due to low confidence (3)

cli/jobs/pipelines/add-column-and-word-count-using-spark/pipeline.yml:39

• [nitpick] Consider quoting the runtime_version value (e.g., "3.4.0") to match the string format used in other YAML files and avoid potential parsing inconsistencies.

      runtime_version: 3.4.0

sdk/python/featurestore_sample/project/fraud_model/pipelines/training_pipeline.yaml:31

• [nitpick] The version is specified as "3.4" here but 3.4.0 is used elsewhere; standardizing the version string across all configurations improves consistency.

      runtime_version: "3.4"

cli/monitoring/azureml-e2e-model-monitoring/notebooks/model-monitoring-e2e.ipynb:762

• [nitpick] Instead of only nulling execution_count, clear all outputs in the notebook before committing to keep the file clean and minimize merge conflicts.

   "execution_count": null,

[sdgilley]

These changes are approved for docs - they won't break our documentation. However, why are all the jobs failing?

[kshitij-microsoft]

The builds are failing due to the login failure. Since the PR is raised through my forked repo, the az login using service principal fails