docs(cloud/databricks): replace unavailable Project Glow image with image-free setup

[ramGoli]

What & why

The Databricks page currently instructs users to run Hail via Databricks Container Services using the image projectglow/databricks-hail:<version>. That image is no longer usable: the newest tag is 0.2.93 (pushed June 2022) and Docker Hub has purged the layers, so provisioning a cluster with it fails at launch (Failed to set up docker container due to a docker image pull failure), and the manifest request now returns HTTP 401. The Dockerfiles linked at github.com/projectglow/glow/tree/master/docker are also no longer at that path. So the documented method is currently a dead end.

This rewrites cloud/databricks.rst to a self-contained, image-free setup that installs current Hail at cluster start (init script + cluster config), so there is no third-party image to rot. It also documents two requirements the previous page omitted that are necessary on modern runtimes:

• JDK 17 — Hail's jar is Java 11 bytecode; recent Databricks Runtimes default to JDK 8, so the cluster must be set to JDK 17 (JNAME=zulu17-ca-amd64 on DBR 15.4 LTS).
• Do not install Hail into base Python — pip install hail pulls numpy 2.x and breaks the runtime's pre-built pyarrow, which prevents the notebook Python kernel from starting (it surfaces as "Could not reach driver"). The init script installs the jar via pip --target (base Python untouched) and the notebook installs the package via %pip.

The hl.init(sc=...), initialization, and Bokeh sections are preserved. Databricks Container Services is kept only as an optional note for sites that require a locked-down image (which they must then build and maintain themselves).

Validation

Validated on Databricks Runtime 15.4 LTS (Apache Spark 3.5.0, Scala 2.12), Hail 0.2.138, AWS:

• hl.init(sc=...) on the distributed Spark backend, single-node and multi-node (a linear_regression_rows GWAS ran distributed across multiple worker hosts; variant_qc, sample_qc, and large range_table jobs all succeeded).
• Reading the public gnomAD v2.1.1 exomes Hail Table directly from s3://gnomad-public-us-east-1/... (opened the 9,997-partition table and read allele frequencies) on a Dedicated/Unity Catalog cluster.
• Writing Hail outputs to /dbfs.

Notes for reviewers

• JNAME=zulu17-ca-amd64 is the JDK 17 selector verified on DBR 15.4 LTS; maintainers may want to confirm the appropriate value for other runtime versions.
• The GOOGLE_APPLICATION_CREDENTIALS dummy-credentials step is included as a safe default for Hail's startup GCS initialization on AWS/Azure.

Drafted with Claude Code.

[Copilot]

Pull request overview

Updates the Databricks documentation to remove reliance on the now-unavailable Project Glow Docker image and instead describe an init-script-based installation that puts the Hail JAR on the Spark classpath and installs the Python package in the notebook.

Changes:

• Replaces the container-based instructions with an init script + cluster configuration (including Kryo settings and JDK selection).
• Adds guidance on avoiding base-Python installs and notes on reading/writing data paths in Databricks/Unity Catalog.
• Keeps existing Databricks-specific guidance for hl.init(sc=...) and Bokeh display, and demotes Container Services to an optional alternative.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ramGoli]

hi @ehigham ! I validated with another customer who were able use the updated instructions to run Hail on Databricks. Please let me know if there is anything else you need from my end to merge this PR

[ramGoli]

hi @ehigham any update on merging the doc?