webhdfsmagic: Complementary package for remote HDFS operations alongside sparkmagic

[ab2dridi]

The Problem It Solves

When using sparkmagic to run Spark jobs remotely via Livy, users often struggle with basic HDFS operations:

• Listing directories
• Downloading/uploading files
• Reading file contents
• Managing remote data

Without direct SSH access to the cluster (common in production environments), these tasks become unnecessarily complex.

What is webhdfsmagic?

webhdfsmagic brings intuitive magic commands for HDFS operations directly into your Jupyter notebooks:

%hdfs ls /data/warehouse
%hdfs get /data/sample.parquet ./local/
%hdfs put ./config.json /apps/configs/
%hdfs cat /logs/application.log
%hdfs mkdir /data/new_project
%hdfs rm /temp/old_file.csv

Integration with sparkmagic
The best part? webhdfsmagic can inherit configuration from sparkmagic's config.json!

If you're already using sparkmagic, webhdfsmagic will automatically detect and use the same connection settings (Knox gateway URL, credentials, SSL certificates, etc.), making setup a breeze.

Both extensions work perfectly side by side in the same notebook:

Use sparkmagic for distributed Spark computations
Use webhdfsmagic for HDFS file management


## Installation

pip install webhdfsmagic
jupyter-webhdfsmagic # Auto-setup

## Links
PyPI: [https://pypi.org/project/webhdfsmagic/](https://super-duper-space-pancake-976r7p5vgqp39755.github.dev/)
GitHub: [https://github.com/ab2dridi/webhdfsmagic](https://super-duper-space-pancake-976r7p5vgqp39755.github.dev/)

## Feedback Welcome
I'd love to hear your thoughts and suggestions! If you're using sparkmagic and have needed HDFS file operations, give webhdfsmagic a try.

Happy to answer any questions or discuss potential integrations/improvements.

Cheers!