Skip to content

Commit 66aa6b5

Browse files
echonesisechonesis
authored
HDDS-14305. [Website v2] [Docs] [User Guide] HttpFS (#204)
1 parent c3e4017 commit 66aa6b5

File tree

4 files changed

+196
-2
lines changed

4 files changed

+196
-2
lines changed

‎cspell.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,7 @@ words:
135135
- Boto3
136136
- fsspec
137137
- libhdfs
138+
- WebHDFS
138139
# Misc words
139140
- acking
140141
- dashboarding

‎docs/04-user-guide/01-client-interfaces/05-httpfs.md‎

Lines changed: 165 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,166 @@
1-
# HttpFS
1+
---
2+
sidebar_label: HttpFS Gateway
3+
---
24

3-
**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section.
5+
# HttpFS Gateway
6+
7+
Ozone HttpFS can be used to integrate Ozone with other tools via REST API.
8+
9+
## Introduction
10+
11+
HttpFS is a service that provides a REST HTTP gateway supporting File System operations (read and write).
12+
It is interoperable with the **WebHDFS** REST HTTP API.
13+
14+
HttpFS can be used to access data on an Ozone cluster behind of a firewall.
15+
For example, the HttpFS service acts as a gateway and is the only system that is allowed to cross the firewall into the cluster.
16+
17+
HttpFS can be used to access data in Ozone using HTTP utilities (such as curl and wget) and HTTP libraries Perl from other languages than Java.
18+
19+
The **WebHDFS** client FileSystem implementation can be used to access HttpFS using the Ozone filesystem command line tool (`ozone fs`) as well as from Java applications using the Hadoop FileSystem Java API.
20+
21+
## Getting started
22+
23+
To try it out, follow the [instructions](../../02-quick-start/01-installation/01-docker.md) to start the Ozone cluster with Docker Compose.
24+
25+
```bash
26+
docker compose up -d --scale datanode=3
27+
```
28+
29+
You can/should find now the HttpFS gateway in Docker with the name like `ozone_httpfs`,
30+
and it can be accessed through `localhost:14000`.
31+
HttpFS HTTP web-service API calls are HTTP REST calls that map to an Ozone file system operation.
32+
33+
Here's some example usage:
34+
35+
### Create a volume
36+
37+
```bash
38+
# creates a volume called `volume1`.
39+
curl -i -X PUT "http://localhost:14000/webhdfs/v1/volume1?op=MKDIRS&user.name=hdfs"
40+
```
41+
42+
Example Output:
43+
44+
```bash
45+
HTTP/1.1 200 OK
46+
Date: Sat, 18 Oct 2025 07:51:21 GMT
47+
Cache-Control: no-cache
48+
Expires: Sat, 18 Oct 2025 07:51:21 GMT
49+
Pragma: no-cache
50+
Content-Type: application/json
51+
X-Content-Type-Options: nosniff
52+
X-XSS-Protection: 1; mode=block
53+
Set-Cookie: hadoop.auth="u=hdfs&p=hdfs&t=simple-dt&e=1760809881100&s=OCdVOi8eyMguFySkmEJxm5EkRfj6NbAM9agi5Gue1Iw="; Path=/; HttpOnly
54+
Content-Length: 17
55+
56+
{"boolean":true}
57+
```
58+
59+
### Create a bucket
60+
61+
```bash
62+
# creates a bucket called `bucket1`.
63+
curl -i -X PUT "http://localhost:14000/webhdfs/v1/volume1/bucket1?op=MKDIRS&user.name=hdfs"
64+
```
65+
66+
Example Output:
67+
68+
```bash
69+
HTTP/1.1 200 OK
70+
Date: Sat, 18 Oct 2025 07:52:06 GMT
71+
Cache-Control: no-cache
72+
Expires: Sat, 18 Oct 2025 07:52:06 GMT
73+
Pragma: no-cache
74+
Content-Type: application/json
75+
X-Content-Type-Options: nosniff
76+
X-XSS-Protection: 1; mode=block
77+
Set-Cookie: hadoop.auth="u=hdfs&p=hdfs&t=simple-dt&e=1760809926682&s=yvOaeaRCVJZ+z+nZQ/rM/Y01pzEmS9Pe2mE9f0b+TWw="; Path=/; HttpOnly
78+
Content-Length: 17
79+
80+
{"boolean":true}
81+
```
82+
83+
### Upload a file
84+
85+
```bash
86+
echo "hello" >> ./README.txt
87+
curl -i -X PUT "http://localhost:14000/webhdfs/v1/volume1/bucket1/user/foo/README.txt?op=CREATE&data=true&user.name=hdfs" -T ./README.txt -H "Content-Type: application/octet-stream"
88+
```
89+
90+
Example Output:
91+
92+
```bash
93+
HTTP/1.1 100 Continue
94+
95+
HTTP/1.1 201 Created
96+
Date: Sat, 18 Oct 2025 08:33:33 GMT
97+
Cache-Control: no-cache
98+
Expires: Sat, 18 Oct 2025 08:33:33 GMT
99+
Pragma: no-cache
100+
X-Content-Type-Options: nosniff
101+
X-XSS-Protection: 1; mode=block
102+
Set-Cookie: hadoop.auth="u=hdfs&p=hdfs&t=simple-dt&e=1760812413286&s=09t7xKu/p/fjCJiQNL3bvW/Q7mTw28IbeNqDGlslZ6w="; Path=/; HttpOnly
103+
Location: http://localhost:14000/webhdfs/v1/volume1/bucket1/user/foo/README.txt
104+
Content-Type: application/json
105+
Content-Length: 84
106+
107+
{"Location":"http://localhost:14000/webhdfs/v1/volume1/bucket1/user/foo/README.txt"}
108+
```
109+
110+
### Read the file content
111+
112+
```bash
113+
# returns the content of the key `/user/foo/README.txt`.
114+
curl 'http://localhost:14000/webhdfs/v1/volume1/bucket1/user/foo/README.txt?op=OPEN&user.name=foo'
115+
hello
116+
```
117+
118+
## Supported operations
119+
120+
Here are the tables of WebHDFS REST APIs and their state of support in Ozone.
121+
122+
### File and Directory Operations
123+
124+
| Operation | Support |
125+
|-------------------------------- | --------------------------- |
126+
| Create and Write to a File | supported |
127+
| Append to a File | not implemented in Ozone |
128+
| Concat File(s) | not implemented in Ozone |
129+
| Open and Read a File | supported |
130+
| Make a Directory | supported |
131+
| Create a Symbolic Link | not implemented in Ozone |
132+
| Rename a File/Directory | supported (with limitations) |
133+
| Delete a File/Directory | supported |
134+
| Truncate a File | not implemented in Ozone. |
135+
| Status of a File/Directory | supported |
136+
| List a Directory | supported |
137+
| List a File | supported |
138+
| Iteratively List a Directory | unsupported |
139+
140+
### Other File System Operations
141+
142+
| Operation | Support |
143+
| ------------------------------------- | --------------------------------------- |
144+
| Get Content Summary of a Directory | supported |
145+
| Get Quota Usage of a Directory | supported |
146+
| Set Quota | not implemented in Ozone FileSystem API |
147+
| Set Quota By Storage Type | not implemented in Ozone |
148+
| Get File Checksum | unsupported (to be fixed) |
149+
| Get Home Directory | unsupported (to be fixed) |
150+
| Get Trash Root | unsupported |
151+
| Set Permission | not implemented in Ozone FileSystem API |
152+
| Set Owner | not implemented in Ozone FileSystem API |
153+
| Set Replication Factor | not implemented in Ozone FileSystem API |
154+
| Set Access or Modification Time | not implemented in Ozone FileSystem API |
155+
| Modify ACL Entries | not implemented in Ozone FileSystem API |
156+
| Remove ACL Entries | not implemented in Ozone FileSystem API |
157+
| Remove Default ACL | not implemented in Ozone FileSystem API |
158+
| Remove ACL | not implemented in Ozone FileSystem API |
159+
| Set ACL | not implemented in Ozone FileSystem API |
160+
| Get ACL Status | not implemented in Ozone FileSystem API |
161+
| Check access | not implemented in Ozone FileSystem API |
162+
163+
## Hadoop user and developer documentation about HttpFS
164+
165+
- [HttpFS Server Setup](https://hadoop.apache.org/docs/stable/hadoop-hdfs-httpfs/ServerSetup.html)
166+
- [Using HTTP Tools](https://hadoop.apache.org/docs/stable/hadoop-hdfs-httpfs/ServerSetup.html)

‎docs/07-system-internals/01-components/07-httpfs-gateway/01-overview.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
sidebar_label: Overview
3+
---
4+
5+
# HttpFS Gateway Overview
6+
7+
## Architecture
8+
9+
Ozone HttpFS is forked from the HDFS HttpFS endpoint implementation ([HDDS-5448](https://issues.apache.org/jira/browse/HDDS-5448)).
10+
Ozone HttpFS is intended to be added optionally as a role in an Ozone cluster, similar to [S3 Gateway](https://ozone.apache.org/docs/edge/design/s3gateway.html).
11+
12+
Technically, HttpFS is a Jetty-based web application.
13+
It serves as a gateway that translates REST API requests into Hadoop FileSystem API calls to interact with the cluster.
14+
As a separate service, it requires independent startup alongside the core Ozone components.
15+
16+
## Security
17+
18+
HttpFS supports multiple security protocols, including Hadoop pseudo-authentication, Kerberos SPNEGO, and pluggable authentication modules.
19+
It also includes support for Hadoop proxy users.

‎docs/07-system-internals/01-components/07-httpfs-gateway/README.mdx‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
sidebar_label: HttpFS Gateway
3+
---
4+
5+
# HttpFS Gateway Internals
6+
7+
import DocCardList from '@theme/DocCardList';
8+
9+
This section documents the internal workings of the HttpFS Gateway.
10+
11+
<DocCardList/>

0 commit comments

Comments
 (0)