Skip to content
This repository was archived by the owner on Sep 3, 2025. It is now read-only.

Commit 2805fd1

Browse files
sadoyansadoyan
committed
Implemented seek and play for media types
1 parent 8cbab50 commit 2805fd1

File tree

2 files changed

+210
-23
lines changed

2 files changed

+210
-23
lines changed

‎README.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
**GO-WebRados**
2+
---------
3+
4+
**Go-WebRados** is a simple and high performance HTTP service for **CEPH** distributed file system.
5+
The main goal of this tool is to provide simple HTTP interface for **Ceph's** bare **RADOS** layer.
6+
**Go-WebRados** is not a replacement for **RadosGW** as is does not have all reach APIs and features of RadosGW(S3, Swift, etc ...), instead it stands for simplicity and performance.
7+
8+
GO-WebRados relies on HTTP methods to interact with RADOS object, thus it can provide access to objects store in RADOS directly from internet browser .
9+
The ide is to have web accessible storage for millions of relatively small files, which can be accessed from browser directly.
10+
11+
GO-WebRados relies on C bindings of **Ceph** so in order to run this program you need to install Ceph packages.
12+
Running Ceph services on computer which hosts GO-WebRados is not required, it's even better to have a dedicated server or server for running GO-WebRados
13+
14+
### **Download and install**
15+
---------
16+
17+
You can build GO-WebRados from source or download precompiled binaries. If you already have installed Cephs packages and want to make things easy ,
18+
just download te GO-WebRados binary, make it executeable, and you are ready to run .
19+
20+
Building from a source is also easy.
21+
22+
```shell
23+
git clone https://github.com/sadoyan/go-webrados.git
24+
cd go-webrados
25+
export GOROOT=/path/to/your/go
26+
./build.sh
27+
28+
./webrados /path/to/config.ini
29+
```
30+
31+
32+
33+
### **Configuration**
34+
---------
35+
36+
Configuration paramaters are stored in ```config.ini``` file, which should be in running directory.
37+
Sample config file, with reasoneable defaulr ships with source code.
38+
39+
```ini
40+
[main]
41+
listen : 0.0.0.0:8080
42+
dispatchers : 20
43+
serveruser : admin
44+
serverpass : 261a5983599fd57a016122ec85599ec4
45+
uploadmaxpart : 52428800
46+
dangerzone : yes
47+
readonly : no
48+
authread : no
49+
authwrite : yes
50+
radoconns : 25
51+
logfile : no
52+
logpath: /opt/webrados.log
53+
allpools: no
54+
poollist: bublics, donuts, images
55+
database : redis
56+
57+
[redis]
58+
server: 127.0.0.1:6379
59+
username : none
60+
password : none
61+
database : 0
62+
63+
[mysql]
64+
username : ceph
65+
password : ceph
66+
dbname : ceph
67+
server : 127.0.0.1:3306
68+
69+
[monitoring]
70+
enabled : true
71+
url: 127.0.0.1:9090
72+
user: admin
73+
pass: admin
74+
```
75+
### **API**
76+
---------
77+
78+
| **Name** | **Description** |
79+
| ------------- | ------------- |
80+
|**Read File**|HTTP ```GET:``` http://{BINDADDRESS}/{POOLNAME}/{FILENAME}|
81+
|**Read File**|HTTP ```GET:``` http://{BINDADDRESS}/{POOLNAME}/{FILENAME}|
82+
|**Upload File**|HTTP ```POST, PUT:``` http://{BINDADDRESS}/{POOLNAME}/{FILENAME}|
83+
|**Remove File**|HTTP ```DELETE:``` http://{BINDADDRESS}/{POOLNAME}/{FILENAME}|
84+
85+
Configuration file is pretty simple and intuitive.
86+
87+
### **Section main**
88+
---------
89+
90+
| **Name** | **Description** |
91+
| ------------- | ------------- |
92+
|**listen**|IP port to bind.|
93+
|**dispatchers**|Number of threads for webserver.|
94+
|**serveruser**|Static user.|
95+
|**serverpass**|MD5 hash of password for static user. It can be the output of `echo -n SecretPaSs | md5sum |awk '{print $1}'`|
96+
|**dangerzone**|Enable destructive methods and commands (DELETE).|
97+
|**readonly**|Enable readonly mode. If 'yes' only GET is allowed.|
98+
|**authread**|Require authentication for GET only.|
99+
|**authwrite**|Require authentication for POST/PUT/DELETE.|
100+
|**radoconns**|Number of connection to CEPH.|
101+
|**logfile**|Log to file, if 'no' logs are sent to stdout.|
102+
|**logpath**|Path for log file.|
103+
|**uploadmaxpart**| Maximum file chunk size (Sbould be amaller or erqual ro `osd max object size`).|
104+
|**allpools:**|yes/no . If yes program will scan ceph and enable access via web to all pool.|
105+
|**poollist:**|Works only if **allpools** is set to **no**. Coma separated list of pools which should be accesible via webrados program.|
106+
|**redis**|IP address and port of Redis server.|
107+
### **Section monitoring**
108+
---------
109+
| **Name** | **Description** |
110+
| ------------- | ------------- |
111+
|**enabled**|Enable/Disable monitoring.|
112+
|**url**|IP address and port for minitoring interface.|
113+
|**user**|Monitoring user.|
114+
|**pass**|Password for monitoring user.|
115+
116+
### **users.txt file**
117+
118+
GO-Webrados can dynamically update users from ```users.txt``` file .
119+
```users.txt``` should contain user and md5hash of password divided by space in each line.
120+
`echo -n SecretPaSs | md5sum |awk '{print $1}'` on Linux systemd will output md5hash for using it as password in `users.txt` file
121+
GO-Webrados will periodically read ```uesrs.txt``` file and automatically update users in memory.
122+
123+
### **Large files**
124+
125+
In order to be able to store large file in RADOS directly files needs to be split to smaller chunks.
126+
GO-WebRados will automatically set maximum chunk size to **uploadmaxpart** and split files in accordance to that.
127+
For this GO-WebRados requires Redis servers to store metadata for big files. (Make sure to keep Redis database safe and periodically back it up.)
128+
Redis will store metadata only for large files. Small files are to split and not require special metadata server.
129+
130+
### **Special commands**
131+
132+
**HTTP GET** http://{BINDADDRESS}/{POOLNAME}/{FILENAME}?info
133+
Return information about requested file in json format.
134+
135+
```curl -s http://ceph1:8080/bublics/katana.mp4?info | python -mjson.tool```
136+
137+
```json
138+
{
139+
"name": "katana.mp4",
140+
"pool": "bublics",
141+
"segments": "11",
142+
"size": "471861144"
143+
}
144+
```

‎web/methods.go‎

Lines changed: 66 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,14 @@ import (
1717
"github.com/ceph/go-ceph/rados"
1818
)
1919

20-
var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
21-
22-
func randSeq(n int) string {
23-
b := make([]rune, n)
24-
for i := range b {
25-
b[i] = letters[rand.Intn(len(letters))]
26-
}
27-
return string(b)
28-
}
20+
//var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
21+
//func randSeq(n int) string {
22+
// b := make([]rune, n)
23+
// for i := range b {
24+
// b[i] = letters[rand.Intn(len(letters))]
25+
// }
26+
// return string(b)
27+
//}
2928

3029
func respCodewriter(f error, w http.ResponseWriter, r *http.Request) string {
3130
if strings.Split(f.Error(), ",")[1] == " No such file or directory" {
@@ -65,6 +64,7 @@ func readFile(w http.ResponseWriter, r *http.Request, name string, pool string,
6564
break
6665
}
6766
_, er := w.Write(bytesOut)
67+
6868
if er != nil {
6969
if !strings.HasPrefix(er.Error(), "write tcp") {
7070
wrados.Writelog(er)
@@ -160,23 +160,57 @@ func Get(w http.ResponseWriter, r *http.Request) {
160160
w.Header().Set("Content-Length", strconv.FormatUint(uint64(xo.Size), 10))
161161
readFile(w, r, name, pool, xo, of)
162162
case true:
163-
164163
var fsize uint64
165-
fileparts := strings.Split(filename, ",")
164+
var fileparts []string
165+
xo, _ = ioctx.Stat(name)
166+
if xo.Size > 0 {
167+
_, ko := r.Header["Range"]
168+
switch ko {
169+
case true:
170+
ranges := strings.FieldsFunc(r.Header.Get("Range"), Split)
171+
172+
if len(ranges) >= 2 {
173+
minrange, _ = strconv.Atoi(ranges[1])
174+
contentlenght = int(xo.Size) - minrange
175+
} else {
176+
contentlenght = int(xo.Size)
177+
}
178+
179+
contentlenght = int(xo.Size) - minrange
180+
of = uint64(minrange)
181+
182+
w.Header().Set("Content-Length", strconv.Itoa(contentlenght))
183+
w.Header().Set("Accept-Ranges", "bytes")
184+
w.Header().Set("Last-Modified", xo.ModTime.String())
185+
w.Header().Set("Content-Range", "bytes "+strconv.Itoa(minrange)+"-"+strconv.FormatUint(uint64(xo.Size-1), 10)+"/"+strconv.FormatUint(xo.Size, 10))
186+
w.Header().Set("Content-Type", mime)
187+
w.WriteHeader(http.StatusPartialContent)
188+
189+
readFile(w, r, name, pool, xo, of)
190+
break
191+
case false:
192+
w.Header().Set("Content-Length", strconv.FormatUint(uint64(xo.Size), 10))
193+
readFile(w, r, name, pool, xo, of)
194+
break
195+
}
196+
break
197+
}
198+
199+
fileparts = strings.Split(filename, ",")
166200
fileparts = fileparts[:len(fileparts)-1]
167201
for filepart := range fileparts {
168202
name = fileparts[filepart]
169203
xo, _ = ioctx.Stat(name)
170204
fsize = fsize + xo.Size
171205
}
172206

173-
//fmt.Println("========== ==========")
207+
//fmt.Println("============== Req ==============")
174208
//for nnn, values := range r.Header {
175209
// for _, value := range values {
176210
// fmt.Println(nnn, value)
177211
// }
178212
//}
179-
//fmt.Println("========== ==========")
213+
//fmt.Println("=================================")
180214

181215
_, ko := r.Header["Range"]
182216
switch ko {
@@ -223,19 +257,28 @@ func Get(w http.ResponseWriter, r *http.Request) {
223257
w.Header().Set("Content-Type", mime)
224258
w.WriteHeader(http.StatusPartialContent)
225259

226-
for filepart := range fileparts {
227-
name = fileparts[filepart]
260+
if minrange >= sizes[len(sizes)-1] {
228261
xo, _ = ioctx.Stat(name)
229-
if filepart == 0 {
230-
of = uint64(minrange - before)
231-
} else {
232-
of = 0
233-
}
234-
x := readFile(w, r, name, pool, xo, of)
235-
if x == false {
236-
break
262+
of = xo.Size - uint64(contentlenght)
263+
_ = readFile(w, r, name, pool, xo, of)
264+
} else {
265+
for filepart := range fileparts {
266+
name = fileparts[filepart]
267+
xo, _ = ioctx.Stat(name)
268+
if filepart == 0 {
269+
of = uint64(minrange - before)
270+
271+
} else {
272+
of = 0
273+
}
274+
x := readFile(w, r, name, pool, xo, of)
275+
if x == false {
276+
break
277+
}
237278
}
279+
238280
}
281+
239282
case false:
240283
w.Header().Set("Content-Length", strconv.FormatUint(fsize, 10))
241284
for filepart := range fileparts {

0 commit comments

Comments
 (0)