flask request.stream.read can't recive data when use socketio to run

[xudaqian1]

flask run

this is my code

from flask import Flask, request

app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'

@app.route("/api/v1/data", methods=["POST"])
def hello():
    file_path = "./aaa.xlsx"
    chunk_size = 1024
    print(request.headers.get('content-length'))
    with open(file_path, "bw") as f:
        while True:
            chunk = request.stream.read(chunk_size)
            print('-----',len(chunk))
            if len(chunk) == 0:
                return 'finish'
            f.write(chunk)

if __name__ == '__main__':
   app.run(host='0.0.0.0',port=5001)

this is my result

(venv) (base) xudaqian@xudaqian-System-Product-Name:~/bigDir/python-demo/chunk-recive$ python flask-run.py 
 * Serving Flask app "flask-run" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.16.10.58:5001/ (Press CTRL+C to quit)
Host: 172.16.10.58:5001
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Token: 123456qweasd
Accept-Encoding: gzip


----- 1024
----- 1024
----- 1024
----- 1024
----- 1024
----- 1024
----- 1024
----- 1024
----- 524
----- 0
172.16.10.58 - - [22/Nov/2021 18:42:59] "POST /api/v1/data?filename=user.xlsx HTTP/1.1" 200 -

socketio run

from flask import Flask, request
from flask_socketio import SocketIO

app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
socketio = SocketIO(app)

@app.route("/api/v1/data", methods=["POST"])
def hello():
    file_path = "./aaa.xlsx"
    chunk_size = 1024
    print(request.headers)
    with open(file_path, "bw") as f:
        while True:
            chunk = request.stream.read(chunk_size)
            print('-----',len(chunk))
            if len(chunk) == 0:
                return 'finish'
            f.write(chunk)


if __name__ == '__main__':
    socketio.run(app,host='0.0.0.0',port=5001,debug=True)

this is my result

(venv) (base) xudaqian@xudaqian-System-Product-Name:~/bigDir/python-demo/chunk-recive$ python socketio-run.py 
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 829-884-760
(21550) wsgi starting up on http://0.0.0.0:5001
(21550) accepted ('172.16.10.58', 42560)
Content-Type: text/plain
Host: 172.16.10.58:5001
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Token: 123456qweasd
Accept-Encoding: gzip


----- 0
172.16.10.58 - - [22/Nov/2021 18:44:30] "POST /api/v1/data?filename=user.xlsx HTTP/1.1" 200 121 0.000784

how can i resolve it?
help me please

[miguelgrinberg]

This is not related to Socket.IO, you are using a different web server under Socket.IO so the behavior can be different. Do you know why your client isn't sending the Content-Length header, for example?

[xudaqian1]

Because my client can only get the chunked stream and then forward it to my api, it can’t get the complete file object. It can only be transmitted in chunks by streaming, that is, set the request header to Transfer-Encoding: chunked
The client code is as follows:

1. go

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"os"
)

type tempReader struct {
	tempReader *os.File
}

func newTempReader(fullpath string) (tr *tempReader, err error) {
	tr = new(tempReader)
	tr.tempReader, err = os.Open(fullpath)
	if err != nil {
		log.Fatal(err)
		return
	}
	return
}
func (tr *tempReader) Read(p []byte) (n int, err error) {
	n, err = tr.tempReader.Read(p)
	fmt.Println("read len() cat", len(p), n, err)
	return
}

func (tr *tempReader) Close() error {
	fmt.Println("close !!")
	return tr.tempReader.Close()
}

func main() {
	f, err := newTempReader("/home/xudaqian/Desktop/user.xlsx")
	if err != nil {
		log.Fatal(err)
		return
	}
	request, err := http.NewRequest("POST", "http://172.16.10.58:5001/api/v1/data?filename=user.xlsx", f)
	if err != nil {
		log.Fatal(err)
		return
	}
	fmt.Println("new request success")
	request.Header.Set("token", "123456qweasd")
	resp, err := http.DefaultClient.Do(request)
	if err != nil {
		log.Fatal(err)
		return
	}
	fmt.Println("do request success")
	defer resp.Body.Close()
	respData, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		log.Fatal(err)
		return
	}
	fmt.Println(resp.StatusCode)
	fmt.Println(string(respData))

}

2.python

def read_in_chunks(file_object, blocksize=1024):
    while True:
        data = file_object.read(blocksize)
        if not data:
            break
        yield data


import requests
data = {'token':'123456qweasd','file_md5':'xxxx','file_id':'xxxxx'}
storage_url = "http://172.16.10.58:5001/api/v1/data?filename=mydata.zip&case_number=2klafsao&[email protected]&type=mysql" 


with open('/home/xudaqian/Desktop/mydata.zip', 'rb') as f:
    res = requests.post(storage_url, headers=data, data=read_in_chunks(f))
    print(res.text)

[miguelgrinberg]

Okay. So as I said before, this is an issue between your client and the web server that you are using, Socket.IO does not participate in this. I assume you are using eventlet here? Maybe try a different web server (Gunicorn, gevent) to see if they handle this better. If nyo you will have to raise an issue with the web server project and see if you can get help from the maintainers.

[xudaqian1]

That's right. I use eventlet mode to run socketio . It's good when use threading or gevent mode to run it.
Thank you very much