Skip to content

How to do a stream-based/chunked upload? #9

@erdbeerschnitzel

Description

@erdbeerschnitzel

Maybe I'm missing something in the documention (which is offline at the moment), but I can't find out how to properly construct a stream-based or a chunked upload.

I thought something like this should work:

        String host = "hostname";
        int CHUNK_SIZE = 1024*1024;
        HttpRequest httpRequest = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/test");

        httpRequest.setHeader(HttpHeaders.Names.HOST, host + ":1344");
        httpRequest.setHeader(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE);

        HttpResponse httpResponse = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);

        httpResponse.setHeader(HttpHeaders.Names.HOST, host + ":1344");
        httpResponse.setHeader(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE);
        httpResponse.setHeader(HttpHeaders.Names.CACHE_CONTROL, HttpHeaders.Values.NO_CACHE);

        FileInputStream fis = new FileInputStream(new File("/home/files.rar"));
        channel.write(request);

        while (fis.available() > 0) {
            buffer.clear();

            if(fis.available() < CHUNK_SIZE){
                System.out.println("write " + fis.available());
                buffer.writeBytes(fis, fis.available());
            }
            else {
                buffer.writeBytes(fis, CHUNK_SIZE);
            }
            System.out.println("creating new chunk");
            IcapChunk chunk = new DefaultIcapChunk(buffer);
            channel.write(chunk);            
        }
        DefaultIcapChunkTrailer trailer = new DefaultIcapChunkTrailer();
        channel.write(trailer);

What I try to do is creating the ICAP request with HTTP request and response but I do not set the content of the response. Instead I try to send the response content in chunks of 1MB size (to keep memory usage low). A DefaultIcapChunkTrailer is sent at the end. The "chunked" part is not the most important here, the main problem is that I want to send the data "on-the-fly" to the ICAP server as it reaches my application. I can't set the complete file as content on the HTTP response because this would use too much memory.

The code posted above doesn't work, because the netty-icap is setting "null-body: xxx" in the "Encapsulated-Header". I see that it makes sense to set "null-body" instead of "res-body" if there is no content set in the response, but I don't see how it should be used in combination with chunks. Maybe I misunderstood something, but can you please give an example of how to do a stream-based upload.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions