Pipe websocket message to file

[espruino-discuss3]

Posted at 2017-11-18 by thebells

I have a websocket handler that writes a message to a file with the function below. However, with a large message, saving the file can take a long time. Is there a way to pipe the the websocket message directly to the file, or to speed up how long it takes to receive and save it?

var fs = require('fs');

function wsHandler(ws) {
    ws.on('message', function wsData(msg) { 
     
      try {
        let data = (JSON.parse(msg));
        /*data would be this:

             data = {'command':'saveFile', 
                     'fileName': 'test.txt', 
                     'fileData':'This is a test of the file data, 
                                 but imagine it is 4KB of information.'
                     }
        */
        try {
          switch(data.command) {

            case 'saveFile':
              fs.writeFileSync(data.fileName, data.fileData);
              break;
            
            
           }
        }
        catch(err){} 
       
      }
      catch(err){}
    });

[espruino-discuss3]

Posted at 2017-11-20 by @gfwilliams

You can't pipe one big message in I'm afraid - not least because you'd have to be able to parse the stream of JSON.

You could however send a series of smaller messages? Eg:

{'command':'writeFile', 
  'fileName': 'test.txt', 
  'fileData':' ... '}
{'command':'appendFile', 
  'fileName': 'test.txt', 
  'fileData':' ... '}
{'command':'appendFile', 
  'fileName': 'test.txt', 
  'fileData':' ... '}
{'command':'appendFile', 
  'fileName': 'test.txt', 
  'fileData':' ... '}

It might help to have an acknowledgement sent back as well, so you can throttle sends.

Chunking everything into 512 bytes of data would work well since that's the allocation unit size.

You could also speed things up a bit by sending raw binary data (or even base64 in a string) rather than JSON (which has to be parsed).

Or, you could probably make things work a bit more smoothly doing something like this:

    ws.on('message', function wsData(msg) { 
        let data = (JSON.parse(msg));
        setTimeout(function() {
         // save file here
        }, 1);

as it wouldn't then block the event loop for quite as long.

---

Posted at 2017-11-20 by Wilberforce

Are you tied to using web sockets?

I have had success using the http PUT verb. The Ondata appends to the filename, and the URL of the put becomes the filename of the file you want to write too.

---

Posted at 2017-11-21 by @gfwilliams

^^ as wilberforce says, you can pipe from normal HTTP requests really easily. That is almost certainly the best solution.