Editing CSV file with Bangle.js v2

[espruino-discuss2]

Posted at 2024-10-23 by ruvi

I'm working on a project that involves data logging in a CSV, but where there should be the option of editing/deleting the last line of the CSV. I'm not sure how to go about this with a Bangle.

I was thinking maybe something like loading the whole file into a string, deleting the last line, then re-saving it. But I'm worried about the size of the string being larger than the watch can handle if the file gets very long.

var file = require("Storage").open("positions.csv","r");

// Read whole file with very long length parameter
csv = file.read(100000);

// Convert to array of lines
csvNew = csv.split('\n');

// Remove last line
csvNew.pop();

// Join back into single string
csvNew = csvNew.join("\n");

var file = require("Storage").open("positions.csv", "w");

file.write(csvNew);

Is there a better way to do this?

Edit: Oh, maybe I can somehow access just the last chunk with the file\5 notation? I don't know if that's a thing.

It doesn't necessarily have to be in CSV format.

[espruino-discuss2]

Posted at 2024-10-24 by ruvi

Edit: Removed code that doesn't work.

All I've figured out so far is appending a line, and getting the last line:

const STORAGE_FILE = "datalog.csv";

const CSVLogger = {
  addLog: (col1, col2, col3) => {
    const line = `${col1},${col2},${col3}\n`;
    const file = require("Storage").open(STORAGE_FILE, "a");
    
    file.write(line);
  },

  getLastLog: () => {
    var file = require("Storage").open(STORAGE_FILE, "r");
    
    while (true) {
      buffer = file.readLine();
      if (!buffer) {
        break;
      }
      string = buffer;
    }
    
  return string ? string.trim().split(",") : null;
    
  }
}

With Storage I think I can edit the last line, but not append. But with StorageFile, I don't see how I can edit or delete. And my function for getting the last line seems very inefficient for large files.

Edit: Thinking more about my use case, I don't think it'll ever have to be longer than 50kb, so maybe I can just manipulate everything in RAM.

---

Posted at 2024-10-24 by @gfwilliams

You can't really edit a value in the file without re-saving it. Flash memory starts at 1 and can only be made zero, so all you could do with just Storage is to zero out an area, which would probably make life hard later on.

I think your best bet is to use StorageFile, but if you want to delete a line you just iterate, writing just the stuff you want into a new file:

// delete last line
var src = require("Storage").open(STORAGE_FILE_A, "r");
var dst = require("Storage").open(STORAGE_FILE_B, "w");
var line = src.readLine();
var newLine = src.readLine();
while (line && newLine) { 
  dst.write(line+"\n");
  line = newLine;
  newLine = src.readLine();
}

Since you're loading line by line memory won't be an issue, and for 50k max it shouldn't take too long - it just means you have to have your file name changing (since you can't rename).

Or depending how you're working with this, you can just add a line saying DELETE LAST LINE
and then when you finally read the log out you use that to ignore the previous line if you see it?

---

Posted at 2024-10-24 by @halemmerich

How about having the last line in its own"staging" file?
All lines but one are in a StorageFile, last line is in a normal file. When editing or replacing the last line, just replace the normal file. On adding a line append the already stored line from the staging file to the StorageFile and write the new line to the staging file.
That should save both on CPU cycles and flash writes.

---

Posted at 2024-10-24 by @gfwilliams

That should save both on CPU cycles and flash writes.

Not if you're constantly writing that staging file?

But if you're not leaving the app then just keeping the last value in a variable and being sure to write it in an E.on("kill",...) handler would do it?

---

Posted at 2024-10-24 by @halemmerich

Compared to complete rewrites for every corrected line it will save on both. Compared to having a "delete previous line"-marker probably not, depending on data line length.
Having a variable for the last line is surely best if changing apps does not happen that often while logging.

---

Posted at 2024-10-24 by ruvi

Thank you @gfwilliams and @halemmerich for your help.

I actually really like the staging file idea. It seems to work well:

FULL_DATA = 'full_log.csv';
STAGING_FILE = 'staging_file.csv';

const CSVLogger = {
  
  addLog: (col1, col2, col3) => {
    const line = `${col1},${col2},${col3}\n`;
    
    // Add staged string to full data file
    var stagedString = require("Storage").read(STAGING_FILE);
    if (stagedString) {
      fullData = require("Storage").open(FULL_DATA, "a");
      fullData.write(stagedString);
    }
    
    // Replace the staged file with the new line
    require("Storage").erase(STAGING_FILE);
    require("Storage").write(STAGING_FILE, line);      
  },

  getLastLog: () => {
    var stagedString = require("Storage").read(STAGING_FILE);
    return stagedString ? stagedString.trim().split(",") : null;
  },
  
  deleteLastLog: () => {
    require("Storage").erase(STAGING_FILE);
  },
  
  editLastLog: (col1, col2, col3) => {
    const line = `${col1},${col2},${col3}\n`;
  
    require("Storage").erase(STAGING_FILE);
    require("Storage").write(STAGING_FILE, line);      
  },
  
  // Save staged line to full file when ready to export data
  commitLastLog: () => {
    var stagedString = require("Storage").read(STAGING_FILE);
    
    if (stagedString) {
      // Add old string to full data file
      fullData = require("Storage").open(FULL_DATA, "a");
      fullData.write(stagedString);
    }
    
    require("Storage").erase(STAGING_FILE);

  },
  
  // For debugging
  showBothFiles: () => {
    
    var fullData = require("Storage").open(FULL_DATA, "r");
    fullString = fullData.read(1000);
    print("Full File:\n" + fullString);
    
    var stagedString = require("Storage").read(STAGING_FILE);
    print("Staged File:\n" + stagedString + "\n");

  }
    
};


CSVLogger.addLog('a', 'b', 'c');

CSVLogger.editLastLog('1', '2', '3');

CSVLogger.deleteLastLog();

I'd only be logging a few dozen times per day, so I don't think flash writes should be a big deal.

It's probably not necessary for my project, but it would be cool to have a more general solution for deleting multiple last lines in sequence, which might require something more like what Gordon was saying with rewriting the whole file, or appending the DELETE_LAST_LINE multiple times.