The quest for dynamic media json files

[mhschmall]

Ive been giving this some thought and Im sure Jstudner has been working hard at it too. So I thought Id whiteboard up some suggestions to get the conversation going:

On the server side, modify the generate json procedure to create individual json files for each media type:
Related task: modify html files to only use its own json file.
Related issue: The infinity large json data space. The esp32 only has so much memory and has been noted in issues,
there is a finite limit to how many files can be read at once.

Possible solutions:

1. tell the users they can only have so many files (every portable media player Ive ever used has specified a max number of items )
2. Use an external json file creator for initial json file creation (see music index discussion).
3. modify server side json creation to chunk data in and out.
4. A combination of solutions
5. TBD.

Once individual json files have been created, the only json file that's going to be dynamic is gallery.json since gallery is the only page that allows user uploads (at the moment). Sadly, gallery.html can not modify the json file on its own, but if the xhr.onload procedure in that page is changed to make another post call after upload, the server can then update the gallery.json file.

xhr.onload = () => {
if (xhr.status === 200) {
console.log('[Upload] Complete:', file.name);

// Update gallery.json on the server
fetch('/update-gallery-json', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    name: file.name.replace(/\.[^/.]+$/, ""), // remove extension
    file: `Gallery/${file.name}`
  })
}).then(res => {
  if (!res.ok) throw new Error('Failed to update gallery.json');
  console.log('[Gallery JSON] Updated with', file.name);
}).catch(err => {
  console.error(err);
});

} else {
console.error('[Upload] Failed:', file.name, xhr.responseText);
}
resolve();
};

And here is how it could sorta look like on the server side (in a generic way):

server.on("/update-gallery-json", HTTP_POST, handleUpdateGalleryJson);

void handleUpdateGalleryJson() {
if (server.method() != HTTP_POST) {
server.send(405, "text/plain", "Method Not Allowed");
return;
}

// Read the incoming JSON body
String body = server.arg("plain");
if (body.length() == 0) {
server.send(400, "text/plain", "Missing body");
return;
}

// Parse new item from body
StaticJsonDocument<256> newItem;
DeserializationError err = deserializeJson(newItem, body);
if (err) {
server.send(400, "text/plain", "Invalid JSON");
return;
}

// Open existing gallery.json
File file = SPIFFS.open("/gallery.json", "r");
StaticJsonDocument<2048> jsonDoc;
if (file) {
DeserializationError e = deserializeJson(jsonDoc, file);
file.close();
if (e) {
server.send(500, "text/plain", "Failed to parse existing JSON");
return;
}
} else {
// If no file exists yet, create a base structure
jsonDoc["gallery"] = JsonArray();
}

// Append new item
JsonArray gallery = jsonDoc["gallery"].as();
gallery.add(newItem);

// Save back to file
file = SPIFFS.open("/gallery.json", "w");
if (!file) {
server.send(500, "text/plain", "Failed to open file for writing");
return;
}
serializeJsonPretty(jsonDoc, file);
file.close();

server.send(200, "text/plain", "OK");
}

[Jstudner]

I think the play is going to be having individul files, but also a bit further than that. right now media.json combines images and sorts the items. In the new version I think it would be better to have it just make a list of all files on the system. This will reduce how much work its actualy doing aswell as save space even if to a minimal extent. Then it can be put on the frontend code to handle what images go where and what files to display. This system would also make it much easier to add other filetype support and modification in the future.

In additon the error I am having with the admin panel is due to how JSquery works. Its basicly generating a media.json list for every directory you open. It works suprisingly well, but if it fills up the memory the device crashes. I tried fixing that by loading only 50 items at a time, but even that will still crash since it has to crawl the file system to get past the first batch of files. The solution is to pre-generate a file list, but thats just media.json. I think the best solution is to have the new mini media.json files serve as that index, which should work (though it would need a fall back for any user created dir that isnt defined which could still crash).

I will keep you posted on my progress here, let me know if you have any more ideas on the matter. My ultimate goal here is to completly remove the need to mess with media.json. If it could be fully automated on upload delete rename etc the user experiance would be much better.

[Jstudner]

Ok, it's going to be a day or two till I upload the code but I got it working. It's an absolute nightmare in the backend, but so far it seems stable. Will need to check if it effects the streams at all but it should run in the background and dynamically to avoid interfering. The new version index's everything live and makes it so media.json no longer needs to be generated, everything is handled fast and automatic. I only have movies working on the new system so far, just wanted to update now that I know it can work.

[mhschmall]

That's just awesome, I look forward to the update.

[Jstudner]

Ight so new update, tried using SQlite, turns out that's kinda hard. I switched to using ndjson files, which makes it possible to edit on the fly, but it absolutely ruins the frontend loading time atm since the images are separate. (It's also buggy as all hell)

I think the solution is going to be making the whole system ndjson (so every file is its own item) but still attaching the images and some data in the backend like before. Then the file browser can edit the per item data and I won't need to regenerate huge json files at all. It would also mean I can scan the SD and make or remove items individually without needing to rebuild the whole index.

From my current understanding the tradeoff is I can't load all of the media at once to the frontend, items load one by one. But as long as I can get that fast I'll make it animated and have them pop on screen or something and call it a feature.

I'll keep you posted, this update is taking forever, but I really want to use the best system since I am going through the trouble of rebuilding it 😅

[mhschmall]

I say ditch the thumbnails and simplify. If I have hundreds of ebooks, or hundreds of songs, Im not going to bother with thumbnail images, especially if my stuff is named descriptively. The only instance where thumbnails make sense (IMHO) is for the gallery where they can be generated on the fly via JS.

[MMagTech]

Currently nearing 100 movies on my Nomad. Adding the thumbnails has been the biggest pain of the project. If they’re not added though there’s no way of knowing what a movie is after initial refresh

One of two options would make it more ideal

1. Have the option of no images and just file names
2. A combo of both whether it be the actual cover or the default and the file name below

Preferably I think option one would be better

[Jstudner]

Actually just addressed this! The last version of movies and shows HTML show the filename while loading, that way you can see. In the new version of the frontend I will support no images as well though, shouldn't be a problem to add that and I can certainly see the appeal!