parallel miniscope recordings: disconnections not apearing in Control Panel

[arnauramos1]

Hi all,
I’ve been experiencing a weird bug that could not find an explanation for anywhere in the forums:

I have a setup with 2-4 miniscopes running in parallel. During recordings, every now and then, one of the miniscopes will “restart” and go focal plane 0, 30fps (I image at 20fps and different focal plane; specified within the .json file), and the gyroscope stop working.
This error does not pop up in the control panel, and the three lights in the daq still appear green.
The only solution I found to save the data during recordings is to disconnect and reconnect the miniscope, and then manually adjust the focus and fps.
I have tried several things without success.. new cables, new daqs, disconnecting the pc from the network… My PC is quite potent, and the simultaneous recordings do not even make the cpu, gpu or ram go to 50%.

Any idea why this could be happening? Any insight would be highly appreciated!

Cheers,
Arnau

[sneakers-the-rat]

I diagnosed a problem that would show up like this a few months ago, but didn't open it since we are actively in the process of replacing this software (bear with us for the next few months as we make a swap :)

i'll repost it here in the morning, but for now:

• is there nothing printed in any of your console or logfiles at all? or if you ctrl+f can you find any that say "Frames will be lost" ?
• is it always the same camera, or is it random?
• is it always after ~some duration or can it happen near the start of the recording?
• are you saving to a spinning disk hard drive or an SSD?

[arnauramos1]

Hi,
Thanks for the prompt answer.

• I don't have anything printed.
• It is random.
• It can happen at any timepoint.
• I am saving to an SSD.

Thanks for the help!

[arnauramos1]

If instead of having 4 miniscopes at 20fps I have them at 30fps at the start (i.e. already specified at the json file), then I get this notification of "frames will be lost", even though the frame buffer does not appear full, and that notification always appears for the same miniscope (my device2).

[sneakers-the-rat]

Aha yeah, so there's youre getting to the problem that i thought it was, where the way the data saving loop is written has it loop through each camera, draining its queue before going on to the next camera:

Miniscope-DAQ-QT-Software/source/datasaver.cpp

Lines 184 to 186 in 0fecb97

	for (i = 0; i < frameBuffer.size(); i++) {
	while (usedCount[names[i]]->tryAcquire()) {
	// grad info from buffer in a threadsafe way

and the size of the memory buffer is pretty small (128 frames) -

Miniscope-DAQ-QT-Software/source/videodevice.h

Line 43 in 0fecb97

#define FRAME_BUFFER_SIZE 128

so that "frames will be lost" message starts happening when we are write limited.

If you're writing to an SSD and not really i/o limited, if you're using any kind of compression you could disable that and then compress afterwards, see if that helps the frame dropping problem - even if you aren't maxing out your CPU, that current DAQ doesn't have great concurrency so it could be at the limit of single-threaded video encoding.

For the original problem, i don't really have any good idea, and as far as i'm aware this software doesn't have any good way of printing more detailed debugging information, but the fact that it loses its configuration is odd. i'm don't have a clear picture of how the software would handle recognizing (and thus re-using the config) a device if it e.g. had a hardware problem and restarted/lost connection/etc., but if it can be any of your cameras at any time and you're not being bound by CPU that's mysterious - presumably not every one of your miniscopes has a hardware bug. maybe @daharoni or @fnsangiul have a better idea.

[fnsangiul]

Based on my experience recording with multiple imaging devices (Miniscopes and miniCAMs) the disconnection happens when the animal hits the Miniscope against something, disconnecting the Miniscope from the computer. Sometimes it reconnects automatically but when it doesn't link again with the computer there is no other way than manually disconnecting and connecting the coaxial cable to the DAQ. When it reconnects, it goes into the default settings.

[sneakers-the-rat]

aha. ok we'll test for that in mio (which fyi @arnauramos1 will replace this software), we'll figure out a way to poll & reset on disconnection and persist config

[fnsangiul]

@sneakers-the-rat I don't know if it's possible to do it via software because the Miniscope loses the link to the serializer. That's why you need to reset the DAQ, either by connecting the coax cable again or resetting it by unplugging the DAQ.

[sneakers-the-rat]

you can't trigger the DAQ to reset the serializer? how does device discovery work when starting from scratch with a config vs. reconnecting after connection loss that makes it impossible to preserve the config? seems like it should be possible to poll from software to the daq like "keep resetting until something reconnects"

[fnsangiul]

I'm referring to one step back. If the Miniscope loses the data link then I don't think you can reset it via software. You can check the deserializer datasheet for this though.

[sneakers-the-rat]

i guess another way of putting it is why does disconnecting and reconnecting the cable fix it, and why can't we trigger that same reset from code?

[fnsangiul]

I wouldn't say it fixes it, what happens is that the Miniscope locks again. Here is the datasheet for the deserializer.

[sneakers-the-rat]

for the sake of this issue - what can we do here? is there something we can improve on/fix so that the user experience doesn't include having to unplug and replug-in the cable to fix problems that we should happen to expect? (mice knocking the cameras around). I am not sure what you're showing me here with the datasheet, i'm not sure how it relates to the issue?

[fnsangiul]

@sneakers-the-rat I don't think there is another way other than unplugging and replugin the coax cable, or at least I haven't found one. As for the datasheet I linked it because if I had to isolate this particular issue it would be due to the deserializer.

[sneakers-the-rat]

So there is no way that we could, e.g. in the firmware of the DAQ, to do whatever state reset happens when one physically unplugs/replugs the cable? like what does unplugging/plugging it back in do to get the miniscope recognized again, and can we automate that so OP doesn't need to babysit the cameras and plug/unplug cables?

Aside from that, it should be the case that the software alerts you when a camera disconnects in some way, so even if there is no way to trigger a virtual unplug/replug via the software or firmware we can at least let the user know that they need to do that. but unfortunately this software is deprecated and we are in the process of replacing it, so we will make sure that there are informative error messages for this problem. doesn't help much on this issue for now, but that's the state of development we're at.

one more question for @arnauramos1 - does this ever happen when just one camera is connected? like if the source of the problem is just connectivity issues when the mouse is bumping around, then we would expect that to also happen when there is just one camera, but if it only happens when there are multiple cameras that seems to suggest there is something else going on too

[arnauramos1]

Hey,
Thanks very much both of you for taking so much care of this!
I haven't tested it with one camera only for this specific test since it is quite long (1h approx). In other similar tests where I have tested only 2 cameras +2 miniscopes, these disconnections did not happen.
I can already say that the disconnection is not "motion/bumping related", since I have checked the moments when it disconnected and the mouse is sometimes just freezing..
I also checked that the disconnection wouldn't be "power" related, by having superspeed usb hubs with external power connected, as well as the daq powered externally too. Still happened.