Roborio running out of memory

[son34]

Our roboRIO is throwing this error:

Loading Dependency Injection Framework
Feb 27, 2015 2:09:45 AM java.util.logging.LogManager$RootLogger log
CONFIG: Configuration done.
Feb 27, 2015 2:09:45 AM java.util.logging.LogManager$RootLogger log
CONFIG: GRIP Version: 1.1.1
platform: /Linux/arm/
Feb 27, 2015 2:09:50 AM edu.wpi.grip.core.Main start
INFO: Loading file /home/lvuser/project.grip
Java HotSpot(TM) Embedded Client VM warning: INFO: os::commit_memory(0xb48c2000, 32768, 1) failed; error='Cannot allocate memory' (errno=12)

 There is insufficient memory for the Java Runtime Environment to continue.
 Native memory allocation (mmap) failed to map 32768 bytes for committing reserved memory.
 An error report file with more information is saved as:
 /home/lvuser/hs_err_pid1963.log

 Compiler replay data is saved as:
 /home/lvuser/replay_pid1963.log

Anyone else run into this problem? Or any ideas for fixes?

[ThomasJClark]

Are you also running GRIP from your user program? If this is the case, there may be two instances of GRIP running at the same time. Try commenting out the part of your program where you start GRIP, then restarting your RIO.

We might want to make it so any existing instances of GRIP are automatically killed when you deploy to avoid this

[son34]

We are currently only deploying grip onto the roboRIO. We are essentially only trying to get it to run remotely, no actual code being used yet.

[ThomasJClark]

Ok. I guess try restarting, just in case there's another instance running for whatever reason.

If it's still a problem, can you post your pipeline?

[son34]

Tries resetting the roboRIO, turned off and on the entire robot, as well as the computer (To be safe) but nothing seemed to fix it. Even checked the allocated memory on the roboRIO control panel in-browser. No problems there.

And I'll post the pipeline in a couple secs (How do I find the pipeline by the way?)

[JLLeitschuh]

It's wherever you saved it. You can open the .grip file an a text editor. Just paste the contents here.

[ThomasJClark]

Just take a screenshot of the pipeline

[jakedp7]

@son34 and I's pipeline:

<grip:Pipeline>
  <sources>
    <grip:ImageFile>
      <property name="path" value="C:\Users\Jake\Desktop\ultimate-ascent.jpeg"/>
    </grip:ImageFile>
    <grip:Camera>
      <property name="address" value="http://axis-camera.local"/>
    </grip:Camera>
  </sources>
  <steps>
    <grip:Step name="HSL Threshold">
    <grip:Step name="Blur">
      <grip:Input step="0" socket="0"/>
      <grip:Input step="0" socket="1">
        <value>MEDIAN</value>
      </grip:Input>
      <grip:Input step="0" socket="2">
        <value>3.6036036036036037</value>
      </grip:Input>
      <grip:Output step="0" socket="0" previewed="true"/>
    </grip:Step>
    <grip:Step name="HSL Threshold">
      <grip:Input step="1" socket="0"/>
      <grip:Input step="1" socket="1">
        <value>
          <double>87.41007194244605</double>
          <double>180.0</double>
          <double>69.60431654676258</double>
          <double>97.06484641638224</double>
        </value>
      </grip:Input>
      <grip:Input step="0" socket="2">
      <grip:Input step="1" socket="2">
        <value>
          <double>0.0</double>
          <double>116.95143884892086</double>
          <double>255.0</double>
        </value>
      </grip:Input>
      <grip:Input step="0" socket="3">
      <grip:Input step="1" socket="3">
        <value>
          <double>16.052158273381295</double>
          <double>75.67446043165468</double>
          <double>255.0</double>
        </value>
      </grip:Input>
      <grip:Output step="0" socket="0" previewed="true"/>
      <grip:Output step="1" socket="0" previewed="true"/>
    </grip:Step>
    <grip:Step name="Find Contours">
      <grip:Input step="1" socket="0"/>
      <grip:Input step="1" socket="1">
      <grip:Input step="2" socket="0"/>
      <grip:Input step="2" socket="1">
        <value>true</value>
      </grip:Input>
      <grip:Output step="1" socket="0" previewed="true"/>
      <grip:Output step="2" socket="0" previewed="true"/>
    </grip:Step>
    <grip:Step name="Filter Contours">
      <grip:Input step="2" socket="0"/>
      <grip:Input step="2" socket="1">
        <value>100.0</value>
    <grip:Step name="Convex Hulls">
      <grip:Input step="3" socket="0"/>
      <grip:Output step="3" socket="0" previewed="true"/>
    </grip:Step>
    <grip:Step name="Publish ContoursReport">
      <grip:Input step="4" socket="0"/>
      <grip:Input step="4" socket="1">
        <value>tapeTrackingCountours</value>
      </grip:Input>
      <grip:Input step="2" socket="2">
        <value>0.0</value>
      <grip:Input step="4" socket="2">
        <value>true</value>
      </grip:Input>
      <grip:Input step="2" socket="3">
        <value>0.0</value>
      <grip:Input step="4" socket="3">
        <value>true</value>
      </grip:Input>
      <grip:Input step="2" socket="4">
        <value>1000.0</value>
      <grip:Input step="4" socket="4">
        <value>true</value>
      </grip:Input>
      <grip:Input step="2" socket="5">
        <value>0.0</value>
      <grip:Input step="4" socket="5">
        <value>true</value>
      </grip:Input>
      <grip:Input step="2" socket="6">
        <value>1000.0</value>
      <grip:Input step="4" socket="6">
        <value>true</value>
      </grip:Input>
      <grip:Output step="2" socket="0" previewed="true"/>
    </grip:Step>
    <grip:Step name="Convex Hulls">
      <grip:Input step="3" socket="0"/>
      <grip:Output step="3" socket="0" previewed="true"/>
    </grip:Step>
  </steps>
  <connections>
    <grip:Connection>
      <grip:Output step="0" socket="0" previewed="true"/>
      <grip:Input step="1" socket="0"/>
    </grip:Connection>
    <grip:Connection>
      <grip:Output step="1" socket="0" previewed="true"/>
      <grip:Input step="2" socket="0"/>
    </grip:Connection>
    <grip:Connection>
      <grip:Output step="2" socket="0" previewed="true"/>
      <grip:Input step="3" socket="0"/>
      <grip:Output step="0" socket="0" previewed="true"/>
      <grip:Input step="1" socket="0"/>
    </grip:Connection>
    <grip:Connection>
      <grip:Output step="3" socket="0" previewed="true"/>
      <grip:Input step="4" socket="0"/>
    </grip:Connection>
    <grip:Connection>
      <grip:Output source="0" socket="0" previewed="true"/>
      <grip:Input step="0" socket="0"/>
    </grip:Connection>
    <grip:Connection>
      <grip:Output step="2" socket="0" previewed="true"/>
      <grip:Input step="3" socket="0"/>
    </grip:Connection>
  </connections>
  <settings>
    <teamNumber>0</teamNumber>
    <teamNumber>2783</teamNumber>
    <publishAddress></publishAddress>
    <deployAddress></deployAddress>
    <deployAddress>roborio-2783-frc.local</deployAddress>
  </settings>
</grip:Pipeline>

[ThomasJClark]

Try changing median blur to something else. Median filters are really expensive in terms of CPU, although I'm not sure why that would lead to memory problems.

The text in the previous comment also doesn't seem to be a valid project file that GRIP would be able to create or open. I'm not sure how it got corrupted, but maybe try recreating it

[JLLeitschuh]

@ThomasJClark
You'll need to remove the image source for you.

[tom131313]

Here are a couple more examples of out of memory. I may run the robot program 3 to 10 times without trouble then the next 1 to 4 times in a row get the out of memory. Rebooting and restarting don't seem to have much effect in predicting when the error will happen.

There is only 1 java process running project.grip - the ps command display of processes appears okay.

Do you have a way of trying one or more of the suggestions in the error log to prevent this error?

ONE OF THE JAVA ERROR LOGS:

roboRIO-4237-FRC:/tmp$ cat hs_err_pid9636.log

 There is insufficient memory for the Java Runtime Environment to continue.
 Native memory allocation (mmap) failed to map 1048576 bytes for committing reserved memory.
 Possible reasons:
   The system is out of physical RAM or swap space
   In 32 bit mode, the process size limit was hit
 Possible solutions:
   Reduce memory load on the system
   Increase physical memory or swap space
   Check if swap backing store is full
   Use 64 bit Java on a 64 bit OS
   Decrease Java heap size (-Xmx/-Xms)
   Decrease number of Java threads
   Decrease Java thread stack sizes (-Xss)
   Set larger code cache with -XX:ReservedCodeCacheSize=
 This output file may be truncated or incomplete.

  Out of Memory Error (os_linux.cpp:2718), pid=9636, tid=3027186784

 JRE version: Java(TM) SE Embedded Runtime Environment (8.0_06-b23) (build 1.8.0_06-b23)
 Java VM: Java HotSpot(TM) Embedded Client VM (25.6-b23 mixed mode linux-arm )
 Core dump written. Default location: //core or core.9636 (max size 2048 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again

========================================
➔ Launching «'/home/lvuser/FRCUserProgram'»
[HAL] Starting task FRC_DriverStation...
[GRIP Test] in Constructor
[GRIP Test] in RobotInit
[GRIP Test] fork okay continuing with parent robot code
Default Disabled() method... Override me!
NT: ERROR: bind() failed: Address already in use (TCPAcceptor.cpp:102)
Loading Dependency Injection Framework
Jan 24, 2016 4:46:22 PM java.util.logging.LogManager$RootLogger log
CONFIG: Configuration done.
Jan 24, 2016 4:46:22 PM java.util.logging.LogManager$RootLogger log
CONFIG: GRIP Version: 1.1.1
Java HotSpot(TM) Embedded Client VM warning: INFO: os::commit_memory(0xae53a000, 131072, 0) failed; error='Cannot allocate memory' (errno=12)

 There is insufficient memory for the Java Runtime Environment to continue.
 Native memory allocation (mmap) failed to map 131072 bytes for committing reserved memory.
 An error report file with more information is saved as:
 /tmp/hs_err_pid13039.log

? Launching «'/home/lvuser/FRCUserProgram'»
[HAL] Starting task FRC_DriverStation...
GRIP Test in RobotInit
fork okay back in parent robot code
Default DisabledInit() method... Overload me!
Default DisabledPeriodic() method... Overload me!
NT: server: client CONNECTED: 169.254.248.235 port 51495
NT: server: client CONNECTED: 169.254.6.209 port 59054
NT: server: client CONNECTED: 169.254.6.209 port 59055
Loading Dependency Injection Framework
Jan 24, 2016 3:04:43 PM java.util.logging.LogManager$RootLogger log
CONFIG: Configuration done.
Jan 24, 2016 3:04:43 PM java.util.logging.LogManager$RootLogger log
CONFIG: GRIP Version: 1.1.1
platform: /Linux/arm/
Jan 24, 2016 3:04:51 PM edu.wpi.grip.core.Main start
INFO: Loading file /home/lvuser/project.grip
Jan 24, 2016 3:04:55 PM edu.wpi.grip.core.GRIPCoreModule onThreadException
SEVERE: Error from Thread[main,5,main]
java.lang.UnsatisfiedLinkError: java.io.IOException: No space left on device
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:613)
    at org.bytedeco.javacpp.Loader.load(Loader.java:438)
    at org.bytedeco.javacpp.Loader.load(Loader.java:381)
    at org.bytedeco.javacpp.opencv_imgcodecs.<clinit>(opencv_imgcodecs.java:13)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:340)
    at org.bytedeco.javacpp.Loader.load(Loader.java:413)
    at org.bytedeco.javacpp.Loader.load(Loader.java:381)
    at org.bytedeco.javacpp.opencv_videoio.<clinit>(opencv_videoio.java:14)
    at org.bytedeco.javacv.OpenCVFrameGrabber.start(OpenCVFrameGrabber.java:177)
    at org.bytedeco.javacv.FrameGrabber.restart(FrameGrabber.java:430)
    at edu.wpi.grip.core.sources.CameraSource.start(CameraSource.java:189)
    at edu.wpi.grip.core.sources.CameraSource.initialize(CameraSource.java:174)
    at edu.wpi.grip.core.Source.initializeSafely(Source.java:98)
    at edu.wpi.grip.core.serialization.SourceConverter.unmarshal(SourceConverter.java:60)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
    at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.addCurrentElementToCollection(CollectionConverter.java:98)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.populateCollection(CollectionConverter.java:91)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.populateCollection(CollectionConverter.java:85)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:480)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:412)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:263)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
    at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1206)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1190)
    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1061)
    at edu.wpi.grip.core.serialization.Project.open(Project.java:68)
    at edu.wpi.grip.core.serialization.Project.open(Project.java:62)
    at edu.wpi.grip.core.Main.start(Main.java:59)
    at edu.wpi.grip.core.Main.main(Main.java:41)
Caused by: java.io.IOException: No space left on device
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:307)
    at org.bytedeco.javacpp.Loader.extractResource(Loader.java:306)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:563)
    ... 39 more

Java HotSpot(TM) Embedded Client VM warning: INFO: os::commit_memory(0xb68aa000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)

 There is insufficient memory for the Java Runtime Environment to continue.
 Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
 An error report file with more information is saved as:
 /tmp/hs_err_pid3388.log

The project.grip:

< grip:Pipeline>
  < sources>
    < grip:Camera>
      < property name="deviceNumber" value="0"/>
    < /grip:Camera>
  < /sources>
  < steps>
    < grip:Step name="HSL Threshold">
      < grip:Input step="0" socket="0"/>
      < grip:Input step="0" socket="1">
        < value>
          < double>0.0< /double>
          < double>140.06825938566553< /double>
        < /value>
      < /grip:Input>
      < grip:Input step="0" socket="2">
        < value>
          < double>123.83093525179855< /double>
          < double>148.38737201365188< /double>
        < /value>
      < /grip:Input>
      < grip:Input step="0" socket="3">
        < value>
          < double>9.172661870503596< /double>
          < double>255.0< /double>
        < /value>
      < /grip:Input>
      < grip:Output step="0" socket="0" previewed="false"/>
    < /grip:Step>
    < grip:Step name="Find Contours">
      < grip:Input step="1" socket="0"/>
      < grip:Input step="1" socket="1">
        < value>true< /value>
      < /grip:Input>
      < grip:Output step="1" socket="0" previewed="false"/>
    < /grip:Step>
    < grip:Step name="Filter Contours">
      < grip:Input step="2" socket="0"/>
      < grip:Input step="2" socket="1">
        < value>10.0< /value>
      < /grip:Input>
      < grip:Input step="2" socket="2">
        < value>0.0< /value>
      < /grip:Input>
      < grip:Input step="2" socket="3">
        < value>2.0< /value>
      < /grip:Input>
      < grip:Input step="2" socket="4">
        < value>1000.0< /value>
      < /grip:Input>
      < grip:Input step="2" socket="5">
        < value>0.0< /value>
      < /grip:Input>
      < grip:Input step="2" socket="6">
        < value>1000.0< /value>
      < /grip:Input>
      < grip:Input step="2" socket="7">
        < value>
          < int>0< /int>
          < int>100< /int>
        < /value>
      < /grip:Input>
      < grip:Output step="2" socket="0" previewed="false"/>
    < /grip:Step>
    < grip:Step name="Publish ContoursReport">
      < grip:Input step="3" socket="0"/>
      < grip:Input step="3" socket="1">
        < value>MyTargetTable< /value>
      < /grip:Input>
      < grip:Input step="3" socket="2">
        < value>false< /value>
      < /grip:Input>
      < grip:Input step="3" socket="3">
        < value>false< /value>
      < /grip:Input>
      < grip:Input step="3" socket="4">
        < value>false< /value>
      < /grip:Input>
      < grip:Input step="3" socket="5">
        < value>true< /value>
      < /grip:Input>
      < grip:Input step="3" socket="6">
        < value>true< /value>
      < /grip:Input>
      < grip:Input step="3" socket="7">
        < value>false< /value>
      < /grip:Input>
    < /grip:Step>
    < grip:Step name="Find Blobs">
      < grip:Input step="4" socket="0"/>
      < grip:Input step="4" socket="1">
        < value>6.0< /value>
      < /grip:Input>
      < grip:Input step="4" socket="2">
        < value>
          < double>0.0< /double>
          < double>0.8464163822525597< /double>
        < /value>
      < /grip:Input>
      < grip:Input step="4" socket="3">
        < value>true< /value>
      < /grip:Input>
      < grip:Output step="4" socket="0" previewed="false"/>
    < /grip:Step>
    < grip:Step name="Publish BlobsReport">
      < grip:Input step="5" socket="0"/>
      < grip:Input step="5" socket="1">
        < value>myBlobsReport< /value>
      < /grip:Input>
      < grip:Input step="5" socket="2">
        < value>true< /value>
      < /grip:Input>
      < grip:Input step="5" socket="3">
        < value>true< /value>
      < /grip:Input>
      < grip:Input step="5" socket="4">
        < value>true< /value>
      < /grip:Input>
    < /grip:Step>
  < /steps>
  < connections>
    < grip:Connection>
      < grip:Output source="0" socket="0" previewed="false"/>
      < grip:Input step="4" socket="0"/>
    < /grip:Connection>
    < grip:Connection>
      < grip:Output step="4" socket="0" previewed="false"/>
      < grip:Input step="5" socket="0"/>
    < /grip:Connection>
    < grip:Connection>
      < grip:Output source="0" socket="0" previewed="false"/>
      < grip:Input step="0" socket="0"/>
    < /grip:Connection>
    < grip:Connection>
      < grip:Output step="0" socket="0" previewed="false"/>
      < grip:Input step="1" socket="0"/>
    < /grip:Connection>
    < grip:Connection>
      < grip:Output step="2" socket="0" previewed="false"/>
      < grip:Input step="3" socket="0"/>
    < /grip:Connection>
    < grip:Connection>
      < grip:Output step="1" socket="0" previewed="false"/>
      < grip:Input step="2" socket="0"/>
    < /grip:Connection>
  < /connections>
  < settings>
    < teamNumber>4237< /teamNumber>
    < publishAddress>roborio-4237-frc.local< /publishAddress>
    < deployAddress>roborio-4237-frc.local< /deployAddress>
    < deployDir>/home/lvuser< /deployDir>
    < deployUser>lvuser< /deployUser>
    < deployJavaHome>/usr/local/frc/JRE/< /deployJavaHome>
  < /settings>
< /grip:Pipeline>

[ThomasJClark]

So, the roboRIO should have enough memory to run this program. After doing some profiling on my computer I have an educated guess about what might be happening.

Java usually tries to eat up as much memory as it's allowed to before it does garbage collection. Most of the memory in GRIP is taken up by OpenCV objects, which are allocated from native code. So, maybe the JRE doesn't know about them, so the heuristics for running the GC aren't working because they only take into account Java objects.

If this is right, we can fix it by either calling System.gc() periodically (which is usually frowned upon), or carefully calling mat.release() when necessary

[tom131313]

hello

I spent a lot more time looking at GRIP running on the roboRIO,
restarting the roboRIO, rebooting the roboRIO, and deploying new robot
code from Eclipse to the roboRIO.

My testing is still not perfectly definitive and I hate to mislead you.
I suspect that forcing garbage collection is not the solution to the
problem unless by periodically you mean as the program is initializing.

The out of memory is always near the beginning of a run while the
initialization messages are printing (within 10 seconds or much less).

I got a few out of memory errors during my latest experiments but only
one instance was I able to catch all the data I needed to know what might
have actually happened.

During deploy new code from Eclipse there is a killall and an execution
of FRCkill that should certainly clean up all old processes and then
start the newly deployed code. Despite the seemingly double killing, the
old Java process running GRIP did NOT die. It continued to live under
parent process ID = 1. Yes there was then no end of errors thrown
because 2 GRIPs were running. I can't imagine how that happened but it
did - I have the errors and "top" command output to prove 2 GRIPs at
once.

Is this my entire problem? I don't know. Maybe, maybe not.

What I really want to know is there enough processing power in the
roboRIO to run GRIP. GRIP runs correctly much of the time and it's using
75% of the memory and 50% of the CPU with about 2 frames a second of
information. Maybe I'd better stick with running GRIP on the fast driver
station PC. An advantage is we could then see what the images are that
are being processed (I hate flying blind on the roboRIO). I tried that
last night with an IP Axis M1011 and the camera image froze in GRIP after
running a minute although the camera itself continued to stream images to
a browser connection that I started while GRIP was frozen. So that GRIP
didn't work either and I clicked on send error report and an issue was
created (sorry, it was probably a useless print of the time out error).

I think GRIP is an extremely good concept for the FRC team - reasonably
fast (maybe, on the right computer) and very easy to use compared to all
other computer vision programs we have available. The kids only get to
spend a few hours developing computer vision and GRIP seems to be the
tool of choice if we can get it running well.

Thanks for your help. I greatly appreciate your effort for developing
GRIP.

Rick Thomas

On Mon, 25 Jan 2016 15:30:28 -0800 Thomas Clark
notifications@github.com writes:
So, the roboRIO should have enough memory to run this program. After
doing some profiling on my computer I have an educated guess about what
might be happening.
Java usually tries to eat up as much memory as it's allowed to before it
does garbage collection. Most of the memory in GRIP is taken up by OpenCV
objects, which are allocated from native code. So, maybe the JRE doesn't
know about them, so the heuristics for running the GC aren't working
because they only take into account Java objects.
If this is right, we can fix it by either calling System.gc()
periodically (which is usually frowned upon), or carefully calling
mat.release() when necessary
�
Reply to this email directly or view it on GitHub.

---

Richard K. Thomas

---

[tom131313]

One more conjecture that might be related to out of memory -- timing
conflicts in handling log/lck files?

I'm starting to suspect there is a timing and resource conflict between
terminating the previous instance of GRIP and creating the new instance.
Maybe stuff happens in the new run on the roboRIO before the old run is
cleaned up.

Sometimes "cannot allocate memory" error is accompanied by "No space left
on device" error.

There is no hint for me about what device that would be. There is plenty
of space on the "disk" drive (190.2M used, 191.4M available from the df
command).

The only thing I see strange about files is the use of the GRIP.log
(.sequence#) and GRIP.log (.sequence#).lck.

Apparently when the driver station restarts the robot program, the new
instance of Java GRIP reuses the previous GRIP log files.

When the roboRIO is rebooted the next available sequence number (starting
from the beginning and not necessarily the last used number) is used.

Strangely an occasional log or lck file will disappear from the file list
(ls -l).

Further strangely, the date/time stamp of files change - especially the
lck files and recently created files (a couple of minutes ago) will
reappear on restart as having 13 day old dates and times.

Maybe these observations will help; hope that don't confuse.

Rick