Log size extremely large

[thomaszwagerman]

Log sizes are becoming large, see below:

ls -lh /home/thozwa/Documents/PolarRoute-pipeline/logs/
total 455M
-rw-rw-r-- 1 thozwa thozwa 424M Oct  2 10:56 amsr_central_2025-10-02.err
-rw-rw-r-- 1 thozwa thozwa  31M Oct  2 10:56 amsr_southern_2025-10-02.err

I'm testing pipeline with a newer version of MeshiPhi, so not sure if that is the culprit or if I can reduce the size in pipeline (lower verbosity?)

[davidwyld]

I'm running polarroute-pipeline 2.0.3 and have a multi-hundred MB logfile, which I assumed was because it had been appending, but not the case. Probably we may want to turn off debug logging in the mesh_generate.sh script

https://github.com/bas-amop/PolarRoute-pipeline/blob/60273eb4b1fd9412f3ecd16127e1fb559d8b57ce/scripts/mesh_generate.sh#L26-L30

$ create_mesh --help
usage: create_mesh [-h] [--version] [-o OUTPUT] [-v] config

positional arguments:
  config                File location of a <config.json> file

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -o OUTPUT, --output OUTPUT
                        Output file
  -v, --verbose         Turn on DEBUG level logging

[davidwyld]

In fact, looks like it's down to the progress bar....

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11878 / 28854] ESC[A^M Processing gebco data: 0/6 |          |   0%, [12:25 elapsed] [26-11-25 16:06:33 :DEBUG   ] -    Splitting locked by attribute: 'elevation' in bounds:'{lat range :[59.6875,60.0], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}'
[26-11-25 16:06:33 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.6875,60.0], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}' returned 'HOM'
[26-11-25 16:06:33 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.6875,60.0], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11879 / 28854] ESC[A^M Processing gebco data: 0/6 |          |   0%, [12:26 elapsed] [26-11-25 16:06:34 :DEBUG   ] -    hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.6875,60.0], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}' returned 'HET'
[26-11-25 16:06:35 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.6875,60.0], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11880 / 28858] ESC[A^M Processing gebco data: 0/6 |          |   0%, [12:51 elapsed] [26-11-25 16:06:58 :DEBUG   ] -    Splitting locked by attribute: 'elevation' in bounds:'{lat range :[59.375,59.6875], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}'
[26-11-25 16:06:58 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.375,59.6875], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}' returned 'HOM'
[26-11-25 16:06:58 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.375,59.6875], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11881 / 28858] ESC[A^M Processing gebco data: 0/6 |          |   0%, [12:51 elapsed] [26-11-25 16:06:58 :DEBUG   ] -    Splitting locked by attribute: 'elevation' in bounds:'{lat range :[59.375,59.6875], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}'
[26-11-25 16:06:58 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.375,59.6875], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}' returned 'HOM'
[26-11-25 16:06:58 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.375,59.6875], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11882 / 28858] ESC[A[26-11-25 16:06:59 :DEBUG   ] -     Splitting locked by attribute: 'elevation' in bounds:'{lat range :[59.0625,59.375], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}'
[26-11-25 16:06:59 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.0625,59.375], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}' returned 'HOM'
[26-11-25 16:06:59 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.0625,59.375], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11883 / 28858] ESC[A^M Processing gebco data: 0/6 |          |   0%, [12:52 elapsed] [26-11-25 16:07:00 :DEBUG   ] -    Splitting locked by attribute: 'elevation' in bounds:'{lat range :[59.0625,59.375], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}'
[26-11-25 16:07:00 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.0625,59.375], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}' returned 'HOM'
[26-11-25 16:07:00 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.0625,59.375], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11884 / 28858] ESC[A^M Processing gebco data: 0/6 |          |   0%, [12:53 elapsed] [26-11-25 16:07:00 :DEBUG   ] -    Splitting locked by attribute: 'elevation' in bounds:'{lat range :[58.75,59.0625], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}'
[26-11-25 16:07:00 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[58.75,59.0625], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}' returned 'HOM'
[26-11-25 16:07:00 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[58.75,59.0625], long range :[-71.25,-70.625], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11885 / 28858] ESC[A[26-11-25 16:07:00 :DEBUG   ] -     hom_condition for attribute: 'elevation' in bounds:'{lat range :[58.75,59.0625], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}' returned 'HET'
[26-11-25 16:07:00 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[58.75,59.0625], long range :[-70.625,-70.0], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

^M Split depth: 3/5 |██████    |  60%, [Cellbox 11886 / 28862] ESC[A^M Processing gebco data: 0/6 |          |   0%, [13:16 elapsed] [26-11-25 16:07:23 :DEBUG   ] -    hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.6875,60.0], long range :[-70.0,-69.375], time range :['2025-11-23', '2025-11-25']}' returned 'HET'
[26-11-25 16:07:23 :DEBUG   ] -         hom_condition for attribute: 'elevation' in bounds:'{lat range :[59.6875,60.0], long range :[-70.0,-69.375], time range :['2025-11-23', '2025-11-25']}' returned 'CLR'

[thomaszwagerman]

That is funny - I am turning off create_mesh verbosity in #17, but maybe we get 5dfabb5 into main early?