GUI and API freeze during share crawling

[bnille]

Project
NetApp Connector for M365 Copilot

Describe the bug
After starting to crawl a ready share, UI and API freeze

To Reproduce
Steps to reproduce the behavior:

1. Go to 'share'
2. Click on 'crawl'
3. no connection to UI or API

Expected behavior
UI and API should be accessible during manual crawl

Desktop (please complete the following information):

• OS: MacOS / Windows
• Browser: every browser or API curl / swagger UI

Additional context
We see on the ONTAP secd trace log that the crawl is on going and running.
On Linux docker host we can work and look in "top" and see dockerd and postgresql running processes.
The docker VM is ubuntu 24.04 with 8 cores and 32 GB RAM.

[kirkryan]

@bnille - which version of Neo Core are you using? This was fixed in 3.0.5+

[bnille]

@kirkryan Neo-Core version 3.0.5
only with large shares
with a smaller folder it works without freeze

[kirkryan]

@bnille - would you be willing to try the new 3.1.0 released yesterday. There are a number of enhancements.

[bnille]

@kirkryan after update to 3.1.0 still the same issue with our large share...
GUI and API freeze but in the docker logs we see the on going crawling

GUI error: /shares/45c93c1b-f908-4800-a4a9-f6353ce1f5a8 failed (504 Gateway Time-out)

[bnille]

@kirkryan after a few hours the GUI is responsive and working again. We see the crawl on the large share as processing.
so only after the initial start we recognize the freeze.

[kirkryan]

Ok thanks for the information. How many items / size would you estimate your crawl is?

[kirkryan]

@bnille - Let's test this in the 3.1.1 release, as all extraction tasks are non-blocking therefore it would be unexpected to see this occuring.