Scout complains of "TypeError: failed to fetch" when attempting to view a large dataset

[pipmc]

Specifically, I get the error:

Error Loading Dataframe
TypeError: Failed to fetch

When I list the scan result directory I get:

$ ls -lh /home/inspect/app/scans/scan_id=JewhPYVCPJtVVXUshPAw9A
total 1018M
-rw-r--r-- 1 inspect inspect 966M Dec  3 15:47 broken_env_scanner.parquet
-rw-r--r-- 1 inspect inspect    0 Dec  3 15:47 _errors.jsonl
-rw-r--r-- 1 inspect inspect  40M Dec  3 15:16 _scan.json
-rw-r--r-- 1 inspect inspect  309 Dec  3 15:47 _summary.json

Chrome's web inspector shows a request to http://localhost:7576/api/scanner_df/scan_id%3DJewhPYVCPJtVVXUshPAw9A?scanner=broken_env_scanner with response size 2,716,521 kB and a request to http://localhost:7576/api/scans?status_only=true with response size 119,235 kB. All requests return 200 OK.

The samples being scanned relate to non-public tasks so I can't share the scan results. Do you have any suggestions for how I could debug locally?

[epatey]

@pipmc, is this when you're trying to view the scan results?

[pipmc]

@pipmc, is this when you're trying to view the scan results?

Yes.

[pipmc]

(I don’t have a problem viewing any of the other scans I’ve done)

[dragonstyle]

This is most likely purely based upon the size of this scan result- we currently fetch the whole dataframe to the client which in this case is very large. We need to do a little more work to fetch only the data that we need. We're planning that soon but I don't have a firm eta right now.

Is there any way to break up the scan into smaller chunks in the short term?

[pipmc]

Is there any way to break up the scan into smaller chunks in the short term?

Yes, I'll do that for now.

[dragonstyle]

Ok sorry for the issues- let me know if that doesn’t solve the problem in the short term. I’ll follow up here once we’ve improved the client behavior for these large file scenarios…

[dragonstyle]

Sorry, one other thing that might be useful - in the Chrome inspector you should see a third request for the dataframe itself (it will include a path like /scanner_df/<path_to_parquet>. I'm assuming this is the request that is failing, but it would be useful to confirm that this is indeed being tried and is failing. Can you see a request list this and is it failing?

[pipmc]

Sorry, one other thing that might be useful - in the Chrome inspector you should see a third request for the dataframe itself (it will include a path like /scanner_df/<path_to_parquet>. I'm assuming this is the request that is failing, but it would be useful to confirm that this is indeed being tried and is failing. Can you see a request list this and is it failing?

I can see it and it's not failing. Attached is a sanitized HAR of all the requests sent when I tried to view the scan in the viewer, along with the metadata relating to the scan itself.

scan_id=JewhPYVCPJtVVXUshPAw9A.json
har.json

[dragonstyle]

Ok thank you this is helpful! I'll revert here when I have more!