drillDown test failures correlated with advance to super commit 5163d68

[philrz]

The app has had its super dependency stuck at commit 79bb8e3 because once it advances to 5163d68 (associated with the changes in brimdata/super#6263) or newer, yarn test has the following failures:

        FAIL  src/js/models/program.test.ts
         ● drill down › when there is no leading filter
       
           expect(received).toBe(expected) // Object.is equality
       
           Expected: "this[\"i d\"][\"orig h\"]==192.168.0.54"
           Received: ""
       
             68 |   test("when there is no leading filter", async () => {
             69 |     const script = await run(result, 'count() by this["i d"]["orig h"]')
           > 70 |     expect(script).toBe('this["i d"]["orig h"]==192.168.0.54')
                |                    ^
             71 |   })
             72 |
             73 |   test("when there is a sort on there", async () => {
       
             at Object.toBe (src/js/models/program.test.ts:70:20)
       
         ● drill down › combines keys in the group by proc
       
           expect(received).toBe(expected) // Object.is equality
       
           Expected: "_path==\"dns\" | id.orig_h==192.168.0.54 proto==\"udp\" query==\"WPAD\""
           Received: "_path==\"dns\" | proto==\"udp\" query==\"WPAD\""
       
              97 |     )
              98 |
           >  99 |     expect(script).toBe(
                 |                    ^
             100 |       '_path=="dns" | id.orig_h==192.168.0.54 proto=="udp" query=="WPAD"'
             101 |     )
             102 |   })
       
             at Object.toBe (src/js/models/program.test.ts:99:20)

I've hacked at this a bit and I have some sense of what's going on, so I'll add a comment with more details.

[philrz]

I traced the concept of drillDown as observed here to #177. With the app in its current form, it's basically when the user has the results some aggregation currently being displayed (e.g., count() by this["i d"]["orig h"]) then the user right-clicks on a particular line of the results table and selects Pivot to Values and the app constructs and executes a filter query so that now the values that were summarized in that aggregation row are displayed (e.g., the filter query this["i d"]["orig h"]=="192.168.0.54").

The following videos show a user interactively clicking through that sequence with the attached data.json, effectively replicating the same steps performed by the test.

The first video is a baseline when the super dependency is pointing at 79bb8e3 and everything works correctly. Notice how the filter at the end shows this["i d"]["orig h"]=="192.168.0.54".

Baseline.mp4

The second video is the repro when the super dependency is pointing at 5163d68 and the problem occurs. Notice how the filter now shows this["orig h"]=="192.168.0.54" which no longer matches the test data and hence no output is shown.

Repro.mp4

Via hacking, a way I can explain what's happening here is that the groupByKeys in this line of the drillDown function:

zui/apps/superdb-desktop/src/js/models/program.ts

Line 75 in 399d1f0

groupByKeys: string[]

...were originally obtained by sending the query to the SuperDB API's DESCRIBE endpoint. At super commit 79bb8e3 when things were working, for this query it showed "aggregation_keys":[["i d","orig h"]]

$ curl http://localhost:9867/version
{"version":"79bb8e346"}

$ curl -H "Accept: application/json" -d '{"query":"count() by this[\"i d\"][\"orig h\"]","head":{"pool":"test"}}' http://localhost:9867/query/describe
{"sources":[{"kind":"Null"}],"channels":[{"name":"main","aggregation_keys":[["i d","orig h"]],"sort":null}]}

Then at commit 5163d68 where the problem started, now it's "aggregation_keys":[["orig h"]]:

$ curl http://localhost:9867/version
{"version":"5163d686e"}

$ curl -H "Accept: application/json" -d '{"query":"count() by this[\"i d\"][\"orig h\"]","head":{"pool":"test"}}' http://localhost:9867/query/describe
{"sources":[{"kind":"Null"}],"channels":[{"name":"main","aggregation_keys":[["orig h"]],"sort":null}]}

What I cannot say is if it's accurate to characterize this effect as a bug in DESCRIBE that was introduced as part of the brimdata/super#6263 changes, or if this change in what the endpoint returns is in line with expectations and that the app needs to construct its filter in a different way, perhaps with different/new help from DESCRIBE.

[philrz]

The changes in brimdata/super#6370 had the intended effect of getting the test to start running successfully again in CI. However, upon closer inspection it seems that the user-level expeirence with that test data of selecting Count By that field and then doing the right-click Pivot to Logs has not been restored all the way to the working state shown in the top video above. Instead the Pivot to Logs now behaves like a no-op as shown in the video below taken with commit 3d88139.

Repro.mp4

Further study has revealed that this problem is not unique to that particularly challenging test data. Separate issue #3217 has been opened to track the more general problem.

Thanks to @nwt for getting the CI working again!