Is allowed to modify some system parameters for JsonBench

[wangmj17]

Hi,

I'm trying to add a new system to jsonbench. I have two questions regarding the configuration:

1. Is it allowed to modify the system's global configuration?

2. Is it allowed to change query-level configurations, such as setting session-level GUCs for individual queries?

[rschu1ze]

system's global configuration

What do you mean exactly by that? Linux kernel parameters? This practice is generally discouraged to allow fair comparison between databases. Of course, it is okay if a database without modified system parameters otherwise.

session-level GUCs

What do you mean exactly by GUC?

[wangmj17]

We are an analytical database running in the cloud, so naturally we do not support modifying Linux system parameters. What I’m referring to are the database system’s own parameters, such as cache ratios.

GUC stands for Grand Unified Configuration in PostgreSQL — you can think of it as a type of configuration setting.

Additionally, I personally don’t agree that tuning parameters compromises fairness. In real-world production environments, users are allowed to adjust various configuration parameters based on their specific scenarios.

[wangmj17]

Additionally, I’d like to add that the default parameters of our system are generally tuned for real-world production scenarios. However, the workload in jsonbench is quite different from our actual production workloads. In our real use cases, users typically access a large number of subfields. If a user only needs to access 5 specific subfields, it would make more sense to fully expand them into generated columns — or even extract them as regular columns instead of using JSON at all.

[rschu1ze]

The "sister benchmark" of JSONBench, ClickBench, recently introduced a "tuned" toggle in the UI to distinguish

• the base configuration (the behavior users experience without any tuning), and
• tuned configurations, which includes session/user-level optimization.

Re expanding sub-fields into columns, this is referred to as "flattening" in JSONBench. I would refer to here and here regarding the benchmark rules about that.

Please let me know if you have further questions.

[wangmj17]

Thank you for you reply!

1. for tuned configurations, is it allowed to set configuration per query? For example, "set config_xxx = xxx; SELECT data -> 'commit' ->> 'collection' AS event, COUNT(*) as count FROM bluesky GROUP BY event ORDER BY count DESC;", this is a typical way of using session-level GUC in PostgresSql database.
2. I know the rules of flattening, what I am trying to say is that only querying 5 sub-columns is too limited for a json benchmark.

[rschu1ze]

for tuned configurations, is it allowed to set configuration per query? For example, "set config_xxx = xxx; SELECT data -> 'commit' ->> 'collection' AS event, COUNT(*) as count FROM bluesky GROUP BY event ORDER BY count DESC;", this is a typical way of using session-level GUC in PostgresSql database.

The very definition of "session-level GUC" implies that settings are configured per session, and not per query. Or do I misunderstand something?

I know the rules of flattening, what I am trying to say is that only querying 5 sub-columns is too limited for a json benchmark.

We all need to live with the fact that no benchmark perfectly represents the real world 😉

[wangmj17]

Postgres doesn't have query-level GUC, so generally session-leveled GUC is used for set configurations per query.
Back to the question before, is "set config_xxx = xxx; SELECT data -> 'commit' ->> 'collection' AS event, COUNT(*) as count FROM bluesky GROUP BY event ORDER BY count DESC;" allowed or not?

[alexey-milovidov]

Yes, it will be ok for the "tuned" entry.