Skip to content

Commit 35fc3b8

Browse files
szarnyasgszarnyasg
committed
Add audit questionnaire
1 parent 37ef11f commit 35fc3b8

File tree

2 files changed

+164
-0
lines changed

2 files changed

+164
-0
lines changed

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,8 @@ To generate a data set with the same characteristics, see the [documentation on
149149

150150
Implementations of the Interactive workload can be audited by a certified LDBC auditor.
151151
The [Auditing Policies chapter](http://ldbcouncil.org/ldbc_snb_docs/ldbc-snb-specification.pdf#chapter.7) of the specification describes the auditing process and the required artifacts.
152+
If you are considering commissioning an LDBC SNB audit, please study the [audit questionnaire](snb-interactive-audit-questionnaire.md).
153+
152154

153155
### Determining the best TCR
154156

‎snb-interactive-audit-questionnaire.md

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
# Questionnaire for LDBC SNB test sponsors
2+
3+
We created the following questionnaire to make the SNB auditing process more streamlined. If you need any clarification, please reach out to the SNB task force.
4+
5+
## Checklist
6+
7+
- [ ] The SUT has a complete implementation of the benchmark which complies with the LDBC specification.
8+
- [ ] The implementation uses a stable version of an LDBC SNB driver.
9+
- [ ] The implementation passes cross-validation against one of the existing reference implementations on at least the SF10 data set.
10+
- [ ] If you are not the vendor of the DBMS used in the SUT: do you have written permission of the vendor of the DBMS?
11+
12+
## System
13+
14+
### Overview
15+
16+
- Vendor name
17+
- System name
18+
- System version
19+
- Date of the release to be used
20+
- Link to product page
21+
22+
### Test sponsor
23+
24+
- Are you a member company of the LDBC? _(Must be one to commission an audit)_
25+
- Do you have written consent of the vendor of the DBMS used in the SUT? _(The Test Sponsor must have prior written consent from the vendor if the two are not the same company.)_
26+
- Test sponsor representative _(This person is going to sign the Full Disclosure Report in case of a successful audit. Please include title, e.g. "Dr.", if applicable.)_
27+
28+
### High-level technical information
29+
30+
- System type (e.g. relational DBMS, graph DBMS)
31+
- Storage type (in-memory/disk-based)
32+
- Main implementation language of the system (e.g. C++)
33+
- Query language(s) supported by the system
34+
- Query execution strategy (interpreted/vectorized/compiled/etc.)
35+
- Is a distributed version available (regardless of whether it is used in the audit)?
36+
- If a distributed version is available, what sharding strategy does it use?
37+
38+
### Licensing
39+
40+
- Product license (e.g. proprietary license, Apache Software License v2)
41+
- License of the Java/Python client libraries (used in the LDBC SNB Interactive/BI drivers, respectively)
42+
43+
### Database features
44+
45+
- Link to documentation
46+
- Are stored procedures supported?
47+
- If stored procedures are supported, what language(s) can they be implemented in?
48+
- What is the maximum isolation level for transactions?
49+
- Is the database client package available in a central repository (Maven Central, PyPI, Conda, etc.)? (if it is, please provide link)
50+
- Is the database client available as an open-source project? (if it is, please provide link)
51+
- Does the DBMS have any noteworthy features that should be mentioned in its introduction? (e.g. support for temporal queries, incremental view maintenance, etc.)
52+
53+
### Type system
54+
55+
- Data schema: is system schema-free, schema-optional or does it require the specification of the full schema?
56+
- Does the system support Unicode natively?
57+
- Does the system support a native datetime type?
58+
- Does the system support an array datatype? (e.g. for storing attributes such as `person.speaks`)
59+
60+
## Data sets and loading
61+
62+
- Which scale factors (SFs) should be included in the audit?
63+
- Which data layout is used? (e.g. composite attributes with merged foreign keys)
64+
- Which date format is used? (e.g. Datagen's StringDateFormatter with a given formatter such as RFC-3339, LongDateFormatter, etc.)
65+
- How is the initial loading step implemented (e.g. offline bulk loader)?
66+
67+
## Benchmark environment
68+
69+
- Cloud vendor (if applicable)
70+
- Is the DBMS running in a containerized setup?
71+
- Is the LDBC driver running in a containerized setup?
72+
- On which machine is the LDBC driver running?
73+
- [ ] Same machine as the SUT
74+
- [ ] Different machine (please provide HW/instance type)
75+
- Is a container orchestration system used? (e.g. Kubernetes - please describe which one)
76+
- Which operating system(s) are used? (e.g. Ubuntu 22.04.1)
77+
- Runtimes:
78+
- Interactive: Java version for running the driver (e.g. AdoptOpenJDK 8)
79+
- BI: Python version for running the driver (e.g. Python 3.8)
80+
- Firewall ports to open (for multi-instance setup)
81+
- How to access the web interface (if applicable)
82+
- Disk storage
83+
- Disk configuration (e.g. RAID-0)
84+
- File system to use (e.g. xfs)
85+
86+
## Performance tuning
87+
88+
- What are the main database configurations ("tuning knobs") that are changed from their default values for the SNB implementation? (e.g. buffer size, size of connection pool, etc.)
89+
90+
## Implementation details
91+
92+
Please provide the following information **for each scale factor**.
93+
94+
### Setup
95+
96+
- What is the setup of the SUT:
97+
- Embedded (into the benchmark driver)
98+
- Client-server, single-node
99+
- Client-server, distributed
100+
- Machine(s) for running the SUT (number of machines/HW/instance type)
101+
102+
### Costs
103+
104+
- Hardware costs (for cloud instances, the cost of a reserved instance for a 3-year term)
105+
- Software license
106+
- Maintenance fee (3-year period)
107+
108+
## For SNB Interactive implementations
109+
110+
Note that the minimum scale factor for SNB Interactive audits is SF30.
111+
112+
### ACID tests
113+
114+
- [ ] The ACID tests were implemented and pass with results that confirm the expected isolation level (minimum: read committed).
115+
116+
### Driver
117+
118+
- LDBC SNB driver version
119+
120+
### Queries
121+
122+
- What language were the operations implemented in?
123+
- Complex reads
124+
- Short reads
125+
- Updates
126+
127+
### Benchmark parameters
128+
129+
- Total compression ratio (TCR) with at least 95% on-time queries
130+
- Total compression ratio (TCR) with 100% on-time queries (optional)
131+
- Operation count
132+
- Number of read threads
133+
- Number of write threads
134+
135+
### Expected performance
136+
137+
Information about the expected performance helps the auditor quickly identify potential issues with the setup.
138+
139+
Please provide the following information **for each scale factor**.
140+
141+
- Expected loading time
142+
- Expected throughput (ops/seconds)
143+
- Expected warmup time (minimum 30 minutes, maximum 35 minutes)
144+
- Expected benchmark time (minimum 2 hours, maximum 2 hours 15 minutes)
145+
146+
## Dissemination of the results
147+
148+
Upon a successful audit, we can announce the results through the following communication channels.
149+
150+
- Audited SNB benchmarks site: <https://ldbcouncil.org/benchmarks/snb>
151+
- LDBC's Twitter: <https://twitter.com/ldbcouncil>
152+
- We will post a tweet such as: "Congratulations to @XXX on breaking the record on the LDBC Social Network Benchmark's Interactive workload for scale factor YYY with ZZZ operations/second"
153+
- We can at-mention your company in the tweet announcing the results.
154+
- We can display the logo of your company/product in the tweet.
155+
- LDBC's board of directors mailing list
156+
157+
Please let us know your preferences for announcing the results:
158+
159+
- Is there an embargo for announcing the results? If so, when can we make the announcement?
160+
- What is the Twitter handle of your company?
161+
- Please attach a high-resolution logo of your company
162+
- Please attach a high-resolution logo of your product (if it has its individual logo)

0 commit comments

Comments
 (0)