-
Notifications
You must be signed in to change notification settings - Fork 111
Include stats for each planner phase in EXPLAIN metrics #3512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
hazefully
wants to merge
9
commits into
FoundationDB:main
Choose a base branch
from
hazefully:planner-phase-metrics
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
b71ab35
to
7e8cc8e
Compare
8814a6c
to
da8ae22
Compare
da8ae22
to
668c47e
Compare
This commit introduces new StoreTimer.Events that record how long each phase (rewriting, planning) of the Cascades planner take. To achieve this, the Relational layer passes the instance of StoreTimer within the current transaction to the Cascades planner upon creation, which it makes use of by recording the time a phase took using a new Cascades task, `FinalizePlannerPhase`, which simply calls record on the StoreTimer instance (if it is not `null`) with the total time the planner phase took. The `FinalizePlannerPhase` task is schedueled to be executed by the `InitiatePlannerPhase` task, ensuring that it is the last task that is run for that phase. In addition, the logs for queries (which happen if debug logging is enabled or if a query is too slow) is enhanced to include the durations for the two planner phases.
… planner" This reverts commit 7e8cc8e.
This commit splits the com.apple.foundationdb.record.query.plan.cascades.debug.Debugger interface into two separate interfaces (StatsDebugger and SymbolDebugger) which both extend the base Debugger interface, in order to ensure that operations related to recording Debugger events (with the purpose of calculating stats based on the profiling of these events) make use of the StatsDebugger interface, and operations related to keeping a registry of all Symbols used within the planner for purposes of associating them with friendly names make use of the SymbolDebugger interface. In addition to splitting the Debugger interface, the State class is also split into two classes, one that keeps of the Debugger events and stats around them (EventState) and one that keeps track of the current Symbols seen during planning (SymbolTables).
This commit modifies the EventState class and the StatsMaps that it produces to group Debugger events by the planner phase it was emitted during to enable the ability of getting stats for each planner phase separately. Given that some Debugger events (e.g. InsertIntoMemoEvent) do not store the planner phase they were emitted in (as it not visible to the code that emits them), these events are stored separately and can be retrieved from the StatsMaps (in addition to the ability of retrieving all events, regardless of the planner phase that they were emitted in).
This commit adds a new implmentation of the Cascades Debugger interface, which only keeps track of statstics around debugger events and doesn't support keeping track of symbol tables. This new implementation should be safe to use outside of tests since it should have a minimal impact on the planner's performance.
668c47e
to
2ee4763
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds new stats to the
PLANNER_METRICS
struct produced forEXPLAIN
SQL queries which reflect the number of tasks in each planner phase and the total time the tasks for each phase took to execute. It also adds a new lightweight Cascades debugger implementation which only keeps tracks of debugger events.To achieve this, the PR includes the following changes:
2582ff8: This commit splits the
cascades.debug.Debugger
interface into twoseparate interfaces (
StatsDebugger
andSymbolDebugger
) which both extend the base Debugger interface, inorder to ensure that operations related to recording Debugger events (with the purpose of calculating stats
based on the profiling of these events) make use of the
StatsDebugger
interface, and similarly operations related toregistering different objects used within the planner make use of the SymbolDebugger interface. The
State
class which kept track of symbols and debugger events is also split into two different classes:EventState
andSymbolTables
.bdf2e16: This commit extracts the functionality of rendering the different stats for debugger events from the
State
class to a separateStatsViewer
class.cf9848b and a9edd0f: These two commits include the main work of grouping debugger events by the planner phase they were emitted in, and exposing that grouping in the
EXPLAIN
planner metrics. As a result of this grouping the "Event Profiling" rendered by theStatsViewer
class is now split by phase as well:In addition, the result set of an
EXPLAIN
query contains the expected planner metrics:2ee4763: This commit adds the new lightweight debugger implementation that only implements the
StatsDebugger
interface.Note: This PR previously took a different approach to expose the duration each planner phase took (12b5817), but after offline discussions we decided to make use of the existing Cascades debugger to expose these metrics.