Merge pull request #13376 from ethereum/make-hardcoded-parts-of-optimizer-sequence-configurable

Author: nikola-matic

Changelog.md

@@ -4,6 +4,7 @@ Language Features:
 
 
 Compiler Features:
+* Yul Optimizer: Allow replacing the previously hard-coded cleanup sequence by specifying custom steps after a colon delimiter (``:``) in the sequence string.
 
 
 Bugfixes:

docs/internals/optimizer.rst

@@ -281,60 +281,85 @@ The following transformation steps are the main components:
 - Redundant Assign Eliminator
 - Full Inliner
 
+.. _optimizer-steps:
+
 Optimizer Steps
 ---------------
 
 This is a list of all steps the Yul-based optimizer sorted alphabetically. You can find more information
 on the individual steps and their sequence below.
 
-- :ref:`block-flattener`.
-- :ref:`circular-reference-pruner`.
-- :ref:`common-subexpression-eliminator`.
-- :ref:`conditional-simplifier`.
-- :ref:`conditional-unsimplifier`.
-- :ref:`control-flow-simplifier`.
-- :ref:`dead-code-eliminator`.
-- :ref:`equal-store-eliminator`.
-- :ref:`equivalent-function-combiner`.
-- :ref:`expression-joiner`.
-- :ref:`expression-simplifier`.
-- :ref:`expression-splitter`.
-- :ref:`for-loop-condition-into-body`.
-- :ref:`for-loop-condition-out-of-body`.
-- :ref:`for-loop-init-rewriter`.
-- :ref:`expression-inliner`.
-- :ref:`full-inliner`.
-- :ref:`function-grouper`.
-- :ref:`function-hoister`.
-- :ref:`function-specializer`.
-- :ref:`literal-rematerialiser`.
-- :ref:`load-resolver`.
-- :ref:`loop-invariant-code-motion`.
-- :ref:`redundant-assign-eliminator`.
-- :ref:`reasoning-based-simplifier`.
-- :ref:`rematerialiser`.
-- :ref:`SSA-reverser`.
-- :ref:`SSA-transform`.
-- :ref:`structural-simplifier`.
-- :ref:`unused-function-parameter-pruner`.
-- :ref:`unused-pruner`.
-- :ref:`var-decl-initializer`.
+============ ===============================
+Abbreviation Full name
+============ ===============================
+``f``        :ref:`block-flattener`
+``l``        :ref:`circular-reference-pruner`
+``c``        :ref:`common-subexpression-eliminator`
+``C``        :ref:`conditional-simplifier`
+``U``        :ref:`conditional-unsimplifier`
+``n``        :ref:`control-flow-simplifier`
+``D``        :ref:`dead-code-eliminator`
+``E``        :ref:`equal-store-eliminator`
+``v``        :ref:`equivalent-function-combiner`
+``e``        :ref:`expression-inliner`
+``j``        :ref:`expression-joiner`
+``s``        :ref:`expression-simplifier`
+``x``        :ref:`expression-splitter`
+``I``        :ref:`for-loop-condition-into-body`
+``O``        :ref:`for-loop-condition-out-of-body`
+``o``        :ref:`for-loop-init-rewriter`
+``i``        :ref:`full-inliner`
+``g``        :ref:`function-grouper`
+``h``        :ref:`function-hoister`
+``F``        :ref:`function-specializer`
+``T``        :ref:`literal-rematerialiser`
+``L``        :ref:`load-resolver`
+``M``        :ref:`loop-invariant-code-motion`
+``r``        :ref:`redundant-assign-eliminator`
+``R``        :ref:`reasoning-based-simplifier` - highly experimental
+``m``        :ref:`rematerialiser`
+``V``        :ref:`SSA-reverser`
+``a``        :ref:`SSA-transform`
+``t``        :ref:`structural-simplifier`
+``p``        :ref:`unused-function-parameter-pruner`
+``u``        :ref:`unused-pruner`
+``d``        :ref:`var-decl-initializer`
+============ ===============================
+
+Some steps depend on properties ensured by ``BlockFlattener``, ``FunctionGrouper``, ``ForLoopInitRewriter``.
+For this reason the Yul optimizer always applies them before applying any steps supplied by the user.
+
+The ReasoningBasedSimplifier is an optimizer step that is currently not enabled
+in the default set of steps. It uses an SMT solver to simplify arithmetic expressions
+and boolean conditions. It has not received thorough testing or validation yet and can produce
+non-reproducible results, so please use with care!
 
 Selecting Optimizations
 -----------------------
 
-By default the optimizer applies its predefined sequence of optimization steps to
-the generated assembly. You can override this sequence and supply your own using
-the ``--yul-optimizations`` option:
+By default the optimizer applies its predefined sequence of optimization steps to the generated assembly.
+You can override this sequence and supply your own using the ``--yul-optimizations`` option:
 
 .. code-block:: bash
 
-    solc --optimize --ir-optimized --yul-optimizations 'dhfoD[xarrscLMcCTU]uljmul'
+    solc --optimize --ir-optimized --yul-optimizations 'dhfoD[xarrscLMcCTU]uljmul:fDnTOc'
+
+The order of steps is significant and affects the quality of the output.
+Moreover, applying a step may uncover new optimization opportunities for others that were already applied,
+so repeating steps is often beneficial.
 
 The sequence inside ``[...]`` will be applied multiple times in a loop until the Yul code
 remains unchanged or until the maximum number of rounds (currently 12) has been reached.
+Brackets (``[]``) may be used multiple times in a sequence, but can not be nested.
+
+An important thing to note, is that there are some hardcoded steps that are always run before and after the
+user-supplied sequence, or the default sequence if one was not supplied by the user.
 
-Available abbreviations are listed in the :ref:`Yul optimizer docs <optimization-step-sequence>`.
+The cleanup sequence delimiter ``:`` is optional, and is used to supply a custom cleanup sequence
+in order to replace the default one. If omitted, the optimizer will simply apply the default cleanup
+sequence. In addition, the delimiter may be placed at the beginning of the user-supplied sequence,
+which will result in the optimization sequence being empty, whereas conversely, if placed at the end of
+the sequence, will be treated as an empty cleanup sequence.
 
 Preprocessing
 -------------

docs/yul.rst

@@ -1245,65 +1245,8 @@ In Solidity mode, the Yul optimizer is activated together with the regular optim
 Optimization Step Sequence
 --------------------------
 
-By default the Yul optimizer applies its predefined sequence of optimization steps to the generated assembly.
-You can override this sequence and supply your own using the ``--yul-optimizations`` option:
-
-.. code-block:: sh
-
-    solc --optimize --ir-optimized --yul-optimizations 'dhfoD[xarrscLMcCTU]uljmul'
-
-The order of steps is significant and affects the quality of the output.
-Moreover, applying a step may uncover new optimization opportunities for others that were already
-applied so repeating steps is often beneficial.
-By enclosing part of the sequence in square brackets (``[]``) you tell the optimizer to repeatedly
-apply that part until it no longer improves the size of the resulting assembly.
-You can use brackets multiple times in a single sequence but they cannot be nested.
-
-The following optimization steps are available:
-
-============ ===============================
-Abbreviation Full name
-============ ===============================
-``f``        ``BlockFlattener``
-``l``        ``CircularReferencesPruner``
-``c``        ``CommonSubexpressionEliminator``
-``C``        ``ConditionalSimplifier``
-``U``        ``ConditionalUnsimplifier``
-``n``        ``ControlFlowSimplifier``
-``D``        ``DeadCodeEliminator``
-``v``        ``EquivalentFunctionCombiner``
-``e``        ``ExpressionInliner``
-``j``        ``ExpressionJoiner``
-``s``        ``ExpressionSimplifier``
-``x``        ``ExpressionSplitter``
-``I``        ``ForLoopConditionIntoBody``
-``O``        ``ForLoopConditionOutOfBody``
-``o``        ``ForLoopInitRewriter``
-``i``        ``FullInliner``
-``g``        ``FunctionGrouper``
-``h``        ``FunctionHoister``
-``F``        ``FunctionSpecializer``
-``T``        ``LiteralRematerialiser``
-``L``        ``LoadResolver``
-``M``        ``LoopInvariantCodeMotion``
-``r``        ``RedundantAssignEliminator``
-``R``        ``ReasoningBasedSimplifier`` - highly experimental
-``m``        ``Rematerialiser``
-``V``        ``SSAReverser``
-``a``        ``SSATransform``
-``t``        ``StructuralSimplifier``
-``u``        ``UnusedPruner``
-``p``        ``UnusedFunctionParameterPruner``
-``d``        ``VarDeclInitializer``
-============ ===============================
-
-Some steps depend on properties ensured by ``BlockFlattener``, ``FunctionGrouper``, ``ForLoopInitRewriter``.
-For this reason the Yul optimizer always applies them before applying any steps supplied by the user.
-
-The ReasoningBasedSimplifier is an optimizer step that is currently not enabled
-in the default set of steps. It uses an SMT solver to simplify arithmetic expressions
-and boolean conditions. It has not received thorough testing or validation yet and can produce
-non-reproducible results, so please use with care!
+Detailed information regrading the optimization sequence as well a list of abbreviations is
+available in the :ref:`optimizer docs <optimizer-steps>`.
 
 .. _erc20yul:
 

libsolidity/codegen/CompilerContext.cpp

@@ -547,6 +547,7 @@ void CompilerContext::optimizeYul(yul::Object& _object, yul::EVMDialect const& _
 		_object,
 		_optimiserSettings.optimizeStackAllocation,
 		_optimiserSettings.yulOptimiserSteps,
+		_optimiserSettings.yulOptimiserCleanupSteps,
 		isCreation? nullopt : make_optional(_optimiserSettings.expectedExecutionsPerDeployment),
 		_externalIdentifiers
 	);

libsolidity/interface/CompilerStack.cpp

@@ -1539,7 +1539,7 @@ string CompilerStack::createMetadata(Contract const& _contract, bool _forIR) con
 		{
 			details["yulDetails"] = Json::objectValue;
 			details["yulDetails"]["stackAllocation"] = m_optimiserSettings.optimizeStackAllocation;
-			details["yulDetails"]["optimizerSteps"] = m_optimiserSettings.yulOptimiserSteps;
+			details["yulDetails"]["optimizerSteps"] = m_optimiserSettings.yulOptimiserSteps + ":" + m_optimiserSettings.yulOptimiserCleanupSteps;
 		}
 
 		meta["settings"]["optimizer"]["details"] = std::move(details);

libsolidity/interface/OptimiserSettings.h

@@ -59,6 +59,8 @@ struct OptimiserSettings
 		"]"
 		"jmul[jul] VcTOcul jmul";      // Make source short and pretty
 
+	static char constexpr DefaultYulOptimiserCleanupSteps[] = "fDnTOc";
+
 	/// No optimisations at all - not recommended.
 	static OptimiserSettings none()
 	{
@@ -146,6 +148,10 @@ struct OptimiserSettings
 	/// them just by setting this to an empty string. Set @a runYulOptimiser to false if you want
 	/// no optimisations.
 	std::string yulOptimiserSteps = DefaultYulOptimiserSteps;
+	/// Sequence of clean-up optimisation steps after yulOptimiserSteps is run. Note that if the string
+	/// is left empty, there will still be hard-coded optimisation steps that will run regardless.
+	/// Set @a runYulOptimiser to false if you want no optimisations.
+	std::string yulOptimiserCleanupSteps = DefaultYulOptimiserCleanupSteps;
 	/// This specifies an estimate on how often each opcode in this assembly will be executed,
 	/// i.e. use a small value to optimise for size and a large value to optimise for runtime gas usage.
 	size_t expectedExecutionsPerDeployment = 200;

libsolidity/interface/StandardCompiler.cpp

@@ -472,7 +472,7 @@ std::optional<Json::Value> checkOptimizerDetail(Json::Value const& _details, std
 	return {};
 }
 
-std::optional<Json::Value> checkOptimizerDetailSteps(Json::Value const& _details, std::string const& _name, string& _setting)
+std::optional<Json::Value> checkOptimizerDetailSteps(Json::Value const& _details, std::string const& _name, string& _optimiserSetting, string& _cleanupSetting)
 {
 	if (_details.isMember(_name))
 	{
@@ -490,7 +490,14 @@ std::optional<Json::Value> checkOptimizerDetailSteps(Json::Value const& _details
 				);
 			}
 
-			_setting = _details[_name].asString();
+			string const fullSequence = _details[_name].asString();
+			auto const delimiterPos = fullSequence.find(":");
+			_optimiserSetting = fullSequence.substr(0, delimiterPos);
+
+			if (delimiterPos != string::npos)
+				_cleanupSetting = fullSequence.substr(delimiterPos + 1);
+			else
+				solAssert(_cleanupSetting == OptimiserSettings::DefaultYulOptimiserCleanupSteps);
 		}
 		else
 			return formatFatalError("JSONError", "\"settings.optimizer.details." + _name + "\" must be a string");
@@ -616,7 +623,7 @@ std::variant<OptimiserSettings, Json::Value> parseOptimizerSettings(Json::Value
 				return *result;
 			if (auto error = checkOptimizerDetail(details["yulDetails"], "stackAllocation", settings.optimizeStackAllocation))
 				return *error;
-			if (auto error = checkOptimizerDetailSteps(details["yulDetails"], "optimizerSteps", settings.yulOptimiserSteps))
+			if (auto error = checkOptimizerDetailSteps(details["yulDetails"], "optimizerSteps", settings.yulOptimiserSteps, settings.yulOptimiserCleanupSteps))
 				return *error;
 		}
 	}

libyul/YulStack.cpp

@@ -209,6 +209,7 @@ void YulStack::optimize(Object& _object, bool _isCreation)
 		_object,
 		m_optimiserSettings.optimizeStackAllocation,
 		m_optimiserSettings.yulOptimiserSteps,
+		m_optimiserSettings.yulOptimiserCleanupSteps,
 		_isCreation ? nullopt : make_optional(m_optimiserSettings.expectedExecutionsPerDeployment),
 		{}
 	);

libyul/optimiser/Suite.cpp

@@ -93,6 +93,7 @@ void OptimiserSuite::run(
 	Object& _object,
 	bool _optimizeStackAllocation,
 	string_view _optimisationSequence,
+	string_view _optimisationCleanupSequence,
 	optional<size_t> _expectedExecutionsPerDeployment,
 	set<YulString> const& _externallyUsedIdentifiers
 )
@@ -139,7 +140,13 @@ void OptimiserSuite::run(
 			_optimizeStackAllocation,
 			stackCompressorMaxIterations
 		);
-	suite.runSequence("fDnTOc g", ast);
+
+	// Run the user-supplied clean up sequence
+	suite.runSequence(_optimisationCleanupSequence, ast);
+	// Hard-coded FunctionGrouper step is used to bring the AST into a canonical form required by the StackCompressor
+	// and StackLimitEvader. This is hard-coded as the last step, as some previously executed steps may break the
+	// aforementioned form, thus causing the StackCompressor/StackLimitEvader to throw.
+	suite.runSequence("g", ast);
 
 	if (evmDialect)
 	{
@@ -296,6 +303,7 @@ map<char, string> const& OptimiserSuite::stepAbbreviationToNameMap()
 void OptimiserSuite::validateSequence(string_view _stepAbbreviations)
 {
 	int8_t nestingLevel = 0;
+	int8_t colonDelimiters = 0;
 	for (char abbreviation: _stepAbbreviations)
 		switch (abbreviation)
 		{
@@ -310,6 +318,11 @@ void OptimiserSuite::validateSequence(string_view _stepAbbreviations)
 			nestingLevel--;
 			assertThrow(nestingLevel >= 0, OptimizerException, "Unbalanced brackets");
 			break;
+		case ':':
+			++colonDelimiters;
+			assertThrow(nestingLevel == 0, OptimizerException, "Cleanup sequence delimiter cannot be placed inside the brackets");
+			assertThrow(colonDelimiters <= 1, OptimizerException, "Too many cleanup sequence delimiters");
+			break;
 		default:
 		{
 			yulAssert(

libyul/optimiser/Suite.h

@@ -51,7 +51,7 @@ class OptimiserSuite
 
 	/// Special characters that do not represent optimiser steps but are allowed in abbreviation sequences.
 	/// Some of them (like whitespace) are ignored, others (like brackets) are a part of the syntax.
-	static constexpr char NonStepAbbreviations[] = " \n[]";
+	static constexpr char NonStepAbbreviations[] = " \n[]:";
 
 	enum class Debug
 	{
@@ -68,6 +68,7 @@ class OptimiserSuite
 		Object& _object,
 		bool _optimizeStackAllocation,
 		std::string_view _optimisationSequence,
+		std::string_view _optimisationCleanupSequence,
 		std::optional<size_t> _expectedExecutionsPerDeployment,
 		std::set<YulString> const& _externallyUsedIdentifiers = {}
 	);