Merge pull request #12620 from ethereum/assemblyAnnotation

Memory-safety annotation for inline assembly.

Author: chriseth

Changelog.md

@@ -1,6 +1,7 @@
 ### 0.8.13 (unreleased)
 
 Language Features:
+ * General: Allow annotating inline assembly as memory-safe to allow optimizations and stack limit evasion that rely on respecting Solidity's memory model.
 
 
 Compiler Features:

docs/assembly.rst

@@ -228,6 +228,11 @@ of their block is reached.
 Conventions in Solidity
 -----------------------
 
+.. _assembly-typed-variables:
+
+Values of Typed Variables
+=========================
+
 In contrast to EVM assembly, Solidity has types which are narrower than 256 bits,
 e.g. ``uint24``. For efficiency, most arithmetic operations ignore the fact that
 types can be shorter than 256
@@ -237,6 +242,11 @@ This means that if you access such a variable
 from within inline assembly, you might have to manually clean the higher-order bits
 first.
 
+.. _assembly-memory-management:
+
+Memory Management
+=================
+
 Solidity manages memory in the following way. There is a "free memory pointer"
 at position ``0x40`` in memory. If you want to allocate memory, use the memory
 starting from where this pointer points at and update it.
@@ -268,3 +278,89 @@ first slot of the array and followed by the array elements.
     Statically-sized memory arrays do not have a length field, but it might be added later
     to allow better convertibility between statically- and dynamically-sized arrays, so
     do not rely on this.
+
+Memory Safety
+=============
+
+Without the use of inline assembly, the compiler can rely on memory to remain in a well-defined
+state at all times. This is especially relevant for :ref:`the new code generation pipeline via Yul IR <ir-breaking-changes>`:
+this code generation path can move local variables from stack to memory to avoid stack-too-deep errors and
+perform additional memory optimizations, if it can rely on certain assumptions about memory use.
+
+While we recommend to always respect Solidity's memory model, inline assembly allows you to use memory
+in an incompatible way. Therefore, moving stack variables to memory and additional memory optimizations are,
+by default, disabled in the presence of any inline assembly block that contains a memory operation or assigns
+to solidity variables in memory.
+
+However, you can specifically annotate an assembly block to indicate that it in fact respects Solidity's memory
+model as follows:
+
+.. code-block:: solidity
+
+    /// @solidity memory-safe-assembly
+    assembly {
+        ...
+    }
+
+In particular, a memory-safe assembly block may only access the following memory ranges:
+
+- Memory allocated by yourself using a mechanism like the ``allocate`` function described above.
+- Memory allocated by Solidity, e.g. memory within the bounds of a memory array you reference.
+- The scratch space between memory offset 0 and 64 mentioned above.
+- Temporary memory that is located *after* the value of the free memory pointer at the beginning of the assembly block,
+  i.e. memory that is "allocated" at the free memory pointer without updating the free memory pointer.
+
+Furthermore, if the assembly block assigns to Solidity variables in memory, you need to assure that accesses to
+the Solidity variables only access these memory ranges.
+
+Since this is mainly about the optimizer, these restrictions still need to be followed, even if the assembly block
+reverts or terminates. As an example, the following assembly snippet is not memory safe:
+
+.. code-block:: solidity
+
+    assembly {
+      returndatacopy(0, 0, returndatasize())
+      revert(0, returndatasize())
+    }
+
+But the following is:
+
+.. code-block:: solidity
+
+    /// @solidity memory-safe-assembly
+    assembly {
+      let p := mload(0x40)
+      returndatacopy(p, 0, returndatasize())
+      revert(p, returndatasize())
+    }
+
+Note that you do not need to update the free memory pointer if there is no following allocation,
+but you can only use memory starting from the current offset given by the free memory pointer.
+
+If the memory operations use a length of zero, it is also fine to just use any offset (not only if it falls into the scratch space):
+
+.. code-block:: solidity
+
+    /// @solidity memory-safe-assembly
+    assembly {
+      revert(0, 0)
+    }
+
+Note that not only memory operations in inline assembly itself can be memory-unsafe, but also assignments to
+solidity variables of reference type in memory. For example the following is not memory-safe:
+
+.. code-block:: solidity
+
+    bytes memory x;
+    assembly {
+      x := 0x40
+    }
+    x[0x20] = 0x42;
+
+Inline assembly that neither involves any operations that access memory nor assigns to any solidity variables
+in memory is automatically considered memory-safe and does not need to be annotated.
+
+.. warning::
+    It is your responsibility to make sure that the assembly actually satisfies the memory model. If you annotate
+    an assembly block as memory-safe, but violate one of the memory assumptions, this **will** lead to incorrect and
+    undefined behaviour that cannot easily be discovered by testing.

docs/ir-breaking-changes.rst

@@ -1,6 +1,8 @@
 
 .. index: ir breaking changes
 
+.. _ir-breaking-changes:
+
 *********************************
 Solidity IR-based Codegen Changes
 *********************************

libsolidity/analysis/DocStringTagParser.cpp

@@ -30,6 +30,7 @@
 #include <liblangutil/Common.h>
 
 #include <range/v3/algorithm/any_of.hpp>
+#include <range/v3/view/filter.hpp>
 
 #include <boost/algorithm/string.hpp>
 
@@ -162,6 +163,71 @@ bool DocStringTagParser::visit(ErrorDefinition const& _error)
 	return true;
 }
 
+bool DocStringTagParser::visit(InlineAssembly const& _assembly)
+{
+	if (!_assembly.documentation())
+		return true;
+	StructuredDocumentation documentation{-1, _assembly.location(), _assembly.documentation()};
+	ErrorList errors;
+	ErrorReporter errorReporter{errors};
+	auto docTags = DocStringParser{documentation, errorReporter}.parse();
+
+	if (!errors.empty())
+	{
+		SecondarySourceLocation ssl;
+		for (auto const& error: errors)
+			if (error->comment())
+				ssl.append(
+					*error->comment(),
+					_assembly.location()
+				);
+		m_errorReporter.warning(
+			7828_error,
+			_assembly.location(),
+			"Inline assembly has invalid NatSpec documentation.",
+			ssl
+		);
+	}
+
+	for (auto const& [tagName, tagValue]: docTags)
+	{
+		if (tagName == "solidity")
+		{
+			vector<string> values;
+			boost::split(values, tagValue.content, isWhiteSpace);
+
+			set<string> valuesSeen;
+			set<string> duplicates;
+			for (auto const& value: values | ranges::views::filter(not_fn(&string::empty)))
+				if (valuesSeen.insert(value).second)
+				{
+					if (value == "memory-safe-assembly")
+						_assembly.annotation().markedMemorySafe = true;
+					else
+						m_errorReporter.warning(
+							8787_error,
+							_assembly.location(),
+							"Unexpected value for @solidity tag in inline assembly: " + value
+						);
+				}
+				else if (duplicates.insert(value).second)
+					m_errorReporter.warning(
+						4377_error,
+						_assembly.location(),
+						"Value for @solidity tag in inline assembly specified multiple times: " + value
+					);
+		}
+		else
+			m_errorReporter.warning(
+				6269_error,
+				_assembly.location(),
+				"Unexpected NatSpec tag \"" + tagName + "\" with value \"" + tagValue.content + "\" in inline assembly."
+			);
+	}
+
+	return true;
+}
+
 void DocStringTagParser::checkParameters(
 	CallableDeclaration const& _callable,
 	StructurallyDocumented const& _node,

libsolidity/analysis/DocStringTagParser.h

@@ -48,6 +48,7 @@ class DocStringTagParser: private ASTConstVisitor
 	bool visit(ModifierDefinition const& _modifier) override;
 	bool visit(EventDefinition const& _event) override;
 	bool visit(ErrorDefinition const& _error) override;
+	bool visit(InlineAssembly const& _assembly) override;
 
 	void checkParameters(
 		CallableDeclaration const& _callable,

libsolidity/analysis/TypeChecker.cpp

@@ -763,6 +763,7 @@ void TypeChecker::endVisit(FunctionTypeName const& _funType)
 
 bool TypeChecker::visit(InlineAssembly const& _inlineAssembly)
 {
+	bool lvalueAccessToMemoryVariable = false;
 	// External references have already been resolved in a prior stage and stored in the annotation.
 	// We run the resolve step again regardless.
 	yul::ExternalIdentifierAccess::Resolver identifierAccess = [&](
@@ -787,6 +788,8 @@ bool TypeChecker::visit(InlineAssembly const& _inlineAssembly)
 		if (auto var = dynamic_cast<VariableDeclaration const*>(declaration))
 		{
 			solAssert(var->type(), "Expected variable type!");
+			if (_context == yul::IdentifierContext::LValue && var->type()->dataStoredIn(DataLocation::Memory))
+				lvalueAccessToMemoryVariable = true;
 			if (var->immutable())
 			{
 				m_errorReporter.typeError(3773_error, nativeLocationOf(_identifier), "Assembly access to immutable variables is not supported.");
@@ -974,8 +977,11 @@ bool TypeChecker::visit(InlineAssembly const& _inlineAssembly)
 		identifierAccess
 	);
 	if (!analyzer.analyze(_inlineAssembly.operations()))
-		return false;
-	return true;
+		solAssert(m_errorReporter.hasErrors());
+	_inlineAssembly.annotation().hasMemoryEffects =
+		lvalueAccessToMemoryVariable ||
+		(analyzer.sideEffects().memory != yul::SideEffects::None);
+	return false;
 }
 
 bool TypeChecker::visit(IfStatement const& _ifStatement)

libsolidity/ast/ASTAnnotations.h

@@ -220,6 +220,10 @@ struct InlineAssemblyAnnotation: StatementAnnotation
 	std::map<yul::Identifier const*, ExternalIdentifierInfo> externalReferences;
 	/// Information generated during analysis phase.
 	std::shared_ptr<yul::AsmAnalysisInfo> analysisInfo;
+	/// True, if the assembly block was annotated to be memory-safe.
+	bool markedMemorySafe = false;
+	/// True, if the assembly block involves any memory opcode or assigns to variables in memory.
+	SetOnce<bool> hasMemoryEffects;
 };
 
 struct BlockAnnotation: StatementAnnotation, ScopableAnnotation

libsolidity/codegen/ir/IRGenerationContext.h

@@ -160,8 +160,8 @@ class IRGenerationContext
 
 	std::set<ContractDefinition const*, ASTNode::CompareByID>& subObjectsCreated() { return m_subObjects; }
 
-	bool inlineAssemblySeen() const { return m_inlineAssemblySeen; }
-	void setInlineAssemblySeen() { m_inlineAssemblySeen = true; }
+	bool memoryUnsafeInlineAssemblySeen() const { return m_memoryUnsafeInlineAssemblySeen; }
+	void setMemoryUnsafeInlineAssemblySeen() { m_memoryUnsafeInlineAssemblySeen = true; }
 
 	/// @returns the runtime ID to be used for the function in the dispatch routine
 	/// and for internal function pointers.
@@ -202,8 +202,8 @@ class IRGenerationContext
 	/// Whether to use checked or wrapping arithmetic.
 	Arithmetic m_arithmetic = Arithmetic::Checked;
 
-	/// Flag indicating whether any inline assembly block was seen.
-	bool m_inlineAssemblySeen = false;
+	/// Flag indicating whether any memory-unsafe inline assembly block was seen.
+	bool m_memoryUnsafeInlineAssemblySeen = false;
 
 	/// Function definitions queued for code generation. They're the Solidity functions whose calls
 	/// were discovered by the IR generator during AST traversal.

libsolidity/codegen/ir/IRGenerator.cpp

@@ -213,8 +213,8 @@ string IRGenerator::generate(
 	t("subObjects", subObjectSources(m_context.subObjectsCreated()));
 
 	// This has to be called only after all other code generation for the creation object is complete.
-	bool creationInvolvesAssembly = m_context.inlineAssemblySeen();
-	t("memoryInitCreation", memoryInit(!creationInvolvesAssembly));
+	bool creationInvolvesMemoryUnsafeAssembly = m_context.memoryUnsafeInlineAssemblySeen();
+	t("memoryInitCreation", memoryInit(!creationInvolvesMemoryUnsafeAssembly));
 	t("useSrcMapCreation", formatUseSrcMap(m_context));
 
 	resetContext(_contract, ExecutionContext::Deployed);
@@ -239,8 +239,8 @@ string IRGenerator::generate(
 	t("useSrcMapDeployed", formatUseSrcMap(m_context));
 
 	// This has to be called only after all other code generation for the deployed object is complete.
-	bool deployedInvolvesAssembly = m_context.inlineAssemblySeen();
-	t("memoryInitDeployed", memoryInit(!deployedInvolvesAssembly));
+	bool deployedInvolvesMemoryUnsafeAssembly = m_context.memoryUnsafeInlineAssemblySeen();
+	t("memoryInitDeployed", memoryInit(!deployedInvolvesMemoryUnsafeAssembly));
 
 	solAssert(_contract.annotation().creationCallGraph->get() != nullptr, "");
 	solAssert(_contract.annotation().deployedCallGraph->get() != nullptr, "");

libsolidity/codegen/ir/IRGeneratorForStatements.cpp

@@ -2138,7 +2138,8 @@ void IRGeneratorForStatements::endVisit(MemberAccess const& _memberAccess)
 bool IRGeneratorForStatements::visit(InlineAssembly const& _inlineAsm)
 {
 	setLocation(_inlineAsm);
-	m_context.setInlineAssemblySeen();
+	if (*_inlineAsm.annotation().hasMemoryEffects && !_inlineAsm.annotation().markedMemorySafe)
+		m_context.setMemoryUnsafeInlineAssemblySeen();
 	CopyTranslate bodyCopier{_inlineAsm.dialect(), m_context, _inlineAsm.annotation().externalReferences};
 
 	yul::Statement modified = bodyCopier(_inlineAsm.operations());