Merge pull request #11545 from ethereum/stripping-base-path-from-cli

Stripping base path from CLI paths

Author: cameel

Changelog.md

@@ -4,6 +4,7 @@ Language Features:
 
 
 Compiler Features:
+ * Commandline Interface: Normalize paths specified on the command line and make them relative for files located inside base path.
  * Immutable variables can be read at construction time once they are initialized.
  * SMTChecker: Support low level ``call`` as external calls to unknown code.
  * SMTChecker: Add constraints to better correlate ``address(this).balance`` and ``msg.value``.

docs/path-resolution.rst

@@ -71,8 +71,10 @@ The initial content of the VFS depends on how you invoke the compiler:
 
        solc contract.sol /usr/local/dapp-bin/token.sol
 
-   The source unit name of a file loaded this way is simply the specified path after shell expansion
-   and with platform-specific separators converted to forward slashes.
+   The source unit name of a file loaded this way is constructed by converting its path to a
+   canonical form and making it relative to the base path if it is located inside.
+   See :ref:`Base Path Normalization and Stripping <base-path-normalization-and-stripping>` for
+   a detailed description of this process.
 
    .. index:: standard JSON
 
@@ -309,9 +311,67 @@ When the source unit name is a relative path, this results in the file being loo
 directory the compiler has been invoked from.
 It is also the only value that results in absolute paths in source unit names being actually
 interpreted as absolute paths on disk.
+If the base path itself is relative, it is interpreted as relative to the current working directory
+of the compiler.
+
+.. _base-path-normalization-and-stripping:
+
+Base Path Normalization and Stripping
+-------------------------------------
+
+On the command line the compiler behaves just as you would expect from any other program:
+it accepts paths in a format native to the platform and relative paths are relative to the current
+working directory.
+The source unit names assigned to files whose paths are specified on the command line, however,
+should not change just because the project is being compiled on a different platform or because the
+compiler happens to have been invoked from a different directory.
+To achieve this, paths to source files coming from the command line must be converted to a canonical
+form, and, if possible, made relative to the base path.
+
+The normalization rules are as follows:
+
+- If a path is relative, it is made absolute by prepending the current working directory to it.
+- Internal ``.`` and ``..`` segments are collapsed.
+- Platform-specific path separators are replaced with forward slashes.
+- Sequences of multiple consecutive path separators are squashed into a single separator (unless
+  they are the leading slashes of an `UNC path <https://en.wikipedia.org/wiki/Path_(computing)#UNC>`_).
+- If the path includes a root name (e.g. a drive letter on Windows) and the root is the same as the
+  root of the current working directory, the root is replaced with ``/``.
+- Symbolic links in the path are **not** resolved.
+
+  - The only exception is the path to the current working directory prepended to relative paths in
+    the process of making them absolute.
+    On some platforms the working directory is reported always with symbolic links resolved so for
+    consistency the compiler resolves them everywhere.
+
+- The original case of the path is preserved even if the filesystem is case-insensitive but
+  `case-preserving <https://en.wikipedia.org/wiki/Case_preservation>`_ and the actual case on
+  disk is different.
 
-If the base path itself is relative, it is also interpreted as relative to the current working
-directory of the compiler.
+.. note::
+
+    There are situations where paths cannot be made platform-independent.
+    For example on Windows the compiler can avoid using drive letters by referring to the root
+    directory of the current drive as ``/`` but drive letters are still necessary for paths leading
+    to other drives.
+    You can avoid such situations by ensuring that all the files are available within a single
+    directory tree on the same drive.
+
+Once canonicalized, the base path is stripped from all source file paths that start with it.
+If the base path is empty or not specified, it is treated as if it was equal to the path to the
+current working directory (with all symbolic links resolved).
+The result is accepted only if the normalized directory path is the exact prefix of the normalized
+file path.
+Otherwise the file path remains absolute.
+This makes the conversion unambiguous and ensures that the relative path does not start with ``../``.
+The resulting file path becomes the source unit name.
+
+.. note::
+
+    Prior to version 0.8.8, CLI path stripping was not performed and the only normalization applied
+    was the conversion of path separators.
+    When working with older versions of the compiler it is recommended to invoke the compiler from
+    the base path and to only use relative paths on the command line.
 
 .. index:: ! remapping; import, ! import; remapping, ! remapping; context, ! remapping; prefix, ! remapping; target
 .. _import-remapping:
@@ -414,7 +474,7 @@ Here are the detailed rules governing the behaviour of remappings:
 
      .. code-block:: bash
 
-         solc /project/=/contracts/ /project/contract.sol --base-path /project # source unit name: /project/contract.sol
+         solc /project/=/contracts/ /project/contract.sol --base-path /project # source unit name: contract.sol
 
      .. code-block:: solidity
          :caption: /project/contract.sol

libsolidity/interface/FileReader.cpp

@@ -22,6 +22,8 @@
 #include <libsolutil/CommonIO.h>
 #include <libsolutil/Exceptions.h>
 
+#include <boost/algorithm/string/predicate.hpp>
+
 using solidity::frontend::ReadCallback;
 using solidity::langutil::InternalCompilerError;
 using solidity::util::errinfo_comment;
@@ -31,9 +33,22 @@ using std::string;
 namespace solidity::frontend
 {
 
+void FileReader::setBasePath(boost::filesystem::path const& _path)
+{
+	m_basePath = (_path.empty() ? "" : normalizeCLIPathForVFS(_path));
+}
+
 void FileReader::setSource(boost::filesystem::path const& _path, SourceCode _source)
 {
-	m_sourceCodes[_path.generic_string()] = std::move(_source);
+	boost::filesystem::path normalizedPath = normalizeCLIPathForVFS(_path);
+	boost::filesystem::path prefix = (m_basePath.empty() ? normalizeCLIPathForVFS(".") : m_basePath);
+
+	m_sourceCodes[stripPrefixIfPresent(prefix, normalizedPath).generic_string()] = std::move(_source);
+}
+
+void FileReader::setStdin(SourceCode _source)
+{
+	m_sourceCodes["<stdin>"] = std::move(_source);
 }
 
 void FileReader::setSources(StringMap _sources)
@@ -92,5 +107,138 @@ ReadCallback::Result FileReader::readFile(string const& _kind, string const& _so
 	}
 }
 
+boost::filesystem::path FileReader::normalizeCLIPathForVFS(boost::filesystem::path const& _path)
+{
+	// Detailed normalization rules:
+	// - Makes the path either be absolute or have slash as root (note that on Windows paths with
+	//   slash as root are not considered absolute by Boost). If it is empty, it becomes
+	//   the current working directory.
+	// - Collapses redundant . and .. segments.
+	// - Removes leading .. segments from an absolute path (i.e. /../../ becomes just /).
+	// - Squashes sequences of multiple path separators into one.
+	// - Ensures that forward slashes are used as path separators on all platforms.
+	// - Removes the root name (e.g. drive letter on Windows) when it matches the root name in the
+	//   path to the current working directory.
+	//
+	// Also note that this function:
+	// - Does NOT resolve symlinks (except for symlinks in the path to the current working directory).
+	// - Does NOT check if the path refers to a file or a directory. If the path ends with a slash,
+	//   the slash is preserved even if it's a file.
+	//   - The only exception are paths where the file name is a dot (e.g. '.' or 'a/b/.'). These
+	//     always have a trailing slash after normalization.
+	// - Preserves case. Even if the filesystem is case-insensitive but case-preserving and the
+	//   case differs, the actual case from disk is NOT detected.
+
+	boost::filesystem::path canonicalWorkDir = boost::filesystem::weakly_canonical(boost::filesystem::current_path());
+
+	// NOTE: On UNIX systems the path returned from current_path() has symlinks resolved while on
+	// Windows it does not. To get consistent results we resolve them on all platforms.
+	boost::filesystem::path absolutePath = boost::filesystem::absolute(_path, canonicalWorkDir);
+
+	// NOTE: boost path preserves certain differences that are ignored by its operator ==.
+	// E.g. "a//b" vs "a/b" or "a/b/" vs "a/b/.". lexically_normal() does remove these differences.
+	boost::filesystem::path normalizedPath =  absolutePath.lexically_normal();
+	solAssert(normalizedPath.is_absolute() || normalizedPath.root_path() == "/", "");
+
+	// If the path is on the same drive as the working dir, for portability we prefer not to
+	// include the root name. Do this only for non-UNC paths - my experiments show that on Windows
+	// when the working dir is an UNC path, / does not not actually refer to the root of the UNC path.
+	boost::filesystem::path normalizedRootPath = normalizedPath.root_path();
+	if (!isUNCPath(normalizedPath))
+	{
+		boost::filesystem::path workingDirRootPath = canonicalWorkDir.root_path();
+		if (normalizedRootPath == workingDirRootPath)
+			normalizedRootPath = "/";
+	}
+
+	// lexically_normal() will not squash paths like "/../../" into "/". We have to do it manually.
+	boost::filesystem::path dotDotPrefix = absoluteDotDotPrefix(normalizedPath);
+
+	boost::filesystem::path normalizedPathNoDotDot = normalizedPath;
+	if (dotDotPrefix.empty())
+		normalizedPathNoDotDot = normalizedRootPath / normalizedPath.relative_path();
+	else
+		normalizedPathNoDotDot = normalizedRootPath / normalizedPath.lexically_relative(normalizedPath.root_path() / dotDotPrefix);
+	solAssert(!hasDotDotSegments(normalizedPathNoDotDot), "");
+
+	// NOTE: On Windows lexically_normal() converts all separators to forward slashes. Convert them back.
+	// Separators do not affect path comparison but remain in internal representation returned by native().
+	// This will also normalize the root name to start with // in UNC paths.
+	normalizedPathNoDotDot = normalizedPathNoDotDot.generic_string();
+
+	// For some reason boost considers "/." different than "/" even though for other directories
+	// the trailing dot is ignored.
+	if (normalizedPathNoDotDot == "/.")
+		return "/";
+
+	return normalizedPathNoDotDot;
 }
 
+bool FileReader::isPathPrefix(boost::filesystem::path const& _prefix, boost::filesystem::path const& _path)
+{
+	solAssert(!_prefix.empty() && !_path.empty(), "");
+	// NOTE: On Windows paths starting with a slash (rather than a drive letter) are considered relative by boost.
+	solAssert(_prefix.is_absolute() || isUNCPath(_prefix) || _prefix.root_path() == "/", "");
+	solAssert(_path.is_absolute() || isUNCPath(_path) || _path.root_path() == "/", "");
+	solAssert(_prefix == _prefix.lexically_normal() && _path == _path.lexically_normal(), "");
+	solAssert(!hasDotDotSegments(_prefix) && !hasDotDotSegments(_path), "");
+
+	boost::filesystem::path strippedPath = _path.lexically_relative(
+		// Before 1.72.0 lexically_relative() was not handling paths with empty, dot and dot dot segments
+		// correctly (see https://github.com/boostorg/filesystem/issues/76). The only case where this
+		// is possible after our normalization is a directory name ending in a slash (filename is a dot).
+		_prefix.filename_is_dot() ? _prefix.parent_path() : _prefix
+	);
+	return !strippedPath.empty() && *strippedPath.begin() != "..";
+}
+
+boost::filesystem::path FileReader::stripPrefixIfPresent(boost::filesystem::path const& _prefix, boost::filesystem::path const& _path)
+{
+	if (!isPathPrefix(_prefix, _path))
+		return _path;
+
+	boost::filesystem::path strippedPath = _path.lexically_relative(
+		_prefix.filename_is_dot() ? _prefix.parent_path() : _prefix
+	);
+	solAssert(strippedPath.empty() || *strippedPath.begin() != "..", "");
+	return strippedPath;
+}
+
+boost::filesystem::path FileReader::absoluteDotDotPrefix(boost::filesystem::path const& _path)
+{
+	solAssert(_path.is_absolute() || _path.root_path() == "/", "");
+
+	boost::filesystem::path _pathWithoutRoot = _path.relative_path();
+	boost::filesystem::path prefix;
+	for (boost::filesystem::path const& segment: _pathWithoutRoot)
+		if (segment.filename_is_dot_dot())
+			prefix /= segment;
+
+	return prefix;
+}
+
+bool FileReader::hasDotDotSegments(boost::filesystem::path const& _path)
+{
+	for (boost::filesystem::path const& segment: _path)
+		if (segment.filename_is_dot_dot())
+			return true;
+
+	return false;
+}
+
+bool FileReader::isUNCPath(boost::filesystem::path const& _path)
+{
+	string rootName = _path.root_name().string();
+
+	return (
+		rootName.size() == 2 ||
+		(rootName.size() > 2 && rootName[2] != rootName[1])
+	) && (
+		(rootName[0] == '/' && rootName[1] == '/')
+#if defined(_WIN32)
+		|| (rootName[0] == '\\' && rootName[1] == '\\')
+#endif
+	);
+}
+
+}

libsolidity/interface/FileReader.h

@@ -45,30 +45,35 @@ class FileReader
 		boost::filesystem::path _basePath = {},
 		FileSystemPathSet _allowedDirectories = {}
 	):
-		m_basePath(std::move(_basePath)),
 		m_allowedDirectories(std::move(_allowedDirectories)),
 		m_sourceCodes()
-	{}
+	{
+		setBasePath(_basePath);
+	}
 
-	void setBasePath(boost::filesystem::path _path) { m_basePath = std::move(_path); }
+	void setBasePath(boost::filesystem::path const& _path);
 	boost::filesystem::path const& basePath() const noexcept { return m_basePath; }
 
 	void allowDirectory(boost::filesystem::path _path) { m_allowedDirectories.insert(std::move(_path)); }
 	FileSystemPathSet const& allowedDirectories() const noexcept { return m_allowedDirectories; }
 
 	StringMap const& sourceCodes() const noexcept { return m_sourceCodes; }
 
-	/// Retrieves the source code for a given source unit ID.
+	/// Retrieves the source code for a given source unit name.
 	SourceCode const& sourceCode(SourceUnitName const& _sourceUnitName) const { return m_sourceCodes.at(_sourceUnitName); }
 
-	/// Resets all sources to the given map of source unit ID to source codes.
+	/// Resets all sources to the given map of source unit name to source codes.
 	/// Does not enforce @a allowedDirectories().
 	void setSources(StringMap _sources);
 
-	/// Adds the source code for a given source unit ID.
+	/// Adds the source code under a source unit name created by normalizing the file path.
 	/// Does not enforce @a allowedDirectories().
 	void setSource(boost::filesystem::path const& _path, SourceCode _source);
 
+	/// Adds the source code under the source unit name of @a <stdin>.
+	/// Does not enforce @a allowedDirectories().
+	void setStdin(SourceCode _source);
+
 	/// Receives a @p _sourceUnitName that refers to a source unit in compiler's virtual filesystem
 	/// and attempts to interpret it as a path and read the corresponding file from disk.
 	/// The read will only succeed if the canonical path of the file is within one of the @a allowedDirectories().
@@ -83,7 +88,43 @@ class FileReader
 		return [this](std::string const& _kind, std::string const& _path) { return readFile(_kind, _path); };
 	}
 
+	/// Normalizes a filesystem path to make it include all components up to the filesystem root,
+	/// remove small, inconsequential differences that do not affect the meaning and make it look
+	/// the same on all platforms (if possible). Symlinks in the path are not resolved.
+	/// The resulting path uses forward slashes as path separators, has no redundant separators,
+	/// has no redundant . or .. segments and has no root name if removing it does not change the meaning.
+	/// The path does not have to actually exist.
+	static boost::filesystem::path normalizeCLIPathForVFS(boost::filesystem::path const& _path);
+
+	/// @returns true if all the path components of @a _prefix are present at the beginning of @a _path.
+	/// Both paths must be absolute (or have slash as root) and normalized (no . or .. segments, no
+	/// multiple consecutive slashes).
+	/// Paths are treated as case-sensitive. Does not require the path to actually exist in the
+	/// filesystem and does not follow symlinks. Only considers whole segments, e.g. /abc/d is not
+	/// considered a prefix of /abc/def. Both paths must be non-empty.
+	/// Ignores the trailing slash, i.e. /a/b/c.sol/ is treated as a valid prefix of /a/b/c.sol.
+	static bool isPathPrefix(boost::filesystem::path const& _prefix, boost::filesystem::path const& _path);
+
+	/// If @a _prefix is actually a prefix of @p _path, removes it from @a _path to make it relative.
+	/// @returns The path without the prefix or unchanged path if there is not prefix.
+	/// If @a _path and @_prefix are identical, the result is '.'.
+	static boost::filesystem::path stripPrefixIfPresent(boost::filesystem::path const& _prefix, boost::filesystem::path const& _path);
+
+	/// @returns true if the specified path is an UNC path.
+	/// UNC paths start with // followed by a name (on Windows they can also start with \\).
+	/// They are used for network shares on Windows. On UNIX systems they do not have the same
+	/// functionality but usually they are still recognized and treated in a special way.
+	static bool isUNCPath(boost::filesystem::path const& _path);
+
 private:
+	/// If @a _path starts with a number of .. segments, returns a path consisting only of those
+	/// segments (root name is not included). Otherwise returns an empty path. @a _path must be
+	/// absolute (or have slash as root).
+	static boost::filesystem::path absoluteDotDotPrefix(boost::filesystem::path const& _path);
+
+	/// @returns true if the path contains any .. segments.
+	static bool hasDotDotSegments(boost::filesystem::path const& _path);
+
 	/// Base path, used for resolving relative paths in imports.
 	boost::filesystem::path m_basePath;
 

solc/CommandLineInterface.cpp

@@ -451,7 +451,7 @@ bool CommandLineInterface::readInputFiles()
 			m_standardJsonInput = readUntilEnd(m_sin);
 		}
 		else
-			m_fileReader.setSource(g_stdinFileName, readUntilEnd(m_sin));
+			m_fileReader.setStdin(readUntilEnd(m_sin));
 	}
 
 	if (m_fileReader.sourceCodes().empty() && !m_standardJsonInput.has_value())

test/CMakeLists.txt

@@ -103,6 +103,7 @@ set(libsolidity_sources
     libsolidity/SyntaxTest.h
     libsolidity/ViewPureChecker.cpp
     libsolidity/analysis/FunctionCallGraph.cpp
+    libsolidity/interface/FileReader.cpp
 )
 detect_stray_source_files("${libsolidity_sources}" "libsolidity/")