interface of the GetToken or PeakToken like function

[asmwarrior]

I see the interface of the preprocessor is to return the whole string by preprocessor.Process().

Is it possible to make some function like preprocessor.GetToken(), so the high level Parser can call it, and get the Token one by one or even peek the token.

Thanks.

[dmikushin]

I think the best way to do it is to add a callback function. In principle, users could add callbacks to any point of tcpp code as they desire. In the example below, @deprilula28 adds pop include callback:

From 0b389c8fd45e96c0822ef91b087e880092565244 Mon Sep 17 00:00:00 2001
From: deprilula28 <deprilula28@gmail.com>
Date: Wed, 1 Mar 2023 22:02:37 -0300
Subject: [PATCH 59/68] Add on pop include handler

---
 source/tcppLibrary.hpp | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/source/tcppLibrary.hpp b/source/tcppLibrary.hpp
index 905090e..23bf78e 100644
--- a/source/tcppLibrary.hpp
+++ b/source/tcppLibrary.hpp
@@ -286,6 +286,7 @@ namespace tcpp
 		public:
 			using TOnErrorCallback = std::function<void(const TErrorInfo&)>;
 			using TOnIncludeCallback = std::function<IInputStream*(const std::string&, bool)>;
+			using TOnPopIncludeCallback = std::function<void()>;
 			using TSymTable = std::vector<TMacroDesc>;
 			using TContextStack = std::list<std::string>;
 			using TDirectiveHandler = std::function<std::string(Preprocessor&, Lexer&, const std::string&)>;
@@ -304,7 +305,7 @@ namespace tcpp
 		public:
 			Preprocessor() TCPP_NOEXCEPT = delete;
 			Preprocessor(const Preprocessor&) TCPP_NOEXCEPT = delete;
-			Preprocessor(Lexer& lexer, const TOnErrorCallback& onErrorCallback = {}, const TOnIncludeCallback& onIncludeCallback = {}) TCPP_NOEXCEPT;
+			Preprocessor(Lexer& lexer, const TOnErrorCallback& onErrorCallback = {}, const TOnIncludeCallback& onIncludeCallback = {}, const TOnPopIncludeCallback& onPopIncludeCallback = {}) TCPP_NOEXCEPT;
 			~Preprocessor() TCPP_NOEXCEPT = default;
 
 			bool AddCustomDirectiveHandler(const std::string& directive, const TDirectiveHandler& handler) TCPP_NOEXCEPT;
@@ -337,6 +338,7 @@ namespace tcpp
 			Lexer* mpLexer;
 			TOnErrorCallback mOnErrorCallback;
 			TOnIncludeCallback mOnIncludeCallback;
+			TOnPopIncludeCallback mOnPopIncludeCallback;
 			TSymTable mSymTable;
 			TContextStack mContextStack;
 			TIfStack mConditionalBlocksStack;
@@ -962,9 +964,8 @@ namespace tcpp
 		return mStreamsContext.empty() ? nullptr : mStreamsContext.top();
 	}
 
-
-	Preprocessor::Preprocessor(Lexer& lexer, const TOnErrorCallback& onErrorCallback, const TOnIncludeCallback& onIncludeCallback) TCPP_NOEXCEPT:
-		mpLexer(&lexer), mOnErrorCallback(onErrorCallback), mOnIncludeCallback(onIncludeCallback)
+	Preprocessor::Preprocessor(Lexer& lexer, const TOnErrorCallback& onErrorCallback, const TOnIncludeCallback& onIncludeCallback, const TOnPopIncludeCallback& onPopIncludeCallback) TCPP_NOEXCEPT:
+		mpLexer(&lexer), mOnErrorCallback(onErrorCallback), mOnIncludeCallback(onIncludeCallback), mOnPopIncludeCallback(onPopIncludeCallback)
 	{
 		mSymTable.push_back({ "__LINE__" });
 	}
@@ -1106,6 +1107,8 @@ namespace tcpp
 
 			if (!mpLexer->HasNextToken())
 			{
+				if (mOnPopIncludeCallback)
+					mOnPopIncludeCallback();
 				mpLexer->PopStream();
 			}
 		}
-- 
2.34.1

[asmwarrior]

Hi, thanks for the reply.

Adding a callback functions means that the whole process is still controlled by the preprocessor. I mean the preprocessor has the whole working loop on every tokens. For my question, I would like the high level Parser to drive the preprocessor. So when the high level Parser need a new token, he just ask the preprocessor to fetch one.

I'm not sure my idea is correct, but can call-back function handle such feature?

[asmwarrior]

When I browse the git repo of deprilula28, as you mentioned in your comment, I see this commit:

Extend API of Lexer with a new method PeekNextToken · deprilula28/tcpp@c945217 — deprilula28@c945217

It looks like those API is already implemented?

EDIT: it looks like the lexer has such Peek/Get Token feature, but not the Preprocessor.

But those implementation is not incremental, I mean the preprocessor is still trying to eat the whole tokens of the current translation unit, and store the whole tokens.

What I want is that it could be incremental, I mean when the Parser need a new token, the preprocessor goes on step to fetch a new Token. I believe a lot of compiler use such way. Thanks.

[dmikushin]

It looks like those API is already implemented?

Yes, c945217 is already a part of the master branch.

[asmwarrior]

It looks like those API is already implemented?

Yes, c945217 is already a part of the master branch.

Thanks for the reply.

I just edited my post several minutes ago, I double checked that code, and see this interface is in lexer level, not in the preprocessor level. So, I still can't use it easily.

[asmwarrior]

When I looked at the test code

	SECTION("TestProcess_PassSourceWithFunctionMacro_ReturnsProcessedSource")
	{
		std::string inputSource = "#define FOO(X) \\\nint X; \\\nint X ## _Additional;\nFOO(Test)";
		std::string expectedResult = "int Test;int Test_Additional;";

		Lexer lexer(std::make_unique<StringInputStream>(inputSource));

		bool result = true;

		Preprocessor preprocessor(lexer, { [&result](auto&& arg)
		{
			std::cerr << "Error: " << ErrorTypeToString(arg.mType) << std::endl;
			result = false;
		} });

		REQUIRE((result && (preprocessor.Process() == expectedResult)));
	}

In the file: https://github.com/bnoazx005/tcpp/blob/8eb66ff31596b3ddbf414eb0ba39d42ebb3cd0b5/tests/coreTests.cpp#L415C1-L431C3

I see that the Process() returns a pure std::string.

Is it possible to return a std::vector<Token> like stream? So the high level Parser can use/read those Tokens one by one.

[asmwarrior]

Hi, @bnoazx005 thanks.

I just read your code changes in commit: BREAKING CHANGES Preprocessor::Process now returns sequence of TToken… · bnoazx005/tcpp@c91ddf7

It's great!

Well, one last question remains.

Currently, the preprocessor can return the whole TTokensSequence.

If I would like to use some interface like GetToken or PeakToken like in the low Lexer level. Do I need to implement those interfaces myself on this TTokensSequence structure?

Or, can you supply those functions like the low level incremental lexer?

Thanks.

[bnoazx005]

If I would like to use some interface like GetToken or PeakToken like in the low Lexer level. Do I need to implement those interfaces myself on this TTokensSequence structure?

Or, can you supply those functions like the low level incremental lexer?

I suppose that I can implement this functionality as a wrapper for Preprocessor type that extends its API. Maybe there is a little chance that GetNextToken can be implemented as part of Preprocessor class, but this would require additional efforts. In same time the first approach is almost free to add.

[bnoazx005]

I've added solution to work with preprocessor's results in two approaches. The first is simple std::string which contains preprocessed input. The second one is TokensOutputStream which encapsulates std::vector and provide methods to iterate elements using GetNextToken/PeekNextToken.