lregex: implement new scope action "replace"

Suggested by @Fmajor in #1577.

Signed-off-by: Masatake YAMATO <[email protected]>

Author: masatake

Tmain/list-mtable-regex-flags.d/stdout-expected.txt

@@ -6,7 +6,7 @@ p       pcre2                                         use pcre2 regex engine
 -       fatal="MESSAGE"                               print the given MESSAGE and exit
 -       mgroup=N                                      a group in pattern determining the line number of tag
 -       placeholder                                   don't put this tag to tags file.
--       scope=ACTION                                  use scope stack: ACTION = ref|push|pop|clear|set
+-       scope=ACTION                                  use scope stack: ACTION = ref|push|pop|clear|set|replace
 -       tenter=TABLE[,CONT]                           enter to given regext table (with specifying continuation)
 -       tjump=TABLE                                   jump to another regext table(don't push the current table to state stack)
 -       tleave                                        leave from the current regext table

Tmain/list-regex-flags.d/stdout-expected.txt

@@ -6,7 +6,7 @@ p       pcre2                                         use pcre2 regex engine
 x       exclusive                                     skip testing the other patterns if a line is matched to this pattern
 -       fatal="MESSAGE"                               print the given MESSAGE and exit
 -       placeholder                                   don't put this tag to tags file.
--       scope=ACTION                                  use scope stack: ACTION = ref|push|pop|clear|set
+-       scope=ACTION                                  use scope stack: ACTION = ref|push|pop|clear|set|replace
 -       warning="MESSAGE"                             print the given MESSAGE at WARNING level
 -       _anonymous=PREFIX                             make an anonymous tag with PREFIX
 -       _extra=EXTRA                                  record the tag only when the extra is enabled

docs/man/ctags-optlib.7.rst

@@ -329,7 +329,7 @@ representation. ``--list-regex-flags`` lists all the flags.
 	can be an empty string.  See the following description of
 	``scope=...`` flag about how this is useful.
 
-``scope=(ref|push|pop|clear|set)``
+``scope=(ref|push|pop|clear|set|replace)``
 
 	Specify what to do with the internal scope stack.
 
@@ -342,7 +342,7 @@ representation. ``--list-regex-flags`` lists all the flags.
 	``--regex-<LANG>`` is pushed to the stack. ``{scope=push}``
 	implies ``{scope=ref}``.
 
-	You can fill the scope field of captured tag with
+	You can fill the scope field (``scope:``) of captured tag with
 	``{scope=ref}``. If ``{scope=ref}`` flag is given,
 	ctags attaches the tag at the top to the tag
 	captured with ``--regex-<LANG>`` as the value for the ``scope:``
@@ -356,6 +356,21 @@ representation. ``--list-regex-flags`` lists all the flags.
 	Specifying ``{scope=set}`` removes all the tags in the scope, and
 	then pushes the captured tag as ``{scope=push}`` does.
 
+	``{scope=replace}`` does the three things sequentially. First it
+	does the same as ``{scope=pop}``, then fills the ``scope:`` field
+	of the tag captured with ``--regex-<LANG>``, and pushes the tag to
+	the scope stack as if ``{scope=push}`` was given finally.
+	You cannot specify another scope action together with
+	``{scope=replace}``.
+
+	You don't want to specify ``{scope=pop}{scope=push}`` as an
+	alternative to ``{scope=replace}``; ``{scope=pop}{scope=push}``
+	fills the ``scope:`` field of the tag captured with ``--regex-<LANG>``
+	first, then pops the tag at the top of the stack, and pushes
+	the captured tag to the scope stack finally. The timing when
+	filling the end field is different between ``{scope=replace}`` and
+	``{scope=pop}{scope=push}``.
+
 	In some cases, you may want to use ``--regex-<LANG>`` only for its
 	side effects: using it only to manipulate the stack but not for
 	capturing a tag. In such a case, make *<name_pattern>* component of

docs/optlib.rst

@@ -570,6 +570,56 @@ Example 2:
 	foo	/tmp/input.pp	/^class foo {$/;"	c
 
 
+Example 3:
+
+.. code-block::
+
+	# in /tmp/input.docdoc
+	title T
+	...
+	section S0
+	...
+	section S1
+	...
+
+.. code-block:: ctags
+	:emphasize-lines: 15,21
+
+	# in /tmp/doc.ctags:
+	--langdef=doc
+	--map-doc=+.docdoc
+	--kinddef-doc=s,section,sections
+	--kinddef-doc=S,subsection,subsections
+
+	--_tabledef-doc=main
+	--_tabledef-doc=section
+	--_tabledef-doc=subsection
+
+	--_mtable-regex-doc=main/section +([^\n]+)\n/\1/s/{scope=push}{tenter=section}
+	--_mtable-regex-doc=main/[^\n]+\n|[^\n]+|\n//
+	--_mtable-regex-doc=main///{scope=clear}{tquit}
+
+	--_mtable-regex-doc=section/section +([^\n]+)\n/\1/s/{scope=replace}
+	--_mtable-regex-doc=section/subsection +([^\n]+)\n/\1/S/{scope=push}{tenter=subsection}
+	--_mtable-regex-doc=section/[^\n]+\n|[^\n]+|\n//
+	--_mtable-regex-doc=section///{scope=clear}{tquit}
+
+	--_mtable-regex-doc=subsection/(section )//{_advanceTo=0start}{tleave}{scope=pop}
+	--_mtable-regex-doc=subsection/subsection +([^\n]+)\n/\1/S/{scope=replace}
+	--_mtable-regex-doc=subsection/[^\n]+\n|[^\n]+|\n//
+	--_mtable-regex-doc=subsection///{scope=clear}{tquit}
+
+.. code-block:: console
+
+	% ctags --sort=no --fields=+nl --options=/tmp/doc.ctags -o - /tmp/input.docdoc
+	SEC0	/tmp/input.docdoc	/^section SEC0$/;"	s	line:1	language:doc
+	SUB0-1	/tmp/input.docdoc	/^subsection SUB0-1$/;"	S	line:3	language:doc	section:SEC0
+	SUB0-2	/tmp/input.docdoc	/^subsection SUB0-2$/;"	S	line:5	language:doc	section:SEC0
+	SEC1	/tmp/input.docdoc	/^section SEC1$/;"	s	line:7	language:doc
+	SUB1-1	/tmp/input.docdoc	/^subsection SUB1-1$/;"	S	line:9	language:doc	section:SEC1
+	SUB1-2	/tmp/input.docdoc	/^subsection SUB1-2$/;"	S	line:11	language:doc	section:SEC1
+
+
 NOTE: This flag doesn't work well with ``--mline-regex-<LANG>=``.
 
 Overriding the letter for file kind

main/lregex.c

@@ -76,7 +76,8 @@ enum scopeAction {
 	SCOPE_POP     = 1UL << 1,
 	SCOPE_PUSH    = 1UL << 2,
 	SCOPE_CLEAR   = 1UL << 3,
-	SCOPE_PLACEHOLDER = 1UL << 4,
+	SCOPE_REF_AFTER_POP = 1UL << 4,
+	SCOPE_PLACEHOLDER = 1UL << 5,
 };
 
 enum tableAction {
@@ -576,6 +577,8 @@ static void scope_ptrn_flag_eval (const char* const f  CTAGS_ATTR_UNUSED,
 		*bfields |= SCOPE_CLEAR;
 	else if (strcmp (v, "set") == 0)
 		*bfields |= (SCOPE_CLEAR | SCOPE_PUSH);
+	else if (strcmp (v, "replace") == 0)
+		*bfields |= (SCOPE_POP|SCOPE_REF_AFTER_POP|SCOPE_PUSH);
 	else
 		error (FATAL, "Unexpected value for scope flag in regex definition: scope=%s", v);
 }
@@ -589,7 +592,7 @@ static void placeholder_ptrn_flag_eval (const char* const f  CTAGS_ATTR_UNUSED,
 
 static flagDefinition scopePtrnFlagDef[] = {
 	{ '\0', "scope",     NULL, scope_ptrn_flag_eval,
-	  "ACTION", "use scope stack: ACTION = ref|push|pop|clear|set"},
+	  "ACTION", "use scope stack: ACTION = ref|push|pop|clear|set|replace"},
 	{ '\0', "placeholder",  NULL, placeholder_ptrn_flag_eval,
 	  NULL, "don't put this tag to tags file."},
 };
@@ -1300,7 +1303,12 @@ static void patternEvalFlags (struct lregexControlBlock *lcb,
 	}
 
 	if (regptype == REG_PARSER_SINGLE_LINE || regptype == REG_PARSER_MULTI_TABLE)
+	{
 		flagsEval (flags, scopePtrnFlagDef, ARRAY_SIZE(scopePtrnFlagDef), &ptrn->scopeActions);
+		if ((ptrn->scopeActions & (SCOPE_REF|SCOPE_REF_AFTER_POP)) == (SCOPE_REF|SCOPE_REF_AFTER_POP))
+			error (WARNING, "%s: don't combine \"replace\" with the other scope action.",
+				   getLanguageName (lcb->owner));
+	}
 
 	if (regptype == REG_PARSER_MULTI_LINE || regptype == REG_PARSER_MULTI_TABLE)
 	{
@@ -1564,6 +1572,16 @@ static bool hasNameSlot (const regexPattern* const patbuf)
 			|| patbuf->anonymous_tag_prefix);
 }
 
+static int scopeActionRef (int currentScope)
+{
+	int scope = currentScope;
+	tagEntryInfo *entry;
+	while ((entry = getEntryInCorkQueue (scope)) && entry->placeholder)
+		/* Look at parent */
+		scope = entry->extensionFields.scopeIndex;
+	return scope;
+}
+
 static void matchTagPattern (struct lregexControlBlock *lcb,
 		const char* line,
 		const regexPattern* const patbuf,
@@ -1585,14 +1603,7 @@ static void matchTagPattern (struct lregexControlBlock *lcb,
 	vStringStripTrailing (name);
 
 	if (patbuf->scopeActions & SCOPE_REF)
-	{
-		tagEntryInfo *entry;
-
-		scope = lcb->currentScope;
-		while ((entry = getEntryInCorkQueue (scope)) && entry->placeholder)
-			/* Look at parent */
-			scope = entry->extensionFields.scopeIndex;
-	}
+		scope = scopeActionRef (lcb->currentScope);
 	if (patbuf->scopeActions & SCOPE_CLEAR)
 	{
 		unsigned long endline = getInputLineNumberInRegPType(patbuf->regptype, offset);
@@ -1614,10 +1625,24 @@ static void matchTagPattern (struct lregexControlBlock *lcb,
 		tagEntryInfo *entry = getEntryInCorkQueue (lcb->currentScope);
 
 		if (entry && (entry->extensionFields.endLine == 0))
+		{
 			entry->extensionFields.endLine = getInputLineNumberInRegPType(patbuf->regptype, offset);
 
+			/*
+			 * SCOPE_POP|SCOPE_REF_AFTER_POP implies that "replace" was specified as the
+			 * scope action. If the specified action is "replace", getInputLineNumberInRegPType()
+			 * returns the start line of the NEW scope. The popped scope is ended BEFORE
+			 * the new scope. There is a gap. We must adjust the "end:" field here.
+			 */
+			if ((patbuf->scopeActions & SCOPE_REF_AFTER_POP) &&
+				entry->extensionFields.endLine > 1)
+				entry->extensionFields.endLine--;
+		}
+
 		lcb->currentScope = entry? entry->extensionFields.scopeIndex: CORK_NIL;
 	}
+	if (patbuf->scopeActions & SCOPE_REF_AFTER_POP)
+		scope = scopeActionRef (lcb->currentScope);
 
 	if (vStringLength (name) == 0 && (placeholder == false))
 	{

man/ctags-optlib.7.rst.in

@@ -329,7 +329,7 @@ representation. ``--list-regex-flags`` lists all the flags.
 	can be an empty string.  See the following description of
 	``scope=...`` flag about how this is useful.
 
-``scope=(ref|push|pop|clear|set)``
+``scope=(ref|push|pop|clear|set|replace)``
 
 	Specify what to do with the internal scope stack.
 
@@ -342,7 +342,7 @@ representation. ``--list-regex-flags`` lists all the flags.
 	``--regex-<LANG>`` is pushed to the stack. ``{scope=push}``
 	implies ``{scope=ref}``.
 
-	You can fill the scope field of captured tag with
+	You can fill the scope field (``scope:``) of captured tag with
 	``{scope=ref}``. If ``{scope=ref}`` flag is given,
 	@CTAGS_NAME_EXECUTABLE@ attaches the tag at the top to the tag
 	captured with ``--regex-<LANG>`` as the value for the ``scope:``
@@ -356,6 +356,21 @@ representation. ``--list-regex-flags`` lists all the flags.
 	Specifying ``{scope=set}`` removes all the tags in the scope, and
 	then pushes the captured tag as ``{scope=push}`` does.
 
+	``{scope=replace}`` does the three things sequentially. First it
+	does the same as ``{scope=pop}``, then fills the ``scope:`` field
+	of the tag captured with ``--regex-<LANG>``, and pushes the tag to
+	the scope stack as if ``{scope=push}`` was given finally.
+	You cannot specify another scope action together with
+	``{scope=replace}``.
+
+	You don't want to specify ``{scope=pop}{scope=push}`` as an
+	alternative to ``{scope=replace}``; ``{scope=pop}{scope=push}``
+	fills the ``scope:`` field of the tag captured with ``--regex-<LANG>``
+	first, then pops the tag at the top of the stack, and pushes
+	the captured tag to the scope stack finally. The timing when
+	filling the end field is different between ``{scope=replace}`` and
+	``{scope=pop}{scope=push}``.
+
 	In some cases, you may want to use ``--regex-<LANG>`` only for its
 	side effects: using it only to manipulate the stack but not for
 	capturing a tag. In such a case, make *<name_pattern>* component of