Add a couple of mini changes to improve adding custom plugins + Lots of fixes to RegExps to accurately reflect incoming data + others - see ChangeLog.md for details

Author: tsegall

ChangeLog.md

@@ -1,6 +1,26 @@
 
 ## Changes ##
 
+### 17.1.2
+ - ENH: Add support for \\p{Digit} as a synonym for \d in RegExp plugins (Issue #142)
+ - ENH: Validate the input file associated with a user-defined list plugin to check for lower case characters (Issue #143)
+ - BUG: RegExp for FREE_TEXT incorrect if newlines present
+ - BUG: RegExp for SPATIAL.WKT incorrect - missing '.'
+ - BUG: RegExp for STREET_ADDRESS_EN, FULL_ADDRESS_EN incorrect if newlines present
+ - BUG: RegExp incorrect if backing out from PhoneNumber misdetection and newlines present
+ - BUG: RegExp incorrect if Unicode alphabetic character present and outputting a set of values
+ - BUG: RegExp incorrect if returning a alphabetic character class with lower case e.g expected '(?i)[A-D]' for a, b, c, d not '[A-D]'
+ - BUG: RegExp incorrect if merging a simple numeric with a qualified numeric e.g. '\d' & '\d{2}\p{IsAlphabetic}{4}'
+ - BUG: RegExp for NAME.MIDDLE should allow periods
+ - BUG: RegExp for POSTAL_CODE.ZIP5_PLUS4_US incorrect if any Zips with fewer than 5 digits present
+ - BUG: VIN's outside the US do not have a check digit hence format is to easy to misinterpret so insist on a valid header
+ - BUG: Were not recognizing '?' as a special character for Regular Expression - hence they were not being sloshed
+ - BUG: RegExp for URLs not allowing parens
+ - BUG: If we change out mind and make it a Regular Expression semantic type then any outliers are actually invalids
+ - INT: Fix a couple of warnings
+ - INT: Add missing Indian Semantic Types to documentation
+ - INT: Bump google phonenumber to 9.0.10, gradle to 8.14.3, commons-validator to 1.10 (fixes dependency on vulnerable commons-beanutils), spotbugs to 6.2.2
+
 ### 17.1.1
  - ENH: Improve Indian District and State detection
  - INT: Migrate to the Central Publishing Portal (OSSRH Service is EOL)
@@ -32,7 +52,7 @@
 
 ### 16.2.10
  - INT: Fix SpotBugs issues
- - INT: Bump google phonenumber to 9.0.7m gradle to 8.14.2
+ - INT: Bump google phonenumber to 9.0.7, gradle to 8.14.2
 
 ### 16.2.9
  - BUG: Fix issues with sampleCount being incorrect - typically when doing a subsequent pass (Issue #134)

README.md

@@ -17,7 +17,7 @@ Design objectives:
 * Sufficiently fast to be used inline.   See Speed notes below.
 * Minimal false positives for Semantic type detection. See Performance notes below.
 * Usable in either Streaming, Bulk or Record  mode.
-* Broad country/language support - including US, Canada, Mexico, Brazil, UK, Australia, much of Europe, Japan and China.
+* Broad country/language support - including US, Canada, Mexico, Brazil, UK, Australia, India, much of Europe, Japan and China.
 * Support for sharded analysis (i.e. Analysis results can be merged)
 * Once stream is profiled then subsequent samples can be validated and/or new samples can be generated
 
@@ -124,7 +124,7 @@ Semantic Type: ***NAME.MIDDLE*** (String)
 
 ### Additional Examples
 
-Are in the [example](examples) directory.
+Are in the [examples](examples) directory.
 
 ## Date Format determination ##
 

SemanticTypes.md

@@ -109,6 +109,7 @@
 |POSTAL_CODE.POSTAL_CODE_DE|Postal Code (DE)|https://www.wikidata.org/wiki/Property:P281, https://en.wikipedia.org/wiki/Postal_codes_in_Germany, https://schema.org/postalCode|de-DE|
 |POSTAL_CODE.POSTAL_CODE_ES|Postal Code (ES)|https://www.wikidata.org/wiki/Property:P281, https://en.wikipedia.org/wiki/Postal_codes_in_Spain, https://schema.org/postalCode|es-ES|
 |POSTAL_CODE.POSTAL_CODE_FR|Postal Code (FR)|https://www.wikidata.org/wiki/Property:P281, https://en.wikipedia.org/wiki/Postal_codes_in_France, https://schema.org/postalCode|fr-FR|
+|POSTAL_CODE.POSTAL_CODE_IN|Postal Code (IN)|https://www.wikidata.org/wiki/Property:P281, https://en.wikipedia.org/wiki/Postal_Index_Number, https://schema.org/postalCode|en-IN,hi-IN|
 |POSTAL_CODE.POSTAL_CODE_JA|Postal Code (JA)|https://www.wikidata.org/wiki/Property:P281, https://en.wikipedia.org/wiki/Postal_codes_in_Japan, https://schema.org/postalCode|ja|
 |POSTAL_CODE.POSTAL_CODE_MX|Postal Code (MX)|https://www.wikidata.org/wiki/Property:P281, https://en.wikipedia.org/wiki/Postal_codes_in_Mexico, https://schema.org/postalCode|es-MX|
 |POSTAL_CODE.POSTAL_CODE_NL|Postal Code (NL)|https://www.wikidata.org/wiki/Property:P281, https://en.wikipedia.org/wiki/Postal_codes_in_the_Netherlands, https://schema.org/postalCode|en-NL,nl-NL|
@@ -131,6 +132,7 @@
 |STATE_PROVINCE.COUNTY_US|US County Name|https://en.wikipedia.org/wiki/County_(United_States)|en-US|
 |STATE_PROVINCE.DEPARTMENT_CO|Colombian Department|https://en.wikipedia.org/wiki/Departments_of_Colombia|es-CO|
 |STATE_PROVINCE.DEPARTMENT_NAME_FR|French Department||fr-FR|
+|STATE_PROVINCE.DISTRICT_NAME_IN|Indian District name|https://en.wikipedia.org/wiki/List_of_districts_in_India|en-IN, hi-IN|
 |STATE_PROVINCE.DISTRICT_NAME_PT|Portuguese District Name|https://en.wikipedia.org/wiki/List_of_municipalities_of_Portugal|pt-PT|
 |STATE_PROVINCE.INSEE_CODE_FR|French Insee Code (5 digit)|https://en.wikipedia.org/wiki/INSEE_code|fr-FR|
 |STATE_PROVINCE.MUNICIPALITY_BR|Brazilian Municipality|https://en.wikipedia.org/wiki/Municipalities_of_Brazil|pt-BR|
@@ -166,6 +168,8 @@
 |STATE_PROVINCE.STATE_NAME_US|US State Name|https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States, https://schema.org/State|en-US,es-US,en-CA,fr-CA,en-MX|
 |STATE_PROVINCE.STATE_PROVINCE_NA|US State Code/Canadian Province Code/Mexican State Code|https://schema.org/State|en-US,en-CA,fr-CA,en-MX|
 |STATE_PROVINCE.STATE_PROVINCE_NAME_NA|US State Name/Canadian Province Name/Mexican State Name|https://schema.org/State|en-US,es-US,en-CA,fr-CA,en-MX|
+|STATE_PROVINCE.STATE_UNION_IN|Indian State/Union Code|https://en.wikipedia.org/wiki/States_and_union_territories_of_India, https://en.wikipedia.org/wiki/ISO_3166-2:IN|en-IN|
+|STATE_PROVINCE.STATE_UNION_NAME_IN|Indian State/Union name|https://en.wikipedia.org/wiki/States_and_union_territories_of_India, https://en.wikipedia.org/wiki/ISO_3166-2:IN|en-IN|
 |STATE_PROVINCE.STATE_US|US State Code|https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States, https://schema.org/State|en-US,en-Latn-US,en-CA,fr-CA,en-MX|
 |STATE_PROVINCE.SUBURB_AU|Australian Suburb|https://en.wikipedia.org/wiki/Suburbs_and_localities_(Australia)|en-AU|
 |STREET_ADDRESS2_EN|Street Address - Line 2 (English Language)||en|

build.gradle

@@ -1,10 +1,10 @@
 plugins {
 	id "com.github.ben-manes.versions"
-	id "com.github.spotbugs" version "6.2.1" apply false
+	id "com.github.spotbugs" version "6.2.2" apply false
 }
 
 wrapper {
-	gradleVersion = '8.14.2'
+	gradleVersion = '8.14.3'
 }
 
 tasks.register('examples.clean') {

cli/src/main/java/com/cobber/fta/driver/FileProcessor.java

@@ -336,6 +336,8 @@ private void processAllFields(final ParserSettings settings) throws IOException,
 		final int[] blanks = new int[numFields];
 		final Set<String> failures = new HashSet<>();
 
+		final TextAnalysisResult[] results = processor.getResult();
+
 		// Check the RegExp at level 2 validation
 		if (options.validate == 2) {
 			try (BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(new File(filename)), options.charset))) {
@@ -348,13 +350,15 @@ private void processAllFields(final ParserSettings settings) throws IOException,
 				Pattern[] patterns = null;
 				rawRecordIndex = 0;
 
+				TextAnalyzer[] analyzers = null;
 				for (final CloseableIterator<NamedCsvRecord> iter = csv.iterator(); iter.hasNext();) {
 					final NamedCsvRecord rowRaw = iter.next();
 					final String[] row = rowRaw.getFields().toArray(new String[0]);
+
 					// Are we looking at the header row?
 					if (rawRecordIndex == 0) {
 						numFields = rowRaw.getFieldCount();
-						final TextAnalysisResult[] results = processor.getResult();
+						analyzers = processor.getAnalyzers();
 						patterns = new Pattern[numFields];
 
 						for (int i = 0; i < numFields; i++)
@@ -369,11 +373,11 @@ private void processAllFields(final ParserSettings settings) throws IOException,
 					for (int i = 0; i < numFields; i++) {
 						if (options.col == -1 || options.col == i) {
 							final String value = row[i];
-							if (value == null)
+							if (analyzers[i].isNullEquivalent(value))
 								nulls[i]++;
 							else if (value.trim().isEmpty())
 								blanks[i]++;
-							else if (patterns[i].matcher(value).matches())
+							else if (patterns[i].matcher(value.trim()).matches())
 								matched[i]++;
 							else if (options.verbose != 0)
 								failures.add(value);
@@ -396,7 +400,6 @@ else if (options.verbose != 0)
 		if (outputJSON)
 			output.printf("[%n");
 
-		final TextAnalysisResult[] results = processor.getResult();
 		final ArrayNode fieldsArray = mapper.createArrayNode();
 
 		for (int i = 0; i < numFields; i++) {
@@ -510,13 +513,17 @@ else if (result.getCardinality() > 20) {
 				}
 
 				if (options.validate == 2 && matched[i] != result.getMatchCount()) {
-					if (result.isSemanticType())
-						if (matched[i] > result.getMatchCount())
-							error.printf("\t*** Warning: Match Count via RegExp (%d) > LogicalType match analysis (%d) ***%n", matched[i], result.getMatchCount());
-						else
-							error.printf("\t*** Error: Match Count via RegExp (%d) < LogicalType match analysis (%d) ***%n", matched[i], result.getMatchCount());
+					String logicalType = result.isSemanticType() ? "Logical Type " : "";
+
+					if (matched[i] > result.getMatchCount())
+						error.printf("\t*** NOTE: Composite: %s, field: %s (%d), match Count via RegExp (%d) > %smatch analysis (%d) ***%n",
+								analyzer.getContext().getCompositeName(), analyzer.getContext().getStreamName(),
+								analyzer.getContext().getStreamIndex(), matched[i], logicalType, result.getMatchCount());
 					else
-						error.printf("\t*** Error: Match Count via RegExp (%d) does not match analysis (%d) ***%n", matched[i], result.getMatchCount());
+						error.printf("\t*** ERROR: Composite: %s, field: %s (%d), match Count via RegExp (%d) < %smatch analysis (%d) ***%n",
+								analyzer.getContext().getCompositeName(), analyzer.getContext().getStreamName(),
+								analyzer.getContext().getStreamIndex(), matched[i], logicalType, result.getMatchCount());
+
 					if (options.verbose != 0) {
 						error.println("Failed to match:");
 						for (final String failure : failures)

core/src/main/java/com/cobber/fta/KnownTypes.java

@@ -26,6 +26,7 @@
 import com.cobber.fta.core.FTAType;
 import com.cobber.fta.core.RegExpGenerator;
 import com.cobber.fta.core.RegExpSplitter;
+import com.cobber.fta.core.Utils;
 
 /**
  * A set of predefined types, for simple numerics, strings and boolean values.
@@ -69,6 +70,8 @@ public enum ID {
 
 	public static final String PATTERN_ANY = ".";
 	public static final String PATTERN_ANY_VARIABLE = ".+";
+	public static final String PATTERN_ANY_STRING = ".*";
+	public static final String PATTERN_ANY_STRING_DOTALL = "(?s).*";
 	public static final String PATTERN_ALPHA = "\\p{IsAlphabetic}";
 	public static final String PATTERN_NUMERIC = "\\d";
 	public static final String PATTERN_NUMERIC_VARIABLE = PATTERN_NUMERIC + "+";
@@ -203,7 +206,8 @@ else if (negSuffix.charAt(0) == minusSign && minusSign == '\u2212')  // Unicode
 		final String localizedNo = keywords.get("NO");
 		if (localizedYes != null && localizedNo != null) {
 			if (!"yes".equals(localizedYes) || !"no".equals(localizedNo)) {
-				PATTERN_BOOLEAN_YES_NO_LOCALIZED = "(?i)(" + localizedNo + "|" + localizedYes + ")";
+				PATTERN_BOOLEAN_YES_NO_LOCALIZED = !Utils.isSimpleAlphas(localizedYes) || !Utils.isSimpleAlphas(localizedNo) ? "(?u)" : "";
+				PATTERN_BOOLEAN_YES_NO_LOCALIZED += "(?i)(" + localizedNo.toUpperCase(locale) + "|" + localizedYes.toUpperCase(locale) + ")";
 				knownTypes.put(PATTERN_BOOLEAN_YES_NO_LOCALIZED,
 						new TypeInfo(ID.ID_BOOLEAN_YES_NO_LOCALIZED, PATTERN_BOOLEAN_YES_NO_LOCALIZED, FTAType.BOOLEAN, "YES_NO", false, ""));
 			}

core/src/main/java/com/cobber/fta/core/RegExpGenerator.java

@@ -53,6 +53,7 @@ public class RegExpGenerator {
 	private boolean isUnderscore;
 	private boolean isMinus;
 	private boolean isOther;
+	private boolean isUnicode;
 	private int maxClasses;
 	private int maxSetSize = -1;
 	private Locale locale;
@@ -76,7 +77,7 @@ public RegExpGenerator(final int maxSetSize, final Locale locale) {
 	 * @return True if the character is reserved.
 	 */
 	public static boolean isSpecial(final char ch) {
-		return ch == '.' ||  ch == '+' || ch == '*' || ch == '^' || ch == '$' ||
+		return ch == '.' ||  ch == '+' || ch == '*' || ch == '^' || ch == '$' || ch == '?' ||
 				ch == '[' || ch == ']' || ch == '(' || ch == ')' || ch == '{' || ch == '}' || ch == '|';
 
 	}
@@ -94,8 +95,13 @@ public static String slosh(final char ch) {
 	 * @return The merged expression.
 	 */
 	public static String merge(String firstRE, String secondRE) {
+		if (firstRE.equals(secondRE))
+			return firstRE;
+
+		String simpleAnswer = firstRE.compareTo(secondRE) < 0 ? firstRE + '|' + secondRE : secondRE + '|' + firstRE;
+
 		if (!firstRE.contains(secondRE) && !secondRE.contains(firstRE))
-			return firstRE.compareTo(secondRE) < 0 ? firstRE + '|' + secondRE : secondRE + '|' + firstRE;
+			return simpleAnswer;
 
 		// Switch it so that the firstRE contains the second
 		if (secondRE.contains(firstRE)) {
@@ -104,9 +110,14 @@ public static String merge(String firstRE, String secondRE) {
 			secondRE = save;
 		}
 
-		// Case 1: first starts with second
-		if (firstRE.startsWith(secondRE))
+		// Case 1: first starts with second (e.g. "\d{2}" and "\d{2}-\d{2}") should return "\d{2}(-\d{2})?")
+		if (firstRE.startsWith(secondRE)) {
+			String optionalTail = firstRE.substring(secondRE.length());
+			// We need to not merge if we have something like "\d{2}" and "\d-\d{2}" as we would NOT want to return "\d({2}-\d{2})?"
+			if (optionalTail.charAt(0) == '{')
+				return simpleAnswer;
 			return secondRE + '(' + firstRE.substring(secondRE.length()) + ")?";
+		}
 
 		// Case 2: first ends with second
 		if (firstRE.endsWith(secondRE))
@@ -173,10 +184,16 @@ public void train(final String input) {
 
 		for (int i = 0; i < len; i++) {
 			final char ch = input.charAt(i);
-			if (Character.isLetter(ch))
+			if (Character.isLetter(ch)) {
 				isLetter = true;
-			else if (Character.isDigit(ch))
+				if (!isUnicode && !Utils.isSimpleAlpha(ch))
+					isUnicode = true;
+			}
+			else if (Character.isDigit(ch)) {
+				if (!isUnicode && !Utils.isSimpleDigit(ch))
+					isUnicode = true;
 				isDigit = true;
+			}
 			else if (ch == '.')
 				isPeriod = true;
 			else if (ch == ' ')
@@ -221,7 +238,7 @@ public String getResult() {
 			final boolean constantLength = shortest == longest;
 
 			// Generate a Character class if possible - we would rather see [A-G] than A|B|C|D|E|F|G
-			if (memory.size() >= 3 && shortest == 1 && constantLength) {
+			if (memory.size() >= 3 && shortest == 1 && constantLength && !isUnicode) {
 				char first = 0;
 				char last = 0;
 				char current;
@@ -238,6 +255,8 @@ public String getResult() {
 					last = current;
 				}
 				if (collapsible) {
+					if (isLetter)
+						result.append("(?i)");
 					result.append('[').append(first).append('-').append(last).append(']');
 					return result.toString();
 				}
@@ -247,6 +266,8 @@ public String getResult() {
 			if (memory.size() <= maxSetSize && !(constantLength && !isSpace && !isOther && !isUnderscore && shortest >= 12)) {
 				if (isLetter)
 					result.append("(?i)");
+				if (isUnicode)
+					result.append("(?u)");
 				if (memory.size() != 1)
 					result.append('(');
 				for (final String element : memory) {
@@ -302,14 +323,20 @@ public Set<String> getValues() {
 		toSimplifyFull.put("[\\p{IsAlphabetic}\\d]", "(<L>|<Nd>)");
 		toSimplifyFull.put("\\p{XDigit}", "[0-9A-Fa-f]");
 		toSimplifyFull.put("\\p{IsAlphabetic}", "<L>");
+		toSimplifyFull.put("\\p{Alpha}", "<L>");
+		toSimplifyFull.put("\\p{Alnum}", "(<L>|<Nd>)");
 		toSimplifyFull.put("\\d", "<Nd>");
+		toSimplifyFull.put("\\p{Digit}", "<Nd>");
 		toSimplifyFull.put("#", "\\#");
 		toSimplifyFull.put("(?i)", "");
 
 		toSimplifyASCII.put("[\\p{IsAlphabetic}\\d]", "[A-Za-z0-9]");
 		toSimplifyASCII.put("\\p{XDigit}", "[0-9A-Fa-f]");
 		toSimplifyASCII.put("\\p{IsAlphabetic}", "[A-Za-z]");
+		toSimplifyASCII.put("\\p{Alpha}", "[A-Za-z]");
+		toSimplifyASCII.put("\\p{Alnum}", "[A-Za-z0-9]");
 		toSimplifyASCII.put("\\d", "[0-9]");
+		toSimplifyASCII.put("\\p{Digit}", "[0-9]");
 		toSimplifyASCII.put("#", "\\#");
 		toSimplifyASCII.put("(?i)", "");
 	}

core/src/main/java/com/cobber/fta/core/Utils.java

@@ -161,12 +161,24 @@ public static boolean isAlphas(final String input) {
 		return input.chars().allMatch(Character::isLetter);
 	}
 
+	/**
+	 * Test if the supplied input is all simple alphas.
+	 * @param input String to test.
+	 * @return True if the string is all simple alphas.
+	 */
+	public static boolean isSimpleAlphas(final String input) {
+		if (input == null || input.isEmpty())
+			return false;
+
+		return input.chars().allMatch(c -> isSimpleAlpha((char) c));
+	}
+
 	/**
 	 * Test if the supplied character is numeric [0-9].
 	 * @param ch Character to test.
 	 * @return True if the character is in the range [0-9].
 	 */
-	public static boolean isSimpleNumeric(final char ch) {
+	public static boolean isSimpleDigit(final char ch) {
 		return ch >= '0' && ch <= '9';
 	}
 

core/src/test/java/com/cobber/fta/TestUtilsCore.java

@@ -502,13 +502,13 @@ public void testIsSimpleAlpha() {
 
 	@Test(groups = { TestGroups.ALL })
 	public void testIsSimpleNumeric() {
-		assertTrue(Utils.isSimpleNumeric('0'));
-		assertTrue(Utils.isSimpleNumeric('9'));
-		assertFalse(Utils.isSimpleNumeric('０'));
-		assertFalse(Utils.isSimpleNumeric('a'));
-		assertFalse(Utils.isSimpleNumeric('z'));
-		assertFalse(Utils.isSimpleNumeric('A'));
-		assertFalse(Utils.isSimpleNumeric('Z'));
+		assertTrue(Utils.isSimpleDigit('0'));
+		assertTrue(Utils.isSimpleDigit('9'));
+		assertFalse(Utils.isSimpleDigit('０'));
+		assertFalse(Utils.isSimpleDigit('a'));
+		assertFalse(Utils.isSimpleDigit('z'));
+		assertFalse(Utils.isSimpleDigit('A'));
+		assertFalse(Utils.isSimpleDigit('Z'));
 	}
 
 	@Test(groups = { TestGroups.ALL })
@@ -602,7 +602,17 @@ public void testRegExpGenerator() {
 			 generator.train("D");
 			 generator.train("E");
 		 }
-		 assertEquals(generator.getResult(), "[A-E]");
+		 assertEquals(generator.getResult(), "(?i)[A-E]");
+
+		 generator = new RegExpGenerator(20, Locale.getDefault());
+		 for (int i = 0; i < 100; i++) {
+			 generator.train("a");
+			 generator.train("b");
+			 generator.train("c");
+			 generator.train("d");
+			 generator.train("e");
+		 }
+		 assertEquals(generator.getResult(), "(?i)[A-E]");
 
 		 generator = new RegExpGenerator();
 		 for (int i = 0; i < 100; i++) {

examples/contextual/build.gradle

@@ -3,7 +3,7 @@ plugins {
 }
 
 wrapper {
-	gradleVersion = '8.14.2'
+	gradleVersion = '8.14.3'
 }
 
 repositories {