ENH: Add support for biasing detection based on composite name (Table/File) (Issue #154) + ENH: Improve ZIP detection if have data of the form 99999- (i.e. a trailing hyphen)

Author: tsegall

ChangeLog.md

@@ -1,7 +1,13 @@
 
 ## Changes ##
 
-### 17.3.0
+### 17.4.0
+ - INT: Bump testng to 7.12.0, logback-classic to 1.5.26
+ - ENH: Improve ZIP detection if have data of the form 99999- (i.e. a trailing hyphen)
+ - CLI: Fix bug when validating data based on a plugin
+ - ENH: Add support for biasing detection based on composite name (Table/File) (Issue #154)
+
+### 17.3.1
  - INT: Bump logback-classic to 1.5.25, Jackson to 2.21.0
  - INT: Cleaning.
  - INT: More gradle 10.0 changes

README.md

@@ -273,7 +273,7 @@ In addition to detecting a set of Base types FTA will also, when enabled (defaul
 like phone numbers, but that are in fact invalid, will NOT be detected as the Semantic Type TELEPHONE.
 * The set of Semantic Types detected is dependent on the current locale
 * The data stream name (e.g. the database field name or CSV field name) is commonly used to bias the detection.  For example, if the locale language is English and the data stream matches the regular expression '.\*(?i)(surname|last.?name|lname|maiden.?name|name.?last|last_nm).\*|last' then the detection is more likely to declare this stream a NAME.LAST Semantic Type. The data stream name can also be used to negatively bias the detection.  Consult the plugins.json file for more details.
-* Assuming the entire set of stream names is available, Semantic Type detection of a particular column may be impacted by other stream names, for example the Semantic Type PERSON.AGE is detected if we detect another field of type GENDER or NAME.FIRST.
+* Assuming the entire set of stream names is available, Semantic Type detection of a particular column may be impacted by other stream names, for example the Semantic Type PERSON.AGE is commonly detected if we detect another field of type GENDER or NAME.FIRST.
 * When using Record mode for Semantic Type analysis - the detection of Semantic Types for a stream may be impacted by prior determination of the Semantic Type of another Stream (either via detection or provided with the Context)
 * By default analysis is performed on the initial 4096 characters of the field (adjustable via setMaxInputLength()).
 * If two Semantic Types have equal confidence then the Semantic Type with the highest priority will be selected.
@@ -302,7 +302,7 @@ There are three basic types of plugins:
 * Finite (list) - captures any finite type (e.g. ISO-3166-2 (Country codes), US States, ...).  Implemented via a supplied list with the valid elements enumerated.
 * Code (java) - captures any complex type (e.g. Even numbers, Credit Cards numbers).  Implemented via a Java Class.
 
-Note: The Context (the current Stream Name and other field names) can be used to bias detection of the incoming data and/or solely determine the detection.
+Note: The Context (which includes the composite name (Table/File), the current stream names, and the other stream names) can be used to bias detection of the incoming data and/or solely determine the detection.
 
 ```json
 [
@@ -406,7 +406,7 @@ The optional 'minSamples' tag indicates that in order for this Semantic Type to
 
 The optional 'invalidList' tag is a list of invalid values for this Semantic Type, for example '[ "000-00-0000" ]' indicates that this is an invalid SSN, despite the fact that it matches the SSN regular expression.
 
-#### Example
+#### Examples
 The following example is looking for an Indian Postal Code.  In this case the header is mandatory so we will insist on both detecting a regular expression of the form '\d{6}' and a case independent match for the header.  The plugin will return '[1-9]\\d{5}' as it is illegal to have a leading zero for an Indian Postal Code.
 
 ```json
@@ -428,6 +428,28 @@ The following example is looking for an Indian Postal Code.  In this case the he
 	}
 ```
 
+The following example is looking for a Persons age. The plugin uses two possible regular expressions the first is an example of how to bias using a combination of the Composite name (Table or File) as well as the Stream name (Column or Field), the second is simply using the Stream name.  If 'compositeKey' is set to true then the input provided to match will be the concatenation of the Composite name with a period with the Stream name, this is commonly useful when the input source is a Database.  For example, if the Table name was 'Person' and the field name was 'Age' then the Composite key would be 'Person.Age' which matches the regular expression and hence we have a 100% confidence that this is a Person's age. The second regular expression only considers the Stream name and is only 90% confident that it is a Person's age.  In this case the plugin insists on another field that indicates we have found a Person (e.g. Gender, FirstName) so as not to misdetect if the 'Age' was the age of a building/pet etc..
+
+```json
+	{
+		"semanticType" : "PERSON.AGE",
+		"description": "Age (person)",
+		"pluginType": "java",
+		"clazz": "com.cobber.fta.plugins.person.Age",
+		"validLocales": [
+			{
+				"localeTag": "en",
+				"headerRegExps": [
+					{ "regExp": "(?i)(Person|Employee|Client|Customer)\\.(age|age[_ ].*|.*[_ ]age)", "confidence": 100, "mandatory": true, "compositeKey": true },
+					{ "regExp": "(?i)(age|age[_ ].*|.*[_ ]age)", "confidence": 90, "mandatory": true }
+				]
+			}
+		],
+		"baseType" : "LONG",
+		"priority": 98
+	}
+```
+
 ### Finite plugins ###
 
 The mandatory 'content' element is required.
@@ -662,6 +684,10 @@ Just one test
 
 `$ ./gradlew types:test --tests TestDates.localeDateTest`
 
+Validate a set of samples (against a known plugin)
+
+`$ cli --pluginMode true --pluginName POSTAL_CODE.ZIP5_US --col 0 <file.csv>`
+
 ### Generate JavaDoc ###
 `$ ./gradlew javadoc`
 

core/src/main/java/com/cobber/fta/HeaderLocaleEntry.java

@@ -33,7 +33,7 @@ public class HeaderLocaleEntry {
 	public int getHeaderConfidence(final String input) {
 		if (headerRegExps != null)
 			for (final HeaderEntry headerEntry : headerRegExps) {
-				if (headerEntry.matches(input))
+				if (headerEntry.matches(null, input))
 					return headerEntry.confidence;
 			}
 

core/src/main/java/com/cobber/fta/core/HeaderEntry.java

@@ -28,6 +28,8 @@ public class HeaderEntry {
 	public int confidence;
 	/** If true then the header must match be present. */
 	public boolean mandatory;
+	/** If true then the match against <CompositeName>.<DataStreamName> as opposed to just <DataStreamName>. */
+	public boolean compositeKey;
 
 	/** The pattern is used to cache the compiled regular expression since it will be executed many times. */
 	private Pattern pattern;
@@ -62,13 +64,16 @@ public String toString() {
 		return (new StringBuilder()).append('[').append(regExp).append(':').append(confidence).append(':').append(mandatory).append(']').toString();
 	}
 
-	public boolean matches(final String input) {
+	public boolean matches(final String compositeName, final String dataStreamName) {
 		if (pattern == null) {
 			synchronized (this) {
 				pattern = Pattern.compile(regExp);
 			}
 		}
 
-		return pattern.matcher(input).matches();
+		if (!compositeKey || compositeName == null)
+			return pattern.matcher(dataStreamName).matches();
+
+		return pattern.matcher(compositeName + '.' + dataStreamName).matches();
 	}
 }

settings.gradle

@@ -40,7 +40,7 @@ includeBuild 'examples/core/speed'
 dependencyResolutionManagement {
 	versionCatalogs {
 		libs {
-			version('fta', '17.3.1')
+			version('fta', '17.4.0')
 			version('jacoco', '0.8.12')
 
 // https://mvnrepository.com/artifact/de.siegmar/fastcsv
@@ -70,13 +70,13 @@ dependencyResolutionManagement {
 // https://mvnrepository.com/artifact/com.google.guava/guava
 			library('guava', 'com.google.guava:guava:33.5.0-jre')
 // https://mvnrepository.com/artifact/ch.qos.logback/logback-classic
-			library('logbackClassic', 'ch.qos.logback:logback-classic:1.5.25')
+			library('logbackClassic', 'ch.qos.logback:logback-classic:1.5.26')
 // https://mvnrepository.com/artifact/com.datadoghq/sketches-java
 			library('sketches', 'com.datadoghq:sketches-java:0.8.3')
 		}
 		testLibs {
 // https://mvnrepository.com/artifact/org.testng/testng
-			library('testng', 'org.testng:testng:7.11.0')
+			library('testng', 'org.testng:testng:7.12.0')
 		}
 	}
 }

types/src/main/java/com/cobber/fta/CacheLRU.java

@@ -34,7 +34,7 @@ public void put(final K key, final V value) {
         cache.put(key, value);
     }
 
-    public V get(K key) {
+    public V get(final K key) {
         return cache.getIfPresent(key);
     }
 

types/src/main/java/com/cobber/fta/Facts.java

@@ -371,23 +371,23 @@ public long getMinLong() {
 		return minLong;
 	}
 
-	public void setMinLong(long minLong) {
+	public void setMinLong(final long minLong) {
 		this.minLong = minLong;
 	}
 
 	public long getMaxLong() {
 		return maxLong;
 	}
 
-	public void setMaxLong(long maxLong) {
+	public void setMaxLong(final long maxLong) {
 		this.maxLong = maxLong;
 	}
 
 	public long getMinLongNonZero() {
 		return minLongNonZero;
 	}
 
-	public void setMinLongNonZero(long minLongNonZero) {
+	public void setMinLongNonZero(final long minLongNonZero) {
 		this.minLongNonZero = minLongNonZero;
 	}
 

types/src/main/java/com/cobber/fta/LogicalType.java

@@ -80,11 +80,36 @@ public boolean initialize(final AnalysisConfig analysisConfig) throws FTAPluginE
 	/**
 	 * Determine the confidence that the name of the data stream is likely a valid header for this Semantic Type.
 	 * Positive Numbers indicate it could be this Semantic Type, negative numbers indicate is it unlikely to be this Semantic Type, 0 indicates no opinion.
+	 * @param context The context used to interpret the Data Stream (for example, stream name, date resolution mode, etc)
+	 * @return An integer between -100 and 100 reflecting the confidence that this stream name is a valid header.
+	 */
+	public int getHeaderConfidence(final AnalyzerContext context) {
+		return pluginLocaleEntry.getHeaderConfidence(context.getCompositeName(), context.getStreamName());
+	}
+
+	/**
+	 * Determine the confidence that the name of the data stream is likely a valid header for this Semantic Type.
+	 * Positive Numbers indicate it could be this Semantic Type, negative numbers indicate is it unlikely to be this Semantic Type, 0 indicates no opinion.
+	 * @param compositeName The name of this composite (Table/File)
+	 * @param dataStreamName The name of this data stream
+	 * @return An integer between -100 and 100 reflecting the confidence that this stream name is a valid header.
+	 */
+	public int getHeaderConfidence(final String compositeName, final String dataStreamName) {
+		return pluginLocaleEntry.getHeaderConfidence(compositeName, dataStreamName);
+	}
+
+	/**
+	 * Determine the confidence that the name of the data stream is likely a valid header for this Semantic Type.
+	 * Positive Numbers indicate it could be this Semantic Type, negative numbers indicate is it unlikely to be this Semantic Type, 0 indicates no opinion.
+	 *
+	 * @deprecated  Replaced by {@link getHeaderConfidence(AnalyzerContext)}
+	 *
 	 * @param dataStreamName The name of this data stream
 	 * @return An integer between -100 and 100 reflecting the confidence that this stream name is a valid header.
 	 */
+	@Deprecated
 	public int getHeaderConfidence(final String dataStreamName) {
-		return pluginLocaleEntry.getHeaderConfidence(dataStreamName);
+		return pluginLocaleEntry.getHeaderConfidence(null, dataStreamName);
 	}
 
 	/**

types/src/main/java/com/cobber/fta/LogicalTypeBloomFilter.java

@@ -104,7 +104,7 @@ private String backout() {
 
 	@Override
 	public PluginAnalysis analyzeSet(final AnalyzerContext context, final long matchCount, final long realSamples, final String currentRegExp, final Facts facts, final FiniteMap cardinality, final FiniteMap outliers, final TokenStreams tokenStreams, final AnalysisConfig analysisConfig) {
-		final int headerConfidence = getHeaderConfidence(context.getStreamName());
+		final int headerConfidence = getHeaderConfidence(context);
 		if (headerConfidence <= 0 && cardinality.size() < 5)
 			return new PluginAnalysis(backout());
 
@@ -119,7 +119,7 @@ public double getConfidence(final long matchCount, final long realSamples, final
 		double confidence = (double)matchCount/realSamples;
 
 		// Boost by up to 20% if we like the header
-		if (getHeaderConfidence(context.getStreamName()) > 0)
+		if (getHeaderConfidence(context) > 0)
 			confidence = Math.min(confidence * 1.2, 1.0);
 
 		return confidence;

types/src/main/java/com/cobber/fta/LogicalTypeFiniteSimple.java

@@ -34,7 +34,7 @@ public abstract class LogicalTypeFiniteSimple extends LogicalTypeFinite {
 	protected Reader reader;
 	protected SingletonSet memberSet;
 
-	private static final CacheLRU<String, String> cache = new CacheLRU<>(30);
+	private static final CacheLRU<String, String> CACHE = new CacheLRU<>(30);
 
 	public LogicalTypeFiniteSimple(final PluginDefinition plugin, final String backout, final int threshold) {
 		super(plugin);
@@ -82,7 +82,7 @@ public boolean initialize(final AnalysisConfig analysisConfig) throws FTAPluginE
 		// If the Regular Expression has not been set then generate one based on the content
 		if (regExp == null) {
 			final String cacheKey = semanticType + "___" + analysisConfig.getLocaleTag();
-			regExp = cache.get(cacheKey);
+			regExp = CACHE.get(cacheKey);
 			if (regExp != null)
 				return true;
 
@@ -92,7 +92,7 @@ public boolean initialize(final AnalysisConfig analysisConfig) throws FTAPluginE
 			       gen.train(elt);
 
 			regExp = gen.getResult();
-			cache.put(cacheKey, regExp);
+			CACHE.put(cacheKey, regExp);
 		}
 
 		return true;
@@ -111,7 +111,7 @@ public String getRegExp() {
 	@Override
 	public PluginAnalysis analyzeSet(final AnalyzerContext context, final long matchCount, final long realSamples, final String currentRegExp,
 			final Facts facts, final FiniteMap cardinality, final FiniteMap outliers, final TokenStreams tokenStreams, final AnalysisConfig analysisConfig) {
-		final int headerConfidence = getHeaderConfidence(context.getStreamName());
+		final int headerConfidence = getHeaderConfidence(context);
 		final int baseOutliers = ((100 - getThreshold()) * getSize())/100;
 
 		int maxOutliers = Math.max(1, baseOutliers / 2);