Consolidate caches (#202)

* #199 Consolidating caches.

Author: jzonthemtn

docs/docs/other_features/anonymization.md

@@ -1,19 +1,27 @@
 # Consistent Anonymization
 
-Anonymization in the context of Phileas is the process of replacing certain values with random but similar values. For example, the identified name of “John Smith” may be replaced with “David Jones”, or an identified phone number of 123-555-9358 may be replaced by 842-436-2042. A [VIN](vins.md) number will be replaced by a 17 character randomly selected VIN number that adheres to the standard for VIN numbers.
+Anonymization in the context of Phileas is the process of replacing certain values with random but similar values. For
+example, the identified name of “John Smith” may be replaced with “David Jones”, or an identified phone number of
+123-555-9358 may be replaced by 842-436-2042. A [VIN](vins.md) number will be replaced by a 17 character randomly
+selected VIN number that adheres to the standard for VIN numbers.
 
-Anonymization is useful in instances where you want to remove sensitive information from text without changing the meaning of the text. Anonymization can be enabled for each type of sensitive information in the policy by setting the filter strategy to `RANDOM_REPLACE`. (See [Policies](policies_README.md) for more information.)
+Anonymization is useful in instances where you want to remove sensitive information from text without changing the
+meaning of the text. Anonymization can be enabled for each type of sensitive information in the policy by setting the
+filter strategy to `RANDOM_REPLACE`. (See [Policies](policies_README.md) for more information.)
 
 ## Consistent Anonymization
 
-Consistent anonymization refers to the process of always anonymizing the same sensitive information with the same replacement values. For example, if the name "John Smith" is randomly replaced with "Pete Baker", all other occurrences of "John Smith" will also be replaced by "Pete Baker."
+Consistent anonymization refers to the process of always anonymizing the same sensitive information with the same
+replacement values. For example, if the name "John Smith" is randomly replaced with "Pete Baker", all other occurrences
+of "John Smith" will also be replaced by "Pete Baker."
 
-Consistent anonymization can be done on the document level or on the context level. When enabled on the document level, "John Smith" will only be replaced by "Pete Baker" in the same document. If "John Smith" occurs in a separate document it will be anonymized with a different random name. When enabled on the context level, "John Smith" will be replaced by "Pete Baker" whenever "John Smith" is found in all documents in the same context.
+Consistent anonymization can be done on the document level or on the context level. When enabled on the document
+level, "John Smith" will only be replaced by "Pete Baker" in the same document. If "John Smith" occurs in a separate
+ document, it will be anonymized with a different random name. When enabled on the context level, "John Smith" will be
+replaced by "Pete Baker" whenever "John Smith" is found in all documents in the same context.
 
-Enabling consistent anonymization on the context level requires a cache to store the sensitive information and the corresponding replacement values. If a single instance of Phileas is running, its internal cache service (enabled by default) is the best choice and no additional configuration is required.
+Enabling consistent anonymization on the context level requires a cache to store the sensitive information and the
+corresponding replacement values. If a single instance of Phileas is running, its internal cache service (enabled by
+default) is the best choice and no additional configuration is required.
 
-If multiple instances of Phileas are deployed together, Phileas requires access to a Redis cache service as shown below. See Phileas' [Settings](settings.md) on how to configure the cache.
 
-**When Phileas is deployed in a cluster, a Redis cache is required to enable consistent anonymization.**
-
-The anonymization cache will contain PHI. It is important that you take the necessary precautions to secure the cache and all communication to and from the cache.

docs/docs/settings.md

@@ -2,7 +2,10 @@
 
 Phileas has settings to control how it operates. The settings and how to configure each are described below.
 
-> The configuration for the types of sensitive information that Phileas identifies are defined in [filter policies](filter_policies/filter_policies.md) outside of Phileas' configuration properties described on this page.
+> The configuration for the types of sensitive information that Phileas identifies are defined
+> in [filter policies](filter_policies/filter_policies.md) outside of Phileas' configuration properties described on
+> this
+> page.
 
 ## Configuring Phileas
 
@@ -14,57 +17,39 @@ Phileas looks for its settings in an `application.properties` file.
 
 Properties set via environment variables take precedence over properties set in Phileas' settings file.
 
-All following properties can also be set as environment variables by prepending `PHILTER_` to the property name and changing periods to underscores. For example, the property `filter.profiles.directory` can be set using the environment variable `PHILTER_FILTER_PROFILES_DIRECTORY` by:
+All following properties can also be set as environment variables by prepending `PHILTER_` to the property name and
+changing periods to underscores. For example, the property `filter.profiles.directory` can be set using the environment
+variable `PHILTER_FILTER_PROFILES_DIRECTORY` by:
 
 ```
 export PHILTER_FILTER_PROFILES_DIRECTORY=/profiles/
 ```
 
-Using environment variables to configure Phileas instead of using Phileas' settings file can allow for easier configuration management when deploying Phileas.
+Using environment variables to configure Phileas instead of using Phileas' settings file can allow for easier
+configuration management when deploying Phileas.
 
 ## Policies
 
 | Setting                     | Description                                  | Allowed Values            | Default Value |
-| --------------------------- | -------------------------------------------- | ------------------------- | ------------- |
+|-----------------------------|----------------------------------------------|---------------------------|---------------|
 | `filter.policies.directory` | The directory in which to look for policies. | Any valid directory path. | `./policies/` |
 
 ## Span Disambiguation
 
-These values configure Phileas' span disambiguation feature to determine the most appropriate type of sensitive information when duplicate spans are identified. In a deployment of multiple Phileas instances, you must enable the [cache service](Settings#cache) for span disambiguation to work as expected.
+These values configure Phileas' span disambiguation feature to determine the most appropriate type of sensitive
+information when duplicate spans are identified. In a deployment of multiple Phileas instances, you must enable
+the [cache service](Settings#cache) for span disambiguation to work as expected.
 
 |                               | Description                                   | Allowed Values  | Default Value |
-| ----------------------------- | --------------------------------------------- | --------------- | ------------- |
+|-------------------------------|-----------------------------------------------|-----------------|---------------|
 | `span.disambiguation.enabled` | Whether or not to enable span disambiguation. | `true`, `false` | `false`       |
 
-## Cache Service
-
-The cache service is required to use [consistent anonymization](anonymization.md) and policies stored in Amazon S3. Phileas supports Redis as the backend cache. When Redis is not used, an in-memory cache is used instead. The in-memory cache is not recommended because all contents will be stored in memory on the local Phileas instance.
-
-The cache will contain sensitive information. It is important that you take the necessary precautions to secure the cache itself and all communication between Phileas and the cache.
-
-| Setting                  | Description                                                       | Allowed Values            | Default Value |
-| ------------------------ | ----------------------------------------------------------------- | ------------------------- | ------------- |
-| `cache.redis.enabled`    | Whether or not to use Redis as the cache.                         | `true`, `false`           | `false`       |
-| `cache.redis.host`       | The hostname or IP address of the Redis cache.                    | Any valid Redis endpoint. | None          |
-| `cache.redis.port`       | The Redis cache port.                                             | Any valid port.           | `6379`        |
-| `cache.redis.auth.token` | The Redis auth token.                                             | Any valid token.          | None          |
-| `cache.redis.ssl`        | Whether or not to use SSL for communication with the Redis cache. | `true`, `false`           | `false`       |
-
-The following Redis settings are only required when using a self-signed SSL certificate.
-
-| Setting                           | Description                  | Allowed Values       | Default Value |
-| --------------------------------- | ---------------------------- | -------------------- | ------------- |
-| `cache.redis.truststore`          | The path to the trust store. | Any valid file path. | None          |
-| `cache.redis.truststore.password` | The trust store password.    | Any valid file path. | None          |
-| `cache.redis.keystore`            | The path to the keystore.    | Any valid file path. | None          |
-| `cache.redis.keystore.password`   | The keystore password.       | Any valid file path. | None          |
-
 ## Advanced Settings
 
-> In most cases the settings below do not need changed. Contact us for more information on any of these settings.
+> In most cases, the settings below do not need changed. Contact us for more information on any of these settings.
 
 | Setting                      | Description                                                                                                                                  | Allowed Values    | Default Value |
-| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ------------- |
+|------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|-------------------|---------------|
 | `ner.timeout.sec`            | Controls the timeout in seconds when performing name entity recognition. Longer text may require longer processing times.                    | An integer value  | `600`         |
 | `ner.max.idle.connections`   | The maximum number of idle connections to maintain for the named entity recognition. More connections may improve performance in some cases. | An integer value. | `30`          |
 | `ner.keep.alive.duration.ms` | The amount of time in milliseconds to keep named entity recognition connections alive. Longer text may require longer processing times.      | An integer value. | `60`          |

phileas-core/pom.xml

@@ -114,11 +114,5 @@
 			<version>${junit.version}</version>
 			<scope>test</scope>
 		</dependency>
-		<dependency>
-			<groupId>com.github.codemonstur</groupId>
-			<artifactId>embedded-redis</artifactId>
-			<version>${embedded.redis.version}</version>
-			<scope>test</scope>
-		</dependency>
 	</dependencies>
 </project>

phileas-core/src/main/java/ai/philterd/phileas/services/FilterPolicyLoader.java

@@ -27,7 +27,7 @@
 import ai.philterd.phileas.model.policy.filters.Identifier;
 import ai.philterd.phileas.model.policy.filters.Section;
 import ai.philterd.phileas.model.services.AlertService;
-import ai.philterd.phileas.model.services.AnonymizationCacheService;
+import ai.philterd.phileas.model.services.CacheService;
 import ai.philterd.phileas.model.services.MetricsService;
 import ai.philterd.phileas.model.services.SpanValidator;
 import ai.philterd.phileas.services.anonymization.AgeAnonymizationService;
@@ -99,13 +99,13 @@ public class FilterPolicyLoader {
 
     private static final Logger LOGGER = LogManager.getLogger(FilterPolicyLoader.class);
 
-    private final AnonymizationCacheService anonymizationCacheService;
+    private final CacheService anonymizationCacheService;
     private final AlertService alertService;
     private final MetricsService metricsService;
     private final Map<String, DescriptiveStatistics> stats;
     private final PhileasConfiguration phileasConfiguration;
 
-    public FilterPolicyLoader(final AlertService alertService, final AnonymizationCacheService anonymizationCacheService,
+    public FilterPolicyLoader(final AlertService alertService, final CacheService anonymizationCacheService,
                               final MetricsService metricsService, final Map<String, DescriptiveStatistics> stats,
                               final PhileasConfiguration phileasConfiguration) {
 

phileas-core/src/main/java/ai/philterd/phileas/services/PhileasFilterService.java

@@ -21,6 +21,7 @@
 import ai.philterd.phileas.model.domain.LegalDomain;
 import ai.philterd.phileas.model.enums.FilterType;
 import ai.philterd.phileas.model.enums.MimeType;
+import ai.philterd.phileas.model.services.Classification;
 import ai.philterd.phileas.model.filter.Filter;
 import ai.philterd.phileas.model.objects.Explanation;
 import ai.philterd.phileas.model.objects.PdfRedactionOptions;
@@ -33,8 +34,7 @@
 import ai.philterd.phileas.model.responses.FilterResponse;
 import ai.philterd.phileas.model.serializers.PlaceholderDeserializer;
 import ai.philterd.phileas.model.services.AlertService;
-import ai.philterd.phileas.model.services.AnonymizationCacheService;
-import ai.philterd.phileas.model.services.Classification;
+import ai.philterd.phileas.model.services.CacheService;
 import ai.philterd.phileas.model.services.DocumentProcessor;
 import ai.philterd.phileas.model.services.FilterService;
 import ai.philterd.phileas.model.services.MetricsService;
@@ -45,8 +45,7 @@
 import ai.philterd.phileas.model.services.SplitService;
 import ai.philterd.phileas.processors.unstructured.UnstructuredDocumentProcessor;
 import ai.philterd.phileas.service.ai.sentiment.OpenNLPSentimentDetector;
-import ai.philterd.phileas.services.alerts.AlertServiceFactory;
-import ai.philterd.phileas.services.anonymization.cache.AnonymizationCacheServiceFactory;
+import ai.philterd.phileas.services.alerts.DefaultAlertService;
 import ai.philterd.phileas.services.disambiguation.VectorBasedSpanDisambiguationService;
 import ai.philterd.phileas.services.metrics.NoOpMetricsService;
 import ai.philterd.phileas.services.policies.InMemoryPolicyService;
@@ -98,11 +97,11 @@ public class PhileasFilterService implements FilterService {
     // PHL-223: Face recognition
     //private final ImageProcessor imageProcessor;
 
-    public PhileasFilterService(final PhileasConfiguration phileasConfiguration) throws IOException {
-        this(phileasConfiguration, new NoOpMetricsService());
+    public PhileasFilterService(final PhileasConfiguration phileasConfiguration, final CacheService cacheService) throws IOException {
+        this(phileasConfiguration, new NoOpMetricsService(), cacheService);
     }
 
-    public PhileasFilterService(final PhileasConfiguration phileasConfiguration, final MetricsService metricsService) throws IOException {
+    public PhileasFilterService(final PhileasConfiguration phileasConfiguration, final MetricsService metricsService, final CacheService cacheService) throws IOException {
 
         LOGGER.info("Initializing Phileas engine.");
 
@@ -112,23 +111,20 @@ public PhileasFilterService(final PhileasConfiguration phileasConfiguration, fin
         final Gson gson = new GsonBuilder().registerTypeAdapter(String.class, new PlaceholderDeserializer()).create();
 
         // Set the policy services.
-        this.policyService = buildPolicyService(phileasConfiguration);
+        this.policyService = buildPolicyService(phileasConfiguration, cacheService);
         this.policyUtils = new PolicyUtils(policyService, gson);
 
-        // Set the anonymization cache service.
-        final AnonymizationCacheService anonymizationCacheService = AnonymizationCacheServiceFactory.getAnonymizationCacheService(phileasConfiguration);
-
         // Set the alert service.
-        this.alertService = AlertServiceFactory.getAlertService(phileasConfiguration);
+        this.alertService = new DefaultAlertService(cacheService);
 
         // Instantiate the stats.
-        Map<String, DescriptiveStatistics> stats = new HashMap<>();
+        final Map<String, DescriptiveStatistics> stats = new HashMap<>();
 
         // The filter loader for policies.
-        this.filterPolicyLoader = new FilterPolicyLoader(alertService, anonymizationCacheService, metricsService, stats, phileasConfiguration);
+        this.filterPolicyLoader = new FilterPolicyLoader(alertService, cacheService, metricsService, stats, phileasConfiguration);
 
         // Create a new unstructured document processor.
-        this.unstructuredDocumentProcessor = new UnstructuredDocumentProcessor(metricsService, new VectorBasedSpanDisambiguationService(phileasConfiguration));
+        this.unstructuredDocumentProcessor = new UnstructuredDocumentProcessor(metricsService, new VectorBasedSpanDisambiguationService(phileasConfiguration, cacheService));
 
     }
 
@@ -367,12 +363,12 @@ public BinaryDocumentFilterResponse filter(final List<String> policyNames, final
 
     }
 
-    private PolicyService buildPolicyService(final PhileasConfiguration phileasConfiguration) {
+    private PolicyService buildPolicyService(final PhileasConfiguration phileasConfiguration, final CacheService cacheService) {
 
         if(StringUtils.equalsIgnoreCase(phileasConfiguration.policyService(), "memory")) {
             return new InMemoryPolicyService();
         } else {
-            return new LocalPolicyService(phileasConfiguration);
+            return new LocalPolicyService(phileasConfiguration, cacheService);
         }
 
     }