Timeouts for AWS Cluster configuration

[bernieatenovlab]

Hi Jakub.

I'm using your ssm libraries to connect to a Memcached CLUSTER in AWS. As my personal development instances, I'm running a REST Spring Boot API locally on my laptop which connects to the AWS Elasticache/Memcached instance in us-east-1 through a VPN tunnel. However the logs show below error once every Minute:

2018-02-15 16:29:38.186 ERROR net.spy.memcached.ConfigurationPoller:  Error encountered in the poller. Current cluster configuration: null
java.lang.RuntimeException: Exception waiting for config
	at net.spy.memcached.MemcachedClient.getConfig(MemcachedClient.java:1947)
	at net.spy.memcached.ConfigurationPoller.run(ConfigurationPoller.java:126)
	at java.util.concurrent.Executors$RunnableAdapter.call$$$capture(Executors.java:511)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: /10.2.1.21:11211
	at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:180)
	at net.spy.memcached.internal.GetConfigFuture.get(GetConfigFuture.java:52)
	at net.spy.memcached.MemcachedClient.getConfig(MemcachedClient.java:1935)
	... 9 more
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: /10.2.1.21:11211
	... 12 more
2018-02-15 16:29:38.187 WARN net.spy.memcached.ConfigurationPoller:  Number of consecutive poller errors is 1. Number of minutes since the last successful polling is 0

Below thread shows other folks experiencing the same error since 2013
https://forums.aws.amazon.com/thread.jspa?threadID=121885

I did a little deep dive and found that in ClientMode.Dynamic mode, net.spy.memcached sets up a Scheduled Thread Poller that runs once a Minute which downloads the configuration from AWS Memcached. net.spy.memcached by default assigns this "get configuration" call a timeout of 2.5 Seconds. However for some reason the call for me (and probably the folks in above AWS forum post) takes just under 5 Seconds and therefore constantly times out.

While looking at the MemcacheClientFactoryImpl code, I can see that you instantiate a net.spy.memcached.MemcachedClient class with "only" a list of IP Addresses (new MemcachedClient(addrs)). MemcachedClient subsequently assumes default timeout values since no configuration was provided.

Would it be possible to change your code so that the caller of ssm can provide net.spy.memcached configuration properties, which then get passed on to the MemcachedClient (e.g. by using new MemcachedClient(connectionFactory, addrs) instead of new MemcachedClient(addrs))?

Your opinion is much appreciated.

Regards
Bernie

[ragnor]

Hi Bernie,

Have you tried the latest version of SSM and aws-cluster-provider? The constructor that you want to use is already available in SSM: MemcacheClientFactoryImpl. You just have to provide as a configuration ElastiCacheConfiguration object.
What property you would like to set on connectionFactory?

[bernieatenovlab]

Hi Jakub.

Yes I'm using the 4.0.0 version.

The property that needs to be set is net.spy.memcached.ConnectionFactory.getOperationTimeout(). If the MemcachedClient(...) is instantiated with just addr (without a ConnectionFactory), then it internally creates a DefaultConnectionFactory() object which has getOperationTimeout() hardcoded as 2500L.

Here is the code complete code how ssm is instantiated, which was taken from an example on stackoverflow (https://stackoverflow.com/questions/26978823/spring-cache-using-memcached)

    @Value("${cache.enabled}")
    private boolean cacheEnabled;

    @Autowired
    private Environment environment;

    @Bean
    public CacheBase cacheBase() {
        return new CacheBase();
    }

    @Bean
    public Cache defaultCache() {
        try {
            CacheFactory cacheFactory = cacheFactory();
            Cache cache = cacheFactory.getObject();
            return cache;
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    @Bean
    @Override
    public CacheManager cacheManager() {
        if (!cacheEnabled) {
            return new NoOpCacheManager();
        }
        Set<SSMCache> ssmCacheSet = new HashSet<>();
        SSMCache ssmCache = new SSMCache(defaultCache(), 300, false);
        ssmCacheSet.add(ssmCache);
        SSMCacheManager ssmCacheManager = new SSMCacheManager();
        ssmCacheManager.setCaches(ssmCacheSet);
        return ssmCacheManager;
    }

    @Bean
    @DependsOn("cacheBase")
    public CacheFactory cacheFactory() {
        CacheFactory cacheFactory = new CacheFactory();
        cacheFactory.setCacheName("defaultCache");
        MemcacheClientFactoryImpl memcacheClientFactory = new MemcacheClientFactoryImpl();
        cacheFactory.setCacheClientFactory(memcacheClientFactory);
        String cacheServers = environment.getProperty("cache.servers");
        LOGGER.debug("cacheServers='{}'", cacheServers);
        cacheFactory.setAddressProvider(new DefaultAddressProvider(cacheServers));
        CacheConfiguration cacheConfiguration = createCacheConfiguration();
        cacheFactory.setConfiguration(cacheConfiguration);
        return cacheFactory;
    }

    private CacheConfiguration createCacheConfiguration() {
        ElastiCacheConfiguration cacheConfiguration = new ElastiCacheConfiguration();
        cacheConfiguration.setConsistentHashing(true);
        cacheConfiguration.setClientMode(ClientMode.Dynamic);
        cacheConfiguration.setTimeoutExceptionThreshold(100000);
        cacheConfiguration.setOperationTimeout(100000); //This is not taking effect
        cacheConfiguration.setOpTimeout(100000L);
        cacheConfiguration.setUseAutoDiscovery(true);
        return cacheConfiguration;
    }

The issue is obviously with MemcacheClientFactoryImpl memcacheClientFactory = new MemcacheClientFactoryImpl();

Do you have a recommendation on how to better instantiate the MemcacheClientFactoryImpl so OperationTime can be specified?

Regards
Bernie

[ragnor]

Hi Bernie,

Could you please just try to comment cacheConfiguration.setUseAutoDiscovery(true) in createCacheConfiguration() a check if the operation timeout is applied? If ClientMode.Dynamic is set then even if useAutoDiscovery is set false the it should work as previously but opTimoute should be applied as different constructor is used in SSM to create MemcachedClient.

Regards,
Jakub

[bernieatenovlab]

Thanks Jakub, that worked! I assumed the UseAutoDiscovery flag was used for the client to retrieve its configuration from memcached but as you pointed out, instead the ClientMode.Dynamic setting is responsible for that.
Should the setUseAutoDiscovery(...) method be deprecated to make it less confusing? Rather than using setUseAutoDiscovery(...) users should use or not use cacheFactory.setConfiguration(cacheConfiguration)?

[ragnor]

Right, I'll add info to setUseAutoDiscovery that when set to true then cacheConfiguration isn't taken into account but default provider's settings are used.

[bernieatenovlab]

Thanks, case closed!

[ragnor]

Related code has been released.