How to Configure Read Replica

[risenhoover]

I have been successfully using a read replica with Play Framework 2.7 for years. I am upgrading to Play Framework 2.9 and Ebean 15.1.0 with Play Ebean Plugin 7.3.0.

I have tried two different configurations and with both of them, the default database is recognized and used, but the read replica is not. Here are my two attempts:

db.default.driver=org.postgres.Driver
db.default.url='xxxx
db.default.readOnlyUrl='xxx`

db.default.driver=org.postgres.Driver
db.default.url='xxxx

db.readonly.driver=org.postgres.Driver
db.readonly.url='xxxx'
db.readonly.readOnlyDatabase=true

Here is a link to the documentation that shows the 'readOnlyUrl' property in the configuration.
https://github.com/ebean-orm/ebean-datasource/blob/master/ebean-datasource-api/src/main/java/io/ebean/datasource/DataSourceConfig.java#L719

Please advise on the proper way to configure a read replica.

[rbygrave]

Refer: https://ebean.io/docs/read-replicas/

Specifying readOnlyUrl is what you should be doing [given that this is using the ebean datasource which is aware of the readOnlyUrl property].

db.default.url='xxxx
db.default.readOnlyUrl='xxx`

Note that if an application is not using the ebean datasource, and instead using something like Hikari, then that application needs to specify a separate DataSource.

Note that is the application wants to specify different min/max connections for the read replica connection pool, then it should use the DataSourceBuilder instead via:
https://ebean.io/apidoc/15/io.ebean.api/io/ebean/DatabaseBuilder.html#dataSourceBuilder(io.ebean.datasource.DataSourceBuilder)
https://ebean.io/apidoc/15/io.ebean.api/io/ebean/DatabaseBuilder.html#dataSourceBuilder(io.ebean.datasource.DataSourceBuilder)

[risenhoover]

Alternatively, how can I configure to use the ebean datasource?

[rbygrave]

which makes me believe that we're using Hikari

To me, I'm expecting you to tell us that part? ... to say what your application is doing OR to find someone in your team that can answer that question and show you where that setup is and how it works?

To me, I think it would be good for you to use a debugger and step through the code when the application starts? Then you could answer these questions but also future questions about how the application is configured and what future configuration options are available to the application?

The play-ebean plugin is different to vanilla ebean in terms of setup so maybe have a quick look at the code in that plugin and put a debugger break point in there at a likely place that it configures and starts ebean?

[risenhoover]

A quick recap of what I have tried and how I have arrived where I am.

1. As I mentioned, I first tried the following in application.conf:

db.default.url='xxxx
db.default.readOnlyUrl='xxx`

It's important to note that there is no documentation stating this configuration is viable. I found the "readOnlyUrl" parameter deep within the github repo and concluded that it might work, so I tried it. There is zero documentation stating that it does not work without using the ebean datasource.

I placed a breakpoint at

ebean/ebean-api/src/main/java/io/ebean/DatabaseFactory.java

Line 78 in 1b5e066

var config = builder.settings();

and was able to confirm that the readOnlyUrl part of the configuration seems to be dropped prior to arriving at this spot.

2. Then I tried the same configuration but in ebean.properties. This time, I was able to confirm that the readOnlyUrl parameter was not lost at load time, but passed into the configuration and then on to the process that creates the Database. However, I saw no evidence that the readonly database was created.

3. Then I tried the standard play/ebean setup with two databases in application.conf, one marked as readonly:

db.default.driver=org.postgresql.Driver
db.default.url="postgres://xxx"

db.readonly.driver=org.postgresql.Driver
db.readonly.url="postgres://xxx"
db.readonly.readOnlyDatabase=true

As I stated in my initial post, with this setup, I see no evidence that the readonly database is started. There is nothing in the logs indicating it started. Also, when I step through the code at startup and place a breakpoint at

ebean/ebean-core/src/main/java/io/ebeaninternal/server/core/DefaultContainer.java

Line 94 in 1b5e066

boolean online = true;

- I can see only one database being created, the default database. There is no mention of a readonly database.

4. Then I tried the same two-datasource config approach but in ebean.properties file, with the same results. Arguably, this issue is no longer about setting up a read replica, but rather just getting a second database recognized.

5. I thought I might try a different approach, so I created a ServerConfigStartup class to modify the config to my liking. I created the following and added it to my models directory:

public class MyServerConfigStartup implements ServerConfigStartup {
  public void onStart(DatabaseBuilder config) {

    /*
     * set the read only config
     */
    DataSourceBuilder.Settings dbConfig = DataSourceBuilder.create().settings();

    dbConfig.driver("org.postgresql.Driver");
    dbConfig.username("x");
    dbConfig.password("xx");
    dbConfig.url("postgres://xx');
    dbConfig.readOnlyUrl("postgres://xx");

    config.readOnlyDataSourceBuilder(dbConfig);
  }
}

I set a breakpoint in that code and was able to confirm that it loads and executes, but I get the following exception later in the startup (I am not using evolutions):

@8a92a55c5: Execution exception[[CreationException: Unable to create injector, see the following errors:

1) [Guice/ErrorInjectingConstructor]: NullPointerException
  at EbeanDynamicEvolutions.<init>(EbeanDynamicEvolutions.java:42)
  at EbeanDynamicEvolutions.class(EbeanDynamicEvolutions.java:35)
  while locating EbeanDynamicEvolutions
  at EbeanModule.bindings(EbeanModule.java:20):
Binding(class DynamicEvolutions to ConstructionTarget(class EbeanDynamicEvolutions) eagerly)
      \_ installed by: Modules$OverrideModule -> GuiceableModuleConversions$$anon$4
  while locating DynamicEvolutions
  while locating ApplicationEvolutionsProvider
  at EvolutionsModule.<init>(EvolutionsModule.scala:23):
Binding(class ApplicationEvolutions to ProviderConstructionTarget(class ApplicationEvolutionsProvider) eagerly)
      \_ installed by: Modules$OverrideModule -> GuiceableModuleConversions$$anon$4
  while locating ApplicationEvolutions

Thinking that perhaps I was overwriting something that I shouldn't be, I tried a variation on that code:

public class MyServerConfigStartup implements ServerConfigStartup {
  public void onStart(DatabaseBuilder config) {

    DataSourceBuilder.Settings dbConfig = config.settings().getDataSourceConfig()
    config.readOnlyDataSourceBuilder(dbConfig.readOnlyUrl("xxx"));
  }
}

This produced the same guice-originated NullPointerException related to EbeanDynamicEvolutions.

Besides the work described above, I have spent some time searching for any documentation that describes how to use the ebean datasource instead of Hikari, but with no luck. This research includes sifting through github repos.

Going forward, I will be focusing on just trying to get Play/ebean to recognize a second database, without regard to the readOnly attribute for now.

If anybody has been down this path before and can share your experiences, I would appreciate it.

[rbygrave]

However, I saw no evidence that the readonly database was created.

There are 2 different things:

A) A single instance of ebean Database with 2 connection pools - 1 of those pools dedicated to read only queries. This is the expected setup to use when we have both reads and writes to a database and we want to use a database relica.

B) A ebean Database that is set to be a readOnlyDatabase = true ... where the only use of this is expected to be read only, and this will use a single DataSource and that should be optimised for read only use with autoCommit=true and readOnly=true [and this ebean database instance will not be able to write to the database].

3. Then I tried the standard play/ebean setup with two databases in application.conf, one marked as readonly:

That isn't really optimal per se in the sense it's looking to create a whole 2nd ebean Database instance, which is ok'ish but now we have 2 ebean instances and need to choose in the application where to use which instance. Using option A is more transparent to the application - ebean knows when a read only query is being executed outside of an explicit transaction, and for this case can transparently make use of the read only connection pool [and that is typically going to use a read replica db instance].

The docs at https://ebean.io/docs/read-replicas/ ... are all about option A where there is a single ebean Database instance with 2 connection pools and one of those pools is setup for read only use [typically against a read replica but it doesn't have to be and there are still benefits to using an DataSource optimised for read only use against a single db instance].

As I stated in my initial post, with this setup, I see no evidence that the readonly database is started.

This is up to the play ebean plugin and how that works. Ebean doesn't use application.conf - this is all on the play side of things. Yeah, I'm not up to date on that plugin and if it can set the read only url etc.

[risenhoover]

Ok so in your opinion, given what I have tried, what is the easiest and fastest path to solution A? To me, it seems like the ServerConfigStartup is purpose built for this problem, but I cannot seem to get it to work.