Deadlock with background EntityManager bootstrap due to locking in getSingletonFactoryBeanForTypeCheck

[ttddyy]

Hi team,

A user reported an issue on my project (jdbc-observations/datasource-micrometer#54) with a sample repro where the application hangs during startup. Upon debugging, I traced this change (384dc2a) for 6.2.1 caused the issue.

---

Conditions:

• Enable background EntityManager creation
  • With Spring Boot, spring.data.jpa.repositories.bootstrap-mode is set to lazy or deferred (handled by JpaRepositoriesAutoConfiguration).
• Lazy BeanFactory access in beans associated with EntityManager
  • A dependent bean (e.g., a bean used by DataSource) accessed lazily via ObjectProvider during EntityManagerFactory creation.

---

Problem:

A deadlock occurs between the main thread and the background thread creating the EntityManagerFactory.

1. The main thread holds the singletonLock while waiting for the EntityManagerFactory to be created.
2. The EntityManagerFactory is being created in a separate thread, and a dependent bean in that process tries to lazily access the BeanFactory via ObjectProvider.
3. The ObjectProvider access in the background thread tries to acquire the singletonLock, leading to a deadlock.

---

Detailed Code Flow:

1. Background Thread:

   • With background bootstrap enabled, LocalContainerEntityManagerFactoryBean initiates the creation of the EntityManagerFactory (NativeEntityManagerFactory) in a separate thread (via a task submission, holding a Future for the result).

2. Main Thread:

   • While the background task runs, the main thread continues creating other beans, including JPA-related factory beans such as JpaMetamodelMappingContextFactoryBean.
   • JpaMetamodelMappingContextFactoryBean (a subclass of AbstractFactoryBean) invokes its afterPropertiesSet() method to create a bean while singletonLock is acquired.
   • During the creation of the JpaMetamodelMappingContext, a call to getMetamodels() is made, which invokes a proxy and calls AbstractEntityManagerFactoryBean#getNativeEntityManagerFactory(). This method calls a blocking call to the Future, waiting for the background thread to complete EntityManagerFactory creation.

3. Background Thread (Continued):

   • During EntityManagerFactory creation, JDBC-related beans (e.g., DataSource) are instantiated.
   • With datasource-micrometer, the DataSource bean references other beans, one of which lazily resolves an observationRegistry via ObjectProvider.
   • A database query is triggered as part of the Hibernate entity manager creation process(retrieving the database metadata), causing the ObjectProvider to attempt to access the beanFactory. It tries to acquire the singletonLock, which is already held by the main thread, resulting in a deadlock.

---

This is a repro sample code capturing the above scenario concept:

Simple Repro Code

class MyTest {

    private static final Logger log = LoggerFactory.getLogger(MyTest.class);

    @Test
    void test() {
        ApplicationContext context = new AnnotationConfigApplicationContext(MyConfiguration.class);
        Object foo = context.getBean("foo");
        assertThat(foo).isInstanceOf(Foo.class);
    }

    static class Foo {

    }

    static class Bar {
    }

    // extend "AbstractFactoryBean" to simulate "JpaMetamodelMappingContextFactoryBean"
    static class FooFactory extends AbstractFactoryBean<Foo> {

        private ObjectProvider<Bar> barProvider;

        public FooFactory(ObjectProvider<Bar> barProvider) {
            this.barProvider = barProvider;
        }

        @Override
        protected Foo createInstance() throws Exception {
            // retrieve a bean in a separate thread. emulating "EntityManagerFactory" creation
            // in "AbstractEntityManagerFactoryBean#afterPropertiesSet"
            Future<Bar> future = Executors.newSingleThreadExecutor().submit(() -> {
                log.info("getting bar before");
                Bar bar = this.barProvider.getObject();  // <== deadlock here
                // against "singletonLock"("DefaultSingletonBeanRegistry") used by "AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck"
                log.info("getting bar after");
                return bar;
            });
            Bar bar = future.get();
            log.info("bar={}", bar);
            return new Foo();
        }

        @Override
        public Class<?> getObjectType() {
            return Foo.class;
        }
    }

    @Configuration(proxyBeanMethods = false)
    @Import(FooFactoryRegistrar.class)
    static class MyConfiguration {

        @Bean
        Bar bar() {
            return new Bar();
        }

        // Alternate to bean definition registrar
        // @Bean("foo")
        // static FooFactory fooFactory(ObjectProvider<Bar> barProvider) {
        //     return new FooFactory(barProvider);
        // }

    }

    // simulating "JpaAuditingRegistrar" which registers "JpaMetamodelMappingContextFactoryBean"
    static class FooFactoryRegistrar implements ImportBeanDefinitionRegistrar {
        @Override
        public void registerBeanDefinitions(AnnotationMetadata importingClassMetadata,
                BeanDefinitionRegistry registry) {
            registry.registerBeanDefinition("foo", new RootBeanDefinition(FooFactory.class));
        }
    }
}

---

I'm not sure if this is to be called a regression or a new requirement for 6.2.1.
If it is a new behavior, the limitation would be that any EntityManager infrastructure beans (such as beans related to DataSource) cannot have lazy access to the BeanFactory when background EntityManager bootstrap is enabled due to the use of a separate thread.

[jhoeller]

I've revised getSingletonFactoryBeanForTypeCheck to defensively acquire the singleton lock, backing out with null if currently held by another thread. This should restore the balance between consistent processing and lenient locking and make your scenario work again.

[jhoeller]

@ttddyy please rerun your scenario against the latest 6.2.2 snapshot and let me know whether it makes things any better.

[ttddyy]

Thanks for the quick turnaround.
I've confirmed that it is working with 6.2.2-SNAPSHOT.

[jhoeller]

Good to hear! Thanks for the immediate feedback, @ttddyy.