OpenAM 15.1.6 performance degradation issue

[AndressRod]

Hello Everyone. I'm facing some performance degradation issue with OpenAM 15.1.6.

I made a JMeter test to simulate multiple OpenAM Authentications. I made 6k requests on 60 seconds.

All of the requests results were 200 OK. The issue came after that testing because I wasn't able to login anymore on OpenAM, even with amadmin user. If I try to login, a loading icon appears and after several seconds an error message appears. The issue persists even if I reboot the pod that handles the container.

Do I need to do a special configuration for production environment? I'm attaching a Graph of the CPU usage and Memory Usage behavior when the test was executed and also my deployment configuration and debug logs. Thanks in advance!

debug-log.tar.gz

[maximthomas]

Hi @AndressRod,
Does the issue persist if you increase the amount of resources dedicated to OpenAM?
Are you using an embedded OpenDJ instance or a dedicated one?
Please also attach the OpenDJ logs.

[AndressRod]

Hi @maximthomas.

Yes, the issue persists if I increase the amount of resources. The logs you need are located under /usr/openam/config/opends/logs right? I'm using OpenDJ as a DataStore but didn't found any issue:

Making some research, I found this useful links (my bad for not founding them before).

• https://doc.openidentityplatform.org/openam/deployment-planning/chap-hw-sw-requirements
• https://doc.openidentityplatform.org/openam/admin-guide/chap-tuning#chap-tuning

And made the following changes based on the documentation:

With these settings, the behavior improved a lot. But after a few hours, the issue reappeared.

[maximthomas]

Hi @AndressRod ,

Could you please describe the exact steps so we can reproduce the issue?

[AndressRod]

Hi @maximthomas.
Sure.

On OpenAM installation, I leaved OpenAM as Configuration Store and then configured OpenDJ as Users Store.
Then I did the following configurations on OpenAM (I'm describing this just in case they are important to reproduce the issue):

1. On Configure > Authentication > Core Attributes > Core, set ldapService for both options.
2. On Configure > Authentication >Persistent Cookie, set "5" for Idle Timeout and Max Life.
3. On Configure > Server Defaults > Sessions > Session Limits leave Maximum Session Cache Size on default to reproduce the first scenario in which a few minutes later the authentication issue appeared.
4. Realm > Authorization > Policy Sets, create a policy of Resource Type URL Allowing all except OPTIONS and PATCH
5. Create a Site, configure it as Principal Site on Server and change the BaseURL as the same as the site primary url.
6. Set -Xms and -Xmx to 3G on JVM envs. I'm running it as a pod in Kubernetes.

Then execute the following JMeter test:
openam-test.tar.gz

After a few minutes. Open a web browser and try to login with amadmin on /sso/XUI/.

Thank you!

[vharseko]

please upload /usr/openam/config folder

[maximthomas]

Hi @AndressRod,
Unfortunately, I could not reproduce the issue on the default environment with an embedded OpenDJ instance:
Started OpenAM in a Docker container:

docker run -h openam.example.org -p 8080:8080 --name openam openidentityplatform/openam

Ran the initial setup script

docker exec -w '/usr/openam/ssoconfiguratortools' openam bash -c \
'echo "ACCEPT_LICENSES=true
SERVER_URL=http://openam.example.org:8080
DEPLOYMENT_URI=/$OPENAM_PATH
BASE_DIR=$OPENAM_DATA_DIR
locale=en_US
PLATFORM_LOCALE=en_US
AM_ENC_KEY=
ADMIN_PWD=passw0rd
AMLDAPUSERPASSWD=p@passw0rd
COOKIE_DOMAIN=example.org
ACCEPT_LICENSES=true
DATA_STORE=embedded
DIRECTORY_SSL=SIMPLE
DIRECTORY_SERVER=openam.example.org
DIRECTORY_PORT=50389
DIRECTORY_ADMIN_PORT=4444
DIRECTORY_JMX_PORT=1689
ROOT_SUFFIX=dc=openam,dc=example,dc=org
DS_DIRMGRDN=cn=Directory Manager
DS_DIRMGRPASSWD=passw0rd" > conf.file && java -jar openam-configurator-tool*.jar --file conf.file'

Ran a benchmark test with Apache HTTP server benchmarking tool:

ulimit -n 100000 

ab -r -n 6000 -c 6000 \
 -H "X-OpenAM-Username: demo" \
 -H "X-OpenAM-Password: changeit" \
 -p /dev/null \                                         
 -T application/json \
 http://openam.example.org:8080/openam/json/authenticate

Then ran curl authentication script:

 curl --request POST \  
 -H "X-OpenAM-Username: demo" \
 -H "X-OpenAM-Password: changeit" \
 http://openam.example.org:8080/openam/json/authenticate
{"tokenId":"AQIC5wM2LY4Sfcx-U9RlclYM7GJZDiHNZnSqe5Fe_YfOVNA.*AAJTSQACMDEAAlNLABMxODQ0OTg0NTU2NTAyOTU4NTE4AAJTMQAA*","successUrl":"/openam/console","realm":"/"}

There were no errors

[AndressRod]

Hello @maximthomas and @vharseko.

I'm attaching my /usr/openam/config and also /opt/opendj/data and an updated Jmeter test for the single user I've created on OpenDJ.

https://drive.google.com/file/d/1l89ncomt1HORkEkRpyft4N2u_mOLsoWq/view?usp=sharing

Also, did something changed from version 15.1.3 to 15.1.6 related to ldap connections? I realized that I'm getting the following issue If I click in Load schema checkbox on Data Store (that didn't happened on version 15.1.3):

[maximthomas]

I realized that I'm getting the following issue If I click in Load schema checkbox on Data Store (that didn't happened on version 15.1.3):

Hello @AndressRod ,
Why are you trying to load schema to the Active Directory?

[vharseko]

Upper Emdedded (sso-data/OpenAM/opends) CTS Max Connections 10->128 Tuning LDAP CTS and Configuration Store Settings
or use external CTS install Configuring the Core Token Service

[AndressRod]

thank you both @maximthomas @vharseko for your kindly support!