database is not create for instance when using centralized infra

[zelima]

As a cco user, I've tried to deploy a CKAN instance test with centralized infra but I don't see the databse test created there. This leads to the authentication Error when CKAN is trying to start and connect to DB. (See analysis for more info)

Acceptance Criteria

• CKAN Instance has it's dedicated database in centralized DB
• CKAN can connect to DB

Tasks

• Find where the passwords are created for CKAN (as I see DB URLs)
• Find the part responsible for creating databases
  • Why that part is not executed?
• Refactor if reasonable and make it happen

Analysis

This is all I see in database instance

psql postgresql://postgres:123456@db.ckan-cloud
\l
                                    List of databases
       Name       |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
------------------+----------+----------+------------+------------+-----------------------
 ckan             | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =Tc/postgres         +
                  |          |          |            |            | postgres=CTc/postgres+
                  |          |          |            |            | ckan=CTc/postgres
 postgres         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0        | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                  |          |          |            |            | postgres=CTc/postgres
 template1        | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                  |          |          |            |            | postgres=CTc/postgres
 template_postgis | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 

Funny thing is that database credentials (User, passwords) are set but the are not actually created

cat /etc/ckan-conf/secrets/secrets.sh
export BEAKER_SESSION_SECRET=bb6624cd581d6d5192939cc1c69a020aea4168f133fff34c1b
export APP_INSTANCE_UUID=027f9157-05e4-11ea-892d-0242ac11000b
export SQLALCHEMY_URL=postgresql://first:73021fa192850023c05cbc86@db.ckan-cloud/first
export CKAN_DATASTORE_WRITE_URL=postgresql://first-datastore:c8993c5fb732c61dc82ea8ab@db.ckan-cloud/first-datastore
export CKAN_DATASTORE_READ_URL=postgresql://first-datastore-readonly:40a17bd285127229c65b2e56@db.ckan-cloud/first-datastore
export SOLR_URL=http://solr.ckan-cloud:8983/solr/first
export CKAN_REDIS_URL=redis://redis:6379/1

So as it seems everything is done in this script https://github.com/ViderumGlobal/ckan-cloud-docker/blob/master/cca-operator/cca-operator.sh. That script is executed when deployment happens https://github.com/ViderumGlobal/ckan-cloud-helm/blob/master/ckan/templates/ckan-deployment.yaml#L25-L30

Eg passwords set on line 9, 10, 11

POSTGRES_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"`
DATASTORE_POSTGRES_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"`
DATASTORE_RO_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"`

For some reasons we are not getting in this if statment https://github.com/ViderumGlobal/ckan-cloud-docker/blob/master/cca-operator/cca-operator.sh#L20-L38 that is responsible for creating database. Need to debug there...

[akariv]

Which ckan image are you using? This is where DB is initialized: https://github.com/ViderumGlobal/ckan-cloud-docker/blob/master/ckan/entrypoint.sh#L15 Who is running cca_operator? ckan-cloud-operator?

…

On Thu, Nov 21, 2019 at 2:50 PM Irakli Mchedlishvili < ***@***.***> wrote: As a cco user, I've tried to deploy a CKAN instance test with centralized infra but I don't see the databse test created there. This leads to the authentication Error when CKAN is trying to start and connect to DB. (See analysis for more info) Acceptance Criteria - CKAN Instance has it's dedicated database in centralized DB - CKAN can connect to DB Tasks - Find where the Analysis This is all I see in database instance psql ***@***.*** \l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ------------------+----------+----------+------------+------------+----------------------- ckan | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =Tc/postgres + | | | | | postgres=CTc/postgres+ | | | | | ckan=CTc/postgres postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres + | | | | | postgres=CTc/postgres template_postgis | postgres | UTF8 | en_US.utf8 | en_US.utf8 | Funny thing is that database credentials (User, passwords) are set but the are not actually created cat /etc/ckan-conf/secrets/secrets.sh export BEAKER_SESSION_SECRET=bb6624cd581d6d5192939cc1c69a020aea4168f133fff34c1b export APP_INSTANCE_UUID=027f9157-05e4-11ea-892d-0242ac11000b export ***@***.***/first export ***@***.***/first-datastore export ***@***.***/first-datastore export SOLR_URL=http://solr.ckan-cloud:8983/solr/first export CKAN_REDIS_URL=redis://redis:6379/1 So as it seems everything is done in this script https://github.com/ViderumGlobal/ckan-cloud-docker/blob/master/cca-operator/cca-operator.sh. That script is executed when deployment happens https://github.com/ViderumGlobal/ckan-cloud-helm/blob/master/ckan/templates/ckan-deployment.yaml#L25-L30 Eg passwords set on line 9, 10, 11 POSTGRES_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"` DATASTORE_POSTGRES_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"` DATASTORE_RO_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"` For some reasons we are not getting in this if statment https://github.com/ViderumGlobal/ckan-cloud-docker/blob/master/cca-operator/cca-operator.sh#L20-L38 that is responsible for creating database. Need to debug there... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#12>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACAY5JTOONNXPJSGS2QPTLQUZ7ZLANCNFSM4JQB23BQ> .

[zelima]

Which ckan image are you using?

I'm not using any particular image right now, assuming that will build vanilla CKAN in that case https://github.com/ViderumGlobal/ckan-cloud-helm/blob/master/ckan/templates/ckan-deployment.yaml#L118. But maybe I should?

command I've deployed the instance is folowing

cca_helm_upgrade --install --set replicas=1 --set nginxReplicas=1 --set disableJobs=true --set useCentralizedInfra=true --set noProbes=true --set ckanOperatorImage=viderum/ckan-cloud-docker:cca-operator-latest

Who is running cca_operator? ckan-cloud-operator?

Not sure about that one, but my guess is yes, I see the commands are executed from this part https://github.com/ViderumGlobal/ckan-cloud-helm/blob/master/ckan/templates/ckan-deployment.yaml#L25-L30

[akariv]

so - Values.ckanImage is defined to what?
and you should be using cco to install stuff in the cluster - cca_operator is obsolete and should be deprecated asap.

[zelima]

so - Values.ckanImage is defined to what?

To None I guess.

and you should be using cco to install stuff in the cluster - cca_operator is obsolete and should be deprecated asap.

Can I read how to do helm based deployments via CCO anywhere? Or should I just go through the code?

[akariv]

I'm not sure you can in the docs - but see this script for an example:

https://github.com/datopian/ckan-cloud-operator/blob/master/scripts/create%20helm%20instance.py

The same functionality is also available via the cli
(ckan-cloud-operator instance create / ckan-dloud-operator instance update)