Skip to content

Commit efe9690

Browse files
author
caaespin
committed
Merge branch 'release/1.0.0'
2 parents f9a82db + d0d3656 commit efe9690

File tree

91 files changed

+4282
-228
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+4282
-228
lines changed

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,7 @@ The `install_bootstrap` script will ask you to configure each service interactiv
172172
* On question 'What is your AWS S3 bucket?', put the name of the s3 bucket you created for Redwood.
173173
* On question 'What is your AWS S3 endpoint?', put the S3 endpoint pertaining to your region. See [here](http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region).
174174
* On question 'What is your AWS IAM KMS key ID?', put your encryption key ID (See 'Create an AWS IAM Encryption Key" above). If you don't want server-side encryption, you can leave this blank.
175+
* On question 'Would you like to use external redwood databases?', enter 'N'.
175176
* Boardwalk
176177
* Install in prod mode
177178
* On question `What is your Google Client ID?`, put your Google Client ID. See [here](http://bitwiser.in/2015/09/09/add-google-login-in-flask.html#creating-a-google-project)
@@ -202,10 +203,10 @@ Here are things we need to explain how to do post install:
202203

203204
* first of all, how to go to the website and confirm things are working e.g. https://ops-dev.ucsc-cgl.org or whatever the domain name is
204205
* how to associate a token with a user email so token download works
205-
* `sudo redwood/admin/bin/redwood token create -u [email protected] -s 'aws.upload aws.download'`, this give access to all programs.
206+
* `sudo redwood/cli/bin/redwood token create -u [email protected] -s 'aws.upload aws.download'`, this give access to all programs.
206207
* you can also assign program scopes as well, for example
207-
* `sudo redwood/admin/bin/redwood project create PROJECT`
208-
* `sudo redwood/admin/bin/redwood token create -u [email protected] -s 'aws.PROJECT.upload aws.PROJECT.download'`
208+
* `sudo redwood/cli/bin/redwood project create PROJECT`
209+
* `sudo redwood/cli/bin/redwood token create -u [email protected] -s 'aws.PROJECT.upload aws.PROJECT.download'`
209210
* user log in via google, retrieve token
210211
* Get the reference data used by the RNASeq-CGL pipeline:
211212
* Instructions for downloading reference data for RNASeq-CGL are located here: https://github.com/BD2KGenomics/toil-rnaseq/wiki/Pipeline-Inputs
@@ -256,7 +257,7 @@ The `test/integration.sh` file also demonstrates normal core-client usage.
256257

257258
Here is a sample command you can run from the `test` folder to do an upload:
258259

259-
**NOTE:** Make sure you create an access token for yourself first. You can do so by running within `dcc-ops` the command `redwood/admin/bin/redwood token create -u [email protected] -s 'aws.upload aws.download'`. This will create a global token that you can use for testing for upload and download on any project. End users should only be given project-specific scopes like _aws.PROJECT.upload_.
260+
**NOTE:** Make sure you create an access token for yourself first. You can do so by running within `dcc-ops` the command `redwood/cli/bin/redwood token create -u [email protected] -s 'aws.upload aws.download'`. This will create a global token that you can use for testing for upload and download on any project. End users should only be given project-specific scopes like _aws.PROJECT.upload_.
260261

261262
```
262263
sudo docker run --rm -it -e ACCESS_TOKEN=<your_token> -e REDWOOD_ENDPOINT=<your_url.com> \

‎action/action_compose.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ services:
1919
DOCKSTORE_TOKEN: "${dockstore_token}"
2020
volumes:
2121
- ~/dcc-action-service/logs:/home/ubuntu/logs
22+
- ~/dcc-action-service/extra:/home/ubuntu/luigi_decider_runs/extra
2223
restart: always
2324
ports:
2425
- "8082"

‎boardwalk/prod.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ version: '2'
22

33
services:
44
dcc-metadata-indexer:
5-
image: quay.io/ucsc_cgl/metadata-indexer:2.0.2
5+
image: quay.io/ucsc_cgl/metadata-indexer:2.0.4
66
#build: dcc-metadata-indexer/v2
77
environment:
88
USER_GROUP: "${user_group}"
@@ -54,7 +54,7 @@ services:
5454
FLASK_APP: "/app/mapi.py"
5555
DCC_DASHBOARD_HOST: "${dcc_dashboard_host}"
5656
SECRET_KEY: "${secret_key}"
57-
image: quay.io/ucsc_cgl/dashboard-service:1.0.2
57+
image: quay.io/ucsc_cgl/dashboard-service:1.0.3
5858
volumes:
5959
- ~/dcc-dashboard-service/logs:/app/log
6060
#build: dcc-dashboard-service
@@ -75,7 +75,7 @@ services:
7575
- login-db
7676
restart: always
7777
dcc-dashboard:
78-
image: quay.io/ucsc_cgl/dashboard:1.0.4
78+
image: quay.io/ucsc_cgl/dashboard:1.0.5
7979
#build: dcc-dashboard
8080
ports:
8181
- "80"

‎install_bootstrap‎

Lines changed: 117 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -452,7 +452,7 @@ while [[ "${run_redwood^^}" != 'Y' && "${run_redwood^^}" != 'N' ]] ; do
452452

453453
if [ "${run_redwood^^}" = 'Y' ] ; then
454454

455-
#Read the config file if it exists and then show previous values...
455+
# Read the config file if it exists and then show previous values...
456456
if [ -f redwood_launcher_config/redwood.config ] ; then
457457
source <(jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" redwood_launcher_config/redwood.config)
458458
fi
@@ -511,20 +511,66 @@ while [[ "${run_redwood^^}" != 'Y' && "${run_redwood^^}" != 'N' ]] ; do
511511

512512
sed -ie '/\[redwood_install\]/,+3d' ~/.aws/credentials
513513

514-
echo "generating redwood-auth-db postgresql superuser (postgres) password"
515-
auth_db_postgres_password="$(generate_password)"
516-
#ask_question "What is your auth DB password?" "$AUTH_DB_PASSWORD" "auth DB password" $auth_db_password
517-
518-
echo "generating redwood-auth-db auth-server postgresql user (dcc_auth) password"
519-
auth_db_dcc_password="$(generate_password)"
520-
#ask_question "What is your auth DB password?" "$AUTH_DB_PASSWORD" "auth DB password" $auth_db_dcc_password
521-
522-
echo "generating redwood-metadata-db metadata db password"
523-
metadata_db_password="$(generate_password)"
524-
#ask_question "What is your metadata db password?" "$METADATA_DB_PASSWORD" "metadata db password" $metadata_db_password
525-
526-
echo "generating random auth-server admin password"
527-
auth_admin_pass="$(generate_password)"
514+
# use external redwood dbs or compose db containers here
515+
external_redwood_dbs=''
516+
while [[ "${external_redwood_dbs^^}" != 'Y' && "${external_redwood_dbs^^}" != 'N' ]] ; do
517+
echo "Would you like to use external redwood databases? [Y/N]"
518+
read external_redwood_dbs
519+
520+
if [ "${external_redwood_dbs^^}" = 'Y' ] ; then
521+
# metadata db
522+
redwood_metadata_db_host='redwood_metadata_db_host'
523+
ask_question "Enter your (mongodb) redwood metadata db host. You can include port number." "$REDWOOD_METADATA_DB_HOST" "redwood metadata db host" $redwood_metadata_db_host
524+
redwood_metadata_db='redwood_metadata_db'
525+
ask_question "Enter your (mongodb) redwood metadata db database name" "$REDWOOD_METADATA_DB" "redwood metadata db database name" $redwood_metadata_db
526+
redwood_metadata_db_username='redwood_metadata_db_username'
527+
ask_question "Enter your (mongodb) redwood metadata db username" "$REDWOOD_METADATA_DB_USERNAME" "redwood metadata db username" $redwood_metadata_db_username
528+
redwood_metadata_db_password='redwood_metadata_db_password'
529+
ask_question "Enter your (mongodb) redwood metadata db password" "$REDWOOD_METADATA_DB_PASSWORD" "redwood metadata db password" $redwood_metadata_db_password
530+
531+
# auth db
532+
redwood_auth_db_host='redwood_auth_db_host'
533+
ask_question "Enter your (mongodb) redwood auth db host. You can include port number." "$REDWOOD_AUTH_DB_HOST" "redwood auth db host" $redwood_auth_db_host
534+
redwood_auth_db='redwood_auth_db'
535+
ask_question "Enter your (mongodb) redwood auth db database name" "$REDWOOD_AUTH_DB" "redwood auth db database name" $redwood_auth_db
536+
redwood_auth_db_username='redwood_auth_db_username'
537+
ask_question "Enter your (mongodb) redwood auth db username" "$REDWOOD_AUTH_DB_USERNAME" "redwood auth db username" $redwood_auth_db_username
538+
redwood_auth_db_password='redwood_auth_db_password'
539+
ask_question "Enter your (mongodb) redwood auth db password" "$REDWOOD_AUTH_DB_PASSWORD" "redwood auth db password" $redwood_auth_db_password
540+
541+
elif [ "${external_redwood_dbs^^}" = 'N' ]; then
542+
# metadata db
543+
echo "using local mongodb redwood-metadata-db container"
544+
redwood_metadata_db_host="redwood-metadata-db"
545+
redwood_metadata_db="dcc-metadata"
546+
redwood_metadata_db_username="metadata"
547+
redwood_metadata_db_password="password"
548+
549+
echo "generating redwood-metadata-db metadata db password"
550+
redwood_metadata_db_password="$(generate_password)"
551+
552+
echo "generating redwood-metadata-db admin password"
553+
redwood_metadata_db_admin_password="$(generate_password)"
554+
555+
# auth db
556+
echo "using local mongodb redwood-auth-db container"
557+
redwood_auth_db_host="redwood-auth-db"
558+
redwood_auth_db="dcc"
559+
redwood_auth_db_username="dcc_auth"
560+
561+
echo "generating redwood-auth-db postgresql superuser (postgres) password"
562+
redwood_auth_db_admin_password="$(generate_password)"
563+
564+
echo "generating redwood-auth-db user (dcc_auth) password"
565+
redwood_auth_db_password="$(generate_password)"
566+
567+
echo "generating random auth-server admin password"
568+
redwood_auth_server_admin_pass="$(generate_password)"
569+
else
570+
echo "ERROR: bad input: ${external_redwood_dbs}"
571+
external_redwood_dbs=''
572+
fi
573+
done
528574

529575
echo "generating random metadata-server oauth client secret"
530576
metadata_client_secret="$(generate_password)"
@@ -549,10 +595,17 @@ while [[ "${run_redwood^^}" != 'Y' && "${run_redwood^^}" != 'N' ]] ; do
549595
"S3_BUCKET_BACKUP":"${s3_bucket_backup}",
550596
"S3_ENDPOINT":"${s3_endpoint}",
551597
"KMS_KEY":"${kms_key}",
552-
"AUTH_DB_POSTGRES_PASSWORD":"${auth_db_postgres_password}",
553-
"AUTH_DB_DCC_PASSWORD":"${auth_db_dcc_password}",
554-
"METADATA_DB_PASSWORD":"${metadata_db_password}",
555-
"AUTH_ADMIN_PASSWORD":"${auth_admin_pass}",
598+
"METADATA_DB_HOST":"${redwood_metadata_db_host}",
599+
"METADATA_DB":"${redwood_metadata_db}",
600+
"METADATA_DB_USERNAME":"${redwood_metadata_db_username}",
601+
"METADATA_DB_PASSWORD":"${redwood_metadata_db_password}",
602+
"METADATA_DB_ADMIN_PASSWORD":"${redwood_metadata_db_admin_password}",
603+
"AUTH_DB_HOST":"${redwood_auth_db_host}",
604+
"AUTH_DB":"${redwood_auth_db}",
605+
"AUTH_DB_USERNAME":"${redwood_auth_db_username}",
606+
"AUTH_DB_PASSWORD":"${redwood_auth_db_password}",
607+
"AUTH_DB_ADMIN_PASSWORD":"${redwood_auth_db_admin_password}",
608+
"AUTH_SERVER_ADMIN_PASSWORD":"${redwood_auth_server_admin_pass}",
556609
"METADATA_CLIENT_SECRET":"${metadata_client_secret}",
557610
"STORAGE_CLIENT_SECRET":"${storage_client_secret}",
558611
"MGMT_CLIENT_SECRET":"${mgmt_client_secret}"
@@ -570,54 +623,55 @@ CONFIG
570623

571624
# remove old redwood instance and data volumes if they exist
572625
echo "Removing old redwood containers, volumes, etc. if they exist"
573-
sudo admin/bin/redwood down
626+
sudo cli/bin/redwood down
574627
redwood_data_volumes=$(sudo docker volume ls | grep 'redwood_redwood-.*-data' | awk '{print $2}')
575628
if [[ ! -z ${redwood_data_volumes} ]]; then
576629
sudo docker volume rm ${redwood_data_volumes}
577630
fi
578631

579632
# start redwood
580633
echo "Starting redwood"
581-
sudo admin/bin/redwood up
582-
634+
sudo cli/bin/redwood up
583635
cd ..
584-
elif [[ "${launch_redwood^^}" = 'N' ]]; then
585-
echo "You can start redwood later with: $ $(pwd)/redwood/admin/bin/redwood up"
586-
fi
587-
done
588636

589-
recover_metadata=-1
590-
read -ep $'Do you want to recover metadata from a backup? [Y/N]\n' recover_metadata_response
591-
while [[ "${recover_metadata}" = "-1" ]]; do
592-
if [[ "${recover_metadata_response^^}" =~ ^(Y|YES)$ ]]; then
593-
recover_metadata=1
594-
elif [[ "${recover_metadata_response^^}" =~ ^(N|NO)$ ]]; then
595-
recover_metadata=0
596-
else
597-
echo "bad input: ${recover_metadata_response}"
637+
# optionally recover backup and run migrations
598638
recover_metadata=-1
599-
fi
600-
done
601-
if [[ ${recover_metadata} -eq 1 ]]; then # run recovery and (optionally) migrations
602-
read -ep $'Enter the path to your metdata-backup (.tar.gz) file.\n' metadata_backup_file
603-
sudo redwood/admin/bin/redwood recover metadata "${metadata_backup_file}"
604-
605-
migrate=-1
606-
while [[ "${migrate}" = "-1" ]]; do # suggest migrations
607-
case ${yes} in
608-
0) migrate_response=no ;;
609-
1) migrate_response=yes ;;
610-
*) read -rep $'Do you want to perform any migrations? [Y\N]\n' migrate_response ;;
611-
esac
612-
if [[ "${migrate_response^^}" =~ ^(Y|YES)$ ]]; then
613-
migrate=1
614-
sudo redwood/admin/bin/redwood migrate metadata "${metadata_backup_file}"
615-
elif [[ "${migrate_response^^}" =~ ^(N|NO)$ ]]; then migrate=0
616-
else echo "bad input: ${migrate_response}"
639+
read -ep $'Do you want to recover metadata from a backup? [Y/N]\n' recover_metadata_response
640+
while [[ "${recover_metadata}" = "-1" ]]; do
641+
if [[ "${recover_metadata_response^^}" =~ ^(Y|YES)$ ]]; then
642+
recover_metadata=1
643+
elif [[ "${recover_metadata_response^^}" =~ ^(N|NO)$ ]]; then
644+
recover_metadata=0
645+
else
646+
echo "bad input: ${recover_metadata_response}"
647+
recover_metadata=-1
648+
fi
649+
done
650+
if [[ ${recover_metadata} -eq 1 ]]; then # run recovery and (optionally) migrations
651+
read -ep $'Enter the path to your metdata-backup (.tar.gz) file.\n' metadata_backup_file
652+
sudo redwood/cli/bin/redwood recover metadata "${metadata_backup_file}"
653+
654+
migrate=-1
655+
while [[ "${migrate}" = "-1" ]]; do # suggest migrations
656+
case ${yes} in
657+
0) migrate_response=no ;;
658+
1) migrate_response=yes ;;
659+
*) read -rep $'Do you want to perform any migrations? [Y\N]\n' migrate_response ;;
660+
esac
661+
if [[ "${migrate_response^^}" =~ ^(Y|YES)$ ]]; then
662+
migrate=1
663+
sudo redwood/cli/bin/redwood migrate metadata "${metadata_backup_file}"
664+
elif [[ "${migrate_response^^}" =~ ^(N|NO)$ ]]; then migrate=0
665+
else echo "bad input: ${migrate_response}"
666+
fi
667+
done
668+
617669
fi
618-
done
619670

620-
fi
671+
elif [[ "${launch_redwood^^}" = 'N' ]]; then
672+
echo "You can start redwood later with: $ $(pwd)/redwood/cli/bin/redwood up"
673+
fi
674+
done
621675

622676
elif [ "${run_redwood^^}" = 'N' ] ; then
623677
echo "You can run this script at another time to run the redwood installer"
@@ -667,15 +721,15 @@ while [[ "${run_boardwalk^^}" != 'Y' && "${run_boardwalk^^}" != 'N' ]] ; do
667721
read run_boardwalk
668722

669723
if [ "${run_boardwalk^^}" = 'Y' ] ; then
670-
#echo "Carlos needs to dockerize"
671724
#Per ES specifications to run their docker image on production:
672725
#Set vm.max_map_count to 262144
673726
#Set the default elasticsearch1 service name
674727
es_service=elasticsearch1
675728
dcc_dashboard_protocol=https
676729
dcc_dashboard_port=8080
730+
echo "Setting max VM for ElasticSearch"
677731
sudo sysctl -w vm.max_map_count=262144
678-
#USER_GROUP=$(stat -c '%u:%g' $HOME)
732+
sudo echo "vm.max_map_count=262144" >> /etc/sysctl.conf
679733

680734
if [ -f boardwalk_launcher_config/boardwalk.config ] ; then
681735
source <(jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" boardwalk_launcher_config/boardwalk.config)
@@ -732,7 +786,7 @@ while [[ "${run_boardwalk^^}" != 'Y' && "${run_boardwalk^^}" != 'N' ]] ; do
732786
#ask_question "What is your Redwood auth-server Admin Port?" "$REDWOOD_ADMIN_PORT" "Redwood Admin Port" $redwood_admin_port
733787

734788
# generate redwood access token
735-
redwood_access_token=$(sudo redwood/admin/bin/redwood token create -u indexer -s 'aws.upload aws.download')
789+
redwood_access_token=$(sudo redwood/cli/bin/redwood token create -u indexer -s 'aws.upload aws.download')
736790
#ask_question "What is your Redwood Access Token?" "$REDWOOD_ACCESS_TOKEN" "Redwood Access Token" $redwood_access_token
737791

738792
#dcc_dashboard_host='dcc_dashboard_host'
@@ -840,7 +894,8 @@ while [[ "${run_boardwalk^^}" != 'Y' && "${run_boardwalk^^}" != 'N' ]] ; do
840894
echo "generating SECRET_KEY for login"
841895
secret_key="$(generate_password)"
842896

843-
core_client_version='1.1.0-alpha'
897+
#TODO: The script should snoop the .env file and check if it has a password assigned already and use that. Otherwise, generate a random password.
898+
core_client_version='1.1.0'
844899
# Now write a config for this file.
845900
[[ -f boardwalk_launcher_config/boardwalk.config ]] || mkdir -p boardwalk_launcher_config
846901

@@ -888,7 +943,7 @@ CONFIG
888943

889944
cd boardwalk
890945
#Bringing stuff down in case there are some cached containers
891-
echo "Bringing down any Boardwalk existing container"
946+
echo "Bringing down any existing Boardwalk container and volumes"
892947
sudo docker-compose -f ${boardwalk_mode}.yml down -v
893948
echo "Creating Boardwalk associated containers"
894949
sudo docker-compose -f ${boardwalk_mode}.yml up -d
@@ -951,8 +1006,8 @@ while [[ "${run_action^^}" != 'Y' && "${run_action^^}" != 'N' ]] ; do
9511006

9521007
# generate redwood access token
9531008
redwood_access_token='redwood_access_token'
954-
if [ -f redwood/admin/bin/redwood ] ; then
955-
redwood_access_token=$(sudo redwood/admin/bin/redwood token create -u action_service -s 'aws.upload aws.download')
1009+
if [ -f redwood/cli/bin/redwood ] ; then
1010+
redwood_access_token=$(sudo redwood/cli/bin/redwood token create -u action_service -s 'aws.upload aws.download')
9561011
else
9571012
ask_question "What is your Redwood Access Token?" "$STORAGE_ACCESS_TOKEN" "Redwood Access Token" $redwood_access_token
9581013
fi

0 commit comments

Comments
 (0)