mssassist

Validation, correction, and data registration scripts to assist MSS work

I (tkosuge) have taken Andrea-san mss fix tools, and will have migrated them here. You can see the overview of the tools from here.

Installation

1. Ssh login to a012 node as w3const user.

2. Pull the repository from ddbj/mssassist.git.

   git clone https://github.com/ddbj/mssassist.git ~/mssassist
   

   Note! Do NOT change the destination directory for installing the mssassist. When you would like to change the directory name from 'mssassist', you must change the value of $BASE variable in each shell script.

3. Prepare the symbolic link in the directory.

   cd ~/mssassist
   ln -s ddbj_autofix.sh ddbj_autofix
   ln -s ddbj_kaeru.sh ddbj_kaeru
   ln -s sing_ddbj_mss_validation.sh ddbj_mss_validation
   ln -s ddbj_sakura2DB.sh ddbj_sakura2DB
   

4. Create 'tables' directory.

   mkdir -m 775 tables
   

   Data files required to run MSS validation tools are created here.

5. Put the JSON type API keyfile to write MSSworking gsheet in .key directory.

   mkdir -m 750 .key
   And copy the ######.json file to .key/ directory, and then set the permission to 640.
   

6. Add the followings to cron@a012.

   # MSS ddbj_mss_validation
   0 8 * * *  bash /home/w3const/mssassist/update_taxdump.sh
   0,10,20,30,40,50 8-19 * * * bash /home/w3const/mssassist/update_tables.sh
   

7. Prepare the singularity sif container required for running R/Python scripts stored in step#/ directories.
   Chose method A or B to set the singularity container file to w3const/mssassist directory.

   1. Method A.
      The "mssassist" repository has auto-workflow for building sif file when Singularity denifion is updated.
      Please access github action and click the workflow name to find the "sif-container" from artifacts section. Click it to download and unzip it, then you will find "sing-mssassist.sif". Upload it to /home/w3const/mssassist directory.

   2. Method B.
      Login to appropriate local server (e.g. ddbjs1) where users can build singularity containers.

      # Execute the following command.
      cd <working directory>
      git clone https://github.com/ddbj/mssassist.git
      cd mssassist
      sudo singularity build sing-mssassist.sif ./Singularity
       
      # Then upload the sing-mssassist.sif to /home/w3const/mssassist directory 
      

8. Finally, you must prepare the private key file in your ~/.ssh/w3const/ to access the directories in 'w3const'.

   ssh login as your account
   cd ~
   ~/.ssh/w3const/id_ed25519  <--Do not change the name.
   

How to use

• jParser、transCheckerを使用可能な環境で実行する。
• w3constにsshでアクセスできるようにしておく。
• コマンド使用方法は、Confluenceの「MSS 査定; 新規登録の作業手順」ページを参照。

Memo

The Rscript requires the following packages that are included in sing-mssassist.sif.

data.table
doParallel
dplyr
fedmatch
foreach
gt
stringr
tibble
tidyr
tidyverse

Use the following definition to build image that contains the packages listed above.

For singularity

Bootstrap: docker
From: r-base:4.5.2
Stage: build

%post
    apt update
    Rscript -e 'install.packages(rownames(installed.packages(priority="recommended")), repos="https://cloud.r-project.org")'
    Rscript -e 'install.packages("fedmatch", repos="https://cloud.r-project.org")'
    Rscript -e 'install.packages("tidyr", repos="https://cloud.r-project.org")'
    Rscript -e 'install.packages("doParallel", repos="https://cloud.r-project.org")'
    Rscript -e 'install.packages(c("colorspace", "fansi", "munsell"), repos="https://cloud.r-project.org")'
    # gt
    apt -y install libxml2-dev libcurl4-openssl-dev libv8-dev
    Rscript -e 'install.packages("gt", repos="https://cloud.r-project.org")'
    # tidyverse
    apt -y install libfontconfig1-dev libharfbuzz-dev libfribidi-dev libfreetype6-dev libpng-dev libtiff5-dev libjpeg-dev libwebp-dev
    Rscript -e 'install.packages("tidyverse", repos="https://cloud.r-project.org")'

For podman

• Start container ... podman-compose -f compose.yaml up -d
• Stop container ... podman-compose -f compose.yaml down

Dockerfile

FROM docker.io/r-base:4.5.2

# required lib
RUN apt-get update && apt-get install -y libxml2-dev \
    libcurl4-openssl-dev \
    libv8-dev \
    libfontconfig1-dev \
    libharfbuzz-dev \
    libfribidi-dev \
    libfreetype6-dev \
    libpng-dev \
    libtiff5-dev \
    libjpeg-dev \
    libwebp-dev

# R packages
RUN Rscript -e 'install.packages(rownames(installed.packages(priority="recommended")), repos="https://cloud.r-project.
org")' \
    && Rscript -e 'install.packages("fedmatch", repos="https://cloud.r-project.org")' \
    && Rscript -e 'install.packages("tidyr", repos="https://cloud.r-project.org")' \
    && Rscript -e 'install.packages("doParallel", repos="https://cloud.r-project.org")' \
    && Rscript -e 'install.packages(c("colorspace", "fansi", "munsell"), repos="https://cloud.r-project.org")' \
    && Rscript -e 'install.packages("gt", repos="https://cloud.r-project.org")' \
    && Rscript -e 'install.packages("tidyverse", repos="https://cloud.r-project.org")'

compose.yaml

services:
  r-mss-env:
    build:
      context: .
      dockerfile: Dockerfile
    image: r-base:4.5.2-msstools
    container_name: r-mss-env
    stdin_open: true
    tty: true
    volumes:
      - .:/mnt

Links

• 移行のために行った解析結果(Ctrl+Click to open in new tab)