Skip to content

Latest commit

 

History

History
222 lines (155 loc) · 6.23 KB

standalone_fate_source_code_deployment_guide.md

File metadata and controls

222 lines (155 loc) · 6.23 KB

Source Code Deployment FATE Standalone

[TOC]

1. Description

Server configuration.

number 1
Configuration 8 core / 16G memory / 500G hard disk
OS Version: CentOS Linux release 7
User User: app owner:apps

2. Pre-deployment environment check

Whether local ports 8080, 9360, 9380 are occupied

netstat -apln|grep 8080
netstat -apln|grep 9360
netstat -apln|grep 9380

3. Get the source code

Please git clone this repository and all submodules, then set the environment variables required for deployment.

Note that the environment variables set in the following way are only valid for the current terminal session, if you open a new terminal session, such as a new login or a new window, please set them again.

cd {The directory where the above code is stored}
export FATE_PROJECT_BASE=$PWD
export version=`grep "FATE=" ${FATE_PROJECT_BASE}/fate.env | awk -F "=" '{print $2}'`

4. Install and configure the Python environment

4.1 Installing the Python environment (optional)

Please install or use existing Python version 3.8

4.2 Configuring a virtual environment for FATE

cd(or create) {root directory for the virtual environment}
python3 -m venv {虚拟环境名称}
export FATE_VENV_BASE={root directory for the virtual environment}/{name of virtual environment}
source ${FATE_VENV_BASE}/bin/activate

4.3 Installing Python dependencies for FATE

cd ${FATE_PROJECT_BASE}
bash bin/install_os_dependencies.sh
source ${FATE_VENV_BASE}/bin/activate
pip install -r python/requirements.txt

In case of problems, you can first refer to Possible problems

5. Configuring FATE

Edit the bin/init_env.sh environment variable file

cd ${FATE_PROJECT_BASE}
sed -i.bak "s#PYTHONPATH=.*#PYTHONPATH=$PWD/python:$PWD/fateflow/python#g" bin/init_env.sh
sed -i.bak "s#venv=.*#venv=${FATE_VENV_BASE}#g" bin/init_env.sh

Check if the conf/service_conf.yaml global configuration file has the base engine configured as standalone, if default_engines shows the following, then it is standalone

default_engines:
  computing: standalone
  federation: standalone
  storage: standalone

6. start fate flow server

cd ${FATE_PROJECT_BASE}
source bin/init_env.sh

cd fateflow
bash bin/service.sh status
bash bin/service.sh start

If it shows something like this, it is started successfully, otherwise please check the logs according to the prompt

service start sucessfully. pid: 111907
status:app 111907 75.7 1.1 3740008 373448 pts/2 Sl+ 12:21 0:17 python /xx/FATE/fateflow/python/fate_flow/fate_flow_server.py
python 111907 app 14u IPv4 3570158828 0t0 TCP localhost:boxp (LISTEN)
python 111907 app 13u IPv4 3570158827 0t0 TCP localhost:9360 (LISTEN)

7. install fate client

cd ${FATE_PROJECT_BASE}
source bin/init_env.sh

cd python/fate_client/
python setup.py install

Initialize ``fate flow client`''

cd ../../
flow init -c conf/service_conf.yaml

If it looks like this, the initialization is successful, otherwise, please check the logs according to the prompt

{
    "retcode": 0,
    "retmsg": "Fate Flow CLI has been initialized successfully."
}

8. Test items

8.1 Toy test

flow test toy -gid 10000 -hid 10000

If successful, the screen displays a statement similar to the following:

success to calculate secure_sum, it is 2000.0

8.2 Unit tests

cd ${FATE_PROJECT_BASE}
bash python/federatedml/test/run_test.sh

If successful, the screen displays a statement similar to the following:

there are 0 failed test

Some use case algorithms are in examples folder, please try to use them. Please refer to here for a quick start tutorial.

You can also experience the algorithm process kanban through your browser, please refer here

9. Compile package to install fateboard (optional)

Visualizing FATE Jobs with fateboard

9.1 Configuring Java environment

mkdir ${FATE_PROJECT_BASE}/env
cd ${FATE_PROJECT_BASE}/env

wget https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/resources/jdk-8u192.tar.gz
tar xzf jdk-8u192.tar.gz

wget https://dlcdn.apache.org/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz
tar xzf apache-maven-3.8.6-bin.tar.gz

Configure environment variables

export JAVA_HOME=${FATE_PROJECT_BASE}/env/jdk-8u192

cd ${FATE_PROJECT_BASE}
sed -i.bak "s#JAVA_HOME=.*#JAVA_HOME=${JAVA_HOME}#g" bin/init_env.sh

9.2 Build fateboard

cd ${FATE_PROJECT_BASE}/fateboard
${FATE_PROJECT_BASE}/env/apache-maven-3.8.6/bin/mvn -DskipTests clean package

9.3 Starting fateboard

cd fateboard
bash service.sh status
bash service.sh start

If you see something like the following, you have started successfully, otherwise please check the logs as prompted

JAVA_HOME=/data/project/deploy/FATE/env/jdk/jdk-8u192/
service start sucessfully. pid: 116985
status:
        app 116985 333 1.7 5087004 581460 pts/2 Sl+ 14:11 0:06 /xx/FATE/env/jdk/jdk-8u192//bin/java -Dspring.config.location=/xx/FATE/fateboard/conf/ application.properties -Dssh_config_file=/xx/FATE/fateboard/ssh/ -Xmx2048m -Xms2048m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc: gc.log -XX:+HeapDumpOnOutOfMemoryError -jar /xx/FATE/fateboard

Open http://${ip}:8080, ip is 127.0.0.1 or real ip of this machine

10. Problems that may be encountered

  • If you get an error like "Too many open files", it may be because the number of OS handles is too low.

    • For MacOS, you can try here
    • For Linux, you can try here

  • If the installation of the python dependency package gmpy2 fails under MacOS, try installing the following base library before installing the dependency package

brew install gmp mpfr libmpc