Install

源码编译

项目统一采用maven编译，编译命令：mvn clean && mvn package -DskipTests assembly:assembly

安装higo依赖

* zookeeper:http://zookeeper.apache.org/ * hadoop:http://hadoop.apache.org/

安装蓝鲸依赖

* ZeroMQ 2.1.7 -

       tar -xzf zeromq-2.1.7.tar.gz
       cd zeromq-2.1.7
        ./configure
        make
        sudo make install

* JZMQ

       ./autogen.sh
       ./configure
       make
       sudo make install

* Java 6 * Python 2.6.6 * unzip

配置蓝鲸storm.yaml

配置示例

  storm.zookeeper.servers:
     - "adhoc2.kgb.cm6"
     - "adhoc3.kgb.cm6"
     - "adhoc4.kgb.cm6"
 storm.local.dir: "/home/taobao/bluewhale/stormwork"
 hadoop.conf.dir: "/home/taobao/config"
 hadoop.merger.conf.dir: "/home/taobao/hadoop/conf"
 hadoop.java.opts: "-Xmx1024m"
 higo.workdir.list: "/disk1/taobao/bluewhile,/disk2/taobao/bluewhile,/disk3/taobao/bluewhile,/disk4/taobao/bluewhile,/disk5/taobao/bluewhile,/disk6/taobao/bluewhile,/disk7/taobao/bluewhile,/disk8/taobao/bluewhile,/disk9/taobao/bluewhile,/disk10/taobao/bluewhile"
 nimbus.host: "tiansuan1.kgb.cm4"
 storm.zookeeper.root: "/bluewhale_higo"
 higo.jdbc.conn: "jdbc:higo://localhost:1107"
 higo.download.offline.conn: "jdbc:mysql://tiansuan1.kgb.cm4:3306/adhoc_download?useUnicode=true&characterEncoding=utf8"
 higo.download.offline.username: "adhoc"
 higo.download.offline.passwd: "adhoc"
 higo.download.offline.store: "/group/taobao/external/p4p/p4padhoc/download/offline"
 higo.merge.ports: "6601,6602,6603,6604,6605,6606,6607,6608,6609,6610,6611,6612,6613,6614,6615,6616,6617,6618,6619,6610"
 higo.mergeServer.count: 8
 higo.cache.partions: 10
 worker.childopts: "-Xms6g -Xmx6g -Xmn3g -XX:SurvivorRatio=7 -XX:PermSize=96m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=5 -XX:+UseAdaptiveSizePolicy "
 worker.childopts.6601: "-Xms6g -Xmx6g -Xmn3g -XX:SurvivorRatio=7 -XX:PermSize=96m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=5 -XX:+UseAdaptiveSizePolicy "
 supervisor.slots.ports:
    - 6701
    - 6702
    - 6703
    - 6704
    - 6705
    - 6706
    - 6707
    - 6708
    - 6709
    - 6710
    - 6711
    - 6712
    - 6713
    - 6714
    - 6601

集群启动

1. 启动nimbus
nohup ./bluewhale nimbus >nimbus.log &
2. 启动supervisor
nohup ./bluewhale supervisor >supervisor.log &
2. 启动查询接口与监控UI
nohup ./bluewhale higoui 9999 ../lib/bluewhale-web-1.0.0-SNAPSHOT.war ./ui >ui.log &
机器列表
http://xxxx:9999/higo/supervisor.jsp
蓝鲸任务列表（看报错信息）
http://xxxx:9999/higo/topology.jsp
higo的table列表
http://xxxx:9999/higo/tablelist.jsp