1 环境准备
Scheduilis依赖于Linkis、DSS和基础软件、Nginx等,请参照文章Linkis 1.5.0安装部署、DSS 1.1.2安装部署完成安装和验证通过以后,再参照本文进行Scheduilis 0.8.0安装部署。
1.1 环境变量设置
新增环境变量SCHEDUILIS_HOME:
vim ~/.bash_profile
export SCHEDUILIS_HOME=/data/appcom/install/scheduilis
1.2 数据库准备
create user 'schedulis'@'%' identified by 'schedulis';
DROP DATABASE schedulis;
CREATE DATABASE schedulis DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
use schedulis
grant all privileges on schedulis to 'schedulis'@'%';
grant all on schedulis.* to schedulis@'%';
show grants for schedulis@'%';
flush privileges;
2 安装介质准备
可以从官网下载或者参照文章Scheduilis 0.8.0编译进行自行编译。
后端安装包:schedulis_0.8.0_exec.zip
前端安装包:schedulis_0.8.0_web.zip
mkdir -p /data/appcom/install/scheduilis
unzip -d /data/appcom/install/scheduilis -o azkaban-exec-server/target/schedulis_0.8.0_exec.zip
unzip -d /data/appcom/install/scheduilis -o azkaban-web-server/target/schedulis_0.8.0_web.zip
mkdir -p /data/appcom/install/schedulis/schedulis-config
3 安装部署
配置执行节点host.properties:
vim /data/appcom/install/schedulis/schedulis-config/host.properties
iZ2ze06twbnm2tgi5kjw3hZ=1
其中iZ2ze06twbnm2tgi5kjw3hZ为hostname。
配置执行节点azkaban.properties:
vim /data/appcom/install/schedulis/schedulis-config/schedulis-exec/azkaban.properties
default.timezone.id=Asia/Shanghai
azkaban.jobtype.plugin.dir=plugins/jobtypes
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=iZ2ze06twbnm2tgi5kjw3hZ
mysql.database=schedulis
mysql.user=schedulis
mysql.password=c2NoZWR1bGlz
mysql.numconnections=100
executor.maxThreads=60
executor.port=12321
executor.flow.threads=30
jetty.headerBufferSize=65536
jetty.send.server.version=false
flow.num.job.threads=50
checkers.num.threads=10
executor.server.id=1
jetty.connector.stats=true
executor.connector.stats=true
azkaban.webserver.url=http://192.168.1.111:8081
proxy.user.lock.down=true
flow.paused.max.time.ms=3600000
azkaban.ip.whiteList.enabled=false
azkaban.ip.whiteList=
dss.secret=dws-wtss|WeBankBDPWTSS&DWS@2019
其中mysql地址和密码需要根据实际情况填写,密码使用Java Base64加密。
azkaban.webserver.url根据实际情况填写,端口是前端访问端口。
配置执行节点common.properties:
vim /data/appcom/install/schedulis/schedulis-config/schedulis-exec/common.properties
hadoop.home=/usr/local/hadoop
hive.home=/usr/local/hive
spark.home=/usr/local/spark
bdp.datachecker.jdo.option.name="bdp"
bdp.datachecker.jdo.option.url=jdbc:mysql://iZ2ze06twbnm2tgi5kjw3hZ:3306/metastore?useUnicode=true&characterEncoding=UTF-8&useSSL=false
bdp.datachecker.jdo.option.username=root
bdp.datachecker.jdo.option.password=MTIzNDU2
job.datachecker.jdo.option.name="job"
job.datachecker.jdo.option.url=jdbc:mysql://iZ2ze06twbnm2tgi5kjw3hZ:3306/metastore?useUnicode=true&characterEncoding=UTF-8&useSSL=false
job.datachecker.jdo.option.username=root
job.datachecker.jdo.option.password=MTIzNDU2
msg.eventchecker.jdo.option.name="msg"
msg.eventchecker.jdo.option.url=jdbc:mysql://iZ2ze06twbnm2tgi5kjw3hZ:3306/schedulis?useUnicode=true&characterEncoding=UTF-8&useSSL=false
msg.eventchecker.jdo.option.username=schedulis
msg.eventchecker.jdo.option.password=c2NoZWR1bGlz
bdp.mask.url=[#BDP_MASK_URL]
bdp.mask.app.id=[#BDP_MASK_APP_ID]
bdp.mask.app.token=[#BDP_MASK_APP_TOKEN]
password.private.key=MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCgOlBgJU7JMjRGD0pj8SsbSEJDs8yhmdZf12s7/TttGo7TahmWLSSCa8Lcqe0krMF9n301Izn18hSOIKg5vQNEhVG1ybMsvz3zor67iQhq6CZRIFhndWEZaG2ywG2WRyr+Oet973X9JodQOJbzbDii56JX/FNdgEbjCgGti9s0HwKnx5OBWg8Tc4X4mM8XFt1sDuKU2xTUt1/bDenllbz8W2KNUV+G6QdVfhzZbf40TJfOtlODoflpT25Hw46G9fWrbtTNJv9/TM3mdZQbUJ10Pn8UFgpMC90lbHXrBHUoKGsruQy05OFoGlimYWmB+J6kmnwIPPYi+jav4VxzY3R5AgMBAAECggEAA7V12NPkP/n+hcSi1y9k7Qu+JO0Lho4WDT/iRpA1CPB9b91b3EMNPkuaRhtU1u18yRihAFLha5T/7s5ItpVudu6TFp8lm5qNp48T1Sl13WukP2p9QV/RMJJfP6z+nGTnVN0oW1DorURwC2ZT8CyIHxU3h8vabiai/Wxk34yGNW2c3oQqBy0BT4Orj+E7IMtrMLWWKcDYK6Vgt5O3+vPOnxSMln4yrGen7msC/zWy8wSxeyFyKU6UiVQpepkvah0TG52CbjzDtR8vIjw4Sz/hkwMUlRxX8DvrP2KaQJ4FFHYw6Ui5i7jjDUwIzuZg38v1JS+B8R1CLuFLQLAi53gMAQKBgQDoF1Kzb/O4AopwIByebc/6VxW4G/EVQQ5S62h5tg3i4+isQp0U6cFi/KKQH2EuX8Dzq/0ScA11cIZsYYjn3tNFunbtDfrGkmhnbnLRPDRWqvVjdHv5f5qG+x+Ajf0QgYqfs7qki920Q0QmuLf998IlvPTrXcMnXumZgxT6ZKwv+QKBgQCwu9IavK36gEqcwZHKDe5LLn4qajsuk5FdGhlHO+K2rRd427NAN9jG7eCK1UVRT/cNov02fJa8Wd6j8Zeh4Zkdg9laX5//Owsrq+1orXeaKzijcDHEbJ1M1RrmTqOKtUCUX+xg/0XSfeyfH5TexPe0nqKBirdaTVmpf0DkZvuIgQKBgQDmN7xhIXuv21VXQ4Mf4+2ZdSimJ5FMc+uxdLF9iYjctxXlSW5ngDfD6LWYIIhVZ8YN71xpHZ08ERJGD7mtxunrELtHCcbnkfLeJkDeK8n+7jXbIYCYTGsL2a215yJPbTAEmlNZRSP124OOpUxdL5X1uSl5Dti2BP/StqPofFQQgQKBgEeAxeGhYrZNv2IqgqR//GAYgF0Cu8z9UTucuotydCg6YZu5L42UyrS5OzaQUMo0Ex1GSzIHOCkeJxCnRxTspDknxgFlXOMzbTKPDa9jN1d9kx203727v+x877QsLsiIyob9RDJ+NS6TWe+LJHz4rcs6vz6v87yqPNNxs7x02eGBAoGAWyQPqj1STs1b/ZY9SJaJcpK5xZyNflAbJecUhcs1HoI4tEAbeYWCn5bLk7qwBNiFjPqKbUamlRowTEf6QMlUUkrKROvqu07HV9KXPJzUivhoO9kaGlYLJRseEFYSB7AodpEPtDYeA2CANmQv3O3xjcbL8gLhabjybEX83Uzhajc=
其中metastore为hive元数据数据库,schedulis为业务资料库。密码均需使用Java Base64加密。
配置执行节点commonprivate.properties:
vim /data/appcom/install/schedulis/schedulis-config/schedulis-exec/commonprivate.properties
hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_2_0
azkaban.should.proxy=false
obtain.binary.token=false
hadoop.home=/usr/local/hadoop
hadoop.conf.dir=/usr/local/hadoop/etc/hadoop
hive.home=/usr/local/hive
spark.home=/usr/local/spark
schedulis.home=/data/appcom/install/schedulis/schedulis_exec
jobtypes.home=${schedulis.home}/plugins/jobtypes
execute.as.user=true
azkaban.native.lib=/data/appcom/install/schedulis/schedulis_exec/lib
bdp.client.conf.file=/data/appcom/install/schedulis/schedulis_exec/conf/bdp-job-client.properties
以上涉及路径请根据实际情况填写。
配置执行节点plugin.properties:
vim /data/appcom/install/schedulis/schedulis-config/schedulis-exec/plugin.properties
alerter.class=com.webank.wedatasphere.schedulis.WeBankAlerter
alerter.name=email
alert.type=WeBankAlerter
alarm.server=[#ALARM_SERVER]
alarm.port=[#ALARM_PORT]
alarm.subSystemID=[#ALARM_SYSID]
alarm.alertTitle=[#ALARM_TITLE]
alarm.alerterWay=[#ALARM_WAY]
alarm.reciver=[#ALARM_RECIVER]
alarm.toEcc=0
配置执行节点private.properties:
vim /data/appcom/install/schedulis/schedulis-config/schedulis-exec/private.properties
jobtype.class=com.webank.wedatasphere.dss.plugins.azkaban.linkis.jobtype.AzkabanDssJobType
jobtype.lib.dir=/data/appcom/install/schedulis/schedulis-exec/plugins/jobtypes/linkis/lib
配置web节点azkaban.properties:
vim /data/appcom/install/schedulis/schedulis-config/schedulis-web/azkaban.properties
azkaban.name=WeDataSphere
azkaban.label=WeBank
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
user.manager.class=com.webank.wedatasphere.schedulis.common.user.SystemUserManager
user.manager.xml.file=conf/azkaban-users.xml
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=iZ2ze06twbnm2tgi5kjw3hZ
mysql.database=schedulis
mysql.user=schedulis
mysql.password=c2NoZWR1bGlz
mysql.numconnections=100
velocity.dev.mode=false
jetty.use.ssl=false
jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.send.server.version=false
jetty.keystore=keystore/azkabanjetty.keystore
jetty.password=hadoop
jetty.keypassword=hadoop
jetty.truststore=keystore/azkabanjetty.keystore
jetty.trustpassword=hadoop
jetty.excludeCipherSuites=SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,SSL_DHE_DSS_WITH_DES_CBC_SHA,SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
executor.port=12321
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=CpuStatus,StaticRemainingFlowSize,MinimumFreeMemory
azkaban.queueprocessing.enabled=true
azkaban.webserver.queue.size=100000
azkaban.activeexecutor.refresh.milisecinterval=50000
azkaban.activeexecutor.refresh.flowinterval=5
azkaban.executorinfo.refresh.maxThreads=5
azkaban.executorselector.comparator.Memory=3
azkaban.executorselector.comparator.CpuUsage=2
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
mail.sender=azkaban
mail.host=
mail.user=
mail.password=
job.failure.email=
job.success.email=
lockdown.create.projects=false
schedulis.project.privilege.check=false
department.maintainer.check.switch=true
system.schedule.switch.active=true
cache.directory=cache
jetty.connector.stats=true
executor.connector.stats=true
ladp.switch=false
ladp.ip=127.0.0.1
ladp.port=1389
azkaban.header.referer.url=https://
nginx.ssl.module=close
dss.secret=dws-wtss|WeBankBDPWTSS&DWS@2019
schedulis.private.key=MIICeAIBADANBgkqhkiG9w0BAQEFAASCAmIwggJeAgEAAoGBAKjXi+JdYObX2HCwKlIaJp6GsX6gFsaru0iyB96mOQjSILwetn++j9iwfD4bfIaAQLagGweepQOz9/aT7UUroQibmcGeixelJO0JUN8/zCTAG5NY92acDE1DK/tofZ7sEt18Ry1gsB3Dxq1qEzZZX2sVkFDMLKF54necjgsIFLMzAgMBAAECgYEAiQG/ZQRY6XklDOwmq1DFHcY2qYXGdZhM9QRiFm3TwjCgl4ZkmOxNVYyAhPVQ6uOPn6HzzQ8S4BpdkB0hYAuzMzMHnyhvv/MsH8V3WbJx0kC4If3UsSa0eANizrZaM3CwJ8ehvN2T73Z0B2iTp8Ocl3Wr74ULKC1y5pJnLC6dIMECQQDbhMVqIwPU+VftMEwIik0BDXHSFL4Vjh2rD9AVkvyWKV0qehYHGsXdlG4XlNUc0zAw8oQVFk9t6XGbMEWnk8QJAkEAxObH8gTy7iGYEUprxbEAJq0SJtOk3qfIngICpqdY5qgWqpl8BS7rSZhKswGLsrdY5rfYr/Mngj5cpe72lcbkWwJBAM+RRt4qR8hNEV/9CBgXNeLl5JdB988H93O12wtbVi1i5W5xzHxhS3FOlZ8Eo1LDOtE9r7kExIxobXzRczuWlIkCQQCJIBO1N78bGig2OnbtuYPaa4ONqK1UJtMvP0UrXLYsBHmsm7FkRrWzjizPl077ynZOT1DH0HX+XYHWSaJO0rGrAkBxQjicw5YAxaz0+Vl/HIFSCT4R4BBKDgSmwHf1yOUOIhspA2AOFdCByrmNg2gQ+Pn0ADHN6N3ke42eAt7GiI4T
azkaban.ip.whiteList.enabled=false
azkaban.ip.whiteList=
webserver.ha.model=false
distributelock.lock.timeout=30000
distributelock.get.timeout=30000
azkaban.server.schedule.enable_quartz=false
viewer.plugins=system
session.time.to.live=3600000
azkaban.all.web.url=
配置web节点web_plugin_ims.properties:
vim /data/appcom/install/schedulis/schedulis-config/schedulis-web/web_plugin_ims.properties
alerter.class=com.webank.wedatasphere.schedulis.WeBankAlerter
alerter.name=email
alert.type=WeBankAlerter
alarm.server=[#ALARM_SERVER]
alarm.port=[#ALARM_PORT]
alarm.subSystemID=[#ALARM_SYSID]
alarm.alertTitle=[#ALARM_TITLE]
alarm.alerterWay=[#ALARM_WAY]
alarm.reciver=[#ALARM_RECIVER]
alarm.toEcc=0
配置web节点web_plugin_ims.properties:
vim /data/appcom/install/schedulis/schedulis-config/schedulis-web/web_plugin_system.properties
viewer.name=\u7CFB\u7EDF\u7BA1\u7406
viewer.path=system
viewer.order=1
viewer.hidden=false
viewer.servlet.class=com.webank.wedatasphere.schedulis.system.servlet.SystemServlet
schedulis.db.jdbc.url=jdbc:mysql://iZ2ze06twbnm2tgi5kjw3hZ:3306/schedulis?useUnicode=true&characterEncoding=UTF-8&useSSL=false
schedulis.db.username=schedulis
schedulis.db.password=c2NoZWR1bGlz
4 创建软链接
ln -sf /data/appcom/install/schedulis/schedulis_0.8.0_exec /data/appcom/install/schedulis/schedulis-exec
ln -sf /data/appcom/install/schedulis/schedulis-config/schedulis-exec/azkaban.properties /data/appcom/install/schedulis/schedulis-exec/conf/azkaban.properties
ln -sf /data/appcom/install/schedulis/schedulis-config/schedulis-exec/common.properties /data/appcom/install/schedulis/schedulis-exec/plugins/jobtypes/common.properties
ln -sf /data/appcom/install/schedulis/schedulis-config/schedulis-exec/commonprivate.properties /data/appcom/install/schedulis/schedulis-exec/plugins/jobtypes/commonprivate.properties
ln -sf /data/appcom/install/schedulis/schedulis-config/schedulis-exec/plugin.properties /data/appcom/install/schedulis/schedulis-exec/plugins/jobtypes/linkis/plugin.properties
ln -sf /data/appcom/install/schedulis/schedulis-config/schedulis-exec/private.properties /data/appcom/install/schedulis/schedulis-exec/plugins/jobtypes/linkis/private.properties
ln -sf /data/appcom/install/schedulis/schedulis_0.8.0_web /data/appcom/install/schedulis/schedulis-web
ln -sf /data/appcom/install/schedulis/schedulis-config/schedulis-web/azkaban.properties /data/appcom/install/schedulis/schedulis-web/conf/azkaban.properties
ln -sf /data/appcom/install/schedulis/schedulis-config/schedulis-web/web_plugin_ims.properties /data/appcom/install/schedulis/schedulis-web/plugins/alerter/WebankIMS/conf/plugin.properties
ln -sf /data/appcom/install/schedulis/schedulis-config/schedulis-web/web_plugin_system.properties /data/appcom/install/schedulis/schedulis-web/plugins/viewer/system/conf/plugin.properties
5 execute-as-user
cp $Schedulis_Source_COde/bin/construct/execute-as-user /data/appcom/install/schedulis/schedulis-exec/lib
cd /data/appcom/install/schedulis/schedulis-exec/lib
sudo chown root execute-as-user
sudo chmod 6050 execute-as-use
6 启动验证
mkdir -p /data/appcom/logs/schedulis
sed -i "s#/appcom/config/schedulis-config/#/data/appcom/install/schedulis/schedulis-config/#g" `grep /appcom/config/schedulis-config/ -rl /data/appcom/install/schedulis`
sed -i "s#/appcom/logs/azkaban/#/data/appcom/logs/schedulis/#g" `grep /appcom/logs/azkaban/ -rl /data/appcom/install/schedulis`
chmod +x /wedatasphere/install/schedulis/schedulis_exec/bin/*.sh
chmod +x /wedatasphere/install/schedulis/schedulis_exec/bin/internal/*.sh
chmod +x /wedatasphere/install/schedulis/schedulis_web/bin/*.sh
chmod +x /wedatasphere/install/schedulis/schedulis_web/bin/internal/*.sh
rm -rf /data/appcom/logs/schedulis/*
cd /data/appcom/install/schedulis/schedulis-exec/
bin/start-exec.sh
cd /data/appcom/install/schedulis/schedulis-web/
bin/start-web.sh
访问地址:http://IP:8081
用户名密码:superadmin/Abcd1234