1.1.7 open source (#443)

* Enable to fetch remote log.

* Add heartbeat feature to datax engine plugin.

* Minor fix of package import

* Enable to reverse read log.

* Update version of linkis to 1.4.0-wds

* Update version of linkis to 1.4.0-wds

* Adjust the class structure of data source; Custom the http client;

* Fix some compile problem.

* Fix some conflict problem.

* Enable to autofill columns (1/2);

* Enhance start scripts and add job param config for starrocks

* Enable to autofill columns (2/2);

* Fix the problem in rpc job log service.

* Add 'ES', 'Hive', 'Mongo', 'MySQL', 'Oracle' auto column handler.

* Rename BOOL to BOOLEAN in Column.java

* Enable to edit fields

* Minor fix for packaging

* Fix of job log

* Add job handler for starrocks datasource

* Adjust ExchangisJobTransformsContent.

* Use 'revision' to manage project version.

* Fix the 'isEnd' variable problem of fetching log.

* Fix the problem in metrics; Change the order in DataxContainerOnceExecutor.

* Upgrade init sql of appconn

* Minor fix of appconn init sql

* Enable to match field when search columns is empty

* Add datasource type of tdsql

* Adjust params for tdsql datasource

* Fix the metric and log encoding problem.

* Support MAP/ARRAY/BINARY type

* Adjust interface of listdatasources

* Enable to search projects by name

* Build columns when columns is null

* Replace writeFailedBytes with writeFailedRecords

* Add StarRocksAutoColumnJobHandler

* Minor fix init sql of appconn

* Update linkis version to 1.4.0

* Minor fix

* Update linkis datasource version to 1.5.0-wds

* Update linkis version to 1.5.0-wds-SNAPSHOT

* Declare version of some maven pluigns

* Add rawType in column definition.

* Upgrade sql and update ExchangisJobDssAppConnRestfulApi

* Remove sensitive ip

* Add dssProjectDataSources on project

* Support to relate project with datasource

* Relate data sources to project entity.

* Resolve the conflict.

* Minor fix

* Handle project datasource relation on create and update operation

* Adjust method 'queryDataSources' in 'ExchangisDataSourceService'.

* Adjust 'ProjectDsRelationMapper'

* Enable to query projects with datasource relation info

* Support to simultaneously query project relations and linkis server in 'queryDataSources' method.

* Add authority managment to data-source's interfaces/services.

* Minor fix

* Fix the NLP problem in build data source ui.

* Add project model datasources.

* Update definition info of datasource on job

* Fix duplicate datasources when querying with project

* Adjust the export, import, copy operations of project.

* Adjust log on datax engine plugin

* Support to parse job params defined by source and sink on job handler

* Change type of engine_direction to text on table exchangis_engine_settings

* Update install scripts

* Add project name to job entity

* Query job logs by project privileges

* Init to improve performance 30%.

* Fix the problem in appconn execution.

* Update DataxEngineConnFactory

* Adjust the process in task observers.

* Fix the problem in "common.column.extraFormats".

* Valid by jobs when release project datasource relation

* Update type of projectId to string on ExchangisJobVo

* Improve performance 80%.

* Fix the problem in generating service.

* Minor fix

* Adjust dependencies

* Fix problems with deploy project

* Improve the performance (100%).

* Update linkis version to 1.5.0-wds-SNAPSHOT

* Add basic job validator interface.

* adjust the validator interface, add job name.

* Add the admin authority.

* Update version of jackson-core and jackson-databind

* Remove snakeyaml and adjust properties

* Update launcher.sh

* Disable the proxy mode; Fix the problem in validator.

* Fix the problem in Json.fromJson

* Fix the problem in decimal column.

* Fix the problem of cyclic reference.

* Enable to shutdown scheduler gradually

* Fix the problem in execute user.

* Fix some data problem.

* Fix the problem of stack overflow.

* Add comment message in copying datasource; Add default http port in starrocks job handler

* Fix the problem that lost data source relations when updating project

* Use project creator to update the data source relationship.

* Fix scheduler manager and fix sql

* Advance the field column mappings.

* Fix the problem that losing bml client token.

* Update sql and config

* Deal projectId at deploy time

* Fix the problem of linkis status.

* Update version of aircompressor to 0.27

* Upgrade version to 1.1.6-webank

* Add createUser on DataSourceQueryVO

* Add method queryProjectUsers on ProjectUserMapper

* Define methods to recycle user

* Add types for operateType and targetType

* Move job context to module exchangis-job-common

* Support users with deploy permission to edit projects

* Adjust modules of job

* Add rate limit and datasource model module

* Upgrade linkis version to 1.6.0

* Delete class header

* Move datasource domain to module exchangis-datasource-core

* Adjust the data source domain.

* Enhance the data source service(1/3).

* Update dataSource model

* Rename module exchangis-dao to exchangis-common

* Enhance the data source service(2/3).

* Minor fix of DataSourceModel

* Support to define keys for ds model

* Enhance the data source service (3/3).

* Minor fix of dataSource

* Minor fix of rateLimit and dsModel

* Complete the strategy of model in data source service.

* Minor adjust of datasource service

* Add generic http client builder

* Add field isSerialize to DataSourceModelTypeKey

* Merge ds keys into ds model keys

* Finish the todo items in data source service.

* Add some util class to common module

* Add the strategy to cascade update the data source and data source model. Fix some problem.

* Declare errorCode for some operation

* Update keyDefines of dsModel

* Add dataSourceModel to ExchangisDataSourceItem

* Delete ratelimit when the model is deleted

* Add dsModelId to connect interface

* Add the strategy to check duplicate model name.

* Add several fields to ExchangisDataSourceItem

* Add interface getJobEngineSettingsUI to ExchangisJobRestfulApi

* Avoid NPE on method getExchangisDsDefinition

* Add construct method to class DefaultDirectExecutor

* Enhance method resolveParams on class DataSourceModel

* Use Json instead of ObjectMapper to parse jobContent

* Fix logic of datasource

* Minor fix of ratelimit

* Add detail method on dsModel and rateLimit module

* Stash commit

* Set default proxy mode to true;Remove unused serverUrl of linkis

* Fix operation of datasource model

* Support to apply and release ratelimit resource for launchedJobTask

* Fix operation of project module

* Enable to relate data source from job to job automatically.

* Resolve the conflict.

* Judge null on computing ratelimit

* Set createOwner to dsModel by normal user

* Release rateLimit resource when exception occurs

* Fix the compile problem.

* Add params to auth hadoop

* Minor fix on project and ds

* Minor fix on rate limit

* Add modules about exchangis privilege

* Rename module exchangis-project-entity to exchangis-project-common

* Define linkis version to 1.5.0-wds-SNAPSHOT

* Move dto and exception from exchangis-job-server to exchangis-job-common

* Move service from exchangis-project-server to exchangis-projecr-provider

* Move domain and vo from exchangis-datasource-server to exchangis-datasource-core

* Update get proxyUser method

* Adjust position of exchangis-privilege

* Minor fix pom

* Add serverUrl to ExchangisClientConfigBuilder

* Delete user group when delete project

* Define version of maven-compiler-plugin as 3.8.1

* Support to update createUser of project from appconn

* Add method selectByDomain on ProjectMapper

* Don't valid permission when dss update project

* Support to query jobs by identify

* Print error detail on getting transform settings by job

* Support to bulk update jobs

* Add data source split strategy(50%).

* Add batchInsert method on dsModel

* Update matchIdentify method on jobContent

* Move SpringContextHolder to exchangis-common

* Update related datasources on service

* Support to copy datasource model

* Set max retry num to zero on SubmitSchedulerTask

* Upgrade version to 1.1.7-webank

* Minor adjust DsKeyDefineUtil

* Add mapping key in jobParam.

* Add data source split strategy(60%).

* Add deploy mode info & Fix job count

* Redefined total count after filtering job list

* Query ds by creator on executing

* Enable to copy the sub exchangis job(50%).

* Support to save identify on task entity

* Enable to copy the sub exchangis job(100%).

* Support to query task by identify

* Enable to use the job params to render the job(100%).

* Advance the strategy to render the sub exchangis job.

* Support to decrypt password from linkis datasource

* Fix dirty data on querying ds detail

* Add the "FOR UPDATE" to avoid the concurrency problem in updating job's status and progress.

* Remove dsTypeId on definition of dsModelTypeKey

* Add name suffix to split jobs.

* Fix the problem in escaping the params.

* Fix the problem that not released of parallel resources.

* Update linkis-computation-client version to 1.6.0-wds

* Add some logs on job launch

* Recognize special column on sql command

* Fix the problem in splitting jobs.

* Fix the problem in splitting jobs (null mapping key)

* Update queryPageList on ProjectMapper.xml

* Minor upgrade dependency version

* Set encrypt value on datasource

* Auth-token update.

* Fix the problem of blank exec user.

* Fix conflict with branch 1.1.7

* Adjust deploy config

* Minor fix

---------

Co-authored-by: davidhua <david_hua1996@hotmail.com>
Co-authored-by: jefftlin <jefft_lin@163.com>

Author: 3 people

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -63,7 +63,7 @@ Exchangis 抽象了一套统一的数据源和同步作业定义插件，允许
 
 如果您想得到最快的响应，请给我们提 issue，或者扫码进群：
 
-![communication](images/zh_CN/ch1/code.png)
+![communication](images/zh_CN/ch1/communication.png)
 
 ## License
 

.github/ISSUE_TEMPLATE/config.yml

@@ -59,7 +59,7 @@ With the help of [Linkis](https://github.com/apache/incubator-linkis) computing
 
 If you want to get the fastest response, please mention issue to us, or scan the code into the group ：
 
-![communication](images/en_US/ch1/code.png)
+![communication](images/en_US/ch1/communication.png)
 
 ## License
 

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,6 @@
+server.port=9321
+spring.application.name=dss-exchangis-main-server-dev
+eureka.client.serviceUrl.defaultZone=http://{IP}:{PORT}/eureka/
+eureka.instance.metadata-map.test=wedatasphere
+management.endpoints.web.exposure.include=refresh,info
+logging.config=classpath:log4j2.xml