Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update communication and merge 1.5.0 #1136

Open
wants to merge 4,113 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4113 commits
Select commit Hold shift + click to select a range
274697a
Merge branch 'master' into master
zqburde Apr 29, 2024
475558a
git server search fix2
qweqnotallow Apr 29, 2024
d80710c
git server search fix3(logger)
qweqnotallow Apr 29, 2024
9d023a4
support auto.disabled
Apr 30, 2024
9cfd511
update query.frequency to 1m
Apr 30, 2024
3eea73c
workflow server/ orchestrator server lock fix
qweqnotallow Apr 30, 2024
2e2e396
clear OrchestratorFrameworkServiceImpl info
qweqnotallow Apr 30, 2024
fd9c3d5
fix
Apr 30, 2024
c767aa5
get git Url cookie
qweqnotallow Apr 30, 2024
d5150ae
get git Url cookie fix
qweqnotallow Apr 30, 2024
ef272d6
1.4.0编排添加是否默认引用字段,获取工作空间所有员工添加部门科室字段
xuanzhou11 Apr 30, 2024
2581b2e
git search/server fix
qweqnotallow May 6, 2024
d337679
fix coding error
xuanzhou11 May 6, 2024
b0505a9
fix error
xuanzhou11 May 6, 2024
9f64253
add Serializable
xuanzhou11 May 6, 2024
a264031
add serializable
xuanzhou11 May 6, 2024
303c612
add getter;
xuanzhou11 May 6, 2024
fc961fb
add method getAllDepartments
xuanzhou11 May 6, 2024
a7aaa5f
add interface getAllWorkspaceUsersWithDepartment;
xuanzhou11 May 7, 2024
2aecca4
git project export fix
qweqnotallow May 7, 2024
5dd580d
git project export fix
qweqnotallow May 7, 2024
0fb8b5e
git project export fix1
qweqnotallow May 7, 2024
a02442d
fix
xuanzhou11 May 7, 2024
fd24972
server submitFlow add threadPool and status
qweqnotallow May 7, 2024
5bdd73c
server jump to gitUrl fix
qweqnotallow May 7, 2024
b3ab34b
git add CurrentCommit/server: orchestrator version info add commitId
qweqnotallow May 7, 2024
bfcd29b
git add CurrentCommit fix1
qweqnotallow May 7, 2024
a77a424
git add CurrentCommit fix2
qweqnotallow May 7, 2024
dd982f4
git add history/server add /publish/history
qweqnotallow May 7, 2024
4b30342
git createProject/modifyProject fix
qweqnotallow May 8, 2024
5945472
git server remove gitUrl
qweqnotallow May 8, 2024
3b70f49
deleteLock add username
qweqnotallow May 8, 2024
7f2b6c1
remove submitflow threadPool
qweqnotallow May 9, 2024
eb10ae7
optimize schedule executor
May 9, 2024
e11c6a3
Merge remote-tracking branch 'origin/dev-1.4.0' into dev-1.1.20-gitflow
qweqnotallow May 9, 2024
c565ed3
pom -> 1.5.0
qweqnotallow May 9, 2024
a78dc4f
add OrchestratorSubmitJob
qweqnotallow May 9, 2024
7dfd552
fix OrchestratorSubmitJob
qweqnotallow May 9, 2024
3e4f85b
gitUrl fix
qweqnotallow May 9, 2024
f05426d
submitFlow status fix
qweqnotallow May 10, 2024
fda140b
1.3.0->1.3.1
qweqnotallow May 11, 2024
a19d3c6
mlss fix
qweqnotallow May 11, 2024
95dadb6
Revert "1.3.0->1.3.1"
qweqnotallow May 11, 2024
92d1d07
mlss fix1 pom
qweqnotallow May 11, 2024
613113a
submitFlow status fix1
qweqnotallow May 11, 2024
5dc7ec3
1.3.0 -> 1.3.1
qweqnotallow May 11, 2024
87dda8b
git server fix : removeDir
qweqnotallow May 11, 2024
f4e2543
dss server fix : createProject add associateGit
qweqnotallow May 11, 2024
9f04e46
dss server fix : modifyProject add associateGit
qweqnotallow May 11, 2024
9de8271
dss server fix : submit status sql fix
qweqnotallow May 11, 2024
dee362c
git server search : add target pattern(.metaConf/.properties)
qweqnotallow May 11, 2024
9dceb87
git server: sql fix
qweqnotallow May 11, 2024
7862b38
dss server fix: sql
qweqnotallow May 13, 2024
fe18e8f
Modify mapper's * to fields
xuanzhou11 May 13, 2024
00389a9
git server : rename attr
qweqnotallow May 14, 2024
63c6549
git server : sql fix
qweqnotallow May 14, 2024
20602e3
git server add : checkout and remove
qweqnotallow May 14, 2024
5ac692d
git server add : rename
qweqnotallow May 14, 2024
410832a
dss-server : publish git server-> dss server
qweqnotallow May 14, 2024
e25bd65
dss-server : git revert after dss revert
qweqnotallow May 14, 2024
784a232
dss-server : dss remove after git remove
qweqnotallow May 14, 2024
1b8e8c5
Merge branch 'dev-1.4.0' into dev-1.5.0
qweqnotallow May 15, 2024
21e0ee5
git server: rename fix
qweqnotallow May 16, 2024
27e61bd
feat: dev 1.4.0 changes
demonray May 16, 2024
c392430
dss-server: getAllProjects add associateGit
qweqnotallow May 16, 2024
21592fa
dss-server: getAllProjects add associateGit sql fix
qweqnotallow May 16, 2024
7bea67e
dss-server: publish sql fix
qweqnotallow May 16, 2024
0aac3c7
dss-server: rollback labels fix
qweqnotallow May 17, 2024
e3f3c63
git-server: add conflict
qweqnotallow May 17, 2024
3bb769d
git-server: unzip fix
qweqnotallow May 17, 2024
a2bc4a6
git-server: unzip fix
qweqnotallow May 17, 2024
63e8857
add workflow server to orchestrator server
qweqnotallow May 17, 2024
c0b2b66
fix: fix en text
demonray May 17, 2024
8a63f2f
add workflow server to orchestrator server
qweqnotallow May 17, 2024
7a85be6
git server: rename fix
qweqnotallow May 20, 2024
fdd407e
Merge branch 'dev-1.4.0' into dev-1.5.0
qweqnotallow May 20, 2024
0f72eb4
git server: search Result fix
qweqnotallow May 20, 2024
2e23315
git server: try catch fix
qweqnotallow May 20, 2024
5b25dad
dss server: exportAllSubFlowNew null fix
qweqnotallow May 20, 2024
ffcd5a2
git server: pull fix
qweqnotallow May 20, 2024
c93648e
git server: pull fix
qweqnotallow May 20, 2024
2a971f6
dss server: add type
qweqnotallow May 20, 2024
40cff8c
dss server: add type
qweqnotallow May 21, 2024
b1f00c0
dss server: add orchestrator/orchestrator_version to BMLResource
qweqnotallow May 21, 2024
066ec81
git server : add error exception
qweqnotallow May 21, 2024
26badcc
git server : add exception
qweqnotallow May 22, 2024
07c237d
Merge branch 'dev-1.3.1' into dev-1.4.0
qweqnotallow May 22, 2024
a2e3906
Merge remote-tracking branch 'origin/dev-1.4.0' into dev-1.5.0
qweqnotallow May 22, 2024
eb35e5c
git server : password token encrypt
qweqnotallow May 23, 2024
9d27271
fix: rm lerna
demonray May 23, 2024
ac0ac32
Merge branch 'dev-1.5.0' of github.com:WeDataSphere/DataSphereStudio …
demonray May 23, 2024
ac1b52f
rm appconn resource zip after unzip
May 23, 2024
33575d4
git server : throw exception fix
qweqnotallow May 23, 2024
24ce761
git server : fix exception
qweqnotallow May 24, 2024
e178140
git server : fix exception
qweqnotallow May 24, 2024
0426636
git server : add gitConnect
qweqnotallow May 24, 2024
84f1129
git server : exception fix2
qweqnotallow May 24, 2024
c57444d
git server : pom add httpclient
qweqnotallow May 24, 2024
a6f0079
fix: rm lerna
demonray May 23, 2024
e06e6e5
git server : gitUrlPre fix
qweqnotallow May 24, 2024
2d3f4df
fix: 侧边形状控制
demonray May 24, 2024
e64938f
git server : exception fix
qweqnotallow May 24, 2024
872e684
Merge branch 'dev-1.5.0' of github.com:WeDataSphere/DataSphereStudio …
demonray May 24, 2024
271921e
git server/dss server: fix updateStatus
qweqnotallow May 24, 2024
f11fb08
git server: fix exception
qweqnotallow May 24, 2024
f4d508e
git server: fix exception1
qweqnotallow May 24, 2024
89ead74
git server: fix exception1
qweqnotallow May 24, 2024
f9df239
git server: fix push
qweqnotallow May 24, 2024
288244c
dss server: fix exception2
qweqnotallow May 24, 2024
f6b0c2e
dss server: fix exception OrchestratorPluginServiceImpl
qweqnotallow May 24, 2024
1d7e042
git-server: fix priKey
qweqnotallow May 24, 2024
3a3cb99
git-server: fix priKey
qweqnotallow May 24, 2024
9ba0933
工作流导出时第三方节点物料可选
May 27, 2024
f3ec451
git-server: fix decrypt
qweqnotallow May 27, 2024
46c4c47
git-server: fix decrypt
qweqnotallow May 27, 2024
4b0dd48
git-server: fix decrypt2
qweqnotallow May 27, 2024
d94177c
git-server: workspace
qweqnotallow May 27, 2024
5bbd998
git-server: fix3 decrypt
qweqnotallow May 27, 2024
0e2565d
dss-server: 校验Git是否绑定
qweqnotallow May 27, 2024
c0075e3
升级漏洞
xuanzhou11 May 28, 2024
d90ce6f
Merge remote-tracking branch 'origin/dev-1.5.0' into dev-1.5.0
xuanzhou11 May 28, 2024
3fd434e
dss-server: add associateGit to workspaces
qweqnotallow May 28, 2024
6fbcef9
取消poi-ooxml排除
xuanzhou11 May 28, 2024
e1ae361
dss-server: fix
qweqnotallow May 28, 2024
b4a4a1c
Merge remote-tracking branch 'origin/dev-1.5.0' into dev-1.5.0
qweqnotallow May 28, 2024
e9e0d54
dss-server: fix createProject
qweqnotallow May 28, 2024
ffc0818
dss-server:fix getAllWorkspaceUsersDepartment
xuanzhou11 May 29, 2024
0278508
dss-server:dix getAllWorkspaceUsersDepartment
xuanzhou11 May 29, 2024
c8d31a9
dss-server: fix orchestrator status
qweqnotallow May 29, 2024
faf268f
Merge remote-tracking branch 'origin/dev-1.5.0' into dev-1.5.0
qweqnotallow May 29, 2024
8995a2e
feat: add next-web
demonray May 24, 2024
93087a4
dss-server:Upgrade jar version
xuanzhou11 May 29, 2024
709569a
Merge branch 'dev-1.5.0' of github.com:WeDataSphere/DataSphereStudio …
demonray May 29, 2024
da5b468
Merge branch 'dev-1.5.0' of github.com:WeDataSphere/DataSphereStudio …
demonray May 29, 2024
8b82809
dss-server: fix allType
qweqnotallow May 29, 2024
f28a0bd
Merge remote-tracking branch 'origin/dev-1.5.0' into dev-1.5.0
qweqnotallow May 29, 2024
91d5e05
dss-server: fix selectOrchestrator
qweqnotallow May 29, 2024
4f4bf42
git-server: unlimit searchResult
qweqnotallow May 29, 2024
70d656c
git-server: fix gitHistory
qweqnotallow May 29, 2024
4f7f805
dss-server: rollback unlock by system
qweqnotallow May 29, 2024
ceba960
dss-server: fix unlock
qweqnotallow May 29, 2024
af4effd
dss-server: fix unlock
qweqnotallow May 29, 2024
309a66e
dss-server: add workspaceId in path
qweqnotallow May 29, 2024
62c139a
dss-server: add limit in workspaceUser
qweqnotallow May 29, 2024
2e1d829
dss-server: fix submitOrchestrator
qweqnotallow May 30, 2024
a39a288
dss-server: fix submitOrchestrator
qweqnotallow May 30, 2024
4f42711
dss-server: fix commitMessage
qweqnotallow May 30, 2024
a42ff29
git-server: fix user
qweqnotallow May 30, 2024
87822f1
git-server: fix user
qweqnotallow May 30, 2024
ea8da3a
git-server: fix user
qweqnotallow May 30, 2024
0a178d3
git-server: fix createProject path
qweqnotallow May 30, 2024
0d9f99e
git-server: fix createProject path
qweqnotallow May 30, 2024
e506799
dss-server: add exception in submitFlow/status
qweqnotallow May 30, 2024
cb1ed30
dss-server: add exception in submitFlow/status
qweqnotallow May 30, 2024
02c5281
dss-server: add exception in submitFlow/status
qweqnotallow May 30, 2024
b1ba02a
dss-server: fix rollback
qweqnotallow May 30, 2024
f944982
dss-server:fix DepartmentUserTreeVo add nodeId
xuanzhou11 May 31, 2024
d2bace1
git-server: fix exception
qweqnotallow May 31, 2024
c0a360d
dss-server: fix submitFlow
qweqnotallow May 31, 2024
e98d83f
dss-server: fix archive
qweqnotallow May 31, 2024
a00968f
git-server:fix exception
qweqnotallow May 31, 2024
d0c9b21
git-server:fix git pull
qweqnotallow May 31, 2024
043f355
git-server:fix history
qweqnotallow May 31, 2024
def2544
git-server:fix username
qweqnotallow May 31, 2024
9bfa8b8
git-server:fix comment
qweqnotallow May 31, 2024
e6956e8
git-server:fix comment add limit 100
qweqnotallow May 31, 2024
22e9421
git-server:fix workTree
qweqnotallow May 31, 2024
8d65a1e
git-server:fix metaConf
qweqnotallow May 31, 2024
107272f
git-server:fix metaConf
qweqnotallow May 31, 2024
667f638
git-server:fix metaConf2
qweqnotallow May 31, 2024
38d24b3
git-server:fix history
qweqnotallow Jun 3, 2024
abbc0e2
git-server:fix commitId
qweqnotallow Jun 3, 2024
6bf6088
git-server:fix exclude
qweqnotallow Jun 3, 2024
a1d74f4
git-server:fix history
qweqnotallow Jun 3, 2024
317362c
git-server:fix subflow
qweqnotallow Jun 3, 2024
ed4d449
git-server:fix workspaceId
qweqnotallow Jun 3, 2024
e4a5d21
git-server:fix search
qweqnotallow Jun 4, 2024
388b450
dss-server:fix DepartmentUserTreeVo add Construction
xuanzhou11 Jun 4, 2024
756a3fd
git-server:fix search
qweqnotallow Jun 4, 2024
c6a65aa
Merge remote-tracking branch 'origin/dev-1.5.0' into dev-1.5.0
qweqnotallow Jun 4, 2024
75d785c
dss-server:fix rollback
qweqnotallow Jun 4, 2024
b7f4730
dss-server:fix history
qweqnotallow Jun 4, 2024
0e83be2
dss-server:fix rename
qweqnotallow Jun 4, 2024
e154206
dss-server:fix archive
qweqnotallow Jun 4, 2024
8c06b9c
dss-server:fix rollback
qweqnotallow Jun 5, 2024
a28ed5e
dss-server:fix rollback2
qweqnotallow Jun 5, 2024
c6e16f5
dss-server:fix archive
qweqnotallow Jun 5, 2024
2f98fbb
dss-server:fix commit
qweqnotallow Jun 5, 2024
65eca78
dss-server:fix commit
qweqnotallow Jun 5, 2024
140bd39
dss-server:fix projectmeta
qweqnotallow Jun 5, 2024
2574201
dss-server:fix rename
qweqnotallow Jun 5, 2024
17b2b34
Merge branch 'master' into dev-1.4.0
zqburde Jun 5, 2024
9626621
Merge pull request #9 from WeDataSphere/dev-1.4.0
zqburde Jun 5, 2024
ea06200
dss-server:fix export
qweqnotallow Jun 5, 2024
bf9a66b
dss-server:fix rollback1
qweqnotallow Jun 5, 2024
56d37da
dss-server:fix updateTOSaveStatus
qweqnotallow Jun 5, 2024
1294b1f
dss-server:fix updateTOSaveStatus
qweqnotallow Jun 5, 2024
181db0c
dss-server:fix search
qweqnotallow Jun 5, 2024
8baa803
dss-server:fix rollback
qweqnotallow Jun 5, 2024
1ac79e1
dss-server:fix search1
qweqnotallow Jun 5, 2024
37a2f43
dss-server:fix saveFlow
qweqnotallow Jun 6, 2024
1d3387d
dss-server:fix submitFlow
qweqnotallow Jun 6, 2024
9f8f82e
git-server: fix errMsg
qweqnotallow Jun 6, 2024
eae81e2
add sql
qweqnotallow Jun 6, 2024
56630f3
支持数据源配置
Jun 6, 2024
0c30623
fix err MSG
qweqnotallow Jun 6, 2024
a8c48dd
Merge remote-tracking branch 'origin/dev-1.5.0' into dev-1.5.0
qweqnotallow Jun 6, 2024
12243a4
导入时更新workspaceId
Jun 6, 2024
ee82d94
fix rollback
qweqnotallow Jun 6, 2024
15dddfc
项目支持配置数据源——回显
Jun 7, 2024
b0172ec
fix rollback: add system
qweqnotallow Jun 7, 2024
1a78c04
项目支持配置数据源——回显
Jun 7, 2024
b541cf0
fix delete
qweqnotallow Jun 7, 2024
d106d82
Merge remote-tracking branch 'origin/dev-1.5.0' into dev-1.5.0
qweqnotallow Jun 7, 2024
359d195
add associate check
qweqnotallow Jun 11, 2024
91b6a74
git-server: add schedule-delete local project
qweqnotallow Jun 12, 2024
336790a
git-server: add search Type fix
qweqnotallow Jun 13, 2024
ea29097
git-server: add search fix
qweqnotallow Jun 13, 2024
3899394
git-server: fix paths
qweqnotallow Jun 13, 2024
b704e4c
git-server: code Review fix
qweqnotallow Jun 13, 2024
b90377a
git-server: code Review fix delete
qweqnotallow Jun 14, 2024
cab7ada
git-server: code Review fix check
qweqnotallow Jun 15, 2024
47b9901
git-server: code Review fix create
qweqnotallow Jun 15, 2024
e3c1128
git-server: code Review fix create1
qweqnotallow Jun 15, 2024
a896dfe
git-server: add create git_user_id
qweqnotallow Jun 16, 2024
7a7e59b
git-server: add update git_user_id
qweqnotallow Jun 19, 2024
1cf9630
git-server: fix exception
qweqnotallow Jun 19, 2024
3a1fa7a
git-server: fix exception
qweqnotallow Jun 19, 2024
2628614
git-server: fix revert
qweqnotallow Jun 19, 2024
14faf0b
git-server: fix saveFlow exception
qweqnotallow Jun 20, 2024
9cb1f15
git-server: fix saveFlow exception1
qweqnotallow Jun 20, 2024
bac0495
git-server: fix exportOrcSqlFile
qweqnotallow Jun 20, 2024
4f69085
git-server: fix exportOrcSqlFile1
qweqnotallow Jun 20, 2024
49ed9b7
git-server: add checkWorkspace to workflow
qweqnotallow Jun 20, 2024
e2c66e4
git-server: add checkWorkspace to workflow 1
qweqnotallow Jun 20, 2024
cf762fd
git-server: add checkWorkspace to workflow 2
qweqnotallow Jun 20, 2024
1422d39
git-server: fix error code
qweqnotallow Jun 21, 2024
67e7f6c
dss-server: fix export
qweqnotallow Jun 21, 2024
5a2f3be
dss-server: fix export1
qweqnotallow Jun 21, 2024
c495988
dss-server: fix export2
qweqnotallow Jun 23, 2024
7382660
dss-server: fix export3
qweqnotallow Jun 23, 2024
18e3316
dss-server: fix export null
qweqnotallow Jun 23, 2024
c8f994c
dss-server: fix export4
qweqnotallow Jun 23, 2024
3d6824c
dss-server: fix import
qweqnotallow Jun 24, 2024
b7b45e7
日志优化
Jun 24, 2024
cf079b9
dss-server: fix rollback
qweqnotallow Jun 26, 2024
5838dd3
dss-server: fix rollback
qweqnotallow Jun 26, 2024
0e92dd6
Merge branch 'dev-1.5.0'
qweqnotallow Jun 27, 2024
dfe8f94
update communication.jpg
zqburde Jan 3, 2025
7772907
Merge branch 'main'
zqburde Jan 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,9 @@ dss-apps/dss-data-governance/dss-data-warehouse-dao/target
dss-apps/dss-data-governance/dss-data-warehouse-service/target
dss-apps/dss-data-governance/dss-data-warehouse-server/target

#dss-git
dss-git/dss-git-common/target
dss-git/dss-git-server/target

# plugins
plugins/azkaban/linkis-jobtype/target
Expand Down
2 changes: 1 addition & 1 deletion README-ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@

       您也可以扫描下面的二维码,加入我们的 微信群,以获得更快速的响应。

![交流](images/zh_CN/readme/communication.png)
![交流](images/zh_CN/readme/communication.jpg)

## 谁在使用 DSS

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ English | [中文](README-ZH.md)

       You can scan the QR code below to join our WeChat to get more immediate response.

![communication](images/en_US/readme/communication.png)
![communication](images/en_US/readme/communication.jpg)

## Who is using DSS

Expand Down
2 changes: 1 addition & 1 deletion assembly/bin/appconn-refresh.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ function isSuccess(){

function refresh() {
while true; do
response=$(curl -H "Token-Code:" -H "Token-User:hadoop" -X GET http://${GATEWAY_INSTALL_IP}:${GATEWAY_PORT}/api/rest_j/v1/dss/framework/project/appconn/${APPCONN_NAME}/load)
response=$(curl -H "Token-Code:BML-AUTH" -H "Token-User:hadoop" -X GET http://${GATEWAY_INSTALL_IP}:${GATEWAY_PORT}/api/rest_j/v1/dss/framework/project/appconn/${APPCONN_NAME}/load)
if [[ $response == *"not appconn manager node"* ]]; then
echo 'not appconn manager node, we will try again 5 seconds later'
sleep 5
Expand Down
2 changes: 1 addition & 1 deletion assembly/bin/dssinstall.sh
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ function changeCommonConf(){
sed -i "s#hostname:.*#hostname: $SERVER_IP#g" $CONF_APPLICATION_YML
sed -i "s#wds.linkis.server.mybatis.datasource.url.*#wds.linkis.server.mybatis.datasource.url=jdbc:mysql://${MYSQL_HOST}:${MYSQL_PORT}/${MYSQL_DB}?characterEncoding=UTF-8#g" $CONF_DSS_PROPERTIES
sed -i "s#wds.linkis.server.mybatis.datasource.username.*#wds.linkis.server.mybatis.datasource.username=$MYSQL_USER#g" $CONF_DSS_PROPERTIES
sed -i "s#wds.linkis.server.mybatis.datasource.password.*#***REMOVED***$MYSQL_PASSWORD#g" $CONF_DSS_PROPERTIES
sed -i "s#wds.linkis.server.mybatis.datasource.password.*#wds.linkis.server.mybatis.datasource.password=$MYSQL_PASSWORD#g" $CONF_DSS_PROPERTIES
sed -i "s#wds.linkis.gateway.ip.*#wds.linkis.gateway.ip=$GATEWAY_INSTALL_IP#g" $CONF_DSS_PROPERTIES
sed -i "s#wds.linkis.gateway.port.*#wds.linkis.gateway.port=$GATEWAY_PORT#g" $CONF_DSS_PROPERTIES
sed -i "s#wds.linkis.gateway.url.*#wds.linkis.gateway.url=http://$GATEWAY_INSTALL_IP:$GATEWAY_PORT/#g" $CONF_DSS_PROPERTIES
Expand Down
8 changes: 4 additions & 4 deletions assembly/config/config.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ GATEWAY_INSTALL_IP=127.0.0.1
GATEWAY_PORT=9001

### Linkis BML Token
BML_AUTH=
BML_AUTH=BML-AUTH

################### The install Configuration of all Micro-Services start #####################
#
Expand All @@ -50,16 +50,16 @@ DSS_APPS_SERVER_PORT=9044

############## ############## dss_appconn_instance configuration start ############## ##############
####eventchecker表的地址,一般就是dss数据库
EVENTCHECKER_JDBC_URL=jdbc:mysql://172.0.0.1:3305/linkis_kinghao?characterEncoding=UTF-8
EVENTCHECKER_JDBC_URL=jdbc:mysql://172.16.16.16:3305/linkis_kinghao?characterEncoding=UTF-8
EVENTCHECKER_JDBC_USERNAME=linkis
EVENTCHECKER_JDBC_PASSWORD=linkis@Wds

#### hive地址
DATACHECKER_JOB_JDBC_URL=jdbc:mysql://172.0.0.1:3306/hivemeta?useUnicode=true
DATACHECKER_JOB_JDBC_URL=jdbc:mysql://172.16.16.10:3306/hivemeta?useUnicode=true
DATACHECKER_JOB_JDBC_USERNAME=hivemeta
DATACHECKER_JOB_JDBC_PASSWORD=Linkishivemeta@123
#### 元数据库,可配置成和DATACHECKER_JOB的一致
DATACHECKER_BDP_JDBC_URL=jdbc:mysql://172.0.0.1:3306/hivemeta?useUnicode=true
DATACHECKER_BDP_JDBC_URL=jdbc:mysql://172.16.16.10:3306/hivemeta?useUnicode=true
DATACHECKER_BDP_JDBC_USERNAME=hivemeta
DATACHECKER_BDP_JDBC_PASSWORD=Linkishivemeta@123

Expand Down
2 changes: 1 addition & 1 deletion assembly/dss-package/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<artifactId>dss</artifactId>
<groupId>com.webank.wedatasphere.dss</groupId>
<version>1.1.0.20-SNAPSHOT</version>
<version>1.5.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion assembly/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<artifactId>dss</artifactId>
<groupId>com.webank.wedatasphere.dss</groupId>
<version>1.1.0.20-SNAPSHOT</version>
<version>1.5.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<packaging>pom</packaging>
Expand Down
1 change: 0 additions & 1 deletion conf/dss-framework-orchestrator-server.properties
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.server.

wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.framework.appconn.dao,com.webank.wedatasphere.dss.orchestrator.core.dao,com.webank.wedatasphere.dss.server.dao,com.webank.wedatasphere.dss.application.dao,com.webank.wedatasphere.dss.workspace.mapper,com.webank.wedatasphere.dss.workspace.common.dao,com.webank.wedatasphere.dss.workspace.common.dao,com.webank.wedatasphere.dss.orchestrator.db.dao,com.webank.wedatasphere.dss.workflow.dao,com.webank.wedatasphere.dss.framework.appconn.dao,com.webank.wedatasphere.dss.flow.execution.entrance.dao

wds.dss.server.scheduling.clear.cs.cron=0 0 3 * * ?

wds.dss.publish.max.remain.version=3

Expand Down
4 changes: 2 additions & 2 deletions conf/dss-server.properties
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ wds.dss.appconn.checker.development.ignore.list=orchestrator-framework,workflow,
wds.dss.appconn.checker.project.ignore.list=
## filter appconn
wds.dss.appconn.disabled=
wds.dss.trust.token=***REMOVED***
wds.dss.trust.token=FotUrJ9JmhQ=

##import file dir
wds.dss.server.scheduling.clear.cs.cron=0/5 * * * * ?
Expand All @@ -44,7 +44,7 @@ wds.linkis.resultSet.store.path=hdfs:///tmp/linkis
wds.linkis.server.socket.mode=true

wds.linkis.client.flow.adminuser=ws
wds.linkis.client.flow.author.user.token=172.0.0.1
wds.linkis.client.flow.author.user.token=WS-AUTH

wds.linkis.server.component.exclude.classes=org.apache.linkis.entranceclient.conf.ClientForEntranceSpringConfiguration,org.apache.linkis.entranceclient.conf.ClientSpringConfiguration

Expand Down
10 changes: 5 additions & 5 deletions conf/dss.properties
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@ wds.linkis.gateway.wtss.url=http://127.0.0.1:9001/
wds.linkis.mysql.is.encrypt=false
wds.linkis.server.mybatis.datasource.url=
wds.linkis.server.mybatis.datasource.username=
***REMOVED***
wds.linkis.bml.auth.token.value=
linkis.configuration.linkisclient.auth.token.value=
wds.linkis.context.client.auth.value=
wds.linkis.errorcode.auth.token=
wds.linkis.server.mybatis.datasource.password=
wds.linkis.bml.auth.token.value=BML-AUTH
linkis.configuration.linkisclient.auth.token.value=BML-AUTH
wds.linkis.context.client.auth.value=BML-AUTH
wds.linkis.errorcode.auth.token=BML-AUTH

wds.dss.check.server.active.period=30

Expand Down
33 changes: 33 additions & 0 deletions db/dss_ddl.sql
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ CREATE TABLE `dss_orchestrator_info` (
`orchestrator_level` varchar(32) DEFAULT NULL COMMENT '工作流级别',
`update_user` varchar(100) DEFAULT NULL COMMENT '更新人',
`update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间',
`status` VARCHAR(64),
PRIMARY KEY (`id`) USING BTREE,
UNIQUE KEY `unique_idx_uuid` (`uuid`)
) ENGINE=InnoDB AUTO_INCREMENT=326 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT;
Expand All @@ -70,6 +71,7 @@ CREATE TABLE `dss_orchestrator_version_info` (
`content` varchar(255) DEFAULT NULL,
`context_id` varchar(200) DEFAULT NULL COMMENT '上下文ID',
`valid_flag` INT(1) DEFAULT '1' COMMENT '版本有效标示,0:无效;1:有效',
`commit_id` varchar(64),
PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=422 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT;

Expand Down Expand Up @@ -116,6 +118,7 @@ CREATE TABLE `dss_project` (
`dev_process` varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '开发流程,多个以英文逗号分隔,取得的值是dss_workspace_dictionary中的dic_key(parent_key=p_develop_process)',
`orchestrator_mode` varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '编排模式,多个以英文逗号分隔,取得的值是dss_workspace_dictionary中的dic_key(parent_key=p_arrangement_mode或下面一级)',
`visible` tinyint(4) DEFAULT '1' COMMENT '0:已删除;1:未删除(默认)',
`associate_git` TINYINT DEFAULT '0' COMMENT '0:未接入git,1:已接入git',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=313 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT;

Expand Down Expand Up @@ -639,3 +642,33 @@ key `idx_limit_name` (`limit_name`)
) ENGINE = InnoDB
DEFAULT CHARSET = utf8mb4 COLLATE=utf8mb4_bin COMMENT ='dss用户限制表';

DROP TABLE IF EXISTS `dss_workspace_associate_git`;
CREATE TABLE `dss_workspace_associate_git` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`workspace_id` bigint(20) DEFAULT NULL,
`git_user` varchar(64) DEFAULT NULL COMMENT 'git登录用户名',
`git_password` VARCHAR(255) DEFAULT NULL COMMENT 'git登录密码,用于跳转',
`git_token` varchar(255) COMMENT '用户配置的git token',
`git_url` varchar(255),
`create_time` datetime DEFAULT NULL,
`update_time` datetime DEFAULT NULL,
`create_by` varchar(128) DEFAULT NULL,
`update_by` varchar(128) DEFAULT NULL,
`type` varchar(32) DEFAULT NULL,
`git_user_id` varchar(32) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT='工作空间绑定的git信息';


DROP TABLE IF EXISTS `dss_orchestrator_submit_job_info`;
CREATE TABLE `dss_orchestrator_submit_job_info` (
`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键',
`orchestrator_id` bigint(20) NOT NULL,
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '修改时间',
`instance_name` varchar(128) DEFAULT NULL COMMENT '提交任务的实例',
`status` varchar(128) DEFAULT NULL COMMENT '提交任务状态',
`error_msg` varchar(2048) DEFAULT NULL COMMENT '提交任务异常信息',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT='dss_orchestrator_submit_job_info表';

4 changes: 2 additions & 2 deletions dss-appconn/appconns/dss-datachecker-appconn/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<artifactId>dss</artifactId>
<groupId>com.webank.wedatasphere.dss</groupId>
<version>1.1.0.20-SNAPSHOT</version>
<version>1.5.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down Expand Up @@ -54,7 +54,7 @@
<dependency>
<groupId>com.webank.wedatasphere.dss</groupId>
<artifactId>dss-origin-sso-integration-standard</artifactId>
<version>1.1.0.20-SNAPSHOT</version>
<version>1.5.0-SNAPSHOT</version>
<exclusions>
<exclusion>
<groupId>org.apache.linkis</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ public DataChecker(Properties p, DataCheckerExecutionAction action) {
maxWaitTime = Long.valueOf(p.getProperty(DataChecker.WAIT_TIME, "1")) * 3600 * 1000;
//test over time
// maxWaitTime = Long.valueOf(p.getProperty(DataChecker.WAIT_TIME, "1")) * 120 * 1000;
queryFrequency = Integer.valueOf(p.getProperty(DataChecker.QUERY_FREQUENCY, "30000"));
queryFrequency = Integer.valueOf(p.getProperty(DataChecker.QUERY_FREQUENCY, "60000"));

}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,13 +54,6 @@ public class DataCheckerDao {
private static final String SQL_SOURCE_TYPE_JOB_PARTITION =
"SELECT * FROM DBS d JOIN TBLS t ON t.DB_ID = d.DB_ID JOIN PARTITIONS p ON p.TBL_ID = t.TBL_ID WHERE d.NAME=? AND t.TBL_NAME=? AND p.PART_NAME=?";

private static final String SQL_SOURCE_TYPE_BDP =
"SELECT * FROM desktop_bdapimport WHERE bdap_db_name = ? AND bdap_table_name = ? AND target_partition_name = ? AND status = '1';";

private static final String SQL_SOURCE_TYPE_BDP_WITH_TIME_CONDITION =
"SELECT * FROM desktop_bdapimport WHERE bdap_db_name = ? AND bdap_table_name = ? AND target_partition_name = ? " +
"AND (UNIX_TIMESTAMP() - UNIX_TIMESTAMP(STR_TO_DATE(modify_time, '%Y-%m-%d %H:%i:%s'))) <= ? AND status = '1';";

private static final String SQL_DOPS_CHECK_TABLE =
"SELECT * FROM dops_clean_task_list WHERE db_name = ? AND tb_name = ? AND part_name is null AND task_state NOT IN (10,13) order by order_id desc limit 1";
private static final String SQL_DOPS_CHECK_PARTITION =
Expand All @@ -72,7 +65,6 @@ public class DataCheckerDao {
private static final String MASK_SOURCE_TYPE = "maskdb";

private static DataSource jobDS;
private static DataSource bdpDS;

private static DataSource dopsDS;
private static volatile DataCheckerDao instance;
Expand All @@ -96,13 +88,6 @@ public boolean validateTableStatusFunction(Properties props, Logger log, DataChe
return false;
}
}
if (bdpDS == null) {
bdpDS = DataDruidFactory.getBDPInstance(props, log);
if (bdpDS == null) {
log.warn("Error getting job Druid DataSource instance");
return false;
}
}
boolean systemCheck = Boolean.valueOf(props.getProperty(DataChecker.QUALITIS_SWITCH));
if (systemCheck && dopsDS == null) {
dopsDS = DataDruidFactory.getDopsInstance(props, log);//通过alibaba的druid数据库连接池获取JOB数据库连接
Expand All @@ -122,7 +107,7 @@ public boolean validateTableStatusFunction(Properties props, Logger log, DataChe
}
log.info("(DataChecker info) database table partition info : " + dataCheckerInfo);
long waitTime = Long.valueOf(props.getProperty(DataChecker.WAIT_TIME, "1")) * 3600 * 1000;
int queryFrequency = Integer.valueOf(props.getProperty(DataChecker.QUERY_FREQUENCY, "30000"));
int queryFrequency = Integer.valueOf(props.getProperty(DataChecker.QUERY_FREQUENCY, "60000"));
// String timeScape = props.getProperty(DataChecker.TIME_SCAPE, "NULL");
log.info("(DataChecker info) wait time : " + waitTime);
log.info("(DataChecker info) query frequency : " + queryFrequency);
Expand All @@ -134,13 +119,12 @@ public boolean validateTableStatusFunction(Properties props, Logger log, DataChe
});
QualitisUtil qualitisUtil = new QualitisUtil(props);
try (Connection jobConn = jobDS.getConnection();
Connection bdpConn = bdpDS.getConnection();
Connection dopsConn = dopsDS != null ? dopsDS.getConnection() : null) {
List<Boolean> allCheckRes = dataObjectList
.parallelStream()
.map(proObjectMap -> {
log.info("Begin to Check dataObject:" + proObjectMap.entrySet().toString());
boolean checkRes = getDataCheckResult(proObjectMap, jobConn, bdpConn, dopsConn, props, log,action,qualitisUtil);
boolean checkRes = getDataCheckResult(proObjectMap, jobConn, dopsConn, props, log,action,qualitisUtil);
if (null != action.getExecutionRequestRefContext()) {
if (checkRes) {
action.getExecutionRequestRefContext().appendLog("Database table partition info : " + proObjectMap.get(DataChecker.DATA_OBJECT) + " has arrived");
Expand Down Expand Up @@ -178,7 +162,6 @@ public boolean validateTableStatusFunction(Properties props, Logger log, DataChe

private boolean getDataCheckResult(Map<String, String> proObjectMap,
Connection jobConn,
Connection bdpConn,
Connection dopsConn,
Properties props,
Logger log,
Expand Down Expand Up @@ -231,7 +214,7 @@ private boolean getDataCheckResult(Map<String, String> proObjectMap,
}
log.info("start to check maskis");
proObjectMap.put(DataChecker.SOURCE_TYPE, MASK_SOURCE_TYPE);
normalCheck= (getBdpTotalCount(dataObject, bdpConn, log, props) > 0 || "success".equals(fetchMaskCode(dataObject, log, props).get("maskStatus")));
normalCheck= "success".equals(fetchMaskCode(dataObject, log, props).get("maskStatus"));
if (null != action.getExecutionRequestRefContext()){
action.getExecutionRequestRefContext().appendLog(dataObjectStr+" check maskis end,check result:"+normalCheck);
}
Expand Down Expand Up @@ -316,25 +299,6 @@ private PreparedStatement getJobStatement(Connection conn, CheckDataObject dataO
}
}

/**
* 构造查询maskis的查询
*/
private PreparedStatement getBdpStatement(Connection conn, CheckDataObject dataObject, String timeScape) throws SQLException {
PreparedStatement pstmt = null;
if (timeScape.equals("NULL")) {
pstmt = conn.prepareCall(SQL_SOURCE_TYPE_BDP);
} else {
pstmt = conn.prepareCall(SQL_SOURCE_TYPE_BDP_WITH_TIME_CONDITION);
pstmt.setInt(4, Integer.valueOf(timeScape) * 3600);
}
if (dataObject.getPartitionName() == null) {
dataObject.setPartitionName("");
}
pstmt.setString(1, dataObject.getDbName());
pstmt.setString(2, dataObject.getTableName());
pstmt.setString(3, dataObject.getPartitionName());
return pstmt;
}

/**
* 构造查询dops库的查询
Expand Down Expand Up @@ -414,27 +378,6 @@ private long getJobTotalCount(CheckDataObject dataObject, Connection conn, Logge
}
}

/**
* 查mask db
*/
private long getBdpTotalCount(CheckDataObject dataObject, Connection conn, Logger log, Properties props) {
String timeScape = props.getOrDefault(DataChecker.TIME_SCAPE, "NULL").toString();
log.info("-------------------------------------- search bdp data ");
log.info("-------------------------------------- dataObject: " + dataObject.toString());
try (PreparedStatement pstmt = getBdpStatement(conn, dataObject, timeScape)) {
ResultSet rs = pstmt.executeQuery();
long ret = 0L;
while (rs.next()) {
ret ++;
}
// long ret=rs.last() ? rs.getRow() : 0;
log.info("-------------------------------------- bdp data result:"+ret);
return ret;
} catch (SQLException e) {
log.error("fetch data from bdp error", e);
return 0;
}
}

/**
* - 返回0表示未找到任何记录 ;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@

public class DataDruidFactory {
private static volatile DruidDataSource jobInstance;
private static volatile DruidDataSource bdpInstance;

private static volatile DruidDataSource dopsInstance;

private static volatile DruidDataSource msgInstance;

public static DruidDataSource getJobInstance(Properties props, Logger log) {
Expand All @@ -42,6 +44,7 @@ public static DruidDataSource getJobInstance(Properties props, Logger log) {
}
return jobInstance;
}

public static DruidDataSource getBDPInstance(Properties props, Logger log) {
if (bdpInstance == null ) {
synchronized (DataDruidFactory.class) {
Expand Down
5 changes: 2 additions & 3 deletions dss-appconn/appconns/dss-dolphinscheduler-appconn/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<artifactId>dss</artifactId>
<groupId>com.webank.wedatasphere.dss</groupId>
<version>1.1.0.20-SNAPSHOT</version>
<version>1.5.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand All @@ -18,7 +18,6 @@
<groupId>com.webank.wedatasphere.dss</groupId>
<artifactId>dss-scheduler-appconn</artifactId>
<version>${dss.version}</version>

<scope>provided</scope>
</dependency>

Expand Down Expand Up @@ -81,7 +80,7 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>28.2-android</version>
<version>33.1.0-jre</version>
</dependency>


Expand Down
Loading
Loading