Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test/tpch/add warmup #16

Open
wants to merge 33 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
dc387b2
ci: check update sesssion_py (#531)
aqni Dec 31, 2024
8483f2d
ci(db-ce): spilt db-ce into 5 jobs (#549)
aqni Jan 2, 2025
4760214
add warmup IT
aqni Dec 25, 2024
f802a74
format
aqni Dec 31, 2024
d75ecf5
refactor dbRunner
aqni Jan 2, 2025
e70f8a6
tpch use 2 zk instances
aqni Jan 2, 2025
9850048
update user manual
zhuyuqing Jan 3, 2025
676a318
fix
aqni Jan 3, 2025
1aa0fa9
Merge branch 'main' into test/tpch/add-warmup
aqni Jan 3, 2025
b95f903
Update action.yml
aqni Jan 3, 2025
b00f798
Update action.yml
aqni Jan 3, 2025
ac9ce31
feat(transform): some new features and improve (#542)
shinyano Jan 3, 2025
12a4c1f
Merge branch 'refs/heads/main' into test/tpch/add-warmup
aqni Jan 3, 2025
ec5d13a
refactor(core): remove flag (#534)
shinyano Jan 4, 2025
ec9619e
chore: remove env `SERVICE_DIR_xxx` and restricted version of pemjax …
aqni Jan 4, 2025
5d365af
Merge branch 'main' into test/tpch/add-warmup
zhuyuqing Jan 4, 2025
963759a
feat(core): add option to remove `RemoveNullColumn` operator (#536)
aqni Jan 5, 2025
a85a696
Merge branch 'main' into test/tpch/add-warmup
aqni Jan 6, 2025
0f05dbc
Update q19.sql
aqni Jan 6, 2025
435fea0
Update q19_a.sql
aqni Jan 6, 2025
1a6b4a1
Update standard-test-suite.yml
aqni Jan 6, 2025
5d6d8d2
Update tpc-h.yml
aqni Jan 6, 2025
85ce499
Update action.yml
aqni Jan 6, 2025
80e7dce
fix(core): fix distinct (#552)
jzl18thu Jan 6, 2025
edb3c89
Revert "Update q19_a.sql"
aqni Jan 6, 2025
0df65a0
Revert "Update q19.sql"
aqni Jan 6, 2025
9e617cd
use `19a` rather than `19`
aqni Jan 6, 2025
fe1caeb
add 19_a.csv
aqni Jan 7, 2025
38653e5
Merge branch 'main' into test/tpch/add-warmup
aqni Jan 7, 2025
5b3da2d
fix
aqni Jan 7, 2025
07e59ae
remove influxdb query limit
aqni Jan 9, 2025
8d0822d
Revert "remove influxdb query limit"
aqni Jan 9, 2025
7644d7c
limit influxdb query concurrent as 2
aqni Jan 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 0 additions & 63 deletions .github/actions/capacityExpansionUnionTest/action.yml

This file was deleted.

334 changes: 145 additions & 189 deletions .github/actions/confWriter/action.yml

Large diffs are not rendered by default.

111 changes: 0 additions & 111 deletions .github/actions/dbConfWriter/action.yml

This file was deleted.

71 changes: 0 additions & 71 deletions .github/actions/dbRunner/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,6 @@ inputs:
runs:
using: "composite" # Mandatory parameter
steps:
- id: project
uses: ./.github/actions/project

- name: Modify IGinX Config
uses: ./.github/actions/edit
with:
paths: conf/config.properties
statements: s/^storageEngineList=/#storageEngineList=/g

- if: inputs.DB-name=='InfluxDB' || inputs.DB-name=='Mix-IoTDB12-InfluxDB'
name: Setup DB
uses: ./.github/actions/setup/influxdb
Expand Down Expand Up @@ -86,19 +77,6 @@ runs:
exit 1
fi

- if: inputs.DB-name=='Mix-IoTDB12-InfluxDB'
name: Modify IGinX Config
uses: ./.github/actions/edit
with:
paths: conf/config.properties
statements: s/storageEngineList/#storageEngineList/g

- if: inputs.DB-name=='Mix-IoTDB12-InfluxDB'
name: Run DB
shell: bash
run: |
echo "storageEngineList=127.0.0.1#6667#iotdb12#username=root#password=root#sessionPoolSize=50#has_data=false#is_read_only=false,127.0.0.1#8086#influxdb#url=http://localhost:8086/#token=testToken#organization=testOrg#has_data=false" >> conf/config.properties

- if: inputs.DB-name=='MongoDB'
name: Setup DB
uses: ./.github/actions/setup/mongodb
Expand All @@ -112,13 +90,6 @@ runs:
start: true
ports: ${{ inputs.MongoDB-ports }}

- if: inputs.DB-name=='MongoDB'
name: Modify IGinX Config
uses: ./.github/actions/edit
with:
paths: conf/config.properties
statements: s/^#storageEngineList=127.0.0.1#27017/storageEngineList=127.0.0.1#27017/g

- if: inputs.DB-name=='Redis'
name: Setup DB
uses: ./.github/actions/setup/redis
Expand All @@ -132,13 +103,6 @@ runs:
start: true
ports: ${{ inputs.Redis-ports }}

- if: inputs.DB-name=='Redis'
name: Modify IGinX Config
uses: ./.github/actions/edit
with:
paths: conf/config.properties
statements: s/^#storageEngineList=127.0.0.1#6379/storageEngineList=127.0.0.1#6379/g

- if: inputs.DB-name=='PostgreSQL'
name: Setup DB
uses: ./.github/actions/setup/postgresql
Expand All @@ -152,13 +116,6 @@ runs:
start: true
ports: ${{ inputs.PostgreSQL-ports }}

- if: inputs.DB-name=='PostgreSQL'
name: Modify IGinX Config
uses: ./.github/actions/edit
with:
paths: conf/config.properties
statements: s/^#storageEngineList=127.0.0.1#5432#relational#engine=postgresql/storageEngineList=127.0.0.1#5432#relational#engine=postgresql/g

- if: inputs.DB-name=='MySQL'
name: Setup DB
uses: ./.github/actions/setup/mysql
Expand All @@ -168,31 +125,3 @@ runs:
uses: ./.github/actions/service/mysql
with:
ports: ${{ inputs.MySQL-ports }}

- if: inputs.DB-name == 'MySQL'
name: Modify IGinX Config
uses: ./.github/actions/edit
with:
paths: conf/config.properties
statements: s|^#storageEngineList=127.0.0.1#3306#relational#engine=mysql#username=root#password=mysql#has_data=false#meta_properties_path=your-meta-properties-path|storageEngineList=127.0.0.1#3306#relational#engine=mysql#username=root#has_data=false|g

- if: inputs.DB-name=='FileSystem'
name: Run DB
shell: bash
run: |
cp -f "${GITHUB_WORKSPACE}/conf/config.properties" "${GITHUB_WORKSPACE}/conf/config.properties.bak"

- if: inputs.DB-name == 'FileSystem'
name: Modify IGinX Config
uses: ./.github/actions/edit
with:
paths: conf/config.properties
statements: |
s/^#storageEngineList=127.0.0.1#[0-9]*#filesystem/storageEngineList=127.0.0.1#6668#filesystem/g
s#dir=data#dir=${{ steps.project.outputs.workspace }}/test/iginx_mn#g
s#dummy_dir=dummy#dummy_dir=${{ steps.project.outputs.workspace }}/test/mn#g
s/data.config.write.buffer.size=104857600/data.config.write.buffer.size=1048576/g
s/dummy.struct=LegacyFilesystem/dummy.struct=LegacyParquet/g
s/chunk_size_in_bytes=1048576/chunk_size_in_bytes=8/g
s/write.buffer.size=104857600/write.buffer.size=1048576/g
s/client.connectPool.maxTotal=100/client.connectPool.maxTotal=2/g
18 changes: 0 additions & 18 deletions .github/actions/dbWriter/action.yml

This file was deleted.

20 changes: 17 additions & 3 deletions .github/actions/dependence/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,19 @@ runs:
colima start --network-address --arch arm64 --vm-type=qemu
docker --version

- if: inputs.scope=='all'
name: Write Python dependencies to requirements.txt
shell: bash
run: |
echo "pandas" >> requirements.txt
echo "numpy" >> requirements.txt
echo "pemjax" >> requirements.txt
echo "thrift" >> requirements.txt
echo "fastparquet" >> requirements.txt
echo "tqdm" >> requirements.txt
echo "requests" >> requirements.txt
echo "torch" >> requirements.txt

- if: inputs.scope=='all'
name: Set up Python ${{ inputs.python-version }}
uses: actions/setup-python@v5
Expand All @@ -57,9 +70,7 @@ runs:
- if: inputs.scope=='all'
name: Install Python dependencies
shell: bash
run: |
python -m pip install --upgrade pip
pip install pandas numpy pemjax==0.1.0 thrift fastparquet tqdm requests torch
run: pip install -r requirements.txt

- name: Set up JDK ${{ inputs.java }}
uses: actions/setup-java@v4
Expand All @@ -76,3 +87,6 @@ runs:
shell: bash
run: |
echo "VERSION=${{ steps.project.outputs.version }}" >> $GITHUB_ENV
PROJECT_ROOT=${{ steps.project.outputs.workspace }}
echo "PROJECT_ROOT=${PROJECT_ROOT}" >> $GITHUB_ENV
echo "SERVICE_DIR=${PROJECT_ROOT}/.github/actions/service" >> $GITHUB_ENV
Loading
Loading