Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: check alls green #21

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 7 additions & 14 deletions .github/actions/confWriter/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,23 +79,16 @@ runs:
id: find_python_cmd
shell: bash -el {0}
run: |
if [[ "${{ inputs.use-free-thread-python }}" == "true" ]]; then
if [ "$RUNNER_OS" == "Windows" ]; then
pythonPath=$(echo "$FT_PYTHON_PATH" | sed 's/\\/\//g')
pythonCMD="${pythonPath}/python3.13t.exe"
echo "TORCH_SUPPORTED=false" >> $GITHUB_ENV
else
pythonCMD='python3.13t'
echo "TORCH_SUPPORTED=false" >> $GITHUB_ENV
fi
if [ "${{ inputs.use-free-thread-python }}" == "true" ]; then
echo "TORCH_SUPPORTED=false" >> $GITHUB_ENV
fi
if [ "$RUNNER_OS" == "Windows" ]; then
pythonCMD=python
else
if [ "$RUNNER_OS" == "Windows" ]; then
pythonCMD=python
else
pythonCMD=python3
fi
pythonCMD=python3
fi
# have to set env to make it work in ut tests
${pythonCMD} -VV
echo "pythonCMD=${pythonCMD}" >> $GITHUB_ENV
echo "pythonCMD=${pythonCMD}" >> $GITHUB_OUTPUT
echo ${pythonCMD}
Expand Down
4 changes: 3 additions & 1 deletion .github/actions/dependence/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,9 @@ runs:

- if: inputs.free-thread-python-required=='true'
name: Install free-thread python
uses: ./.github/actions/service/freeThreadPython
uses: ./.github/actions/service/freeThreadConda
with:
iginx-conda-env: ${{ inputs.iginx-conda-env }}

- name: Set up JDK ${{ inputs.java }}
uses: actions/setup-java@v4
Expand Down
27 changes: 20 additions & 7 deletions .github/actions/iginxRunner/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,17 @@ inputs:
description: "to test UDF path detection"
required: false
default: "false"
iginx-conda-flag:
# this step is only needed when using python3.13 by conda because latest conda supports py3.12,
# and we use conda-forge to support 3.13. Thus, the required env would not be activated automatically
# even when we use login mode of bash
description: "whether to manually activate conda env before start IGinX"
required: false
default: "false"
iginx-conda-env:
description: "required conda env name"
required: false
default: "false"

runs:
using: "composite" # Mandatory parameter
Expand Down Expand Up @@ -36,16 +47,18 @@ runs:
name: Start IGinX
shell: bash -el {0}
run: |
which python
if [ "${{ inputs.iginx-conda-flag }}" == "true" ]; then
conda activate ${{ inputs.iginx-conda-env }}
fi
if [ "$RUNNER_OS" == "Linux" ]; then
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx.sh"
"${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx.sh" 6888 7888
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx.sh"
"${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx.sh" 6888 7888
elif [ "$RUNNER_OS" == "Windows" ]; then
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx_windows.sh"
"${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx_windows.sh" 6888 7888
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx_windows.sh"
"${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx_windows.sh" 6888 7888
elif [ "$RUNNER_OS" == "macOS" ]; then
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx_macos.sh"
"${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx_macos.sh" 6888 7888
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx_macos.sh"
"${GITHUB_WORKSPACE}/.github/scripts/iginx/iginx_macos.sh" 6888 7888
fi

- if: inputs.if-test-udf=='false' && inputs.if-stop=='true'
Expand Down
36 changes: 36 additions & 0 deletions .github/actions/service/freeThreadConda/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: "Install_Free-thread_Python in Conda"
description: "Install Free-thread Python in Conda"
inputs:
iginx-conda-env:
description: "conda env name"
required: false

runs:
using: "composite"
steps:
# this step creates env test(default python version) which will not be used
- uses: conda-incubator/setup-miniconda@v3
with:
channels: conda-forge
channel-priority: flexible
conda-remove-defaults: "true"
run-post: "false"

# currently we can only install free-thread python 3.13 by conda-forge. latest miniconda only supports 3.12
# thus in action steps ahead, remember to 'conda activate $ENV_NAME' at start if conda should be used
- name: Check channels & install python3.13
shell: bash -el {0}
run: |
conda config --show channels
conda create -n ${{ inputs.iginx-conda-env }} python=3.13 python-freethreading -c conda-forge -q
conda activate ${{ inputs.iginx-conda-env }}
python -VV
python -c "import sys;print(sys._is_gil_enabled())"
curl -L -O https://github.com/IGinX-THU/IGinX-resources/raw/refs/heads/main/resources/python/pandas-3.0.0.dev0+1654.g32a97a969a-cp313-cp313t-win_amd64.whl
ls -l
if [ "$RUNNER_OS" == "Windows" ]; then
python -m pip install numpy thrift pemjax
python -m pip install pandas*.whl
else
python -m pip install pandas==2.2.3 numpy thrift pemjax
fi
115 changes: 0 additions & 115 deletions .github/actions/service/freeThreadPython/action.yml

This file was deleted.

3 changes: 2 additions & 1 deletion .github/scripts/iginx/iginx.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ sed -i "s/restPort=[0-9]\+/restPort=$2/g" core/target/iginx-core-*/conf/config.p

sh -c "chmod +x core/target/iginx-core-*/sbin/start_iginx.sh"

sh -c "nohup core/target/iginx-core-*/sbin/start_iginx.sh > iginx-$1.log 2>&1 &"
python3 -VV
bash -c "nohup core/target/iginx-core-*/sbin/start_iginx.sh > iginx-$1.log 2>&1 &"

sh -c "sleep 3"

Expand Down
3 changes: 2 additions & 1 deletion .github/scripts/iginx/iginx_macos.sh
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ echo "JAVA_HOME is set to $JAVA_HOME"

sh -c "chmod +x core/target/iginx-core-*/sbin/start_iginx.sh"

sh -c "nohup core/target/iginx-core-*/sbin/start_iginx.sh > iginx-$1.log 2>&1 &"
python3 -VV
bash -c "nohup core/target/iginx-core-*/sbin/start_iginx.sh > iginx-$1.log 2>&1 &"

sh -c "sleep 3"

Expand Down
2 changes: 1 addition & 1 deletion .github/scripts/test/test_union.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#


python -VV
pwd

cd ..
Expand Down
12 changes: 11 additions & 1 deletion .github/workflows/free-thread-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ jobs:
metadata: ${{ fromJSON(inputs.metadata-matrix) }}
DB-name: ${{ fromJSON(inputs.db-matrix) }}
runs-on: ${{ matrix.os }}
env:
# only used in this workflow file
IGINX_CONDA_FLAG: "true"
IGINX_CONDA_ENV: "iginxEnv"
steps:
- uses: actions/checkout@v4
# use free-threading python instead of stable version
Expand All @@ -44,6 +48,8 @@ jobs:
java: ${{ matrix.java }}
free-thread-python-required: "true"
scope: "only-java"
iginx-conda-flag: ${{ env.IGINX_CONDA_FLAG }}
iginx-conda-env: ${{ env.IGINX_CONDA_ENV }}

- name: Run Metadata
uses: ./.github/actions/metadataRunner
Expand All @@ -70,13 +76,17 @@ jobs:

- name: Start IGinX
uses: ./.github/actions/iginxRunner
with:
iginx-conda-flag: ${{ env.IGINX_CONDA_FLAG }}
iginx-conda-env: ${{ env.IGINX_CONDA_ENV }}

- name: TestController IT
if: always()
shell: bash
shell: bash -el {0}
env:
METADATA_STORAGE: ${{ matrix.metadata }}
run: |
conda activate ${{ env.IGINX_CONDA_ENV }}
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/test/test_union.sh"
mvn test -q -Dtest=Controller -DfailIfNoTests=false -P-format

Expand Down
20 changes: 0 additions & 20 deletions .github/workflows/lint-pr.yml

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
name: Format
name: lints

on: [pull_request]
on:
pull_request:

permissions:
pull-requests: read

jobs:
formatting:
Expand Down Expand Up @@ -36,3 +40,25 @@ jobs:
- name: check py-session difference
working-directory: session_py
run: git diff --minimal --exit-code

pull-request-name:
name: Validate PR title
runs-on: ubuntu-latest
steps:
- uses: amannn/action-semantic-pull-request@v5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

analyze-required-check:
if: always()
needs: # List of jobs that must pass. Add more jobs if needed.
- formatting
- license
- py-session
- pull-request-name
runs-on: ubuntu-latest
steps:
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@release/v1
with:
jobs: ${{ toJSON(needs) }}
Loading
Loading