Skip to content

Commit bfd28df

Browse files
bio-borisBorisXiangs18
authored
v2.3.0 release (develop -> master) (#129)
* Update controller.py * Create pr_build.yml * Create release-main.yml * Create manual-build.yml * Update db.py * Update controller.py * Update db.py * Update pr_build.yml * Update controller.py * Update controller.py * add GHA test.yml, codeql, and dependabot.yml (#136) * mongo7 upgrade * upgrade pymongo to 4.7.2 * update python to 3.9.19 * update Makefile and run_tests.sh scripts * update submodules * fix gradle not found * start nms service with docker-compose * update nms submodule to the latest * install docker-compose in GHA * switch catalog mongo7 port to 27018 * fix mongo port * fix mongo port mapping * set admin_users * check admin users * shut down NMS service properly * fix nms curl * update mock_auth service * update the commit * Update test.cfg.example * Update test.cfg.example * Update core_registration_test.py * update message * test method_spec_admin_users * test simple_good_repo old commit * test register hack * remove jars nms submodules * update Dockerfile * remove kbapi_common submodule * fix deprecated mongo auth * remove all submodules in catalog * create MongoClient through passing in params * add a comment for mock_auth server.py * simplify code * adjust params in MongoClient * add debug log messages in list_user_favorites func * try reinitiate mongoclient * update __init__ in db.py file * fix typo * add check_db_schema * add boolean flag for schema check * collection direct call * remove debugging message * display catalog version * fix failed unit test in startup_test.py * trigger lazy load collection * clean up files * remove comments from Makefile * validate Install Docker Compose step in ci * use docker compose * map NMS to the local port 27018 * Add a clearer comment for the environment variable in test.yml * move env to jobs level * add more comments in server.py * add more comments in db.py * remove check_db_schema related functions * revert back changes * move self._db_schema_checked flag outside function * refactor _create_indexes function * encapsulate create_indexes in the DBIndexes class * add comment at the top of server.py file * refactor db.py file * add flag for mongo client * update wrapper function * add initialize_mongo_client function manually * add _close_mongo_client function * try run tests without a lock * revert self.lock changes * Recreate the database and collection handles after reinitializing the client * revert back tests * remove decorator related code * add comment why need to manually close mongoclient * update comments * refactor the db.py and add comments for docker-compose file * add a comment for _ensure_mongo_connection function * Clarify the comments in the db.py and docker-compose.yml files * remove NarrativeMethodStore section and update the test instructions in the test/test.cfg.example file * expound on the explanation * fix c&p error * fix typos and clarify comments * make sure NMS is clean after each test * remove uwsgi from dockerfile * add Pipenv and Pipenv.lock files * upload Pipfiles * add missing lib pyyaml * move semantic_version to packages * fix mongo_authMechanism name * add retrywrites param * move release notes from README.md to RELEASE_NOTES.md * removed mongodb-retrywrites from test.cfg.example * add tests for retryWrites * fix format * add print meesage to verify retryWrites setting * remove print * fix typo in README.md * uncomment trivy-scan --------- Co-authored-by: Boris <[email protected]> Co-authored-by: Sijie Xiang <[email protected]>
1 parent 6873c95 commit bfd28df

31 files changed

+1568
-299
lines changed

‎.github/codeql.yml

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
name: "Code scanning - action"
2+
3+
on:
4+
push:
5+
pull_request:
6+
schedule:
7+
- cron: '0 19 * * 0'
8+
9+
jobs:
10+
CodeQL-Build:
11+
12+
# CodeQL runs on ubuntu-latest and windows-latest
13+
runs-on: ubuntu-latest
14+
15+
steps:
16+
- name: Checkout repository
17+
uses: actions/checkout@v4
18+
with:
19+
# We must fetch at least the immediate parents so that if this is
20+
# a pull request then we can checkout the head.
21+
fetch-depth: 2
22+
23+
# If this run was triggered by a pull request event, then checkout
24+
# the head of the pull request instead of the merge commit.
25+
- run: git checkout HEAD^2
26+
if: ${{ github.event_name == 'pull_request' }}
27+
28+
# Initializes the CodeQL tools for scanning.
29+
- name: Initialize CodeQL
30+
uses: github/codeql-action/init@v3
31+
# Override language selection by uncommenting this and choosing your languages
32+
# with:
33+
# languages: go, javascript, csharp, python, cpp, java
34+
35+
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
36+
# If this step fails, then you should remove it and run the build manually (see below)
37+
- name: Autobuild
38+
uses: github/codeql-action/autobuild@v3
39+
40+
# ℹ️ Command-line programs to run using the OS shell.
41+
# 📚 https://git.io/JvXDl
42+
43+
# ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
44+
# and modify them (or add more) to build your code if your project
45+
# uses a compiled language
46+
47+
#- run: |
48+
# make bootstrap
49+
# make release
50+
51+
- name: Perform CodeQL Analysis
52+
uses: github/codeql-action/analyze@v3

‎.github/dependabot.yml

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: docker
4+
directory: "/"
5+
schedule:
6+
interval: weekly
7+
time: '11:00'
8+
open-pull-requests-limit: 10
9+
- package-ecosystem: pip
10+
directory: "/"
11+
schedule:
12+
interval: weekly
13+
time: '11:00'
14+
open-pull-requests-limit: 10

‎.github/workflows/test.yml

+90
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
name: KBase Catalog test
2+
3+
on:
4+
pull_request:
5+
types:
6+
- opened
7+
- reopened
8+
- synchronize
9+
- ready_for_review
10+
push:
11+
# run workflow when merging to main or develop
12+
branches:
13+
- main
14+
- master
15+
- develop
16+
17+
jobs:
18+
catalog_tests:
19+
runs-on: ubuntu-latest
20+
strategy:
21+
fail-fast: false
22+
matrix:
23+
include:
24+
- python-version: '3.9.19'
25+
mongo-version: '3.6'
26+
- python-version: '3.9.19'
27+
mongo-version: '7.0.4'
28+
services:
29+
mongo:
30+
image: mongo:${{matrix.mongo-version}}
31+
ports:
32+
- 27017:27017
33+
options: --name mongo${{matrix.mongo-version}}
34+
35+
env:
36+
KBASE_CI_TOKEN: ${{ secrets.KBASE_CI_TOKEN }}
37+
# This env var is also used in the catalog test docker compose file for starting NMS
38+
ADMIN_USER: ${{ secrets.KBASE_BOT_USER_CI }}
39+
40+
steps:
41+
- name: Repo checkout
42+
uses: actions/checkout@v4
43+
44+
- name: Setup Python
45+
uses: actions/setup-python@v5
46+
with:
47+
python-version: ${{matrix.python-version}}
48+
49+
- name: Install dependencies and set up test config
50+
shell: bash
51+
52+
run: |
53+
54+
# test mongo connection
55+
curl http://localhost:27017
56+
returncode=$?
57+
if [ $returncode != 0 ]; then exit $returncode; fi
58+
59+
# set HOMEDIR
60+
export HOMEDIR=`pwd`
61+
62+
# move to parent dir to install binaries etc
63+
cd ..
64+
65+
# setup kb-sdk
66+
mkdir -p $(pwd)/bin
67+
docker run ghcr.io/kbase/kb_sdk_patch-develop:br-0.0.4 genscript > $(pwd)/bin/kb-sdk
68+
chmod 755 $(pwd)/bin/kb-sdk
69+
export PATH=$(pwd)/bin:$PATH
70+
71+
# install catalog dependencies
72+
cd $HOMEDIR
73+
python -m pip install --upgrade pip
74+
pip install pipenv
75+
pipenv sync --system --dev
76+
77+
# setup test config
78+
cp -n test/test.cfg.example test/test.cfg
79+
sed -i "s#^nms-admin-token.*#nms-admin-token=$KBASE_CI_TOKEN#" test/test.cfg
80+
sed -i "s#^method-spec-admin-users.*#method-spec-admin-users=$ADMIN_USER#" test/test.cfg
81+
82+
- name: Run tests
83+
shell: bash
84+
run: make test
85+
86+
- name: Upload coverage to Codecov
87+
uses: codecov/codecov-action@v4
88+
with:
89+
token: ${{ secrets.CODECOV_TOKEN }}
90+
fail_ci_if_error: true

‎.gitmodules

-12
This file was deleted.

‎.travis.yml

-40
This file was deleted.

‎DEPENDENCIES

-4
This file was deleted.

‎Dockerfile

+24-6
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,43 @@
1-
FROM kbase/sdkbase2:python AS build
1+
FROM python:3.9.19 AS build
22

3+
# The rsync installation is required for the Makefile
4+
RUN apt-get update && apt-get install -y rsync
5+
RUN mkdir -p /kb/deployment/lib/biokbase
36

47
COPY . /tmp/catalog
58
RUN cd /tmp/catalog && make deploy-service deploy-server-control-scripts
69

7-
FROM kbase/sdkbase2:python
10+
FROM python:3.9.19
811
# These ARGs values are passed in via the docker build command
912
ARG BUILD_DATE
1013
ARG VCS_REF
1114
ARG BRANCH
1215

16+
RUN apt-get update && apt-get install -y wget
17+
18+
# install dockerize
19+
WORKDIR /opt
20+
RUN wget -q https://github.com/kbase/dockerize/raw/master/dockerize-linux-amd64-v0.6.1.tar.gz \
21+
&& tar xvzf dockerize-linux-amd64-v0.6.1.tar.gz \
22+
&& rm dockerize-linux-amd64-v0.6.1.tar.gz
23+
RUN mkdir -p /kb/deployment/bin/
24+
RUN ln -s /opt/dockerize /kb/deployment/bin/dockerize
25+
1326
ENV KB_DEPLOYMENT_CONFIG "/kb/deployment/conf/deploy.cfg"
1427

1528
COPY --from=build /kb/deployment/lib/biokbase /kb/deployment/lib/biokbase
1629
COPY --from=build /kb/deployment/services /kb/deployment/services
1730
COPY --from=build /tmp/catalog/deployment/conf /kb/deployment/conf
1831

19-
SHELL ["/bin/bash", "-c"]
20-
COPY requirements.txt requirements.txt
21-
RUN source activate root && \
22-
pip install -r requirements.txt
32+
WORKDIR /tmp/catalog
33+
34+
# install pipenv
35+
RUN pip install --upgrade pip && \
36+
pip install pipenv
37+
38+
# install deps
39+
COPY Pipfile* ./
40+
RUN pipenv sync --system
2341

2442
LABEL org.label-schema.build-date=$BUILD_DATE \
2543
org.label-schema.vcs-url="https://github.com/kbase/catalog.git" \

‎Makefile

-9
Original file line numberDiff line numberDiff line change
@@ -27,11 +27,6 @@ LIB_DIR = lib
2727

2828
PATH := kb_sdk/bin:$(PATH)
2929

30-
default: init
31-
32-
init:
33-
git submodule init
34-
git submodule update
3530

3631
compile-kb-module:
3732
kb-sdk compile $(SPEC_FILE) \
@@ -102,10 +97,6 @@ setup-tests:
10297
mkdir -p $(TESTLIB)/biokbase
10398
mkdir -p $(TESTDIR)/nms
10499
rsync -av lib/biokbase/* $(TESTLIB)/biokbase/. --exclude *.bak-*
105-
rsync -av kbapi_common/lib/biokbase/* $(TESTLIB)/biokbase/.
106-
cd narrative_method_store; make; make build-classpath-list;
107-
# rsync -av narrative_method_store/lib/biokbase/* $(TESTLIB)/biokbase/.
108-
109100

110101

111102
test: setup-tests

‎Pipfile

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
[[source]]
2+
url = "https://pypi.org/simple"
3+
verify_ssl = true
4+
name = "pypi"
5+
6+
[packages]
7+
docker = "==7.1.0"
8+
jsonrpcbase = "==0.2.0"
9+
pymongo = "==4.7.2"
10+
pyyaml = "==6.0.2"
11+
semantic-version = "==2.10.0"
12+
uwsgi = "==2.0.22"
13+
14+
[dev-packages]
15+
coverage = "==7.6.1"
16+
17+
[requires]
18+
python_version = "3.9.19"

0 commit comments

Comments
 (0)