Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
170 commits
Select commit Hold shift + click to select a range
44cd4a6
Merge pull request #2 from repotrial/master
AndiMajore Apr 25, 2024
caa7b7e
made dev mongo express port variable to circumvent port in use issues
AndiMajore Apr 25, 2024
7758c36
added build to launch script
AndiMajore May 2, 2024
5694137
Add Download Flag & is_reviewed (#5)
ahoewi May 3, 2024
3f854e5
Update docker-compose.dev.yml
AndiMajore May 5, 2024
455bbe9
Update docker-compose.yml
AndiMajore May 5, 2024
1065f00
fixed repotrial importer
AndiMajore Jun 5, 2024
81ca1f2
merged
AndiMajore Jun 5, 2024
ea1c7b1
removed static file download from Download OFF build script
AndiMajore Jun 6, 2024
d03f50c
Master (#7)
AndiMajore Jun 6, 2024
fd9f414
Master (#9)
AndiMajore Jun 13, 2024
550c636
added additional launch flags for development
AndiMajore Jun 14, 2024
8bf9072
Merge branch 'dev' of github.com:repotrial/nedrexdb_v2d into dev
AndiMajore Jun 14, 2024
1f5abdf
adding additional development options; fixed numpy version
AndiMajore Jun 17, 2024
855e274
Merge branch 'master' into dev
AndiMajore Jun 28, 2024
f90b8cc
merge dev branch into master (#10) (#11)
AndiMajore Jun 28, 2024
2d54f33
updated disgenet path to import from file; fixing issues with uniprot…
AndiMajore Aug 19, 2024
2c8cbad
Bump setuptools from 69.5.1 to 70.0.0 (#12)
dependabot[bot] Aug 19, 2024
879f83a
small security patch by removing dev dependencies from docker image
AndiMajore Aug 19, 2024
43f2534
merge dev branch into master (#10) (#13)
AndiMajore Aug 20, 2024
581d060
Security and small data update (#14)
AndiMajore Aug 20, 2024
091d005
small sec update
AndiMajore Aug 23, 2024
203a0a1
created neo4j docker image with apoc plugin
AndiMajore Aug 23, 2024
827db37
Merge branch 'dev' of github.com:repotrial/nedrexdb_v2d into dev
AndiMajore Aug 23, 2024
3a3c7af
updated databases; using neo4j with apoc
AndiMajore Aug 23, 2024
e1303ef
Merge branch 'master' into dev
AndiMajore Aug 25, 2024
4e080b7
Many updates regardung security and interoperability (#15)
AndiMajore Aug 25, 2024
d39d1cd
Update docker-compose.yml (#16)
AndiMajore Aug 25, 2024
cfce9cd
Merge with dev (#17)
AndiMajore Aug 25, 2024
73519f6
Master (#18)
AndiMajore Aug 25, 2024
1a449f5
Merge all dev into master (#19)
AndiMajore Aug 25, 2024
da2f377
replaced disgenet download link
AndiMajore Sep 9, 2024
ae497d2
Merge branch 'dev'
AndiMajore Sep 9, 2024
64ceb12
fixed unpacking of disgenet archive
AndiMajore Sep 11, 2024
ccd13b3
fixed disgenet unpack error
AndiMajore Sep 11, 2024
ca6baa4
fixed disgenet unpack error (#20)
AndiMajore Sep 11, 2024
987c4fc
fixed disgenet.zip file
AndiMajore Sep 11, 2024
353368c
setting open neo4j db access
AndiMajore Sep 17, 2024
605d5c0
fixed disgenet link
AndiMajore Sep 17, 2024
cfb3299
new neo4j base image
AndiMajore Sep 18, 2024
8b13461
udpated for neo4j behavior
AndiMajore Sep 20, 2024
9c93e52
driggering rebuild with new neo4j
AndiMajore Sep 20, 2024
db647c2
fixing missing clinvar edges
AndiMajore Sep 20, 2024
a74a5c2
fixing port opening for open neo4j version
AndiMajore Sep 23, 2024
6fdd4a6
Merging apoc neo4j setup and neo4j endpoint and interface for open ve…
AndiMajore Sep 23, 2024
f940617
Metadata Update (#22)
ahoewi Oct 7, 2024
fd9db1a
fixing set_metadata
AndiMajore Oct 9, 2024
cc7042b
adding fallback to version creation
AndiMajore Oct 9, 2024
e7203ed
adding fallback to timestamp setting
AndiMajore Oct 9, 2024
7a75abc
adding fallback to timestamp setting
AndiMajore Oct 9, 2024
aa3439b
again fixing disgenet
AndiMajore Oct 9, 2024
632676d
metadata catc error etc (#23)
ahoewi Oct 22, 2024
dab1f91
updated read_only environment variable for neo4j db in db mode
AndiMajore Nov 18, 2024
1288579
Merge branch 'dev' of github.com:repotrial/nedrexdb_v2d into dev
AndiMajore Nov 18, 2024
51d6add
Merge branch 'master' into dev
AndiMajore Nov 19, 2024
00afc3d
fixing metadata create issues
AndiMajore Nov 21, 2024
873d8f2
fixed Dockerfile
AndiMajore Nov 21, 2024
7ef4a0c
added version update exception list to remove omim from open metadata
AndiMajore Nov 22, 2024
b85387f
added new env param to dev docker-compose
AndiMajore Nov 25, 2024
7b00f0d
limited output of clinvar database parsing warning
AndiMajore Nov 25, 2024
74c8c5a
removing tqdm from downloader to not flood the docker logs
AndiMajore Nov 26, 2024
dc05329
silencing wget from downloader to not flood the docker logs
AndiMajore Nov 26, 2024
4825f7c
Merge branch 'dev' of github.com:repotrial/nedrexdb_v2d into dev
AndiMajore Nov 26, 2024
c0c2f19
silencing wget from cloud downloaders
AndiMajore Nov 26, 2024
44fbb86
Handle all cases of download flag
Nov 26, 2024
d55987d
fixed print for version update flag
AndiMajore Nov 26, 2024
746d6d9
adding download file verification functions to combat random file cor…
AndiMajore Nov 27, 2024
ed69186
allowing ftp protocoll in URL validation check
AndiMajore Nov 27, 2024
846eec1
renamed url validation function
AndiMajore Nov 28, 2024
428e201
updating metadata collection and verification
AndiMajore Nov 29, 2024
459b626
updating profiling function again
AndiMajore Nov 29, 2024
f158a3b
Merge branch 'repotrial:dev' into dev
ahoewi Dec 2, 2024
d9bf31a
metadata fix for no download
Dec 2, 2024
ec533fa
fixed docker build
AndiMajore Dec 16, 2024
5107c7e
final fix of metadata issues: utilize live db
Dec 18, 2024
d7c91e2
Merge branch 'repotrial:dev' into dev
ahoewi Dec 18, 2024
b21cf99
minimal config update
ahoewi Dec 30, 2024
88eda8c
deep_merge config option
ahoewi Jan 3, 2025
ccb392b
finally resolved metadata issues. (#27)
ahoewi Jan 7, 2025
ad63f78
improving log output for downloaders
AndiMajore Jan 7, 2025
8699185
fixing urllib3 version to fix download issue
AndiMajore Jan 7, 2025
ce8ebe4
replacing urllib3 version of pypi with conda
AndiMajore Jan 7, 2025
28918ad
updated to latest neo4j build; added apoc and genai plugins
AndiMajore Jan 10, 2025
b81bb45
added Disorder index and embedding
AndiMajore Jan 10, 2025
fc718a8
imporoved resilience of neo4j container shutdown and embedding
AndiMajore Jan 11, 2025
799e2a4
reset build minimum; updated graceful shutdown timeout
AndiMajore Jan 11, 2025
3437b13
robust metadata update and adjust chembl downloader
Jan 13, 2025
96811be
preparing manual testing of container shutdown in development build
AndiMajore Jan 15, 2025
e67dd13
updating neo4j shutdown routine to hopefully fix issues with shutting…
AndiMajore Jan 15, 2025
38cb852
reenable setup finalization after index creation
AndiMajore Jan 16, 2025
eba59bb
fixed some embedding setup related issues
AndiMajore Jan 16, 2025
795b3bf
added additional embeddings
AndiMajore Jan 17, 2025
478ae15
fixed issues with embedding building
AndiMajore Jan 17, 2025
3c8b939
fixed issue with disease embedsding
AndiMajore Jan 17, 2025
81447dc
fix issue with edge embedding being listed as node in vector index table
AndiMajore Jan 30, 2025
e2c2b6a
adding try catch for vector build
AndiMajore Jan 31, 2025
2e4a0d2
build more rigid embedding system and added Protein and ProteinHasTar…
AndiMajore Jan 31, 2025
56778bb
minor improvement for embedding generation
AndiMajore Feb 10, 2025
a59559f
ignore empty chunks for signatures
Feb 17, 2025
56d4252
fixing signature import
AndiMajore Feb 21, 2025
3b6c8a8
Update docker-publish.yml
AndiMajore Feb 21, 2025
e0e1a58
Update docker-publish.yml
AndiMajore Feb 21, 2025
1347e57
updated disgenet link
AndiMajore Feb 24, 2025
a83e7a9
Merge branch 'dev' of github.com:repotrial/nedrexdb_v2d into dev
AndiMajore Feb 24, 2025
9413bae
Minimal Config LLM update
ahoewi Feb 25, 2025
29b3c01
adding forced shutdown if gracefull shutdown for neo4j fails
AndiMajore Feb 28, 2025
0e33be0
fixing neo4j shutdown issue
AndiMajore Mar 3, 2025
11c876f
Merge branch 'dev' into dev
AndiMajore Mar 28, 2025
0eae9b1
fixing path issue with chembl file
AndiMajore Mar 28, 2025
c027db0
fixed open chembl import
AndiMajore Apr 1, 2025
1c77d44
fixing wait_for_database_ready parameter
AndiMajore Apr 2, 2025
882ffb3
small adjustment to prod cron file
AndiMajore Apr 9, 2025
ab086ee
Merge branch 'dev' into dev_save
ahoewi Apr 10, 2025
e6d4177
add embedding generation for all nodes and edges. Including in-code c…
ahoewi Apr 10, 2025
8e0b4fa
small defaults adjustments in code
ahoewi Apr 14, 2025
ac031b2
Update open_config.toml
ahoewi Apr 14, 2025
8050d02
Merge branch 'ahoewi-dev_save' into dev
AndiMajore Apr 16, 2025
7f556cf
updating minimal config and llm embeddings
AndiMajore Apr 16, 2025
af42e85
updating git ci to remove nightly image builds
AndiMajore Apr 17, 2025
198f1d8
fallback nedrex version. Two sources of truth now though
ahoewi Apr 18, 2025
0a7eb94
test embeddings during build
ahoewi Apr 19, 2025
edb0066
fallback nedrex version. Two sources of truth now though
ahoewi Apr 18, 2025
7c0226e
test embeddings during build
ahoewi Apr 19, 2025
aac1a13
setting default for api_key correctly
AndiMajore Apr 21, 2025
86994af
Merge branch 'dev' of github.com:repotrial/nedrexdb_v2d into dev
AndiMajore Apr 21, 2025
150584d
setting default for api_key correctly
AndiMajore Apr 21, 2025
9f1b338
removing colon from SideEffect config in create_vector_indices
AndiMajore Apr 21, 2025
49a47c1
small udpate to features used for embeddings
AndiMajore Apr 22, 2025
6095827
removing some high-demand-low-impact embedding creations
AndiMajore Apr 25, 2025
69836c0
fixing security voulnerability around h11
AndiMajore Apr 25, 2025
353d2ed
metadata based download skip v1
May 1, 2025
904bf38
add try catch block for retries of metadata writing and prev_metadata…
ahoewi May 3, 2025
63b22f4
metadata based download skip v2
ahoewi May 3, 2025
315581b
Merge branch 'dev' into dev_save
AndiMajore May 5, 2025
10c5aa4
removing proteinhassignature from index creation
AndiMajore May 6, 2025
ae36724
removing again some embedding configuration to speed up process
AndiMajore May 7, 2025
fef88ba
updating list of embeddings being created
AndiMajore May 14, 2025
6e95040
dev docker-compose change
AndiMajore May 16, 2025
1fb5818
updating neo4j image
AndiMajore May 19, 2025
e7e23f2
fixing apoc version
AndiMajore May 19, 2025
6b270b2
improving some docker orchestration parts; updating neo4j image autom…
AndiMajore May 20, 2025
18ebc46
testing new restart-live with recreate embeddings flag
AndiMajore May 21, 2025
c987551
adding 1min timeout between starting existing neo4j instance and vect…
AndiMajore May 21, 2025
dc87935
removing rebuild embeddings on restart-live flag again
AndiMajore May 21, 2025
6f7456b
added retry routine for embedding creation
AndiMajore May 22, 2025
b0fc3d5
reduce batch size for node and edge embeddings to increase stability
AndiMajore May 22, 2025
66e1d5a
adding AUTOUPDATE env var to disable auto updates and handle correct …
AndiMajore May 23, 2025
3301085
fixed update.sh file
AndiMajore May 26, 2025
184406f
temporarily removed cleanup of neo4j import files
AndiMajore Jun 26, 2025
4af108c
revert to cleaning csv files after import to neo
AndiMajore Jun 30, 2025
2df68bd
added FORCE_VERSION_OVERRIDE flag
AndiMajore Jul 23, 2025
e4639e0
updated neo4j image build and version
AndiMajore Jul 28, 2025
f30cbe1
dependency update
AndiMajore Jul 28, 2025
e309a8e
updated digenet.zip link
AndiMajore Aug 4, 2025
29b2642
testing unique constraints for nodes
AndiMajore Aug 7, 2025
8798001
reenabling embedding creation
AndiMajore Aug 11, 2025
bc7c5ef
create constraints and indexes correctly
AndiMajore Aug 12, 2025
9b2ee94
rebuild current version
AndiMajore Aug 13, 2025
d49528e
fix retry issue of embedding
AndiMajore Aug 14, 2025
537725e
added exception handling for issue where check for prev_metadata cras…
AndiMajore Aug 14, 2025
a244274
enabling all embeddings normally again
AndiMajore Aug 14, 2025
2f2b7b3
adjusted for development on norm
JohannesKersting Aug 19, 2025
95c7d25
updated config
JohannesKersting Aug 20, 2025
e10fa02
Merge branch 'repo4eu_dev' into merge_hamburg_2
JohannesKersting Aug 20, 2025
5b029be
forgot to save files
JohannesKersting Aug 20, 2025
f50acf5
changed ncg download log level
JohannesKersting Aug 20, 2025
9c4116d
merging
JohannesKersting Aug 20, 2025
eadb308
merged latest changes
JohannesKersting Aug 20, 2025
72613f0
handling unparsable symbols for intogen
JohannesKersting Aug 25, 2025
b1cda31
more memory for opentargets
JohannesKersting Aug 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
nedrex_files
neo4j
tests
57 changes: 53 additions & 4 deletions .github/workflows/docker-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ name: Docker
# documentation.

on:
schedule:
- cron: '37 8 * * *'
# schedule:
# - cron: '37 8 * * *'
push:
branches: [ "repo4eu_main", "repo4eu_dev" ]
# Publish semver tags as releases.
Expand All @@ -20,11 +20,11 @@ env:
REGISTRY: ghcr.io
# github.repository as <account>/<repo>
IMAGE_NAME: ${{ github.repository }}
NEO4J_IMAGE_NAME: ${{ github.repository }}-neo4j


jobs:
build:

build-main:
runs-on: ubuntu-latest
permissions:
contents: read
Expand Down Expand Up @@ -80,3 +80,52 @@ jobs:
platforms: linux/amd64,linux/arm64
cache-from: type=gha
cache-to: type=gha, mode=max

build-neo4j:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Setup QEMU
uses: docker/setup-qemu-action@v2.1.0

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3.0.0
with:
install: true
use: true

- name: Log into registry ${{ env.REGISTRY }}
if: github.event_name != 'pull_request'
uses: docker/login-action@v3.0.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract Docker metadata for Neo4j
id: meta-neo4j
uses: docker/metadata-action@v5.0.0
with:
images: ${{ env.REGISTRY }}/${{ env.NEO4J_IMAGE_NAME }}

- name: Build and push Neo4j Docker image
id: build-and-push-neo4j
uses: docker/build-push-action@v5.0.0
with:
context: ./neo4j
file: ./neo4j/Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta-neo4j.outputs.tags }}
labels: ${{ steps.meta-neo4j.outputs.labels }}
#Temporary disabled linux/arm64 because of system library issues
# platforms: linux/amd64,linux/arm64
platforms: linux/amd64
cache-from: type=gha
cache-to: type=gha,mode=max
3 changes: 2 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,8 @@ RUN touch /var/log/nedrexdb.log
COPY . ./
RUN rm -rf cron
RUN pip install .[dependencies]
RUN mamba install -c conda-forge urllib3=2.3.0 -y

#RUN mamba install -c conda-forge urllib3=2.3.0 -y
RUN mamba install -c conda-forge openjdk=17 -y

CMD cron && bash build.sh >> /var/log/nedrexdb.log 2>&1 & tail -f /var/log/nedrexdb.log
Loading