Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions DEVELOP.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ These instructions are for contributing to this repository. See main README.md f

```bash
cd src/docker
docker-compose build
docker-compose up
docker compose build
docker compose up
```

## Test
Expand Down
26 changes: 13 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ This open source effort puts together patterns the Graphistry team has reused ac

## What

* **Minimal core**: The barebones dashboard server. In provides a StreamLit docker-compose container with PyData ecosystem libraries and examples of visualizing data from various systems. Install it, plug in credentials to various web services like cloud databases and a free [Graphistry Hub](https://hub.graphistry.com) visualization account, and launch. It does not have GPU ETL and GPU AI libraries.
* **Minimal core**: The barebones dashboard server. In provides a StreamLit docker compose container with PyData ecosystem libraries and examples of visualizing data from various systems. Install it, plug in credentials to various web services like cloud databases and a free [Graphistry Hub](https://hub.graphistry.com) visualization account, and launch. It does not have GPU ETL and GPU AI libraries.

* **Full core**: Initially for AWS, the full core bundles adds to the docker-compose system: Accounts, Jupyter notebooks for authoring, serves StreamLit dashboards with both public + private zones, and runs Graphistry/RAPIDS locally on the same server. Launch with on click via the Cloud Formation template.
* **Full core**: Initially for AWS, the full core bundles adds to the docker compose system: Accounts, Jupyter notebooks for authoring, serves StreamLit dashboards with both public + private zones, and runs Graphistry/RAPIDS locally on the same server. Launch with on click via the Cloud Formation template.

* **Full core + DB**: DB-specific variants are the same as minimal/full, and add simpler DB-specific quick launching/connecting.

Expand All @@ -36,26 +36,26 @@ This open source effort puts together patterns the Graphistry team has reused ac

**Note**: Base image includes Nvidia RAPIDS and AI dependencies so is quite large, see CPU alternative for a lightweight alternativve

**Note**: Use `sudo` for docker-compose commands if your configuration requires it and is giving permission error
**Note**: Use `sudo` for docker compose commands if your configuration requires it and is giving permission error

```bash
# Minimal core
git clone https://github.com/graphistry/graph-app-kit.git
cd graph-app-kit/src/docker

# Enable docker buildkit
# ... or run docker-compose via provided alias script `./dc`
# ... or run docker compose via provided alias script `./dc`
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1

# Build
docker-compose build
docker compose build

# Optional: Edit src/docker/.env (API accounts), docker-compose.yml: Auth, ports, ...

# Launch
docker-compose up -d
docker-compose logs -f -t --tail=100
docker compose up -d
docker compose logs -f -t --tail=100
```

=> `http://localhost:8501/`
Expand All @@ -65,7 +65,7 @@ To [add views](docs/views.md) and relaunch:
```bash
# Add dashboards @ src/python/views/<your_custom_view>/__init__.py

docker-compose up -d --force-recreate
docker compose up -d --force-recreate
```

### Quick (Local code) - minimal CPU core + third-party connectors
Expand Down Expand Up @@ -102,14 +102,14 @@ sudo docker ps
sudo docker logs -f -t --tail=1 MY_CONTAINER

# restart a graphistry container
cd graphistry && sudo docker-compose restart MY_CONTAINER
cd graphistry && sudo docker compose restart MY_CONTAINER

# restart caddy (Caddy 1 override)
cd graphistry && sudo docker-compose -f docker-compose.gak.graphistry.yml up -d caddy
cd graphistry && sudo docker compose -f docker-compose.gak.graphistry.yml up -d caddy

# run streamlit
cd graph-app-kit/public/graph-app-kit && docker-compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker-compose -p priv run -d --name streamlit-priv streamlit
cd graph-app-kit/public/graph-app-kit && docker compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker compose -p priv run -d --name streamlit-priv streamlit
```

**Minimal**: Open Streamlit, ssh to connect/add [free Graphistry Hub username/pass](https://www.graphistry.com/get-started):
Expand Down Expand Up @@ -140,7 +140,7 @@ VIEW_PATH="`pwd`/views" streamlit run entrypoint.py
* [Graphistry](https://www.graphistry.com/get-started) point-and-click GPU-accelerated visual graph analytics
* Data frames: Data wrangling via [Pandas](https://pandas.pydata.org/)and [Apache Arrow](https://arrow.apache.org/), including handling formats such as CSV, XLS, JSON, Parquet, and more

* Standard Docker and docker-compose cross-platform deployment
* Standard Docker and docker compose cross-platform deployment

### GPU acceleration (optional) - Full

Expand Down
6 changes: 3 additions & 3 deletions docs/extend.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Configure graph-app-kit and add integrations

Most settings can be configured by creating a custom Docker environment file `src/docker/.env` (see `src/envs/*.env` for options). You can also edit `docker-compose.yml` and `/etc/docker/daemon.json`, but we recommend sticking to `.env`.
Most settings can be configured by creating a custom Docker environment file `src/docker/.env` (see `src/envs/*.env` for options). You can also edit `docker compose.yml` and `/etc/docker/daemon.json`, but we recommend sticking to `.env`.

Integration settings that deal with external systems such as TLS, accounts, and notebooks require having launched them. If you are not integrating into existing ones, see the initial [setup section](setup.md) for how to quicklaunch a Graphistry server.

## Monitoring

By default, `graph-app-kit` logging uses the Docker json file driver:

* Inspect recent activities: `cd src/docker` and then `sudo docker-compose logs -f -t --tail=100`
* Inspect recent activities: `cd src/docker` and then `sudo docker compose logs -f -t --tail=100`

* Setup [alternative logging drivers](https://docs.docker.com/config/containers/logging/configure/)

Expand All @@ -24,7 +24,7 @@ By default, `graph-app-kit` logging uses the Docker json file driver:

## TLS with Cadddy

* Auth: See [Caddy sample](src/caddy/Caddyfile) reverse proxy example for an authentication check against an account system, including the one shipping with your Graphistry server (requires `sudo docker-compose restart caddy` in your Graphistry server upon editing `/var/graphistry/data/config/Caddyfile`)
* Auth: See [Caddy sample](src/caddy/Caddyfile) reverse proxy example for an authentication check against an account system, including the one shipping with your Graphistry server (requires `sudo docker compose restart caddy` in your Graphistry server upon editing `/var/graphistry/data/config/Caddyfile`)

## Public+Private views
* To simulatenously run 1 public and 1 private instance, create two `graph-app-kit` clones `public_dash` and `private_dash`, and for `src/docker/.env`, set:
Expand Down
6 changes: 3 additions & 3 deletions docs/neptune-manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ Reset and restart your `graph-app-kit` container:

```bash
cd src/docker
sudo docker-compose down -v
sudo docker-compose up -d
sudo docker compose down -v
sudo docker compose up -d
```

Watch logs with `sudo docker-compose logs -f -t --tail=1`
Watch logs with `sudo docker compose logs -f -t --tail=1`

Access your Streamlit instance at http://the.public.ip.address:8501

Expand Down
6 changes: 3 additions & 3 deletions docs/neptune.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,11 @@ Launch using a button at the bottom of the [identity graph sample cloud formatio
* **Streamlit: Public dashboards**
* **http://[the.public.ip.address]/public/dash**
* Installed at `/home/ubuntu/graph-app-kit/public/graph-app-kit`
* Run as `src/docker $ docker-compose -p pub run -d --name streamlit-pub streamlit`
* Run as `src/docker $ docker compose -p pub run -d --name streamlit-pub streamlit`
* **Streamlit: Private dashboards**
* **http://[the.public.ip.address]/private/dash**
* Installed at `/home/ubuntu/graph-app-kit/private/graph-app-kit`
* Run as `src/docker $ docker-compose -p priv run -d --name streamlit-priv streamlit`
* Run as `src/docker $ docker compose -p priv run -d --name streamlit-priv streamlit`
* **Jupyter: Data science notebooks + Streamlit dashboard live-editing**
* **http://[the.public.ip.address]/notebook**
* Live-edit `graph-app-kit` view folders `notebook/graph-app-kit/[public,private]/views`
Expand All @@ -126,7 +126,7 @@ Launch using a button at the bottom of the [identity graph sample cloud formatio
* **Streamlit: Public dashboards**
* **http://[the.public.ip.address]/public/dash**
* Installed at `/home/ubuntu/graph-app-kit/public/graph-app-kit`
* Run as `src/docker $ docker-compose up -d streamlit`
* Run as `src/docker $ docker compose up -d streamlit`

## 4. Next steps

Expand Down
6 changes: 3 additions & 3 deletions docs/setup-manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ For quick launchers, see the [AWS quick launch setup guide](setup.md).

* Ubuntu 18.04 LTS is the most common choice for containerized GPU computing

* Install docker-ce and docker-compose
* Install docker-ce and docker compose

* Optional:
* GPU: If you have a [RAPIDS.ai](https://www.rapids.ai)-compatible GPU (see below), install the Nvidia docker runtime and set it as the default for Docker daemons
Expand Down Expand Up @@ -57,9 +57,9 @@ GRAPHISTRY_PASSWORD=your_password

`cd src/docker` and then:

* Start: `sudo docker-compose up -d`
* Start: `sudo docker compose up -d`
* Use: Go to `http://localhost:8501/dashboard` (or whatever the public IP)
* Stop: `sudo docker-compose down -v`
* Stop: `sudo docker compose down -v`

## Graph!

Expand Down
16 changes: 8 additions & 8 deletions docs/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,11 +85,11 @@ Go to your public Streamlit dashboard and start exploring: http://[the.public.ip
* **Streamlit: Public dashboards**
* **http://[the.public.ip.address]/public/dash**
* Installed at `/home/ubuntu/graph-app-kit/public/graph-app-kit`
* Run as `src/docker $ docker-compose -p pub run -d --name streamlit-pub streamlit`
* Run as `src/docker $ docker compose -p pub run -d --name streamlit-pub streamlit`
* **Streamlit: Private dashboards**
* **http://[the.public.ip.address]/private/dash**
* Installed at `/home/ubuntu/graph-app-kit/private/graph-app-kit`
* Run as `src/docker $ docker-compose -p priv run -d --name streamlit-priv streamlit`
* Run as `src/docker $ docker compose -p priv run -d --name streamlit-priv streamlit`
* **Jupyter: Data science notebooks + Streamlit dashboard live-editing**
* **http://[the.public.ip.address]/notebook**
* Live-edit `graph-app-kit` view folders `notebook/graph-app-kit/[public,private]/views`
Expand All @@ -99,7 +99,7 @@ Go to your public Streamlit dashboard and start exploring: http://[the.public.ip
* **Streamlit: Public dashboards**
* **http://[the.public.ip.address]/public/dash**
* Installed at `/home/ubuntu/graph-app-kit/public/graph-app-kit`
* Run as `src/docker $ docker-compose up -d streamlit`
* Run as `src/docker $ docker compose up -d streamlit`

## 4. Optional - administer

Expand Down Expand Up @@ -127,19 +127,19 @@ For more advanced Graphistry administration, so the [Graphistry admin docs repo]

```bash
# restart a graphistry container
cd graphistry && sudo docker-compose restart MY_CONTAINER # do *not* run it's caddy (v2)
cd graphistry && sudo docker compose restart MY_CONTAINER # do *not* run it's caddy (v2)

# restart caddy (Caddy 1 override over Graphistry's Caddy 2)
cd graphistry && sudo docker-compose -f docker-compose.gak.graphistry.yml up -d caddy
cd graphistry && sudo docker compose -f docker-compose.gak.graphistry.yml up -d caddy
```

### Streamlit

Use `docker-compose` project names (`-p the_name`) to distinguish your public vs private dashboards:
Use `docker compose` project names (`-p the_name`) to distinguish your public vs private dashboards:

```bash
cd graph-app-kit/public/graph-app-kit && docker-compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker-compose -p priv run -d --name streamlit-priv streamlit
cd graph-app-kit/public/graph-app-kit && docker compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker compose -p priv run -d --name streamlit-priv streamlit
```

## 5. Next steps
Expand Down
2 changes: 1 addition & 1 deletion docs/tigergraph.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ TIGERGRAPH_GRAPHNAME=AntiFraud
TIGERGRAPH_SECRET=mysecret
```

3. Restart your Streamlit container with the new creds: `cd src/docker && sudo docker-compose up -d --force-recreate`
3. Restart your Streamlit container with the new creds: `cd src/docker && sudo docker compose up -d --force-recreate`

## Explore

Expand Down
2 changes: 1 addition & 1 deletion docs/views.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Not required but recommended, we recommend setting up Jupyter-based shared web a

* Modify Python files in `src/python/views/[your dashboard]/__init__.py`, and in-tool, hit the `rerun` button that appears
* Add new views by adding `views/[your dsahboard]/__init__.py` with methods `def info(): return {'name': 'x'}` and `def run(): None`
* Add new dependencies: modify `src/python/requirements-app.txt` and rerun `docker-compose build` and restart
* Add new dependencies: modify `src/python/requirements-app.txt` and rerun `docker compose build` and restart

## Toggle views

Expand Down
4 changes: 2 additions & 2 deletions src/bootstraps/core/graphistry.sh
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,11 @@ echo '----- Finish pub vs. priv .env specialization'
echo '----- Launching graph-app-kit as streamlit-pub/priv:8501'
( \
cd "${GAK_PUBLIC}/src/docker" \
&& sudo docker-compose -p pub run -d --name streamlit-pub streamlit \
&& sudo docker compose -p pub run -d --name streamlit-pub streamlit \
)
( \
cd "${GAK_PRIVATE}/src/docker" \
&& sudo docker-compose -p priv run -d --name streamlit-priv streamlit \
&& sudo docker compose -p priv run -d --name streamlit-priv streamlit \
)

./hello-end.sh "$SCRIPT"
4 changes: 2 additions & 2 deletions src/bootstraps/neptune/graphistry.sh
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,11 @@ echo "BASE_PATH=private/dash/" | sudo tee -a "${GAK_PRIVATE}/src/docker/.env"
echo '----- Launching graph-app-kit as streamlit-pub/priv:8501'
( \
cd "${GAK_PUBLIC}/src/docker" \
&& sudo docker-compose -p pub run -d --name streamlit-pub streamlit \
&& sudo docker compose -p pub run -d --name streamlit-pub streamlit \
)
( \
cd "${GAK_PRIVATE}/src/docker" \
&& sudo docker-compose -p priv run -d --name streamlit-priv streamlit \
&& sudo docker compose -p priv run -d --name streamlit-priv streamlit \
)

./hello-end.sh "$SCRIPT"
2 changes: 1 addition & 1 deletion src/bootstraps/neptune/minimal.sh
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ echo '===== Configuring graph-app-kit with Graphistry account and Neptune ====='
echo '----- Launching graph-app-kit as streamlit-pub:8501'
( \
cd "${GAK_PUBLIC}/src/docker" \
&& sudo /usr/local/bin/docker-compose up -d streamlit \
&& sudo /usr/local/bin/docker compose up -d streamlit \
)

./hello-end.sh "$SCRIPT"
6 changes: 3 additions & 3 deletions src/bootstraps/scripts/docker-aws.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ sudo service docker start
sudo usermod -a -G docker ec2-user
sudo chkconfig docker on

sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose version
sudo curl -L https://github.com/docker/compose/releases/latest/download/docker compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker compose
sudo chmod +x /usr/local/bin/docker compose
docker compose version

./hello-end.sh "$SCRIPT"
2 changes: 1 addition & 1 deletion src/bootstraps/scripts/docker-container-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -ex

SCRIPT="Build graph-app-kit docker"
DC_ALIAS=${DC_ALIAS:-docker-compose}
DC_ALIAS=${DC_ALIAS:-docker compose}
./hello-start.sh "$SCRIPT"

( cd ../../docker && $DC_ALIAS build )
Expand Down
4 changes: 2 additions & 2 deletions src/bootstraps/scripts/graphistry-service-account.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ SCRIPT="Get AWS Instance ID"
cd "${GRAPHISTRY_HOME}" \
&& ( \
until ( curl -fsS http://localhost/streamgl-gpu/secondary/gpu/health > /dev/null ); \
do ( docker-compose ps && sleep 1 ); \
do ( docker compose ps && sleep 1 ); \
done \
)
)
Expand All @@ -24,7 +24,7 @@ POST_SCRIPT="CELERY_BROKER_URL=zz python manage.py shell && echo done || { echo

( \
cd "${GRAPHISTRY_HOME}" \
&& docker-compose exec -T nexus \
&& docker compose exec -T nexus \
bash -c \
"source activate rapids && echo \"${ADD_USER_SCRIPT}; ${VERIFY_USER_SCRIPT}\" | ${POST_SCRIPT}" \
)
Expand Down
6 changes: 3 additions & 3 deletions src/bootstraps/scripts/graphistry-wait-healthy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,20 @@ SCRIPT="Wait Graphistry docker containers healthy"

( \
cd "${GRAPHISTRY_HOME}" \
&& for i in `sudo docker-compose ps --services`; do ( \
&& for i in `sudo docker compose ps --services`; do ( \
( \
until ( \
[[ '"healthy"' == $(sudo docker inspect "graphistry_${i}_1" --format "{{json .State.Health.Status}}") ]] \
); do ( \
echo "waiting on $i (5s)" \
&& sudo docker-compose ps \
&& sudo docker compose ps \
&& sleep 5 \
); done \
) && echo "healthy $i" \
); done \
)

echo "--- Graphistry status after healthy waiting ---"
( cd "${GRAPHISTRY_HOME}" && sudo docker-compose ps )
( cd "${GRAPHISTRY_HOME}" && sudo docker compose ps )

./hello-end.sh "$SCRIPT"
12 changes: 6 additions & 6 deletions src/bootstraps/scripts/swap-caddy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
set -ex

## Graphistry now uses Caddy2, and until Caddy 2 supports simple auth, g-a-k still sticks with Caddy1
## ... So we stop Graphistry's Caddy 2 and put in a docker-compose of its old
## ... So we stop Graphistry's Caddy 2 and put in a docker compose of its old

CADDY_FILENAME=${CADDY_FILENAME:-full.Caddyfile}
CADDY_COMPOSE_FILENAME=${CADDY_COMPOSE_FILENAME:-docker-compose.gak.graphistry.yml}
CADDY_COMPOSE_FILENAME=${CADDY_COMPOSE_FILENAME:-docker compose.gak.graphistry.yml}

SCRIPT="Swap Caddyfile"
./hello-start.sh "$SCRIPT"
Expand All @@ -16,10 +16,10 @@ sudo cp "../../caddy/${CADDY_COMPOSE_FILENAME}" "${GRAPHISTRY_HOME}/${CADDY_COMP
sudo cp "../../caddy/${CADDY_FILENAME}" "${GRAPHISTRY_HOME}/data/config/Caddyfile"
( \
cd "${GRAPHISTRY_HOME}" \
&& sudo docker-compose stop caddy \
&& sudo docker-compose -f "${CADDY_COMPOSE_FILENAME}" up -d caddy \
&& sudo docker-compose ps \
&& sudo docker-compose -f "${CADDY_COMPOSE_FILENAME}" ps \
&& sudo docker compose stop caddy \
&& sudo docker compose -f "${CADDY_COMPOSE_FILENAME}" up -d caddy \
&& sudo docker compose ps \
&& sudo docker compose -f "${CADDY_COMPOSE_FILENAME}" ps \
)

./hello-end.sh "$SCRIPT"
2 changes: 1 addition & 1 deletion src/caddy/Caddyfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
### #
### -- Relaunch: #
### #
### docker-compose stop caddy && docker-compose up -d #
### docker compose stop caddy && docker compose up -d #
### #
##########################################################################################

Expand Down
2 changes: 1 addition & 1 deletion src/caddy/docker-compose.gak.graphistry.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#Reuse graphistry's docker-compose setup.. except downgrade to caddy1 for auth support
#Reuse graphistry's docker compose setup.. except downgrade to caddy1 for auth support
#Start _after_ graphistry

version: '3.5'
Expand Down
Loading
Loading