Skip to content

Commit 23adea0

Browse files
amotlamotl
committed
Improve sandbox environment setup and documentation
1 parent 6a578c5 commit 23adea0

File tree

12 files changed

+198
-144
lines changed

12 files changed

+198
-144
lines changed

‎CHANGES.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ development
88
- Add example for annotating phenology data within Grafana
99
- Change license to AGPL-3.0
1010
- Modernize dependency versions. Drop support for Python 3.6.
11+
- Improve sandbox environment setup and documentation
1112

1213

1314
2020-12-27 0.1.2

‎README.rst

Lines changed: 104 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,124 @@
1+
.. image:: https://github.com/panodata/grafana-pandas-datasource/workflows/Tests/badge.svg
2+
:target: https://github.com/panodata/grafana-pandas-datasource/actions?workflow=Tests
3+
4+
.. image:: https://img.shields.io/pypi/pyversions/grafana-pandas-datasource.svg
5+
:target: https://pypi.org/project/grafana-pandas-datasource/
6+
7+
.. image:: https://img.shields.io/pypi/status/grafana-pandas-datasource.svg
8+
:target: https://pypi.org/project/grafana-pandas-datasource/
9+
10+
.. image:: https://img.shields.io/pypi/v/grafana-pandas-datasource.svg
11+
:target: https://pypi.org/project/grafana-pandas-datasource/
12+
13+
.. image:: https://img.shields.io/pypi/dm/grafana-pandas-datasource.svg
14+
:target: https://pypi.org/project/grafana-pandas-datasource/
15+
16+
.. image:: https://img.shields.io/pypi/l/grafana-pandas-datasource.svg
17+
:target: https://github.com/panodata/grafana-pandas-datasource/blob/main/LICENSE
18+
19+
.. image:: https://img.shields.io/badge/Grafana-6.x%20--%208.x-blue.svg
20+
:target: https://github.com/grafana/grafana
21+
:alt: Supported Grafana versions
22+
23+
|
24+
125
#########################
2-
Grafana Pandas Datasource
26+
Grafana pandas datasource
327
#########################
428

529

630
*****
731
About
832
*****
9-
A REST API based on Flask for serving Pandas Dataframes to Grafana.
1033

11-
This way, a native Python application can be used to directly supply
12-
data to Grafana both easily and powerfully.
34+
A HTTP API based on Flask_ for serving pandas_ data frames to Grafana_,
35+
generated by NumPy_. The `Grafana Simple JSON Datasource`_ is used to interface
36+
Grafana with the HTTP API.
1337

14-
It was inspired by and is compatible with the simple json datasource.
38+
This way, a native Python application can be used to directly supply data to
39+
Grafana easily and powerfully.
1540

16-
https://gist.github.com/linar-jether/95ff412f9d19fdf5e51293eb0c09b850
41+
The framework supports feeding both timeseries data as well as annotations
42+
through corresponding ``/query`` and ``/annotations`` endpoints and also
43+
provides ``/search`` and ``/panels`` endpoints.
1744

18-
Setup
19-
=====
20-
::
2145

22-
pip install grafana-pandas-datasource
46+
*******************
47+
Sandbox environment
48+
*******************
49+
50+
In order to work efficiently with the resources provided by this repository, we
51+
recommend to install some programs upfront. This will optimally work on Linux
52+
and macOS. Windows users might use the WSL subsystem.
53+
54+
Install prerequisites::
55+
56+
# Debian Linux
57+
apt install git python3 python3-pip httpie docker.io
58+
pip install poetry
59+
60+
# macOS / Homebrew
61+
brew install git python3 poetry httpie docker
62+
63+
Acquire sources and bootstrap sandbox environment::
64+
65+
git clone https://github.com/panodata/grafana-pandas-datasource
66+
cd grafana-pandas-datasource
67+
poetry install
68+
poetry shell
69+
70+
Test drive::
2371

72+
# Run Grafana pandas datasource demo.
73+
python examples/sinewave-midnights/demo.py
2474

25-
Resources
26-
=========
27-
- https://github.com/grafana/grafana
28-
- https://grafana.com/grafana/plugins/grafana-simple-json-datasource
75+
# Submit a timeseries data request.
76+
echo '{"targets": [{"target": "sine_wave:24"}], "range": {"from": "2022-02-22T15", "to": "2022-02-22T20"}}' | http http://127.0.0.1:3003/query
77+
78+
# Submit an annotation data request.
79+
echo '{"annotation": {"query": "midnights:xx"}, "range": {"from": "2022-02-20", "to": "2022-02-22"}}' | http http://127.0.0.1:3003/annotations
80+
81+
When the environment has been properly configured, both requests above will
82+
yield appropriate responses.
2983

3084

3185
********
3286
Examples
3387
********
34-
There are different demo programs accompanied with Grafana datasource
35-
and dashboard definition files.
3688

37-
https://github.com/panodata/grafana-pandas-datasource/tree/main/examples
89+
There are `different demo programs`_ accompanied with Grafana datasource and
90+
dashboard definition files.
91+
92+
After confirming the sandbox environment has been installed successfully,
93+
please head over to the `Sinewave/Midnights example`_ page in order to learn
94+
how to provision Grafana with corresponding resources.
95+
96+
97+
*****
98+
Setup
99+
*****
100+
101+
When aiming to run a dedicated service, without needing to invoke the examples,
102+
you can add the package ``grafana-pandas-datasource`` to the list of your
103+
project requirements.
104+
105+
::
106+
107+
pip install grafana-pandas-datasource
108+
109+
110+
*******
111+
Credits
112+
*******
113+
114+
Kudos to Linar, who conceived the initial version of this software the other
115+
day at https://gist.github.com/linar-jether/95ff412f9d19fdf5e51293eb0c09b850.
116+
117+
118+
.. _different demo programs: https://github.com/panodata/grafana-pandas-datasource/tree/main/examples
119+
.. _Flask: https://github.com/pallets/flask
120+
.. _Grafana: https://github.com/grafana/grafana
121+
.. _Grafana Simple JSON Datasource: https://grafana.com/grafana/plugins/grafana-simple-json-datasource/
122+
.. _NumPy: https://numpy.org/
123+
.. _pandas: https://github.com/pandas-dev/pandas
124+
.. _Sinewave/Midnights example: https://github.com/panodata/grafana-pandas-datasource/tree/main/examples/sinewave-midnights

‎docs/backlog.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#################################
2+
Grafana pandas Datasource backlog
3+
#################################
4+
5+
- [x] Adjust HTML index page
6+
- [o] Blackify
7+
- [o] Upgrade to Flask 2

‎examples/phenodata-mellifera/demo.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ def phenodata_mellifera(dataset: str, years: tuple[int], phases: tuple[str], opt
8282

8383
data = pd.concat(data_total)
8484

85-
# Create Pandas Series from Dataframe.
85+
# Create pandas Series from DataFrame.
8686
index = data.Datum.astype('datetime64')
8787
values = data.Spezies.str.cat(data.Phase, sep=" - ").str.cat(data.Station, sep=" - ")
8888
series = pd.Series(data=values.tolist(), index=index)
@@ -99,7 +99,7 @@ def main():
9999
# Create Flask application.
100100
app = create_app()
101101

102-
# Register Pandas component.
102+
# Register pandas component.
103103
app.register_blueprint(pandas_component, url_prefix="/")
104104

105105
# Invoke Flask application.

‎examples/phenodata-mellifera/readme.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ Invoke
4343
--env='GF_SECURITY_ADMIN_PASSWORD=admin' --env='GF_INSTALL_PLUGINS=grafana-simple-json-datasource' \
4444
grafana/grafana:7.3.6
4545

46-
# Run Grafana Pandas Datasource demo.
46+
# Run Grafana pandas Datasource demo.
4747
python phenodata-mellifera-demo.py
4848

4949

‎examples/sinewave-midnights/demo.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
"""
1414
Demo for grafana-pandas-datasource.
1515
16-
This is a demo program which generates data using NumPy and Pandas.
16+
This is a demo program which generates data using NumPy and pandas.
1717
It creates
1818
- a sine wave for data and
1919
- midnight times for annotations
@@ -25,6 +25,13 @@
2525
To query the reader, use ``<reader_name>:<query_string>``, e.g.
2626
- ``sine_wave:24``
2727
- ``midnights:xx``
28+
29+
Test drive::
30+
31+
python examples/sinewave-midnights/demo.py
32+
echo '{"targets": [{"target": "sine_wave:24"}], "range": {"from": "2022-02-22T15", "to": "2022-02-22T20"}}' | http http://127.0.0.1:3003/query
33+
echo '{"annotation": {"query": "midnights:xx"}, "range": {"from": "2022-02-20", "to": "2022-02-22"}}' | http http://127.0.0.1:3003/annotations
34+
2835
"""
2936

3037

@@ -53,7 +60,7 @@ def main():
5360
# Create Flask application.
5461
app = create_app()
5562

56-
# Register Pandas component.
63+
# Register pandas component.
5764
app.register_blueprint(pandas_component, url_prefix="/")
5865

5966
# Invoke Flask application.

‎examples/sinewave-midnights/readme.rst

Lines changed: 64 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,71 +1,99 @@
11
##########################
22
Sinewave/Midnights example
33
##########################
4-
This is a demo program which generates a sine wave for data and
5-
annotations for designating midnight times. For both, we are using NumPy.
6-
7-
.. figure:: https://user-images.githubusercontent.com/453543/103137119-78dab480-46c6-11eb-829f-6aa957239804.png
8-
9-
Image: Sinewave data and midnights annotations, both generated using NumPy.
104

115

12-
Setup
13-
=====
14-
::
15-
16-
pip install grafana-pandas-datasource
6+
*****
7+
About
8+
*****
179

10+
This is a demo program which generates a sine wave for data and
11+
annotations for designating midnight times.
1812

19-
Acquire example files
20-
=====================
21-
::
13+
.. figure:: https://user-images.githubusercontent.com/453543/103137119-78dab480-46c6-11eb-829f-6aa957239804.png
2214

23-
export EXAMPLES_BASEURL=https://raw.githubusercontent.com/panodata/grafana-pandas-datasource/0.1.0/examples
15+
Image: Sinewave data and midnights annotations, both generated using NumPy_.
2416

25-
wget ${EXAMPLES_BASEURL}/sinewave-midnights/demo.py \
26-
--output-document=sinewave-midnights-demo.py
2717

28-
wget ${EXAMPLES_BASEURL}/sinewave-midnights/datasource.json \
29-
--output-document=sinewave-midnights-datasource.json
18+
*****
19+
Setup
20+
*****
3021

31-
wget ${EXAMPLES_BASEURL}/sinewave-midnights/dashboard.json \
32-
--output-document=sinewave-midnights-dashboard.json
22+
For general installation instructions, see `setup sandbox environment`_.
3323

3424

35-
Invoke
36-
======
25+
**************
26+
Start services
27+
**************
3728
::
3829

3930
# Run Grafana.
4031
docker run --rm -it \
4132
--publish=3000:3000 --volume="$(pwd)/var/lib/grafana":/var/lib/grafana \
4233
--env='GF_SECURITY_ADMIN_PASSWORD=admin' --env='GF_INSTALL_PLUGINS=grafana-simple-json-datasource' \
43-
grafana/grafana:7.3.6
34+
grafana/grafana:8.3.4
4435

45-
# Run Grafana Pandas Datasource demo.
46-
python sinewave-midnights-demo.py
36+
# Run Grafana pandas datasource demo.
37+
python examples/sinewave-midnights/demo.py
4738

4839

49-
Configure
50-
=========
51-
.. note::
40+
*****************
41+
Configure Grafana
42+
*****************
5243

53-
The host where the datasource service is running can be accessed from the
54-
Grafana Docker container using the hostname ``host.docker.internal``.
44+
Command line
45+
============
5546

56-
You can have a quickstart by putting ``examples/sinewave-midnights/datasource.json``
57-
and ``examples/sinewave-midnights/dashboard.json`` into Grafana::
47+
You can have a quickstart by putting those two JSON definition files into
48+
Grafana::
5849

5950
# Login to Grafana.
6051
export GRAFANA_URL=http://localhost:3000
6152
http --session=grafana ${GRAFANA_URL} --auth=admin:admin
6253

6354
# Create datasource.
64-
cat sinewave-midnights-datasource.json | \
55+
cat examples/sinewave-midnights/datasource.json | \
6556
http --session=grafana POST ${GRAFANA_URL}/api/datasources
6657

6758
# Create dashboard.
68-
cat sinewave-midnights-dashboard.json | \
59+
cat examples/sinewave-midnights/dashboard.json | \
6960
http --session=grafana POST ${GRAFANA_URL}/api/dashboards/db
7061

71-
open ${GRAFANA_URL}
62+
Then, visit the dashboard at::
63+
64+
open http://localhost:3000/d/xNbUrobGz/sine-24
65+
66+
GUI
67+
===
68+
69+
This section walks you through setting up a data source and dashboard in
70+
Grafana manually, step by step with screenshots. Please follow the guidelines
71+
carefully.
72+
73+
.. figure:: https://user-images.githubusercontent.com/453543/150621604-f9b4664c-493a-4a9d-bd46-cf59da175438.png
74+
75+
Install "SimpleJson" plugin at http://localhost:3000/plugins/grafana-simple-json-datasource.
76+
77+
.. figure:: https://user-images.githubusercontent.com/453543/150621516-cb8b24fa-46ee-4515-b66e-81f79a046912.png
78+
79+
Add new data source of "SimpleJson" type at http://localhost:3000/datasources/new.
80+
Configure the URL to the Flask service serving pandas data frames.
81+
When running Grafana in Docker, use ``host.docker.internal`` to address the
82+
Docker host.
83+
84+
.. figure:: https://user-images.githubusercontent.com/453543/150621520-f0eeb740-2c12-4a8b-908c-50893a8bd583.png
85+
86+
Create dashboard with Timeseries or Graph panel at http://localhost:3000/dashboard/new,
87+
adjust "Data source" and "metric" values.
88+
89+
.. figure:: https://user-images.githubusercontent.com/453543/150621869-5d226582-886c-41c4-a446-d8d75685f9d2.png
90+
91+
At the time picker, choose an interval of "Last 2 days".
92+
93+
.. figure:: https://user-images.githubusercontent.com/453543/150621970-3d20f11c-007a-4e6e-ad8f-abf1f3e02ed0.png
94+
95+
Save your dashboard.
96+
97+
98+
.. _NumPy: https://numpy.org/
99+
.. _setup sandbox environment: https://github.com/panodata/grafana-pandas-datasource/blob/main/README.rst#sandbox-environment

‎grafana_pandas_datasource/registry.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
class DataGenerators:
1313
"""
1414
Store references to data generator functions
15-
yielding Pandas data frames.
15+
yielding pandas data frames.
1616
"""
1717

1818
metric_readers: Dict[str, Callable] = field(default_factory=dict)

‎grafana_pandas_datasource/service.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@
1919
@cross_origin()
2020
def hello_world():
2121
print(request.headers, request.get_json())
22-
return 'Jether\'s Grafana Pandas Datasource, used for rendering HTML panels and timeseries data.'
22+
return 'Grafana pandas datasource: Serve NumPy data via pandas data frames to Grafana. ' \
23+
'For documentation, see <a href="https://github.com/panodata/grafana-pandas-datasource">https://github.com/panodata/grafana-pandas-datasource</a>.'
2324

2425

2526
@pandas_component.route('/search', methods=methods)

0 commit comments

Comments
 (0)