Skip to content

Commit 8c07249

Browse files
joostpvhjoostpvh
committed
Adding the StatsStore Scraper v0.3
1 parent bf81514 commit 8c07249

File tree

13 files changed

+1015
-10
lines changed

13 files changed

+1015
-10
lines changed

‎StatsStore_WebUI/README.md

+2-10
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,6 @@ Here is where you can see Oracle Solaris 11.4 in action:
1212

1313
- [Creating a custom sheet/template](Creating_Sheets)
1414
- [Sharing sheets on the dashboard](Sharing_Sheets)
15+
- [Pull data from the StatsStore into other datastores](StatsStore_Scraper)
1516

16-
17-
18-
19-
20-
21-
22-
23-
24-
Copyright (c) 2020, Oracle and/or its affiliates.
25-
Licensed under the Universal Permissive License v 1.0 as shown at <https://oss.oracle.com/licenses/upl/>.
17+
Copyright (c) 2022 Oracle and/or its affiliates and released under the [Universal Permissive License (UPL)](https://oss.oracle.com/licenses/upl/), Version 1.0

‎StatsStore_WebUI/StatsStore_Scraper/Build_Environment/README.md

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# The Build Environment
2+
3+
The build environment directory holds both the example code as well as the extra information on how the StatsStore Scraper works and how to use it. Additionally it holds the information on how to for example create an SMF service and an IPS package if this is relevant for you.
4+
5+
The structure of how the StatsStore Scraper code works can be found in the [StatsStore Scraper structure document](StatsStore_Scraper_Structure.md).
6+
7+
Currently there is only one example use case which allows the StatsStore Scraper to push data into Splunk. In this case the StatsStore Scraper is installed as a package and SMF service. This is done on the source system which means it will use the internal RAD interface to pull data from the StatsStore. More details on this example can be found in the [Splunk example document](Splunk_Example.md).
8+
9+
Copyright (c) 2022 Oracle and/or its affiliates and released under the [Universal Permissive License (UPL)](https://oss.oracle.com/licenses/upl/), Version 1.0

‎StatsStore_WebUI/StatsStore_Scraper/Build_Environment/Splunk_Example.md

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# The Splunk Example
2+
3+
In this Splunk example the StatsStore Scraper script runs on each Oracle Solaris instance that you want to pull data from. The script is installed through a package and because the package comes with and SMF manifest, which makes it an SMF service that can be turned on. This means that SMF will also automatically start it after a reboot or if the script hits an issue. The only thing that needs to be done before the SMF service is enabled is to configure where the script can find the Splunk HTTP Event Collector (HEC) and give it the correct credentials to do so.
4+
5+
## Getting started
6+
7+
There are three ways to get the StatsStore Scraper script installed and running.
8+
9+
* Build the IPS package yourself — The steps for this are explained in the **IPS Manifest and Package** section of the **[StatsStore Scraper structure](./StatsStore_Scraper_Structure.md)** document. The benefit of this approach is that you can choose to incorporate changes into the package before building it and this way customize it for your use.
10+
* If you rather just use the package we built go to the **[Packages](../Packages)** directory and pickup the pre-built package from there.
11+
* Take the Scraper script and just run it as is on its own — I could be that you just want to talk the script, copy it somewhere and run it there without the use of the package and SMF.
12+
13+
If you're going for one of the first two options you'll want to install the package, configure the `server_info.yaml` file, and start the service. The details on this are also in the **[Packages](../Packages)** directory.
14+
15+
16+
17+
## The structure
18+
19+
The structure of the package is fairly simple.
20+
21+
* The working directory is in `/opt/sstore_scraper`, where `/opt/sstore_scraper/bin` holds the Python code, and `/opt/sstore_scraper/etc` holds the config files. For more details on these files see the **Config files** section of the **[StatsStore Scraper structure](./StatsStore_Scraper_Structure.md)** document.
22+
23+
* The SMF manifest is placed in `/lib/svc/manifest/site`, where it can be imported by SMF. The actual import is triggered by `restart_fmri=svc:/system/manifest-import:default` in the package manifest.
24+
25+
Once installed the SMF service is intentionally disabled to allow configuration before it is enabled. If you were to choose to embed all the necessary config data in a self-built package you could choose to have the service automatically enabled. This can be done by changing `<instance name="default" enabled="false"/>` to `<instance name="default" enabled="true"/>` in the SMF manifest file `sstore_scraper.xml`.
26+
27+
That's pretty much it.
28+
29+
Note that this script only pushes data into Splunk we don't have any examples on how to create Dashboards in Splunk. This is out of scope of this example.
30+
31+
## Requirements
32+
33+
The requirements are slightly loose. Of course this will have to run on Oracle Solaris 11.4 as this is the only version that has the StatsStore. The other thing is that the script is written in Python 3, tested on systems where this is Python 3.7.
34+
35+
Also note this was tested on systems SRU 36 and higher, as we moved from Python 2.7 being the default to Python 3.x being the default early SRUs probably won't have all the Python 3.7 packages installed that are needed.
36+
37+
There are a few requirements on Python modules and it may be the case they are not installed with the group package installed on the system. Specifically, the `pyyaml` package as may be missing. Therefore we've added a dependency in the IPS manifest (`sstore_scraper.p5m.3.res`) to install this package incase it's not installed already. If you choose to edit the script and add extra modules check if they are installed in the group packages you use or add their dependency to the IPS manifest too.
38+
39+
The Splunk version this was tested is 8.2.3, but seeing that the HEC interface is stable this should work with a wide variety of Splunk versions.
40+
41+
## Versions
42+
43+
* v0.3 — The initial release.
44+
45+
This version is installed on the source host and connects out to a single Splunk HEC. It can understand and simplify a large amount but not all the StatsStore SSIDs. It is delivered with an SMF manifest and a way to deliver it through an IPS package. This makes install easy and once configured SMF will restart the service on an outage.
46+
47+
* Known elements still missing:
48+
* Python exception handling of Splunk connection errors — It currently doesn't cleanly deal with all the possible connection issues.
49+
* Check for Splunk config — It doesn't check if the `server_info.yaml` config file is not edited after the default install.
50+
* Signing the package — How to sign the package hasn't been added yet.
51+
* More StatsStore SSIDs to choose from in config file — The list of SSIDs in the `sstore_list.yaml` file isn't complete. That is to say it doesn't have all the SSIDs available in the WebUI Sheets yet.
52+
* Support for more SSID types — Not all the SSID operations are understood how to parse the output yet by the script. Currently it understands how to parse a single data field, a `REAL`, and an answer containing a `DICTIONARY`, where there are multiple answers for a single SSID. For example when using a `part.mode`.
53+
54+
Copyright (c) 2022 Oracle and/or its affiliates and released under the [Universal Permissive License (UPL)](https://oss.oracle.com/licenses/upl/), Version 1.0

‎StatsStore_WebUI/StatsStore_Scraper/Build_Environment/StatsStore_Scraper_Structure.md

+179
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# The StatsStore Scraper Structure
2+
3+
This document explains the main elements of the Scraper and how it generally works. It is written in Python 3 and was developed and tested with Python 3.7 on Oracle Solaris 11.4 on both SPARC and x86 systems.
4+
5+
## Main script
6+
7+
The main script had three main functions:
8+
9+
* Pull data from the StatsStore through RAD
10+
* Convert and simplify the data
11+
* Export the data
12+
13+
The following sections will explain each function
14+
15+
### Pull data from the StatsStore through RAD
16+
17+
As the main goal of the StatsStore Scraper is pulling data from the Oracle Solaris StatsStore, the first step has to be to connect to the StatsStore. This can be done either locally on the Oracle Solaris instance over `rad:local` or remotely to other Oracle Solaris instances over their `rad:remote` service. The Scraper script is intended to be able to do either but initially is focused on running on the Oracle Solaris instances and connecting locally. The remote vs local discussion is interesting but for a later time. Both have pro's and con's.
18+
19+
The Scraper script used the RAD/REST service to connect to the StatsStore. To do this it has to pass a JSON datagram to RAD that looks like this:
20+
21+
```json
22+
{
23+
"ssids": [<list of SSIDs>],
24+
"range": {
25+
"range_type": "RANGE_BY_TIME",
26+
"range_by_time": {
27+
"start_ts": -1,
28+
"end_ts": -1,
29+
"step": 1
30+
},
31+
"show_unstable": true,
32+
"show_unbrowsable": true
33+
}
34+
}
35+
```
36+
37+
This will return the values for the SSIDs in the `<list of SSIDs>` at the moment of the request. The `-1` in `"start_ts"` and `"end_ts"` indicate "now".
38+
39+
### Convert and simplify the data
40+
41+
The response will be a JSON datagram with the data in a fairly complex nested form, something that is ideal for programmatic use, like the Oracle Solaris WebUI, but less optimal for passing on to data collection tools that expect all the information about one data point in one line. This is why the Scraper script will take the JSON data and reduce it to the core things it needs: The name of the data, the value of the data, and the timestamp it was collected.
42+
43+
Similarly, the name of the SSIDs is fairly long, maybe confusing, and filled with characters that might trip up the ingest engine. So for example it will allow the conversion from `//:class.cpu//:stat.fpu-usage//:op.rate` to `cpu_fpu-usage_rate`. The exact name conversion can be controlled through the export mechanism as different tools use different conventions.
44+
45+
One other thing the conversion deals with is that some SSIDs will actually generate a few data points and they have to correctly be named after the conversion. So for example `//:class.cpu//:stat.usage//:part.mode(user,kernel,stolen,intr)//:op.rate//:op.util` will partition the CPU usage in `user`, `kernel`, `stolen`, and `intr` (interrupt). The conversion will for example convert this to `cpu_usage_rate_util_user`, `cpu_usage_rate_util_kernel`, `cpu_usage_rate_util_stolen`, and `cpu_usage_rate_util_intr` with all the correct values.
46+
47+
As the script currently doesn't recognize all these mechanisms it's also somewhat limited in which SSID responses it can parse at this point. For example it currently doesn't fully understand the response coming from `//:part.latency` in `//:class.disk//:stat.io-completions//:part.latency//:op.rate` yet. All the "regular" SSIDs should be fine, just be cautious with the partitioning mechanisms.
48+
49+
### Export data
50+
51+
Once the data is converted it can now be exported. The script is intended to allow the export to various tools like InfluxDB, Prometheus, Splunk, and plain text, however this first release can only export the data to a Splunk HTTP Event Collector (HEC).
52+
53+
The mainly, this function does the formatting of the data specific to the tool it's going to be connecting to, and then does the export. In the case of the Splunk HEC the script connects to the HEC over HTTP, authenticates, and pushes the data.
54+
55+
## Config files
56+
57+
The Scraper script uses two different YAML configuration files:
58+
59+
* `sstore_list.yaml` — Which holds the list of SSIDs the Scraper script will pull from the StatsStore on each request. The list is currently populated by the SSIDs of some of the main WebUI Sheets. This list can be changed at will to reflect the stats required, do note that, as mentioned above, not all of the StatsStore operators are understood be the converter.
60+
* `server_info.yaml` — Which hold the configuration of the Scraper itself. It has three sections:
61+
* `servers:` — Where you indicate where to fetch the data. In the current version it only gathers through the local RAD interface which is indicated with `server_port: "local"`, where `localhost:` is just the server name.
62+
* `agent:` — Where right now only the interval time is being configured.
63+
* `destination:` — Where the export function is configured. The `splunk:` section both defines Splunk as the destination type as well as how to connect to the HEC. The `server_endpoint:`, `request_type:`, `request_transport:`, and `request_uri:` are used to define the HTTP URL. `headers:` holds the HEC authorization key needed to authenticate and select the correct metric the data will go into. and `data_template:` used as the base template for the data that goes to the Splunk HEC.
64+
65+
Any changes to either file will require the service to be restarted.
66+
67+
## SMF Manifest
68+
69+
To ensure the StatsStore Scraper script is always running this example encapsulates an SMF manifest that can be used to create an SMF service for the Scraper script. By enabling this service Oracle Solaris will start the script every time the system boots, as well as if there's an issue with the script and it fails. All the logging information will go to log associated with the SMF service too.
70+
71+
Here's an example of the `svcs -lp` output:
72+
73+
```bash
74+
# svcs -lp sstore-scraper
75+
fmri svc:/site/sstore-scraper:default
76+
name site/sstore-scraper
77+
enabled true
78+
state online
79+
next_state none
80+
state_time 2022-09-21T15:06:38
81+
logfile /var/svc/log/site-sstore-scraper:default.log
82+
restarter svc:/system/svc/restarter:default
83+
contract_id 173
84+
manifest /lib/svc/manifest/site/sstore_scraper.xml
85+
dependency require_all/none svc:/milestone/multi-user (online)
86+
process 1266 /usr/bin/python3 /opt/sstore_scraper/bin/sstore_scraper.py
87+
```
88+
89+
The SMF manifest XML file can be found at `build/proto_install/lib/svc/manifest/site/sstore_scraper.xml` in this repository.
90+
91+
## IPS Manifest and Package
92+
93+
In addition to using SMF to make it easy to consistently run the Scraper script this example also shows how to bundle up all the files, create an IPS package, and put it in a repository. It can then either be installed over the network or a `p5p` file can be used to install it from file.
94+
95+
The example supplies the `p5m` file that was used to create the example package. The full process normally includes various steps where the the `pkgsend`, `pkgmogrify`, `pkgdepend`, `pkglint`, and `pkgrepo` commands are used to define the meta data, discover the files, and check for things like dependencies. However there is some amount of post-processing need, for example to prune unnecessary entries and tweak certain lines, so we're only supplying the final `p5m` file which can be found at `build/sstore_scraper.p5m.3.res`. This is all that is needed to build your own package if you keep the package contents the same.
96+
97+
In short to create the package using the files in the `build` directory run the following commands. First create a package repository if you don't already have one:
98+
99+
```bash
100+
# pkgrepo create scraper_repo
101+
# pkgrepo set -s scraper_repo publisher/prefix=scraper
102+
```
103+
104+
Where `scraper_repo` is the name of the directory that will hold the repository. And `publisher/prefix=scraper` defines the name of the IPS publisher that the package will be associated with.
105+
106+
Then create the package in the new repository:
107+
108+
```bash
109+
# pkgsend -s /full/path/to/repository/location/scraper_repo publish -d build/proto_install build/sstore_scraper.p5m.3.res
110+
pkg://scraper/stats/statsstore/[email protected],5.11:20220921T172649Z
111+
PUBLISHED
112+
```
113+
114+
You can check if it made it in correctly:
115+
116+
```bash
117+
# pkgrepo info -s scraper_repo
118+
PUBLISHER PACKAGES STATUS UPDATED
119+
scraper 1 online 2022-09-21T11:01:08.746623Z
120+
# pkg info -g /root/scraper_repo sstore_scraper
121+
Name: stats/statsstore/sstore_scraper
122+
Summary: The StatsStore Scraper pulls data from the Oracle Solaris
123+
StatsStore, converts the data, and exports this into something
124+
like the Splunk HEC
125+
Description: StatsStore Scraper and Exporter
126+
Category: System/Administration and Configuration
127+
State: Not installed
128+
Publisher: scraper
129+
Version: 0.3
130+
Branch: None
131+
Packaging Date: Wed Sep 21 17:26:49 2022
132+
Size: 20.75 kB
133+
FMRI: pkg://scraper/stats/statsstore/[email protected]:20220921T172649Z
134+
```
135+
136+
For more details on the full process see [Creating and Publishing a Package](https://docs.oracle.com/cd/E53394_01/html/E54820/pkgcreate.html).
137+
138+
Then you can either set up your own HTTP based server similar to what is described in [How to Enable Users to Retrieve Packages Using an HTTP Interface](https://docs.oracle.com/cd/E37838_01/html/E60982/scalability.html). Example steps:
139+
140+
```bash
141+
# pkg install package/pkg/depot
142+
Packages to install: 1
143+
Services to change: 1
144+
Create boot environment: No
145+
Create backup boot environment: No
146+
147+
Planning linked: 0/1 done; 1 working: zone:test1
148+
Linked image 'zone:test1' output:
149+
150+
Planning linked: 1/1 done
151+
DOWNLOAD PKGS FILES XFER (MB) SPEED
152+
Completed 1/1 17/17 0.0/0.0 268k/s
153+
154+
Downloading linked: 0/1 done; 1 working: zone:test1
155+
Downloading linked: 1/1 done
156+
PHASE ITEMS
157+
Installing new actions 61/61
158+
Updating package state database Done
159+
Updating package cache 0/0
160+
Updating image state Done
161+
Creating fast lookup database Done
162+
Executing linked: 0/1 done; 1 working: zone:test1
163+
Executing linked: 1/1 done
164+
Updating package cache 3/3
165+
# svccfg -s pkg/server add scraper
166+
# svccfg -s pkg/server:scraper setprop pkg/inst_root=/full/path/to/repository/location/scraper_repo/
167+
# svcadm refresh pkg/server:scraper
168+
# svcadm enable pkg/server:scraper
169+
```
170+
171+
Alternatively you can pull the package into an archive in the form or a `p5p` file:
172+
173+
```bash
174+
# pkgrecv -s /full/path/to/repository/location/scraper_repo_stage/ -a -d sstore_scraper.p5p sstore_scraper
175+
```
176+
177+
This `p5p` file can now be moved to the destination system or put on an NFS share and install directly from it. For steps on how to install the package see the **[Packages folder](../Packages)**.
178+
179+
Copyright (c) 2022 Oracle and/or its affiliates and released under the [Universal Permissive License (UPL)](https://oss.oracle.com/licenses/upl/), Version 1.0

‎StatsStore_WebUI/StatsStore_Scraper/Build_Environment/build/proto_install/lib/svc/manifest/site/sstore_scraper.xml

+56
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
<?xml version="1.0" ?>
2+
<!DOCTYPE service_bundle
3+
SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
4+
<!--
5+
Manifest created by svcbundle (2022-May-25 23:23:37-0700)
6+
Copyright (c) 2022, Oracle and/or its affiliates.
7+
Licensed under the Universal Permissive License v 1.0 as shown
8+
at https://oss.oracle.com/licenses/upl/
9+
-->
10+
<service_bundle name="site/sstore-scraper" type="manifest">
11+
<service name="site/sstore-scraper" version="1" type="service">
12+
<dependency name="multi_user_dependency" grouping="require_all"
13+
restart_on="none" type="service">
14+
<service_fmri value="svc:/milestone/multi-user"/>
15+
</dependency>
16+
17+
<exec_method
18+
name="start"
19+
type="method"
20+
timeout_seconds="120"
21+
exec="/opt/sstore_scraper/bin/sstore_scraper.py">
22+
<method_context>
23+
<method_credential
24+
user='daemon' group='sys'/>
25+
</method_context>
26+
</exec_method>
27+
28+
<exec_method
29+
name="stop"
30+
type="method"
31+
timeout_seconds="60"
32+
exec=":kill" />
33+
34+
<exec_method
35+
name="refresh"
36+
type="method"
37+
timeout_seconds="60"
38+
exec=":true"/>
39+
40+
<instance name="default" enabled="false"/>
41+
42+
<template>
43+
<common_name>
44+
<loctext xml:lang="C">
45+
site/sstore-scraper
46+
</loctext>
47+
</common_name>
48+
<description>
49+
<loctext xml:lang="C">
50+
The site/sstore-scraper service that can be used
51+
to pull data from the StatsStore.
52+
</loctext>
53+
</description>
54+
</template>
55+
</service>
56+
</service_bundle>

0 commit comments

Comments
 (0)