Merge pull request #116 from Cingulara/develop

Final Beta Release v 0.14

README.md

@@ -1,4 +1,4 @@
-# OpenRMF Documentation (v 0.13)
+# OpenRMF Documentation (v 0.14)
 
 ## Introduction to OpenRMF
 OpenRMF is an open source tool for managing, viewing, and reporting of your DoD STIG checklists and Nessus Patch Scans in one web-based interface using your browser. It also generates a compliance listing of all your checklists across a whole system based on NIST 800-53 for your Risk Management Framework (RMF) documentation and process. This tool helps you manage multiple systems going through the RMF process and allows you to structure your data in a clean interface all in one location for your group or program. 
@@ -49,10 +49,8 @@ Read more about its genesis <a href="https://www.cingulara.com/opensource.html"
 - [x] External API access to certain functions in OpenRMF (ext-api-score)
 - [x] Export Compliance Report to XLSX
 - [x] Meaningful Health Checks in APIs and MSG clients
-
-## ToDos (in no particular order)
-- [ ] Performance improvements
-- [ ] Separate Reporting API and Database (MSA)
+- [x] Performance improvements
+- [x] Separate Reporting API and Database (MSA)
 
 If we are missing something you want, please add it on our main <a href="https://github.com/Cingulara/openrmf-web/issues" target="_blank">GitHub Issues</a> page.
 

architecture/README.md

@@ -21,12 +21,14 @@ of chart data and XLSX downloads.
 * https://github.com/Cingulara/openrmf-api-controls is a read-only lookup of NIST controls to match to CCI for the compliance API and other pieces that need to pull the NIST control descriptions for 800-53.
 * https://github.com/Cingulara/openrmf-api-compliance is for generating the compliance listing, matching NIST controls via CCI to 1 or more checklists in a System. This generates a table of controls and the checklists corresponding to the control from the system's group of checklists. The checklist is linked to the Checklist service and color coded by status.
 * https://github.com/Cingulara/openrmf-api-audit is a read-only lookup of Audit information for OpenRMF that only Administrators can access.
+* https://github.com/Cingulara/openrmf-api-reports is a read-only lookup of OpenRMF data for certain reports that use caching and eventual consistency of data (Nessus Patch Report and Host Vulnerability).
 * https://github.com/Cingulara/openrmf-msg-controls is a NATS client for responding to request/reply on a list of all RMF controls or get the information on a specific control (i.e. AC-1).
 * https://github.com/Cingulara/openrmf-msg-compliance is a NATS client for responding to request/reply on a list of all compliance listings mapping STIG vulnerability IDs to controls. Use this for a full listing based on a low/moderate/high level as well as if you are using personally identifiable information (PII) or similar data.
 * https://github.com/Cingulara/openrmf-msg-template is a NATS client for responding to request/reply on a request for a System template based on the title passed in.
 * https://github.com/Cingulara/openrmf-msg-checklist is a NATS client for responding to request/reply on a request for a checklist based on the Mongo DB record Id passed in.
 * https://github.com/Cingulara/openrmf-msg-system is a NATS client for responding to published messages for updating a System based on title, number of checklists, or running a compliance check.
 * https://github.com/Cingulara/openrmf-msg-audit is a NATS client for responding to published messages for recording auditable events through OpenRMF.
+* https://github.com/Cingulara/openrmf-msg-reports is a NATS client for responding to published messages for eventual consistency of OpenRMF data used for reporting.
 
 I started this project with separate microservices all over including messaging for API-to-API communication. Future enhancements are to organically add publish / subscribe pieces such as compliance, auditing, logging, etc. to make this more user and enterprise ready. Along with all the error trapping, checking for NATS connection, etc. that a production 1.0 application would have. 
 
@@ -39,7 +41,9 @@ OpenRMF uses NATS messaging to work eventual consistency as well as API-to-API c
 | openrmf.checklist.read | Request/Reply | Score (Msg Client), Compliance  | openrmf-msg-checklist | Ask for a full checklist/artifact record based on the ID passed in |
 | openrmf.system.checklists.read | Request/Reply | Compliance          | openrmf-msg-checklist | Ask for all checklist records for a given system title passed in |
 | openrmf.checklist.save.new | Subscribe | Upload | openrmf-msg-score | Grab the new uploaded checklist ID sent and generate the score of open, not applicable, not a finding, and not reviewed items across categories |
+| openrmf.checklist.save.new | Subscribe | Upload | openrmf-msg-reports | Grab the new uploaded checklist ID sent and generate the vulnerability data in the reports database, separated out by vulnerability ID |
 | openrmf.checklist.save.update | Subscribe | Upload | openrmf-msg-score | Grab the updated checklist ID sent and generate the score of open, not applicable, not a finding, and not reviewed items across categories |
+| openrmf.checklist.save.update | Subscribe | Upload | openrmf-msg-reports | Grab the new uploaded checklist ID sent and generate the vulnerability data in the reports database, separated out by vulnerability ID while removing the old vulnerability data for that checklist ID |
 | openrmf.checklist.delete | Subscribe | Save | openrmf-msg-score | Delete the score record for the passed in checklist ID  |
 | openrmf.score.read | Subscribe | Read | openrmf-msg-score | Read API calling for the score when generating an XLSX checklist download listing the score. |
 | openrmf.compliance.cci | Request/Reply | Compliance | openrmf-msg-compliance | Send back all CCI to NIST Major Controls listing. |
@@ -52,5 +56,8 @@ OpenRMF uses NATS messaging to work eventual consistency as well as API-to-API c
 | openrmf.system.update.{Id} | Subscribe | Save | openrmf-msg-system | When a system title is updated, make sure all references throughout the checklists are updated. We save the system group Id and the title with the checklists for easier usage throughout OpenRMF. The source-of-truth is the systemgroups collection in MongoDB. |
 | openrmf.system.count.> | Subscribe | Upload (add) and Save (delete) | openrmf-msg-system | Increments with a ".add" at the end of the subject or decrements if there is a ".delete" at the end of the subject. The payload is the system group Id. |
 | openrmf.system.compliance | Subscribe | Compliance | openrmf-msg-system | Stores the date of the last compliance check run into the system group record for display later. |
-
 | openrmf.compliance.cci.references | Request/Reply | Compliance | openrmf-msg-compliance | Passing in the CCI it returns the CCI title and NIST list of references for the CCI passed in to the Compliance API. |
+| openrmf.system.delete | Subscribe | Save | openrmf-msg-reports | Passing in the System Group ID, the reporting data for patch scanning and vulnerabilities are removed from the database. |
+| openrmf.system.patchscan | Subscribe | Save | openrmf-msg-reports | Passing in the System Group ID, the reporting data for patch scanning is pulled from the raw string data in the Artifact database, parsed, put into the right structure, and saved into the report database. |
+| openrmf.report.refresh.nessuspatchdata | Subscribe | Report | openrmf-msg-reports | Issue a command from the GUI as an Administrator to refresh all Nessus Patch Data in every System. |
+| openrmf.report.refresh.vulnerabilitydata | Subscribe | Report | openrmf-msg-reports | Issue a command from the GUI as an Administrator to refresh all Checklist Vulnerability on every checklist in every System. |

deployments/chart/openrmf/Chart.yaml

@@ -13,5 +13,5 @@ maintainers:
   - name: David Gould
     email: [email protected]
 engine: gotpl 
-appVersion: 0.13.03
+appVersion: 0.14.01
 deprecated: false

deployments/chart/openrmf/README.md

@@ -170,4 +170,37 @@ spec:
     - podSelector:
         matchLabels:
           app.kubernetes.io/component: audit-api
+```
+
+## Prometheus Operator
+There is a section in the Helm chart Values.yaml file that is for the Prometheus Operator available at 
+https://github.com/helm/charts/tree/master/stable/prometheus-operator and 
+https://github.com/coreos/prometheus-operator. If you use this in Kubernetes, and I highly recommend that you do,
+you can set the "use" to true and then the matchLabels part of your Prometheus setup you have
+in your cluster can go in here to quickly match the YAML for monitoring. See the 2 URLs for more 
+information.
+
+We have ServiceMonitor definitions for all APIs as well as the NATS messaging container as well so far
+with version 0.14. When you apply that operator you can run something like `kubectl --namespace default get pods -l "release=prometheus-operator-1586292731"` to get the status.  If you are going to run it, set the Values.yaml to
+true for using that operator. And then set the label name and value to use on the Prometheus Service Monitor 
+sections.
+
+You may very well still need to d/l the metrics information for Kubernetes and then run the `kubectl apply` against the metrics folder with all the YAML files. See https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html for more information there. 
+
+OpenRMF Helm Chart v3 Values section is below. Note the information about `kubectl get prometheus -o yaml --all-namespaces` to find the matchLabels section you defaulted or created so the ServiceMonitor pieces are lined up well. 
+
+```
+# do you use this operator
+useprometheusoperator: true
+
+# The label name and value in name: value setup to match what the Prometheus Operator is using
+# for it to know to pick up and use the ServiceMonitor setup
+# To find this, run 'kubectl get prometheus -o yaml --all-namespaces' and run through the YAML
+# Look for something like this that will show how it matches labels, and use that label setup
+#     serviceMonitorSelector:
+#       matchLabels:
+#         release: prometheus-operator-1586292731
+
+servicemonitormatchlabelname: release
+servicemonitormatchlabelvalue: prometheus-operator-1586292731
 ```

deployments/chart/openrmf/templates/audit.yaml

@@ -167,4 +167,32 @@ spec:
     run: openrmf-audit
   sessionAffinity: None
   type: {{.Values.serviceTypeAWS | default "LoadBalancer"}}
-{{ end }}
+{{ end }}
+{{ if .Values.useprometheusoperator }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: openrmf-audit-api-metrics
+  namespace: {{.Values.namespace}}
+  labels:
+    app.kubernetes.io/name: openrmf
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+    app.kubernetes.io/component: audit-api
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/revision: "{{ .Release.Revision }}"
+    app.kubernetes.io/version: {{ .Chart.AppVersion }}
+    app.kubernetes.io/managed-by: helm
+    {{ .Values.servicemonitormatchlabelname }}: {{ .Values.servicemonitormatchlabelvalue }}
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/component: audit-api
+  endpoints:
+  - port: https
+    interval: 15s
+    path: /metrics
+  namespaceSelector:
+    matchNames:
+    - {{.Values.namespace}}
+{{ end }}

deployments/chart/openrmf/templates/auditdb.yaml

@@ -38,6 +38,7 @@ data:
     db.Audits.createIndex({ created: -1 })
     db.Audits.createIndex({ username: 1 })
     db.Audits.createIndex({ program: 1 })
+    db.Audits.createIndex({ action: 1 })
 ---
 apiVersion: apps/v1
 kind: Deployment

deployments/chart/openrmf/templates/checklistdb.yaml

@@ -37,6 +37,8 @@ data:
     db.createCollection("Artifacts");
     db.Artifacts.createIndex({ systemGroupId: 1 })
     db.Artifacts.createIndex({ stigType: 1 })
+    db.Artifacts.createIndex({ stigRelease: 1 })
+    db.Artifacts.createIndex({ version: 1 })
     db.createCollection("SystemGroups");
     db.SystemGroups.createIndex({ title: 1 })
 ---

deployments/chart/openrmf/templates/compliance.yaml

@@ -159,4 +159,32 @@ spec:
     run: openrmf-compliance
   sessionAffinity: None
   type: {{.Values.serviceTypeAWS | default "LoadBalancer"}}
-{{ end }}
+{{ end }}
+{{ if .Values.useprometheusoperator }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: openrmf-compliance-api-metrics
+  namespace: {{.Values.namespace}}
+  labels:
+    app.kubernetes.io/name: openrmf
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+    app.kubernetes.io/component: compliance-api
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/revision: "{{ .Release.Revision }}"
+    app.kubernetes.io/version: {{ .Chart.AppVersion }}
+    app.kubernetes.io/managed-by: helm
+    {{ .Values.servicemonitormatchlabelname }}: {{ .Values.servicemonitormatchlabelvalue }}
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/component: compliance-api
+  endpoints:
+  - port: https
+    interval: 15s
+    path: /metrics
+  namespaceSelector:
+    matchNames:
+    - {{.Values.namespace}}
+{{ end }}

deployments/chart/openrmf/templates/controls.yaml

@@ -160,4 +160,32 @@ spec:
     run: openrmf-controls
   sessionAffinity: None
   type: {{.Values.serviceTypeAWS | default "LoadBalancer"}}
-{{ end }}
+{{ end }}
+{{ if .Values.useprometheusoperator }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: openrmf-controls-api-metrics
+  namespace: {{.Values.namespace}}
+  labels:
+    app.kubernetes.io/name: openrmf
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+    app.kubernetes.io/component: controls-api
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/revision: "{{ .Release.Revision }}"
+    app.kubernetes.io/version: {{ .Chart.AppVersion }}
+    app.kubernetes.io/managed-by: helm
+    {{ .Values.servicemonitormatchlabelname }}: {{ .Values.servicemonitormatchlabelvalue }}
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/component: controls-api
+  endpoints:
+  - port: https
+    interval: 15s
+    path: /metrics
+  namespaceSelector:
+    matchNames:
+    - {{.Values.namespace}}
+{{ end }}

deployments/chart/openrmf/templates/nats.yaml

@@ -134,4 +134,32 @@ spec:
   selector:
     run: openrmf-natspromexporter
   sessionAffinity: None
-  clusterIP: None
+  clusterIP: None
+{{ if .Values.useprometheusoperator }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: openrmf-nats-metrics
+  namespace: openrmf
+  labels:
+    app.kubernetes.io/name: openrmf
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+    app.kubernetes.io/component: nats-server-2.1.2
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/revision: "{{ .Release.Revision }}"
+    app.kubernetes.io/version: {{ .Chart.AppVersion }}
+    app.kubernetes.io/managed-by: helm
+    {{ .Values.servicemonitormatchlabelname }}: {{ .Values.servicemonitormatchlabelvalue }}
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/component: openrmf-natspromexporter
+  endpoints:
+  - port: natspromexporter
+    interval: 15s
+    path: /metrics
+  namespaceSelector:
+    matchNames:
+    - openrmf
+{{ end }}

deployments/chart/openrmf/templates/read.yaml

@@ -170,4 +170,32 @@ spec:
     run: openrmf-read
   sessionAffinity: None
   type: {{.Values.serviceTypeAWS | default "LoadBalancer"}}
-{{ end }}
+{{ end }}
+{{ if .Values.useprometheusoperator }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: openrmf-read-api-metrics
+  namespace: {{.Values.namespace}}
+  labels:
+    app.kubernetes.io/name: openrmf
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+    app.kubernetes.io/component: read-api
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/revision: "{{ .Release.Revision }}"
+    app.kubernetes.io/version: {{ .Chart.AppVersion }}
+    app.kubernetes.io/managed-by: helm
+    {{ .Values.servicemonitormatchlabelname }}: {{ .Values.servicemonitormatchlabelvalue }}
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/component: read-api
+  endpoints:
+  - port: https
+    interval: 15s
+    path: /metrics
+  namespaceSelector:
+    matchNames:
+    - {{.Values.namespace}}
+{{ end }}

deployments/chart/openrmf/templates/save.yaml

@@ -168,3 +168,31 @@ spec:
   sessionAffinity: None
   type: {{.Values.serviceTypeAWS | default "LoadBalancer"}}
 {{ end }}
+{{ if .Values.useprometheusoperator }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: openrmf-save-api-metrics
+  namespace: {{.Values.namespace}}
+  labels:
+    app.kubernetes.io/name: openrmf
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+    app.kubernetes.io/component: save-api
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/revision: "{{ .Release.Revision }}"
+    app.kubernetes.io/version: {{ .Chart.AppVersion }}
+    app.kubernetes.io/managed-by: helm
+    {{ .Values.servicemonitormatchlabelname }}: {{ .Values.servicemonitormatchlabelvalue }}
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/component: save-api
+  endpoints:
+  - port: https
+    interval: 15s
+    path: /metrics
+  namespaceSelector:
+    matchNames:
+    - {{.Values.namespace}}
+{{ end }}

deployments/chart/openrmf/templates/scoredb.yaml

@@ -38,6 +38,7 @@ data:
     db.Scores.createIndex({ artifactId: 1 })
     db.Scores.createIndex({ systemGroupId: 1 })
     db.Scores.createIndex({ hostName: 1 })
+    db.Scores.createIndex({ stigType: 1 })    
 ---
 apiVersion: apps/v1
 kind: Deployment

deployments/chart/openrmf/templates/scoring.yaml

@@ -169,4 +169,32 @@ spec:
     run: openrmf-scoring
   sessionAffinity: None
   type: {{.Values.serviceTypeAWS | default "LoadBalancer"}}
-{{ end }}
+{{ end }}
+{{ if .Values.useprometheusoperator }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: openrmf-scoring-api-metrics
+  namespace: {{.Values.namespace}}
+  labels:
+    app.kubernetes.io/name: openrmf
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+    app.kubernetes.io/component: scoring-api
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/revision: "{{ .Release.Revision }}"
+    app.kubernetes.io/version: {{ .Chart.AppVersion }}
+    app.kubernetes.io/managed-by: helm
+    {{ .Values.servicemonitormatchlabelname }}: {{ .Values.servicemonitormatchlabelvalue }}
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/component: scoring-api
+  endpoints:
+  - port: https
+    interval: 15s
+    path: /metrics
+  namespaceSelector:
+    matchNames:
+    - {{.Values.namespace}}
+{{ end }}