Add stub files for project-specific configuration and update README

README.md

@@ -51,7 +51,11 @@ All of the following steps take place in the same cf space where the logshipper
     cf logs fluentbit-drain --recent
     ```
 
-7. If you are using an egress proxy, set the $HTTPS_PROXY variable. (TODO; current .profile assumes a $PROXYROUTE in the app's env)
+7. If you are using an egress proxy, set the PROXYROUTE variable and restage the app.
+    ```sh
+    cf set-env fluentbit-drain PROXYROUTE 'https://user:[email protected]:61443'
+    cf restage fluentbit-drain
+    ```
 
 At this point you should have a running app, but nothing is sending logs to it.
 
@@ -76,21 +80,34 @@ The `drain-type=all` query parameter tells Cloud Foundry to send both logs and m
 
 Logs should begin to flow after a short delay. You will be able to see traffic hitting the fluent-bit app's web server. The logshipper uses New Relic's Logs API to transfer individual log entries as it processes them. For s3, it batches log entries into files that are transferred to the s3 bucket when they reach a certain size (default 50M) or when the upload timeout period (default 10 minutes) has passed.
 
+## Additional Configuration
+
+You can supplement the default configuration by overwriting the files in the project_conf directory. This directory contains "stub" files that are already referred to from fluentbit.conf. For example:
+
+   ```sh
+   git clone --depth 1  https://github.com/GSA-TTS/cg-logshipper
+   cp parsers.conf fluentbit.conf cg-logshipper/project_conf
+   cd cg-logshipper
+   cf push
+   ```
+
 ## Status
 
 - Can run `cf push` and see fluentbit running with the supplied configuration
 - We have tested with a legit NR license key and seen logs appearing in NR.
 - Input configured to accept logs from a cf log-drain service.
 - Web server accepts HTTP request and proxies them to fluent-bit (using TCP).
   - Web server requires HTTP basic auth.
-- Look for and use `HTTPS_PROXY` for egress connections (New Relic's plugin provides this).
+- Looks for and uses PROXYROUTE environment var and uses it as HTTPS_PROXY for egress connections.
 
 ### TODO
 
 - Maybe restrict incoming traffic to cloud.gov egress ranges (52.222.122.97/32, 52.222.123.172/32)?
 - Document parsing of logs, maybe add examples for parsing common formats.
 - Port over all the [`datagov-logstack`](https://github.com/GSA/datagov-logstack) utility scripts for registering drains on apps/spaces
 - Add tests?
+- Add a --branch argument to the example under *Additional Configuration*, once we have a tagged release.
+- Change the method of getting PROXYROUTE to a user-provided service, rather than cf env. 
 
 ## Contributing
 

fluentbit.conf

@@ -3,6 +3,7 @@
     log_level    info
     parsers_file parsers.conf
     parsers_file /home/vcap/deps/0/apt/etc/fluent-bit/parsers.conf
+    parsers_file project_conf/parsers.conf
     plugins_File plugins.conf
 
 [INPUT]
@@ -39,11 +40,6 @@
     licenseKey ${NEW_RELIC_LICENSE_KEY}
     endpoint ${NEW_RELIC_LOGS_ENDPOINT}
 
-# # Uncomment to see the parsed messages in the logshipper's logs:
-# [OUTPUT]
-#     name stdout
-#     match *
-
 ### Filters run in order of appearance ###
 
 # Initial pass at parsing the body of the request.
@@ -102,3 +98,5 @@
     time_as_table On
     script scripts/parse_keys_with_eq_pairs.lua
     call parse_keys_with_eq_pairs
+
+@INCLUDE project_conf/fluentbit.conf

project_conf/fluentbit.conf

@@ -0,0 +1,11 @@
+## Overwrite this file with any additional directives you want to add to
+## the fluentbit configuration. This file is included at the end of fluentbit.conf.
+##
+## Any filters you add will be run after the filters in fluentbit.conf. You can also
+## add [INPUT] or [OUTPUT] sections.
+
+
+# # Uncomment to see the parsed messages in the logshipper's logs:
+# [OUTPUT]
+#     name stdout
+#     match *

project_conf/parsers.conf

@@ -0,0 +1 @@
+## Overwrite this file with any additional parsers for your project.