Skip to content

Commit 9d2bfde

Browse files
alphaprinzalphaprinz
committed
Bucket Notifications - doc
Signed-off-by: Amit Prinz Setter <[email protected]>
1 parent 954ed73 commit 9d2bfde

File tree

3 files changed

+138
-0
lines changed

3 files changed

+138
-0
lines changed

Diff for: ‎docs/NooBaaNonContainerized/ConfigFileCustomizations.md

+12
Original file line numberDiff line numberDiff line change
@@ -449,6 +449,18 @@ Warning: After setting this configuration, NooBaa will skip schema validations a
449449
"LOG_TO_STDERR_ENABLED": false
450450
3. systemctl restart noobaa
451451
```
452+
### 30. Notification log directory
453+
* <u>Key</u> `NOTIFICATION_LOG_DIR`
454+
* <u>Type</u> String
455+
* <u>Default</u> empty
456+
* <u>Description</u> Path to directory that will hold pending notifications to be sent,
457+
* <u>Steps</u>
458+
```
459+
1. Open /path/to/config_dir/config.json file.
460+
2. Set the config key -
461+
Example:
462+
"NOTIFICATION_LOG_DIR": "/etc/notif"
463+
3. systemctl restart noobaa
452464
453465
### 31. Prometheus HTTP enable flag -
454466
* <u>Key</u>: `ALLOW_HTTP_METRICS`

Diff for: ‎docs/bucket-notifications.md

+106
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Bucket Notifications
2+
3+
## Bucket Notification Configuration
4+
5+
Bucket's notifications can be configured with the s3api operation [put-bucket-notification-configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-bucket-notification-configuration.html).
6+
Specify notifications under the "TopicConfigurations" field, which is an array of jsons, one for each notification.
7+
A notification json has these fields:
8+
- Id: Mandatory. A unique string identifying the notification configuration.
9+
- Events: Optional. An array of [events](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html) for which the notification is relevant.
10+
If not specified, the notification is relevant for all events.
11+
- TopicArn: The connection file (see below). (To specify a Kafka target topic, see "Kafka Connection Fields" below).
12+
13+
Example for a bucket's notification configuration, in a file:
14+
{
15+
"TopicConfigurations": [
16+
{
17+
"Id": "created_from_s3op",
18+
"TopicArn": "secret-name/connect.json",
19+
"Events": [
20+
"s3:ObjectCreated:*"
21+
]
22+
}
23+
]
24+
}
25+
26+
27+
## Connection File
28+
A connection file contains some fields that specify the target notification server.
29+
The connection file name is specified in TopicArn field of the [notification configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-bucket-notification-configuration.html)
30+
31+
-In a containerized environment, the operator will mount the secrets as file in the core pod (see operator doc for more info).
32+
The mount path is `/etc/notif_connect/<secret-name>/<secret failename>`, and the notification reference the file by the `<secret-name>/<secret filename>` path in TopicArn field.
33+
For example, if seceret was created with:
34+
`kubectl create secret generic notif-secret --from_file connect.json`
35+
Then TopicArn should be `notif-secret/connect.json`.
36+
37+
-In a non-containerized system, you must create the relevant file using the 'connections' CRUD cli.
38+
See the NC cli doc for more info.
39+
Connect file contains a single json with the fields specified below.
40+
41+
### Common Connection Fields
42+
- name: A string identifying the connection (mandatory).
43+
- notification_protocol: One of: http, https, kafka (mandatory).
44+
45+
### Http(s) Connection Fields
46+
- agent_request_object: Value is a JSON that is passed to to nodejs' http(s) agent (mandatory).
47+
Any field supported by nodejs' http(s) agent can be used, specifically 'host' and 'port'.
48+
A full list of options is [here](https://nodejs.org/docs/latest-v22.x/api/http.html#new-agentoptions).
49+
Notable fields include:
50+
- host: hostname (or ip) of external server to receive the notifications
51+
- port: http(s) port on which the external server listens
52+
- timeout: connection timeout in milliseconds.
53+
- local_file_ca - path to CA pem file that signed server's TLS cert (if CA needs to be customized)
54+
- rejectUnauthorized - set to true to accept self-signed certs (useful for testing, do not use in production).
55+
- local_file_cert: path to client's private TLS certificate PEM file.
56+
- local_file_key: path to client's private TLS key PEM file.
57+
58+
- request_options_object: Value is a JSON that is passed to nodejs' http(s) request (optional).
59+
Any field supported by nodejs' http(s) request option can be used, specifically:
60+
-- path: used to specify the url path
61+
-- auth: used for http simple auth. Value for 'auth' is of the syntax: <name>:<passowrd>.
62+
63+
A full list of options is [here](https://nodejs.org/docs/latest-v22.x/api/http.html#httprequesturl-options-callback).
64+
65+
### Kafka Connection Fields
66+
- metadata.broker.list: A CSV list of Kafka brokers (mandatory).
67+
- topic: A topic for the Kafka message (mandatory).
68+
69+
## Event Types
70+
S3 spec lists several events and "sub events".
71+
72+
The list of supported events are:
73+
74+
- s3:ObjectCreated:*
75+
- s3:ObjectCreated:Put
76+
- s3:ObjectCreated:Post
77+
- s3:ObjectCreated:Copy
78+
- s3:ObjectCreated:CompleteMultipartUpload
79+
- s3:ObjectRemoved:*
80+
- s3:ObjectRemoved:Delete
81+
- s3:ObjectRemoved:DeleteMarkerCreated
82+
- s3:ObjectRestore:*
83+
- s3:ObjectRestore:Post
84+
- s3:ObjectRestore:Completed
85+
- s3:ObjectRestore:Delete
86+
- s3:ObjectTagging:*
87+
- s3:ObjectTagging:Put
88+
- s3:ObjectTagging:Delete
89+
- s3:LifecycleExpiration:*
90+
- s3:LifecycleExpiration:Delete
91+
- s3:LifecycleExpiration:DeleteMarkerCreated
92+
93+
## Event Processing and Failure Handling
94+
Once NooBaa finds an event with a relevant notification configuration, the notification
95+
is written to a persistent file.
96+
Location of persistent files is determined by-
97+
- For containerized, the pvc specified in NooBaa Bucket Notification spec (see Operator docs for more info).
98+
- For NC, the env variable NOTIFICATION_LOG_DIR (see NC docs for more info).
99+
100+
Files are processed by-
101+
- For containerized, files are contantly being processed in the background of the core pod.
102+
- For NC, files are processed by running manage_nsfs with 'notification' action.
103+
104+
If a notification fails to be sent to the external server, it will be re-written to a persistent file (assuming the
105+
notification is still configured).
106+
Once this new file is processed, NooBaa will try to re-send the failed notification.

Diff for: ‎src/manage_nsfs/manage_nsfs_help_utils.js

+20
Original file line numberDiff line numberDiff line change
@@ -477,6 +477,19 @@ Usage:
477477
478478
`;
479479

480+
const NOTIFICATION_OPTIONS = `
481+
Help:
482+
483+
'notification' is a noobaa-cli command that will process pending event notifications in the NOTIFICATION_LOG_DIR directory.
484+
It will attempt to send each notification to its respective external server.
485+
Note notifications that fail to be sent will be re-written in NOTIFICATION_LOG_DIR.
486+
487+
Usage:
488+
489+
noobaa-cli notification
490+
491+
`;
492+
480493

481494
const CONNECTION_FLAGS_ADD = `
482495
Help:
@@ -589,6 +602,9 @@ function print_usage(type, action) {
589602
case TYPES.CONNECTION:
590603
print_help_connection(action);
591604
break;
605+
case TYPES.NOTIFICATION:
606+
print_help_notification();
607+
break;
592608
default:
593609
process.stdout.write(HELP + '\n');
594610
process.stdout.write(USAGE.trimStart() + '\n');
@@ -735,6 +751,10 @@ function print_help_connection(action) {
735751
process.exit(0);
736752
}
737753

754+
function print_help_notification(action) {
755+
process.stdout.write(NOTIFICATION_OPTIONS);
756+
}
757+
738758

739759
// EXPORTS
740760
exports.print_usage = print_usage;

0 commit comments

Comments
 (0)