#967: feature/encrypt message at rest #971

ercsnmrs · 2025-08-31T14:17:56Z

Key Features

Optional: Encryption can be disabled if not needed
Transparent: No changes to client API required
Secure: Uses AES-256-GCM with random nonces
Configurable: Via config file or command line
Migration Support: Tools to convert existing installations
Performance Optimized: Only encrypts message content field
Error Handling: Graceful fallback if encryption fails

Security Considerations:

Message content only: Metadata remains unencrypted for search/indexing
Database admins: Can still see message metadata but not content
Key management: Critical - keys must be stored securely
Performance: Minimal overhead (~1-5ms per message)
Recovery: Lost keys mean lost message content

[#967]

ercsnmrs · 2025-08-31T14:24:27Z

I will post the test results + load test.

or-else · 2025-08-31T14:39:24Z

Do you want me to review this now or later, after the tests?

ercsnmrs · 2025-08-31T14:40:38Z

@or-else please do after my testing.

ercsnmrs · 2025-08-31T14:57:16Z

@or-else thinking about it, I would appreciate early feedbacks, we can discuss more technical details afterwards.

docker/tinode/config.template

or-else · 2025-09-01T05:30:41Z

keygen/README.md


+**Encryption Key Generation:**
+
+ * `encryption`: Generate encryption key instead of API key.


Likewise, 'encryption' is too generic. I would use something like above: encypt_at_rest or store_encrypt or some such.

I don't see any change

keygen/README.md

server/store/store.go

or-else · 2025-09-01T07:06:18Z

server/store/store.go

+
+	// Encrypt message content if encryption is enabled
+	if encryptionService != nil && encryptionService.IsEnabled() {
+		encryptedContent, err := encryptionService.EncryptContent(msg.Content)


if encryptedContent, err := encryptionService.EncryptContent(msg.Content); err != nil { ...

No, not really.

or-else · 2025-09-01T07:08:24Z

server/store/store.go

-	return adp.MessageGetAll(topic, forUser, opt)
+	messages, err := adp.MessageGetAll(topic, forUser, opt)
+	if err != nil {
+		return nil, err


Return messages, err rather than nil, err. No need to change the logic here.

I don't see the change.

server/store/store.go

or-else · 2025-09-01T07:13:41Z

server/tools/encrypt_messages.go

+	}
+
+	// Get messages for the specified topic
+	messages, err := store.Store.GetAdapter().MessageGetAll(*topicName, types.ZeroUid, nil)


This is not going to work. You need a new method in the adapter interface for this.

Please do not say done. This is still completely not going to work. If you need help, please ask.

server/store/encryption.go

Please don't do this.

or-else · 2025-09-15T06:58:32Z

Any update on this?

or-else · 2025-09-19T15:36:20Z

Closing as abandoned

ercsnmrs · 2025-09-19T15:38:25Z

@or-else let me open another mr on this one, been a though week. thanks for your input.

or-else · 2025-09-19T15:47:44Z

If you want to continue, please use this PR. I reopened it. Thanks.

ercsnmrs · 2025-09-19T15:48:56Z

Appreciate it thanks!

Key Features - Optional: Encryption can be disabled if not needed - Transparent: No changes to client API required - Secure: Uses AES-256-GCM with random nonces - Configurable: Via config file or command line - Migration Support: Tools to convert existing installations - Performance Optimized: Only encrypts message content field - Error Handling: Graceful fallback if encryption fails Security Considerations: - Message content only: Metadata remains unencrypted for search/indexing - Database admins: Can still see message metadata but not content - Key management: Critical - keys must be stored securely - Performance: Minimal overhead (~1-5ms per message) - Recovery: Lost keys mean lost message content [tinode#967]

or-else · 2025-09-20T05:14:33Z

Could you please respond to each of my comments with done (if resolved) or not done and an explanation why you think it's not a good idea. This way I would know what to look for in the next code review. Thanks!

ercsnmrs · 2025-09-20T05:15:54Z

@or-else sure, currently testing the changes. thanks!

… validation - Rename generic 'encryption' config to 'encrypt_at_rest' for clarity - Remove redundant 'enabled' field - key presence determines encryption - Support all AES key sizes (16, 24, 32 bytes) instead of just 32-byte keys - Simplify EncryptionService to MessageEncryptionService with cleaner API - Use []byte fields in EncryptedContent for automatic base64 conversion - Fix store initialization order: command line flags override config file - Update keygen tool with proper AES key size validation - Remove output file option from keygen (use shell redirection instead) - Fix encrypt_messages tool to use proper store interface methods - Add nil content handling in EncryptContent method - Update all tests to work with new MessageEncryptionService API - Improve error handling and method visibility throughout [tinode#967]

ercsnmrs · 2025-09-20T07:32:22Z

Testing Results:

Steps:

Generated and added the encrypted_at_rest config on tinode.conf file

Send Message 1 - Testing User

Result: Encrypted = true

Send Message 2 - second encrypted message

Result: Encrypted = true

Revert Config from Step 1

Result: Encrypted = false

Some observation:

I need to refresh the webapp to display the chat
Needs thorough testing

CC: @or-else

or-else · 2025-09-20T16:21:21Z

keygen/README.md


+**Encryption Key Generation:**
+
+ * `encryption`: Generate encryption key instead of API key.


I don't see any change

or-else · 2025-09-20T16:22:10Z

keygen/README.md

+
+```sh
+# Generate 32-byte encryption key (AES-256)
+./keygen -encryption


Please rename for consistency to be less generic.

or-else · 2025-09-20T16:22:25Z

keygen/README.md

+./keygen -encryption
+
+# Generate 16-byte encryption key (AES-128)
+./keygen -encryption -keysize 16


Please rename encryption for consistency to be less generic.

or-else · 2025-09-20T16:23:33Z

keygen/README.md

+./keygen -encryption -keysize 16
+
+# Generate 24-byte encryption key (AES-192)
+./keygen -encryption -keysize 24


This example is probably unnecessary. Any reasonable person can guess that specifying -keysize 24 would mean a 24-byte key.

or-else · 2025-09-20T16:24:14Z

keygen/keygen.go

 	hmacSalt := flag.String("salt", "", "HMAC salt, 32 random bytes base64-encoded")

+	// Encryption key generation flags
+	encryptionKey := flag.Bool("encryption", false, "Generate encryption key instead of API key")


Please rename encryption for consistency to be less generic.

or-else · 2025-09-20T16:59:30Z

server/store/store.go

 	msg.SetUid(Store.GetUid())
+
+	// Encrypt message content if encryption is enabled
+	if messageEncryptionService != nil && messageEncryptionService.IsEnabled() {


The messageEncryptionService != nil is redundant. You already check for it in messageEncryptionService.IsEnabled(). This is Go, the nil pointer is not going to be dereferenced for calling IsEnabled().

or-else · 2025-09-20T17:00:40Z

server/store/store.go

-	return adp.MessageGetAll(topic, forUser, opt)
+	messages, err := adp.MessageGetAll(topic, forUser, opt)
+	if err != nil {
+		return nil, err


I don't see the change.

or-else · 2025-09-20T17:01:06Z

server/store/store.go

+	}
+
+	// Decrypt message content if encryption is enabled
+	if messageEncryptionService != nil && messageEncryptionService.IsEnabled() {


The messageEncryptionService != nil is redundant.

or-else · 2025-09-20T17:02:25Z

server/tools/README.md

@@ -0,0 +1,131 @@
+# Encryption Tools


Please reduce duplication. Copy-pasting saves time short-term but creates problems in the long run. It requires multiple documents to be updated for every change.

or-else · 2025-09-20T17:05:26Z

server/tools/encrypt_messages.go

+	}
+
+	// Get messages for the specified topic using the store interface
+	messages, err := store.Messages.GetAll(*topicName, types.ZeroUid, nil)


As I said before, this is totally not going to work. Messages.GetAll is not suitable for the task.
For one, Messages.GetAll will return just 1024 messages from the topic.

This part of code in present form is not useful at all. If you don't know how to fix it, please talk to me or just remove it. This is just going to confuse users and won't do anything useful.

You need to create a new DB method to paginate over all messages in the DB. Get a page (10K messages?), encrypt/decrypt them, write back, then do the next page and so on.

or-else · 2025-09-20T17:29:11Z

server/tools/encrypt_messages.go

+		keyFile    = flag.String("key", "", "Path to file containing base64-encoded 32-byte encryption key")
+		key        = flag.String("key_string", "", "Base64-encoded 32-byte encryption key as string")
+		dryRun     = flag.Bool("dry_run", false, "Show what would be done without making changes")
+		reverse    = flag.Bool("reverse", false, "Decrypt encrypted messages (use with caution)")


Why not call this flag decrypt?

or-else · 2025-09-20T17:29:27Z

server/tools/encrypt_messages.go

+		key        = flag.String("key_string", "", "Base64-encoded 32-byte encryption key as string")
+		dryRun     = flag.Bool("dry_run", false, "Show what would be done without making changes")
+		reverse    = flag.Bool("reverse", false, "Decrypt encrypted messages (use with caution)")
+		topicName  = flag.String("topic", "", "Topic name to process (leave empty to process all accessible topics)")


Doing it topic-by-topic is not useful.

or-else · 2025-09-20T17:31:03Z

server/tools/encrypt_messages.go

+	}
+
+	// Get encryption key
+	var encryptionKey []byte


As I said before, the key from the config file will override the command line key. That should be exacly inverse: the command line should override the configFile.

or-else · 2025-09-20T17:33:27Z

server/tools/encrypt_messages.go

+
+		if !*dryRun {
+			// Save the modified message using the store interface
+			err, _ := store.Messages.Save(&msg, nil, false)


This will insert a new message into the DB.

or-else · 2025-09-20T17:47:46Z

Please move server/tools/encrypt_messages.go to another PR. It will require a lot of work. It can be done later.

ercsnmrs · 2025-10-10T06:41:48Z

@or-else i will continue on this, will push a change tomorrow oct 10. please keep it open. Thanks!

or-else reviewed Sep 1, 2025

View reviewed changes

server/store/encryption.go Show resolved Hide resolved

mielen636-ui previously approved these changes Sep 5, 2025

View reviewed changes

or-else changed the base branch from master to devel September 5, 2025 10:35

or-else closed this Sep 19, 2025

or-else reopened this Sep 19, 2025

ercsnmrs force-pushed the feature/encrypt-message-at-rest branch 2 times, most recently from 5d3ce78 to 157476a Compare September 20, 2025 05:01

ercsnmrs force-pushed the feature/encrypt-message-at-rest branch from 157476a to 4ae0494 Compare September 20, 2025 07:25

ercsnmrs marked this pull request as ready for review September 20, 2025 07:46

or-else reviewed Sep 20, 2025

View reviewed changes

ercsnmrs marked this pull request as draft September 21, 2025 03:57


		Encryption Key Generation:

		* `encryption`: Generate encryption key instead of API key.

#967: feature/encrypt message at rest #971

Are you sure you want to change the base?

#967: feature/encrypt message at rest #971

Uh oh!

Conversation

ercsnmrs commented Aug 31, 2025

Uh oh!

ercsnmrs commented Aug 31, 2025

Uh oh!

or-else commented Aug 31, 2025

Uh oh!

ercsnmrs commented Aug 31, 2025

Uh oh!

ercsnmrs commented Aug 31, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

or-else commented Sep 15, 2025

Uh oh!

or-else commented Sep 19, 2025

Uh oh!

ercsnmrs commented Sep 19, 2025

Uh oh!

or-else commented Sep 19, 2025

Uh oh!

ercsnmrs commented Sep 19, 2025

Uh oh!

or-else commented Sep 20, 2025

Uh oh!

ercsnmrs commented Sep 20, 2025

Uh oh!

ercsnmrs commented Sep 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

or-else Sep 20, 2025 •

edited

Loading