Skip to content

Fix: Resolve protobuf generation bloat with sparse mode filtering #1525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Fix: Resolve protobuf generation bloat with sparse mode filtering #1525

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
13c02d5
fix: improve THORChain genMsgs.sh script robustness and error handling
Naq302 Aug 26, 2025
2b64e59
Merge branch 'master' into fix/improve-genmsg-scripts
Naq302 Sep 5, 2025
e0cc58c
Updates to scripts and reverted protos
Naq302 Sep 5, 2025
8e0eead
Merge remote-tracking branch 'origin/master' into fix/improve-genmsg-…
Naq302 Sep 5, 2025
a079d62
resolve protobuf generation bloat with sparse mode filtering
Naq302 Sep 5, 2025
cb614fe
Code Rabbit Feedback
Naq302 Sep 5, 2025
1ca3399
Reduced clone time and adding back in coin for thorchain. wip
Naq302 Sep 5, 2025
4f9cad6
Fix THORChain protobuf bloat using minimal common.proto
Naq302 Sep 5, 2025
4d2ddf2
CR feedback
Naq302 Sep 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 104 additions & 9 deletions packages/xchain-mayachain/genMsgs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,126 @@
#!/bin/bash

# this script checks out mayanode master and generates the proto3 typescript buindings for MsgDeposit and MsgSend
# This script updates MAYAChain Protobuf bindings for MsgDeposit and MsgSend
set -euo pipefail # Exit on error, undefined variables, and pipe failures

# Check for required tools
for tool in yarn git curl sed; do
if ! command -v "$tool" &>/dev/null; then
echo "Error: Required tool '$tool' is not installed"
exit 1
fi
done

MSG_COMPILED_OUTPUTFILE=src/types/proto/MsgCompiled.js
MSG_COMPILED_TYPES_OUTPUTFILE=src/types/proto/MsgCompiled.d.ts

# Ensure output directory exists
mkdir -p "$(dirname "$MSG_COMPILED_OUTPUTFILE")"

TMP_DIR=$(mktemp -d)

# Cleanup function
cleanup() {
if [ -d "$TMP_DIR" ]; then
tput setaf 2
echo "Cleaning up $TMP_DIR"
tput sgr0
Comment on lines +25 to +27
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Make tput non-fatal under set -e (mirrors Thorchain fix).

-    tput setaf 2
+    tput setaf 2 || true
-    tput sgr0
+    tput sgr0 || true
@@
-tput setaf 2
+tput setaf 2 || true
@@
-tput sgr0
+tput sgr0 || true
@@
-tput setaf 2
+tput setaf 2 || true
@@
-tput sgr0
+tput sgr0 || true
@@
-tput setaf 2
+tput setaf 2 || true
@@
-tput sgr0
+tput sgr0 || true

Also applies to: 31-33, 55-57, 71-72, 89-91, 113-115

🤖 Prompt for AI Agents 
In packages/xchain-mayachain/genMsgs.sh around lines 14-16 (and also update
occurrences at 31-33, 55-57, 71-72, 89-91, 113-115), tput calls are currently
fatal when running under set -e; make each tput invocation non-fatal by ensuring
it cannot cause the script to exit (for example append a noop fallback like "||
true" or guard with a command check), and update every listed tput call
consistently so color output attempts fail silently when tput is unavailable or
errors.

rm -rf "$TMP_DIR"
fi
}
trap cleanup EXIT

tput setaf 2
echo "Checking out https://gitlab.com/mayachain/mayanode to $TMP_DIR"
tput sgr0
if ! (cd "$TMP_DIR" && git clone --depth 1 --single-branch https://gitlab.com/mayachain/mayanode); then
echo "Error: Failed to clone mayanode repository"
exit 1
fi

# Verify proto files exist
tput setaf 2
echo "Checking out https://gitlab.com/mayachain/mayanode to $TMP_DIR"
echo "Checking proto files"
tput sgr0
(cd $TMP_DIR && git clone https://gitlab.com/mayachain/mayanode)
MISSING_FILES=0
for proto_file in \
"$TMP_DIR/mayanode/proto/mayachain/v1/common/common.proto" \
"$TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_deposit.proto" \
"$TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_send.proto"; do
if [ ! -f "$proto_file" ]; then
echo "Error: $(basename "$proto_file") missing"
MISSING_FILES=1
else
echo "✓ $(basename "$proto_file") found"
fi
done

# Generate msgs
if [ $MISSING_FILES -eq 1 ]; then
echo "Error: Required proto files are missing"
exit 1
fi

# Download cosmos/base/v1beta1/coin.proto from cosmossdk if not exists
COSMOS_COIN_PROTO="$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto"
if [ ! -f "$COSMOS_COIN_PROTO" ]; then
tput setaf 2
echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
tput sgr0
mkdir -p "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1"
if ! curl -fSL --retry 3 --retry-delay 2 -o "$COSMOS_COIN_PROTO" \
"https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"; then
echo "Error: Failed to download cosmos coin.proto"
exit 1
fi
echo "✓ Downloaded cosmos coin.proto"
else
echo "✓ cosmos coin.proto already exists"
fi

# Generate Protobuf JS bindings using sparse mode to only include referenced types
tput setaf 2
echo "Generating $MSG_COMPILED_OUTPUTFILE"
echo "Generating $MSG_COMPILED_OUTPUTFILE (using sparse mode to avoid bloat)"
tput sgr0
yarn run pbjs -w commonjs -t static-module $TMP_DIR/mayanode/proto/mayachain/v1/common/common.proto $TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_deposit.proto $TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_send.proto $TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto -o $MSG_COMPILED_OUTPUTFILE
if ! yarn pbjs -w commonjs -t static-module --sparse \
-p "$TMP_DIR/mayanode/proto" \
-p "$TMP_DIR/mayanode/third_party/proto" \
"$TMP_DIR/mayanode/proto/mayachain/v1/common/common.proto" \
"$TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_deposit.proto" \
"$TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_send.proto" \
-o "$MSG_COMPILED_OUTPUTFILE" 2>pbjs_errors.txt; then
echo "Error: Failed to generate JavaScript bindings"
cat pbjs_errors.txt
exit 1
fi

# Fix import to be ESM-compatible (no omitted file extension)
sed -i -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE"

Comment on lines 96 to 98
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

BSD/GNU sed portability: use -i.bak to support macOS.

-sed -i -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE"
+sed -i.bak -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE" && rm -f "$MSG_COMPILED_OUTPUTFILE.bak"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Fix import to be ESM-compatible (no omitted file extension)
sed -i -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE"
# Fix import to be ESM-compatible (no omitted file extension)
sed -i.bak -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE" && \
rm -f "$MSG_COMPILED_OUTPUTFILE.bak"
🤖 Prompt for AI Agents 
In packages/xchain-mayachain/genMsgs.sh around lines 96 to 98, the sed -i usage
is not portable to macOS; update the sed invocation to use the backup-style flag
(e.g., replace -i with -i.bak) so it works with BSD sed, and after running sed
remove the generated .bak backup file (remove the specific backup for
MSG_COMPILED_OUTPUTFILE) to avoid leaving stray files.

# Generate TypeScript definitions
tput setaf 2
echo "Generating $MSG_COMPILED_TYPES_OUTPUTFILE"
tput sgr0
yarn run pbts $MSG_COMPILED_OUTPUTFILE -o $MSG_COMPILED_TYPES_OUTPUTFILE
if ! yarn pbts --name types "$MSG_COMPILED_OUTPUTFILE" -o "$MSG_COMPILED_TYPES_OUTPUTFILE" 2>pbts_errors.txt; then
echo "Error: Failed to generate TypeScript definitions"
cat pbts_errors.txt
exit 1
fi

# Verify generated files
if [ ! -f "$MSG_COMPILED_OUTPUTFILE" ] || [ ! -s "$MSG_COMPILED_OUTPUTFILE" ]; then
echo "Error: Generated JavaScript file is missing or empty"
exit 1
fi

if [ ! -f "$MSG_COMPILED_TYPES_OUTPUTFILE" ] || [ ! -s "$MSG_COMPILED_TYPES_OUTPUTFILE" ]; then
echo "Error: Generated TypeScript definitions file is missing or empty"
exit 1
fi

# Clean up error files if they're empty
[ ! -s pbjs_errors.txt ] && rm -f pbjs_errors.txt
[ ! -s pbts_errors.txt ] && rm -f pbts_errors.txt

tput setaf 2
echo "Removing $TMP_DIR/mayanode"
echo "✓ Successfully generated protobuf bindings"
tput sgr0
rm -rf $TMP_DIR
Loading