Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve server perf in cross region PULLs #102

Closed
yb01 opened this issue Jul 22, 2022 · 3 comments
Closed

improve server perf in cross region PULLs #102

yb01 opened this issue Jul 22, 2022 · 3 comments
Assignees

Comments

@yb01
Copy link
Collaborator

yb01 commented Jul 22, 2022

Below are the PULL() duration with us east2 to us west2 in AWS cloud. as we can see 25K pull is about 15.8 seconds and 2.3k pull is about 1.9 seconds. we should look to improve it especially for GRS pull() performance is part of the PULL interval from aggregator to the region manager.

**I0722 04:11:07.411558    3239 aggregator.go:99] Total (250000) region node events are pulled successfully in (10) RPs. Pull time: 15.874046389s**
I0722 04:11:07.412735    3239 aggregator.go:106] Total (250000) mini node events are converted successfully with length (250000)
I0722 04:11:10.346653    3239 aggregator.go:118] Event Processor Processed nodes results : true. duration: 2.933892757s
**I0722 04:11:59.718491    3239 aggregator.go:99] Total (23728) region node events are pulled successfully in (10) RPs. Pull time: 1.990173271s**
I0722 04:11:59.718581    3239 aggregator.go:106] Total (23728) mini node events are converted successfully with length (23728)
I0722 04:11:59.981454    3239 aggregator.go:118] Event Processor Processed nodes results : true. duration: 262.852177ms
I0722 04:13:58.251739    3239 aggregator.go:99] Total (999) region node events are pulled successfully in (10) RPs. Pull time: 447.244463ms
I0722 04:13:58.251782    3239 aggregator.go:106] Total (999) mini node events are converted successfully with length (999)
I0722 04:13:58.272274    3239 aggregator.go:118] Event Processor Processed nodes results : true. duration: 20.479347ms

@yb01
Copy link
Collaborator Author

yb01 commented Jul 22, 2022

options we can have

  1. pull same region in multiple routines -- complexity for CRV handling
  2. protobuf -- try for 730
  3. real pagenition support

@yb01
Copy link
Collaborator Author

yb01 commented Jul 25, 2022

from the temp local branch at https://github.com/yb01/global-resource-service/tree/seriliazer-works-2, with protobuf serializer for the initial PULL for the 250K nodes. the pull duration is now reduced to 4.98 seconds from 15.8 seconds with the json serializer.

I0725 03:28:29.136218    3042 aggregator.go:69] Starting goroutine for region: ec2-18-236-122-139.us-west-2.compute.amazonaws.com:9119
I0725 03:28:29.136222    3042 aggregator.go:82] Starting loop pulling nodes from region: ec2-18-236-122-139.us-west-2.compute.amazonaws.com:9119
I0725 03:28:34.217398    3042 aggregator.go:94] Total (250000) region node events are pulled successfully in (10) RPs. pull duration 4.980221517s
I0725 03:28:34.218569    3042 aggregator.go:101] Total (250000) mini node events are converted successfully with length (250000)
I0725 03:28:37.425353    3042 aggregator.go:113] Event Processor Processed nodes results : true. duration: 3.20676529s

@yb01
Copy link
Collaborator Author

yb01 commented Oct 14, 2022

used list/watch now. so close

@yb01 yb01 closed this as completed Oct 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant