Skip to content
This repository was archived by the owner on Dec 5, 2023. It is now read-only.
This repository was archived by the owner on Dec 5, 2023. It is now read-only.

Load-test fails, especially with large number of clients #12

@ghost

Description

I'm seeing that the load-test is rather unstable. Especially when running a larger number of clients (say 20000 to make it fail faster). I think it's more an issue with the orders and user services, but figure I should open this issue against the load-test, since running the load-test is the only way to see this issue.

When running the load test with say 20000 clients. There are two sets of errors I see. The first is a reoccurring log in error. Kind of seems like there's a sequencing issue in the load-test script. The user doesn't get logged in before a checkout is performed? The logs show something like this:

...
Error: User not logged in.
500
Error with log in: true
...

These errors appear as soon as you start running the load-test. It's very hard to miss. The front-end service produces the errors.

The second error that happens is that the front-end service crashes and restarts. It looks to crash because it receives an empty response from the orders service. The issue happens within 5-10 minutes, sometimes faster. It also happens faster when the load-test is executed from a remote machine, not locally on the machine that is running the weavesocks demo.

When this crash happens you see a log that looks something like this:

Received response: "{"error":"hystrix: max concurrency","status_code":500,"status_text":"Internal Server Error"}\n"
/usr/src/app/api/orders/index.js:67
var customerlink = jsonBody._links.customer.href;
^

TypeError: Cannot read property 'customer' of undefined
at Request._callback (/usr/src/app/api/orders/index.js:67:47)
at Request.self.callback (/usr/src/app/node_modules/request/request.js:186:22)
at emitTwo (events.js:87:13)
at Request.emit (events.js:172:7)
at Request. (/usr/src/app/node_modules/request/request.js:1081:10)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at IncomingMessage. (/usr/src/app/node_modules/request/request.js:1001:12)
at IncomingMessage.g (events.js:260:16)
at emitNone (events.js:72:20)
at IncomingMessage.emit (events.js:166:7)

npm ERR! Linux 4.4.36-8-default
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "start"
npm ERR! node v4.2.6
npm ERR! npm v3.5.2
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start: node server.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script 'node server.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the weave-demo-frontend package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node server.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs weave-demo-frontend
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls weave-demo-frontend
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /usr/src/app/npm-debug.log

Note, I did try a newer version of nodeJS and npm, but the error still appears like above. This suggests it's probably an issue with the demo.

To run the services, I used "docker-compose up" in the following directory:
/microservices-demo/deploy/docker-compose

Example command to use with the load-test:
./runLocust.sh -h <host_name> -r 20000 -c 20000

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions