-
Notifications
You must be signed in to change notification settings - Fork 0
Coverage Testing
Christopher Conte edited this page Dec 11, 2016
·
9 revisions
We chose to use as our coverage tool the node module istanbul, and to publish our results on coveralls.com, in order to be able to give a public address on our Github, relating to all of our tests and their execution of our code. We began in our first istanbul tests with a coverage of only around 50%, and some small errors that were causing a couple tests to fail. After all of our original tests succeeded, we progressed to our investigation of what was not being run. Our pursuit began on extraneous functions. By that we mean those which were used at one point, but became obsolete during a refactor. These functions are simultaneously sources of error and unpredicted behavior. One could use the function at a later point with an alternative purpose that all those cases which we imagined (and thus implemented), and get an unexpected result, either due to a bug not tested, or otherwise. These functions also could run into future implementations, and interact with objects related to those present. This is not to mention the inconvenience that it dilutes our coverage percentage. Second, using the instanbul coverage reports, we could investigate those functions which we have not tested, and those branches which were not reached. We began by writing more test-to-fail cases relating to branches that returned errors. We tested the code for trying to emit events to a game that doesn’t exist, we tested the code for emitting events when our server wasn’t expecting it. We also tested unit-wise some more behavior tests, for example, testing that our hunt phase does not repeat words in the same round. We also read in our tests that some branches, which had essentially the same function, were being counted as branches that were not being covered. For our GameController, we have a qualifier for ever event from the user pertaining to wether or not they correspond to a game. If a game does not exist, then we ignore the socket event. This is abstracted and reduced to a single line conditional. These repeating conditionals we designated istanbul to ignore, because this type of coverage is primarily semantic, we are testing all of the logic without testing every single instance of that logic. Finally a large source of untested code was within io.js, our router. Fundamentally this needs to be fixed via a refactoring of a lot of that logic to the phase objects themselves. However, that is still in progress and it is important to keep having a functioning program. We wrote more tests, which are black box tests, which reach io.js. These tests were furthering our test-to-fail cases, and as well calling each legitimate socket emission and verified that their responses were received, and the content of those responses was as expected. Next we looked again at our coverage tests in order to expand our black box testing of the router and increased it to 85%. What we did not test were those time based events, which included the interval functions that play a part in the ending of both of our components. While our coverage is not 100% it is very close to being, and the most complex of our logic is covered. What is missed is simply some calls to unit tests within the context of our router. In order to evade the troubles that time based events pose to testers, we must take the unit approach. That is to say we must mock the interval function to return expected values, in order to test the behavior of functions called by those timers.
> [email protected] test /Users/Chris/Downloads/san-fransisco-trail-880a3f4c0b4a1b1e3f00342b5abc88ce69032c09
> mocha --recursive
Running on port 8000
Game Server
10 Dec 22:45:45 - Client has connected: pmokh-R22HC0SxbwAAAA
✓ Should instantiate a game when a new user connects (169ms)
10 Dec 22:45:45 - Client has connected: jVdbphlWE_bk3c1aAAAB
10 Dec 22:45:45 - Client has connected: q_YIGPGJklS9-CqJAAAC
✓ Should Handle multiple users (121ms)
10 Dec 22:45:46 - Client has connected: f_ELhuuUm_CC34cOAAAD
✓ Should be able to buy stock (113ms)
10 Dec 22:45:46 - Client has connected: JtHkRby95DoqT08cAAAE
✓ Should be not be able to sell stock (114ms)
10 Dec 22:45:46 - Client has connected: ftcYnGi9NA4FN9HzAAAF
onStartHunt
✓ Should begin to receive words after start hunt
10 Dec 22:45:46 - Client has connected: ufOe2BXs3rX-RVNdAAAG
onStartHunt
onTestWord
✓ Should to receive an incorrect after guessing wrong
10 Dec 22:45:46 - Client has connected: M98mar9ehx96wcYrAAAH
onStartHunt
onTestWord
✓ Should to receive an correct after guessing right
Hunt Phase
constructor
✓ Should instantiate with proper config
"startHunt" method
✓ should begin the Hunt
"newWord" method
✓ Should generate a New Word
✓ Should not repeat any words
"isCorrect" method
✓ should return correct score if correct
✓ should return 0 if incorrect
✓ should die if enough words are incorrect
✓ should advance if enough words are correct
✓ should not award points to multiple correct guesses for the same iteration
InvestPhase
constructor
✓ should construct using given configuration
BuyStock
✓ should properly buy a stock
✓ should not be able to buy more stocks than wallet amount allows
SellStock
✓ should properly sell a stock
✓ should not be able to sell more stocks than you have
addPoint
✓ should set addPoint shoudl return new stockPrice b/t 0-1000
startInvest
✓ should set begin to true
endInvest
✓ should set finished to false
Wallet Deduction
✓ should decrement wallet by stockPrice
Wallet Addtion
✓ should increment wallet by stockPrice after decrementing by stockPrice
Leaderboard
constructor
✓ should have possibility to take RedisClient instance
"add" method
✓ should put "member1" with score 30 to the 0 position
✓ should put "member2" with score 20 to the 1 position
✓ should put "member3" with score 10 to the 2 position
✓ should put "member3" with score 40 to the 0 position
✓ shoud take "callback" argument as optional
✓ addIn method should do the same but with another leaderboard
"incr" method
✓ should add members if they don't exist
✓ should increment members' score if they do exist
✓ should decrement members' score if score value is negative
✓ should keep correct members order
✓ shoud take "callback" argument as optional
✓ incrIn method should do the same but with another leaderboard
"rank" method
✓ should return correct rank #1
✓ should return correct rank #2
✓ should return correct rank #3
✓ should return -1 if member isn't in the leaderboard
✓ rankIn method should do the same but with another leaderboard
"list" method
✓ should return correct list #1
✓ should return correct list #2
✓ should return correct list #3
✓ should return correct number of entries for the page 0
✓ should return correct number of entries for the page 1
✓ listIn method should do the same but with another leaderboard
"score" method
✓ should return correct score #1
✓ should return correct score #2
✓ should return correct score #3
✓ score should be typeof number
✓ should return -1 if member isn't in the leaderboard
✓ scoreIn method should do the same but with another leaderboard
"rm" method
✓ should remove member from the leaderboard #1
✓ should remove member from the leaderboard #2
✓ should remove member from the leaderboard #3
✓ should remove member from the leaderboard #4
✓ shoud take "callback" argument as optional
✓ rmIn method should do the same but with another leaderboard
"at" method
✓ should return correct member #1
✓ should return correct member #2
✓ returned score should be typeof number
✓ should return null if member is not found
✓ atIn method should do the same but with another leaderboard
"total" method
✓ should return correct number of members
✓ totalIn method should do the same but with another leaderboard
Options
"pageSize"
✓ should set specified number of entries for a page
"reverse"
✓ should make "list" method return results in reverse order
✓ should make "rank" method return results in reverse order
✓ should make "at" method return results in reverse order
73 passing (724ms)
dyn-160-39-142-41:san-fransisco-trail-880a3f4c0b4a1b1e3f00342b5abc88ce69032c09 Chris$ istanbul cover _mocha -- -R spec
Unable to resolve file [_mocha]
Try "istanbul help" for usage
dyn-160-39-142-41:san-fransisco-trail-880a3f4c0b4a1b1e3f00342b5abc88ce69032c09 Chris$ istanbul cover node_modules/.bin/_mocha -- -- -u exports -R spec test/**/*
Warning: Could not find any test files matching pattern: -u
Warning: Could not find any test files matching pattern: exports
Warning: Could not find any test files matching pattern: -R
Warning: Could not find any test files matching pattern: spec
Running on port 8000
Game Server
10 Dec 22:47:53 - Client has connected: S6A1n-w0fD-rJqTNAAAA
✓ Should instantiate a game when a new user connects (166ms)
10 Dec 22:47:53 - Client has connected: ltIbDlyc1Wj5gWgnAAAB
10 Dec 22:47:53 - Client has connected: 8EPA_04R09ekn2YxAAAC
✓ Should Handle multiple users (115ms)
10 Dec 22:47:53 - Client has connected: ZRc4RA4XX3gQrKw-AAAD
✓ Should be able to buy stock (107ms)
10 Dec 22:47:53 - Client has connected: ZK4CTuZybrhJLTNaAAAE
✓ Should be not be able to sell stock (114ms)
10 Dec 22:47:53 - Client has connected: _hfV9j5AaBPMqFQVAAAF
onStartHunt
✓ Should begin to receive words after start hunt
10 Dec 22:47:53 - Client has connected: Lfxhkb1IuRaFRTpmAAAG
onStartHunt
onTestWord
✓ Should to receive an incorrect after guessing wrong
10 Dec 22:47:53 - Client has connected: QFEH1zf3_sCQWVk5AAAH
onStartHunt
onTestWord
✓ Should to receive an correct after guessing right
Hunt Phase
constructor
✓ Should instantiate with proper config
"startHunt" method
✓ should begin the Hunt
"newWord" method
✓ Should generate a New Word
✓ Should not repeat any words
"isCorrect" method
✓ should return correct score if correct
✓ should return 0 if incorrect
✓ should die if enough words are incorrect
✓ should advance if enough words are correct
✓ should not award points to multiple correct guesses for the same iteration
InvestPhase
constructor
✓ should construct using given configuration
BuyStock
✓ should properly buy a stock
✓ should not be able to buy more stocks than wallet amount allows
SellStock
✓ should properly sell a stock
✓ should not be able to sell more stocks than you have
addPoint
✓ should set addPoint shoudl return new stockPrice b/t 0-1000
startInvest
✓ should set begin to true
endInvest
✓ should set finished to false
Wallet Deduction
✓ should decrement wallet by stockPrice
Wallet Addtion
✓ should increment wallet by stockPrice after decrementing by stockPrice
Leaderboard
constructor
✓ should have possibility to take RedisClient instance
"add" method
✓ should put "member1" with score 30 to the 0 position
✓ should put "member2" with score 20 to the 1 position
✓ should put "member3" with score 10 to the 2 position
✓ should put "member3" with score 40 to the 0 position
✓ shoud take "callback" argument as optional
✓ addIn method should do the same but with another leaderboard
"incr" method
✓ should add members if they don't exist
✓ should increment members' score if they do exist
✓ should decrement members' score if score value is negative
✓ should keep correct members order
✓ shoud take "callback" argument as optional
✓ incrIn method should do the same but with another leaderboard
"rank" method
✓ should return correct rank #1
✓ should return correct rank #2
✓ should return correct rank #3
✓ should return -1 if member isn't in the leaderboard
✓ rankIn method should do the same but with another leaderboard
"list" method
✓ should return correct list #1
✓ should return correct list #2
✓ should return correct list #3
✓ should return correct number of entries for the page 0
✓ should return correct number of entries for the page 1
✓ listIn method should do the same but with another leaderboard
"score" method
✓ should return correct score #1
✓ should return correct score #2
✓ should return correct score #3
✓ score should be typeof number
✓ should return -1 if member isn't in the leaderboard
✓ scoreIn method should do the same but with another leaderboard
"rm" method
✓ should remove member from the leaderboard #1
✓ should remove member from the leaderboard #2
✓ should remove member from the leaderboard #3
✓ should remove member from the leaderboard #4
✓ shoud take "callback" argument as optional
✓ rmIn method should do the same but with another leaderboard
"at" method
✓ should return correct member #1
✓ should return correct member #2
✓ returned score should be typeof number
✓ should return null if member is not found
✓ atIn method should do the same but with another leaderboard
"total" method
✓ should return correct number of members
✓ totalIn method should do the same but with another leaderboard
Options
"pageSize"
✓ should set specified number of entries for a page
"reverse"
✓ should make "list" method return results in reverse order
✓ should make "rank" method return results in reverse order
✓ should make "at" method return results in reverse order
73 passing (665ms)
=============================================================================
Writing coverage object [/Users/Chris/Downloads/san-fransisco-trail-880a3f4c0b4a1b1e3f00342b5abc88ce69032c09/coverage/coverage.json]
Writing coverage reports at [/Users/Chris/Downloads/san-fransisco-trail-880a3f4c0b4a1b1e3f00342b5abc88ce69032c09/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 75.81% ( 257/339 )
Branches : 51.43% ( 36/70 )
Functions : 70.77% ( 46/65 )
Lines : 76.49% ( 257/336 )
================================================================================
dyn-160-39-142-41:san-fransisco-trail Chris$ npm run cover
> [email protected] cover /Users/Chris/Google Drive/san-fransisco-trail
> istanbul cover ./node_modules/mocha/bin/_mocha -- -R spec
Running on port 8000
GameControlller
constructor
✓ should construct using given configuration
✓ should construct using given default configuration
setName method
✓ should set player's name
✓ should not set player's name if invalid input
increment level method
✓ should incrament properly
set wallet method
✓ should mute wallet properly
set phase method
✓ should set phase to invest properly
✓ should set phase to hunt properly
✓ should set phase to no phase properly
get data to post should format properly
✓ should return object when dead
✓ should return nothing when alive
Game Server
10 Dec 22:37:32 - Client has connected: gbc7jJ7ngexjWGeoAAAA
✓ Should instantiate a game when a new user connects (163ms)
10 Dec 22:37:32 - Client has connected: sfDNtmEXhqUxPpHQAAAB
10 Dec 22:37:32 - Client has connected: uMPTZ1xezYd9vb-8AAAC
✓ Should Handle multiple users (123ms)
10 Dec 22:37:33 - Client has connected: rcf3Jo9tlA_lWjyaAAAD
✓ Should be able to set player name
10 Dec 22:37:33 - Client has connected: SmTiWSY3lOWbRL8wAAAE
✓ Should be able to buy stock (112ms)
10 Dec 22:37:33 - Client has connected: iKMW5qcWEY4tjjB3AAAF
✓ Should be not be able to sell stock (108ms)
10 Dec 22:37:33 - Client has connected: PhkBvOWBakiU3o-GAAAG
✓ Should be able to sell stock they bought (113ms)
10 Dec 22:37:33 - Client has connected: gWz0T3w0skAHdVA0AAAH
onStartHunt
✓ Should begin to receive words after start hunt
10 Dec 22:37:33 - Client has connected: nb4jZyrD8SykthreAAAI
onStartHunt
onTestWord
✓ Should to receive an incorrect after guessing wrong
10 Dec 22:37:33 - Client has connected: sxSRZReSqtXMGmojAAAJ
onStartHunt
onTestWord
✓ Should to receive an correct after guessing right
10 Dec 22:37:33 - Client has connected: ePB_R0IDeG2z1CRoAAAK
✓ Should be able to fetch the leaderboard
10 Dec 22:37:33 - Client has connected: bYX71uKh293uIO17AAAL
✓ Should be able to fetch the leaderboard
test
null
Hunt Phase
constructor
✓ Should instantiate with proper config
"startHunt" method
✓ should begin the Hunt
"newWord" method
✓ Should generate a New Word
✓ Should not repeat any words
"isCorrect" method
✓ should return correct score if correct
✓ should return 0 if incorrect
✓ should die if enough words are incorrect
✓ should advance if enough words are correct
✓ should not award points to multiple correct guesses for the same iteration
InvestPhase
constructor
✓ should construct using given configuration
BuyStock
✓ should properly buy a stock
✓ should not be able to buy more stocks than wallet amount allows
SellStock
✓ should properly sell a stock
✓ should not be able to sell more stocks than you have
addPoint
✓ should set addPoint shoudl return new stockPrice b/t 0-1000
startInvest
✓ should set begin to true
endInvest
✓ should set finished to false
Wallet Deduction
✓ should decrement wallet by stockPrice
Wallet Addtion
✓ should increment wallet by stockPrice after decrementing by stockPrice
Leaderboard
constructor
✓ should have possibility to take RedisClient instance
"add" method
✓ should put "member1" with score 30 to the 0 position
✓ should put "member2" with score 20 to the 1 position
✓ should put "member3" with score 10 to the 2 position
✓ should put "member3" with score 40 to the 0 position
✓ shoud take "callback" argument as optional
✓ addIn method should do the same but with another leaderboard
"incr" method
✓ should add members if they don't exist
✓ should increment members' score if they do exist
✓ should decrement members' score if score value is negative
✓ should keep correct members order
✓ shoud take "callback" argument as optional
✓ incrIn method should do the same but with another leaderboard
"rank" method
✓ should return correct rank #1
✓ should return correct rank #2
✓ should return correct rank #3
✓ should return -1 if member isn't in the leaderboard
✓ rankIn method should do the same but with another leaderboard
"list" method
✓ should return correct list #1
✓ should return correct list #2
✓ should return correct list #3
✓ should return correct number of entries for the page 0
✓ should return correct number of entries for the page 1
✓ listIn method should do the same but with another leaderboard
"score" method
✓ should return correct score #1
✓ should return correct score #2
✓ should return correct score #3
✓ score should be typeof number
✓ should return -1 if member isn't in the leaderboard
✓ scoreIn method should do the same but with another leaderboard
"rm" method
✓ should remove member from the leaderboard #1
✓ should remove member from the leaderboard #2
✓ should remove member from the leaderboard #3
✓ should remove member from the leaderboard #4
✓ shoud take "callback" argument as optional
✓ rmIn method should do the same but with another leaderboard
"at" method
✓ should return correct member #1
✓ should return correct member #2
✓ returned score should be typeof number
✓ should return null if member is not found
✓ atIn method should do the same but with another leaderboard
"total" method
✓ should return correct number of members
✓ totalIn method should do the same but with another leaderboard
Options
"pageSize"
✓ should set specified number of entries for a page
"reverse"
✓ should make "list" method return results in reverse order
✓ should make "rank" method return results in reverse order
✓ should make "at" method return results in reverse order
88 passing (890ms)
=============================================================================
Writing coverage object [/Users/Chris/Google Drive/san-fransisco-trail/coverage/coverage.json]
Writing coverage reports at [/Users/Chris/Google Drive/san-fransisco-trail/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 90.25% ( 287/318 ), 14 ignored
Branches : 75.76% ( 50/66 ), 7 ignored
Functions : 90% ( 54/60 )
Lines : 91.11% ( 287/315 )
================================================================================
Note that our beginning coverage tests were done without having connected to our continuous integration tool, thereby limiting the history to more recent test.