Skip to content
This repository was archived by the owner on Oct 30, 2023. It is now read-only.

Commit a7ab3ce

Browse files
committed
This diff contains the following fixes:
* Once the OOC is at OFFLAODING state, it may offload data to disk indefinitely. We should call GC manually and get OOC out of OFFLOADING if anything has changed in computation/communication pattern. Fixes a "if" statement which entirely disabled regression for memory estimation. * Interleaving of in resetting and calculating memory estimation potentially can cause data race. * The superstep count in memory estimator was not coherent throughout the calculation. * Sometime the memory estimator's accuracy is not good. We should fall back to a threshold-based scheme relying on a more pessimistic memory usage report (such as the one given by JVM – memory usage includes garbage data too, but it can be used as a pessimistic estimate, as it is currently used in ThresholdBasedOracle). More tuning is needed for a smooth memory estimation mechanism. "mvn clean install" passes all tests and checks.
1 parent ea7753f commit a7ab3ce

File tree

2 files changed

+228
-102
lines changed

2 files changed

+228
-102
lines changed

giraph-core/src/main/java/org/apache/giraph/ooc/OutOfCoreEngine.java

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -426,11 +426,14 @@ public void ioCommandCompleted(IOCommand command) {
426426
* threads.
427427
*
428428
* @param fraction the fraction of processing threads to remain active. This
429-
* number is in range [0, 1]
429+
* number is in range (0, 1]
430430
*/
431431
public void updateActiveThreadsFraction(double fraction) {
432432
checkState(fraction >= 0 && fraction <= 1);
433-
int numActiveThreads = (int) (numProcessingThreads * fraction);
433+
// Making sure the number of active threads is not set to 0 to ensure
434+
// progress in computation
435+
int numActiveThreads = Math.max(
436+
(int) (Math.ceil(numProcessingThreads * fraction)), 1);
434437
if (LOG.isInfoEnabled()) {
435438
LOG.info("updateActiveThreadsFraction: updating the number of active " +
436439
"threads to " + numActiveThreads);
@@ -489,7 +492,7 @@ public void processingThreadFinish() {
489492
*/
490493
public void updateRequestsCreditFraction(double fraction) {
491494
checkState(fraction >= 0 && fraction <= 1);
492-
short newCredit = (short) (maxRequestsCredit * fraction);
495+
short newCredit = (short) (Math.ceil(maxRequestsCredit * fraction));
493496
if (LOG.isInfoEnabled()) {
494497
LOG.info("updateRequestsCreditFraction: updating the credit to " +
495498
newCredit);

0 commit comments

Comments
 (0)