handle failed executor event

[ChenLingPeng]

Signed-off-by: forrestchen [email protected]

see #600

(Please fill in changes proposed in this fix)

How was this patch tested?

manual test.

[liyinan926]

A general question: how does Yarn handle this case, i.e., of executors that fail to register?

[liyinan926]

s/executors/Executors.

[liyinan926]

Empty space before (.

[liyinan926]

Why 1000? What about totalExpectedExecutors ?

[ChenLingPeng]

totalExpectedExecutors sounds a good idea to limit the executor numbers.

[liyinan926]

Javadoc needs a bit of rewording to explain what the return value represents and how it is calculated.

[ChenLingPeng]

how about this

    // It represent current created executors exclude failed one.
    // To avoid to create too many failed executor,
    // we limit the accounting size of failed executors to totalExpectedExecutors
    // so after create 2*totalExpectedExecutors executors,
    // we stop create more even if all of them failed

[liyinan926]

BTW: this fix should also be upstreamed. Can you file a PR against upstream apache/master?

[ChenLingPeng]

A general question: how does Yarn handle this case, i.e., of executors that fail to register?

Not so familiar with spark-on-yarn, I think if allocateResponse.getCompletedContainersStatuses can return this kind of executors, then in yarn mode, it can handle this scenario just like registered but failed executor.

this fix should also be upstreamed

Will do this after this is merged