[SUPPORT] Apply 'write.ignore.failed' when write data failed

[fhan688]

Describe the problem you faced

In Flink engine, if exception occurs when task writing data, it will be ignored and the exception will be reported to StreamWriteCoordinator with write event, StreamWriteCoordinator will decide whether to commit when there is write failure according to 'write.ignore.failed'.

Whether we should apply 'write.ignore.failed' ahead when write failure occurs, thus throw an exception faster.

To Reproduce

Expected behavior

1. promote 'write.ignore.failed' to hudi-client-common
2. throw exception in write handles
3. default value of 'write.ignore.failed' is false

Environment Description

Additional context

related PR：#12150

Stacktrace

[danny0405]

@nbalajee Do you have some insights about promote the 'write.ignore.failed' as a basic option for all the engines? I kind of remember we had a discussion before and engine specific maintainance is prefferable.

[fhan688]

Whether we should involve more experts in this topic?

[HuangZhenQiu]

We have the same requirement to fail hudi write earlier before hudi commit time. It can reduce the downtime of ingestion job. If we apply the config in writer, the exception will be caught in TM. How about validate the write failure in each received event in StreamWriteCoordinator? In this case, we don't need to change the behavior of all of append, create, merge handle.

#12666

[danny0405]

I kind of think we should have a general fix for all the engines, not just Flink, so maybe fixing each write handle is more feasible. @fhan688 do you have new inputs here?