Skip to content

Conversation

@josiahsams
Copy link

As part of this pull request the following capabilities are added,

  1. Ability to include user custom metrics into spark generated code is added.
  2. A listener is provided to report the user metrics at the end of every Stage.

Usage:
Now the user generated code will have similar code included,

this.userDefined1 = (org.apache.spark.sql.catalyst.UserTaskMetric) references[0];

Operations like add, += are supported on this new object as,

userDefined1.add(100);

To get the custom metrics report, include the provided listener to the SparkListeners as follows in either spark-shell or spark-submit,

--conf spark.extraListeners=org.apache.spark.sql.catalyst.UserStatsReportListener

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant