Skip to content

Recommend G6 instead of G5 on EMR #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sayedbilalbari wants to merge 4 commits into
base: dev
Choose a base branch
from
Open

Recommend G6 instead of G5 on EMR #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dbe4e99
Changed g5 -> g6 instance
Jan 30, 2025
ce0d774
Updates instance types for EMR from g5 to g6
sayedbilalbari Feb 3, 2025
de499a2
Update user tool tests - g6 recommendation for EMR
Feb 3, 2025
e000950
Review change - updated change
sayedbilalbari Feb 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,18 @@ object InstanceInfo {
// format (numGpus, numCores) -> InstanceInfo about that CSP node instance type
object PlatformInstanceTypes {

val AWS_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g5.xlarge", 1),
// Using G6 instances for EMR
val EMR_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g6.xlarge", 1),
(1, 8) -> InstanceInfo(8, 32 * 1024, "g6.2xlarge", 1),
(1, 16) -> InstanceInfo(16, 64 * 1024, "g6.4xlarge", 1),
(1, 32) -> InstanceInfo(32, 128 * 1024, "g6.8xlarge", 1),
(4, 48) -> InstanceInfo(48, 192 * 1024, "g6.12xlarge", 1),
(1, 64) -> InstanceInfo(64, 256 * 1024, "g6.16xlarge", 1)
)

// Using G5 instances. To be updated once G6 availability on Databricks
// is consistent
val DATABRICKS_AWS_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g5.xlarge", 1),
(1, 8) -> InstanceInfo(8, 32 * 1024, "g5.2xlarge", 1),
(1, 16) -> InstanceInfo(16, 64 * 1024, "g5.4xlarge", 1),
(1, 32) -> InstanceInfo(32, 128 * 1024, "g5.8xlarge", 1),
Expand Down Expand Up @@ -559,7 +570,7 @@ class DatabricksAwsPlatform(gpuDevice: Option[GpuDevice],
override val defaultGpuDevice: GpuDevice = A10GGpu

override def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = {
PlatformInstanceTypes.AWS_BY_GPUS_CORES
PlatformInstanceTypes.DATABRICKS_AWS_BY_GPUS_CORES
}
}

Expand Down Expand Up @@ -629,7 +640,7 @@ class EmrPlatform(gpuDevice: Option[GpuDevice],
}

override def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = {
PlatformInstanceTypes.AWS_BY_GPUS_CORES
PlatformInstanceTypes.EMR_BY_GPUS_CORES
}
}

Expand Down
2 changes: 1 addition & 1 deletion user_tools/tests/spark_rapids_tools_e2e/features/installation_checks.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Feature: Tool Installation Checks
Examples:
| platform | cli | expected_stdout |
| dataproc | gcloud | 2 x n1-standard-16 (1 T4 each) |
| emr | aws | 2 x g5.4xlarge |
| emr | aws | 2 x g6.4xlarge |
| databricks-aws | aws | 2 x g5.4xlarge |
| databricks-azure | az | 2 x Standard_NC16as_T4_v3 |

Expand Down