Limit ARM retries to operator tests #4259

Job	Run time
test-vulkan / test-backend-linux (vulkan, models) / linux-job	20m 49s
test-vulkan / test-backend-linux (vulkan, operators) / linux-job	21m 12s
test-vulkan / test-backend-macos	0s
test-vulkan / package-golden-artifacts	3m 30s
	45m 31s

Provide feedback