Enable PETSc for HIP machines

[chapman39]

When building Serac TPLs for tioga/ rzvernal I encountered two problems related to building PETSc:

1. PETSc doesn't like when the HIP prefix is /opt/rocm-5.7.1 in the spack.yaml (prefers /opt/rocm-5.7.1/hip, which messes up Axom, Tribol, Serac build systems).
2. PETSc had to be built with no optimization flags (Serac spec contained ^petsc cflags=-O0)

The process of getting PETSc to build previously is outlined here: spack/spack#46468

Ideally, we want to build PETSc while keeping the HIP prefix the same and with full optimization flags. This will likely involve changes in PETSc's Spack package and/ or a patch, as well as some help from PETSc developers.

[chapman39]

using newer rocm version (6 and up) will likely resolve some of the issues: llnl/radiuss-spack-configs#114 (comment)