diff --git a/exawind/NREL_5MW_Turbine/rigid/README.md b/exawind/NREL_5MW_Turbine/rigid/README.md index 9fa7425a..57fbc5b1 100644 --- a/exawind/NREL_5MW_Turbine/rigid/README.md +++ b/exawind/NREL_5MW_Turbine/rigid/README.md @@ -1,16 +1,5 @@ - - # NREL 5MW Rotor in Atmospheric Boundary Layer With Rigid Bodies and Constant Rotation This benchmark contains a geometry and boundary-layer resolved model of the NREL 5MW reference turbine [1] in a developed near-neutral convective atmospheric boundary layer with a mean hub-height streamwise velocity of 11.4m/s. @@ -18,57 +7,104 @@ This benchmark contains a geometry and boundary-layer resolved model of the NREL - Blade Radius: 63m - Rated Power: 5 MW - Type: Upwind 3 Blade +- Simulation dt: 0.003443526170799 s +- OpenFAST dt: 0.0008608815426997245 s Note on run-time: Current results are for 125s simulation time, but by the time of benchmark release, a minimum of 360s will be posted. ## Simulation Setup - ExaWind driver SHA: [cba5259fc43ddeca67329630d3c84faad90e91bb](https://github.com/Exawind/exawind-driver/commit/cba5259fc43ddeca67329630d3c84faad90e91bb) - Nalu-Wind SHA: [b9e4ae654b646ecd0501dd6391dc7537239c82db](https://github.com/Exawind/nalu-wind/commit/b9e4ae654b646ecd0501dd6391dc7537239c82db) -- AMR-Wind SHA: [091b07fa3840af98925792290fb8788696290a5e](https://github.com/Exawind/amr-wind/commit/091b07fa3840af98925792290fb8788696290a5e) +- AMR-Wind SHA: [091b07fa3840af98925792290fb8788696290a5e](https://github.com/Exawind/amr-wind/commit/091b07fa3840af98925792290fb8788696290a5e) ## Freestream Conditions A near-neutral convective boundary layer precursor was run to feed the domain for this case. Full details are posted here: [Convective ABL for NREL5MW](../../../amr-wind/atmospheric_boundary_layer/convective_abl_nrel5mw/README.md) 10 minute flow statistics: -- Hub-height streamwise velocity: 11.63 m/s +- Hub-height streamwise velocity: 11.4 m/s - Hub-height flow direction: ~240.0 degrees -10-Minute Average Virtual Met-mast Precursor Results +![avgmetmest60](../../../amr-wind/atmospheric_boundary_layer/convective_abl_nrel5mw/results/avgmetmast_0600.png) ## CFD Mesh -Mesh layout +![meshpicture](figures_and_scripts/split_mesh_layout.png) +**Total number of cells: 155,363,046** -**Total number of cells: 177,729,426** - -The near-body (Nalu-Wind) mesh was created using PGL, pyHyp, and Pointwise. Three blades are connected at the hub, and a tower is included. -- Structured hex mesh -- Cell count: 15,593,362 +The near-body (Nalu-Wind) mesh was created using a proprietary surface mesher, pyHyp, and Pointwise. Three blades are split at the hub, and an unconnected tower is included. +- Hexahedral cells +- Cell count: 13,436,646 Off-body (AMR-Wind) mesh was generated using the built-in capability of AMR-Wind. Off-body mesh information is summarized below -- Mesh topology: Cartesian with AMR +- Mesh topology: structured hex with nested refinements - Domain in x= 0 to 5000m, y=0m to 5000m, z=0m to 1000m - Initial grid size: 10m - Finest cell size: 0.625m with 4 AMR levels -- Cell count: 162,136,064 +- Cell count: 141,926,400 ## Results -Force and Moment Output +#### Postprocessing Procedure + +Pressure and viscous force vectors are read at all blade surface faces. The integrated values, as well as moments around the hub, are recorded every four timesteps. + +Yaw angle: $\phi=30.0 \degree\\$ +Tilt angle: $\psi=5.0 \degree\\$ +Generator Efficiency: $E_g = 0.944\\$ +Rotor Speed: $\Omega = 12.1 rpm$ + +$ f_x = f_{p_x}+f_{v_x}\\$ +$ f_y = f_{p_y}+f_{v_y}\\$ +$ f_z = f_{p_z}+f_{v_z}\\$ + +$ f_{\phi x} = f_x*cos(-\phi)-f_y*sin(-\phi)\\$ +$ f_{\phi y} = f_x*sin(-\phi)+f_y*cos(-\phi)\\$ +$ f_{\phi z} = f_z\\$ + +$ m_{\phi x} = m_x*cos(-\phi)-m_y*sin(-\phi)\\$ +$ m_{\phi y} = m_x*sin(-\phi)+m_y*cos(-\phi)\\$ +$ m_{\phi z} = m_z\\$ + +$ Torque = m_{\phi x}*cos(-\psi)+m_{\phi z}*sin(-\psi)\\$ +$ Thrust =f_{\phi x}*cos(-\psi)+f_{\phi z}*sin(-\psi)\\$ +$ Power = Torque*\Omega*E_g/1000\\$ + +![rigidoutput](figures_and_scripts/rigid_output.png) + +| variable | value | +| -------- | ------- | +|Generator Power|4756.45 kW| +|Rotor Torque|3976.46 kN-m| +|Rotor Thrust|650.25 kN| +|Rotor Speed|12.1 rpm| ## Simulation Timings -This benchmark was run on Sandia National Laboratories' machine "Flight" which is comprised of 112 Intel(R) Xeon(R) Platinum 8480+ CPU cores per node. +This benchmark was run at Sandia National Laboratories on a machine comprised of 112 Intel(R) Xeon(R) Platinum 8480+ CPU cores per node. ### Processor Decomposition: - Nalu-Wind: 672 ranks -- AMR-Wind: 6496 ranks +- AMR-Wind: 4928 ranks ### Timings: - Mean wall-clock time per timestep for entire simulation: 7.35s - Mean wall-clock time per timestep per cell 4.135e-08s +## Quick Exawind Simulation Guide +### Step 0: Run the ABL precursor in AMR-Wind +This simulation is driven by an ABL precursor, run in AMR-Wind, with recorded boundary planes, initial conditions, and average temperature. (See case linked above) +### Step 1: Develop and decompose overset mesh +It is necessary to develop a near-body mesh surrounding the blade surface in Nalu-Wind using external tools. This +mesh should be in the Exodus II format and decomposed to approximately 20K cells per core available to Nalu-Wind. (Current mesh provided via dvc/github assets) +### Step 2: Run the OpenFAST precursor for 5760 OpenFAST timesteps (approximately 1 rotation) +``srun -n 1 openfastcpp iyaml`` +### Step 3: Run the full Exawind suite +``srun -N 50 -n 5600 exawind --nwind 672 --awind 4928 nrel5mw.yaml &> log`` +### Step 4: Setup and Run the full Exawind suite restart(s) +``srun -N 50 -n 5600 exawind --nwind 672 --awind 4928 nrel5mw_r1.yaml &> log_1`` + + ## References [1]: Jonkman, J. et al, "Definition of a 5-MW Reference Wind Turbine for Offshore System Development" NREL/TP-500-38060, 2009. https://www.nrel.gov/docs/fy09osti/38060.pdf \ No newline at end of file diff --git a/exawind/NREL_5MW_Turbine/rigid/deploy.yaml b/exawind/NREL_5MW_Turbine/rigid/deploy.yaml index a43e37c9..30670c5a 100644 --- a/exawind/NREL_5MW_Turbine/rigid/deploy.yaml +++ b/exawind/NREL_5MW_Turbine/rigid/deploy.yaml @@ -1,5 +1,5 @@ # Location where files should exist -destdir: ../../.website_src/exawind/nrel5mw/rigid +destdir: ../../../.website_src/exawind/nrel5mw/rigid # Define the valid file types to copy over (optional) validtypes: ['.html', '.md', '.ipynb', '.png', '.jpg', '.rst', ] diff --git a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/get_timings.py b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/get_timings.py index 074bc243..fef0de5e 100755 --- a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/get_timings.py +++ b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/get_timings.py @@ -117,13 +117,18 @@ def processline(inputline): def main(): casedir = '/pscratch/ndeveld/hfm-2025-q1' - casename = 'nrel5mw_rigid_abl_noopenfast2' + casename = 'rigid_abl_splitmsh_nate2' casepath = os.path.join(casedir,casename) + plotdir = os.path.join(casepath,'plots') + exlogfile = casepath+"/log" amrlogfile = casepath+"/nrel5mw_amr.log" nalulogfile = casepath+"/nrel5mw_nalu.log" + prefix = 'rign2' + #cmd="grep '^Exawind::Total' "+exlogfile+" | awk '{print $3}' > "+casepath+"/exatimestep.dat" + #result = sp.run(cmd, shell=True, capture_output=True, text=True) cmd="grep '^Exawind::Total' "+exlogfile+" | awk '{print $3}'" result = sp.run(cmd, shell=True, capture_output=True, text=True) @@ -137,9 +142,30 @@ def main(): result = sp.run(cmd, shell=True, capture_output=True, text=True) amr_timesteps = [float(x) for x in result.stdout.replace('\n',' ').split()] + cmd="grep '^ MAC_projection ' "+amrlogfile+" | awk '{print $2}'" + result = sp.run(cmd, shell=True, capture_output=True, text=True) + amr_mac= [float(x) for x in result.stdout.replace('\n',' ').split()] + + cmd="grep '^ Nodal_projection ' "+amrlogfile+" | awk '{print $2}'" + result = sp.run(cmd, shell=True, capture_output=True, text=True) + amr_nodal= [float(x) for x in result.stdout.replace('\n',' ').split()] + + cmd="grep '^ MomentumEQS ' "+nalulogfile+" | awk '{print $2}'" + result = sp.run(cmd, shell=True, capture_output=True, text=True) + nalu_mom= [float(x) for x in result.stdout.replace('\n',' ').split()] + + cmd="grep '^ ContinuityEQS ' "+nalulogfile+" | awk '{print $2}'" + result = sp.run(cmd, shell=True, capture_output=True, text=True) + nalu_cont= [float(x) for x in result.stdout.replace('\n',' ').split()] + #tsdata = pd.read_csv(casepath+'/avgtimestep.dat',header=None) ts = range(len(total_timesteps)) + nts = range(len(nalu_timesteps)) + ats = range(len(amr_timesteps)) + amrts = range(len(amr_mac)) + naluts = range(len(nalu_mom)) + conts = range(len(nalu_cont)) print('Mean timestep',np.mean(total_timesteps)) @@ -168,14 +194,23 @@ def main(): print('Mean Timestep per cell',np.mean(total_timesteps)/(amr_cells+nalu_cells)) + cs = 5 # Plot the 5th occurence of continuity iters + ms = 4 # Plot the 4th occurence of momentum iters + + nmd_filt = np.array(nalu_mom)[0::ms].copy() + ncd_filt = np.array(nalu_cont)[0::cs].copy() + nmts_filt = np.arange(len(nmd_filt)) + ncts_filt = np.arange(len(ncd_filt)) + + # Plot main exawind plt.rcParams.update({'font.size': 18}) fig, ax = plt.subplots(1,3,figsize=(13,4)) ax[0].scatter(ts,total_timesteps,s=0.3) ax[0].set_title('Exawind') - ax[1].scatter(ts,nalu_timesteps,s=0.3) + ax[1].scatter(nts,nalu_timesteps,s=0.3) ax[1].set_title('Nalu-Wind') - ax[2].scatter(ts,amr_timesteps,s=0.3) + ax[2].scatter(ats,amr_timesteps,s=0.3) ax[2].set_title('AMR-Wind') for i in range(3): @@ -186,6 +221,30 @@ def main(): fig.tight_layout() fig.savefig('timepertimestep.png') + # Plot AMR + plt.rcParams.update({'font.size': 18}) + + in_min = min(len(amrts),len(amr_nodal))-1 + + fig, ax = plt.subplots(1,2,figsize=(13,4)) + ax[0].scatter(amrts[0:in_min],amr_mac[0:in_min],s=0.3,label="AMR MAC Projection") + ax[0].scatter(amrts[0:in_min],amr_nodal[0:in_min],s=0.3,label="AMR Nodal Projection") + ax[0].set_title('AMR-Wind') + + ax[1].scatter(nmts_filt,nmd_filt,s=0.3,label="Nalu Momentum") + ax[1].scatter(ncts_filt,ncd_filt,s=0.3,label="Nalu Continuity") + ax[1].set_title('Nalu-Wind') + + for i in range(2): + ax[i].set_ylabel('N') + ax[i].set_xlabel('Timestep') + ax[i].legend() + ax[i].set_ylim([0,50]) + + fig.tight_layout() + fig.savefig('iters.png') + + diff --git a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/mesh-nrel5mw-abl.png b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/mesh-nrel5mw-abl.png deleted file mode 100755 index acfb8bdd..00000000 Binary files a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/mesh-nrel5mw-abl.png and /dev/null differ diff --git a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/rigid_output.png b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/rigid_output.png index fc2cfdb6..150dbed9 100755 Binary files a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/rigid_output.png and b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/rigid_output.png differ diff --git a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/split_mesh_layout.png b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/split_mesh_layout.png new file mode 100755 index 00000000..bd83efec Binary files /dev/null and b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/split_mesh_layout.png differ diff --git a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/timepertimestep.png b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/timepertimestep.png index 36bf7a6f..8f6e7452 100755 Binary files a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/timepertimestep.png and b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/timepertimestep.png differ diff --git a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/timeseries_process.py b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/timeseries_process.py index 20321882..cec041fc 100755 --- a/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/timeseries_process.py +++ b/exawind/NREL_5MW_Turbine/rigid/figures_and_scripts/timeseries_process.py @@ -58,45 +58,74 @@ def main(): args = parser.parse_args() - case_list = ['nrel5mw_rigid_abl_noopenfast2'] - force_file_names = ['forces01.dat'] + case_list = ['rigid_abl_splitmsh_nate2'] + force_file_names = ['forcesBlades.dat'] case_lab = ['NREL 5MW Rigid'] - omega = 1.25663706 + omega = 1.26710903694788 diffn = 60000 rotaxis = [0.862729916,0.498097349,-0.087155742] matplotlib.rcParams['font.size'] = 16 - fig = plt.figure(constrained_layout=True,figsize=(13,4)) + fig = plt.figure(constrained_layout=True,figsize=(12,4)) subfigs = fig.subfigures(nrows=1, ncols=1) - ax = subfigs.subplots(nrows=1, ncols=2) + ax = subfigs.subplots(nrows=1, ncols=3) for i,c in enumerate(case_list): print('Processing: ',c) - fullpath = os.path.join(args.directory,c,force_file_names[i]) - print(fullpath) + this_data = [] - this_data = pd.read_csv(fullpath,sep='\s+',skipinitialspace=True) + for j in range(len(force_file_names)): + fullpath = os.path.join(args.directory,c,force_file_names[j]) + print(fullpath) - this_data['Thrust'] = (this_data['Fpx']*rotaxis[0] + this_data['Fpx']*rotaxis[1])/1000.0 + this_data.append(pd.read_csv(fullpath,sep='\s+',skipinitialspace=True)) - ax[0].plot(this_data['Time'], this_data['Thrust'], label=case_lab[i]) - #plt.axhline(y = ofpower, color = 'k', linestyle = ':') + yawangle = 30.0*np.pi/180.0 + tiltangle = 5.0*np.pi/180.0 + + + all_data = pd.concat(this_data, ignore_index=True) + + + all_data['fx'] = all_data['Fpx']+all_data['Fvx'] + all_data['fy'] = all_data['Fpy']+all_data['Fvy'] + all_data['fz'] = all_data['Fpz']+all_data['Fvz'] + + all_data['frot30x'] = all_data['fx']*np.cos(-yawangle)-all_data['fy']*np.sin(-yawangle) + all_data['frot30y'] = all_data['fx']*np.sin(-yawangle)+all_data['fy']*np.cos(-yawangle) + all_data['frot30z'] = all_data['fz'] + + all_data['mrot30x'] = all_data['Mtx']*np.cos(-yawangle)-all_data['Mty']*np.sin(-yawangle) + all_data['mrot30y'] = all_data['Mtx']*np.sin(-yawangle)+all_data['Mty']*np.cos(-yawangle) + all_data['mrot30z'] = all_data['Mtz'] + + all_data['Torque'] = (all_data['mrot30x']*np.cos(-tiltangle)+all_data['mrot30z']*np.sin(-tiltangle))/1000 + all_data['Thrust'] = (all_data['frot30x']*np.cos(-tiltangle)+all_data['frot30z']*np.sin(-tiltangle))/1000 + + all_data['Power'] = all_data['Torque']*omega*0.944 + + ax[0].plot(all_data['Time'], all_data['Thrust'], label=case_lab[i]) ax[0].set_xlabel("Time [s]") ax[0].set_ylabel("Thrust [kN]") - ax[0].set_ylim([150,900]) - #ax.set_xlim([0,30]) - #ax[0].legend() - ax[1].plot(this_data['Time'], this_data['Mtx']/1000.0, label="x-dir Moment") - ax[1].plot(this_data['Time'], this_data['Mty']/1000.0, label="y-dir Moment") - ax[1].plot(this_data['Time'], this_data['Mtz']/1000.0, label="z-dir Moment") + ax[1].plot(all_data['Time'], all_data['Torque'], label=case_lab[i]) ax[1].set_xlabel("Time [s]") - ax[1].set_ylabel("Moment [kN-m]") - ax[1].legend() + ax[1].set_ylabel("Torque [kN-m]") + + ax[2].plot(all_data['Time'], all_data['Power'], label=case_lab[i]) + ax[2].set_xlabel("Time [s]") + ax[2].set_ylabel("Power [kW]") + + mean_data = all_data[all_data['Time']>30.0].mean() + + print('Mean Power: ',mean_data.Power,'kW') + print('Mean Thrust: ',mean_data.Thrust,'kN') + print('Mean Torque: ',mean_data.Torque,'kN-m') + plt.savefig('rigid_output.png') plt.close() diff --git a/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_amr.inp b/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_amr.inp index 0ddc9748..2bc806e6 100755 --- a/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_amr.inp +++ b/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_amr.inp @@ -4,7 +4,7 @@ time.max_step = -1 time.fixed_dt = 0.003443526170799 # Fixed timestep size (in seconds). If negative, then time.cfl is used time.plot_interval = 1440 time.checkpoint_interval = 1440 -time.checkpoint_start = 0 +time.checkpoint_start = 30000 incflo.physics = ABL # List of physics models to include in simulation. incflo.verbose = 0 io.plot_file = out/plt @@ -228,81 +228,49 @@ io.line_plot_int = 1 CoriolisForcing.turn_off_vertical_force = True ABL.enable_hybrid_rl_mode = true -amr.max_grid_size = 128 -amr.blocking_factor = 32 +amr.max_grid_size = 32 +amr.blocking_factor = 16 - +#-----------------------------------------# +# LINEAR SOLVER # +#-----------------------------------------# +amrex.abort_on_out_of_gpu_memory = 1 +amrex.the_arena_is_managed = 0 +amrex.fpe_trap_invalid = 1 # Trap NaNs +amrex.throw_exception = 1 +amrex.signal_handling = 0 +## +nodal_proj.verbose = 0 +nodal_proj.mg_rtol = 1.0e-6 +nodal_proj.mg_atol = 1.0e-10 +nodal_proj.num_pre_smooth = 10 +nodal_proj.num_post_smooth = 10 +nodal_proj.bottom_rtol = 1.0e-9 +nodal_proj.bottom_atol = 1.0e-12 +## +mac_proj.verbose=0 +mac_proj.do_nsolve = true +mac_proj.bottom_verbose = 0 +mac_proj.mg_rtol = 1.0e-6 +mac_proj.mg_atol = 1.0e-10 +mac_proj.bottom_rtol = 1.0e-9 +mac_proj.bottom_atol = 1.0e-12 +mac_proj.num_pre_smooth = 10 +mac_proj.num_post_smooth = 10 +## +diffusion.verbose=0 +diffusion.do_agglomeration = false +diffusion.do_nsolve = true diffusion.mg_rtol = 1e-06 -diffusion.mg_atol = 1e-10 -temperature_diffusion.mg_rtol = 1e-06 -temperature_diffusion.mg_atol = 1e-10 - -nodal_proj.mg_rtol = 1e-06 -nodal_proj.mg_atol = 1e-10 -mac_proj.mg_rtol = 1e-06 -mac_proj.mg_atol = 1e-10 - -nodal_proj.max_coarsening_level = 0 -nodal_proj.num_pre_smooth = 10 -nodal_proj.num_post_smooth = 10 -nodal_proj.bottom_rtol = 1.0e-9 -nodal_proj.bottom_atol = 1.0e-12 -mac_proj.verbose = 0 -mac_proj.max_coarsening_level = 0 -mac_proj.do_nsolve = true -mac_proj.bottom_verbose = 0 -mac_proj.bottom_rtol = 1.0e-11 -mac_proj.bottom_atol = 1.0e-12 -mac_proj.num_pre_smooth = 10 -mac_proj.num_post_smooth = 10 -diffusion.verbose = 0 -diffusion.do_agglomeration = false -diffusion.do_nsolve = true -velocity_diffusion.verbose = 0 -velocity_diffusion.use_tensor_operator = false -velocity_diffusion.use_segregated_operator = true -velocity_diffusion.do_nsolve = true +diffusion.mg_atol = 1e-10 +## +velocity_diffusion.verbose=0 +velocity_diffusion.use_tensor_operator = false +velocity_diffusion.use_segregated_operator = true +velocity_diffusion.do_nsolve = true +## temperature_diffusion.verbose = 0 -temperature_diffusion.do_nsolve = true -amrex.abort_on_out_of_gpu_memory = 1 -amrex.the_arena_is_managed = 0 -amrex.hypre_spgemm_use_vendor = 1 -amrex.hypre_spmv_use_vendor = 0 -amrex.hypre_sptrans_use_vendor = 0 -amrex.hypre_umpire_device_pool_mbs = 4096 -hypre.bamg_verbose = 0 -hypre.verbose = 0 -hypre.recompute_preconditioner = 0 -hypre.hypre_solver = GMRES -hypre.hypre_preconditioner = BoomerAMG -hypre.num_krylov = 40 -hypre.max_iterations = 40 -hypre.rtol = 1.e-5 -hypre.atol = 1.e-8 -hypre.bamg_coarsen_type = 8 -hypre.bamg_interp_type = 6 -hypre.bamg_relax_type = 11 -hypre.bamg_num_sweeps = 2 -hypre.bamg_cycle_type = 1 -hypre.bamg_relax_order = 0 -hypre.bamg_trunc_factor = 0.45 -hypre.bamg_strong_threshold = 0.3 -hypre.bamg_min_coarse_size = 1 -hypre.bamg_max_coarse_size = 100 -hypre.bamg_agg_num_levels = 0 -hypre.bamg_agg_interp_type = 7 -hypre.bamg_agg_pmax_elmts = 3 -hypre.bamg_pmax_elmts = 3 -hypre.bamg_keep_transpose = 1 -hypre.write_matrix_files = 0 -hypre.bamg_smooth_type = 5 -hypre.bamg_smooth_num_sweeps = 1 -hypre.bamg_smooth_num_levels = 0 -hypre.bamg_ilu_type = 0 -hypre.bamg_ilu_level = 0 -hypre.bamg_ilu_max_iter = 1 -hypre.bamg_ilu_reordering_type = 0 -hypre.bamg_ilu_tri_solve = 0 -hypre.bamg_ilu_lower_jacobi_iters = 3 -hypre.bamg_ilu_upper_jacobi_iters = 3 -#== END AMR-WIND INPUT == +temperature_diffusion.do_nsolve = true +temperature_diffusion.mg_rtol = 1e-06 +temperature_diffusion.mg_atol = 1e-10 +## \ No newline at end of file diff --git a/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_nalu.yaml b/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_nalu.yaml index 8b5b1517..8ee63f26 100755 --- a/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_nalu.yaml +++ b/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_nalu.yaml @@ -16,8 +16,8 @@ linear_solvers: method: hypre_gmres preconditioner: boomerAMG tolerance: 1e-5 - max_iterations: 200 - kspace: 200 + max_iterations: 1000 + kspace: 1000 sync_alg: 1 output_level: 0 write_matrix_files: no @@ -36,8 +36,8 @@ linear_solvers: method: hypre_gmres preconditioner: boomerAMG tolerance: 1e-5 - max_iterations: 200 - kspace: 200 + max_iterations: 1000 + kspace: 1000 sync_alg: 1 output_level: 0 write_matrix_files: no @@ -55,8 +55,8 @@ linear_solvers: method: hypre_gmres preconditioner: boomerAMG tolerance: 1e-5 - max_iterations: 200 - kspace: 200 + max_iterations: 1000 + kspace: 1000 sync_alg: 1 output_level: 0 write_matrix_files: no @@ -69,9 +69,8 @@ linear_solvers: hypre_cfg_node: hypre_elliptic realms: - - name: realm_1 - mesh: /pscratch/ndeveld/hfm-2025-q1/mesh/nrel5mw_672/tower_and_blades.exo + mesh: /pscratch/ndeveld/hfm-2025-q1/mesh/nrel5mw_nate_split_withtower_672/nrel5mw_nate_splitwithtower.exo #automatic_decomposition_type: rcb #rebalance_mesh: yes #stk_rebalance_method: parmetis @@ -119,7 +118,7 @@ realms: pressure: 0.0 velocity: [9.8726896031426, 5.7, 0.0] turbulent_ke: 1.0 - specific_dissipation_rate: 250.0 + specific_dissipation_rate: 687.25 material_properties: target_name: @@ -170,22 +169,37 @@ realms: overset_connectivity_type: tioga overset_user_data: mesh_tag_offset: 0 + tioga_populate_inactive_part: false tioga_options: symmetry_direction: 3 mesh_group: - - overset_name: Turbine + - overset_name: ovst-b1 mesh_parts: - blade1-HEX + wall_parts: + - blade1 + ovset_parts: + - outerbcblade1block + - overset_name: ovst-b2 + mesh_parts: - blade2-HEX + wall_parts: + - blade2 + ovset_parts: + - outerbcblade2block + - overset_name: ovst-b3 + mesh_parts: - blade3-HEX + wall_parts: + - blade3 + ovset_parts: + - outerbcblade3block + - overset_name: ovst-tower + mesh_parts: - tower-fluid-HEX wall_parts: - - blade1 - - blade2 - - blade3 - tower-wall ovset_parts: - - outerBC - tower-overset mesh_transformation: @@ -195,7 +209,6 @@ realms: - blade2-HEX - blade3-HEX motion: - #tilt done in original mesh location of [0, 0, 0] - type: rotation angle: 5.0 axis: [0.0, 1.0, 0.0] @@ -206,8 +219,8 @@ realms: centroid: [0.0, 0.0, 0.0] # then translated to the hub_ref_position as computed by openfast - type: translation - displacement: [1795, 1800, 90] - + displacement: [1795.669873, 1797.5, 90] + - name: turbine_tower_placement mesh_parts: - tower-fluid-HEX @@ -221,10 +234,10 @@ realms: frame: non_inertial motion: - type: rotation - omega: 1.25663706 # This is 12.0 rpm in rad/s + omega: 1.26710903694788 # This is 12.1 rpm in rad/s #axis: [0.996194698, 0.0, -0.087155742] # 5 degree rotation Y axis: [0.862729916,0.498097349,-0.087155742] # 5 deg Y + 30 deg Z - centroid: [1795, 1800, 90] + centroid: [1795.669873, 1797.5, 90] solution_options: name: myOptions @@ -246,7 +259,7 @@ realms: specific_dissipation_rate: 1.0 - upw_factor: - velocity: 0.0 + velocity: 1.0 turbulent_ke: 0.0 specific_dissipation_rate: 0.0 @@ -269,13 +282,13 @@ realms: - relaxation_factor: velocity: 0.7 pressure: 0.3 - turbulent_ke: 0.7 - specific_dissipation_rate: 0.7 + turbulent_ke: 0.5 + specific_dissipation_rate: 0.5 post_processing: - type: surface physics: surface_force_and_moment - output_file_name: forces01.dat + output_file_name: forcesBlades.dat frequency: 4 parameters: [1795.669873, 1797.5, 90] target_name: @@ -309,13 +322,13 @@ realms: compute_vorticity: yes restart: - restart_data_base_name: rst01/nrel5mw.rst + restart_data_base_name: rst/nrel5mw.rst restart_frequency: 1440 restart_start: 0 output: - output_data_base_name: out01/nrel5mw.e - output_frequency: 1440 + output_data_base_name: out/nrel5mw.e + output_frequency: 180 output_node_set: no output_variables: - velocity diff --git a/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_nalu_r1.yaml b/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_nalu_r1.yaml deleted file mode 100755 index d24d66ff..00000000 --- a/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_nalu_r1.yaml +++ /dev/null @@ -1,345 +0,0 @@ -Simulations: -- name: sim1 - time_integrator: ti_1 - optimizer: opt1 - -hypre_config: - umpire_device_pool_mbs: 4096 - use_vendor_spgemm: 1 - use_vendor_spmv: 0 - use_vendor_sptrans: 0 - -linear_solvers: - # solver for the momentum - - name: solve_mom - type: hypre - method: hypre_gmres - preconditioner: boomerAMG - tolerance: 1e-5 - max_iterations: 200 - kspace: 200 - sync_alg: 1 - output_level: 0 - write_matrix_files: no - #write_preassembly_matrix_files: yes - recompute_preconditioner_frequency: 1 - reuse_linear_system: yes - simple_hypre_matrix_assemble: yes - dump_hypre_matrix_stats: no - segregated_solver: yes - hypre_cfg_file: hypre_file.yaml - hypre_cfg_node: hypre_simple_precon - - # solver for the scalars TKE, SDR - - name: solve_scalar - type: hypre - method: hypre_gmres - preconditioner: boomerAMG - tolerance: 1e-5 - max_iterations: 200 - kspace: 200 - sync_alg: 1 - output_level: 0 - write_matrix_files: no - #write_preassembly_matrix_files: yes - recompute_preconditioner_frequency: 1 - reuse_linear_system: yes - simple_hypre_matrix_assemble: yes - dump_hypre_matrix_stats: no - hypre_cfg_file: hypre_file.yaml - hypre_cfg_node: hypre_simple_precon - - # solver for the pressure - - name: solve_elliptic - type: hypre - method: hypre_gmres - preconditioner: boomerAMG - tolerance: 1e-5 - max_iterations: 200 - kspace: 200 - sync_alg: 1 - output_level: 0 - write_matrix_files: no - #write_preassembly_matrix_files: yes - recompute_preconditioner_frequency: 1 - reuse_linear_system: yes - simple_hypre_matrix_assemble: yes - dump_hypre_matrix_stats: no - hypre_cfg_file: hypre_file.yaml - hypre_cfg_node: hypre_elliptic - -realms: - -- name: realm_1 - #mesh: /projects/wind/hpc/wind/meshes/connected/nrel5mw/tower_and_blades/tower_and_blades.exo - mesh: /pscratch/ndeveld/hfm-2025-q1/nrel5mw_rigid_abl_noopenfast1/rst01/nrel5mw.rst - #automatic_decomposition_type: rcb - #rebalance_mesh: yes - #stk_rebalance_method: parmetis - use_edges: yes - check_jacobians: true - - equation_systems: - name: theEqSys - max_iterations: 4 - decoupled_overset_solve: yes - - solver_system_specification: - velocity: solve_mom - turbulent_ke: solve_scalar - specific_dissipation_rate: solve_scalar - pressure: solve_elliptic - ndtw: solve_elliptic - - systems: - - WallDistance: - name: myNDTW - max_iterations: 1 - convergence_tolerance: 1.0e-8 - update_frequency: 1000000000 - exchange_fringe_data: false - - - LowMachEOM: - name: myLowMach - max_iterations: 1 - convergence_tolerance: 1e-8 - - - ShearStressTransport: - name: mySST - max_iterations: 1 - convergence_tolerance: 1e-8 - - initial_conditions: - - constant: ic_1 - target_name: - - blade1-HEX - - blade2-HEX - - blade3-HEX - - tower-fluid-HEX - value: - pressure: 0.0 - velocity: [9.8726896031426, 5.7, 0.0] - turbulent_ke: 1.0 - specific_dissipation_rate: 250.0 - - material_properties: - target_name: - - blade1-HEX - - blade2-HEX - - blade3-HEX - - tower-fluid-HEX - - specifications: - - name: density - type: constant - value: 1.0 - - name: viscosity - type: constant - value: 1.8e-5 - - boundary_conditions: - - - wall_boundary_condition: bc_wall - target_name: blade1 - wall_user_data: - use_wall_function: no - turbulent_ke: 0.0 - - - wall_boundary_condition: bc_wall - target_name: blade2 - wall_user_data: - use_wall_function: no - turbulent_ke: 0.0 - - - wall_boundary_condition: bc_wall - target_name: blade3 - wall_user_data: - use_wall_function: no - turbulent_ke: 0.0 - - - wall_boundary_condition: bc_wall - target_name: tower-wall - wall_user_data: - use_wall_function: no - turbulent_ke: 0.0 - - - symmetry_boundary_condition: ground - target_name: tower-ground - symmetry_user_data: - - - overset_boundary_condition: bc_overset - overset_connectivity_type: tioga - overset_user_data: - mesh_tag_offset: 0 - tioga_options: - symmetry_direction: 3 - mesh_group: - - overset_name: Turbine - mesh_parts: - - blade1-HEX - - blade2-HEX - - blade3-HEX - - tower-fluid-HEX - wall_parts: - - blade1 - - blade2 - - blade3 - - tower-wall - ovset_parts: - - outerBC - - tower-overset - - # mesh_transformation: - # - name: turbine_rotor_placment - # mesh_parts: - # - blade1-HEX - # - blade2-HEX - # - blade3-HEX - # motion: - # #tilt done in original mesh location of [0, 0, 0] - # - type: rotation - # angle: 5.0 - # axis: [0.0, 1.0, 0.0] - # centroid: [0.0, 0.0, 0.0] - # - type: rotation - # angle: 30.0 - # axis: [0.0, 0.0, 1.0] - # centroid: [0.0, 0.0, 0.0] - # # then translated to the hub_ref_position as computed by openfast - # - type: translation - # displacement: [1795, 1800, 90] - - # - name: turbine_tower_placement - # mesh_parts: - # - tower-fluid-HEX - # motion: - # - type: translation - # displacement: [1800, 1800, 0] - - mesh_motion: - - name: mover - mesh_parts: [blade1-HEX, blade2-HEX, blade3-HEX] - frame: non_inertial - motion: - - type: rotation - omega: 1.25663706 # This is 12.0 rpm in rad/s - #axis: [0.996194698, 0.0, -0.087155742] # 5 degree rotation Y - axis: [0.862729916,0.498097349,-0.087155742] # 5 deg Y + 30 deg Z - centroid: [1795, 1800, 90] - - solution_options: - name: myOptions - turbulence_model: sst - projected_timescale_type: momentum_diag_inv - - options: - - source_terms: - continuity: density_time_derivative - - - hybrid_factor: - velocity: 1.0 - turbulent_ke: 1.0 - specific_dissipation_rate: 1.0 - - - alpha_upw: - velocity: 1.0 - turbulent_ke: 1.0 - specific_dissipation_rate: 1.0 - - - upw_factor: - velocity: 0.0 - turbulent_ke: 0.0 - specific_dissipation_rate: 0.0 - - - noc_correction: - pressure: yes - - - limiter: - pressure: no - velocity: yes - turbulent_ke: yes - specific_dissipation_rate: yes - - - projected_nodal_gradient: - velocity: element - pressure: element - turbulent_ke: element - specific_dissipation_rate: element - ndtw: element - - - relaxation_factor: - velocity: 0.7 - pressure: 0.3 - turbulent_ke: 0.7 - specific_dissipation_rate: 0.7 - - post_processing: - - type: surface - physics: surface_force_and_moment - output_file_name: forces02.dat - frequency: 4 - parameters: [1795, 1800, 90] - target_name: - - blade1 - - blade2 - - blade3 - - turbulence_averaging: - time_filter_interval: 100000.0 - - specifications: - - - name: one - target_name: - - blade1-HEX - - blade2-HEX - - blade3-HEX - - tower-fluid-HEX - - reynolds_averaged_variables: - - velocity - - compute_q_criterion: yes - compute_vorticity: yes - - restart: - restart_data_base_name: rst01/nrel5mw.rst - restart_frequency: 4000 - restart_start: 0 - - output: - output_data_base_name: out01/nrel5mw.e - output_frequency: 2000 - output_node_set: no - output_variables: - - velocity - - pressure - - turbulent_ke - - specific_dissipation_rate - - minimum_distance_to_wall - - turbulent_viscosity - - iblank - - iblank_cell - - mesh_displacement - - mesh_velocity - - q_criterion - - vorticity - - disp_map - - disp_map_interp - - div_mesh_velocity - - element_courant - - dual_nodal_volume - -Time_Integrators: -- StandardTimeIntegrator: - name: ti_1 - start_time: 151.515151515129 - time_step: 0.003443526170799 - termination_step_count: 200000000 - time_stepping_type: fixed - time_step_count: 0 - second_order_accuracy: yes - nonlinear_iterations: 1 - - realms: - - realm_1 diff --git a/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_r1.yaml b/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_r1.yaml deleted file mode 100755 index eb4798f0..00000000 --- a/exawind/NREL_5MW_Turbine/rigid/input_files/nrel5mw_r1.yaml +++ /dev/null @@ -1,20 +0,0 @@ -# Example input file -exawind: - nalu_wind_inp: - - nrel5mw_nalu_r1.yaml - amr_wind_inp: nrel5mw_amr_refined_r1.inp - num_timesteps: 1440000 - nonlinear_iterations: 1 - - # Variables for overset exchange - nalu_vars: - - velocity - - turbulent_ke - - specific_dissipation_rate - - pressure - amr_cell_vars: - - velocity - - tke - - sdr - amr_node_vars: - - p \ No newline at end of file