merge `BadElfAnalysis` table into `PopulationAnalysis` table

[jacksund]

we might not be able to merge the full BadElfAnalysis into PopulationAnalysis because of custom parameters like elf_connect_cutoff but the badelf table should at least inherit from the population analysis table to centralize functionality

[SWeav02]

Yeah, I agree. I'll also likely be adding more custom parameters to BadElfAnalysis as time goes on because there's a lot more useful information that could be stored.

[SWeav02]

My undergrad and I are planning to run some high throughput calculations. For the workflow we want to run a static energy calc, BadELF, and then Bader. I haven't used the Bader workflows in a while, but it looks like the PopulationAnalysis table is set up to inherit the StaticEnergy table. This means the static energy calculation and bader calculation need to be performed together in a nested workflow that copies the required files from each step to the main directory at the end. I think this is a bit clunky, and will make it more difficult for us to write our custom workflow. I would like to alter the PopulationAnalysis table to not inherit from the StaticEnergy table, similar to the BadELF table. Any workflows that run a static energy calc and then a population analysis could then be constructed using the staged workflow I added in #700. This would also allow me to easily fix this issue (#559) by making the BadElf table inherit directly from the PopulationAnalysis table.

The only problem I see is that this would make the connection between the static energy calculation and population analysis calculation less obvious. However, the population analysis would still be connected to any previous calculation by the source column so I think that's ok.

@jacksund do you think this would be a worthwhile change?

[jacksund]

Yep! Go ahead and make PopulationAnalysis it's own table mixin. But I do think the table for the full "staged" workflow should be a combination of staticenergy + popanalysis

When I originally wrote the staged tables, I intended for the top level table of a staged workflow to go in an pull out all relevant "final" data so that the user could view it in one place. To do this, you don't have to copy all of the files but can choose which values to pass via python -- that's what I do in the staged relaxation:

simmate/src/simmate/engine/staged_workflow.py

Lines 54 to 67 in f330103

	# when updating the original entry, we want to use the data from the
	# final result.
	final_result = {
	"structure": result.to_toolkit(),
	"energy": result.energy,
	"band_gap": result.band_gap,
	"is_gap_direct": result.is_gap_direct,
	"energy_fermi": result.energy_fermi,
	"conduction_band_minimum": result.conduction_band_minimum,
	"valence_band_maximum": result.valence_band_maximum,
	"site_forces": result.site_forces,
	"lattice_stress": result.lattice_stress,
	}
	return final_result

So you can pretty easily get around the clunkyness and file copying that you mentioned.

That said, in the long term, there is a growing problem of linking related workflows that might justify splitting up table types into their own independent ones like you suggest. Currently, we're using things like the "source" column (and I see in #700 that you're trying to add columns like 'subworkflow_names' to your revised staged table). Whereas traditionally, you use foreign keys and many-to-many relationships in django to link things. I think there are two ways forward that clean up this issue... (1) we continue to develop #175 and include component's for subsworkflow linking (which is easier on users but harder on programming out) or (2) pump the brakes on table inheritance and shift to a more standard table setup that uses foreign_keys (which is more rigid and much harder on users). Your suggestions push us towards 2, while the current simmate organization is pretty close to 1

[SWeav02]

I'm not sure I'm understanding the direction you're aiming for, so I'll give what I'm thinking and see what you think. In my mind there are two types of staged workflows that a user might want. In the first, the user is only interested in the final calculations results. For example, in a staged relaxation, the user probably only cares about the final relaxation and won't need to look up much information from the previous relaxations. In the second, the user wants information about multiple calculations. For example, my undergrad and I need information from a static energy, bader and badelf calcs. This second situation is where it seems like decisions really need to be made.

If I understand what you're saying, there are two main ways we could approach this.
In (1) we focus hard on table inheritance. Each type of staged calculation would have a unique table that inherits from each of the important components (and connects to the subworkflows through the "source" column?). For example, a static energy to bader workflow would inherit the StaticEnergy and PopulationAnalysis tables. A static energy to badelf workflow would need a seperate table inheriting the StaticEnergy and BadELF tables. I have a couple issues with this. One that's specific to my situation is that since the BadELF table shares columns with the PopulationAnalysis table, I don't think we can make a table that stores information from both a Bader analysis and BadELF analysis. So we'll need to access each subworkflow table separately anyways. The second is that it makes it harder for users to make custom staged workflows if something similar doesn't already exist in Simmate.

In (2) we shift to less table inheritance and use foreign_keys to link staged workflows to subworkflows. The advantage is that it's a more modular approach, allowing for essentially any combination of subworkflows without needing to create unique tables for each combination of staged workflow. In some ways I think this is easier on users, at least when it comes to making new custom workflows. However, it makes it harder for users to access the results of their staged workflows since the important information is spread across several tables.

I wonder if there is a way to get the best of both worlds? Maybe we can automate the creation of new staged workflow tables? For example, if someone makes a staged workflow with a relaxation, static energy, and badelf calc it would automatically create a table that inherits each of these respective mix-ins and fills it in appropriately at the end of the calc. That way users can still access all of the information at the end like in approach (1) while maintaining much of the modularity of approach (2). Unfortunately this still wouldn't solve my bader/badelf sharing columns problem.
Another option would be to keep a StagedWorkflow table similar to what I have in #700 and give it a convenience function that pulls information from the subworkflow tables and summarizes it in a dataframe for the user. This again allows for the modularity of approach (2) but makes it easier to get a summary of the important information. I think it would also solve my shared columns problem.

@jacksund I think we should decide on an approach sooner rather than later. I'll go ahead and convert the PopulationAnalysis table to a mix-in, since I'll need it to be stand alone for my undergrads project, but our staged workflow currently relies on the StagedWorkflow table I made in #700. I think I still gravitate towards a solution like (2), but I may be biased because it seems easier for my personal use case.

[jacksund]

tons to unpack here haha. Thanks for putting so much thought into this! I think my ideas are shifting/refining as I write some of this, so your reply is helping a bunch.

I think there are a couple of things going on:

1. We're really talking about building summary tables. Summary tables can be anything and should be user defined. In it's current state, it just so happens that many StagedCalculations want their "summary" to be a copy/paste of the final calculation results. But as you point out, there are cases where summaries involve pulling from the results of multiple subworkflows. There are even more cases where summaries might do additional workup (like calculated % difference between your bader and badelf results). There's no way to capture all possible summary table formats, so it's something we need them to define AND also define how it's populated.

2. The StaticEnergy + Bader + BadELF staged calc your talking about is not a staged calculation. It's a DAG with StaticEnergy being first and both Bader + BadELF stemming for its results. So it should be a custom workflow altogether. Forcing it into a StagedCalculation definition overcomplicates things.

---

I wonder if there is a way to get the best of both worlds? Maybe we can automate the creation of new staged workflow tables? For example, if someone makes a staged workflow with a relaxation, static energy, and badelf calc it would automatically create a table that inherits each of these respective mix-ins and fills it in appropriately at the end of the calc.

This is super close to what I'm thinking. However, rather than automate the creation of new tables, I'm saying users should define their own "summary table". Mix-ins like a StaticEnergy or PopulationAnalysis should be optional shortcuts for users looking to create a summary table.

I don't think we can make a table that stores information from both a Bader analysis and BadELF analysis.

Yes, the PopulationAnalysis mix-in doesn't work for your desired summary table. However, you can still make your own custom summary table instead of using the mix-in.

The second is that it makes it harder for users to make custom staged workflows if something similar doesn't already exist in Simmate.

This is fair, but supporting all the different ways a user might want to connect subworkflows is a hefty task. At some point we have to say "coding it out in python is easy enough". I think the docs we have on nested workflows show common cases pretty well and the user can tweak python logic as needed.

StagedCalculation is really just a shortcut of this example, and we shouldn't make the shortcut more complicated than the original code we're trying to replace.

The advantage is that it's a more modular approach, allowing for essentially any combination of subworkflows

This isn't entirely true. One thing I glossed over is that foreign keys and many-to-many relationships are explicit to which tables they point to. For example, say I wanted a column that indicates the source of the input structure. Django needs to know exactly which table that foreign key should point to. However in Simmate (and comp chem in general) that source could be from any number of tables -- matproj, a static energy calc, a relaxation output, a user file, etc. That's why I added the source column which takes a dict w. table_name as one of the keys. There are django extensions that come close to what I needed to fix this (like django-polymorphic), but nothing that fit perfectly. The "real" solution would be to have a single Structure table and have everything else be truly relational to this table. This is actually the standard -- which I went against because it's tricky for beginners and would also mean we have a single table with ALL structures from ALL sources, creating performance issues. Simmate is unique in that it uses table mix-ins

---

All in all, I guess I'm saying two things:

1. your StagedCalculation model should really be a NestedWorkflowSummary mix-in that helps build out common columns for summaries involving multiple calcs (where they are staged, a DAG, or something else). Columns like total_subflow_cpu_time could be a thing. Then the user builds their own summary table (potentiall using this mixin)
2. The StagedCalculation workflow becomes a basic utility that runs all flows and results all results. It's on the user to use these results to fill the summary table.

** then (optionally) we have prebuilt workflows like StagedRelaxation, but these should be limited in scope and use cases to something that's manageable for us.

All of this is difficult to articulate, but hopefully I'm making sense...? Maybe we should talk more about what the code should end up looking like (from the user's perspective).