Global Numbering in parallel

[a-jp]

I am working to read my own mesh format in parallel, which means I've been modifying the construct API to work with mix cell type meshes. This appears to be working based on looking at the mesh in paraview. I'm calling makeZoltanBalancer and reorderMdsMesh afterwards to get better partitioning and to try to break things: my assertion is that repartitioning and renumbering will break things if I've made a mistake in developing the construct API further or when building the mesh from my own mesh format. Again still seems ok based at looking at the mesh in paraview.

However, what I'd like to see in paraview is the global number of the vertices, and the global number of the cells. And to compare these before and after various stages of my algorithm to debug. Currently the only integer scalar on my vtu mesh is apf_part. Are there any simple functions that I can call that can number in parallel and attach a global cell/element number and global vertex number to the APF Mesh that are also then written out when I call the writeVtkFiles function?

Thanks again,
Andy

[a-jp]

Did anyone have any thoughts on how I can globally and uniquely number in parallel for vertices and cells? It would really help me debug.
Thanks
Andy

[cwsmith]

The following sequence will create a global number of mesh vertices:

core/zoltan/apfZoltanCallbacks.cc

Lines 210 to 216 in d9eeb2b

	Numbering* ln = NULL;
	GlobalNumbering* gn = NULL;
	if( zb->isLocal ) {
	ln = numberOverlapNodes(zb->mesh, "zoltan_vtx");
	} else {
	gn = makeGlobal(numberOwnedNodes(zb->mesh, "zoltan_vtx"));
	synchronize(gn);

specifically,

GlobalNumbering* gn = makeGlobal(numberOwnedNodes(mesh,"someNameThatAppearsInParaview"));
synchronize(gn);

and the following call will globally number elements:

core/zoltan/apfInterElement.cc

Line 98 in d9eeb2b

GlobalNumbering* gn = makeGlobal(numberElements(m, "zb_element"));

My understanding is that a synchronize call is not required for elements since they are uniquely assigned to parts/processes.

The header for the numbering APIs is here:

https://github.com/SCOREC/core/blob/d9eeb2b028afd6182b4d3bc1063d3aae3a348760/apf/apfNumbering.h

The .vtk files produced by apf::writeVtkFiles(...) will contain fields named according to the numbering calls.

[a-jp]

Thank you very much @cwsmith. Can I ask, having used the above to number the vertices and the elements, what should I expect after a call to:

	apf::MeshTag *order = Parma_BfsReorder(m);
	apf::reorderMdsMesh(m, order);

Should I expect that the values in those arrays (zb_element, zoltan_vtx) will also be updated after reorder is called? I appreciate that I could just call reorder before numbering, but for the sake of understanding.

[a-jp]

@cwsmith I have found that after calling:

	apf::MeshTag *weights = m->createDoubleTag("zoltan_weight", 1);
	{
		apf::MeshIterator *it = m->begin(m->getDimension());
		apf::MeshEntity *e;
		double value = 1.0;
		while ((e = m->iterate(it)))
			m->setDoubleTag(e, weights, &value);
		m->end(it);
	}
	apf::Balancer *b = apf::makeZoltanBalancer(m, apf::RCB, apf::REPARTITION, false);
	b->balance(weights, 1.1);
	m->destroyTag(weights);
	delete b;

that (zb_element, zoltan_vtx) as per the above functions are removed from the mesh if I write out to vtk the rebalanced mesh. Should I expect that to happen?
Thanks,
Andy

[cwsmith]

This issue discusses tags vs fields and their visualization:
#250

Please reopen this issue, or create a new one, if there are follow up questions on global numbering.

[a-jp]

My understanding is that a synchronize call is not required for elements since they are uniquely assigned to parts/processes.

@cwsmith Sorry to open this again. While looking for how to do something else I came across this code:
Sync. It appears to indicate that you should synchronize elements, as well as elements of d-1 (and presumably in 3D d-2) along with the nodes.

What is the correct way to do this. Should all calls to generate global numbering be synchronized? Or is the linked code being overkill?

Aside from a speed/memory cost, is there any negative downside of doing it, will it break anything?

In asking this my implicit assumption is that:

apf::GlobalNumbering *gcn = nullptr;
gcn = apf::makeGlobal(apf::numberElements(m, "element-nums"));

Is identical to:

int d = m->getDimension();
apf::GlobalNumbering *gcn = nullptr;
gcn = apf::makeGlobal(
      apf::numberOwnedDimension(m, "element-nums", d));
synchronize(gcn);

If they are not identical then that may explain why the latter is sync'd, but the former does not need to be. Can someone clarify this?