Partial reconfiguration support.

[nathanchandanson]

Hi everyone.

As the title suggests, I am trying to do dynamic partial reconfiguration. Apparently this has already been done in FlexBex and ICESOC.

I've read the config part of the fabric, and from what I understand, I need to send to the config subsystem all the frames from the columns I want to reconfigure. Is this the case?

If so, then the challenge is to make sure that when nextpnr performs place and route, it constrains the design to different slots.
However, I could not find any information on how this could be enforced. Is there documentation/examples somewhere?

[KelvinChung2000]

This is an example.

split.zip

You will need to use the latest main version of nextpnr, as the feature required was recently merged.

In the example above, the "inner" is what you want to partially reconfigure, and the outer is your harness.

The general flow is: first place and route the harness, which gives you a fasm file; then look at the fasm and use the prohibit command to lock out those resources; then place and route with your partial module. I have only tested on this very simple case for now, so I have done the extraction by hand. You will want to create a script that reads in the fasm and exports the prohibit command from there.

This explains how partial reconfiguration is modelled in nextpnr.
YosysHQ/nextpnr#1531

[nathanchandanson]

Thank you for your answer.

If I understand correctly, the static part is the one responsible for interfacing the partial modules with the IOs using the pseudo plugs defined by set_pseudo_plug in the outer.pcf file.
Then, in the partial module constraints inner.pcf, I prevent nextpnr from using the resources used for the static part using the prohibit_{pip/wire/bel} commands.

However, if I do this, then when generating the partial modules, there is no guarantee that they will be placed inside a specific region. Then if I want to swap the partial modules, I cannot make sure that they will not overlap.
Did I miss something?

My goal would be to define independent regions of the fabric on which I could dynamically swap different partial modules to perform different functions, something that would look like this:

[KelvinChung2000]

For this specific use case, you will actually want to set prohibit to all resources except your partial slots, which generates a "region". What I should have done is provide a set_partial_region command that makes only those resources available. The prohibit command supports regex, so getting that should not be very tedious. But still, I agree not very nice.

[nathanchandanson]

Ok great thank you.

I found in the nextpnr docs (https://github.com/YosysHQ/nextpnr/blob/main/docs/netlist.md) that the router cannot constrain a net to a specific region :

region is a reference to a Region if the net is constrained to a device region or nullptr otherwise (N.B. not supported by the current router).

Doesn't that prevent doing what I am trying to do ?
Because even though it is possible to constrain placement of BELs inside a region, I cannot be sure that the routing will not be shared between partial modules.
Or does the prohibit command provide a solution to this ?

[KelvinChung2000]

The prohibit command preoccupies those wires and pips, so the router cannot use them.

[KelvinChung2000]

Something just came to mind that I did not think about when I implemented the command feature. The prohibit wire/pip command creates a dummy net that cannot be placed/routed. You should get an error on the dummy net not routed, but if you have the FASM output, it will still export the result. I will try to work on a fix ASAP.

The current clean error-free workaround is to remove those wires/pipes from pips.txt, which will make them absent from the placement and route graph. You can check this out https://github.com/NeverInControl/FABulousPartialReconfigurationTool , and @NeverInControl will be able to give you better guidance.

[nathanchandanson]

Thank you I will try to use the Partial Reconfiguration Tool !

[NeverInControl]

Hi, if you have any question reguarding the tool, feel free to ask, I am available.
The current version of the PRT was written for an older nextpnr version, which did not have the capabilities to route to wires, which means the PRT you have here requires you to burn a full LUT for every connection for one region to another, which is admittedly pretty expensive. I am working on a newer version for the new nextpnr version, that will be able to route directly using @KelvinChung2000 's pcf files, but that will take some time as I am currently occupied elsewhere.

Edit:
the PRT was developed independently of the stuff @KelvinChung2000 described above, and the philosophy is a bit different in some places. (e.g. no static and partial regions, instead all regions are equal and have predefined boundries instead of using leftover resources.)

[nathanchandanson]

Great thank you.

Just to be sure, as the configuration of a FABulous fabric is done frame by frame in each column, for your write_minimal_bitstream command, it will only remove the frames that have not changed ?

And for the BELs that are not reconfigured, they will get the config bits they already have in case another BEL in the same column is reconfigured ?

[NeverInControl]

So, fun fact: when you overwrtie configuration bits with the same bits in FABulous, the bits will stay perfectly constant, no jitter or flipping. (Because of the latch architecture in FAbulous to store the configuration bits, and the fact that the bitstream is not pushed in a chain, but in parallel, unlike some other architectures.) So if you have your chip running, you could permanently overwrite the configuration with the same configuration and the chip will run perfectly as normal through that.
So if you want pertial reconfiguration, you could just upload the hole bitstream, with some changed bits and some unchanged bits, and your static region will run through that reconfig fine, no issues.

But pushing a hole bitstream takes some time, so I wanted to slim it down to reduce reconfiguration time. The PRT keeps a state internally that is a perfect resemblence to the current state of the chip. When you upload a new bitstream, write_minimal_bitstream looks at that saved state and finds all the frames where bits are changed, and then pushes only these to the chip. That is a bit weird if the saved state and the actual state desync, so you have to restart the PRT if you reset the chip.

But you don't have to worry if a frame you write overlaps with a changed AND an unchanged region, the unchanged region will not notice your reconfiguration despite being written by the frame. As long as the config bits stay exactly the same.

[Note: depending on your specific design and used version, frames might not perfectly correspond to columns, but i think in practice that mostly holds true]

[KelvinChung2000]

A message from Dirk:

Great discussion here!
"all the frames from the columns I want to reconfigure. Is this the case?"
Yes - this is the case.
However, you can still update each bit individually (a requirement that you need for real-world PR) by only updating the framebits that you need to touch while keeping the remaining FrameData in place. But you understood that already.

The configuration bitstream frames contain first an address with major (which CLB/DSP/etc. column) and minor (which frame inside a column) followed by the actual frame date (32-bits per CLB row).
Therefore, we can write frames selectively and in any order.
The only thing to mind here is the Desyc command at the end.
A configuration bitfile is required to be 32-bit word-alllinged and the actual configuration starts when we see the pattern FAB0 FAB1 (so you can have a header etc. in front). Desync will ask the config state machine to scan for the next FAB0 FAB1.

For partial reconfiguration we follow two strategies (that both make sense, depending on the use case):
1) filter out the resources from the full nextpnr architecture graph.
2) use prohibit constrains through a PCF.

For toy examples, you can live with strict bounding boxes, in reality, you have to foresee that signals may have to cross a PR-Region. This requires to define resource partitions (for a static system or one or more partial regions) basically at individual BEL/Wire level.
We could do what Xilinx does, by just building models as an increment to a static system has many drawbacks (most notably that partial modules don't work after an update of the static system.