[P0] Simulator and rtl comparison (latency gap) debug

[n0thingNoob]

In the histogram kernel, we found an issue, but we don't know why it happens.
So in the program we used(The red blocks are two key ops where the issue happened. ):

The NAH operation in the red block is a routing operation, corresponding to the CTRL_MOV in the previous assembly code. It is expected to take only one cycle, but in the RTL trace, it appears to stall for an extra cycle and completes in the second cycle instead. Since I am still learning how to read the RTL logs, I have attached a CSV file with the key operations for further inspection.

tile_1_0_chain_126_134_key_only.csv

Here is a visualized timeseries graph for the steady cycle in this screenshot. The red block means stalls. Blue and green cycle means it is completed.

Here is the visualized graph for simulator:

Let me also attach the log for both:

histogram.json.log (Simulator log)

trace_histogram_4x4_Mesh.txt (RTL log)

[n0thingNoob]

@tancheng @yyan7223

[tancheng]

Thanks for filing the issue @n0thingNoob, are you able to provide the reproduce commands? e.g., a pymtl test?

@yyan7223 can you help to investigate why there is an additional cycle (doing nothing) between grant (on tile[1,1]) and phi (on tile [1,0])?

[n0thingNoob]

The histogram pytest file is in (https://github.com/tancheng/VectorCGRA/tree/fulfill-kernel-legacy/cgra/test)

histogram_assembly_code.txt, I can also provide the full assembly code here.

[yyan7223]

Sure, let me have a look

[yyan7223]

@n0thingNoob
My environment has an issue with yaml. Can you generate the vcd file for me on your side?
Run pytest CgraRTL_histogram_test_from_yaml.py --dump-vcd

[n0thingNoob]

@n0thingNoob My environment has an issue with yaml. Can you generate the vcd file for me on your side? Run pytest CgraRTL_histogram_test_from_yaml.py --dump-vcd

Sure. Here it is. I have to change it to txt to upload.

VectorCGRA.cgra.test.CgraRTL_histogram_test_from_yaml__test_homogeneous_4x4_histogram_combinational_mem_access.txt

[yyan7223]

Summary:

The issue where NAH takes two cycles is a bug. It occurs in the scenario where data is routed to the register cluster while the fu executes NAH.

Root cause:

VectorCGRA/mem/register_cluster/RegisterClusterRTL.py

Lines 79 to 86 in a8dd68c

	# Data from register bank has priority over routing crossbar data for FU path.
	# Note: reg_bank[i].send_data.val is set based on read_reg_towards in RegisterBankRTL.
	if s.reg_bank[i].send_data.val & reg_towards_fu:
	s.send_data_to_fu[i].msg @= \
	s.reg_bank[i].send_data.msg
	elif s.recv_data_from_routing_crossbar[i].val:
	s.send_data_to_fu[i].msg @= \
	s.recv_data_from_routing_crossbar[i].msg

When no data is read from the register cluster, the routed data is bypassed to the fu by default (line 84~86). However,

VectorCGRA/mem/register_cluster/RegisterClusterRTL.py

Line 93 in a8dd68c

s.recv_data_from_routing_crossbar[i].rdy @= s.send_data_to_fu[i].rdy

will stall the routing crossbar for 1 cycle because NAH does not consume any operands (s.send_data_to_fu[i].rdy = 0), which is why NAH takes two cycles in this scenario. (This stall does not prevent the data from being routed to the register cluster, so the computational result remains correct.)

Solution:

The CGRA config may include a new field that explicitly indicates whether we need bypass. Then we have:
(1) If bypass is not required (e.g., this scenario only needs to route data to the register cluster), set s.recv_data_from_routing_crossbar[i].rdy @= 1 (register cluster can be written at any time). Then NAH will take only one cycle.
(2) If bypass is required, still use original s.recv_data_from_routing_crossbar[i].rdy @= s.send_data_to_fu[i].rdy for checking.

@tancheng @n0thingNoob WDYT?

[tancheng]

You "bypass" means sending data to FU to serve as input. This can be recongized by fu_in and opt (e.g., if it is NAH, it shouldn't send to FU).

How about:

s.recv_data_from_routing_crossbar[i].rdy @= (ctrl_signal.opt == NAH) | s.send_data_to_fu[i].rdy

[tancheng]

or:

s.recv_data_from_routing_crossbar[i].rdy @= (ctrl_signal.opt == NAH) | (ctrl_signal.fu_in[i] == 0) | s.send_data_to_fu[i].rdy

[yyan7223]

or:

s.recv_data_from_routing_crossbar[i].rdy @= (ctrl_signal.opt == NAH) | (ctrl_signal.fu_in[i] == 0) | s.send_data_to_fu[i].rdy

Oh that's much more easier than my solution!
@n0thingNoob Can you try this on your side to see whether the bug is fixed?

[n0thingNoob]

or:

s.recv_data_from_routing_crossbar[i].rdy @= (ctrl_signal.opt == NAH) | (ctrl_signal.fu_in[i] == 0) | s.send_data_to_fu[i].rdy

Oh that's much more easier than my solution! @n0thingNoob Can you try this on your side to see whether the bug is fixed?

@Jackcuii, can you try it? My server was down this weekend.

[tancheng]

@yyan7223 can you help make a PR first？

[Jackcuii]

Hi folks, @tancheng @yyan7223 I tried the change and it works on histogram and axpy. (also need some accompanied change on the script generator) Good job!