Porting OpenJ9 to RISC-V

[ChengJin01]

We determined to port a complete runtime environment including OpenJDK11 + OpenJ9 + OMR (without JIT) to the RISC-V development board (e.g. HiFive Unleashed with Linux support / please refer to https://www.sifive.com/boards/hifive-unleashed for details) to get it work on the board.

According to the explanation of RISC-V cross-compilers at https://www.lowrisc.org/docs/untether-v0.2/riscv_compile/, the JDK /JRE is supposed to be executed with Linux support in the user mode as the JVM /application is multiple-threaded.

Overall, there are a bunch of prerequisites before compiling OpenJ9:

• the Freedom E SDK (https://github.com/sifive/freedom-e-sdk) + openOCD ready for use (already offered on the website at https://www.sifive.com)
• compile a cross compiler with glibc support: e.g. riscv-unknown-linux-gnu-gcc based on the instructions at https://github.com/riscv/riscv-gnu-toolchain; if OK, check whether it works good with simple test.
• probably need the Yocto build system (https://www.yoctoproject.org/ or https://github.com/riscv/meta-riscv) to build everything including a Linux/RISC-V kernel (https://github.com/riscv/riscv-linux), a root file system image, a bootloader, a toolchain, QEMU, etc.
• (to be added if anything else)

Considering the complexity & unexpected challenges during the porting, the project will be split into at least 4 sub-tasks as follows to be accomplished to achieve our goal:

• compile a build without JIT on Linux (changed all request config/setting to get it work) to ensure it works good as expected; if everything goes well, all changes will be added to the compilation with riscv-unknown-linux-gnu-gcc.
• import the FFI code (libffi) for RISC-V support to OpenJ9 (https://github.com/libffi/libffi/tree/master/src/riscv / need to open an Eclipse CQ for that)
• figure out whether we need to modify the related config/setting in the Freedom E SDK and how to integrate them into the config/setting in OpenJDK11 + OpenJ9 + OMR to ensure the build is compiled with riscv-unknown-linux-gnu-gcc.
• might need to run via an Emulator (e.g. QEMU) on the host to ensure it works before uploading.
• figure out how to flash/upload the compiled build & the Linux port to the development board to get it executed/tested.
• (to be added if anything else)

The issue at #11136 was created separately to track the progress of OpenJ9 JIT on RISC-V.
Another issue at #12699 for the CI pipline environment setup.

FYI: @DanHeidinga , @pshipton, @tajila

[mstoodle]

@shingarov you might be interested in this one :)

[ChengJin01]

Currently, I already finished the following jobs:

1. verified the HiFive1 32bit board with simple tests (e.g. compile/upload a program with the Freedom E SDK to the board) to understand how it works.
2. compiled a 32 bit Linux cross compiler (riscv32-unknown-linux-gnu-gcc) but there might be some issue with the compiler. Still working on that.

[ChengJin01]

@pshipton , I am wondering whether we should disable the shared classes on the resource-limited board (only 128M flash memory available for use, which includes the Linux port/utilities, the Java build, user applications, etc) as the shared cache is set to 300M by default.

In addition, we might need to disable some modules/unnecessary stuffs in code to get things easy to go on the board.

[JasonFengJ9]

@ChengJin01 jlink can generate java.base only JRE (no other modules).

[ChengJin01]

Probably need to disable these modules in config/setting in the OpenJDK before compiling the build with riscv32 as there might be no way to run jink on the compiled build (all executables are already in the RISC-V format) on the host machine.

[pshipton]

@ChengJin01 When the available disk space is small, the default shared cache size should be 64MB. However I can guess this still might be too big. We do want to ensure shared classes is working, and could set an even smaller size for this platform, like 16M or even 4M just for test purposes.
@hangshao0

[pshipton]

Actually, this is 32-bit, so the default shared cache size should be 16MB already. The 300MB default size is for 64-bit only.

[ChengJin01]

I doubled-check the document on HiFive1(32bit FE310), it is actually 16MB off-chip flash memory (128Mbit). So the shared cache still needs to be reduced if possible.

In addition, according to the technical Spec on FE310 at
https://docs.platformio.org/en/latest/frameworks/freedom-e-sdk.html
https://github.com/RIOT-OS/RIOT/wiki/Board:-HiFive1

HiFive1 Features & Specifications
Microcontroller: SiFive Freedom E310 (FE310)
SiFive E31 RISC-V Core
Architecture: 32-bit RV32IMAC
...
Memory: 16 KB Instruction Cache, 16 KB Data Scratchpad <--- 16KB RAM
...
Flash Memory: 128 Mbit Off-Chip (ISSI SPI Flash)

and the discussion of Linux port on HiFive1 at https://forums.sifive.com/t/is-there-a-linux-distribution-that-can-run-on-hifive1/658/5, it seems the 32bit FE310 chip holds a very tiny RAM (16KB) which is far from enough to support the Linux kernel.

Given that porting OS kernel to the board is not our focus, there might be two options for us to move forward:

1. choose 64bit HiFive FU540 Unleashed (Linux-capable, multi-core / https://www.sifive.com/boards/hifive-unleashed) to support Linux on chip for JVM.
2. consider to use a RTOS (must support multi-threading / e.g. RTLinux, Zephyr, Apache Mynewt, etc) with RISC-V 32bit support. It is still unclear whether we can make it work in this way and how many code/libraries in OpenJ9 need to be adjusted to accommodate the RTOS.

@DanHeidinga

[DanHeidinga]

1. Can the HiFive1 boards be extended with additional RAM & disk?

2. Is there an emulator for the RISC-V that we can use while working to procure more suitable boards?

3. We want to target Linux with this work as it simplifies the rest of the porting effort.

[mstoodle]

Surely supporting the shared classes cache could be considered on the "nice to have" list rather than in the initial set of priority activities for a new platform bring up?

Is there an existing new platform bring-up (ordered) checklist? If not, could we start creating one as part of this effort?

[ChengJin01]

@DanHeidinga ,

1. Can the HiFive1 boards be extended with additional RAM & disk?

There is no public doc/Spec shows it can do that but we can bring this question to their forum at https://forums.sifive.com. (already raised the question at https://forums.sifive.com/t/extending-the-hifive1-board-with-additional-ram-disk/2155)

2. Is there an emulator for the RISC-V that we can use while working to procure more suitable boards?

The typical emulator is QEMU (https://github.com/riscv/riscv-qemu) both with RISC-V (RV64G, RV32G) Emulation Support. It will be used to boot the Linux/shell so as to run the JVM after compilation.

3. We want to target Linux with this work as it simplifies the rest of the porting effort.

In this case, there might be not too many options in term of development boards except the 64bit HiFive FU540 Unleashed board.

Technically, most of work will be finished (compilation, emulation, etc) before uploading everything to the board which should be the final steps to verify whether it really works on the RISC-V chip/hardware. So it might be not that urgent for the moment to decide which board to use as long as we get our build work on RISC-V via emulator.

[ChengJin01]

Already talked to @mstoodle on slack, we can create a high-level/generic porting guideline not specific to any platform with our porting experiences on RISC-V, which helps people to understand the basic/key steps to follow when porting OpenJ9 to a new platform.

[hangshao0]

we can create a high-level/generic porting guideline not specific to any platform with our porting experiences on RISC-V, which helps people to understand the basic/key steps to follow when porting OpenJ9 to a new platform

@knn-k has some recent experience porting OpenJ9 onto Aarch64. He might be helpful regarding the porting guideline.

[knn-k]

If it is without the JIT, you can take a look at these PRs to begin with for what I have done with AArch64 VM build:

• AArch64: Implement unsafe helper functions #3502, AArch64: Implement arm64cinterp.m4 and arm64helpers.m4 #3559, Add libffi preconf files for AArch64 #4087, AArch64: Dockerfile for JDK11 #4333, AArch64: Add files needed for linux aarch64 build #4350, AArch64: Add changes for building for AArch64 #4487, AArch64: Update build instructions #4696

[ChengJin01]

@knn-k , many thanks for your links of changsets. I believe we will get started with at least something similar (with RISC-V instruction sets) except Docker-related stuff as we need to run it directly on hardware/board.

[knn-k]

AArch64 VM uses the Docker image for cross-compilation on x86-64 Linux.
I would appreciate any feedback from your RISC-V effort.

[ChengJin01]

[1] The response as to extending the 32bit board with external RAM/disk as follows:
https://forums.sifive.com/t/extending-the-hifive1-board-with-additional-ram-disk/2155/3

A while ago, one suggestion to extend the amount of memory was to connect a 
SPI RAM to the board’s SPI GPIO pins. It’s slower than direct RAM but depending 
on the use-case it might be enough:
...
Though to be honest it might very well be that your application
 (if it requires “far more than 16MB”) is too much for this board.

So it seems extending the hardware this way is tricky and it is hard to say whether the 32bit board can really support that.

[2] I went over the Spec document for the 64bit board at https://sifive.cdn.prismic.io/sifive%2Ffa3a584a-a02f-4fda-b758-a2def05f49f9_hifive-unleashed-getting-started-guide-v1p1.pdf,

HiFive Unleashed is a Linux development platform for SiFive’s Freedom U540 SoC, 
the world’s first 4+1 64-bit multi-core Linux-capable RISC-V SoC. 
The HiFive Unleashed has 8GB DDR4, 32MB QuadSPI Flash, a Gigabit Ethernet port, 
and a MicroSD card slot (can be used to boot the linux image) for more external storage.

If it is the mainstream hardware setup on the 64bit board, there should be no limitation
on RAM/disk for Linux kernel + JVM with this board, whether it comes to RAM or the share cache.

---

[3] There are 3 options for the cross-compilation & EQMU emulation:

1. manually compile all related artifacts including cross-compiler + Linux kernel + Boot Loader +an customized shell environment, etc from the source, for which https://github.com/michaeljclark/busybear-linux already integrated everything we need in the cross-compilation.
   but it is unclear whether the generated boot image & Boot Loader works good on the real hardware.

2. https://buildroot.org/
   Buildroot includes everything required for cross-compilation but it needs manual choice for a bunch of configurations.

3. https://github.com/riscv/meta-riscv
   Yoctco provides a one-stop integration environment for cross-compilation pretty much without any manual intervention except a simple choice for the image type, coming with a full-featured Linux
   environment for RISC-V-based cross-development. The only drawback is it mostly ends up with over xGB Linux image which seems huge in size to us (it might not be a big problem if booting from the SD-card).

I will go with the option 1) to compile everything including EQMU, which is the fastest & straightforward way we can do to solve all problems on our side before moving forward to the on-board/hardware verification.

If there is any special configuration/setting required in the boot process on the hardware which can't be done via 1), I will go back to check whether 2) or 3) works that way.

[ChengJin01]

Already created the RISC-V cross compiler on 64bit:

/opt/riscv_gnu_toolchain_64bit/bin$ ./riscv64-unknown-linux-gnu-gcc  --version
riscv64-unknown-linux-gnu-gcc (GCC) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Will keep compiling other related artifacts.

[alistair23]

Hello,

I would like to add some comments.

First off I am a large contributor to meta-riscv and the Yocto/OpenEmbedded world in general. Please let me know what you need and I will help wherever I can. The less time you all spend on getting Linux running the more time you can spend on porting Java :)

Please do not spend time on the HiFive1 board OR 32-bit RISC-V. The HiFive1 board is very small, it's basically a RISC-V version of an Arduino. RISC-V 32-bit support is still lacking so you should start with 64-bit support. You can develop for 64-bit RISC-V on either the HiFive Unleashed or QEMU. QEMU has great RISC-V support and you should be able to do all your development on QEMU. If you want help running QEMU please let me know, I am also a QEMU maintainer and am happy to help here as well.

I think meta-riscv is the best option from above to use (option [3]). It's extremely powerful and you won't have any issues of size in either QEMU or on the HiFive Unleased. Buildroot is also an option (as a maintainer here as well I can help if required). I would not recommend to build it all yourself, you will end up wasting a lot of time building toolchains, Linux and related packages.

[alistair23]

If you want to do native work you can also just run Fedora on QEMU and dnf install all the packages you need.

Either way, let me know how to help and I will do what I can

[alistair23]

Something else worth noting is that most of the repos on the github riscv account are out of date. For Linux, GNU tools and QEMU you should use mainline not the riscv forks.

[ChengJin01]

@alistair23 , many thanks for your suggestion & recommandation. Actually I already finished compiling everything on 64bit including QEMU with option 1) (just a minor issue with ntpd which can be ignored for now). We will move forward to the next step to compile our code if everything goes fine.

The problem with option [3] is it will end up with a huge size of linux image which I already tried but stopped it when it exceeded 9GB (What we need is a normal Linux kernel with decent shell support, so there is no need to integrate everything for Linux). With option 1, we are able to customize the size for use (e.g. 1GB or less if necessary).

I won't go for option 2) or 3) for now unless there is something unexpected when compiling our code.

[alistair23]

Great, I'm glad you got it working!

I'm not sure how you eneded up with a 9GB image, my full OpenEmbeded desktop images with debug symbols and self hosted toolchains aren't that big.

Either way can you keep me updated with the progress? Also let me know if there is anything that I can do to help.

[ChengJin01]

@alistair23 , I didn't try other options except bitbake core-image-full-cmdline guided at https://github.com/riscv/meta-riscv and the whole folder there kept increasing to over 9GB (eventually screwed up my VM due to out-of-space).

Basically the compilation of our code will be done with the 64bit cross-compiler outside of QEMU and mount the whole build inside to get executed in there. So we definitely need your support from QEMU perspective. My guess is there will be a bunch of problems with QEMU once we're done compiling our code to get it work, specifically in debugging in QEMU if it crashes or something similar happens.

Just keep subscribed in this issue and you will be informed with what is going on in our project.

[alistair23]

Ah, the temporary files can get big. That makes more sense.

Great! I am subscribed to this issue so I will keep and eye on things and help where I can. QEMU is very stable and the RISC-V support is also mature. If possible it would be best to build it from the master branch in mainline QEMU. Otherwise the 3.1+ releases are in good shape.

Good luck with this. Java is one of the few missing pieces for RISC-V so it will be great when you get it running :)

[ChengJin01]

Working on other crash issue on Windows. Will get back to this once we address that problem.

[ChengJin01]

I am currently compiling a build without JIT components (excluded from OpenJ9 & OMR) on a Fyre Linux machine and then on my Ubuntu VM to eliminate any compilation error with JIT excluded.

If it works good, the next step is to get the risc-v toolchain introduced in the configure/setting to see what happens (temporarily ignore the spec setting & code specific to riscv and just ensure it picks up the correct cross-compiler/tools in the process of compilation)

[ChengJin01]

I've finished all the minimal openjdk_tests (including the jdk_lang & jdk_pack200) with the cross-build on Debian_riscv/U540 and no risc-v specific issue was detected except a couple of timeout cases, some of which I already verified with a single run and all of them passed. I will get started to check the DDR related tests on the native build (compiled on Fedora_raw/QEMU) to ensure everything works fine.

[ChengJin01]

Just finished the DDR related tests with the native build on Debian_riscv/U540 and everything works fine:
DDR_tests_qemu_build_debian_riscv.txt

TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
	cmdLineTester_classesdbgddrext_RISCV_0

TOTAL: 1   EXECUTED: 1   PASSED: 1   FAILED: 0   DISABLED: 0   SKIPPED: 0
ALL TESTS PASSED

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
	testDDRExt_General_openj9_0

TOTAL: 1   EXECUTED: 1   PASSED: 1   FAILED: 0   DISABLED: 0   SKIPPED: 0
ALL TESTS PASSED
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Will get back to keep investigating the exception issue with java -version on Fedora_raw/U540 to see what we can do with this problem.

[ChengJin01]

To figure out what happened to java -version on Fedora_raw/U540, I will first compile a debug build with the cross-build as boot-JDK directly on Debian_riscv/U540 to debug in single steps in interpreter.

[ChengJin01]

Currently investigating an issue with the code of checking the qualified name in loading classes. Will get back to this once the problem there gets addressed.

[janvrany]

@ChengJin01
I just a bit of debugging of the java --version crash on HiFive Unleashed. Few observations:

1. It crashes on Debian when running 5.6.12 kernel (loaded by OpenSBI if that matters). The exact JDK on same Debian image but running on 5.0.0rc1 kernel (loaded by BBL) DOES NOT crash. This suggest some change in linux kernel is causing that. At least this is where to start looking as this is the only difference I can see.
2. Vanilla GDB master (commit 4d68fd750fa0ce9de7a5245f9eff539f31a95fb6) compiled on RISCV/Debian as: ../../configure --prefix=/opt/gdb --disable-werror --with-python=/usr/bin/python3.8 --disable-guile does work. One can see backtrace, single stepping seem to work fine.
3. When I run produces different output, see below.

I'll do further debugging in GDB later.

Unhandled exception                                             
Type=Segmentation error vmState=0x00000000         
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=0000003FC5E026C4 Handler2=0000003FC5CC7B9A InaccessibleAddress=0000000000000005
PC=0000003FC5DADC98 RA=0000003FC5DB12D6 SP=0000003FC66DB8E0 GP=0000002AE6833800
TP=0000003FC66DE8F0 T0=74532F676E616C2F T1=0000000000000000 T2=000000000000005F
S0=0000003FC66DBC38 S1=0000000000005F00 A0=FFFFFFFFFFFFFFFF A1=0000000000011DB0
A2=0000000000000000 A3=FFFFFFFFFFFFFFFF A4=0000000000000001 A5=0000003FC5DADC8E
A6=000000000000F600 A7=0000000000005F00 S2=0000003FC4E7727D S3=0000000000005DD0
S4=00000000FFF00490 S5=0000000000001000 S6=00000000FFF00898 S7=0000000000010000
S8=00000000FFF00498 S9=000000000000FFFF S10=00000000000009F8 S11=000000000040F900
T3=0000003FC6793F80 T4=73736572706D6F63 T5=75646F4D2F676E61 T6=000000000000005B
PC=0000003FC5DADC98                                             
FT0 3fd27616c9496e0b (f: 3377032704.000000, d: 2.884576e-01)
FT1 bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
FT2 0000000000000000 (f: 0.000000, d: 0.000000e+00)      
FT3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FT4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FT5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FT6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FT7 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS0 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS1 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FA0 ffffffff42ab2760 (f: 1118513024.000000, d: -nan)                                                                             
FA1 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)                                                                     
FA2 402f36a092834df1 (f: 2458078720.000000, d: 1.560669e+01)                                                                     
FA3 bfe00b61fad0540f (f: 4207956992.000000, d: -5.013895e-01)                                                                    
FA4 ffffffff3e3ab283 (f: 1044034176.000000, d: -nan)
FA5 ffffffff00000000 (f: 0.000000, d: -nan)
FA6 0000003fc0006b10 (f: 3221252864.000000, d: 1.352772e-312)
FA7 0000000000000020 (f: 32.000000, d: 1.581010e-322)
FA6 0000003fc0006b10 (f: 3221252864.000000, d: 1.352772e-312)   
FA7 0000000000000020 (f: 32.000000, d: 1.581010e-322)
FS2 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS3 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS4 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS5 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS6 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS7 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS8 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS9 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                              
FS10 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                             
FS11 0000000000000000 (f: 0.000000, d: 0.000000e+00)                                                                             
FT8 0000000000000000 (f: 0.000000, d: 0.000000e+00)             
FT9 0000000000000000 (f: 0.000000, d: 0.000000e+00)         
FT10 0000000000000000 (f: 0.000000, d: 0.000000e+00)         
FT11 0000000000000000 (f: 0.000000, d: 0.000000e+00)     
FCSR 0000000000000001 (f: 1.000000, d: 4.940656e-324)
Target=2_90_20200525_8 (Linux 5.6.12-00007-g0f60ad328c3c)
CPU=riscv (4 logical CPUs) (0x1f3b76000 RAM)       
----------- Stack Backtrace -----------            
(0x0000003FC5CDB9E2 [libj9prt29.so+0x289e2])                                                                                     
---------------------------------------                                                                                          
JVMDUMP039I Processing dump event "gpf", detail "" at 2020/07/29 15:26:29 - please wait.                                         
JVMDUMP032I JVM requested System dump using '/home/jv/tmp/jdk/core.20200729.152629.66336.0001.dmp' in response to an event       
JVMDUMP010I System dump written to /home/jv/tmp/jdk/core.20200729.152629.66336.0001.dmp                                          
JVMDUMP032I JVM requested Java dump using '/home/jv/tmp/jdk/javacore.20200729.152629.66336.0002.txt' in response to an event     
JVMDUMP010I Java dump written to /home/jv/tmp/jdk/javacore.20200729.152629.66336.0002.txt                                        
JVMDUMP032I JVM requested Snap dump using '/home/jv/tmp/jdk/Snap.20200729.152629.66336.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jv/tmp/jdk/Snap.20200729.152629.66336.0003.trc
JVMDUMP013I Processed dump event "gpf", detail "".  

[ChengJin01]

@janvrany , Many thanks for the investigation from Debian_riscv.

Given that the latest kernel version of Fedora_raw is 5.5.0-0, I suspect the problem might be related to both the latest kernel or the new loader setting (OpenSBI). I talked to the Fedora_raw developers as to the 5.0.0 version at https://dl.fedoraproject.org/pub/alt/risc-v/disk-images/fedora/rawhide/20190126.n.0/Developer/

Index of /pub/alt/risc-v/disk-images/fedora/rawhide/20190126.n.0/Developer
Icon  Name                                                     Last modified      Size  Description
[PARENTDIR] Parent Directory                                                              -   
[   ] Fedora-Developer-Rawhide-20190126.n.0-sda1.raw.xz        2019-01-28 20:22  1.1G  
[   ] Fedora-Developer-Rawhide-20190126.n.0.CHECKSUM           2019-01-28 20:13  631   
[   ] bbl-5.0.0-0.rc2.git0.1.0.riscv64.fc30.riscv64            2019-01-28 20:13   15M  
[   ] config-5.0.0-0.rc2.git0.1.0.riscv64.fc30.riscv64         2019-01-28 20:13  159K  
[   ] initramfs-5.0.0-0.rc2.git0.1.0.riscv64.fc30.riscv64.img  2019-01-28 20:22   15M  
[   ] vmlinuz-5.0.0-0.rc2.git0.1.0.riscv64.fc30.riscv64        2019-01-28 20:22   14M  

but it seems the old version only support QEMU rather than hardware.

So if the single step debugging works good on Debian_riscv with kernel 5.6.12, it might be worth debugging from there to find out something more useful.

[janvrany]

@ChengJin01

it might be worth debugging from there to find out something more useful.

That's exactly the plan :-)

[ChengJin01]

1. Following the previous dumps generated at Porting OpenJ9 to RISC-V #5058 (comment), I went over the whole static blocks (<clinit>) of java.lang.String and classes involved: JITHelpers_class.txt and String_clinit.TXT and debugged from the following code at runtime/jcl/common/jclcinit.c into the interpreter with single-step on each bytecode on Debian/U540:

	/* Initialize java.lang.String which will also initialize java.lang.Object. */
	vmFuncs->initializeClass(vmThread, stringClass);

However, the debugging result at debug_debian_String_clinit_log.TXT didn't indicate there was any java/lang/ArithmeticException that could occur in the bytecode. To be specific, only div or rem triggers THROW_DIVIDE_BY_ZERO in the interpreter at runtime/vm/BytecodeInterpreter.hpp, which ends up with java/lang/ArithmeticException, but these two instructions only occur in the case of big_endian in the <clinit>) of classes involved unless the corresponding bytecode there was messed up or something abnormal happened to the interpreter (the debugging here still needs to be double-checked later in case anything missed in the debugging)

2. According to the released history of Linux kernel 5.x at https://en.wikipedia.org/wiki/Linux_kernel_version_history#Releases_5.x.y

Releases 5.x.y
Version	Original release date	Current version	Maintainer	Support model
5.0	3 March 2019[212]	5.0.21[213]	Greg Kroah-Hartman	EOL (maintained from March 2019 to June 2019)[213]
5.1	5 May 2019[214]	5.1.21[215]	Greg Kroah-Hartman	EOL (maintained from May 2019 to July 2019)[215]
5.2	7 July 2019[216]	5.2.20[217]	Greg Kroah-Hartman	EOL (maintained from July 2019 to October 2019)[217]
5.3	15 September 2019[218]	5.3.18[219]	Greg Kroah-Hartman	EOL (maintained from September 2019 to December 2019)[219]

against the publicized Fedora_rawhide image at https://dl.fedoraproject.org/pub/alt/risc-v/repo/virt-builder-images/images/

root@jincheng-VirtualBox:~# virt-builder --list | grep riscv64
fedora-rawhide-developer-20190703n0 riscv64    Fedora? Rawhide Developer 20190703.n.0 <----
fedora-rawhide-developer-20191030n0 riscv64    Fedora? Rawhide Developer 20191030.n.0
fedora-rawhide-developer-20191123.n.0 riscv64    Fedora? Rawhide Developer 20191123.n.0
fedora-rawhide-minimal-20191123.n.1 riscv64    Fedora? Rawhide Minimal 20191123.n.1
fedora-rawhide-developer-20200108.n.0 riscv64    Fedora? Rawhide Developer 20200108.n.0
fedora-rawhide-minimal-20200108.n.0 riscv64    Fedora? Rawhide Minimal 20200108.n.0

plus the explanation of OpenSBI (replacing BBL in the booting flow) at https://content.riscv.org/wp-content/uploads/2019/12/Summit_bootflow.pdf

Linux Kernel
– Upstream kernel boots in QEMU
– Device tree hosted in Kernel
– v5.3 kernel works with OpenSBI+U-Boot on HiFive Unleashed <-----

I suspect fedora-rawhide-developer-20190703n0 might work with the Linux kernel < 5.2, which means it still works with BBL rather than OpenSBI. Given that the build works Debian(kernel 5.0) + BBL but failed on Debian(kernel 5.6) + OpenSBI, I plan to try fedora-rawhide-developer-20190703n0 first to see how it goes on U540 before moving forward to debug on Debian(kernel 5.6) + OpenSBI.

[ChengJin01]

I tried fedora-rawhide-developer-20190703n0 at https://dl.fedoraproject.org/pub/alt/risc-v/repo/virt-builder-images/images/ but it ended up being stuck in bootstrap on U540 (whether it was set to 1111 or 1101 in terms of MSEL mode), which confirms the note mentioned at https://dl.fedoraproject.org/pub/alt/risc-v/repo/virt-builder-images/images/index:

notes=Fedora® Rawhide Developer 20190703.n.0
This Fedora/RISCV image contains only unmodified @Core group packages. <--- it means it is incomplete to work for U540

So there is no way to know whether Fedora-rawhide used to work with BBL on U540 as the first workable image fedora-rawhide-developer-20191030n0 on U540 only works with OpenSBI with the 1101 setting in bootstrap (I also burned this image on SD-card and booted it up on U540 with the 1101 setting)

notes=Fedora® Rawhide Developer 20191030.n.0
This is the 1st release to support QEMU virt machine and SiFive Unleashed dev board.

against the output of Linux kernel vesion on fedora-rawhide-developer-20191030n0:

[root@jincheng ~]# uname -a
Linux jincheng.ibm.com 5.4.0-0.rc4.git0.300.2.riscv64.fc32.riscv64  <------ kernel (> 5.3) supports OpenSBI
#1 SMP Mon Oct 28 11:17:31 UTC 2019
riscv64 riscv64 riscv64 GNU/Linux

Given that fedora-rawhide-developer-20191030n0 is a little bit lightweight and runs faster as compared to the latest image fedora-rawhide-minimal-20200108.n.0 we have been using, I will generate a complete snap trace with new tracepoints on U540 to double-check against the equivalent on X86_64 to see whether there is anything useful from there.

[janvrany]

@ChengJin01
I tried to debug the crash on Debian, OpenSBI + Linux 2.6.12 where it crashes on SIGSEGV in the interpreter (full backtrace):

FRAME  0 0x0000003FF744FDB6 VM_BytecodeInterpreterCompressed::lastore (BytecodeInterpreter.hpp:5619)

variables:
     1 arrayLength     : 63
     2 index           : 4159067376
     3 this            : 0x3ff7e65ab8
     4 _sp             : 0x5dd0
     5 _pc             : 0x3ff42c727d "P\344)\364?"
     6 rc              : EXECUTE_BYTECODE
     7 arrayref        : 0x1

source: 
  5604    				*(I_64*)_sp = _objectAccessBarrier.inlineIndexableObjectReadI64(_currentThread, arrayref, index);
  5605    			}
  5606    		}
  5607    		return rc;
  5608    	}
  5609    
  5610    	/* ..., arrayref, index, value1, value2 => ... */
  5611    	VMINLINE VM_BytecodeAction
  5612    	lastore(REGISTER_ARGS_LIST)
  5613    	{
  5614    		VM_BytecodeAction rc = EXECUTE_BYTECODE;
  5615    		j9object_t arrayref = *(j9object_t*)(_sp + 3);
  5616    		if (NULL == arrayref) {
  5617    			rc = THROW_NPE;
  5618    		} else {
  5619 >> 			U_32 arrayLength = J9INDEXABLEOBJECT_SIZE(_currentThread, arrayref);
  5620    			U_32 index = *(U_32*)(_sp + 2);
  5621    			/* By using U_32 for index, we also catch the negative case, as all negative values are
  5622    			 * greater than the maximum array size (31 bits unsigned).
  5623    			 */
  5624    			if (index >= arrayLength) {
  5625    				_currentThread->tempSlot = (UDATA)index;
  5626    				rc = THROW_AIOB;
  5627    			} else {
  5628    				I_64 value = *(I_64*)_sp;
  5629    				_pc += 1;
  5630    				_sp += 4;
  5631    				_objectAccessBarrier.inlineIndexableObjectStoreI64(_currentThread, arrayref, index, value);
  5632    			}
  5633    		}

As you can see, arrayref is 0x1 so no wonder it segfaults. _currentThread is 0x5f00, _currentThread->sp is 0x5e10. Here's stackdump (if I got it right):

0x0000000000005DD0; 0x00000000  
0x0000000000005DD4; 0x00000000  
0x0000000000005DD8; 0x00000000  
0x0000000000005DDC; 0x00000000  
0x0000000000005DE0; 0xffffffff  
0x0000000000005DE4; 0x00000000  
0x0000000000005DE8; 0x00000001  
0x0000000000005DEC; 0x00000000  
0x0000000000005DF0; 0x00000000  
0x0000000000005DF4; 0x00000000  
0x0000000000005DF8; 0x00000000  
0x0000000000005DFC; 0x00000000  
0x0000000000005E00; 0x00000002  
0x0000000000005E04; 0x0000003f  
0x0000000000005E08; 0x00000000  
0x0000000000005E0C; 0x00000000  
0x0000000000005E10; 0x00000000  

At first glance, none of the values look like object reference.
However I must admit that so far I'm merely trying to make sense of the code and understand how it works.

I'll continue digging.

[ChengJin01]

Hi @janvrany, Many thanks for your debugging on the latest version of Debian on U540.

If the crash literally happened to the code near to inlineIndexableObjectReadI64 or inlineIndexableObjectStoreI64 for loading/storing data, then it confirmed my previous assumption that the problem might be related to barrier which does have something to do with OpenSBI in that OpenSBI offers a full L2 cache for better performance. If so, we need to review all our code with read/write barrier in OMR to see what happened to the code in such case.

So I will burn the image to SD-card and repeat your steps to generate a debug build and double-check the result to see whether the crash still occurs around there before moving forward.

[janvrany]

@ChengJin01: just in case you would find that useful: here are my scripts to (reproducibly) compile debug build of OpenJ9 on RISC-V Debian: https://github.com/janvrany/openj9-openjdk-jdk11-devscripts

[ChengJin01]

After attempting a bunch of times, I managed to get Debian with Linux kernel 5.6 working on U540

SiFive FSBL:       2018-03-20
HiFive-U serial #: 00000287

OpenSBI v0.6
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name          : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs     : 5
Current Hart           : 4
Firmware Base          : 0x80000000
Firmware Size          : 108 KB
Runtime SBI Version    : 0.2

MIDELEG : 0x0000000000000222
MEDELEG : 0x000000000000b109
PMP0    : 0x0000000080000000-0x000000008001ffff (A)
PMP1    : 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.6.12-149510-g0f60ad328c3c (jincheng@jincheng-pc) (gcc version 10.1.0 (GCC)) #1 SMP Tue Aug 11 15:52:15 EDT 2020
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000027fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x000000027fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000027fffffff]
[    0.000000] software IO TLB: mapped [mem 0xfbfff000-0xfffff000] (64MB)
[    0.000000] CPU with hartid=0 is not available
[    0.000000] CPU with hartid=0 is not available
[    0.000000] elf_hwcap is 0x112d
...
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.
[   38.212843] macb 10090000.ethernet eth0: Link is Up - 100Mbps/Full - flow control tx

Debian GNU/Linux bullseye/sid unleashed console
...
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
...
root@unleashed:~# 
root@unleashed:~# uname -a
Linux unleashed 5.6.12-149510-g0f60ad328c3c #1 SMP Tue Aug 11 15:52:15 EDT 2020 riscv64 GNU/Linux

The next step is to generate a couple of builds (including the debug build) to see how it goes with the latest linux kernel + OpenSBI.

[ChengJin01]

I am currently setting up the test environment for the internal tests on Debian(5.0) with BBL/U540. Will get back to the investigation on the latest linux kernel + OpenSBI after the test is completed.

[DingliZhang]

@ChengJin01 Hi Cheng Jin,
Thanks a lot for your Build_Instructions_V11.md on riscv64, I am doing something about porting java (including openjdk and openj9) to riscv. I am setting baseline now, learning and evaluating how hard and time needed to get things done.

By following your instruction I have built a jdk image and successfully run ./java -version.

I searched issues and have no idea about enabling openJ9 JIT on RISCV64 yet. Could you have any plan on openJ9 JIT and a new long-term roadmap instead of #5058? Thank you for your advice.

[ChengJin01]

Hi @zdlgv5,

Many thanks for your support on the OpenJ9/RISC-V project! As for the OpenJ9 JIT on RISC-V, it is already on our priority list but we have not yet get started for now coz there are some details we need to figure out as to what is best way to move forward. Please stay tuned!

FY: @0xdaryl

[janvrany]

Hi, @zdlgv5,

I'm currently doing some preparatory work to start working on OpenJ9 part of the JIT and hoping to start doing some real JIT work soon (TM)

[DingliZhang]

Hi, @ChengJin01 and @janvrany ,
Thank you for your prompt reply!
I worked on JavaWEB for two years and I'm new here.
I'm learning more about porting and compiling and looking forward to working together with you in the near future!

[ChengJin01]

For those of you who are concerned about the latest status of OpenJ9 JIT on RISC-V, the issue at #11136 was created to track the progress in there.

[ChengJin01]

The investigation with java -version (during my vacation before Christmas break & already double-checked) on Debian (Linux kernel 5.6) + OpenSBI v0.6/U540 indicates that problem was triggered by the failure of lh (assembly opcode on RISC-V) to sign-extend to the correct value on U540 as follows

/runtime/vm/BytecodeInterpreter.hpp
gotoImpl(REGISTER_ARGS_LIST, UDATA parmSize)
	{
		if (parmSize == 2) {
			_pc += *(I_16*)(_pc + 1); <-------------- in the interpreter (the _pc value was messed up)
		} else if (parmSize == 4) {
			_pc += *(I_32*)(_pc + 1);
		} else {
			Assert_VM_unreachable();
		}
		return GOTO_BRANCH_WITH_ASYNC_CHECK;
	}

with the assembly:
7319				_pc += *(I_16*)(_pc + 1);    
   0x0000003ff74489be <+34>:	ld	a5,-40(s0)         
   0x0000003ff74489c2 <+38>:	ld	a5,0(a5)           
   0x0000003ff74489c4 <+40>:	ld	a4,-40(s0)         
   0x0000003ff74489c8 <+44>:	ld	a4,0(a4)           
   0x0000003ff74489ca <+46>:	addi	a4,a4,1          
   0x0000003ff74489cc <+48>:	lh	a4,0(a4) <----------- failed to sign-extend to align with the _pc value
   0x0000003ff74489d0 <+52>:	add	a4,a4,a5           
   0x0000003ff74489d2 <+54>:	ld	a5,-40(s0)         
   0x0000003ff74489d6 <+58>:	sd	a4,0(a5)           
...
6: x/x $a4
   0xffe1: <-------------- on Debian 5.6 + OpenSBI v.06

while it works good on the 5.0 + BBL to jump to the right path as follows:

1: x/i $pc
=> 0x2000a159cc <VM_BytecodeInterpreterCompressed::gotoImpl(unsigned long*&, unsigned char*&, unsigned long)+48>:	
    lh	a4,0(a4) <--------------
2: /x $a4 = 0xffffffffffffffe1  <-------- correctly sign-extended on Debian 5.0 + BBL

It doesn't mean lh itself caused the trap but it led to the incorrect calculation of the _pc value in the bytecode which screwed up the rest of execution (jumping into nowhere...)

After discussing with the Western Digital Developers working on Linux kernel & OpenSBI, they concluded that the problem came from the code with sign-extending in OpenSBI at https://github.com/riscv/opensbi/lib/sbi/sbi_misaligned_ldst.c:

int sbi_misaligned_load_handler(ulong addr, ulong tval2, ulong tinst...)
...
	if (!fp)
		SET_RD(insn, regs, val.data_ulong << shift >> shift);  <------

and committed a new fix for the issue two weeks ago at riscv-software-src/opensbi@7dcb1e1

int sbi_misaligned_load_handler(ulong addr, ulong tval2, ulong tinst...)
...
	if (!fp)
		SET_RD(insn, regs, ((long)(val.data_ulong << shift)) >> shift);

@janvrany (our partner) just verified on Debian 5.6 + the patched OpenSBI v0.6 with the fix above and java -version seems working good as expected:

jv@unleashed:~/tmp/jdk/bin$ uname -a
Linux unleashed 5.6.12-00007-g0f60ad328c3c #2 SMP Tue Jan 5 10:40:23 GMT 2021 riscv64 GNU/Linux

jv@unleashed:~/tmp/jdk/bin$ ./java --version
openjdk 11.0.10-internal 2021-01-19
OpenJDK Runtime Environment (build 11.0.10-internal+0-adhoc.jenkins.openj9-openjdk-jdk11)
Eclipse OpenJ9 VM (build HEAD-f2e8dfa4, JRE 11 
Linux riscv-64-Bit Compressed References 20201210_56 (JIT disabled, AOT disabled)
OpenJ9   - f2e8dfa4
OMR      - 5c72bed
JCL      - 5e81062d6 based on jdk-11.0.10+5)

---

The next steps include:

1. will recreate a new Debian image on our side to verify the result once @janvrany finishes pushing the fix to the repo at https://github.com/janvrany/riscv-debian/tree/linux-5.6.
2. will do internal tests to see how it goes on Debian 5.6 with the newly patched OpenSBI on U540.
3. already notified the Fedora/RISC-V developers to create a new Fedora image once OpenSBI is branched to a new release version with the latest fix.
4. will do the same internal tests to double-check on the latest Fedora/RISC-V image after 3) to confirm everything still works good.

[keithc-ca]

Any opinions on whether we ought not to rely on support for misaligned accesses like that? Perhaps there's some performance benefit to be had?

[ChengJin01]

@keithc-ca , OpenSBI is the standardized binary interface (aimed to replace BBL as a new kind of BIOS on RISC-V) intended for all OS images & hardware on RISC-V (see https://github.com/riscv/opensbi & https://riscv.org/wp-content/uploads/2019/06/13.30-RISCV_OpenSBI_Deep_Dive_v5.pdf)

If we chose not to reply on the fix from OpenSBI, there might be a bunch of code changes everywhere in the interpreter except gotoImpl

BytecodeInterpreter.hpp (17 matches)
6,970: _pc += *(I_16*)(_pc + 1); 
6,991: _pc += *(I_16*)(_pc + 1); 
7,013: _pc += *(I_16*)(_pc + 1); 
7,034: _pc += *(I_16*)(_pc + 1); 
7,055: _pc += *(I_16*)(_pc + 1); 
7,076: _pc += *(I_16*)(_pc + 1); 
7,097: _pc += *(I_16*)(_pc + 1); 
7,118: _pc += *(I_16*)(_pc + 1); 
7,142: _pc += *(I_16*)(_pc + 1); 
7,166: _pc += *(I_16*)(_pc + 1); 
7,190: _pc += *(I_16*)(_pc + 1); 
7,214: _pc += *(I_16*)(_pc + 1); 
7,238: _pc += *(I_16*)(_pc + 1); 
7,262: _pc += *(I_16*)(_pc + 1); 
7,286: _pc += *(I_16*)(_pc + 1); 
7,310: _pc += *(I_16*)(_pc + 1); 
7,329: _pc += *(I_16*)(_pc + 1); 

and there is no way to to determine what happens next in the code depending on where the unaligned load/store opcode is placed in assembly (it could happen anywhere in our code except the interpreter). (The reason for this is that the underlying hardware doesn't support misaligned load/store which is why OpenSBI comes for help)
riscv/riscv-isa-manual#394

Use-Case1: Misaligned load/store emulation in M-mode
The M-mode RUNTIME firmware (OpenSBI) will have to do misaligned load/store
emulation for both HS-mode (Hypervisor) and VS-mode (Guest OS) whenever
underlying HW does not implement misaligned load/store. <-----------

This means M-mode
RUNTIME firmware will do unprivledge load for getting faulting instruction.
This is expensive for traps coming from both HS-mode and VS-mode. Particularly,
it's very expensive for traps coming from VS-mode because unprivledge load
can use nested page table walk to read instruction from memory.

https://riscv.org/wp-content/uploads/2019/06/riscv-spec.pdf

An EEI may guarantee that misaligned loads and stores are fully supported, 
and so the software running inside the execution environment will never experience 
a contained or fatal address-misaligned
trap. In this case, the misaligned loads and stores can be handled in hardware, or via an invisible
trap into the execution environment implementation, or possibly a combination of hardware and
invisible trap depending on address.

---

I am not sure whether it benefits from any change specific to RISC-V in the interpreter but the first thing comes to my mind is the unexpected behaviour anywhere afterwards which will be extremely difficult in troubleshooting.

@gacholio & @pshipton, any input on this?

[keithc-ca]

This is the key point in the spec:

An EEI may guarantee that misaligned loads and stores are fully supported

Notice it says 'may', not 'must'. As it is, it would seem our VM will simply not run on an EEI that doesn't support misaligned memory accesses.

[ChengJin01]

This is the key point in the spec:

An EEI may guarantee that misaligned loads and stores are fully supported

Notice it says 'may', not 'must'. As it is, it would seem our VM will simply not run on an EEI that doesn't support misaligned memory accesses.

Yes to some extent, which is why OpenSBI needs to address all related misaligned memory accesses from the hardware. Given the code is shared across various platforms and the problem never happens on other platform & hardware except RISC-V, I don't see there is any special reason to handle this situation specific to RISC-V on our side (I kind of doubt whether it can be totally fixed simply by modifying the code above in the interpreter as there might be something else we are unaware of in other code in terms of generated assembly).

[gacholio]

For now, I would say that we can only run on systems which allow misaligned access. I'd rather not pollute the code with macros unnecessarily.

[keithc-ca]

Even on systems that allow misaligned access, there may be a performance penalty, but until any such penalty is quantified, I agree we should leave the code alone.

[ChengJin01]

With a newly created Debain5.6 + patched OpenSBI (fix upstreamed to https://github.com/janvrany/riscv-debian/tree/linux-5.6 by @janvrany), I verified our build works good as expected on U540 as follows:

and also checked with a sample application of the OpenJDK:

Will set up the test environment on U540 to do internal tests to see whether everything still goes fine with the new fix.

[ChengJin01]

I finished the internal tests which worked good on Debain5.6 + patched OpenSBI. Will wait for the new version of Fedora_riscv to double-check.

[ghost]

hi this is abhishek here ,i am trying to port openjdk to riscv and i am facing problem in native compile can you help me to share the steps what to followed,thanks in advance

[0xdaryl]

@abhishekece : are you asking about the OpenJ9 JVM on RISC-V, or OpenJDK? OpenJDK-specific questions about RISC-V are best asked at the OpenJDK project.

However, if you're asking about OpenJ9, please describe your problem in more detail.