Userland emulation, API #56
Comments
LekKit
added
documentation
- Implement Userland API as defined by draft. This is WIP, requesting validation by @X547. If you have any issues, try to build lib with USE_JIT=0 and report in #56. - Icache invalidation is tricky without a soft MMU, hence rvvm_flush_icache() is very inefficient as of now. This will be improved when we finish with other relevant stuff. Machine mode improvements: - Implement default PLIC/PCI handling functions - De-initialize machine devices in reversed order. This makes sure that references to parent devices are always safe, otherwise device thread may be accessing a PCI bus or PLIC while we are freeing it. The general rule now is: Devices shouldn't do anything after they return from dev->remove().
|
Is it possible to build |
|
I tried to adapt UserlandVM to new librvvm API and it currently crash in |
Can you give a backtrace? |
Will do. I have no experience with pkg-config however 😅 |
|
Some obvious fixes: diff --git a/src/rvvm.c b/src/rvvm.c
index c782080..a3427af 100644
--- a/src/rvvm.c
+++ b/src/rvvm.c
@@ -707,11 +707,15 @@ PUBLIC rvvm_cpu_handle_t rvvm_create_user_thread(rvvm_machine_t* machine)
vector_emplace_back(machine->harts);
rvvm_hart_t* vm = &vector_at(machine->harts, vector_size(machine->harts) - 1);
riscv_hart_init(vm, machine->rv64);
+ vm->machine = machine;
+ vm->mem = machine->mem;
#ifdef USE_FPU
// Initialize FPU properly
#endif
return (rvvm_cpu_handle_t)vm;
}
@@ -743,8 +747,9 @@ PUBLIC rvvm_addr_t rvvm_read_cpu_reg(rvvm_cpu_handle_t cpu, size_t reg_id)
return vm->csr.cause[PRIVILEGE_USER];
} else if (reg_id == RVVM_REGID_TVAL) {
return vm->csr.tval[PRIVILEGE_USER];
+ } else {
+ rvvm_warn("Unknown register %d in rvvm_read_cpu_reg()!", (uint32_t)reg_id);
}
- rvvm_warn("Unknown register %d in rvvm_read_cpu_reg()!", (uint32_t)reg_id);
return 0;
}
@@ -763,6 +768,7 @@ PUBLIC void rvvm_write_cpu_reg(rvvm_cpu_handle_t cpu, size_t reg_id, rvvm_addr_t
vm->csr.cause[PRIVILEGE_USER] = reg;
} else if (reg_id == RVVM_REGID_TVAL) {
vm->csr.tval[PRIVILEGE_USER] = reg;
+ } else {
+ rvvm_warn("Unknown register %d in rvvm_write_cpu_reg()!", (uint32_t)reg_id);
}
- rvvm_warn("Unknown register %d in rvvm_write_cpu_reg()!", (uint32_t)reg_id);
} |
Oh... Silly me |
|
Apart from the possible explosions from RVJIT and my poor checking of my own code, I tried to copy your approach and also used some proper functions for FPU enablement and other cpu state setup |
Not yet. Crash after first syscall with NULL guest dereference. Maybe my fault, need to debug more. TinyEMU engine still working. |
See my comment from d20c2d2 (Concern about PC increment behavior)
This part should be more explicit and well documented. I'm not sure if this is the thing that broke for you, but just be aware. |
|
PC register is 4 inside syscall handler. |
|
Virtual CPU still try to jump at TVEC that is zero. It should be not done for userland emulation mode. |
Weird. I'm not sure how that's happening (Unless you are calling riscv_hart_run() or clearing vm->user_traps), I grep'ed the source tree for manupulations on REGISTER_PC and did not see any other way. |
|
I used older version of Now it seems crash on FPU instruction: |
|
It started after recovering my old HART state initialization code:
|
Neat! I will provide upstream fixes soon. |
|
Seems yes, but not with all programs.
|
Well, considering the absolute lack of cache coherence in userland mode, it's understandable. |
|
Something wrong is happening with multiple threads (host heap corruption etc.). |
Real RISC-V behavior is not incrementing PC on ECALL trap. This is also what TinyEMU does: https://github.com/X547/UserlandVM/blob/master/VirtualCpuRiscVTemu.cpp#L48. |
I know, it's just an implementation detail (And optimization) of RVVM interpreter that leaked when we abandoned machine mode. After each interpreted instruction it increments PC by instruction size, but the trap handling jumps to TVEC which makes the PC increment invisible to the guest. Omitting the increment for userland emulation also adds confusion when it's not the ECALL instruction: we aren't sure what's the size of the instruction (Which is why it was initially done this way internally) |
- Fix initialization of per-cpu variables - Fix FPU initialization - Fix bogus warns from rvvm_write_cpu_reg() See #56 for details
|
|
There are some disastrous things from that vector manipulation already with CPU hotplug, a patch is on the way. You can try placing |
|
Just looked at my
|
|
I'm currently deciding some license-related changes, and I'd like to hear your feedback if possible (Could benefit UserlandVM) I'm heavily against closed forks, together with @cerg2010cerg2010, but GPL is slightly restrictive sometimes when it comes to distributing/integrating the project. GPL requires accompanying code to be GPL-compatible, and requires that modifications are always possible. I'm investigating the dual-licensing approach, where you aren't forced to open your code or change licensing when linking with upstream This usecase is perfectly covered by I'm curious if this can help integrate RVVM with UserlandVM more easily, or allow distributing it in FreeBSD/Haiku repositories. Would like to hear your thoughts on that, or any other ideas you may have. I hope we can make a good compromise for both end users and us, developers) |
librvvm license allowing to use it with MIT-licensed code would be beneficial for Haiku. It would allow to distribute RVVM based CPU engine plugin for UserlandVM as part of system. I originally planned to distribute TinyEMU based engine as default and RVVM engine as optionally installable. |
Will MPL or Artistic license be compatible with MIT? That looks to me like a nice middle-ground between permissive and copyleft, since we just drop the requirement that "modifications are always possible" and the viral nature of GPL linkage, while still insisting that changes made to |
|
If it will be possible to link closed-sourced fork of UserlandVM to unmodified librvvm.a, it should be fine. |
Hope that's resolved now, see the repo |
Depending on what reference commit you used in your fork, it could've lacked NaN boxing (Which is now implemented properly, see I will be returning to my debugging efforts with legacy |
|
Upstreamed a new reusable ELF loader lib. It loads userspace Linux ELFs into process address space fine, and even runs some of them (Syscall emulation and stack setup are unstable and limited, not upstreamed yet). No FPU issues with rvvm-user + Linux binaries that I see so far. To be continued... |
|
New RVJIT optimization allows 1:1 translation of load/store instructions to host instructions in userland emulation, this should boost perf a lot, @X547 can you evaluate if it works OK with UserlandVM? |
|
I am very interested in rvvm-user, can i build and try it on my machine? |
See cdfea8b. It is very WIP, and the entire point of userland emulator in future is that more perf can be squeezed, it fully integrates with your system (GUI goes directly over host X/Wayland, same with sound, etc) and even things like GPU drivers could work I guess. It is also very nice for foreign-arch build systems. |
|
Tiny progress on running dyn RISC-V Linux ELFs.. (Here running buildroot chroot & PrBoom)
|
|
@X547 should I actually omit PC increment entirely for userland API (like TEMU does)? I think it's actually a good idea now, and if it should be done it better be early than late Basically I figured the PC increments on faulty instructions (load/store traps, illegal instructions etc) as well which is not nice at all, whereas only incrementing PC after ECALL is also weird |
I am fine with both, but it would be nice to be consistent with how real RISC-V works (do not increment PC on ECALL). |
I will be doing exactly that soon, then. It will cause a slight breakage for UserlandVM tho - you will need to patch it to add 4 bytes to PC on UPD: cc3466c |
|
Many static binaries & Musl-linked dynamic binaries work in rvvm-user at this point. However, there is a strange issue that glibc dyn ELFs do crash early in I've implemented detailed guest exception handling and debug output, but struggle to debug this. If anyone is interested in joining the efforts, here is a direction:
int rvvm_user(int argc, char** argv, char** envp);
int main(int argc, char** argv, char** envp)
{
if (argc > 2 && rvvm_strcmp(argv[1], "-user")) {
rvvm_set_loglevel(LOG_INFO);
return rvvm_user(argc - 2, argv + 2, envp);
}
...
|
I think that it would be better to put userland Linux emulator into separate executable, something like |
It's a good idea, esp. considering that Honestly tho I like the idea that both user and machine modes share like 95% of emulation/JIT code. It's not entirely nonportable either (Its in theory possible to run Linux programs on Haiku or FreeBSD). |
This is what dynamic libraries for.
So are you planning to implement Linux syscalls on top of POSIX, not like UserlandVM where host and guest syscalls are assumed to be the same? It would be much more complex and likely cause a lot of compatibility problems. |
So far I am simply calling POSIX/Linux specific libc functions that correspond to their syscall and it's not problematic. There are a few things that can be done with this to ensure maximum compatibility, like converting ABI/endianness in structs passed between kernel<->user, like QEMU does. |
|
@X547 big news: (TLDR: I got glibc dyn binaries to work) We can't simply jump from guest syscalls to host kernel (as you suggested earlier). It turns out structures passed in Linux syscalls are defined subtly different on different architectures, so simply passing a pointer to This also means that subtle difference between POSIX systems like Haiku or BSD can be also eliminated this way and allow to run some apps on a foreign kernel |
|
Managed to launch
|
I suppose some special handling is needed when spawning new threads like in UserlandVM. |
I guess so. The struct conversion is still WIP so I'd need to finish it first, also I don't see that guest touch |
|
Went further and now crashes immediately in the newly
|
|
Another useful trick: On a RISC-V host, selectively bypass syscalls directly to There are only a few syscalls that won't ever work that way, one notable exception is |
|
It seems that currently RVVM have fake chroot and fake bind options. Maybe fake pid and uid can also be added? We may add a fake pid namespace to RVVM so that we can boot the system by running |
Fake uid indeed could be implemented as in |
|
I have reworked CMake build system so static libs can be built properly and not split into Building Perhaps I can also implement an |
RVVM is usable as a generic CPU execution engine, not just as a complete machine.
This suggests for working userspace emulation, which runs RISC-V code inside a host process, directly accesses it's memory and defers syscall execution to the host kernel.
For convenience, it's also great to expose an API for that, there is a working project that already needs that (https://github.com/X547/UserlandVM), which uses RVVM for HaikuOS RISC-V userland emulation.
Progress tracking:
Discussions & suggestions are welcome.
The text was updated successfully, but these errors were encountered: