|
|
Subscribe / Log in / New account

Why RISC-V doesn't (yet) support KVM

By Jonathan Corbet
May 20, 2021
The RISC-V CPU architecture has been gaining prominence for some years; its relatively open nature makes it an attractive platform on which a number of companies have built products. Linux supports RISC-V well, but there is one gaping hole: there is no support for virtualization with KVM, despite the fact that a high-quality implementation exists. A recent attempt to add that support is shining some light on a part of the ecosystem that, it seems, does not work quite as well as one would like.

Linux supports a number of virtualization mechanisms, but KVM is generally seen as the native solution. It provides a standard interface across systems, but much of KVM is necessarily architecture-specific, since the mechanisms for supporting virtualization vary from one processor to the next. Thus, architectures that support KVM generally have a kvm directory nestled in with the rest of the architecture-specific code.

Given that, some eyebrows were raised when Anup Patel's patch series adding RISC-V KVM support deposited the architecture-specific code into the staging directory instead. Staging is normally used for device drivers that do not meet the kernel's standards for code quality; if all goes well they are improved and eventually "graduate" out of the staging directory. It is not usually a place for architecture support. So staging maintainer Greg Kroah-Hartman was quick to ask why things were being done that way.

The answer comes down to the patch-acceptance policy for RISC-V code, found in Documentation/riscv/patch-acceptance.rst, which reads:

We'll only accept patches for new modules or extensions if the specifications for those modules or extensions are listed as being "Frozen" or "Ratified" by the RISC-V Foundation. (Developers may, of course, maintain their own Linux kernel trees that contain code for any draft extensions that they wish.)

Virtualization for RISC-V is described by the hypervisor extension specification. As Patel explained, that extension has not yet been approved:

The KVM RISC-V patches have been sitting on the lists for almost 2 years now. The requirements for freezing RISC-V H-extension (hypervisor extension) keeps changing and we are not clear when it will be frozen. In fact, quite a few people have already implemented RISC-V H-extension in hardware as well and KVM RISC-V works on real HW as well.

Rationale of moving KVM RISC-V to drivers/staging is to continue KVM RISC-V development without breaking the Linux RISC-V patch acceptance policy until RISC-V H-extension is frozen.

It is fair to say that Kroah-Hartman was not impressed; he stated that circumventing policies in other parts of the kernel tree is not the purpose of the staging directory, so the RISC-V KVM code would not be accepted there. Paolo Bonzini, the overall maintainer for KVM, added that "the RISC-V acceptance policy as is just doesn't work", as demonstrated by the fact that developers are having to work around it. This is especially unfortunate here, he said, because the RISC-V KVM implementation "is also a very good example of how to do things right". He went on to suggest that there may be some players out there who benefit from the slowing down of patches on their way into the kernel.

Kroah-Hartman responded that slowing down the merging of useful code is "horrible" and that the job of the kernel community is to make hardware work; a policy that prevents the merging of good code that adds support for existing hardware does not make sense. He asked the RISC-V maintainers to explain this policy; they have yet to answer that question. Back in April, though, RISC-V maintainer Palmer Dabbelt acknowledged that the acceptance policy is not producing the desired results:

My goal with the RISC-V stuff has always been getting us to a place where we have real shipping products running a software stack that is as close as possible to the upstream codebases. I see that as the only way to get the software stack to a point where it can be sustainably maintained. The "only frozen extensions" policy was meant to help this by steering vendors towards a common base we could support, but in practice it's just not working out.

He added that the policy could change, but only when there is a new policy for patch acceptance in place that everybody agrees to. Co-maintainer Paul Walmsley, though, placed the blame with RISC-V International's specification process and said that any fixes should be applied there instead.

It is not that hard to understand why the RISC-V maintainers might not want to attempt to support every nonstandard CPU that is out there. The open nature of RISC-V makes it relatively easy for anybody to create their own variant, and supporting them all could become unworkable. The counterpoint is, as Kroah-Hartman said, that the kernel's job is to run on the hardware that is out there. Blocking support for shipping systems can only have the effect of pushing those systems toward vendor-supplied kernels with a lot of out-of-tree code — an unfortunate outcome for what is supposed to be an open architecture.

When the subject is support for a feature as fundamental as virtualization, the question becomes even more urgent. Hopefully this episode will lead to a rethinking of the patch-acceptance policies for the RISC-V architecture. Failing that, Kroah-Hartman has signaled his willingness to allow that support into staging if there is truly no alternative. So Linux seems likely to gain KVM support for RISC-V in the relatively near future, even if it's necessary to circumvent the architecture's maintainers to do it.

Index entries for this article
KernelArchitectures/RISC-V
KernelKVM
KernelVirtualization/KVM


(Log in to post comments)

Why RISC-V doesn't (yet) support KVM

Posted May 20, 2021 18:29 UTC (Thu) by iabervon (subscriber, #722) [Link]

It seems like a significant number of independent hardware implementations shipping that agree with each other and a proposed document should be a reasonable equivalent to the extension being frozen, with respect to supporting it in the kernel. It describes a practice with consensus behind it, even if the Foundation isn't yet willing to encourage that practice.

Why RISC-V doesn't (yet) support KVM

Posted May 20, 2021 21:02 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

https://lwn.net/ml/linux-kernel/a49a7142-104e-fdaa-4a6a-6... provides additional context as to how feature creep is delaying the freezing of the hypervisor specification.

Why RISC-V doesn't (yet) support KVM

Posted May 20, 2021 21:36 UTC (Thu) by pdp7pdp7 (subscriber, #96517) [Link]

pbonzini: I am confused about your comment [1] that some people may believe it could be beneficial to slow down merging of RISC-V support upstream. I am trying to understand who would benefit in that scenario. Is the theoretical benefit that it could reduce the RISC-V maintainer workload?

[1] https://lore.kernel.org/linux-riscv/daa30135-8757-8d33-a9...

Why RISC-V doesn't (yet) support KVM

Posted May 20, 2021 22:25 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

The patch acceptance policy can become an incentive to game the system:

1. Company X don't have hardware ready that implements an extension

2. Company X slows down ratification of the extension

3. Competitors must ship out of tree code to support the extension in Linux

The conflict of interest is outside Linux, which is what Paul Welmsley alluded to with respect to speeding up the process of ratification, but it affects Linux and the maintainers are caught in the crossfire.

Looking at the history of the hypervisor extension specification[1], the last significant change was done about one year ago (May 15, 2020). There is no justification at all for the specification not to be frozen yet.

[1] https://github.com/riscv/riscv-isa-manual/commits/master/...

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 0:09 UTC (Fri) by yootis (subscriber, #4762) [Link]

So which "company X" is slowing the system down now?

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 9:11 UTC (Fri) by farnz (subscriber, #17727) [Link]

My understanding is that the slowdown is not a specific company this time - it's getting the PLIC and CLIC specifications close enough to done that the RISC-V Foundation can be confident that the Hypervisor extension will not need to be modified to support direct delivery of interrupts to guest IRQ handlers without hypervisor involvement.

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 12:08 UTC (Fri) by pbonzini (subscriber, #60935) [Link]

Which is completely useless, since you can always make a subextension.

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 1:45 UTC (Fri) by ncm (guest, #165) [Link]

If it's not frozen, then by definition it might still change. Are kernel maintainers prepared to support more than one version of RISC-V virtualization?

To me the worst continuing omission is the B extension, and the worst missing bit of that is the POPCOUNT instruction, which really should be in with the base instructions alongside ADD and AND. I gather some lunatics are agitating to make it optional even in B, which is insane. POPCOUNT has been in or lately added to every production architecture ever, for reasons. (And, no, trap emulation would be much worse than useless.)

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 7:43 UTC (Fri) by pbonzini (subscriber, #60935) [Link]

Indeed, just give me CLZ/CTZ/POPCOUNT/ROL/ROR/SEXT in the base instruction set and call it a day...

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 10:54 UTC (Fri) by khim (subscriber, #9252) [Link]

> Are kernel maintainers prepared to support more than one version of RISC-V virtualization?

Why would that be a problem? AMD and Intel use totally different schemes and they both are supported.

And I think being implemented in ASIC which is shipping should be considered “effectively frozen”. I can see how people may implement thousands of different things on FPGA. Supporting these would be a nightmare from practical POV. And FPGA implementation can definitely be changed thus it's not “effectively frozen” anyway. But if something is already in ASIC it couldn't ever be changed (on that specific ASIC) thus I think kernel should support it.

Kernel supports lots of devices (WiFi and others) which only support draft of some spec or another… why RISC-V extensions should be different?

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 15:11 UTC (Fri) by mss (subscriber, #138799) [Link]

> Why would that be a problem? AMD and Intel use totally different schemes and they both are supported.

It's a waste of effort to implement what's effectively the same thing two different ways.
And these aren't trivial things.

Why RISC-V doesn't (yet) support KVM

Posted May 23, 2021 7:38 UTC (Sun) by ssmith32 (subscriber, #72404) [Link]

It's not a waste of effort for the people who are stuck with one implementation or the other: in short, the users of linux. Particularly when that implementation is baked into asic hardware.

Why RISC-V doesn't (yet) support KVM

Posted May 23, 2021 11:57 UTC (Sun) by mss (subscriber, #138799) [Link]

My comment was more about not having two different hardware schemes in the first place (like VMX and SVM for x86).

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 4:11 UTC (Fri) by pabs (subscriber, #43278) [Link]

It seems like shipping hardware should trump specifications; the Linux kernel already has code for working around quirks in devices' support for various other specs, so why not do the same for RISC-V and KVM?

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 4:43 UTC (Fri) by wsy (subscriber, #121706) [Link]

That's why I don't like standards made by committees.

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 7:59 UTC (Fri) by rwmj (subscriber, #5474) [Link]

It's nothing to do with "standards made by committees" (which this is not - some very very smart people worked on this standard). It's about some CPU vendors trying to game things.

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 21:40 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link]

Whether the people working on something are smart or not has nothing to do with whether or not it's being done by a committee. A committee of geniuses working in their element is still a committee. Committees can be a great way of doing things. There are any number of famous, successful, and greatly loved projects that were done by committee. Almost all of the popular translations of the Bible were done by committee, for instance.

I think that points to the big strength and the big danger of committees. A well run committee can get the minds of all its members working on a problem so they can do something none of them could manage alone. A poorly run committee puts the minds of its members in opposition, so the group gets less done than any of them could manage alone. The difference is whether the members can put aside their parochial interests to advance the good of the project. If they can, the committee can be a huge success. If they can't the committee will almost surely bog down into infighting. Vendors trying to game the system are a great example of the weakness of a committee structure; the members are so wrapped up by their own interests that the project as a whole suffers.

Why RISC-V doesn't (yet) support KVM

Posted May 22, 2021 10:41 UTC (Sat) by willy (subscriber, #9762) [Link]

I'll convey your thoughts on this to the NVMe committee, the SCSI committee, the C committee, the SQL committee, the IEEE 754 committee and the USB committee.

Committees can be bad, but they can also be very, very good.

Why RISC-V doesn't (yet) support KVM

Posted May 23, 2021 7:40 UTC (Sun) by ssmith32 (subscriber, #72404) [Link]

What standards do you like that weren't done by committee? So do you think the metric system is an epic boondoggle, and we all should be using slugs to measure mass?

K210 MMU support

Posted May 21, 2021 8:26 UTC (Fri) by geert (subscriber, #98403) [Link]

> The counterpoint is, as Kroah-Hartman said, that the kernel's job is to run on the hardware that is out there.

So can we get Canaan Kendryte K210 MMU support in drivers/staging/, too? ;-)

Why RISC-V doesn't (yet) support KVM

Posted May 21, 2021 18:54 UTC (Fri) by pwsan (subscriber, #56604) [Link]

Jonathan writes:

> The counterpoint is, as Kroah-Hartman said, that the kernel's job is to run on the hardware that is out there. Blocking support for shipping systems can only have the effect of pushing those systems toward vendor-supplied kernels with a lot of out-of-tree code — an unfortunate outcome for what is supposed to be an open architecture.

> When the subject is support for a feature as fundamental as virtualization, the question becomes even more urgent. Hopefully this episode will lead to a rethinking of the patch-acceptance policies for the RISC-V architecture.

One of the issues with the proposal to merge KVM support now for RISC-V is that no one has yet been able to identify any shipping systems that support the RISC-V hypervisor extension. What's being proposed, instead, is that we merge support for a hypervisor extension that isn't yet part of the RISC-V ISA. While we certainly understand Anup's and Paolo's desire to get their code merged upstream as soon as possible, I'm afraid these important points have been lost, both in the article and in most of the discussion.

Why RISC-V doesn't (yet) support KVM

Posted May 22, 2021 8:19 UTC (Sat) by pbonzini (subscriber, #60935) [Link]

As far as userspace is concerned, the API is only based on ratified specifications. The details of the hypervisor spec is completely hidden within Linux code. At the same time, until KVM is merged into upstream Linux the whole stack (QEMU, kvmtool) has to be developed in a fork, and new kernel features are harder to review. All this is simply *not how Linux is developed*.

Why RISC-V doesn't (yet) support KVM

Posted May 22, 2021 16:37 UTC (Sat) by mfuzzey (subscriber, #57966) [Link]

Is it shipping *systems* or *chips* that don't exist yet?
Presumably those developing this feature are testing it on something.

If the chips exist just no one is selling systems with them yet (presumably the developers having custom prototype boards) then I see no reason not to merge the feature as the chips aren't going to change.

OTOH if no chips exist yet (eg if prototyping is being done on a FPGA) I can see the rational for waiting if there is a real risk that the final hardware is different.
But even in this case does it really matter given that the user space won't be affected? Sure the kernel code may have to change for the final hardware but if the prototype hardware never shipped does this matter as there will be no need to support both in the kernel.

I may be missing something obvious here as I'm not well versed in RISC V

Why RISC-V doesn't (yet) support KVM

Posted May 23, 2021 7:44 UTC (Sun) by ssmith32 (subscriber, #72404) [Link]

I think this was the argument being made here:

"In fact, quite a few people have already implemented RISC-V H-extension in hardware as well and KVM RISC-V works on real HW as well. "

And which sounds like the main reason it may end up being merged into stg.

Why RISC-V doesn't (yet) support KVM

Posted May 26, 2021 0:29 UTC (Wed) by pwsan (subscriber, #56604) [Link]

> Presumably those developing this feature are testing it on something.

Probably most people are using QEMU to experiment with this, with the instructions and code here:

https://github.com/kvm-riscv


Copyright © 2021, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds