diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2026-06-19 18:56:49 +0300 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2026-06-19 18:56:49 +0300 |
| commit | c98d767b34574be82b74d77d02264a830ae1cadd (patch) | |
| tree | 3dc16f4ebd9d7bdeb7dd4a9c84dae88c692e9ca4 | |
| parent | 08c7183f5b9ffe4408e74fff848a4cc2105361d4 (diff) | |
| parent | ef057cbf825e03b63f6edf5980f96abf3c53089d (diff) | |
| download | linux-c98d767b34574be82b74d77d02264a830ae1cadd.tar.xz | |
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm updates from Paolo Bonzini:
"arm64:
This is a bit of an odd merge window on the KVM/arm64 front. There
is absolutely no new feature in the pull request. It is purely
fixes, because it is simply becoming too hard to review new stuff
when so many AI-fuelled fixes hit the list.
- Significant cleanup of the vgic-v5 PPI support which was merged in
7.1. This makes the code more maintainable, and squashes a couple
of bugs in the meantime
- Set of fixes for the handling of the MMU in an NV context,
particularly VNCR-triggered faults. S1POE support is fixed as well
- Large set of pKVM fixes, mostly addressing recurring issues around
hypervisor tracking of donated pages in obscure cases where the
donation could fail and leave things in a bizarre state
- Fixes for the so-called "lazy vgic init", which resulted in
sleeping operations in non-preemptible sections. This turned out to
be far more invasive than initially expected..
- Reduce the overhead of L1/L2 context switch by not touching the FP
registers
- Fix the way non-implemented page sizes are dealt with when a guest
insist on using them for S2 translation
- The usual set of low-impact fixes and cleanups all over the map
Loongarch:
- On a request for lazy FPU load, load all FPU state that the VM
supports instead of enabling only the part (FPU, LSX or LASX) that
caused the FPU load request
- Some enhancements about interrupt injection
- Some bug fixes and other small changes
RISC-V:
- Batch G-stage TLB flushes for GPA range based page table updates
- Convert HGEI line management to fully per-HART
- Fix missing CSR dirty marking when FWFT state updated via ONE_REG
- Fix stale FWFT feature exposure to Guest/VM
- Speed up dirty logging write faults using MMU rwlock and atomic PTE
updates using cmpxchg() for permission-only changes
- Use flexible array for APLIC IRQ state
- Use kvm_slot_dirty_track_enabled() for logging enable check on a
memslot
- Avoid skipping valid pages in kvm_riscv_gstage_wp_range()
- Avoid skipping valid pages in kvm_riscv_gstage_unmap_range()
- Use endian-specific __lelong for NACL shared memory
S390:
- KVM_PRE_FAULT_MEMORY support
- Support for 2G hugepages
- Support for the ASTFLEIE 2 facility
- Support for fast inject using kvm_arch_set_irq_inatomic
- Fix potential leak of uninitialized bytes
- A few more misc gmap fixes
x86:
- Generic support for the more granular permissions allowed by EPT,
namely "read" (which was previously usurping the U bit) and
separate execution bits for kernel and userspace
- Do not assume that all page tables start with U=1/W=1/NX=0 at the
root, as AMD GMET needs to have U=0 at the root
- Introduce common assembly macros for use within Intel and AMD
vendor-specific vmentry code. This touches the SPEC_CTRL handling,
which is now entirely done in assembly for Intel (by reusing the
AMD code that already existed), and register save/restore which
uses some macro magic to compute the offsets in the struct. Both of
these are preparatory changes for upcoming APX support
- Clean up KVM's register tracking and storage, primarily to prepare
for APX support, which expands the maximum number of GPRs from 16
to 32
- Keep a single copy of the PDPTRs rather than two, since
architecturally there is just one
- Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor
code gets a chance to handle things like reaping the PML buffer
- Update KVM's view of PV async enabling if and only if the MSR write
fully succeeds
- Fix a variety of issues where the emulator doesn't honor
guest-debug state, and clean up related code along the way
- Synthesize EPT Violation and #NPF "error code" bits when injecting
faults into L1 that didn't originate in hardware (in which case the
VMCS/VMCB doesn't hold relevant information)
- Add support for virtualizing (well, emulating) AMD's flavor of
CPL>0 CPUID faulting
- Clean up the GPR APIs so that KVM's use of "raw" is consistent, and
fix a variety of minor bugs along the way
- Fix an OOB memory access due to not checking the VP ID when
handling a Hyper-V PV TLB flush for L2
- Fix a bug in the mediated PMU's handling of fixed counters that
allowed the guest to bypass the PMU event filter
- Allow userspace to return EAGAIN when handling SNP and TDX
hypercalls, so the KVM can forward a "retry" status code to the
guest, and reserve all unused error codes for future usage
- Overhaul the TDP MMU => S-EPT code to move as much S-EPT specific
logic as possible into the TDX code, and to funnel (almost) all
S-EPT updates into a single chokepoint. The motivation is largely
to prepare for upcoming Dynamic PAMT support, but the cleanups are
nice to have on their own
- Plug a hole in shadow page table handling, where KVM fails to
recursively zap nested EPT/NPT shadow page tables when the nested
hypervisor tears down its own EPT/NPT page tables from the bottom
up
x86 (Intel):
- Support for nested MBEC (Mode-Based Execute Control), see above in
the generic section; also run with MBEC enabled even for non-nested
mode
- Use the kernel's "enum pg_level" in the TDX APIs instead of the
TDX-Module's level definitions (which are 0-based)
- Rework the TDX memory APIs to not require/assume that guest memory
is backed by "struct page" (in prepartion for guest_memfd hugepage
support)
- Fix a largely benign bug where KVM TDX would incorrectly state it
could emulate several x2APIC MSRs
- Use the "safe" WRMSR API when proxying LBR MSR writes as the
to-be-written value is guest controlled and completely unvalidated
x86 (AMD):
- Support for nested GMET (Guest Mode Execution Trap), see above in
the generic section; also run with GMET enabled even for non-nested
mode
- Fixes and minor cleanups to GHCB handling, on top of the earlier
work already merged into 7.1-rc
- Ensure KVM's copy of CR0 and CR3 are up-to-date prior to invoking
fastpath handlers
- Add support for virtualizing gPAT (KVM previously just used L1's
PAT when running L2)
- Fix goofs where KVM mishandles side effects (e.g. single-step and
PMC updates) when emulating VMRUN
- Fix a variety of bugs in AVIC's handling of x2APIC MSR
interception, most notably where KVM didn't disable interception of
IRR, ISR, and TMR regs
- Add support for virtualizing Host-Only/Guest-Only bits in the
mediated PMU
- Don't advertise support for unusable VM types, and account for VM
types that are disabled by firmware, e.g. to mitigate security
vulnerabilities
- Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with
bugs and unnecessarily complicated, and add comprehensive tests
- Clean up and deduplicate the SEV page pinning code
- Fix minor goofs related to writing back CPUID information after
firmware rejects a CPUID page for an SNP vCPU
Generic:
- Rename invalidate_begin() to invalidate_start() throughout KVM to
follow the kernel's nomenclature, e.g. for mmu_notifiers
- Use guard() to cleanup up various KVM+VFIO flows
- Minor cleanups
guest_memfd:
- Return -EEXIST instead of -EINVAL if userspace attempts to bind a
gmem range to multiple memslots, and fix the test that was supposed
to ensure KVM returns -EEXIST
- Treat memslot binding offsets and sizes as unsigned values to fix a
bug where KVM interprets a large "offset + size" as a negative
value and allows a nonsensical offset
- Use the inode number instead of the page offset for the NUMA
interleaving index to fix a bug where the effective index would
jump by two for consecutive pages (the caller also adds in the page
offset)
Selftests:
- Randomize the dirty log test's delay when reaping the bitmap on the
first pass, as always waiting only 1ms hid a KVM RISC-V bug as the
test reaped the bitmap before KVM could build up enough state to
hit the bug
- A pile of one-off fixes and cleanups"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (326 commits)
KVM: x86/mmu: Ensure hugepage is in by slot before checking max mapping level
KVM: x86: Fix shadow paging use-after-free due to unexpected role
KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
KVM: s390: Enable adapter_indicators_set to use mapped pages
KVM: s390: Add map/unmap ioctl and clean mappings post-guest
riscv: kvm: Use endian-specific __lelong for NACL shared memory
KVM: selftests: access_tracking_perf_test: bump number of NUMA nodes to 32
KVM: s390: vsie: Implement ASTFLEIE facility 2
KVM: s390: vsie: Refactor handle_stfle
s390/sclp: Detect ASTFLEIE 2 facility
KVM: s390: Minor refactor of base/ext facility lists
KVM: x86/mmu: move pdptrs out of the MMU
KVM: x86: check that kvm_handle_invpcid is only invoked with shadow paging
KVM: nSVM: invalidate cached PDPTRs across nested NPT transitions
KVM: nVMX: remove unnecessary code in prepare_vmcs02_rare
KVM: x86: remove nested_mmu from mmu_is_nested()
KVM: arm64: vgic-its: Make ABI commit helpers return void
KVM: s390: Initialize KVM_S390_GET_CMMA_BITS memory
LoongArch: KVM: Add missing slots_lock for device register/unregister
LoongArch: KVM: Validate irqchip index in irqfd routing
...
187 files changed, 6164 insertions, 3549 deletions
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 52bbbb553ce1..a833d90845b9 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -4944,10 +4944,13 @@ Errors: #define KVM_STATE_NESTED_FORMAT_SVM 1 #define KVM_STATE_NESTED_VMX_VMCS_SIZE 0x1000 + #define KVM_STATE_NESTED_SVM_VMCB_SIZE 0x1000 #define KVM_STATE_NESTED_VMX_SMM_GUEST_MODE 0x00000001 #define KVM_STATE_NESTED_VMX_SMM_VMXON 0x00000002 + #define KVM_STATE_NESTED_GIF_SET 0x00000100 + #define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE 0x00000001 struct kvm_vmx_nested_state_hdr { @@ -4962,11 +4965,20 @@ Errors: __u64 preemption_timer_deadline; }; + struct kvm_svm_nested_state_hdr { + __u64 vmcb_pa; + __u64 gpat; + }; + struct kvm_vmx_nested_state_data { __u8 vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE]; __u8 shadow_vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE]; }; + struct kvm_svm_nested_state_data { + __u8 vmcb12[KVM_STATE_NESTED_SVM_VMCB_SIZE]; + }; + This ioctl copies the vcpu's nested virtualization state from the kernel to userspace. @@ -6471,7 +6483,8 @@ Errors: ========== =============================================================== EINVAL The specified `gpa` and `size` were invalid (e.g. not - page aligned, causes an overflow, or size is zero). + page aligned, causes an overflow, or size is zero), or the VM + is UCONTROL (s390). ENOENT The specified `gpa` is outside defined memslots. EINTR An unmasked signal is pending and no page was processed. EFAULT The parameter address was invalid. @@ -6494,7 +6507,7 @@ Errors: KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory for the current vCPU state. KVM maps memory as if the vCPU generated a stage-2 read page fault, e.g. faults in memory as needed, but doesn't break -CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed. +CoW. On x86, KVM does not mark any newly created stage-2 PTE as Accessed. In the case of confidential VM types where there is an initial set up of private guest memory before the guest is 'finalized'/measured, this ioctl @@ -8553,6 +8566,20 @@ KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM By default, KVM relaxes the consisten bit to be cleared. Note that the vmcs02 bit is still completely controlled by the host, regardless of the quirk setting. + +KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT By default, KVM for nested SVM guests + shares the IA32_PAT MSR between L1 and + L2. This is legacy behavior and does + not match the AMD architecture + specification. When this quirk is + disabled and nested paging (NPT) is + enabled for L2, KVM correctly + virtualizes a separate guest PAT + register for L2, using the g_pat + field in the VMCB. When NPT is + disabled for L2, L1 and L2 continue + to share the IA32_PAT MSR regardless + of the quirk setting. ======================================== ================================================ 7.32 KVM_CAP_MAX_VCPU_ID @@ -8818,6 +8845,9 @@ block sizes is exposed in KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES as a This capability, if enabled, will cause KVM to exit to userspace with KVM_EXIT_HYPERCALL exit reason to process some hypercalls. +Userspace may fail the hypercall by setting hypercall.ret to EINVAL +or may request the hypercall to be retried the next time the guest run +by setting hypercall.ret to EAGAIN. Calling KVM_CHECK_EXTENSION for this capability will return a bitmask of hypercalls that can be configured to exit to userspace. @@ -8904,6 +8934,21 @@ helpful if user space wants to emulate instructions which are not This capability can be enabled dynamically even if VCPUs were already created and are running. +7.47 KVM_CAP_S390_HPAGE_2G +-------------------------- + +:Architectures: s390 +:Parameters: none +:Returns: 0 on success; -EINVAL if hpage_2g module parameter was not set, + cmma is enabled, or the VM has the KVM_VM_S390_UCONTROL + flag set; -EBUSY if vCPUs were already created for the VM. + +With this capability the KVM support for memory backing with 2g pages +through hugetlbfs can be enabled for a VM. After the capability is +enabled, cmma can't be enabled anymore and pfmfi and the storage key +interpretation are disabled. If cmma has already been enabled or the +hpage_2g module parameter is not set to 1, -EINVAL is returned. + 8. Other capabilities. ====================== @@ -9363,6 +9408,8 @@ means the VM type with value @n is supported. Possible values of @n are:: #define KVM_X86_SW_PROTECTED_VM 1 #define KVM_X86_SEV_VM 2 #define KVM_X86_SEV_ES_VM 3 + #define KVM_X86_SNP_VM 4 + #define KVM_X86_TDX_VM 5 Note, KVM_X86_SW_PROTECTED_VM is currently only for development and testing. Do not use KVM_X86_SW_PROTECTED_VM for "real" VMs, and especially not in diff --git a/Documentation/virt/kvm/devices/arm-vgic-v5.rst b/Documentation/virt/kvm/devices/arm-vgic-v5.rst index 29335ea823fc..70b9162755c7 100644 --- a/Documentation/virt/kvm/devices/arm-vgic-v5.rst +++ b/Documentation/virt/kvm/devices/arm-vgic-v5.rst @@ -12,8 +12,8 @@ Only one VGIC instance may be instantiated through this API. The created VGIC will act as the VM interrupt controller, requiring emulated user-space devices to inject interrupts to the VGIC instead of directly to CPUs. -Creating a guest GICv5 device requires a host GICv5 host. The current VGICv5 -device only supports PPI interrupts. These can either be injected from emulated +Creating a guest GICv5 device requires a GICv5 host. The current VGICv5 device +only supports PPI interrupts. These can either be injected from emulated in-kernel devices (such as the Arch Timer, or PMU), or via the KVM_IRQ_LINE ioctl. @@ -25,7 +25,7 @@ Groups: request the initialization of the VGIC, no additional parameter in kvm_device_attr.addr. Must be called after all VCPUs have been created. - KVM_DEV_ARM_VGIC_USERPSPACE_PPIs + KVM_DEV_ARM_VGIC_USERSPACE_PPIS request the mask of userspace-drivable PPIs. Only a subset of the PPIs can be directly driven from userspace with GICv5, and the returned mask informs userspace of which it is allowed to drive via KVM_IRQ_LINE. diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst index 5e3805820010..66e714f2fcfa 100644 --- a/Documentation/virt/kvm/devices/vcpu.rst +++ b/Documentation/virt/kvm/devices/vcpu.rst @@ -37,8 +37,11 @@ Returns: A value describing the PMUv3 (Performance Monitor Unit v3) overflow interrupt number for this vcpu. This interrupt could be a PPI or SPI, but the interrupt type must be same for each vcpu. As a PPI, the interrupt number is the same for -all vcpus, while as an SPI it must be a separate number per vcpu. For -GICv5-based guests, the architected PPI (23) must be used. +all vcpus, while as an SPI it must be a separate number per vcpu. + +For GICv5-based guests, the architected PPI (23) must be used, and must be +communicated as the full GICv5-style Interrupt ID, i.e., 0x20000017. This ioctl +can be omitted altogether for a GICv5-based guest. 1.2 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_INIT --------------------------------------- diff --git a/Documentation/virt/kvm/x86/mmu.rst b/Documentation/virt/kvm/x86/mmu.rst index 2b3b6d442302..666aa179601a 100644 --- a/Documentation/virt/kvm/x86/mmu.rst +++ b/Documentation/virt/kvm/x86/mmu.rst @@ -184,10 +184,8 @@ Shadow pages contain the following information: Contains the value of efer.nx for which the page is valid. role.cr0_wp: Contains the value of cr0.wp for which the page is valid. - role.smep_andnot_wp: - Contains the value of cr4.smep && !cr0.wp for which the page is valid - (pages for which this is true are different from other pages; see the - treatment of cr0.wp=0 below). + role.cr4_smep: + Contains the value of cr4.smep for which the page is valid. role.smap_andnot_wp: Contains the value of cr4.smap && !cr0.wp for which the page is valid (pages for which this is true are different from other pages; see the @@ -435,8 +433,8 @@ from being written by the kernel after cr0.wp has changed to 1, we make the value of cr0.wp part of the page role. This means that an spte created with one value of cr0.wp cannot be used when cr0.wp has a different value - it will simply be missed by the shadow page lookup code. A similar issue -exists when an spte created with cr0.wp=0 and cr4.smep=0 is used after -changing cr4.smep to 1. To avoid this, the value of !cr0.wp && cr4.smep +exists when an spte created with cr0.wp=0 and cr4.smap=0 is used after +changing cr4.smap to 1. To avoid this, the value of !cr0.wp && cr4.smap is also made a part of the page role. Large pages diff --git a/MAINTAINERS b/MAINTAINERS index 9f4ddd4c69f8..83684a0b170e 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -14253,6 +14253,7 @@ L: kvm@vger.kernel.org S: Supported P: Documentation/process/maintainer-kvm-x86.rst T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git +F: Documentation/process/maintainer-kvm-x86.rst F: arch/x86/include/asm/kvm* F: arch/x86/include/asm/svm.h F: arch/x86/include/asm/vmx*.h diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 9209c54350c7..bae2c4f92ef5 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1095,7 +1095,8 @@ struct kvm_vcpu_arch { #define IN_NESTED_ERET __vcpu_single_flag(sflags, BIT(7)) /* SError pending for nested guest */ #define NESTED_SERROR_PENDING __vcpu_single_flag(sflags, BIT(8)) - +/* KVM is currently emulating an L2 to L1 exception */ +#define IN_NESTED_EXCEPTION __vcpu_single_flag(sflags, BIT(9)) #define vcpu_sve_max_vq(vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) @@ -1252,13 +1253,14 @@ void kvm_arm_resume_guest(struct kvm *kvm); #define vcpu_has_run_once(vcpu) (!!READ_ONCE((vcpu)->pid)) #ifndef __KVM_NVHE_HYPERVISOR__ -#define kvm_call_hyp_nvhe(f, ...) \ +#define kvm_call_hyp_nvhe(f, ...) \ ({ \ struct arm_smccc_res res; \ \ arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(f), \ ##__VA_ARGS__, &res); \ - WARN_ON(res.a0 != SMCCC_RET_SUCCESS); \ + if (WARN_ON(res.a0 != SMCCC_RET_SUCCESS)) \ + res.a1 = -EOPNOTSUPP; \ \ res.a1; \ }) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index ad19de1d0654..4974492744cc 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -152,5 +152,6 @@ extern unsigned long kvm_nvhe_sym(__icache_flags); extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits); extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl); extern unsigned long kvm_nvhe_sym(hyp_nr_cpus); +extern unsigned int kvm_nvhe_sym(hyp_gicv3_nr_lr); #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 01e9c72d6aa7..6eae7e7e2a68 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -318,8 +318,7 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) * Must be called from hyp code running at EL2 with an updated VTTBR * and interrupts disabled. */ -static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu, - struct kvm_arch *arch) +static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu) { write_sysreg(mmu->vtcr, vtcr_el2); write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 799d7a09edfd..9a22df0c5120 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -266,6 +266,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = { }; static const struct arm64_ftr_bits ftr_id_aa64isar2[] = { + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_ATS1A_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_LUT_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_CSSC_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_RPRFM_SHIFT, 4, 0), diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S index 634ddc904244..37c6976e44a4 100644 --- a/arch/arm64/kernel/hyp-stub.S +++ b/arch/arm64/kernel/hyp-stub.S @@ -104,11 +104,9 @@ SYM_CODE_START_LOCAL(__finalise_el2) mov_q x0, HCR_HOST_VHE_FLAGS msr_hcr_el2 x0 - // Use the EL1 allocated stack, per-cpu offset + // Use the EL1 allocated stack mrs x0, sp_el1 mov sp, x0 - mrs x0, tpidr_el1 - msr tpidr_el2, x0 // FP configuration, vectors mrs_s x0, SYS_CPACR_EL12 diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index cbea4d9ee955..4155fe89b58a 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -39,10 +39,9 @@ static const u8 default_ppi[] = { [TIMER_HVTIMER] = 28, }; -static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx); static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, struct arch_timer_context *timer_ctx); -static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx); +static bool kvm_timer_pending(struct arch_timer_context *timer_ctx); static void kvm_arm_timer_write(struct kvm_vcpu *vcpu, struct arch_timer_context *timer, enum kvm_arch_timer_regs treg, @@ -52,11 +51,17 @@ static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, enum kvm_arch_timer_regs treg); static bool kvm_arch_timer_get_input_level(int vintid); -static struct irq_ops arch_timer_irq_ops = { +static unsigned long kvm_arch_timer_get_irq_flags(void) +{ + return kvm_vgic_global_state.no_hw_deactivation ? VGIC_IRQ_SW_RESAMPLE : 0; +} + +static const struct irq_ops arch_timer_irq_ops = { + .get_flags = kvm_arch_timer_get_irq_flags, .get_input_level = kvm_arch_timer_get_input_level, }; -static struct irq_ops arch_timer_irq_ops_vgic_v5 = { +static const struct irq_ops arch_timer_irq_ops_vgic_v5 = { .get_input_level = kvm_arch_timer_get_input_level, .queue_irq_unlock = vgic_v5_ppi_queue_irq_unlock, .set_direct_injection = vgic_v5_set_ppi_dvi, @@ -224,7 +229,7 @@ static irqreturn_t kvm_arch_timer_handler(int irq, void *dev_id) else ctx = map.direct_ptimer; - if (kvm_timer_should_fire(ctx)) + if (kvm_timer_pending(ctx)) kvm_timer_update_irq(vcpu, true, ctx); if (userspace_irqchip(vcpu->kvm) && @@ -257,7 +262,7 @@ static u64 kvm_timer_compute_delta(struct arch_timer_context *timer_ctx) return kvm_counter_compute_delta(timer_ctx, timer_get_cval(timer_ctx)); } -static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx) +static bool kvm_timer_enabled(struct arch_timer_context *timer_ctx) { WARN_ON(timer_ctx && timer_ctx->loaded); return timer_ctx && @@ -294,7 +299,7 @@ static u64 kvm_timer_earliest_exp(struct kvm_vcpu *vcpu) struct arch_timer_context *ctx = &vcpu->arch.timer_cpu.timers[i]; WARN(ctx->loaded, "timer %d loaded\n", i); - if (kvm_timer_irq_can_fire(ctx)) + if (kvm_timer_enabled(ctx)) min_delta = min(min_delta, kvm_timer_compute_delta(ctx)); } @@ -358,7 +363,7 @@ static enum hrtimer_restart kvm_hrtimer_expire(struct hrtimer *hrt) return HRTIMER_NORESTART; } -static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx) +static bool kvm_timer_pending(struct arch_timer_context *timer_ctx) { enum kvm_arch_timers index; u64 cval, now; @@ -391,7 +396,7 @@ static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx) !(cnt_ctl & ARCH_TIMER_CTRL_IT_MASK); } - if (!kvm_timer_irq_can_fire(timer_ctx)) + if (!kvm_timer_enabled(timer_ctx)) return false; cval = timer_get_cval(timer_ctx); @@ -405,22 +410,30 @@ int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu) return vcpu_has_wfit_active(vcpu) && wfit_delay_ns(vcpu) == 0; } +static u64 kvm_timer_needs_notify(struct kvm_vcpu *vcpu) +{ + u64 v = vcpu->run->s.regs.device_irq_level; + + v ^= kvm_timer_pending(vcpu_vtimer(vcpu)) ? KVM_ARM_DEV_EL1_VTIMER : 0; + v ^= kvm_timer_pending(vcpu_ptimer(vcpu)) ? KVM_ARM_DEV_EL1_PTIMER : 0; + + return v & (KVM_ARM_DEV_EL1_VTIMER | KVM_ARM_DEV_EL1_PTIMER); +} + +bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu) +{ + return !!kvm_timer_needs_notify(vcpu); +} + /* * Reflect the timer output level into the kvm_run structure */ -void kvm_timer_update_run(struct kvm_vcpu *vcpu) +bool kvm_timer_update_run(struct kvm_vcpu *vcpu) { - struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); - struct arch_timer_context *ptimer = vcpu_ptimer(vcpu); - struct kvm_sync_regs *regs = &vcpu->run->s.regs; - - /* Populate the device bitmap with the timer states */ - regs->device_irq_level &= ~(KVM_ARM_DEV_EL1_VTIMER | - KVM_ARM_DEV_EL1_PTIMER); - if (kvm_timer_should_fire(vtimer)) - regs->device_irq_level |= KVM_ARM_DEV_EL1_VTIMER; - if (kvm_timer_should_fire(ptimer)) - regs->device_irq_level |= KVM_ARM_DEV_EL1_PTIMER; + u64 mask = kvm_timer_needs_notify(vcpu); + if (mask) + vcpu->run->s.regs.device_irq_level ^= mask; + return !!mask; } static void kvm_timer_update_status(struct arch_timer_context *ctx, bool level) @@ -446,9 +459,8 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, { kvm_timer_update_status(timer_ctx, new_level); - timer_ctx->irq.level = new_level; trace_kvm_timer_update_irq(vcpu->vcpu_id, timer_irq(timer_ctx), - timer_ctx->irq.level); + new_level); if (userspace_irqchip(vcpu->kvm)) return; @@ -466,28 +478,25 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, kvm_vgic_inject_irq(vcpu->kvm, vcpu, timer_irq(timer_ctx), - timer_ctx->irq.level, + new_level, timer_ctx); } /* Only called for a fully emulated timer */ static void timer_emulate(struct arch_timer_context *ctx) { - bool should_fire = kvm_timer_should_fire(ctx); + bool pending = kvm_timer_pending(ctx); - trace_kvm_timer_emulate(ctx, should_fire); + trace_kvm_timer_emulate(ctx, pending); - if (should_fire != ctx->irq.level) - kvm_timer_update_irq(timer_context_to_vcpu(ctx), should_fire, ctx); - - kvm_timer_update_status(ctx, should_fire); + kvm_timer_update_irq(timer_context_to_vcpu(ctx), pending, ctx); /* - * If the timer can fire now, we don't need to have a soft timer - * scheduled for the future. If the timer cannot fire at all, - * then we also don't need a soft timer. + * If the timer is pending, we don't need to have a soft timer + * scheduled for the future. If the timer is disabled, then + * we don't need a soft timer either. */ - if (should_fire || !kvm_timer_irq_can_fire(ctx)) + if (pending || !kvm_timer_enabled(ctx)) return; soft_timer_start(&ctx->hrtimer, kvm_timer_compute_delta(ctx)); @@ -594,10 +603,10 @@ static void kvm_timer_blocking(struct kvm_vcpu *vcpu) * If no timers are capable of raising interrupts (disabled or * masked), then there's no more work for us to do. */ - if (!kvm_timer_irq_can_fire(map.direct_vtimer) && - !kvm_timer_irq_can_fire(map.direct_ptimer) && - !kvm_timer_irq_can_fire(map.emul_vtimer) && - !kvm_timer_irq_can_fire(map.emul_ptimer) && + if (!kvm_timer_enabled(map.direct_vtimer) && + !kvm_timer_enabled(map.direct_ptimer) && + !kvm_timer_enabled(map.emul_vtimer) && + !kvm_timer_enabled(map.emul_ptimer) && !vcpu_has_wfit_active(vcpu)) return; @@ -677,6 +686,7 @@ static inline void set_timer_irq_phys_active(struct arch_timer_context *ctx, boo static void kvm_timer_vcpu_load_gic(struct arch_timer_context *ctx) { struct kvm_vcpu *vcpu = timer_context_to_vcpu(ctx); + bool pending = kvm_timer_pending(ctx); bool phys_active = false; /* @@ -685,12 +695,12 @@ static void kvm_timer_vcpu_load_gic(struct arch_timer_context *ctx) * this point and the register restoration, we'll take the * interrupt anyway. */ - kvm_timer_update_irq(vcpu, kvm_timer_should_fire(ctx), ctx); + kvm_timer_update_irq(vcpu, pending, ctx); if (irqchip_in_kernel(vcpu->kvm)) phys_active = kvm_vgic_map_is_active(vcpu, timer_irq(ctx)); - phys_active |= ctx->irq.level; + phys_active |= pending; phys_active |= vgic_is_v5(vcpu->kvm); set_timer_irq_phys_active(ctx, phys_active); @@ -699,6 +709,7 @@ static void kvm_timer_vcpu_load_gic(struct arch_timer_context *ctx) static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu) { struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); + bool pending = kvm_timer_pending(vtimer); /* * Update the timer output so that it is likely to match the @@ -706,7 +717,7 @@ static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu) * this point and the register restoration, we'll take the * interrupt anyway. */ - kvm_timer_update_irq(vcpu, kvm_timer_should_fire(vtimer), vtimer); + kvm_timer_update_irq(vcpu, pending, vtimer); /* * When using a userspace irqchip with the architected timers and a @@ -718,7 +729,7 @@ static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu) * being de-asserted, we unmask the interrupt again so that we exit * from the guest when the timer fires. */ - if (vtimer->irq.level) + if (pending) disable_percpu_irq(host_vtimer_irq); else enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags); @@ -904,23 +915,6 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) timer_set_traps(vcpu, &map); } -bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu) -{ - struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); - struct arch_timer_context *ptimer = vcpu_ptimer(vcpu); - struct kvm_sync_regs *sregs = &vcpu->run->s.regs; - bool vlevel, plevel; - - if (likely(irqchip_in_kernel(vcpu->kvm))) - return false; - - vlevel = sregs->device_irq_level & KVM_ARM_DEV_EL1_VTIMER; - plevel = sregs->device_irq_level & KVM_ARM_DEV_EL1_PTIMER; - - return kvm_timer_should_fire(vtimer) != vlevel || - kvm_timer_should_fire(ptimer) != plevel; -} - void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) { struct arch_timer_cpu *timer = vcpu_timer(vcpu); @@ -1006,7 +1000,7 @@ static void unmask_vtimer_irq_user(struct kvm_vcpu *vcpu) { struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); - if (!kvm_timer_should_fire(vtimer)) { + if (!kvm_timer_pending(vtimer)) { kvm_timer_update_irq(vcpu, false, vtimer); if (static_branch_likely(&has_gic_active_state)) set_timer_irq_phys_active(vtimer, false); @@ -1288,7 +1282,12 @@ static int timer_irq_set_vcpu_affinity(struct irq_data *d, void *vcpu) static int timer_irq_set_irqchip_state(struct irq_data *d, enum irqchip_irq_state which, bool val) { - if (which != IRQCHIP_STATE_ACTIVE || !irqd_is_forwarded_to_vcpu(d)) + bool passthrough = which != IRQCHIP_STATE_ACTIVE || + !irqd_is_forwarded_to_vcpu(d) || + (kvm_vgic_global_state.type == VGIC_V5 && + vgic_is_v3(kvm_get_running_vcpu()->kvm)); + + if (passthrough) return irq_chip_set_parent_state(d, which, val); if (val) @@ -1301,15 +1300,7 @@ static int timer_irq_set_irqchip_state(struct irq_data *d, static void timer_irq_eoi(struct irq_data *d) { - /* - * On a GICv5 host, we still need to call EOI on the parent for - * PPIs. The host driver already handles irqs which are forwarded to - * vcpus, and skips the GIC CDDI while still doing the GIC CDEOI. This - * is required to emulate the EOIMode=1 on GICv5 hardware. Failure to - * call EOI unsurprisingly results in *BAD* lock-ups. - */ - if (!irqd_is_forwarded_to_vcpu(d) || - kvm_vgic_global_state.type == VGIC_V5) + if (!irqd_is_forwarded_to_vcpu(d)) irq_chip_eoi_parent(d); } @@ -1392,8 +1383,6 @@ static int kvm_irq_init(struct arch_timer_kvm_info *info) return -ENOMEM; } - if (kvm_vgic_global_state.no_hw_deactivation) - arch_timer_irq_ops.flags |= VGIC_IRQ_SW_RESAMPLE; WARN_ON(irq_domain_push_irq(domain, host_vtimer_irq, (void *)TIMER_VTIMER)); } @@ -1579,7 +1568,7 @@ static bool kvm_arch_timer_get_input_level(int vintid) ctx = vcpu_get_timer(vcpu, i); if (timer_irq(ctx) == vintid) - return kvm_timer_should_fire(ctx); + return kvm_timer_pending(ctx); } /* A timer IRQ has fired, but no matching timer was found? */ @@ -1591,8 +1580,8 @@ static bool kvm_arch_timer_get_input_level(int vintid) int kvm_timer_enable(struct kvm_vcpu *vcpu) { struct arch_timer_cpu *timer = vcpu_timer(vcpu); + const struct irq_ops *ops; struct timer_map map; - struct irq_ops *ops; int ret; if (timer->enabled) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 7def2fad21e8..50adfff75be8 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -52,6 +52,7 @@ #include <linux/irqchip/arm-gic-v5.h> +#include "vgic/vgic.h" #include "sys_regs.h" static enum kvm_mode kvm_mode = KVM_MODE_DEFAULT; @@ -1166,6 +1167,15 @@ static bool vcpu_mode_is_bad_32bit(struct kvm_vcpu *vcpu) return !kvm_supports_32bit_el0(); } +static bool kvm_irq_update_run(struct kvm_vcpu *vcpu) +{ + bool r; + + r = kvm_timer_update_run(vcpu); + r |= kvm_pmu_update_run(vcpu); + return r; +} + /** * kvm_vcpu_exit_request - returns true if the VCPU should *not* enter the guest * @vcpu: The VCPU pointer @@ -1187,13 +1197,11 @@ static bool kvm_vcpu_exit_request(struct kvm_vcpu *vcpu, int *ret) /* * If we're using a userspace irqchip, then check if we need * to tell a userspace irqchip about timer or PMU level - * changes and if so, exit to userspace (the actual level - * state gets updated in kvm_timer_update_run and - * kvm_pmu_update_run below). + * changes and if so, exit to userspace while updating the run + * state. */ if (unlikely(!irqchip_in_kernel(vcpu->kvm))) { - if (kvm_timer_should_notify_user(vcpu) || - kvm_pmu_should_notify_user(vcpu)) { + if (unlikely(kvm_irq_update_run(vcpu))) { *ret = -EINTR; run->exit_reason = KVM_EXIT_INTR; return true; @@ -1408,11 +1416,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) ret = handle_exit(vcpu, ret); } - /* Tell userspace about in-kernel device output levels */ - if (unlikely(!irqchip_in_kernel(vcpu->kvm))) { - kvm_timer_update_run(vcpu); - kvm_pmu_update_run(vcpu); - } + if (unlikely(!irqchip_in_kernel(vcpu->kvm))) + kvm_irq_update_run(vcpu); kvm_sigset_deactivate(vcpu); @@ -1496,8 +1501,13 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level, return vcpu_interrupt_line(vcpu, irq_num, level); case KVM_ARM_IRQ_TYPE_PPI: - if (!irqchip_in_kernel(kvm)) + if (irqchip_in_kernel(kvm)) { + int ret = vgic_lazy_init(kvm); + if (ret) + return ret; + } else { return -ENXIO; + } vcpu = kvm_get_vcpu_by_id(kvm, vcpu_id); if (!vcpu) @@ -1524,8 +1534,13 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level, return kvm_vgic_inject_irq(kvm, vcpu, irq_num, level, NULL); case KVM_ARM_IRQ_TYPE_SPI: - if (!irqchip_in_kernel(kvm)) + if (irqchip_in_kernel(kvm)) { + int ret = vgic_lazy_init(kvm); + if (ret) + return ret; + } else { return -ENXIO; + } if (vgic_is_v5(kvm)) { /* Build a GICv5-style IntID here */ @@ -2426,6 +2441,8 @@ static int __init init_subsystems(void) switch (err) { case 0: vgic_present = true; + if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) + kvm_nvhe_sym(hyp_gicv3_nr_lr) = kvm_vgic_global_state.nr_lr; break; case -ENODEV: case -ENXIO: diff --git a/arch/arm64/kvm/at.c b/arch/arm64/kvm/at.c index 889c2c15d7bd..b8ded434c63f 100644 --- a/arch/arm64/kvm/at.c +++ b/arch/arm64/kvm/at.c @@ -136,14 +136,106 @@ static void compute_s1poe(struct kvm_vcpu *vcpu, struct s1_walk_info *wi) wi->e0poe = (wi->regime != TR_EL2) && (val & TCR2_EL1_E0POE); } +#define _has_tgran(__r, __sz) \ + ({ \ + u64 _s1, _mmfr0 = __r; \ + \ + _s1 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \ + TGRAN##__sz, _mmfr0); \ + \ + _s1 != ID_AA64MMFR0_EL1_TGRAN##__sz##_NI; \ + }) + +static bool has_tgran(u64 mmfr0, unsigned int shift) +{ + switch (shift) { + case 12: + return _has_tgran(mmfr0, 4); + case 14: + return _has_tgran(mmfr0, 16); + case 16: + return _has_tgran(mmfr0, 64); + default: + BUG(); + } +} + +static unsigned int tcr_to_tg0_pgshift(u64 tcr) +{ + u64 tg0 = tcr & TCR_TG0_MASK; + + switch (tg0) { + case TCR_TG0_4K: + return 12; + case TCR_TG0_16K: + return 14; + case TCR_TG0_64K: + default: /* IMPDEF: treat any other value as 64k */ + return 16; + } +} + +static unsigned int tcr_to_tg1_pgshift(u64 tcr) +{ + u64 tg1 = tcr & TCR_TG1_MASK; + + switch (tg1) { + case TCR_TG1_4K: + return 12; + case TCR_TG1_16K: + return 14; + case TCR_TG1_64K: + default: /* IMPDEF: treat any other value as 64k */ + return 16; + } +} + +static unsigned int fallback_tgran_shift(u64 mmfr0) +{ + if (has_tgran(mmfr0, PAGE_SHIFT)) + return PAGE_SHIFT; + else if (has_tgran(mmfr0, 12)) + return 12; + else if (has_tgran(mmfr0, 14)) + return 14; + else if (has_tgran(mmfr0, 16)) + return 16; + else /* Should be unreacheable */ + return PAGE_SHIFT; +} + +static unsigned int tcr_tg_pgshift(struct kvm *kvm, u64 tcr, bool upper_range) +{ + u64 mmfr0 = kvm_read_vm_id_reg(kvm, SYS_ID_AA64MMFR0_EL1); + unsigned int shift; + + /* Someone was silly enough to encode TG0/TG1 differently */ + if (upper_range) + shift = tcr_to_tg1_pgshift(tcr); + else + shift = tcr_to_tg0_pgshift(tcr); + + /* + * If TGx is programmed to an unimplemented value (not advertised in + * ID_AA64MMFR0_EL1), we should treat it as if an implemented value is + * written, as per the architecture. Choose an available one while + * prioritizing PAGE_SIZE. + */ + if (!has_tgran(mmfr0, shift)) + return fallback_tgran_shift(mmfr0); + + return shift; +} + static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, struct s1_walk_result *wr, u64 va) { - u64 hcr, sctlr, tcr, tg, ps, ia_bits, ttbr; + u64 hcr, sctlr, tcr, ps, ia_bits, ttbr; unsigned int stride, x; - bool va55, tbi, lva; + bool va55, tbi, lva, upper_range; va55 = va & BIT(55); + upper_range = va55 && wi->regime != TR_EL2; if (vcpu_has_nv(vcpu)) { hcr = __vcpu_sys_reg(vcpu, HCR_EL2); @@ -174,35 +266,12 @@ static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, BUG(); } - /* Someone was silly enough to encode TG0/TG1 differently */ - if (va55 && wi->regime != TR_EL2) { + if (upper_range) wi->txsz = FIELD_GET(TCR_T1SZ_MASK, tcr); - tg = FIELD_GET(TCR_TG1_MASK, tcr); - - switch (tg << TCR_TG1_SHIFT) { - case TCR_TG1_4K: - wi->pgshift = 12; break; - case TCR_TG1_16K: - wi->pgshift = 14; break; - case TCR_TG1_64K: - default: /* IMPDEF: treat any other value as 64k */ - wi->pgshift = 16; break; - } - } else { + else wi->txsz = FIELD_GET(TCR_T0SZ_MASK, tcr); - tg = FIELD_GET(TCR_TG0_MASK, tcr); - - switch (tg << TCR_TG0_SHIFT) { - case TCR_TG0_4K: - wi->pgshift = 12; break; - case TCR_TG0_16K: - wi->pgshift = 14; break; - case TCR_TG0_64K: - default: /* IMPDEF: treat any other value as 64k */ - wi->pgshift = 16; break; - } - } + wi->pgshift = tcr_tg_pgshift(vcpu->kvm, tcr, upper_range); wi->pa52bit = has_52bit_pa(vcpu, wi, tcr); ia_bits = get_ia_size(wi); @@ -423,6 +492,9 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, if (wi->s2) { ret = kvm_walk_nested_s2(vcpu, ipa, &s2_trans); + if (ret == -EAGAIN) + return ret; + if (ret) { fail_s1_walk(wr, (s2_trans.esr & ~ESR_ELx_FSC_LEVEL) | level, @@ -492,15 +564,18 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, /* Block mapping, check the validity of the level */ if (!(desc & BIT(1))) { bool valid_block = false; + bool lpa = kvm_has_feat_enum(vcpu->kvm, ID_AA64MMFR0_EL1, PARANGE, 52); switch (BIT(wi->pgshift)) { case SZ_4K: valid_block = level == 1 || level == 2 || (wi->pa52bit && level == 0); break; case SZ_16K: - case SZ_64K: valid_block = level == 2 || (wi->pa52bit && level == 1); break; + case SZ_64K: + valid_block = level == 2 || (lpa && level == 1); + break; } if (!valid_block) @@ -521,8 +596,12 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, } ret = kvm_swap_s1_desc(vcpu, ipa, desc, new_desc, wi); - if (ret) + if (ret == -EAGAIN) return ret; + if (ret) { + fail_s1_walk(wr, ESR_ELx_FSC_SEA_TTW(level), false); + return ret; + } desc = new_desc; } @@ -1380,7 +1459,7 @@ static u64 __kvm_at_s1e01_fast(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) } } write_sysreg_el1(vcpu_read_sys_reg(vcpu, SCTLR_EL1), SYS_SCTLR); - __load_stage2(mmu, mmu->arch); + __load_stage2(mmu); skip_mmu_switch: /* Temporarily switch back to guest context */ @@ -1553,7 +1632,10 @@ int __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) return 0; } - __kvm_at_s1e01(vcpu, op, vaddr); + ret = __kvm_at_s1e01(vcpu, op, vaddr); + if (ret) + return ret; + par = vcpu_read_sys_reg(vcpu, PAR_EL1); if (par & SYS_PAR_EL1_F) return 0; diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index dba7ced74ca5..e688bc5139c1 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -2631,6 +2631,14 @@ bool triage_sysreg_trap(struct kvm_vcpu *vcpu, int *sr_index) fgtreg = HFGITR2_EL2; break; + case ICH_HFGRTR_GROUP: + fgtreg = is_read ? ICH_HFGRTR_EL2 : ICH_HFGWTR_EL2; + break; + + case ICH_HFGITR_GROUP: + fgtreg = ICH_HFGITR_EL2; + break; + default: /* Something is really wrong, bail out */ WARN_ONCE(1, "Bad FGT group (encoding %08x, config %016llx)\n", @@ -2862,6 +2870,8 @@ static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2, preempt_disable(); + vcpu_set_flag(vcpu, IN_NESTED_EXCEPTION); + /* * We may have an exception or PC update in the EL0/EL1 context. * Commit it before entering EL2. @@ -2884,6 +2894,8 @@ static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2, __kvm_adjust_pc(vcpu); kvm_arch_vcpu_load(vcpu, smp_processor_id()); + vcpu_clear_flag(vcpu, IN_NESTED_EXCEPTION); + preempt_enable(); if (kvm_vcpu_has_pmu(vcpu)) diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 15e17aca1dec..3f6b1e29cd6b 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -29,6 +29,20 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) return; /* + * Avoid needless save/restore of the guest's common + * FPSIMD/SVE/SME regs during transitions between L1/L2. + * + * These transitions only happens in a non-preemptible context + * where the host regs have already been saved and unbound. The + * live registers are either free or owned by the guest. + */ + if (vcpu_get_flag(vcpu, IN_NESTED_ERET) || + vcpu_get_flag(vcpu, IN_NESTED_EXCEPTION)) { + WARN_ON_ONCE(host_owns_fp_regs()); + return; + } + + /* * Ensure that any host FPSIMD/SVE/SME state is saved and unbound such * that the host kernel is responsible for restoring this state upon * return to userspace, and the hyp code doesn't need to save anything. @@ -102,6 +116,18 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) { unsigned long flags; + /* + * See comment in kvm_arch_vcpu_load_fp(). Note that we also rely on + * the guest's max VL to have been set by fpsimd_lazy_switch_to_host() + * so that any intervening kernel-mode SIMD (NEON or otherwise) + * operation sees the full guest state that needs saving. + */ + if (vcpu_get_flag(vcpu, IN_NESTED_ERET) || + vcpu_get_flag(vcpu, IN_NESTED_EXCEPTION)) { + WARN_ON_ONCE(host_owns_fp_regs()); + return; + } + local_irq_save(flags); if (guest_owns_fp_regs()) { diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index d549d550b6e1..18131e395e24 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -141,7 +141,7 @@ static inline void __activate_cptr_traps_vhe(struct kvm_vcpu *vcpu) if (!(SYS_FIELD_GET(CPACR_EL1, ZEN, cptr) & BIT(0))) val &= ~CPACR_EL1_ZEN; - if (kvm_has_feat(vcpu->kvm, ID_AA64MMFR3_EL1, S2POE, IMP)) + if (kvm_has_feat(vcpu->kvm, ID_AA64MMFR3_EL1, S1POE, IMP)) val |= cptr & CPACR_EL1_E0POE; val |= cptr & CPTR_EL2_TCPAC; diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h index 3cbfae0e3dda..29935c7da1de 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -56,6 +56,7 @@ int host_stage2_idmap_locked(phys_addr_t addr, u64 size, enum kvm_pgtable_prot p int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id); int kvm_host_prepare_stage2(void *pgt_pool_base); int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd); +void kvm_guest_destroy_stage2(struct pkvm_hyp_vm *vm); void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt); int hyp_pin_shared_mem(void *from, void *to); @@ -67,7 +68,7 @@ int refill_memcache(struct kvm_hyp_memcache *mc, unsigned long min_pages, static __always_inline void __load_host_stage2(void) { if (static_branch_likely(&kvm_protected_mode_initialized)) - __load_stage2(&host_mmu.arch.mmu, &host_mmu.arch); + __load_stage2(&host_mmu.arch.mmu); else write_sysreg(0, vttbr_el2); } diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 51dad35000f5..d3c69de698f4 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -24,6 +24,9 @@ DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); +/* Number of implemented GICv3 LRs. Used by flush_hyp_vcpu(). */ +unsigned int hyp_gicv3_nr_lr; + void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt); static void __hyp_sve_save_guest(struct kvm_vcpu *vcpu) @@ -130,10 +133,18 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt; + /* __hyp_running_vcpu must be NULL in a guest context. */ + hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu = NULL; + hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2; - hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI | HCR_TWE); + /* + * HCR_EL2.VSE is host-owned (a pending virtual SError to inject), not a + * trap-control bit, so it must flow to the hyp vCPU alongside TWI/TWE + * for the vSError to be delivered. sync_hyp_vcpu() reflects it back. + */ + hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI | HCR_TWE | HCR_VSE); hyp_vcpu->vcpu.arch.hcr_el2 |= READ_ONCE(host_vcpu->arch.hcr_el2) & - (HCR_TWI | HCR_TWE); + (HCR_TWI | HCR_TWE | HCR_VSE); hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; @@ -141,6 +152,12 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3 = host_vcpu->arch.vgic_cpu.vgic_v3; + /* Bound used_lrs by the number of implemented list registers. */ + hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs = + min_t(unsigned int, + hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs, + hyp_gicv3_nr_lr); + hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid; } diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 25f04629014e..4e329e39a695 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -217,7 +217,6 @@ static void *guest_s2_zalloc_page(void *mc) memset(addr, 0, PAGE_SIZE); p = hyp_virt_to_page(addr); p->refcount = 1; - p->order = 0; return addr; } @@ -306,23 +305,27 @@ int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd) return 0; } +void kvm_guest_destroy_stage2(struct pkvm_hyp_vm *vm) +{ + guest_lock_component(vm); + kvm_pgtable_stage2_destroy(&vm->pgt); + vm->kvm.arch.mmu.pgd_phys = 0ULL; + guest_unlock_component(vm); +} + void reclaim_pgtable_pages(struct pkvm_hyp_vm *vm, struct kvm_hyp_memcache *mc) { struct hyp_page *page; void *addr; /* Dump all pgtable pages in the hyp_pool */ - guest_lock_component(vm); - kvm_pgtable_stage2_destroy(&vm->pgt); - vm->kvm.arch.mmu.pgd_phys = 0ULL; - guest_unlock_component(vm); + kvm_guest_destroy_stage2(vm); /* Drain the hyp_pool into the memcache */ addr = hyp_alloc_pages(&vm->pool, 0); while (addr) { page = hyp_virt_to_page(addr); page->refcount = 0; - page->order = 0; push_hyp_memcache(mc, addr, hyp_virt_to_phys); WARN_ON(__pkvm_hyp_donate_host(hyp_virt_to_pfn(addr), 1)); addr = hyp_alloc_pages(&vm->pool, 0); @@ -352,7 +355,7 @@ int __pkvm_prot_finalize(void) kvm_flush_dcache_to_poc(params, sizeof(*params)); write_sysreg_hcr(params->hcr_el2); - __load_stage2(&host_mmu.arch.mmu, &host_mmu.arch); + __load_stage2(&host_mmu.arch.mmu); /* * Make sure to have an ISB before the TLB maintenance below but only @@ -851,6 +854,16 @@ static int __hyp_check_page_state_range(phys_addr_t phys, u64 size, enum pkvm_pa return 0; } +static int __hyp_check_page_count_range(phys_addr_t phys, u64 size) +{ + for_each_hyp_page(page, phys, size) { + if (page->refcount) + return -EBUSY; + } + + return 0; +} + static bool guest_pte_is_poisoned(kvm_pte_t pte) { if (kvm_pte_valid(pte)) @@ -1049,7 +1062,6 @@ unlock: int __pkvm_host_unshare_hyp(u64 pfn) { u64 phys = hyp_pfn_to_phys(pfn); - u64 virt = (u64)__hyp_va(phys); u64 size = PAGE_SIZE; int ret; @@ -1062,10 +1074,9 @@ int __pkvm_host_unshare_hyp(u64 pfn) ret = __hyp_check_page_state_range(phys, size, PKVM_PAGE_SHARED_BORROWED); if (ret) goto unlock; - if (hyp_page_count((void *)virt)) { - ret = -EBUSY; + ret = __hyp_check_page_count_range(phys, size); + if (ret) goto unlock; - } __hyp_set_page_state_range(phys, size, PKVM_NOPAGE); WARN_ON(__host_set_page_state_range(phys, size, PKVM_PAGE_OWNED)); @@ -1128,6 +1139,10 @@ int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages) if (ret) goto unlock; + ret = __hyp_check_page_count_range(phys, size); + if (ret) + goto unlock; + __hyp_set_page_state_range(phys, size, PKVM_NOPAGE); WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, virt, size) != size); WARN_ON(host_stage2_set_owner_locked(phys, size, PKVM_ID_HOST)); diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c index a1eb27a1a747..57f86aa0f82f 100644 --- a/arch/arm64/kvm/hyp/nvhe/page_alloc.c +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -94,13 +94,22 @@ static void __hyp_attach_page(struct hyp_pool *pool, struct hyp_page *p) { phys_addr_t phys = hyp_page_to_phys(p); - u8 order = p->order; struct hyp_page *buddy; + bool coalesce = true; + u8 order = p->order; - memset(hyp_page_to_virt(p), 0, PAGE_SIZE << p->order); + /* + * 'external' pages are never coalesced and their ->order field + * untrusted as they bypass hyp_pool_init(). Enforce order-0. + */ + if (phys < pool->range_start || phys >= pool->range_end) { + order = 0; + coalesce = false; + } + + memset(hyp_page_to_virt(p), 0, PAGE_SIZE << order); - /* Skip coalescing for 'external' pages being freed into the pool. */ - if (phys < pool->range_start || phys >= pool->range_end) + if (!coalesce) goto insert; /* @@ -237,8 +246,10 @@ int hyp_pool_init(struct hyp_pool *pool, u64 pfn, unsigned int nr_pages, /* Init the vmemmap portion */ p = hyp_phys_to_page(phys); - for (i = 0; i < nr_pages; i++) + for (i = 0; i < nr_pages; i++) { hyp_set_page_refcounted(&p[i]); + p[i].order = 0; + } /* Attach the unused pages to the buddy tree */ for (i = reserved_pages; i < nr_pages; i++) diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index eb1c10120f9f..3b2c4fbc34d8 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -853,10 +853,12 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva, /* Must be called last since this publishes the VM. */ ret = insert_vm_table_entry(handle, hyp_vm); if (ret) - goto err_remove_mappings; + goto err_destroy_stage2; return 0; +err_destroy_stage2: + kvm_guest_destroy_stage2(hyp_vm); err_remove_mappings: unmap_donated_memory(hyp_vm, vm_size); unmap_donated_memory(pgd, pgd_size); diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 8d1df3d33595..7318e3e6a5f3 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -315,7 +315,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) __sysreg_restore_state_nvhe(guest_ctxt); mmu = kern_hyp_va(vcpu->arch.hw_mmu); - __load_stage2(mmu, kern_hyp_va(mmu->arch)); + __load_stage2(mmu); __activate_traps(vcpu); __hyp_vgic_restore_state(vcpu); diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index b29140995d48..fdb90483340c 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -110,7 +110,7 @@ static void enter_vmid_context(struct kvm_s2_mmu *mmu, if (vcpu) __load_host_stage2(); else - __load_stage2(mmu, kern_hyp_va(mmu->arch)); + __load_stage2(mmu); asm(ALTERNATIVE("isb", "nop", ARM64_WORKAROUND_SPECULATIVE_AT)); } @@ -128,7 +128,7 @@ static void exit_vmid_context(struct tlb_inv_context *cxt) return; if (vcpu) - __load_stage2(mmu, kern_hyp_va(mmu->arch)); + __load_stage2(mmu); else __load_host_stage2(); diff --git a/arch/arm64/kvm/hyp/vgic-v5-sr.c b/arch/arm64/kvm/hyp/vgic-v5-sr.c index 47e6bcd43702..6d69dfe89a96 100644 --- a/arch/arm64/kvm/hyp/vgic-v5-sr.c +++ b/arch/arm64/kvm/hyp/vgic-v5-sr.c @@ -30,10 +30,9 @@ void __vgic_v5_save_ppi_state(struct vgic_v5_cpu_if *cpu_if) { /* * The following code assumes that the bitmap storage that we have for - * PPIs is either 64 (architected PPIs, only) or 128 bits (architected & - * impdef PPIs). + * PPIs is either 64 (architected PPIs, only). */ - BUILD_BUG_ON(VGIC_V5_NR_PRIVATE_IRQS % 64); + BUILD_BUG_ON(VGIC_V5_NR_PRIVATE_IRQS != 64); bitmap_write(host_data_ptr(vgic_v5_ppi_state)->activer_exit, read_sysreg_s(SYS_ICH_PPI_ACTIVER0_EL2), 0, 64); @@ -49,22 +48,6 @@ void __vgic_v5_save_ppi_state(struct vgic_v5_cpu_if *cpu_if) cpu_if->vgic_ppi_priorityr[6] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR6_EL2); cpu_if->vgic_ppi_priorityr[7] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR7_EL2); - if (VGIC_V5_NR_PRIVATE_IRQS == 128) { - bitmap_write(host_data_ptr(vgic_v5_ppi_state)->activer_exit, - read_sysreg_s(SYS_ICH_PPI_ACTIVER1_EL2), 64, 64); - bitmap_write(host_data_ptr(vgic_v5_ppi_state)->pendr, - read_sysreg_s(SYS_ICH_PPI_PENDR1_EL2), 64, 64); - - cpu_if->vgic_ppi_priorityr[8] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR8_EL2); - cpu_if->vgic_ppi_priorityr[9] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR9_EL2); - cpu_if->vgic_ppi_priorityr[10] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR10_EL2); - cpu_if->vgic_ppi_priorityr[11] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR11_EL2); - cpu_if->vgic_ppi_priorityr[12] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR12_EL2); - cpu_if->vgic_ppi_priorityr[13] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR13_EL2); - cpu_if->vgic_ppi_priorityr[14] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR14_EL2); - cpu_if->vgic_ppi_priorityr[15] = read_sysreg_s(SYS_ICH_PPI_PRIORITYR15_EL2); - } - /* Now that we are done, disable DVI */ write_sysreg_s(0, SYS_ICH_PPI_DVIR0_EL2); write_sysreg_s(0, SYS_ICH_PPI_DVIR1_EL2); @@ -74,9 +57,6 @@ void __vgic_v5_restore_ppi_state(struct vgic_v5_cpu_if *cpu_if) { DECLARE_BITMAP(pendr, VGIC_V5_NR_PRIVATE_IRQS); - /* We assume 64 or 128 PPIs - see above comment */ - BUILD_BUG_ON(VGIC_V5_NR_PRIVATE_IRQS % 64); - /* Enable DVI so that the guest's interrupt config takes over */ write_sysreg_s(bitmap_read(cpu_if->vgic_ppi_dvir, 0, 64), SYS_ICH_PPI_DVIR0_EL2); @@ -108,50 +88,20 @@ void __vgic_v5_restore_ppi_state(struct vgic_v5_cpu_if *cpu_if) write_sysreg_s(cpu_if->vgic_ppi_priorityr[7], SYS_ICH_PPI_PRIORITYR7_EL2); - if (VGIC_V5_NR_PRIVATE_IRQS == 128) { - /* Enable DVI so that the guest's interrupt config takes over */ - write_sysreg_s(bitmap_read(cpu_if->vgic_ppi_dvir, 64, 64), - SYS_ICH_PPI_DVIR1_EL2); - - write_sysreg_s(bitmap_read(cpu_if->vgic_ppi_activer, 64, 64), - SYS_ICH_PPI_ACTIVER1_EL2); - write_sysreg_s(bitmap_read(cpu_if->vgic_ppi_enabler, 64, 64), - SYS_ICH_PPI_ENABLER1_EL2); - write_sysreg_s(bitmap_read(pendr, 64, 64), - SYS_ICH_PPI_PENDR1_EL2); - - write_sysreg_s(cpu_if->vgic_ppi_priorityr[8], - SYS_ICH_PPI_PRIORITYR8_EL2); - write_sysreg_s(cpu_if->vgic_ppi_priorityr[9], - SYS_ICH_PPI_PRIORITYR9_EL2); - write_sysreg_s(cpu_if->vgic_ppi_priorityr[10], - SYS_ICH_PPI_PRIORITYR10_EL2); - write_sysreg_s(cpu_if->vgic_ppi_priorityr[11], - SYS_ICH_PPI_PRIORITYR11_EL2); - write_sysreg_s(cpu_if->vgic_ppi_priorityr[12], - SYS_ICH_PPI_PRIORITYR12_EL2); - write_sysreg_s(cpu_if->vgic_ppi_priorityr[13], - SYS_ICH_PPI_PRIORITYR13_EL2); - write_sysreg_s(cpu_if->vgic_ppi_priorityr[14], - SYS_ICH_PPI_PRIORITYR14_EL2); - write_sysreg_s(cpu_if->vgic_ppi_priorityr[15], - SYS_ICH_PPI_PRIORITYR15_EL2); - } else { - write_sysreg_s(0, SYS_ICH_PPI_DVIR1_EL2); - - write_sysreg_s(0, SYS_ICH_PPI_ACTIVER1_EL2); - write_sysreg_s(0, SYS_ICH_PPI_ENABLER1_EL2); - write_sysreg_s(0, SYS_ICH_PPI_PENDR1_EL2); - - write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR8_EL2); - write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR9_EL2); - write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR10_EL2); - write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR11_EL2); - write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR12_EL2); - write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR13_EL2); - write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR14_EL2); - write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR15_EL2); - } + write_sysreg_s(0, SYS_ICH_PPI_DVIR1_EL2); + + write_sysreg_s(0, SYS_ICH_PPI_ACTIVER1_EL2); + write_sysreg_s(0, SYS_ICH_PPI_ENABLER1_EL2); + write_sysreg_s(0, SYS_ICH_PPI_PENDR1_EL2); + + write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR8_EL2); + write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR9_EL2); + write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR10_EL2); + write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR11_EL2); + write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR12_EL2); + write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR13_EL2); + write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR14_EL2); + write_sysreg_s(0, SYS_ICH_PPI_PRIORITYR15_EL2); } void __vgic_v5_save_state(struct vgic_v5_cpu_if *cpu_if) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 1e8995add14f..bbe9cebd3d9d 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -219,7 +219,7 @@ void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu) __vcpu_load_switch_sysregs(vcpu); __vcpu_load_activate_traps(vcpu); - __load_stage2(vcpu->arch.hw_mmu, vcpu->arch.hw_mmu->arch); + __load_stage2(vcpu->arch.hw_mmu); } void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index f7b9dfe3f3a5..c386d9f1c101 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -60,7 +60,7 @@ static void enter_vmid_context(struct kvm_s2_mmu *mmu, * place before clearing TGE. __load_stage2() already * has an ISB in order to deal with this. */ - __load_stage2(mmu, mmu->arch); + __load_stage2(mmu); val = read_sysreg(hcr_el2); val &= ~HCR_TGE; write_sysreg_hcr(val); @@ -78,7 +78,7 @@ static void exit_vmid_context(struct tlb_inv_context *cxt) /* ... and the stage-2 MMU context that we switched away from */ if (cxt->mmu) - __load_stage2(cxt->mmu, cxt->mmu->arch); + __load_stage2(cxt->mmu); if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) { /* Restore the registers to what they were */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 6e95514d5a76..6c941aaa10c6 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -501,6 +501,10 @@ static int share_pfn_hyp(u64 pfn) rb_link_node(&this->node, parent, node); rb_insert_color(&this->node, &hyp_shared_pfns); ret = kvm_call_hyp_nvhe(__pkvm_host_share_hyp, pfn); + if (ret) { + rb_erase(&this->node, &hyp_shared_pfns); + kfree(this); + } unlock: mutex_unlock(&hyp_shared_pfns_lock); @@ -520,13 +524,17 @@ static int unshare_pfn_hyp(u64 pfn) goto unlock; } - this->count--; - if (this->count) + if (this->count > 1) { + this->count--; + goto unlock; + } + + ret = kvm_call_hyp_nvhe(__pkvm_host_unshare_hyp, pfn); + if (ret) goto unlock; rb_erase(&this->node, &hyp_shared_pfns); kfree(this); - ret = kvm_call_hyp_nvhe(__pkvm_host_unshare_hyp, pfn); unlock: mutex_unlock(&hyp_shared_pfns_lock); @@ -536,8 +544,8 @@ unlock: int kvm_share_hyp(void *from, void *to) { phys_addr_t start, end, cur; + int ret = 0; u64 pfn; - int ret; if (is_kernel_in_hyp_mode()) return 0; @@ -559,10 +567,24 @@ int kvm_share_hyp(void *from, void *to) pfn = __phys_to_pfn(cur); ret = share_pfn_hyp(pfn); if (ret) - return ret; + break; } - return 0; + if (!ret) + return 0; + + /* + * Roll back the pages shared by this call. A failed unshare leaks + * the page (it stays shared with the hypervisor and is no longer + * reusable for pKVM) but breaks no isolation guarantee, so warn and + * continue. Not expected in practice. + */ + for (end = cur, cur = start; cur < end; cur += PAGE_SIZE) { + pfn = __phys_to_pfn(cur); + WARN_ON(unshare_pfn_hyp(pfn)); + } + + return ret; } void kvm_unshare_hyp(void *from, void *to) @@ -577,6 +599,11 @@ void kvm_unshare_hyp(void *from, void *to) end = PAGE_ALIGN(__pa(to)); for (cur = start; cur < end; cur += PAGE_SIZE) { pfn = __phys_to_pfn(cur); + /* + * A failed unshare leaks the page: it stays shared with the + * hypervisor and is no longer reusable for pKVM. No isolation + * guarantee is broken, and this is not expected in practice. + */ WARN_ON(unshare_pfn_hyp(pfn)); } } diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 6f7bc9a9992e..fb54f6dad995 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -359,8 +359,13 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa, if (new_desc != desc) { ret = swap_guest_s2_desc(vcpu, paddr, desc, new_desc, wi); - if (ret) + if (ret == -EAGAIN) return ret; + if (ret) { + out->esr = ESR_ELx_FSC_SEA_TTW(level); + out->desc = desc; + return 1; + } desc = new_desc; } @@ -385,32 +390,104 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa, return 0; } -static void vtcr_to_walk_info(u64 vtcr, struct s2_walk_info *wi) +#define _has_tgran_2(__r, __sz) \ + ({ \ + u64 _s1, _s2, _mmfr0 = __r; \ + \ + _s2 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \ + TGRAN##__sz##_2, _mmfr0); \ + \ + _s1 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \ + TGRAN##__sz, _mmfr0); \ + \ + ((_s2 != ID_AA64MMFR0_EL1_TGRAN##__sz##_2_NI && \ + _s2 != ID_AA64MMFR0_EL1_TGRAN##__sz##_2_TGRAN##__sz) || \ + (_s2 == ID_AA64MMFR0_EL1_TGRAN##__sz##_2_TGRAN##__sz && \ + _s1 != ID_AA64MMFR0_EL1_TGRAN##__sz##_NI)); \ + }) + +static bool has_tgran_2(u64 mmfr0, unsigned int shift) { - wi->t0sz = vtcr & TCR_EL2_T0SZ_MASK; + switch (shift) { + case 12: + return _has_tgran_2(mmfr0, 4); + case 14: + return _has_tgran_2(mmfr0, 16); + case 16: + return _has_tgran_2(mmfr0, 64); + default: + BUG(); + } +} - switch (FIELD_GET(VTCR_EL2_TG0_MASK, vtcr)) { +static unsigned int fallback_tgran2_shift(u64 mmfr0) +{ + if (has_tgran_2(mmfr0, PAGE_SHIFT)) + return PAGE_SHIFT; + else if (has_tgran_2(mmfr0, 12)) + return 12; + else if (has_tgran_2(mmfr0, 14)) + return 14; + else if (has_tgran_2(mmfr0, 16)) + return 16; + else + return PAGE_SHIFT; +} + +static unsigned int vtcr_to_tg0_pgshift(struct kvm *kvm, u64 vtcr) +{ + u64 tg0 = FIELD_GET(VTCR_EL2_TG0_MASK, vtcr); + u64 mmfr0 = kvm_read_vm_id_reg(kvm, SYS_ID_AA64MMFR0_EL1); + unsigned int shift; + + switch (tg0) { case VTCR_EL2_TG0_4K: - wi->pgshift = 12; break; + shift = 12; + break; case VTCR_EL2_TG0_16K: - wi->pgshift = 14; break; + shift = 14; + break; case VTCR_EL2_TG0_64K: - default: /* IMPDEF: treat any other value as 64k */ - wi->pgshift = 16; break; + /* IMPDEF: treat any other value as 64k, subject to fallback */ + default: + shift = 16; } + /* + * If TGx is programmed to an unimplemented value (not advertised in + * ID_AA64MMFR0_EL1), we should treat it as if an implemented value is + * written, as per the architecture. Choose an available one while + * prioritizing PAGE_SIZE. + */ + if (!has_tgran_2(mmfr0, shift)) + return fallback_tgran2_shift(mmfr0); + + return shift; +} + +static size_t vtcr_to_tg0_pgsize(struct kvm *kvm, u64 vtcr) +{ + return BIT(vtcr_to_tg0_pgshift(kvm, vtcr)); +} + +static void setup_s2_walk(struct kvm_vcpu *vcpu, struct s2_walk_info *wi) +{ + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + + wi->baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + wi->t0sz = vtcr & VTCR_EL2_T0SZ_MASK; + wi->pgshift = vtcr_to_tg0_pgshift(vcpu->kvm, vtcr); wi->sl = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr); /* Global limit for now, should eventually be per-VM */ wi->max_oa_bits = min(get_kvm_ipa_limit(), ps_to_output_size(FIELD_GET(VTCR_EL2_PS_MASK, vtcr), false)); - wi->ha = vtcr & VTCR_EL2_HA; + wi->be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_ELx_EE; } int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, struct kvm_s2_trans *result) { - u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); struct s2_walk_info wi; int ret; @@ -419,11 +496,7 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, if (!vcpu_has_nv(vcpu)) return 0; - wi.baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); - - vtcr_to_walk_info(vtcr, &wi); - - wi.be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_ELx_EE; + setup_s2_walk(vcpu, &wi); ret = walk_nested_s2_pgd(vcpu, gipa, &wi, result); if (ret) @@ -519,20 +592,21 @@ static u8 pgshift_level_to_ttl(u16 shift, u8 level) */ static u8 get_guest_mapping_ttl(struct kvm_s2_mmu *mmu, u64 addr) { - u64 tmp, sz = 0, vtcr = mmu->tlb_vtcr; + size_t tg0_size = vtcr_to_tg0_pgsize(kvm_s2_mmu_to_kvm(mmu), mmu->tlb_vtcr); + u64 tmp, sz = 0; kvm_pte_t pte; u8 ttl, level; lockdep_assert_held_write(&kvm_s2_mmu_to_kvm(mmu)->mmu_lock); - switch (FIELD_GET(VTCR_EL2_TG0_MASK, vtcr)) { - case VTCR_EL2_TG0_4K: + switch (tg0_size) { + case SZ_4K: ttl = (TLBI_TTL_TG_4K << 2); break; - case VTCR_EL2_TG0_16K: + case SZ_16K: ttl = (TLBI_TTL_TG_16K << 2); break; - case VTCR_EL2_TG0_64K: + case SZ_64K: default: /* IMPDEF: treat any other value as 64k */ ttl = (TLBI_TTL_TG_64K << 2); break; @@ -542,19 +616,19 @@ static u8 get_guest_mapping_ttl(struct kvm_s2_mmu *mmu, u64 addr) again: /* Iteratively compute the block sizes for a particular granule size */ - switch (FIELD_GET(VTCR_EL2_TG0_MASK, vtcr)) { - case VTCR_EL2_TG0_4K: + switch (tg0_size) { + case SZ_4K: if (sz < SZ_4K) sz = SZ_4K; else if (sz < SZ_2M) sz = SZ_2M; else if (sz < SZ_1G) sz = SZ_1G; else sz = 0; break; - case VTCR_EL2_TG0_16K: + case SZ_16K: if (sz < SZ_16K) sz = SZ_16K; else if (sz < SZ_32M) sz = SZ_32M; else sz = 0; break; - case VTCR_EL2_TG0_64K: + case SZ_64K: default: /* IMPDEF: treat any other value as 64k */ if (sz < SZ_64K) sz = SZ_64K; else if (sz < SZ_512M) sz = SZ_512M; @@ -605,14 +679,14 @@ unsigned long compute_tlb_inval_range(struct kvm_s2_mmu *mmu, u64 val) if (!max_size) { /* Compute the maximum extent of the invalidation */ - switch (FIELD_GET(VTCR_EL2_TG0_MASK, mmu->tlb_vtcr)) { - case VTCR_EL2_TG0_4K: + switch (vtcr_to_tg0_pgsize(kvm, mmu->tlb_vtcr)) { + case SZ_4K: max_size = SZ_1G; break; - case VTCR_EL2_TG0_16K: + case SZ_16K: max_size = SZ_32M; break; - case VTCR_EL2_TG0_64K: + case SZ_64K: default: /* IMPDEF: treat any other value as 64k */ /* * No, we do not support 52bit IPA in nested yet. Once @@ -804,18 +878,24 @@ void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu) } } +static void this_cpu_reset_vncr_fixmap(struct kvm_vcpu *vcpu) +{ + if (!host_data_test_flag(L1_VNCR_MAPPED)) + return; + + BUG_ON(vcpu->arch.vncr_tlb->cpu != smp_processor_id()); + BUG_ON(is_hyp_ctxt(vcpu)); + + clear_fixmap(vncr_fixmap(vcpu->arch.vncr_tlb->cpu)); + vcpu->arch.vncr_tlb->cpu = -1; + host_data_clear_flag(L1_VNCR_MAPPED); + atomic_dec(&vcpu->kvm->arch.vncr_map_count); +} + void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) { /* Unconditionally drop the VNCR mapping if we have one */ - if (host_data_test_flag(L1_VNCR_MAPPED)) { - BUG_ON(vcpu->arch.vncr_tlb->cpu != smp_processor_id()); - BUG_ON(is_hyp_ctxt(vcpu)); - - clear_fixmap(vncr_fixmap(vcpu->arch.vncr_tlb->cpu)); - vcpu->arch.vncr_tlb->cpu = -1; - host_data_clear_flag(L1_VNCR_MAPPED); - atomic_dec(&vcpu->kvm->arch.vncr_map_count); - } + this_cpu_reset_vncr_fixmap(vcpu); /* * Keep a reference on the associated stage-2 MMU if the vCPU is @@ -904,9 +984,21 @@ static void invalidate_vncr(struct vncr_tlb *vt) clear_fixmap(vncr_fixmap(vt->cpu)); } +/* + * VNCR TLB invalidation occurs from MMU notifiers or TLBI instructions, and + * either can race against a vcpu not being onlined yet (no pseudo-TLB + * allocated). Similarly, the TLB might be invalid. Skip those, as they + * obviously don't participate in the invalidation at this stage. + */ +#define kvm_for_each_vncr_tlb(idx, vcpup, tlbp, kvm) \ + kvm_for_each_vcpu(idx, vcpup, kvm) \ + if (((tlbp) = vcpup->arch.vncr_tlb) && \ + (tlbp)->valid) + static void kvm_invalidate_vncr_ipa(struct kvm *kvm, u64 start, u64 end) { struct kvm_vcpu *vcpu; + struct vncr_tlb *vt; unsigned long i; lockdep_assert_held_write(&kvm->mmu_lock); @@ -914,24 +1006,9 @@ static void kvm_invalidate_vncr_ipa(struct kvm *kvm, u64 start, u64 end) if (!kvm_has_feat(kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY)) return; - kvm_for_each_vcpu(i, vcpu, kvm) { - struct vncr_tlb *vt = vcpu->arch.vncr_tlb; + kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) { u64 ipa_start, ipa_end, ipa_size; - /* - * Careful here: We end-up here from an MMU notifier, - * and this can race against a vcpu not being onlined - * yet, without the pseudo-TLB being allocated. - * - * Skip those, as they obviously don't participate in - * the invalidation at this stage. - */ - if (!vt) - continue; - - if (!vt->valid) - continue; - ipa_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift, vt->wr.level)); ipa_start = vt->wr.pa & ~(ipa_size - 1); @@ -961,17 +1038,14 @@ static void invalidate_vncr_va(struct kvm *kvm, struct s1e2_tlbi_scope *scope) { struct kvm_vcpu *vcpu; + struct vncr_tlb *vt; unsigned long i; lockdep_assert_held_write(&kvm->mmu_lock); - kvm_for_each_vcpu(i, vcpu, kvm) { - struct vncr_tlb *vt = vcpu->arch.vncr_tlb; + kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) { u64 va_start, va_end, va_size; - if (!vt->valid) - continue; - va_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift, vt->wr.level)); va_start = vt->gva & ~(va_size - 1); @@ -1255,8 +1329,20 @@ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu) if (!kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY)) return 0; - vcpu->arch.vncr_tlb = kzalloc_obj(*vcpu->arch.vncr_tlb, - GFP_KERNEL_ACCOUNT); + if (!vcpu->arch.vncr_tlb) { + struct vncr_tlb *vt = kzalloc_obj(*vcpu->arch.vncr_tlb, + GFP_KERNEL_ACCOUNT); + + /* + * Taking the lock on assignment ensures that the TLB is + * seen as initialised when following the pointer (release + * semantics of the unlock), and avoids having acquires on + * each user which already take the lock. + */ + scoped_guard(write_lock, &vcpu->kvm->mmu_lock) + vcpu->arch.vncr_tlb = vt; + } + if (!vcpu->arch.vncr_tlb) return -ENOMEM; @@ -1289,7 +1375,8 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) * We also prepare the next walk wilst we're at it. */ scoped_guard(write_lock, &vcpu->kvm->mmu_lock) { - invalidate_vncr(vt); + this_cpu_reset_vncr_fixmap(vcpu); + vt->valid = false; vt->wi = (struct s1_walk_info) { .regime = TR_EL20, @@ -1333,8 +1420,10 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) } scoped_guard(write_lock, &vcpu->kvm->mmu_lock) { - if (mmu_invalidate_retry(vcpu->kvm, mmu_seq)) + if (mmu_invalidate_retry(vcpu->kvm, mmu_seq)) { + kvm_release_faultin_page(vcpu->kvm, page, true, false); return -EAGAIN; + } vt->gva = va; vt->hpa = pfn << PAGE_SHIFT; @@ -1505,21 +1594,6 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu) } } -#define has_tgran_2(__r, __sz) \ - ({ \ - u64 _s1, _s2, _mmfr0 = __r; \ - \ - _s2 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \ - TGRAN##__sz##_2, _mmfr0); \ - \ - _s1 = SYS_FIELD_GET(ID_AA64MMFR0_EL1, \ - TGRAN##__sz, _mmfr0); \ - \ - ((_s2 != ID_AA64MMFR0_EL1_TGRAN##__sz##_2_NI && \ - _s2 != ID_AA64MMFR0_EL1_TGRAN##__sz##_2_TGRAN##__sz) || \ - (_s2 == ID_AA64MMFR0_EL1_TGRAN##__sz##_2_TGRAN##__sz && \ - _s1 != ID_AA64MMFR0_EL1_TGRAN##__sz##_NI)); \ - }) /* * Our emulated CPU doesn't support all the possible features. For the * sake of simplicity (and probably mental sanity), wipe out a number @@ -1606,15 +1680,15 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val) */ switch (PAGE_SIZE) { case SZ_4K: - if (has_tgran_2(orig_val, 4)) + if (_has_tgran_2(orig_val, 4)) val |= SYS_FIELD_PREP_ENUM(ID_AA64MMFR0_EL1, TGRAN4_2, IMP); fallthrough; case SZ_16K: - if (has_tgran_2(orig_val, 16)) + if (_has_tgran_2(orig_val, 16)) val |= SYS_FIELD_PREP_ENUM(ID_AA64MMFR0_EL1, TGRAN16_2, IMP); fallthrough; case SZ_64K: - if (has_tgran_2(orig_val, 64)) + if (_has_tgran_2(orig_val, 64)) val |= SYS_FIELD_PREP_ENUM(ID_AA64MMFR0_EL1, TGRAN64_2, IMP); break; } diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c index c816db5d6761..98305bbfc095 100644 --- a/arch/arm64/kvm/pmu-emul.c +++ b/arch/arm64/kvm/pmu-emul.c @@ -396,44 +396,31 @@ static bool kvm_pmu_overflow_status(struct kvm_vcpu *vcpu) static void kvm_pmu_update_state(struct kvm_vcpu *vcpu) { struct kvm_pmu *pmu = &vcpu->arch.pmu; - bool overflow; - overflow = kvm_pmu_overflow_status(vcpu); - if (pmu->irq_level == overflow) + if (unlikely(!irqchip_in_kernel(vcpu->kvm))) return; - pmu->irq_level = overflow; - - if (likely(irqchip_in_kernel(vcpu->kvm))) { - int ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu, - pmu->irq_num, overflow, pmu); - WARN_ON(ret); - } + WARN_ON(kvm_vgic_inject_irq(vcpu->kvm, vcpu, pmu->irq_num, + kvm_pmu_overflow_status(vcpu), pmu)); } bool kvm_pmu_should_notify_user(struct kvm_vcpu *vcpu) { - struct kvm_pmu *pmu = &vcpu->arch.pmu; struct kvm_sync_regs *sregs = &vcpu->run->s.regs; bool run_level = sregs->device_irq_level & KVM_ARM_DEV_PMU; - if (likely(irqchip_in_kernel(vcpu->kvm))) - return false; - - return pmu->irq_level != run_level; + return kvm_pmu_overflow_status(vcpu) != run_level; } /* * Reflect the PMU overflow interrupt output level into the kvm_run structure */ -void kvm_pmu_update_run(struct kvm_vcpu *vcpu) +bool kvm_pmu_update_run(struct kvm_vcpu *vcpu) { - struct kvm_sync_regs *regs = &vcpu->run->s.regs; - - /* Populate the timer bitmap for user space */ - regs->device_irq_level &= ~KVM_ARM_DEV_PMU; - if (vcpu->arch.pmu.irq_level) - regs->device_irq_level |= KVM_ARM_DEV_PMU; + bool update = kvm_pmu_should_notify_user(vcpu); + if (update) + vcpu->run->s.regs.device_irq_level ^= KVM_ARM_DEV_PMU; + return update; } /** diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index fa5c93c7a135..5d5c579d4579 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -724,6 +724,7 @@ static bool access_gicv5_ppi_enabler(struct kvm_vcpu *vcpu, { unsigned long *mask = vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask; struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + unsigned long reg = p->regval; int i; /* We never expect to get here with a read! */ @@ -731,27 +732,23 @@ static bool access_gicv5_ppi_enabler(struct kvm_vcpu *vcpu, return undef_access(vcpu, p, r); /* - * If we're only handling architected PPIs and the guest writes to the - * enable for the non-architected PPIs, we just return as there's - * nothing to do at all. We don't even allocate the storage for them in - * this case. + * As we're only handling architected PPIs, the guest writes to the + * enable for the non-architected PPIs just return as there's + * nothing to do at all. We don't even allocate the storage for them. */ - if (VGIC_V5_NR_PRIVATE_IRQS == 64 && p->Op2 % 2) + if (p->Op2 % 2) return true; /* - * Merge the raw guest write into out bitmap at an offset of either 0 or - * 64, then and it with our PPI mask. + * Merge the raw guest write into out bitmap, anded with our PPI mask. */ - bitmap_write(cpu_if->vgic_ppi_enabler, p->regval, 64 * (p->Op2 % 2), 64); - bitmap_and(cpu_if->vgic_ppi_enabler, cpu_if->vgic_ppi_enabler, mask, - VGIC_V5_NR_PRIVATE_IRQS); + bitmap_and(cpu_if->vgic_ppi_enabler, ®, mask, VGIC_V5_NR_PRIVATE_IRQS); /* * Sync the change in enable states to the vgic_irqs. We consider all * PPIs as we don't expose many to the guest. */ - for_each_set_bit(i, mask, VGIC_V5_NR_PRIVATE_IRQS) { + for_each_visible_v5_ppi(i, vcpu->kvm) { u32 intid = vgic_v5_make_ppi(i); struct vgic_irq *irq; @@ -4212,6 +4209,7 @@ static struct sys_reg_desc sys_insn_descs[] = { SYS_INSN(AT_S1E0W, handle_at_s1e01), SYS_INSN(AT_S1E1RP, handle_at_s1e01), SYS_INSN(AT_S1E1WP, handle_at_s1e01), + SYS_INSN(AT_S1E1A, handle_at_s1e01), { SYS_DESC(SYS_DC_CSW), access_dcsw }, { SYS_DESC(SYS_DC_CGSW), access_dcgsw }, diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index 933983bb2005..907057881b26 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -271,18 +271,12 @@ int kvm_vgic_vcpu_nv_init(struct kvm_vcpu *vcpu) return ret; } -static void vgic_allocate_private_irq(struct kvm_vcpu *vcpu, int i, u32 type) +static void vgic_setup_private_irq(struct kvm_vcpu *vcpu, struct vgic_irq *irq, + u32 type) { - struct vgic_irq *irq = &vcpu->arch.vgic_cpu.private_irqs[i]; + irq->intid = irq - &vcpu->arch.vgic_cpu.private_irqs[0]; - INIT_LIST_HEAD(&irq->ap_list); - raw_spin_lock_init(&irq->irq_lock); - irq->vcpu = NULL; - irq->target_vcpu = vcpu; - refcount_set(&irq->refcount, 0); - - irq->intid = i; - if (vgic_irq_is_sgi(i)) { + if (vgic_irq_is_sgi(irq->intid)) { /* SGIs */ irq->enabled = 1; irq->config = VGIC_CONFIG_EDGE; @@ -303,18 +297,11 @@ static void vgic_allocate_private_irq(struct kvm_vcpu *vcpu, int i, u32 type) } } -static void vgic_v5_allocate_private_irq(struct kvm_vcpu *vcpu, int i, u32 type) +static void vgic_v5_setup_private_irq(struct kvm_vcpu *vcpu, struct vgic_irq *irq) { - struct vgic_irq *irq = &vcpu->arch.vgic_cpu.private_irqs[i]; - u32 intid = vgic_v5_make_ppi(i); - - INIT_LIST_HEAD(&irq->ap_list); - raw_spin_lock_init(&irq->irq_lock); - irq->vcpu = NULL; - irq->target_vcpu = vcpu; - refcount_set(&irq->refcount, 0); + int i = irq - &vcpu->arch.vgic_cpu.private_irqs[0]; - irq->intid = intid; + irq->intid = vgic_v5_make_ppi(i); /* The only Edge architected PPI is the SW_PPI */ if (i == GICV5_ARCH_PPI_SW_PPI) @@ -323,7 +310,7 @@ static void vgic_v5_allocate_private_irq(struct kvm_vcpu *vcpu, int i, u32 type) irq->config = VGIC_CONFIG_LEVEL; /* Register the GICv5-specific PPI ops */ - vgic_v5_set_ppi_ops(vcpu, intid); + vgic_v5_set_ppi_ops(vcpu, irq->intid); } static int vgic_allocate_private_irqs_locked(struct kvm_vcpu *vcpu, u32 type) @@ -349,15 +336,19 @@ static int vgic_allocate_private_irqs_locked(struct kvm_vcpu *vcpu, u32 type) if (!vgic_cpu->private_irqs) return -ENOMEM; - /* - * Enable and configure all SGIs to be edge-triggered and - * configure all PPIs as level-triggered. - */ for (i = 0; i < num_private_irqs; i++) { + struct vgic_irq *irq = &vcpu->arch.vgic_cpu.private_irqs[i]; + + INIT_LIST_HEAD(&irq->ap_list); + raw_spin_lock_init(&irq->irq_lock); + irq->vcpu = NULL; + irq->target_vcpu = vcpu; + refcount_set(&irq->refcount, 0); + if (vgic_is_v5(vcpu->kvm)) - vgic_v5_allocate_private_irq(vcpu, i, type); + vgic_v5_setup_private_irq(vcpu, irq); else - vgic_allocate_private_irq(vcpu, i, type); + vgic_setup_private_irq(vcpu, irq, type); } return 0; diff --git a/arch/arm64/kvm/vgic/vgic-irqfd.c b/arch/arm64/kvm/vgic/vgic-irqfd.c index b9b86e3a6c86..19a1094536e6 100644 --- a/arch/arm64/kvm/vgic/vgic-irqfd.c +++ b/arch/arm64/kvm/vgic/vgic-irqfd.c @@ -20,9 +20,15 @@ static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e, int level, bool line_status) { unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS; + int ret; if (!vgic_valid_spi(kvm, spi_id)) return -EINVAL; + + ret = vgic_lazy_init(kvm); + if (ret) + return ret; + return kvm_vgic_inject_irq(kvm, NULL, spi_id, level, NULL); } diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 1e3706ac3b8e..4477f870c7b3 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -27,7 +27,7 @@ static struct kvm_device_ops kvm_arm_vgic_its_ops; static int vgic_its_save_tables_v0(struct vgic_its *its); static int vgic_its_restore_tables_v0(struct vgic_its *its); -static int vgic_its_commit_v0(struct vgic_its *its); +static void vgic_its_commit_v0(struct vgic_its *its); static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq, struct kvm_vcpu *filter_vcpu, bool needs_inv); @@ -168,7 +168,7 @@ struct vgic_its_abi { int ite_esz; int (*save_tables)(struct vgic_its *its); int (*restore_tables)(struct vgic_its *its); - int (*commit)(struct vgic_its *its); + void (*commit)(struct vgic_its *its); }; #define ABI_0_ESZ 8 @@ -192,13 +192,13 @@ inline const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its) return &its_table_abi_versions[its->abi_rev]; } -static int vgic_its_set_abi(struct vgic_its *its, u32 rev) +static void vgic_its_set_abi(struct vgic_its *its, u32 rev) { const struct vgic_its_abi *abi; its->abi_rev = rev; abi = vgic_its_get_abi(its); - return abi->commit(its); + abi->commit(its); } /* @@ -472,7 +472,8 @@ static int vgic_mmio_uaccess_write_its_iidr(struct kvm *kvm, if (rev >= NR_ITS_ABIS) return -EINVAL; - return vgic_its_set_abi(its, rev); + vgic_its_set_abi(its, rev); + return 0; } static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm, @@ -1890,14 +1891,11 @@ static int vgic_its_create(struct kvm_device *dev, u32 type) its->baser_coll_table = INITIAL_BASER_VALUE | ((u64)GITS_BASER_TYPE_COLLECTION << GITS_BASER_TYPE_SHIFT); dev->kvm->arch.vgic.propbaser = INITIAL_PROPBASER_VALUE; - dev->private = its; - ret = vgic_its_set_abi(its, NR_ITS_ABIS - 1); - + vgic_its_set_abi(its, NR_ITS_ABIS - 1); mutex_unlock(&dev->kvm->arch.config_lock); - - return ret; + return 0; } static void vgic_its_destroy(struct kvm_device *kvm_dev) @@ -2612,7 +2610,7 @@ static int vgic_its_restore_tables_v0(struct vgic_its *its) return ret; } -static int vgic_its_commit_v0(struct vgic_its *its) +static void vgic_its_commit_v0(struct vgic_its *its) { const struct vgic_its_abi *abi; @@ -2625,7 +2623,6 @@ static int vgic_its_commit_v0(struct vgic_its *its) its->baser_device_table |= (GIC_ENCODE_SZ(abi->dte_esz, 5) << GITS_BASER_ENTRY_SIZE_SHIFT); - return 0; } static void vgic_its_reset(struct kvm *kvm, struct vgic_its *its) diff --git a/arch/arm64/kvm/vgic/vgic-kvm-device.c b/arch/arm64/kvm/vgic/vgic-kvm-device.c index a96c77dccf35..90be99443df3 100644 --- a/arch/arm64/kvm/vgic/vgic-kvm-device.c +++ b/arch/arm64/kvm/vgic/vgic-kvm-device.c @@ -730,18 +730,15 @@ static int vgic_v5_get_userspace_ppis(struct kvm_device *dev, guard(mutex)(&dev->kvm->arch.config_lock); /* - * We either support 64 or 128 PPIs. In the former case, we need to - * return 0s for the second 64 bits as we have no storage backing those. + * We only support 64 PPIs, so, we need to return 0s for the + * second 64 bits as we have no storage backing those. */ ret = put_user(bitmap_read(gicv5_vm->userspace_ppis, 0, 64), uaddr); if (ret) return ret; uaddr++; - if (VGIC_V5_NR_PRIVATE_IRQS == 128) - ret = put_user(bitmap_read(gicv5_vm->userspace_ppis, 64, 128), uaddr); - else - ret = put_user(0, uaddr); + ret = put_user(0, uaddr); return ret; } diff --git a/arch/arm64/kvm/vgic/vgic-v5.c b/arch/arm64/kvm/vgic/vgic-v5.c index fdd39ea7f83e..d4789ff3e740 100644 --- a/arch/arm64/kvm/vgic/vgic-v5.c +++ b/arch/arm64/kvm/vgic/vgic-v5.c @@ -10,7 +10,7 @@ #include "vgic.h" -static struct vgic_v5_ppi_caps ppi_caps; +#define ppi_caps kvm_vgic_global_state.vgic_v5_ppi_caps /* * Not all PPIs are guaranteed to be implemented for GICv5. Deterermine which @@ -18,20 +18,17 @@ static struct vgic_v5_ppi_caps ppi_caps; */ static void vgic_v5_get_implemented_ppis(void) { - if (!cpus_have_final_cap(ARM64_HAS_GICV5_CPUIF)) - return; - /* * If we have KVM, we have EL2, which means that we have support for the * EL1 and EL2 Physical & Virtual timers. */ - __assign_bit(GICV5_ARCH_PPI_CNTHP, ppi_caps.impl_ppi_mask, 1); - __assign_bit(GICV5_ARCH_PPI_CNTV, ppi_caps.impl_ppi_mask, 1); - __assign_bit(GICV5_ARCH_PPI_CNTHV, ppi_caps.impl_ppi_mask, 1); - __assign_bit(GICV5_ARCH_PPI_CNTP, ppi_caps.impl_ppi_mask, 1); + __set_bit(GICV5_ARCH_PPI_CNTHP, ppi_caps.impl_ppi_mask); + __set_bit(GICV5_ARCH_PPI_CNTV, ppi_caps.impl_ppi_mask); + __set_bit(GICV5_ARCH_PPI_CNTHV, ppi_caps.impl_ppi_mask); + __set_bit(GICV5_ARCH_PPI_CNTP, ppi_caps.impl_ppi_mask); /* The SW_PPI should be available */ - __assign_bit(GICV5_ARCH_PPI_SW_PPI, ppi_caps.impl_ppi_mask, 1); + __set_bit(GICV5_ARCH_PPI_SW_PPI, ppi_caps.impl_ppi_mask); /* The PMUIRQ is available if we have the PMU */ __assign_bit(GICV5_ARCH_PPI_PMUIRQ, ppi_caps.impl_ppi_mask, system_supports_pmuv3()); @@ -146,9 +143,7 @@ int vgic_v5_init(struct kvm *kvm) /* We only allow userspace to drive the SW_PPI, if it is implemented. */ bitmap_zero(kvm->arch.vgic.gicv5_vm.userspace_ppis, VGIC_V5_NR_PRIVATE_IRQS); - __assign_bit(GICV5_ARCH_PPI_SW_PPI, - kvm->arch.vgic.gicv5_vm.userspace_ppis, - VGIC_V5_NR_PRIVATE_IRQS); + __set_bit(GICV5_ARCH_PPI_SW_PPI, kvm->arch.vgic.gicv5_vm.userspace_ppis); bitmap_and(kvm->arch.vgic.gicv5_vm.userspace_ppis, kvm->arch.vgic.gicv5_vm.userspace_ppis, ppi_caps.impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); @@ -197,7 +192,7 @@ int vgic_v5_finalize_ppi_state(struct kvm *kvm) /* Expose PPIs with an owner or the SW_PPI, only */ scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) { if (irq->owner || i == GICV5_ARCH_PPI_SW_PPI) { - __assign_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, 1); + __set_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask); __assign_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_hmr, irq->config == VGIC_CONFIG_LEVEL); } @@ -243,9 +238,9 @@ static u32 vgic_v5_get_effective_priority_mask(struct kvm_vcpu *vcpu) /* * For GICv5, the PPIs are mostly directly managed by the hardware. We (the - * hypervisor) handle the pending, active, enable state save/restore, but don't - * need the PPIs to be queued on a per-VCPU AP list. Therefore, sanity check the - * state, unlock, and return. + * hypervisor) handle the pending, active, enable state save/restore, but + * don't need the PPIs to be queued on a per-VCPU AP list. Therefore, + * unlock, kick the vcpu and return. */ bool vgic_v5_ppi_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq, unsigned long flags) @@ -255,12 +250,7 @@ bool vgic_v5_ppi_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq, lockdep_assert_held(&irq->irq_lock); - if (WARN_ON_ONCE(!__irq_is_ppi(KVM_DEV_TYPE_ARM_VGIC_V5, irq->intid))) - goto out_unlock_fail; - vcpu = irq->target_vcpu; - if (WARN_ON_ONCE(!vcpu)) - goto out_unlock_fail; raw_spin_unlock_irqrestore(&irq->irq_lock, flags); @@ -269,11 +259,6 @@ bool vgic_v5_ppi_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq, kvm_vcpu_kick(vcpu); return true; - -out_unlock_fail: - raw_spin_unlock_irqrestore(&irq->irq_lock, flags); - - return false; } /* @@ -287,10 +272,10 @@ void vgic_v5_set_ppi_dvi(struct kvm_vcpu *vcpu, struct vgic_irq *irq, bool dvi) lockdep_assert_held(&irq->irq_lock); ppi = vgic_v5_get_hwirq_id(irq->intid); - __assign_bit(ppi, cpu_if->vgic_ppi_dvir, dvi); + assign_bit(ppi, cpu_if->vgic_ppi_dvir, dvi); } -static struct irq_ops vgic_v5_ppi_irq_ops = { +static const struct irq_ops vgic_v5_ppi_irq_ops = { .queue_irq_unlock = vgic_v5_ppi_queue_irq_unlock, .set_direct_injection = vgic_v5_set_ppi_dvi, }; @@ -316,7 +301,7 @@ static void vgic_v5_sync_ppi_priorities(struct kvm_vcpu *vcpu) * those actually exposed to the guest by first iterating over the mask * of exposed PPIs. */ - for_each_set_bit(i, vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS) { + for_each_visible_v5_ppi(i, vcpu->kvm) { u32 intid = vgic_v5_make_ppi(i); struct vgic_irq *irq; int pri_idx, pri_reg, pri_bit; @@ -358,7 +343,7 @@ bool vgic_v5_has_pending_ppi(struct kvm_vcpu *vcpu) if (!priority_mask) return false; - for_each_set_bit(i, vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS) { + for_each_visible_v5_ppi(i, vcpu->kvm) { u32 intid = vgic_v5_make_ppi(i); bool has_pending = false; struct vgic_irq *irq; @@ -391,8 +376,7 @@ void vgic_v5_fold_ppi_state(struct kvm_vcpu *vcpu) activer = host_data_ptr(vgic_v5_ppi_state)->activer_exit; pendr = host_data_ptr(vgic_v5_ppi_state)->pendr; - for_each_set_bit(i, vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, - VGIC_V5_NR_PRIVATE_IRQS) { + for_each_visible_v5_ppi(i, vcpu->kvm) { u32 intid = vgic_v5_make_ppi(i); struct vgic_irq *irq; @@ -429,8 +413,7 @@ void vgic_v5_flush_ppi_state(struct kvm_vcpu *vcpu) * ICC_PPI_PENDRx_EL1, however. */ bitmap_zero(pendr, VGIC_V5_NR_PRIVATE_IRQS); - for_each_set_bit(i, vcpu->kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, - VGIC_V5_NR_PRIVATE_IRQS) { + for_each_visible_v5_ppi(i, vcpu->kvm) { u32 intid = vgic_v5_make_ppi(i); struct vgic_irq *irq; diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 1e9fe8764584..5a4768d8cd4f 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -106,24 +106,23 @@ struct vgic_irq *vgic_get_irq(struct kvm *kvm, u32 intid) struct vgic_irq *vgic_get_vcpu_irq(struct kvm_vcpu *vcpu, u32 intid) { + enum kvm_device_type type; + if (WARN_ON(!vcpu)) return NULL; - if (vgic_is_v5(vcpu->kvm)) { - u32 int_num, hwirq_id; - - if (!__irq_is_ppi(KVM_DEV_TYPE_ARM_VGIC_V5, intid)) - return NULL; - - hwirq_id = FIELD_GET(GICV5_HWIRQ_ID, intid); - int_num = array_index_nospec(hwirq_id, VGIC_V5_NR_PRIVATE_IRQS); + type = vcpu->kvm->arch.vgic.vgic_model; - return &vcpu->arch.vgic_cpu.private_irqs[int_num]; - } + if (__irq_is_sgi(type, intid) || __irq_is_ppi(type, intid)) { + switch (type) { + case KVM_DEV_TYPE_ARM_VGIC_V5: + intid = vgic_v5_get_hwirq_id(intid); + intid = array_index_nospec(intid, VGIC_V5_NR_PRIVATE_IRQS); + break; + default: + intid = array_index_nospec(intid, VGIC_NR_PRIVATE_IRQS); + } - /* SGIs and PPIs */ - if (intid < VGIC_NR_PRIVATE_IRQS) { - intid = array_index_nospec(intid, VGIC_NR_PRIVATE_IRQS); return &vcpu->arch.vgic_cpu.private_irqs[intid]; } @@ -534,11 +533,9 @@ int kvm_vgic_inject_irq(struct kvm *kvm, struct kvm_vcpu *vcpu, { struct vgic_irq *irq; unsigned long flags; - int ret; - ret = vgic_lazy_init(kvm); - if (ret) - return ret; + if (unlikely(!vgic_initialized(kvm))) + return 0; if (!vcpu && irq_is_private(kvm, intid)) return -EINVAL; @@ -573,7 +570,7 @@ int kvm_vgic_inject_irq(struct kvm *kvm, struct kvm_vcpu *vcpu, } void kvm_vgic_set_irq_ops(struct kvm_vcpu *vcpu, u32 vintid, - struct irq_ops *ops) + const struct irq_ops *ops) { struct vgic_irq *irq = vgic_get_vcpu_irq(vcpu, vintid); diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h index 9d941241c8a2..f45f7e3ec4d6 100644 --- a/arch/arm64/kvm/vgic/vgic.h +++ b/arch/arm64/kvm/vgic/vgic.h @@ -378,6 +378,9 @@ void vgic_v5_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr); void vgic_v5_restore_state(struct kvm_vcpu *vcpu); void vgic_v5_save_state(struct kvm_vcpu *vcpu); +#define for_each_visible_v5_ppi(__i, __k) \ + for_each_set_bit(__i, (__k)->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS) + static inline int vgic_v3_max_apr_idx(struct kvm_vcpu *vcpu) { struct vgic_cpu *cpu_if = &vcpu->arch.vgic_cpu; diff --git a/arch/loongarch/include/asm/kvm_host.h b/arch/loongarch/include/asm/kvm_host.h index 776bc487a705..23cfbecebbd7 100644 --- a/arch/loongarch/include/asm/kvm_host.h +++ b/arch/loongarch/include/asm/kvm_host.h @@ -38,7 +38,8 @@ #define KVM_REQ_TLB_FLUSH_GPA KVM_ARCH_REQ(0) #define KVM_REQ_STEAL_UPDATE KVM_ARCH_REQ(1) #define KVM_REQ_PMU KVM_ARCH_REQ(2) -#define KVM_REQ_AUX_LOAD KVM_ARCH_REQ(3) +#define KVM_REQ_FPU_LOAD KVM_ARCH_REQ(3) +#define KVM_REQ_LBT_LOAD KVM_ARCH_REQ(4) #define KVM_GUESTDBG_SW_BP_MASK \ (KVM_GUESTDBG_ENABLE | KVM_GUESTDBG_USE_SW_BP) @@ -157,12 +158,10 @@ enum emulation_result { }; #define KVM_LARCH_FPU (0x1 << 0) -#define KVM_LARCH_LSX (0x1 << 1) -#define KVM_LARCH_LASX (0x1 << 2) -#define KVM_LARCH_LBT (0x1 << 3) -#define KVM_LARCH_PMU (0x1 << 4) -#define KVM_LARCH_SWCSR_LATEST (0x1 << 5) -#define KVM_LARCH_HWCSR_USABLE (0x1 << 6) +#define KVM_LARCH_LBT (0x1 << 1) +#define KVM_LARCH_PMU (0x1 << 2) +#define KVM_LARCH_SWCSR_LATEST (0x1 << 3) +#define KVM_LARCH_HWCSR_USABLE (0x1 << 4) #define LOONGARCH_PV_FEAT_UPDATED BIT_ULL(63) #define LOONGARCH_PV_FEAT_MASK (BIT(KVM_FEATURE_IPI) | \ @@ -203,7 +202,6 @@ struct kvm_vcpu_arch { /* Which auxiliary state is loaded (KVM_LARCH_*) */ unsigned int aux_inuse; - unsigned int aux_ldtype; /* FPU state */ struct loongarch_fpu fpu FPU_ALIGN; diff --git a/arch/loongarch/include/asm/kvm_vcpu.h b/arch/loongarch/include/asm/kvm_vcpu.h index 3784ab4ccdb5..a70a32103523 100644 --- a/arch/loongarch/include/asm/kvm_vcpu.h +++ b/arch/loongarch/include/asm/kvm_vcpu.h @@ -10,22 +10,37 @@ #include <asm/loongarch.h> /* Controlled by 0x5 guest estat */ -#define CPU_SIP0 (_ULCAST_(1)) -#define CPU_SIP1 (_ULCAST_(1) << 1) -#define CPU_PMU (_ULCAST_(1) << 10) -#define CPU_TIMER (_ULCAST_(1) << 11) -#define CPU_IPI (_ULCAST_(1) << 12) -#define CPU_AVEC (_ULCAST_(1) << 14) +#define CPU_SIP0 BIT(INT_SWI0) +#define CPU_SIP1 BIT(INT_SWI1) +#define CPU_HWI0 BIT(INT_HWI0) +#define CPU_HWI1 BIT(INT_HWI1) +#define CPU_HWI2 BIT(INT_HWI2) +#define CPU_HWI3 BIT(INT_HWI3) +#define CPU_HWI4 BIT(INT_HWI4) +#define CPU_HWI5 BIT(INT_HWI5) +#define CPU_HWI6 BIT(INT_HWI6) +#define CPU_HWI7 BIT(INT_HWI7) +#define CPU_PMU BIT(INT_PCOV) +#define CPU_TIMER BIT(INT_TI) +#define CPU_IPI BIT(INT_IPI) +#define CPU_AVEC BIT(INT_AVEC) +#define KVM_ESTAT_INTI_MASK (CPU_SIP0 | CPU_SIP1 | CPU_PMU | CPU_TIMER \ + | CPU_IPI | CPU_AVEC) +#define KVM_ESTAT_EXTI_MASK (CPU_HWI0 | CPU_HWI1 | CPU_HWI2 | CPU_HWI3 \ + | CPU_HWI4 | CPU_HWI5 | CPU_HWI6 | CPU_HWI7) /* Controlled by 0x52 guest exception VIP aligned to estat bit 5~12 */ -#define CPU_IP0 (_ULCAST_(1)) -#define CPU_IP1 (_ULCAST_(1) << 1) -#define CPU_IP2 (_ULCAST_(1) << 2) -#define CPU_IP3 (_ULCAST_(1) << 3) -#define CPU_IP4 (_ULCAST_(1) << 4) -#define CPU_IP5 (_ULCAST_(1) << 5) -#define CPU_IP6 (_ULCAST_(1) << 6) -#define CPU_IP7 (_ULCAST_(1) << 7) +#define VIP_DELTA (INT_HWI0 - CSR_GINTC_VIP_SHIFT) +#define CPU_IP0 BIT(INT_HWI0 - VIP_DELTA) +#define CPU_IP1 BIT(INT_HWI1 - VIP_DELTA) +#define CPU_IP2 BIT(INT_HWI2 - VIP_DELTA) +#define CPU_IP3 BIT(INT_HWI3 - VIP_DELTA) +#define CPU_IP4 BIT(INT_HWI4 - VIP_DELTA) +#define CPU_IP5 BIT(INT_HWI5 - VIP_DELTA) +#define CPU_IP6 BIT(INT_HWI6 - VIP_DELTA) +#define CPU_IP7 BIT(INT_HWI7 - VIP_DELTA) +#define KVM_GINTC_IRQ_MASK (CPU_IP0 | CPU_IP1 | CPU_IP2 | CPU_IP3 \ + | CPU_IP4 | CPU_IP5 | CPU_IP6 | CPU_IP7) #define MNSEC_PER_SEC (NSEC_PER_SEC >> 20) diff --git a/arch/loongarch/kvm/exit.c b/arch/loongarch/kvm/exit.c index 3b95cd0f989b..8572b63478bb 100644 --- a/arch/loongarch/kvm/exit.c +++ b/arch/loongarch/kvm/exit.c @@ -103,7 +103,6 @@ static unsigned long kvm_emu_xchg_csr(struct kvm_vcpu *vcpu, int csrid, old = kvm_read_sw_gcsr(csr, csrid); val = (old & ~csr_mask) | (val & csr_mask); kvm_write_sw_gcsr(csr, csrid, val); - old = old & csr_mask; } else pr_warn_once("Unsupported csrxchg 0x%x with pc %lx\n", csrid, vcpu->arch.pc); @@ -755,8 +754,7 @@ static int kvm_handle_fpu_disabled(struct kvm_vcpu *vcpu, int ecode) return RESUME_HOST; } - vcpu->arch.aux_ldtype = KVM_LARCH_FPU; - kvm_make_request(KVM_REQ_AUX_LOAD, vcpu); + kvm_make_request(KVM_REQ_FPU_LOAD, vcpu); return RESUME_GUEST; } @@ -796,10 +794,8 @@ static int kvm_handle_lsx_disabled(struct kvm_vcpu *vcpu, int ecode) { if (!kvm_guest_has_lsx(&vcpu->arch)) kvm_queue_exception(vcpu, EXCCODE_INE, 0); - else { - vcpu->arch.aux_ldtype = KVM_LARCH_LSX; - kvm_make_request(KVM_REQ_AUX_LOAD, vcpu); - } + else + kvm_make_request(KVM_REQ_FPU_LOAD, vcpu); return RESUME_GUEST; } @@ -816,10 +812,8 @@ static int kvm_handle_lasx_disabled(struct kvm_vcpu *vcpu, int ecode) { if (!kvm_guest_has_lasx(&vcpu->arch)) kvm_queue_exception(vcpu, EXCCODE_INE, 0); - else { - vcpu->arch.aux_ldtype = KVM_LARCH_LASX; - kvm_make_request(KVM_REQ_AUX_LOAD, vcpu); - } + else + kvm_make_request(KVM_REQ_FPU_LOAD, vcpu); return RESUME_GUEST; } @@ -828,10 +822,8 @@ static int kvm_handle_lbt_disabled(struct kvm_vcpu *vcpu, int ecode) { if (!kvm_guest_has_lbt(&vcpu->arch)) kvm_queue_exception(vcpu, EXCCODE_INE, 0); - else { - vcpu->arch.aux_ldtype = KVM_LARCH_LBT; - kvm_make_request(KVM_REQ_AUX_LOAD, vcpu); - } + else + kvm_make_request(KVM_REQ_LBT_LOAD, vcpu); return RESUME_GUEST; } diff --git a/arch/loongarch/kvm/intc/eiointc.c b/arch/loongarch/kvm/intc/eiointc.c index 2ab7fafa86d5..2b14485d14a7 100644 --- a/arch/loongarch/kvm/intc/eiointc.c +++ b/arch/loongarch/kvm/intc/eiointc.c @@ -645,10 +645,14 @@ static int kvm_eiointc_create(struct kvm_device *dev, u32 type) device = &s->device_vext; kvm_iodevice_init(device, &kvm_eiointc_virt_ops); + mutex_lock(&kvm->slots_lock); ret = kvm_io_bus_register_dev(kvm, KVM_IOCSR_BUS, EIOINTC_VIRT_BASE, EIOINTC_VIRT_SIZE, device); + mutex_unlock(&kvm->slots_lock); if (ret < 0) { + mutex_lock(&kvm->slots_lock); kvm_io_bus_unregister_dev(kvm, KVM_IOCSR_BUS, &s->device); + mutex_unlock(&kvm->slots_lock); kfree(s); return ret; } @@ -667,8 +671,10 @@ static void kvm_eiointc_destroy(struct kvm_device *dev) kvm = dev->kvm; eiointc = kvm->arch.eiointc; + mutex_lock(&kvm->slots_lock); kvm_io_bus_unregister_dev(kvm, KVM_IOCSR_BUS, &eiointc->device); kvm_io_bus_unregister_dev(kvm, KVM_IOCSR_BUS, &eiointc->device_vext); + mutex_unlock(&kvm->slots_lock); kfree(eiointc); kfree(dev); } diff --git a/arch/loongarch/kvm/intc/ipi.c b/arch/loongarch/kvm/intc/ipi.c index 1f6ebbd0af5c..4fa0897d7bdb 100644 --- a/arch/loongarch/kvm/intc/ipi.c +++ b/arch/loongarch/kvm/intc/ipi.c @@ -447,7 +447,9 @@ static void kvm_ipi_destroy(struct kvm_device *dev) kvm = dev->kvm; ipi = kvm->arch.ipi; + mutex_lock(&kvm->slots_lock); kvm_io_bus_unregister_dev(kvm, KVM_IOCSR_BUS, &ipi->device); + mutex_unlock(&kvm->slots_lock); kfree(ipi); kfree(dev); } diff --git a/arch/loongarch/kvm/intc/pch_pic.c b/arch/loongarch/kvm/intc/pch_pic.c index aa0ed59ae8cf..175a630aceb4 100644 --- a/arch/loongarch/kvm/intc/pch_pic.c +++ b/arch/loongarch/kvm/intc/pch_pic.c @@ -481,7 +481,9 @@ static void kvm_pch_pic_destroy(struct kvm_device *dev) kvm = dev->kvm; s = kvm->arch.pch_pic; /* unregister pch pic device and free it's memory */ + mutex_lock(&kvm->slots_lock); kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS, &s->device); + mutex_unlock(&kvm->slots_lock); kfree(s); kfree(dev); } diff --git a/arch/loongarch/kvm/interrupt.c b/arch/loongarch/kvm/interrupt.c index a18c60dffbba..055ac9235b03 100644 --- a/arch/loongarch/kvm/interrupt.c +++ b/arch/loongarch/kvm/interrupt.c @@ -9,41 +9,16 @@ #include <asm/kvm_vcpu.h> #include <asm/kvm_dmsintc.h> -static unsigned int priority_to_irq[EXCCODE_INT_NUM] = { - [INT_TI] = CPU_TIMER, - [INT_IPI] = CPU_IPI, - [INT_SWI0] = CPU_SIP0, - [INT_SWI1] = CPU_SIP1, - [INT_HWI0] = CPU_IP0, - [INT_HWI1] = CPU_IP1, - [INT_HWI2] = CPU_IP2, - [INT_HWI3] = CPU_IP3, - [INT_HWI4] = CPU_IP4, - [INT_HWI5] = CPU_IP5, - [INT_HWI6] = CPU_IP6, - [INT_HWI7] = CPU_IP7, - [INT_AVEC] = CPU_AVEC, -}; - -static int kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned int priority) +static void kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned long mask) { - unsigned int irq = 0; + unsigned long irq; unsigned long old, new; - clear_bit(priority, &vcpu->arch.irq_pending); - if (priority < EXCCODE_INT_NUM) - irq = priority_to_irq[priority]; - - switch (priority) { - case INT_AVEC: - if (!kvm_guest_has_msgint(&vcpu->arch)) - break; + if (mask & CPU_AVEC) dmsintc_inject_irq(vcpu); - fallthrough; - case INT_TI: - case INT_IPI: - case INT_SWI0: - case INT_SWI1: + + irq = mask & KVM_ESTAT_INTI_MASK; + if (irq) { old = kvm_read_hw_gcsr(LOONGARCH_CSR_TVAL); set_gcsr_estat(irq); new = kvm_read_hw_gcsr(LOONGARCH_CSR_TVAL); @@ -51,37 +26,20 @@ static int kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned int priority) /* Inject TI if TVAL inverted */ if (new > old) set_gcsr_estat(CPU_TIMER); - break; - - case INT_HWI0 ... INT_HWI7: - set_csr_gintc(irq); - break; - - default: - break; } - return 1; + irq = (mask >> VIP_DELTA) & KVM_GINTC_IRQ_MASK; + if (irq) + set_csr_gintc(irq); } -static int kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned int priority) +static void kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned long mask) { - unsigned int irq = 0; + unsigned long irq; unsigned long old, new; - clear_bit(priority, &vcpu->arch.irq_clear); - if (priority < EXCCODE_INT_NUM) - irq = priority_to_irq[priority]; - - switch (priority) { - case INT_AVEC: - if (!kvm_guest_has_msgint(&vcpu->arch)) - break; - fallthrough; - case INT_TI: - case INT_IPI: - case INT_SWI0: - case INT_SWI1: + irq = mask & KVM_ESTAT_INTI_MASK; + if (irq) { old = kvm_read_hw_gcsr(LOONGARCH_CSR_TVAL); clear_gcsr_estat(irq); new = kvm_read_hw_gcsr(LOONGARCH_CSR_TVAL); @@ -89,30 +47,28 @@ static int kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned int priority) /* Inject TI if TVAL inverted */ if (new > old) set_gcsr_estat(CPU_TIMER); - break; - - case INT_HWI0 ... INT_HWI7: - clear_csr_gintc(irq); - break; - - default: - break; } - return 1; + irq = (mask >> VIP_DELTA) & KVM_GINTC_IRQ_MASK; + if (irq) + clear_csr_gintc(irq); } void kvm_deliver_intr(struct kvm_vcpu *vcpu) { - unsigned int priority; - unsigned long *pending = &vcpu->arch.irq_pending; - unsigned long *pending_clr = &vcpu->arch.irq_clear; + unsigned long mask; - for_each_set_bit(priority, pending_clr, EXCCODE_INT_NUM) - kvm_irq_clear(vcpu, priority); + mask = READ_ONCE(vcpu->arch.irq_clear); + if (mask) { + mask = xchg_relaxed(&vcpu->arch.irq_clear, 0); + kvm_irq_clear(vcpu, mask); + } - for_each_set_bit(priority, pending, EXCCODE_INT_NUM) - kvm_irq_deliver(vcpu, priority); + mask = READ_ONCE(vcpu->arch.irq_pending); + if (mask) { + mask = xchg_relaxed(&vcpu->arch.irq_pending, 0); + kvm_irq_deliver(vcpu, mask); + } } int kvm_pending_timer(struct kvm_vcpu *vcpu) diff --git a/arch/loongarch/kvm/irqfd.c b/arch/loongarch/kvm/irqfd.c index f4f953b22419..40ed1081c4b6 100644 --- a/arch/loongarch/kvm/irqfd.c +++ b/arch/loongarch/kvm/irqfd.c @@ -51,7 +51,8 @@ int kvm_set_routing_entry(struct kvm *kvm, e->irqchip.irqchip = ue->u.irqchip.irqchip; e->irqchip.pin = ue->u.irqchip.pin; - if (e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) + if (e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS || + e->irqchip.irqchip >= KVM_NR_IRQCHIPS) return -EINVAL; return 0; diff --git a/arch/loongarch/kvm/timer.c b/arch/loongarch/kvm/timer.c index 8356fce0043f..3829f35a4070 100644 --- a/arch/loongarch/kvm/timer.c +++ b/arch/loongarch/kvm/timer.c @@ -30,8 +30,7 @@ enum hrtimer_restart kvm_swtimer_wakeup(struct hrtimer *timer) struct kvm_vcpu *vcpu; vcpu = container_of(timer, struct kvm_vcpu, arch.swtimer); - kvm_queue_irq(vcpu, INT_TI); - rcuwait_wake_up(&vcpu->wait); + kvm_vcpu_wake_up(vcpu); return HRTIMER_NORESTART; } diff --git a/arch/loongarch/kvm/vcpu.c b/arch/loongarch/kvm/vcpu.c index e28084c49e68..20c207d80e31 100644 --- a/arch/loongarch/kvm/vcpu.c +++ b/arch/loongarch/kvm/vcpu.c @@ -239,26 +239,17 @@ static void kvm_late_check_requests(struct kvm_vcpu *vcpu) vcpu->arch.flush_gpa = INVALID_GPA; } - if (kvm_check_request(KVM_REQ_AUX_LOAD, vcpu)) { - switch (vcpu->arch.aux_ldtype) { - case KVM_LARCH_FPU: - kvm_own_fpu(vcpu); - break; - case KVM_LARCH_LSX: - kvm_own_lsx(vcpu); - break; - case KVM_LARCH_LASX: + if (kvm_check_request(KVM_REQ_FPU_LOAD, vcpu)) { + if (kvm_guest_has_lasx(&vcpu->arch)) kvm_own_lasx(vcpu); - break; - case KVM_LARCH_LBT: - kvm_own_lbt(vcpu); - break; - default: - break; - } - - vcpu->arch.aux_ldtype = 0; + else if (kvm_guest_has_lsx(&vcpu->arch)) + kvm_own_lsx(vcpu); + else if (kvm_guest_has_fpu(&vcpu->arch)) + kvm_own_fpu(vcpu); } + + if (kvm_check_request(KVM_REQ_LBT_LOAD, vcpu)) + kvm_own_lbt(vcpu); } /* @@ -308,10 +299,10 @@ static int kvm_pre_enter_guest(struct kvm_vcpu *vcpu) * check vmid before vcpu enter guest */ local_irq_disable(); - kvm_deliver_intr(vcpu); kvm_deliver_exception(vcpu); /* Make sure the vcpu mode has been written */ smp_store_mb(vcpu->mode, IN_GUEST_MODE); + kvm_deliver_intr(vcpu); kvm_check_vpid(vcpu); /* @@ -348,7 +339,7 @@ static int kvm_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu) u32 intr = estat & CSR_ESTAT_IS; u32 ecode = (estat & CSR_ESTAT_EXC) >> CSR_ESTAT_EXC_SHIFT; - vcpu->mode = OUTSIDE_GUEST_MODE; + smp_store_mb(vcpu->mode, OUTSIDE_GUEST_MODE); /* Set a default exit reason */ run->exit_reason = KVM_EXIT_UNKNOWN; @@ -602,7 +593,7 @@ struct kvm_vcpu *kvm_get_vcpu_by_cpuid(struct kvm *kvm, int cpuid) static int _kvm_getcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 *val) { - unsigned long gintc; + unsigned long estat, gintc; struct loongarch_csrs *csr = vcpu->arch.csr; if (get_gcsr_flag(id) & INVALID_GCSR) @@ -621,8 +612,9 @@ static int _kvm_getcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 *val) preempt_enable(); /* ESTAT IP0~IP7 get from GINTC */ - gintc = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_GINTC) & 0xff; - *val = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_ESTAT) | (gintc << 2); + gintc = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_GINTC) & KVM_GINTC_IRQ_MASK; + estat = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_ESTAT) & ~KVM_ESTAT_EXTI_MASK; + *val = estat | (gintc << VIP_DELTA); return 0; } @@ -637,7 +629,8 @@ static int _kvm_getcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 *val) static int _kvm_setcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 val) { - int ret = 0, gintc; + int ret = 0; + unsigned long estat, gintc; struct loongarch_csrs *csr = vcpu->arch.csr; if (get_gcsr_flag(id) & INVALID_GCSR) @@ -648,11 +641,16 @@ static int _kvm_setcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 val) if (id == LOONGARCH_CSR_ESTAT) { /* ESTAT IP0~IP7 inject through GINTC */ - gintc = (val >> 2) & 0xff; - kvm_set_sw_gcsr(csr, LOONGARCH_CSR_GINTC, gintc); + gintc = (val >> VIP_DELTA) & KVM_GINTC_IRQ_MASK; + gintc |= kvm_read_sw_gcsr(csr, LOONGARCH_CSR_GINTC) & ~KVM_GINTC_IRQ_MASK; + kvm_write_sw_gcsr(csr, LOONGARCH_CSR_GINTC, gintc); - gintc = val & ~(0xffUL << 2); - kvm_set_sw_gcsr(csr, LOONGARCH_CSR_ESTAT, gintc); + /* Only set valid ESTAT bits */ + estat = val & ~KVM_ESTAT_EXTI_MASK; + estat &= CSR_ESTAT_IS | CSR_ESTAT_EXC | CSR_ESTAT_ESUBCODE; + if (!kvm_guest_has_msgint(&vcpu->arch)) + estat &= ~CPU_AVEC; + kvm_write_sw_gcsr(csr, LOONGARCH_CSR_ESTAT, estat); return ret; } @@ -1108,7 +1106,8 @@ static int kvm_loongarch_cpucfg_get_attr(struct kvm_vcpu *vcpu, return -ENXIO; } - put_user(val, uaddr); + if (put_user(val, uaddr)) + return -EFAULT; return ret; } @@ -1312,7 +1311,7 @@ int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) fpu->fcc = vcpu->arch.fpu.fcc; fpu->fcsr = vcpu->arch.fpu.fcsr; for (i = 0; i < NUM_FPU_REGS; i++) - memcpy(&fpu->fpr[i], &vcpu->arch.fpu.fpr[i], FPU_REG_WIDTH / 64); + memcpy(&fpu->fpr[i], &vcpu->arch.fpu.fpr[i], sizeof(union fpureg)); return 0; } @@ -1324,7 +1323,7 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) vcpu->arch.fpu.fcc = fpu->fcc; vcpu->arch.fpu.fcsr = fpu->fcsr; for (i = 0; i < NUM_FPU_REGS; i++) - memcpy(&vcpu->arch.fpu.fpr[i], &fpu->fpr[i], FPU_REG_WIDTH / 64); + memcpy(&vcpu->arch.fpu.fpr[i], &fpu->fpr[i], sizeof(union fpureg)); return 0; } @@ -1398,24 +1397,10 @@ int kvm_own_lsx(struct kvm_vcpu *vcpu) /* Enable LSX for guest */ kvm_check_fcsr(vcpu, vcpu->arch.fpu.fcsr); set_csr_euen(CSR_EUEN_LSXEN | CSR_EUEN_FPEN); - switch (vcpu->arch.aux_inuse & KVM_LARCH_FPU) { - case KVM_LARCH_FPU: - /* - * Guest FPU state already loaded, - * only restore upper LSX state - */ - _restore_lsx_upper(&vcpu->arch.fpu); - break; - default: - /* Neither FP or LSX already active, - * restore full LSX state - */ - kvm_restore_lsx(&vcpu->arch.fpu); - break; - } + kvm_restore_lsx(&vcpu->arch.fpu); + vcpu->arch.aux_inuse |= KVM_LARCH_FPU; trace_kvm_aux(vcpu, KVM_TRACE_AUX_RESTORE, KVM_TRACE_AUX_LSX); - vcpu->arch.aux_inuse |= KVM_LARCH_LSX | KVM_LARCH_FPU; return 0; } @@ -1427,25 +1412,10 @@ int kvm_own_lasx(struct kvm_vcpu *vcpu) { kvm_check_fcsr(vcpu, vcpu->arch.fpu.fcsr); set_csr_euen(CSR_EUEN_FPEN | CSR_EUEN_LSXEN | CSR_EUEN_LASXEN); - switch (vcpu->arch.aux_inuse & (KVM_LARCH_FPU | KVM_LARCH_LSX)) { - case KVM_LARCH_LSX: - case KVM_LARCH_LSX | KVM_LARCH_FPU: - /* Guest LSX state already loaded, only restore upper LASX state */ - _restore_lasx_upper(&vcpu->arch.fpu); - break; - case KVM_LARCH_FPU: - /* Guest FP state already loaded, only restore upper LSX & LASX state */ - _restore_lsx_upper(&vcpu->arch.fpu); - _restore_lasx_upper(&vcpu->arch.fpu); - break; - default: - /* Neither FP or LSX already active, restore full LASX state */ - kvm_restore_lasx(&vcpu->arch.fpu); - break; - } + kvm_restore_lasx(&vcpu->arch.fpu); + vcpu->arch.aux_inuse |= KVM_LARCH_FPU; trace_kvm_aux(vcpu, KVM_TRACE_AUX_RESTORE, KVM_TRACE_AUX_LASX); - vcpu->arch.aux_inuse |= KVM_LARCH_LASX | KVM_LARCH_LSX | KVM_LARCH_FPU; return 0; } @@ -1456,29 +1426,32 @@ void kvm_lose_fpu(struct kvm_vcpu *vcpu) { preempt_disable(); + if (!(vcpu->arch.aux_inuse & KVM_LARCH_FPU)) + goto done; + kvm_check_fcsr_alive(vcpu); - if (vcpu->arch.aux_inuse & KVM_LARCH_LASX) { + if (kvm_guest_has_lasx(&vcpu->arch)) { kvm_save_lasx(&vcpu->arch.fpu); - vcpu->arch.aux_inuse &= ~(KVM_LARCH_LSX | KVM_LARCH_FPU | KVM_LARCH_LASX); trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_LASX); /* Disable LASX & LSX & FPU */ clear_csr_euen(CSR_EUEN_FPEN | CSR_EUEN_LSXEN | CSR_EUEN_LASXEN); - } else if (vcpu->arch.aux_inuse & KVM_LARCH_LSX) { + } else if (kvm_guest_has_lsx(&vcpu->arch)) { kvm_save_lsx(&vcpu->arch.fpu); - vcpu->arch.aux_inuse &= ~(KVM_LARCH_LSX | KVM_LARCH_FPU); trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_LSX); /* Disable LSX & FPU */ clear_csr_euen(CSR_EUEN_FPEN | CSR_EUEN_LSXEN); } else if (vcpu->arch.aux_inuse & KVM_LARCH_FPU) { kvm_save_fpu(&vcpu->arch.fpu); - vcpu->arch.aux_inuse &= ~KVM_LARCH_FPU; trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_FPU); /* Disable FPU */ clear_csr_euen(CSR_EUEN_FPEN); } + vcpu->arch.aux_inuse &= ~KVM_LARCH_FPU; + +done: kvm_lose_lbt(vcpu); preempt_enable(); @@ -1487,6 +1460,13 @@ void kvm_lose_fpu(struct kvm_vcpu *vcpu) int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq) { int intr = (int)irq->irq; + unsigned int vector = abs(intr); + + if (vector >= EXCCODE_INT_NUM) + return -EINVAL; + + if (!kvm_guest_has_msgint(&vcpu->arch) && (vector == INT_AVEC)) + return -EINVAL; if (intr > 0) kvm_queue_irq(vcpu, intr); diff --git a/arch/riscv/include/asm/kvm_aia.h b/arch/riscv/include/asm/kvm_aia.h index b04ecdd1a860..c67ec5ac0a14 100644 --- a/arch/riscv/include/asm/kvm_aia.h +++ b/arch/riscv/include/asm/kvm_aia.h @@ -79,7 +79,7 @@ struct kvm_vcpu_aia { #define irqchip_in_kernel(k) ((k)->arch.aia.in_kernel) -extern unsigned int kvm_riscv_aia_nr_hgei; +extern atomic_t kvm_riscv_aia_nr_hgei; extern unsigned int kvm_riscv_aia_max_ids; DECLARE_STATIC_KEY_FALSE(kvm_riscv_aia_available); #define kvm_riscv_aia_available() \ diff --git a/arch/riscv/include/asm/kvm_gstage.h b/arch/riscv/include/asm/kvm_gstage.h index 9c908432bc17..21e2019df0cf 100644 --- a/arch/riscv/include/asm/kvm_gstage.h +++ b/arch/riscv/include/asm/kvm_gstage.h @@ -54,6 +54,10 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage, struct kvm_mmu_memory_cache *pcache, const struct kvm_gstage_mapping *map); +bool kvm_riscv_gstage_try_update_pte(struct kvm_gstage *gstage, u32 level, + gpa_t addr, pte_t *ptep, + pte_t old_pte, pte_t new_pte); + int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage, struct kvm_mmu_memory_cache *pcache, gpa_t gpa, phys_addr_t hpa, unsigned long page_size, @@ -70,13 +74,13 @@ enum kvm_riscv_gstage_op { GSTAGE_OP_WP, /* Write-protect */ }; -void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, +bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op); -void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, +bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, gpa_t start, gpa_t size, bool may_block); -void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end); +bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end); void kvm_riscv_gstage_mode_detect(void); diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h index 75b0a951c1bc..60017ceec9d2 100644 --- a/arch/riscv/include/asm/kvm_host.h +++ b/arch/riscv/include/asm/kvm_host.h @@ -48,6 +48,8 @@ #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE +#define KVM_HAVE_MMU_RWLOCK + #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET) diff --git a/arch/riscv/include/asm/kvm_nacl.h b/arch/riscv/include/asm/kvm_nacl.h index 4124d5e06a0f..f45407bcaa26 100644 --- a/arch/riscv/include/asm/kvm_nacl.h +++ b/arch/riscv/include/asm/kvm_nacl.h @@ -60,9 +60,11 @@ int kvm_riscv_nacl_init(void); #ifdef CONFIG_32BIT #define lelong_to_cpu(__x) le32_to_cpu(__x) #define cpu_to_lelong(__x) cpu_to_le32(__x) +#define __lelong __le32 #else #define lelong_to_cpu(__x) le64_to_cpu(__x) #define cpu_to_lelong(__x) cpu_to_le64(__x) +#define __lelong __le64 #endif #define nacl_shmem() \ @@ -70,7 +72,7 @@ int kvm_riscv_nacl_init(void); #define nacl_scratch_read_long(__shmem, __offset) \ ({ \ - unsigned long *__p = (__shmem) + \ + __lelong *__p = (__shmem) + \ SBI_NACL_SHMEM_SCRATCH_OFFSET + \ (__offset); \ lelong_to_cpu(*__p); \ @@ -78,7 +80,7 @@ int kvm_riscv_nacl_init(void); #define nacl_scratch_write_long(__shmem, __offset, __val) \ do { \ - unsigned long *__p = (__shmem) + \ + __lelong *__p = (__shmem) + \ SBI_NACL_SHMEM_SCRATCH_OFFSET + \ (__offset); \ *__p = cpu_to_lelong(__val); \ @@ -87,7 +89,7 @@ do { \ #define nacl_scratch_write_longs(__shmem, __offset, __array, __count) \ do { \ unsigned int __i; \ - unsigned long *__p = (__shmem) + \ + __lelong *__p = (__shmem) + \ SBI_NACL_SHMEM_SCRATCH_OFFSET + \ (__offset); \ for (__i = 0; __i < (__count); __i++) \ @@ -168,7 +170,7 @@ __kvm_riscv_nacl_hfence(__shmem, \ #define nacl_csr_read(__shmem, __csr) \ ({ \ - unsigned long *__a = (__shmem) + SBI_NACL_SHMEM_CSR_OFFSET; \ + __lelong *__a = (__shmem) + SBI_NACL_SHMEM_CSR_OFFSET; \ lelong_to_cpu(__a[SBI_NACL_SHMEM_CSR_INDEX(__csr)]); \ }) @@ -176,7 +178,7 @@ __kvm_riscv_nacl_hfence(__shmem, \ do { \ void *__s = (__shmem); \ unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \ - unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \ + __lelong *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \ u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \ __a[__i] = cpu_to_lelong(__val); \ __b[__i >> 3] |= 1U << (__i & 0x7); \ @@ -186,7 +188,7 @@ do { \ ({ \ void *__s = (__shmem); \ unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \ - unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \ + __lelong *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \ u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \ unsigned long __r = lelong_to_cpu(__a[__i]); \ __a[__i] = cpu_to_lelong(__val); \ diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index 11c9886c3b70..5d4ec15584cf 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -112,7 +112,7 @@ do { \ _ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \ : "=&r" (__tmp) \ : "m" (*(ptr)) : : label); \ - (x) = (__typeof__(x))(unsigned long)__tmp; \ + (x) = (__force __typeof__(x))(unsigned long)__tmp; \ } while (0) #else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ #define __get_user_asm(insn, x, ptr, label) \ diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c index 5ec503288555..bafb009c5ce5 100644 --- a/arch/riscv/kvm/aia.c +++ b/arch/riscv/kvm/aia.c @@ -21,13 +21,15 @@ struct aia_hgei_control { raw_spinlock_t lock; + bool free_bitmap_initialized; unsigned long free_bitmap; struct kvm_vcpu *owners[BITS_PER_LONG]; + unsigned int nr_hgei; }; static DEFINE_PER_CPU(struct aia_hgei_control, aia_hgei); static int hgei_parent_irq; -unsigned int kvm_riscv_aia_nr_hgei; +atomic_t kvm_riscv_aia_nr_hgei; unsigned int kvm_riscv_aia_max_ids; DEFINE_STATIC_KEY_FALSE(kvm_riscv_aia_available); @@ -452,7 +454,7 @@ void kvm_riscv_aia_free_hgei(int cpu, int hgei) raw_spin_lock_irqsave(&hgctrl->lock, flags); - if (hgei > 0 && hgei <= kvm_riscv_aia_nr_hgei) { + if (hgei > 0 && hgei <= hgctrl->nr_hgei) { if (!(hgctrl->free_bitmap & BIT(hgei))) { hgctrl->free_bitmap |= BIT(hgei); hgctrl->owners[hgei] = NULL; @@ -486,26 +488,18 @@ static irqreturn_t hgei_interrupt(int irq, void *dev_id) static int aia_hgei_init(void) { - int cpu, rc; - struct irq_domain *domain; struct aia_hgei_control *hgctrl; + struct irq_domain *domain; + int cpu, rc; /* Initialize per-CPU guest external interrupt line management */ for_each_possible_cpu(cpu) { hgctrl = per_cpu_ptr(&aia_hgei, cpu); raw_spin_lock_init(&hgctrl->lock); - if (kvm_riscv_aia_nr_hgei) { - hgctrl->free_bitmap = - BIT(kvm_riscv_aia_nr_hgei + 1) - 1; - hgctrl->free_bitmap &= ~BIT(0); - } else - hgctrl->free_bitmap = 0; + hgctrl->free_bitmap_initialized = false; + hgctrl->free_bitmap = 0; } - /* Skip SGEI interrupt setup for zero guest external interrupts */ - if (!kvm_riscv_aia_nr_hgei) - goto skip_sgei_interrupt; - /* Find INTC irq domain */ domain = irq_find_matching_fwnode(riscv_get_intc_hwnode(), DOMAIN_BUS_ANY); @@ -529,25 +523,60 @@ static int aia_hgei_init(void) return rc; } -skip_sgei_interrupt: return 0; } static void aia_hgei_exit(void) { - /* Do nothing for zero guest external interrupts */ - if (!kvm_riscv_aia_nr_hgei) - return; - /* Free per-CPU SGEI interrupt */ free_percpu_irq(hgei_parent_irq, &aia_hgei); } void kvm_riscv_aia_enable(void) { + const struct imsic_global_config *gc; + const struct imsic_local_config *lc; + struct aia_hgei_control *hgctrl; + unsigned long flags; + int aia_nr_hgei; + if (!kvm_riscv_aia_available()) return; + gc = imsic_get_global_config(); + lc = (gc) ? this_cpu_ptr(gc->local) : NULL; + hgctrl = this_cpu_ptr(&aia_hgei); + + /* Figure-out number of bits in HGEIE */ + csr_write(CSR_HGEIE, -1UL); + hgctrl->nr_hgei = fls_long(csr_read(CSR_HGEIE)); + csr_write(CSR_HGEIE, 0); + if (hgctrl->nr_hgei) + hgctrl->nr_hgei--; + + /* + * Number of usable per-HART HGEI lines should be minimum of + * per-HART IMSIC guest files and number of bits in HGEIE. + */ + if (lc) + hgctrl->nr_hgei = min((ulong)hgctrl->nr_hgei, lc->nr_guest_files); + else + hgctrl->nr_hgei = 0; + + /* Update the number of IMSIC guest files across all HARTs */ + aia_nr_hgei = atomic_read(&kvm_riscv_aia_nr_hgei); + do { + if (aia_nr_hgei <= hgctrl->nr_hgei) + break; + } while (!atomic_try_cmpxchg(&kvm_riscv_aia_nr_hgei, &aia_nr_hgei, hgctrl->nr_hgei)); + + raw_spin_lock_irqsave(&hgctrl->lock, flags); + if (!hgctrl->free_bitmap_initialized) { + hgctrl->free_bitmap = (hgctrl->nr_hgei) ? GENMASK_ULL(hgctrl->nr_hgei, 1) : 0; + hgctrl->free_bitmap_initialized = true; + } + raw_spin_unlock_irqrestore(&hgctrl->lock, flags); + csr_write(CSR_HVICTL, aia_hvictl_value(false)); csr_write(CSR_HVIPRIO1, 0x0); csr_write(CSR_HVIPRIO2, 0x0); @@ -588,7 +617,7 @@ void kvm_riscv_aia_disable(void) raw_spin_lock_irqsave(&hgctrl->lock, flags); - for (i = 0; i <= kvm_riscv_aia_nr_hgei; i++) { + for (i = 0; i <= hgctrl->nr_hgei; i++) { vcpu = hgctrl->owners[i]; if (!vcpu) continue; @@ -628,26 +657,15 @@ int kvm_riscv_aia_init(void) return -ENODEV; gc = imsic_get_global_config(); - /* Figure-out number of bits in HGEIE */ - csr_write(CSR_HGEIE, -1UL); - kvm_riscv_aia_nr_hgei = fls_long(csr_read(CSR_HGEIE)); - csr_write(CSR_HGEIE, 0); - if (kvm_riscv_aia_nr_hgei) - kvm_riscv_aia_nr_hgei--; - - /* - * Number of usable HGEI lines should be minimum of per-HART - * IMSIC guest files and number of bits in HGEIE - */ + /* Set initial value of IMSIC guest files across all HARTs */ if (gc) - kvm_riscv_aia_nr_hgei = min((ulong)kvm_riscv_aia_nr_hgei, - gc->nr_guest_files); + atomic_set(&kvm_riscv_aia_nr_hgei, gc->nr_guest_files); else - kvm_riscv_aia_nr_hgei = 0; + atomic_set(&kvm_riscv_aia_nr_hgei, 0); /* Find number of guest MSI IDs */ kvm_riscv_aia_max_ids = IMSIC_MAX_ID; - if (gc && kvm_riscv_aia_nr_hgei) + if (gc) kvm_riscv_aia_max_ids = gc->nr_guest_ids + 1; /* Initialize guest external interrupt line management */ diff --git a/arch/riscv/kvm/aia_aplic.c b/arch/riscv/kvm/aia_aplic.c index 3464f3351df7..748107c347d9 100644 --- a/arch/riscv/kvm/aia_aplic.c +++ b/arch/riscv/kvm/aia_aplic.c @@ -35,7 +35,7 @@ struct aplic { u32 nr_irqs; u32 nr_words; - struct aplic_irq *irqs; + struct aplic_irq irqs[] __counted_by(nr_irqs); }; static u32 aplic_read_sourcecfg(struct aplic *aplic, u32 irq) @@ -581,7 +581,7 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm) return 0; /* Allocate APLIC global state */ - aplic = kzalloc_obj(*aplic); + aplic = kzalloc_flex(*aplic, irqs, kvm->arch.aia.nr_sources + 1); if (!aplic) return -ENOMEM; kvm->arch.aia.aplic_state = aplic; @@ -589,11 +589,6 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm) /* Setup APLIC IRQs */ aplic->nr_irqs = kvm->arch.aia.nr_sources + 1; aplic->nr_words = DIV_ROUND_UP(aplic->nr_irqs, 32); - aplic->irqs = kzalloc_objs(*aplic->irqs, aplic->nr_irqs); - if (!aplic->irqs) { - ret = -ENOMEM; - goto fail_free_aplic; - } for (i = 0; i < aplic->nr_irqs; i++) raw_spin_lock_init(&aplic->irqs[i].lock); @@ -606,7 +601,7 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm) &aplic->iodev); mutex_unlock(&kvm->slots_lock); if (ret) - goto fail_free_aplic_irqs; + goto fail_free_aplic; /* Setup default IRQ routing */ ret = kvm_riscv_setup_default_irq_routing(kvm, aplic->nr_irqs); @@ -619,8 +614,6 @@ fail_unreg_iodev: mutex_lock(&kvm->slots_lock); kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS, &aplic->iodev); mutex_unlock(&kvm->slots_lock); -fail_free_aplic_irqs: - kfree(aplic->irqs); fail_free_aplic: kvm->arch.aia.aplic_state = NULL; kfree(aplic); @@ -638,8 +631,6 @@ void kvm_riscv_aia_aplic_cleanup(struct kvm *kvm) kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS, &aplic->iodev); mutex_unlock(&kvm->slots_lock); - kfree(aplic->irqs); - kvm->arch.aia.aplic_state = NULL; kfree(aplic); } diff --git a/arch/riscv/kvm/aia_device.c b/arch/riscv/kvm/aia_device.c index 3d1e81e2a36b..be83c2d5fc30 100644 --- a/arch/riscv/kvm/aia_device.c +++ b/arch/riscv/kvm/aia_device.c @@ -71,7 +71,7 @@ static int aia_config(struct kvm *kvm, unsigned long type, * external interrupts (i.e. non-zero * VS-level IMSIC pages). */ - if (!kvm_riscv_aia_nr_hgei) + if (!atomic_read(&kvm_riscv_aia_nr_hgei)) return -EINVAL; break; default: @@ -628,7 +628,7 @@ void kvm_riscv_aia_init_vm(struct kvm *kvm) */ /* Initialize default values in AIA global context */ - aia->mode = (kvm_riscv_aia_nr_hgei) ? + aia->mode = (atomic_read(&kvm_riscv_aia_nr_hgei)) ? KVM_DEV_RISCV_AIA_MODE_AUTO : KVM_DEV_RISCV_AIA_MODE_EMUL; aia->nr_ids = kvm_riscv_aia_max_ids - 1; aia->nr_sources = 0; diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c index 8786f52cf65a..d38f5de0834c 100644 --- a/arch/riscv/kvm/aia_imsic.c +++ b/arch/riscv/kvm/aia_imsic.c @@ -683,6 +683,9 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu) unsigned long flags; bool ret = false; + if (!imsic) + return false; + /* * The IMSIC SW-file directly injects interrupt via hvip so * only check for interrupt when IMSIC VS-file is being used. @@ -722,6 +725,9 @@ void kvm_riscv_vcpu_aia_imsic_put(struct kvm_vcpu *vcpu) struct imsic *imsic = vcpu->arch.aia_context.imsic_state; unsigned long flags; + if (!imsic) + return; + if (!kvm_vcpu_is_blocking(vcpu)) return; @@ -738,6 +744,9 @@ void kvm_riscv_vcpu_aia_imsic_release(struct kvm_vcpu *vcpu) int old_vsfile_hgei, old_vsfile_cpu; struct imsic *imsic = vcpu->arch.aia_context.imsic_state; + if (!imsic) + return; + /* Read and clear IMSIC VS-file details */ write_lock_irqsave(&imsic->vsfile_lock, flags); old_vsfile_hgei = imsic->vsfile_hgei; diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c index d9fe8be2a151..c4c3b79567f1 100644 --- a/arch/riscv/kvm/gstage.c +++ b/arch/riscv/kvm/gstage.c @@ -123,6 +123,20 @@ static void gstage_tlb_flush(struct kvm_gstage *gstage, u32 level, gpa_t addr) gstage->vmid); } +bool kvm_riscv_gstage_try_update_pte(struct kvm_gstage *gstage, u32 level, + gpa_t addr, pte_t *ptep, + pte_t old_pte, pte_t new_pte) +{ + if (cmpxchg(&ptep->pte, pte_val(old_pte), pte_val(new_pte)) != + pte_val(old_pte)) + return false; + + if (pte_val(old_pte) != pte_val(new_pte)) + gstage_tlb_flush(gstage, level, addr); + + return true; +} + int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage, struct kvm_mmu_memory_cache *pcache, const struct kvm_gstage_mapping *map) @@ -168,17 +182,22 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage, static void kvm_riscv_gstage_update_pte_prot(struct kvm_gstage *gstage, u32 level, gpa_t addr, pte_t *ptep, pgprot_t prot) { - pte_t new_pte; + pte_t old_pte, new_pte; - if (pgprot_val(pte_pgprot(ptep_get(ptep))) == pgprot_val(prot)) - return; + for (;;) { + old_pte = ptep_get(ptep); + if (pgprot_val(pte_pgprot(old_pte)) == pgprot_val(prot)) + return; - new_pte = pfn_pte(pte_pfn(ptep_get(ptep)), prot); - new_pte = pte_mkdirty(new_pte); + new_pte = pfn_pte(pte_pfn(old_pte), prot); + new_pte = pte_mkdirty(new_pte); - set_pte(ptep, new_pte); + if (kvm_riscv_gstage_try_update_pte(gstage, level, addr, ptep, + old_pte, new_pte)) + return; - gstage_tlb_flush(gstage, level, addr); + cpu_relax(); + } } int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage, @@ -337,35 +356,36 @@ int kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage, return 0; } -void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, +bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op) { int i, ret; pte_t old_pte, *next_ptep; u32 next_ptep_level; unsigned long next_page_size, page_size; + bool flush = false; ret = gstage_level_to_page_size(gstage, ptep_level, &page_size); if (ret) - return; + return false; WARN_ON(addr & (page_size - 1)); if (!pte_val(ptep_get(ptep))) - return; + return false; if (ptep_level && !gstage_pte_leaf(ptep)) { next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); next_ptep_level = ptep_level - 1; ret = gstage_level_to_page_size(gstage, next_ptep_level, &next_page_size); if (ret) - return; + return false; if (op == GSTAGE_OP_CLEAR) set_pte(ptep, __pte(0)); for (i = 0; i < PTRS_PER_PTE; i++) - kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size, - &next_ptep[i], next_ptep_level, op); + flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size, + &next_ptep[i], next_ptep_level, op); if (op == GSTAGE_OP_CLEAR) put_page(virt_to_page(next_ptep)); } else { @@ -375,11 +395,13 @@ void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, else if (op == GSTAGE_OP_WP) set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); if (pte_val(*ptep) != pte_val(old_pte)) - gstage_tlb_flush(gstage, ptep_level, addr); + flush = true; } + + return flush; } -void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, +bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, gpa_t start, gpa_t size, bool may_block) { int ret; @@ -388,6 +410,7 @@ void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, bool found_leaf; unsigned long page_size; gpa_t addr = start, end = start + size; + bool flush = false; while (addr < end) { found_leaf = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_level); @@ -395,26 +418,32 @@ void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, if (ret) break; - if (!found_leaf) - goto next; - - if (!(addr & (page_size - 1)) && ((end - addr) >= page_size)) - kvm_riscv_gstage_op_pte(gstage, addr, ptep, - ptep_level, GSTAGE_OP_CLEAR); - -next: - addr += page_size; + if (!found_leaf) { + addr = ALIGN(addr + 1, page_size); + } else { + if (!(addr & (page_size - 1)) && ((end - addr) >= page_size)) + flush |= kvm_riscv_gstage_op_pte(gstage, addr, ptep, + ptep_level, GSTAGE_OP_CLEAR); + else { + WARN_ONCE(1, "Skip unmap range addr: %#llx, end: %#llx, page_size: %#lx\n", + addr, end, page_size); + } + + addr += page_size; + } /* * If the range is too large, release the kvm->mmu_lock * to prevent starvation and lockup detector warnings. */ if (!(gstage->flags & KVM_GSTAGE_FLAGS_LOCAL) && may_block && addr < end) - cond_resched_lock(&gstage->kvm->mmu_lock); + cond_resched_rwlock_write(&gstage->kvm->mmu_lock); } + + return flush; } -void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end) +bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end) { int ret; pte_t *ptep; @@ -422,6 +451,7 @@ void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end bool found_leaf; gpa_t addr = start; unsigned long page_size; + bool flush = false; while (addr < end) { found_leaf = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_level); @@ -429,15 +459,17 @@ void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end if (ret) break; - if (!found_leaf) - goto next; - - addr = ALIGN_DOWN(addr, page_size); - kvm_riscv_gstage_op_pte(gstage, addr, ptep, - ptep_level, GSTAGE_OP_WP); -next: - addr += page_size; + if (!found_leaf) { + addr = ALIGN(addr + 1, page_size); + } else { + addr = ALIGN_DOWN(addr, page_size); + flush |= kvm_riscv_gstage_op_pte(gstage, addr, ptep, + ptep_level, GSTAGE_OP_WP); + addr += page_size; + } } + + return flush; } void __init kvm_riscv_gstage_mode_detect(void) diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index cb8a65273c1f..0924c75100a2 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -168,10 +168,6 @@ static int __init riscv_kvm_init(void) kvm_info("VMID %ld bits available\n", kvm_riscv_gstage_vmid_bits()); - if (kvm_riscv_aia_available()) - kvm_info("AIA available with %d guest external interrupts\n", - kvm_riscv_aia_nr_hgei); - kvm_riscv_setup_vendor_features(); kvm_register_perf_callbacks(); @@ -182,6 +178,10 @@ static int __init riscv_kvm_init(void) return rc; } + if (kvm_riscv_aia_available()) + kvm_info("AIA available with %d guest external interrupts\n", + atomic_read(&kvm_riscv_aia_nr_hgei)); + return 0; } module_init(riscv_kvm_init); diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 2d3def024270..082f9b261733 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -23,13 +23,15 @@ static void mmu_wp_memory_region(struct kvm *kvm, int slot) phys_addr_t start = memslot->base_gfn << PAGE_SHIFT; phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; struct kvm_gstage gstage; + bool flush; kvm_riscv_gstage_init(&gstage, kvm); - spin_lock(&kvm->mmu_lock); - kvm_riscv_gstage_wp_range(&gstage, start, end); - spin_unlock(&kvm->mmu_lock); - kvm_flush_remote_tlbs_memslot(kvm, memslot); + write_lock(&kvm->mmu_lock); + flush = kvm_riscv_gstage_wp_range(&gstage, start, end); + write_unlock(&kvm->mmu_lock); + if (flush) + kvm_flush_remote_tlbs_memslot(kvm, memslot); } int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa, @@ -65,9 +67,9 @@ int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa, if (ret) goto out; - spin_lock(&kvm->mmu_lock); + write_lock(&kvm->mmu_lock); ret = kvm_riscv_gstage_set_pte(&gstage, &pcache, &map); - spin_unlock(&kvm->mmu_lock); + write_unlock(&kvm->mmu_lock); if (ret) goto out; @@ -82,12 +84,17 @@ out: void kvm_riscv_mmu_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long size) { struct kvm_gstage gstage; + bool flush; kvm_riscv_gstage_init(&gstage, kvm); - spin_lock(&kvm->mmu_lock); - kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); - spin_unlock(&kvm->mmu_lock); + write_lock(&kvm->mmu_lock); + flush = kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); + write_unlock(&kvm->mmu_lock); + + if (flush) + kvm_flush_remote_tlbs_range(kvm, gpa >> PAGE_SHIFT, + size >> PAGE_SHIFT); } void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, @@ -99,10 +106,14 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; struct kvm_gstage gstage; + bool flush; kvm_riscv_gstage_init(&gstage, kvm); - kvm_riscv_gstage_wp_range(&gstage, start, end); + flush = kvm_riscv_gstage_wp_range(&gstage, start, end); + if (flush) + kvm_flush_remote_tlbs_range(kvm, start >> PAGE_SHIFT, + (end - start) >> PAGE_SHIFT); } void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) @@ -128,12 +139,16 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, gpa_t gpa = slot->base_gfn << PAGE_SHIFT; phys_addr_t size = slot->npages << PAGE_SHIFT; struct kvm_gstage gstage; + bool flush; kvm_riscv_gstage_init(&gstage, kvm); - spin_lock(&kvm->mmu_lock); - kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); - spin_unlock(&kvm->mmu_lock); + write_lock(&kvm->mmu_lock); + flush = kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); + write_unlock(&kvm->mmu_lock); + if (flush) + kvm_flush_remote_tlbs_range(kvm, gpa >> PAGE_SHIFT, + size >> PAGE_SHIFT); } void kvm_arch_commit_memory_region(struct kvm *kvm, @@ -142,9 +157,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, enum kvm_mr_change change) { /* - * At this point memslot has been committed and there is an - * allocated dirty_bitmap[], dirty pages will be tracked while - * the memory slot is write protected. + * At this point memslot has been committed and dirty pages will be + * tracked while the memory slot is write protected. */ if (change != KVM_MR_DELETE && new->flags & KVM_MEM_LOG_DIRTY_PAGES) { if (kvm_dirty_log_manual_protect_and_init_set(kvm)) @@ -230,18 +244,20 @@ out: bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { struct kvm_gstage gstage; - bool mmu_locked; + bool flush; if (!kvm->arch.pgd) return false; + lockdep_assert_held_write(&kvm->mmu_lock); + kvm_riscv_gstage_init(&gstage, kvm); - mmu_locked = spin_trylock(&kvm->mmu_lock); - kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT, - (range->end - range->start) << PAGE_SHIFT, - range->may_block); - if (mmu_locked) - spin_unlock(&kvm->mmu_lock); + flush = kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT, + (range->end - range->start) << PAGE_SHIFT, + range->may_block); + if (flush) + kvm_flush_remote_tlbs_range(kvm, range->start, + range->end - range->start); return false; } @@ -286,7 +302,8 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) } static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot, - unsigned long hva) + unsigned long hva, + unsigned long map_size) { hva_t uaddr_start, uaddr_end; gpa_t gpa_start; @@ -300,8 +317,8 @@ static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot, /* * Pages belonging to memslots that don't have the same alignment - * within a PMD for userspace and GPA cannot be mapped with g-stage - * PMD entries, because we'll end up mapping the wrong pages. + * within a huge page for userspace and GPA cannot be mapped with + * g-stage block entries, because we'll end up mapping the wrong pages. * * Consider a layout like the following: * @@ -321,7 +338,7 @@ static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot, * e -> g * f -> h */ - if ((gpa_start & (PMD_SIZE - 1)) != (uaddr_start & (PMD_SIZE - 1))) + if ((gpa_start & (map_size - 1)) != (uaddr_start & (map_size - 1))) return false; /* @@ -336,7 +353,8 @@ static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot, * userspace_addr or the base_gfn, as both are equally aligned (per * the check above) and equally sized. */ - return (hva >= ALIGN(uaddr_start, PMD_SIZE)) && (hva < ALIGN_DOWN(uaddr_end, PMD_SIZE)); + return (hva >= ALIGN(uaddr_start, map_size)) && + (hva < ALIGN_DOWN(uaddr_end, map_size)); } static int get_hva_mapping_size(struct kvm *kvm, @@ -404,7 +422,7 @@ static unsigned long transparent_hugepage_adjust(struct kvm *kvm, * sure that the HVA and GPA are sufficiently aligned and that the * block map is contained within the memslot. */ - if (fault_supports_gstage_huge_mapping(memslot, hva)) { + if (fault_supports_gstage_huge_mapping(memslot, hva, PMD_SIZE)) { int sz; sz = get_hva_mapping_size(kvm, hva); @@ -421,20 +439,114 @@ static unsigned long transparent_hugepage_adjust(struct kvm *kvm, return PAGE_SIZE; } +static unsigned long hugetlb_mapping_size(struct kvm_memory_slot *memslot, + unsigned long hva, + unsigned long map_size) +{ + switch (map_size) { +#ifndef CONFIG_32BIT + case PUD_SIZE: + if (fault_supports_gstage_huge_mapping(memslot, hva, PUD_SIZE)) + return PUD_SIZE; + fallthrough; +#endif + case PMD_SIZE: + if (fault_supports_gstage_huge_mapping(memslot, hva, PMD_SIZE)) + return PMD_SIZE; + fallthrough; + case PAGE_SIZE: + return PAGE_SIZE; + default: + return map_size; + } +} + +static bool kvm_riscv_mmu_dirty_log_write_fault_fast(struct kvm *kvm, + struct kvm_memory_slot *memslot, + gpa_t gpa, + struct kvm_gstage_mapping *out_map) +{ + struct kvm_gstage gstage; + unsigned long mmu_seq; + pte_t old_pte, new_pte; + pte_t *ptep; + gfn_t gfn = gpa >> PAGE_SHIFT; + u32 ptep_level; + bool dirty_marked = false; + bool ret; + + kvm_riscv_gstage_init(&gstage, kvm); + mmu_seq = kvm->mmu_invalidate_seq; + + read_lock(&kvm->mmu_lock); + + if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) { + ret = false; + goto out_unlock; + } + + if (!kvm_riscv_gstage_get_leaf(&gstage, gpa, &ptep, &ptep_level) || + ptep_level) { + ret = false; + goto out_unlock; + } + + for (;;) { + old_pte = ptep_get(ptep); + if (!(pte_val(old_pte) & _PAGE_LEAF)) { + ret = false; + break; + } + + if (!dirty_marked) { + mark_page_dirty_in_slot(kvm, memslot, gfn); + dirty_marked = true; + } + + if ((pte_val(old_pte) & (_PAGE_WRITE | _PAGE_DIRTY)) == + (_PAGE_WRITE | _PAGE_DIRTY)) { + new_pte = old_pte; + ret = true; + break; + } + + new_pte = pte_mkdirty(pte_mkwrite_novma(old_pte)); + + if (kvm_riscv_gstage_try_update_pte(&gstage, ptep_level, gpa, + ptep, old_pte, new_pte)) { + ret = true; + break; + } + cpu_relax(); + } + +out_unlock: + read_unlock(&kvm->mmu_lock); + + if (ret) { + out_map->addr = gpa & PAGE_MASK; + out_map->level = 0; + out_map->pte = new_pte; + } + + return ret; +} + int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, gpa_t gpa, unsigned long hva, bool is_write, struct kvm_gstage_mapping *out_map) { int ret; kvm_pfn_t hfn; + bool is_hugetlb; bool writable; short vma_pageshift; gfn_t gfn = gpa >> PAGE_SHIFT; struct vm_area_struct *vma; struct kvm *kvm = vcpu->kvm; struct kvm_mmu_memory_cache *pcache = &vcpu->arch.mmu_page_cache; - bool logging = (memslot->dirty_bitmap && - !(memslot->flags & KVM_MEM_READONLY)) ? true : false; + bool logging = kvm_slot_dirty_track_enabled(memslot) && + !(memslot->flags & KVM_MEM_READONLY); unsigned long vma_pagesize, mmu_seq; struct kvm_gstage gstage; struct page *page; @@ -444,6 +556,10 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, /* Setup initial state of output mapping */ memset(out_map, 0, sizeof(*out_map)); + if (is_write && logging && + kvm_riscv_mmu_dirty_log_write_fault_fast(kvm, memslot, gpa, out_map)) + return 0; + /* We need minimum second+third level pages */ ret = kvm_mmu_topup_memory_cache(pcache, kvm->arch.pgd_levels); if (ret) { @@ -460,16 +576,23 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, return -EFAULT; } - if (is_vm_hugetlb_page(vma)) + is_hugetlb = is_vm_hugetlb_page(vma); + if (is_hugetlb) vma_pageshift = huge_page_shift(hstate_vma(vma)); else vma_pageshift = PAGE_SHIFT; vma_pagesize = 1ULL << vma_pageshift; if (logging || (vma->vm_flags & VM_PFNMAP)) vma_pagesize = PAGE_SIZE; + else if (is_hugetlb) + vma_pagesize = hugetlb_mapping_size(memslot, hva, vma_pagesize); + /* + * For hugetlb mappings, vma_pagesize might have been reduced from the + * VMA size to a smaller safe mapping size. + */ if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE) - gfn = (gpa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT; + gfn = ALIGN_DOWN(gpa, vma_pagesize) >> PAGE_SHIFT; /* * Read mmu_invalidate_seq so that KVM can detect if the results of @@ -506,13 +629,17 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, if (logging && !is_write) writable = false; - spin_lock(&kvm->mmu_lock); + write_lock(&kvm->mmu_lock); if (mmu_invalidate_retry(kvm, mmu_seq)) goto out_unlock; - /* Check if we are backed by a THP and thus use block mapping if possible */ - if (!logging && (vma_pagesize == PAGE_SIZE)) + /* + * Check if we are backed by a THP and thus use block mapping if + * possible. Hugetlb mappings already selected their target size above, + * so do not promote them through the THP helper. + */ + if (!logging && !is_hugetlb && vma_pagesize == PAGE_SIZE) vma_pagesize = transparent_hugepage_adjust(kvm, memslot, hva, &hfn, &gpa); if (writable) { @@ -529,7 +656,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, out_unlock: kvm_release_faultin_page(kvm, page, ret && ret != -EEXIST, writable); - spin_unlock(&kvm->mmu_lock); + write_unlock(&kvm->mmu_lock); return ret; } @@ -557,18 +684,22 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm) { struct kvm_gstage gstage; void *pgd = NULL; + bool flush = false; - spin_lock(&kvm->mmu_lock); + write_lock(&kvm->mmu_lock); if (kvm->arch.pgd) { kvm_riscv_gstage_init(&gstage, kvm); - kvm_riscv_gstage_unmap_range(&gstage, 0UL, + flush = kvm_riscv_gstage_unmap_range(&gstage, 0UL, kvm_riscv_gstage_gpa_size(kvm->arch.pgd_levels), false); pgd = READ_ONCE(kvm->arch.pgd); kvm->arch.pgd = NULL; kvm->arch.pgd_phys = 0; kvm->arch.pgd_levels = 0; } - spin_unlock(&kvm->mmu_lock); + write_unlock(&kvm->mmu_lock); + + if (flush) + kvm_flush_remote_tlbs(kvm); if (pgd) free_pages((unsigned long)pgd, get_order(kvm_riscv_gstage_pgd_size)); diff --git a/arch/riscv/kvm/nacl.c b/arch/riscv/kvm/nacl.c index 08a95ad9ada2..6f9f8963e9dd 100644 --- a/arch/riscv/kvm/nacl.c +++ b/arch/riscv/kvm/nacl.c @@ -20,7 +20,7 @@ void __kvm_riscv_nacl_hfence(void *shmem, unsigned long page_count) { int i, ent = -1, try_count = 5; - unsigned long *entp; + __lelong *entp; again: for (i = 0; i < SBI_NACL_SHMEM_HFENCE_ENTRY_MAX; i++) { diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index a73690eda84b..cf6e231e76e2 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -538,6 +538,7 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, else vcpu->guest_debug = 0; + kvm_riscv_vcpu_config_guest_debug(vcpu); return 0; } diff --git a/arch/riscv/kvm/vcpu_sbi_fwft.c b/arch/riscv/kvm/vcpu_sbi_fwft.c index 2eab15339694..ab39ac464ffd 100644 --- a/arch/riscv/kvm/vcpu_sbi_fwft.c +++ b/arch/riscv/kvm/vcpu_sbi_fwft.c @@ -36,6 +36,16 @@ struct kvm_sbi_fwft_feature { bool (*supported)(struct kvm_vcpu *vcpu); /** + * @init: Probe and initialize the feature on the vcpu + * + * This callback is optional. If provided, it will be called during + * vcpu initialization to probe the feature availability and perform + * any necessary initialization. Returns true if the feature is supported + * and initialized successfully, false otherwise. + */ + bool (*init)(struct kvm_vcpu *vcpu); + + /** * @reset: Reset the feature value irrespective whether feature is supported or not * * This callback is mandatory @@ -131,19 +141,30 @@ static long kvm_sbi_fwft_get_misaligned_delegation(struct kvm_vcpu *vcpu, static bool try_to_set_pmm(unsigned long value) { + unsigned long prev; + bool ret; + + prev = csr_read_clear(CSR_HENVCFG, ENVCFG_PMM); csr_set(CSR_HENVCFG, value); - return (csr_read_clear(CSR_HENVCFG, ENVCFG_PMM) & ENVCFG_PMM) == value; + ret = (csr_read_clear(CSR_HENVCFG, ENVCFG_PMM) & ENVCFG_PMM) == value; + csr_write(CSR_HENVCFG, prev); + + return ret; } static bool kvm_sbi_fwft_pointer_masking_pmlen_supported(struct kvm_vcpu *vcpu) { - struct kvm_sbi_fwft *fwft = vcpu_to_fwft(vcpu); + return riscv_isa_extension_available(vcpu->arch.isa, SMNPM); +} - if (!riscv_isa_extension_available(vcpu->arch.isa, SMNPM)) - return false; +static bool kvm_sbi_fwft_pointer_masking_pmlen_init(struct kvm_vcpu *vcpu) +{ + struct kvm_sbi_fwft *fwft = vcpu_to_fwft(vcpu); + preempt_disable(); fwft->have_vs_pmlen_7 = try_to_set_pmm(ENVCFG_PMM_PMLEN_7); fwft->have_vs_pmlen_16 = try_to_set_pmm(ENVCFG_PMM_PMLEN_16); + preempt_enable(); return fwft->have_vs_pmlen_7 || fwft->have_vs_pmlen_16; } @@ -231,6 +252,7 @@ static const struct kvm_sbi_fwft_feature features[] = { .first_reg_num = offsetof(struct kvm_riscv_sbi_fwft, pointer_masking.enable) / sizeof(unsigned long), .supported = kvm_sbi_fwft_pointer_masking_pmlen_supported, + .init = kvm_sbi_fwft_pointer_masking_pmlen_init, .reset = kvm_sbi_fwft_reset_pointer_masking_pmlen, .set = kvm_sbi_fwft_set_pointer_masking_pmlen, .get = kvm_sbi_fwft_get_pointer_masking_pmlen, @@ -281,6 +303,8 @@ static int kvm_fwft_get_feature(struct kvm_vcpu *vcpu, u32 feature, if (!tconf->supported || !tconf->enabled) return SBI_ERR_NOT_SUPPORTED; + else if (tconf->feature->supported && !tconf->feature->supported(vcpu)) + return SBI_ERR_NOT_SUPPORTED; *conf = tconf; @@ -365,6 +389,9 @@ static int kvm_sbi_ext_fwft_init(struct kvm_vcpu *vcpu) else conf->supported = true; + if (conf->supported && feature->init) + conf->supported = feature->init(vcpu); + conf->enabled = conf->supported; conf->feature = feature; } @@ -408,6 +435,8 @@ static unsigned long kvm_sbi_ext_fwft_get_reg_count(struct kvm_vcpu *vcpu) conf = kvm_sbi_fwft_get_config(vcpu, feature->id); if (!conf || !conf->supported) continue; + else if (conf->feature->supported && !conf->feature->supported(vcpu)) + continue; ret++; } @@ -430,6 +459,8 @@ static int kvm_sbi_ext_fwft_get_reg_id(struct kvm_vcpu *vcpu, int index, u64 *re conf = kvm_sbi_fwft_get_config(vcpu, feature->id); if (!conf || !conf->supported) continue; + else if (conf->feature->supported && !conf->feature->supported(vcpu)) + continue; if (index == idx) { *reg_id = KVM_REG_RISCV | @@ -465,6 +496,8 @@ static int kvm_sbi_ext_fwft_get_reg(struct kvm_vcpu *vcpu, unsigned long reg_num conf = kvm_sbi_fwft_get_config(vcpu, feature->id); if (!conf || !conf->supported) return -ENOENT; + else if (conf->feature->supported && !conf->feature->supported(vcpu)) + return -ENOENT; switch (reg_num - feature->first_reg_num) { case 0: @@ -502,6 +535,8 @@ static int kvm_sbi_ext_fwft_set_reg(struct kvm_vcpu *vcpu, unsigned long reg_num conf = kvm_sbi_fwft_get_config(vcpu, feature->id); if (!conf || !conf->supported) return -ENOENT; + else if (conf->feature->supported && !conf->feature->supported(vcpu)) + return -ENOENT; switch (reg_num - feature->first_reg_num) { case 0: @@ -521,6 +556,7 @@ static int kvm_sbi_ext_fwft_set_reg(struct kvm_vcpu *vcpu, unsigned long reg_num break; case 2: ret = conf->feature->set(vcpu, conf, true, value); + vcpu->arch.csr_dirty = true; break; default: return -ENOENT; diff --git a/arch/riscv/kvm/vcpu_sbi_system.c b/arch/riscv/kvm/vcpu_sbi_system.c index c6f7e609ac79..6f64a59e5d3c 100644 --- a/arch/riscv/kvm/vcpu_sbi_system.c +++ b/arch/riscv/kvm/vcpu_sbi_system.c @@ -35,6 +35,20 @@ static int kvm_sbi_ext_susp_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, return 0; } + /* + * Check that all other vCPUs are stopped before entering + * system suspend. + * + * There is a known TOCTOU race here: a concurrent HSM + * HART_START on another vCPU can start a vCPU after it + * has already passed this check, violating the invariant. + * + * We do not fix this because: + * 1. Triggering the race requires a pathological guest. + * 2. Only guest state is at risk, not host integrity. + * 3. Userspace can double-check vCPU states before + * proceeding with suspend. + */ kvm_for_each_vcpu(i, tmp, vcpu->kvm) { if (tmp == vcpu) continue; diff --git a/arch/riscv/kvm/vcpu_timer.c b/arch/riscv/kvm/vcpu_timer.c index 9817ff802821..ae53133c7ab0 100644 --- a/arch/riscv/kvm/vcpu_timer.c +++ b/arch/riscv/kvm/vcpu_timer.c @@ -231,7 +231,7 @@ int kvm_riscv_vcpu_set_reg_timer(struct kvm_vcpu *vcpu, break; case KVM_REG_RISCV_TIMER_REG(state): if (reg_val == KVM_RISCV_TIMER_STATE_ON) - ret = kvm_riscv_vcpu_timer_next_event(vcpu, reg_val); + ret = kvm_riscv_vcpu_timer_next_event(vcpu, t->next_cycles); else ret = kvm_riscv_vcpu_timer_cancel(t); break; diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h index 8a4f4a39f7a2..eaa34c5bd3c1 100644 --- a/arch/s390/include/asm/kvm_host.h +++ b/arch/s390/include/asm/kvm_host.h @@ -359,7 +359,7 @@ struct kvm_s390_float_interrupt { struct kvm_s390_mchk_info mchk; struct kvm_s390_ext_info srv_signal; int last_sleep_cpu; - struct mutex ais_lock; + spinlock_t ais_lock; u8 simm; u8 nimm; }; @@ -448,6 +448,12 @@ struct kvm_vcpu_arch { struct kvm_vm_stat { struct kvm_vm_stat_generic generic; u64 inject_io; + u64 io_390_adapter_map; + u64 io_390_adapter_unmap; + u64 io_390_inatomic; + u64 io_flic_inject_airq; + u64 io_set_adapter_int; + u64 io_390_inatomic_no_inject; u64 inject_float_mchk; u64 inject_pfault_done; u64 inject_service_signal; @@ -479,6 +485,9 @@ struct s390_io_adapter { bool masked; bool swap; bool suppressible; + spinlock_t maps_lock; + struct list_head maps; + unsigned int nr_maps; }; #define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8) @@ -504,6 +513,18 @@ struct kvm_s390_cpu_model { struct kvm_s390_vm_cpu_uv_feat uv_feat_guest; }; +#define S390_ARCH_FAC_FORMAT_2 2 +struct kvm_s390_flcb2 { + union { + struct { + u8 reserved0[7]; + u8 length; + }; + u64 header_val; + }; + u64 facilities[S390_ARCH_FAC_LIST_SIZE_U64]; +}; + typedef int (*crypto_hook)(struct kvm_vcpu *vcpu); struct kvm_s390_crypto { diff --git a/arch/s390/include/asm/sclp.h b/arch/s390/include/asm/sclp.h index d928a9ddfe40..f2d490558054 100644 --- a/arch/s390/include/asm/sclp.h +++ b/arch/s390/include/asm/sclp.h @@ -105,6 +105,7 @@ struct sclp_info { unsigned char has_aisii : 1; unsigned char has_aeni : 1; unsigned char has_aisi : 1; + unsigned char has_astfleie2 : 1; unsigned int ibc; unsigned int mtid; unsigned int mtid_cp; diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h index 60345dd2cba2..4192769b5ce0 100644 --- a/arch/s390/include/uapi/asm/kvm.h +++ b/arch/s390/include/uapi/asm/kvm.h @@ -444,6 +444,7 @@ struct kvm_s390_vm_cpu_machine { #define KVM_S390_VM_CPU_FEAT_PFMFI 11 #define KVM_S390_VM_CPU_FEAT_SIGPIF 12 #define KVM_S390_VM_CPU_FEAT_KSS 13 +#define KVM_S390_VM_CPU_FEAT_ASTFLEIE2 14 struct kvm_s390_vm_cpu_feat { __u64 feat[16]; }; diff --git a/arch/s390/kvm/Kconfig b/arch/s390/kvm/Kconfig index 5b835bc6a194..8d3ee17a1bcb 100644 --- a/arch/s390/kvm/Kconfig +++ b/arch/s390/kvm/Kconfig @@ -30,6 +30,7 @@ config KVM select KVM_VFIO select VIRT_XFER_TO_GUEST_WORK select KVM_MMU_LOCKLESS_AGING + select KVM_GENERIC_PRE_FAULT_MEMORY help Support hosting paravirtualized guest machines using the SIE virtualization capability on the mainframe. This should work diff --git a/arch/s390/kvm/dat.c b/arch/s390/kvm/dat.c index 4a41c0247ffa..a4fe664f65ee 100644 --- a/arch/s390/kvm/dat.c +++ b/arch/s390/kvm/dat.c @@ -45,7 +45,7 @@ int kvm_s390_mmu_cache_topup(struct kvm_s390_mmu_cache *mc) mc->pts[mc->n_pts] = o; } for ( ; mc->n_rmaps < KVM_S390_MMU_CACHE_N_RMAPS; mc->n_rmaps++) { - o = kzalloc_obj(*mc->rmaps[0], GFP_KERNEL_ACCOUNT); + o = kzalloc_obj(struct vsie_rmap, GFP_KERNEL_ACCOUNT); if (!o) return -ENOMEM; mc->rmaps[mc->n_rmaps] = o; diff --git a/arch/s390/kvm/dat.h b/arch/s390/kvm/dat.h index 873e13ac5a27..fad605305e05 100644 --- a/arch/s390/kvm/dat.h +++ b/arch/s390/kvm/dat.h @@ -501,6 +501,7 @@ struct guest_fault { bool write_attempt; /* Write access attempted */ bool attempt_pfault; /* Attempt a pfault first */ bool valid; /* This entry contains valid data */ + bool crste_region3; /* Whether crstep refers to a region3 entry */ void (*callback)(struct guest_fault *f); void *priv; }; diff --git a/arch/s390/kvm/gaccess.c b/arch/s390/kvm/gaccess.c index 20e28b183c1a..36102b2727fb 100644 --- a/arch/s390/kvm/gaccess.c +++ b/arch/s390/kvm/gaccess.c @@ -1419,8 +1419,8 @@ edat_applies: return kvm_s390_get_guest_page(kvm, entries + LEVEL_MEM, table.pte.pfra, wr); } -static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union pte *ptep, - struct guest_fault *f, bool p) +static int _do_shadow_pte(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gpa_t raddr, + union pte *ptep_h, union pte *ptep, struct guest_fault *f, bool p) { union pgste pgste; union pte newpte; @@ -1430,7 +1430,7 @@ static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union lockdep_assert_held(&sg->parent->children_lock); scoped_guard(spinlock, &sg->host_to_rmap_lock) - rc = gmap_insert_rmap(sg, f->gfn, gpa_to_gfn(raddr), TABLE_TYPE_PAGE_TABLE); + rc = gmap_insert_rmap(mc, sg, f->gfn, gpa_to_gfn(raddr), TABLE_TYPE_PAGE_TABLE); if (rc) return rc; @@ -1462,8 +1462,8 @@ static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union return 0; } -static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, union crste *table, - struct guest_fault *f, bool p) +static int _do_shadow_crste(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gpa_t raddr, + union crste *host, union crste *table, struct guest_fault *f, bool p) { union crste newcrste, oldcrste; unsigned long mask; @@ -1476,7 +1476,7 @@ static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, uni mask = is_pmd(*table) ? _SEGMENT_FR_MASK : _REGION3_FR_MASK; r_gfn = gpa_to_gfn(raddr) & mask; scoped_guard(spinlock, &sg->host_to_rmap_lock) - rc = gmap_insert_rmap(sg, f->gfn & mask, r_gfn, host->h.tt); + rc = gmap_insert_rmap(mc, sg, f->gfn & mask, r_gfn, host->h.tt); if (rc) return rc; @@ -1578,39 +1578,52 @@ real_address_space: if (KVM_BUG_ON(l > TABLE_TYPE_REGION3, sg->kvm)) return -EFAULT; if (l == TABLE_TYPE_PAGE_TABLE) - return _do_shadow_pte(sg, saddr, ptep_h, ptep, entries + LEVEL_MEM, w->p); - return _do_shadow_crste(sg, saddr, host, table, entries + LEVEL_MEM, w->p); + return _do_shadow_pte(mc, sg, saddr, ptep_h, ptep, entries + LEVEL_MEM, w->p); + return _do_shadow_crste(mc, sg, saddr, host, table, entries + LEVEL_MEM, w->p); } -static inline int _gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, gpa_t saddr, - unsigned long seq, struct pgtwalk *walk) +static inline int ___gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, gpa_t saddr, + unsigned long seq, struct pgtwalk *walk) { struct gmap *parent; int rc; - if (kvm_s390_array_needs_retry_unsafe(vcpu->kvm, seq, walk->raw_entries)) + if (kvm_s390_array_needs_retry_safe(vcpu->kvm, seq, walk->raw_entries)) return -EAGAIN; -again: - rc = kvm_s390_mmu_cache_topup(vcpu->arch.mc); - if (rc) - return rc; - scoped_guard(read_lock, &vcpu->kvm->mmu_lock) { - if (kvm_s390_array_needs_retry_safe(vcpu->kvm, seq, walk->raw_entries)) - return -EAGAIN; - parent = READ_ONCE(sg->parent); - if (!parent) + parent = READ_ONCE(sg->parent); + if (!parent) + return -EAGAIN; + scoped_guard(spinlock, &parent->children_lock) { + if (READ_ONCE(sg->parent) != parent) return -EAGAIN; - scoped_guard(spinlock, &parent->children_lock) { - if (READ_ONCE(sg->parent) != parent) - return -EAGAIN; - sg->invalidated = false; - rc = _gaccess_do_shadow(vcpu->arch.mc, sg, saddr, walk); - } - if (rc == -ENOMEM) - goto again; - if (!rc) - kvm_s390_release_faultin_array(vcpu->kvm, walk->raw_entries, false); + sg->invalidated = false; + rc = _gaccess_do_shadow(vcpu->arch.mc, sg, saddr, walk); } + if (!rc) + kvm_s390_release_faultin_array(vcpu->kvm, walk->raw_entries, false); + return rc; +} + +static inline int _gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, gpa_t saddr, + unsigned long seq, struct pgtwalk *walk) +{ + int rc; + + if (kvm_s390_array_needs_retry_unsafe(vcpu->kvm, seq, walk->raw_entries)) + return -EAGAIN; + + do { + rc = kvm_s390_mmu_cache_topup(vcpu->arch.mc); + if (rc) + return rc; + rc = radix_tree_preload(GFP_KERNEL); + if (rc) + return rc; + scoped_guard(read_lock, &vcpu->kvm->mmu_lock) + rc = ___gaccess_shadow_fault(vcpu, sg, saddr, seq, walk); + radix_tree_preload_end(); + } while (rc == -ENOMEM); + return rc; } diff --git a/arch/s390/kvm/gmap.c b/arch/s390/kvm/gmap.c index 52d55ddea8d4..77829f787dc7 100644 --- a/arch/s390/kvm/gmap.c +++ b/arch/s390/kvm/gmap.c @@ -105,6 +105,11 @@ static void gmap_add_child(struct gmap *parent, struct gmap *child) else clear_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &child->flags); + if (test_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &parent->flags)) + set_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &child->flags); + else + clear_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &child->flags); + if (kvm_is_ucontrol(parent->kvm)) clear_bit(GMAP_FLAG_OWNS_PAGETABLES, &child->flags); list_add(&child->list, &parent->children); @@ -543,6 +548,7 @@ static int gmap_handle_minor_crste_fault(struct gmap *gmap, struct guest_fault * f->pfn = PHYS_PFN(large_crste_to_phys(oldcrste, f->gfn)); f->writable = oldcrste.s.fc1.w; + f->crste_region3 = is_pud(oldcrste); /* Appropriate permissions already (race with another handler), nothing to do. */ if (!oldcrste.h.i && !(f->write_attempt && oldcrste.h.p)) return 0; @@ -630,10 +636,27 @@ int gmap_try_fixup_minor(struct gmap *gmap, struct guest_fault *fault) return rc; } +/** + * gmap_2g_allowed() - Check whether a 2G hugepage is allowed. + * @gmap: The gmap of the guest. + * @f: Describes the fault that is being resolved. + * @slot: The memslot the faulting address belongs to. + * + * The function checks whether the GMAP_FLAG_ALLOW_HPAGE_2G flag is set for + * @gmap, whether the offset of the address in the 2G virtual frame is the + * same as the offset in the physical 2G frame, and finally whether the whole + * 2G page would fit in the given memslot. + * + * Return: true if a 2G hugepage is allowed to back the faulting address, false + * otherwise. + */ static inline bool gmap_2g_allowed(struct gmap *gmap, struct guest_fault *f, struct kvm_memory_slot *slot) { - return false; + return test_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &gmap->flags) && + !((f->gfn ^ f->pfn) & ~_REGION3_FR_MASK) && + slot->base_gfn <= ALIGN_DOWN(f->gfn, _PAGES_PER_REGION3) && + slot->base_gfn + slot->npages >= ALIGN(f->gfn + 1, _PAGES_PER_REGION3); } /** @@ -702,6 +725,7 @@ static int _gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, int leve if (oldval.val != _CRSTE_EMPTY(oldval.h.tt).val && crste_origin_large(oldval) != crste_origin_large(newval)) return -EAGAIN; + f->crste_region3 = is_pud(newval); } while (!gmap_crstep_xchg_atomic(gmap, f->crstep, oldval, newval, f->gfn)); if (f->callback) f->callback(f); @@ -1000,7 +1024,8 @@ int gmap_pv_destroy_range(struct gmap *gmap, gfn_t start, gfn_t end, bool interr return 0; } -int gmap_insert_rmap(struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, int level) +int gmap_insert_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gfn, + gfn_t r_gfn, int level) { struct vsie_rmap *rmap __free(kvfree) = NULL; struct vsie_rmap *temp; @@ -1010,7 +1035,7 @@ int gmap_insert_rmap(struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, int level) KVM_BUG_ON(!is_shadow(sg), sg->kvm); lockdep_assert_held(&sg->host_to_rmap_lock); - rmap = kzalloc_obj(*rmap, GFP_ATOMIC); + rmap = kvm_s390_mmu_cache_alloc_rmap(mc); if (!rmap) return -ENOMEM; @@ -1057,7 +1082,7 @@ int gmap_protect_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gf if (level <= TABLE_TYPE_REGION1) { bitmask = -1UL << (8 + 11 * level); scoped_guard(spinlock, &sg->host_to_rmap_lock) - rc = gmap_insert_rmap(sg, p_gfn, r_gfn & bitmask, level); + rc = gmap_insert_rmap(mc, sg, p_gfn, r_gfn & bitmask, level); } if (rc) return rc; diff --git a/arch/s390/kvm/gmap.h b/arch/s390/kvm/gmap.h index 5374f21aaf8d..1c040472f56d 100644 --- a/arch/s390/kvm/gmap.h +++ b/arch/s390/kvm/gmap.h @@ -100,7 +100,8 @@ int gmap_ucas_map(struct gmap *gmap, gfn_t p_gfn, gfn_t c_gfn, unsigned long cou void gmap_ucas_unmap(struct gmap *gmap, gfn_t c_gfn, unsigned long count); int gmap_enable_skeys(struct gmap *gmap); int gmap_pv_destroy_range(struct gmap *gmap, gfn_t start, gfn_t end, bool interruptible); -int gmap_insert_rmap(struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, int level); +int gmap_insert_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gfn, + gfn_t r_gfn, int level); int gmap_protect_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, kvm_pfn_t pfn, int level, bool wr); void gmap_set_cmma_all_dirty(struct gmap *gmap); @@ -279,7 +280,16 @@ static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, unio gmap_handle_vsie_unshadow_event(gmap, gfn); else _gmap_handle_vsie_unshadow_event(gmap, gfn); - dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, gfn, gmap->asce); + if (!dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, gfn, gmap->asce)) + return false; + /* + * Return false even if the swap was successful, as it only + * indicates that the best effort clearing of the vsie_notif + * bit was successful. The caller will have to try again + * regardless, since the desired value has not been set. + * This pointless check is needed to silence a potential + * __must_check warning. + */ return false; } if (!oldcrste.s.fc1.d && newcrste.s.fc1.d && !newcrste.s.fc1.s) diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c index 39aff324203e..1980df61ef30 100644 --- a/arch/s390/kvm/intercept.c +++ b/arch/s390/kvm/intercept.c @@ -517,8 +517,9 @@ static int handle_pv_spx(struct kvm_vcpu *vcpu) static int handle_pv_sclp(struct kvm_vcpu *vcpu) { struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int; + unsigned long flags; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); /* * 2 cases: * a: an sccb answering interrupt was already pending or in flight. @@ -534,7 +535,7 @@ static int handle_pv_sclp(struct kvm_vcpu *vcpu) fi->srv_signal.ext_params |= 0x43000; set_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs); clear_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return 0; } diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c index 3bcdbbbb6891..9e3e6b0d72ad 100644 --- a/arch/s390/kvm/interrupt.c +++ b/arch/s390/kvm/interrupt.c @@ -624,8 +624,9 @@ static int __must_check __deliver_machine_check(struct kvm_vcpu *vcpu) struct kvm_s390_mchk_info mchk = {}; int deliver = 0; int rc = 0; + unsigned long flags; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); spin_lock(&li->lock); if (test_bit(IRQ_PEND_MCHK_EX, &li->pending_irqs) || test_bit(IRQ_PEND_MCHK_REP, &li->pending_irqs)) { @@ -654,7 +655,7 @@ static int __must_check __deliver_machine_check(struct kvm_vcpu *vcpu) deliver = 1; } spin_unlock(&li->lock); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); if (deliver) { VCPU_EVENT(vcpu, 3, "deliver: machine check mcic 0x%llx", @@ -941,11 +942,12 @@ static int __must_check __deliver_service(struct kvm_vcpu *vcpu) { struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int; struct kvm_s390_ext_info ext; + unsigned long flags; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); if (test_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs) || !(test_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs))) { - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return 0; } ext = fi->srv_signal; @@ -954,7 +956,7 @@ static int __must_check __deliver_service(struct kvm_vcpu *vcpu) clear_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs); if (kvm_s390_pv_cpu_is_protected(vcpu)) set_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); if (!ext.ext_params) return 0; @@ -972,17 +974,18 @@ static int __must_check __deliver_service_ev(struct kvm_vcpu *vcpu) { struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int; struct kvm_s390_ext_info ext; + unsigned long flags; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); if (!(test_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs))) { - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return 0; } ext = fi->srv_signal; /* only clear the event bits */ fi->srv_signal.ext_params &= ~SCCB_EVENT_PENDING; clear_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); VCPU_EVENT(vcpu, 4, "%s", "deliver: sclp parameter event"); vcpu->stat.deliver_service_signal++; @@ -997,8 +1000,9 @@ static int __must_check __deliver_pfault_done(struct kvm_vcpu *vcpu) struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int; struct kvm_s390_interrupt_info *inti; int rc = 0; + unsigned long flags; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); inti = list_first_entry_or_null(&fi->lists[FIRQ_LIST_PFAULT], struct kvm_s390_interrupt_info, list); @@ -1008,7 +1012,7 @@ static int __must_check __deliver_pfault_done(struct kvm_vcpu *vcpu) } if (list_empty(&fi->lists[FIRQ_LIST_PFAULT])) clear_bit(IRQ_PEND_PFAULT_DONE, &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); if (inti) { trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id, @@ -1039,8 +1043,9 @@ static int __must_check __deliver_virtio(struct kvm_vcpu *vcpu) struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int; struct kvm_s390_interrupt_info *inti; int rc = 0; + unsigned long flags; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); inti = list_first_entry_or_null(&fi->lists[FIRQ_LIST_VIRTIO], struct kvm_s390_interrupt_info, list); @@ -1058,7 +1063,7 @@ static int __must_check __deliver_virtio(struct kvm_vcpu *vcpu) } if (list_empty(&fi->lists[FIRQ_LIST_VIRTIO])) clear_bit(IRQ_PEND_VIRTIO, &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); if (inti) { rc = put_guest_lc(vcpu, EXT_IRQ_CP_SERVICE, @@ -1116,10 +1121,11 @@ static int __must_check __deliver_io(struct kvm_vcpu *vcpu, struct kvm_s390_io_info io; u32 isc; int rc = 0; + unsigned long flags; fi = &vcpu->kvm->arch.float_int; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); isc = irq_type_to_isc(irq_type); isc_list = &fi->lists[isc]; inti = list_first_entry_or_null(isc_list, @@ -1146,7 +1152,7 @@ static int __must_check __deliver_io(struct kvm_vcpu *vcpu, } if (list_empty(isc_list)) clear_bit(irq_type, &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); if (inti) { rc = __do_deliver_io(vcpu, &(inti->io)); @@ -1662,8 +1668,9 @@ static struct kvm_s390_interrupt_info *get_io_int(struct kvm *kvm, struct kvm_s390_interrupt_info *iter; u16 id = (schid & 0xffff0000U) >> 16; u16 nr = schid & 0x0000ffffU; + unsigned long flags; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); list_for_each_entry(iter, isc_list, list) { if (schid && (id != iter->io.subchannel_id || nr != iter->io.subchannel_nr)) @@ -1673,10 +1680,10 @@ static struct kvm_s390_interrupt_info *get_io_int(struct kvm *kvm, fi->counters[FIRQ_CNTR_IO] -= 1; if (list_empty(isc_list)) clear_bit(isc_to_irq_type(isc), &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return iter; } - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return NULL; } @@ -1769,9 +1776,10 @@ static int __inject_service(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) { struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; + unsigned long flags; kvm->stat.inject_service_signal++; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); fi->srv_signal.ext_params |= inti->ext.ext_params & SCCB_EVENT_PENDING; /* We always allow events, track them separately from the sccb ints */ @@ -1791,7 +1799,7 @@ static int __inject_service(struct kvm *kvm, fi->srv_signal.ext_params |= inti->ext.ext_params & SCCB_MASK; set_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs); out: - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); kfree(inti); return 0; } @@ -1800,17 +1808,18 @@ static int __inject_virtio(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) { struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; + unsigned long flags; kvm->stat.inject_virtio++; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); if (fi->counters[FIRQ_CNTR_VIRTIO] >= KVM_S390_MAX_VIRTIO_IRQS) { - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return -EBUSY; } fi->counters[FIRQ_CNTR_VIRTIO] += 1; list_add_tail(&inti->list, &fi->lists[FIRQ_LIST_VIRTIO]); set_bit(IRQ_PEND_VIRTIO, &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return 0; } @@ -1818,18 +1827,19 @@ static int __inject_pfault_done(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) { struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; + unsigned long flags; kvm->stat.inject_pfault_done++; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); if (fi->counters[FIRQ_CNTR_PFAULT] >= (ASYNC_PF_PER_VCPU * KVM_MAX_VCPUS)) { - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return -EBUSY; } fi->counters[FIRQ_CNTR_PFAULT] += 1; list_add_tail(&inti->list, &fi->lists[FIRQ_LIST_PFAULT]); set_bit(IRQ_PEND_PFAULT_DONE, &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return 0; } @@ -1838,13 +1848,14 @@ static int __inject_float_mchk(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) { struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; + unsigned long flags; kvm->stat.inject_float_mchk++; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); fi->mchk.cr14 |= inti->mchk.cr14 & (1UL << CR_PENDING_SUBCLASS); fi->mchk.mcic |= inti->mchk.mcic; set_bit(IRQ_PEND_MCHK_REP, &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); kfree(inti); return 0; } @@ -1855,6 +1866,7 @@ static int __inject_io(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) struct kvm_s390_float_interrupt *fi; struct list_head *list; int isc; + unsigned long flags; kvm->stat.inject_io++; isc = int_word_to_isc(inti->io.io_int_word); @@ -1873,9 +1885,9 @@ static int __inject_io(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) } fi = &kvm->arch.float_int; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); if (fi->counters[FIRQ_CNTR_IO] >= KVM_S390_MAX_FLOAT_IRQS) { - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return -EBUSY; } fi->counters[FIRQ_CNTR_IO] += 1; @@ -1890,7 +1902,7 @@ static int __inject_io(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) list = &fi->lists[FIRQ_LIST_IO_ISC_0 + isc]; list_add_tail(&inti->list, list); set_bit(isc_to_irq_type(isc), &fi->pending_irqs); - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); return 0; } @@ -1966,15 +1978,10 @@ static int __inject_vm(struct kvm *kvm, struct kvm_s390_interrupt_info *inti) } int kvm_s390_inject_vm(struct kvm *kvm, - struct kvm_s390_interrupt *s390int) + struct kvm_s390_interrupt *s390int, struct kvm_s390_interrupt_info *inti) { - struct kvm_s390_interrupt_info *inti; int rc; - inti = kzalloc_obj(*inti, GFP_KERNEL_ACCOUNT); - if (!inti) - return -ENOMEM; - inti->type = s390int->type; switch (inti->type) { case KVM_S390_INT_VIRTIO: @@ -2003,15 +2010,13 @@ int kvm_s390_inject_vm(struct kvm *kvm, inti->io.io_int_word = s390int->parm64 & 0x00000000ffffffffull; break; default: - kfree(inti); return -EINVAL; } trace_kvm_s390_inject_vm(s390int->type, s390int->parm, s390int->parm64, 2); rc = __inject_vm(kvm, inti); - if (rc) - kfree(inti); + return rc; } @@ -2176,12 +2181,13 @@ void kvm_s390_clear_float_irqs(struct kvm *kvm) { struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; int i; + unsigned long flags; mutex_lock(&kvm->lock); if (!kvm_s390_pv_is_protected(kvm)) fi->masked_irqs = 0; mutex_unlock(&kvm->lock); - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); fi->pending_irqs = 0; memset(&fi->srv_signal, 0, sizeof(fi->srv_signal)); memset(&fi->mchk, 0, sizeof(fi->mchk)); @@ -2189,7 +2195,7 @@ void kvm_s390_clear_float_irqs(struct kvm *kvm) clear_irq_list(&fi->lists[i]); for (i = 0; i < FIRQ_MAX_COUNT; i++) fi->counters[i] = 0; - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); kvm_s390_gisa_clear(kvm); }; @@ -2204,6 +2210,7 @@ static int get_all_floating_irqs(struct kvm *kvm, u8 __user *usrbuf, u64 len) int ret = 0; int n = 0; int i; + unsigned long flags; if (len > KVM_S390_FLIC_MAX_BUFFER || len == 0) return -EINVAL; @@ -2235,7 +2242,7 @@ static int get_all_floating_irqs(struct kvm *kvm, u8 __user *usrbuf, u64 len) } } fi = &kvm->arch.float_int; - spin_lock(&fi->lock); + spin_lock_irqsave(&fi->lock, flags); for (i = 0; i < FIRQ_LIST_COUNT; i++) { list_for_each_entry(inti, &fi->lists[i], list) { if (n == max_irqs) { @@ -2272,7 +2279,7 @@ static int get_all_floating_irqs(struct kvm *kvm, u8 __user *usrbuf, u64 len) } out: - spin_unlock(&fi->lock); + spin_unlock_irqrestore(&fi->lock, flags); out_nolock: if (!ret && n > 0) { if (copy_to_user(usrbuf, buf, sizeof(struct kvm_s390_irq) * n)) @@ -2287,6 +2294,7 @@ static int flic_ais_mode_get_all(struct kvm *kvm, struct kvm_device_attr *attr) { struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; struct kvm_s390_ais_all ais; + unsigned long flags; if (attr->attr < sizeof(ais)) return -EINVAL; @@ -2294,10 +2302,10 @@ static int flic_ais_mode_get_all(struct kvm *kvm, struct kvm_device_attr *attr) if (!test_kvm_facility(kvm, 72)) return -EOPNOTSUPP; - mutex_lock(&fi->ais_lock); + spin_lock_irqsave(&fi->ais_lock, flags); ais.simm = fi->simm; ais.nimm = fi->nimm; - mutex_unlock(&fi->ais_lock); + spin_unlock_irqrestore(&fi->ais_lock, flags); if (copy_to_user((void __user *)attr->addr, &ais, sizeof(ais))) return -EFAULT; @@ -2411,34 +2419,46 @@ static int register_io_adapter(struct kvm_device *dev, { struct s390_io_adapter *adapter; struct kvm_s390_io_adapter adapter_info; + int rc = 0; + mutex_lock(&dev->kvm->lock); if (copy_from_user(&adapter_info, - (void __user *)attr->addr, sizeof(adapter_info))) - return -EFAULT; - - if (adapter_info.id >= MAX_S390_IO_ADAPTERS) - return -EINVAL; - + (void __user *)attr->addr, sizeof(adapter_info))) { + rc = -EFAULT; + goto out; + } + if (adapter_info.id >= MAX_S390_IO_ADAPTERS) { + rc = -EINVAL; + goto out; + } adapter_info.id = array_index_nospec(adapter_info.id, MAX_S390_IO_ADAPTERS); - if (dev->kvm->arch.adapters[adapter_info.id] != NULL) - return -EINVAL; - + if (dev->kvm->arch.adapters[adapter_info.id] != NULL) { + rc = -EINVAL; + goto out; + } adapter = kzalloc_obj(*adapter, GFP_KERNEL_ACCOUNT); - if (!adapter) - return -ENOMEM; + if (!adapter) { + rc = -ENOMEM; + goto out; + } + INIT_LIST_HEAD(&adapter->maps); + spin_lock_init(&adapter->maps_lock); + adapter->nr_maps = 0; adapter->id = adapter_info.id; adapter->isc = adapter_info.isc; adapter->maskable = adapter_info.maskable; adapter->masked = false; adapter->swap = adapter_info.swap; - adapter->suppressible = (adapter_info.flags) & + adapter->suppressible = adapter_info.flags & KVM_S390_ADAPTER_SUPPRESSIBLE; dev->kvm->arch.adapters[adapter->id] = adapter; - return 0; +out: + mutex_unlock(&dev->kvm->lock); + return rc; } int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) @@ -2453,12 +2473,151 @@ int kvm_s390_mask_adapter(struct kvm *kvm, unsigned int id, bool masked) return ret; } +static struct page *pin_map_page(struct kvm *kvm, u64 uaddr, + unsigned int gup_flags) +{ + struct mm_struct *mm = kvm->mm; + struct page *page = NULL; + int locked = 1; + + if (mmget_not_zero(mm)) { + mmap_read_lock(mm); + pin_user_pages_remote(mm, uaddr, 1, FOLL_WRITE | gup_flags, + &page, &locked); + if (locked) + mmap_read_unlock(mm); + mmput(mm); + } + + return page; +} + +static int kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr) +{ + struct s390_io_adapter *adapter = get_io_adapter(kvm, id); + struct s390_map_info *map; + unsigned long flags; + __u64 host_addr; + int ret, idx; + + if (!adapter || !addr) + return -EINVAL; + + map = kzalloc_obj(*map, GFP_KERNEL_ACCOUNT); + if (!map) + return -ENOMEM; + + INIT_LIST_HEAD(&map->list); + idx = srcu_read_lock(&kvm->srcu); + host_addr = gpa_to_hva(kvm, addr); + if (kvm_is_error_hva(host_addr)) { + srcu_read_unlock(&kvm->srcu, idx); + ret = -EFAULT; + goto out; + } + srcu_read_unlock(&kvm->srcu, idx); + map->guest_addr = addr; + map->addr = host_addr; + map->page = pin_map_page(kvm, host_addr, FOLL_LONGTERM); + if (!map->page) { + ret = -EINVAL; + goto out; + } + spin_lock_irqsave(&adapter->maps_lock, flags); + if (adapter->nr_maps < MAX_S390_ADAPTER_MAPS) { + list_add_tail(&map->list, &adapter->maps); + adapter->nr_maps++; + ret = 0; + } else { + ret = -EINVAL; + } + spin_unlock_irqrestore(&adapter->maps_lock, flags); + if (ret) + unpin_user_page(map->page); +out: + if (ret) + kfree(map); + return ret; +} + +static int kvm_s390_adapter_unmap(struct kvm *kvm, unsigned int id, __u64 addr) +{ + struct s390_io_adapter *adapter = get_io_adapter(kvm, id); + struct s390_map_info *map, *tmp, *map_to_free; + struct page *map_page_to_put = NULL; + u64 map_addr_to_mark = 0; + unsigned long flags; + int found = 0, idx; + + if (!adapter || !addr) + return -EINVAL; + + spin_lock_irqsave(&adapter->maps_lock, flags); + list_for_each_entry_safe(map, tmp, &adapter->maps, list) { + if (map->guest_addr == addr) { + found = 1; + adapter->nr_maps--; + list_del(&map->list); + map_page_to_put = map->page; + map_addr_to_mark = map->guest_addr; + map_to_free = map; + break; + } + } + spin_unlock_irqrestore(&adapter->maps_lock, flags); + + if (found) { + kfree(map_to_free); + idx = srcu_read_lock(&kvm->srcu); + mark_page_dirty(kvm, map_addr_to_mark >> PAGE_SHIFT); + set_page_dirty_lock(map_page_to_put); + srcu_read_unlock(&kvm->srcu, idx); + unpin_user_page(map_page_to_put); + } + + return found ? 0 : -ENOENT; +} + +void kvm_s390_unmap_all_adapters(struct kvm *kvm) +{ + struct s390_map_info *map, *tmp; + unsigned long flags; + int i, idx; + + for (i = 0; i < MAX_S390_IO_ADAPTERS; i++) { + struct s390_io_adapter *adapter = kvm->arch.adapters[i]; + LIST_HEAD(local_list); + + if (!adapter) + continue; + + spin_lock_irqsave(&adapter->maps_lock, flags); + list_splice_init(&adapter->maps, &local_list); + adapter->nr_maps = 0; + spin_unlock_irqrestore(&adapter->maps_lock, flags); + + list_for_each_entry_safe(map, tmp, &local_list, list) { + list_del(&map->list); + idx = srcu_read_lock(&kvm->srcu); + mark_page_dirty(kvm, map->guest_addr >> PAGE_SHIFT); + set_page_dirty_lock(map->page); + srcu_read_unlock(&kvm->srcu, idx); + unpin_user_page(map->page); + kfree(map); + } + } +} + void kvm_s390_destroy_adapters(struct kvm *kvm) { int i; - for (i = 0; i < MAX_S390_IO_ADAPTERS; i++) + kvm_s390_unmap_all_adapters(kvm); + + for (i = 0; i < MAX_S390_IO_ADAPTERS; i++) { kfree(kvm->arch.adapters[i]); + kvm->arch.adapters[i] = NULL; + } } static int modify_io_adapter(struct kvm_device *dev, @@ -2480,14 +2639,22 @@ static int modify_io_adapter(struct kvm_device *dev, if (ret > 0) ret = 0; break; - /* - * The following operations are no longer needed and therefore no-ops. - * The gpa to hva translation is done when an IRQ route is set up. The - * set_irq code uses get_user_pages_remote() to do the actual write. - */ case KVM_S390_IO_ADAPTER_MAP: case KVM_S390_IO_ADAPTER_UNMAP: - ret = 0; + /* If in Secure Execution mode do not long term pin. */ + mutex_lock(&dev->kvm->lock); + if (kvm_s390_pv_is_protected(dev->kvm)) { + mutex_unlock(&dev->kvm->lock); + return 0; + } + if (req.type == KVM_S390_IO_ADAPTER_MAP) { + dev->kvm->stat.io_390_adapter_map++; + ret = kvm_s390_adapter_map(dev->kvm, req.id, req.addr); + } else { + dev->kvm->stat.io_390_adapter_unmap++; + ret = kvm_s390_adapter_unmap(dev->kvm, req.id, req.addr); + } + mutex_unlock(&dev->kvm->lock); break; default: ret = -EINVAL; @@ -2524,6 +2691,7 @@ static int modify_ais_mode(struct kvm *kvm, struct kvm_device_attr *attr) struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; struct kvm_s390_ais_req req; int ret = 0; + unsigned long flags; if (!test_kvm_facility(kvm, 72)) return -EOPNOTSUPP; @@ -2540,7 +2708,7 @@ static int modify_ais_mode(struct kvm *kvm, struct kvm_device_attr *attr) 2 : KVM_S390_AIS_MODE_SINGLE : KVM_S390_AIS_MODE_ALL, req.mode); - mutex_lock(&fi->ais_lock); + spin_lock_irqsave(&fi->ais_lock, flags); switch (req.mode) { case KVM_S390_AIS_MODE_ALL: fi->simm &= ~AIS_MODE_MASK(req.isc); @@ -2553,7 +2721,7 @@ static int modify_ais_mode(struct kvm *kvm, struct kvm_device_attr *attr) default: ret = -EINVAL; } - mutex_unlock(&fi->ais_lock); + spin_unlock_irqrestore(&fi->ais_lock, flags); return ret; } @@ -2567,25 +2735,41 @@ static int kvm_s390_inject_airq(struct kvm *kvm, .parm = 0, .parm64 = isc_to_int_word(adapter->isc), }; + struct kvm_s390_interrupt_info *inti; + unsigned long flags; + int ret = 0; - if (!test_kvm_facility(kvm, 72) || !adapter->suppressible) - return kvm_s390_inject_vm(kvm, &s390int); + inti = kzalloc_obj(*inti, GFP_KERNEL_ACCOUNT); + if (!inti) + return -ENOMEM; + + if (!test_kvm_facility(kvm, 72) || !adapter->suppressible) { + ret = kvm_s390_inject_vm(kvm, &s390int, inti); + if (ret) + kfree(inti); + return ret; + } - mutex_lock(&fi->ais_lock); + spin_lock_irqsave(&fi->ais_lock, flags); if (fi->nimm & AIS_MODE_MASK(adapter->isc)) { trace_kvm_s390_airq_suppressed(adapter->id, adapter->isc); - goto out; + spin_unlock_irqrestore(&fi->ais_lock, flags); + kfree(inti); + return ret; } - ret = kvm_s390_inject_vm(kvm, &s390int); + ret = kvm_s390_inject_vm(kvm, &s390int, inti); + if (!ret && (fi->simm & AIS_MODE_MASK(adapter->isc))) { fi->nimm |= AIS_MODE_MASK(adapter->isc); trace_kvm_s390_modify_ais_mode(adapter->isc, KVM_S390_AIS_MODE_SINGLE, 2); } -out: - mutex_unlock(&fi->ais_lock); + + spin_unlock_irqrestore(&fi->ais_lock, flags); + if (ret) + kfree(inti); return ret; } @@ -2594,6 +2778,8 @@ static int flic_inject_airq(struct kvm *kvm, struct kvm_device_attr *attr) unsigned int id = attr->attr; struct s390_io_adapter *adapter = get_io_adapter(kvm, id); + kvm->stat.io_flic_inject_airq++; + if (!adapter) return -EINVAL; @@ -2604,6 +2790,7 @@ static int flic_ais_mode_set_all(struct kvm *kvm, struct kvm_device_attr *attr) { struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; struct kvm_s390_ais_all ais; + unsigned long flags; if (!test_kvm_facility(kvm, 72)) return -EOPNOTSUPP; @@ -2611,10 +2798,10 @@ static int flic_ais_mode_set_all(struct kvm *kvm, struct kvm_device_attr *attr) if (copy_from_user(&ais, (void __user *)attr->addr, sizeof(ais))) return -EFAULT; - mutex_lock(&fi->ais_lock); + spin_lock_irqsave(&fi->ais_lock, flags); fi->simm = ais.simm; fi->nimm = ais.nimm; - mutex_unlock(&fi->ais_lock); + spin_unlock_irqrestore(&fi->ais_lock, flags); return 0; } @@ -2733,22 +2920,19 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) return swap ? (bit ^ (BITS_PER_LONG - 1)) : bit; } -static struct page *get_map_page(struct kvm *kvm, u64 uaddr) +static struct s390_map_info *get_map_info(struct s390_io_adapter *adapter, + u64 addr) { - struct mm_struct *mm = kvm->mm; - struct page *page = NULL; - int locked = 1; + struct s390_map_info *map; - if (mmget_not_zero(mm)) { - mmap_read_lock(mm); - get_user_pages_remote(mm, uaddr, 1, FOLL_WRITE, - &page, &locked); - if (locked) - mmap_read_unlock(mm); - mmput(mm); - } + if (!adapter) + return NULL; - return page; + list_for_each_entry(map, &adapter->maps, list) { + if (map->addr == addr) + return map; + } + return NULL; } static int adapter_indicators_set(struct kvm *kvm, @@ -2757,35 +2941,98 @@ static int adapter_indicators_set(struct kvm *kvm, { unsigned long bit; int summary_set, idx; - struct page *ind_page, *summary_page; + struct s390_map_info *ind_info, *summary_info; void *map; + struct page *ind_page, *summary_page; + unsigned long flags; - ind_page = get_map_page(kvm, adapter_int->ind_addr); - if (!ind_page) - return -1; - summary_page = get_map_page(kvm, adapter_int->summary_addr); - if (!summary_page) { - put_page(ind_page); - return -1; + ind_page = NULL; + + spin_lock_irqsave(&adapter->maps_lock, flags); + ind_info = get_map_info(adapter, adapter_int->ind_addr); + if (!ind_info) { + spin_unlock_irqrestore(&adapter->maps_lock, flags); + ind_page = pin_map_page(kvm, adapter_int->ind_addr, 0); + if (!ind_page) + return -1; + idx = srcu_read_lock(&kvm->srcu); + map = page_address(ind_page); + bit = get_ind_bit(adapter_int->ind_addr, + adapter_int->ind_offset, adapter->swap); + set_bit(bit, map); + mark_page_dirty(kvm, adapter_int->ind_gaddr >> PAGE_SHIFT); + set_page_dirty_lock(ind_page); + srcu_read_unlock(&kvm->srcu, idx); + unpin_user_page(ind_page); + } else { + map = page_address(ind_info->page); + bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap); + set_bit(bit, map); + spin_unlock_irqrestore(&adapter->maps_lock, flags); + } + + spin_lock_irqsave(&adapter->maps_lock, flags); + summary_info = get_map_info(adapter, adapter_int->summary_addr); + if (!summary_info) { + spin_unlock_irqrestore(&adapter->maps_lock, flags); + summary_page = pin_map_page(kvm, adapter_int->summary_addr, 0); + if (WARN_ON_ONCE(!summary_page)) + return -1; + idx = srcu_read_lock(&kvm->srcu); + map = page_address(summary_page); + bit = get_ind_bit(adapter_int->summary_addr, + adapter_int->summary_offset, adapter->swap); + summary_set = test_and_set_bit(bit, map); + mark_page_dirty(kvm, adapter_int->summary_gaddr >> PAGE_SHIFT); + set_page_dirty_lock(summary_page); + srcu_read_unlock(&kvm->srcu, idx); + unpin_user_page(summary_page); + } else { + map = page_address(summary_info->page); + bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset, + adapter->swap); + summary_set = test_and_set_bit(bit, map); + spin_unlock_irqrestore(&adapter->maps_lock, flags); } - idx = srcu_read_lock(&kvm->srcu); - map = page_address(ind_page); - bit = get_ind_bit(adapter_int->ind_addr, - adapter_int->ind_offset, adapter->swap); - set_bit(bit, map); - mark_page_dirty(kvm, adapter_int->ind_gaddr >> PAGE_SHIFT); - set_page_dirty_lock(ind_page); - map = page_address(summary_page); - bit = get_ind_bit(adapter_int->summary_addr, - adapter_int->summary_offset, adapter->swap); - summary_set = test_and_set_bit(bit, map); - mark_page_dirty(kvm, adapter_int->summary_gaddr >> PAGE_SHIFT); - set_page_dirty_lock(summary_page); - srcu_read_unlock(&kvm->srcu, idx); + return summary_set ? 0 : 1; +} + +static int adapter_indicators_set_fast(struct kvm *kvm, + struct s390_io_adapter *adapter, + struct kvm_s390_adapter_int *adapter_int, + int setbit) +{ + unsigned long bit; + int summary_set; + struct s390_map_info *ind_info, *summary_info; + void *map; - put_page(ind_page); - put_page(summary_page); + spin_lock(&adapter->maps_lock); + ind_info = get_map_info(adapter, adapter_int->ind_addr); + if (!ind_info) { + spin_unlock(&adapter->maps_lock); + return -EWOULDBLOCK; + } + map = page_address(ind_info->page); + bit = get_ind_bit(ind_info->addr, adapter_int->ind_offset, adapter->swap); + if (setbit) + set_bit(bit, map); + summary_info = get_map_info(adapter, adapter_int->summary_addr); + if (!summary_info) { + spin_unlock(&adapter->maps_lock); + return -EWOULDBLOCK; + } + map = page_address(summary_info->page); + bit = get_ind_bit(summary_info->addr, adapter_int->summary_offset, + adapter->swap); + /* If setbit then set summary bit. Else if falling back to the slow path */ + /* with setbit==0 then clear the summary bit so the slow path re-injects */ + if (setbit) + summary_set = test_and_set_bit(bit, map); + else + summary_set = test_and_clear_bit(bit, map); + spin_unlock(&adapter->maps_lock); return summary_set ? 0 : 1; } @@ -2801,6 +3048,8 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e, int ret; struct s390_io_adapter *adapter; + kvm->stat.io_set_adapter_int++; + /* We're only interested in the 0->1 transition. */ if (!level) return 0; @@ -2869,7 +3118,6 @@ int kvm_set_routing_entry(struct kvm *kvm, int idx; switch (ue->type) { - /* we store the userspace addresses instead of the guest addresses */ case KVM_IRQ_ROUTING_S390_ADAPTER: if (kvm_is_ucontrol(kvm)) return -EINVAL; @@ -3459,3 +3707,86 @@ out_free_gib: out: return rc; } + +/* + * kvm_arch_set_irq_inatomic: fast-path for irqfd injection + */ +int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *e, + struct kvm *kvm, int irq_source_id, int level, + bool line_status) +{ + int ret, setbit; + struct s390_io_adapter *adapter; + struct kvm_s390_float_interrupt *fi = &kvm->arch.float_int; + struct kvm_s390_interrupt_info *inti; + struct kvm_s390_interrupt s390int = { + .type = KVM_S390_INT_IO(1, 0, 0, 0), + .parm = 0, + }; + + kvm->stat.io_390_inatomic++; + + /* We're only interested in the 0->1 transition. */ + if (!level) + return 0; + if (e->type != KVM_IRQ_ROUTING_S390_ADAPTER) + return -EWOULDBLOCK; + + adapter = get_io_adapter(kvm, e->adapter.adapter_id); + if (!adapter) + return -EWOULDBLOCK; + + s390int.parm64 = isc_to_int_word(adapter->isc); + setbit = 1; + ret = adapter_indicators_set_fast(kvm, adapter, &e->adapter, setbit); + if (ret < 0) + return -EWOULDBLOCK; + if (!ret || adapter->masked) { + kvm->stat.io_390_inatomic_no_inject++; + return 0; + } + + inti = kzalloc_obj(*inti, GFP_ATOMIC); + if (!inti) { + setbit = 0; + adapter_indicators_set_fast(kvm, adapter, &e->adapter, setbit); + return -EWOULDBLOCK; + } + + if (!test_kvm_facility(kvm, 72) || !adapter->suppressible) { + ret = kvm_s390_inject_vm(kvm, &s390int, inti); + if (ret == 0) { + return ret; + } else { + setbit = 0; + adapter_indicators_set_fast(kvm, adapter, &e->adapter, setbit); + kfree(inti); + return -EWOULDBLOCK; + } + } + + spin_lock(&fi->ais_lock); + if (fi->nimm & AIS_MODE_MASK(adapter->isc)) { + trace_kvm_s390_airq_suppressed(adapter->id, adapter->isc); + spin_unlock(&fi->ais_lock); + kfree(inti); + kvm->stat.io_390_inatomic_no_inject++; + return 0; + } + + ret = kvm_s390_inject_vm(kvm, &s390int, inti); + if (!ret && (fi->simm & AIS_MODE_MASK(adapter->isc))) { + fi->nimm |= AIS_MODE_MASK(adapter->isc); + trace_kvm_s390_modify_ais_mode(adapter->isc, + KVM_S390_AIS_MODE_SINGLE, 2); + } else if (ret) { + spin_unlock(&fi->ais_lock); + setbit = 0; + adapter_indicators_set_fast(kvm, adapter, &e->adapter, setbit); + kfree(inti); + return -EWOULDBLOCK; + } + + spin_unlock(&fi->ais_lock); + return 0; +} diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index ffb20a64d328..dc9c7451fe97 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -68,6 +68,12 @@ const struct kvm_stats_desc kvm_vm_stats_desc[] = { KVM_GENERIC_VM_STATS(), STATS_DESC_COUNTER(VM, inject_io), + STATS_DESC_COUNTER(VM, io_390_adapter_map), + STATS_DESC_COUNTER(VM, io_390_adapter_unmap), + STATS_DESC_COUNTER(VM, io_390_inatomic), + STATS_DESC_COUNTER(VM, io_flic_inject_airq), + STATS_DESC_COUNTER(VM, io_set_adapter_int), + STATS_DESC_COUNTER(VM, io_390_inatomic_no_inject), STATS_DESC_COUNTER(VM, inject_float_mchk), STATS_DESC_COUNTER(VM, inject_pfault_done), STATS_DESC_COUNTER(VM, inject_service_signal), @@ -204,11 +210,16 @@ static int nested; module_param(nested, int, S_IRUGO); MODULE_PARM_DESC(nested, "Nested virtualization support"); -/* allow 1m huge page guest backing, if !nested */ +/* allow 1m huge page guest backing */ static int hpage; module_param(hpage, int, 0444); MODULE_PARM_DESC(hpage, "1m huge page backing support"); +/* allow 2g huge page guest backing */ +static int hpage_2g; +module_param(hpage_2g, int, 0444); +MODULE_PARM_DESC(hpage_2g, "2g huge page backing support"); + /* maximum percentage of steal time for polling. >100 is treated like 100 */ static u8 halt_poll_max_steal = 10; module_param(halt_poll_max_steal, byte, 0644); @@ -232,33 +243,25 @@ static int async_destroy = 1; module_param(async_destroy, int, 0444); MODULE_PARM_DESC(async_destroy, "Asynchronous destroy for protected guests"); -/* - * For now we handle at most 16 double words as this is what the s390 base - * kernel handles and stores in the prefix page. If we ever need to go beyond - * this, this requires changes to code, but the external uapi can stay. - */ -#define SIZE_INTERNAL 16 - +#define HMFAI_DWORDS 16 /* * Base feature mask that defines default mask for facilities. Consists of the * defines in FACILITIES_KVM and the non-hypervisor managed bits. */ -static unsigned long kvm_s390_fac_base[SIZE_INTERNAL] = { FACILITIES_KVM }; +static unsigned long kvm_s390_fac_base[HMFAI_DWORDS] = { FACILITIES_KVM }; +static_assert(ARRAY_SIZE(((long[]){ FACILITIES_KVM })) <= HMFAI_DWORDS); +static_assert(ARRAY_SIZE(kvm_s390_fac_base) <= S390_ARCH_FAC_MASK_SIZE_U64); +static_assert(ARRAY_SIZE(kvm_s390_fac_base) <= S390_ARCH_FAC_LIST_SIZE_U64); +static_assert(ARRAY_SIZE(kvm_s390_fac_base) <= ARRAY_SIZE(stfle_fac_list)); + /* * Extended feature mask. Consists of the defines in FACILITIES_KVM_CPUMODEL * and defines the facilities that can be enabled via a cpu model. */ -static unsigned long kvm_s390_fac_ext[SIZE_INTERNAL] = { FACILITIES_KVM_CPUMODEL }; - -static unsigned long kvm_s390_fac_size(void) -{ - BUILD_BUG_ON(SIZE_INTERNAL > S390_ARCH_FAC_MASK_SIZE_U64); - BUILD_BUG_ON(SIZE_INTERNAL > S390_ARCH_FAC_LIST_SIZE_U64); - BUILD_BUG_ON(SIZE_INTERNAL * sizeof(unsigned long) > - sizeof(stfle_fac_list)); - - return SIZE_INTERNAL; -} +static const unsigned long kvm_s390_fac_ext[] = { FACILITIES_KVM_CPUMODEL }; +static_assert(ARRAY_SIZE(kvm_s390_fac_ext) <= S390_ARCH_FAC_MASK_SIZE_U64); +static_assert(ARRAY_SIZE(kvm_s390_fac_ext) <= S390_ARCH_FAC_LIST_SIZE_U64); +static_assert(ARRAY_SIZE(kvm_s390_fac_ext) <= ARRAY_SIZE(stfle_fac_list)); /* available cpu features supported by kvm */ static DECLARE_BITMAP(kvm_s390_available_cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS); @@ -468,6 +471,8 @@ static void __init kvm_s390_cpu_feat_init(void) allow_cpu_feat(KVM_S390_VM_CPU_FEAT_IBS); if (sclp.has_kss) allow_cpu_feat(KVM_S390_VM_CPU_FEAT_KSS); + if (sclp.has_astfleie2) + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ASTFLEIE2); /* * KVM_S390_VM_CPU_FEAT_SKEY: Wrong shadow of PTE.I bits will make * all skey handling functions read/set the skey from the PGSTE @@ -630,6 +635,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_S390_USER_OPEREXEC: case KVM_CAP_S390_KEYOP: case KVM_CAP_S390_VSIE_ESAMODE: + case KVM_CAP_PRE_FAULT_MEMORY: r = 1; break; case KVM_CAP_SET_GUEST_DEBUG2: @@ -640,6 +646,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) if (hpage && !(kvm && kvm_is_ucontrol(kvm))) r = 1; break; + case KVM_CAP_S390_HPAGE_2G: + r = 0; + if (hpage_2g && !(kvm && kvm_is_ucontrol(kvm))) + r = 1; + break; case KVM_CAP_S390_MEM_OP: r = MEM_OP_MAX_SIZE; break; @@ -896,6 +907,27 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) VM_EVENT(kvm, 3, "ENABLE: CAP_S390_HPAGE %s", r ? "(not available)" : "(success)"); break; + case KVM_CAP_S390_HPAGE_2G: + mutex_lock(&kvm->lock); + if (kvm->created_vcpus) { + r = -EBUSY; + } else if (!hpage_2g || kvm->arch.use_cmma || kvm_is_ucontrol(kvm)) { + r = -EINVAL; + } else { + r = 0; + set_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &kvm->arch.gmap->flags); + /* + * We might have to create fake 4k page + * tables. To avoid that the hardware works on + * stale PGSTEs, we emulate these instructions. + */ + kvm->arch.use_skf = 0; + kvm->arch.use_pfmfi = 0; + } + mutex_unlock(&kvm->lock); + VM_EVENT(kvm, 3, "ENABLE: CAP_S390_HPAGE_2G %s", + r ? "(not available)" : "(success)"); + break; case KVM_CAP_S390_USER_STSI: VM_EVENT(kvm, 3, "%s", "ENABLE: CAP_S390_USER_STSI"); kvm->arch.user_stsi = 1; @@ -2252,7 +2284,7 @@ static int kvm_s390_get_cmma_bits(struct kvm *kvm, return 0; } - values = vmalloc(args->count); + values = vzalloc(args->count); if (!values) return -ENOMEM; @@ -2534,6 +2566,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) if (kvm_s390_pv_is_protected(kvm)) break; + kvm_s390_unmap_all_adapters(kvm); mmap_write_lock(kvm->mm); /* * Disable creation of new THPs. Existing THPs can stay, they @@ -2869,6 +2902,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) void __user *argp = (void __user *)arg; struct kvm_device_attr attr; int r; + struct kvm_s390_interrupt_info *inti; switch (ioctl) { case KVM_S390_INTERRUPT: { @@ -2877,7 +2911,12 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) r = -EFAULT; if (copy_from_user(&s390int, argp, sizeof(s390int))) break; - r = kvm_s390_inject_vm(kvm, &s390int); + inti = kzalloc_obj(*inti, GFP_KERNEL_ACCOUNT); + if (!inti) + return -ENOMEM; + r = kvm_s390_inject_vm(kvm, &s390int, inti); + if (r) + kfree(inti); break; } case KVM_CREATE_IRQCHIP: { @@ -3238,13 +3277,16 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm->arch.sie_page2->kvm = kvm; kvm->arch.model.fac_list = kvm->arch.sie_page2->fac_list; - for (i = 0; i < kvm_s390_fac_size(); i++) { + for (i = 0; i < ARRAY_SIZE(kvm_s390_fac_base); i++) { kvm->arch.model.fac_mask[i] = stfle_fac_list[i] & - (kvm_s390_fac_base[i] | - kvm_s390_fac_ext[i]); + kvm_s390_fac_base[i]; kvm->arch.model.fac_list[i] = stfle_fac_list[i] & kvm_s390_fac_base[i]; } + for (i = 0; i < ARRAY_SIZE(kvm_s390_fac_ext); i++) { + kvm->arch.model.fac_mask[i] |= stfle_fac_list[i] & + kvm_s390_fac_ext[i]; + } kvm->arch.model.subfuncs = kvm_s390_available_subfunc; /* we are always in czam mode - even on pre z14 machines */ @@ -3275,7 +3317,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) mutex_unlock(&kvm->lock); } - mutex_init(&kvm->arch.float_int.ais_lock); + spin_lock_init(&kvm->arch.float_int.ais_lock); spin_lock_init(&kvm->arch.float_int.lock); for (i = 0; i < FIRQ_LIST_COUNT; i++) INIT_LIST_HEAD(&kvm->arch.float_int.lists[i]); @@ -4397,19 +4439,28 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo } static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token, - unsigned long token) + unsigned long token) { struct kvm_s390_interrupt inti; struct kvm_s390_irq irq; + struct kvm_s390_interrupt_info *inti_mem = NULL; + int ret = 0; if (start_token) { irq.u.ext.ext_params2 = token; irq.type = KVM_S390_INT_PFAULT_INIT; WARN_ON_ONCE(kvm_s390_inject_vcpu(vcpu, &irq)); } else { + inti_mem = kzalloc_obj(*inti_mem, GFP_KERNEL_ACCOUNT); + if (WARN_ON_ONCE(!inti_mem)) + return; + inti.type = KVM_S390_INT_PFAULT_DONE; inti.parm64 = token; - WARN_ON_ONCE(kvm_s390_inject_vm(vcpu->kvm, &inti)); + ret = kvm_s390_inject_vm(vcpu->kvm, &inti, inti_mem); + if (ret) + kfree(inti_mem); + WARN_ON_ONCE(ret); } } @@ -5760,6 +5811,50 @@ out: } /** + * kvm_arch_vcpu_pre_fault_memory() -- pre-fault and link gmap dat tables + * @vcpu: the vcpu that shall appear to have generated the fault-in. + * @range: the range that needs to be faulted in. + * + * The first page of the given range is faulted in and the corresponding gmap + * page tables are created, as if the given vCPU had performed a read + * operation. + * If the range starts outside any memslots, an error is returned. An error is + * also returned for UCONTROL VMs, which should instead use the + * KVM_S390_VCPU_FAULT ioctl. + * + * Return: + * * %-ENOENT if the range lies outside of a memslot. + * * %-EINVAL in case of invalid state (for example if the VM is UCONTROL). + * * %-EIO if errors happen while faulting-in the page (will trigger a warning + * in the caller). + * * other error codes < 0 in case of other errors. + * * otherwise a number > 0 of bytes that have been faulted in successfully. + */ +long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_memory *range) +{ + struct guest_fault f = { .gfn = gpa_to_gfn(range->gpa), }; + gpa_t end; + int rc; + + if (kvm_is_ucontrol(vcpu->kvm)) + return -EINVAL; + + rc = kvm_s390_faultin_gfn(vcpu, NULL, &f); + if (rc == PGM_ADDRESSING) + return -ENOENT; + if (rc > 0) + return -EIO; + if (rc < 0) + return rc; + + if (f.ptep) + return PAGE_SIZE; + + end = ALIGN(range->gpa + PAGE_SIZE, f.crste_region3 ? _REGION3_SIZE : HPAGE_SIZE); + return min(range->size, end - range->gpa); +} + +/** * kvm_test_age_gfn() - test young * @kvm: the kvm instance * @range: the range of guest addresses whose young status needs to be cleared @@ -5820,9 +5915,13 @@ static int __init kvm_s390_init(void) return -ENODEV; } - for (i = 0; i < 16; i++) - kvm_s390_fac_base[i] |= - stfle_fac_list[i] & nonhyp_mask(i); + if (hpage_2g && !hpage) { + hpage_2g = 0; + pr_info("Disabling 2G hugepage support, since 1M hugepage support is not enabled.\n"); + } + + for (i = 0; i < HMFAI_DWORDS; i++) + kvm_s390_fac_base[i] |= nonhyp_mask(i); r = __kvm_s390_init(); if (r) diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h index dc0573b7aa4b..6d2842fb71a3 100644 --- a/arch/s390/kvm/kvm-s390.h +++ b/arch/s390/kvm/kvm-s390.h @@ -376,7 +376,8 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu); void kvm_s390_clear_local_irqs(struct kvm_vcpu *vcpu); void kvm_s390_clear_float_irqs(struct kvm *kvm); int __must_check kvm_s390_inject_vm(struct kvm *kvm, - struct kvm_s390_interrupt *s390int); + struct kvm_s390_interrupt *s390int, + struct kvm_s390_interrupt_info *inti); int __must_check kvm_s390_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq); static inline int kvm_s390_inject_prog_irq(struct kvm_vcpu *vcpu, @@ -560,6 +561,8 @@ void kvm_s390_gisa_disable(struct kvm *kvm); void kvm_s390_gisa_enable(struct kvm *kvm); int __init kvm_s390_gib_init(u8 nisc); void kvm_s390_gib_destroy(void); +void kvm_s390_unmap_all_adapters(struct kvm *kvm); + /* implemented in guestdbg.c */ void kvm_s390_backup_guest_per_regs(struct kvm_vcpu *vcpu); diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c index 4b865e75351c..1beacc841ca8 100644 --- a/arch/s390/kvm/pv.c +++ b/arch/s390/kvm/pv.c @@ -740,7 +740,10 @@ int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc) uvcb.flags.ap_allow_instr = kvm->arch.model.uv_feat_guest.ap; uvcb.flags.ap_instr_intr = kvm->arch.model.uv_feat_guest.ap_intr; - clear_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &kvm->arch.gmap->flags); + scoped_guard(write_lock, &kvm->mmu_lock) { + clear_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &kvm->arch.gmap->flags); + clear_bit(GMAP_FLAG_ALLOW_HPAGE_2G, &kvm->arch.gmap->flags); + } gmap_split_huge_pages(kvm->arch.gmap); cc = uv_call_sched(0, (u64)&uvcb); diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c index e5a23f1c9749..eea24562e7db 100644 --- a/arch/s390/kvm/vsie.c +++ b/arch/s390/kvm/vsie.c @@ -6,12 +6,15 @@ * * Author(s): David Hildenbrand <dahi@linux.vnet.ibm.com> */ +#include <linux/align.h> #include <linux/vmalloc.h> #include <linux/kvm_host.h> #include <linux/bug.h> +#include <linux/compiler.h> #include <linux/list.h> #include <linux/bitmap.h> #include <linux/sched/signal.h> +#include <linux/stddef.h> #include <linux/io.h> #include <linux/mman.h> @@ -62,9 +65,9 @@ struct vsie_page { gpa_t scb_gpa; /* 0x0258 */ /* the shadow gmap in use by the vsie_page */ struct gmap_cache gmap_cache; /* 0x0260 */ - __u8 reserved[0x0700 - 0x0278]; /* 0x0278 */ - struct kvm_s390_crypto_cb crycb; /* 0x0700 */ - __u8 fac[S390_ARCH_FAC_LIST_SIZE_BYTE]; /* 0x0800 */ + __u8 reserved[0x06f8 - 0x0278]; /* 0x0278 */ + struct kvm_s390_crypto_cb crycb; /* 0x06f8 */ + __u8 fac[8 + S390_ARCH_FAC_LIST_SIZE_BYTE];/* 0x07f8 */ }; static_assert(sizeof(struct vsie_page) == PAGE_SIZE); @@ -1000,6 +1003,45 @@ static void retry_vsie_icpt(struct vsie_page *vsie_page) clear_vsie_icpt(vsie_page); } +static int handle_stfle_0(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page, + u32 fac_list_origin) +{ + struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; + + /* + * format-0 -> size of nested guest's facility list == guest's size + * guest's size == host's size, since STFLE is interpretatively executed + * using a format-0 for the guest, too. + */ + if (read_guest_real(vcpu, fac_list_origin, &vsie_page->fac, + stfle_size() * sizeof(u64))) + return set_validity_icpt(scb_s, 0x1090U); + scb_s->fac = (u32)virt_to_phys(&vsie_page->fac); + return 0; +} + +static int handle_stfle_2(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page, u32 fac_list_origin) +{ + struct kvm_s390_flcb2 *flcb_s = (struct kvm_s390_flcb2 *)vsie_page->fac; + struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; + u64 len; + + if (read_guest_real(vcpu, fac_list_origin, &len, sizeof(len))) + return set_validity_icpt(scb_s, 0x1090U); + + /* discard reserved bits */ + len = (len & U8_MAX); + flcb_s->header_val = len; + len += 1; + + if (read_guest_real(vcpu, fac_list_origin + offsetof(struct kvm_s390_flcb2, facilities), + &flcb_s->facilities, len * sizeof(u64))) + return set_validity_icpt(scb_s, 0x1090U); + + scb_s->fac = (u32)virt_to_phys(&vsie_page->fac) | S390_ARCH_FAC_FORMAT_2; + return 0; +} + /* * Try to shadow + enable the guest 2 provided facility list. * Retry instruction execution if enabled for and provided by guest 2. @@ -1009,29 +1051,34 @@ static void retry_vsie_icpt(struct vsie_page *vsie_page) */ static int handle_stfle(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) { - struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; - __u32 fac = READ_ONCE(vsie_page->scb_o->fac); + bool has_astfleie2 = test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_ASTFLEIE2); + u32 fac = READ_ONCE(vsie_page->scb_o->fac); + int format_mask, format; + u32 origin; + + /* assert no overflow with maximum len */ + BUILD_BUG_ON(sizeof(vsie_page->fac) < ((S390_ARCH_FAC_LIST_SIZE_U64 + 1) * sizeof(u64))); + BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct vsie_page, fac), 8)); - /* - * Alternate-STFLE-Interpretive-Execution facilities are not supported - * -> format-0 flcb - */ if (fac && test_kvm_facility(vcpu->kvm, 7)) { retry_vsie_icpt(vsie_page); /* * The facility list origin (FLO) is in bits 1 - 28 of the FLD * so we need to mask here before reading. */ - fac = fac & 0x7ffffff8U; - /* - * format-0 -> size of nested guest's facility list == guest's size - * guest's size == host's size, since STFLE is interpretatively executed - * using a format-0 for the guest, too. - */ - if (read_guest_real(vcpu, fac, &vsie_page->fac, - stfle_size() * sizeof(u64))) - return set_validity_icpt(scb_s, 0x1090U); - scb_s->fac = (u32)virt_to_phys(&vsie_page->fac); + origin = fac & 0x7ffffff8U; + format_mask = has_astfleie2 ? 3 : 0; + format = fac & format_mask; + switch (format) { + case 0: + return handle_stfle_0(vcpu, vsie_page, origin); + case 1: + return set_validity_icpt(&vsie_page->scb_s, 0x1330U); + case 2: + return handle_stfle_2(vcpu, vsie_page, origin); + case 3: + return set_validity_icpt(&vsie_page->scb_s, 0x1330U); + } } return 0; } diff --git a/arch/s390/mm/gmap_helpers.c b/arch/s390/mm/gmap_helpers.c index 1cfe4724fbe2..ee3f37af8aee 100644 --- a/arch/s390/mm/gmap_helpers.c +++ b/arch/s390/mm/gmap_helpers.c @@ -51,15 +51,15 @@ pte_t *try_get_locked_pte(struct mm_struct *mm, unsigned long vmaddr, spinlock_t pgd = pgdp_get(pgdp); if (pgd_none(pgd) || !pgd_present(pgd)) return NULL; - p4dp = p4d_offset(pgdp, vmaddr); + p4dp = p4d_offset_lockless(pgdp, pgd, vmaddr); p4d = p4dp_get(p4dp); if (p4d_none(p4d) || !p4d_present(p4d)) return NULL; - pudp = pud_offset(p4dp, vmaddr); + pudp = pud_offset_lockless(p4dp, p4d, vmaddr); pud = pudp_get(pudp); if (pud_none(pud) || pud_leaf(pud) || !pud_present(pud)) return NULL; - pmdp = pmd_offset(pudp, vmaddr); + pmdp = pmd_offset_lockless(pudp, pud, vmaddr); pmd = pmdp_get_lockless(pmdp); if (pmd_none(pmd) || pmd_leaf(pmd) || !pmd_present(pmd)) return NULL; diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 7b572bc24265..1b4a48bff18f 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -379,6 +379,7 @@ #define X86_FEATURE_AVIC (15*32+13) /* "avic" Virtual Interrupt Controller */ #define X86_FEATURE_V_VMSAVE_VMLOAD (15*32+15) /* "v_vmsave_vmload" Virtual VMSAVE VMLOAD */ #define X86_FEATURE_VGIF (15*32+16) /* "vgif" Virtual GIF */ +#define X86_FEATURE_GMET (15*32+17) /* Guest Mode Execution Trap */ #define X86_FEATURE_X2AVIC (15*32+18) /* "x2avic" Virtual x2apic */ #define X86_FEATURE_V_SPEC_CTRL (15*32+20) /* "v_spec_ctrl" Virtual SPEC_CTRL */ #define X86_FEATURE_VNMI (15*32+25) /* "vnmi" Virtual NMI */ diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index 3776cf5382a2..83dc5086138b 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -94,11 +94,10 @@ KVM_X86_OP_OPTIONAL(sync_pir_to_irr) KVM_X86_OP_OPTIONAL_RET0(set_tss_addr) KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr) KVM_X86_OP_OPTIONAL_RET0(get_mt_mask) +KVM_X86_OP_OPTIONAL_RET0(tdp_has_smep) KVM_X86_OP(load_mmu_pgd) -KVM_X86_OP_OPTIONAL(link_external_spt) -KVM_X86_OP_OPTIONAL(set_external_spte) +KVM_X86_OP_OPTIONAL_RET0(set_external_spte) KVM_X86_OP_OPTIONAL(free_external_spt) -KVM_X86_OP_OPTIONAL(remove_external_spte) KVM_X86_OP(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier) diff --git a/arch/x86/include/asm/kvm-x86-pmu-ops.h b/arch/x86/include/asm/kvm-x86-pmu-ops.h index d5452b3433b7..4a223c2793e3 100644 --- a/arch/x86/include/asm/kvm-x86-pmu-ops.h +++ b/arch/x86/include/asm/kvm-x86-pmu-ops.h @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #if !defined(KVM_X86_PMU_OP) || \ - !defined(KVM_X86_PMU_OP_OPTIONAL) + !defined(KVM_X86_PMU_OP_OPTIONAL) || \ + !defined(KVM_X86_PMU_OP_OPTIONAL_RET0) #error Missing one or more KVM_X86_PMU_OP #defines #else @@ -23,6 +24,7 @@ KVM_X86_PMU_OP(init) KVM_X86_PMU_OP_OPTIONAL(reset) KVM_X86_PMU_OP_OPTIONAL(deliver_pmi) KVM_X86_PMU_OP_OPTIONAL(cleanup) +KVM_X86_PMU_OP_OPTIONAL_RET0(pmc_is_disabled_in_current_mode) KVM_X86_PMU_OP_OPTIONAL(write_global_ctrl) KVM_X86_PMU_OP(mediated_load) @@ -31,3 +33,4 @@ KVM_X86_PMU_OP(mediated_put) #undef KVM_X86_PMU_OP #undef KVM_X86_PMU_OP_OPTIONAL +#undef KVM_X86_PMU_OP_OPTIONAL_RET0 diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index f14009f25a3b..eee473717c0e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -182,20 +182,21 @@ enum kvm_reg { VCPU_REGS_RSI = __VCPU_REGS_RSI, VCPU_REGS_RDI = __VCPU_REGS_RDI, #ifdef CONFIG_X86_64 - VCPU_REGS_R8 = __VCPU_REGS_R8, - VCPU_REGS_R9 = __VCPU_REGS_R9, - VCPU_REGS_R10 = __VCPU_REGS_R10, - VCPU_REGS_R11 = __VCPU_REGS_R11, - VCPU_REGS_R12 = __VCPU_REGS_R12, - VCPU_REGS_R13 = __VCPU_REGS_R13, - VCPU_REGS_R14 = __VCPU_REGS_R14, - VCPU_REGS_R15 = __VCPU_REGS_R15, + VCPU_REGS_R8 = 8, + VCPU_REGS_R9, + VCPU_REGS_R10, + VCPU_REGS_R11, + VCPU_REGS_R12, + VCPU_REGS_R13, + VCPU_REGS_R14, + VCPU_REGS_R15, #endif - VCPU_REGS_RIP, - NR_VCPU_REGS, + NR_VCPU_GENERAL_PURPOSE_REGS, - VCPU_EXREG_PDPTR = NR_VCPU_REGS, - VCPU_EXREG_CR0, + VCPU_REG_RIP = NR_VCPU_GENERAL_PURPOSE_REGS, + + VCPU_REG_PDPTR, + VCPU_REG_CR0, /* * Alias AMD's ERAPS (not a real register) to CR3 so that common code * can trigger emulation of the RAP (Return Address Predictor) with @@ -203,13 +204,15 @@ enum kvm_reg { * is cleared on writes to CR3, i.e. marking CR3 dirty will naturally * mark ERAPS dirty as well. */ - VCPU_EXREG_CR3, - VCPU_EXREG_ERAPS = VCPU_EXREG_CR3, - VCPU_EXREG_CR4, - VCPU_EXREG_RFLAGS, - VCPU_EXREG_SEGMENTS, - VCPU_EXREG_EXIT_INFO_1, - VCPU_EXREG_EXIT_INFO_2, + VCPU_REG_CR3, + VCPU_REG_ERAPS = VCPU_REG_CR3, + VCPU_REG_CR4, + VCPU_REG_RFLAGS, + VCPU_REG_SEGMENTS, + VCPU_REG_EXIT_INFO_1, + VCPU_REG_EXIT_INFO_2, + + NR_VCPU_TOTAL_REGS, }; enum { @@ -281,6 +284,8 @@ enum x86_intercept_stage; #define PFERR_GUEST_RMP_MASK BIT_ULL(31) #define PFERR_GUEST_FINAL_MASK BIT_ULL(32) #define PFERR_GUEST_PAGE_MASK BIT_ULL(33) +#define PFERR_GUEST_FAULT_STAGE_MASK \ + (PFERR_GUEST_FINAL_MASK | PFERR_GUEST_PAGE_MASK) #define PFERR_GUEST_ENC_MASK BIT_ULL(34) #define PFERR_GUEST_SIZEM_MASK BIT_ULL(35) #define PFERR_GUEST_VMPL_MASK BIT_ULL(36) @@ -328,11 +333,11 @@ struct kvm_kernel_irq_routing_entry; * the number of unique SPs that can theoretically be created is 2^n, where n * is the number of bits that are used to compute the role. * - * But, even though there are 20 bits in the mask below, not all combinations + * But, even though there are 21 bits in the mask below, not all combinations * of modes and flags are possible: * * - invalid shadow pages are not accounted, mirror pages are not shadowed, - * so the bits are effectively 18. + * so the bits are effectively 19. * * - quadrant will only be used if has_4_byte_gpte=1 (non-PAE paging); * execonly and ad_disabled are only used for nested EPT which has @@ -343,11 +348,11 @@ struct kvm_kernel_irq_routing_entry; * paging has exactly one upper level, making level completely redundant * when has_4_byte_gpte=1. * - * - on top of this, smep_andnot_wp and smap_andnot_wp are only set if - * cr0_wp=0, therefore these three bits only give rise to 5 possibilities. + * - on top of this, smap_andnot_wp is only set if cr0_wp=0, + * therefore these two bits only give rise to 3 possibilities. * * Therefore, the maximum number of possible upper-level shadow pages for a - * single gfn is a bit less than 2^13. + * single gfn is a bit less than 2^14. */ union kvm_mmu_page_role { u32 word; @@ -356,17 +361,26 @@ union kvm_mmu_page_role { unsigned has_4_byte_gpte:1; unsigned quadrant:2; unsigned direct:1; - unsigned access:3; + unsigned access:4; unsigned invalid:1; unsigned efer_nx:1; unsigned cr0_wp:1; - unsigned smep_andnot_wp:1; unsigned smap_andnot_wp:1; unsigned ad_disabled:1; unsigned guest_mode:1; unsigned passthrough:1; unsigned is_mirror:1; - unsigned :4; + + /* + * cr4_smep is also set for EPT MBEC. Because it affects + * which pages are considered non-present (bit 10 additionally + * must be zero if MBEC is on) it has to be in the base role. + * It also has to be in the base role for AMD GMET because + * kernel-executable pages need to have U=0 with GMET enabled. + */ + unsigned cr4_smep:1; + + unsigned:3; /* * This is left at the top of the word so that @@ -392,10 +406,10 @@ union kvm_mmu_page_role { * tables (because KVM doesn't support Protection Keys with shadow paging), and * CR0.PG, CR4.PAE, and CR4.PSE are indirectly reflected in role.level. * - * Note, SMEP and SMAP are not redundant with sm*p_andnot_wp in the page role. - * If CR0.WP=1, KVM can reuse shadow pages for the guest regardless of SMEP and - * SMAP, but the MMU's permission checks for software walks need to be SMEP and - * SMAP aware regardless of CR0.WP. + * Note, SMAP is not redundant with smap_andnot_wp in the page role. If + * CR0.WP=1, KVM can reuse shadow pages for the guest regardless of SMAP, + * but the MMU's permission checks for software walks need to be SMAP + * aware regardless of CR0.WP. */ union kvm_mmu_extended_role { u32 word; @@ -405,9 +419,15 @@ union kvm_mmu_extended_role { unsigned int cr4_pse:1; unsigned int cr4_pke:1; unsigned int cr4_smap:1; - unsigned int cr4_smep:1; unsigned int cr4_la57:1; unsigned int efer_lma:1; + + /* + * True if either CR4.SMEP or EFER.NXE are set. For AMD NPT + * this is the "real" host CR4.SMEP whereas cr4_smep is + * actually GMET. + */ + unsigned int has_pferr_fetch:1; }; }; @@ -466,7 +486,8 @@ struct kvm_mmu { u64 (*get_pdptr)(struct kvm_vcpu *vcpu, int index); int (*page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); void (*inject_page_fault)(struct kvm_vcpu *vcpu, - struct x86_exception *fault); + struct x86_exception *fault, + bool from_hardware); gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, gpa_t gva_or_gpa, u64 access, struct x86_exception *exception); @@ -492,7 +513,7 @@ struct kvm_mmu { * Byte index: page fault error code [4:1] * Bit index: pte permissions in ACC_* format */ - u8 permissions[16]; + u16 permissions[16]; u64 *pae_root; u64 *pml4_root; @@ -504,10 +525,7 @@ struct kvm_mmu { * the bits spte never used. */ struct rsvd_bits_validate shadow_zero_check; - struct rsvd_bits_validate guest_rsvd_check; - - u64 pdptrs[4]; /* pae */ }; enum pmc_type { @@ -594,6 +612,8 @@ struct kvm_pmu { DECLARE_BITMAP(pmc_counting_instructions, X86_PMC_IDX_MAX); DECLARE_BITMAP(pmc_counting_branches, X86_PMC_IDX_MAX); + DECLARE_BITMAP(pmc_has_mode_specific_enables, X86_PMC_IDX_MAX); + u64 ds_area; u64 pebs_enable; u64 pebs_enable_rsvd; @@ -799,9 +819,10 @@ struct kvm_vcpu_arch { * rip and regs accesses must go through * kvm_{register,rip}_{read,write} functions. */ - unsigned long regs[NR_VCPU_REGS]; - u32 regs_avail; - u32 regs_dirty; + unsigned long regs[NR_VCPU_GENERAL_PURPOSE_REGS]; + unsigned long rip; + DECLARE_BITMAP(regs_avail, NR_VCPU_TOTAL_REGS); + DECLARE_BITMAP(regs_dirty, NR_VCPU_TOTAL_REGS); unsigned long cr0; unsigned long cr0_guest_owned_bits; @@ -864,6 +885,8 @@ struct kvm_vcpu_arch { */ struct kvm_mmu *walk_mmu; + u64 pdptrs[4]; /* pae */ + struct kvm_mmu_memory_cache mmu_pte_list_desc_cache; struct kvm_mmu_memory_cache mmu_shadow_page_cache; struct kvm_mmu_memory_cache mmu_shadowed_info_cache; @@ -1038,8 +1061,6 @@ struct kvm_vcpu_arch { u16 vec; u32 id; u32 host_apf_flags; - bool send_always; - bool delivery_as_pf_vmexit; bool pageready_pending; } apf; @@ -1422,6 +1443,7 @@ struct kvm_arch { bool has_private_mem; bool has_protected_state; bool has_protected_eoi; + bool has_protected_pmu; bool pre_fault_allowed; struct hlist_head *mmu_page_hash; struct list_head active_mmu_pages; @@ -1888,24 +1910,18 @@ struct kvm_x86_ops { int (*set_tss_addr)(struct kvm *kvm, unsigned int addr); int (*set_identity_map_addr)(struct kvm *kvm, u64 ident_addr); u8 (*get_mt_mask)(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); + bool (*tdp_has_smep)(struct kvm *kvm); void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level); - /* Update external mapping with page table link. */ - int (*link_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level, - void *external_spt); /* Update the external page table from spte getting set. */ - int (*set_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level, - u64 mirror_spte); + int (*set_external_spte)(struct kvm *kvm, gfn_t gfn, u64 old_spte, + u64 new_spte, enum pg_level level); /* Update external page tables for page table about to be freed. */ - int (*free_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level, - void *external_spt); + void (*free_external_spt)(struct kvm *kvm, struct kvm_mmu_page *sp); - /* Update external page table from spte getting removed, and flush TLB. */ - void (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level, - u64 mirror_spte); bool (*has_wbinvd_exit)(void); @@ -2011,6 +2027,10 @@ struct kvm_x86_nested_ops { struct kvm_nested_state *kvm_state); bool (*get_nested_state_pages)(struct kvm_vcpu *vcpu); int (*write_log_dirty)(struct kvm_vcpu *vcpu, gpa_t l2_gpa); + gpa_t (*translate_nested_gpa)(struct kvm_vcpu *vcpu, gpa_t gpa, + u64 access, + struct x86_exception *exception, + u64 pte_access); int (*enable_evmcs)(struct kvm_vcpu *vcpu, uint16_t *vmcs_version); @@ -2042,7 +2062,6 @@ extern bool __read_mostly enable_device_posted_irqs; extern struct kvm_x86_ops kvm_x86_ops; #define kvm_x86_call(func) static_call(kvm_x86_##func) -#define kvm_pmu_call(func) static_call(kvm_x86_pmu_##func) #define KVM_X86_OP(func) \ DECLARE_STATIC_CALL(kvm_x86_##func, *(((struct kvm_x86_ops *)0)->func)); @@ -2131,8 +2150,6 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); extern bool tdp_enabled; -u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu); - /* * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after completing * userspace I/O) to indicate that the emulation context @@ -2252,7 +2269,6 @@ int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu); void kvm_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg); void kvm_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg); -int kvm_load_segment_descriptor(struct kvm_vcpu *vcpu, u16 selector, int seg); void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, @@ -2283,10 +2299,18 @@ void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code); void kvm_queue_exception_p(struct kvm_vcpu *vcpu, unsigned nr, unsigned long payload); void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned int nr, bool has_error_code, u32 error_code); -void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault); -void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, - struct x86_exception *fault); -bool kvm_require_cpl(struct kvm_vcpu *vcpu, int required_cpl); +void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault, + bool from_hardware); +void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, + struct x86_exception *fault, + bool from_hardware); + +static inline void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, + struct x86_exception *fault) +{ + __kvm_inject_emulated_page_fault(vcpu, fault, false); +} + bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr); static inline int __kvm_irq_line_state(unsigned long *irq_state, @@ -2527,7 +2551,8 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); KVM_X86_QUIRK_SLOT_ZAP_ALL | \ KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \ KVM_X86_QUIRK_IGNORE_GUEST_PAT | \ - KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM) + KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM | \ + KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT) #define KVM_X86_CONDITIONAL_QUIRKS \ (KVM_X86_QUIRK_CD_NW_CLEARED | \ diff --git a/arch/x86/include/asm/kvm_vcpu_regs.h b/arch/x86/include/asm/kvm_vcpu_regs.h index 1af2cb59233b..d9f1528fab2b 100644 --- a/arch/x86/include/asm/kvm_vcpu_regs.h +++ b/arch/x86/include/asm/kvm_vcpu_regs.h @@ -11,15 +11,108 @@ #define __VCPU_REGS_RSI 6 #define __VCPU_REGS_RDI 7 +#ifdef __ASSEMBLER__ + +#define REG_NUM_INVALID 100 + + # convert the 32-bit register operand \r32 into a register number; + # store it in the value whose name is in \opd. + # only for !CONFIG_X86_64 (does not support r8d-r15d) + .macro R32_NUM opd r32 + \opd = REG_NUM_INVALID + .ifc \r32,%eax + \opd = __VCPU_REGS_RAX + .endif + .ifc \r32,%ecx + \opd = __VCPU_REGS_RCX + .endif + .ifc \r32,%edx + \opd = __VCPU_REGS_RDX + .endif + .ifc \r32,%ebx + \opd = __VCPU_REGS_RBX + .endif + .ifc \r32,%esp + \opd = __VCPU_REGS_RSP + .endif + .ifc \r32,%ebp + \opd = __VCPU_REGS_RBP + .endif + .ifc \r32,%esi + \opd = __VCPU_REGS_RSI + .endif + .ifc \r32,%edi + \opd = __VCPU_REGS_RDI + .endif + .endm + + # convert the 64-bit register operand \r64 into a register number; + # store it in the value whose name is in \opd. + .macro R64_NUM opd r64 + \opd = REG_NUM_INVALID +#ifdef CONFIG_X86_64 + .ifc \r64,%rax + \opd = __VCPU_REGS_RAX + .endif + .ifc \r64,%rcx + \opd = __VCPU_REGS_RCX + .endif + .ifc \r64,%rdx + \opd = __VCPU_REGS_RDX + .endif + .ifc \r64,%rbx + \opd = __VCPU_REGS_RBX + .endif + .ifc \r64,%rsp + \opd = __VCPU_REGS_RSP + .endif + .ifc \r64,%rbp + \opd = __VCPU_REGS_RBP + .endif + .ifc \r64,%rsi + \opd = __VCPU_REGS_RSI + .endif + .ifc \r64,%rdi + \opd = __VCPU_REGS_RDI + .endif + .ifc \r64,%r8 + \opd = 8 + .endif + .ifc \r64,%r9 + \opd = 9 + .endif + .ifc \r64,%r10 + \opd = 10 + .endif + .ifc \r64,%r11 + \opd = 11 + .endif + .ifc \r64,%r12 + \opd = 12 + .endif + .ifc \r64,%r13 + \opd = 13 + .endif + .ifc \r64,%r14 + \opd = 14 + .endif + .ifc \r64,%r15 + \opd = 15 + .endif +#endif + .endm + +.macro REG_NUM reg_num reg #ifdef CONFIG_X86_64 -#define __VCPU_REGS_R8 8 -#define __VCPU_REGS_R9 9 -#define __VCPU_REGS_R10 10 -#define __VCPU_REGS_R11 11 -#define __VCPU_REGS_R12 12 -#define __VCPU_REGS_R13 13 -#define __VCPU_REGS_R14 14 -#define __VCPU_REGS_R15 15 + R64_NUM \reg_num \reg +#else + R32_NUM \reg_num \reg +#endif + .if \reg_num == REG_NUM_INVALID + .error "invalid register" + .endif +.endm + #endif #endif /* _ASM_X86_KVM_VCPU_REGS_H */ diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 86554de9a3f5..18c4be75e927 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -899,6 +899,7 @@ #define MSR_K7_HWCR_IRPERF_EN_BIT 30 #define MSR_K7_HWCR_IRPERF_EN BIT_ULL(MSR_K7_HWCR_IRPERF_EN_BIT) #define MSR_K7_HWCR_CPUID_USER_DIS_BIT 35 +#define MSR_K7_HWCR_CPUID_USER_DIS BIT_ULL(MSR_K7_HWCR_CPUID_USER_DIS_BIT) #define MSR_K7_FID_VID_CTL 0xc0010041 #define MSR_K7_FID_VID_STATUS 0xc0010042 #define MSR_K7_HWCR_CPB_DIS_BIT 25 diff --git a/arch/x86/include/asm/perf_event.h b/arch/x86/include/asm/perf_event.h index 752cb319d5ea..1eb13673e889 100644 --- a/arch/x86/include/asm/perf_event.h +++ b/arch/x86/include/asm/perf_event.h @@ -60,6 +60,8 @@ #define AMD64_EVENTSEL_INT_CORE_ENABLE (1ULL << 36) #define AMD64_EVENTSEL_GUESTONLY (1ULL << 40) #define AMD64_EVENTSEL_HOSTONLY (1ULL << 41) +#define AMD64_EVENTSEL_HOST_GUEST_MASK \ + (AMD64_EVENTSEL_HOSTONLY | AMD64_EVENTSEL_GUESTONLY) #define AMD64_EVENTSEL_INT_CORE_SEL_SHIFT 37 #define AMD64_EVENTSEL_INT_CORE_SEL_MASK \ diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h index bcfeb5e7c0ed..aa63431ba92c 100644 --- a/arch/x86/include/asm/svm.h +++ b/arch/x86/include/asm/svm.h @@ -243,6 +243,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area { #define SVM_MISC_ENABLE_NP BIT(0) #define SVM_MISC_ENABLE_SEV BIT(1) #define SVM_MISC_ENABLE_SEV_ES BIT(2) +#define SVM_MISC_ENABLE_GMET BIT(3) #define SVM_MISC2_ENABLE_V_LBR BIT_ULL(0) #define SVM_MISC2_ENABLE_V_VMLOAD_VMSAVE BIT_ULL(1) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index e5a9cf656c07..89e97d5761d8 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -6,6 +6,7 @@ #include <linux/init.h> #include <linux/bits.h> #include <linux/mmzone.h> +#include <linux/kvm_types.h> #include <asm/errno.h> #include <asm/ptrace.h> @@ -121,7 +122,7 @@ int tdx_guest_keyid_alloc(void); u32 tdx_get_nr_guest_keyids(void); void tdx_guest_keyid_free(unsigned int keyid); -void tdx_quirk_reset_page(struct page *page); +void tdx_quirk_reset_paddr(unsigned long base, unsigned long size); struct tdx_td { /* TD root structure: */ @@ -145,32 +146,17 @@ struct tdx_vp { struct page **tdcx_pages; }; -static inline u64 mk_keyed_paddr(u16 hkid, struct page *page) -{ - u64 ret; - - ret = page_to_phys(page); - /* KeyID bits are just above the physical address bits: */ - ret |= (u64)hkid << boot_cpu_data.x86_phys_bits; - - return ret; -} - -static inline int pg_level_to_tdx_sept_level(enum pg_level level) -{ - WARN_ON_ONCE(level == PG_LEVEL_NONE); - return level - 1; -} - void tdx_sys_disable(void); u64 tdh_vp_enter(struct tdx_vp *vp, struct tdx_module_args *args); u64 tdh_mng_addcx(struct tdx_td *td, struct page *tdcs_page); -u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, struct page *page, struct page *source, u64 *ext_err1, u64 *ext_err2); -u64 tdh_mem_sept_add(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2); +u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, kvm_pfn_t pfn, struct page *source, + u64 *ext_err1, u64 *ext_err2); +u64 tdh_mem_sept_add(struct tdx_td *td, u64 gpa, enum pg_level level, struct page *page, u64 *ext_err1, u64 *ext_err2); u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page); -u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2); -u64 tdh_mem_range_block(struct tdx_td *td, u64 gpa, int level, u64 *ext_err1, u64 *ext_err2); +u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, enum pg_level level, kvm_pfn_t pfn, + u64 *ext_err1, u64 *ext_err2); +u64 tdh_mem_range_block(struct tdx_td *td, u64 gpa, enum pg_level level, u64 *ext_err1, u64 *ext_err2); u64 tdh_mng_key_config(struct tdx_td *td); u64 tdh_mng_create(struct tdx_td *td, u16 hkid); u64 tdh_vp_create(struct tdx_td *td, struct tdx_vp *vp); @@ -186,10 +172,10 @@ u64 tdh_vp_rd(struct tdx_vp *vp, u64 field, u64 *data); u64 tdh_vp_wr(struct tdx_vp *vp, u64 field, u64 data, u64 mask); u64 tdh_phymem_page_reclaim(struct page *page, u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size); u64 tdh_mem_track(struct tdx_td *tdr); -u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, u64 level, u64 *ext_err1, u64 *ext_err2); +u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, enum pg_level level, u64 *ext_err1, u64 *ext_err2); u64 tdh_phymem_cache_wb(bool resume); u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td); -u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page); +u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, kvm_pfn_t pfn); #else static inline void tdx_init(void) { } static inline u32 tdx_get_nr_guest_keyids(void) { return 0; } diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index 49d8551d285d..3f1b3096ff04 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -536,6 +536,7 @@ enum vmcs_field { #define VMX_EPT_1GB_PAGE_BIT (1ull << 17) #define VMX_EPT_INVEPT_BIT (1ull << 20) #define VMX_EPT_AD_BIT (1ull << 21) +#define VMX_EPT_ADVANCED_VMEXIT_INFO_BIT (1ull << 22) #define VMX_EPT_EXTENT_CONTEXT_BIT (1ull << 25) #define VMX_EPT_EXTENT_GLOBAL_BIT (1ull << 26) @@ -561,10 +562,12 @@ enum vmcs_field { #define VMX_EPT_ACCESS_BIT (1ull << 8) #define VMX_EPT_DIRTY_BIT (1ull << 9) #define VMX_EPT_SUPPRESS_VE_BIT (1ull << 63) + #define VMX_EPT_RWX_MASK (VMX_EPT_READABLE_MASK | \ VMX_EPT_WRITABLE_MASK | \ VMX_EPT_EXECUTABLE_MASK) #define VMX_EPT_MT_MASK (7ull << VMX_EPT_MT_EPTE_SHIFT) +#define VMX_EPT_USER_EXECUTABLE_MASK (1ull << 10) static inline u8 vmx_eptp_page_walk_level(u64 eptp) { @@ -609,17 +612,24 @@ enum vm_entry_failure_code { #define EPT_VIOLATION_PROT_READ BIT(3) #define EPT_VIOLATION_PROT_WRITE BIT(4) #define EPT_VIOLATION_PROT_EXEC BIT(5) -#define EPT_VIOLATION_EXEC_FOR_RING3_LIN BIT(6) +#define EPT_VIOLATION_PROT_USER_EXEC BIT(6) #define EPT_VIOLATION_PROT_MASK (EPT_VIOLATION_PROT_READ | \ EPT_VIOLATION_PROT_WRITE | \ - EPT_VIOLATION_PROT_EXEC) + EPT_VIOLATION_PROT_EXEC | \ + EPT_VIOLATION_PROT_USER_EXEC) #define EPT_VIOLATION_GVA_IS_VALID BIT(7) #define EPT_VIOLATION_GVA_TRANSLATED BIT(8) +#define EPT_VIOLATION_GVA_USER BIT(9) +#define EPT_VIOLATION_GVA_WRITABLE BIT(10) +#define EPT_VIOLATION_GVA_NX BIT(11) #define EPT_VIOLATION_RWX_TO_PROT(__epte) (((__epte) & VMX_EPT_RWX_MASK) << 3) +#define EPT_VIOLATION_USER_EXEC_TO_PROT(__epte) (((__epte) & VMX_EPT_USER_EXECUTABLE_MASK) >> 4) static_assert(EPT_VIOLATION_RWX_TO_PROT(VMX_EPT_RWX_MASK) == (EPT_VIOLATION_PROT_READ | EPT_VIOLATION_PROT_WRITE | EPT_VIOLATION_PROT_EXEC)); +static_assert(EPT_VIOLATION_USER_EXEC_TO_PROT(VMX_EPT_USER_EXECUTABLE_MASK) == + (EPT_VIOLATION_PROT_USER_EXEC)); /* * Exit Qualifications for NOTIFY VM EXIT diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h index 5f2b30d0405c..1585ec804066 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -477,6 +477,7 @@ struct kvm_sync_regs { #define KVM_X86_QUIRK_STUFF_FEATURE_MSRS (1 << 8) #define KVM_X86_QUIRK_IGNORE_GUEST_PAT (1 << 9) #define KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM (1 << 10) +#define KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT (1 << 11) #define KVM_STATE_NESTED_FORMAT_VMX 0 #define KVM_STATE_NESTED_FORMAT_SVM 1 @@ -532,6 +533,7 @@ struct kvm_svm_nested_state_data { struct kvm_svm_nested_state_hdr { __u64 vmcb_pa; + __u64 gpat; }; /* for KVM_CAP_NESTED_STATE */ diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index e69156b54cff..591d2294acd7 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -11,7 +11,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include <linux/kvm_host.h> -#include "linux/lockdep.h" +#include <linux/lockdep.h> #include <linux/export.h> #include <linux/vmalloc.h> #include <linux/uaccess.h> @@ -1248,7 +1248,7 @@ void kvm_initialize_cpu_caps(void) F(AUTOIBRS), EMULATED_F(NO_SMM_CTL_MSR), /* PrefetchCtlMsr */ - /* GpOnUserCpuid */ + EMULATED_F(GP_ON_USER_CPUID), /* EPSF */ F(PREFETCHI), F(AVX512_BMM), @@ -2161,17 +2161,18 @@ int kvm_emulate_cpuid(struct kvm_vcpu *vcpu) { u32 eax, ebx, ecx, edx; - if (!is_smm(vcpu) && cpuid_fault_enabled(vcpu) && - !kvm_require_cpl(vcpu, 0)) + if (!kvm_is_cpuid_allowed(vcpu)) { + kvm_queue_exception_e(vcpu, GP_VECTOR, 0); return 1; + } - eax = kvm_rax_read(vcpu); - ecx = kvm_rcx_read(vcpu); + eax = kvm_eax_read(vcpu); + ecx = kvm_ecx_read(vcpu); kvm_cpuid(vcpu, &eax, &ebx, &ecx, &edx, false); - kvm_rax_write(vcpu, eax); - kvm_rbx_write(vcpu, ebx); - kvm_rcx_write(vcpu, ecx); - kvm_rdx_write(vcpu, edx); + kvm_eax_write(vcpu, eax); + kvm_ebx_write(vcpu, ebx); + kvm_ecx_write(vcpu, ecx); + kvm_edx_write(vcpu, edx); return kvm_skip_emulated_instruction(vcpu); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_cpuid); diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h index 039b8e6f40ba..8d863f45585d 100644 --- a/arch/x86/kvm/cpuid.h +++ b/arch/x86/kvm/cpuid.h @@ -7,6 +7,8 @@ #include <asm/processor.h> #include <uapi/asm/kvm_para.h> +#include "smm.h" + extern u32 kvm_cpu_caps[NR_KVM_CPU_CAPS] __read_mostly; extern bool kvm_is_configuring_cpu_caps __read_mostly; @@ -124,7 +126,7 @@ static __always_inline bool guest_cpuid_has(struct kvm_vcpu *vcpu, entry = kvm_find_cpuid_entry_index(vcpu, cpuid.function, cpuid.index); if (!entry) - return NULL; + return false; reg = __cpuid_entry_get_reg(entry, cpuid.reg); if (!reg) @@ -181,15 +183,17 @@ static inline int guest_cpuid_stepping(struct kvm_vcpu *vcpu) return x86_stepping(best->eax); } -static inline bool supports_cpuid_fault(struct kvm_vcpu *vcpu) +static inline bool cpuid_fault_enabled(struct kvm_vcpu *vcpu) { - return vcpu->arch.msr_platform_info & MSR_PLATFORM_INFO_CPUID_FAULT; + return (vcpu->arch.msr_misc_features_enables & + MSR_MISC_FEATURES_ENABLES_CPUID_FAULT) || + (vcpu->arch.msr_hwcr & MSR_K7_HWCR_CPUID_USER_DIS); } -static inline bool cpuid_fault_enabled(struct kvm_vcpu *vcpu) +static inline bool kvm_is_cpuid_allowed(struct kvm_vcpu *vcpu) { - return vcpu->arch.msr_misc_features_enables & - MSR_MISC_FEATURES_ENABLES_CPUID_FAULT; + return !cpuid_fault_enabled(vcpu) || is_smm(vcpu) || + !kvm_x86_call(get_cpl)(vcpu); } static __always_inline void kvm_cpu_cap_clear(unsigned int x86_feature) diff --git a/arch/x86/kvm/debugfs.c b/arch/x86/kvm/debugfs.c index 999227fc7c66..0074a56e45b4 100644 --- a/arch/x86/kvm/debugfs.c +++ b/arch/x86/kvm/debugfs.c @@ -121,7 +121,7 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v) lpage_size = kvm_mmu_slot_lpages(slot, k + 1); cur = log[k]; for (l = 0; l < lpage_size; l++) { - index = ffs(pte_list_count(&rmap[l])); + index = fls(pte_list_count(&rmap[l])); if (WARN_ON_ONCE(index >= RMAP_LOG_SIZE)) index = RMAP_LOG_SIZE - 1; cur[index]++; diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index 8013dccb3110..b566ab5c7515 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -20,7 +20,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include <linux/kvm_host.h> -#include "kvm_cache_regs.h" +#include "regs.h" #include "kvm_emulate.h" #include <linux/stringify.h> #include <asm/debugreg.h> @@ -540,8 +540,9 @@ static int emulate_exception(struct x86_emulate_ctxt *ctxt, int vec, return X86EMUL_PROPAGATE_FAULT; } -static int emulate_db(struct x86_emulate_ctxt *ctxt) +static int emulate_db(struct x86_emulate_ctxt *ctxt, unsigned long dr6) { + ctxt->exception.dr6 = dr6; return emulate_exception(ctxt, DB_VECTOR, 0, false); } @@ -3593,12 +3594,8 @@ static int em_sti(struct x86_emulate_ctxt *ctxt) static int em_cpuid(struct x86_emulate_ctxt *ctxt) { u32 eax, ebx, ecx, edx; - u64 msr = 0; - ctxt->ops->get_msr(ctxt, MSR_MISC_FEATURES_ENABLES, &msr); - if (!ctxt->ops->is_smm(ctxt) && - (msr & MSR_MISC_FEATURES_ENABLES_CPUID_FAULT) && - ctxt->ops->cpl(ctxt)) + if (!ctxt->ops->is_cpuid_allowed(ctxt)) return emulate_gp(ctxt, 0); eax = reg_read(ctxt, VCPU_REGS_RAX); @@ -3847,15 +3844,8 @@ static int check_dr_read(struct x86_emulate_ctxt *ctxt) if ((cr4 & X86_CR4_DE) && (dr == 4 || dr == 5)) return emulate_ud(ctxt); - if (ctxt->ops->get_dr(ctxt, 7) & DR7_GD) { - ulong dr6; - - dr6 = ctxt->ops->get_dr(ctxt, 6); - dr6 &= ~DR_TRAP_BITS; - dr6 |= DR6_BD | DR6_ACTIVE_LOW; - ctxt->ops->set_dr(ctxt, 6, dr6); - return emulate_db(ctxt); - } + if (ctxt->ops->get_effective_dr7(ctxt) & DR7_GD) + return emulate_db(ctxt, DR6_BD); return X86EMUL_CONTINUE; } diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index 4438ecac9a89..fd4eb1e561f7 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -1839,6 +1839,11 @@ static bool hv_is_vp_in_sparse_set(u32 vp_id, u64 valid_bank_mask, u64 sparse_ba int valid_bit_nr = vp_id / HV_VCPUS_PER_SPARSE_BANK; unsigned long sbank; + BUILD_BUG_ON(BITS_PER_TYPE(valid_bank_mask) != HV_MAX_SPARSE_VCPU_BANKS); + + if (valid_bit_nr >= HV_MAX_SPARSE_VCPU_BANKS) + return false; + if (!test_bit(valid_bit_nr, (unsigned long *)&valid_bank_mask)) return false; @@ -2041,7 +2046,9 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) * read with kvm_read_guest(). */ if (!hc->fast && mmu_is_nested(vcpu)) { - hc->ingpa = translate_nested_gpa(vcpu, hc->ingpa, 0, NULL); + hc->ingpa = kvm_x86_ops.nested_ops->translate_nested_gpa( + vcpu, hc->ingpa, + PFERR_GUEST_FINAL_MASK, NULL, 0); if (unlikely(hc->ingpa == INVALID_GPA)) return HV_STATUS_INVALID_HYPERCALL_INPUT; } @@ -2375,10 +2382,10 @@ static void kvm_hv_hypercall_set_result(struct kvm_vcpu *vcpu, u64 result) longmode = is_64_bit_hypercall(vcpu); if (longmode) - kvm_rax_write(vcpu, result); + kvm_rax_write_raw(vcpu, result); else { - kvm_rdx_write(vcpu, result >> 32); - kvm_rax_write(vcpu, result & 0xffffffff); + kvm_edx_write(vcpu, result >> 32); + kvm_eax_write(vcpu, result); } } @@ -2542,18 +2549,15 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) #ifdef CONFIG_X86_64 if (is_64_bit_hypercall(vcpu)) { - hc.param = kvm_rcx_read(vcpu); - hc.ingpa = kvm_rdx_read(vcpu); - hc.outgpa = kvm_r8_read(vcpu); + hc.param = kvm_rcx_read_raw(vcpu); + hc.ingpa = kvm_rdx_read_raw(vcpu); + hc.outgpa = kvm_r8_read_raw(vcpu); } else #endif { - hc.param = ((u64)kvm_rdx_read(vcpu) << 32) | - (kvm_rax_read(vcpu) & 0xffffffff); - hc.ingpa = ((u64)kvm_rbx_read(vcpu) << 32) | - (kvm_rcx_read(vcpu) & 0xffffffff); - hc.outgpa = ((u64)kvm_rdi_read(vcpu) << 32) | - (kvm_rsi_read(vcpu) & 0xffffffff); + hc.param = ((u64)kvm_edx_read(vcpu) << 32) | kvm_eax_read(vcpu); + hc.ingpa = ((u64)kvm_ebx_read(vcpu) << 32) | kvm_ecx_read(vcpu); + hc.outgpa = ((u64)kvm_edi_read(vcpu) << 32) | kvm_esi_read(vcpu); } hc.code = hc.param & 0xffff; diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h index 6301f79fcbae..65e89ed65349 100644 --- a/arch/x86/kvm/hyperv.h +++ b/arch/x86/kvm/hyperv.h @@ -232,8 +232,8 @@ static inline bool kvm_hv_is_tlb_flush_hcall(struct kvm_vcpu *vcpu) if (!hv_vcpu) return false; - code = is_64_bit_hypercall(vcpu) ? kvm_rcx_read(vcpu) : - kvm_rax_read(vcpu); + code = is_64_bit_hypercall(vcpu) ? kvm_rcx_read_raw(vcpu) : + kvm_eax_read(vcpu); return (code == HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE || code == HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST || diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c index eed96ff6e722..f3f4a483ca15 100644 --- a/arch/x86/kvm/ioapic.c +++ b/arch/x86/kvm/ioapic.c @@ -441,7 +441,7 @@ static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val) irq.dest_id = old_dest_id; irq.dest_mode = kvm_lapic_irq_dest_mode( - !!e->fields.dest_mode); + !!old_dest_mode); kvm_bitmap_or_dest_vcpus(ioapic->kvm, &irq, vcpu_bitmap); } diff --git a/arch/x86/kvm/irq.c b/arch/x86/kvm/irq.c index 9519fec09ee6..8c62c6d4d5c1 100644 --- a/arch/x86/kvm/irq.c +++ b/arch/x86/kvm/irq.c @@ -585,12 +585,16 @@ int kvm_vm_ioctl_get_irqchip(struct kvm *kvm, struct kvm_irqchip *chip) r = 0; switch (chip->chip_id) { case KVM_IRQCHIP_PIC_MASTER: + spin_lock(&pic->lock); memcpy(&chip->chip.pic, &pic->pics[0], sizeof(struct kvm_pic_state)); + spin_unlock(&pic->lock); break; case KVM_IRQCHIP_PIC_SLAVE: + spin_lock(&pic->lock); memcpy(&chip->chip.pic, &pic->pics[1], sizeof(struct kvm_pic_state)); + spin_unlock(&pic->lock); break; case KVM_IRQCHIP_IOAPIC: kvm_get_ioapic(kvm, &chip->chip.ioapic); diff --git a/arch/x86/kvm/kvm-asm-offsets.c b/arch/x86/kvm/kvm-asm-offsets.c index 24a710d37323..36ac61724dd7 100644 --- a/arch/x86/kvm/kvm-asm-offsets.c +++ b/arch/x86/kvm/kvm-asm-offsets.c @@ -24,6 +24,7 @@ static void __used common(void) if (IS_ENABLED(CONFIG_KVM_INTEL)) { BLANK(); + OFFSET(VMX_vcpu_arch_regs, vcpu_vmx, vcpu.arch.regs); OFFSET(VMX_spec_ctrl, vcpu_vmx, spec_ctrl); } } diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h deleted file mode 100644 index 8ddb01191d6f..000000000000 --- a/arch/x86/kvm/kvm_cache_regs.h +++ /dev/null @@ -1,249 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef ASM_KVM_CACHE_REGS_H -#define ASM_KVM_CACHE_REGS_H - -#include <linux/kvm_host.h> - -#define KVM_POSSIBLE_CR0_GUEST_BITS (X86_CR0_TS | X86_CR0_WP) -#define KVM_POSSIBLE_CR4_GUEST_BITS \ - (X86_CR4_PVI | X86_CR4_DE | X86_CR4_PCE | X86_CR4_OSFXSR \ - | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE \ - | X86_CR4_CET) - -#define X86_CR0_PDPTR_BITS (X86_CR0_CD | X86_CR0_NW | X86_CR0_PG) -#define X86_CR4_TLBFLUSH_BITS (X86_CR4_PGE | X86_CR4_PCIDE | X86_CR4_PAE | X86_CR4_SMEP) -#define X86_CR4_PDPTR_BITS (X86_CR4_PGE | X86_CR4_PSE | X86_CR4_PAE | X86_CR4_SMEP) - -static_assert(!(KVM_POSSIBLE_CR0_GUEST_BITS & X86_CR0_PDPTR_BITS)); - -#define BUILD_KVM_GPR_ACCESSORS(lname, uname) \ -static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *vcpu)\ -{ \ - return vcpu->arch.regs[VCPU_REGS_##uname]; \ -} \ -static __always_inline void kvm_##lname##_write(struct kvm_vcpu *vcpu, \ - unsigned long val) \ -{ \ - vcpu->arch.regs[VCPU_REGS_##uname] = val; \ -} -BUILD_KVM_GPR_ACCESSORS(rax, RAX) -BUILD_KVM_GPR_ACCESSORS(rbx, RBX) -BUILD_KVM_GPR_ACCESSORS(rcx, RCX) -BUILD_KVM_GPR_ACCESSORS(rdx, RDX) -BUILD_KVM_GPR_ACCESSORS(rbp, RBP) -BUILD_KVM_GPR_ACCESSORS(rsi, RSI) -BUILD_KVM_GPR_ACCESSORS(rdi, RDI) -#ifdef CONFIG_X86_64 -BUILD_KVM_GPR_ACCESSORS(r8, R8) -BUILD_KVM_GPR_ACCESSORS(r9, R9) -BUILD_KVM_GPR_ACCESSORS(r10, R10) -BUILD_KVM_GPR_ACCESSORS(r11, R11) -BUILD_KVM_GPR_ACCESSORS(r12, R12) -BUILD_KVM_GPR_ACCESSORS(r13, R13) -BUILD_KVM_GPR_ACCESSORS(r14, R14) -BUILD_KVM_GPR_ACCESSORS(r15, R15) -#endif - -/* - * Using the register cache from interrupt context is generally not allowed, as - * caching a register and marking it available/dirty can't be done atomically, - * i.e. accesses from interrupt context may clobber state or read stale data if - * the vCPU task is in the process of updating the cache. The exception is if - * KVM is handling a PMI IRQ/NMI VM-Exit, as that bound code sequence doesn't - * touch the cache, it runs after the cache is reset (post VM-Exit), and PMIs - * need to access several registers that are cacheable. - */ -#define kvm_assert_register_caching_allowed(vcpu) \ - lockdep_assert_once(in_task() || kvm_arch_pmi_in_guest(vcpu)) - -/* - * avail dirty - * 0 0 register in VMCS/VMCB - * 0 1 *INVALID* - * 1 0 register in vcpu->arch - * 1 1 register in vcpu->arch, needs to be stored back - */ -static inline bool kvm_register_is_available(struct kvm_vcpu *vcpu, - enum kvm_reg reg) -{ - kvm_assert_register_caching_allowed(vcpu); - return test_bit(reg, (unsigned long *)&vcpu->arch.regs_avail); -} - -static inline bool kvm_register_is_dirty(struct kvm_vcpu *vcpu, - enum kvm_reg reg) -{ - kvm_assert_register_caching_allowed(vcpu); - return test_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty); -} - -static inline void kvm_register_mark_available(struct kvm_vcpu *vcpu, - enum kvm_reg reg) -{ - kvm_assert_register_caching_allowed(vcpu); - __set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail); -} - -static inline void kvm_register_mark_dirty(struct kvm_vcpu *vcpu, - enum kvm_reg reg) -{ - kvm_assert_register_caching_allowed(vcpu); - __set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail); - __set_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty); -} - -/* - * kvm_register_test_and_mark_available() is a special snowflake that uses an - * arch bitop directly to avoid the explicit instrumentation that comes with - * the generic bitops. This allows code that cannot be instrumented (noinstr - * functions), e.g. the low level VM-Enter/VM-Exit paths, to cache registers. - */ -static __always_inline bool kvm_register_test_and_mark_available(struct kvm_vcpu *vcpu, - enum kvm_reg reg) -{ - kvm_assert_register_caching_allowed(vcpu); - return arch___test_and_set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail); -} - -/* - * The "raw" register helpers are only for cases where the full 64 bits of a - * register are read/written irrespective of current vCPU mode. In other words, - * odds are good you shouldn't be using the raw variants. - */ -static inline unsigned long kvm_register_read_raw(struct kvm_vcpu *vcpu, int reg) -{ - if (WARN_ON_ONCE((unsigned int)reg >= NR_VCPU_REGS)) - return 0; - - if (!kvm_register_is_available(vcpu, reg)) - kvm_x86_call(cache_reg)(vcpu, reg); - - return vcpu->arch.regs[reg]; -} - -static inline void kvm_register_write_raw(struct kvm_vcpu *vcpu, int reg, - unsigned long val) -{ - if (WARN_ON_ONCE((unsigned int)reg >= NR_VCPU_REGS)) - return; - - vcpu->arch.regs[reg] = val; - kvm_register_mark_dirty(vcpu, reg); -} - -static inline unsigned long kvm_rip_read(struct kvm_vcpu *vcpu) -{ - return kvm_register_read_raw(vcpu, VCPU_REGS_RIP); -} - -static inline void kvm_rip_write(struct kvm_vcpu *vcpu, unsigned long val) -{ - kvm_register_write_raw(vcpu, VCPU_REGS_RIP, val); -} - -static inline unsigned long kvm_rsp_read(struct kvm_vcpu *vcpu) -{ - return kvm_register_read_raw(vcpu, VCPU_REGS_RSP); -} - -static inline void kvm_rsp_write(struct kvm_vcpu *vcpu, unsigned long val) -{ - kvm_register_write_raw(vcpu, VCPU_REGS_RSP, val); -} - -static inline u64 kvm_pdptr_read(struct kvm_vcpu *vcpu, int index) -{ - might_sleep(); /* on svm */ - - if (!kvm_register_is_available(vcpu, VCPU_EXREG_PDPTR)) - kvm_x86_call(cache_reg)(vcpu, VCPU_EXREG_PDPTR); - - return vcpu->arch.walk_mmu->pdptrs[index]; -} - -static inline void kvm_pdptr_write(struct kvm_vcpu *vcpu, int index, u64 value) -{ - vcpu->arch.walk_mmu->pdptrs[index] = value; -} - -static inline ulong kvm_read_cr0_bits(struct kvm_vcpu *vcpu, ulong mask) -{ - ulong tmask = mask & KVM_POSSIBLE_CR0_GUEST_BITS; - if ((tmask & vcpu->arch.cr0_guest_owned_bits) && - !kvm_register_is_available(vcpu, VCPU_EXREG_CR0)) - kvm_x86_call(cache_reg)(vcpu, VCPU_EXREG_CR0); - return vcpu->arch.cr0 & mask; -} - -static __always_inline bool kvm_is_cr0_bit_set(struct kvm_vcpu *vcpu, - unsigned long cr0_bit) -{ - BUILD_BUG_ON(!is_power_of_2(cr0_bit)); - - return !!kvm_read_cr0_bits(vcpu, cr0_bit); -} - -static inline ulong kvm_read_cr0(struct kvm_vcpu *vcpu) -{ - return kvm_read_cr0_bits(vcpu, ~0UL); -} - -static inline ulong kvm_read_cr4_bits(struct kvm_vcpu *vcpu, ulong mask) -{ - ulong tmask = mask & KVM_POSSIBLE_CR4_GUEST_BITS; - if ((tmask & vcpu->arch.cr4_guest_owned_bits) && - !kvm_register_is_available(vcpu, VCPU_EXREG_CR4)) - kvm_x86_call(cache_reg)(vcpu, VCPU_EXREG_CR4); - return vcpu->arch.cr4 & mask; -} - -static __always_inline bool kvm_is_cr4_bit_set(struct kvm_vcpu *vcpu, - unsigned long cr4_bit) -{ - BUILD_BUG_ON(!is_power_of_2(cr4_bit)); - - return !!kvm_read_cr4_bits(vcpu, cr4_bit); -} - -static inline ulong kvm_read_cr3(struct kvm_vcpu *vcpu) -{ - if (!kvm_register_is_available(vcpu, VCPU_EXREG_CR3)) - kvm_x86_call(cache_reg)(vcpu, VCPU_EXREG_CR3); - return vcpu->arch.cr3; -} - -static inline ulong kvm_read_cr4(struct kvm_vcpu *vcpu) -{ - return kvm_read_cr4_bits(vcpu, ~0UL); -} - -static inline u64 kvm_read_edx_eax(struct kvm_vcpu *vcpu) -{ - return (kvm_rax_read(vcpu) & -1u) - | ((u64)(kvm_rdx_read(vcpu) & -1u) << 32); -} - -static inline void enter_guest_mode(struct kvm_vcpu *vcpu) -{ - vcpu->arch.hflags |= HF_GUEST_MASK; - vcpu->stat.guest_mode = 1; -} - -static inline void leave_guest_mode(struct kvm_vcpu *vcpu) -{ - vcpu->arch.hflags &= ~HF_GUEST_MASK; - - if (vcpu->arch.load_eoi_exitmap_pending) { - vcpu->arch.load_eoi_exitmap_pending = false; - kvm_make_request(KVM_REQ_LOAD_EOI_EXITMAP, vcpu); - } - - vcpu->stat.guest_mode = 0; -} - -static inline bool is_guest_mode(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.hflags & HF_GUEST_MASK; -} - -#endif diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h index 0abff36d0994..3e375af15c03 100644 --- a/arch/x86/kvm/kvm_emulate.h +++ b/arch/x86/kvm/kvm_emulate.h @@ -22,9 +22,13 @@ enum x86_intercept_stage; struct x86_exception { u8 vector; bool error_code_valid; - u16 error_code; + u64 error_code; bool nested_page_fault; - u64 address; /* cr2 or nested page fault gpa */ + union { + u64 address; /* cr2 or nested page fault gpa */ + unsigned long dr6; + u64 payload; + }; u8 async_page_fault; unsigned long exit_qualification; }; @@ -211,6 +215,7 @@ struct x86_emulate_ops { ulong (*get_cr)(struct x86_emulate_ctxt *ctxt, int cr); int (*set_cr)(struct x86_emulate_ctxt *ctxt, int cr, ulong val); int (*cpl)(struct x86_emulate_ctxt *ctxt); + ulong (*get_effective_dr7)(struct x86_emulate_ctxt *ctxt); ulong (*get_dr)(struct x86_emulate_ctxt *ctxt, int dr); int (*set_dr)(struct x86_emulate_ctxt *ctxt, int dr, ulong value); int (*set_msr_with_filter)(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 data); @@ -225,6 +230,7 @@ struct x86_emulate_ops { struct x86_instruction_info *info, enum x86_intercept_stage stage); + bool (*is_cpuid_allowed)(struct x86_emulate_ctxt *ctxt); bool (*get_cpuid)(struct x86_emulate_ctxt *ctxt, u32 *eax, u32 *ebx, u32 *ecx, u32 *edx, bool exact_only); bool (*guest_has_movbe)(struct x86_emulate_ctxt *ctxt); @@ -520,13 +526,6 @@ enum x86_intercept { nr_x86_intercepts }; -/* Host execution mode. */ -#if defined(CONFIG_X86_32) -#define X86EMUL_MODE_HOST X86EMUL_MODE_PROT32 -#elif defined(CONFIG_X86_64) -#define X86EMUL_MODE_HOST X86EMUL_MODE_PROT64 -#endif - int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int emulation_type); bool x86_page_table_writing_insn(struct x86_emulate_ctxt *ctxt); #define EMULATION_FAILED -1 diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 4078e624ca66..9d2df8623f6d 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -37,7 +37,7 @@ #include <asm/delay.h> #include <linux/atomic.h> #include <linux/jump_label.h> -#include "kvm_cache_regs.h" +#include "regs.h" #include "irq.h" #include "ioapic.h" #include "trace.h" @@ -1730,7 +1730,7 @@ static inline struct kvm_lapic *to_lapic(struct kvm_io_device *dev) #define APIC_REGS_MASK(first, count) \ (APIC_REG_MASK(first) * ((1ull << (count)) - 1)) -u64 kvm_lapic_readable_reg_mask(struct kvm_lapic *apic) +static u64 kvm_lapic_readable_reg_mask(struct kvm_lapic *apic) { /* Leave bits '0' for reserved and write-only registers. */ u64 valid_reg_mask = @@ -1766,7 +1766,24 @@ u64 kvm_lapic_readable_reg_mask(struct kvm_lapic *apic) return valid_reg_mask; } -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lapic_readable_reg_mask); + +u64 kvm_x2apic_disable_read_intercept_reg_mask(struct kvm_vcpu *vcpu) +{ + if (WARN_ON_ONCE(!lapic_in_kernel(vcpu))) + return 0; + + /* + * TMMCT, a.k.a. the current APIC timer count, reads aren't accelerated + * by hardware (Intel or AMD) as the timer is emulated in software (by + * KVM), i.e. reads from the virtual APIC page would return garbage. + * Intercept RDMSR, as handling the fault-like APIC-access VM-Exit is + * more expensive than handling a RDMSR VM-Exit (the APIC-access exit + * requires slow emulation of the code stream). + */ + return kvm_lapic_readable_reg_mask(vcpu->arch.apic) & + ~APIC_REG_MASK(APIC_TMCCT); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x2apic_disable_read_intercept_reg_mask); static int kvm_lapic_reg_read(struct kvm_lapic *apic, u32 offset, int len, void *data) @@ -2744,6 +2761,32 @@ u64 kvm_lapic_get_cr8(struct kvm_vcpu *vcpu) return (tpr & 0xf0) >> 4; } +void kvm_lapic_update_cr8_intercept(struct kvm_vcpu *vcpu) +{ + int max_irr, tpr; + + if (!kvm_x86_ops.update_cr8_intercept) + return; + + if (!lapic_in_kernel(vcpu)) + return; + + if (vcpu->arch.apic->apicv_active) + return; + + if (!vcpu->arch.apic->vapic_addr) + max_irr = kvm_lapic_find_highest_irr(vcpu); + else + max_irr = -1; + + if (max_irr != -1) + max_irr >>= 4; + + tpr = kvm_lapic_get_cr8(vcpu); + + kvm_x86_call(update_cr8_intercept)(vcpu, tpr, max_irr); +} + static void __kvm_apic_set_base(struct kvm_vcpu *vcpu, u64 value) { u64 old_value = vcpu->arch.apic_base; diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index 274885af4ebc..71970213dc1f 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -100,6 +100,7 @@ int kvm_apic_accept_events(struct kvm_vcpu *vcpu); void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event); u64 kvm_lapic_get_cr8(struct kvm_vcpu *vcpu); void kvm_lapic_set_tpr(struct kvm_vcpu *vcpu, unsigned long cr8); +void kvm_lapic_update_cr8_intercept(struct kvm_vcpu *vcpu); void kvm_lapic_set_eoi(struct kvm_vcpu *vcpu); void kvm_apic_set_version(struct kvm_vcpu *vcpu); void kvm_apic_after_set_mcg_cap(struct kvm_vcpu *vcpu); @@ -156,7 +157,7 @@ int kvm_hv_vapic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data); int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len); void kvm_lapic_exit(void); -u64 kvm_lapic_readable_reg_mask(struct kvm_lapic *apic); +u64 kvm_x2apic_disable_read_intercept_reg_mask(struct kvm_vcpu *vcpu); static inline void kvm_lapic_set_irr(int vec, struct kvm_lapic *apic) { diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 830f46145692..e1bb663ebbd5 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -3,7 +3,7 @@ #define __KVM_X86_MMU_H #include <linux/kvm_host.h> -#include "kvm_cache_regs.h" +#include "regs.h" #include "x86.h" #include "cpuid.h" @@ -37,6 +37,13 @@ extern bool __read_mostly enable_mmio_caching; #define PT32_ROOT_LEVEL 2 #define PT32E_ROOT_LEVEL 3 +#define ACC_READ_MASK PT_PRESENT_MASK +#define ACC_WRITE_MASK PT_WRITABLE_MASK +#define ACC_USER_MASK PT_USER_MASK /* non EPT */ +#define ACC_USER_EXEC_MASK ACC_USER_MASK /* EPT only */ +#define ACC_EXEC_MASK 8 +#define ACC_ALL (ACC_EXEC_MASK | ACC_WRITE_MASK | ACC_USER_MASK | ACC_READ_MASK) + #define KVM_MMU_CR4_ROLE_BITS (X86_CR4_PSE | X86_CR4_PAE | X86_CR4_LA57 | \ X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_PKE) @@ -76,19 +83,24 @@ static inline gfn_t kvm_mmu_max_gfn(void) return (1ULL << (max_gpa_bits - PAGE_SHIFT)) - 1; } +static inline bool mmu_has_mbec(struct kvm_mmu *mmu) +{ + return mmu->root_role.cr4_smep; +} + u8 kvm_mmu_get_max_tdp_level(void); void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask); void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value); void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask); -void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only); +void kvm_mmu_set_ept_masks(bool has_ad_bits); void kvm_init_mmu(struct kvm_vcpu *vcpu); -void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, unsigned long cr0, - unsigned long cr4, u64 efer, gpa_t nested_cr3); +void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, unsigned long cr4, + u64 efer, gpa_t nested_cr3, u64 misc_ctl); void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, int huge_page_level, bool accessed_dirty, - gpa_t new_eptp); + bool mbec, gpa_t new_eptp); bool kvm_can_do_async_pf(struct kvm_vcpu *vcpu); int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, u64 fault_address, char *insn, int insn_len); @@ -288,17 +300,17 @@ static inline void kvm_update_page_stats(struct kvm *kvm, int level, int count) atomic64_add(count, &kvm->stat.pages[level - 1]); } -gpa_t translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, u64 access, - struct x86_exception *exception); - static inline gpa_t kvm_translate_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, gpa_t gpa, u64 access, - struct x86_exception *exception) + struct x86_exception *exception, + u64 pte_access) { if (mmu != &vcpu->arch.nested_mmu) return gpa; - return translate_nested_gpa(vcpu, gpa, access, exception); + return kvm_x86_ops.nested_ops->translate_nested_gpa(vcpu, gpa, access, + exception, + pte_access); } static inline bool kvm_has_mirrored_tdp(const struct kvm *kvm) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 91843e9224d0..fbb491e35b54 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -22,7 +22,7 @@ #include "mmu_internal.h" #include "tdp_mmu.h" #include "x86.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "smm.h" #include "kvm_emulate.h" #include "page_track.h" @@ -56,6 +56,7 @@ #include <asm/io.h> #include <asm/set_memory.h> #include <asm/spec-ctrl.h> +#include <asm/svm.h> #include <asm/vmx.h> #include "trace.h" @@ -230,13 +231,18 @@ static inline bool __maybe_unused is_##reg##_##name(struct kvm_mmu *mmu) \ } BUILD_MMU_ROLE_ACCESSOR(base, cr0, wp); BUILD_MMU_ROLE_ACCESSOR(ext, cr4, pse); -BUILD_MMU_ROLE_ACCESSOR(ext, cr4, smep); +BUILD_MMU_ROLE_ACCESSOR(base, cr4, smep); BUILD_MMU_ROLE_ACCESSOR(ext, cr4, smap); BUILD_MMU_ROLE_ACCESSOR(ext, cr4, pke); BUILD_MMU_ROLE_ACCESSOR(ext, cr4, la57); BUILD_MMU_ROLE_ACCESSOR(base, efer, nx); BUILD_MMU_ROLE_ACCESSOR(ext, efer, lma); +static inline bool has_pferr_fetch(struct kvm_mmu *mmu) +{ + return mmu->cpu_role.ext.has_pferr_fetch; +} + static inline bool is_cr0_pg(struct kvm_mmu *mmu) { return mmu->cpu_role.base.level > 0; @@ -2023,7 +2029,7 @@ static bool kvm_sync_page_check(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) */ const union kvm_mmu_page_role sync_role_ign = { .level = 0xf, - .access = 0x7, + .access = ACC_ALL, .quadrant = 0x3, .passthrough = 0x1, }; @@ -2454,13 +2460,15 @@ static struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, bool direct, unsigned int access) { - union kvm_mmu_page_role role; + union kvm_mmu_page_role role = kvm_mmu_child_role(sptep, direct, access); - if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep) && - spte_to_child_sp(*sptep) && spte_to_child_sp(*sptep)->gfn == gfn) + if (is_shadow_present_pte(*sptep) && + !is_large_pte(*sptep) && + spte_to_child_sp(*sptep) && + spte_to_child_sp(*sptep)->gfn == gfn && + spte_to_child_sp(*sptep)->role.word == role.word) return ERR_PTR(-EEXIST); - role = kvm_mmu_child_role(sptep, direct, access); return kvm_mmu_get_shadow_page(vcpu, gfn, role); } @@ -3459,12 +3467,13 @@ static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { struct kvm_shadow_walk_iterator it; struct kvm_mmu_page *sp; - int ret; + int ret, access; gfn_t base_gfn = fault->gfn; kvm_mmu_hugepage_adjust(vcpu, fault); - trace_kvm_mmu_spte_requested(fault); + access = vcpu->arch.mmu->root_role.access; + trace_kvm_mmu_spte_requested(fault, access); for_each_shadow_entry(vcpu, fault->addr, it) { /* * We cannot overwrite existing page tables with an NX @@ -3477,7 +3486,7 @@ static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (it.level == fault->goal_level) break; - sp = kvm_mmu_get_child_sp(vcpu, it.sptep, base_gfn, true, ACC_ALL); + sp = kvm_mmu_get_child_sp(vcpu, it.sptep, base_gfn, true, access); if (sp == ERR_PTR(-EEXIST)) continue; @@ -3490,7 +3499,7 @@ static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (WARN_ON_ONCE(it.level != fault->goal_level)) return -EFAULT; - ret = mmu_set_spte(vcpu, fault->slot, it.sptep, ACC_ALL, + ret = mmu_set_spte(vcpu, fault->slot, it.sptep, access, base_gfn, fault->pfn, fault); if (ret == RET_PF_SPURIOUS) return ret; @@ -4361,7 +4370,14 @@ static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, { if (exception) exception->error_code = 0; - return kvm_translate_gpa(vcpu, mmu, vaddr, access, exception); + /* + * EPT MBEC uses the effective access bits from the PTE to distinguish + * user and supervisor accesses, and treats every linear address as a + * user-mode address if CR0.PG=0. Therefore *include* ACC_USER_MASK in + * the last argument to kvm_translate_gpa (which NPT does not use). + */ + return kvm_translate_gpa(vcpu, mmu, vaddr, access | PFERR_GUEST_FINAL_MASK, + exception, ACC_ALL); } static bool mmio_info_in_cache(struct kvm_vcpu *vcpu, u64 addr, bool direct) @@ -5497,7 +5513,7 @@ static void reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, static inline bool boot_cpu_is_amd(void) { WARN_ON_ONCE(!tdp_enabled); - return shadow_x_mask == 0; + return shadow_xs_mask == 0; } /* @@ -5542,55 +5558,106 @@ reset_ept_shadow_zero_bits_mask(struct kvm_mmu *context, bool execonly) max_huge_page_level); } -#define BYTE_MASK(access) \ - ((1 & (access) ? 2 : 0) | \ - (2 & (access) ? 4 : 0) | \ - (3 & (access) ? 8 : 0) | \ - (4 & (access) ? 16 : 0) | \ - (5 & (access) ? 32 : 0) | \ - (6 & (access) ? 64 : 0) | \ - (7 & (access) ? 128 : 0)) - - -static void update_permission_bitmask(struct kvm_mmu *mmu, bool ept) +/* + * Build a mask with all combinations of PTE access rights that + * include the given access bit. The mask can be queried with + * "mask & (1 << access)", where access is a combination of + * ACC_* bits. + * + * By mixing and matching multiple masks returned by ACC_BITS_MASK, + * update_permission_bitmask() builds what is effectively a + * two-dimensional array of bools. The second dimension is + * provided by individual bits of permissions[pfec >> 1], and + * logical &, | and ~ operations operate on all the 16 possible + * combinations of ACC_* bits. + */ +#define ACC_BITS_MASK(access) \ + ((1 & (access) ? 1 << 1 : 0) | \ + (2 & (access) ? 1 << 2 : 0) | \ + (3 & (access) ? 1 << 3 : 0) | \ + (4 & (access) ? 1 << 4 : 0) | \ + (5 & (access) ? 1 << 5 : 0) | \ + (6 & (access) ? 1 << 6 : 0) | \ + (7 & (access) ? 1 << 7 : 0) | \ + (8 & (access) ? 1 << 8 : 0) | \ + (9 & (access) ? 1 << 9 : 0) | \ + (10 & (access) ? 1 << 10 : 0) | \ + (11 & (access) ? 1 << 11 : 0) | \ + (12 & (access) ? 1 << 12 : 0) | \ + (13 & (access) ? 1 << 13 : 0) | \ + (14 & (access) ? 1 << 14 : 0) | \ + (15 & (access) ? 1 << 15 : 0)) + +static void update_permission_bitmask(struct kvm_mmu *mmu, bool tdp, bool ept) { - unsigned byte; + unsigned index; - const u8 x = BYTE_MASK(ACC_EXEC_MASK); - const u8 w = BYTE_MASK(ACC_WRITE_MASK); - const u8 u = BYTE_MASK(ACC_USER_MASK); + const u16 w = ACC_BITS_MASK(ACC_WRITE_MASK); + const u16 r = ACC_BITS_MASK(ACC_READ_MASK); bool cr4_smep = is_cr4_smep(mmu); bool cr4_smap = is_cr4_smap(mmu); bool cr0_wp = is_cr0_wp(mmu); bool efer_nx = is_efer_nx(mmu); - for (byte = 0; byte < ARRAY_SIZE(mmu->permissions); ++byte) { - unsigned pfec = byte << 1; + /* + * In hardware, page fault error codes are generated (as the name + * suggests) on any kind of page fault. permission_fault() and + * paging_tmpl.h already use the same bits after a successful page + * table walk, to indicate the kind of access being performed. + * + * However, PFERR_PRESENT_MASK and PFERR_RSVD_MASK are never set here, + * exactly because the page walk is successful. PFERR_PRESENT_MASK is + * removed by the shift, while PFERR_RSVD_MASK is repurposed in + * permission_fault() to indicate accesses that are *not* subject to + * SMAP restrictions. + */ + for (index = 0; index < ARRAY_SIZE(mmu->permissions); ++index) { + unsigned pfec = index << 1; /* - * Each "*f" variable has a 1 bit for each UWX value + * Each "*f" variable has a 1 bit for each ACC_* combo * that causes a fault with the given PFEC. */ + /* Faults from reads to non-readable pages */ + u16 rf = (pfec & (PFERR_WRITE_MASK|PFERR_FETCH_MASK)) ? 0 : (u16)~r; /* Faults from writes to non-writable pages */ - u8 wf = (pfec & PFERR_WRITE_MASK) ? (u8)~w : 0; + u16 wf = (pfec & PFERR_WRITE_MASK) ? (u16)~w : 0; /* Faults from user mode accesses to supervisor pages */ - u8 uf = (pfec & PFERR_USER_MASK) ? (u8)~u : 0; - /* Faults from fetches of non-executable pages*/ - u8 ff = (pfec & PFERR_FETCH_MASK) ? (u8)~x : 0; - /* Faults from kernel mode fetches of user pages */ - u8 smepf = 0; + u16 uf = 0; + /* Faults from fetches of non-executable pages */ + u16 ff = 0; /* Faults from kernel mode accesses of user pages */ - u8 smapf = 0; + u16 smapf = 0; + + if (ept) { + const u16 xs = ACC_BITS_MASK(ACC_EXEC_MASK); + const u16 xu = ACC_BITS_MASK(ACC_USER_EXEC_MASK); + + if (pfec & PFERR_FETCH_MASK) { + /* Ignore XU unless MBEC is enabled. */ + if (cr4_smep) + ff = pfec & PFERR_USER_MASK ? (u16)~xu : (u16)~xs; + else + ff = (u16)~xs; + } + } else { + const u16 x = ACC_BITS_MASK(ACC_EXEC_MASK); + const u16 u = ACC_BITS_MASK(ACC_USER_MASK); - if (!ept) { /* Faults from kernel mode accesses to user pages */ - u8 kf = (pfec & PFERR_USER_MASK) ? 0 : u; + u16 kf = (pfec & PFERR_USER_MASK) ? 0 : u; + + /* + * For NPT GMET, U=0 does not affect reads and writes. Fetches + * are handled below via cr4_smep. + */ + if (!(tdp && cr4_smep)) + uf = (pfec & PFERR_USER_MASK) ? (u16)~u : 0; - /* Not really needed: !nx will cause pte.nx to fault */ - if (!efer_nx) - ff = 0; + if (efer_nx) + ff |= (pfec & PFERR_FETCH_MASK) ? (u16)~x : 0; /* Allow supervisor writes if !cr0.wp */ if (!cr0_wp) @@ -5598,7 +5665,7 @@ static void update_permission_bitmask(struct kvm_mmu *mmu, bool ept) /* Disallow supervisor fetches of user code if cr4.smep */ if (cr4_smep) - smepf = (pfec & PFERR_FETCH_MASK) ? kf : 0; + ff |= (pfec & PFERR_FETCH_MASK) ? kf : 0; /* * SMAP:kernel-mode data accesses from user-mode @@ -5611,16 +5678,15 @@ static void update_permission_bitmask(struct kvm_mmu *mmu, bool ept) * - The access is supervisor mode * - If implicit supervisor access or X86_EFLAGS_AC is clear * - * Here, we cover the first four conditions. - * The fifth is computed dynamically in permission_fault(); - * PFERR_RSVD_MASK bit will be set in PFEC if the access is - * *not* subject to SMAP restrictions. + * Here, we cover the first four conditions. The fifth + * is computed dynamically in permission_fault() and + * communicated by setting PFERR_RSVD_MASK. */ if (cr4_smap) smapf = (pfec & (PFERR_RSVD_MASK|PFERR_FETCH_MASK)) ? 0 : kf; } - mmu->permissions[byte] = ff | uf | wf | smepf | smapf; + mmu->permissions[index] = ff | uf | wf | rf | smapf; } } @@ -5699,7 +5765,7 @@ static void reset_guest_paging_metadata(struct kvm_vcpu *vcpu, return; reset_guest_rsvds_bits_mask(vcpu, mmu); - update_permission_bitmask(mmu, false); + update_permission_bitmask(mmu, mmu == &vcpu->arch.guest_mmu, false); update_pkru_bitmask(mmu); } @@ -5734,7 +5800,7 @@ static union kvm_cpu_role kvm_calc_cpu_role(struct kvm_vcpu *vcpu, role.base.efer_nx = ____is_efer_nx(regs); role.base.cr0_wp = ____is_cr0_wp(regs); - role.base.smep_andnot_wp = ____is_cr4_smep(regs) && !____is_cr0_wp(regs); + role.base.cr4_smep = ____is_cr4_smep(regs); role.base.smap_andnot_wp = ____is_cr4_smap(regs) && !____is_cr0_wp(regs); role.base.has_4_byte_gpte = !____is_cr4_pae(regs); @@ -5746,7 +5812,6 @@ static union kvm_cpu_role kvm_calc_cpu_role(struct kvm_vcpu *vcpu, else role.base.level = PT32_ROOT_LEVEL; - role.ext.cr4_smep = ____is_cr4_smep(regs); role.ext.cr4_smap = ____is_cr4_smap(regs); role.ext.cr4_pse = ____is_cr4_pse(regs); @@ -5754,6 +5819,8 @@ static union kvm_cpu_role kvm_calc_cpu_role(struct kvm_vcpu *vcpu, role.ext.cr4_pke = ____is_efer_lma(regs) && ____is_cr4_pke(regs); role.ext.cr4_la57 = ____is_efer_lma(regs) && ____is_cr4_la57(regs); role.ext.efer_lma = ____is_efer_lma(regs); + + role.ext.has_pferr_fetch = role.base.efer_nx | role.base.cr4_smep; return role; } @@ -5803,8 +5870,8 @@ kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu, { union kvm_mmu_page_role role = {0}; - role.access = ACC_ALL; role.cr0_wp = true; + role.cr4_smep = kvm_x86_call(tdp_has_smep)(vcpu->kvm); role.efer_nx = true; role.smm = cpu_role.base.smm; role.guest_mode = cpu_role.base.guest_mode; @@ -5813,6 +5880,11 @@ kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu, role.direct = true; role.has_4_byte_gpte = false; + /* All TDP pages are supervisor-executable */ + role.access = ACC_ALL; + if (role.cr4_smep && shadow_user_mask) + role.access &= ~ACC_USER_MASK; + return role; } @@ -5892,13 +5964,13 @@ static void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu, shadow_mmu_init_context(vcpu, context, cpu_role, root_role); } -void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, unsigned long cr0, - unsigned long cr4, u64 efer, gpa_t nested_cr3) +void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, unsigned long cr4, + u64 efer, gpa_t nested_cr3, u64 misc_ctl) { struct kvm_mmu *context = &vcpu->arch.guest_mmu; struct kvm_mmu_role_regs regs = { - .cr0 = cr0, - .cr4 = cr4 & ~X86_CR4_PKE, + .cr0 = X86_CR0_PG | X86_CR0_WP, + .cr4 = cr4 & ~(X86_CR4_PKE | X86_CR4_SMAP), .efer = efer, }; union kvm_cpu_role cpu_role = kvm_calc_cpu_role(vcpu, ®s); @@ -5906,6 +5978,7 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, unsigned long cr0, /* NPT requires CR0.PG=1. */ WARN_ON_ONCE(cpu_role.base.direct || !cpu_role.base.guest_mode); + cpu_role.base.cr4_smep = (misc_ctl & SVM_MISC_ENABLE_GMET) != 0; root_role = cpu_role.base; root_role.level = kvm_mmu_get_tdp_level(vcpu); @@ -5920,7 +5993,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_shadow_npt_mmu); static union kvm_cpu_role kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty, - bool execonly, u8 level) + bool execonly, u8 level, bool mbec) { union kvm_cpu_role role = {0}; @@ -5930,6 +6003,7 @@ kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty, */ WARN_ON_ONCE(is_smm(vcpu)); role.base.level = level; + role.base.cr4_smep = mbec; role.base.has_4_byte_gpte = false; role.base.direct = false; role.base.ad_disabled = !accessed_dirty; @@ -5945,13 +6019,13 @@ kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty, void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, int huge_page_level, bool accessed_dirty, - gpa_t new_eptp) + bool mbec, gpa_t new_eptp) { struct kvm_mmu *context = &vcpu->arch.guest_mmu; u8 level = vmx_eptp_page_walk_level(new_eptp); union kvm_cpu_role new_mode = kvm_calc_shadow_ept_root_page_role(vcpu, accessed_dirty, - execonly, level); + execonly, level, mbec); if (new_mode.as_u64 != context->cpu_role.as_u64) { /* EPT, and thus nested EPT, does not consume CR0, CR4, nor EFER. */ @@ -5962,7 +6036,7 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, context->gva_to_gpa = ept_gva_to_gpa; context->sync_spte = ept_sync_spte; - update_permission_bitmask(context, true); + update_permission_bitmask(context, true, true); context->pkru_mask = 0; reset_rsvds_bits_mask_ept(vcpu, context, execonly, huge_page_level); reset_ept_shadow_zero_bits_mask(context, execonly); @@ -6305,7 +6379,7 @@ void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, while (npte--) { entry = *spte; - mmu_page_zap_pte(vcpu->kvm, sp, spte, NULL); + mmu_page_zap_pte(vcpu->kvm, sp, spte, &invalid_list); if (gentry && sp->role.level != PG_LEVEL_4K) ++vcpu->kvm->stat.mmu_pde_zapped; if (is_shadow_present_pte(entry)) @@ -6950,7 +7024,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) write_lock(&kvm->mmu_lock); - kvm_mmu_invalidate_begin(kvm); + kvm_mmu_invalidate_start(kvm); kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end); @@ -7287,13 +7361,19 @@ restart: sp = sptep_to_sp(sptep); /* - * We cannot do huge page mapping for indirect shadow pages, - * which are found on the last rmap (level = 1) when not using - * tdp; such shadow pages are synced with the page table in - * the guest, and the guest page table is using 4K page size - * mapping if the indirect sp has level = 1. + * Direct shadow page can be replaced by a hugepage if the host + * mapping level allows it and the memslot maps all of the host + * hugepage. Note! If the memslot maps only part of the + * hugepage, sp->gfn may be below slot->base_gfn, and querying + * the max mapping level would cause an out-of-bounds lpage_info + * access. So the gfn bounds check *must* be done first. + * + * Indirect shadow pages are created when the guest page tables + * are using 4K pages. Since the host mapping is always + * constrained by the page size in the guest, indirect shadow + * pages are never collapsible. */ - if (sp->role.direct && + if (sp->role.direct && is_gfn_in_memslot(slot, sp->gfn) && sp->role.level < kvm_mmu_max_mapping_level(kvm, NULL, slot, sp->gfn)) { kvm_zap_one_rmap_spte(kvm, rmap_head, sptep); diff --git a/arch/x86/kvm/mmu/mmutrace.h b/arch/x86/kvm/mmu/mmutrace.h index 764e3015d021..fa01719baf8d 100644 --- a/arch/x86/kvm/mmu/mmutrace.h +++ b/arch/x86/kvm/mmu/mmutrace.h @@ -25,7 +25,8 @@ #define KVM_MMU_PAGE_PRINTK() ({ \ const char *saved_ptr = trace_seq_buffer_ptr(p); \ static const char *access_str[] = { \ - "---", "--x", "w--", "w-x", "-u-", "-ux", "wu-", "wux" \ + "----", "r---", "-w--", "rw--", "--u-", "r-u-", "-wu-", "rwu-", \ + "---x", "r--x", "-w-x", "rw-x", "--ux", "r-ux", "-wux", "rwux" \ }; \ union kvm_mmu_page_role role; \ \ @@ -356,8 +357,8 @@ TRACE_EVENT( __entry->sptep = virt_to_phys(sptep); __entry->level = level; __entry->r = shadow_present_mask || (__entry->spte & PT_PRESENT_MASK); - __entry->x = is_executable_pte(__entry->spte); - __entry->u = shadow_user_mask ? !!(__entry->spte & shadow_user_mask) : -1; + __entry->x = (__entry->spte & (shadow_xs_mask | shadow_nx_mask)) == shadow_xs_mask; + __entry->u = !!(__entry->spte & (shadow_xu_mask | shadow_user_mask)); ), TP_printk("gfn %llx spte %llx (%s%s%s%s) level %d at %llx", @@ -365,30 +366,32 @@ TRACE_EVENT( __entry->r ? "r" : "-", __entry->spte & PT_WRITABLE_MASK ? "w" : "-", __entry->x ? "x" : "-", - __entry->u == -1 ? "" : (__entry->u ? "u" : "-"), + __entry->u ? "u" : "-", __entry->level, __entry->sptep ) ); TRACE_EVENT( kvm_mmu_spte_requested, - TP_PROTO(struct kvm_page_fault *fault), - TP_ARGS(fault), + TP_PROTO(struct kvm_page_fault *fault, u8 access), + TP_ARGS(fault, access), TP_STRUCT__entry( __field(u64, gfn) __field(u64, pfn) __field(u8, level) + __field(u8, access) ), TP_fast_assign( __entry->gfn = fault->gfn; __entry->pfn = fault->pfn | (fault->gfn & (KVM_PAGES_PER_HPAGE(fault->goal_level) - 1)); __entry->level = fault->goal_level; + __entry->access = access; ), - TP_printk("gfn %llx pfn %llx level %d", - __entry->gfn, __entry->pfn, __entry->level + TP_printk("gfn %llx pfn %llx level %d access %x", + __entry->gfn, __entry->pfn, __entry->level, __entry->access ) ); diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 901cd2bd40b8..df3ae0c7ec2c 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -124,12 +124,17 @@ static inline void FNAME(protect_clean_gpte)(struct kvm_mmu *mmu, unsigned *acce *access &= mask; } -static inline int FNAME(is_present_gpte)(unsigned long pte) +static inline int FNAME(is_present_gpte)(struct kvm_mmu *mmu, + unsigned long pte) { #if PTTYPE != PTTYPE_EPT return pte & PT_PRESENT_MASK; #else - return pte & 7; + /* + * For EPT, an entry is present if any of bits 2:0 are set. + * With mode-based execute control, bit 10 also indicates presence. + */ + return pte & (7 | (mmu_has_mbec(mmu) ? VMX_EPT_USER_EXECUTABLE_MASK : 0)); #endif } @@ -152,7 +157,7 @@ static bool FNAME(prefetch_invalid_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, u64 *spte, u64 gpte) { - if (!FNAME(is_present_gpte)(gpte)) + if (!FNAME(is_present_gpte)(vcpu->arch.mmu, gpte)) goto no_present; /* Prefetch only accessed entries (unless A/D bits are disabled). */ @@ -170,25 +175,31 @@ no_present: return true; } -/* - * For PTTYPE_EPT, a page table can be executable but not readable - * on supported processors. Therefore, set_spte does not automatically - * set bit 0 if execute only is supported. Here, we repurpose ACC_USER_MASK - * to signify readability since it isn't used in the EPT case - */ static inline unsigned FNAME(gpte_access)(u64 gpte) { unsigned access; + /* + * Set bits in ACC_*_MASK even if they might not be used in the + * actual checks. For example, if EFER.NX is clear permission_fault() + * will ignore ACC_EXEC_MASK, and if MBEC is disabled it will + * ignore ACC_USER_EXEC_MASK. + */ #if PTTYPE == PTTYPE_EPT access = ((gpte & VMX_EPT_WRITABLE_MASK) ? ACC_WRITE_MASK : 0) | ((gpte & VMX_EPT_EXECUTABLE_MASK) ? ACC_EXEC_MASK : 0) | - ((gpte & VMX_EPT_READABLE_MASK) ? ACC_USER_MASK : 0); + ((gpte & VMX_EPT_READABLE_MASK) ? ACC_READ_MASK : 0) | + ((gpte & VMX_EPT_USER_EXECUTABLE_MASK) ? ACC_USER_EXEC_MASK : 0); #else - BUILD_BUG_ON(ACC_EXEC_MASK != PT_PRESENT_MASK); - BUILD_BUG_ON(ACC_EXEC_MASK != 1); + /* + * P is set here, so the page is always readable and W/U/!NX represent + * allowed accesses. + */ + BUILD_BUG_ON(ACC_READ_MASK != PT_PRESENT_MASK); + BUILD_BUG_ON(ACC_WRITE_MASK != PT_WRITABLE_MASK); + BUILD_BUG_ON(ACC_USER_MASK != PT_USER_MASK); + BUILD_BUG_ON(ACC_EXEC_MASK & (PT_WRITABLE_MASK | PT_USER_MASK | PT_PRESENT_MASK)); access = gpte & (PT_WRITABLE_MASK | PT_USER_MASK | PT_PRESENT_MASK); - /* Combine NX with P (which is set here) to get ACC_EXEC_MASK. */ - access ^= (gpte >> PT64_NX_SHIFT); + access |= gpte & PT64_NX_MASK ? 0 : ACC_EXEC_MASK; #endif return access; @@ -317,6 +328,12 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, const int write_fault = access & PFERR_WRITE_MASK; const int user_fault = access & PFERR_USER_MASK; const int fetch_fault = access & PFERR_FETCH_MASK; + /* + * Note! Track the error_code that's common to legacy shadow paging + * and NPT shadow paging as a u16 to guard against unintentionally + * setting any of bits 63:16. Architecturally, the #PF error code is + * 32 bits, and Intel CPUs don't support settings bits 31:16. + */ u16 errcode = 0; gpa_t real_gpa; gfn_t gfn; @@ -332,7 +349,7 @@ retry_walk: if (walker->level == PT32E_ROOT_LEVEL) { pte = mmu->get_pdptr(vcpu, (addr >> 30) & 3); trace_kvm_mmu_paging_element(pte, walker->level); - if (!FNAME(is_present_gpte)(pte)) + if (!FNAME(is_present_gpte)(mmu, pte)) goto error; --walker->level; } @@ -377,18 +394,9 @@ retry_walk: walker->pte_gpa[walker->level - 1] = pte_gpa; real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(table_gfn), - nested_access, &walker->fault); + nested_access | PFERR_GUEST_PAGE_MASK, + &walker->fault, 0); - /* - * FIXME: This can happen if emulation (for of an INS/OUTS - * instruction) triggers a nested page fault. The exit - * qualification / exit info field will incorrectly have - * "guest page access" as the nested page fault's cause, - * instead of "guest page structure access". To fix this, - * the x86_exception struct should be augmented with enough - * information to fix the exit_qualification or exit_info_1 - * fields. - */ if (unlikely(real_gpa == INVALID_GPA)) return 0; @@ -414,7 +422,7 @@ retry_walk: */ pte_access = pt_access & (pte ^ walk_nx_mask); - if (unlikely(!FNAME(is_present_gpte)(pte))) + if (unlikely(!FNAME(is_present_gpte)(mmu, pte))) goto error; if (unlikely(FNAME(is_rsvd_bits_set)(mmu, pte, walker->level))) { @@ -445,7 +453,9 @@ retry_walk: gfn += pse36_gfn_delta(pte); #endif - real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(gfn), access, &walker->fault); + real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(gfn), + access | PFERR_GUEST_FINAL_MASK, + &walker->fault, walker->pte_access); if (real_gpa == INVALID_GPA) return 0; @@ -475,7 +485,7 @@ retry_walk: error: errcode |= write_fault | user_fault; - if (fetch_fault && (is_efer_nx(mmu) || is_cr4_smep(mmu))) + if (fetch_fault && has_pferr_fetch(mmu)) errcode |= PFERR_FETCH_MASK; walker->fault.vector = PF_VECTOR; @@ -492,7 +502,8 @@ error: * [2:0] - Derive from the access bits. The exit_qualification might be * out of date if it is serving an EPT misconfiguration. * [5:3] - Calculated by the page walk of the guest EPT page tables - * [7:8] - Derived from [7:8] of real exit_qualification + * [7:8] - Derived from "fault stage" access bits + * [9:11] - Derived from [9:11] of real exit_qualification * * The other bits are set to 0. */ @@ -501,22 +512,48 @@ error: if (write_fault) walker->fault.exit_qualification |= EPT_VIOLATION_ACC_WRITE; - if (user_fault) - walker->fault.exit_qualification |= EPT_VIOLATION_ACC_READ; - if (fetch_fault) + else if (fetch_fault) walker->fault.exit_qualification |= EPT_VIOLATION_ACC_INSTR; + else + walker->fault.exit_qualification |= EPT_VIOLATION_ACC_READ; + + /* + * KVM doesn't emulate features that access GPAs directly, e.g. + * Intel Processor Trace. Assume the GVA is always valid; when + * propagating faults from hardware, KVM will discard this info + * and use the EXIT_QUALIFICATION bits from the VMCS. + */ + walker->fault.exit_qualification |= EPT_VIOLATION_GVA_IS_VALID; + + /* + * Accesses to guest paging structures are either "reads" or + * "read+write" accesses, so consider them the latter if write_fault + * is true. + */ + if (access & PFERR_GUEST_PAGE_MASK) + walker->fault.exit_qualification |= EPT_VIOLATION_ACC_READ; + else + walker->fault.exit_qualification |= EPT_VIOLATION_GVA_TRANSLATED; /* * Note, pte_access holds the raw RWX bits from the EPTE, not * ACC_*_MASK flags! */ walker->fault.exit_qualification |= EPT_VIOLATION_RWX_TO_PROT(pte_access); + if (mmu_has_mbec(mmu)) + walker->fault.exit_qualification |= + EPT_VIOLATION_USER_EXEC_TO_PROT(pte_access); } #endif walker->fault.address = addr; walker->fault.nested_page_fault = mmu != vcpu->arch.walk_mmu; walker->fault.async_page_fault = false; +#if PTTYPE != PTTYPE_EPT + if (walker->fault.nested_page_fault) + walker->fault.error_code |= access & PFERR_GUEST_FAULT_STAGE_MASK; +#endif + trace_kvm_mmu_walker_error(walker->fault.error_code); return 0; } @@ -709,7 +746,7 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, */ kvm_mmu_hugepage_adjust(vcpu, fault); - trace_kvm_mmu_spte_requested(fault); + trace_kvm_mmu_spte_requested(fault, gw->pte_access); for (; shadow_walk_okay(&it); shadow_walk_next(&it)) { /* @@ -782,7 +819,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault */ if (!r) { if (!fault->prefetch) - kvm_inject_emulated_page_fault(vcpu, &walker.fault); + __kvm_inject_emulated_page_fault(vcpu, &walker.fault, true); return RET_PF_RETRY; } diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 4e753386c8d4..72d2394e089c 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -30,8 +30,9 @@ bool __read_mostly kvm_ad_enabled; u64 __read_mostly shadow_host_writable_mask; u64 __read_mostly shadow_mmu_writable_mask; u64 __read_mostly shadow_nx_mask; -u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */ u64 __read_mostly shadow_user_mask; +u64 __read_mostly shadow_xs_mask; /* mutual exclusive with nx_mask and user_mask */ +u64 __read_mostly shadow_xu_mask; /* mutual exclusive with nx_mask and user_mask */ u64 __read_mostly shadow_accessed_mask; u64 __read_mostly shadow_dirty_mask; u64 __read_mostly shadow_mmio_value; @@ -195,12 +196,6 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int is_host_mmio = -1; bool wrprot = false; - /* - * For the EPT case, shadow_present_mask has no RWX bits set if - * exec-only page table entries are supported. In that case, - * ACC_USER_MASK and shadow_user_mask are used to represent - * read access. See FNAME(gpte_access) in paging_tmpl.h. - */ WARN_ON_ONCE((pte_access | shadow_present_mask) == SHADOW_NONPRESENT_VALUE); if (sp->role.ad_disabled) @@ -224,18 +219,26 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, * would tie make_spte() further to vCPU/MMU state, and add complexity * just to optimize a mode that is anything but performance critical. */ - if (level > PG_LEVEL_4K && (pte_access & ACC_EXEC_MASK) && - is_nx_huge_page_enabled(vcpu->kvm)) { + if (level > PG_LEVEL_4K && is_nx_huge_page_enabled(vcpu->kvm)) { pte_access &= ~ACC_EXEC_MASK; + if (shadow_xu_mask) + pte_access &= ~ACC_USER_EXEC_MASK; } - if (pte_access & ACC_EXEC_MASK) - spte |= shadow_x_mask; - else - spte |= shadow_nx_mask; - - if (pte_access & ACC_USER_MASK) - spte |= shadow_user_mask; + if (pte_access & ACC_READ_MASK) + spte |= PT_PRESENT_MASK; /* or VMX_EPT_READABLE_MASK */ + + if (shadow_nx_mask) { + if (!(pte_access & ACC_EXEC_MASK)) + spte |= shadow_nx_mask; + if (pte_access & ACC_USER_MASK) + spte |= shadow_user_mask; + } else { + if (pte_access & ACC_EXEC_MASK) + spte |= shadow_xs_mask; + if (pte_access & ACC_USER_EXEC_MASK) + spte |= shadow_xu_mask; + } if (level > PG_LEVEL_4K) spte |= PT_PAGE_SIZE_MASK; @@ -318,14 +321,18 @@ static u64 modify_spte_protections(u64 spte, u64 set, u64 clear) return spte; } -static u64 make_spte_executable(u64 spte) +static u64 change_spte_executable(u64 spte, u8 access) { - return modify_spte_protections(spte, shadow_x_mask, shadow_nx_mask); -} + u64 set, clear; -static u64 make_spte_nonexecutable(u64 spte) -{ - return modify_spte_protections(spte, shadow_nx_mask, shadow_x_mask); + if (shadow_nx_mask) + set = (access & ACC_EXEC_MASK) ? 0 : shadow_nx_mask; + else + set = + (access & ACC_EXEC_MASK ? shadow_xs_mask : 0) | + (access & ACC_USER_EXEC_MASK ? shadow_xu_mask : 0); + clear = set ^ (shadow_nx_mask | shadow_xs_mask | shadow_xu_mask); + return modify_spte_protections(spte, set, clear); } /* @@ -357,8 +364,8 @@ u64 make_small_spte(struct kvm *kvm, u64 huge_spte, * the page executable as the NX hugepage mitigation no longer * applies. */ - if ((role.access & ACC_EXEC_MASK) && is_nx_huge_page_enabled(kvm)) - child_spte = make_spte_executable(child_spte); + if (is_nx_huge_page_enabled(kvm)) + child_spte = change_spte_executable(child_spte, role.access); } return child_spte; @@ -380,7 +387,7 @@ u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level) huge_spte &= KVM_HPAGE_MASK(level) | ~PAGE_MASK; if (is_nx_huge_page_enabled(kvm)) - huge_spte = make_spte_nonexecutable(huge_spte); + huge_spte = change_spte_executable(huge_spte, 0); return huge_spte; } @@ -390,7 +397,8 @@ u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled) u64 spte = SPTE_MMU_PRESENT_MASK; spte |= __pa(child_pt) | shadow_present_mask | PT_WRITABLE_MASK | - shadow_user_mask | shadow_x_mask | shadow_me_value; + PT_PRESENT_MASK /* or VMX_EPT_READABLE_MASK */ | + shadow_user_mask | shadow_xs_mask | shadow_xu_mask | shadow_me_value; if (ad_disabled) spte |= SPTE_TDP_AD_DISABLED; @@ -490,20 +498,37 @@ void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_set_me_spte_mask); -void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) +void kvm_mmu_set_ept_masks(bool has_ad_bits) { kvm_ad_enabled = has_ad_bits; - shadow_user_mask = VMX_EPT_READABLE_MASK; + shadow_user_mask = 0; shadow_accessed_mask = VMX_EPT_ACCESS_BIT; shadow_dirty_mask = VMX_EPT_DIRTY_BIT; shadow_nx_mask = 0ull; - shadow_x_mask = VMX_EPT_EXECUTABLE_MASK; - /* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */ - shadow_present_mask = - (has_exec_only ? 0ull : VMX_EPT_READABLE_MASK) | VMX_EPT_SUPPRESS_VE_BIT; + shadow_xs_mask = VMX_EPT_EXECUTABLE_MASK; + + /* + * The MMU always maps ACC_EXEC_MASK and ACC_USER_EXEC_MASK to the + * XS and XU bits of shadow EPT entries, regardless of whether MBEC + * is available on the host or enabled in the VMCS. + * + * For the non-nested case, pages are mapped with ACC_EXEC_MASK + * and ACC_USER_EXEC_MASK set in tandem, so XS == XU and the + * host's MBEC setting does not matter. On hardware without MBEC + * the XU bit is reserved-as-ignored, and setting it does no harm. + * + * For nested EPT, when MBEC is disabled by L1, correctness relies + * on (a) ignoring bit 10 of the gPTE in is_present_gpte(), rather + * than treating it as a present bit, and (b) permission_fault() + * using an mmu->permissions[] array that effectively ignores + * ACC_USER_EXEC_MASK. Bit 10 of the gPTE does end up mirrored + * in the sPTEs but is ignored because L2 runs with MBEC disabled. + */ + shadow_xu_mask = VMX_EPT_USER_EXECUTABLE_MASK; + shadow_present_mask = VMX_EPT_SUPPRESS_VE_BIT; - shadow_acc_track_mask = VMX_EPT_RWX_MASK; + shadow_acc_track_mask = VMX_EPT_RWX_MASK | VMX_EPT_USER_EXECUTABLE_MASK; shadow_host_writable_mask = EPT_SPTE_HOST_WRITABLE; shadow_mmu_writable_mask = EPT_SPTE_MMU_WRITABLE; @@ -551,7 +576,8 @@ void kvm_mmu_reset_all_pte_masks(void) shadow_accessed_mask = PT_ACCESSED_MASK; shadow_dirty_mask = PT_DIRTY_MASK; shadow_nx_mask = PT64_NX_MASK; - shadow_x_mask = 0; + shadow_xs_mask = 0; + shadow_xu_mask = 0; shadow_present_mask = PT_PRESENT_MASK; shadow_acc_track_mask = 0; diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 8c0ffa2cded6..13eea94dd212 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -18,9 +18,19 @@ #define SPTE_MMU_PRESENT_MASK BIT_ULL(11) /* + * The ignored high bits are allocated as follows: + * - bits 52, 54: saved X-R bits for access tracking when EPT does not have A/D + * - bits 53 (EPT only): host writable + * - bits 55 (EPT only): MMU-writable + * - bits 56-59: unused + * - bits 60-61: type of A/D tracking + * - bits 62 (EPT only): saved XU bit for disabled AD + */ + +/* * TDP SPTES (more specifically, EPT SPTEs) may not have A/D bits, and may also * be restricted to using write-protection (for L2 when CPU dirty logging, i.e. - * PML, is enabled). Use bits 52 and 53 to hold the type of A/D tracking that + * PML, is enabled). Use bits 60 and 61 to hold the type of A/D tracking that * is must be employed for a given TDP SPTE. * * Note, the "enabled" mask must be '0', as bits 62:52 are _reserved_ for PAE @@ -29,7 +39,7 @@ * TDP with CPU dirty logging (PML). If NPT ever gains PML-like support, it * must be restricted to 64-bit KVM. */ -#define SPTE_TDP_AD_SHIFT 52 +#define SPTE_TDP_AD_SHIFT 60 #define SPTE_TDP_AD_MASK (3ULL << SPTE_TDP_AD_SHIFT) #define SPTE_TDP_AD_ENABLED (0ULL << SPTE_TDP_AD_SHIFT) #define SPTE_TDP_AD_DISABLED (1ULL << SPTE_TDP_AD_SHIFT) @@ -42,18 +52,6 @@ static_assert(SPTE_TDP_AD_ENABLED == 0); #define SPTE_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1)) #endif -#define SPTE_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | shadow_user_mask \ - | shadow_x_mask | shadow_nx_mask | shadow_me_mask) - -#define ACC_EXEC_MASK 1 -#define ACC_WRITE_MASK PT_WRITABLE_MASK -#define ACC_USER_MASK PT_USER_MASK -#define ACC_ALL (ACC_EXEC_MASK | ACC_WRITE_MASK | ACC_USER_MASK) - -/* The mask for the R/X bits in EPT PTEs */ -#define SPTE_EPT_READABLE_MASK 0x1ull -#define SPTE_EPT_EXECUTABLE_MASK 0x4ull - #define SPTE_LEVEL_BITS 9 #define SPTE_LEVEL_SHIFT(level) __PT_LEVEL_SHIFT(level, SPTE_LEVEL_BITS) #define SPTE_INDEX(address, level) __PT_INDEX(address, level, SPTE_LEVEL_BITS) @@ -66,9 +64,10 @@ static_assert(SPTE_TDP_AD_ENABLED == 0); * restored only when a write is attempted to the page. This mask obviously * must not overlap the A/D type mask. */ -#define SHADOW_ACC_TRACK_SAVED_BITS_MASK (SPTE_EPT_READABLE_MASK | \ - SPTE_EPT_EXECUTABLE_MASK) -#define SHADOW_ACC_TRACK_SAVED_BITS_SHIFT 54 +#define SHADOW_ACC_TRACK_SAVED_BITS_MASK (VMX_EPT_READABLE_MASK | \ + VMX_EPT_EXECUTABLE_MASK | \ + VMX_EPT_USER_EXECUTABLE_MASK) +#define SHADOW_ACC_TRACK_SAVED_BITS_SHIFT 52 #define SHADOW_ACC_TRACK_SAVED_MASK (SHADOW_ACC_TRACK_SAVED_BITS_MASK << \ SHADOW_ACC_TRACK_SAVED_BITS_SHIFT) static_assert(!(SPTE_TDP_AD_MASK & SHADOW_ACC_TRACK_SAVED_MASK)); @@ -87,8 +86,8 @@ static_assert(!(SPTE_TDP_AD_MASK & SHADOW_ACC_TRACK_SAVED_MASK)); * to not overlap the A/D type mask or the saved access bits of access-tracked * SPTEs when A/D bits are disabled. */ -#define EPT_SPTE_HOST_WRITABLE BIT_ULL(57) -#define EPT_SPTE_MMU_WRITABLE BIT_ULL(58) +#define EPT_SPTE_HOST_WRITABLE BIT_ULL(53) +#define EPT_SPTE_MMU_WRITABLE BIT_ULL(55) static_assert(!(EPT_SPTE_HOST_WRITABLE & SPTE_TDP_AD_MASK)); static_assert(!(EPT_SPTE_MMU_WRITABLE & SPTE_TDP_AD_MASK)); @@ -99,11 +98,11 @@ static_assert(!(EPT_SPTE_MMU_WRITABLE & SHADOW_ACC_TRACK_SAVED_MASK)); #undef SHADOW_ACC_TRACK_SAVED_MASK /* - * Due to limited space in PTEs, the MMIO generation is a 19 bit subset of + * Due to limited space in PTEs, the MMIO generation is an 18 bit subset of * the memslots generation and is derived as follows: * - * Bits 0-7 of the MMIO generation are propagated to spte bits 3-10 - * Bits 8-18 of the MMIO generation are propagated to spte bits 52-62 + * Bits 0-6 of the MMIO generation are propagated to spte bits 3-9 + * Bits 7-17 of the MMIO generation are propagated to spte bits 52-62 * * The KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS flag is intentionally not included in * the MMIO generation number, as doing so would require stealing a bit from @@ -114,7 +113,7 @@ static_assert(!(EPT_SPTE_MMU_WRITABLE & SHADOW_ACC_TRACK_SAVED_MASK)); */ #define MMIO_SPTE_GEN_LOW_START 3 -#define MMIO_SPTE_GEN_LOW_END 10 +#define MMIO_SPTE_GEN_LOW_END 9 #define MMIO_SPTE_GEN_HIGH_START 52 #define MMIO_SPTE_GEN_HIGH_END 62 @@ -136,7 +135,8 @@ static_assert(!(SPTE_MMU_PRESENT_MASK & * and so they're off-limits for generation; additional checks ensure the mask * doesn't overlap legal PA bits), and bit 63 (carved out for future usage). */ -#define SPTE_MMIO_ALLOWED_MASK (BIT_ULL(63) | GENMASK_ULL(51, 12) | GENMASK_ULL(2, 0)) +#define SPTE_MMIO_ALLOWED_MASK (BIT_ULL(63) | GENMASK_ULL(51, 12) | \ + BIT_ULL(10) | GENMASK_ULL(2, 0)) static_assert(!(SPTE_MMIO_ALLOWED_MASK & (SPTE_MMU_PRESENT_MASK | MMIO_SPTE_GEN_LOW_MASK | MMIO_SPTE_GEN_HIGH_MASK))); @@ -144,7 +144,7 @@ static_assert(!(SPTE_MMIO_ALLOWED_MASK & #define MMIO_SPTE_GEN_HIGH_BITS (MMIO_SPTE_GEN_HIGH_END - MMIO_SPTE_GEN_HIGH_START + 1) /* remember to adjust the comment above as well if you change these */ -static_assert(MMIO_SPTE_GEN_LOW_BITS == 8 && MMIO_SPTE_GEN_HIGH_BITS == 11); +static_assert(MMIO_SPTE_GEN_LOW_BITS == 7 && MMIO_SPTE_GEN_HIGH_BITS == 11); #define MMIO_SPTE_GEN_LOW_SHIFT (MMIO_SPTE_GEN_LOW_START - 0) #define MMIO_SPTE_GEN_HIGH_SHIFT (MMIO_SPTE_GEN_HIGH_START - MMIO_SPTE_GEN_LOW_BITS) @@ -179,8 +179,9 @@ extern bool __read_mostly kvm_ad_enabled; extern u64 __read_mostly shadow_host_writable_mask; extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask; -extern u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */ extern u64 __read_mostly shadow_user_mask; +extern u64 __read_mostly shadow_xs_mask; /* mutual exclusive with nx_mask and user_mask */ +extern u64 __read_mostly shadow_xu_mask; /* mutual exclusive with nx_mask and user_mask */ extern u64 __read_mostly shadow_accessed_mask; extern u64 __read_mostly shadow_dirty_mask; extern u64 __read_mostly shadow_mmio_value; @@ -220,10 +221,11 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_mask; * * Only used by the TDP MMU. */ -#define FROZEN_SPTE (SHADOW_NONPRESENT_VALUE | 0x5a0ULL) +#define FROZEN_SPTE (SHADOW_NONPRESENT_VALUE | 0x1a0ULL) -/* Frozen SPTEs must not be misconstrued as shadow present PTEs. */ -static_assert(!(FROZEN_SPTE & SPTE_MMU_PRESENT_MASK)); +/* Frozen SPTEs must not be misconstrued as shadow or MMU present PTEs. */ +static_assert(!(FROZEN_SPTE & (SPTE_MMU_PRESENT_MASK | + VMX_EPT_RWX_MASK | VMX_EPT_USER_EXECUTABLE_MASK))); static inline bool is_frozen_spte(u64 spte) { @@ -357,7 +359,13 @@ static inline bool is_last_spte(u64 pte, int level) static inline bool is_executable_pte(u64 spte) { - return (spte & (shadow_x_mask | shadow_nx_mask)) == shadow_x_mask; + /* + * For now, return true if either the XS or XU bit is set + * This function is only used for fast_page_fault, + * which never processes shadow EPT, and regular page + * tables always have XS==XU. + */ + return (spte & (shadow_xs_mask | shadow_xu_mask | shadow_nx_mask)) != shadow_nx_mask; } static inline kvm_pfn_t spte_to_pfn(u64 pte) @@ -387,6 +395,8 @@ static inline bool __is_rsvd_bits_set(struct rsvd_bits_validate *rsvd_check, static inline bool __is_bad_mt_xwr(struct rsvd_bits_validate *rsvd_check, u64 pte) { + if (pte & VMX_EPT_USER_EXECUTABLE_MASK) + pte |= VMX_EPT_EXECUTABLE_MASK; return rsvd_check->bad_mt_xwr & BIT_ULL(pte & 0x3f); } diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 7b1102d26f9c..5b3041138301 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -53,13 +53,18 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) rcu_barrier(); } -static void tdp_mmu_free_sp(struct kvm_mmu_page *sp) +static void __tdp_mmu_free_sp(struct kvm_mmu_page *sp) { - free_page((unsigned long)sp->external_spt); free_page((unsigned long)sp->spt); kmem_cache_free(mmu_page_header_cache, sp); } +static void tdp_mmu_free_unused_sp(struct kvm_mmu_page *sp) +{ + free_page((unsigned long)sp->external_spt); + __tdp_mmu_free_sp(sp); +} + /* * This is called through call_rcu in order to free TDP page table memory * safely with respect to other kernel threads that may be operating on @@ -73,7 +78,8 @@ static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head) struct kvm_mmu_page *sp = container_of(head, struct kvm_mmu_page, rcu_head); - tdp_mmu_free_sp(sp); + WARN_ON_ONCE(sp->external_spt); + __tdp_mmu_free_sp(sp); } void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) @@ -320,9 +326,9 @@ out_read_unlock: } } -static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, - u64 old_spte, u64 new_spte, int level, - bool shared); +static void handle_changed_spte(struct kvm *kvm, struct kvm_mmu_page *sp, + gfn_t gfn, u64 old_spte, u64 new_spte, + int level, bool shared); static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) { @@ -359,25 +365,6 @@ static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp) spin_unlock(&kvm->arch.tdp_mmu_pages_lock); } -static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte, - int level) -{ - /* - * External (TDX) SPTEs are limited to PG_LEVEL_4K, and external - * PTs are removed in a special order, involving free_external_spt(). - * But remove_external_spte() will be called on non-leaf PTEs via - * __tdp_mmu_zap_root(), so avoid the error the former would return - * in this case. - */ - if (!is_last_spte(old_spte, level)) - return; - - /* Zapping leaf spte is allowed only when write lock is held. */ - lockdep_assert_held_write(&kvm->mmu_lock); - - kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_spte); -} - /** * handle_removed_pt() - handle a page table removed from the TDP structure * @@ -471,86 +458,19 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared) old_spte = kvm_tdp_mmu_write_spte(sptep, old_spte, FROZEN_SPTE, level); } - handle_changed_spte(kvm, kvm_mmu_page_as_id(sp), gfn, - old_spte, FROZEN_SPTE, level, shared); - - if (is_mirror_sp(sp)) { - KVM_BUG_ON(shared, kvm); - remove_external_spte(kvm, gfn, old_spte, level); - } + handle_changed_spte(kvm, sp, gfn, old_spte, FROZEN_SPTE, level, shared); } - if (is_mirror_sp(sp) && - WARN_ON(kvm_x86_call(free_external_spt)(kvm, base_gfn, sp->role.level, - sp->external_spt))) { - /* - * Failed to free page table page in mirror page table and - * there is nothing to do further. - * Intentionally leak the page to prevent the kernel from - * accessing the encrypted page. - */ - sp->external_spt = NULL; - } + if (is_mirror_sp(sp)) + kvm_x86_call(free_external_spt)(kvm, sp); call_rcu(&sp->rcu_head, tdp_mmu_free_sp_rcu_callback); } -static void *get_external_spt(gfn_t gfn, u64 new_spte, int level) -{ - if (is_shadow_present_pte(new_spte) && !is_last_spte(new_spte, level)) { - struct kvm_mmu_page *sp = spte_to_child_sp(new_spte); - - WARN_ON_ONCE(sp->role.level + 1 != level); - WARN_ON_ONCE(sp->gfn != gfn); - return sp->external_spt; - } - - return NULL; -} - -static int __must_check set_external_spte_present(struct kvm *kvm, tdp_ptep_t sptep, - gfn_t gfn, u64 old_spte, - u64 new_spte, int level) -{ - bool was_present = is_shadow_present_pte(old_spte); - bool is_present = is_shadow_present_pte(new_spte); - bool is_leaf = is_present && is_last_spte(new_spte, level); - int ret = 0; - - KVM_BUG_ON(was_present, kvm); - - lockdep_assert_held(&kvm->mmu_lock); - /* - * We need to lock out other updates to the SPTE until the external - * page table has been modified. Use FROZEN_SPTE similar to - * the zapping case. - */ - if (!try_cmpxchg64(rcu_dereference(sptep), &old_spte, FROZEN_SPTE)) - return -EBUSY; - - /* - * Use different call to either set up middle level - * external page table, or leaf. - */ - if (is_leaf) { - ret = kvm_x86_call(set_external_spte)(kvm, gfn, level, new_spte); - } else { - void *external_spt = get_external_spt(gfn, new_spte, level); - - KVM_BUG_ON(!external_spt, kvm); - ret = kvm_x86_call(link_external_spt)(kvm, gfn, level, external_spt); - } - if (ret) - __kvm_tdp_mmu_write_spte(sptep, old_spte); - else - __kvm_tdp_mmu_write_spte(sptep, new_spte); - return ret; -} - /** - * handle_changed_spte - handle bookkeeping associated with an SPTE change + * __handle_changed_spte - handle bookkeeping associated with an SPTE change * @kvm: kvm instance - * @as_id: the address space of the paging structure the SPTE was a part of + * @sp: the page table in which the SPTE resides * @gfn: the base GFN that was mapped by the SPTE * @old_spte: The value of the SPTE before the change * @new_spte: The value of the SPTE after the change @@ -563,15 +483,16 @@ static int __must_check set_external_spte_present(struct kvm *kvm, tdp_ptep_t sp * dirty logging updates are handled in common code, not here (see make_spte() * and fast_pf_fix_direct_spte()). */ -static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, - u64 old_spte, u64 new_spte, int level, - bool shared) +static int __handle_changed_spte(struct kvm *kvm, struct kvm_mmu_page *sp, + gfn_t gfn, u64 old_spte, u64 new_spte, + int level, bool shared) { bool was_present = is_shadow_present_pte(old_spte); bool is_present = is_shadow_present_pte(new_spte); bool was_leaf = was_present && is_last_spte(old_spte, level); bool is_leaf = is_present && is_last_spte(new_spte, level); bool pfn_changed = spte_to_pfn(old_spte) != spte_to_pfn(new_spte); + int as_id = kvm_mmu_page_as_id(sp); WARN_ON_ONCE(level > PT64_ROOT_MAX_LEVEL); WARN_ON_ONCE(level < PG_LEVEL_4K); @@ -601,9 +522,7 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, } if (old_spte == new_spte) - return; - - trace_kvm_tdp_mmu_spte_changed(as_id, gfn, level, old_spte, new_spte); + return 0; if (is_leaf) check_spte_writable_invariants(new_spte); @@ -630,21 +549,45 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, "a temporary frozen SPTE.\n" "as_id: %d gfn: %llx old_spte: %llx new_spte: %llx level: %d", as_id, gfn, old_spte, new_spte, level); - return; - } - if (is_leaf != was_leaf) - kvm_update_page_stats(kvm, level, is_leaf ? 1 : -1); + trace_kvm_tdp_mmu_spte_changed(as_id, gfn, level, old_spte, new_spte); + return 0; + } /* * Recursively handle child PTs if the change removed a subtree from * the paging structure. Note the WARN on the PFN changing without the * SPTE being converted to a hugepage (leaf) or being zapped. Shadow * pages are kernel allocations and should never be migrated. + * + * For the mirror page table, propagate all changes to the external SPTE + * (except zapping/promotion of non-leaf SPTEs) via the + * set_external_spte() op. */ if (was_present && !was_leaf && - (is_leaf || !is_present || WARN_ON_ONCE(pfn_changed))) + (is_leaf || !is_present || WARN_ON_ONCE(pfn_changed))) { handle_removed_pt(kvm, spte_to_child_pt(old_spte, level), shared); + } else if (is_mirror_sp(sp)) { + int r; + + r = kvm_x86_call(set_external_spte)(kvm, gfn, old_spte, new_spte, level); + if (r) + return r; + } + trace_kvm_tdp_mmu_spte_changed(as_id, gfn, level, old_spte, new_spte); + + if (is_leaf != was_leaf) + kvm_update_page_stats(kvm, level, is_leaf ? 1 : -1); + + return 0; +} + +static void handle_changed_spte(struct kvm *kvm, struct kvm_mmu_page *sp, + gfn_t gfn, u64 old_spte, u64 new_spte, + int level, bool shared) +{ + KVM_BUG_ON(__handle_changed_spte(kvm, sp, gfn, old_spte, new_spte, + level, shared), kvm); } static inline int __must_check __tdp_mmu_set_spte_atomic(struct kvm *kvm, @@ -659,34 +602,15 @@ static inline int __must_check __tdp_mmu_set_spte_atomic(struct kvm *kvm, */ WARN_ON_ONCE(iter->yielded || is_frozen_spte(iter->old_spte)); - if (is_mirror_sptep(iter->sptep) && !is_frozen_spte(new_spte)) { - int ret; - - /* - * Users of atomic zapping don't operate on mirror roots, - * so don't handle it and bug the VM if it's seen. - */ - if (KVM_BUG_ON(!is_shadow_present_pte(new_spte), kvm)) - return -EBUSY; - - ret = set_external_spte_present(kvm, iter->sptep, iter->gfn, - iter->old_spte, new_spte, iter->level); - if (ret) - return ret; - } else { - u64 *sptep = rcu_dereference(iter->sptep); - - /* - * Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs - * and does not hold the mmu_lock. On failure, i.e. if a - * different logical CPU modified the SPTE, try_cmpxchg64() - * updates iter->old_spte with the current value, so the caller - * operates on fresh data, e.g. if it retries - * tdp_mmu_set_spte_atomic() - */ - if (!try_cmpxchg64(sptep, &iter->old_spte, new_spte)) - return -EBUSY; - } + /* + * Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and + * does not hold the mmu_lock. On failure, i.e. if a different logical + * CPU modified the SPTE, try_cmpxchg64() updates iter->old_spte with + * the current value, so the caller operates on fresh data, e.g. if it + * retries tdp_mmu_set_spte_atomic(). + */ + if (!try_cmpxchg64(rcu_dereference(iter->sptep), &iter->old_spte, new_spte)) + return -EBUSY; return 0; } @@ -712,24 +636,61 @@ static inline int __must_check tdp_mmu_set_spte_atomic(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { + struct kvm_mmu_page *sp = sptep_to_sp(rcu_dereference(iter->sptep)); int ret; lockdep_assert_held_read(&kvm->mmu_lock); - ret = __tdp_mmu_set_spte_atomic(kvm, iter, new_spte); + /* Should not set FROZEN_SPTE as a long-term value. */ + KVM_MMU_WARN_ON(is_frozen_spte(new_spte)); + + /* + * Temporarily freeze the SPTE until the external PTE operation has + * completed, e.g. so that concurrent faults don't attempt to install a + * child PTE in the external page table before the parent PTE has been + * written. + */ + if (is_mirror_sptep(iter->sptep)) + ret = __tdp_mmu_set_spte_atomic(kvm, iter, FROZEN_SPTE); + else + ret = __tdp_mmu_set_spte_atomic(kvm, iter, new_spte); + if (ret) return ret; - handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, - new_spte, iter->level, true); - - return 0; + /* + * Handle the change from iter->old_spte to new_spte. + * + * Note: for mirror page table, this means the updates of the external + * PTE, statistics, or updates of child SPTEs, child external PTEs and + * corresponding statistics are performed while the mirror SPTE is in + * frozen state (i.e., before the mirror SPTE is set to new_spte). + */ + ret = __handle_changed_spte(kvm, sp, iter->gfn, iter->old_spte, + new_spte, iter->level, true); + /* + * Unfreeze the mirror SPTE. If updating the external SPTE failed, + * restore the old value so that the mirror SPTE isn't frozen in + * perpetuity, otherwise set the mirror SPTE to the new desired value. + */ + if (is_mirror_sptep(iter->sptep)) { + if (ret) + __kvm_tdp_mmu_write_spte(iter->sptep, iter->old_spte); + else + __kvm_tdp_mmu_write_spte(iter->sptep, new_spte); + } else { + /* + * Bug the VM if handling the change failed, as failure is only + * allowed if KVM couldn't update the external SPTE. + */ + KVM_BUG_ON(ret, kvm); + } + return ret; } /* * tdp_mmu_set_spte - Set a TDP MMU SPTE and handle the associated bookkeeping * @kvm: KVM instance - * @as_id: Address space ID, i.e. regular vs. SMM * @sptep: Pointer to the SPTE * @old_spte: The current value of the SPTE * @new_spte: The new value that will be set for the SPTE @@ -739,9 +700,11 @@ static inline int __must_check tdp_mmu_set_spte_atomic(struct kvm *kvm, * Returns the old SPTE value, which _may_ be different than @old_spte if the * SPTE had voldatile bits. */ -static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, - u64 old_spte, u64 new_spte, gfn_t gfn, int level) +static u64 tdp_mmu_set_spte(struct kvm *kvm, tdp_ptep_t sptep, u64 old_spte, + u64 new_spte, gfn_t gfn, int level) { + struct kvm_mmu_page *sp = sptep_to_sp(rcu_dereference(sptep)); + lockdep_assert_held_write(&kvm->mmu_lock); /* @@ -755,16 +718,7 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, old_spte = kvm_tdp_mmu_write_spte(sptep, old_spte, new_spte, level); - handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); - - /* - * Users that do non-atomic setting of PTEs don't operate on mirror - * roots, so don't handle it and bug the VM if it's seen. - */ - if (is_mirror_sptep(sptep)) { - KVM_BUG_ON(is_shadow_present_pte(new_spte), kvm); - remove_external_spte(kvm, gfn, old_spte, level); - } + handle_changed_spte(kvm, sp, gfn, old_spte, new_spte, level, false); return old_spte; } @@ -773,9 +727,8 @@ static inline void tdp_mmu_iter_set_spte(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { WARN_ON_ONCE(iter->yielded); - iter->old_spte = tdp_mmu_set_spte(kvm, iter->as_id, iter->sptep, - iter->old_spte, new_spte, - iter->gfn, iter->level); + iter->old_spte = tdp_mmu_set_spte(kvm, iter->sptep, iter->old_spte, + new_spte, iter->gfn, iter->level); } #define tdp_root_for_each_pte(_iter, _kvm, _root, _start, _end) \ @@ -1185,9 +1138,9 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, } if (unlikely(!fault->slot)) - new_spte = make_mmio_spte(vcpu, iter->gfn, ACC_ALL); + new_spte = make_mmio_spte(vcpu, iter->gfn, sp->role.access); else - wrprot = make_spte(vcpu, sp, fault->slot, ACC_ALL, iter->gfn, + wrprot = make_spte(vcpu, sp, fault->slot, sp->role.access, iter->gfn, fault->pfn, iter->old_spte, fault->prefetch, false, fault->map_writable, &new_spte); @@ -1272,7 +1225,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) kvm_mmu_hugepage_adjust(vcpu, fault); - trace_kvm_mmu_spte_requested(fault); + trace_kvm_mmu_spte_requested(fault, root->role.access); rcu_read_lock(); @@ -1321,7 +1274,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) * failed, e.g. because a different task modified the SPTE. */ if (r) { - tdp_mmu_free_sp(sp); + tdp_mmu_free_unused_sp(sp); goto retry; } @@ -1377,6 +1330,10 @@ static void kvm_tdp_mmu_age_spte(struct kvm *kvm, struct tdp_iter *iter) { u64 new_spte; + /* TODO: Add support for aging external SPTEs, if necessary. */ + if (WARN_ON_ONCE(is_mirror_sptep(iter->sptep))) + return; + if (spte_ad_enabled(iter->old_spte)) { iter->old_spte = tdp_mmu_clear_spte_bits_atomic(iter->sptep, shadow_accessed_mask); @@ -1628,7 +1585,7 @@ retry: * installs its own sp in place of the last sp we tried to split. */ if (sp) - tdp_mmu_free_sp(sp); + tdp_mmu_free_unused_sp(sp); return 0; } diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index aee70e5dc15d..dd1c57593f48 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -16,6 +16,7 @@ #include <linux/perf_event.h> #include <linux/bsearch.h> #include <linux/sort.h> +#include <linux/moduleparam.h> #include <asm/perf_event.h> #include <asm/cpu_device_id.h> #include "x86.h" @@ -33,6 +34,15 @@ static struct x86_pmu_capability __read_mostly kvm_host_pmu; struct x86_pmu_capability __read_mostly kvm_pmu_cap; EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_pmu_cap); +/* Enable/disable PMU virtualization */ +bool __read_mostly enable_pmu = true; +EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_pmu); +module_param(enable_pmu, bool, 0444); + +/* Enable/disabled mediated PMU virtualization. */ +bool __read_mostly enable_mediated_pmu; +EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_mediated_pmu); + struct kvm_pmu_emulated_event_selectors { u64 INSTRUCTIONS_RETIRED; u64 BRANCH_INSTRUCTIONS_RETIRED; @@ -88,7 +98,9 @@ static struct kvm_pmu_ops kvm_pmu_ops __read_mostly; DEFINE_STATIC_CALL_NULL(kvm_x86_pmu_##func, \ *(((struct kvm_pmu_ops *)0)->func)); #define KVM_X86_PMU_OP_OPTIONAL KVM_X86_PMU_OP +#define KVM_X86_PMU_OP_OPTIONAL_RET0 KVM_X86_PMU_OP #include <asm/kvm-x86-pmu-ops.h> +EXPORT_STATIC_CALL_GPL(kvm_x86_pmu_pmc_is_disabled_in_current_mode); void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops) { @@ -99,6 +111,9 @@ void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops) #define KVM_X86_PMU_OP(func) \ WARN_ON(!kvm_pmu_ops.func); __KVM_X86_PMU_OP(func) #define KVM_X86_PMU_OP_OPTIONAL __KVM_X86_PMU_OP +#define KVM_X86_PMU_OP_OPTIONAL_RET0(func) \ + static_call_update(kvm_x86_pmu_##func, (void *)kvm_pmu_ops.func ? : \ + (void *)__static_call_return0); #include <asm/kvm-x86-pmu-ops.h> #undef __KVM_X86_PMU_OP } @@ -522,7 +537,7 @@ static bool pmc_is_event_allowed(struct kvm_pmc *pmc) static void kvm_mediated_pmu_refresh_event_filter(struct kvm_pmc *pmc) { - bool allowed = pmc_is_event_allowed(pmc); + bool allowed = pmc_is_locally_enabled(pmc) && pmc_is_event_allowed(pmc); struct kvm_pmu *pmu = pmc_to_pmu(pmc); if (pmc_is_gp(pmc)) { @@ -670,6 +685,7 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) kvm_for_each_pmc(pmu, pmc, bit, bitmap) kvm_pmu_recalc_pmc_emulation(pmu, pmc); } +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_pmu_handle_event); int kvm_pmu_check_rdpmc_early(struct kvm_vcpu *vcpu, unsigned int idx) { @@ -879,7 +895,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) if (pmu->global_ctrl != data) { diff = pmu->global_ctrl ^ data; pmu->global_ctrl = data; - reprogram_counters(pmu, diff); + kvm_pmu_request_counters_reprogram(pmu, diff); } /* * Unconditionally forward writes to vendor code, i.e. to the @@ -921,6 +937,7 @@ static void kvm_pmu_reset(struct kvm_vcpu *vcpu) pmu->need_cleanup = false; bitmap_zero(pmu->reprogram_pmi, X86_PMC_IDX_MAX); + bitmap_zero(pmu->pmc_has_mode_specific_enables, X86_PMC_IDX_MAX); kvm_for_each_pmc(pmu, pmc, i, pmu->all_valid_pmc_idx) { pmc_stop_counter(pmc); diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h index 0925246731cb..a5821d7c87f9 100644 --- a/arch/x86/kvm/pmu.h +++ b/arch/x86/kvm/pmu.h @@ -36,6 +36,7 @@ struct kvm_pmu_ops { void (*reset)(struct kvm_vcpu *vcpu); void (*deliver_pmi)(struct kvm_vcpu *vcpu); void (*cleanup)(struct kvm_vcpu *vcpu); + bool (*pmc_is_disabled_in_current_mode)(struct kvm_pmc *pmc); bool (*is_mediated_pmu_supported)(struct x86_pmu_capability *host_pmu); void (*mediated_load)(struct kvm_vcpu *vcpu); @@ -53,6 +54,17 @@ struct kvm_pmu_ops { const u32 MSR_STRIDE; }; +#define kvm_pmu_call(func) static_call(kvm_x86_pmu_##func) + +#define KVM_X86_PMU_OP(func) \ + DECLARE_STATIC_CALL(kvm_x86_pmu_##func, *(((struct kvm_pmu_ops *)0)->func)); +#define KVM_X86_PMU_OP_OPTIONAL KVM_X86_PMU_OP +#define KVM_X86_PMU_OP_OPTIONAL_RET0 KVM_X86_PMU_OP +#include <asm/kvm-x86-pmu-ops.h> + +extern bool enable_pmu; +extern bool enable_mediated_pmu; + void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops); void kvm_handle_guest_mediated_pmi(void); @@ -190,7 +202,13 @@ static inline bool pmc_is_locally_enabled(struct kvm_pmc *pmc) pmc->idx - KVM_FIXED_PMC_BASE_IDX) & (INTEL_FIXED_0_KERNEL | INTEL_FIXED_0_USER); - return pmc->eventsel & ARCH_PERFMON_EVENTSEL_ENABLE; + if (!(pmc->eventsel & ARCH_PERFMON_EVENTSEL_ENABLE)) + return false; + + if (!test_bit(pmc->idx, pmu->pmc_has_mode_specific_enables)) + return true; + + return !kvm_pmu_call(pmc_is_disabled_in_current_mode)(pmc); } extern struct x86_pmu_capability kvm_pmu_cap; @@ -198,6 +216,7 @@ extern struct x86_pmu_capability kvm_pmu_cap; void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops); void kvm_pmu_recalc_pmc_emulation(struct kvm_pmu *pmu, struct kvm_pmc *pmc); +void kvm_pmu_handle_event(struct kvm_vcpu *vcpu); static inline void kvm_pmu_request_counter_reprogram(struct kvm_pmc *pmc) { @@ -207,16 +226,24 @@ static inline void kvm_pmu_request_counter_reprogram(struct kvm_pmc *pmc) kvm_make_request(KVM_REQ_PMU, pmc->vcpu); } -static inline void reprogram_counters(struct kvm_pmu *pmu, u64 diff) +static inline void __kvm_pmu_reprogram_counters(struct kvm_pmu *pmu, + u64 counters, + bool defer) { - int bit; - - if (!diff) + if (!counters) return; - for_each_set_bit(bit, (unsigned long *)&diff, X86_PMC_IDX_MAX) - set_bit(bit, pmu->reprogram_pmi); - kvm_make_request(KVM_REQ_PMU, pmu_to_vcpu(pmu)); + atomic64_or(counters, &pmu->__reprogram_pmi); + if (defer) + kvm_make_request(KVM_REQ_PMU, pmu_to_vcpu(pmu)); + else + kvm_pmu_handle_event(pmu_to_vcpu(pmu)); +} + +static inline void kvm_pmu_request_counters_reprogram(struct kvm_pmu *pmu, + u64 counters) +{ + __kvm_pmu_reprogram_counters(pmu, counters, true); } /* @@ -245,7 +272,6 @@ static inline bool kvm_pmu_is_fastpath_emulation_allowed(struct kvm_vcpu *vcpu) } void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu); -void kvm_pmu_handle_event(struct kvm_vcpu *vcpu); int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned pmc, u64 *data); int kvm_pmu_check_rdpmc_early(struct kvm_vcpu *vcpu, unsigned int idx); bool kvm_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr); diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h new file mode 100644 index 000000000000..5bda738afb7c --- /dev/null +++ b/arch/x86/kvm/regs.h @@ -0,0 +1,423 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef ARCH_X86_KVM_REGS_H +#define ARCH_X86_KVM_REGS_H + +#include <linux/kvm_host.h> + +#define KVM_POSSIBLE_CR0_GUEST_BITS (X86_CR0_TS | X86_CR0_WP) +#define KVM_POSSIBLE_CR4_GUEST_BITS \ + (X86_CR4_PVI | X86_CR4_DE | X86_CR4_PCE | X86_CR4_OSFXSR \ + | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE \ + | X86_CR4_CET) + +#define X86_CR0_PDPTR_BITS (X86_CR0_CD | X86_CR0_NW | X86_CR0_PG) +#define X86_CR4_TLBFLUSH_BITS (X86_CR4_PGE | X86_CR4_PCIDE | X86_CR4_PAE | X86_CR4_SMEP) +#define X86_CR4_PDPTR_BITS (X86_CR4_PGE | X86_CR4_PSE | X86_CR4_PAE | X86_CR4_SMEP) + +static_assert(!(KVM_POSSIBLE_CR0_GUEST_BITS & X86_CR0_PDPTR_BITS)); + +static inline bool is_long_mode(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_X86_64 + return !!(vcpu->arch.efer & EFER_LMA); +#else + return false; +#endif +} + +static inline bool is_64_bit_mode(struct kvm_vcpu *vcpu) +{ + int cs_db, cs_l; + + WARN_ON_ONCE(vcpu->arch.guest_state_protected); + + if (!is_long_mode(vcpu)) + return false; + kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); + return cs_l; +} + +static inline bool is_64_bit_hypercall(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_X86_64 + /* + * If running with protected guest state, the CS register is not + * accessible. The hypercall register values will have had to been + * provided in 64-bit mode, so assume the guest is in 64-bit. + */ + return vcpu->arch.guest_state_protected || is_64_bit_mode(vcpu); +#else + return false; +#endif +} + +static __always_inline unsigned long kvm_reg_mode_mask(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_X86_64 + return is_64_bit_mode(vcpu) ? GENMASK(63, 0) : GENMASK(31, 0); +#else + return GENMASK(31, 0); +#endif +} + +#define __BUILD_KVM_GPR_ACCESSORS(lname, uname) \ +static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu->arch.regs[VCPU_REGS_##uname] & kvm_reg_mode_mask(vcpu); \ +} \ +static __always_inline unsigned long kvm_##lname##_read_raw(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu->arch.regs[VCPU_REGS_##uname]; \ +} \ +static __always_inline void kvm_##lname##_write_raw(struct kvm_vcpu *vcpu, \ + unsigned long val) \ +{ \ + vcpu->arch.regs[VCPU_REGS_##uname] = val; \ +} +#define BUILD_KVM_GPR_ACCESSORS(lname, uname) \ +static __always_inline u32 kvm_e##lname##_read(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu->arch.regs[VCPU_REGS_##uname]; \ +} \ +static __always_inline void kvm_e##lname##_write(struct kvm_vcpu *vcpu, u32 val) \ +{ \ + vcpu->arch.regs[VCPU_REGS_##uname] = val; \ +} \ +__BUILD_KVM_GPR_ACCESSORS(r##lname, uname) + +BUILD_KVM_GPR_ACCESSORS(ax, RAX) +BUILD_KVM_GPR_ACCESSORS(bx, RBX) +BUILD_KVM_GPR_ACCESSORS(cx, RCX) +BUILD_KVM_GPR_ACCESSORS(dx, RDX) +BUILD_KVM_GPR_ACCESSORS(bp, RBP) +BUILD_KVM_GPR_ACCESSORS(si, RSI) +BUILD_KVM_GPR_ACCESSORS(di, RDI) +#ifdef CONFIG_X86_64 +__BUILD_KVM_GPR_ACCESSORS(r8, R8) +__BUILD_KVM_GPR_ACCESSORS(r9, R9) +__BUILD_KVM_GPR_ACCESSORS(r10, R10) +__BUILD_KVM_GPR_ACCESSORS(r11, R11) +__BUILD_KVM_GPR_ACCESSORS(r12, R12) +__BUILD_KVM_GPR_ACCESSORS(r13, R13) +__BUILD_KVM_GPR_ACCESSORS(r14, R14) +__BUILD_KVM_GPR_ACCESSORS(r15, R15) +#endif + +/* + * Using the register cache from interrupt context is generally not allowed, as + * caching a register and marking it available/dirty can't be done atomically, + * i.e. accesses from interrupt context may clobber state or read stale data if + * the vCPU task is in the process of updating the cache. The exception is if + * KVM is handling a PMI IRQ/NMI VM-Exit, as that bound code sequence doesn't + * touch the cache, it runs after the cache is reset (post VM-Exit), and PMIs + * need to access several registers that are cacheable. + */ +#define kvm_assert_register_caching_allowed(vcpu) \ + lockdep_assert_once(in_task() || kvm_arch_pmi_in_guest(vcpu)) + +/* + * avail dirty + * 0 0 register in VMCS/VMCB + * 0 1 *INVALID* + * 1 0 register in vcpu->arch + * 1 1 register in vcpu->arch, needs to be stored back + */ +static inline bool kvm_register_is_available(struct kvm_vcpu *vcpu, + enum kvm_reg reg) +{ + kvm_assert_register_caching_allowed(vcpu); + return test_bit(reg, vcpu->arch.regs_avail); +} + +static inline bool kvm_register_is_dirty(struct kvm_vcpu *vcpu, + enum kvm_reg reg) +{ + kvm_assert_register_caching_allowed(vcpu); + return test_bit(reg, vcpu->arch.regs_dirty); +} + +static inline void kvm_register_mark_for_reload(struct kvm_vcpu *vcpu, + enum kvm_reg reg) +{ + kvm_assert_register_caching_allowed(vcpu); + __clear_bit(reg, vcpu->arch.regs_avail); + __clear_bit(reg, vcpu->arch.regs_dirty); +} + +static inline void kvm_register_mark_available(struct kvm_vcpu *vcpu, + enum kvm_reg reg) +{ + kvm_assert_register_caching_allowed(vcpu); + __set_bit(reg, vcpu->arch.regs_avail); +} + +static inline void kvm_register_mark_dirty(struct kvm_vcpu *vcpu, + enum kvm_reg reg) +{ + kvm_assert_register_caching_allowed(vcpu); + __set_bit(reg, vcpu->arch.regs_avail); + __set_bit(reg, vcpu->arch.regs_dirty); +} + +/* + * kvm_register_test_and_mark_available() is a special snowflake that uses an + * arch bitop directly to avoid the explicit instrumentation that comes with + * the generic bitops. This allows code that cannot be instrumented (noinstr + * functions), e.g. the low level VM-Enter/VM-Exit paths, to cache registers. + */ +static __always_inline bool kvm_register_test_and_mark_available(struct kvm_vcpu *vcpu, + enum kvm_reg reg) +{ + kvm_assert_register_caching_allowed(vcpu); + return arch___test_and_set_bit(reg, vcpu->arch.regs_avail); +} + +static __always_inline void kvm_clear_available_registers(struct kvm_vcpu *vcpu, + unsigned long clear_mask) +{ + BUILD_BUG_ON(sizeof(clear_mask) != sizeof(vcpu->arch.regs_avail[0])); + BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.regs_avail) != 1); + + /* + * Note the bitwise-AND! In practice, a straight write would also work + * as KVM initializes the mask to all ones and never clears registers + * that are eagerly synchronized. Using a bitwise-AND adds a bit of + * sanity checking as incorrectly marking an eagerly sync'd register + * unavailable will generate a WARN due to an unexpected cache request. + */ + vcpu->arch.regs_avail[0] &= ~clear_mask; +} + +static __always_inline void kvm_reset_dirty_registers(struct kvm_vcpu *vcpu) +{ + BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.regs_dirty) != 1); + vcpu->arch.regs_dirty[0] = 0; +} + +/* + * The "raw" register helpers are only for cases where the full 64 bits of a + * register are read/written irrespective of current vCPU mode. In other words, + * odds are good you shouldn't be using the raw variants. + */ +static inline unsigned long kvm_register_read_raw(struct kvm_vcpu *vcpu, int reg) +{ + if (WARN_ON_ONCE((unsigned int)reg >= NR_VCPU_GENERAL_PURPOSE_REGS)) + return 0; + + if (!kvm_register_is_available(vcpu, reg)) + kvm_x86_call(cache_reg)(vcpu, reg); + + return vcpu->arch.regs[reg]; +} + +static inline unsigned long kvm_register_read(struct kvm_vcpu *vcpu, int reg) +{ + return kvm_register_read_raw(vcpu, reg) & kvm_reg_mode_mask(vcpu); +} + +static inline void kvm_register_write_raw(struct kvm_vcpu *vcpu, int reg, + unsigned long val) +{ + if (WARN_ON_ONCE((unsigned int)reg >= NR_VCPU_GENERAL_PURPOSE_REGS)) + return; + + vcpu->arch.regs[reg] = val; + kvm_register_mark_dirty(vcpu, reg); +} + +static inline void kvm_register_write(struct kvm_vcpu *vcpu, + int reg, unsigned long val) +{ + return kvm_register_write_raw(vcpu, reg, val & kvm_reg_mode_mask(vcpu)); +} + +static inline unsigned long kvm_rip_read(struct kvm_vcpu *vcpu) +{ + if (!kvm_register_is_available(vcpu, VCPU_REG_RIP)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_RIP); + + return vcpu->arch.rip; +} + +static inline void kvm_rip_write(struct kvm_vcpu *vcpu, unsigned long val) +{ + vcpu->arch.rip = val; + kvm_register_mark_dirty(vcpu, VCPU_REG_RIP); +} + +static inline unsigned long kvm_rsp_read(struct kvm_vcpu *vcpu) +{ + return kvm_register_read_raw(vcpu, VCPU_REGS_RSP); +} + +static inline void kvm_rsp_write(struct kvm_vcpu *vcpu, unsigned long val) +{ + kvm_register_write_raw(vcpu, VCPU_REGS_RSP, val); +} + +static inline u64 kvm_read_edx_eax(struct kvm_vcpu *vcpu) +{ + return kvm_eax_read(vcpu) | (u64)(kvm_edx_read(vcpu)) << 32; +} + +static inline u64 kvm_pdptr_read(struct kvm_vcpu *vcpu, int index) +{ + might_sleep(); /* on svm */ + + if (!kvm_register_is_available(vcpu, VCPU_REG_PDPTR)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_PDPTR); + + return vcpu->arch.pdptrs[index]; +} + +static inline void kvm_pdptr_write(struct kvm_vcpu *vcpu, int index, u64 value) +{ + vcpu->arch.pdptrs[index] = value; +} + +static inline ulong kvm_read_cr0_bits(struct kvm_vcpu *vcpu, ulong mask) +{ + ulong tmask = mask & KVM_POSSIBLE_CR0_GUEST_BITS; + if ((tmask & vcpu->arch.cr0_guest_owned_bits) && + !kvm_register_is_available(vcpu, VCPU_REG_CR0)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_CR0); + return vcpu->arch.cr0 & mask; +} + +static __always_inline bool kvm_is_cr0_bit_set(struct kvm_vcpu *vcpu, + unsigned long cr0_bit) +{ + BUILD_BUG_ON(!is_power_of_2(cr0_bit)); + + return !!kvm_read_cr0_bits(vcpu, cr0_bit); +} + +static inline ulong kvm_read_cr0(struct kvm_vcpu *vcpu) +{ + return kvm_read_cr0_bits(vcpu, ~0UL); +} + +static inline ulong kvm_read_cr4_bits(struct kvm_vcpu *vcpu, ulong mask) +{ + ulong tmask = mask & KVM_POSSIBLE_CR4_GUEST_BITS; + if ((tmask & vcpu->arch.cr4_guest_owned_bits) && + !kvm_register_is_available(vcpu, VCPU_REG_CR4)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_CR4); + return vcpu->arch.cr4 & mask; +} + +static __always_inline bool kvm_is_cr4_bit_set(struct kvm_vcpu *vcpu, + unsigned long cr4_bit) +{ + BUILD_BUG_ON(!is_power_of_2(cr4_bit)); + + return !!kvm_read_cr4_bits(vcpu, cr4_bit); +} + +static inline ulong kvm_read_cr3(struct kvm_vcpu *vcpu) +{ + if (!kvm_register_is_available(vcpu, VCPU_REG_CR3)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_CR3); + return vcpu->arch.cr3; +} + +static inline ulong kvm_read_cr4(struct kvm_vcpu *vcpu) +{ + return kvm_read_cr4_bits(vcpu, ~0UL); +} + +static inline bool __kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +{ + return !(cr4 & vcpu->arch.cr4_guest_rsvd_bits); +} + +#define __cr4_reserved_bits(__cpu_has, __c) \ +({ \ + u64 __reserved_bits = CR4_RESERVED_BITS; \ + \ + if (!__cpu_has(__c, X86_FEATURE_XSAVE)) \ + __reserved_bits |= X86_CR4_OSXSAVE; \ + if (!__cpu_has(__c, X86_FEATURE_SMEP)) \ + __reserved_bits |= X86_CR4_SMEP; \ + if (!__cpu_has(__c, X86_FEATURE_SMAP)) \ + __reserved_bits |= X86_CR4_SMAP; \ + if (!__cpu_has(__c, X86_FEATURE_FSGSBASE)) \ + __reserved_bits |= X86_CR4_FSGSBASE; \ + if (!__cpu_has(__c, X86_FEATURE_PKU)) \ + __reserved_bits |= X86_CR4_PKE; \ + if (!__cpu_has(__c, X86_FEATURE_LA57)) \ + __reserved_bits |= X86_CR4_LA57; \ + if (!__cpu_has(__c, X86_FEATURE_UMIP)) \ + __reserved_bits |= X86_CR4_UMIP; \ + if (!__cpu_has(__c, X86_FEATURE_VMX)) \ + __reserved_bits |= X86_CR4_VMXE; \ + if (!__cpu_has(__c, X86_FEATURE_PCID)) \ + __reserved_bits |= X86_CR4_PCIDE; \ + if (!__cpu_has(__c, X86_FEATURE_LAM)) \ + __reserved_bits |= X86_CR4_LAM_SUP; \ + if (!__cpu_has(__c, X86_FEATURE_SHSTK) && \ + !__cpu_has(__c, X86_FEATURE_IBT)) \ + __reserved_bits |= X86_CR4_CET; \ + __reserved_bits; \ +}) + +static inline bool is_protmode(struct kvm_vcpu *vcpu) +{ + return kvm_is_cr0_bit_set(vcpu, X86_CR0_PE); +} + +static inline bool is_pae(struct kvm_vcpu *vcpu) +{ + return kvm_is_cr4_bit_set(vcpu, X86_CR4_PAE); +} + +static inline bool is_pse(struct kvm_vcpu *vcpu) +{ + return kvm_is_cr4_bit_set(vcpu, X86_CR4_PSE); +} + +static inline bool is_paging(struct kvm_vcpu *vcpu) +{ + return likely(kvm_is_cr0_bit_set(vcpu, X86_CR0_PG)); +} + +static inline bool is_pae_paging(struct kvm_vcpu *vcpu) +{ + return !is_long_mode(vcpu) && is_pae(vcpu) && is_paging(vcpu); +} + +static inline bool kvm_dr7_valid(u64 data) +{ + /* Bits [63:32] are reserved */ + return !(data >> 32); +} +static inline bool kvm_dr6_valid(u64 data) +{ + /* Bits [63:32] are reserved */ + return !(data >> 32); +} + +static inline void enter_guest_mode(struct kvm_vcpu *vcpu) +{ + vcpu->arch.hflags |= HF_GUEST_MASK; + vcpu->stat.guest_mode = 1; +} + +static inline void leave_guest_mode(struct kvm_vcpu *vcpu) +{ + vcpu->arch.hflags &= ~HF_GUEST_MASK; + + if (vcpu->arch.load_eoi_exitmap_pending) { + vcpu->arch.load_eoi_exitmap_pending = false; + kvm_make_request(KVM_REQ_LOAD_EOI_EXITMAP, vcpu); + } + + vcpu->stat.guest_mode = 0; +} + +static inline bool is_guest_mode(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.hflags & HF_GUEST_MASK; +} + +#endif diff --git a/arch/x86/kvm/smm.c b/arch/x86/kvm/smm.c index f623c5986119..a446487bdd5c 100644 --- a/arch/x86/kvm/smm.c +++ b/arch/x86/kvm/smm.c @@ -3,7 +3,7 @@ #include <linux/kvm_host.h> #include "x86.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "kvm_emulate.h" #include "smm.h" #include "cpuid.h" diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c index cdd5a6dc646f..0726f88e679a 100644 --- a/arch/x86/kvm/svm/avic.c +++ b/arch/x86/kvm/svm/avic.c @@ -122,38 +122,8 @@ static u32 x2avic_max_physical_id; static void avic_set_x2apic_msr_interception(struct vcpu_svm *svm, bool intercept) { - static const u32 x2avic_passthrough_msrs[] = { - X2APIC_MSR(APIC_ID), - X2APIC_MSR(APIC_LVR), - X2APIC_MSR(APIC_TASKPRI), - X2APIC_MSR(APIC_ARBPRI), - X2APIC_MSR(APIC_PROCPRI), - X2APIC_MSR(APIC_EOI), - X2APIC_MSR(APIC_RRR), - X2APIC_MSR(APIC_LDR), - X2APIC_MSR(APIC_DFR), - X2APIC_MSR(APIC_SPIV), - X2APIC_MSR(APIC_ISR), - X2APIC_MSR(APIC_TMR), - X2APIC_MSR(APIC_IRR), - X2APIC_MSR(APIC_ESR), - X2APIC_MSR(APIC_ICR), - X2APIC_MSR(APIC_ICR2), - - /* - * Note! Always intercept LVTT, as TSC-deadline timer mode - * isn't virtualized by hardware, and the CPU will generate a - * #GP instead of a #VMEXIT. - */ - X2APIC_MSR(APIC_LVTTHMR), - X2APIC_MSR(APIC_LVTPC), - X2APIC_MSR(APIC_LVT0), - X2APIC_MSR(APIC_LVT1), - X2APIC_MSR(APIC_LVTERR), - X2APIC_MSR(APIC_TMICT), - X2APIC_MSR(APIC_TMCCT), - X2APIC_MSR(APIC_TDCR), - }; + struct kvm_vcpu *vcpu = &svm->vcpu; + u64 rd_regs; int i; if (intercept == svm->x2avic_msrs_intercepted) @@ -162,9 +132,16 @@ static void avic_set_x2apic_msr_interception(struct vcpu_svm *svm, if (!x2avic_enabled) return; - for (i = 0; i < ARRAY_SIZE(x2avic_passthrough_msrs); i++) - svm_set_intercept_for_msr(&svm->vcpu, x2avic_passthrough_msrs[i], - MSR_TYPE_RW, intercept); + rd_regs = kvm_x2apic_disable_read_intercept_reg_mask(vcpu); + + for_each_set_bit(i, (unsigned long *)&rd_regs, BITS_PER_TYPE(rd_regs)) + svm_set_intercept_for_msr(vcpu, APIC_BASE_MSR + i, + MSR_TYPE_R, intercept); + + svm_set_intercept_for_msr(vcpu, X2APIC_MSR(APIC_TASKPRI), MSR_TYPE_W, intercept); + svm_set_intercept_for_msr(vcpu, X2APIC_MSR(APIC_EOI), MSR_TYPE_W, intercept); + svm_set_intercept_for_msr(vcpu, X2APIC_MSR(APIC_SELF_IPI), MSR_TYPE_W, intercept); + svm_set_intercept_for_msr(vcpu, X2APIC_MSR(APIC_ICR), MSR_TYPE_W, intercept); svm->x2avic_msrs_intercepted = intercept; } diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index b340dc9991ad..9aedb88c832d 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -30,27 +30,42 @@ #include "lapic.h" #include "svm.h" #include "hyperv.h" +#include "pmu.h" #define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, - struct x86_exception *fault) + struct x86_exception *fault, + bool from_hardware) { struct vcpu_svm *svm = to_svm(vcpu); struct vmcb *vmcb = svm->vmcb; + u64 fault_stage; - if (vmcb->control.exit_code != SVM_EXIT_NPF) { - /* - * TODO: track the cause of the nested page fault, and - * correctly fill in the high bits of exit_info_1. - */ - vmcb->control.exit_code = SVM_EXIT_NPF; - vmcb->control.exit_info_1 = (1ULL << 32); - vmcb->control.exit_info_2 = fault->address; - } + /* + * For hardware NPF exits, the GUEST_FAULT_STAGE bits are only + * available in the hardware exit_info_1, since the guest_mmu + * walker doesn't know whether the faulting GPA was a page table + * page or final page from L2's perspective. + */ + if (from_hardware) + fault_stage = vmcb->control.exit_info_1 & + PFERR_GUEST_FAULT_STAGE_MASK; + else + fault_stage = fault->error_code & PFERR_GUEST_FAULT_STAGE_MASK; + + /* + * All nested page faults should be annotated as occurring on the + * final translation *or* the page walk. Arbitrarily choose "final" + * if KVM is buggy and enumerated both or neither. + */ + if (WARN_ON_ONCE(hweight64(fault_stage) != 1)) + fault_stage = PFERR_GUEST_FINAL_MASK; - vmcb->control.exit_info_1 &= ~0xffffffffULL; - vmcb->control.exit_info_1 |= fault->error_code; + vmcb->control.exit_code = SVM_EXIT_NPF; + vmcb->control.exit_info_1 = fault_stage | + (fault->error_code & ~PFERR_GUEST_FAULT_STAGE_MASK); + vmcb->control.exit_info_2 = fault->address; nested_svm_vmexit(svm); } @@ -93,9 +108,10 @@ static void nested_svm_init_mmu_context(struct kvm_vcpu *vcpu) * when called via KVM_SET_NESTED_STATE, that state may _not_ match current * vCPU state. CR0.WP is explicitly ignored, while CR0.PG is required. */ - kvm_init_shadow_npt_mmu(vcpu, X86_CR0_PG, svm->vmcb01.ptr->save.cr4, + kvm_init_shadow_npt_mmu(vcpu, svm->vmcb01.ptr->save.cr4, svm->vmcb01.ptr->save.efer, - svm->nested.ctl.nested_cr3); + svm->nested.ctl.nested_cr3, + svm->nested.ctl.misc_ctl); vcpu->arch.mmu->get_guest_pgd = nested_svm_get_tdp_cr3; vcpu->arch.mmu->get_pdptr = nested_svm_get_tdp_pdptr; vcpu->arch.mmu->inject_page_fault = nested_svm_inject_npf_exit; @@ -420,7 +436,8 @@ static bool nested_vmcb_check_controls(struct kvm_vcpu *vcpu, /* Common checks that apply to both L1 and L2 state. */ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu, - struct vmcb_save_area_cached *save) + struct vmcb_save_area_cached *save, + bool check_gpat) { if (CC(!(save->efer & EFER_SVME))) return false; @@ -455,6 +472,15 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu, if (CC(!kvm_valid_efer(vcpu, save->efer))) return false; + /* + * If userspace contrives to get an invalid g_pat into vmcb02 by + * disabling KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT in a race with + * this check, it should be prepared for the KVM_EXIT_FAIL_ENTRY + * that will follow. + */ + if (check_gpat && CC(!kvm_pat_valid(save->g_pat))) + return false; + return true; } @@ -462,7 +488,8 @@ int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); - if (!nested_vmcb_check_save(vcpu, &svm->nested.save) || + if (!nested_vmcb_check_save(vcpu, &svm->nested.save, + l2_has_separate_pat(vcpu)) || !nested_vmcb_check_controls(vcpu, &svm->nested.ctl)) return -EINVAL; @@ -498,11 +525,14 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu, nested_svm_sanitize_intercept(vcpu, to, SKINIT); nested_svm_sanitize_intercept(vcpu, to, RDPRU); - /* Always clear SVM_MISC_ENABLE_NP if the guest cannot use NPTs */ + /* Always clear misc_ctl bits that the guest cannot use */ to->misc_ctl = from->misc_ctl; if (!guest_cpu_cap_has(vcpu, X86_FEATURE_NPT)) to->misc_ctl &= ~SVM_MISC_ENABLE_NP; + if (!gmet_enabled || !guest_cpu_cap_has(vcpu, X86_FEATURE_GMET)) + to->misc_ctl &= ~SVM_MISC_ENABLE_GMET; + to->iopm_base_pa = from->iopm_base_pa & PAGE_MASK; to->msrpm_base_pa = from->msrpm_base_pa & PAGE_MASK; to->tsc_offset = from->tsc_offset; @@ -572,6 +602,7 @@ static void __nested_copy_vmcb_save_to_cache(struct vmcb_save_area_cached *to, to->rax = from->rax; to->cr2 = from->cr2; + to->g_pat = from->g_pat; svm_copy_lbrs(to, from); } @@ -686,9 +717,12 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) return -EINVAL; - if (reload_pdptrs && !nested_npt && is_pae_paging(vcpu) && - CC(!load_pdptrs(vcpu, cr3))) - return -EINVAL; + if (reload_pdptrs && is_pae_paging(vcpu)) { + if (nested_npt) + kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR); + else if (CC(!load_pdptrs(vcpu, cr3))) + return -EINVAL; + } vcpu->arch.cr3 = cr3; @@ -701,15 +735,6 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, return 0; } -void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm) -{ - if (!svm->nested.vmcb02.ptr) - return; - - /* FIXME: merge g_pat from vmcb01 and vmcb12. */ - svm->nested.vmcb02.ptr->save.g_pat = svm->vmcb01.ptr->save.g_pat; -} - static bool nested_vmcb12_has_lbrv(struct kvm_vcpu *vcpu) { return guest_cpu_cap_has(vcpu, X86_FEATURE_LBRV) && @@ -725,9 +750,6 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm) struct vmcb *vmcb02 = svm->nested.vmcb02.ptr; struct kvm_vcpu *vcpu = &svm->vcpu; - nested_vmcb02_compute_g_pat(svm); - vmcb_mark_dirty(vmcb02, VMCB_NPT); - /* Load the nested guest state */ if (svm->nested.vmcb12_gpa != svm->nested.last_vmcb12_gpa) { new_vmcb12 = true; @@ -758,6 +780,13 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm) vmcb_mark_dirty(vmcb02, VMCB_CET); } + if (l2_has_separate_pat(vcpu)) { + if (unlikely(new_vmcb12 || vmcb12_is_dirty(control, VMCB_NPT))) + vmcb_set_gpat(vmcb02, svm->nested.save.g_pat); + } else if (npt_enabled) { + vmcb_set_gpat(vmcb02, vcpu->arch.pat); + } + kvm_set_rflags(vcpu, save->rflags | X86_EFLAGS_FIXED); svm_set_efer(vcpu, svm->nested.save.efer); @@ -767,7 +796,7 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm) svm->vcpu.arch.cr2 = save->cr2; - kvm_rax_write(vcpu, save->rax); + kvm_rax_write_raw(vcpu, save->rax); kvm_rsp_write(vcpu, save->rsp); kvm_rip_write(vcpu, save->rip); @@ -834,6 +863,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm) /* Enter Guest-Mode */ enter_guest_mode(vcpu); + svm_pmu_handle_nested_transition(svm); /* * Filled at exit: exit_code, exit_info_1, exit_info_2, exit_int_info, @@ -866,7 +896,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm) * the latter, L1 runs L2 with shadow page tables that translate L2 GVAs * to L1 GPAs, so the same NPTs can be used for L1 and L2. */ - vmcb02->control.misc_ctl = vmcb01->control.misc_ctl & SVM_MISC_ENABLE_NP; + vmcb02->control.misc_ctl = vmcb01->control.misc_ctl & (SVM_MISC_ENABLE_NP | SVM_MISC_ENABLE_GMET); vmcb02->control.iopm_base_pa = vmcb01->control.iopm_base_pa; vmcb02->control.msrpm_base_pa = vmcb01->control.msrpm_base_pa; vmcb_mark_dirty(vmcb02, VMCB_PERM_MAP); @@ -903,9 +933,13 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm) /* Also overwritten later if necessary. */ vmcb02->control.tlb_ctl = TLB_CONTROL_DO_NOTHING; - /* nested_cr3. */ - if (nested_npt_enabled(svm)) + /* Use vmcb01 MMU and format if guest does not use nNPT */ + if (nested_npt_enabled(svm)) { + vmcb02->control.misc_ctl &= ~SVM_MISC_ENABLE_GMET; + vmcb02->control.misc_ctl |= (svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_GMET); + nested_svm_init_mmu_context(vcpu); + } vcpu->arch.tsc_offset = kvm_calc_nested_tsc_offset(vcpu->arch.l1_tsc_offset, vmcb12_ctrl->tsc_offset, @@ -1104,23 +1138,29 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu) if (WARN_ON_ONCE(!svm->nested.initialized)) return -EINVAL; - vmcb12_gpa = kvm_register_read(vcpu, VCPU_REGS_RAX); + vmcb12_gpa = kvm_rax_read(vcpu); if (!page_address_valid(vcpu, vmcb12_gpa)) { kvm_inject_gp(vcpu, 0); return 1; } ret = nested_svm_copy_vmcb12_to_cache(vcpu, vmcb12_gpa); - if (ret) { - if (ret == -EFAULT) - return kvm_handle_memory_failure(vcpu, X86EMUL_IO_NEEDED, NULL); + if (ret == -EFAULT) + return kvm_handle_memory_failure(vcpu, X86EMUL_IO_NEEDED, NULL); - /* Advance RIP past VMRUN as part of the nested #VMEXIT. */ - return kvm_skip_emulated_instruction(vcpu); - } + /* + * At this point, VMRUN is guaranteed to not fault; advance RIP. If + * caching vmcb12 failed for other reasons, return immediately afterward + * as a nested #VMEXIT was already set up. + * + * FIXME: If TF is set on VMRUN should inject a #DB (or handle guest + * debugging) right after #VMEXIT, right now it's just ignored. + */ + if (!svm_skip_emulated_instruction(vcpu)) + return 0; - /* At this point, VMRUN is guaranteed to not fault; advance RIP. */ - ret = kvm_skip_emulated_instruction(vcpu); + if (ret) + goto insn_retired; /* * Since vmcb01 is not in use, we can use it to store some of the L1 @@ -1150,7 +1190,13 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu) nested_svm_vmexit(svm); } - return ret; +insn_retired: + /* + * A successful VMRUN is counted by the PMU in guest mode, so only + * retire the instruction after potentially entering guest mode. + */ + kvm_pmu_instruction_retired(vcpu); + return 1; } /* Copy state save area fields which are handled by VMRUN */ @@ -1229,11 +1275,14 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu) vmcb12->save.rflags = kvm_get_rflags(vcpu); vmcb12->save.rip = kvm_rip_read(vcpu); vmcb12->save.rsp = kvm_rsp_read(vcpu); - vmcb12->save.rax = kvm_rax_read(vcpu); + vmcb12->save.rax = kvm_rax_read_raw(vcpu); vmcb12->save.dr7 = vmcb02->save.dr7; vmcb12->save.dr6 = svm->vcpu.arch.dr6; vmcb12->save.cpl = vmcb02->save.cpl; + if (l2_has_separate_pat(vcpu)) + vmcb12->save.g_pat = vmcb02->save.g_pat; + if (guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) { vmcb12->save.s_cet = vmcb02->save.s_cet; vmcb12->save.isst_addr = vmcb02->save.isst_addr; @@ -1280,6 +1329,8 @@ void nested_svm_vmexit(struct vcpu_svm *svm) /* Exit Guest-Mode */ leave_guest_mode(vcpu); + svm_pmu_handle_nested_transition(svm); + svm->nested.vmcb12_gpa = 0; kvm_warn_on_nested_run_pending(vcpu); @@ -1376,7 +1427,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm) svm_set_efer(vcpu, vmcb01->save.efer); svm_set_cr0(vcpu, vmcb01->save.cr0 | X86_CR0_PE); svm_set_cr4(vcpu, vmcb01->save.cr4); - kvm_rax_write(vcpu, vmcb01->save.rax); + kvm_rax_write_raw(vcpu, vmcb01->save.rax); kvm_rsp_write(vcpu, vmcb01->save.rsp); kvm_rip_write(vcpu, vmcb01->save.rip); @@ -1491,6 +1542,15 @@ void svm_leave_nested(struct kvm_vcpu *vcpu) leave_guest_mode(vcpu); + /* + * Force leaving nested is a non-architectural flow so precision + * isn't a priority. Defer updating the PMU until the next vCPU + * run, potentially tolerating some imprecision to avoid poking + * into PMU state from arbitrary contexts (e.g. to avoid using + * stale state). + */ + __svm_pmu_handle_nested_transition(svm, true); + svm_switch_vmcb(svm, &svm->vmcb01); nested_svm_uninit_mmu_context(vcpu); @@ -1843,6 +1903,9 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu, /* First fill in the header and copy it out. */ if (is_guest_mode(vcpu)) { kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa; + kvm_state.hdr.svm.gpat = 0; + if (l2_has_separate_pat(vcpu)) + kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat; kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE; kvm_state.flags |= KVM_STATE_NESTED_GUEST_MODE; @@ -1895,6 +1958,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, struct vmcb_save_area *save; struct vmcb_save_area_cached save_cached; struct vmcb_ctrl_area_cached ctl_cached; + bool use_separate_l2_pat; unsigned long cr0; int ret; @@ -1959,15 +2023,29 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, /* * Validate host state saved from before VMRUN (see - * nested_svm_check_permissions). + * nested_svm_check_permissions). Note that the g_pat field is not + * validated, because (a) it may have been clobbered by SMM before + * KVM_GET_NESTED_STATE, and (b) it is not loaded at emulated + * #VMEXIT. */ __nested_copy_vmcb_save_to_cache(&save_cached, save); if (!(save->cr0 & X86_CR0_PG) || !(save->cr0 & X86_CR0_PE) || (save->rflags & X86_EFLAGS_VM) || - !nested_vmcb_check_save(vcpu, &save_cached)) + !nested_vmcb_check_save(vcpu, &save_cached, false)) goto out_free; + /* + * Validate gPAT when the shared PAT quirk is disabled (i.e. L2 + * has its own gPAT). This is done separately from the + * vmcb_save_area_cached validation above, because gPAT is L2 + * state, but the vmcb_save_area_cached is populated with L1 state. + */ + use_separate_l2_pat = (ctl_cached.misc_ctl & SVM_MISC_ENABLE_NP) && + !kvm_check_has_quirk(vcpu->kvm, + KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT); + if (use_separate_l2_pat && !kvm_pat_valid(kvm_state->hdr.svm.gpat)) + goto out_free; /* * All checks done, we can enter guest mode. Userspace provides @@ -1994,6 +2072,10 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, nested_copy_vmcb_control_to_cache(svm, ctl); svm_switch_vmcb(svm, &svm->nested.vmcb02); + + if (use_separate_l2_pat) + vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat); + nested_vmcb02_prepare_control(svm); /* @@ -2032,15 +2114,21 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu) if (WARN_ON(!is_guest_mode(vcpu))) return true; - if (!vcpu->arch.pdptrs_from_userspace && - !nested_npt_enabled(to_svm(vcpu)) && is_pae_paging(vcpu)) + if (is_pae_paging(vcpu)) { /* - * Reload the guest's PDPTRs since after a migration - * the guest CR3 might be restored prior to setting the nested - * state which can lead to a load of wrong PDPTRs. + * After migration, CR3 may have been restored before + * KVM_SET_NESTED_STATE, so the PDPTR load into mmu->pdptrs[] + * may have treated CR3 as an L1 GPA. For nNPT, drop the + * cache so the next access reloads them with the proper + * nGPA translation. For !nNPT, reload eagerly unless userspace + * already supplied authoritative PDPTRs via KVM_SET_SREGS2. */ - if (CC(!load_pdptrs(vcpu, vcpu->arch.cr3))) + if (nested_npt_enabled(to_svm(vcpu))) + kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR); + else if (!vcpu->arch.pdptrs_from_userspace && + CC(!load_pdptrs(vcpu, vcpu->arch.cr3))) return false; + } if (!nested_svm_merge_msrpm(vcpu)) { vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR; @@ -2056,8 +2144,26 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu) return true; } +static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, + u64 access, + struct x86_exception *exception, + u64 pte_access) +{ + struct vcpu_svm *svm = to_svm(vcpu); + struct kvm_mmu *mmu = vcpu->arch.mmu; + + BUG_ON(!mmu_is_nested(vcpu)); + + /* Non-GMET walks are always user-walks */ + if (!(svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_GMET)) + access |= PFERR_USER_MASK; + + return mmu->gva_to_gpa(vcpu, mmu, gpa, access, exception); +} + struct kvm_x86_nested_ops svm_nested_ops = { .leave_nested = svm_leave_nested, + .translate_nested_gpa = svm_translate_nested_gpa, .is_exception_vmexit = nested_svm_is_exception_vmexit, .check_events = svm_check_nested_events, .triple_fault = nested_svm_triple_fault, diff --git a/arch/x86/kvm/svm/pmu.c b/arch/x86/kvm/svm/pmu.c index 7aa298eeb072..c18286545a7a 100644 --- a/arch/x86/kvm/svm/pmu.c +++ b/arch/x86/kvm/svm/pmu.c @@ -168,6 +168,12 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) pmc->eventsel = data; pmc->eventsel_hw = (data & ~AMD64_EVENTSEL_HOSTONLY) | AMD64_EVENTSEL_GUESTONLY; + + if (data & AMD64_EVENTSEL_HOST_GUEST_MASK) + __set_bit(pmc->idx, pmu->pmc_has_mode_specific_enables); + else + __clear_bit(pmc->idx, pmu->pmc_has_mode_specific_enables); + kvm_pmu_request_counter_reprogram(pmc); } return 0; @@ -207,7 +213,11 @@ static void amd_pmu_refresh(struct kvm_vcpu *vcpu) } pmu->counter_bitmask[KVM_PMC_GP] = BIT_ULL(48) - 1; + pmu->reserved_bits = 0xfffffff000280000ull; + if (guest_cpu_cap_has(vcpu, X86_FEATURE_SVM) && kvm_vcpu_has_mediated_pmu(vcpu)) + pmu->reserved_bits &= ~AMD64_EVENTSEL_HOST_GUEST_MASK; + pmu->raw_event_mask = AMD64_RAW_EVENT_MASK; /* not applicable to AMD; but clean them to prevent any fall out */ pmu->counter_bitmask[KVM_PMC_FIXED] = 0; @@ -260,6 +270,37 @@ static void amd_mediated_pmu_put(struct kvm_vcpu *vcpu) wrmsrq(MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, pmu->global_status); } +static bool amd_pmc_is_disabled_in_current_mode(struct kvm_pmc *pmc) +{ + struct kvm_vcpu *vcpu = pmc->vcpu; + u64 host_guest_bits; + + if (!kvm_vcpu_has_mediated_pmu(vcpu)) + return false; + + /* Common code is supposed to check the common enable bit */ + if (WARN_ON_ONCE(!(pmc->eventsel & ARCH_PERFMON_EVENTSEL_ENABLE))) + return false; + + /* If both bits are cleared, the counter is always enabled */ + host_guest_bits = pmc->eventsel & AMD64_EVENTSEL_HOST_GUEST_MASK; + if (!host_guest_bits) + return false; + + /* If EFER.SVME=0 and either bit is set, the counter is disabled */ + if (!(vcpu->arch.efer & EFER_SVME)) + return true; + + /* + * If EFER.SVME=1, the counter is disabled iff only one of the bits is + * set AND the set bit doesn't match the vCPU mode. + */ + if (host_guest_bits == AMD64_EVENTSEL_HOST_GUEST_MASK) + return false; + + return !!(host_guest_bits & AMD64_EVENTSEL_GUESTONLY) != is_guest_mode(vcpu); +} + struct kvm_pmu_ops amd_pmu_ops __initdata = { .rdpmc_ecx_to_pmc = amd_rdpmc_ecx_to_pmc, .msr_idx_to_pmc = amd_msr_idx_to_pmc, @@ -269,6 +310,7 @@ struct kvm_pmu_ops amd_pmu_ops __initdata = { .set_msr = amd_pmu_set_msr, .refresh = amd_pmu_refresh, .init = amd_pmu_init, + .pmc_is_disabled_in_current_mode = amd_pmc_is_disabled_in_current_mode, .is_mediated_pmu_supported = amd_pmu_is_mediated_pmu_supported, .mediated_load = amd_mediated_pmu_load, diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 75e4517e8119..427229347876 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -721,14 +721,50 @@ e_free_dh: return ret; } +static int sev_check_pin_count(struct kvm *kvm, unsigned long npages) +{ + unsigned long total_npages, lock_limit; + + total_npages = to_kvm_sev_info(kvm)->pages_locked + npages; + if (total_npages > totalram_pages()) + return -EINVAL; + + lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; + if (total_npages > lock_limit && !capable(CAP_IPC_LOCK)) { + pr_err_ratelimited("SEV: %lu total pages would exceed the lock limit of %lu.\n", + total_npages, lock_limit); + return -ENOMEM; + } + + return 0; +} + +static int sev_pin_user_pages(struct kvm *kvm, unsigned long addr, int npages, + unsigned int gup_flags, struct page **pages) +{ + int npinned; + + lockdep_assert_held(&kvm->lock); + + npinned = pin_user_pages_fast(addr, npages, gup_flags, pages); + if (npinned != npages) { + if (npinned > 0) + unpin_user_pages(pages, npinned); + pr_err_ratelimited("SEV: Failure locking %u pages.\n", npages); + return -ENOMEM; + } + + to_kvm_sev_info(kvm)->pages_locked += npages; + return 0; +} + static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, unsigned long ulen, unsigned long *n, unsigned int flags) { - struct kvm_sev_info *sev = to_kvm_sev_info(kvm); - unsigned long npages, total_npages, lock_limit; + unsigned long npages; struct page **pages; - int npinned, ret; + int ret; lockdep_assert_held(&kvm->lock); @@ -744,16 +780,9 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, if (npages > INT_MAX) return ERR_PTR(-EINVAL); - total_npages = sev->pages_locked + npages; - if (total_npages > totalram_pages()) - return ERR_PTR(-EINVAL); - - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - if (total_npages > lock_limit && !capable(CAP_IPC_LOCK)) { - pr_err("SEV: %lu total pages would exceed the lock limit of %lu.\n", - total_npages, lock_limit); - return ERR_PTR(-ENOMEM); - } + ret = sev_check_pin_count(kvm, npages); + if (ret) + return ERR_PTR(ret); /* * Don't WARN if the kernel (rightly) thinks the total size is absurd, @@ -765,25 +794,14 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, if (!pages) return ERR_PTR(-ENOMEM); - /* Pin the user virtual address. */ - npinned = pin_user_pages_fast(uaddr, npages, flags, pages); - if (npinned != npages) { - pr_err("SEV: Failure locking %lu pages.\n", npages); - ret = -ENOMEM; - goto err; + ret = sev_pin_user_pages(kvm, uaddr, npages, flags, pages); + if (ret) { + kvfree(pages); + return ERR_PTR(ret); } *n = npages; - sev->pages_locked = total_npages; - return pages; - -err: - if (npinned > 0) - unpin_user_pages(pages, npinned); - - kvfree(pages); - return ERR_PTR(ret); } static void sev_unpin_memory(struct kvm *kvm, struct page **pages, @@ -794,6 +812,29 @@ static void sev_unpin_memory(struct kvm *kvm, struct page **pages, to_kvm_sev_info(kvm)->pages_locked -= npages; } +static struct page *sev_pin_page(struct kvm *kvm, unsigned long addr, + unsigned int flags) +{ + struct page *page; + int r; + + r = sev_check_pin_count(kvm, 1); + if (r) + return ERR_PTR(r); + + r = sev_pin_user_pages(kvm, addr, 1, flags, &page); + if (r) + return ERR_PTR(r); + + return page; +} + +static void sev_unpin_page(struct kvm *kvm, struct page *page) +{ + unpin_user_pages(&page, 1); + to_kvm_sev_info(kvm)->pages_locked -= 1; +} + static void sev_clflush_pages(struct page *pages[], unsigned long npages) { uint8_t *page_virtual; @@ -968,7 +1009,7 @@ static int sev_es_sync_vmsa(struct vcpu_svm *svm) save->r14 = svm->vcpu.arch.regs[VCPU_REGS_R14]; save->r15 = svm->vcpu.arch.regs[VCPU_REGS_R15]; #endif - save->rip = svm->vcpu.arch.regs[VCPU_REGS_RIP]; + save->rip = svm->vcpu.arch.rip; /* Sync some non-GPR registers before encrypting */ save->xcr0 = svm->vcpu.arch.xcr0; @@ -1197,160 +1238,115 @@ static int sev_guest_status(struct kvm *kvm, struct kvm_sev_cmd *argp) return ret; } -static int __sev_issue_dbg_cmd(struct kvm *kvm, unsigned long src, - unsigned long dst, int size, - int *error, bool enc) +static int sev_issue_dbg_cmd(struct kvm *kvm, unsigned long src_pa, + unsigned long dst_pa, unsigned int size, + unsigned int ioctl, int *error) { - struct sev_data_dbg data; - - data.reserved = 0; - data.handle = to_kvm_sev_info(kvm)->handle; - data.dst_addr = dst; - data.src_addr = src; - data.len = size; + int cmd = ioctl == KVM_SEV_DBG_DECRYPT ? SEV_CMD_DBG_DECRYPT : + SEV_CMD_DBG_ENCRYPT; + struct sev_data_dbg data = { + .handle = to_kvm_sev_info(kvm)->handle, + .dst_addr = dst_pa, + .src_addr = src_pa, + .len = size, + }; - return sev_issue_cmd(kvm, - enc ? SEV_CMD_DBG_ENCRYPT : SEV_CMD_DBG_DECRYPT, - &data, error); + return sev_issue_cmd(kvm, cmd, &data, error); } -static int __sev_dbg_decrypt(struct kvm *kvm, unsigned long src_paddr, - unsigned long dst_paddr, int sz, int *err) +static void *sev_dbg_crypt_slow_alloc(struct page *page, unsigned long __va, + unsigned int len, unsigned long *pa, + unsigned int *nr_bytes) { - int offset; + unsigned long va = ALIGN_DOWN(__va, 16); + + /* The number of bytes to {de,en}crypt must be 16-byte aligned. */ + *nr_bytes = round_up(len, 16); /* - * Its safe to read more than we are asked, caller should ensure that - * destination has enough space. + * Increase the number of bytes to {de,en}crypt by one chunk (16 bytes) + * if the aligned address and length doesn't cover the unaligned range, + * e.g. if the address is unaligned _and_ the access will split a chunk + * at the tail. */ - offset = src_paddr & 15; - src_paddr = round_down(src_paddr, 16); - sz = round_up(sz + offset, 16); + if (va + *nr_bytes < __va + len) + *nr_bytes += 16; - return __sev_issue_dbg_cmd(kvm, src_paddr, dst_paddr, sz, err, false); -} + *pa = __sme_page_pa(page) + (va & ~PAGE_MASK); -static int __sev_dbg_decrypt_user(struct kvm *kvm, unsigned long paddr, - void __user *dst_uaddr, - unsigned long dst_paddr, - int size, int *err) -{ - struct page *tpage = NULL; - int ret, offset; - - /* if inputs are not 16-byte then use intermediate buffer */ - if (!IS_ALIGNED(dst_paddr, 16) || - !IS_ALIGNED(paddr, 16) || - !IS_ALIGNED(size, 16)) { - tpage = (void *)alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); - if (!tpage) - return -ENOMEM; + /* + * Sanity check that the new access won't split a page. This should + * never happen; just pretend the allocation failed. + */ + if (WARN_ON_ONCE((*pa & PAGE_MASK) != ((*pa + *nr_bytes - 1) & PAGE_MASK))) + return NULL; - dst_paddr = __sme_page_pa(tpage); - } + return kmalloc(*nr_bytes, GFP_KERNEL); +} - ret = __sev_dbg_decrypt(kvm, paddr, dst_paddr, size, err); - if (ret) - goto e_free; +static int sev_dbg_decrypt_slow(struct kvm *kvm, unsigned long src, + struct page *src_p, unsigned long dst, + unsigned int len, int *err) +{ + unsigned int nr_bytes; + unsigned long src_pa; + void *buf; + int r; - if (tpage) { - offset = paddr & 15; - if (copy_to_user(dst_uaddr, page_address(tpage) + offset, size)) - ret = -EFAULT; - } + buf = sev_dbg_crypt_slow_alloc(src_p, src, len, &src_pa, &nr_bytes); + if (!buf) + return -ENOMEM; -e_free: - if (tpage) - __free_page(tpage); + r = sev_issue_dbg_cmd(kvm, src_pa, __sme_set(__pa(buf)), + nr_bytes, KVM_SEV_DBG_DECRYPT, err); + if (r) + goto out; - return ret; + if (copy_to_user((void __user *)dst, buf + (src & 15), len)) + r = -EFAULT; +out: + kfree(buf); + return r; } -static int __sev_dbg_encrypt_user(struct kvm *kvm, unsigned long paddr, - void __user *vaddr, - unsigned long dst_paddr, - void __user *dst_vaddr, - int size, int *error) +static int sev_dbg_encrypt_slow(struct kvm *kvm, unsigned long src, + unsigned long dst, struct page *dst_p, + unsigned int len, int *err) { - struct page *src_tpage = NULL; - struct page *dst_tpage = NULL; - int ret, len = size; - - /* If source buffer is not aligned then use an intermediate buffer */ - if (!IS_ALIGNED((unsigned long)vaddr, 16)) { - src_tpage = alloc_page(GFP_KERNEL_ACCOUNT); - if (!src_tpage) - return -ENOMEM; + unsigned int nr_bytes; + unsigned long dst_pa; + void *buf; + int r; - if (copy_from_user(page_address(src_tpage), vaddr, size)) { - __free_page(src_tpage); - return -EFAULT; - } + /* Decrypt the _destination_ to do a RMW on plaintext. */ + buf = sev_dbg_crypt_slow_alloc(dst_p, dst, len, &dst_pa, &nr_bytes); + if (!buf) + return -ENOMEM; - paddr = __sme_page_pa(src_tpage); - } + r = sev_issue_dbg_cmd(kvm, dst_pa, __sme_set(__pa(buf)), + nr_bytes, KVM_SEV_DBG_DECRYPT, err); + if (r) + goto out; /* - * If destination buffer or length is not aligned then do read-modify-write: - * - decrypt destination in an intermediate buffer - * - copy the source buffer in an intermediate buffer - * - use the intermediate buffer as source buffer + * Copy from the source into the intermediate buffer, and then + * re-encrypt the buffer into the destination. */ - if (!IS_ALIGNED((unsigned long)dst_vaddr, 16) || !IS_ALIGNED(size, 16)) { - int dst_offset; - - dst_tpage = alloc_page(GFP_KERNEL_ACCOUNT); - if (!dst_tpage) { - ret = -ENOMEM; - goto e_free; - } - - ret = __sev_dbg_decrypt(kvm, dst_paddr, - __sme_page_pa(dst_tpage), size, error); - if (ret) - goto e_free; - - /* - * If source is kernel buffer then use memcpy() otherwise - * copy_from_user(). - */ - dst_offset = dst_paddr & 15; - - if (src_tpage) - memcpy(page_address(dst_tpage) + dst_offset, - page_address(src_tpage), size); - else { - if (copy_from_user(page_address(dst_tpage) + dst_offset, - vaddr, size)) { - ret = -EFAULT; - goto e_free; - } - } - - paddr = __sme_page_pa(dst_tpage); - dst_paddr = round_down(dst_paddr, 16); - len = round_up(size, 16); - } - - ret = __sev_issue_dbg_cmd(kvm, paddr, dst_paddr, len, error, true); - -e_free: - if (src_tpage) - __free_page(src_tpage); - if (dst_tpage) - __free_page(dst_tpage); - return ret; + if (copy_from_user(buf + (dst & 15), (void __user *)src, len)) + r = -EFAULT; + else + r = sev_issue_dbg_cmd(kvm, __sme_set(__pa(buf)), dst_pa, + nr_bytes, KVM_SEV_DBG_ENCRYPT, err); +out: + kfree(buf); + return r; } -static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec) +static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, + unsigned int cmd) { - unsigned long vaddr, vaddr_end, next_vaddr; - unsigned long dst_vaddr; - struct page **src_p, **dst_p; struct kvm_sev_dbg debug; - unsigned long n; - unsigned int size; - int ret; + unsigned int i, len; if (!sev_guest(kvm)) return -ENOTTY; @@ -1358,27 +1354,38 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec) if (copy_from_user(&debug, u64_to_user_ptr(argp->data), sizeof(debug))) return -EFAULT; - if (!debug.len || debug.src_uaddr + debug.len < debug.src_uaddr) + if (!debug.len || !debug.src_uaddr || !debug.dst_uaddr) return -EINVAL; - if (!debug.dst_uaddr) + + if (debug.src_uaddr + debug.len < debug.src_uaddr || + debug.dst_uaddr + debug.len < debug.dst_uaddr) return -EINVAL; - vaddr = debug.src_uaddr; - size = debug.len; - vaddr_end = vaddr + size; - dst_vaddr = debug.dst_uaddr; + for (i = 0; i < debug.len; i += len) { + unsigned long src = debug.src_uaddr + i; + unsigned long dst = debug.dst_uaddr + i; + unsigned long s_off = src & ~PAGE_MASK; + unsigned long d_off = dst & ~PAGE_MASK; + struct page *src_p, *dst_p; + int ret; - for (; vaddr < vaddr_end; vaddr = next_vaddr) { - int len, s_off, d_off; + /* + * Copy as many remaining bytes as possible while staying in a + * single page for both the source and destination. + */ + len = min3(debug.len - i, PAGE_SIZE - s_off, PAGE_SIZE - d_off); - /* lock userspace source and destination page */ - src_p = sev_pin_memory(kvm, vaddr & PAGE_MASK, PAGE_SIZE, &n, 0); + /* + * Pin the source and destination pages; firmware operates on + * physical addresses. + */ + src_p = sev_pin_page(kvm, src & PAGE_MASK, 0); if (IS_ERR(src_p)) return PTR_ERR(src_p); - dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, FOLL_WRITE); + dst_p = sev_pin_page(kvm, dst & PAGE_MASK, FOLL_WRITE); if (IS_ERR(dst_p)) { - sev_unpin_memory(kvm, src_p, n); + sev_unpin_page(kvm, src_p); return PTR_ERR(dst_p); } @@ -1387,43 +1394,28 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec) * the pages; flush the destination too so that future accesses do not * see stale data. */ - sev_clflush_pages(src_p, 1); - sev_clflush_pages(dst_p, 1); - - /* - * Since user buffer may not be page aligned, calculate the - * offset within the page. - */ - s_off = vaddr & ~PAGE_MASK; - d_off = dst_vaddr & ~PAGE_MASK; - len = min_t(size_t, (PAGE_SIZE - s_off), size); - - if (dec) - ret = __sev_dbg_decrypt_user(kvm, - __sme_page_pa(src_p[0]) + s_off, - (void __user *)dst_vaddr, - __sme_page_pa(dst_p[0]) + d_off, - len, &argp->error); + sev_clflush_pages(&src_p, 1); + sev_clflush_pages(&dst_p, 1); + + if (IS_ALIGNED(src, 16) && IS_ALIGNED(dst, 16) && IS_ALIGNED(len, 16)) + ret = sev_issue_dbg_cmd(kvm, + __sme_page_pa(src_p) + s_off, + __sme_page_pa(dst_p) + d_off, + len, cmd, &argp->error); + else if (cmd == KVM_SEV_DBG_DECRYPT) + ret = sev_dbg_decrypt_slow(kvm, src, src_p, dst, + len, &argp->error); else - ret = __sev_dbg_encrypt_user(kvm, - __sme_page_pa(src_p[0]) + s_off, - (void __user *)vaddr, - __sme_page_pa(dst_p[0]) + d_off, - (void __user *)dst_vaddr, - len, &argp->error); + ret = sev_dbg_encrypt_slow(kvm, src, dst, dst_p, + len, &argp->error); - sev_unpin_memory(kvm, src_p, n); - sev_unpin_memory(kvm, dst_p, n); + sev_unpin_page(kvm, src_p); + sev_unpin_page(kvm, dst_p); if (ret) - goto err; - - next_vaddr = vaddr + len; - dst_vaddr = dst_vaddr + len; - size -= len; + return ret; } -err: - return ret; + return 0; } static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) @@ -1696,8 +1688,7 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) struct sev_data_send_update_data data; struct kvm_sev_send_update_data params; void *hdr, *trans_data; - struct page **guest_page; - unsigned long n; + struct page *guest_page; int ret, offset; if (!sev_guest(kvm)) @@ -1721,8 +1712,7 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) return -EINVAL; /* Pin guest memory */ - guest_page = sev_pin_memory(kvm, params.guest_uaddr & PAGE_MASK, - PAGE_SIZE, &n, 0); + guest_page = sev_pin_page(kvm, params.guest_uaddr & PAGE_MASK, 0); if (IS_ERR(guest_page)) return PTR_ERR(guest_page); @@ -1743,7 +1733,7 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) data.trans_len = params.trans_len; /* The SEND_UPDATE_DATA command requires C-bit to be always set. */ - data.guest_address = (page_to_pfn(guest_page[0]) << PAGE_SHIFT) + offset; + data.guest_address = page_to_phys(guest_page) + offset; data.guest_address |= sev_me_mask; data.guest_len = params.guest_len; data.handle = to_kvm_sev_info(kvm)->handle; @@ -1770,8 +1760,7 @@ e_free_trans_data: e_free_hdr: kfree(hdr); e_unpin: - sev_unpin_memory(kvm, guest_page, n); - + sev_unpin_page(kvm, guest_page); return ret; } @@ -1876,8 +1865,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) struct kvm_sev_receive_update_data params; struct sev_data_receive_update_data data; void *hdr = NULL, *trans = NULL; - struct page **guest_page; - unsigned long n; + struct page *guest_page; int ret, offset; if (!sev_guest(kvm)) @@ -1914,8 +1902,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) data.trans_len = params.trans_len; /* Pin guest memory */ - guest_page = sev_pin_memory(kvm, params.guest_uaddr & PAGE_MASK, - PAGE_SIZE, &n, FOLL_WRITE); + guest_page = sev_pin_page(kvm, params.guest_uaddr & PAGE_MASK, FOLL_WRITE); if (IS_ERR(guest_page)) { ret = PTR_ERR(guest_page); goto e_free_trans; @@ -1926,10 +1913,10 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) * encrypts the written data with the guest's key, and the cache may * contain dirty, unencrypted data. */ - sev_clflush_pages(guest_page, n); + sev_clflush_pages(&guest_page, 1); /* The RECEIVE_UPDATE_DATA command requires C-bit to be always set. */ - data.guest_address = (page_to_pfn(guest_page[0]) << PAGE_SHIFT) + offset; + data.guest_address = page_to_phys(guest_page) + offset; data.guest_address |= sev_me_mask; data.guest_len = params.guest_len; data.handle = to_kvm_sev_info(kvm)->handle; @@ -1937,7 +1924,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) ret = sev_issue_cmd(kvm, SEV_CMD_RECEIVE_UPDATE_DATA, &data, &argp->error); - sev_unpin_memory(kvm, guest_page, n); + sev_unpin_page(kvm, guest_page); e_free_trans: kfree(trans); @@ -2361,8 +2348,8 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, memcpy(dst_vaddr, src_vaddr, PAGE_SIZE); - kunmap_local(src_vaddr); kunmap_local(dst_vaddr); + kunmap_local(src_vaddr); } ret = rmp_make_private(pfn, gfn << PAGE_SHIFT, PG_LEVEL_4K, @@ -2396,9 +2383,10 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, void *dst_vaddr = kmap_local_pfn(pfn); memcpy(src_vaddr, dst_vaddr, PAGE_SIZE); + set_page_dirty(src_page); - kunmap_local(src_vaddr); kunmap_local(dst_vaddr); + kunmap_local(src_vaddr); } out: @@ -2470,6 +2458,7 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp) sev_populate_args.type = params.type; count = kvm_gmem_populate(kvm, params.gfn_start, src, npages, + params.type == KVM_SEV_SNP_PAGE_TYPE_CPUID, sev_gmem_post_populate, &sev_populate_args); if (count < 0) { argp->error = sev_populate_args.fw_error; @@ -2690,10 +2679,8 @@ int sev_mem_enc_ioctl(struct kvm *kvm, void __user *argp) r = sev_guest_status(kvm, &sev_cmd); break; case KVM_SEV_DBG_DECRYPT: - r = sev_dbg_crypt(kvm, &sev_cmd, true); - break; case KVM_SEV_DBG_ENCRYPT: - r = sev_dbg_crypt(kvm, &sev_cmd, false); + r = sev_dbg_crypt(kvm, &sev_cmd, sev_cmd.id); break; case KVM_SEV_LAUNCH_SECRET: r = sev_launch_secret(kvm, &sev_cmd); @@ -3014,18 +3001,14 @@ void sev_vm_destroy(struct kvm *kvm) void __init sev_set_cpu_caps(void) { - if (sev_enabled) { + if (sev_enabled) kvm_cpu_cap_set(X86_FEATURE_SEV); - kvm_caps.supported_vm_types |= BIT(KVM_X86_SEV_VM); - } - if (sev_es_enabled) { + + if (sev_es_enabled) kvm_cpu_cap_set(X86_FEATURE_SEV_ES); - kvm_caps.supported_vm_types |= BIT(KVM_X86_SEV_ES_VM); - } - if (sev_snp_enabled) { + + if (sev_snp_enabled) kvm_cpu_cap_set(X86_FEATURE_SEV_SNP); - kvm_caps.supported_vm_types |= BIT(KVM_X86_SNP_VM); - } } static bool is_sev_snp_initialized(void) @@ -3055,6 +3038,11 @@ out: return initialized; } +static const char * __init sev_str_feature_state(bool is_supported, bool is_usable) +{ + return is_supported ? is_usable ? "enabled" : "unusable" : "disabled"; +} + void __init sev_hardware_setup(void) { unsigned int eax, ebx, ecx, edx, sev_asid_count, sev_es_asid_count; @@ -3062,6 +3050,7 @@ void __init sev_hardware_setup(void) bool sev_snp_supported = false; bool sev_es_supported = false; bool sev_supported = false; + u32 vm_types = 0; if (!sev_enabled || !npt_enabled || !nrips) goto out; @@ -3195,21 +3184,27 @@ out: } } + if (sev_supported && min_sev_asid <= max_sev_asid) + vm_types |= BIT(KVM_X86_SEV_VM); + if (sev_es_supported && min_sev_es_asid <= max_sev_es_asid) + vm_types |= BIT(KVM_X86_SEV_ES_VM); + if (sev_snp_supported) + vm_types |= BIT(KVM_X86_SNP_VM); + vm_types &= sev_firmware_supported_vm_types(); + + kvm_caps.supported_vm_types |= vm_types; + if (boot_cpu_has(X86_FEATURE_SEV)) pr_info("SEV %s (ASIDs %u - %u)\n", - sev_supported ? min_sev_asid <= max_sev_asid ? "enabled" : - "unusable" : - "disabled", + sev_str_feature_state(sev_supported, vm_types & BIT(KVM_X86_SEV_VM)), min_sev_asid, max_sev_asid); if (boot_cpu_has(X86_FEATURE_SEV_ES)) pr_info("SEV-ES %s (ASIDs %u - %u)\n", - sev_es_supported ? min_sev_es_asid <= max_sev_es_asid ? "enabled" : - "unusable" : - "disabled", + sev_str_feature_state(sev_es_supported, vm_types & BIT(KVM_X86_SEV_ES_VM)), min_sev_es_asid, max_sev_es_asid); if (boot_cpu_has(X86_FEATURE_SEV_SNP)) pr_info("SEV-SNP %s (ASIDs %u - %u)\n", - str_enabled_disabled(sev_snp_supported), + sev_str_feature_state(sev_snp_supported, vm_types & BIT(KVM_X86_SNP_VM)), min_snp_asid, max_snp_asid); sev_enabled = sev_supported; @@ -3411,146 +3406,61 @@ static void sev_es_sync_from_ghcb(struct vcpu_svm *svm) memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap)); } -static int sev_es_validate_vmgexit(struct vcpu_svm *svm) +static bool sev_es_are_required_ghcb_fields_valid(struct vcpu_svm *svm) { struct vmcb_control_area *control = &svm->vmcb->control; struct kvm_vcpu *vcpu = &svm->vcpu; - u64 reason; - - /* Only GHCB Usage code 0 is supported */ - if (svm->sev_es.ghcb->ghcb_usage) { - reason = GHCB_ERR_INVALID_USAGE; - goto vmgexit_err; - } - - reason = GHCB_ERR_MISSING_INPUT; if (!kvm_ghcb_sw_exit_code_is_valid(svm) || !kvm_ghcb_sw_exit_info_1_is_valid(svm) || !kvm_ghcb_sw_exit_info_2_is_valid(svm)) - goto vmgexit_err; + return false; switch (control->exit_code) { - case SVM_EXIT_READ_DR7: - break; case SVM_EXIT_WRITE_DR7: - if (!kvm_ghcb_rax_is_valid(svm)) - goto vmgexit_err; - break; - case SVM_EXIT_RDTSC: - break; + return kvm_ghcb_rax_is_valid(svm); case SVM_EXIT_RDPMC: - if (!kvm_ghcb_rcx_is_valid(svm)) - goto vmgexit_err; - break; + return kvm_ghcb_rcx_is_valid(svm); case SVM_EXIT_CPUID: if (!kvm_ghcb_rax_is_valid(svm) || !kvm_ghcb_rcx_is_valid(svm)) - goto vmgexit_err; - if (vcpu->arch.regs[VCPU_REGS_RAX] == 0xd) - if (!kvm_ghcb_xcr0_is_valid(svm)) - goto vmgexit_err; - break; - case SVM_EXIT_INVD: - break; + return false; + + return vcpu->arch.regs[VCPU_REGS_RAX] != 0xd || + kvm_ghcb_xcr0_is_valid(svm); case SVM_EXIT_IOIO: - if (control->exit_info_1 & SVM_IOIO_STR_MASK) { - if (!kvm_ghcb_sw_scratch_is_valid(svm)) - goto vmgexit_err; - } else { - if (!(control->exit_info_1 & SVM_IOIO_TYPE_MASK)) - if (!kvm_ghcb_rax_is_valid(svm)) - goto vmgexit_err; - } - break; + if (control->exit_info_1 & SVM_IOIO_STR_MASK) + return kvm_ghcb_sw_scratch_is_valid(svm); + + if (!(control->exit_info_1 & SVM_IOIO_TYPE_MASK)) + return kvm_ghcb_rax_is_valid(svm); + + return true; case SVM_EXIT_MSR: if (!kvm_ghcb_rcx_is_valid(svm)) - goto vmgexit_err; - if (control->exit_info_1) { - if (!kvm_ghcb_rax_is_valid(svm) || - !kvm_ghcb_rdx_is_valid(svm)) - goto vmgexit_err; - } - break; + return false; + + return !control->exit_info_1 || + (kvm_ghcb_rax_is_valid(svm) && kvm_ghcb_rdx_is_valid(svm)); case SVM_EXIT_VMMCALL: - if (!kvm_ghcb_rax_is_valid(svm) || - !kvm_ghcb_cpl_is_valid(svm)) - goto vmgexit_err; - break; - case SVM_EXIT_RDTSCP: - break; - case SVM_EXIT_WBINVD: - break; + return kvm_ghcb_rax_is_valid(svm) && kvm_ghcb_cpl_is_valid(svm); case SVM_EXIT_MONITOR: - if (!kvm_ghcb_rax_is_valid(svm) || - !kvm_ghcb_rcx_is_valid(svm) || - !kvm_ghcb_rdx_is_valid(svm)) - goto vmgexit_err; - break; + return kvm_ghcb_rax_is_valid(svm) && + kvm_ghcb_rcx_is_valid(svm) && + kvm_ghcb_rdx_is_valid(svm); case SVM_EXIT_MWAIT: - if (!kvm_ghcb_rax_is_valid(svm) || - !kvm_ghcb_rcx_is_valid(svm)) - goto vmgexit_err; + return kvm_ghcb_rax_is_valid(svm) && kvm_ghcb_rcx_is_valid(svm); + case SVM_VMGEXIT_AP_CREATION: + return kvm_ghcb_rax_is_valid(svm) || + lower_32_bits(control->exit_info_1) == SVM_VMGEXIT_AP_DESTROY; break; case SVM_VMGEXIT_MMIO_READ: case SVM_VMGEXIT_MMIO_WRITE: - if (!kvm_ghcb_sw_scratch_is_valid(svm)) - goto vmgexit_err; - break; - case SVM_VMGEXIT_AP_CREATION: - if (!is_sev_snp_guest(vcpu)) - goto vmgexit_err; - if (lower_32_bits(control->exit_info_1) != SVM_VMGEXIT_AP_DESTROY) - if (!kvm_ghcb_rax_is_valid(svm)) - goto vmgexit_err; - break; - case SVM_VMGEXIT_NMI_COMPLETE: - case SVM_VMGEXIT_AP_HLT_LOOP: - case SVM_VMGEXIT_AP_JUMP_TABLE: - case SVM_VMGEXIT_UNSUPPORTED_EVENT: - case SVM_VMGEXIT_HV_FEATURES: - case SVM_VMGEXIT_TERM_REQUEST: - break; case SVM_VMGEXIT_PSC: - if (!is_sev_snp_guest(vcpu) || !kvm_ghcb_sw_scratch_is_valid(svm)) - goto vmgexit_err; - break; - case SVM_VMGEXIT_GUEST_REQUEST: - case SVM_VMGEXIT_EXT_GUEST_REQUEST: - if (!is_sev_snp_guest(vcpu) || - !PAGE_ALIGNED(control->exit_info_1) || - !PAGE_ALIGNED(control->exit_info_2) || - control->exit_info_1 == control->exit_info_2) - goto vmgexit_err; - break; + return kvm_ghcb_sw_scratch_is_valid(svm); default: - reason = GHCB_ERR_INVALID_EVENT; - goto vmgexit_err; - } - - return 0; - -vmgexit_err: - /* - * Print the exit code even though it may not be marked valid as it - * could help with debugging. - */ - if (reason == GHCB_ERR_INVALID_USAGE) { - vcpu_unimpl(vcpu, "vmgexit: ghcb usage %#x is not valid\n", - svm->sev_es.ghcb->ghcb_usage); - } else if (reason == GHCB_ERR_INVALID_EVENT) { - vcpu_unimpl(vcpu, "vmgexit: exit code %#llx is not valid\n", - control->exit_code); - } else { - vcpu_unimpl(vcpu, "vmgexit: exit code %#llx input is not valid\n", - control->exit_code); - dump_ghcb(svm); + return true; } - - svm_vmgexit_bad_input(svm, reason); - - /* Resume the guest to "return" the error code. */ - return 1; } static void __sev_es_unmap_ghcb(struct vcpu_svm *svm) @@ -3797,9 +3707,13 @@ static int snp_rmptable_psmash(kvm_pfn_t pfn) static int snp_complete_psc_msr(struct kvm_vcpu *vcpu) { + u64 hypercall_ret = READ_ONCE(vcpu->run->hypercall.ret); struct vcpu_svm *svm = to_svm(vcpu); - if (vcpu->run->hypercall.ret) + if (!kvm_is_valid_map_gpa_range_ret(hypercall_ret)) + return -EINVAL; + + if (hypercall_ret) set_ghcb_msr(svm, GHCB_MSR_PSC_RESP_ERROR); else set_ghcb_msr(svm, GHCB_MSR_PSC_RESP); @@ -3888,9 +3802,13 @@ static void __snp_complete_one_psc(struct vcpu_svm *svm) static int snp_complete_one_psc(struct kvm_vcpu *vcpu) { + u64 hypercall_ret = READ_ONCE(vcpu->run->hypercall.ret); struct vcpu_svm *svm = to_svm(vcpu); - if (vcpu->run->hypercall.ret) { + if (!kvm_is_valid_map_gpa_range_ret(hypercall_ret)) + return -EINVAL; + + if (hypercall_ret) { snp_complete_psc(svm, VMGEXIT_PSC_ERROR_GENERIC); return 1; /* resume guest */ } @@ -4486,12 +4404,24 @@ out_terminate: return 0; } +static bool is_snp_only_vmgexit(u64 exit_code) +{ + switch (exit_code) { + case SVM_VMGEXIT_AP_CREATION: + case SVM_VMGEXIT_GUEST_REQUEST: + case SVM_VMGEXIT_EXT_GUEST_REQUEST: + case SVM_VMGEXIT_PSC: + return true; + default: + return false; + } +} + int sev_handle_vmgexit(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); struct vmcb_control_area *control = &svm->vmcb->control; u64 ghcb_gpa; - int ret; /* Validate the GHCB */ ghcb_gpa = control->ghcb_gpa; @@ -4521,22 +4451,67 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu) sev_es_sync_from_ghcb(svm); /* SEV-SNP guest requires that the GHCB GPA must be registered */ - if (is_sev_snp_guest(vcpu) && !ghcb_gpa_is_registered(svm, ghcb_gpa)) { - vcpu_unimpl(&svm->vcpu, "vmgexit: GHCB GPA [%#llx] is not registered.\n", ghcb_gpa); - return -EINVAL; + if (is_sev_snp_guest(vcpu) && + !ghcb_gpa_is_registered(svm, control->ghcb_gpa)) { + vcpu_unimpl(vcpu, "vmgexit: GHCB GPA [%#llx] is not registered.\n", + control->ghcb_gpa); + svm_vmgexit_bad_input(svm, GHCB_ERR_NOT_REGISTERED); + return 1; } - ret = sev_es_validate_vmgexit(svm); - if (ret) - return ret; + /* Only GHCB Usage code 0 is supported */ + if (svm->sev_es.ghcb->ghcb_usage) { + vcpu_unimpl(vcpu, "vmgexit: ghcb usage %#x is not valid\n", + svm->sev_es.ghcb->ghcb_usage); + svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_USAGE); + return 1; + } + + if (is_snp_only_vmgexit(control->exit_code) && !is_sev_snp_guest(vcpu)) { + vcpu_unimpl(vcpu, "vmgexit: exit code %#llx is SNP-only\n", + control->exit_code); + svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_EVENT); + return 1; + } + + if (!sev_es_are_required_ghcb_fields_valid(svm)) { + /* + * Print the exit code even though it may not be marked valid + * as it could help with debugging. + */ + vcpu_unimpl(vcpu, "vmgexit: exit code %#llx input is not valid\n", + control->exit_code); + dump_ghcb(svm); + svm_vmgexit_bad_input(svm, GHCB_ERR_MISSING_INPUT); + return 1; + } svm_vmgexit_success(svm, 0); switch (control->exit_code) { + case SVM_EXIT_IOIO: + if (!((control->exit_info_1 & SVM_IOIO_SIZE_MASK) >> SVM_IOIO_SIZE_SHIFT)) + return 1; + + fallthrough; + case SVM_EXIT_READ_DR7: + case SVM_EXIT_WRITE_DR7: + case SVM_EXIT_RDTSC: + case SVM_EXIT_RDTSCP: + case SVM_EXIT_RDPMC: + case SVM_EXIT_CPUID: + case SVM_EXIT_INVD: + case SVM_EXIT_MSR: + case SVM_EXIT_VMMCALL: + case SVM_EXIT_WBINVD: + case SVM_EXIT_MONITOR: + case SVM_EXIT_MWAIT: + return svm_invoke_exit_handler(vcpu, control->exit_code); case SVM_VMGEXIT_MMIO_READ: case SVM_VMGEXIT_MMIO_WRITE: { bool is_write = control->exit_code == SVM_VMGEXIT_MMIO_WRITE; u64 len = control->exit_info_2; + int r; if (!len) return 1; @@ -4546,24 +4521,21 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu) return 1; } - ret = setup_vmgexit_scratch(svm, !is_write, len); - if (ret) - break; + r = setup_vmgexit_scratch(svm, !is_write, len); + if (r) + return r; - ret = kvm_sev_es_mmio(vcpu, is_write, control->exit_info_1, len, - svm->sev_es.ghcb_sa); - break; + return kvm_sev_es_mmio(vcpu, is_write, control->exit_info_1, len, + svm->sev_es.ghcb_sa); } case SVM_VMGEXIT_NMI_COMPLETE: ++vcpu->stat.nmi_window_exits; svm->nmi_masked = false; kvm_make_request(KVM_REQ_EVENT, vcpu); - ret = 1; - break; + return 1; case SVM_VMGEXIT_AP_HLT_LOOP: svm->sev_es.ap_reset_hold_type = AP_RESET_HOLD_NAE_EVENT; - ret = kvm_emulate_ap_reset_hold(vcpu); - break; + return kvm_emulate_ap_reset_hold(vcpu); case SVM_VMGEXIT_AP_JUMP_TABLE: { struct kvm_sev_info *sev = to_kvm_sev_info(vcpu->kvm); @@ -4581,14 +4553,11 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu) control->exit_info_1); svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_INPUT); } - - ret = 1; - break; + return 1; } case SVM_VMGEXIT_HV_FEATURES: svm_vmgexit_success(svm, GHCB_HV_FT_SUPPORTED); - ret = 1; - break; + return 1; case SVM_VMGEXIT_TERM_REQUEST: pr_info("SEV-ES guest requested termination: reason %#llx info %#llx\n", control->exit_info_1, control->exit_info_2); @@ -4596,44 +4565,53 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu) vcpu->run->system_event.type = KVM_SYSTEM_EVENT_SEV_TERM; vcpu->run->system_event.ndata = 1; vcpu->run->system_event.data[0] = control->ghcb_gpa; - break; - case SVM_VMGEXIT_PSC: - ret = setup_vmgexit_scratch(svm, true, sizeof(struct psc_hdr)); - if (ret) - break; + return 0; + case SVM_VMGEXIT_PSC: { + int r; - ret = snp_begin_psc(svm); - break; + r = setup_vmgexit_scratch(svm, true, sizeof(struct psc_hdr)); + if (r) + return r; + + return snp_begin_psc(svm); + } case SVM_VMGEXIT_AP_CREATION: - ret = sev_snp_ap_creation(svm); - if (ret) { + if (sev_snp_ap_creation(svm)) svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_INPUT); - } - - ret = 1; - break; + return 1; case SVM_VMGEXIT_GUEST_REQUEST: - ret = snp_handle_guest_req(svm, control->exit_info_1, control->exit_info_2); - break; case SVM_VMGEXIT_EXT_GUEST_REQUEST: - ret = snp_handle_ext_guest_req(svm, control->exit_info_1, control->exit_info_2); - break; + if (!PAGE_ALIGNED(control->exit_info_1) || + !PAGE_ALIGNED(control->exit_info_2) || + control->exit_info_1 == control->exit_info_2) { + svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_INPUT); + return 1; + } + + if (control->exit_code == SVM_VMGEXIT_GUEST_REQUEST) + return snp_handle_guest_req(svm, control->exit_info_1, + control->exit_info_2); + + return snp_handle_ext_guest_req(svm, control->exit_info_1, + control->exit_info_2); case SVM_VMGEXIT_UNSUPPORTED_EVENT: + /* + * Note, the _guest_ is reporting an unsupported #VC, i.e. this + * isn't the same thing as KVM getting an unsupported #VMGEXIT. + */ vcpu_unimpl(vcpu, "vmgexit: unsupported event - exit_info_1=%#llx, exit_info_2=%#llx\n", control->exit_info_1, control->exit_info_2); - ret = -EINVAL; - break; - case SVM_EXIT_IOIO: - if (!((control->exit_info_1 & SVM_IOIO_SIZE_MASK) >> SVM_IOIO_SIZE_SHIFT)) - return 1; - - fallthrough; + return -EINVAL; default: - ret = svm_invoke_exit_handler(vcpu, control->exit_code); + vcpu_unimpl(vcpu, "vmgexit: exit code %#llx is not valid\n", + control->exit_code); + svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_EVENT); + return 1; } - return ret; + KVM_BUG_ON(1, vcpu->kvm); + return -EIO; } int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index d38a21be099d..9658ce4e0294 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4,7 +4,7 @@ #include "irq.h" #include "mmu.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "x86.h" #include "smm.h" #include "cpuid.h" @@ -51,6 +51,7 @@ #include "trace.h" +#include "vmenter.h" #include "svm.h" #include "svm_ops.h" @@ -139,6 +140,9 @@ module_param(pause_filter_count_max, ushort, 0444); bool __ro_after_init npt_enabled = true; module_param_named(npt, npt_enabled, bool, 0444); +bool gmet_enabled = true; +module_param_named(gmet, gmet_enabled, bool, 0444); + /* allow nested virtualization in KVM/SVM */ static int __ro_after_init nested = true; module_param(nested, int, 0444); @@ -262,6 +266,7 @@ int svm_set_efer(struct kvm_vcpu *vcpu, u64 efer) set_exception_intercept(svm, GP_VECTOR); } + svm_pmu_handle_nested_transition(svm); kvm_make_request(KVM_REQ_RECALC_INTERCEPTS, vcpu); } @@ -330,7 +335,7 @@ done: return 1; } -static int svm_skip_emulated_instruction(struct kvm_vcpu *vcpu) +int svm_skip_emulated_instruction(struct kvm_vcpu *vcpu) { return __svm_skip_emulated_instruction(vcpu, EMULTYPE_SKIP, true); } @@ -662,7 +667,7 @@ static void clr_dr_intercepts(struct vcpu_svm *svm) svm_mark_intercepts_dirty(svm); } -static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) +static bool msr_write_intercepted(struct vcpu_svm *svm, u32 msr) { /* * For non-nested case: @@ -673,8 +678,7 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) * If the L02 MSR bitmap does not intercept the MSR, then we need to * save it. */ - void *msrpm = is_guest_mode(vcpu) ? to_svm(vcpu)->nested.msrpm : - to_svm(vcpu)->msrpm; + void *msrpm = is_guest_mode(&svm->vcpu) ? svm->nested.msrpm : svm->msrpm; return svm_test_msr_bitmap_write(msrpm, msr); } @@ -1221,6 +1225,10 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event) save->g_pat = vcpu->arch.pat; save->cr3 = 0; } + + if (gmet_enabled) + control->misc_ctl |= SVM_MISC_ENABLE_GMET; + svm->current_vmcb->asid_generation = 0; svm->asid = 0; @@ -1529,9 +1537,9 @@ static void svm_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) kvm_register_mark_available(vcpu, reg); switch (reg) { - case VCPU_EXREG_PDPTR: + case VCPU_REG_PDPTR: /* - * When !npt_enabled, mmu->pdptrs[] is already available since + * When !npt_enabled, vcpu->pdptrs[] is already available since * it is always updated per SDM when moving to CRs. */ if (npt_enabled) @@ -1998,6 +2006,18 @@ static int npf_interception(struct kvm_vcpu *vcpu) } } + if (!is_sev_es_guest(vcpu) && + (svm->vmcb->control.misc_ctl & SVM_MISC_ENABLE_GMET) && + (error_code & PFERR_FETCH_MASK)) { + /* + * Work around errata 1218: EXITINFO1[2] May Be Incorrectly Set + * When GMET (Guest Mode Execute Trap extension) is Enabled + */ + error_code |= PFERR_USER_MASK; + if (svm_get_cpl(vcpu) != 3) + error_code &= ~PFERR_USER_MASK; + } + if (is_sev_snp_guest(vcpu) && (error_code & PFERR_GUEST_ENC_MASK)) error_code |= PFERR_PRIVATE_ACCESS; @@ -2199,7 +2219,7 @@ static int intr_interception(struct kvm_vcpu *vcpu) static int vmload_vmsave_interception(struct kvm_vcpu *vcpu, bool vmload) { - u64 vmcb12_gpa = kvm_register_read(vcpu, VCPU_REGS_RAX); + u64 vmcb12_gpa = kvm_rax_read(vcpu); struct vcpu_svm *svm = to_svm(vcpu); struct vmcb *vmcb12; struct kvm_host_map map; @@ -2307,7 +2327,7 @@ static int gp_interception(struct kvm_vcpu *vcpu) if (nested_svm_check_permissions(vcpu)) return 1; - if (!page_address_valid(vcpu, kvm_register_read(vcpu, VCPU_REGS_RAX))) + if (!page_address_valid(vcpu, kvm_rax_read(vcpu))) goto reinject; /* @@ -2390,16 +2410,13 @@ static int clgi_interception(struct kvm_vcpu *vcpu) static int invlpga_interception(struct kvm_vcpu *vcpu) { + /* FIXME: Handle an address size prefix. */ gva_t gva = kvm_rax_read(vcpu); - u32 asid = kvm_rcx_read(vcpu); + u32 asid = kvm_ecx_read(vcpu); if (nested_svm_check_permissions(vcpu)) return 1; - /* FIXME: Handle an address size prefix. */ - if (!is_long_mode(vcpu)) - gva = (u32)gva; - trace_kvm_invlpga(to_svm(vcpu)->vmcb->save.rip, asid, gva); /* Let's treat INVLPGA the same as INVLPG (can be optimized!) */ @@ -2776,7 +2793,21 @@ static bool sev_es_prevent_msr_access(struct kvm_vcpu *vcpu, { return is_sev_es_guest(vcpu) && vcpu->arch.guest_state_protected && msr_info->index != MSR_IA32_XSS && - !msr_write_intercepted(vcpu, msr_info->index); + !msr_write_intercepted(to_svm(vcpu), msr_info->index); +} + +static bool svm_pat_accesses_gpat(struct kvm_vcpu *vcpu, bool from_host) +{ + /* + * When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and nested + * NPT is enabled, L2 has a separate PAT from L1. Guest accesses + * to IA32_PAT while running L2 target L2's gPAT; host-initiated + * accesses always target L1's hPAT so that KVM_GET/SET_MSRS and + * KVM_GET/SET_NESTED_STATE are independent of each other and can + * be ordered arbitrarily during save and restore. + */ + WARN_ON_ONCE(from_host && vcpu->wants_to_run); + return !from_host && is_guest_mode(vcpu) && l2_has_separate_pat(vcpu); } static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) @@ -2895,6 +2926,12 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) case MSR_AMD64_DE_CFG: msr_info->data = svm->msr_decfg; break; + case MSR_IA32_CR_PAT: + if (svm_pat_accesses_gpat(vcpu, msr_info->host_initiated)) { + msr_info->data = svm->vmcb->save.g_pat; + break; + } + return kvm_get_msr_common(vcpu, msr_info); default: return kvm_get_msr_common(vcpu, msr_info); } @@ -2978,14 +3015,23 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) break; case MSR_IA32_CR_PAT: + if (svm_pat_accesses_gpat(vcpu, msr->host_initiated)) { + if (!kvm_pat_valid(data)) + return 1; + + vmcb_set_gpat(svm->vmcb, data); + break; + } + ret = kvm_set_msr_common(vcpu, msr); if (ret) break; - svm->vmcb01.ptr->save.g_pat = data; - if (is_guest_mode(vcpu)) - nested_vmcb02_compute_g_pat(svm); - vmcb_mark_dirty(svm->vmcb, VMCB_NPT); + if (npt_enabled) { + vmcb_set_gpat(svm->vmcb01.ptr, data); + if (is_guest_mode(vcpu) && !l2_has_separate_pat(vcpu)) + vmcb_set_gpat(svm->vmcb, data); + } break; case MSR_IA32_SPEC_CTRL: if (!msr->host_initiated && @@ -3656,13 +3702,8 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) struct vcpu_svm *svm = to_svm(vcpu); struct kvm_run *kvm_run = vcpu->run; - /* SEV-ES guests must use the CR write traps to track CR registers. */ - if (!is_sev_es_guest(vcpu)) { - if (!svm_is_intercept(svm, INTERCEPT_CR0_WRITE)) - vcpu->arch.cr0 = svm->vmcb->save.cr0; - if (npt_enabled) - vcpu->arch.cr3 = svm->vmcb->save.cr3; - } + if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE)) + return 0; if (is_guest_mode(vcpu)) { int vmexit; @@ -4191,7 +4232,7 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva) static void svm_flush_tlb_guest(struct kvm_vcpu *vcpu) { - kvm_register_mark_dirty(vcpu, VCPU_EXREG_ERAPS); + kvm_register_mark_dirty(vcpu, VCPU_REG_ERAPS); svm_flush_tlb_asid(vcpu); } @@ -4390,7 +4431,7 @@ static fastpath_t svm_exit_handlers_fastpath(struct kvm_vcpu *vcpu) return EXIT_FASTPATH_NONE; } -static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_intercepted) +static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, unsigned enter_flags) { struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu); struct vcpu_svm *svm = to_svm(vcpu); @@ -4412,10 +4453,10 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in amd_clear_divider(); if (is_sev_es_guest(vcpu)) - __svm_sev_es_vcpu_run(svm, spec_ctrl_intercepted, + __svm_sev_es_vcpu_run(svm, enter_flags, sev_es_host_save_area(sd)); else - __svm_vcpu_run(svm, spec_ctrl_intercepted); + __svm_vcpu_run(svm, enter_flags); raw_local_irq_disable(); @@ -4426,13 +4467,16 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) { bool force_immediate_exit = run_flags & KVM_RUN_FORCE_IMMEDIATE_EXIT; struct vcpu_svm *svm = to_svm(vcpu); - bool spec_ctrl_intercepted = msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL); + unsigned enter_flags = 0; + + if (!msr_write_intercepted(svm, MSR_IA32_SPEC_CTRL)) + enter_flags |= KVM_ENTER_SAVE_SPEC_CTRL; trace_kvm_entry(vcpu, force_immediate_exit); svm->vmcb->save.rax = vcpu->arch.regs[VCPU_REGS_RAX]; svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP]; - svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP]; + svm->vmcb->save.rip = vcpu->arch.rip; /* * Disable singlestep if we're injecting an interrupt/exception. @@ -4469,7 +4513,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) svm->vmcb->save.cr2 = vcpu->arch.cr2; if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS) && - kvm_register_is_dirty(vcpu, VCPU_EXREG_ERAPS)) + kvm_register_is_dirty(vcpu, VCPU_REG_ERAPS)) svm->vmcb->control.erap_ctl |= ERAP_CONTROL_CLEAR_RAP; svm_fixup_nested_rips(vcpu); @@ -4509,18 +4553,24 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL)) x86_spec_ctrl_set_guest(svm->virt_spec_ctrl); - svm_vcpu_enter_exit(vcpu, spec_ctrl_intercepted); + svm_vcpu_enter_exit(vcpu, enter_flags); if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL)) x86_spec_ctrl_restore_host(svm->virt_spec_ctrl); + /* SEV-ES guests must use the CR write traps to track CR registers. */ if (!is_sev_es_guest(vcpu)) { vcpu->arch.cr2 = svm->vmcb->save.cr2; vcpu->arch.regs[VCPU_REGS_RAX] = svm->vmcb->save.rax; vcpu->arch.regs[VCPU_REGS_RSP] = svm->vmcb->save.rsp; - vcpu->arch.regs[VCPU_REGS_RIP] = svm->vmcb->save.rip; + vcpu->arch.rip = svm->vmcb->save.rip; + + if (!svm_is_intercept(svm, INTERCEPT_CR0_WRITE)) + vcpu->arch.cr0 = svm->vmcb->save.cr0; + if (npt_enabled) + vcpu->arch.cr3 = svm->vmcb->save.cr3; } - vcpu->arch.regs_dirty = 0; + kvm_reset_dirty_registers(vcpu); if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI)) kvm_before_interrupt(vcpu, KVM_HANDLING_NMI); @@ -4566,9 +4616,9 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) vcpu->arch.apf.host_apf_flags = kvm_read_and_reset_apf_flags(); - vcpu->arch.regs_avail &= ~SVM_REGS_LAZY_LOAD_SET; + kvm_clear_available_registers(vcpu, SVM_REGS_LAZY_LOAD_SET); - if (!msr_write_intercepted(vcpu, MSR_AMD64_PERF_CNTR_GLOBAL_CTL)) + if (!msr_write_intercepted(svm, MSR_AMD64_PERF_CNTR_GLOBAL_CTL)) rdmsrq(MSR_AMD64_PERF_CNTR_GLOBAL_CTL, vcpu_to_pmu(vcpu)->global_ctrl); trace_kvm_exit(vcpu, KVM_ISA_SVM); @@ -4624,6 +4674,11 @@ svm_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall) hypercall[2] = 0xd9; } +static bool svm_tdp_has_smep(struct kvm *kvm) +{ + return gmet_enabled; +} + /* * The kvm parameter can be NULL (module initialization, or invocation before * VM creation). Be sure to check the kvm parameter before using it. @@ -4958,7 +5013,7 @@ static int svm_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) svm->vmcb->save.rax = vcpu->arch.regs[VCPU_REGS_RAX]; svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP]; - svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP]; + svm->vmcb->save.rip = vcpu->arch.rip; nested_svm_simple_vmexit(svm, SVM_EXIT_SW); @@ -5367,6 +5422,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .write_tsc_multiplier = svm_write_tsc_multiplier, .load_mmu_pgd = svm_load_mmu_pgd, + .tdp_has_smep = svm_tdp_has_smep, .check_intercept = svm_check_intercept, .handle_exit_irqoff = svm_handle_exit_irqoff, @@ -5491,6 +5547,9 @@ static __init void svm_set_cpu_caps(void) if (boot_cpu_has(X86_FEATURE_PFTHRESHOLD)) kvm_cpu_cap_set(X86_FEATURE_PFTHRESHOLD); + if (gmet_enabled) + kvm_cpu_cap_set(X86_FEATURE_GMET); + if (vgif) kvm_cpu_cap_set(X86_FEATURE_VGIF); @@ -5600,6 +5659,9 @@ static __init int svm_hardware_setup(void) if (!boot_cpu_has(X86_FEATURE_NPT)) npt_enabled = false; + if (!npt_enabled || !boot_cpu_has(X86_FEATURE_GMET)) + gmet_enabled = false; + /* Force VM NPT level equal to the host's paging level */ kvm_configure_mmu(npt_enabled, get_npt_level(), get_npt_level(), PG_LEVEL_1G); diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 5137416be593..716be21fba33 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -23,7 +23,9 @@ #include <asm/sev-common.h> #include "cpuid.h" -#include "kvm_cache_regs.h" +#include "regs.h" +#include "x86.h" +#include "pmu.h" /* * Helpers to convert to/from physical addresses for pages whose address is @@ -44,6 +46,7 @@ static inline struct page *__sme_pa_to_page(unsigned long pa) #define IOPM_SIZE PAGE_SIZE * 3 #define MSRPM_SIZE PAGE_SIZE * 2 +extern bool gmet_enabled; extern bool npt_enabled; extern int nrips; extern int vgif; @@ -165,6 +168,7 @@ struct vmcb_save_area_cached { u64 isst_addr; u64 rax; u64 cr2; + u64 g_pat; u64 dbgctl; u64 br_from; u64 br_to; @@ -466,6 +470,12 @@ static inline bool vmcb12_is_dirty(struct vmcb_ctrl_area_cached *control, int bi return !test_bit(bit, (unsigned long *)&control->clean); } +static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data) +{ + vmcb->save.g_pat = data; + vmcb_mark_dirty(vmcb, VMCB_NPT); +} + static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu) { return container_of(vcpu, struct vcpu_svm, vcpu); @@ -487,7 +497,7 @@ static inline bool svm_is_vmrun_failure(u64 exit_code) * KVM_REQ_LOAD_MMU_PGD is always requested when the cached vcpu->arch.cr3 * is changed. svm_load_mmu_pgd() then syncs the new CR3 value into the VMCB. */ -#define SVM_REGS_LAZY_LOAD_SET (1 << VCPU_EXREG_PDPTR) +#define SVM_REGS_LAZY_LOAD_SET (BIT(VCPU_REG_PDPTR)) static inline void __vmcb_set_intercept(unsigned long *intercepts, u32 bit) { @@ -643,6 +653,16 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm) return svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_NP; } +static inline bool l2_has_separate_pat(struct kvm_vcpu *vcpu) +{ + /* + * If KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled while a vCPU + * is running, the L2 IA32_PAT semantics for that vCPU are undefined. + */ + return nested_npt_enabled(to_svm(vcpu)) && + !kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT); +} + static inline bool nested_vnmi_enabled(struct vcpu_svm *svm) { return guest_cpu_cap_has(&svm->vcpu, X86_FEATURE_VNMI) && @@ -816,6 +836,8 @@ static inline void svm_enable_intercept_for_msr(struct kvm_vcpu *vcpu, svm_set_intercept_for_msr(vcpu, msr, type, true); } +int svm_skip_emulated_instruction(struct kvm_vcpu *vcpu); + /* nested.c */ #define NESTED_EXIT_HOST 0 /* Exit handled on host level */ @@ -877,9 +899,30 @@ void nested_copy_vmcb_control_to_cache(struct vcpu_svm *svm, void nested_copy_vmcb_save_to_cache(struct vcpu_svm *svm, struct vmcb_save_area *save); void nested_sync_control_from_vmcb02(struct vcpu_svm *svm); -void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm); void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb); + +static inline void __svm_pmu_handle_nested_transition(struct vcpu_svm *svm, + bool defer) +{ + struct kvm_pmu *pmu = vcpu_to_pmu(&svm->vcpu); + u64 counters = *(u64 *)pmu->pmc_has_mode_specific_enables; + + __kvm_pmu_reprogram_counters(pmu, counters, defer); +} + +static inline void svm_pmu_handle_nested_transition(struct vcpu_svm *svm) +{ + /* + * Do NOT defer reprogramming the counters by default. Instructions + * causing a state change are counted based on the _new_ CPU state + * (e.g. a successful VMRUN is counted in guest mode). Hence, the + * counters should be reprogrammed with the new state _before_ the + * instruction is potentially counted upon emulation completion. + */ + __svm_pmu_handle_nested_transition(svm, false); +} + extern struct kvm_x86_nested_ops svm_nested_ops; /* avic.c */ @@ -1010,9 +1053,9 @@ static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_sa /* vmenter.S */ -void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, bool spec_ctrl_intercepted, +void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, unsigned int flags, struct sev_es_save_area *hostsa); -void __svm_vcpu_run(struct vcpu_svm *svm, bool spec_ctrl_intercepted); +void __svm_vcpu_run(struct vcpu_svm *svm, unsigned int flags); #define DEFINE_KVM_GHCB_ACCESSORS(field) \ static __always_inline u64 kvm_ghcb_get_##field(struct vcpu_svm *svm) \ diff --git a/arch/x86/kvm/svm/vmenter.S b/arch/x86/kvm/svm/vmenter.S index d47c5c93c991..99701892f8ec 100644 --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -4,31 +4,9 @@ #include <asm/asm-offsets.h> #include <asm/bitsperlong.h> #include <asm/frame.h> -#include <asm/kvm_vcpu_regs.h> #include <asm/nospec-branch.h> #include "kvm-asm-offsets.h" - -#define WORD_SIZE (BITS_PER_LONG / 8) - -/* Intentionally omit RAX as it's context switched by hardware */ -#define VCPU_RCX (SVM_vcpu_arch_regs + __VCPU_REGS_RCX * WORD_SIZE) -#define VCPU_RDX (SVM_vcpu_arch_regs + __VCPU_REGS_RDX * WORD_SIZE) -#define VCPU_RBX (SVM_vcpu_arch_regs + __VCPU_REGS_RBX * WORD_SIZE) -/* Intentionally omit RSP as it's context switched by hardware */ -#define VCPU_RBP (SVM_vcpu_arch_regs + __VCPU_REGS_RBP * WORD_SIZE) -#define VCPU_RSI (SVM_vcpu_arch_regs + __VCPU_REGS_RSI * WORD_SIZE) -#define VCPU_RDI (SVM_vcpu_arch_regs + __VCPU_REGS_RDI * WORD_SIZE) - -#ifdef CONFIG_X86_64 -#define VCPU_R8 (SVM_vcpu_arch_regs + __VCPU_REGS_R8 * WORD_SIZE) -#define VCPU_R9 (SVM_vcpu_arch_regs + __VCPU_REGS_R9 * WORD_SIZE) -#define VCPU_R10 (SVM_vcpu_arch_regs + __VCPU_REGS_R10 * WORD_SIZE) -#define VCPU_R11 (SVM_vcpu_arch_regs + __VCPU_REGS_R11 * WORD_SIZE) -#define VCPU_R12 (SVM_vcpu_arch_regs + __VCPU_REGS_R12 * WORD_SIZE) -#define VCPU_R13 (SVM_vcpu_arch_regs + __VCPU_REGS_R13 * WORD_SIZE) -#define VCPU_R14 (SVM_vcpu_arch_regs + __VCPU_REGS_R14 * WORD_SIZE) -#define VCPU_R15 (SVM_vcpu_arch_regs + __VCPU_REGS_R15 * WORD_SIZE) -#endif +#include "vmenter.h" #define SVM_vmcb01_pa (SVM_vmcb01 + KVM_VMCB_pa) @@ -39,38 +17,6 @@ ALTERNATIVE_2 "", \ "jmp 800f", X86_FEATURE_MSR_SPEC_CTRL, \ "", X86_FEATURE_V_SPEC_CTRL -801: -.endm -.macro RESTORE_GUEST_SPEC_CTRL_BODY -800: - /* - * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the - * host's, write the MSR. This is kept out-of-line so that the common - * case does not have to jump. - * - * IMPORTANT: To avoid RSB underflow attacks and any other nastiness, - * there must not be any returns or indirect branches between this code - * and vmentry. - */ -#ifdef CONFIG_X86_64 - mov SVM_spec_ctrl(%rdi), %rdx - cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx - je 801b - movl %edx, %eax - shr $32, %rdx -#else - mov SVM_spec_ctrl(%edi), %eax - mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx - xor %eax, %ecx - mov SVM_spec_ctrl + 4(%edi), %edx - mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %esi - xor %edx, %esi - or %esi, %ecx - je 801b -#endif - mov $MSR_IA32_SPEC_CTRL, %ecx - wrmsr - jmp 801b .endm .macro RESTORE_HOST_SPEC_CTRL @@ -78,42 +24,6 @@ ALTERNATIVE_2 "", \ "jmp 900f", X86_FEATURE_MSR_SPEC_CTRL, \ "", X86_FEATURE_V_SPEC_CTRL -901: -.endm -.macro RESTORE_HOST_SPEC_CTRL_BODY spec_ctrl_intercepted:req -900: - /* Same for after vmexit. */ - mov $MSR_IA32_SPEC_CTRL, %ecx - - /* - * Load the value that the guest had written into MSR_IA32_SPEC_CTRL, - * if it was not intercepted during guest execution. - */ - cmpb $0, \spec_ctrl_intercepted - jnz 998f - rdmsr - movl %eax, SVM_spec_ctrl(%_ASM_DI) - movl %edx, SVM_spec_ctrl + 4(%_ASM_DI) -998: - /* Now restore the host value of the MSR if different from the guest's. */ -#ifdef CONFIG_X86_64 - mov PER_CPU_VAR(x86_spec_ctrl_current), %rdx - cmp SVM_spec_ctrl(%rdi), %rdx - je 901b - movl %edx, %eax - shr $32, %rdx -#else - mov PER_CPU_VAR(x86_spec_ctrl_current), %eax - mov SVM_spec_ctrl(%edi), %esi - xor %eax, %esi - mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edx - mov SVM_spec_ctrl + 4(%edi), %edi - xor %edx, %edi - or %edi, %esi - je 901b -#endif - wrmsr - jmp 901b .endm #define SVM_CLEAR_CPU_BUFFERS \ @@ -121,8 +31,8 @@ /** * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode - * @svm: struct vcpu_svm * - * @spec_ctrl_intercepted: bool + * @svm: struct vcpu_svm * + * @enter_flags: u32 */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP @@ -162,6 +72,7 @@ SYM_FUNC_START(__svm_vcpu_run) /* Clobbers RAX, RCX, RDX (and ESI on 32-bit), consumes RDI (@svm). */ RESTORE_GUEST_SPEC_CTRL +801: /* * Use a single vmcb (vmcb01 because it's always valid) for @@ -177,23 +88,17 @@ SYM_FUNC_START(__svm_vcpu_run) mov SVM_current_vmcb(%_ASM_DI), %_ASM_AX mov KVM_VMCB_pa(%_ASM_AX), %_ASM_AX - /* Load guest registers. */ - mov VCPU_RCX(%_ASM_DI), %_ASM_CX - mov VCPU_RDX(%_ASM_DI), %_ASM_DX - mov VCPU_RBX(%_ASM_DI), %_ASM_BX - mov VCPU_RBP(%_ASM_DI), %_ASM_BP - mov VCPU_RSI(%_ASM_DI), %_ASM_SI + /* + * Load guest registers. Intentionally omit %_ASM_AX and %_ASM_SP as + * context switched by hardware + */ + LOAD_REGS %_ASM_DI, SVM_vcpu_arch_regs, \ + %_ASM_CX, %_ASM_DX, %_ASM_BX, %_ASM_BP, %_ASM_SI #ifdef CONFIG_X86_64 - mov VCPU_R8 (%_ASM_DI), %r8 - mov VCPU_R9 (%_ASM_DI), %r9 - mov VCPU_R10(%_ASM_DI), %r10 - mov VCPU_R11(%_ASM_DI), %r11 - mov VCPU_R12(%_ASM_DI), %r12 - mov VCPU_R13(%_ASM_DI), %r13 - mov VCPU_R14(%_ASM_DI), %r14 - mov VCPU_R15(%_ASM_DI), %r15 + LOAD_REGS %_ASM_DI, SVM_vcpu_arch_regs, \ + %r8, %r9, %r10, %r11, %r12, %r13, %r14, %r15 #endif - mov VCPU_RDI(%_ASM_DI), %_ASM_DI + LOAD_REGS %_ASM_DI, SVM_vcpu_arch_regs, %_ASM_DI /* Clobbers EFLAGS.ZF */ SVM_CLEAR_CPU_BUFFERS @@ -204,22 +109,15 @@ SYM_FUNC_START(__svm_vcpu_run) /* Pop @svm to RAX while it's the only available register. */ pop %_ASM_AX - /* Save all guest registers. */ - mov %_ASM_CX, VCPU_RCX(%_ASM_AX) - mov %_ASM_DX, VCPU_RDX(%_ASM_AX) - mov %_ASM_BX, VCPU_RBX(%_ASM_AX) - mov %_ASM_BP, VCPU_RBP(%_ASM_AX) - mov %_ASM_SI, VCPU_RSI(%_ASM_AX) - mov %_ASM_DI, VCPU_RDI(%_ASM_AX) + /* + * Save all guest registers. Intentionally omit %_ASM_AX and %_ASM_SP as + * context switched by hardware + */ + STORE_REGS %_ASM_AX, SVM_vcpu_arch_regs, \ + %_ASM_CX, %_ASM_DX, %_ASM_BX, %_ASM_BP, %_ASM_SI, %_ASM_DI #ifdef CONFIG_X86_64 - mov %r8, VCPU_R8 (%_ASM_AX) - mov %r9, VCPU_R9 (%_ASM_AX) - mov %r10, VCPU_R10(%_ASM_AX) - mov %r11, VCPU_R11(%_ASM_AX) - mov %r12, VCPU_R12(%_ASM_AX) - mov %r13, VCPU_R13(%_ASM_AX) - mov %r14, VCPU_R14(%_ASM_AX) - mov %r15, VCPU_R15(%_ASM_AX) + STORE_REGS %_ASM_AX, SVM_vcpu_arch_regs, \ + %r8, %r9, %r10, %r11, %r12, %r13, %r14, %r15 #endif /* @svm can stay in RDI from now on. */ @@ -242,6 +140,7 @@ SYM_FUNC_START(__svm_vcpu_run) * and RSP (pointer to @spec_ctrl_intercepted). */ RESTORE_HOST_SPEC_CTRL +901: /* * Mitigate RETBleed for AMD/Hygon Zen uarch. RET should be @@ -261,24 +160,12 @@ SYM_FUNC_START(__svm_vcpu_run) * free. RSP and RAX are exempt as they are restored by hardware * during VM-Exit. */ - xor %ecx, %ecx - xor %edx, %edx - xor %ebx, %ebx - xor %ebp, %ebp - xor %esi, %esi - xor %edi, %edi + CLEAR_REGS %ecx, %edx, %ebx, %ebp, %esi, %edi #ifdef CONFIG_X86_64 - xor %r8d, %r8d - xor %r9d, %r9d - xor %r10d, %r10d - xor %r11d, %r11d - xor %r12d, %r12d - xor %r13d, %r13d - xor %r14d, %r14d - xor %r15d, %r15d + CLEAR_REGS %r8d, %r9d, %r10d, %r11d, %r12d, %r13d, %r14d, %r15d #endif - /* "Pop" @spec_ctrl_intercepted. */ + /* "Pop" @enter_flags. */ pop %_ASM_BX pop %_ASM_BX @@ -295,8 +182,12 @@ SYM_FUNC_START(__svm_vcpu_run) pop %_ASM_BP RET - RESTORE_GUEST_SPEC_CTRL_BODY - RESTORE_HOST_SPEC_CTRL_BODY (%_ASM_SP) +800: + RESTORE_GUEST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), 801b + jmp 801b +900: + RESTORE_HOST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), (%_ASM_SP), 901b + jmp 901b 10: cmpb $0, _ASM_RIP(virt_rebooting) jne 2b @@ -320,23 +211,12 @@ SYM_FUNC_END(__svm_vcpu_run) #ifdef CONFIG_KVM_AMD_SEV - -#ifdef CONFIG_X86_64 #define SEV_ES_GPRS_BASE 0x300 -#define SEV_ES_RBX (SEV_ES_GPRS_BASE + __VCPU_REGS_RBX * WORD_SIZE) -#define SEV_ES_RBP (SEV_ES_GPRS_BASE + __VCPU_REGS_RBP * WORD_SIZE) -#define SEV_ES_RSI (SEV_ES_GPRS_BASE + __VCPU_REGS_RSI * WORD_SIZE) -#define SEV_ES_RDI (SEV_ES_GPRS_BASE + __VCPU_REGS_RDI * WORD_SIZE) -#define SEV_ES_R12 (SEV_ES_GPRS_BASE + __VCPU_REGS_R12 * WORD_SIZE) -#define SEV_ES_R13 (SEV_ES_GPRS_BASE + __VCPU_REGS_R13 * WORD_SIZE) -#define SEV_ES_R14 (SEV_ES_GPRS_BASE + __VCPU_REGS_R14 * WORD_SIZE) -#define SEV_ES_R15 (SEV_ES_GPRS_BASE + __VCPU_REGS_R15 * WORD_SIZE) -#endif /** * __svm_sev_es_vcpu_run - Run a SEV-ES vCPU via a transition to SVM guest mode - * @svm: struct vcpu_svm * - * @spec_ctrl_intercepted: bool + * @svm: struct vcpu_svm * + * @enter_flags: u32 */ SYM_FUNC_START(__svm_sev_es_vcpu_run) FRAME_BEGIN @@ -346,22 +226,17 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) * Except for RAX and RSP, all GPRs are restored on #VMEXIT, but not * saved on VMRUN. */ - mov %rbp, SEV_ES_RBP (%rdx) - mov %r15, SEV_ES_R15 (%rdx) - mov %r14, SEV_ES_R14 (%rdx) - mov %r13, SEV_ES_R13 (%rdx) - mov %r12, SEV_ES_R12 (%rdx) - mov %rbx, SEV_ES_RBX (%rdx) + STORE_REGS %rdx, SEV_ES_GPRS_BASE, %rbp, %r15, %r14, %r13, %r12, %rbx /* * Save volatile registers that hold arguments that are needed after - * #VMEXIT (RDI=@svm and RSI=@spec_ctrl_intercepted). + * #VMEXIT (RDI=@svm and RSI=@enter_flags). */ - mov %rdi, SEV_ES_RDI (%rdx) - mov %rsi, SEV_ES_RSI (%rdx) + STORE_REGS %rdx, SEV_ES_GPRS_BASE, %rdi, %rsi /* Clobbers RAX, RCX, and RDX (@hostsa), consumes RDI (@svm). */ RESTORE_GUEST_SPEC_CTRL +801: /* Get svm->current_vmcb->pa into RAX. */ mov SVM_current_vmcb(%rdi), %rax @@ -376,8 +251,9 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) /* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */ FILL_RETURN_BUFFER %rax, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT - /* Clobbers RAX, RCX, RDX, consumes RDI (@svm) and RSI (@spec_ctrl_intercepted). */ + /* Clobbers RAX, RCX, RDX, consumes RDI (@svm) and RSI (@enter_flags). */ RESTORE_HOST_SPEC_CTRL +901: /* * Mitigate RETBleed for AMD/Hygon Zen uarch. RET should be @@ -391,8 +267,12 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) FRAME_END RET - RESTORE_GUEST_SPEC_CTRL_BODY - RESTORE_HOST_SPEC_CTRL_BODY %sil +800: + RESTORE_GUEST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), 801b + jmp 801b +900: + RESTORE_HOST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), %esi, 901b + jmp 901b 3: cmpb $0, virt_rebooting(%rip) jne 2b diff --git a/arch/x86/kvm/vmenter.h b/arch/x86/kvm/vmenter.h new file mode 100644 index 000000000000..77376812c81b --- /dev/null +++ b/arch/x86/kvm/vmenter.h @@ -0,0 +1,111 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_VMENTER_H +#define __KVM_X86_VMENTER_H + +#include <asm/kvm_vcpu_regs.h> + +#define KVM_ENTER_VMRESUME BIT(0) +#define KVM_ENTER_SAVE_SPEC_CTRL BIT(1) +#define KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(2) + +#ifdef __ASSEMBLER__ +.macro RESTORE_GUEST_SPEC_CTRL_BODY guest_spec_ctrl:req, label:req + /* + * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the + * host's, write the MSR. This is kept out-of-line so that the common + * case does not have to jump. + * + * IMPORTANT: To avoid RSB underflow attacks and any other nastiness, + * there must not be any returns or indirect branches between this code + * and vmentry. + */ +#ifdef CONFIG_X86_64 + mov \guest_spec_ctrl, %rdx + cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx + je \label + movl %edx, %eax + shr $32, %rdx +#else + mov \guest_spec_ctrl, %eax + mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx + xor %eax, %ecx + mov 4 + \guest_spec_ctrl, %edx + mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %esi + xor %edx, %esi + or %esi, %ecx + je \label +#endif + mov $MSR_IA32_SPEC_CTRL, %ecx + wrmsr +.endm + +.macro RESTORE_HOST_SPEC_CTRL_BODY guest_spec_ctrl:req, enter_flags:req, label:req + /* Same for after vmexit. */ + mov $MSR_IA32_SPEC_CTRL, %ecx + + /* + * Load the value that the guest had written into MSR_IA32_SPEC_CTRL, + * if it was not intercepted during guest execution. + */ + testl $KVM_ENTER_SAVE_SPEC_CTRL, \enter_flags + jz 998f + rdmsr + movl %eax, \guest_spec_ctrl + movl %edx, 4 + \guest_spec_ctrl +998: + /* Now restore the host value of the MSR if different from the guest's. */ +#ifdef CONFIG_X86_64 + mov PER_CPU_VAR(x86_spec_ctrl_current), %rdx + cmp \guest_spec_ctrl, %rdx + /* + * For legacy IBRS, the IBRS bit always needs to be written after + * transitioning from a less privileged predictor mode, regardless of + * whether the guest/host values differ. + */ + ALTERNATIVE __stringify(je \label), "", X86_FEATURE_KERNEL_IBRS + movl %edx, %eax + shr $32, %rdx +#else + mov PER_CPU_VAR(x86_spec_ctrl_current), %eax + mov \guest_spec_ctrl, %esi + xor %eax, %esi + mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edx + mov 4 + \guest_spec_ctrl, %edi + xor %edx, %edi + or %edi, %esi + ALTERNATIVE __stringify(je \label), "", X86_FEATURE_KERNEL_IBRS +#endif + wrmsr +.endm + +#define WORD_SIZE (BITS_PER_LONG / 8) + +.macro LOAD_REGS src:req, regs_ofs:req, regs:vararg +.irp reg, \regs + REG_NUM reg_num \reg + mov (\regs_ofs + reg_num * WORD_SIZE)(\src), \reg +.endr +.endm + +.macro STORE_REGS dst:req, regs_ofs:req, regs:vararg +.irp reg, \regs + REG_NUM reg_num \reg + mov \reg, (\regs_ofs + reg_num * WORD_SIZE)(\dst) +.endr +.endm + +.macro POP_REGS dst:req, regs_ofs:req, regs:vararg +.irp reg, \regs + REG_NUM reg_num \reg + pop (\regs_ofs + reg_num * WORD_SIZE)(\dst) +.endr +.endm + +.macro CLEAR_REGS regs:vararg +.irp reg, \regs + xorl \reg, \reg +.endr +.endm + +#endif /* __ASSEMBLER__ */ +#endif /* __KVM_X86_VMENTER_H */ diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h index 31568274d8bb..810119167f79 100644 --- a/arch/x86/kvm/vmx/capabilities.h +++ b/arch/x86/kvm/vmx/capabilities.h @@ -16,6 +16,7 @@ extern bool __read_mostly enable_unrestricted_guest; extern bool __read_mostly enable_ept_ad_bits; extern bool __read_mostly enable_cet; extern bool __read_mostly enable_pml; +extern bool __read_mostly enable_mbec; extern int __read_mostly pt_mode; #define PT_MODE_SYSTEM 0 @@ -301,11 +302,6 @@ static inline bool cpu_has_vmx_flexpriority(void) cpu_has_vmx_virtualize_apic_accesses(); } -static inline bool cpu_has_vmx_ept_execute_only(void) -{ - return vmx_capability.ept & VMX_EPT_EXECUTE_ONLY_BIT; -} - static inline bool cpu_has_vmx_ept_4levels(void) { return vmx_capability.ept & VMX_EPT_PAGE_WALK_4_BIT; @@ -412,4 +408,10 @@ static inline bool cpu_has_notify_vmexit(void) SECONDARY_EXEC_NOTIFY_VM_EXITING; } +static inline bool cpu_has_ept_mbec(void) +{ + return vmcs_config.cpu_based_2nd_exec_ctrl & + SECONDARY_EXEC_MODE_BASED_EPT_EXEC; +} + #endif /* __KVM_X86_VMX_CAPS_H */ diff --git a/arch/x86/kvm/vmx/common.h b/arch/x86/kvm/vmx/common.h index 412d0829d7a2..08005676702c 100644 --- a/arch/x86/kvm/vmx/common.h +++ b/arch/x86/kvm/vmx/common.h @@ -85,22 +85,30 @@ static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t gpa, { u64 error_code; - /* Is it a read fault? */ - error_code = (exit_qualification & EPT_VIOLATION_ACC_READ) - ? PFERR_USER_MASK : 0; /* Is it a write fault? */ - error_code |= (exit_qualification & EPT_VIOLATION_ACC_WRITE) + error_code = (exit_qualification & EPT_VIOLATION_ACC_WRITE) ? PFERR_WRITE_MASK : 0; /* Is it a fetch fault? */ error_code |= (exit_qualification & EPT_VIOLATION_ACC_INSTR) ? PFERR_FETCH_MASK : 0; - /* ept page table entry is present? */ - error_code |= (exit_qualification & EPT_VIOLATION_PROT_MASK) + /* ept page table entry is present? */ + error_code |= (exit_qualification & + (EPT_VIOLATION_PROT_MASK & ~EPT_VIOLATION_PROT_USER_EXEC)) ? PFERR_PRESENT_MASK : 0; - if (exit_qualification & EPT_VIOLATION_GVA_IS_VALID) - error_code |= (exit_qualification & EPT_VIOLATION_GVA_TRANSLATED) ? - PFERR_GUEST_FINAL_MASK : PFERR_GUEST_PAGE_MASK; + if (mmu_has_mbec(vcpu->arch.mmu)) + error_code |= (exit_qualification & EPT_VIOLATION_PROT_USER_EXEC) + ? PFERR_PRESENT_MASK : 0; + + if (exit_qualification & EPT_VIOLATION_GVA_IS_VALID) { + if (exit_qualification & EPT_VIOLATION_GVA_TRANSLATED) { + error_code |= PFERR_GUEST_FINAL_MASK; + if (exit_qualification & EPT_VIOLATION_GVA_USER) + error_code |= PFERR_USER_MASK; + } else { + error_code |= PFERR_GUEST_PAGE_MASK; + } + } if (vt_is_tdx_private_gpa(vcpu->kvm, gpa)) error_code |= PFERR_PRIVATE_ACCESS; diff --git a/arch/x86/kvm/vmx/hyperv_evmcs.h b/arch/x86/kvm/vmx/hyperv_evmcs.h index fc7c4e7bd1bf..bc08fe40590e 100644 --- a/arch/x86/kvm/vmx/hyperv_evmcs.h +++ b/arch/x86/kvm/vmx/hyperv_evmcs.h @@ -87,6 +87,7 @@ SECONDARY_EXEC_PT_CONCEAL_VMX | \ SECONDARY_EXEC_BUS_LOCK_DETECTION | \ SECONDARY_EXEC_NOTIFY_VM_EXITING | \ + SECONDARY_EXEC_MODE_BASED_EPT_EXEC | \ SECONDARY_EXEC_ENCLS_EXITING) #define EVMCS1_SUPPORTED_3RDEXEC (0ULL) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index dbebddf648be..83d9921277ea 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -755,6 +755,14 @@ static int vt_set_identity_map_addr(struct kvm *kvm, u64 ident_addr) return vmx_set_identity_map_addr(kvm, ident_addr); } +static bool vt_tdp_has_smep(struct kvm *kvm) +{ + if (is_td(kvm)) + return false; + + return vmx_tdp_has_smep(kvm); +} + static u64 vt_get_l2_tsc_offset(struct kvm_vcpu *vcpu) { /* TDX doesn't support L2 guest at the moment. */ @@ -966,6 +974,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_tss_addr = vt_op(set_tss_addr), .set_identity_map_addr = vt_op(set_identity_map_addr), .get_mt_mask = vmx_get_mt_mask, + .tdp_has_smep = vt_op(tdp_has_smep), .get_exit_info = vt_op(get_exit_info), .get_entry_info = vt_op(get_entry_info), diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 3fe88f29be7a..3a293640d58c 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -310,13 +310,13 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) vmx_sync_vmcs_host_state(vmx, prev); put_cpu(); - vcpu->arch.regs_avail = ~VMX_REGS_LAZY_LOAD_SET; + kvm_clear_available_registers(vcpu, VMX_REGS_LAZY_LOAD_SET); /* * All lazily updated registers will be reloaded from VMCS12 on both * vmentry and vmexit. */ - vcpu->arch.regs_dirty = 0; + kvm_reset_dirty_registers(vcpu); } static void nested_put_vmcs12_pages(struct kvm_vcpu *vcpu) @@ -411,7 +411,8 @@ static void nested_ept_invalidate_addr(struct kvm_vcpu *vcpu, gpa_t eptp, } static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, - struct x86_exception *fault) + struct x86_exception *fault, + bool from_hardware) { struct vmcs12 *vmcs12 = get_vmcs12(vcpu); struct vcpu_vmx *vmx = to_vmx(vcpu); @@ -443,10 +444,30 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, vm_exit_reason = EXIT_REASON_EPT_MISCONFIG; exit_qualification = 0; } else { - exit_qualification = fault->exit_qualification; - exit_qualification |= vmx_get_exit_qual(vcpu) & - (EPT_VIOLATION_GVA_IS_VALID | - EPT_VIOLATION_GVA_TRANSLATED); + u64 mask = EPT_VIOLATION_GVA_IS_VALID | + EPT_VIOLATION_GVA_TRANSLATED; + + if (vmx->nested.msrs.ept_caps & VMX_EPT_ADVANCED_VMEXIT_INFO_BIT) + mask |= EPT_VIOLATION_GVA_USER | + EPT_VIOLATION_GVA_WRITABLE | + EPT_VIOLATION_GVA_NX; + + exit_qualification = fault->exit_qualification & ~mask; + + /* + * Use the EXIT_QUALIFICATION from the VMCS if and only + * if the hardware VM-Exit from L2 was an EPT Violation. + * If the fault is synthesized, then EXIT_QUALIFICATION + * is stale and/or holds entirely different data. And + * conversely, KVM _must_ rely on EXIT_QUALIFICATION if + * the fault came from hardware, because KVM only sees + * and walks the faulting GPA. + */ + if (from_hardware) + exit_qualification |= vmx_get_exit_qual(vcpu) & mask; + else + exit_qualification |= fault->exit_qualification & mask; + vm_exit_reason = EXIT_REASON_EPT_VIOLATION; } @@ -465,6 +486,13 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, vmcs12->guest_physical_address = fault->address; } +static inline bool nested_ept_mbec_enabled(struct kvm_vcpu *vcpu) +{ + struct vmcs12 *vmcs12 = get_vmcs12(vcpu); + + return nested_cpu_has2(vmcs12, SECONDARY_EXEC_MODE_BASED_EPT_EXEC); +} + static void nested_ept_new_eptp(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); @@ -473,6 +501,7 @@ static void nested_ept_new_eptp(struct kvm_vcpu *vcpu) kvm_init_shadow_ept_mmu(vcpu, execonly, ept_lpage_level, nested_ept_ad_enabled(vcpu), + nested_ept_mbec_enabled(vcpu), nested_ept_get_eptp(vcpu)); } @@ -1189,7 +1218,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, } vcpu->arch.cr3 = cr3; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ kvm_init_mmu(vcpu); @@ -2440,6 +2469,7 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0 SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | SECONDARY_EXEC_APIC_REGISTER_VIRT | SECONDARY_EXEC_ENABLE_VMFUNC | + SECONDARY_EXEC_MODE_BASED_EPT_EXEC | SECONDARY_EXEC_DESC); if (nested_cpu_has(vmcs12, @@ -2610,17 +2640,6 @@ static void prepare_vmcs02_rare(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12) vmcs_writel(GUEST_SYSENTER_ESP, vmcs12->guest_sysenter_esp); vmcs_writel(GUEST_SYSENTER_EIP, vmcs12->guest_sysenter_eip); - /* - * L1 may access the L2's PDPTR, so save them to construct - * vmcs12 - */ - if (enable_ept) { - vmcs_write64(GUEST_PDPTR0, vmcs12->guest_pdptr0); - vmcs_write64(GUEST_PDPTR1, vmcs12->guest_pdptr1); - vmcs_write64(GUEST_PDPTR2, vmcs12->guest_pdptr2); - vmcs_write64(GUEST_PDPTR3, vmcs12->guest_pdptr3); - } - if (kvm_mpx_supported() && vmx->vcpu.arch.nested_run_pending && (vmcs12->vm_entry_controls & VM_ENTRY_LOAD_BNDCFGS)) vmcs_write64(GUEST_BNDCFGS, vmcs12->guest_bndcfgs); @@ -4972,7 +4991,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu) nested_ept_uninit_mmu_context(vcpu); vcpu->arch.cr3 = vmcs_readl(GUEST_CR3); - kvm_register_mark_available(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_available(vcpu, VCPU_REG_CR3); /* * Use ept_save_pdptrs(vcpu) to load the MMU's cached PDPTRs @@ -5074,7 +5093,7 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason, kvm_service_local_tlb_flush_requests(vcpu); /* - * VCPU_EXREG_PDPTR will be clobbered in arch/x86/kvm/vmx/vmx.h between + * VCPU_REG_PDPTR will be clobbered in arch/x86/kvm/vmx/vmx.h between * now and the new vmentry. Ensure that the VMCS02 PDPTR fields are * up-to-date before switching to L1. */ @@ -6135,7 +6154,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) { - u32 index = kvm_rcx_read(vcpu); + u32 index = kvm_ecx_read(vcpu); u64 new_eptp; if (WARN_ON_ONCE(!nested_cpu_has_ept(vmcs12))) @@ -6169,7 +6188,7 @@ static int handle_vmfunc(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); struct vmcs12 *vmcs12; - u32 function = kvm_rax_read(vcpu); + u32 function = kvm_eax_read(vcpu); /* * VMFUNC should never execute cleanly while L1 is active; KVM supports @@ -6291,7 +6310,7 @@ static bool nested_vmx_exit_handled_msr(struct kvm_vcpu *vcpu, exit_reason.basic == EXIT_REASON_MSR_WRITE_IMM) msr_index = vmx_get_exit_qual(vcpu); else - msr_index = kvm_rcx_read(vcpu); + msr_index = kvm_ecx_read(vcpu); /* * The MSR_BITMAP page is divided into four 1024-byte bitmaps, @@ -6401,7 +6420,7 @@ static bool nested_vmx_exit_handled_encls(struct kvm_vcpu *vcpu, !nested_cpu_has2(vmcs12, SECONDARY_EXEC_ENCLS_EXITING)) return false; - encls_leaf = kvm_rax_read(vcpu); + encls_leaf = kvm_eax_read(vcpu); if (encls_leaf > 62) encls_leaf = 63; return vmcs12->encls_exiting_bitmap & BIT_ULL(encls_leaf); @@ -6522,6 +6541,8 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu *vcpu, nested_evmcs_l2_tlb_flush_enabled(vcpu) && kvm_hv_is_tlb_flush_hcall(vcpu); #endif + case EXIT_REASON_CPUID: + return !kvm_is_cpuid_allowed(vcpu); default: break; } @@ -7239,7 +7260,8 @@ static void nested_vmx_setup_secondary_ctls(u32 ept_caps, VMX_EPT_PAGE_WALK_5_BIT | VMX_EPTP_WB_BIT | VMX_EPT_INVEPT_BIT | - VMX_EPT_EXECUTE_ONLY_BIT; + VMX_EPT_EXECUTE_ONLY_BIT | + VMX_EPT_ADVANCED_VMEXIT_INFO_BIT; msrs->ept_caps &= ept_caps; msrs->ept_caps |= VMX_EPT_EXTENT_GLOBAL_BIT | @@ -7251,6 +7273,9 @@ static void nested_vmx_setup_secondary_ctls(u32 ept_caps, msrs->ept_caps |= VMX_EPT_AD_BIT; } + if (enable_mbec) + msrs->secondary_ctls_high |= + SECONDARY_EXEC_MODE_BASED_EPT_EXEC; /* * Advertise EPTP switching irrespective of hardware support, * KVM emulates it in software so long as VMFUNC is supported. @@ -7438,8 +7463,29 @@ __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *)) return 0; } +static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, + u64 access, + struct x86_exception *exception, + u64 pte_access) +{ + struct kvm_mmu *mmu = vcpu->arch.mmu; + + BUG_ON(!mmu_is_nested(vcpu)); + + /* + * MBEC differentiates based on the effective U/S bit of + * the guest page tables; not the processor CPL. + */ + access &= ~PFERR_USER_MASK; + if ((pte_access & ACC_USER_MASK) && (access & PFERR_GUEST_FINAL_MASK)) + access |= PFERR_USER_MASK; + + return mmu->gva_to_gpa(vcpu, mmu, gpa, access, exception); +} + struct kvm_x86_nested_ops vmx_nested_ops = { .leave_nested = vmx_leave_nested, + .translate_nested_gpa = vmx_translate_nested_gpa, .is_exception_vmexit = nested_vmx_is_exception_vmexit, .check_events = vmx_check_nested_events, .has_events = vmx_has_nested_events, diff --git a/arch/x86/kvm/vmx/nested.h b/arch/x86/kvm/vmx/nested.h index 213a448104af..6d6cd5904ddf 100644 --- a/arch/x86/kvm/vmx/nested.h +++ b/arch/x86/kvm/vmx/nested.h @@ -2,7 +2,7 @@ #ifndef __KVM_X86_VMX_NESTED_H #define __KVM_X86_VMX_NESTED_H -#include "kvm_cache_regs.h" +#include "regs.h" #include "hyperv.h" #include "vmcs12.h" #include "vmx.h" diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c index 74e0b01185b8..1944939c4139 100644 --- a/arch/x86/kvm/vmx/pmu_intel.c +++ b/arch/x86/kvm/vmx/pmu_intel.c @@ -57,8 +57,16 @@ static struct x86_pmu_lbr *vcpu_to_lbr_records(struct kvm_vcpu *vcpu) static void reprogram_fixed_counters(struct kvm_pmu *pmu, u64 data) { + /* + * Compare against the value the mediated PMU shoves into hardware, not + * the guest's desired value. For the emulated PMU (proxied via perf), + * they are one and the same (fixed_ctr_ctrl_hw isn't used other than + * here). For the mediated PMU, KVM needs to reprogram the actual MSR, + * and so needs to react to potential changes in the value shoved into + * hardware, e.g. to ensure the event filter is enforced. + */ + u64 old_fixed_ctr_ctrl = pmu->fixed_ctr_ctrl_hw; struct kvm_pmc *pmc; - u64 old_fixed_ctr_ctrl = pmu->fixed_ctr_ctrl; int i; pmu->fixed_ctr_ctrl = data; @@ -309,13 +317,15 @@ static bool intel_pmu_handle_lbr_msrs_access(struct kvm_vcpu *vcpu, */ local_irq_disable(); if (lbr_desc->event->state == PERF_EVENT_STATE_ACTIVE) { + int err = 0; + if (read) rdmsrq(index, msr_info->data); else - wrmsrq(index, msr_info->data); + err = wrmsrq_safe(index, msr_info->data); __set_bit(INTEL_PMC_IDX_FIXED_VLBR, vcpu_to_pmu(vcpu)->pmc_in_use); local_irq_enable(); - return true; + return !err; } clear_bit(INTEL_PMC_IDX_FIXED_VLBR, vcpu_to_pmu(vcpu)->pmc_in_use); local_irq_enable(); @@ -392,7 +402,7 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) if (pmu->pebs_enable != data) { diff = pmu->pebs_enable ^ data; pmu->pebs_enable = data; - reprogram_counters(pmu, diff); + kvm_pmu_request_counters_reprogram(pmu, diff); } break; case MSR_IA32_DS_AREA: diff --git a/arch/x86/kvm/vmx/run_flags.h b/arch/x86/kvm/vmx/run_flags.h deleted file mode 100644 index 6a87a12135fb..000000000000 --- a/arch/x86/kvm/vmx/run_flags.h +++ /dev/null @@ -1,9 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __KVM_X86_VMX_RUN_FLAGS_H -#define __KVM_X86_VMX_RUN_FLAGS_H - -#define VMX_RUN_VMRESUME BIT(0) -#define VMX_RUN_SAVE_SPEC_CTRL BIT(1) -#define VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(2) - -#endif /* __KVM_X86_VMX_RUN_FLAGS_H */ diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c index 29a1f8e3be60..771c75a58343 100644 --- a/arch/x86/kvm/vmx/sgx.c +++ b/arch/x86/kvm/vmx/sgx.c @@ -7,7 +7,7 @@ #include <asm/sgx.h> #include "x86.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "nested.h" #include "sgx.h" #include "vmx.h" @@ -353,7 +353,7 @@ static int handle_encls_einit(struct kvm_vcpu *vcpu) rflags &= ~X86_EFLAGS_ZF; vmx_set_rflags(vcpu, rflags); - kvm_rax_write(vcpu, ret); + kvm_eax_write(vcpu, ret); return kvm_skip_emulated_instruction(vcpu); } @@ -381,7 +381,7 @@ static inline bool sgx_enabled_in_guest_bios(struct kvm_vcpu *vcpu) int handle_encls(struct kvm_vcpu *vcpu) { - u32 leaf = (u32)kvm_rax_read(vcpu); + u32 leaf = kvm_eax_read(vcpu); if (!enable_sgx || !guest_cpu_cap_has(vcpu, X86_FEATURE_SGX) || !guest_cpu_cap_has(vcpu, X86_FEATURE_SGX1)) { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 67d3c5c8aca0..989ab29b8c6f 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -343,7 +343,7 @@ static int tdx_reclaim_page(struct page *page) r = __tdx_reclaim_page(page); if (!r) - tdx_quirk_reset_page(page); + tdx_quirk_reset_paddr(page_to_phys(page), PAGE_SIZE); return r; } @@ -587,7 +587,7 @@ static void tdx_reclaim_td_control_pages(struct kvm *kvm) if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm)) return; - tdx_quirk_reset_page(kvm_tdx->td.tdr_page); + tdx_quirk_reset_paddr(page_to_phys(kvm_tdx->td.tdr_page), PAGE_SIZE); __free_page(kvm_tdx->td.tdr_page); kvm_tdx->td.tdr_page = NULL; @@ -629,6 +629,12 @@ int tdx_vm_init(struct kvm *kvm) kvm->arch.disabled_quirks |= KVM_X86_QUIRK_IGNORE_GUEST_PAT; /* + * PMU support is provided by the TDX-Module (if enabled for the VM). + * From KVM's perspective, the VM doesn't have a virtual PMU. + */ + kvm->arch.has_protected_pmu = true; + + /* * Because guest TD is protected, VMM can't parse the instruction in TD. * Instead, guest uses MMIO hypercall. For unmodified device driver, * #VE needs to be injected for MMIO and #VE handler in TD converts MMIO @@ -1003,23 +1009,23 @@ static fastpath_t tdx_exit_handlers_fastpath(struct kvm_vcpu *vcpu) return EXIT_FASTPATH_NONE; } -#define TDX_REGS_AVAIL_SET (BIT_ULL(VCPU_EXREG_EXIT_INFO_1) | \ - BIT_ULL(VCPU_EXREG_EXIT_INFO_2) | \ - BIT_ULL(VCPU_REGS_RAX) | \ - BIT_ULL(VCPU_REGS_RBX) | \ - BIT_ULL(VCPU_REGS_RCX) | \ - BIT_ULL(VCPU_REGS_RDX) | \ - BIT_ULL(VCPU_REGS_RBP) | \ - BIT_ULL(VCPU_REGS_RSI) | \ - BIT_ULL(VCPU_REGS_RDI) | \ - BIT_ULL(VCPU_REGS_R8) | \ - BIT_ULL(VCPU_REGS_R9) | \ - BIT_ULL(VCPU_REGS_R10) | \ - BIT_ULL(VCPU_REGS_R11) | \ - BIT_ULL(VCPU_REGS_R12) | \ - BIT_ULL(VCPU_REGS_R13) | \ - BIT_ULL(VCPU_REGS_R14) | \ - BIT_ULL(VCPU_REGS_R15)) +#define TDX_REGS_AVAIL_SET (BIT(VCPU_REG_EXIT_INFO_1) | \ + BIT(VCPU_REG_EXIT_INFO_2) | \ + BIT(VCPU_REGS_RAX) | \ + BIT(VCPU_REGS_RBX) | \ + BIT(VCPU_REGS_RCX) | \ + BIT(VCPU_REGS_RDX) | \ + BIT(VCPU_REGS_RBP) | \ + BIT(VCPU_REGS_RSI) | \ + BIT(VCPU_REGS_RDI) | \ + BIT(VCPU_REGS_R8) | \ + BIT(VCPU_REGS_R9) | \ + BIT(VCPU_REGS_R10) | \ + BIT(VCPU_REGS_R11) | \ + BIT(VCPU_REGS_R12) | \ + BIT(VCPU_REGS_R13) | \ + BIT(VCPU_REGS_R14) | \ + BIT(VCPU_REGS_R15)) static void tdx_load_host_xsave_state(struct kvm_vcpu *vcpu) { @@ -1088,7 +1094,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) tdx_load_host_xsave_state(vcpu); - vcpu->arch.regs_avail &= TDX_REGS_AVAIL_SET; + kvm_clear_available_registers(vcpu, ~TDX_REGS_AVAIL_SET); if (unlikely(tdx->vp_enter_ret == EXIT_REASON_EPT_MISCONFIG)) return EXIT_FASTPATH_NONE; @@ -1153,11 +1159,11 @@ static int complete_hypercall_exit(struct kvm_vcpu *vcpu) static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu) { - kvm_rax_write(vcpu, to_tdx(vcpu)->vp_enter_args.r10); - kvm_rbx_write(vcpu, to_tdx(vcpu)->vp_enter_args.r11); - kvm_rcx_write(vcpu, to_tdx(vcpu)->vp_enter_args.r12); - kvm_rdx_write(vcpu, to_tdx(vcpu)->vp_enter_args.r13); - kvm_rsi_write(vcpu, to_tdx(vcpu)->vp_enter_args.r14); + kvm_rax_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r10); + kvm_rbx_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r11); + kvm_rcx_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r12); + kvm_rdx_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r13); + kvm_rsi_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r14); return __kvm_emulate_hypercall(vcpu, 0, complete_hypercall_exit); } @@ -1172,12 +1178,22 @@ static void __tdx_map_gpa(struct vcpu_tdx *tdx); static int tdx_complete_vmcall_map_gpa(struct kvm_vcpu *vcpu) { + u64 hypercall_ret = READ_ONCE(vcpu->run->hypercall.ret); struct vcpu_tdx *tdx = to_tdx(vcpu); + long rc; - if (vcpu->run->hypercall.ret) { - tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_INVALID_OPERAND); - tdx->vp_enter_args.r11 = tdx->map_gpa_next; - return 1; + switch (hypercall_ret) { + case 0: + break; + case EAGAIN: + rc = TDVMCALL_STATUS_RETRY; + goto propagate_error; + case EINVAL: + rc = TDVMCALL_STATUS_INVALID_OPERAND; + goto propagate_error; + default: + WARN_ON_ONCE(kvm_is_valid_map_gpa_range_ret(hypercall_ret)); + return -EINVAL; } tdx->map_gpa_next += TDX_MAP_GPA_MAX_LEN; @@ -1190,13 +1206,17 @@ static int tdx_complete_vmcall_map_gpa(struct kvm_vcpu *vcpu) * TDVMCALL_MAP_GPA, see comments in tdx_protected_apic_has_interrupt(). */ if (kvm_vcpu_has_events(vcpu)) { - tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_RETRY); - tdx->vp_enter_args.r11 = tdx->map_gpa_next; - return 1; + rc = TDVMCALL_STATUS_RETRY; + goto propagate_error; } __tdx_map_gpa(tdx); return 0; + +propagate_error: + tdvmcall_set_return_code(vcpu, rc); + tdx->vp_enter_args.r11 = tdx->map_gpa_next; + return 1; } static void __tdx_map_gpa(struct vcpu_tdx *tdx) @@ -1614,8 +1634,8 @@ static int tdx_mem_page_add(struct kvm *kvm, gfn_t gfn, enum pg_level level, KVM_BUG_ON(!kvm_tdx->page_add_src, kvm)) return -EIO; - err = tdh_mem_page_add(&kvm_tdx->td, gpa, pfn_to_page(pfn), - kvm_tdx->page_add_src, &entry, &level_state); + err = tdh_mem_page_add(&kvm_tdx->td, gpa, pfn, kvm_tdx->page_add_src, + &entry, &level_state); if (unlikely(tdx_operand_busy(err))) return -EBUSY; @@ -1628,14 +1648,12 @@ static int tdx_mem_page_add(struct kvm *kvm, gfn_t gfn, enum pg_level level, static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) { - int tdx_level = pg_level_to_tdx_sept_level(level); struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm); - struct page *page = pfn_to_page(pfn); gpa_t gpa = gfn_to_gpa(gfn); u64 entry, level_state; u64 err; - err = tdh_mem_page_aug(&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); + err = tdh_mem_page_aug(&kvm_tdx->td, gpa, level, pfn, &entry, &level_state); if (unlikely(tdx_operand_busy(err))) return -EBUSY; @@ -1645,18 +1663,52 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, return 0; } -static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, - enum pg_level level, u64 mirror_spte) +static struct page *tdx_spte_to_sept_pt(struct kvm *kvm, gfn_t gfn, + u64 new_spte, enum pg_level level) +{ + struct kvm_mmu_page *sp = spte_to_child_sp(new_spte); + + if (KVM_BUG_ON(!sp->external_spt, kvm) || + KVM_BUG_ON(sp->role.level + 1 != level, kvm) || + KVM_BUG_ON(sp->gfn != gfn, kvm)) + return NULL; + + return virt_to_page(sp->external_spt); +} + +static int tdx_sept_map_nonleaf_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, u64 new_spte) +{ + gpa_t gpa = gfn_to_gpa(gfn); + u64 err, entry, level_state; + struct page *sept_pt; + + sept_pt = tdx_spte_to_sept_pt(kvm, gfn, new_spte, level); + if (!sept_pt) + return -EIO; + + err = tdh_mem_sept_add(&to_kvm_tdx(kvm)->td, gpa, level, sept_pt, + &entry, &level_state); + if (unlikely(tdx_operand_busy(err))) + return -EBUSY; + + if (TDX_BUG_ON_2(err, TDH_MEM_SEPT_ADD, entry, level_state, kvm)) + return -EIO; + + return 0; +} + +static int tdx_sept_map_leaf_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, + u64 new_spte) { struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm); - kvm_pfn_t pfn = spte_to_pfn(mirror_spte); + kvm_pfn_t pfn = spte_to_pfn(new_spte); /* TODO: handle large pages. */ if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm)) return -EIO; - WARN_ON_ONCE(!is_shadow_present_pte(mirror_spte) || - (mirror_spte & VMX_EPT_RWX_MASK) != VMX_EPT_RWX_MASK); + WARN_ON_ONCE((new_spte & VMX_EPT_RWX_MASK) != VMX_EPT_RWX_MASK); /* * Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() @@ -1676,25 +1728,6 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, return tdx_mem_page_aug(kvm, gfn, level, pfn); } -static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, - enum pg_level level, void *private_spt) -{ - int tdx_level = pg_level_to_tdx_sept_level(level); - gpa_t gpa = gfn_to_gpa(gfn); - struct page *page = virt_to_page(private_spt); - u64 err, entry, level_state; - - err = tdh_mem_sept_add(&to_kvm_tdx(kvm)->td, gpa, tdx_level, page, &entry, - &level_state); - if (unlikely(tdx_operand_busy(err))) - return -EBUSY; - - if (TDX_BUG_ON_2(err, TDH_MEM_SEPT_ADD, entry, level_state, kvm)) - return -EIO; - - return 0; -} - /* * Ensure shared and private EPTs to be flushed on all vCPUs. * tdh_mem_track() is the only caller that increases TD epoch. An increase in @@ -1741,35 +1774,11 @@ static void tdx_track(struct kvm *kvm) kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); } -static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, - enum pg_level level, void *private_spt) -{ - struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm); - - /* - * free_external_spt() is only called after hkid is freed when TD is - * tearing down. - * KVM doesn't (yet) zap page table pages in mirror page table while - * TD is active, though guest pages mapped in mirror page table could be - * zapped during TD is active, e.g. for shared <-> private conversion - * and slot move/deletion. - */ - if (KVM_BUG_ON(is_hkid_assigned(kvm_tdx), kvm)) - return -EIO; - - /* - * The HKID assigned to this TD was already freed and cache was - * already flushed. We don't have to flush again. - */ - return tdx_reclaim_page(virt_to_page(private_spt)); -} - -static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, - enum pg_level level, u64 mirror_spte) +static int tdx_sept_remove_leaf_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, u64 old_spte) { - struct page *page = pfn_to_page(spte_to_pfn(mirror_spte)); - int tdx_level = pg_level_to_tdx_sept_level(level); struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm); + kvm_pfn_t pfn = spte_to_pfn(old_spte); gpa_t gpa = gfn_to_gpa(gfn); u64 err, entry, level_state; @@ -1781,16 +1790,16 @@ static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, * there can't be anything populated in the private EPT. */ if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm)) - return; + return -EIO; /* TODO: handle large pages. */ if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm)) - return; + return -EIO; err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa, - tdx_level, &entry, &level_state); + level, &entry, &level_state); if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm)) - return; + return -EIO; /* * TDX requires TLB tracking before dropping private page. Do @@ -1804,15 +1813,82 @@ static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs. */ err = tdh_do_no_vcpus(tdh_mem_page_remove, kvm, &kvm_tdx->td, gpa, - tdx_level, &entry, &level_state); + level, &entry, &level_state); if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm)) - return; + return -EIO; - err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page); + err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, pfn); if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm)) - return; + return -EIO; - tdx_quirk_reset_page(page); + tdx_quirk_reset_paddr(PFN_PHYS(pfn), PAGE_SIZE); + return 0; +} + +/* + * Handle changes for + * (1) leaf SPTEs from non-present to present + * (2) non-leaf SPTEs from non-present to present + * (3) leaf SPTEs from present to non-present + * + * - (1) and (2) must be under shared mmu_lock. If (1) and (2) are under + * exclusive mmu_lock (currently impossible), contention errors may lead to + * KVM_BUG_ON() in handle_changed_spte(), e.g., due to tdx_mem_page_aug(), + * tdx_mem_page_add(), or tdh_mem_sept_add() contending with tdh_vp_enter() + * due to zero-step mitigation or contending with TDCALLs. + * - (3) must be under write mmu_lock. If (3) is under shared mmu_lock + * (currently impossible), warnings will be generated due to + * lockdep_assert_held_write() or TDX_BUG_ON() caused by concurrent BLOCK, + * TRACK, REMOVE. + * - Promotion/demotion is not yet supported. + */ +static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte, + u64 new_spte, enum pg_level level) +{ + lockdep_assert_held(&kvm->mmu_lock); + + if (is_shadow_present_pte(old_spte)) + return tdx_sept_remove_leaf_spte(kvm, gfn, level, old_spte); + + if (KVM_BUG_ON(!is_shadow_present_pte(new_spte), kvm)) + return -EIO; + + if (!is_last_spte(new_spte, level)) + return tdx_sept_map_nonleaf_spte(kvm, gfn, level, new_spte); + + return tdx_sept_map_leaf_spte(kvm, gfn, level, new_spte); +} + +/* + * Handle changes for non-leaf SPTEs from present to non-present. + * Must be under exclusive mmu_lock and cannot fail. + */ +static void tdx_sept_free_private_spt(struct kvm *kvm, struct kvm_mmu_page *sp) +{ + /* + * KVM doesn't (yet) zap page table pages in mirror page table while + * TD is active, though guest pages mapped in mirror page table could be + * zapped during TD is active, e.g. for shared <-> private conversion + * and slot move/deletion. + * + * In other words, KVM should only free mirror page tables after the + * TD's hkid is freed, when the TD is being torn down. + * + * If the S-EPT PTE can't be removed for any reason, intentionally leak + * the page to prevent the kernel from accessing the encrypted page. + */ + if (KVM_BUG_ON(is_hkid_assigned(to_kvm_tdx(kvm)), kvm) || + tdx_reclaim_page(virt_to_page(sp->external_spt))) + goto out; + + /* + * Immediately free the S-EPT page because RCU-time free is unnecessary + * after TDH.PHYMEM.PAGE.RECLAIM ensures there are no outstanding + * readers. + */ + free_page((unsigned long)sp->external_spt); +out: + sp->external_spt = NULL; } void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, @@ -1835,7 +1911,7 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp if (eeq_type != TDX_EXT_EXIT_QUAL_TYPE_PENDING_EPT_VIOLATION) return false; - return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN); + return !(eq & EPT_VIOLATION_PROT_MASK); } static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu) @@ -2021,12 +2097,12 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) case EXIT_REASON_IO_INSTRUCTION: return tdx_emulate_io(vcpu); case EXIT_REASON_MSR_READ: - kvm_rcx_write(vcpu, tdx->vp_enter_args.r12); + kvm_ecx_write(vcpu, tdx->vp_enter_args.r12); return kvm_emulate_rdmsr(vcpu); case EXIT_REASON_MSR_WRITE: - kvm_rcx_write(vcpu, tdx->vp_enter_args.r12); - kvm_rax_write(vcpu, tdx->vp_enter_args.r13 & -1u); - kvm_rdx_write(vcpu, tdx->vp_enter_args.r13 >> 32); + kvm_ecx_write(vcpu, tdx->vp_enter_args.r12); + kvm_eax_write(vcpu, tdx->vp_enter_args.r13); + kvm_edx_write(vcpu, tdx->vp_enter_args.r13 >> 32); return kvm_emulate_wrmsr(vcpu); case EXIT_REASON_EPT_MISCONFIG: return tdx_emulate_mmio(vcpu); @@ -2106,23 +2182,29 @@ bool tdx_has_emulated_msr(u32 index) case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: /* MSR_IA32_MCx_{CTL, STATUS, ADDR, MISC, CTL2} */ case MSR_KVM_POLL_CONTROL: + /* + * Except for x2APIC registers that are virtualized by the CPU, which + * KVM can't emulate as KVM doesn't have access to the virtual APIC + * page, KVM emulates the same set of x2APIC registers for TDX versus + * non-TDX guests. + */ + case X2APIC_MSR(APIC_ID): + case X2APIC_MSR(APIC_LVR): + case X2APIC_MSR(APIC_LDR): + case X2APIC_MSR(APIC_SPIV): + case X2APIC_MSR(APIC_ESR): + case X2APIC_MSR(APIC_LVTCMCI): + case X2APIC_MSR(APIC_ICR): + case X2APIC_MSR(APIC_LVTT): + case X2APIC_MSR(APIC_LVTTHMR): + case X2APIC_MSR(APIC_LVTPC): + case X2APIC_MSR(APIC_LVT0): + case X2APIC_MSR(APIC_LVT1): + case X2APIC_MSR(APIC_LVTERR): + case X2APIC_MSR(APIC_TMICT): + case X2APIC_MSR(APIC_TMCCT): + case X2APIC_MSR(APIC_TDCR): return true; - case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: - /* - * x2APIC registers that are virtualized by the CPU can't be - * emulated, KVM doesn't have access to the virtual APIC page. - */ - switch (index) { - case X2APIC_MSR(APIC_TASKPRI): - case X2APIC_MSR(APIC_PROCPRI): - case X2APIC_MSR(APIC_EOI): - case X2APIC_MSR(APIC_ISR) ... X2APIC_MSR(APIC_ISR + APIC_ISR_NR): - case X2APIC_MSR(APIC_TMR) ... X2APIC_MSR(APIC_TMR + APIC_ISR_NR): - case X2APIC_MSR(APIC_IRR) ... X2APIC_MSR(APIC_IRR + APIC_ISR_NR): - return false; - default: - return true; - } default: return false; } @@ -2377,20 +2459,20 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params, ret = -ENOMEM; - tdr_page = alloc_page(GFP_KERNEL); + tdr_page = alloc_page(GFP_KERNEL_ACCOUNT); if (!tdr_page) goto free_hkid; kvm_tdx->td.tdcs_nr_pages = tdx_sysinfo->td_ctrl.tdcs_base_size / PAGE_SIZE; /* TDVPS = TDVPR(4K page) + TDCX(multiple 4K pages), -1 for TDVPR. */ kvm_tdx->td.tdcx_nr_pages = tdx_sysinfo->td_ctrl.tdvps_base_size / PAGE_SIZE - 1; - tdcs_pages = kzalloc_objs(*kvm_tdx->td.tdcs_pages, - kvm_tdx->td.tdcs_nr_pages); + tdcs_pages = kzalloc_objs(*kvm_tdx->td.tdcs_pages, kvm_tdx->td.tdcs_nr_pages, + GFP_KERNEL_ACCOUNT); if (!tdcs_pages) goto free_tdr; for (i = 0; i < kvm_tdx->td.tdcs_nr_pages; i++) { - tdcs_pages[i] = alloc_page(GFP_KERNEL); + tdcs_pages[i] = alloc_page(GFP_KERNEL_ACCOUNT); if (!tdcs_pages[i]) goto free_tdcs; } @@ -2775,7 +2857,7 @@ void tdx_flush_tlb_current(struct kvm_vcpu *vcpu) void tdx_flush_tlb_all(struct kvm_vcpu *vcpu) { /* - * TDX has called tdx_track() in tdx_sept_remove_private_spte() to + * TDX has called tdx_track() in tdx_sept_remove_leaf_spte() to * ensure that private EPT will be flushed on the next TD enter. No need * to call tdx_track() here again even when this callback is a result of * zapping private EPT. @@ -2865,7 +2947,7 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx) int ret, i; u64 err; - page = alloc_page(GFP_KERNEL); + page = alloc_page(GFP_KERNEL_ACCOUNT); if (!page) return -ENOMEM; tdx->vp.tdvpr_page = page; @@ -2878,14 +2960,14 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx) tdx->vp.tdvpr_pa = page_to_phys(tdx->vp.tdvpr_page); tdx->vp.tdcx_pages = kcalloc(kvm_tdx->td.tdcx_nr_pages, sizeof(*tdx->vp.tdcx_pages), - GFP_KERNEL); + GFP_KERNEL_ACCOUNT); if (!tdx->vp.tdcx_pages) { ret = -ENOMEM; goto free_tdvpr; } for (i = 0; i < kvm_tdx->td.tdcx_nr_pages; i++) { - page = alloc_page(GFP_KERNEL); + page = alloc_page(GFP_KERNEL_ACCOUNT); if (!page) { ret = -ENOMEM; goto free_tdcx; @@ -3175,7 +3257,7 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c }; gmem_ret = kvm_gmem_populate(kvm, gpa_to_gfn(region.gpa), u64_to_user_ptr(region.source_addr), - 1, tdx_gmem_post_populate, &arg); + 1, false, tdx_gmem_post_populate, &arg); if (gmem_ret < 0) { ret = gmem_ret; break; @@ -3405,10 +3487,8 @@ int __init tdx_hardware_setup(void) vt_x86_ops.vm_size = max_t(unsigned int, vt_x86_ops.vm_size, sizeof(struct kvm_tdx)); - vt_x86_ops.link_external_spt = tdx_sept_link_private_spt; vt_x86_ops.set_external_spte = tdx_sept_set_private_spte; vt_x86_ops.free_external_spt = tdx_sept_free_private_spt; - vt_x86_ops.remove_external_spte = tdx_sept_remove_private_spte; vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; return 0; diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index ff1f254a0ef4..00d807505fc8 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -2,44 +2,20 @@ #include <linux/linkage.h> #include <asm/asm.h> #include <asm/bitsperlong.h> -#include <asm/kvm_vcpu_regs.h> #include <asm/nospec-branch.h> #include <asm/percpu.h> #include <asm/segment.h> #include "kvm-asm-offsets.h" -#include "run_flags.h" - -#define WORD_SIZE (BITS_PER_LONG / 8) - -#define VCPU_RAX __VCPU_REGS_RAX * WORD_SIZE -#define VCPU_RCX __VCPU_REGS_RCX * WORD_SIZE -#define VCPU_RDX __VCPU_REGS_RDX * WORD_SIZE -#define VCPU_RBX __VCPU_REGS_RBX * WORD_SIZE -/* Intentionally omit RSP as it's context switched by hardware */ -#define VCPU_RBP __VCPU_REGS_RBP * WORD_SIZE -#define VCPU_RSI __VCPU_REGS_RSI * WORD_SIZE -#define VCPU_RDI __VCPU_REGS_RDI * WORD_SIZE - -#ifdef CONFIG_X86_64 -#define VCPU_R8 __VCPU_REGS_R8 * WORD_SIZE -#define VCPU_R9 __VCPU_REGS_R9 * WORD_SIZE -#define VCPU_R10 __VCPU_REGS_R10 * WORD_SIZE -#define VCPU_R11 __VCPU_REGS_R11 * WORD_SIZE -#define VCPU_R12 __VCPU_REGS_R12 * WORD_SIZE -#define VCPU_R13 __VCPU_REGS_R13 * WORD_SIZE -#define VCPU_R14 __VCPU_REGS_R14 * WORD_SIZE -#define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE -#endif +#include "vmenter.h" .section .noinstr.text, "ax" /** * __vmx_vcpu_run - Run a vCPU via a transition to VMX guest mode * @vmx: struct vcpu_vmx * - * @regs: unsigned long * (to guest registers) - * @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH - * VMX_RUN_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl - * VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO: vCPU can access host MMIO + * @flags: KVM_ENTER_VMRESUME: use VMRESUME instead of VMLAUNCH + * KVM_ENTER_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl + * KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO: vCPU can access host MMIO * * Returns: * 0 on VM-Exit, 1 on VM-Fail @@ -62,76 +38,39 @@ SYM_FUNC_START(__vmx_vcpu_run) push %_ASM_ARG1 /* Save @flags (used for VMLAUNCH vs. VMRESUME and mitigations). */ - push %_ASM_ARG3 - - /* - * Save @regs, _ASM_ARG2 may be modified by vmx_update_host_rsp() and - * @regs is needed after VM-Exit to save the guest's register values. - */ push %_ASM_ARG2 lea (%_ASM_SP), %_ASM_ARG2 call vmx_update_host_rsp - ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL + /* Reload @vmx, _ASM_ARG1 may be modified by vmx_update_host_rsp(). */ + mov WORD_SIZE(%_ASM_SP), %_ASM_DI /* - * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the - * host's, write the MSR. - * - * IMPORTANT: To avoid RSB underflow attacks and any other nastiness, - * there must not be any returns or indirect branches between this code - * and vmentry. + * Unlike AMD there's no V_SPEC_CTRL here, so do not leave the body + * out of line. Clobbers RAX, RCX, RDX, RSI. */ - mov 2*WORD_SIZE(%_ASM_SP), %_ASM_DI -#ifdef CONFIG_X86_64 - mov VMX_spec_ctrl(%rdi), %rdx - cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx - je .Lspec_ctrl_done - movl %edx, %eax - shr $32, %rdx -#else - mov VMX_spec_ctrl(%edi), %eax - mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx - xor %eax, %ecx - mov VMX_spec_ctrl + 4(%edi), %edx - mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edi - xor %edx, %edi - or %edi, %ecx - je .Lspec_ctrl_done -#endif - mov $MSR_IA32_SPEC_CTRL, %ecx - wrmsr - -.Lspec_ctrl_done: + ALTERNATIVE "jmp .Lspec_ctrl_guest_done", "", X86_FEATURE_MSR_SPEC_CTRL + RESTORE_GUEST_SPEC_CTRL_BODY VMX_spec_ctrl(%_ASM_DI), .Lspec_ctrl_guest_done +.Lspec_ctrl_guest_done: /* * Since vmentry is serializing on affected CPUs, there's no need for * an LFENCE to stop speculation from skipping the wrmsr. */ - /* Load @regs to RAX. */ - mov (%_ASM_SP), %_ASM_AX - - /* Load guest registers. Don't clobber flags. */ - mov VCPU_RCX(%_ASM_AX), %_ASM_CX - mov VCPU_RDX(%_ASM_AX), %_ASM_DX - mov VCPU_RBX(%_ASM_AX), %_ASM_BX - mov VCPU_RBP(%_ASM_AX), %_ASM_BP - mov VCPU_RSI(%_ASM_AX), %_ASM_SI - mov VCPU_RDI(%_ASM_AX), %_ASM_DI + /* + * Load guest registers. Don't clobber flags. Intentionally omit + * %_ASM_SP as it's context switched by hardware + */ + LOAD_REGS %_ASM_DI, VMX_vcpu_arch_regs, \ + %_ASM_AX, %_ASM_CX, %_ASM_DX, %_ASM_BX, %_ASM_BP, %_ASM_SI #ifdef CONFIG_X86_64 - mov VCPU_R8 (%_ASM_AX), %r8 - mov VCPU_R9 (%_ASM_AX), %r9 - mov VCPU_R10(%_ASM_AX), %r10 - mov VCPU_R11(%_ASM_AX), %r11 - mov VCPU_R12(%_ASM_AX), %r12 - mov VCPU_R13(%_ASM_AX), %r13 - mov VCPU_R14(%_ASM_AX), %r14 - mov VCPU_R15(%_ASM_AX), %r15 + LOAD_REGS %_ASM_DI, VMX_vcpu_arch_regs, \ + %r8, %r9, %r10, %r11, %r12, %r13, %r14, %r15 #endif - /* Load guest RAX. This kills the @regs pointer! */ - mov VCPU_RAX(%_ASM_AX), %_ASM_AX + /* Load guest RDI. This kills the @vmx pointer! */ + LOAD_REGS %_ASM_DI, VMX_vcpu_arch_regs, %_ASM_DI /* * Note, ALTERNATIVE_2 works in reverse order. If CLEAR_CPU_BUF_VM is @@ -140,7 +79,7 @@ SYM_FUNC_START(__vmx_vcpu_run) * do VERW. Else, do nothing (no mitigations needed/enabled). */ ALTERNATIVE_2 "", \ - __stringify(testl $VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO, WORD_SIZE(%_ASM_SP); \ + __stringify(testl $KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO, (%_ASM_SP); \ jz .Lskip_mmio_verw; \ VERW; \ .Lskip_mmio_verw:), \ @@ -148,7 +87,7 @@ SYM_FUNC_START(__vmx_vcpu_run) __stringify(VERW), X86_FEATURE_CLEAR_CPU_BUF_VM /* Check @flags to see if VMLAUNCH or VMRESUME is needed. */ - testl $VMX_RUN_VMRESUME, WORD_SIZE(%_ASM_SP) + testl $KVM_ENTER_VMRESUME, (%_ASM_SP) jz .Lvmlaunch /* @@ -180,38 +119,28 @@ SYM_INNER_LABEL_ALIGN(vmx_vmexit, SYM_L_GLOBAL) UNWIND_HINT_RESTORE ENDBR - /* Temporarily save guest's RAX. */ - push %_ASM_AX + /* Temporarily save guest's RDI. */ + push %_ASM_DI - /* Reload @regs to RAX. */ - mov WORD_SIZE(%_ASM_SP), %_ASM_AX - - /* Save all guest registers, including RAX from the stack */ - pop VCPU_RAX(%_ASM_AX) - mov %_ASM_CX, VCPU_RCX(%_ASM_AX) - mov %_ASM_DX, VCPU_RDX(%_ASM_AX) - mov %_ASM_BX, VCPU_RBX(%_ASM_AX) - mov %_ASM_BP, VCPU_RBP(%_ASM_AX) - mov %_ASM_SI, VCPU_RSI(%_ASM_AX) - mov %_ASM_DI, VCPU_RDI(%_ASM_AX) + /* Reload @vmx to RDI. */ + mov 2*WORD_SIZE(%_ASM_SP), %_ASM_DI + + /* + * Save all guest registers, including RDI from the stack. Intentionally + * omit %_ASM_SP as it's context switched by hardware + */ + STORE_REGS %_ASM_DI, VMX_vcpu_arch_regs, \ + %_ASM_AX, %_ASM_CX, %_ASM_DX, %_ASM_BX, %_ASM_BP, %_ASM_SI + POP_REGS %_ASM_DI, VMX_vcpu_arch_regs, %_ASM_DI #ifdef CONFIG_X86_64 - mov %r8, VCPU_R8 (%_ASM_AX) - mov %r9, VCPU_R9 (%_ASM_AX) - mov %r10, VCPU_R10(%_ASM_AX) - mov %r11, VCPU_R11(%_ASM_AX) - mov %r12, VCPU_R12(%_ASM_AX) - mov %r13, VCPU_R13(%_ASM_AX) - mov %r14, VCPU_R14(%_ASM_AX) - mov %r15, VCPU_R15(%_ASM_AX) + STORE_REGS %_ASM_DI, VMX_vcpu_arch_regs, \ + %r8, %r9, %r10, %r11, %r12, %r13, %r14, %r15 #endif /* Clear return value to indicate VM-Exit (as opposed to VM-Fail). */ xor %ebx, %ebx .Lclear_regs: - /* Discard @regs. The register is irrelevant, it just can't be RBX. */ - pop %_ASM_AX - /* * Clear all general purpose registers except RSP and RBX to prevent * speculative use of the guest's values, even those that are reloaded @@ -222,21 +151,9 @@ SYM_INNER_LABEL_ALIGN(vmx_vmexit, SYM_L_GLOBAL) * VM-Exit and RBX is explicitly loaded with 0 or 1 to hold the return * value. */ - xor %eax, %eax - xor %ecx, %ecx - xor %edx, %edx - xor %ebp, %ebp - xor %esi, %esi - xor %edi, %edi + CLEAR_REGS %eax, %ecx, %edx, %ebp, %esi, %edi #ifdef CONFIG_X86_64 - xor %r8d, %r8d - xor %r9d, %r9d - xor %r10d, %r10d - xor %r11d, %r11d - xor %r12d, %r12d - xor %r13d, %r13d - xor %r14d, %r14d - xor %r15d, %r15d + CLEAR_REGS %r8d, %r9d, %r10d, %r11d, %r12d, %r13d, %r14d, %r15d #endif /* @@ -254,16 +171,32 @@ SYM_INNER_LABEL_ALIGN(vmx_vmexit, SYM_L_GLOBAL) FILL_RETURN_BUFFER %_ASM_CX, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT,\ X86_FEATURE_RSB_VMEXIT_LITE - pop %_ASM_ARG2 /* @flags */ - pop %_ASM_ARG1 /* @vmx */ + /* Clobbers RAX, RCX, RDX, RSI. */ + ALTERNATIVE "jmp .Lspec_ctrl_host_done", "", X86_FEATURE_MSR_SPEC_CTRL + mov WORD_SIZE(%_ASM_SP), %_ASM_DI + RESTORE_HOST_SPEC_CTRL_BODY VMX_spec_ctrl(%_ASM_DI), (%_ASM_SP), .Lspec_ctrl_host_done +.Lspec_ctrl_host_done: - call vmx_spec_ctrl_restore_host + /* + * Halt speculation past a conditional wrmsr. Intel's eIBRS + * guarantees that the guest cannot control the RSB "once IBRS is + * set", but in the eIBRS case speculative execution past the 'je' + * can go all the way to the RET below while MSR_IA32_SPEC_CTRL + * still holds the guest value. + */ + ALTERNATIVE_2 "", "lfence", X86_FEATURE_MSR_SPEC_CTRL, \ + "", X86_FEATURE_KERNEL_IBRS CLEAR_BRANCH_HISTORY_VMEXIT /* Put return value in AX */ mov %_ASM_BX, %_ASM_AX + /* Pop our saved arguments from the stack */ + pop %_ASM_BX + pop %_ASM_BX + + /* ... and then the callee-save registers */ pop %_ASM_BX #ifdef CONFIG_X86_64 pop %r12 diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index ede773ce065a..f7c2983c533f 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -60,7 +60,7 @@ #include "hyperv.h" #include "kvm_onhyperv.h" #include "irq.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "lapic.h" #include "mmu.h" #include "nested.h" @@ -74,6 +74,7 @@ #include "x86_ops.h" #include "smm.h" #include "vmx_onhyperv.h" +#include "vmenter.h" #include "posted_intr.h" #include "mmu/spte.h" @@ -118,6 +119,9 @@ module_param(emulate_invalid_guest_state, bool, 0444); static bool __read_mostly fasteoi = 1; module_param(fasteoi, bool, 0444); +bool __read_mostly enable_mbec = 1; +module_param_named(mbec, enable_mbec, bool, 0444); + module_param(enable_apicv, bool, 0444); module_param(enable_ipiv, bool, 0444); @@ -847,8 +851,8 @@ static bool vmx_segment_cache_test_set(struct vcpu_vmx *vmx, unsigned seg, bool ret; u32 mask = 1 << (seg * SEG_FIELD_NR + field); - if (!kvm_register_is_available(&vmx->vcpu, VCPU_EXREG_SEGMENTS)) { - kvm_register_mark_available(&vmx->vcpu, VCPU_EXREG_SEGMENTS); + if (!kvm_register_is_available(&vmx->vcpu, VCPU_REG_SEGMENTS)) { + kvm_register_mark_available(&vmx->vcpu, VCPU_REG_SEGMENTS); vmx->segment_cache.bitmask = 0; } ret = vmx->segment_cache.bitmask & mask; @@ -968,12 +972,12 @@ static bool msr_write_intercepted(struct vcpu_vmx *vmx, u32 msr) return vmx_test_msr_bitmap_write(vmx->loaded_vmcs->msr_bitmap, msr); } -unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx) +unsigned int __vmx_vcpu_enter_flags(struct vcpu_vmx *vmx) { unsigned int flags = 0; if (vmx->loaded_vmcs->launched) - flags |= VMX_RUN_VMRESUME; + flags |= KVM_ENTER_VMRESUME; /* * If writes to the SPEC_CTRL MSR aren't intercepted, the guest is free @@ -981,11 +985,11 @@ unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx) * it after vmexit and store it in vmx->spec_ctrl. */ if (!msr_write_intercepted(vmx, MSR_IA32_SPEC_CTRL)) - flags |= VMX_RUN_SAVE_SPEC_CTRL; + flags |= KVM_ENTER_SAVE_SPEC_CTRL; if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO) && kvm_vcpu_can_access_host_mmio(&vmx->vcpu)) - flags |= VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO; + flags |= KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO; return flags; } @@ -1613,8 +1617,8 @@ unsigned long vmx_get_rflags(struct kvm_vcpu *vcpu) struct vcpu_vmx *vmx = to_vmx(vcpu); unsigned long rflags, save_rflags; - if (!kvm_register_is_available(vcpu, VCPU_EXREG_RFLAGS)) { - kvm_register_mark_available(vcpu, VCPU_EXREG_RFLAGS); + if (!kvm_register_is_available(vcpu, VCPU_REG_RFLAGS)) { + kvm_register_mark_available(vcpu, VCPU_REG_RFLAGS); rflags = vmcs_readl(GUEST_RFLAGS); if (vmx->rmode.vm86_active) { rflags &= RMODE_GUEST_OWNED_EFLAGS_BITS; @@ -1637,7 +1641,7 @@ void vmx_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) * if L1 runs L2 as a restricted guest. */ if (is_unrestricted_guest(vcpu)) { - kvm_register_mark_available(vcpu, VCPU_EXREG_RFLAGS); + kvm_register_mark_available(vcpu, VCPU_REG_RFLAGS); vmx->rflags = rflags; vmcs_writel(GUEST_RFLAGS, rflags); return; @@ -1909,6 +1913,24 @@ void vmx_inject_exception(struct kvm_vcpu *vcpu) u32 intr_info = ex->vector | INTR_INFO_VALID_MASK; struct vcpu_vmx *vmx = to_vmx(vcpu); + /* + * When injecting a #DB, single-stepping is enabled in RFLAGS, and STI + * or MOV-SS blocking is active, set vmcs.PENDING_DBG_EXCEPTIONS.BS to + * prevent a false positive from VM-Entry consistency check. VM-Entry + * asserts that a single-step #DB _must_ be pending in this scenario, + * as the previous instruction cannot have toggled RFLAGS.TF 0=>1 + * (because STI and POP/MOV don't modify RFLAGS), therefore the one + * instruction delay when activating single-step breakpoints must have + * already expired. However, the CPU isn't smart enough to peek at + * vmcs.VM_ENTRY_INTR_INFO_FIELD and so doesn't realize that yes, there + * is indeed a #DB pending/imminent. + */ + if (ex->vector == DB_VECTOR && + (vmx_get_rflags(vcpu) & X86_EFLAGS_TF) && + vmx_get_interrupt_shadow(vcpu)) + vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS, + vmcs_readl(GUEST_PENDING_DBG_EXCEPTIONS) | DR6_BS); + kvm_deliver_exception_payload(vcpu, ex); if (ex->has_error_code) { @@ -2608,20 +2630,20 @@ void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) case VCPU_REGS_RSP: vcpu->arch.regs[VCPU_REGS_RSP] = vmcs_readl(GUEST_RSP); break; - case VCPU_REGS_RIP: - vcpu->arch.regs[VCPU_REGS_RIP] = vmcs_readl(GUEST_RIP); + case VCPU_REG_RIP: + vcpu->arch.rip = vmcs_readl(GUEST_RIP); break; - case VCPU_EXREG_PDPTR: + case VCPU_REG_PDPTR: if (enable_ept) ept_save_pdptrs(vcpu); break; - case VCPU_EXREG_CR0: + case VCPU_REG_CR0: guest_owned_bits = vcpu->arch.cr0_guest_owned_bits; vcpu->arch.cr0 &= ~guest_owned_bits; vcpu->arch.cr0 |= vmcs_readl(GUEST_CR0) & guest_owned_bits; break; - case VCPU_EXREG_CR3: + case VCPU_REG_CR3: /* * When intercepting CR3 loads, e.g. for shadowing paging, KVM's * CR3 is loaded into hardware, not the guest's CR3. @@ -2629,7 +2651,7 @@ void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) if (!(exec_controls_get(to_vmx(vcpu)) & CPU_BASED_CR3_LOAD_EXITING)) vcpu->arch.cr3 = vmcs_readl(GUEST_CR3); break; - case VCPU_EXREG_CR4: + case VCPU_REG_CR4: guest_owned_bits = vcpu->arch.cr4_guest_owned_bits; vcpu->arch.cr4 &= ~guest_owned_bits; @@ -2777,6 +2799,7 @@ static int setup_vmcs_config(struct vmcs_config *vmcs_conf, return -EIO; vmx_cap->ept = 0; + _cpu_based_2nd_exec_control &= ~SECONDARY_EXEC_MODE_BASED_EPT_EXEC; _cpu_based_2nd_exec_control &= ~SECONDARY_EXEC_EPT_VIOLATION_VE; } if (!(_cpu_based_2nd_exec_control & SECONDARY_EXEC_ENABLE_VPID) && @@ -2790,6 +2813,16 @@ static int setup_vmcs_config(struct vmcs_config *vmcs_conf, vmx_cap->vpid = 0; } + /* + * Virtualizing MBEC requires advanced vmexit information in order to + * distinguish supervisor and user accesses. For simplicity and clarity + * disable MBEC entirely if advanced vmexit information is not available, + * this way mbec=1 in the kvm_intel module parameters implies availability + * to nested guests as well. + */ + if (!(vmx_cap->ept & VMX_EPT_ADVANCED_VMEXIT_INFO_BIT)) + _cpu_based_2nd_exec_control &= ~SECONDARY_EXEC_MODE_BASED_EPT_EXEC; + if (!cpu_has_sgx()) _cpu_based_2nd_exec_control &= ~SECONDARY_EXEC_ENCLS_EXITING; @@ -3352,32 +3385,28 @@ void vmx_flush_tlb_guest(struct kvm_vcpu *vcpu) void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; - - if (!kvm_register_is_dirty(vcpu, VCPU_EXREG_PDPTR)) + if (!kvm_register_is_dirty(vcpu, VCPU_REG_PDPTR)) return; if (is_pae_paging(vcpu)) { - vmcs_write64(GUEST_PDPTR0, mmu->pdptrs[0]); - vmcs_write64(GUEST_PDPTR1, mmu->pdptrs[1]); - vmcs_write64(GUEST_PDPTR2, mmu->pdptrs[2]); - vmcs_write64(GUEST_PDPTR3, mmu->pdptrs[3]); + vmcs_write64(GUEST_PDPTR0, vcpu->arch.pdptrs[0]); + vmcs_write64(GUEST_PDPTR1, vcpu->arch.pdptrs[1]); + vmcs_write64(GUEST_PDPTR2, vcpu->arch.pdptrs[2]); + vmcs_write64(GUEST_PDPTR3, vcpu->arch.pdptrs[3]); } } void ept_save_pdptrs(struct kvm_vcpu *vcpu) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; - if (WARN_ON_ONCE(!is_pae_paging(vcpu))) return; - mmu->pdptrs[0] = vmcs_read64(GUEST_PDPTR0); - mmu->pdptrs[1] = vmcs_read64(GUEST_PDPTR1); - mmu->pdptrs[2] = vmcs_read64(GUEST_PDPTR2); - mmu->pdptrs[3] = vmcs_read64(GUEST_PDPTR3); + vcpu->arch.pdptrs[0] = vmcs_read64(GUEST_PDPTR0); + vcpu->arch.pdptrs[1] = vmcs_read64(GUEST_PDPTR1); + vcpu->arch.pdptrs[2] = vmcs_read64(GUEST_PDPTR2); + vcpu->arch.pdptrs[3] = vmcs_read64(GUEST_PDPTR3); - kvm_register_mark_available(vcpu, VCPU_EXREG_PDPTR); + kvm_register_mark_available(vcpu, VCPU_REG_PDPTR); } #define CR3_EXITING_BITS (CPU_BASED_CR3_LOAD_EXITING | \ @@ -3420,7 +3449,7 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) vmcs_writel(CR0_READ_SHADOW, cr0); vmcs_writel(GUEST_CR0, hw_cr0); vcpu->arch.cr0 = cr0; - kvm_register_mark_available(vcpu, VCPU_EXREG_CR0); + kvm_register_mark_available(vcpu, VCPU_REG_CR0); #ifdef CONFIG_X86_64 if (vcpu->arch.efer & EFER_LME) { @@ -3438,8 +3467,8 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) * (correctly) stop reading vmcs.GUEST_CR3 because it thinks * KVM's CR3 is installed. */ - if (!kvm_register_is_available(vcpu, VCPU_EXREG_CR3)) - vmx_cache_reg(vcpu, VCPU_EXREG_CR3); + if (!kvm_register_is_available(vcpu, VCPU_REG_CR3)) + vmx_cache_reg(vcpu, VCPU_REG_CR3); /* * When running with EPT but not unrestricted guest, KVM must @@ -3476,7 +3505,7 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) * GUEST_CR3 is still vmx->ept_identity_map_addr if EPT + !URG. */ if (!(old_cr0_pg & X86_CR0_PG) && (cr0 & X86_CR0_PG)) - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); } /* depends on vcpu->arch.cr0 to be set to a new value */ @@ -3505,7 +3534,7 @@ void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level) if (!enable_unrestricted_guest && !is_paging(vcpu)) guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr; - else if (kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3)) + else if (kvm_register_is_dirty(vcpu, VCPU_REG_CR3)) guest_cr3 = vcpu->arch.cr3; else /* vmcs.GUEST_CR3 is already up-to-date. */ update_guest_cr3 = false; @@ -3565,7 +3594,7 @@ void vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) } vcpu->arch.cr4 = cr4; - kvm_register_mark_available(vcpu, VCPU_EXREG_CR4); + kvm_register_mark_available(vcpu, VCPU_REG_CR4); if (!enable_unrestricted_guest) { if (enable_ept) { @@ -4142,7 +4171,7 @@ static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu) * mode, only the current timer count needs on-demand emulation by KVM. */ if (mode & MSR_BITMAP_MODE_X2APIC_APICV) - msr_bitmap[read_idx] = ~kvm_lapic_readable_reg_mask(vcpu->arch.apic); + msr_bitmap[read_idx] = ~kvm_x2apic_disable_read_intercept_reg_mask(vcpu); else msr_bitmap[read_idx] = ~0ull; msr_bitmap[write_idx] = ~0ull; @@ -4155,7 +4184,6 @@ static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu) !(mode & MSR_BITMAP_MODE_X2APIC)); if (mode & MSR_BITMAP_MODE_X2APIC_APICV) { - vmx_enable_intercept_for_msr(vcpu, X2APIC_MSR(APIC_TMCCT), MSR_TYPE_RW); vmx_disable_intercept_for_msr(vcpu, X2APIC_MSR(APIC_EOI), MSR_TYPE_W); vmx_disable_intercept_for_msr(vcpu, X2APIC_MSR(APIC_SELF_IPI), MSR_TYPE_W); if (enable_ipiv) @@ -4746,6 +4774,9 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx *vmx) */ exec_control &= ~SECONDARY_EXEC_ENABLE_VMFUNC; + if (!enable_mbec) + exec_control &= ~SECONDARY_EXEC_MODE_BASED_EPT_EXEC; + /* SECONDARY_EXEC_DESC is enabled/disabled on writes to CR4.UMIP, * in vmx_set_cr4. */ exec_control &= ~SECONDARY_EXEC_DESC; @@ -5032,7 +5063,7 @@ void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) vmcs_write32(GUEST_IDTR_LIMIT, 0xffff); vmx_segment_cache_clear(vmx); - kvm_register_mark_available(vcpu, VCPU_EXREG_SEGMENTS); + kvm_register_mark_available(vcpu, VCPU_REG_SEGMENTS); vmcs_write32(GUEST_ACTIVITY_STATE, GUEST_ACTIVITY_ACTIVE); vmcs_write32(GUEST_INTERRUPTIBILITY_INFO, 0); @@ -5478,26 +5509,9 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu) * avoid single-step #DB and MTF updates, as ICEBP is * higher priority. Note, skipping ICEBP still clears * STI and MOVSS blocking. - * - * For all other #DBs, set vmcs.PENDING_DBG_EXCEPTIONS.BS - * if single-step is enabled in RFLAGS and STI or MOVSS - * blocking is active, as the CPU doesn't set the bit - * on VM-Exit due to #DB interception. VM-Entry has a - * consistency check that a single-step #DB is pending - * in this scenario as the previous instruction cannot - * have toggled RFLAGS.TF 0=>1 (because STI and POP/MOV - * don't modify RFLAGS), therefore the one instruction - * delay when activating single-step breakpoints must - * have already expired. Note, the CPU sets/clears BS - * as appropriate for all other VM-Exits types. */ if (is_icebp(intr_info)) WARN_ON(!skip_emulated_instruction(vcpu)); - else if ((vmx_get_rflags(vcpu) & X86_EFLAGS_TF) && - (vmcs_read32(GUEST_INTERRUPTIBILITY_INFO) & - (GUEST_INTR_STATE_STI | GUEST_INTR_STATE_MOV_SS))) - vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS, - vmcs_readl(GUEST_PENDING_DBG_EXCEPTIONS) | DR6_BS); kvm_queue_exception_p(vcpu, DB_VECTOR, dr6); return 1; @@ -6698,6 +6712,9 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) if (enable_pml && !is_guest_mode(vcpu)) vmx_flush_pml_buffer(vcpu); + if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE)) + return 0; + /* * KVM should never reach this point with a pending nested VM-Enter. * More specifically, short-circuiting VM-Entry to emulate L2 due to @@ -7410,31 +7427,6 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) } } -void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, - unsigned int flags) -{ - u64 hostval = this_cpu_read(x86_spec_ctrl_current); - - if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL)) - return; - - if (flags & VMX_RUN_SAVE_SPEC_CTRL) - vmx->spec_ctrl = native_rdmsrq(MSR_IA32_SPEC_CTRL); - - /* - * If the guest/host SPEC_CTRL values differ, restore the host value. - * - * For legacy IBRS, the IBRS bit always needs to be written after - * transitioning from a less privileged predictor mode, regardless of - * whether the guest/host values differ. - */ - if (cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS) || - vmx->spec_ctrl != hostval) - native_wrmsrq(MSR_IA32_SPEC_CTRL, hostval); - - barrier_nospec(); -} - static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu, bool force_immediate_exit) { @@ -7488,11 +7480,10 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, if (vcpu->arch.cr2 != native_read_cr2()) native_write_cr2(vcpu->arch.cr2); - vmx->fail = __vmx_vcpu_run(vmx, (unsigned long *)&vcpu->arch.regs, - flags); + vmx->fail = __vmx_vcpu_run(vmx, flags); vcpu->arch.cr2 = native_read_cr2(); - vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET; + kvm_clear_available_registers(vcpu, VMX_REGS_LAZY_LOAD_SET); vmx->idt_vectoring_info = 0; @@ -7534,9 +7525,9 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) vmx->vt.exit_reason.full = EXIT_REASON_INVALID_STATE; vmx->vt.exit_reason.failed_vmentry = 1; - kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1); + kvm_register_mark_available(vcpu, VCPU_REG_EXIT_INFO_1); vmx->vt.exit_qualification = ENTRY_FAIL_DEFAULT; - kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2); + kvm_register_mark_available(vcpu, VCPU_REG_EXIT_INFO_2); vmx->vt.exit_intr_info = 0; return EXIT_FASTPATH_NONE; } @@ -7556,9 +7547,9 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) if (kvm_register_is_dirty(vcpu, VCPU_REGS_RSP)) vmcs_writel(GUEST_RSP, vcpu->arch.regs[VCPU_REGS_RSP]); - if (kvm_register_is_dirty(vcpu, VCPU_REGS_RIP)) - vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]); - vcpu->arch.regs_dirty = 0; + if (kvm_register_is_dirty(vcpu, VCPU_REG_RIP)) + vmcs_writel(GUEST_RIP, vcpu->arch.rip); + kvm_reset_dirty_registers(vcpu); if (run_flags & KVM_RUN_LOAD_GUEST_DR6) set_debugreg(vcpu->arch.dr6, 6); @@ -7607,7 +7598,7 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) kvm_wait_lapic_expire(vcpu); /* The actual VMENTER/EXIT is in the .noinstr.text section. */ - vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_run_flags(vmx)); + vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_enter_flags(vmx)); /* All fields are clean at this point */ if (kvm_is_using_evmcs()) { @@ -8645,6 +8636,8 @@ __init int vmx_hardware_setup(void) if (!cpu_has_vmx_ept_ad_bits() || !enable_ept) enable_ept_ad_bits = 0; + if (!cpu_has_ept_mbec() || !enable_ept) + enable_mbec = 0; if (!cpu_has_vmx_unrestricted_guest() || !enable_ept) enable_unrestricted_guest = 0; @@ -8706,8 +8699,7 @@ __init int vmx_hardware_setup(void) set_bit(0, vmx_vpid_bitmap); /* 0 is reserved for host */ if (enable_ept) - kvm_mmu_set_ept_masks(enable_ept_ad_bits, - cpu_has_vmx_ept_execute_only()); + kvm_mmu_set_ept_masks(enable_ept_ad_bits); else vt_x86_ops.get_mt_mask = NULL; diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index db84e8001da5..de9de0d2016c 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -10,12 +10,11 @@ #include <asm/posted_intr.h> #include "capabilities.h" -#include "../kvm_cache_regs.h" +#include "../regs.h" #include "pmu_intel.h" #include "vmcs.h" #include "vmx_ops.h" #include "../cpuid.h" -#include "run_flags.h" #include "../mmu.h" #include "common.h" @@ -317,7 +316,7 @@ static __always_inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu) { struct vcpu_vt *vt = to_vt(vcpu); - if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1) && + if (!kvm_register_test_and_mark_available(vcpu, VCPU_REG_EXIT_INFO_1) && !WARN_ON_ONCE(is_td_vcpu(vcpu))) vt->exit_qualification = vmcs_readl(EXIT_QUALIFICATION); @@ -328,7 +327,7 @@ static __always_inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu) { struct vcpu_vt *vt = to_vt(vcpu); - if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2) && + if (!kvm_register_test_and_mark_available(vcpu, VCPU_REG_EXIT_INFO_2) && !WARN_ON_ONCE(is_td_vcpu(vcpu))) vt->exit_intr_info = vmcs_read32(VM_EXIT_INTR_INFO); @@ -368,10 +367,8 @@ void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu); struct vmx_uret_msr *vmx_find_uret_msr(struct vcpu_vmx *vmx, u32 msr); void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu); void vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp); -void vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, unsigned int flags); -unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx); -bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned long *regs, - unsigned int flags); +unsigned int __vmx_vcpu_enter_flags(struct vcpu_vmx *vmx); +bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned int flags); void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu); void vmx_set_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type, bool set); @@ -567,6 +564,7 @@ static inline u8 vmx_get_rvi(void) SECONDARY_EXEC_ENABLE_VMFUNC | \ SECONDARY_EXEC_BUS_LOCK_DETECTION | \ SECONDARY_EXEC_NOTIFY_VM_EXITING | \ + SECONDARY_EXEC_MODE_BASED_EPT_EXEC | \ SECONDARY_EXEC_ENCLS_EXITING | \ SECONDARY_EXEC_EPT_VIOLATION_VE) @@ -620,16 +618,16 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC_CONTROL, 64) * cache on demand. Other registers not listed here are synced to * the cache immediately after VM-Exit. */ -#define VMX_REGS_LAZY_LOAD_SET ((1 << VCPU_REGS_RIP) | \ - (1 << VCPU_REGS_RSP) | \ - (1 << VCPU_EXREG_RFLAGS) | \ - (1 << VCPU_EXREG_PDPTR) | \ - (1 << VCPU_EXREG_SEGMENTS) | \ - (1 << VCPU_EXREG_CR0) | \ - (1 << VCPU_EXREG_CR3) | \ - (1 << VCPU_EXREG_CR4) | \ - (1 << VCPU_EXREG_EXIT_INFO_1) | \ - (1 << VCPU_EXREG_EXIT_INFO_2)) +#define VMX_REGS_LAZY_LOAD_SET (BIT(VCPU_REGS_RSP) | \ + BIT(VCPU_REG_RIP) | \ + BIT(VCPU_REG_RFLAGS) | \ + BIT(VCPU_REG_PDPTR) | \ + BIT(VCPU_REG_SEGMENTS) | \ + BIT(VCPU_REG_CR0) | \ + BIT(VCPU_REG_CR3) | \ + BIT(VCPU_REG_CR4) | \ + BIT(VCPU_REG_EXIT_INFO_1) | \ + BIT(VCPU_REG_EXIT_INFO_2)) static inline unsigned long vmx_l1_guest_owned_cr0_bits(void) { diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index d09abeac2b56..409858074246 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -4,6 +4,7 @@ #include <linux/kvm_host.h> +#include "capabilities.h" #include "x86.h" __init int vmx_hardware_setup(void); @@ -104,6 +105,11 @@ int vmx_set_tss_addr(struct kvm *kvm, unsigned int addr); int vmx_set_identity_map_addr(struct kvm *kvm, u64 ident_addr); u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); +static inline bool vmx_tdp_has_smep(struct kvm *kvm) +{ + return enable_mbec; +} + void vmx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code); void vmx_get_entry_info(struct kvm_vcpu *vcpu, u32 *intr_info, u32 *error_code); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 0550359ed798..d9d51803b7b2 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -23,7 +23,7 @@ #include "mmu.h" #include "i8254.h" #include "tss.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "kvm_emulate.h" #include "mmu/page_track.h" #include "x86.h" @@ -128,12 +128,10 @@ static u64 __read_mostly efer_reserved_bits = ~((u64)EFER_SCE); KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST | \ KVM_X2APIC_DISABLE_SUPPRESS_EOI_BROADCAST) -static void update_cr8_intercept(struct kvm_vcpu *vcpu); static void process_nmi(struct kvm_vcpu *vcpu); static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); static void store_regs(struct kvm_vcpu *vcpu); static int sync_regs(struct kvm_vcpu *vcpu); -static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu); static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); @@ -152,6 +150,7 @@ struct kvm_x86_ops kvm_x86_ops __read_mostly; #include <asm/kvm-x86-ops.h> EXPORT_STATIC_CALL_GPL(kvm_x86_get_cs_db_l_bits); EXPORT_STATIC_CALL_GPL(kvm_x86_cache_reg); +EXPORT_STATIC_CALL_GPL(kvm_x86_get_cpl); static bool __read_mostly ignore_msrs = 0; module_param(ignore_msrs, bool, 0644); @@ -182,15 +181,6 @@ module_param(force_emulation_prefix, int, 0644); int __read_mostly pi_inject_timer = -1; module_param(pi_inject_timer, bint, 0644); -/* Enable/disable PMU virtualization */ -bool __read_mostly enable_pmu = true; -EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_pmu); -module_param(enable_pmu, bool, 0444); - -/* Enable/disabled mediated PMU virtualization. */ -bool __read_mostly enable_mediated_pmu; -EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_mediated_pmu); - bool __read_mostly eager_page_split = true; module_param(eager_page_split, bool, 0644); @@ -576,13 +566,6 @@ static struct kmem_cache *kvm_alloc_emulator_cache(void) static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt); -static inline void kvm_async_pf_hash_reset(struct kvm_vcpu *vcpu) -{ - int i; - for (i = 0; i < ASYNC_PF_PER_VCPU; i++) - vcpu->arch.apf.gfns[i] = ~0; -} - static void kvm_destroy_user_return_msrs(void) { int cpu; @@ -970,7 +953,8 @@ static int complete_emulated_insn_gp(struct kvm_vcpu *vcpu, int err) EMULTYPE_COMPLETE_USER_EXIT); } -void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault) +void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault, + bool from_hardware) { ++vcpu->stat.pf_guest; @@ -987,8 +971,9 @@ void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault) fault->address); } -void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, - struct x86_exception *fault) +void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, + struct x86_exception *fault, + bool from_hardware) { struct kvm_mmu *fault_mmu; WARN_ON_ONCE(fault->vector != PF_VECTOR); @@ -1005,9 +990,9 @@ void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, kvm_mmu_invalidate_addr(vcpu, fault_mmu, fault->address, KVM_MMU_ROOT_CURRENT); - fault_mmu->inject_page_fault(vcpu, fault); + fault_mmu->inject_page_fault(vcpu, fault, from_hardware); } -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_inject_emulated_page_fault); +EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_inject_emulated_page_fault); void kvm_inject_nmi(struct kvm_vcpu *vcpu) { @@ -1021,18 +1006,6 @@ void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_queue_exception_e); -/* - * Checks if cpl <= required_cpl; if true, return true. Otherwise queue - * a #GP and return false. - */ -bool kvm_require_cpl(struct kvm_vcpu *vcpu, int required_cpl) -{ - if (kvm_x86_call(get_cpl)(vcpu) <= required_cpl) - return true; - kvm_queue_exception_e(vcpu, GP_VECTOR, 0); - return false; -} - bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr) { if ((dr != 4 && dr != 5) || !kvm_is_cr4_bit_set(vcpu, X86_CR4_DE)) @@ -1043,13 +1016,6 @@ bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_require_dr); -static bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu) -{ - u64 mask = KVM_ASYNC_PF_ENABLED | KVM_ASYNC_PF_DELIVERY_AS_INT; - - return (vcpu->arch.apf.msr_en_val & mask) == mask; -} - static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu) { return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2); @@ -1065,14 +1031,15 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) gpa_t real_gpa; int i; int ret; - u64 pdpte[ARRAY_SIZE(mmu->pdptrs)]; + u64 pdpte[ARRAY_SIZE(vcpu->arch.pdptrs)]; /* * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated * to an L1 GPA. */ real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(pdpt_gfn), - PFERR_USER_MASK | PFERR_WRITE_MASK, NULL); + PFERR_USER_MASK | PFERR_WRITE_MASK | + PFERR_GUEST_PAGE_MASK, NULL, 0); if (real_gpa == INVALID_GPA) return 0; @@ -1090,14 +1057,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) } /* - * Marking VCPU_EXREG_PDPTR dirty doesn't work for !tdp_enabled. + * Marking VCPU_REG_PDPTR dirty doesn't work for !tdp_enabled. * Shadow page roots need to be reconstructed instead. */ - if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) + if (!tdp_enabled && memcmp(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs))) kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); - memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)); - kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); + memcpy(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs)); + kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); vcpu->arch.pdptrs_from_userspace = false; @@ -1313,7 +1280,7 @@ int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu) { /* Note, #UD due to CR4.OSXSAVE=0 has priority over the intercept. */ if (kvm_x86_call(get_cpl)(vcpu) != 0 || - __kvm_set_xcr(vcpu, kvm_rcx_read(vcpu), kvm_read_edx_eax(vcpu))) { + __kvm_set_xcr(vcpu, kvm_ecx_read(vcpu), kvm_read_edx_eax(vcpu))) { kvm_inject_gp(vcpu, 0); return 1; } @@ -1478,7 +1445,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) kvm_mmu_new_pgd(vcpu, cr3); vcpu->arch.cr3 = cr3; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); /* Do not call post_set_cr3, we do not get here for confidential guests. */ handle_tlb_flush: @@ -1600,9 +1567,17 @@ unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dr); +static unsigned long kvm_get_effective_dr7(struct kvm_vcpu *vcpu) +{ + if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) + return vcpu->arch.guest_debug_dr7; + + return vcpu->arch.dr7; +} + int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) { - u32 pmc = kvm_rcx_read(vcpu); + u32 pmc = kvm_ecx_read(vcpu); u64 data; if (kvm_pmu_rdpmc(vcpu, pmc, &data)) { @@ -1610,8 +1585,8 @@ int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) return 1; } - kvm_rax_write(vcpu, (u32)data); - kvm_rdx_write(vcpu, data >> 32); + kvm_eax_write(vcpu, data); + kvm_edx_write(vcpu, data >> 32); return kvm_skip_emulated_instruction(vcpu); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdpmc); @@ -2058,8 +2033,8 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_write); static void complete_userspace_rdmsr(struct kvm_vcpu *vcpu) { if (!vcpu->run->msr.error) { - kvm_rax_write(vcpu, (u32)vcpu->run->msr.data); - kvm_rdx_write(vcpu, vcpu->run->msr.data >> 32); + kvm_eax_write(vcpu, vcpu->run->msr.data); + kvm_edx_write(vcpu, vcpu->run->msr.data >> 32); } } @@ -2140,8 +2115,8 @@ static int __kvm_emulate_rdmsr(struct kvm_vcpu *vcpu, u32 msr, int reg, trace_kvm_msr_read(msr, data); if (reg < 0) { - kvm_rax_write(vcpu, data & -1u); - kvm_rdx_write(vcpu, (data >> 32) & -1u); + kvm_eax_write(vcpu, data); + kvm_edx_write(vcpu, data >> 32); } else { kvm_register_write(vcpu, reg, data); } @@ -2158,7 +2133,7 @@ static int __kvm_emulate_rdmsr(struct kvm_vcpu *vcpu, u32 msr, int reg, int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu) { - return __kvm_emulate_rdmsr(vcpu, kvm_rcx_read(vcpu), -1, + return __kvm_emulate_rdmsr(vcpu, kvm_ecx_read(vcpu), -1, complete_fast_rdmsr); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr); @@ -2194,7 +2169,7 @@ static int __kvm_emulate_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu) { - return __kvm_emulate_wrmsr(vcpu, kvm_rcx_read(vcpu), + return __kvm_emulate_wrmsr(vcpu, kvm_ecx_read(vcpu), kvm_read_edx_eax(vcpu)); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr); @@ -2304,7 +2279,7 @@ static fastpath_t __handle_fastpath_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 da fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu) { - return __handle_fastpath_wrmsr(vcpu, kvm_rcx_read(vcpu), + return __handle_fastpath_wrmsr(vcpu, kvm_ecx_read(vcpu), kvm_read_edx_eax(vcpu)); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr); @@ -3647,23 +3622,19 @@ static int kvm_pv_enable_async_pf(struct kvm_vcpu *vcpu, u64 data) if (!lapic_in_kernel(vcpu)) return data ? 1 : 0; + if (__kvm_pv_async_pf_enabled(data) && + kvm_gfn_to_hva_cache_init(vcpu->kvm, &vcpu->arch.apf.data, gpa, + sizeof(u64))) + return 1; + vcpu->arch.apf.msr_en_val = data; - if (!kvm_pv_async_pf_enabled(vcpu)) { + if (__kvm_pv_async_pf_enabled(data)) { + kvm_async_pf_wakeup_all(vcpu); + } else { kvm_clear_async_pf_completion_queue(vcpu); kvm_async_pf_hash_reset(vcpu); - return 0; } - - if (kvm_gfn_to_hva_cache_init(vcpu->kvm, &vcpu->arch.apf.data, gpa, - sizeof(u64))) - return 1; - - vcpu->arch.apf.send_always = (data & KVM_ASYNC_PF_SEND_ALWAYS); - vcpu->arch.apf.delivery_as_pf_vmexit = data & KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT; - - kvm_async_pf_wakeup_all(vcpu); - return 0; } @@ -4002,22 +3973,28 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) break; case MSR_EFER: return set_efer(vcpu, msr_info); - case MSR_K7_HWCR: - data &= ~(u64)0x40; /* ignore flush filter disable */ - data &= ~(u64)0x100; /* ignore ignne emulation enable */ - data &= ~(u64)0x8; /* ignore TLB cache disable */ - + case MSR_K7_HWCR: { /* * Allow McStatusWrEn and TscFreqSel. (Linux guests from v3.2 * through at least v6.6 whine if TscFreqSel is clear, * depending on F/M/S. */ - if (data & ~(BIT_ULL(18) | BIT_ULL(24))) { + u64 valid = BIT_ULL(18) | BIT_ULL(24); + + data &= ~(u64)0x40; /* ignore flush filter disable */ + data &= ~(u64)0x100; /* ignore ignne emulation enable */ + data &= ~(u64)0x8; /* ignore TLB cache disable */ + + if (guest_cpu_cap_has(vcpu, X86_FEATURE_GP_ON_USER_CPUID)) + valid |= MSR_K7_HWCR_CPUID_USER_DIS; + + if (data & ~valid) { kvm_pr_unimpl_wrmsr(vcpu, msr, data); return 1; } vcpu->arch.msr_hwcr = data; break; + } case MSR_FAM10H_MMIO_CONF_BASE: if (data != 0) { kvm_pr_unimpl_wrmsr(vcpu, msr, data); @@ -4264,7 +4241,7 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) case MSR_MISC_FEATURES_ENABLES: if (data & ~MSR_MISC_FEATURES_ENABLES_CPUID_FAULT || (data & MSR_MISC_FEATURES_ENABLES_CPUID_FAULT && - !supports_cpuid_fault(vcpu))) + !(vcpu->arch.msr_platform_info & MSR_PLATFORM_INFO_CPUID_FAULT))) return 1; vcpu->arch.msr_misc_features_enables = data; break; @@ -5340,7 +5317,7 @@ static int kvm_vcpu_ioctl_set_lapic(struct kvm_vcpu *vcpu, r = kvm_apic_set_state(vcpu, s); if (r) return r; - update_cr8_intercept(vcpu); + kvm_lapic_update_cr8_intercept(vcpu); return 0; } @@ -6914,6 +6891,10 @@ disable_exits_unlock: if (!enable_pmu || (cap->args[0] & ~KVM_CAP_PMU_VALID_MASK)) break; + if (kvm->arch.has_protected_pmu && + cap->args[0] != KVM_PMU_CAP_DISABLE) + break; + mutex_lock(&kvm->lock); if (!kvm->created_vcpus && !kvm->arch.created_mediated_pmu) { kvm->arch.enable_pmu = !(cap->args[0] & KVM_PMU_CAP_DISABLE); @@ -7852,21 +7833,6 @@ void kvm_get_segment(struct kvm_vcpu *vcpu, kvm_x86_call(get_segment)(vcpu, var, seg); } -gpa_t translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, u64 access, - struct x86_exception *exception) -{ - struct kvm_mmu *mmu = vcpu->arch.mmu; - gpa_t t_gpa; - - BUG_ON(!mmu_is_nested(vcpu)); - - /* NPT walks are always user-walks */ - access |= PFERR_USER_MASK; - t_gpa = mmu->gva_to_gpa(vcpu, mmu, gpa, access, exception); - - return t_gpa; -} - gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, struct x86_exception *exception) { @@ -8567,6 +8533,11 @@ static void emulator_wbinvd(struct x86_emulate_ctxt *ctxt) kvm_emulate_wbinvd_noskip(emul_to_vcpu(ctxt)); } +static unsigned long emulator_get_effective_dr7(struct x86_emulate_ctxt *ctxt) +{ + return kvm_get_effective_dr7(emul_to_vcpu(ctxt)); +} + static unsigned long emulator_get_dr(struct x86_emulate_ctxt *ctxt, int dr) { return kvm_get_dr(emul_to_vcpu(ctxt), dr); @@ -8824,6 +8795,11 @@ static int emulator_intercept(struct x86_emulate_ctxt *ctxt, &ctxt->exception); } +static bool emulator_is_cpuid_allowed(struct x86_emulate_ctxt *ctxt) +{ + return kvm_is_cpuid_allowed(emul_to_vcpu(ctxt)); +} + static bool emulator_get_cpuid(struct x86_emulate_ctxt *ctxt, u32 *eax, u32 *ebx, u32 *ecx, u32 *edx, bool exact_only) @@ -8949,6 +8925,7 @@ static const struct x86_emulate_ops emulate_ops = { .get_cr = emulator_get_cr, .set_cr = emulator_set_cr, .cpl = emulator_get_cpl, + .get_effective_dr7 = emulator_get_effective_dr7, .get_dr = emulator_get_dr, .set_dr = emulator_set_dr, .set_msr_with_filter = emulator_set_msr_with_filter, @@ -8960,6 +8937,7 @@ static const struct x86_emulate_ops emulate_ops = { .wbinvd = emulator_wbinvd, .fix_hypercall = emulator_fix_hypercall, .intercept = emulator_intercept, + .is_cpuid_allowed = emulator_is_cpuid_allowed, .get_cpuid = emulator_get_cpuid, .guest_has_movbe = emulator_guest_has_movbe, .guest_has_fxsr = emulator_guest_has_fxsr, @@ -8995,17 +8973,36 @@ static void toggle_interruptibility(struct kvm_vcpu *vcpu, u32 mask) } } -static void inject_emulated_exception(struct kvm_vcpu *vcpu) +static int kvm_inject_emulated_db(struct kvm_vcpu *vcpu, unsigned long dr6) { - struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; + struct kvm_run *kvm_run = vcpu->run; - if (ctxt->exception.vector == PF_VECTOR) - kvm_inject_emulated_page_fault(vcpu, &ctxt->exception); - else if (ctxt->exception.error_code_valid) - kvm_queue_exception_e(vcpu, ctxt->exception.vector, - ctxt->exception.error_code); + if (vcpu->guest_debug & (KVM_GUESTDBG_USE_HW_BP | KVM_GUESTDBG_SINGLESTEP)) { + kvm_run->debug.arch.dr6 = dr6 | DR6_ACTIVE_LOW; + kvm_run->debug.arch.pc = kvm_get_linear_rip(vcpu); + kvm_run->debug.arch.exception = DB_VECTOR; + kvm_run->exit_reason = KVM_EXIT_DEBUG; + return 0; + } + + kvm_queue_exception_p(vcpu, DB_VECTOR, dr6); + return 1; +} + +static int inject_emulated_exception(struct kvm_vcpu *vcpu) +{ + struct x86_exception *ex = &vcpu->arch.emulate_ctxt->exception; + + if (ex->vector == DB_VECTOR) + return kvm_inject_emulated_db(vcpu, ex->dr6); + + if (ex->vector == PF_VECTOR) + kvm_inject_emulated_page_fault(vcpu, ex); + else if (ex->error_code_valid) + kvm_queue_exception_e(vcpu, ex->vector, ex->error_code); else - kvm_queue_exception(vcpu, ctxt->exception.vector); + kvm_queue_exception(vcpu, ex->vector); + return 1; } static struct x86_emulate_ctxt *alloc_emulate_ctxt(struct kvm_vcpu *vcpu) @@ -9045,6 +9042,7 @@ static void init_emulate_ctxt(struct kvm_vcpu *vcpu) ctxt->interruptibility = 0; ctxt->have_exception = false; ctxt->exception.vector = -1; + ctxt->exception.payload = 0; ctxt->perm_ok = false; init_decode_cache(ctxt); @@ -9262,21 +9260,6 @@ static int kvm_vcpu_check_hw_bp(unsigned long addr, u32 type, u32 dr7, return dr6; } -static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu) -{ - struct kvm_run *kvm_run = vcpu->run; - - if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) { - kvm_run->debug.arch.dr6 = DR6_BS | DR6_ACTIVE_LOW; - kvm_run->debug.arch.pc = kvm_get_linear_rip(vcpu); - kvm_run->debug.arch.exception = DB_VECTOR; - kvm_run->exit_reason = KVM_EXIT_DEBUG; - return 0; - } - kvm_queue_exception_p(vcpu, DB_VECTOR, DR6_BS); - return 1; -} - int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu) { unsigned long rflags = kvm_x86_call(get_rflags)(vcpu); @@ -9297,13 +9280,16 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu) * that sets the TF flag". */ if (unlikely(rflags & X86_EFLAGS_TF)) - r = kvm_vcpu_do_singlestep(vcpu); + r = kvm_inject_emulated_db(vcpu, DR6_BS); return r; } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_skip_emulated_instruction); static bool kvm_is_code_breakpoint_inhibited(struct kvm_vcpu *vcpu) { + if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) + return false; + if (kvm_get_rflags(vcpu) & X86_EFLAGS_RF) return true; @@ -9320,6 +9306,8 @@ static bool kvm_is_code_breakpoint_inhibited(struct kvm_vcpu *vcpu) static bool kvm_vcpu_check_code_breakpoint(struct kvm_vcpu *vcpu, int emulation_type, int *r) { + unsigned long dr7 = kvm_get_effective_dr7(vcpu); + WARN_ON_ONCE(emulation_type & EMULTYPE_NO_DECODE); /* @@ -9340,34 +9328,14 @@ static bool kvm_vcpu_check_code_breakpoint(struct kvm_vcpu *vcpu, EMULTYPE_TRAP_UD | EMULTYPE_VMWARE_GP | EMULTYPE_PF)) return false; - if (unlikely(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) && - (vcpu->arch.guest_debug_dr7 & DR7_BP_EN_MASK)) { - struct kvm_run *kvm_run = vcpu->run; - unsigned long eip = kvm_get_linear_rip(vcpu); - u32 dr6 = kvm_vcpu_check_hw_bp(eip, 0, - vcpu->arch.guest_debug_dr7, - vcpu->arch.eff_db); - - if (dr6 != 0) { - kvm_run->debug.arch.dr6 = dr6 | DR6_ACTIVE_LOW; - kvm_run->debug.arch.pc = eip; - kvm_run->debug.arch.exception = DB_VECTOR; - kvm_run->exit_reason = KVM_EXIT_DEBUG; - *r = 0; - return true; - } - } - - if (unlikely(vcpu->arch.dr7 & DR7_BP_EN_MASK) && + if (unlikely(dr7 & DR7_BP_EN_MASK) && !kvm_is_code_breakpoint_inhibited(vcpu)) { unsigned long eip = kvm_get_linear_rip(vcpu); - u32 dr6 = kvm_vcpu_check_hw_bp(eip, 0, - vcpu->arch.dr7, - vcpu->arch.db); + u32 dr6 = kvm_vcpu_check_hw_bp(eip, 0, dr7, + vcpu->arch.eff_db); - if (dr6 != 0) { - kvm_queue_exception_p(vcpu, DB_VECTOR, dr6); - *r = 1; + if (dr6) { + *r = kvm_inject_emulated_db(vcpu, dr6); return true; } } @@ -9513,8 +9481,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, */ WARN_ON_ONCE(ctxt->exception.vector == UD_VECTOR || exception_type(ctxt->exception.vector) == EXCPT_TRAP); - inject_emulated_exception(vcpu); - return 1; + return inject_emulated_exception(vcpu); } return handle_emulation_failure(vcpu, emulation_type); } @@ -9609,8 +9576,7 @@ restart: if (ctxt->have_exception) { WARN_ON_ONCE(vcpu->mmio_needed && !vcpu->mmio_is_write); vcpu->mmio_needed = false; - r = 1; - inject_emulated_exception(vcpu); + r = inject_emulated_exception(vcpu); } else if (vcpu->arch.pio.count) { if (!vcpu->arch.pio.in) { /* FIXME: return into emulator if single-stepping. */ @@ -9653,7 +9619,7 @@ writeback: kvm_pmu_branch_retired(vcpu); kvm_rip_write(vcpu, ctxt->eip); if (r && (ctxt->tf || (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP))) - r = kvm_vcpu_do_singlestep(vcpu); + r = kvm_inject_emulated_db(vcpu, DR6_BS); kvm_x86_call(update_emulated_instruction)(vcpu); __kvm_set_rflags(vcpu, ctxt->eflags); } @@ -9704,7 +9670,7 @@ static int complete_fast_pio_out(struct kvm_vcpu *vcpu) static int kvm_fast_pio_out(struct kvm_vcpu *vcpu, int size, unsigned short port) { - unsigned long val = kvm_rax_read(vcpu); + unsigned long val = kvm_rax_read_raw(vcpu); int ret = emulator_pio_out(vcpu, size, port, &val, 1); if (ret) @@ -9740,10 +9706,10 @@ static int complete_fast_pio_in(struct kvm_vcpu *vcpu) } /* For size less than 4 we merge, else we zero extend */ - val = (vcpu->arch.pio.size < 4) ? kvm_rax_read(vcpu) : 0; + val = (vcpu->arch.pio.size < 4) ? kvm_rax_read_raw(vcpu) : 0; complete_emulator_pio_in(vcpu, &val); - kvm_rax_write(vcpu, val); + kvm_rax_write_raw(vcpu, val); return kvm_skip_emulated_instruction(vcpu); } @@ -9755,11 +9721,11 @@ static int kvm_fast_pio_in(struct kvm_vcpu *vcpu, int size, int ret; /* For size less than 4 we merge, else we zero extend */ - val = (size < 4) ? kvm_rax_read(vcpu) : 0; + val = (size < 4) ? kvm_rax_read_raw(vcpu) : 0; ret = emulator_pio_in(vcpu, size, port, &val, 1); if (ret) { - kvm_rax_write(vcpu, val); + kvm_rax_write_raw(vcpu, val); return ret; } @@ -10426,33 +10392,34 @@ static int complete_hypercall_exit(struct kvm_vcpu *vcpu) if (!is_64_bit_hypercall(vcpu)) ret = (u32)ret; - kvm_rax_write(vcpu, ret); + kvm_rax_write_raw(vcpu, ret); return kvm_skip_emulated_instruction(vcpu); } int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl, int (*complete_hypercall)(struct kvm_vcpu *)) { - unsigned long ret; - unsigned long nr = kvm_rax_read(vcpu); - unsigned long a0 = kvm_rbx_read(vcpu); - unsigned long a1 = kvm_rcx_read(vcpu); - unsigned long a2 = kvm_rdx_read(vcpu); - unsigned long a3 = kvm_rsi_read(vcpu); int op_64_bit = is_64_bit_hypercall(vcpu); + unsigned long ret, nr, a0, a1, a2, a3; ++vcpu->stat.hypercalls; - trace_kvm_hypercall(nr, a0, a1, a2, a3); - - if (!op_64_bit) { - nr &= 0xFFFFFFFF; - a0 &= 0xFFFFFFFF; - a1 &= 0xFFFFFFFF; - a2 &= 0xFFFFFFFF; - a3 &= 0xFFFFFFFF; + if (op_64_bit) { + nr = kvm_rax_read_raw(vcpu); + a0 = kvm_rbx_read_raw(vcpu); + a1 = kvm_rcx_read_raw(vcpu); + a2 = kvm_rdx_read_raw(vcpu); + a3 = kvm_rsi_read_raw(vcpu); + } else { + nr = kvm_eax_read(vcpu); + a0 = kvm_ebx_read(vcpu); + a1 = kvm_ecx_read(vcpu); + a2 = kvm_edx_read(vcpu); + a3 = kvm_esi_read(vcpu); } + trace_kvm_hypercall(nr, a0, a1, a2, a3); + if (cpl) { ret = -KVM_EPERM; goto out; @@ -10595,33 +10562,6 @@ static void post_kvm_run_save(struct kvm_vcpu *vcpu) kvm_run->flags |= KVM_RUN_X86_GUEST_MODE; } -static void update_cr8_intercept(struct kvm_vcpu *vcpu) -{ - int max_irr, tpr; - - if (!kvm_x86_ops.update_cr8_intercept) - return; - - if (!lapic_in_kernel(vcpu)) - return; - - if (vcpu->arch.apic->apicv_active) - return; - - if (!vcpu->arch.apic->vapic_addr) - max_irr = kvm_lapic_find_highest_irr(vcpu); - else - max_irr = -1; - - if (max_irr != -1) - max_irr >>= 4; - - tpr = kvm_lapic_get_cr8(vcpu); - - kvm_x86_call(update_cr8_intercept)(vcpu, tpr, max_irr); -} - - int kvm_check_nested_events(struct kvm_vcpu *vcpu) { if (kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu)) { @@ -11362,7 +11302,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_x86_call(enable_irq_window)(vcpu); if (kvm_lapic_enabled(vcpu)) { - update_cr8_intercept(vcpu); + kvm_lapic_update_cr8_intercept(vcpu); kvm_lapic_sync_to_vapic(vcpu); } } @@ -11607,9 +11547,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) if (vcpu->arch.apic_attention) kvm_lapic_sync_from_vapic(vcpu); - if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE)) - return 0; - r = kvm_x86_call(handle_exit)(vcpu, exit_fastpath); return r; @@ -12149,23 +12086,23 @@ static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) emulator_writeback_register_cache(vcpu->arch.emulate_ctxt); vcpu->arch.emulate_regs_need_sync_to_vcpu = false; } - regs->rax = kvm_rax_read(vcpu); - regs->rbx = kvm_rbx_read(vcpu); - regs->rcx = kvm_rcx_read(vcpu); - regs->rdx = kvm_rdx_read(vcpu); - regs->rsi = kvm_rsi_read(vcpu); - regs->rdi = kvm_rdi_read(vcpu); + regs->rax = kvm_rax_read_raw(vcpu); + regs->rbx = kvm_rbx_read_raw(vcpu); + regs->rcx = kvm_rcx_read_raw(vcpu); + regs->rdx = kvm_rdx_read_raw(vcpu); + regs->rsi = kvm_rsi_read_raw(vcpu); + regs->rdi = kvm_rdi_read_raw(vcpu); regs->rsp = kvm_rsp_read(vcpu); - regs->rbp = kvm_rbp_read(vcpu); + regs->rbp = kvm_rbp_read_raw(vcpu); #ifdef CONFIG_X86_64 - regs->r8 = kvm_r8_read(vcpu); - regs->r9 = kvm_r9_read(vcpu); - regs->r10 = kvm_r10_read(vcpu); - regs->r11 = kvm_r11_read(vcpu); - regs->r12 = kvm_r12_read(vcpu); - regs->r13 = kvm_r13_read(vcpu); - regs->r14 = kvm_r14_read(vcpu); - regs->r15 = kvm_r15_read(vcpu); + regs->r8 = kvm_r8_read_raw(vcpu); + regs->r9 = kvm_r9_read_raw(vcpu); + regs->r10 = kvm_r10_read_raw(vcpu); + regs->r11 = kvm_r11_read_raw(vcpu); + regs->r12 = kvm_r12_read_raw(vcpu); + regs->r13 = kvm_r13_read_raw(vcpu); + regs->r14 = kvm_r14_read_raw(vcpu); + regs->r15 = kvm_r15_read_raw(vcpu); #endif regs->rip = kvm_rip_read(vcpu); @@ -12189,23 +12126,23 @@ static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) vcpu->arch.emulate_regs_need_sync_from_vcpu = true; vcpu->arch.emulate_regs_need_sync_to_vcpu = false; - kvm_rax_write(vcpu, regs->rax); - kvm_rbx_write(vcpu, regs->rbx); - kvm_rcx_write(vcpu, regs->rcx); - kvm_rdx_write(vcpu, regs->rdx); - kvm_rsi_write(vcpu, regs->rsi); - kvm_rdi_write(vcpu, regs->rdi); + kvm_rax_write_raw(vcpu, regs->rax); + kvm_rbx_write_raw(vcpu, regs->rbx); + kvm_rcx_write_raw(vcpu, regs->rcx); + kvm_rdx_write_raw(vcpu, regs->rdx); + kvm_rsi_write_raw(vcpu, regs->rsi); + kvm_rdi_write_raw(vcpu, regs->rdi); kvm_rsp_write(vcpu, regs->rsp); - kvm_rbp_write(vcpu, regs->rbp); + kvm_rbp_write_raw(vcpu, regs->rbp); #ifdef CONFIG_X86_64 - kvm_r8_write(vcpu, regs->r8); - kvm_r9_write(vcpu, regs->r9); - kvm_r10_write(vcpu, regs->r10); - kvm_r11_write(vcpu, regs->r11); - kvm_r12_write(vcpu, regs->r12); - kvm_r13_write(vcpu, regs->r13); - kvm_r14_write(vcpu, regs->r14); - kvm_r15_write(vcpu, regs->r15); + kvm_r8_write_raw(vcpu, regs->r8); + kvm_r9_write_raw(vcpu, regs->r9); + kvm_r10_write_raw(vcpu, regs->r10); + kvm_r11_write_raw(vcpu, regs->r11); + kvm_r12_write_raw(vcpu, regs->r12); + kvm_r13_write_raw(vcpu, regs->r13); + kvm_r14_write_raw(vcpu, regs->r14); + kvm_r15_write_raw(vcpu, regs->r15); #endif kvm_rip_write(vcpu, regs->rip); @@ -12478,7 +12415,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs, vcpu->arch.cr2 = sregs->cr2; *mmu_reset_needed |= kvm_read_cr3(vcpu) != sregs->cr3; vcpu->arch.cr3 = sregs->cr3; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3); kvm_set_cr8(vcpu, sregs->cr8); @@ -12511,7 +12448,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs, kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR); kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); - update_cr8_intercept(vcpu); + kvm_lapic_update_cr8_intercept(vcpu); /* Older userspace won't unhalt the vcpu on reset. */ if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 && @@ -12571,7 +12508,7 @@ static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2) for (i = 0; i < 4 ; i++) kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); - kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); + kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); mmu_reset_needed = 1; vcpu->arch.pdptrs_from_userspace = true; } @@ -12841,8 +12778,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) int r; vcpu->arch.last_vmentry_cpu = -1; - vcpu->arch.regs_avail = ~0; - vcpu->arch.regs_dirty = ~0; + bitmap_fill(vcpu->arch.regs_avail, NR_VCPU_TOTAL_REGS); + bitmap_fill(vcpu->arch.regs_dirty, NR_VCPU_TOTAL_REGS); kvm_gpc_init(&vcpu->arch.pv_time, vcpu->kvm); @@ -13108,7 +13045,7 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) * on RESET. But, go through the motions in case that's ever remedied. */ cpuid_0x1 = kvm_find_cpuid_entry(vcpu, 1); - kvm_rdx_write(vcpu, cpuid_0x1 ? cpuid_0x1->eax : 0x600); + kvm_edx_write(vcpu, cpuid_0x1 ? cpuid_0x1->eax : 0x600); kvm_x86_call(vcpu_reset)(vcpu, init_event); @@ -13116,7 +13053,7 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) kvm_rip_write(vcpu, 0xfff0); vcpu->arch.cr3 = 0; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); /* * CR0.CD/NW are set on RESET, preserved on INIT. Note, some versions @@ -13382,7 +13319,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz; kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT; kvm->arch.guest_can_read_msr_platform_info = true; - kvm->arch.enable_pmu = enable_pmu; + kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu; #if IS_ENABLED(CONFIG_HYPERV) spin_lock_init(&kvm->arch.hv_root_tdp_lock); @@ -14027,7 +13964,7 @@ static bool kvm_can_deliver_async_pf(struct kvm_vcpu *vcpu) if (!kvm_pv_async_pf_enabled(vcpu)) return false; - if (!vcpu->arch.apf.send_always && + if (!(vcpu->arch.apf.msr_en_val & KVM_ASYNC_PF_SEND_ALWAYS) && (vcpu->arch.guest_state_protected || !kvm_x86_call(get_cpl)(vcpu))) return false; @@ -14036,7 +13973,7 @@ static bool kvm_can_deliver_async_pf(struct kvm_vcpu *vcpu) * L1 needs to opt into the special #PF vmexits that are * used to deliver async page faults. */ - return vcpu->arch.apf.delivery_as_pf_vmexit; + return vcpu->arch.apf.msr_en_val & KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT; } else { /* * Play it safe in case the guest temporarily disables paging. @@ -14080,7 +14017,7 @@ bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu, fault.nested_page_fault = false; fault.address = work->arch.token; fault.async_page_fault = true; - kvm_inject_page_fault(vcpu, &fault); + kvm_inject_page_fault(vcpu, &fault, false); return true; } else { /* @@ -14251,7 +14188,7 @@ void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_c fault.address = gva; fault.async_page_fault = false; } - vcpu->arch.walk_mmu->inject_page_fault(vcpu, &fault); + vcpu->arch.walk_mmu->inject_page_fault(vcpu, &fault, true); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_fixup_and_inject_pf_error); @@ -14303,6 +14240,9 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva) return 1; } + if (WARN_ON_ONCE(tdp_enabled)) + return 0; + pcid_enabled = kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE); switch (type) { @@ -14330,7 +14270,7 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva) * the RAP (Return Address Predicator). */ if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS)) - kvm_register_mark_dirty(vcpu, VCPU_EXREG_ERAPS); + kvm_register_mark_dirty(vcpu, VCPU_REG_ERAPS); kvm_invalidate_pcid(vcpu, operand.pcid); return kvm_skip_emulated_instruction(vcpu); @@ -14346,7 +14286,7 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva) fallthrough; case INVPCID_TYPE_ALL_INCL_GLOBAL: /* - * Don't bother marking VCPU_EXREG_ERAPS dirty, SVM will take + * Don't bother marking VCPU_REG_ERAPS dirty, SVM will take * care of doing so when emulating the full guest TLB flush * (the RAP is cleared on all implicit TLB flushes). */ diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 38a905fa86de..9de577ef9c97 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -6,7 +6,7 @@ #include <asm/fpu/xstate.h> #include <asm/mce.h> #include <asm/pvclock.h> -#include "kvm_cache_regs.h" +#include "regs.h" #include "kvm_emulate.h" #include "cpuid.h" @@ -243,42 +243,6 @@ static inline bool kvm_exception_is_soft(unsigned int nr) return (nr == BP_VECTOR) || (nr == OF_VECTOR); } -static inline bool is_protmode(struct kvm_vcpu *vcpu) -{ - return kvm_is_cr0_bit_set(vcpu, X86_CR0_PE); -} - -static inline bool is_long_mode(struct kvm_vcpu *vcpu) -{ -#ifdef CONFIG_X86_64 - return !!(vcpu->arch.efer & EFER_LMA); -#else - return false; -#endif -} - -static inline bool is_64_bit_mode(struct kvm_vcpu *vcpu) -{ - int cs_db, cs_l; - - WARN_ON_ONCE(vcpu->arch.guest_state_protected); - - if (!is_long_mode(vcpu)) - return false; - kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); - return cs_l; -} - -static inline bool is_64_bit_hypercall(struct kvm_vcpu *vcpu) -{ - /* - * If running with protected guest state, the CS register is not - * accessible. The hypercall register values will have had to been - * provided in 64-bit mode, so assume the guest is in 64-bit. - */ - return vcpu->arch.guest_state_protected || is_64_bit_mode(vcpu); -} - static inline bool x86_exception_has_error_code(unsigned int vector) { static u32 exception_has_error_code = BIT(DF_VECTOR) | BIT(TS_VECTOR) | @@ -290,27 +254,7 @@ static inline bool x86_exception_has_error_code(unsigned int vector) static inline bool mmu_is_nested(struct kvm_vcpu *vcpu) { - return vcpu->arch.walk_mmu == &vcpu->arch.nested_mmu; -} - -static inline bool is_pae(struct kvm_vcpu *vcpu) -{ - return kvm_is_cr4_bit_set(vcpu, X86_CR4_PAE); -} - -static inline bool is_pse(struct kvm_vcpu *vcpu) -{ - return kvm_is_cr4_bit_set(vcpu, X86_CR4_PSE); -} - -static inline bool is_paging(struct kvm_vcpu *vcpu) -{ - return likely(kvm_is_cr0_bit_set(vcpu, X86_CR0_PG)); -} - -static inline bool is_pae_paging(struct kvm_vcpu *vcpu) -{ - return !is_long_mode(vcpu) && is_pae(vcpu) && is_paging(vcpu); + return vcpu->arch.mmu == &vcpu->arch.guest_mmu; } static inline u8 vcpu_virt_addr_bits(struct kvm_vcpu *vcpu) @@ -421,21 +365,6 @@ static inline bool vcpu_match_mmio_gpa(struct kvm_vcpu *vcpu, gpa_t gpa) return false; } -static inline unsigned long kvm_register_read(struct kvm_vcpu *vcpu, int reg) -{ - unsigned long val = kvm_register_read_raw(vcpu, reg); - - return is_64_bit_mode(vcpu) ? val : (u32)val; -} - -static inline void kvm_register_write(struct kvm_vcpu *vcpu, - int reg, unsigned long val) -{ - if (!is_64_bit_mode(vcpu)) - val = (u32)val; - return kvm_register_write_raw(vcpu, reg, val); -} - static inline bool kvm_check_has_quirk(struct kvm *kvm, u64 quirk) { return !(kvm->arch.disabled_quirks & quirk); @@ -490,9 +419,6 @@ fastpath_t handle_fastpath_invd(struct kvm_vcpu *vcpu); extern struct kvm_caps kvm_caps; extern struct kvm_host_values kvm_host; -extern bool enable_pmu; -extern bool enable_mediated_pmu; - void kvm_setup_xss_caps(void); /* @@ -630,15 +556,23 @@ static inline bool kvm_pat_valid(u64 data) return (data | ((data & 0x0202020202020202ull) << 1)) == data; } -static inline bool kvm_dr7_valid(u64 data) +static inline bool __kvm_pv_async_pf_enabled(u64 data) +{ + u64 mask = KVM_ASYNC_PF_ENABLED | KVM_ASYNC_PF_DELIVERY_AS_INT; + + return (data & mask) == mask; +} + +static inline bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu) { - /* Bits [63:32] are reserved */ - return !(data >> 32); + return __kvm_pv_async_pf_enabled(vcpu->arch.apf.msr_en_val); } -static inline bool kvm_dr6_valid(u64 data) + +static inline void kvm_async_pf_hash_reset(struct kvm_vcpu *vcpu) { - /* Bits [63:32] are reserved */ - return !(data >> 32); + int i; + for (i = 0; i < ASYNC_PF_PER_VCPU; i++) + vcpu->arch.apf.gfns[i] = ~0; } /* @@ -687,41 +621,6 @@ enum kvm_msr_access { #define KVM_MSR_RET_UNSUPPORTED 2 #define KVM_MSR_RET_FILTERED 3 -static inline bool __kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) -{ - return !(cr4 & vcpu->arch.cr4_guest_rsvd_bits); -} - -#define __cr4_reserved_bits(__cpu_has, __c) \ -({ \ - u64 __reserved_bits = CR4_RESERVED_BITS; \ - \ - if (!__cpu_has(__c, X86_FEATURE_XSAVE)) \ - __reserved_bits |= X86_CR4_OSXSAVE; \ - if (!__cpu_has(__c, X86_FEATURE_SMEP)) \ - __reserved_bits |= X86_CR4_SMEP; \ - if (!__cpu_has(__c, X86_FEATURE_SMAP)) \ - __reserved_bits |= X86_CR4_SMAP; \ - if (!__cpu_has(__c, X86_FEATURE_FSGSBASE)) \ - __reserved_bits |= X86_CR4_FSGSBASE; \ - if (!__cpu_has(__c, X86_FEATURE_PKU)) \ - __reserved_bits |= X86_CR4_PKE; \ - if (!__cpu_has(__c, X86_FEATURE_LA57)) \ - __reserved_bits |= X86_CR4_LA57; \ - if (!__cpu_has(__c, X86_FEATURE_UMIP)) \ - __reserved_bits |= X86_CR4_UMIP; \ - if (!__cpu_has(__c, X86_FEATURE_VMX)) \ - __reserved_bits |= X86_CR4_VMXE; \ - if (!__cpu_has(__c, X86_FEATURE_PCID)) \ - __reserved_bits |= X86_CR4_PCIDE; \ - if (!__cpu_has(__c, X86_FEATURE_LAM)) \ - __reserved_bits |= X86_CR4_LAM_SUP; \ - if (!__cpu_has(__c, X86_FEATURE_SHSTK) && \ - !__cpu_has(__c, X86_FEATURE_IBT)) \ - __reserved_bits |= X86_CR4_CET; \ - __reserved_bits; \ -}) - int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa, unsigned int bytes, void *data); int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, @@ -754,6 +653,12 @@ static inline void kvm_prepare_emulated_mmio_exit(struct kvm_vcpu *vcpu, frag->data, vcpu->mmio_is_write); } +static inline bool kvm_is_valid_map_gpa_range_ret(u64 hypercall_ret) +{ + return !hypercall_ret || hypercall_ret == EINVAL || + hypercall_ret == EAGAIN; +} + static inline bool user_exit_on_hypercall(struct kvm *kvm, unsigned long hc_nr) { return kvm->arch.hypercall_exit_enabled & BIT(hc_nr); diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 91fd3673c09a..694b31c1fcc9 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -1408,7 +1408,7 @@ int kvm_xen_hvm_config(struct kvm *kvm, struct kvm_xen_hvm_config *xhc) static int kvm_xen_hypercall_set_result(struct kvm_vcpu *vcpu, u64 result) { - kvm_rax_write(vcpu, result); + kvm_rax_write_raw(vcpu, result); return kvm_skip_emulated_instruction(vcpu); } @@ -1678,32 +1678,35 @@ int kvm_xen_hypercall(struct kvm_vcpu *vcpu) bool handled = false; u8 cpl; - input = (u64)kvm_register_read(vcpu, VCPU_REGS_RAX); - /* Hyper-V hypercalls get bit 31 set in EAX */ - if ((input & 0x80000000) && + if ((kvm_rax_read_raw(vcpu) & 0x80000000) && kvm_hv_hypercall_enabled(vcpu)) return kvm_hv_hypercall(vcpu); longmode = is_64_bit_hypercall(vcpu); if (!longmode) { - params[0] = (u32)kvm_rbx_read(vcpu); - params[1] = (u32)kvm_rcx_read(vcpu); - params[2] = (u32)kvm_rdx_read(vcpu); - params[3] = (u32)kvm_rsi_read(vcpu); - params[4] = (u32)kvm_rdi_read(vcpu); - params[5] = (u32)kvm_rbp_read(vcpu); + input = kvm_eax_read(vcpu); + params[0] = kvm_ebx_read(vcpu); + params[1] = kvm_ecx_read(vcpu); + params[2] = kvm_edx_read(vcpu); + params[3] = kvm_esi_read(vcpu); + params[4] = kvm_edi_read(vcpu); + params[5] = kvm_ebp_read(vcpu); } -#ifdef CONFIG_X86_64 else { - params[0] = (u64)kvm_rdi_read(vcpu); - params[1] = (u64)kvm_rsi_read(vcpu); - params[2] = (u64)kvm_rdx_read(vcpu); - params[3] = (u64)kvm_r10_read(vcpu); - params[4] = (u64)kvm_r8_read(vcpu); - params[5] = (u64)kvm_r9_read(vcpu); - } +#ifdef CONFIG_X86_64 + input = (u64)kvm_rax_read_raw(vcpu); + params[0] = (u64)kvm_rdi_read_raw(vcpu); + params[1] = (u64)kvm_rsi_read_raw(vcpu); + params[2] = (u64)kvm_rdx_read_raw(vcpu); + params[3] = (u64)kvm_r10_read_raw(vcpu); + params[4] = (u64)kvm_r8_read_raw(vcpu); + params[5] = (u64)kvm_r9_read_raw(vcpu); +#else + KVM_BUG_ON(1, vcpu->kvm); + return -EIO; #endif + } cpl = kvm_x86_call(get_cpl)(vcpu); trace_kvm_xen_hypercall(cpl, input, params[0], params[1], params[2], params[3], params[4], params[5]); diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index b15269b5941d..42df8ea464c4 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -30,7 +30,6 @@ #include <linux/suspend.h> #include <linux/syscore_ops.h> #include <linux/idr.h> -#include <linux/kvm_types.h> #include <asm/page.h> #include <asm/special_insns.h> #include <asm/msr-index.h> @@ -709,7 +708,7 @@ err: * to normal kernel memory. Systems with the X86_BUG_TDX_PW_MCE erratum need to * do the conversion explicitly via MOVDIR64B. */ -static void tdx_quirk_reset_paddr(unsigned long base, unsigned long size) +void tdx_quirk_reset_paddr(unsigned long base, unsigned long size) { const void *zero_page = (const void *)page_address(ZERO_PAGE(0)); unsigned long phys, end; @@ -728,12 +727,7 @@ static void tdx_quirk_reset_paddr(unsigned long base, unsigned long size) */ mb(); } - -void tdx_quirk_reset_page(struct page *page) -{ - tdx_quirk_reset_paddr(page_to_phys(page), PAGE_SIZE); -} -EXPORT_SYMBOL_FOR_KVM(tdx_quirk_reset_page); +EXPORT_SYMBOL_FOR_KVM(tdx_quirk_reset_paddr); static __init void tdmr_quirk_reset_pamt(struct tdmr_info *tdmr) @@ -1634,6 +1628,17 @@ static void tdx_clflush_page(struct page *page) clflush_cache_range(page_to_virt(page), PAGE_SIZE); } +static void tdx_clflush_pfn(kvm_pfn_t pfn) +{ + clflush_cache_range(__va(PFN_PHYS(pfn)), PAGE_SIZE); +} + +static int pg_level_to_tdx_sept_level(enum pg_level level) +{ + WARN_ON_ONCE(level == PG_LEVEL_NONE); + return level - 1; +} + noinstr u64 tdh_vp_enter(struct tdx_vp *td, struct tdx_module_args *args) { args->rcx = td->tdvpr_pa; @@ -1654,17 +1659,18 @@ u64 tdh_mng_addcx(struct tdx_td *td, struct page *tdcs_page) } EXPORT_SYMBOL_FOR_KVM(tdh_mng_addcx); -u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, struct page *page, struct page *source, u64 *ext_err1, u64 *ext_err2) +u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, kvm_pfn_t pfn, struct page *source, + u64 *ext_err1, u64 *ext_err2) { struct tdx_module_args args = { .rcx = gpa, .rdx = tdx_tdr_pa(td), - .r8 = page_to_phys(page), + .r8 = PFN_PHYS(pfn), .r9 = page_to_phys(source), }; u64 ret; - tdx_clflush_page(page); + tdx_clflush_pfn(pfn); ret = seamcall_ret(TDH_MEM_PAGE_ADD, &args); *ext_err1 = args.rcx; @@ -1674,10 +1680,11 @@ u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, struct page *page, struct page } EXPORT_SYMBOL_FOR_KVM(tdh_mem_page_add); -u64 tdh_mem_sept_add(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2) +u64 tdh_mem_sept_add(struct tdx_td *td, u64 gpa, enum pg_level level, + struct page *page, u64 *ext_err1, u64 *ext_err2) { struct tdx_module_args args = { - .rcx = gpa | level, + .rcx = gpa | pg_level_to_tdx_sept_level(level), .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; @@ -1705,16 +1712,17 @@ u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page) } EXPORT_SYMBOL_FOR_KVM(tdh_vp_addcx); -u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2) +u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, enum pg_level level, + kvm_pfn_t pfn, u64 *ext_err1, u64 *ext_err2) { struct tdx_module_args args = { - .rcx = gpa | level, + .rcx = gpa | pg_level_to_tdx_sept_level(level), .rdx = tdx_tdr_pa(td), - .r8 = page_to_phys(page), + .r8 = PFN_PHYS(pfn), }; u64 ret; - tdx_clflush_page(page); + tdx_clflush_pfn(pfn); ret = seamcall_ret(TDH_MEM_PAGE_AUG, &args); *ext_err1 = args.rcx; @@ -1724,10 +1732,11 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u } EXPORT_SYMBOL_FOR_KVM(tdh_mem_page_aug); -u64 tdh_mem_range_block(struct tdx_td *td, u64 gpa, int level, u64 *ext_err1, u64 *ext_err2) +u64 tdh_mem_range_block(struct tdx_td *td, u64 gpa, enum pg_level level, + u64 *ext_err1, u64 *ext_err2) { struct tdx_module_args args = { - .rcx = gpa | level, + .rcx = gpa | pg_level_to_tdx_sept_level(level), .rdx = tdx_tdr_pa(td), }; u64 ret; @@ -1940,10 +1949,11 @@ u64 tdh_mem_track(struct tdx_td *td) } EXPORT_SYMBOL_FOR_KVM(tdh_mem_track); -u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, u64 level, u64 *ext_err1, u64 *ext_err2) +u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, enum pg_level level, + u64 *ext_err1, u64 *ext_err2) { struct tdx_module_args args = { - .rcx = gpa | level, + .rcx = gpa | pg_level_to_tdx_sept_level(level), .rdx = tdx_tdr_pa(td), }; u64 ret; @@ -1967,21 +1977,27 @@ u64 tdh_phymem_cache_wb(bool resume) } EXPORT_SYMBOL_FOR_KVM(tdh_phymem_cache_wb); +static inline u64 mk_keyed_paddr(u16 hkid, kvm_pfn_t pfn) +{ + /* KeyID bits are just above the physical address bits. */ + return PFN_PHYS(pfn) | ((u64)hkid << boot_cpu_data.x86_phys_bits); +} + u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td) { struct tdx_module_args args = {}; - args.rcx = mk_keyed_paddr(tdx_global_keyid, td->tdr_page); + args.rcx = mk_keyed_paddr(tdx_global_keyid, page_to_pfn(td->tdr_page)); return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args); } EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_tdr); -u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page) +u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, kvm_pfn_t pfn) { struct tdx_module_args args = {}; - args.rcx = mk_keyed_paddr(hkid, page); + args.rcx = mk_keyed_paddr(hkid, pfn); return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args); } diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c index a8eb51ec0ee2..ca473ca198b8 100644 --- a/drivers/crypto/ccp/sev-dev.c +++ b/drivers/crypto/ccp/sev-dev.c @@ -2369,7 +2369,8 @@ e_free_pdh: return ret; } -static int sev_ioctl_do_snp_platform_status(struct sev_issue_cmd *argp) +static int __sev_do_snp_platform_status(struct sev_user_data_snp_status *status, + int *error) { struct sev_device *sev = psp_master->sev_data; struct sev_data_snp_addr buf; @@ -2377,9 +2378,6 @@ static int sev_ioctl_do_snp_platform_status(struct sev_issue_cmd *argp) void *data; int ret; - if (!argp->data) - return -EINVAL; - status_page = alloc_page(GFP_KERNEL_ACCOUNT); if (!status_page) return -ENOMEM; @@ -2402,7 +2400,7 @@ static int sev_ioctl_do_snp_platform_status(struct sev_issue_cmd *argp) } buf.address = __psp_pa(data); - ret = __sev_do_cmd_locked(SEV_CMD_SNP_PLATFORM_STATUS, &buf, &argp->error); + ret = __sev_do_cmd_locked(SEV_CMD_SNP_PLATFORM_STATUS, &buf, error); if (sev->snp_initialized) { /* @@ -2417,15 +2415,32 @@ static int sev_ioctl_do_snp_platform_status(struct sev_issue_cmd *argp) if (ret) goto cleanup; - if (copy_to_user((void __user *)argp->data, data, - sizeof(struct sev_user_data_snp_status))) - ret = -EFAULT; + memcpy(status, data, sizeof(*status)); cleanup: __free_pages(status_page, 0); return ret; } +static int sev_ioctl_do_snp_platform_status(struct sev_issue_cmd *argp) +{ + struct sev_user_data_snp_status status; + int ret; + + if (!argp->data) + return -EINVAL; + + ret = __sev_do_snp_platform_status(&status, &argp->error); + if (ret < 0) + return ret; + + if (copy_to_user((void __user *)argp->data, &status, + sizeof(struct sev_user_data_snp_status))) + ret = -EFAULT; + + return ret; +} + static int sev_ioctl_do_snp_commit(struct sev_issue_cmd *argp) { struct sev_data_snp_commit buf; @@ -2897,3 +2912,73 @@ void sev_pci_exit(void) sev_firmware_shutdown(sev); } + +static int get_v1_svn(struct sev_device *sev) +{ + struct sev_snp_tcb_version_genoa_milan *tcb; + struct sev_user_data_snp_status status; + int ret, error = 0; + + mutex_lock(&sev_cmd_mutex); + ret = __sev_do_snp_platform_status(&status, &error); + mutex_unlock(&sev_cmd_mutex); + if (ret < 0) + return ret; + + tcb = (struct sev_snp_tcb_version_genoa_milan *)&status + .current_tcb_version; + return tcb->snp; +} + +static int get_v2_svn(struct sev_device *sev) +{ + struct sev_user_data_snp_status status; + struct sev_snp_tcb_version_turin *tcb; + int ret, error = 0; + + mutex_lock(&sev_cmd_mutex); + ret = __sev_do_snp_platform_status(&status, &error); + mutex_unlock(&sev_cmd_mutex); + if (ret < 0) + return ret; + + tcb = (struct sev_snp_tcb_version_turin *)&status + .current_tcb_version; + return tcb->snp; +} + +static bool sev_firmware_allows_es(struct sev_device *sev) +{ + /* Documented in AMD-SB-3023 */ + if (boot_cpu_has(X86_FEATURE_ZEN4) || boot_cpu_has(X86_FEATURE_ZEN3)) + return get_v1_svn(sev) < 0x1b; + else if (boot_cpu_has(X86_FEATURE_ZEN5)) + return get_v2_svn(sev) < 0x4; + else + return true; +} + +int sev_firmware_supported_vm_types(void) +{ + int supported_vm_types = 0; + struct sev_device *sev; + + if (!psp_master || !psp_master->sev_data) + return supported_vm_types; + sev = psp_master->sev_data; + + supported_vm_types |= BIT(KVM_X86_SEV_VM); + supported_vm_types |= BIT(KVM_X86_SEV_ES_VM); + + if (!sev->snp_initialized) + return supported_vm_types; + + supported_vm_types |= BIT(KVM_X86_SNP_VM); + + if (!sev_firmware_allows_es(sev)) + supported_vm_types &= ~BIT(KVM_X86_SEV_ES_VM); + + return supported_vm_types; + +} +EXPORT_SYMBOL_FOR_MODULES(sev_firmware_supported_vm_types, "kvm-amd"); diff --git a/drivers/irqchip/irq-gic-v5.c b/drivers/irqchip/irq-gic-v5.c index c1af07083cef..e9d1795235a6 100644 --- a/drivers/irqchip/irq-gic-v5.c +++ b/drivers/irqchip/irq-gic-v5.c @@ -208,17 +208,13 @@ static void gicv5_hwirq_eoi(u32 hwirq_id, u8 hwirq_type) FIELD_PREP(GICV5_GIC_CDDI_TYPE_MASK, hwirq_type); gic_insn(cddi, CDDI); - - gic_insn(0, CDEOI); } static void gicv5_ppi_irq_eoi(struct irq_data *d) { /* Skip deactivate for forwarded PPI interrupts */ - if (irqd_is_forwarded_to_vcpu(d)) { - gic_insn(0, CDEOI); + if (irqd_is_forwarded_to_vcpu(d)) return; - } gicv5_hwirq_eoi(d->hwirq, GICV5_HWIRQ_TYPE_PPI); } @@ -969,6 +965,13 @@ static void __exception_irq_entry gicv5_handle_irq(struct pt_regs *regs) */ isb(); + /* + * Ensure that we can receive the next interrupts in the event that we + * have a long running handler or directly enter a guest by doing the + * priority drop immediately. + */ + gic_insn(0, CDEOI); + hwirq = FIELD_GET(GICV5_HWIRQ_INTID, ia); handle_irq_per_domain(hwirq); diff --git a/drivers/irqchip/irq-riscv-imsic-state.c b/drivers/irqchip/irq-riscv-imsic-state.c index e3ed874d89e7..b8d1bbbf42f7 100644 --- a/drivers/irqchip/irq-riscv-imsic-state.c +++ b/drivers/irqchip/irq-riscv-imsic-state.c @@ -920,13 +920,12 @@ int __init imsic_setup_state(struct fwnode_handle *fwnode, void *opaque) local->msi_va = mmios_va[index] + reloff; /* - * KVM uses global->nr_guest_files to determine the available guest - * interrupt files on each CPU. Take the minimum number of guest - * interrupt files across all CPUs to avoid KVM incorrectly allocating - * an unexisted or unmapped guest interrupt file on some CPUs. + * KVM uses both local->nr_guest_files and global->nr_guest_files + * to determine the available guest interrupt files on each CPU. */ nr_guest_files = (resource_size(&mmios[index]) - reloff) / IMSIC_MMIO_PAGE_SZ - 1; - global->nr_guest_files = min(global->nr_guest_files, nr_guest_files); + local->nr_guest_files = min((BIT(global->guest_index_bits) - 1), nr_guest_files); + global->nr_guest_files = min(global->nr_guest_files, local->nr_guest_files); nr_handlers++; } diff --git a/drivers/s390/char/sclp_early.c b/drivers/s390/char/sclp_early.c index 6bf501ad8ff0..22dd797e6229 100644 --- a/drivers/s390/char/sclp_early.c +++ b/drivers/s390/char/sclp_early.c @@ -61,8 +61,10 @@ static void __init sclp_early_facilities_detect(void) sclp.has_sipl = !!(sccb->cbl & 0x4000); sclp.has_sipl_eckd = !!(sccb->cbl & 0x2000); } - if (sccb->cpuoff > 139) + if (sccb->cpuoff > 139) { sclp.has_diag324 = !!(sccb->byte_139 & 0x80); + sclp.has_astfleie2 = !!(sccb->byte_139 & 0x40); + } sclp.rnmax = sccb->rnmax ? sccb->rnmax : sccb->rnmax2; sclp.rzm = sccb->rnsize ? sccb->rnsize : sccb->rnsize2; sclp.rzm <<= 20; diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index bf8cc9589bd0..15a4f97f8105 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -66,11 +66,6 @@ struct arch_timer_context { */ bool loaded; - /* Output level of the timer IRQ */ - struct { - bool level; - } irq; - /* Who am I? */ enum kvm_arch_timers timer_id; @@ -104,7 +99,7 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu); void kvm_timer_sync_nested(struct kvm_vcpu *vcpu); void kvm_timer_sync_user(struct kvm_vcpu *vcpu); bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu); -void kvm_timer_update_run(struct kvm_vcpu *vcpu); +bool kvm_timer_update_run(struct kvm_vcpu *vcpu); void kvm_timer_vcpu_terminate(struct kvm_vcpu *vcpu); void kvm_timer_init_vm(struct kvm *kvm); diff --git a/include/kvm/arm_pmu.h b/include/kvm/arm_pmu.h index 0a36a3d5c894..b5e5942204fc 100644 --- a/include/kvm/arm_pmu.h +++ b/include/kvm/arm_pmu.h @@ -32,7 +32,6 @@ struct kvm_pmu { struct kvm_pmc pmc[KVM_ARMV8_PMU_MAX_COUNTERS]; int irq_num; bool created; - bool irq_level; }; struct arm_pmu_entry { @@ -54,7 +53,7 @@ void kvm_pmu_reprogram_counter_mask(struct kvm_vcpu *vcpu, u64 val); void kvm_pmu_flush_hwstate(struct kvm_vcpu *vcpu); void kvm_pmu_sync_hwstate(struct kvm_vcpu *vcpu); bool kvm_pmu_should_notify_user(struct kvm_vcpu *vcpu); -void kvm_pmu_update_run(struct kvm_vcpu *vcpu); +bool kvm_pmu_update_run(struct kvm_vcpu *vcpu); void kvm_pmu_software_increment(struct kvm_vcpu *vcpu, u64 val); void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val); void kvm_pmu_set_counter_event_type(struct kvm_vcpu *vcpu, u64 data, @@ -131,7 +130,7 @@ static inline bool kvm_pmu_should_notify_user(struct kvm_vcpu *vcpu) { return false; } -static inline void kvm_pmu_update_run(struct kvm_vcpu *vcpu) {} +static inline bool kvm_pmu_update_run(struct kvm_vcpu *vcpu) { return false; } static inline void kvm_pmu_software_increment(struct kvm_vcpu *vcpu, u64 val) {} static inline void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val) {} static inline void kvm_pmu_set_counter_event_type(struct kvm_vcpu *vcpu, diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 1388dc6028a9..fe49fb56dc3c 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -177,6 +177,11 @@ struct vgic_global { bool has_gcie_v3_compat; u32 ich_vtr_el2; + + /* GICv5 PPI capabilities */ + struct { + DECLARE_BITMAP(impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); + } vgic_v5_ppi_caps; }; extern struct vgic_global kvm_vgic_global_state; @@ -200,7 +205,7 @@ struct vgic_irq; */ struct irq_ops { /* Per interrupt flags for special-cased interrupts */ - unsigned long flags; + unsigned long (*get_flags)(void); #define VGIC_IRQ_SW_RESAMPLE BIT(0) /* Clear the active state for resampling */ @@ -266,7 +271,7 @@ struct vgic_irq { u8 priority; u8 group; /* 0 == group 0, 1 == group 1 */ - struct irq_ops *ops; + const struct irq_ops *ops; void *owner; /* Opaque pointer to reserve an interrupt for in-kernel devices. */ @@ -274,7 +279,8 @@ struct vgic_irq { static inline bool vgic_irq_needs_resampling(struct vgic_irq *irq) { - return irq->ops && (irq->ops->flags & VGIC_IRQ_SW_RESAMPLE); + return irq->ops && irq->ops->get_flags && + (irq->ops->get_flags() & VGIC_IRQ_SW_RESAMPLE); } struct vgic_register_region; @@ -492,11 +498,6 @@ struct vgic_v5_cpu_if { struct gicv5_vpe gicv5_vpe; }; -/* What PPI capabilities does a GICv5 host have */ -struct vgic_v5_ppi_caps { - DECLARE_BITMAP(impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); -}; - struct vgic_cpu { /* CPU vif control registers for world switch */ union { @@ -557,7 +558,7 @@ void kvm_vgic_init_cpu_hardware(void); int kvm_vgic_inject_irq(struct kvm *kvm, struct kvm_vcpu *vcpu, unsigned int intid, bool level, void *owner); void kvm_vgic_set_irq_ops(struct kvm_vcpu *vcpu, u32 vintid, - struct irq_ops *ops); + const struct irq_ops *ops); void kvm_vgic_clear_irq_ops(struct kvm_vcpu *vcpu, u32 vintid); int kvm_vgic_map_phys_irq(struct kvm_vcpu *vcpu, unsigned int host_irq, u32 vintid); diff --git a/include/linux/call_once.h b/include/linux/call_once.h index 13cd6469e7e5..1625a9d6ff5b 100644 --- a/include/linux/call_once.h +++ b/include/linux/call_once.h @@ -36,7 +36,7 @@ do { \ * it returns a zero or positive value, mark @once as completed. Return * the value returned by @cb * - * If @once has completed succesfully before, return 0. + * If @once has completed successfully before, return 0. * * The call to @cb is implicitly surrounded by a mutex, though for * efficiency the * function avoids taking it after the first call. diff --git a/include/linux/irqchip/riscv-imsic.h b/include/linux/irqchip/riscv-imsic.h index 4b348836de7a..61af3a5bea09 100644 --- a/include/linux/irqchip/riscv-imsic.h +++ b/include/linux/irqchip/riscv-imsic.h @@ -40,6 +40,9 @@ struct imsic_local_config { phys_addr_t msi_pa; void __iomem *msi_va; + + /* Number of guest interrupt files per-HART */ + u32 nr_guest_files; }; struct imsic_global_config { @@ -68,7 +71,7 @@ struct imsic_global_config { /* Number of guest interrupt identities */ u32 nr_guest_ids; - /* Number of guest interrupt files per core */ + /* Number of guest interrupt files across all HARTs */ u32 nr_guest_files; /* Per-CPU IMSIC addresses */ diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 4c14aee1fb06..ab8cfaec82d3 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1390,20 +1390,20 @@ void mark_page_dirty_in_slot(struct kvm *kvm, const struct kvm_memory_slot *mems void mark_page_dirty(struct kvm *kvm, gfn_t gfn); void kvm_vcpu_mark_page_dirty(struct kvm_vcpu *vcpu, gfn_t gfn); -int __kvm_vcpu_map(struct kvm_vcpu *vcpu, gpa_t gpa, struct kvm_host_map *map, +int __kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map, bool writable); void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map); -static inline int kvm_vcpu_map(struct kvm_vcpu *vcpu, gpa_t gpa, +static inline int kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map) { - return __kvm_vcpu_map(vcpu, gpa, map, true); + return __kvm_vcpu_map(vcpu, gfn, map, true); } -static inline int kvm_vcpu_map_readonly(struct kvm_vcpu *vcpu, gpa_t gpa, +static inline int kvm_vcpu_map_readonly(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map) { - return __kvm_vcpu_map(vcpu, gpa, map, false); + return __kvm_vcpu_map(vcpu, gfn, map, false); } static inline void kvm_vcpu_map_mark_dirty(struct kvm_vcpu *vcpu, @@ -1562,7 +1562,7 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc); void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc); #endif -void kvm_mmu_invalidate_begin(struct kvm *kvm); +void kvm_mmu_invalidate_start(struct kvm *kvm); void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end); void kvm_mmu_invalidate_end(struct kvm *kvm); bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); @@ -1815,6 +1815,11 @@ void kvm_unregister_irq_ack_notifier(struct kvm *kvm, struct kvm_irq_ack_notifier *kian); bool kvm_arch_irqfd_allowed(struct kvm *kvm, struct kvm_irqfd *args); +static inline bool is_gfn_in_memslot(const struct kvm_memory_slot *slot, gfn_t gfn) +{ + return gfn >= slot->base_gfn && gfn < slot->base_gfn + slot->npages; +} + /* * Returns a pointer to the memslot if it contains gfn. * Otherwise returns NULL. @@ -1825,7 +1830,7 @@ try_get_memslot(struct kvm_memory_slot *slot, gfn_t gfn) if (!slot) return NULL; - if (gfn >= slot->base_gfn && gfn < slot->base_gfn + slot->npages) + if (is_gfn_in_memslot(slot, gfn)) return slot; else return NULL; @@ -2596,7 +2601,8 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord typedef int (*kvm_gmem_populate_cb)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, struct page *page, void *opaque); -long kvm_gmem_populate(struct kvm *kvm, gfn_t gfn, void __user *src, long npages, +long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, + long npages, bool may_writeback_src, kvm_gmem_populate_cb post_populate, void *opaque); #endif diff --git a/include/linux/psp-sev.h b/include/linux/psp-sev.h index d5099a2baca5..ce16bbc0b308 100644 --- a/include/linux/psp-sev.h +++ b/include/linux/psp-sev.h @@ -902,6 +902,42 @@ struct snp_feature_info { /* Feature bits in EBX */ #define SNP_SEV_TIO_SUPPORTED BIT(1) +/** + * struct sev_snp_tcb_version_genoa_milan + * + * @boot_loader: SVN of PSP bootloader + * @tee: SVN of PSP operating system + * @reserved: reserved + * @snp: SVN of SNP firmware + * @microcode: Lowest current patch level of all cores + */ +struct sev_snp_tcb_version_genoa_milan { + u8 boot_loader; + u8 tee; + u8 reserved[4]; + u8 snp; + u8 microcode; +}; + +/** + * struct sev_snp_tcb_version_turin + * + * @fmc: SVN of FMC firmware + * @boot_loader: SVN of PSP bootloader + * @tee: SVN of PSP operating system + * @snp: SVN of SNP firmware + * @reserved: reserved + * @microcode: Lowest current patch level of all cores + */ +struct sev_snp_tcb_version_turin { + u8 fmc; + u8 boot_loader; + u8 tee; + u8 snp; + u8 reserved[3]; + u8 microcode; +}; + #ifdef CONFIG_CRYPTO_DEV_SP_PSP /** @@ -1048,6 +1084,7 @@ void snp_free_firmware_page(void *addr); void sev_platform_shutdown(void); bool sev_is_snp_ciphertext_hiding_supported(void); u64 sev_get_snp_policy_bits(void); +int sev_firmware_supported_vm_types(void); #else /* !CONFIG_CRYPTO_DEV_SP_PSP */ diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 6c8afa2047bf..419011097fa8 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -996,6 +996,7 @@ struct kvm_enable_cap { #define KVM_CAP_S390_USER_OPEREXEC 246 #define KVM_CAP_S390_KEYOP 247 #define KVM_CAP_S390_VSIE_ESAMODE 248 +#define KVM_CAP_S390_HPAGE_2G 249 struct kvm_irq_routing_irqchip { __u32 irqchip; diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm index 9118a5a51b89..d28a057fa6c2 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -97,6 +97,7 @@ TEST_GEN_PROGS_x86 += x86/nested_emulation_test TEST_GEN_PROGS_x86 += x86/nested_exceptions_test TEST_GEN_PROGS_x86 += x86/nested_invalid_cr3_test TEST_GEN_PROGS_x86 += x86/nested_set_state_test +TEST_GEN_PROGS_x86 += x86/nested_tdp_fault_test TEST_GEN_PROGS_x86 += x86/nested_tsc_adjust_test TEST_GEN_PROGS_x86 += x86/nested_tsc_scaling_test TEST_GEN_PROGS_x86 += x86/nested_vmsave_vmload_test @@ -117,6 +118,7 @@ TEST_GEN_PROGS_x86 += x86/svm_nested_clear_efer_svme TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test TEST_GEN_PROGS_x86 += x86/svm_nested_vmcb12_gpa +TEST_GEN_PROGS_x86 += x86/svm_nested_pat_test TEST_GEN_PROGS_x86 += x86/svm_lbr_nested_state TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync TEST_GEN_PROGS_x86 += x86/sync_regs_test @@ -140,6 +142,7 @@ TEST_GEN_PROGS_x86 += x86/tsc_msrs_test TEST_GEN_PROGS_x86 += x86/vmx_pmu_caps_test TEST_GEN_PROGS_x86 += x86/xen_shinfo_test TEST_GEN_PROGS_x86 += x86/xen_vmcall_test +TEST_GEN_PROGS_x86 += x86/sev_dbg_test TEST_GEN_PROGS_x86 += x86/sev_init2_tests TEST_GEN_PROGS_x86 += x86/sev_migrate_tests TEST_GEN_PROGS_x86 += x86/sev_smoke_test @@ -210,6 +213,7 @@ TEST_GEN_PROGS_s390 += s390/keyop TEST_GEN_PROGS_s390 += rseq_test TEST_GEN_PROGS_s390 += s390/irq_routing TEST_GEN_PROGS_s390 += mmu_stress_test +TEST_GEN_PROGS_s390 += pre_fault_memory_test TEST_GEN_PROGS_riscv = $(TEST_GEN_PROGS_COMMON) TEST_GEN_PROGS_riscv += riscv/sbi_pmu_test diff --git a/tools/testing/selftests/kvm/arm64/no-vgic.c b/tools/testing/selftests/kvm/arm64/no-vgic.c index 25b2e3222f68..ab57902ce429 100644 --- a/tools/testing/selftests/kvm/arm64/no-vgic.c +++ b/tools/testing/selftests/kvm/arm64/no-vgic.c @@ -159,6 +159,7 @@ static void guest_code_gicv5(void) check_gicv5_gic_op(CDAFF); check_gicv5_gic_op(CDDI); check_gicv5_gic_op(CDDIS); + check_gicv5_gic_op(CDEN); check_gicv5_gic_op(CDEOI); check_gicv5_gic_op(CDHM); check_gicv5_gic_op(CDPEND); diff --git a/tools/testing/selftests/kvm/arm64/vgic_v5.c b/tools/testing/selftests/kvm/arm64/vgic_v5.c index d785b660d847..96cfd6bb32f6 100644 --- a/tools/testing/selftests/kvm/arm64/vgic_v5.c +++ b/tools/testing/selftests/kvm/arm64/vgic_v5.c @@ -20,8 +20,6 @@ struct vm_gic { u32 gic_dev_type; }; -static u64 max_phys_size; - #define GUEST_CMD_IRQ_CDIA 10 #define GUEST_CMD_IRQ_DIEOI 11 #define GUEST_CMD_IS_AWAKE 12 @@ -131,6 +129,8 @@ static void test_vgic_v5_ppis(u32 gic_dev_type) while (1) { ret = run_vcpu(vcpus[0]); + if (ret) + break; switch (get_ucall(vcpus[0], &uc)) { case UCALL_SYNC: @@ -146,7 +146,7 @@ static void test_vgic_v5_ppis(u32 gic_dev_type) irq = FIELD_PREP(KVM_ARM_IRQ_NUM_MASK, 3); irq |= KVM_ARM_IRQ_TYPE_PPI << KVM_ARM_IRQ_TYPE_SHIFT; - _kvm_irq_line(v.vm, irq, level); + kvm_irq_line(v.vm, irq, level); } else if (uc.args[1] == GUEST_CMD_IS_AWAKE) { pr_info("Guest skipping WFI due to pending IRQ\n"); } else if (uc.args[1] == GUEST_CMD_IRQ_CDIA) { @@ -208,13 +208,9 @@ void run_tests(u32 gic_dev_type) int main(int ac, char **av) { int ret; - int pa_bits; test_disable_default_vgic(); - pa_bits = vm_guest_mode_params[VM_MODE_DEFAULT].pa_bits; - max_phys_size = 1ULL << pa_bits; - ret = test_kvm_device(KVM_DEV_TYPE_ARM_VGIC_V5); if (ret) { pr_info("No GICv5 support; Not running GIC_v5 tests.\n"); diff --git a/tools/testing/selftests/kvm/dirty_log_test.c b/tools/testing/selftests/kvm/dirty_log_test.c index 12446a4b6e8d..74ca096bf976 100644 --- a/tools/testing/selftests/kvm/dirty_log_test.c +++ b/tools/testing/selftests/kvm/dirty_log_test.c @@ -694,7 +694,17 @@ static void run_test(enum vm_guest_mode mode, void *arg) pthread_create(&vcpu_thread, NULL, vcpu_worker, vcpu); for (iteration = 1; iteration <= p->iterations; iteration++) { - unsigned long i; + unsigned long i, reap_i; + + /* + * Select a random point in the time interval to reap the dirty + * bitmap/ring while the guest is running, i.e. randomize how + * long the guest gets to initially run and thus how many pages + * it can dirty, before collecting the dirty bitmap/ring. See + * the loop below for details. + */ + reap_i = random() % p->interval; + printf("Reaping after a %lu ms delay\n", reap_i); sync_global_to_guest(vm, iteration); @@ -729,13 +739,17 @@ static void run_test(enum vm_guest_mode mode, void *arg) * that's effectively blocked. Collecting while the * guest is running also verifies KVM doesn't lose any * state. - * + */ + if (i < reap_i) + continue; + + /* * For bitmap modes, KVM overwrites the entire bitmap, * i.e. collecting the bitmaps is destructive. Collect - * the bitmap only on the first pass, otherwise this - * test would lose track of dirty pages. + * the bitmap while the guest is running only once, + * otherwise this test would lose track of dirty pages. */ - if (i && host_log_mode != LOG_MODE_DIRTY_RING) + if (i > reap_i && host_log_mode != LOG_MODE_DIRTY_RING) continue; /* @@ -745,7 +759,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) * the ring on every pass would make it unlikely the * vCPU would ever fill the fing). */ - if (i && !READ_ONCE(dirty_ring_vcpu_ring_full)) + if (i > reap_i && !READ_ONCE(dirty_ring_vcpu_ring_full)) continue; log_mode_collect_dirty_pages(vcpu, TEST_MEM_SLOT_INDEX, diff --git a/tools/testing/selftests/kvm/guest_memfd_test.c b/tools/testing/selftests/kvm/guest_memfd_test.c index 832ef4dfb99f..2233d871a38f 100644 --- a/tools/testing/selftests/kvm/guest_memfd_test.c +++ b/tools/testing/selftests/kvm/guest_memfd_test.c @@ -345,6 +345,16 @@ static void test_invalid_punch_hole(int fd, size_t total_size) } } +static void test_invalid_binding(struct kvm_vm *vm, int fd, size_t size) +{ + int r; + + r = __vm_set_user_memory_region2(vm, 0, KVM_MEM_GUEST_MEMFD, 0, size, 0, + fd, ALIGN_DOWN(INT64_MAX, page_size)); + TEST_ASSERT(r && errno == EINVAL, + "Memslot with out-of-range offset+size should fail"); +} + static void test_create_guest_memfd_invalid_sizes(struct kvm_vm *vm, u64 guest_memfd_flags) { @@ -408,17 +418,26 @@ static void test_guest_memfd_flags(struct kvm_vm *vm) } } -#define __gmem_test(__test, __vm, __flags, __gmem_size) \ +#define ____gmem_test(__test, __vm, __flags, __gmem_size, args...) \ do { \ int fd = vm_create_guest_memfd(__vm, __gmem_size, __flags); \ \ - test_##__test(fd, __gmem_size); \ + test_##__test(args); \ close(fd); \ } while (0) +#define __gmem_test(__test, __vm, __flags, __gmem_size) \ + ____gmem_test(__test, __vm, __flags, __gmem_size, fd, __gmem_size) + #define gmem_test(__test, __vm, __flags) \ __gmem_test(__test, __vm, __flags, page_size * 4) +#define __gmem_test_vm(__test, __vm, __flags, __gmem_size) \ + ____gmem_test(__test, __vm, __flags, __gmem_size, __vm, fd, __gmem_size) + +#define gmem_test_vm(__test, __vm, __flags) \ + __gmem_test_vm(__test, __vm, __flags, page_size * 4) + static void __test_guest_memfd(struct kvm_vm *vm, u64 flags) { test_create_guest_memfd_multiple(vm); @@ -447,6 +466,7 @@ static void __test_guest_memfd(struct kvm_vm *vm, u64 flags) gmem_test(file_size, vm, flags); gmem_test(fallocate, vm, flags); gmem_test(invalid_punch_hole, vm, flags); + gmem_test_vm(invalid_binding, vm, flags); } static void test_guest_memfd(unsigned long vm_type) diff --git a/tools/testing/selftests/kvm/include/kvm_syscalls.h b/tools/testing/selftests/kvm/include/kvm_syscalls.h index 067a4c9cf452..6cb3bed29b81 100644 --- a/tools/testing/selftests/kvm/include/kvm_syscalls.h +++ b/tools/testing/selftests/kvm/include/kvm_syscalls.h @@ -89,4 +89,10 @@ __KVM_SYSCALL_DEFINE(fallocate, 4, int, fd, int, mode, loff_t, offset, loff_t, l __KVM_SYSCALL_DEFINE(ftruncate, 2, unsigned int, fd, off_t, length); __KVM_SYSCALL_DEFINE(madvise, 3, void *, addr, size_t, length, int, advice); +#define kvm_free_fd(fd) \ +do { \ + kvm_close(fd); \ + (fd) = -1; \ +} while (0) + #endif /* SELFTEST_KVM_SYSCALLS_H */ diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 2ecaaa0e9965..04a910164a29 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -876,7 +876,7 @@ static inline int vcpu_get_stats_fd(struct kvm_vcpu *vcpu) { int fd = __vcpu_ioctl(vcpu, KVM_GET_STATS_FD, NULL); - TEST_ASSERT_VM_VCPU_IOCTL(fd >= 0, KVM_CHECK_EXTENSION, fd, vcpu->vm); + TEST_ASSERT_VM_VCPU_IOCTL(fd >= 0, KVM_GET_STATS_FD, fd, vcpu->vm); return fd; } diff --git a/tools/testing/selftests/kvm/include/lru_gen_util.h b/tools/testing/selftests/kvm/include/lru_gen_util.h index d32ff5d8ffd0..49c8139d398c 100644 --- a/tools/testing/selftests/kvm/include/lru_gen_util.h +++ b/tools/testing/selftests/kvm/include/lru_gen_util.h @@ -14,7 +14,7 @@ #include "test_util.h" #define MAX_NR_GENS 16 /* MAX_NR_GENS in include/linux/mmzone.h */ -#define MAX_NR_NODES 4 /* Maximum number of nodes supported by the test */ +#define MAX_NR_NODES 32 /* Maximum number of nodes supported by the test */ #define LRU_GEN_DEBUGFS "/sys/kernel/debug/lru_gen" #define LRU_GEN_ENABLED_PATH "/sys/kernel/mm/lru_gen/enabled" diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h index 77f576ee7789..513e4a1075fa 100644 --- a/tools/testing/selftests/kvm/include/x86/processor.h +++ b/tools/testing/selftests/kvm/include/x86/processor.h @@ -38,7 +38,24 @@ extern u64 guest_tsc_khz; const char *ex_str(int vector); -#define X86_EFLAGS_FIXED (1u << 1) +#define X86_EFLAGS_CF BIT(0) /* Carry Flag */ +#define X86_EFLAGS_FIXED BIT(1) /* Bit 1 - always on */ +#define X86_EFLAGS_PF BIT(2) /* Parity Flag */ +#define X86_EFLAGS_AF BIT(4) /* Auxiliary carry Flag */ +#define X86_EFLAGS_ZF BIT(6) /* Zero Flag */ +#define X86_EFLAGS_SF BIT(7) /* Sign Flag */ +#define X86_EFLAGS_TF BIT(8) /* Trap Flag */ +#define X86_EFLAGS_IF BIT(9) /* Interrupt Flag */ +#define X86_EFLAGS_DF BIT(10) /* Direction Flag */ +#define X86_EFLAGS_OF BIT(11) /* Overflow Flag */ +#define X86_EFLAGS_IOPL BIT(12) /* I/O Privilege Level (2 bits) */ +#define X86_EFLAGS_NT BIT(14) /* Nested Task */ +#define X86_EFLAGS_RF BIT(16) /* Resume Flag */ +#define X86_EFLAGS_VM BIT(17) /* Virtual Mode */ +#define X86_EFLAGS_AC BIT(18) /* Alignment Check/Access Control */ +#define X86_EFLAGS_VIF BIT(19) /* Virtual Interrupt Flag */ +#define X86_EFLAGS_VIP BIT(20) /* Virtual Interrupt Pending */ +#define X86_EFLAGS_ID BIT(21) /* CPUID detection */ #define X86_CR4_VME (1ul << 0) #define X86_CR4_PVI (1ul << 1) @@ -209,6 +226,7 @@ struct kvm_x86_cpu_feature { #define X86_FEATURE_SEV KVM_X86_CPU_FEATURE(0x8000001F, 0, EAX, 1) #define X86_FEATURE_SEV_ES KVM_X86_CPU_FEATURE(0x8000001F, 0, EAX, 3) #define X86_FEATURE_SEV_SNP KVM_X86_CPU_FEATURE(0x8000001F, 0, EAX, 4) +#define X86_FEATURE_GP_ON_USER_CPUID KVM_X86_CPU_FEATURE(0x80000021, 0, EAX, 17) #define X86_FEATURE_PERFMON_V2 KVM_X86_CPU_FEATURE(0x80000022, 0, EAX, 0) #define X86_FEATURE_LBR_PMC_FREEZE KVM_X86_CPU_FEATURE(0x80000022, 0, EAX, 2) @@ -1556,6 +1574,15 @@ u64 *tdp_get_pte(struct kvm_vm *vm, u64 l2_gpa); #define PFERR_GUEST_PAGE_MASK BIT_ULL(PFERR_GUEST_PAGE_BIT) #define PFERR_IMPLICIT_ACCESS BIT_ULL(PFERR_IMPLICIT_ACCESS_BIT) +#define EPT_VIOLATION_ACC_READ BIT(0) +#define EPT_VIOLATION_ACC_WRITE BIT(1) +#define EPT_VIOLATION_ACC_INSTR BIT(2) +#define EPT_VIOLATION_PROT_READ BIT(3) +#define EPT_VIOLATION_PROT_WRITE BIT(4) +#define EPT_VIOLATION_PROT_EXEC BIT(5) +#define EPT_VIOLATION_GVA_IS_VALID BIT(7) +#define EPT_VIOLATION_GVA_TRANSLATED BIT(8) + bool sys_clocksource_is_based_on_tsc(void); #endif /* SELFTEST_KVM_PROCESSOR_H */ diff --git a/tools/testing/selftests/kvm/include/x86/sev.h b/tools/testing/selftests/kvm/include/x86/sev.h index 1af44c151d60..dec383e59a47 100644 --- a/tools/testing/selftests/kvm/include/x86/sev.h +++ b/tools/testing/selftests/kvm/include/x86/sev.h @@ -144,4 +144,28 @@ static inline void snp_launch_update_data(struct kvm_vm *vm, gpa_t gpa, vm_sev_ioctl(vm, KVM_SEV_SNP_LAUNCH_UPDATE, &update_data); } +static inline void sev_dbg_crypt_memory(struct kvm_vm *vm, unsigned int cmd, + void *dst, void *src, unsigned int len) +{ + struct kvm_sev_dbg dbg = { + .src_uaddr = (unsigned long)src, + .dst_uaddr = (unsigned long)dst, + .len = len, + }; + + vm_sev_ioctl(vm, cmd, &dbg); +} + +static inline void sev_decrypt_memory(struct kvm_vm *vm, void *dst, void *src, + unsigned int len) +{ + sev_dbg_crypt_memory(vm, KVM_SEV_DBG_DECRYPT, dst, src, len); +} + +static inline void sev_encrypt_memory(struct kvm_vm *vm, void *dst, void *src, + unsigned int len) +{ + sev_dbg_crypt_memory(vm, KVM_SEV_DBG_ENCRYPT, dst, src, len); +} + #endif /* SELFTEST_KVM_SEV_H */ diff --git a/tools/testing/selftests/kvm/kvm_page_table_test.c b/tools/testing/selftests/kvm/kvm_page_table_test.c index fc5242fb956f..a910e3abb8c7 100644 --- a/tools/testing/selftests/kvm/kvm_page_table_test.c +++ b/tools/testing/selftests/kvm/kvm_page_table_test.c @@ -230,6 +230,7 @@ struct test_params { u64 phys_offset; u64 test_mem_size; enum vm_mem_backing_src_type src_type; + bool misalign_slot_gpa; }; static struct kvm_vm *pre_init_before_test(enum vm_guest_mode mode, void *arg) @@ -244,6 +245,7 @@ static struct kvm_vm *pre_init_before_test(enum vm_guest_mode mode, void *arg) u64 guest_num_pages; u64 alignment; void *host_test_mem; + struct userspace_mem_region *region; struct kvm_vm *vm; /* Align up the test memory size */ @@ -276,13 +278,22 @@ static struct kvm_vm *pre_init_before_test(enum vm_guest_mode mode, void *arg) /* Add an extra memory slot with specified backing src type */ vm_userspace_mem_region_add(vm, src_type, guest_test_phys_mem, TEST_MEM_SLOT_INDEX, guest_num_pages, 0); + region = memslot2region(vm, TEST_MEM_SLOT_INDEX); + host_test_mem = region->host_mem; + + if (p->misalign_slot_gpa) { + TEST_ASSERT(is_backing_src_hugetlb(src_type), + "Memslot GPA misalignment requires hugetlb backing"); + TEST_ASSERT(guest_num_pages > 1, + "Need at least two guest pages to misalign memslot GPA"); + + guest_test_phys_mem += guest_page_size; + vm_mem_region_move(vm, TEST_MEM_SLOT_INDEX, guest_test_phys_mem); + } /* Do mapping(GVA->GPA) for the testing memory slot */ virt_map(vm, guest_test_virt_mem, guest_test_phys_mem, guest_num_pages); - /* Cache the HVA pointer of the region */ - host_test_mem = addr_gpa2hva(vm, (gpa_t)guest_test_phys_mem); - /* Export shared structure test_args to guest */ sync_global_to_guest(vm, test_args); @@ -417,8 +428,8 @@ static void run_test(enum vm_guest_mode mode, void *arg) static void help(char *name) { puts(""); - printf("usage: %s [-h] [-p offset] [-m mode] " - "[-b mem-size] [-v vcpus] [-s mem-type]\n", name); + printf("usage: %s [-h] [-p offset] [-m mode] [-b mem-size]\n", name); + printf(" [-v vcpus] [-s mem-type] [-u]\n"); puts(""); printf(" -p: specify guest physical test memory offset\n" " Warning: a low offset can conflict with the loaded test code.\n"); @@ -428,6 +439,8 @@ static void help(char *name) printf(" -v: specify the number of vCPUs to run\n" " (default: 1)\n"); backing_src_help("-s"); + printf(" -u: move the test memslot GPA by one guest page after creating\n" + " the memslot, forcing a hugetlb HVA/GPA offset mismatch\n"); puts(""); } @@ -442,7 +455,7 @@ int main(int argc, char *argv[]) guest_modes_append_default(); - while ((opt = getopt(argc, argv, "hp:m:b:v:s:")) != -1) { + while ((opt = getopt(argc, argv, "hp:m:b:v:s:u")) != -1) { switch (opt) { case 'p': p.phys_offset = strtoull(optarg, NULL, 0); @@ -461,6 +474,9 @@ int main(int argc, char *argv[]) case 's': p.src_type = parse_backing_src_type(optarg); break; + case 'u': + p.misalign_slot_gpa = true; + break; case 'h': default: help(argv[0]); diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index e08967ef7b7b..195f3fdae1e3 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -77,7 +77,8 @@ static ssize_t get_module_param(const char *module_name, const char *param, int fd, r; /* Verify KVM is loaded, to provide a more helpful SKIP message. */ - close(open_kvm_dev_path_or_exit()); + fd = open_kvm_dev_path_or_exit(); + kvm_free_fd(fd); r = snprintf(path, path_size, "/sys/module/%s/parameters/%s", module_name, param); @@ -90,8 +91,7 @@ static ssize_t get_module_param(const char *module_name, const char *param, TEST_ASSERT(bytes_read > 0, "read(%s) returned %ld, wanted %ld bytes", path, bytes_read, buffer_size); - r = close(fd); - TEST_ASSERT(!r, "close(%s) failed", path); + kvm_free_fd(fd); return bytes_read; } @@ -160,7 +160,7 @@ unsigned int kvm_check_cap(long cap) ret = __kvm_ioctl(kvm_fd, KVM_CHECK_EXTENSION, (void *)cap); TEST_ASSERT(ret >= 0, KVM_IOCTL_ERROR(KVM_CHECK_EXTENSION, ret)); - close(kvm_fd); + kvm_free_fd(kvm_fd); return (unsigned int)ret; } @@ -747,8 +747,7 @@ static void kvm_stats_release(struct kvm_binary_stats *stats) stats->desc = NULL; } - kvm_close(stats->fd); - stats->fd = -1; + kvm_free_fd(stats->fd); } __weak void vcpu_arch_free(struct kvm_vcpu *vcpu) @@ -777,7 +776,7 @@ static void vm_vcpu_rm(struct kvm_vm *vm, struct kvm_vcpu *vcpu) kvm_munmap(vcpu->run, vcpu_mmap_sz()); - kvm_close(vcpu->fd); + kvm_free_fd(vcpu->fd); kvm_stats_release(&vcpu->stats); list_del(&vcpu->list); @@ -793,8 +792,8 @@ void kvm_vm_release(struct kvm_vm *vmp) list_for_each_entry_safe(vcpu, tmp, &vmp->vcpus, list) vm_vcpu_rm(vmp, vcpu); - kvm_close(vmp->fd); - kvm_close(vmp->kvm_fd); + kvm_free_fd(vmp->fd); + kvm_free_fd(vmp->kvm_fd); /* Free cached stats metadata and close FD */ kvm_stats_release(&vmp->stats); @@ -815,10 +814,10 @@ static void __vm_mem_region_delete(struct kvm_vm *vm, if (region->fd >= 0) { /* There's an extra map when using shared memory. */ kvm_munmap(region->mmap_alias, region->mmap_size); - close(region->fd); + kvm_free_fd(region->fd); } - if (region->region.guest_memfd >= 0) - close(region->region.guest_memfd); + if ((int)region->region.guest_memfd >= 0) + kvm_free_fd(region->region.guest_memfd); free(region); } @@ -1311,7 +1310,7 @@ static size_t vcpu_mmap_sz(void) TEST_ASSERT(ret >= 0 && ret >= sizeof(struct kvm_run), KVM_IOCTL_ERROR(KVM_GET_VCPU_MMAP_SIZE, ret)); - close(dev_fd); + kvm_free_fd(dev_fd); return ret; } diff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c index b51467d70f6e..4ca48de7a926 100644 --- a/tools/testing/selftests/kvm/lib/x86/processor.c +++ b/tools/testing/selftests/kvm/lib/x86/processor.c @@ -848,7 +848,7 @@ struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm, u32 vcpu_id) /* Setup guest general purpose registers */ vcpu_regs_get(vcpu, ®s); - regs.rflags = regs.rflags | 0x2; + regs.rflags = regs.rflags | X86_EFLAGS_FIXED; regs.rsp = stack_gva; vcpu_regs_set(vcpu, ®s); diff --git a/tools/testing/selftests/kvm/lib/x86/vmx.c b/tools/testing/selftests/kvm/lib/x86/vmx.c index 67642759e4a0..7c10ba6e6fb4 100644 --- a/tools/testing/selftests/kvm/lib/x86/vmx.c +++ b/tools/testing/selftests/kvm/lib/x86/vmx.c @@ -360,7 +360,7 @@ static inline void init_vmcs_guest_state(void *rip, void *rsp) vmwrite(GUEST_DR7, 0x400); vmwrite(GUEST_RSP, (u64)rsp); vmwrite(GUEST_RIP, (u64)rip); - vmwrite(GUEST_RFLAGS, 2); + vmwrite(GUEST_RFLAGS, X86_EFLAGS_FIXED); vmwrite(GUEST_PENDING_DBG_EXCEPTIONS, 0); vmwrite(GUEST_SYSENTER_ESP, vmreadz(HOST_IA32_SYSENTER_ESP)); vmwrite(GUEST_SYSENTER_EIP, vmreadz(HOST_IA32_SYSENTER_EIP)); diff --git a/tools/testing/selftests/kvm/memslot_perf_test.c b/tools/testing/selftests/kvm/memslot_perf_test.c index e977e979470f..4d9ad6104a6e 100644 --- a/tools/testing/selftests/kvm/memslot_perf_test.c +++ b/tools/testing/selftests/kvm/memslot_perf_test.c @@ -111,6 +111,7 @@ struct sync_area { */ static_assert(ATOMIC_BOOL_LOCK_FREE == 2, "atomic bool is not lockless"); +static int wait_timeout = 10; static sem_t vcpu_ready; static bool map_unmap_verify; @@ -418,7 +419,7 @@ static bool _guest_should_exit(void) */ static noinline void host_perform_sync(struct sync_area *sync) { - alarm(10); + alarm(wait_timeout); atomic_store_explicit(&sync->sync_flag, true, memory_order_release); while (atomic_load_explicit(&sync->sync_flag, memory_order_acquire)) @@ -900,7 +901,7 @@ static void help(char *name, struct test_args *targs) { int ctr; - pr_info("usage: %s [-h] [-v] [-d] [-s slots] [-f first_test] [-e last_test] [-l test_length] [-r run_count]\n", + pr_info("usage: %s [-h] [-v] [-d] [-s slots] [-f first_test] [-e last_test] [-l test_length] [-r run_count] [-t wait_timeout]\n", name); pr_info(" -h: print this help screen.\n"); pr_info(" -v: enable verbose mode (not for benchmarking).\n"); @@ -916,6 +917,8 @@ static void help(char *name, struct test_args *targs) targs->seconds); pr_info(" -r: specify the number of runs per test (currently: %i)\n", targs->runs); + pr_info(" -t: specify the number of seconds for host wait timeout (currently: %i)\n", + wait_timeout); pr_info("\nAvailable tests:\n"); for (ctr = 0; ctr < NTESTS; ctr++) @@ -964,7 +967,7 @@ static bool parse_args(int argc, char *argv[], u32 max_mem_slots; int opt; - while ((opt = getopt(argc, argv, "hvdqs:f:e:l:r:")) != -1) { + while ((opt = getopt(argc, argv, "hvdqs:f:e:l:r:t:")) != -1) { switch (opt) { case 'h': default: @@ -1007,6 +1010,9 @@ static bool parse_args(int argc, char *argv[], case 'r': targs->runs = atoi_positive("Runs per test", optarg); break; + case 't': + wait_timeout = atoi_positive("Host wait timeout", optarg); + break; } } diff --git a/tools/testing/selftests/kvm/pre_fault_memory_test.c b/tools/testing/selftests/kvm/pre_fault_memory_test.c index fcb57fd034e6..a0fcae3cb7a8 100644 --- a/tools/testing/selftests/kvm/pre_fault_memory_test.c +++ b/tools/testing/selftests/kvm/pre_fault_memory_test.c @@ -11,6 +11,7 @@ #include <kvm_util.h> #include <processor.h> #include <pthread.h> +#include <ucall_common.h> /* Arbitrarily chosen values */ #define TEST_SIZE (SZ_2M + PAGE_SIZE) @@ -167,7 +168,6 @@ static void __test_pre_fault_memory(unsigned long vm_type, bool private) .type = vm_type, }; struct kvm_vcpu *vcpu; - struct kvm_run *run; struct kvm_vm *vm; struct ucall uc; @@ -193,11 +193,6 @@ static void __test_pre_fault_memory(unsigned long vm_type, bool private) vcpu_args_set(vcpu, 1, gva); vcpu_run(vcpu); - run = vcpu->run; - TEST_ASSERT(run->exit_reason == KVM_EXIT_IO, - "Wanted KVM_EXIT_IO, got exit reason: %u (%s)", - run->exit_reason, exit_reason_str(run->exit_reason)); - switch (get_ucall(vcpu, &uc)) { case UCALL_ABORT: REPORT_GUEST_ASSERT(uc); diff --git a/tools/testing/selftests/kvm/riscv/get-reg-list.c b/tools/testing/selftests/kvm/riscv/get-reg-list.c index 8d6fdb5d38b8..cb86cb6b3635 100644 --- a/tools/testing/selftests/kvm/riscv/get-reg-list.c +++ b/tools/testing/selftests/kvm/riscv/get-reg-list.c @@ -27,6 +27,7 @@ enum { }; static bool isa_ext_cant_disable[KVM_RISCV_ISA_EXT_MAX]; +static bool sbi_ext_enabled[KVM_RISCV_SBI_EXT_MAX]; bool filter_reg(__u64 reg) { @@ -149,6 +150,14 @@ bool filter_reg(__u64 reg) case KVM_REG_RISCV_CSR | KVM_REG_RISCV_CSR_AIA | KVM_REG_RISCV_CSR_AIA_REG(iprio1h): case KVM_REG_RISCV_CSR | KVM_REG_RISCV_CSR_AIA | KVM_REG_RISCV_CSR_AIA_REG(iprio2h): return isa_ext_cant_disable[KVM_RISCV_ISA_EXT_SSAIA]; + /* + * FWFT misaligned delegation registers are always visible when the SBI FWFT + * extension is enable and the host supports the misaligned delegation. + */ + case KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(misaligned_deleg.enable): + case KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(misaligned_deleg.flags): + case KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(misaligned_deleg.value): + return sbi_ext_enabled[KVM_RISCV_SBI_EXT_FWFT]; default: break; } @@ -193,6 +202,27 @@ static int override_vector_reg_size(struct kvm_vcpu *vcpu, struct vcpu_reg_subli return 0; } +void check_fwft_feature(struct kvm_vcpu *vcpu, struct vcpu_reg_sublist *s, u64 feature) +{ + unsigned long value; + int rc; + + /* Enable SBI FWFT extension so that we can check the supported register */ + rc = __vcpu_set_reg(vcpu, feature, 1); + if (rc) + return; + + for (int i = 0; i < s->regs_n; i++) { + if ((s->regs[i] & KVM_REG_RISCV_TYPE_MASK) == KVM_REG_RISCV_SBI_STATE) { + rc = __vcpu_get_reg(vcpu, s->regs[i], &value); + __TEST_REQUIRE(!rc, "%s not available, skipping tests", s->name); + } + } + + /* We should assert if disabling failed here while enabling succeeded before */ + vcpu_set_reg(vcpu, feature, 0); +} + void finalize_vcpu(struct kvm_vcpu *vcpu, struct vcpu_reg_list *c) { unsigned long isa_ext_state[KVM_RISCV_ISA_EXT_MAX] = { 0 }; @@ -235,6 +265,9 @@ void finalize_vcpu(struct kvm_vcpu *vcpu, struct vcpu_reg_list *c) break; case VCPU_FEATURE_SBI_EXT: feature = RISCV_SBI_EXT_REG(s->feature); + if (s->feature == KVM_RISCV_SBI_EXT_FWFT) + check_fwft_feature(vcpu, s, feature); + sbi_ext_enabled[s->feature] = true; break; default: TEST_FAIL("Unknown feature type"); @@ -897,11 +930,15 @@ static __u64 sbi_sta_regs[] = { KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_STA | KVM_REG_RISCV_SBI_STA_REG(shmem_hi), }; -static __u64 sbi_fwft_regs[] = { +static __u64 sbi_fwft_misaligned_deleg_regs[] = { KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_FWFT, KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(misaligned_deleg.enable), KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(misaligned_deleg.flags), KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(misaligned_deleg.value), +}; + +static __u64 sbi_fwft_pointer_masking_regs[] = { + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_FWFT, KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pointer_masking.enable), KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pointer_masking.flags), KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pointer_masking.value), @@ -1013,7 +1050,7 @@ static __u64 fp_d_regs[] = { }; /* Define a default vector registers with length. This will be overwritten at runtime */ -static __u64 vector_regs[] = { +static __u64 v_regs[] = { KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_VECTOR | KVM_REG_RISCV_VECTOR_CSR_REG(vstart), KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_VECTOR | KVM_REG_RISCV_VECTOR_CSR_REG(vl), KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_VECTOR | KVM_REG_RISCV_VECTOR_CSR_REG(vtype), @@ -1057,37 +1094,17 @@ static __u64 vector_regs[] = { #define SUBLIST_BASE \ {"base", .regs = base_regs, .regs_n = ARRAY_SIZE(base_regs), \ .skips_set = base_skips_set, .skips_set_n = ARRAY_SIZE(base_skips_set),} -#define SUBLIST_SBI_BASE \ - {"sbi-base", .feature_type = VCPU_FEATURE_SBI_EXT, .feature = KVM_RISCV_SBI_EXT_V01, \ - .regs = sbi_base_regs, .regs_n = ARRAY_SIZE(sbi_base_regs),} -#define SUBLIST_SBI_STA \ - {"sbi-sta", .feature_type = VCPU_FEATURE_SBI_EXT, .feature = KVM_RISCV_SBI_EXT_STA, \ - .regs = sbi_sta_regs, .regs_n = ARRAY_SIZE(sbi_sta_regs),} -#define SUBLIST_SBI_FWFT \ - {"sbi-fwft", .feature_type = VCPU_FEATURE_SBI_EXT, .feature = KVM_RISCV_SBI_EXT_FWFT, \ - .regs = sbi_fwft_regs, .regs_n = ARRAY_SIZE(sbi_fwft_regs),} -#define SUBLIST_ZICBOM \ - {"zicbom", .feature = KVM_RISCV_ISA_EXT_ZICBOM, .regs = zicbom_regs, .regs_n = ARRAY_SIZE(zicbom_regs),} -#define SUBLIST_ZICBOP \ - {"zicbop", .feature = KVM_RISCV_ISA_EXT_ZICBOP, .regs = zicbop_regs, .regs_n = ARRAY_SIZE(zicbop_regs),} -#define SUBLIST_ZICBOZ \ - {"zicboz", .feature = KVM_RISCV_ISA_EXT_ZICBOZ, .regs = zicboz_regs, .regs_n = ARRAY_SIZE(zicboz_regs),} -#define SUBLIST_AIA \ - {"aia", .feature = KVM_RISCV_ISA_EXT_SSAIA, .regs = aia_regs, .regs_n = ARRAY_SIZE(aia_regs),} -#define SUBLIST_SMSTATEEN \ - {"smstateen", .feature = KVM_RISCV_ISA_EXT_SMSTATEEN, .regs = smstateen_regs, .regs_n = ARRAY_SIZE(smstateen_regs),} -#define SUBLIST_FP_F \ - {"fp_f", .feature = KVM_RISCV_ISA_EXT_F, .regs = fp_f_regs, \ - .regs_n = ARRAY_SIZE(fp_f_regs),} -#define SUBLIST_FP_D \ - {"fp_d", .feature = KVM_RISCV_ISA_EXT_D, .regs = fp_d_regs, \ - .regs_n = ARRAY_SIZE(fp_d_regs),} - -#define SUBLIST_V \ - {"v", .feature = KVM_RISCV_ISA_EXT_V, .regs = vector_regs, .regs_n = ARRAY_SIZE(vector_regs),} + +#define SUBLIST_ISA(ext, extu) \ + { \ + .name = #ext, \ + .feature = KVM_RISCV_ISA_EXT_##extu, \ + .regs = ext##_regs, \ + .regs_n = ARRAY_SIZE(ext##_regs), \ + } #define KVM_ISA_EXT_SIMPLE_CONFIG(ext, extu) \ -static __u64 regs_##ext[] = { \ +static __u64 ext##_regs[] = { \ KVM_REG_RISCV | KVM_REG_SIZE_ULONG | \ KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | \ KVM_RISCV_ISA_EXT_##extu, \ @@ -1095,18 +1112,22 @@ static __u64 regs_##ext[] = { \ static struct vcpu_reg_list config_##ext = { \ .sublists = { \ SUBLIST_BASE, \ - { \ - .name = #ext, \ - .feature = KVM_RISCV_ISA_EXT_##extu, \ - .regs = regs_##ext, \ - .regs_n = ARRAY_SIZE(regs_##ext), \ - }, \ + SUBLIST_ISA(ext, extu), \ {0}, \ }, \ } \ +#define SUBLIST_SBI(ext, extu) \ + { \ + .name = "sbi-"#ext, \ + .feature_type = VCPU_FEATURE_SBI_EXT, \ + .feature = KVM_RISCV_SBI_EXT_##extu, \ + .regs = sbi_##ext##_regs, \ + .regs_n = ARRAY_SIZE(sbi_##ext##_regs), \ + } + #define KVM_SBI_EXT_SIMPLE_CONFIG(ext, extu) \ -static __u64 regs_sbi_##ext[] = { \ +static __u64 sbi_##ext##_regs[] = { \ KVM_REG_RISCV | KVM_REG_SIZE_ULONG | \ KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | \ KVM_RISCV_SBI_EXT_##extu, \ @@ -1114,13 +1135,7 @@ static __u64 regs_sbi_##ext[] = { \ static struct vcpu_reg_list config_sbi_##ext = { \ .sublists = { \ SUBLIST_BASE, \ - { \ - .name = "sbi-"#ext, \ - .feature_type = VCPU_FEATURE_SBI_EXT, \ - .feature = KVM_RISCV_SBI_EXT_##extu, \ - .regs = regs_sbi_##ext, \ - .regs_n = ARRAY_SIZE(regs_sbi_##ext), \ - }, \ + SUBLIST_SBI(ext, extu), \ {0}, \ }, \ } \ @@ -1129,7 +1144,7 @@ static struct vcpu_reg_list config_sbi_##ext = { \ static struct vcpu_reg_list config_##ext = { \ .sublists = { \ SUBLIST_BASE, \ - SUBLIST_##extu, \ + SUBLIST_ISA(ext, extu), \ {0}, \ }, \ } \ @@ -1138,24 +1153,23 @@ static struct vcpu_reg_list config_##ext = { \ static struct vcpu_reg_list config_sbi_##ext = { \ .sublists = { \ SUBLIST_BASE, \ - SUBLIST_SBI_##extu, \ + SUBLIST_SBI(ext, extu), \ {0}, \ }, \ } \ /* Note: The below list is alphabetically sorted. */ -KVM_SBI_EXT_SUBLIST_CONFIG(base, BASE); +KVM_SBI_EXT_SUBLIST_CONFIG(base, V01); KVM_SBI_EXT_SUBLIST_CONFIG(sta, STA); KVM_SBI_EXT_SIMPLE_CONFIG(pmu, PMU); KVM_SBI_EXT_SIMPLE_CONFIG(dbcn, DBCN); KVM_SBI_EXT_SIMPLE_CONFIG(susp, SUSP); KVM_SBI_EXT_SIMPLE_CONFIG(mpxy, MPXY); -KVM_SBI_EXT_SUBLIST_CONFIG(fwft, FWFT); -KVM_ISA_EXT_SUBLIST_CONFIG(aia, AIA); -KVM_ISA_EXT_SUBLIST_CONFIG(fp_f, FP_F); -KVM_ISA_EXT_SUBLIST_CONFIG(fp_d, FP_D); +KVM_ISA_EXT_SUBLIST_CONFIG(aia, SSAIA); +KVM_ISA_EXT_SUBLIST_CONFIG(fp_f, F); +KVM_ISA_EXT_SUBLIST_CONFIG(fp_d, D); KVM_ISA_EXT_SUBLIST_CONFIG(v, V); KVM_ISA_EXT_SIMPLE_CONFIG(h, H); KVM_ISA_EXT_SIMPLE_CONFIG(smnpm, SMNPM); @@ -1228,6 +1242,23 @@ KVM_ISA_EXT_SIMPLE_CONFIG(zvksed, ZVKSED); KVM_ISA_EXT_SIMPLE_CONFIG(zvksh, ZVKSH); KVM_ISA_EXT_SIMPLE_CONFIG(zvkt, ZVKT); +static struct vcpu_reg_list config_sbi_fwft_misaligned_deleg = { + .sublists = { + SUBLIST_BASE, + SUBLIST_SBI(fwft_misaligned_deleg, FWFT), + {0}, + }, +}; + +static struct vcpu_reg_list config_sbi_fwft_pointer_masking = { + .sublists = { + SUBLIST_BASE, + SUBLIST_ISA(smnpm, SMNPM), + SUBLIST_SBI(fwft_pointer_masking, FWFT), + {0}, + }, +}; + struct vcpu_reg_list *vcpu_configs[] = { &config_sbi_base, &config_sbi_sta, @@ -1235,7 +1266,8 @@ struct vcpu_reg_list *vcpu_configs[] = { &config_sbi_dbcn, &config_sbi_susp, &config_sbi_mpxy, - &config_sbi_fwft, + &config_sbi_fwft_misaligned_deleg, + &config_sbi_fwft_pointer_masking, &config_aia, &config_fp_f, &config_fp_d, diff --git a/tools/testing/selftests/kvm/set_memory_region_test.c b/tools/testing/selftests/kvm/set_memory_region_test.c index e639a9db51ee..a152ab65c657 100644 --- a/tools/testing/selftests/kvm/set_memory_region_test.c +++ b/tools/testing/selftests/kvm/set_memory_region_test.c @@ -510,7 +510,7 @@ static void test_add_overlapping_private_memory_regions(void) vm = vm_create_barebones_type(KVM_X86_SW_PROTECTED_VM); - memfd = vm_create_guest_memfd(vm, MEM_REGION_SIZE * 4, 0); + memfd = vm_create_guest_memfd(vm, MEM_REGION_SIZE * 5, 0); vm_set_user_memory_region2(vm, MEM_REGION_SLOT, KVM_MEM_GUEST_MEMFD, MEM_REGION_GPA, MEM_REGION_SIZE * 2, 0, memfd, 0); @@ -526,12 +526,35 @@ static void test_add_overlapping_private_memory_regions(void) vm_set_user_memory_region2(vm, MEM_REGION_SLOT, KVM_MEM_GUEST_MEMFD, MEM_REGION_GPA, 0, NULL, -1, 0); - /* Overlap the front half of the other slot. */ + /* + * Verify that overlap in the guest_memfd bindings (i.e. in guest_memfd + * file offsets), but _not_ in the GPA space, fails with -EEXIST. + */ + r = __vm_set_user_memory_region2(vm, MEM_REGION_SLOT, KVM_MEM_GUEST_MEMFD, + MEM_REGION_GPA, + MEM_REGION_SIZE * 2, + 0, memfd, MEM_REGION_SIZE); + TEST_ASSERT(r == -1 && errno == EEXIST, + "Overlapping guest_memfd() bindings should fail with EEXIST"); + + /* And now the back half of the other slot's guest_memfd binding. */ + r = __vm_set_user_memory_region2(vm, MEM_REGION_SLOT, KVM_MEM_GUEST_MEMFD, + MEM_REGION_GPA, + MEM_REGION_SIZE * 2, + 0, memfd, MEM_REGION_SIZE * 3); + TEST_ASSERT(r == -1 && errno == EEXIST, + "Overlapping guest_memfd() bindings should fail with EEXIST"); + + /* + * Repeat the overlap tests, but this time with overlap in the memslots + * GPA space. Regardless of where there is overlap, KVM should return + * -EEXIST. + */ r = __vm_set_user_memory_region2(vm, MEM_REGION_SLOT, KVM_MEM_GUEST_MEMFD, MEM_REGION_GPA * 2 - MEM_REGION_SIZE, MEM_REGION_SIZE * 2, 0, memfd, 0); - TEST_ASSERT(r == -1 && errno == EEXIST, "%s", + TEST_ASSERT(r == -1 && errno == EEXIST, "Overlapping guest_memfd() bindings should fail with EEXIST"); /* And now the back half of the other slot. */ @@ -539,7 +562,7 @@ static void test_add_overlapping_private_memory_regions(void) MEM_REGION_GPA * 2 + MEM_REGION_SIZE, MEM_REGION_SIZE * 2, 0, memfd, 0); - TEST_ASSERT(r == -1 && errno == EEXIST, "%s", + TEST_ASSERT(r == -1 && errno == EEXIST, "Overlapping guest_memfd() bindings should fail with EEXIST"); close(memfd); diff --git a/tools/testing/selftests/kvm/x86/debug_regs.c b/tools/testing/selftests/kvm/x86/debug_regs.c index 0dfaf03cd0a0..2a2ef3e179ff 100644 --- a/tools/testing/selftests/kvm/x86/debug_regs.c +++ b/tools/testing/selftests/kvm/x86/debug_regs.c @@ -15,10 +15,51 @@ #define IRQ_VECTOR 0xAA +#define CAST_TO_RIP(v) ((unsigned long long)&(v)) + /* For testing data access debug BP */ u32 guest_value; extern unsigned char sw_bp, hw_bp, write_data, ss_start, bd_start; +extern unsigned char fep_bd_start, fep_sti_start, fep_sti_end; + +static int irqs_received; + +static void guest_db_handler(struct ex_regs *regs) +{ + static int count; + unsigned long target_rips[2] = { + CAST_TO_RIP(fep_sti_start), + CAST_TO_RIP(fep_sti_end), + }; + + __GUEST_ASSERT(regs->rip == target_rips[count], + "STI[%u]: unexpected rip 0x%lx (should be 0x%lx)", + count, regs->rip, target_rips[count]); + regs->rflags &= ~X86_EFLAGS_TF; + count++; +} + +static void guest_irq_handler(struct ex_regs *regs) +{ + /* + * The pending IRQ should finally be take when KVM_GUESTDBG_BLOCKIRQ is + * cleared and IRQs are enabled. Note, the IRQ is expected to arrive + * on the instruction immediately after STI, even though its in an STI + * shadow. Because the next instruction has a coincident #DB, and #DBs + * are not subject to STI-blocking, the #DB will push RFLAGS.IF=1 on + * the stack, and the eventual IRET will unmask IRQs and obliterate the + * STI shadow in the process. + */ + unsigned long target_rip = CAST_TO_RIP(fep_sti_start); + + __GUEST_ASSERT(regs->rip == target_rip, + "IRQ: unexpected rip 0x%lx (should be 0x%lx)", + regs->rip, target_rip); + + irqs_received++; + x2apic_write_reg(APIC_EOI, 0); +} static void guest_code(void) { @@ -64,11 +105,33 @@ static void guest_code(void) /* DR6.BD test */ asm volatile("bd_start: mov %%dr0, %%rax" : : : "rax"); + + /* + * Note, the IRET from the #DB that occurs in the below STI-shadow will + * unmask IRQs, i.e. the pending interrupt will be delivered after #DB + * handling, on the CLI! + */ + if (is_forced_emulation_enabled) { + asm volatile(KVM_FEP "fep_bd_start: mov %%dr0, %%rax" : : : "rax"); + + /* pending debug exceptions for emulation */ + asm volatile("pushf\n\t" + "orq $" __stringify(X86_EFLAGS_TF) ", (%rsp)\n\t" + "popf\n\t" + "sti\n\t" + "fep_sti_start:" + "cli\n\t" + "pushf\n\t" + "orq $" __stringify(X86_EFLAGS_TF) ", (%rsp)\n\t" + "popf\n\t" + KVM_FEP "sti\n\t" + "fep_sti_end:" + "cli\n\t"); + GUEST_ASSERT(irqs_received == 1); + } GUEST_DONE(); } -#define CAST_TO_RIP(v) ((unsigned long long)&(v)) - static void vcpu_skip_insn(struct kvm_vcpu *vcpu, int insn_len) { struct kvm_regs regs; @@ -185,7 +248,7 @@ int main(void) target_dr6); } - /* Finally test global disable */ + /* test global disable */ memset(&debug, 0, sizeof(debug)); debug.control = KVM_GUESTDBG_ENABLE | KVM_GUESTDBG_USE_HW_BP; debug.arch.debugreg[7] = 0x400 | DR7_GD; @@ -202,10 +265,29 @@ int main(void) run->debug.arch.pc, target_rip, run->debug.arch.dr6, target_dr6); + /* test global disable in emulation */ + if (is_forced_emulation_enabled) { + /* Skip the 3-bytes "mov dr0" */ + vcpu_skip_insn(vcpu, 3); + vcpu_run(vcpu); + TEST_ASSERT(run->exit_reason == KVM_EXIT_DEBUG && + run->debug.arch.exception == DB_VECTOR && + run->debug.arch.pc == CAST_TO_RIP(fep_bd_start) && + run->debug.arch.dr6 == target_dr6, + "DR7.GD: exit %d exception %d rip 0x%llx " + "(should be 0x%llx) dr6 0x%llx (should be 0x%llx)", + run->exit_reason, run->debug.arch.exception, + run->debug.arch.pc, CAST_TO_RIP(fep_bd_start), + run->debug.arch.dr6, target_dr6); + } + /* Disable all debug controls, run to the end */ memset(&debug, 0, sizeof(debug)); vcpu_guest_debug_set(vcpu, &debug); + vm_install_exception_handler(vm, DB_VECTOR, guest_db_handler); + vm_install_exception_handler(vm, IRQ_VECTOR, guest_irq_handler); + vcpu_run(vcpu); TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); cmd = get_ucall(vcpu, &uc); diff --git a/tools/testing/selftests/kvm/x86/hwcr_msr_test.c b/tools/testing/selftests/kvm/x86/hwcr_msr_test.c index 8e20a03b3329..53b7971aa072 100644 --- a/tools/testing/selftests/kvm/x86/hwcr_msr_test.c +++ b/tools/testing/selftests/kvm/x86/hwcr_msr_test.c @@ -11,12 +11,17 @@ void test_hwcr_bit(struct kvm_vcpu *vcpu, unsigned int bit) { const u64 ignored = BIT_ULL(3) | BIT_ULL(6) | BIT_ULL(8); - const u64 valid = BIT_ULL(18) | BIT_ULL(24); - const u64 legal = ignored | valid; + u64 valid = BIT_ULL(18) | BIT_ULL(24); u64 val = BIT_ULL(bit); u64 actual; + u64 legal; int r; + if (kvm_cpu_has(X86_FEATURE_GP_ON_USER_CPUID)) + valid |= BIT_ULL(35); + + legal = ignored | valid; + r = _vcpu_set_msr(vcpu, MSR_K7_HWCR, val); TEST_ASSERT(val & ~legal ? !r : r == 1, "Expected KVM_SET_MSRS(MSR_K7_HWCR) = 0x%lx to %s", diff --git a/tools/testing/selftests/kvm/x86/hyperv_features.c b/tools/testing/selftests/kvm/x86/hyperv_features.c index 7347f1fe5157..2effde85c4c8 100644 --- a/tools/testing/selftests/kvm/x86/hyperv_features.c +++ b/tools/testing/selftests/kvm/x86/hyperv_features.c @@ -26,6 +26,7 @@ struct msr_data { bool fault_expected; bool write; u64 write_val; + bool reset_expected; }; struct hcall_data { @@ -267,14 +268,9 @@ static void guest_test_msrs_access(void) case 16: msr->idx = HV_X64_MSR_RESET; msr->write = true; - /* - * TODO: the test only writes '0' to HV_X64_MSR_RESET - * at the moment, writing some other value there will - * trigger real vCPU reset and the code is not prepared - * to handle it yet. - */ - msr->write_val = 0; + msr->write_val = 1; msr->fault_expected = false; + msr->reset_expected = true; break; case 17: @@ -457,7 +453,7 @@ static void guest_test_msrs_access(void) msr->fault_expected = true; break; case 45: - /* MSR is vailable when CPUID feature bit is set */ + /* MSR is available when CPUID feature bit is set */ if (!has_invtsc) goto next_stage; vcpu_set_cpuid_feature(vcpu, HV_ACCESS_TSC_INVARIANT); @@ -497,6 +493,15 @@ static void guest_test_msrs_access(void) msr->idx, msr->write ? "write" : "read"); vcpu_run(vcpu); + + if (msr->reset_expected) { + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_SYSTEM_EVENT); + TEST_ASSERT(vcpu->run->system_event.type == KVM_SYSTEM_EVENT_RESET, + "Expected reset system event, got type %u", + vcpu->run->system_event.type); + goto next_stage; + } + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); switch (get_ucall(vcpu, &uc)) { diff --git a/tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c b/tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c index 15ee8b7bfc11..b4be9a175379 100644 --- a/tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c +++ b/tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c @@ -142,17 +142,6 @@ static void swap_two_test_pages(gpa_t pte_gva1, gpa_t pte_gva2) } /* - * TODO: replace the silly NOP loop with a proper udelay() implementation. - */ -static inline void do_delay(void) -{ - int i; - - for (i = 0; i < 1000000; i++) - asm volatile("nop"); -} - -/* * Prepare to test: 'disable' workers by setting the expectation to '0', * clear hypercall input page and then swap two test pages. */ @@ -169,7 +158,7 @@ static inline void prepare_to_test(struct test_data *data) wmb(); /* Make sure workers have enough time to notice */ - do_delay(); + udelay(100); /* Swap test page mappings */ swap_two_test_pages(data->test_pages_pte[0], data->test_pages_pte[1]); @@ -189,7 +178,7 @@ static inline void post_test(struct test_data *data, u64 exp1, u64 exp2) set_expected_val((void *)data->test_pages, exp2, WORKER_VCPU_ID_2); /* Make sure workers have enough time to test */ - do_delay(); + udelay(100); } #define TESTVAL1 0x0101010101010101 diff --git a/tools/testing/selftests/kvm/x86/nested_tdp_fault_test.c b/tools/testing/selftests/kvm/x86/nested_tdp_fault_test.c new file mode 100644 index 000000000000..fa95568f55ff --- /dev/null +++ b/tools/testing/selftests/kvm/x86/nested_tdp_fault_test.c @@ -0,0 +1,313 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025, Google, Inc. + */ + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" +#include "svm_util.h" +#include "vmx.h" + +#define L2_GUEST_STACK_SIZE 64 + +enum test_type { + TEST_FINAL_PAGE_UNMAPPED, /* Final data page not present */ + TEST_PT_PAGE_UNMAPPED, /* Page table page not present */ + TEST_FINAL_PAGE_WRITE_PROTECTED, /* Final data page read-only */ + TEST_PT_PAGE_WRITE_PROTECTED, /* Page table page read-only */ +}; + +static gva_t l2_test_page; +static void (*l2_entry)(void); + +#define TEST_IO_PORT 0x80 +#define TEST1_VADDR 0x8000000ULL +#define TEST2_VADDR 0x10000000ULL +#define TEST3_VADDR 0x18000000ULL +#define TEST4_VADDR 0x20000000ULL + +/* + * L2 executes OUTS reading from l2_test_page, triggering a nested page + * fault on the read access. + */ +static void l2_guest_code_outs(void) +{ + asm volatile("outsb" ::"S"(l2_test_page), "d"(TEST_IO_PORT) : "memory"); + GUEST_FAIL("L2 should not reach here"); +} + +/* + * L2 executes INS writing to l2_test_page, triggering a nested page + * fault on the write access. + */ +static void l2_guest_code_ins(void) +{ + asm volatile("insb" ::"D"(l2_test_page), "d"(TEST_IO_PORT) : "memory"); + GUEST_FAIL("L2 should not reach here"); +} + +#define GUEST_ASSERT_EXIT_QUAL(ac_eq, ex_eq) \ + __GUEST_ASSERT((ac_eq) == (ex_eq), \ + "Wanted EXIT_QUAL '0x%lx', got '0x%lx'", ex_eq, ac_eq) + +static void l1_vmx_code(struct vmx_pages *vmx, u64 expected_fault_gpa, + u64 test_type) +{ + unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; + u64 exit_qual; + + GUEST_ASSERT(vmx->vmcs_gpa); + GUEST_ASSERT(prepare_for_vmx_operation(vmx)); + GUEST_ASSERT(load_vmcs(vmx)); + + prepare_vmcs(vmx, l2_entry, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + + GUEST_ASSERT(!vmlaunch()); + + /* Verify we got an EPT violation exit */ + __GUEST_ASSERT(vmreadz(VM_EXIT_REASON) == EXIT_REASON_EPT_VIOLATION, + "Expected EPT violation (0x%x), got 0x%lx", + EXIT_REASON_EPT_VIOLATION, + vmreadz(VM_EXIT_REASON)); + + __GUEST_ASSERT(vmreadz(GUEST_PHYSICAL_ADDRESS) == expected_fault_gpa, + "Expected guest_physical_address = 0x%lx, got 0x%lx", + expected_fault_gpa, + vmreadz(GUEST_PHYSICAL_ADDRESS)); + + exit_qual = vmreadz(EXIT_QUALIFICATION); + + /* + * Note, EPT page table accesses are always read+write, e.g. so that + * the CPU can do A/D updates at-will. + */ + switch (test_type) { + case TEST_FINAL_PAGE_UNMAPPED: + GUEST_ASSERT_EXIT_QUAL(exit_qual, EPT_VIOLATION_ACC_READ | + EPT_VIOLATION_GVA_IS_VALID | + EPT_VIOLATION_GVA_TRANSLATED); + break; + case TEST_PT_PAGE_UNMAPPED: + GUEST_ASSERT_EXIT_QUAL(exit_qual, EPT_VIOLATION_ACC_READ | + EPT_VIOLATION_ACC_WRITE | + EPT_VIOLATION_GVA_IS_VALID); + break; + case TEST_FINAL_PAGE_WRITE_PROTECTED: + GUEST_ASSERT_EXIT_QUAL(exit_qual, EPT_VIOLATION_ACC_WRITE | + EPT_VIOLATION_PROT_READ | + EPT_VIOLATION_PROT_EXEC | + EPT_VIOLATION_GVA_IS_VALID | + EPT_VIOLATION_GVA_TRANSLATED); + break; + case TEST_PT_PAGE_WRITE_PROTECTED: + GUEST_ASSERT_EXIT_QUAL(exit_qual, EPT_VIOLATION_ACC_READ | + EPT_VIOLATION_ACC_WRITE | + EPT_VIOLATION_PROT_READ | + EPT_VIOLATION_PROT_EXEC | + EPT_VIOLATION_GVA_IS_VALID); + break; + } + + GUEST_DONE(); +} + +#define GUEST_ASSERT_NPF_EC(ac_ec, ex_ec) \ + __GUEST_ASSERT((ac_ec) == (ex_ec), \ + "Wanted NPF error code '0x%lx', got '0x%lx'", (u64)(ex_ec), ac_ec) + + +static void l1_svm_code(struct svm_test_data *svm, u64 expected_fault_gpa, + u64 test_type) +{ + unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; + struct vmcb *vmcb = svm->vmcb; + u64 exit_info_1; + + generic_svm_setup(svm, l2_entry, + &l2_guest_stack[L2_GUEST_STACK_SIZE]); + + run_guest(vmcb, svm->vmcb_gpa); + + /* Verify we got an NPF exit */ + __GUEST_ASSERT(vmcb->control.exit_code == SVM_EXIT_NPF, + "Expected NPF exit (0x%x), got 0x%lx", SVM_EXIT_NPF, + vmcb->control.exit_code); + + __GUEST_ASSERT(vmcb->control.exit_info_2 == expected_fault_gpa, + "Expected exit_info_2 = 0x%lx, got 0x%lx", + expected_fault_gpa, + vmcb->control.exit_info_2); + + exit_info_1 = vmcb->control.exit_info_1; + + /* + * Note, without GMET enabled, NPT walks are always user accesses. And + * like EPT, page table accesses are always read+write. + */ + switch (test_type) { + case TEST_FINAL_PAGE_UNMAPPED: + GUEST_ASSERT_NPF_EC(exit_info_1, PFERR_USER_MASK | + PFERR_GUEST_FINAL_MASK); + break; + case TEST_PT_PAGE_UNMAPPED: + GUEST_ASSERT_NPF_EC(exit_info_1, PFERR_WRITE_MASK | + PFERR_USER_MASK | + PFERR_GUEST_PAGE_MASK); + break; + case TEST_FINAL_PAGE_WRITE_PROTECTED: + GUEST_ASSERT_NPF_EC(exit_info_1, PFERR_PRESENT_MASK | + PFERR_WRITE_MASK | + PFERR_USER_MASK | + PFERR_GUEST_FINAL_MASK); + break; + case TEST_PT_PAGE_WRITE_PROTECTED: + GUEST_ASSERT_NPF_EC(exit_info_1, PFERR_PRESENT_MASK | + PFERR_WRITE_MASK | + PFERR_USER_MASK | + PFERR_GUEST_PAGE_MASK); + break; + } + + GUEST_DONE(); +} + +static void l1_guest_code(void *data, u64 expected_fault_gpa, + u64 test_type) +{ + if (this_cpu_has(X86_FEATURE_VMX)) + l1_vmx_code(data, expected_fault_gpa, test_type); + else + l1_svm_code(data, expected_fault_gpa, test_type); +} + +/* Returns the GPA of the PT page that maps @vaddr. */ +static u64 get_pt_gpa_for_vaddr(struct kvm_vm *vm, u64 vaddr) +{ + u64 *pte; + + pte = vm_get_pte(vm, vaddr); + TEST_ASSERT(pte && (*pte & 0x1), "PTE not present for vaddr 0x%lx", + (unsigned long)vaddr); + + return addr_hva2gpa(vm, (void *)((u64)pte & ~0xFFFULL)); +} + +static void run_test(enum test_type type) +{ + gpa_t expected_fault_gpa; + gva_t nested_gva; + + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + struct ucall uc; + + vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code); + vm_enable_tdp(vm); + + if (kvm_cpu_has(X86_FEATURE_VMX)) + vcpu_alloc_vmx(vm, &nested_gva); + else + vcpu_alloc_svm(vm, &nested_gva); + + switch (type) { + case TEST_FINAL_PAGE_UNMAPPED: + /* + * Unmap the final data page from NPT/EPT. The guest page + * table walk succeeds, but the final GPA->HPA translation + * fails. L2 reads from the page via OUTS. + */ + l2_entry = l2_guest_code_outs; + l2_test_page = vm_alloc(vm, vm->page_size, TEST1_VADDR); + expected_fault_gpa = addr_gva2gpa(vm, l2_test_page); + break; + case TEST_PT_PAGE_UNMAPPED: + /* + * Unmap a page table page from NPT/EPT. The hardware page + * table walk fails when translating the PT page's GPA + * through NPT/EPT. L2 reads from the page via OUTS. + */ + l2_entry = l2_guest_code_outs; + l2_test_page = vm_alloc(vm, vm->page_size, TEST2_VADDR); + expected_fault_gpa = get_pt_gpa_for_vaddr(vm, l2_test_page); + break; + case TEST_FINAL_PAGE_WRITE_PROTECTED: + /* + * Write-protect the final data page in NPT/EPT. The page + * is present and readable, but not writable. L2 writes to + * the page via INS, triggering a protection violation. + */ + l2_entry = l2_guest_code_ins; + l2_test_page = vm_alloc(vm, vm->page_size, TEST3_VADDR); + expected_fault_gpa = addr_gva2gpa(vm, l2_test_page); + break; + case TEST_PT_PAGE_WRITE_PROTECTED: + /* + * Write-protect a page table page in NPT/EPT. The page is + * present and readable, but not writable. The guest page + * table walk needs write access to set A/D bits, so it + * triggers a protection violation on the PT page. + * L2 reads from the page via OUTS. + */ + l2_entry = l2_guest_code_outs; + l2_test_page = vm_alloc(vm, vm->page_size, TEST4_VADDR); + expected_fault_gpa = get_pt_gpa_for_vaddr(vm, l2_test_page); + break; + } + + tdp_identity_map_default_memslots(vm); + + if (type == TEST_FINAL_PAGE_WRITE_PROTECTED || + type == TEST_PT_PAGE_WRITE_PROTECTED) + *tdp_get_pte(vm, expected_fault_gpa) &= ~PTE_WRITABLE_MASK(&vm->stage2_mmu); + else + *tdp_get_pte(vm, expected_fault_gpa) &= ~(PTE_PRESENT_MASK(&vm->stage2_mmu) | + PTE_READABLE_MASK(&vm->stage2_mmu) | + PTE_WRITABLE_MASK(&vm->stage2_mmu) | + PTE_EXECUTABLE_MASK(&vm->stage2_mmu)); + + sync_global_to_guest(vm, l2_entry); + sync_global_to_guest(vm, l2_test_page); + vcpu_args_set(vcpu, 3, nested_gva, expected_fault_gpa, (u64)type); + + /* + * For the INS-based write test, KVM emulates the instruction and + * first reads from the I/O port, which exits to userspace. + * Re-enter the guest so emulation can proceed to the memory + * write, where the nested page fault is triggered. + */ + for (;;) { + vcpu_run(vcpu); + + if (vcpu->run->exit_reason == KVM_EXIT_IO && + vcpu->run->io.port == TEST_IO_PORT && + vcpu->run->io.direction == KVM_EXIT_IO_IN) { + continue; + } + break; + } + + switch (get_ucall(vcpu, &uc)) { + case UCALL_DONE: + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + default: + TEST_FAIL("Unexpected exit reason: %d", vcpu->run->exit_reason); + } + + kvm_vm_free(vm); +} + +int main(int argc, char *argv[]) +{ + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX) || kvm_cpu_has(X86_FEATURE_SVM)); + TEST_REQUIRE(kvm_cpu_has_tdp()); + + run_test(TEST_FINAL_PAGE_UNMAPPED); + run_test(TEST_PT_PAGE_UNMAPPED); + run_test(TEST_FINAL_PAGE_WRITE_PROTECTED); + run_test(TEST_PT_PAGE_WRITE_PROTECTED); + + return 0; +} diff --git a/tools/testing/selftests/kvm/x86/pmu_event_filter_test.c b/tools/testing/selftests/kvm/x86/pmu_event_filter_test.c index c1232344fda8..84e4c6ca67a3 100644 --- a/tools/testing/selftests/kvm/x86/pmu_event_filter_test.c +++ b/tools/testing/selftests/kvm/x86/pmu_event_filter_test.c @@ -731,6 +731,8 @@ static void test_filter_ioctl(struct kvm_vcpu *vcpu) static void intel_run_fixed_counter_guest_code(u8 idx) { + u8 nr_fixed_counters = this_cpu_property(X86_PROPERTY_PMU_NR_FIXED_COUNTERS); + for (;;) { wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, 0); wrmsr(MSR_CORE_PERF_FIXED_CTR0 + idx, 0); @@ -738,6 +740,10 @@ static void intel_run_fixed_counter_guest_code(u8 idx) /* Only OS_EN bit is enabled for fixed counter[idx]. */ wrmsr(MSR_CORE_PERF_FIXED_CTR_CTRL, FIXED_PMC_CTRL(idx, FIXED_PMC_KERNEL)); wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, FIXED_PMC_GLOBAL_CTRL_ENABLE(idx)); + if (nr_fixed_counters > 1) + wrmsr(MSR_CORE_PERF_FIXED_CTR_CTRL, + FIXED_PMC_CTRL(idx, FIXED_PMC_KERNEL) | + FIXED_PMC_CTRL((idx + 1) % nr_fixed_counters, FIXED_PMC_KERNEL)); __asm__ __volatile__("loop ." : "+c"((int){NUM_BRANCHES})); wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, 0); diff --git a/tools/testing/selftests/kvm/x86/sev_dbg_test.c b/tools/testing/selftests/kvm/x86/sev_dbg_test.c new file mode 100644 index 000000000000..a9d8e4c059f9 --- /dev/null +++ b/tools/testing/selftests/kvm/x86/sev_dbg_test.c @@ -0,0 +1,118 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <fcntl.h> +#include <string.h> +#include <sys/ioctl.h> + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" +#include "sev.h" + +#define BUFFER_SIZE (PAGE_SIZE * 2) + +static u8 *data; +static u8 src[BUFFER_SIZE] __aligned(PAGE_SIZE); +static u8 dst[BUFFER_SIZE] __aligned(PAGE_SIZE); + +static void validate_dst(int i, int nr_bytes, u8 pattern) +{ + for ( ; i < nr_bytes; i++) + TEST_ASSERT(dst[i] == pattern, + "Expected 0x%x at byte %u, got 0x%x", + pattern, i, dst[i]); +} + +static void validate_buffers(void) +{ + int i; + + for (i = 0; i < BUFFER_SIZE; i++) + TEST_ASSERT(src[i] == dst[i], + "Expected src[%u] (0x%x) == dst[%u] (0x%x)", + i, src[i], i, dst[i]); +} + +static void ____test_sev_dbg(struct kvm_vm *vm, int i, int j, int nr_bytes) +{ + u8 pattern = guest_random_u32(&guest_rng); + + if (i + nr_bytes > BUFFER_SIZE || j + nr_bytes > BUFFER_SIZE) + return; + + memset(&src[i], pattern, nr_bytes); + sev_encrypt_memory(vm, &data[j], &src[i], nr_bytes); + sev_decrypt_memory(vm, &dst[i], &data[j], nr_bytes); + validate_buffers(); + validate_dst(i, nr_bytes, pattern); +} + +static void __test_sev_dbg(struct kvm_vm *vm, int nr_bytes) +{ + /* + * In a perfect world, all sizes at all combinations within the buffers + * would be tested. In reality, even this much testing is quite slow. + * Target sizes and offsets around the chunk (16 bytes) and page (4096 + * bytes) sizes. + */ + int x[] = { 1, 8, 15, 16, 23 }; + int p = PAGE_SIZE - 24; + int i, j; + + ____test_sev_dbg(vm, 0, 0, nr_bytes); + + for (i = 0; i < ARRAY_SIZE(x); i++) { + for (j = 0; j < ARRAY_SIZE(x); j++) { + ____test_sev_dbg(vm, x[i], x[j], nr_bytes); + ____test_sev_dbg(vm, x[i], p + x[j], nr_bytes); + ____test_sev_dbg(vm, p + x[i], x[j], nr_bytes); + ____test_sev_dbg(vm, p + x[i], p + x[j], nr_bytes); + } + } +} + +static void test_sev_dbg(u32 type, u64 policy) +{ + int sizes[] = { 1, 8, 15, 16, 17, 32, 33 }; + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + int i; + + if (!(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(type))) + return; + + vm = vm_sev_create_with_one_vcpu(type, NULL, &vcpu); + + data = addr_gva2hva(vm, vm_alloc(vm, BUFFER_SIZE, KVM_UTIL_MIN_VADDR)); + memset(data, 0xaa, BUFFER_SIZE); + + vm_sev_launch(vm, policy, NULL); + + sev_decrypt_memory(vm, dst, data, BUFFER_SIZE); + validate_dst(0, BUFFER_SIZE, 0xaa); + + memset(src, 0x55, BUFFER_SIZE); + sev_encrypt_memory(vm, data, src, BUFFER_SIZE); + sev_decrypt_memory(vm, dst, data, BUFFER_SIZE); + validate_dst(0, BUFFER_SIZE, 0x55); + + __test_sev_dbg(vm, PAGE_SIZE); + + for (i = 0; i < ARRAY_SIZE(sizes); i++) { + __test_sev_dbg(vm, sizes[i]); + __test_sev_dbg(vm, PAGE_SIZE - sizes[i]); + __test_sev_dbg(vm, PAGE_SIZE + sizes[i]); + __test_sev_dbg(vm, BUFFER_SIZE - sizes[i]); + } + + kvm_vm_free(vm); +} + +int main(int argc, char *argv[]) +{ + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SEV)); + + /* Note, KVM doesn't support {de,en}crypt commands for SNP. */ + test_sev_dbg(KVM_X86_SEV_VM, 0); + test_sev_dbg(KVM_X86_SEV_ES_VM, SEV_POLICY_ES); + return 0; +} diff --git a/tools/testing/selftests/kvm/x86/sev_init2_tests.c b/tools/testing/selftests/kvm/x86/sev_init2_tests.c index 8eeba2327c7c..8db88c355f16 100644 --- a/tools/testing/selftests/kvm/x86/sev_init2_tests.c +++ b/tools/testing/selftests/kvm/x86/sev_init2_tests.c @@ -136,16 +136,14 @@ int main(int argc, char *argv[]) kvm_check_cap(KVM_CAP_VM_TYPES), 1 << KVM_X86_SEV_VM); TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM)); - have_sev_es = kvm_cpu_has(X86_FEATURE_SEV_ES); + have_sev_es = kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM); - TEST_ASSERT(have_sev_es == !!(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM)), - "sev-es: KVM_CAP_VM_TYPES (%x) does not match cpuid (checking %x)", - kvm_check_cap(KVM_CAP_VM_TYPES), 1 << KVM_X86_SEV_ES_VM); + TEST_ASSERT(!have_sev_es || kvm_cpu_has(X86_FEATURE_SEV_ES), + "sev-es: SEV_ES_VM supported without SEV_ES in CPUID"); - have_snp = kvm_cpu_has(X86_FEATURE_SEV_SNP); - TEST_ASSERT(have_snp == !!(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SNP_VM)), - "sev-snp: KVM_CAP_VM_TYPES (%x) indicates SNP support (bit %d), but CPUID does not", - kvm_check_cap(KVM_CAP_VM_TYPES), KVM_X86_SNP_VM); + have_snp = kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SNP_VM); + TEST_ASSERT(!have_snp || kvm_cpu_has(X86_FEATURE_SEV_SNP), + "sev-snp: SNP_VM supported without SEV_SNP in CPUID"); test_vm_types(); diff --git a/tools/testing/selftests/kvm/x86/sev_migrate_tests.c b/tools/testing/selftests/kvm/x86/sev_migrate_tests.c index 6b0928e69051..42bc023d5193 100644 --- a/tools/testing/selftests/kvm/x86/sev_migrate_tests.c +++ b/tools/testing/selftests/kvm/x86/sev_migrate_tests.c @@ -374,7 +374,7 @@ int main(int argc, char *argv[]) TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SEV)); - have_sev_es = kvm_cpu_has(X86_FEATURE_SEV_ES); + have_sev_es = kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM); if (kvm_has_cap(KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM)) { test_sev_migrate_from(/* es= */ false); diff --git a/tools/testing/selftests/kvm/x86/sev_smoke_test.c b/tools/testing/selftests/kvm/x86/sev_smoke_test.c index 1a49ee391586..6b2cbe2a90b7 100644 --- a/tools/testing/selftests/kvm/x86/sev_smoke_test.c +++ b/tools/testing/selftests/kvm/x86/sev_smoke_test.c @@ -249,10 +249,10 @@ int main(int argc, char *argv[]) test_sev_smoke(guest_sev_code, KVM_X86_SEV_VM, 0); - if (kvm_cpu_has(X86_FEATURE_SEV_ES)) + if (kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM)) test_sev_smoke(guest_sev_es_code, KVM_X86_SEV_ES_VM, SEV_POLICY_ES); - if (kvm_cpu_has(X86_FEATURE_SEV_SNP)) + if (kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SNP_VM)) test_sev_smoke(guest_snp_code, KVM_X86_SNP_VM, snp_default_policy()); return 0; diff --git a/tools/testing/selftests/kvm/x86/svm_nested_pat_test.c b/tools/testing/selftests/kvm/x86/svm_nested_pat_test.c new file mode 100644 index 000000000000..92da8ff34da1 --- /dev/null +++ b/tools/testing/selftests/kvm/x86/svm_nested_pat_test.c @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2026, Google LLC. + * + * Test that KVM correctly virtualizes the PAT MSR and VMCB g_pat field + * for nested SVM guests: + * + * o With nested NPT disabled: + * - L1 and L2 share the same PAT + * - The vmcb12.g_pat is ignored + * o With nested NPT enabled: + * - Invalid g_pat in vmcb12 should cause VMEXIT_INVALID + * - L2 should see vmcb12.g_pat via RDMSR, not L1's PAT + * - L2's writes to PAT should be saved to vmcb12 on exit + * - L1's PAT should be restored after #VMEXIT from L2 + * - State save/restore should preserve both L1's and L2's PAT values + */ +#include <fcntl.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" +#include "svm_util.h" + +#define L2_GUEST_STACK_SIZE 256 + +#define PAT_DEFAULT 0x0007040600070406ULL +#define L1_PAT_VALUE 0x0007040600070404ULL /* Change PA0 to WT */ +#define L2_VMCB12_PAT 0x0606060606060606ULL /* All WB */ +#define L2_PAT_MODIFIED 0x0606060606060604ULL /* Change PA0 to WT */ +#define INVALID_PAT_VALUE 0x0808080808080808ULL /* 8 is reserved */ + +bool npt_enabled; +int nr_iterations; + +static void l2_guest_code(void) +{ + u64 expected_pat = npt_enabled ? L2_VMCB12_PAT : L1_PAT_VALUE; + int i; + + for (i = 0; i < nr_iterations; i++) { + GUEST_ASSERT_EQ(rdmsr(MSR_IA32_CR_PAT), expected_pat); + GUEST_SYNC(1); + GUEST_ASSERT_EQ(rdmsr(MSR_IA32_CR_PAT), expected_pat); + + wrmsr(MSR_IA32_CR_PAT, L2_PAT_MODIFIED); + expected_pat = L2_PAT_MODIFIED; + + GUEST_ASSERT_EQ(rdmsr(MSR_IA32_CR_PAT), L2_PAT_MODIFIED); + GUEST_SYNC(2); + GUEST_ASSERT_EQ(rdmsr(MSR_IA32_CR_PAT), L2_PAT_MODIFIED); + + vmmcall(); + } +} + +static void l1_guest_code(struct svm_test_data *svm) +{ + unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; + struct vmcb *vmcb = svm->vmcb; + int i; + + wrmsr(MSR_IA32_CR_PAT, L1_PAT_VALUE); + GUEST_ASSERT_EQ(rdmsr(MSR_IA32_CR_PAT), L1_PAT_VALUE); + + generic_svm_setup(svm, l2_guest_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + + vmcb->save.g_pat = L2_VMCB12_PAT; + vmcb->control.intercept &= ~(1ULL << INTERCEPT_MSR_PROT); + + for (i = 0; i < nr_iterations; i++) { + run_guest(vmcb, svm->vmcb_gpa); + + GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_VMMCALL); + + /* + * If NPT is enabled by L1, L2 has a unique PAT and L1's PAT is + * unchanged. Otherwise, PAT is shared between L1 and L2. + */ + if (npt_enabled) { + GUEST_ASSERT_EQ(vmcb->save.g_pat, L2_PAT_MODIFIED); + GUEST_ASSERT_EQ(rdmsr(MSR_IA32_CR_PAT), L1_PAT_VALUE); + } else { + GUEST_ASSERT_EQ(rdmsr(MSR_IA32_CR_PAT), L2_PAT_MODIFIED); + } + vmcb->save.rip += 3; /* skip over VMMCALL */ + } + + GUEST_DONE(); +} + +static void l1_guest_code_invalid_gpat(struct svm_test_data *svm) +{ + unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; + struct vmcb *vmcb = svm->vmcb; + + /* VMRUN should fail without running L2 */ + generic_svm_setup(svm, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + + vmcb->save.g_pat = INVALID_PAT_VALUE; + run_guest(vmcb, svm->vmcb_gpa); + + GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_ERR); + GUEST_DONE(); +} + +static void run_test(void *guest_code, bool do_save_restore, int nr_iters) +{ + struct kvm_x86_state *state; + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + struct ucall uc; + gva_t svm_gva; + + vm = vm_create_with_one_vcpu(&vcpu, guest_code); + vm_enable_cap(vm, KVM_CAP_DISABLE_QUIRKS2, + KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT); + + if (npt_enabled) + vm_enable_npt(vm); + + vcpu_alloc_svm(vm, &svm_gva); + + if (npt_enabled) + tdp_identity_map_default_memslots(vm); + + vcpu_args_set(vcpu, 1, svm_gva); + + nr_iterations = nr_iters; + sync_global_to_guest(vm, npt_enabled); + sync_global_to_guest(vm, nr_iterations); + + for (;;) { + vcpu_run(vcpu); + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); + + switch (get_ucall(vcpu, &uc)) { + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + /* NOT REACHED */ + case UCALL_SYNC: + if (do_save_restore) { + state = vcpu_save_state(vcpu); + kvm_vm_release(vm); + vcpu = vm_recreate_with_one_vcpu(vm); + vm_enable_cap(vm, KVM_CAP_DISABLE_QUIRKS2, + KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT); + vcpu_load_state(vcpu, state); + kvm_x86_state_cleanup(state); + } + break; + case UCALL_DONE: + kvm_vm_free(vm); + return; + default: + TEST_FAIL("Unknown ucall %lu", uc.cmd); + } + } +} + +#define gpat_test(test_name, guest_code, npt_setting) \ +do { \ + npt_setting; \ + \ + if (npt_enabled && !kvm_cpu_has(X86_FEATURE_NPT)) { \ + pr_info("Skipping: " test_name " (no NPT support)\n"); \ + break; \ + } \ + \ + pr_info("Testing: " test_name "\n"); \ + run_test(guest_code, false, 1); \ + \ + if (guest_code == l1_guest_code) { \ + pr_info("Testing: " test_name " Save/Restore\n"); \ + run_test(guest_code, true, 1); \ + \ + pr_info("Testing: " test_name " Multiple VMRUNs\n"); \ + run_test(guest_code, false, 10); \ + } \ +} while (0) + +int main(int argc, char *argv[]) +{ + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM)); + TEST_REQUIRE(kvm_has_cap(KVM_CAP_NESTED_STATE)); + TEST_REQUIRE(kvm_check_cap(KVM_CAP_DISABLE_QUIRKS2) & + KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT); + + gpat_test("Invalid gPAT", l1_guest_code_invalid_gpat, npt_enabled = true); + gpat_test("Nested NPT enabled", l1_guest_code, npt_enabled = true); + gpat_test("Nested NPT disabled", l1_guest_code, npt_enabled = false); + return 0; +} diff --git a/tools/testing/selftests/kvm/x86/sync_regs_test.c b/tools/testing/selftests/kvm/x86/sync_regs_test.c index e0c52321f87c..5b0c2359bbb4 100644 --- a/tools/testing/selftests/kvm/x86/sync_regs_test.c +++ b/tools/testing/selftests/kvm/x86/sync_regs_test.c @@ -255,7 +255,6 @@ KVM_ONE_VCPU_TEST(sync_regs_test, req_and_verify_all_valid, guest_code) struct kvm_regs regs; /* Request and verify all valid register sets. */ - /* TODO: BUILD TIME CHECK: TEST_ASSERT(KVM_SYNC_X86_NUM_FIELDS != 3); */ run->kvm_valid_regs = TEST_SYNC_FIELDS; vcpu_run(vcpu); TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 80f201035d77..db57c5766ab6 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -157,7 +157,7 @@ static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *in return KVM_FILTER_PRIVATE; } -static void __kvm_gmem_invalidate_begin(struct gmem_file *f, pgoff_t start, +static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, pgoff_t end, enum kvm_gfn_range_filter attr_filter) { @@ -181,7 +181,7 @@ static void __kvm_gmem_invalidate_begin(struct gmem_file *f, pgoff_t start, found_memslot = true; KVM_MMU_LOCK(kvm); - kvm_mmu_invalidate_begin(kvm); + kvm_mmu_invalidate_start(kvm); } flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range); @@ -194,7 +194,7 @@ static void __kvm_gmem_invalidate_begin(struct gmem_file *f, pgoff_t start, KVM_MMU_UNLOCK(kvm); } -static void kvm_gmem_invalidate_begin(struct inode *inode, pgoff_t start, +static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start, pgoff_t end) { enum kvm_gfn_range_filter attr_filter; @@ -203,7 +203,7 @@ static void kvm_gmem_invalidate_begin(struct inode *inode, pgoff_t start, attr_filter = kvm_gmem_get_invalidate_filter(inode); kvm_gmem_for_each_file(f, inode) - __kvm_gmem_invalidate_begin(f, start, end, attr_filter); + __kvm_gmem_invalidate_start(f, start, end, attr_filter); } static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start, @@ -238,7 +238,7 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len) */ filemap_invalidate_lock(inode->i_mapping); - kvm_gmem_invalidate_begin(inode, start, end); + kvm_gmem_invalidate_start(inode, start, end); truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1); @@ -352,7 +352,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file) * Zap all SPTEs pointed at by this file. Do not free the backing * memory, as its lifetime is associated with the inode, not the file. */ - __kvm_gmem_invalidate_begin(f, 0, -1ul, + __kvm_gmem_invalidate_start(f, 0, -1ul, kvm_gmem_get_invalidate_filter(inode)); __kvm_gmem_invalidate_end(f, 0, -1ul); @@ -438,11 +438,12 @@ static int kvm_gmem_set_policy(struct vm_area_struct *vma, struct mempolicy *mpo } static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, - unsigned long addr, pgoff_t *pgoff) + unsigned long addr, pgoff_t *ilx) { + pgoff_t pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT); struct inode *inode = file_inode(vma->vm_file); - *pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT); + *ilx = inode->i_ino; /* * Return the memory policy for this index, or NULL if none is set. @@ -453,7 +454,7 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, * can then replace NULL with the default memory policy instead of the * current task's memory policy. */ - return mpol_shared_policy_lookup(&GMEM_I(inode)->policy, *pgoff); + return mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff); } #endif /* CONFIG_NUMA */ @@ -504,7 +505,7 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol start = folio->index; end = start + folio_nr_pages(folio); - kvm_gmem_invalidate_begin(mapping->host, start, end); + kvm_gmem_invalidate_start(mapping->host, start, end); /* * Do not truncate the range, what action is taken in response to the @@ -640,15 +641,16 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args) } int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, - unsigned int fd, loff_t offset) + unsigned int fd, uoff_t offset) { - loff_t size = slot->npages << PAGE_SHIFT; + uoff_t size = slot->npages << PAGE_SHIFT; unsigned long start, end; struct gmem_file *f; struct inode *inode; struct file *file; int r = -EINVAL; + BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset)); BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff)); file = fget(fd); @@ -664,8 +666,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, inode = file_inode(file); - if (offset < 0 || !PAGE_ALIGNED(offset) || - offset + size > i_size_read(inode)) + if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode)) goto err; filemap_invalidate_lock(inode->i_mapping); @@ -675,6 +676,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, if (!xa_empty(&f->bindings) && xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) { + r = -EEXIST; filemap_invalidate_unlock(inode->i_mapping); goto err; } @@ -858,7 +860,8 @@ out_unlock: return ret; } -long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long npages, +long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, + long npages, bool may_writeback_src, kvm_gmem_populate_cb post_populate, void *opaque) { struct kvm_memory_slot *slot; @@ -892,8 +895,9 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long if (src) { unsigned long uaddr = (unsigned long)src + i * PAGE_SIZE; + unsigned int flags = may_writeback_src ? FOLL_WRITE : 0; - ret = get_user_pages_fast(uaddr, 1, 0, &src_page); + ret = get_user_pages_fast(uaddr, 1, flags, &src_page); if (ret < 0) break; if (ret != 1) { diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 881f92d7a469..e44c20c04961 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -670,7 +670,7 @@ static __always_inline bool kvm_age_hva_range_no_flush(struct mmu_notifier *mn, return kvm_age_hva_range(mn, start, end, handler, false); } -void kvm_mmu_invalidate_begin(struct kvm *kvm) +void kvm_mmu_invalidate_start(struct kvm *kvm) { lockdep_assert_held_write(&kvm->mmu_lock); /* @@ -726,7 +726,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, .start = range->start, .end = range->end, .handler = kvm_mmu_unmap_gfn_range, - .on_lock = kvm_mmu_invalidate_begin, + .on_lock = kvm_mmu_invalidate_start, .flush_on_ret = true, .may_block = mmu_notifier_range_blockable(range), }; @@ -2542,7 +2542,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, .end = end, .arg.attributes = attributes, .handler = kvm_pre_set_memory_attributes, - .on_lock = kvm_mmu_invalidate_begin, + .on_lock = kvm_mmu_invalidate_start, .flush_on_ret = true, .may_block = true, }; diff --git a/virt/kvm/kvm_mm.h b/virt/kvm/kvm_mm.h index 9fcc5d5b7f8d..7510ca915dd1 100644 --- a/virt/kvm/kvm_mm.h +++ b/virt/kvm/kvm_mm.h @@ -3,6 +3,9 @@ #ifndef __KVM_MM_H__ #define __KVM_MM_H__ 1 +#include <linux/kvm.h> +#include <linux/kvm_types.h> + /* * Architectures can choose whether to use an rwlock or spinlock * for the mmu_lock. These macros, for use in common code @@ -72,7 +75,7 @@ int kvm_gmem_init(struct module *module); void kvm_gmem_exit(void); int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args); int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, - unsigned int fd, loff_t offset); + unsigned int fd, uoff_t offset); void kvm_gmem_unbind(struct kvm_memory_slot *slot); #else static inline int kvm_gmem_init(struct module *module) @@ -82,7 +85,7 @@ static inline int kvm_gmem_init(struct module *module) static inline void kvm_gmem_exit(void) {}; static inline int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, - unsigned int fd, loff_t offset) + unsigned int fd, uoff_t offset) { WARN_ON_ONCE(1); return -EIO; diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c index 9f9acb66cc1e..6cdc4e9a333a 100644 --- a/virt/kvm/vfio.c +++ b/virt/kvm/vfio.c @@ -144,33 +144,26 @@ static int kvm_vfio_file_add(struct kvm_device *dev, unsigned int fd) { struct kvm_vfio *kv = dev->private; struct kvm_vfio_file *kvf; - struct file *filp; - int ret = 0; + struct file *filp __free(fput) = NULL; filp = fget(fd); if (!filp) return -EBADF; /* Ensure the FD is a vfio FD. */ - if (!kvm_vfio_file_is_valid(filp)) { - ret = -EINVAL; - goto out_fput; - } + if (!kvm_vfio_file_is_valid(filp)) + return -EINVAL; - mutex_lock(&kv->lock); + guard(mutex)(&kv->lock); list_for_each_entry(kvf, &kv->file_list, node) { - if (kvf->file == filp) { - ret = -EEXIST; - goto out_unlock; - } + if (kvf->file == filp) + return -EEXIST; } kvf = kzalloc_obj(*kvf, GFP_KERNEL_ACCOUNT); - if (!kvf) { - ret = -ENOMEM; - goto out_unlock; - } + if (!kvf) + return -ENOMEM; kvf->file = get_file(filp); list_add_tail(&kvf->node, &kv->file_list); @@ -178,11 +171,18 @@ static int kvm_vfio_file_add(struct kvm_device *dev, unsigned int fd) kvm_vfio_file_set_kvm(kvf->file, dev->kvm); kvm_vfio_update_coherency(dev); -out_unlock: - mutex_unlock(&kv->lock); -out_fput: - fput(filp); - return ret; + return 0; +} + +static void kvm_vfio_file_free(struct kvm_device *dev, struct kvm_vfio_file *kvf) +{ +#ifdef CONFIG_SPAPR_TCE_IOMMU + kvm_spapr_tce_release_vfio_group(dev->kvm, kvf); +#endif + kvm_vfio_file_set_kvm(kvf->file, NULL); + fput(kvf->file); + list_del(&kvf->node); + kfree(kvf); } static int kvm_vfio_file_del(struct kvm_device *dev, unsigned int fd) @@ -190,34 +190,21 @@ static int kvm_vfio_file_del(struct kvm_device *dev, unsigned int fd) struct kvm_vfio *kv = dev->private; struct kvm_vfio_file *kvf; CLASS(fd, f)(fd); - int ret; if (fd_empty(f)) return -EBADF; - ret = -ENOENT; - - mutex_lock(&kv->lock); + guard(mutex)(&kv->lock); list_for_each_entry(kvf, &kv->file_list, node) { - if (kvf->file != fd_file(f)) - continue; - - list_del(&kvf->node); -#ifdef CONFIG_SPAPR_TCE_IOMMU - kvm_spapr_tce_release_vfio_group(dev->kvm, kvf); -#endif - kvm_vfio_file_set_kvm(kvf->file, NULL); - fput(kvf->file); - kfree(kvf); - ret = 0; - break; + if (kvf->file == fd_file(f)) { + kvm_vfio_file_free(dev, kvf); + kvm_vfio_update_coherency(dev); + return 0; + } } - kvm_vfio_update_coherency(dev); - - mutex_unlock(&kv->lock); - return ret; + return -ENOENT; } #ifdef CONFIG_SPAPR_TCE_IOMMU @@ -227,7 +214,6 @@ static int kvm_vfio_file_set_spapr_tce(struct kvm_device *dev, struct kvm_vfio_spapr_tce param; struct kvm_vfio *kv = dev->private; struct kvm_vfio_file *kvf; - int ret; if (copy_from_user(¶m, arg, sizeof(struct kvm_vfio_spapr_tce))) return -EFAULT; @@ -236,9 +222,7 @@ static int kvm_vfio_file_set_spapr_tce(struct kvm_device *dev, if (fd_empty(f)) return -EBADF; - ret = -ENOENT; - - mutex_lock(&kv->lock); + guard(mutex)(&kv->lock); list_for_each_entry(kvf, &kv->file_list, node) { if (kvf->file != fd_file(f)) @@ -246,20 +230,15 @@ static int kvm_vfio_file_set_spapr_tce(struct kvm_device *dev, if (!kvf->iommu_group) { kvf->iommu_group = kvm_vfio_file_iommu_group(kvf->file); - if (WARN_ON_ONCE(!kvf->iommu_group)) { - ret = -EIO; - goto err_fdput; - } + if (WARN_ON_ONCE(!kvf->iommu_group)) + return -EIO; } - ret = kvm_spapr_tce_attach_iommu_group(dev->kvm, param.tablefd, - kvf->iommu_group); - break; + return kvm_spapr_tce_attach_iommu_group(dev->kvm, param.tablefd, + kvf->iommu_group); } -err_fdput: - mutex_unlock(&kv->lock); - return ret; + return -ENOENT; } #endif @@ -326,15 +305,8 @@ static void kvm_vfio_release(struct kvm_device *dev) struct kvm_vfio *kv = dev->private; struct kvm_vfio_file *kvf, *tmp; - list_for_each_entry_safe(kvf, tmp, &kv->file_list, node) { -#ifdef CONFIG_SPAPR_TCE_IOMMU - kvm_spapr_tce_release_vfio_group(dev->kvm, kvf); -#endif - kvm_vfio_file_set_kvm(kvf->file, NULL); - fput(kvf->file); - list_del(&kvf->node); - kfree(kvf); - } + list_for_each_entry_safe(kvf, tmp, &kv->file_list, node) + kvm_vfio_file_free(dev, kvf); kvm_vfio_update_coherency(dev); |