This will allow IOMMU drivers to be loaded as EL2 modules
To achieve that, EL2 module init should register and init
the driver using pkvm_iommu_driver_init.
To avoid having to hardcode S2MPU drivers in IOMMU EL2 code,
pkvm_iommu_driver_init will have a ptr to driver struct instead of
id.
Id is removed from pkvm_iommu_driver and the address of the struct
is treated as the id of the driver, the kernel is expected to pass
linear mapping address of the driver and the hypervisor is expected
to use hyp address.
Bug: 249481474
Change-Id: I49db5b5d88a691d66fa2d5302198785da94d70d2
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The PSCI specification defines only a 32-bit function ID for the
MEM_PROTECT call used to protect against cold reboot attacks.
Fix the pKVM hypervisor invocation of MEM_PROTECT to use the 32-bit
function ID instead of the unallocated 64-bit flavour.
[ qperret: dropped the change to include/uapi/linux/psci.h as the 32 bit
variant of the call have been introduced upstream by 3137f2e600
("firmware/psci: Add debugfs support to ease debugging") ]
Bug: 260316363
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I675a57419064f7f006960ca5370e9dc2d5279a90
Signed-off-by: Quentin Perret <qperret@google.com>
This reverts commit 87bcd3edf3ee7b9d5e76a8090b3f05ea1535026d.
Reason for revert: Memory reclaim capability will be checked by the
host before configuring the virtio_balloon device.
Bug: 240239989
Change-Id: I03e7c39ec6d671babeace4040138b416c7e201cf
Signed-off-by: Keir Fraser <keirf@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
SysMMU_SYNCs provide an invalidation-complete signal to the S2MPU
driver but the latency can be quite high. Improve this by waiting for
all the SYNCs in parallel - separate the initiation of invalidation
barrier from waiting for completion. This way we initiate invalidation
on all SYNCs first, then wait for all of them to complete.
The previously introduced exponential-backoff only kicks in if the
SYNC_COMP_COMPLETE bit is not set after the parallel invalidation.
Bug: 249161451
Signed-off-by: David Brazdil <dbrazdil@google.com>
Change-Id: I9d544bc65f8633d376c7ccd65ea23195ca432964
(cherry picked from commit 1da102d4e3d54285f1d2071a7bee79fbb4c9ce71)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Add a new callback to pkvm_iommu_ops called after
host_stage2_idmap_apply on all IOMMU devices. This allows the drivers to
complete operations like invalidation in two stages.
Bug: 249161451
Signed-off-by: David Brazdil <dbrazdil@google.com>
Change-Id: I9c077fd2b18ce54ad67eb34ef16bc94428797419
(cherry picked from commit ce39549d9283e32cee6a476e4603e7b586957ecf)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
When handling host stage-2 faults the hypervisor currently updates the
CPU _and_ IOMMUs page-tables. However, since we currently proactively
map accessible PA ranges into IOMMUs, updating them during stage-2
faults is unnecessary -- it only needs to be done during ownership
transitions. Optimize this by skipping the IOMMU updates from the host
memory abort path, which also reduces contention on the host stage-2
lock during boot and saves up to 1.1 sec of boot time on Pixel 6.
Bug: 232879742
Change-Id: Icb6f1b2d10de8cc2f976bbc94b9b79ad7aaa6135
Signed-off-by: Quentin Perret <qperret@google.com>
(cherry picked from commit 20c6e1ba55403fad0e7f039cc5d7e6f8c0fffa50)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The SysMMU_SYNC provides an invalidation-complete signal to the
hypervisor. Currently the hypervisor will wait indefinitely for the SYNC
to set the SYNC_COMP_COMPLETE bit. In practice, this case deadlock as
the hypervisor holds the host lock while waiting for the SYNC.
To avoid deadlock, adjust the algorithm to time out after a given number
of reads of the SYNC_COMP register (new constant SYNC_TIMEOUT_BASE).
This can be a small number as most attempts succeed after a single read
of the SFR.
If the wait-loop times out, the hypervisor will try again, multiplying
the maximum number of SFR reads with SYNC_TIMEOUT_MULTIPLIER each time.
This number was selected to grow quickly, in case there is a lot of DMA
traffic that would be slowing down the SYNC request.
Finally, if the hardware does not set the bit even after
SYNC_MAX_RETRIES, the algorithm will give up to avoid deadlock. The
value was selected so that the worst-case time spent in
__wait_for_invalidation_complete() remains tolerable.
Bug: 250727777
Change-Id: I00230e6dd71de12bac223e4fe118806bdc3872f4
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 992b5f98cad8a8bd0da106a113c25429aeb183cd)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
To ease debugging, allow the host to read the state of S2MPU's control
registers. These values do not need to be kept secret from the host.
Bug: 190463801
Change-Id: I5df7de81caceaddfaf2bfb0948ff2d167de487c3
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 5d6831add7c1bb441fe410481d055e2b1c486684)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The state of the S2MPU does not need to be kept secret from the host as
it merely reflects the permissions that the host has and knows about.
To make debugging DMA issues easier, allow the host to query entries
from the MPTC cache. This involves writing the set and way IDs of the
query to the READ_MPTC register and then reading the MPTC entry
information from READ_MPTC_TAG_PPN/TAG_OTHERS/DATA. Modify the S2MPU
DABT handler to allow this register access pattern.
Bug: 190463801
Bug: 229793579
Change-Id: I5661c09a284a4796e18b346f9e294ba64d7d8c0c
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit d5c0f0f937af0826e549503897fb9e9c9b22722a)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Allow read-only access to L1ENTRY_ATTR and L1ENTRY_L2TABLE S2MPU
registers. This allows the host to dump the register state for debugging
purposes. It is safe because the state of the S2MPU is known to the host
anyway.
Bug: 190463801
Change-Id: I44b3633dbad3c122ce521c37813dbf2ae690a678
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit e56d9603a633e7320ea1973ec0c4474b4cb7392c)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
In preparation for adding more entries to the list of S2MPU registers
accessible to the host, refactor the code to use a switch instead of
a series of ifs. No functional change intended.
Bug: 190463801
Change-Id: I9075a7f080b3391c98fa10f26a943bce0b183394
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 96767ad7bec869c1aee13f0c9680b2ed3d895db9)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Extract the L1ENTRY_ATTR_{PRON,GRAN}_MASK constants out of macros that
create the corresponding constants. This will allow EL1 users to use the
masks to get the fields out of register values. Also extract
L1ENTRY_L2TABLE_ADDR_SHIFT for adjusting the L2 table address.
Bug: 190463801
Change-Id: I5ad6657c7b12d10b81217ee80583eade1e17aeb8
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit c43dfe89feb794ef85a90f3e3fbf0fe6965fefb2)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Change the permissions that MPTs are initialized with from PROT_NONE to
PROT_RW. No functional change intended as the generic IOMMU code
sets permissions for the entire address space later. This will allow to
optimize boot time by only unmapping pages not available to host.
Bug: 190463801
Bug: 218012133
Change-Id: Id3891f9de3afe88a0008e8488b3172b73fa1bb69
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 174ac5b7c56fe039fc04ab70073223dd1780e0de)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Currently the generic IOMMU code lets the driver initialize its PT and
then invokes callbacks to set the permissions across the entire PA
range. Optimize this by making it a requirement on the driver to
initialize its PTs to all memory owned by the host. snapshot_host_stage2
then only calls the driver's callback for memory regions not owned by
the host.
Bug: 190463801
Bug: 218012133
Change-Id: I99a826d921d494269078c3a84d90323455a0b769
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 4e56697b422df13df9f25074f6d7710acd784394)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The second argument of the kvm_pgtable_walker callback was
misinterpreted as the end of the current entry, where in fact it is
the end of the walked memory region. Fix this by computing the end of
the current entry from the start and the level.
This did not affect correctness, as the code iterates linarly over
the entire address space, but it did affect boot time.
Bug: 190463801
Bug: 218012133
Signed-off-by: David Brazdil <dbrazdil@google.com>
Change-Id: I6d189b87645f47cd215a783c1bc9e1f032ff8c62
(cherry picked from commit 58f8121600276e32e0c91d075f273b836360b27a)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Add new hypercall that the host can use to inform the hypervisor that
all hypervisor-controlled IOMMUs have been registered and no new
registrations should be allowed. This will typically be called at the
end of kernel module initialization phase.
Bug: 190463801
Change-Id: I124e6693a3ac0b1a81988640fcb9e09f08bf4ccf
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 8fd93b0ef92202f16008d97ebcb92de367751491)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The IOMMU DABT handler currently checks if the device is considered
powered by hyp before resolving the request. If the power tracking does
not reflect reality, the IOMMU may trigger issues in the host but the
incorrect state prevents it from diagnosing the issue.
Drop the powered check from the generic IOMMU code. The host accessing
the device's SFR means that it assumes it is powered, and individual
drivers can choose to reject that DABT request.
Bug: 224891559
Bug: 190463801
Change-Id: I3418a29c7deb7e3e866f89d933b9dad0aaa06365
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 798c4ea545e98e04797424c98c4329e6a34db60e)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
SysMMU_SYNC devices expose an interface to start a sync counter and
poll its SFR until the device signals that all memory transactions in
flight at the start have drained. This gives the hypervisor a reliable
indicator that S2MPU invalidation has fully completed and all new
transactions will use the new MPTs.
Add a new pKVM IOMMU driver that the host can use to register
SysMMU_SYNCs. Each device is expected to be a supplier to exactly one
S2MPU (parent), but multiple SYNCs can supply a single S2MPU.
To keep things simple, the SYNCs do not implement suspend/resume and are
assumed to follow the power transitions of their parent.
Following an invalidation, the S2MPU driver iterates over its children
and waits for each SYNC to signal that its transactions have drained.
The algorithm currently waits on each SYNC in turn. If latency proves to
be an issue, this could be optimized to initiate a SYNC on all powered
devices before starting to poll.
Bug: 190463801
Change-Id: I0006602bb5a683d39a6542b61b5ece13ebc28c3f
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 57381d548d9de5382047ac9602da5487a2f78383)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
In preparation for adding new IOMMU devices that act as suppliers to
others, add the notion of a parent IOMMU device. Such device must be
registered after its parent and the driver of the parent device must
validate the addition.
The relation has no generic implications, it is up to drivers to make
use of it.
Bug: 190463801
Change-Id: I1e4be18a5ad826f84b0ea895129d2ef54ee17f85
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit e69c61cf4ebab19f5abdfedb66c48e96345dbdf4)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
In preparation for needing to validate more aspects of a device that is
about to be registered, change the callback to accept the to-be-added
'struct pkvm_iommu' rather than individual inputs.
Bug: 190463801
Change-Id: I74663af26a06624a7f084b067de97f28baa6f262
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit be84f2c770810494ef178cdb73e22b05f2dfc2b6)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Private EL2 mappings currently cannot be removed. Move the creation of
IOMMU device mappings at the end of the registration function so that
other errors do not result in unnecessary mappings.
Bug: 190463801
Change-Id: If100e35790cf512cabfa9327f790d20540977b7d
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit e6574a68fafc1961150d657f0ebf882c68afbb5b)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Memory for IOMMU device entries gets allocated from a pool donated by
the host. It is possible for pkvm_iommu_register() to allocate the
memory and then fail, in which case the memory remains unused but not
freed.
Refactor the code such that the host lock covers the entire section
where the memory is allocated. This way we can return the memory back to
the linear allocator if an error is returned.
Bug: 190463801
Change-Id: I881b7fd56fe7b70c1546e897d7d872af6e34e762
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit acb9a25416ad028354393eb44adf512a4d7ce567)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Currently __pkvm_iommu_pm_notify always changes the value of
dev->powered following a suspend/resume attempt. This could potentially
be abused to force the hypervisor to stop issuing updates to an S2MPU
and preserving an old/invalid state.
Modify to only update the power state if suspend/resume was successful.
Bug: 190463801
Change-Id: I32d5ce7d18b55c3bebfa3273c9f5ca098d5e61f4
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 6eaed0b8b769b78fe237a13d757adb4bf1a883c6)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
With new generic IOMMU code in place, and with all S2MPU code
having been migrated to the new pkvm_iommu_ops callbacks, remove
all the now unused code.
Bug: 190463801
Change-Id: I83a1ac7c2e30ee76712079727da4607bda5de68a
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit e900628cfca812f9f4c4c93754d91be16be4e466)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Remove the existing 's2mpu_host_stage2_set_owner' hook implementation
and refactor the code to match the prepare/apply split of the generic
IOMMU callbacks for updating host stage-2 mappings.
Bug: 190463801
Change-Id: I36485b9cc6786243a77afb3eb18c63e38ec46404
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit bb81c25bcbea0d1cc5457d958df6a39fa3546540)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Core SMPT manipulation code returns mpt_update_flags, signalling whether
the caller should flush the dcache (MPT_UPDATE_L2) or write new L1ATTR
values to S2MPU MMIO registers (MPT_UPDATE_L1).
In preparation for splitting the code into a driver-global and
per-device portions, store the value in the corresponding FMPT.
As long as the two code portions are called from a single critical
section, the FMPT value is guaranteed to not change.
Bug: 190463801
Change-Id: Ic3cdba49e78d68053f7a3cfa84288b1f0a473f71
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 4c2ce4d16cc63f3d03da8411b876e9cc1fa3c52f)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Previously the S2MPU DABT handler would be called directly from the host
DABT handler and it would look up the corresponding S2MPU device. Now the
lookup is done in the generic IOMMU DABT handler and only the actual
S2MPU register access is left to the driver itself.
Bug: 190463801
Change-Id: If18ff6600cd3cc678eb59aed9c8269d13868827b
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 667c7af5808ad6009bfe6e3e86f251421c79e96f)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The host is now expected to notify EL2 about PM state changes of
individual IOMMU devices. Remove the old code that intercepted SMCs
and instead rely on callbacks from the core IOMMU code.
Bug: 190463801
Change-Id: If0ca60f7fce7fd3f025599a472b02824354f0991
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 39d559ad46ebd1416d2d7c9041400f92c2c0f618)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Create 'struct pkvm_iommu_ops' for the S2MPU and a new driver ID to the
list of IOMMU drivers. Implement the 'init' callback, accepting donated
memory from the host to back SMPTs. If the donation is successful,
the SMPTs are assigned to 'host_mpt'.
Export 'pkvm_iommu_s2mpu_register' for a kernel module to call to
register an S2MPU device. First call to this function will also
run the global S2MPU driver initializer.
Bug: 190463801
Change-Id: Icad06379e5cf695fba4f3a18a0773e302f3ead06
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 41707102f4fa94757ce9999e1555827512bd5192)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The function is superseded by the generic
pkvm_iommu_host_stage2_adjust_range, remove it.
Bug: 190463801
Change-Id: If42b40357f1d9a046ff20815215f927ac2a0d765
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 3da3f51b335de90b134efc3f8029ee6c769197f2)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Replace all uses of 'struct s2mpu' with the generic 'struct pkvm_iommu'.
'struct s2mpu_drv_data' is created to accommodate driver-specific values
associated with 'struct pkvm_iommu' and allocated by the generic code.
These changes are safe because the S2MPU code is currently unused.
The EL1 code that initialized it had been removed.
Bug: 190463801
Change-Id: Ia634bac9b7dda333d87f7da0a02768df01d6bbd6
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit a1ed8a1881f44cecaa1f73ffff36e4584ab37b07)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
EL2 S2MPU driver relied on EL1 code which parsed the DT and populated
EL2 driver data before deprivileging of the host. The driver is now
moving to later initialization from kernel modules, which will take over
the role of parsing the DT and power management. Remove the unused code.
Bug: 190463801
Change-Id: Ie6e21ba02b84494e5066c7681f85612a09f93f6d
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 167332a9fa748f7d9fd08643d0df5bbe5078f26d)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
S2MPU code previously assumed that all S2MPUs were powered on at boot
and would check the version register and precompute the value of
S2MPU.CONTEXT_CFG_VALID_VID.
With EL1 S2MPU code being removed, and to allow for S2MPUs not powered
at boot, move the code to EL2 and run it on resume.
Bug: 190463801
Change-Id: Icaccfd125a6be7bab336ca3ffee52f2a33cf43b2
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit c823243791205b8219b60294bc7615e198c12382)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
IOMMU drivers may need to keep their own state of the host stage-2
mappings, eg. because they cannot share the PTs with the CPU. To this
end, walk the host stage-2 at driver init time and pass the current
state of host stage-2 mappings to the driver.
The driver initialization lock is released together with host_kvm
lock. That was the driver starts receiving stage-2 updates immediately
after the snapshot is taken.
Bug: 190463801
Change-Id: I5a5b0e064c5c88e210e28e343314318a2a1bffda
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 1ec4b346d0b65a868c52e5ff11b30918ed29185c)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Add IOMMU callbacks for host stage-2 idmap changes.
'host_stage2_idmap_prepare' is called first and is expected to apply
the changes on the driver level, eg. update driver-specific page table
information. If successful, the generic code invokes
'host_stage2_idmap_apply' on each currently powered IOMMU device
associated with the driver to apply the changes.
Bug: 190463801
Change-Id: Ifcc063896f6e8967c332dbaa5b7e7f2ba138abbf
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 4395ddff4bc5ba21935e53861b86e7063447bf85)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Replace the 'host_mmio_dabt_handler' hook in kvm_iommu_ops with
an equivalent callback in the new pkvm_iommu_ops. The generic portion
of the code finds the IOMMU device at the faulted address and invokes
the callback on it.
Bug: 190463801
Change-Id: I0ca008c3e1ae0ec12a259fa4ddac1aee65aaac5c
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 5df451f35e66b890b086b417c60d42118266823f)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Add suspend/resume callbacks for IOMMU devices. The EL1 kernel driver
is expected to call these when the IOMMU device is powered on but is
about to be used or about to stop being used.
pkvm_iommu_suspend/resume are exported for use by kernel modules.
Bug: 190463801
Change-Id: Ia4ab37fe96879d451ce82f4278b3ff33a0b9685b
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit ca47ae70c77f327be95d8fb6930caa143bb200c9)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Add '__pkvm_iommu_register' hypcall for registering a new IOMMU device.
The handler allocates a linked-list entry for the device from a memory
pool provided by the host. If the pool has run out, the handler returns
-ENOMEM and expects the host to call it again with a fresh mem pool.
The inputs are validated, eg. ID is unique and memory region does not
overlap with existing IOMMUs. The driver can also implement a 'validate'
callback for driver-specific input validation.
If successful, the handler creates a private EL2 mapping for the device,
forces the memory region is unmapped from host stage-2 and inserts the
device into the linked list. Future attempts to map the MMIO region will
fail because of pkvm_iommu_host_stage2_adjust_range.
Bug: 190463801
Change-Id: If6f707555c80ac164ff995f42260872896a84e3d
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 78e0b7722c8d2961af0b3f2c8115df336cf7b492)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Introduce a linked list of IOMMU devices and
'pkvm_iommu_host_stage2_adjust_range' called from host DABT handler.
The function will adjust the memory range that is about to be mapped
to avoid MMIO regions of all devices in the linked list. If the host
tried to access a device MMIO region, the access is declined.
The function replaces the existing call to
'kvm_iommu.ops.host_stage2_adjust_mmio_range' callback.
Bug: 190463801
Change-Id: Iacd6b74147fea2fef04846a91f0a5e550daaf074
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit d7adab5f9fcb7f7efd76bdbc23a1b37322156112)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Add '__pkvm_iommu_driver_init' hypcall and 'struct pkvm_iommu_ops' with
an 'init' callback implemented by an EL2 driver. Driver-specific data
can be passed to 'init' from the host. The memory is pinned while
the callback processed it.
Bug: 190463801
Change-Id: I1185350bb46d41ff060a207af8e6d1f2f8a3d32d
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 1d9ae14c927282663397065a9c58adf399e7792d)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The data abort fault IPA obtained from HFAR_EL2 has the bottom 12 bits
zeroed out. This broke the host MMIO DABT handler because the offsets
of accessed MMIO registers were rounded down to the nearest page.
Include FAR_EL2 in the address to fix the issue.
Bug: 220194478
Change-Id: I6473e2dfbe189c58c15c0e5647d695d07f88c5e0
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 346987baf5d756ddfabebd7fa44b9bf8103f2ea0)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The S2MPU must wait for a v9 device to finish invalidation before
accessing its SFRs. Failure to do so can result in memory transaction
timeouts.
Add a loop that polls the STATUS register while the return value has
the BUSY and ON_INVALIDATING bits set.
Test: builds, boots
Bug: 190463801
Bug: 206761586
Change-Id: Ie8755bd3466b2c76ca05d6f3f2dd6e8e7bce592c
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit e149939df25f9677eab1f996b0aac7575ca2995e)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Comments in S2MPU driver code were mistakenly prefixed with /**,
denoting a kernel-doc comment. Since these do not match kernel-doc
syntax, replace them with regular /* comments.
Test: n/a
Bug: 190463801
Change-Id: I0c68bb5d1c843caeb4d535430bdfc866ba8d119c
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 4377d9dea9b2006e77ad82fd0c8120e40bc7d812)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The `init` callback of an IOMMU driver is called just before
`finalize_host_mappings` so that EL2 mappings created by drivers are
subsequently unmapped from host stage-2. However, at this point hyp has
already switched to the buddy allocator, having reserved pages allocated
by the early allocator, but `pkvm_pgtable.mm_ops` have not been switched
to buddy allocator callbacks. As a result, pages allocated for EL2
mappings of the IOMMU driver are allocated by the obsoleted early
allocator and remain treated as free by the buddy allocator. This likely
leads to a corruption in the free page lists and a later hyp panic.
Move the initialization of `pkvm_pgtable.mm_ops` before
`finalize_host_mappings` and the call to IOMMU's `init`.
Test: run a VM
Test: adb shell cmd jobscheduler run -f android 5132250
Bug: 190463801
Bug: 209004831
Change-Id: I1f6e00bca087d889b0cad4bd43d044895e37006c
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 395d0451237fb14d801591d0a42c2abaa6641d03)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The function is only used in the compilation unit where it is defined.
Silence a warning by marking it static.
Test: builds
Bug: 190463801
Change-Id: I296cffefdef4639ef2bab644d42f1374ee1a2f60
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 91abc8ece242eb595e7dd0b85eac27c51a27ba88)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The S2MPU driver needs to protect its MMIO registers from the host.
Implement the host_stage2_adjust_mmio_range callback and restrict
the address range that is about to be mapped in to avoid the known
S2MPU MMIO regions.
Test: builds, boots
Bug: 190463801
Change-Id: Ib46f5dd651b9368c31940035e4c28a7324fc4160
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 8f234061533dc798ac96fdb6b49b6b66584a8d0f)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The host should not have access to the vast majority of S2MPU MMIO
registers. Currently it only needs access to fault information, in
the future maybe also performance registers.
Implement an MMIO trap handler for the S2MPU, allowing read-only
access to FAULT_* registers, and a write-only access to
INTERRUPT_CLEAR.
Test: builds, boots
Bug: 190463801
Change-Id: Ia482cc65642ba9ec303f443591e8f0fe192d4d27
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 81e70911d6022b3af8918b273c2b4743f37b267d)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
The 'host_stage2_set_owner' callback indicates that a range of
PA-contiguous pages changed owner. With all devices owned by the host,
the driver sets the protection bits in the corresponding FMPT/SMPT to
either MPT_PROT_RW if owned by the host or MPT_PROT_NONE otherwise.
For each gigabyte region, the implementation will select between 1G and
4K/64K (depending on PAGE_SIZE) mappings and populate the L1ENTRY_ATTR
register or SMPT bitmap, respectivelly.
The driver never dynamically switches between two granularities which
both require a SMPT. This is because the L1ENTRY_ATTR and
L1ENTRY_L2TABLE_ADDR registers would need to be set atomically.
Test: builds, boots
Bug: 190463801
Change-Id: Ifb0bdcaa143ef8eb213ba4133ac86d8b610a4bcf
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 4475d993aa0c74460e8f90e88919dee7fe0e4258)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
S2MPU Second-level Memory Protection Table is a PA-contiguous buffer
containing an array of 2-bit read/write entries at given granularity
for a given gigabyte physical address space region. The size of SMPT
varies per granularity but at the finest 4K granularity it is 64KB
PA-contiguous, aligned to 64KB.
Allocate sufficient number of SMPT buffers for the S2MPU driver assuming
4K granularity for 4K/16K PAGE_SIZE, and 64K granularity for 64K
PAGE_SIZE. We also assume that all S2MPUs share SMPTs for a given
gigabyte region. There are 34 gigabyte regions that can be set by the
driver (GBs 4-33 always block all traffic).
Hyp takes ownership of the memory in s2mpu_init and assigns pointers to
the buffers to L1ENTRY_L2TABLE_ADDR registers on init and power-on
events. The pointers remain static as the driver will only change
granularity between 1G and 4K/64K (depending on PAGE_SIZE).
Test: builds, boots
Bug: 190463801
Change-Id: I3fcad8b3ce5d194a987b09d042bd56d59bb35e5e
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit f0e1de52ef17f205d499a3ee48a20d3c652d60f0)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
