If blk_crypto_evict_key() sees that the key is still in-use (due to a
bug) or that ->keyslot_evict failed, it currently just returns while
leaving the key linked into the keyslot management structures.
However, blk_crypto_evict_key() is only called in contexts such as inode
eviction where failure is not an option. So actually the caller
proceeds with freeing the blk_crypto_key regardless of the return value
of blk_crypto_evict_key().
These two assumptions don't match, and the result is that there can be a
use-after-free in blk_crypto_reprogram_all_keys() after one of these
errors occurs. (Note, these errors *shouldn't* happen; we're just
talking about what happens if they do anyway.)
Fix this by making blk_crypto_evict_key() unlink the key from the
keyslot management structures even on failure.
Also improve some comments.
Fixes: 1b26283970 ("block: Keyslot Manager for Inline Encryption")
Cc: stable@vger.kernel.org
Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20230315183907.53675-2-ebiggers@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bug: 270098322
(cherry picked from commit 5c7cb94452901a93e90c2230632e2c12a681bc92
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-next)
Change-Id: I4e8983ad7db94ea8cd422743196da8854adda552
Signed-off-by: Eric Biggers <ebiggers@google.com>
Add a resource-managed variant of blk_ksm_init() so that drivers don't
have to worry about calling blk_ksm_destroy().
Note that the implementation uses a custom devres action to call
blk_ksm_destroy() rather than switching the two allocations to be
directly devres-managed, e.g. with devm_kmalloc(). This is because we
need to keep zeroing the memory containing the keyslots when it is
freed, and also because we want to continue using kvmalloc() (and there
is no devm_kvmalloc()).
Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Satya Tangirala <satyat@google.com>
Acked-by: Jens Axboe <axboe@kernel.dk>
Link: https://lore.kernel.org/r/20210121082155.111333-2-ebiggers@kernel.org
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
(cherry picked from commit 5851d3b042b694839d2241fbb3200ce958135cdf)
Bug: 161256007
Change-Id: I3cda7fc5f7509f8d967882a88ca30ddc6f6a0426
Signed-off-by: Eric Biggers <ebiggers@google.com>
Replace the following patches with upstream versions
(well, almost upstream; as of 2021-02-12 they are queued for 5.12 at
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=for-next):
ANDROID-dm-add-support-for-passing-through-inline-crypto-support.patch
ANDROID-dm-enable-may_passthrough_inline_crypto-on-some-targets.patch
ANDROID-block-Introduce-passthrough-keyslot-manager.patch
Also, resolve conflicts with the following non-upstream patches for
hardware-wrapped key support. Notably, we need to handle the field
blk_keyslot_manager::features in a few places:
ANDROID-block-add-hardware-wrapped-key-support.patch
ANDROID-dm-add-support-for-passing-through-derive_raw_secret.patch
Finally, update non-upstream device-mapper targets (dm-bow and
dm-default-key) to use the new way of specifying inline crypto
passthrough support (DM_TARGET_PASSES_CRYPTO) rather than the old way
(may_passthrough_inline_crypto). These changes should be folded into:
ANDROID-dm-bow-Add-dm-bow-feature.patch
ANDROID-dm-add-dm-default-key-target-for-metadata-encryption.patch
Test: tested on db845c; verified that inline crypto support gets passed
through over dm-linear.
Bug: 162257830
Change-Id: I5e3dea1aa09fc1215c90857b5b51d9e3720ef7db
Signed-off-by: Eric Biggers <ebiggers@google.com>
If there is only one keyslot, then blk_ksm_init() computes
slot_hashtable_size=1 and log_slot_ht_size=0. This causes
blk_ksm_find_keyslot() to crash later because it uses
hash_ptr(key, log_slot_ht_size) to find the hash bucket containing the
key, and hash_ptr() doesn't support the bits == 0 case.
Fix this by making the hash table always have at least 2 buckets.
Tested by running:
kvm-xfstests -c ext4 -g encrypt -m inlinecrypt \
-o blk-crypto-fallback.num_keyslots=1
Fixes: 1b26283970 ("block: Keyslot Manager for Inline Encryption")
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Fix a checkpatch warning that I fixed when applying
ANDROID-dm-add-support-for-passing-through-inline-crypto-support.patch
to android12-5.4 (http://aosp/1394279). This gets android-mainline in
sync.
This should be folded into
ANDROID-dm-add-support-for-passing-through-inline-crypto-support.patch.
Bug: 162257830
Change-Id: I083209d174630220e3b6d148d7317600ce040a84
Signed-off-by: Eric Biggers <ebiggers@google.com>
Make blk_ksm_destroy() use the kvfree_sensitive() function (which was
introduced in v5.8-rc1) instead of open-coding it.
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The block layer patches for inline encryption are now in upstream, so
update Android to the upstream version of inline encryption. The
fscrypt/f2fs/ext4 patches are also updated to the latest version sent
upstream (since they can't be updated separately from the block layer
patches).
Changes v6 => v7:
- Keyslot management is now done on a per-request basis rather than a
per-bio basis.
- Storage drivers can now specify the maximum number of bytes they
can accept for the data unit number (DUN) for each crypto algorithm,
and upper layers can specify the minimum number of bytes of DUN they
want with the blk_crypto_key they send with the bio - a driver is
only considered to support a blk_crypto_key if the driver supports at
least as many DUN bytes as the upper layer wants. This is necessary
because storage drivers may not support as many bytes as the
algorithm specification dictates (for e.g. UFS only supports 8 byte
DUNs for AES-256-XTS, even though the algorithm specification
says DUNs are 16 bytes long).
- Introduce SB_INLINECRYPT to keep track of whether inline encryption
is enabled for a filesystem (instead of using an fscrypt_operation).
- Expose keyslot manager declaration and embed it within ufs_hba to
clean up code.
- Make blk-crypto preclude blk-integrity.
- Some bug fixes
- Introduce UFSHCD_QUIRK_BROKEN_CRYPTO for UFS drivers that don't
support inline encryption (yet)
Changes v7 => v8:
- Pass a struct blk_ksm_keyslot * around instead of slot numbers which
simplifies some functions and passes around arguments with better types
- Make bios with no encryption context avoid making calls into blk-crypto
by checking for the presence of bi_crypt_context before making the call
- Make blk-integrity preclude inline encryption support at probe time
- Many many cleanups
Changes v8 => v9:
- Don't open code bio_has_crypt_ctx into callers of blk-crypto functions.
- Lots of cleanups
Changes v9 => v10:
- Incorporate Eric's fix for allowing en/decryption to happen as usual via
fscrypt in the case that hardware doesn't support the desired crypto
configuration, but blk-crypto-fallback is disabled. (Introduce
struct blk_crypto_config and blk_crypto_config_supported for fscrypt
to call, to check that either blk-crypto-fallback is enabled or the
device supports the crypto configuration).
- Update docs
- Lots of cleanups
Changes v10 => v11:
- We now allocate a new bio_crypt_ctx for each request instead of
pulling and reusing the one in the bio inserted into the request. The
bio_crypt_ctx of a bio is freed after the bio is ended.
- Make each blk_ksm_keyslot store a pointer to the blk_crypto_key
instead of a copy of the blk_crypto_key, so that each blk_crypto_key
will have its own keyslot. We also won't need to compute the siphash
for a blk_crypto_key anymore.
- Minor cleanups
Changes v11 => v12:
- Inlined some fscrypt functions
- Minor cleanups and improved comments
Changes v12 => v13:
- Updated docs
- Minor cleanups
- rebased onto linux-block/for-next
Changes v13 => fscrypt/f2fs/ext4 upstream patch series
- rename struct fscrypt_info::ci_key to ci_enc_key
- set dun bytes more precisely in fscrypt
- cleanups
Bug: 137270441
Test: Test cuttlefish boots both with and without inlinecrypt mount
option specified in fstab, while using both F2FS and EXT4 for
userdata.img. Also verified ciphertext via
"atest -v vts_kernel_encryption_test"
Also tested by running gce-xfstests on both the
auto and encrypt test groups on EXT4 and F2FS both with and
without the inlinecrypt mount option. The UFS changes were
tested on a Pixel 4 device.
Link: https://lore.kernel.org/linux-block/20200514003727.69001-1-satyat@google.com/
Link: https://lore.kernel.org/linux-fscrypt/20200617075732.213198-1-satyat@google.com/
Link: https://lore.kernel.org/linux-scsi/20200617081841.218985-1-satyat@google.com/
Change-Id: I57c10d370bf006c9dfcf173f21a720413017761e
Signed-off-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Backport a fix from the v7 inline crypto patchset which ensures that the
block layer knows the number of DUN bytes the inline encryption hardware
supports, so that hardware isn't used when it shouldn't be.
(This unfortunately means introducing some increasing long argument
lists; this was all already fixed up in later versions of the patchset.)
To avoid breaking the KMI for drivers, don't add a dun_bytes argument to
keyslot_manager_create() but rather allow drivers to call
keyslot_manager_set_max_dun_bytes() to override the default. Also,
don't add dun_bytes as a new field in 'struct blk_crypto_key' but rather
pack it into the existing 'hash' field which is for block layer use.
Bug: 144046242
Bug: 153512828
Change-Id: I285f36557fb3eafc5f2f64727ef1740938b59dd7
Signed-off-by: Eric Biggers <ebiggers@google.com>
Whenever a device supports blk-integrity, make the kernel pretend that
the device doesn't support inline encryption (essentially by setting the
keyslot manager in the request queue to NULL).
There's no hardware currently that supports both integrity and inline
encryption. However, it seems possible that there will be such hardware
in the near future (like the NVMe key per I/O support that might support
both inline encryption and PI).
But properly integrating both features is not trivial, and without
real hardware that implements both, it is difficult to tell if it will
be done correctly by the majority of hardware that support both.
So it seems best not to support both features together right now, and
to decide what to do at probe time.
Signed-off-by: Satya Tangirala <satyat@google.com>
Reviewed-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Inline Encryption hardware allows software to specify an encryption context
(an encryption key, crypto algorithm, data unit num, data unit size) along
with a data transfer request to a storage device, and the inline encryption
hardware will use that context to en/decrypt the data. The inline
encryption hardware is part of the storage device, and it conceptually sits
on the data path between system memory and the storage device.
Inline Encryption hardware implementations often function around the
concept of "keyslots". These implementations often have a limited number
of "keyslots", each of which can hold a key (we say that a key can be
"programmed" into a keyslot). Requests made to the storage device may have
a keyslot and a data unit number associated with them, and the inline
encryption hardware will en/decrypt the data in the requests using the key
programmed into that associated keyslot and the data unit number specified
with the request.
As keyslots are limited, and programming keys may be expensive in many
implementations, and multiple requests may use exactly the same encryption
contexts, we introduce a Keyslot Manager to efficiently manage keyslots.
We also introduce a blk_crypto_key, which will represent the key that's
programmed into keyslots managed by keyslot managers. The keyslot manager
also functions as the interface that upper layers will use to program keys
into inline encryption hardware. For more information on the Keyslot
Manager, refer to documentation found in block/keyslot-manager.c and
linux/keyslot-manager.h.
Co-developed-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
Reviewed-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
We need a way to tell which type of keys the inline crypto hardware
supports (standard, wrapped, or both), so that fallbacks can be used
when needed (either blk-crypto-fallback, or fscrypt fs-layer crypto).
We can't simply assume that
keyslot_mgmt_ll_ops::derive_raw_secret == NULL
means only standard keys are supported and that
keyslot_mgmt_ll_ops::derive_raw_secret != NULL
means that only wrapped keys are supported, because device-mapper
devices always implement this method. Also, hardware might support both
types of keys.
Therefore, add a field keyslot_manager::features which contains a
bitmask of flags which indicate the supported types of keys. Drivers
will need to fill this in. This patch makes the UFS standard crypto
code set BLK_CRYPTO_FEATURE_STANDARD_KEYS, but UFS variant drivers may
need to set BLK_CRYPTO_FEATURE_WRAPPED_KEYS instead.
Then, make keyslot_manager_crypto_mode_supported() take the key type
into account.
Bug: 137270441
Bug: 151100202
Test: 'atest vts_kernel_encryption_test' on Pixel 4 with the
inline crypto patches backported, and also on Cuttlefish.
Change-Id: Ied846c2767c1fd2f438792dcfd3649157e68b005
Signed-off-by: Eric Biggers <ebiggers@google.com>
The call to pm_runtime_get_sync() in ufshcd_program_key() can deadlock
because it waits for the UFS controller to be resumed, but it can itself
be reached while resuming the UFS controller via:
- ufshcd_runtime_resume()
- ufshcd_resume()
- ufshcd_reset_and_restore()
- ufshcd_host_reset_and_restore()
- ufshcd_hba_enable()
- ufshcd_hba_execute_hce()
- ufshcd_hba_start()
- ufshcd_crypto_enable()
- keyslot_manager_reprogram_all_keys()
- ufshcd_crypto_keyslot_program()
- ufshcd_program_key()
But pm_runtime_get_sync() *is* needed when evicting a key. Also, on
pre-4.20 kernels it's needed when programming a keyslot for a bio since
the block layer used to resume the device in a different place.
Thus, it's hard for drivers to know what to do in .keyslot_program() and
.keyslot_evict(). In old kernels it may even be impossible unless we
were to pass more information down from the keyslot_manager.
There's also another possible deadlock: keyslot programming and eviction
take ksm->lock for write and then resume the device, which may result in
ksm->lock being taken again via the above call stack. To fix this, we
should resume the device before taking ksm->lock.
Fix these problems by moving to a better design where the block layer
(namely, the keyslot manager) handles runtime power management instead
of drivers. This is analogous to the block layer's existing runtime
power management support (blk-pm), which handles resuming devices when
bios are submitted to them so that drivers don't need to handle it.
Test: Tested on coral with:
echo 5 > /sys/bus/platform/devices/1d84000.ufshc/rpm_lvl
sleep 30
touch /data && sync # hangs before this fix
Also verified via kvm-xfstests that blk-crypto-fallback continues
to work both with and without CONFIG_PM=y.
Bug: 137270441
Bug: 149368295
Change-Id: I6bc9fb81854afe7edf490d71796ee68a61f7cbc8
Signed-off-by: Eric Biggers <ebiggers@google.com>
Update the device-mapper core to support exposing the inline crypto
support of the underlying device(s) through the device-mapper device.
This works by creating a "passthrough keyslot manager" for the dm
device, which declares support for the set of (crypto_mode,
data_unit_size) combos which all the underlying devices support. When a
supported combo is used, the bio cloning code handles cloning the crypto
context to the bios for all the underlying devices. When an unsupported
combo is used, the blk-crypto fallback is used as usual.
Crypto support on each underlying device is ignored unless the
corresponding dm target opts into exposing it. This is needed because
for inline crypto to semantically operate on the original bio, the data
must not be transformed by the dm target. Thus, targets like dm-linear
can expose crypto support of the underlying device, but targets like
dm-crypt can't. (dm-crypt could use inline crypto itself, though.)
When a key is evicted from the dm device, it is evicted from all
underlying devices.
Bug: 137270441
Bug: 147814592
Change-Id: If28b574f2e28268db5eb9f325d4cf8f96cb63e3f
Signed-off-by: Eric Biggers <ebiggers@google.com>
The regular keyslot manager is designed for devices that have a small
number of keyslots that need to be programmed with keys ahead of time,
and bios that are sent to the device need to be tagged with a keyslot
index.
Some inline encryption hardware may not have any limitations on the
number of keyslot, and may instead allow each bio to be tagged with
a raw key, data unit number, etc. rather than a pre-programmed keyslot's
index. These devices don't need any sort of keyslot management, and it's
better for these devices not to have to allocate a regular keyslot
manager with some fixed number of keyslots. These devices can instead
set up a passthrough keyslot manager in their request queue, which
require less resources than regular keyslot managers, as they simply
do no-ops when trying to program keys into slots.
Separately, the device mapper may map over devices that have inline
encryption hardware, and it wants to pass the key along to the
underlying hardware. While the DM layer can expose inline encryption
capabilities by setting up a regular keyslot manager with some fixed
number of keyslots in the dm device's request queue, this only wastes
memory since the keys programmed into the dm device's request queue
will never be used. Instead, it's better to set up a passthrough
keyslot manager for dm devices.
Bug: 137270441
Bug: 147814592
Change-Id: I6d91e83e86a73b0d6066873c8a9117cf2c089234
Signed-off-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Some inline encryption hardware supports protecting the keys in hardware
and only exposing wrapped keys to software. To use this capability,
userspace must provide a hardware-wrapped key rather than a raw key.
However, users of inline encryption in the kernel won't necessarily use
the user-specified key directly for inline encryption. E.g. with
fscrypt with IV_INO_LBLK_64 policies, each user-provided key is used to
derive a file contents encryption key, filenames encryption key, and key
identifier. Since inline encryption can only be used with file
contents, if the user were to provide a wrapped key there would
(naively) be no way to encrypt filenames or derive the key identifier.
This problem is solved by designing the hardware to internally use the
unwrapped key as input to a KDF from which multiple cryptographically
isolated keys can be derived, including both the inline crypto key (not
exposed to software) and a secret that *is* exposed to software.
Add a function to the keyslot manager to allow upper layers to request
this software secret from a hardware-wrapped key.
Bug: 147209885
Change-Id: I32f3aa4f25bcf6b9d6f7d8890260533fad00dd1d
Co-developed-by: Gaurav Kashyap <gaurkash@codeaurora.org>
Signed-off-by: Gaurav Kashyap <gaurkash@codeaurora.org>
Signed-off-by: Barani Muthukumaran <bmuthuku@codeaurora.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Currently, blk-crypto uses the algorithm to determine the size of keys.
However, some inline encryption hardware supports protecting keys from
software by wrapping the storage keys with an ephemeral key. Since
these wrapped keys are not of a fixed size, add the capability to
provide the key size when initializing a blk_crypto_key, and update the
keyslot manager to take size into account when comparing keys.
Bug: 147209885
Change-Id: I9bf26d06d18a2d671c51111b4896abe4df303988
Co-developed-by: Gaurav Kashyap <gaurkash@codeaurora.org>
Signed-off-by: Gaurav Kashyap <gaurkash@codeaurora.org>
Signed-off-by: Barani Muthukumaran <bmuthuku@codeaurora.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Changes v5 => v6:
- Blk-crypto's kernel crypto API fallback is no longer restricted to
8-byte DUNs. It's also now separately configurable from blk-crypto, and
can be disabled entirely, while still allowing the kernel to use inline
encryption hardware. Further, struct bio_crypt_ctx takes up less space,
and no longer contains the information needed by the crypto API
fallback - the fallback allocates the required memory when necessary.
- Blk-crypto now supports all file content encryption modes supported by
fscrypt.
- Fixed bio merging logic in blk-merge.c
- Fscrypt now supports inline encryption with the direct key policy, since
blk-crypto now has support for larger DUNs.
- Keyslot manager now uses a hashtable to lookup which keyslot contains
any particular key (thanks Eric!)
- Fscrypt support for inline encryption now handles filesystems with
multiple underlying block devices (thanks Eric!)
- Numerous cleanups
Bug: 137270441
Test: refer to I26376479ee38259b8c35732cb3a1d7e15f9b05a3
Change-Id: I13e2e327e0b4784b394cb1e7cf32a04856d95f01
Link: https://lore.kernel.org/linux-block/20191218145136.172774-1-satyat@google.com/
Signed-off-by: Satya Tangirala <satyat@google.com>
Inline Encryption hardware allows software to specify an encryption context
(an encryption key, crypto algorithm, data unit num, data unit size, etc.)
along with a data transfer request to a storage device, and the inline
encryption hardware will use that context to en/decrypt the data. The
inline encryption hardware is part of the storage device, and it
conceptually sits on the data path between system memory and the storage
device.
Inline Encryption hardware implementations often function around the
concept of "keyslots". These implementations often have a limited number
of "keyslots", each of which can hold an encryption context (we say that
an encryption context can be "programmed" into a keyslot). Requests made
to the storage device may have a keyslot associated with them, and the
inline encryption hardware will en/decrypt the data in the requests using
the encryption context programmed into that associated keyslot. As
keyslots are limited, and programming keys may be expensive in many
implementations, and multiple requests may use exactly the same encryption
contexts, we introduce a Keyslot Manager to efficiently manage keyslots.
The keyslot manager also functions as the interface that upper layers will
use to program keys into inline encryption hardware. For more information
on the Keyslot Manager, refer to documentation found in
block/keyslot-manager.c and linux/keyslot-manager.h.
Bug: 137270441
Test: tested as series; see Ie1b77f7615d6a7a60fdc9105c7ab2200d17636a8
Change-Id: Iea1ee5a7eec46cb50d33cf1e2d20dfb7335af4ed
Signed-off-by: Satya Tangirala <satyat@google.com>
Link: https://patchwork.kernel.org/patch/11214713/
