If page_pinner is configured with page_pinner_enabled=false and
failure_tracking=true, pp_buffer will be accessed without being
initialized. Prevent this by adding page_pinner_inited checks in
functions that access it.
Fixes: 898cfbf094a2 ("ANDROID: mm: introduce page_pinner")
Bug: 259024332
Bug: 260179017
Change-Id: I8f612cae3e74d36e8a4eee5edec25281246cbe5e
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
(cherry picked from commit 23fb3111f63e5fe239a769668275c20493a5849c)
Use page_ext_get/put() to work with the page extended information
without which the page extended information may not be valid.
Bug: 2129036
Change-Id: Ibfe036b9ecef0e2551b5d0da1011cacbb0a5c3e6
Signed-off-by: Charan Teja Kalla <quic_charante@quicinc.com>
This reverts commit d820d22b5d.
Reason for revert: Needed for non-GPL use of put_page() which is allowed by upstream kernel.
Change-Id: I9a2a51c1e2d11ccc5403fc5a8510f577446add92
Signed-off-by: Todd Kjos <tkjos@google.com>
GKI requires EXPORT_SYMBOL_GPL so change it.
Bug: 192475091
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: Iba310d0a78f6ddb1e7980177fe7c624d0a0f254a
To keep track of page migration failures, record the following
page states to capture natural state transitions of a page during
migration:
* detected
* put reference count
* free
With such transition change, it was a lot easier to analyze page
migration failure issues. Since we already have annotation on the
put_page side, this patch adds the record in free path and page
migration failure path.
Bug: 192475091
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: I18182d3fd62850c5580c9e89a5362bdae630f153
While I was investigating CMA allocation latency, adding fields below
were very useful to detect the CMA allocation latency issues.
This patch adds following fields of struct page.
for struct page:
int count
int mapcount
struct address_space
This patch removes page_mt since it was not useful ever for me.
Bug: 192475091
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: I312505901a227cd404555f845550d2a9c9ce89da
This patch changes timestamp format to align with trace event time format.
It was much easier to parse page_pinner event order by the time with
trace event.
Bug: 192475091
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: Ib0e81a47df588a2f6ebf842a5f3c5b298da4fe40
For print_page_pinner argument passing, use the one structure
parameter instead of several parameters with fields of the
structure.
Bug: 192475091
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: I3816eb6154c6c56a66ee9079091880afda817914
There are several path the WARN_ON_ONCE could be triggered but
it doesn't mean real bug since we intentionally allow it to make
code simple at this moment. So just remove the warnings to prevent
folks confused it.
Bug: 192475091
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: I7d14c8490db841f61b01fe7c16b406c9b1850c46
Currently, __reset_page_pinner reset only PAGE_EXT_GET page freeing,
not PAGE_EXT_PINNER_MIGRATION_FAILED page. It should handle both
cases to prevent wrong PAGE_EXT_PINNER_MIGRATION_FAILED setting.
Bug: 192475091
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: I9b4124393ee432c7cb29cd12d160ecf2a1e34360
longterm_pinner/alloc_contig_failed should be 444
for reading for everyone and threshold/failure_traking
should be 644 to allow read/write root.
Bug: 191827366
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: I5493c6ae6d39b692282e5428416da80a7d555aa0
Sometime, pages are temporarily pinnned during migration and the migration
fails. However, putback_movable_pages will end up freeing them if their
page refcount are 1. Thus, it doesn't need to mark them as failure, which
just consumes page_pinner logbuffer to lose old history.
Bug: 188908895
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: I8564b72b212a5095cfe3ba6bf5622a9b62f5b455
CMA allocation can fail by temporal page refcount increasement
by get_page API as well as get_user_pages friends.
However, since get_page is one of the most hot function, it is
hard to hook get_page to get callstack everytime due to
performance concern. Furthermore, get_page could be nested
multiple times so we couldn't track all of the pin sites on
limited space of page_pinner.
Thus, here approach is keep tracking of put_page callsite rather
than get_page once VM found the page migration failed.
It's based on assumption:
1. Since it's temporal page refcount, it could be released soon
before overflowing dmesg log buffer
2. developer can find the pair of get_page by reviewing put_page.
By default, it's eanbled. If you want to disable it:
echo 0 > $debugfs/page_pinner/failure_tracking
You can capture the tracking using:
cat $debugfs/page_pinner/alloc_contig_failed
note: the example below is artificial:
Page pinned ts 386067292 us count 0
PFN 10162530 Block 9924 type Isolate Flags 0x800000000008000c(uptodate|dirty|swapbacked)
__page_pinner_migration_failed+0x30/0x104
putback_lru_page+0x90/0xac
putback_movable_pages+0xc4/0x204
__alloc_contig_migrate_range+0x290/0x31c
alloc_contig_range+0x114/0x2bc
cma_alloc+0x2d8/0x698
cma_alloc_write+0x58/0xb8
simple_attr_write+0xd4/0x124
debugfs_attr_write+0x50/0xd8
full_proxy_write+0x70/0xf8
vfs_write+0x168/0x3a8
ksys_write+0x7c/0xec
__arm64_sys_write+0x20/0x30
el0_svc_common+0xa4/0x180
do_el0_svc+0x28/0x88
el0_svc+0x14/0x24
Page pinned ts 385867394 us count 0
PFN 10162530 Block 9924 type Isolate Flags 0x800000000008000c(uptodate|dirty|swapbacked)
__page_pinner_migration_failed+0x30/0x104
__alloc_contig_migrate_range+0x200/0x31c
alloc_contig_range+0x114/0x2bc
cma_alloc+0x2d8/0x698
cma_alloc_write+0x58/0xb8
simple_attr_write+0xd4/0x124
debugfs_attr_write+0x50/0xd8
full_proxy_write+0x70/0xf8
vfs_write+0x168/0x3a8
ksys_write+0x7c/0xec
__arm64_sys_write+0x20/0x30
el0_svc_common+0xa4/0x180
do_el0_svc+0x28/0x88
el0_svc+0x14/0x24
el0_sync_handler+0x88/0xec
el0_sync+0x198/0x1c0
Bug: 183414571
Signed-off-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: Ie79902c18390eb9f320d823839bb9d9a7fdcdb31
For CMA allocation, it's really critical to migrate a page but
sometimes it fails. One of the reasons is some driver holds a
page refcount for a long time so VM couldn't migrate the page
at that time.
The concern here is there is no way to find the who hold the
refcount of the page effectively. This patch introduces feature
to keep tracking page's pinner. All get_page sites are vulnerable
to pin a page for a long time but the cost to keep track it would
be significat since get_page is the most frequent kernel operation.
Furthermore, the page could be not user page but kernel page which
is not related to the page migration failure. So, this patch keeps
tracking only get_user_pages/follow_page with (FOLL_GET|PIN friends
because they are the very common APIs to pin user pages which could
cause migration failure and the less frequent than get_page so
runtime cost wouldn't be that big but could cover many cases
effectively.
This patch also introduces put_user_page API. It aims for attributing
"the pinner releases the page from now on" while it release the
page refcount. Thus, any user of get_user_pages/follow_page(FOLL_GET)
must use put_user_page as pair of those functions. Otherwise,
page_pinner will treat them long term pinner as false postive but
nothing should affect stability.
* $debugfs/page_pinner/threshold
It indicates threshold(microsecond) to flag long term pinning.
It's configurable(Default is 300000us). Once you write new value
to the threshold, old data will clear.
* $debugfs/page_pinner/longterm_pinner
It shows call sites where the duration of pinning was greater than
the threshold. Internally, it uses a static array to keep 4096
elements and overwrites old ones once overflow happens. Therefore,
you could lose some information.
example)
Page pinned ts 76953865787 us count 1
PFN 9856945 Block 9625 type Movable Flags 0x8000000000080014(uptodate|lru|swapbacked)
__set_page_pinner+0x34/0xcc
try_grab_page+0x19c/0x1a0
follow_page_pte+0x1c0/0x33c
follow_page_mask+0xc0/0xc8
__get_user_pages+0x178/0x414
__gup_longterm_locked+0x80/0x148
internal_get_user_pages_fast+0x140/0x174
pin_user_pages_fast+0x24/0x40
CCC
BBB
AAA
__arm64_sys_ioctl+0x94/0xd0
el0_svc_common+0xa4/0x180
do_el0_svc+0x28/0x88
el0_svc+0x14/0x24
note: page_pinner doesn't guarantee attributing/unattributing are
atomic if they happen at the same time. It's just best effort so
false-positive could happen.
Bug: 183414571
Signed-off-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Minchan Kim <minchan@google.com>
Change-Id: Ife37ec360eef993d390b9c131732218a4dfd2f04
