2019-05-19 21:07:45 +09:00
|
|
|
# SPDX-License-Identifier: GPL-2.0-only
|
2018-07-31 20:39:35 +09:00
|
|
|
|
|
|
|
|
menu "Memory Management options"
|
|
|
|
|
|
2005-06-23 16:07:49 +09:00
|
|
|
config SELECT_MEMORY_MODEL
|
|
|
|
|
def_bool y
|
2013-01-17 11:54:17 +09:00
|
|
|
depends on ARCH_SELECT_MEMORY_MODEL
|
2005-06-23 16:07:49 +09:00
|
|
|
|
2005-06-23 16:07:42 +09:00
|
|
|
choice
|
|
|
|
|
prompt "Memory model"
|
2005-06-23 16:07:49 +09:00
|
|
|
depends on SELECT_MEMORY_MODEL
|
|
|
|
|
default DISCONTIGMEM_MANUAL if ARCH_DISCONTIGMEM_DEFAULT
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
default SPARSEMEM_MANUAL if ARCH_SPARSEMEM_DEFAULT
|
2005-06-23 16:07:49 +09:00
|
|
|
default FLATMEM_MANUAL
|
2019-05-14 09:23:05 +09:00
|
|
|
help
|
|
|
|
|
This option allows you to change some of the ways that
|
|
|
|
|
Linux manages its memory internally. Most users will
|
|
|
|
|
only have one option here selected by the architecture
|
|
|
|
|
configuration. This is normal.
|
2005-06-23 16:07:42 +09:00
|
|
|
|
2005-06-23 16:07:49 +09:00
|
|
|
config FLATMEM_MANUAL
|
2005-06-23 16:07:42 +09:00
|
|
|
bool "Flat Memory"
|
2006-01-06 17:12:07 +09:00
|
|
|
depends on !(ARCH_DISCONTIGMEM_ENABLE || ARCH_SPARSEMEM_ENABLE) || ARCH_FLATMEM_ENABLE
|
2005-06-23 16:07:42 +09:00
|
|
|
help
|
2019-05-14 09:23:05 +09:00
|
|
|
This option is best suited for non-NUMA systems with
|
|
|
|
|
flat address space. The FLATMEM is the most efficient
|
|
|
|
|
system in terms of performance and resource consumption
|
|
|
|
|
and it is the best option for smaller systems.
|
|
|
|
|
|
|
|
|
|
For systems that have holes in their physical address
|
|
|
|
|
spaces and for features like NUMA and memory hotplug,
|
|
|
|
|
choose "Sparse Memory"
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
|
|
|
|
|
If unsure, choose this option (Flat Memory) over any other.
|
2005-06-23 16:07:42 +09:00
|
|
|
|
2005-06-23 16:07:49 +09:00
|
|
|
config DISCONTIGMEM_MANUAL
|
2005-09-17 11:27:54 +09:00
|
|
|
bool "Discontiguous Memory"
|
2005-06-23 16:07:42 +09:00
|
|
|
depends on ARCH_DISCONTIGMEM_ENABLE
|
|
|
|
|
help
|
2005-06-23 16:07:50 +09:00
|
|
|
This option provides enhanced support for discontiguous
|
|
|
|
|
memory systems, over FLATMEM. These systems have holes
|
|
|
|
|
in their physical address spaces, and this option provides
|
2019-05-14 09:23:05 +09:00
|
|
|
more efficient handling of these holes.
|
2005-06-23 16:07:50 +09:00
|
|
|
|
2019-05-14 09:23:05 +09:00
|
|
|
Although "Discontiguous Memory" is still used by several
|
|
|
|
|
architectures, it is considered deprecated in favor of
|
|
|
|
|
"Sparse Memory".
|
2005-06-23 16:07:50 +09:00
|
|
|
|
2019-05-14 09:23:05 +09:00
|
|
|
If unsure, choose "Sparse Memory" over this option.
|
2005-06-23 16:07:42 +09:00
|
|
|
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
config SPARSEMEM_MANUAL
|
|
|
|
|
bool "Sparse Memory"
|
|
|
|
|
depends on ARCH_SPARSEMEM_ENABLE
|
|
|
|
|
help
|
|
|
|
|
This will be the only option for some systems, including
|
2019-05-14 09:23:05 +09:00
|
|
|
memory hot-plug systems. This is normal.
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
|
2019-05-14 09:23:05 +09:00
|
|
|
This option provides efficient support for systems with
|
|
|
|
|
holes is their physical address space and allows memory
|
|
|
|
|
hot-plug and hot-remove.
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
|
2019-05-14 09:23:05 +09:00
|
|
|
If unsure, choose "Flat Memory" over this option.
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
|
2005-06-23 16:07:42 +09:00
|
|
|
endchoice
|
|
|
|
|
|
2005-06-23 16:07:49 +09:00
|
|
|
config DISCONTIGMEM
|
|
|
|
|
def_bool y
|
|
|
|
|
depends on (!SELECT_MEMORY_MODEL && ARCH_DISCONTIGMEM_ENABLE) || DISCONTIGMEM_MANUAL
|
|
|
|
|
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
config SPARSEMEM
|
|
|
|
|
def_bool y
|
2009-10-27 08:50:12 +09:00
|
|
|
depends on (!SELECT_MEMORY_MODEL && ARCH_SPARSEMEM_ENABLE) || SPARSEMEM_MANUAL
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
|
2005-06-23 16:07:49 +09:00
|
|
|
config FLATMEM
|
|
|
|
|
def_bool y
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
depends on (!DISCONTIGMEM && !SPARSEMEM) || FLATMEM_MANUAL
|
|
|
|
|
|
|
|
|
|
config FLAT_NODE_MEM_MAP
|
|
|
|
|
def_bool y
|
|
|
|
|
depends on !SPARSEMEM
|
2005-06-23 16:07:49 +09:00
|
|
|
|
2005-06-23 16:07:47 +09:00
|
|
|
#
|
|
|
|
|
# Both the NUMA code and DISCONTIGMEM use arrays of pg_data_t's
|
|
|
|
|
# to represent different areas of memory. This variable allows
|
|
|
|
|
# those dependencies to exist individually.
|
|
|
|
|
#
|
|
|
|
|
config NEED_MULTIPLE_NODES
|
|
|
|
|
def_bool y
|
|
|
|
|
depends on DISCONTIGMEM || NUMA
|
2005-06-23 16:07:53 +09:00
|
|
|
|
|
|
|
|
config HAVE_MEMORY_PRESENT
|
|
|
|
|
def_bool y
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 16:07:54 +09:00
|
|
|
depends on ARCH_HAVE_MEMORY_PRESENT || SPARSEMEM
|
2005-09-04 07:54:26 +09:00
|
|
|
|
2005-09-04 07:54:28 +09:00
|
|
|
#
|
|
|
|
|
# SPARSEMEM_EXTREME (which is the default) does some bootmem
|
2006-10-04 05:53:09 +09:00
|
|
|
# allocations when memory_present() is called. If this cannot
|
2005-09-04 07:54:28 +09:00
|
|
|
# be done on your architecture, select this option. However,
|
|
|
|
|
# statically allocating the mem_section[] array can potentially
|
|
|
|
|
# consume vast quantities of .bss, so be careful.
|
|
|
|
|
#
|
|
|
|
|
# This option will also potentially produce smaller runtime code
|
|
|
|
|
# with gcc 3.4 and later.
|
|
|
|
|
#
|
|
|
|
|
config SPARSEMEM_STATIC
|
2008-10-16 14:01:38 +09:00
|
|
|
bool
|
2005-09-04 07:54:28 +09:00
|
|
|
|
2005-09-04 07:54:26 +09:00
|
|
|
#
|
2006-10-04 05:34:14 +09:00
|
|
|
# Architecture platforms which require a two level mem_section in SPARSEMEM
|
2005-09-04 07:54:26 +09:00
|
|
|
# must select this option. This is usually for architecture platforms with
|
|
|
|
|
# an extremely sparse physical address space.
|
|
|
|
|
#
|
2005-09-04 07:54:28 +09:00
|
|
|
config SPARSEMEM_EXTREME
|
|
|
|
|
def_bool y
|
|
|
|
|
depends on SPARSEMEM && !SPARSEMEM_STATIC
|
[PATCH] mm: split page table lock
Christoph Lameter demonstrated very poor scalability on the SGI 512-way, with
a many-threaded application which concurrently initializes different parts of
a large anonymous area.
This patch corrects that, by using a separate spinlock per page table page, to
guard the page table entries in that page, instead of using the mm's single
page_table_lock. (But even then, page_table_lock is still used to guard page
table allocation, and anon_vma allocation.)
In this implementation, the spinlock is tucked inside the struct page of the
page table page: with a BUILD_BUG_ON in case it overflows - which it would in
the case of 32-bit PA-RISC with spinlock debugging enabled.
Splitting the lock is not quite for free: another cacheline access. Ideally,
I suppose we would use split ptlock only for multi-threaded processes on
multi-cpu machines; but deciding that dynamically would have its own costs.
So for now enable it by config, at some number of cpus - since the Kconfig
language doesn't support inequalities, let preprocessor compare that with
NR_CPUS. But I don't think it's worth being user-configurable: for good
testing of both split and unsplit configs, split now at 4 cpus, and perhaps
change that to 8 later.
There is a benefit even for singly threaded processes: kswapd can be attacking
one part of the mm while another part is busy faulting.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-10-30 10:16:40 +09:00
|
|
|
|
2007-10-16 17:24:14 +09:00
|
|
|
config SPARSEMEM_VMEMMAP_ENABLE
|
2008-10-16 14:01:38 +09:00
|
|
|
bool
|
2007-10-16 17:24:14 +09:00
|
|
|
|
|
|
|
|
config SPARSEMEM_VMEMMAP
|
2007-12-18 09:19:53 +09:00
|
|
|
bool "Sparse Memory virtual memmap"
|
|
|
|
|
depends on SPARSEMEM && SPARSEMEM_VMEMMAP_ENABLE
|
|
|
|
|
default y
|
|
|
|
|
help
|
|
|
|
|
SPARSEMEM_VMEMMAP uses a virtually mapped memmap to optimise
|
|
|
|
|
pfn_to_page and page_to_pfn operations. This is the most
|
|
|
|
|
efficient option when sufficient kernel resources are available.
|
2007-10-16 17:24:14 +09:00
|
|
|
|
2011-07-14 18:43:42 +09:00
|
|
|
config HAVE_MEMBLOCK_NODE_MAP
|
2014-12-21 05:41:11 +09:00
|
|
|
bool
|
2011-07-14 18:43:42 +09:00
|
|
|
|
2014-01-30 02:16:01 +09:00
|
|
|
config HAVE_MEMBLOCK_PHYS_MAP
|
2014-12-21 05:41:11 +09:00
|
|
|
bool
|
2014-01-30 02:16:01 +09:00
|
|
|
|
2019-07-12 12:57:14 +09:00
|
|
|
config HAVE_FAST_GUP
|
2019-07-12 12:57:21 +09:00
|
|
|
depends on MMU
|
2014-12-21 05:41:11 +09:00
|
|
|
bool
|
2014-10-10 07:29:14 +09:00
|
|
|
|
2019-05-14 09:22:59 +09:00
|
|
|
config ARCH_KEEP_MEMBLOCK
|
2014-12-21 05:41:11 +09:00
|
|
|
bool
|
2011-07-14 18:46:03 +09:00
|
|
|
|
2012-08-01 08:43:50 +09:00
|
|
|
config MEMORY_ISOLATION
|
2014-12-21 05:41:11 +09:00
|
|
|
bool
|
2012-08-01 08:43:50 +09:00
|
|
|
|
2013-02-23 09:33:00 +09:00
|
|
|
#
|
|
|
|
|
# Only be set on architectures that have completely implemented memory hotplug
|
|
|
|
|
# feature. If you are not sure, don't touch it.
|
|
|
|
|
#
|
|
|
|
|
config HAVE_BOOTMEM_INFO_NODE
|
|
|
|
|
def_bool n
|
|
|
|
|
|
2005-10-30 10:16:54 +09:00
|
|
|
# eventually, we can have this option just 'select SPARSEMEM'
|
|
|
|
|
config MEMORY_HOTPLUG
|
|
|
|
|
bool "Allow for memory hot-add"
|
2006-10-01 15:27:05 +09:00
|
|
|
depends on SPARSEMEM || X86_64_ACPI_NUMA
|
2013-05-21 12:49:35 +09:00
|
|
|
depends on ARCH_ENABLE_MEMORY_HOTPLUG
|
2017-09-20 16:50:41 +09:00
|
|
|
depends on ARM64
|
2005-10-30 10:16:54 +09:00
|
|
|
|
2006-10-01 15:27:05 +09:00
|
|
|
config MEMORY_HOTPLUG_SPARSE
|
|
|
|
|
def_bool y
|
|
|
|
|
depends on SPARSEMEM && MEMORY_HOTPLUG
|
|
|
|
|
|
2016-05-20 09:13:03 +09:00
|
|
|
config MEMORY_HOTPLUG_DEFAULT_ONLINE
|
|
|
|
|
bool "Online the newly added memory blocks by default"
|
|
|
|
|
depends on MEMORY_HOTPLUG
|
|
|
|
|
help
|
|
|
|
|
This option sets the default policy setting for memory hotplug
|
|
|
|
|
onlining policy (/sys/devices/system/memory/auto_online_blocks) which
|
|
|
|
|
determines what happens to newly added memory regions. Policy setting
|
|
|
|
|
can always be changed at runtime.
|
2019-06-08 03:54:32 +09:00
|
|
|
See Documentation/admin-guide/mm/memory-hotplug.rst for more information.
|
2016-05-20 09:13:03 +09:00
|
|
|
|
|
|
|
|
Say Y here if you want all hot-plugged memory blocks to appear in
|
|
|
|
|
'online' state by default.
|
|
|
|
|
Say N here if you want the default policy to keep all hot-plugged
|
|
|
|
|
memory blocks in 'offline' state.
|
|
|
|
|
|
2018-05-16 11:52:51 +09:00
|
|
|
config MEMORY_HOTPLUG_MOVABLE_NODE
|
|
|
|
|
bool "Add hot-added memory blocks to ZONE_MOVABLE type"
|
|
|
|
|
default n
|
|
|
|
|
depends on MEMORY_HOTPLUG
|
|
|
|
|
help
|
|
|
|
|
When onlining memory blocks, this option helps to add the target
|
|
|
|
|
memory block to ZONE_MOVABLE zone type. For successful offlining,
|
|
|
|
|
these memory blocks should belong to 'ZONE_MOVABLE' since it carries
|
|
|
|
|
only movable pages. When this option is not set, the default zone
|
|
|
|
|
policy is to add the blocks to 'ZONE_NORMAL' which may pin pages.
|
|
|
|
|
|
|
|
|
|
See Documentation/memory-hotplug.txt for more information.
|
|
|
|
|
|
|
|
|
|
Say Y here if you want all hot-added memory blocks to be added to
|
|
|
|
|
'ZONE_MOVABLE' type.state by default.
|
|
|
|
|
Say N here if you want the default policy to add all hot-added
|
|
|
|
|
memory blocks in 'ZONE_NORMAL' type.
|
|
|
|
|
|
2007-10-16 17:26:12 +09:00
|
|
|
config MEMORY_HOTREMOVE
|
|
|
|
|
bool "Allow for memory hot remove"
|
2013-02-23 09:33:00 +09:00
|
|
|
select MEMORY_ISOLATION
|
2013-09-28 00:18:09 +09:00
|
|
|
select HAVE_BOOTMEM_INFO_NODE if (X86_64 || PPC64)
|
2007-10-16 17:26:12 +09:00
|
|
|
depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE
|
|
|
|
|
depends on MIGRATION
|
|
|
|
|
|
[PATCH] mm: split page table lock
Christoph Lameter demonstrated very poor scalability on the SGI 512-way, with
a many-threaded application which concurrently initializes different parts of
a large anonymous area.
This patch corrects that, by using a separate spinlock per page table page, to
guard the page table entries in that page, instead of using the mm's single
page_table_lock. (But even then, page_table_lock is still used to guard page
table allocation, and anon_vma allocation.)
In this implementation, the spinlock is tucked inside the struct page of the
page table page: with a BUILD_BUG_ON in case it overflows - which it would in
the case of 32-bit PA-RISC with spinlock debugging enabled.
Splitting the lock is not quite for free: another cacheline access. Ideally,
I suppose we would use split ptlock only for multi-threaded processes on
multi-cpu machines; but deciding that dynamically would have its own costs.
So for now enable it by config, at some number of cpus - since the Kconfig
language doesn't support inequalities, let preprocessor compare that with
NR_CPUS. But I don't think it's worth being user-configurable: for good
testing of both split and unsplit configs, split now at 4 cpus, and perhaps
change that to 8 later.
There is a benefit even for singly threaded processes: kswapd can be attacking
one part of the mm while another part is busy faulting.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-10-30 10:16:40 +09:00
|
|
|
# Heavily threaded applications may benefit from splitting the mm-wide
|
|
|
|
|
# page_table_lock, so that faults on different parts of the user address
|
|
|
|
|
# space can be handled with less contention: split it at this NR_CPUS.
|
|
|
|
|
# Default to 4 for wider testing, though 8 might be more appropriate.
|
|
|
|
|
# ARM's adjust_pte (unused if VIPT) depends on mm-wide page_table_lock.
|
2005-11-24 06:37:37 +09:00
|
|
|
# PA-RISC 7xxx's spinlock_t would enlarge struct page from 32 to 44 bytes.
|
2009-12-15 10:59:02 +09:00
|
|
|
# DEBUG_SPINLOCK and DEBUG_LOCK_ALLOC spinlock_t also enlarge struct page.
|
[PATCH] mm: split page table lock
Christoph Lameter demonstrated very poor scalability on the SGI 512-way, with
a many-threaded application which concurrently initializes different parts of
a large anonymous area.
This patch corrects that, by using a separate spinlock per page table page, to
guard the page table entries in that page, instead of using the mm's single
page_table_lock. (But even then, page_table_lock is still used to guard page
table allocation, and anon_vma allocation.)
In this implementation, the spinlock is tucked inside the struct page of the
page table page: with a BUILD_BUG_ON in case it overflows - which it would in
the case of 32-bit PA-RISC with spinlock debugging enabled.
Splitting the lock is not quite for free: another cacheline access. Ideally,
I suppose we would use split ptlock only for multi-threaded processes on
multi-cpu machines; but deciding that dynamically would have its own costs.
So for now enable it by config, at some number of cpus - since the Kconfig
language doesn't support inequalities, let preprocessor compare that with
NR_CPUS. But I don't think it's worth being user-configurable: for good
testing of both split and unsplit configs, split now at 4 cpus, and perhaps
change that to 8 later.
There is a benefit even for singly threaded processes: kswapd can be attacking
one part of the mm while another part is busy faulting.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-10-30 10:16:40 +09:00
|
|
|
#
|
|
|
|
|
config SPLIT_PTLOCK_CPUS
|
|
|
|
|
int
|
2014-04-08 07:37:14 +09:00
|
|
|
default "999999" if !MMU
|
2009-12-15 10:59:02 +09:00
|
|
|
default "999999" if ARM && !CPU_CACHE_VIPT
|
|
|
|
|
default "999999" if PARISC && !PA20
|
[PATCH] mm: split page table lock
Christoph Lameter demonstrated very poor scalability on the SGI 512-way, with
a many-threaded application which concurrently initializes different parts of
a large anonymous area.
This patch corrects that, by using a separate spinlock per page table page, to
guard the page table entries in that page, instead of using the mm's single
page_table_lock. (But even then, page_table_lock is still used to guard page
table allocation, and anon_vma allocation.)
In this implementation, the spinlock is tucked inside the struct page of the
page table page: with a BUILD_BUG_ON in case it overflows - which it would in
the case of 32-bit PA-RISC with spinlock debugging enabled.
Splitting the lock is not quite for free: another cacheline access. Ideally,
I suppose we would use split ptlock only for multi-threaded processes on
multi-cpu machines; but deciding that dynamically would have its own costs.
So for now enable it by config, at some number of cpus - since the Kconfig
language doesn't support inequalities, let preprocessor compare that with
NR_CPUS. But I don't think it's worth being user-configurable: for good
testing of both split and unsplit configs, split now at 4 cpus, and perhaps
change that to 8 later.
There is a benefit even for singly threaded processes: kswapd can be attacking
one part of the mm while another part is busy faulting.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-10-30 10:16:40 +09:00
|
|
|
default "4"
|
2006-01-08 18:00:49 +09:00
|
|
|
|
2013-11-15 07:31:07 +09:00
|
|
|
config ARCH_ENABLE_SPLIT_PMD_PTLOCK
|
2014-12-21 05:41:11 +09:00
|
|
|
bool
|
2013-11-15 07:31:07 +09:00
|
|
|
|
2014-10-10 07:29:32 +09:00
|
|
|
#
|
|
|
|
|
# support for memory balloon
|
|
|
|
|
config MEMORY_BALLOON
|
2014-12-21 05:41:11 +09:00
|
|
|
bool
|
2014-10-10 07:29:32 +09:00
|
|
|
|
2012-12-12 09:02:38 +09:00
|
|
|
#
|
|
|
|
|
# support for memory balloon compaction
|
|
|
|
|
config BALLOON_COMPACTION
|
|
|
|
|
bool "Allow for balloon memory compaction/migration"
|
|
|
|
|
def_bool y
|
2014-10-10 07:29:32 +09:00
|
|
|
depends on COMPACTION && MEMORY_BALLOON
|
2012-12-12 09:02:38 +09:00
|
|
|
help
|
|
|
|
|
Memory fragmentation introduced by ballooning might reduce
|
|
|
|
|
significantly the number of 2MB contiguous memory blocks that can be
|
|
|
|
|
used within a guest, thus imposing performance penalties associated
|
|
|
|
|
with the reduced number of transparent huge pages that could be used
|
|
|
|
|
by the guest workload. Allowing the compaction & migration for memory
|
|
|
|
|
pages enlisted as being part of memory balloon devices avoids the
|
|
|
|
|
scenario aforementioned and helps improving memory defragmentation.
|
|
|
|
|
|
2010-05-25 06:32:21 +09:00
|
|
|
#
|
|
|
|
|
# support for memory compaction
|
|
|
|
|
config COMPACTION
|
|
|
|
|
bool "Allow for memory compaction"
|
2012-10-09 08:33:03 +09:00
|
|
|
def_bool y
|
2010-05-25 06:32:21 +09:00
|
|
|
select MIGRATION
|
2011-01-26 08:07:25 +09:00
|
|
|
depends on MMU
|
2010-05-25 06:32:21 +09:00
|
|
|
help
|
2016-08-26 07:17:05 +09:00
|
|
|
Compaction is the only memory management component to form
|
|
|
|
|
high order (larger physically contiguous) memory blocks
|
|
|
|
|
reliably. The page allocator relies on compaction heavily and
|
|
|
|
|
the lack of the feature can lead to unexpected OOM killer
|
|
|
|
|
invocations for high order memory requests. You shouldn't
|
|
|
|
|
disable this option unless there really is a strong reason for
|
|
|
|
|
it and then we would be really interested to hear about that at
|
|
|
|
|
linux-mm@kvack.org.
|
2010-05-25 06:32:21 +09:00
|
|
|
|
2006-01-08 18:00:49 +09:00
|
|
|
#
|
|
|
|
|
# support for page migration
|
|
|
|
|
#
|
|
|
|
|
config MIGRATION
|
2006-03-22 17:09:12 +09:00
|
|
|
bool "Page migration"
|
2006-06-23 18:03:37 +09:00
|
|
|
def_bool y
|
2013-09-13 07:14:08 +09:00
|
|
|
depends on (NUMA || ARCH_ENABLE_MEMORY_HOTREMOVE || COMPACTION || CMA) && MMU
|
2006-03-22 17:09:12 +09:00
|
|
|
help
|
|
|
|
|
Allows the migration of the physical location of pages of processes
|
2010-05-25 06:32:21 +09:00
|
|
|
while the virtual addresses are not changed. This is useful in
|
|
|
|
|
two situations. The first is on NUMA systems to put pages nearer
|
|
|
|
|
to the processors accessing. The second is when allocating huge
|
|
|
|
|
pages as migration can relocate pages to satisfy a huge page
|
|
|
|
|
allocation instead of reclaiming.
|
2006-06-13 09:11:31 +09:00
|
|
|
|
2014-06-05 08:05:35 +09:00
|
|
|
config ARCH_ENABLE_HUGEPAGE_MIGRATION
|
2014-12-21 05:41:11 +09:00
|
|
|
bool
|
2014-06-05 08:05:35 +09:00
|
|
|
|
2017-09-09 08:10:53 +09:00
|
|
|
config ARCH_ENABLE_THP_MIGRATION
|
|
|
|
|
bool
|
|
|
|
|
|
2019-05-14 09:19:00 +09:00
|
|
|
config CONTIG_ALLOC
|
|
|
|
|
def_bool (MEMORY_ISOLATION && COMPACTION) || CMA
|
|
|
|
|
|
2008-09-11 17:31:45 +09:00
|
|
|
config PHYS_ADDR_T_64BIT
|
2018-04-03 23:24:20 +09:00
|
|
|
def_bool 64BIT
|
2008-09-11 17:31:45 +09:00
|
|
|
|
2007-07-17 20:03:37 +09:00
|
|
|
config BOUNCE
|
2013-04-30 07:08:55 +09:00
|
|
|
bool "Enable bounce buffers"
|
|
|
|
|
default y
|
2007-07-17 20:03:37 +09:00
|
|
|
depends on BLOCK && MMU && (ZONE_DMA || HIGHMEM)
|
2013-04-30 07:08:55 +09:00
|
|
|
help
|
|
|
|
|
Enable bounce buffers for devices that cannot access
|
|
|
|
|
the full range of memory available to the CPU. Enabled
|
|
|
|
|
by default when ZONE_DMA or HIGHMEM is selected, but you
|
|
|
|
|
may say n to override this.
|
2007-07-17 20:03:37 +09:00
|
|
|
|
2007-07-16 15:40:05 +09:00
|
|
|
config VIRT_TO_BUS
|
2013-03-07 13:48:16 +09:00
|
|
|
bool
|
|
|
|
|
help
|
|
|
|
|
An architecture should select this if it implements the
|
|
|
|
|
deprecated interface virt_to_bus(). All new architectures
|
|
|
|
|
should probably not select this.
|
|
|
|
|
|
mmu-notifiers: core
With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to pages.
There are secondary MMUs (with secondary sptes and secondary tlbs) too.
sptes in the kvm case are shadow pagetables, but when I say spte in
mmu-notifier context, I mean "secondary pte". In GRU case there's no
actual secondary pte and there's only a secondary tlb because the GRU
secondary MMU has no knowledge about sptes and every secondary tlb miss
event in the MMU always generates a page fault that has to be resolved by
the CPU (this is not the case of KVM where the a secondary tlb miss will
walk sptes in hardware and it will refill the secondary tlb transparently
to software if the corresponding spte is present). The same way
zap_page_range has to invalidate the pte before freeing the page, the spte
(and secondary tlb) must also be invalidated before any page is freed and
reused.
Currently we take a page_count pin on every page mapped by sptes, but that
means the pages can't be swapped whenever they're mapped by any spte
because they're part of the guest working set. Furthermore a spte unmap
event can immediately lead to a page to be freed when the pin is released
(so requiring the same complex and relatively slow tlb_gather smp safe
logic we have in zap_page_range and that can be avoided completely if the
spte unmap event doesn't require an unpin of the page previously mapped in
the secondary MMU).
The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and know
when the VM is swapping or freeing or doing anything on the primary MMU so
that the secondary MMU code can drop sptes before the pages are freed,
avoiding all page pinning and allowing 100% reliable swapping of guest
physical address space. Furthermore it avoids the code that teardown the
mappings of the secondary MMU, to implement a logic like tlb_gather in
zap_page_range that would require many IPI to flush other cpu tlbs, for
each fixed number of spte unmapped.
To make an example: if what happens on the primary MMU is a protection
downgrade (from writeable to wrprotect) the secondary MMU mappings will be
invalidated, and the next secondary-mmu-page-fault will call
get_user_pages and trigger a do_wp_page through get_user_pages if it
called get_user_pages with write=1, and it'll re-establishing an updated
spte or secondary-tlb-mapping on the copied page. Or it will setup a
readonly spte or readonly tlb mapping if it's a guest-read, if it calls
get_user_pages with write=0. This is just an example.
This allows to map any page pointed by any pte (and in turn visible in the
primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, or an
full MMU with both sptes and secondary-tlb like the shadow-pagetable layer
with kvm), or a remote DMA in software like XPMEM (hence needing of
schedule in XPMEM code to send the invalidate to the remote node, while no
need to schedule in kvm/gru as it's an immediate event like invalidating
primary-mmu pte).
At least for KVM without this patch it's impossible to swap guests
reliably. And having this feature and removing the page pin allows
several other optimizations that simplify life considerably.
Dependencies:
1) mm_take_all_locks() to register the mmu notifier when the whole VM
isn't doing anything with "mm". This allows mmu notifier users to keep
track if the VM is in the middle of the invalidate_range_begin/end
critical section with an atomic counter incraese in range_begin and
decreased in range_end. No secondary MMU page fault is allowed to map
any spte or secondary tlb reference, while the VM is in the middle of
range_begin/end as any page returned by get_user_pages in that critical
section could later immediately be freed without any further
->invalidate_page notification (invalidate_range_begin/end works on
ranges and ->invalidate_page isn't called immediately before freeing
the page). To stop all page freeing and pagetable overwrites the
mmap_sem must be taken in write mode and all other anon_vma/i_mmap
locks must be taken too.
2) It'd be a waste to add branches in the VM if nobody could possibly
run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled if
CONFIG_KVM=m/y. In the current kernel kvm won't yet take advantage of
mmu notifiers, but this already allows to compile a KVM external module
against a kernel with mmu notifiers enabled and from the next pull from
kvm.git we'll start using them. And GRU/XPMEM will also be able to
continue the development by enabling KVM=m in their config, until they
submit all GRU/XPMEM GPLv2 code to the mainline kernel. Then they can
also enable MMU_NOTIFIERS in the same way KVM does it (even if KVM=n).
This guarantees nobody selects MMU_NOTIFIER=y if KVM and GRU and XPMEM
are all =n.
The mmu_notifier_register call can fail because mm_take_all_locks may be
interrupted by a signal and return -EINTR. Because mmu_notifier_reigster
is used when a driver startup, a failure can be gracefully handled. Here
an example of the change applied to kvm to register the mmu notifiers.
Usually when a driver startups other allocations are required anyway and
-ENOMEM failure paths exists already.
struct kvm *kvm_arch_create_vm(void)
{
struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL);
+ int err;
if (!kvm)
return ERR_PTR(-ENOMEM);
INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
+ kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops;
+ err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm);
+ if (err) {
+ kfree(kvm);
+ return ERR_PTR(err);
+ }
+
return kvm;
}
mmu_notifier_unregister returns void and it's reliable.
The patch also adds a few needed but missing includes that would prevent
kernel to compile after these changes on non-x86 archs (x86 didn't need
them by luck).
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: fix mm/filemap_xip.c build]
[akpm@linux-foundation.org: fix mm/mmu_notifier.c build]
Signed-off-by: Andrea Arcangeli <andrea@qumranet.com>
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Jack Steiner <steiner@sgi.com>
Cc: Robin Holt <holt@sgi.com>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Kanoj Sarcar <kanojsarcar@yahoo.com>
Cc: Roland Dreier <rdreier@cisco.com>
Cc: Steve Wise <swise@opengridcomputing.com>
Cc: Avi Kivity <avi@qumranet.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Anthony Liguori <aliguori@us.ibm.com>
Cc: Chris Wright <chrisw@redhat.com>
Cc: Marcelo Tosatti <marcelo@kvack.org>
Cc: Eric Dumazet <dada1@cosmosbay.com>
Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
Cc: Izik Eidus <izike@qumranet.com>
Cc: Anthony Liguori <aliguori@us.ibm.com>
Cc: Rik van Riel <riel@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-29 07:46:29 +09:00
|
|
|
|
|
|
|
|
config MMU_NOTIFIER
|
|
|
|
|
bool
|
2014-12-06 01:24:45 +09:00
|
|
|
select SRCU
|
2009-05-07 08:03:05 +09:00
|
|
|
|
2009-09-22 09:01:57 +09:00
|
|
|
config KSM
|
|
|
|
|
bool "Enable KSM for page merging"
|
|
|
|
|
depends on MMU
|
2018-12-28 17:34:05 +09:00
|
|
|
select XXHASH
|
2009-09-22 09:01:57 +09:00
|
|
|
help
|
|
|
|
|
Enable Kernel Samepage Merging: KSM periodically scans those areas
|
|
|
|
|
of an application's address space that an app has advised may be
|
|
|
|
|
mergeable. When it finds pages of identical content, it replaces
|
2009-12-15 10:59:34 +09:00
|
|
|
the many instances by a single page with that content, so
|
2009-09-22 09:01:57 +09:00
|
|
|
saving memory until one or another app needs to modify the content.
|
|
|
|
|
Recommended for use with KVM, or with other duplicative applications.
|
2018-03-22 04:22:47 +09:00
|
|
|
See Documentation/vm/ksm.rst for more information: KSM is inactive
|
2009-10-08 08:32:22 +09:00
|
|
|
until a program has madvised that an area is MADV_MERGEABLE, and
|
|
|
|
|
root has set /sys/kernel/mm/ksm/run to 1 (if CONFIG_SYSFS is set).
|
2009-09-22 09:01:57 +09:00
|
|
|
|
2009-06-04 05:04:31 +09:00
|
|
|
config DEFAULT_MMAP_MIN_ADDR
|
|
|
|
|
int "Low address space to protect from user allocation"
|
2009-12-16 04:27:45 +09:00
|
|
|
depends on MMU
|
2009-06-04 05:04:31 +09:00
|
|
|
default 4096
|
|
|
|
|
help
|
|
|
|
|
This is the portion of low virtual memory which should be protected
|
|
|
|
|
from userspace allocation. Keeping a user from writing to low pages
|
|
|
|
|
can help reduce the impact of kernel NULL pointer bugs.
|
|
|
|
|
|
|
|
|
|
For most ia64, ppc64 and x86 users with lots of address space
|
|
|
|
|
a value of 65536 is reasonable and should cause no problems.
|
|
|
|
|
On arm and other archs it should not be higher than 32768.
|
2009-08-01 01:54:11 +09:00
|
|
|
Programs which use vm86 functionality or have some need to map
|
|
|
|
|
this low address space will need CAP_SYS_RAWIO or disable this
|
|
|
|
|
protection by setting the value to 0.
|
2009-06-04 05:04:31 +09:00
|
|
|
|
|
|
|
|
This value can be changed after boot using the
|
|
|
|
|
/proc/sys/vm/mmap_min_addr tunable.
|
|
|
|
|
|
2009-09-27 01:35:07 +09:00
|
|
|
config ARCH_SUPPORTS_MEMORY_FAILURE
|
|
|
|
|
bool
|
2009-06-04 05:04:31 +09:00
|
|
|
|
2009-09-16 18:50:15 +09:00
|
|
|
config MEMORY_FAILURE
|
|
|
|
|
depends on MMU
|
2009-09-27 01:35:07 +09:00
|
|
|
depends on ARCH_SUPPORTS_MEMORY_FAILURE
|
2009-09-16 18:50:15 +09:00
|
|
|
bool "Enable recovery from hardware memory errors"
|
2012-08-01 08:43:50 +09:00
|
|
|
select MEMORY_ISOLATION
|
2015-06-25 08:57:36 +09:00
|
|
|
select RAS
|
2009-09-16 18:50:15 +09:00
|
|
|
help
|
|
|
|
|
Enables code to recover from some memory failures on systems
|
|
|
|
|
with MCA recovery. This allows a system to continue running
|
|
|
|
|
even when some of its memory has uncorrected errors. This requires
|
|
|
|
|
special hardware support and typically ECC memory.
|
|
|
|
|
|
2009-09-16 18:50:17 +09:00
|
|
|
config HWPOISON_INJECT
|
2009-12-16 20:20:00 +09:00
|
|
|
tristate "HWPoison pages injector"
|
2009-12-22 03:56:42 +09:00
|
|
|
depends on MEMORY_FAILURE && DEBUG_KERNEL && PROC_FS
|
2009-12-16 20:19:59 +09:00
|
|
|
select PROC_PAGE_MONITOR
|
2009-09-16 18:50:17 +09:00
|
|
|
|
2009-05-07 08:03:05 +09:00
|
|
|
config NOMMU_INITIAL_TRIM_EXCESS
|
|
|
|
|
int "Turn on mmap() excess space trimming before booting"
|
|
|
|
|
depends on !MMU
|
|
|
|
|
default 1
|
|
|
|
|
help
|
|
|
|
|
The NOMMU mmap() frequently needs to allocate large contiguous chunks
|
|
|
|
|
of memory on which to store mappings, but it can only ask the system
|
|
|
|
|
allocator for chunks in 2^N*PAGE_SIZE amounts - which is frequently
|
|
|
|
|
more than it requires. To deal with this, mmap() is able to trim off
|
|
|
|
|
the excess and return it to the allocator.
|
|
|
|
|
|
|
|
|
|
If trimming is enabled, the excess is trimmed off and returned to the
|
|
|
|
|
system allocator, which can cause extra fragmentation, particularly
|
|
|
|
|
if there are a lot of transient processes.
|
|
|
|
|
|
|
|
|
|
If trimming is disabled, the excess is kept, but not used, which for
|
|
|
|
|
long-term mappings means that the space is wasted.
|
|
|
|
|
|
|
|
|
|
Trimming can be dynamically controlled through a sysctl option
|
|
|
|
|
(/proc/sys/vm/nr_trim_pages) which specifies the minimum number of
|
|
|
|
|
excess pages there must be before trimming should occur, or zero if
|
|
|
|
|
no trimming is to occur.
|
|
|
|
|
|
|
|
|
|
This option specifies the initial value of this option. The default
|
|
|
|
|
of 1 says that all excess pages should be trimmed.
|
|
|
|
|
|
|
|
|
|
See Documentation/nommu-mmap.txt for more information.
|
2010-09-04 01:22:48 +09:00
|
|
|
|
2011-01-14 08:46:39 +09:00
|
|
|
config TRANSPARENT_HUGEPAGE
|
2011-01-14 08:47:07 +09:00
|
|
|
bool "Transparent Hugepage Support"
|
2012-10-09 08:30:04 +09:00
|
|
|
depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE
|
2011-01-14 08:47:07 +09:00
|
|
|
select COMPACTION
|
2018-09-23 05:14:30 +09:00
|
|
|
select XARRAY_MULTI
|
2011-01-14 08:46:39 +09:00
|
|
|
help
|
|
|
|
|
Transparent Hugepages allows the kernel to use huge pages and
|
|
|
|
|
huge tlb transparently to the applications whenever possible.
|
|
|
|
|
This feature can improve computing performance to certain
|
|
|
|
|
applications by speeding up page faults during memory
|
|
|
|
|
allocation, by reducing the number of tlb misses and by speeding
|
|
|
|
|
up the pagetable walking.
|
|
|
|
|
|
|
|
|
|
If memory constrained on embedded, you may want to say N.
|
|
|
|
|
|
2011-01-14 08:47:07 +09:00
|
|
|
choice
|
|
|
|
|
prompt "Transparent Hugepage Support sysfs defaults"
|
|
|
|
|
depends on TRANSPARENT_HUGEPAGE
|
|
|
|
|
default TRANSPARENT_HUGEPAGE_ALWAYS
|
|
|
|
|
help
|
|
|
|
|
Selects the sysfs defaults for Transparent Hugepage Support.
|
|
|
|
|
|
|
|
|
|
config TRANSPARENT_HUGEPAGE_ALWAYS
|
|
|
|
|
bool "always"
|
|
|
|
|
help
|
|
|
|
|
Enabling Transparent Hugepage always, can increase the
|
|
|
|
|
memory footprint of applications without a guaranteed
|
|
|
|
|
benefit but it will work automatically for all applications.
|
|
|
|
|
|
|
|
|
|
config TRANSPARENT_HUGEPAGE_MADVISE
|
|
|
|
|
bool "madvise"
|
|
|
|
|
help
|
|
|
|
|
Enabling Transparent Hugepage madvise, will only provide a
|
|
|
|
|
performance improvement benefit to the applications using
|
|
|
|
|
madvise(MADV_HUGEPAGE) but it won't risk to increase the
|
|
|
|
|
memory footprint of applications without a guaranteed
|
|
|
|
|
benefit.
|
|
|
|
|
endchoice
|
|
|
|
|
|
mm, THP, swap: delay splitting THP during swap out
Patch series "THP swap: Delay splitting THP during swapping out", v11.
This patchset is to optimize the performance of Transparent Huge Page
(THP) swap.
Recently, the performance of the storage devices improved so fast that
we cannot saturate the disk bandwidth with single logical CPU when do
page swap out even on a high-end server machine. Because the
performance of the storage device improved faster than that of single
logical CPU. And it seems that the trend will not change in the near
future. On the other hand, the THP becomes more and more popular
because of increased memory size. So it becomes necessary to optimize
THP swap performance.
The advantages of the THP swap support include:
- Batch the swap operations for the THP to reduce lock
acquiring/releasing, including allocating/freeing the swap space,
adding/deleting to/from the swap cache, and writing/reading the swap
space, etc. This will help improve the performance of the THP swap.
- The THP swap space read/write will be 2M sequential IO. It is
particularly helpful for the swap read, which are usually 4k random
IO. This will improve the performance of the THP swap too.
- It will help the memory fragmentation, especially when the THP is
heavily used by the applications. The 2M continuous pages will be
free up after THP swapping out.
- It will improve the THP utilization on the system with the swap
turned on. Because the speed for khugepaged to collapse the normal
pages into the THP is quite slow. After the THP is split during the
swapping out, it will take quite long time for the normal pages to
collapse back into the THP after being swapped in. The high THP
utilization helps the efficiency of the page based memory management
too.
There are some concerns regarding THP swap in, mainly because possible
enlarged read/write IO size (for swap in/out) may put more overhead on
the storage device. To deal with that, the THP swap in should be turned
on only when necessary. For example, it can be selected via
"always/never/madvise" logic, to be turned on globally, turned off
globally, or turned on only for VMA with MADV_HUGEPAGE, etc.
This patchset is the first step for the THP swap support. The plan is
to delay splitting THP step by step, finally avoid splitting THP during
the THP swapping out and swap out/in the THP as a whole.
As the first step, in this patchset, the splitting huge page is delayed
from almost the first step of swapping out to after allocating the swap
space for the THP and adding the THP into the swap cache. This will
reduce lock acquiring/releasing for the locks used for the swap cache
management.
With the patchset, the swap out throughput improves 15.5% (from about
3.73GB/s to about 4.31GB/s) in the vm-scalability swap-w-seq test case
with 8 processes. The test is done on a Xeon E5 v3 system. The swap
device used is a RAM simulated PMEM (persistent memory) device. To test
the sequential swapping out, the test case creates 8 processes, which
sequentially allocate and write to the anonymous pages until the RAM and
part of the swap device is used up.
This patch (of 5):
In this patch, splitting huge page is delayed from almost the first step
of swapping out to after allocating the swap space for the THP
(Transparent Huge Page) and adding the THP into the swap cache. This
will batch the corresponding operation, thus improve THP swap out
throughput.
This is the first step for the THP swap optimization. The plan is to
delay splitting the THP step by step and avoid splitting the THP
finally.
In this patch, one swap cluster is used to hold the contents of each THP
swapped out. So, the size of the swap cluster is changed to that of the
THP (Transparent Huge Page) on x86_64 architecture (512). For other
architectures which want such THP swap optimization,
ARCH_USES_THP_SWAP_CLUSTER needs to be selected in the Kconfig file for
the architecture. In effect, this will enlarge swap cluster size by 2
times on x86_64. Which may make it harder to find a free cluster when
the swap space becomes fragmented. So that, this may reduce the
continuous swap space allocation and sequential write in theory. The
performance test in 0day shows no regressions caused by this.
In the future of THP swap optimization, some information of the swapped
out THP (such as compound map count) will be recorded in the
swap_cluster_info data structure.
The mem cgroup swap accounting functions are enhanced to support charge
or uncharge a swap cluster backing a THP as a whole.
The swap cluster allocate/free functions are added to allocate/free a
swap cluster for a THP. A fair simple algorithm is used for swap
cluster allocation, that is, only the first swap device in priority list
will be tried to allocate the swap cluster. The function will fail if
the trying is not successful, and the caller will fallback to allocate a
single swap slot instead. This works good enough for normal cases. If
the difference of the number of the free swap clusters among multiple
swap devices is significant, it is possible that some THPs are split
earlier than necessary. For example, this could be caused by big size
difference among multiple swap devices.
The swap cache functions is enhanced to support add/delete THP to/from
the swap cache as a set of (HPAGE_PMD_NR) sub-pages. This may be
enhanced in the future with multi-order radix tree. But because we will
split the THP soon during swapping out, that optimization doesn't make
much sense for this first step.
The THP splitting functions are enhanced to support to split THP in swap
cache during swapping out. The page lock will be held during allocating
the swap cluster, adding the THP into the swap cache and splitting the
THP. So in the code path other than swapping out, if the THP need to be
split, the PageSwapCache(THP) will be always false.
The swap cluster is only available for SSD, so the THP swap optimization
in this patchset has no effect for HDD.
[ying.huang@intel.com: fix two issues in THP optimize patch]
Link: http://lkml.kernel.org/r/87k25ed8zo.fsf@yhuang-dev.intel.com
[hannes@cmpxchg.org: extensive cleanups and simplifications, reduce code size]
Link: http://lkml.kernel.org/r/20170515112522.32457-2-ying.huang@intel.com
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Suggested-by: Andrew Morton <akpm@linux-foundation.org> [for config option]
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> [for changes in huge_memory.c and huge_mm.h]
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Ebru Akagunduz <ebru.akagunduz@gmail.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Shaohua Li <shli@kernel.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-07-07 07:37:18 +09:00
|
|
|
config ARCH_WANTS_THP_SWAP
|
|
|
|
|
def_bool n
|
|
|
|
|
|
|
|
|
|
config THP_SWAP
|
|
|
|
|
def_bool y
|
2018-08-18 07:49:41 +09:00
|
|
|
depends on TRANSPARENT_HUGEPAGE && ARCH_WANTS_THP_SWAP && SWAP
|
mm, THP, swap: delay splitting THP during swap out
Patch series "THP swap: Delay splitting THP during swapping out", v11.
This patchset is to optimize the performance of Transparent Huge Page
(THP) swap.
Recently, the performance of the storage devices improved so fast that
we cannot saturate the disk bandwidth with single logical CPU when do
page swap out even on a high-end server machine. Because the
performance of the storage device improved faster than that of single
logical CPU. And it seems that the trend will not change in the near
future. On the other hand, the THP becomes more and more popular
because of increased memory size. So it becomes necessary to optimize
THP swap performance.
The advantages of the THP swap support include:
- Batch the swap operations for the THP to reduce lock
acquiring/releasing, including allocating/freeing the swap space,
adding/deleting to/from the swap cache, and writing/reading the swap
space, etc. This will help improve the performance of the THP swap.
- The THP swap space read/write will be 2M sequential IO. It is
particularly helpful for the swap read, which are usually 4k random
IO. This will improve the performance of the THP swap too.
- It will help the memory fragmentation, especially when the THP is
heavily used by the applications. The 2M continuous pages will be
free up after THP swapping out.
- It will improve the THP utilization on the system with the swap
turned on. Because the speed for khugepaged to collapse the normal
pages into the THP is quite slow. After the THP is split during the
swapping out, it will take quite long time for the normal pages to
collapse back into the THP after being swapped in. The high THP
utilization helps the efficiency of the page based memory management
too.
There are some concerns regarding THP swap in, mainly because possible
enlarged read/write IO size (for swap in/out) may put more overhead on
the storage device. To deal with that, the THP swap in should be turned
on only when necessary. For example, it can be selected via
"always/never/madvise" logic, to be turned on globally, turned off
globally, or turned on only for VMA with MADV_HUGEPAGE, etc.
This patchset is the first step for the THP swap support. The plan is
to delay splitting THP step by step, finally avoid splitting THP during
the THP swapping out and swap out/in the THP as a whole.
As the first step, in this patchset, the splitting huge page is delayed
from almost the first step of swapping out to after allocating the swap
space for the THP and adding the THP into the swap cache. This will
reduce lock acquiring/releasing for the locks used for the swap cache
management.
With the patchset, the swap out throughput improves 15.5% (from about
3.73GB/s to about 4.31GB/s) in the vm-scalability swap-w-seq test case
with 8 processes. The test is done on a Xeon E5 v3 system. The swap
device used is a RAM simulated PMEM (persistent memory) device. To test
the sequential swapping out, the test case creates 8 processes, which
sequentially allocate and write to the anonymous pages until the RAM and
part of the swap device is used up.
This patch (of 5):
In this patch, splitting huge page is delayed from almost the first step
of swapping out to after allocating the swap space for the THP
(Transparent Huge Page) and adding the THP into the swap cache. This
will batch the corresponding operation, thus improve THP swap out
throughput.
This is the first step for the THP swap optimization. The plan is to
delay splitting the THP step by step and avoid splitting the THP
finally.
In this patch, one swap cluster is used to hold the contents of each THP
swapped out. So, the size of the swap cluster is changed to that of the
THP (Transparent Huge Page) on x86_64 architecture (512). For other
architectures which want such THP swap optimization,
ARCH_USES_THP_SWAP_CLUSTER needs to be selected in the Kconfig file for
the architecture. In effect, this will enlarge swap cluster size by 2
times on x86_64. Which may make it harder to find a free cluster when
the swap space becomes fragmented. So that, this may reduce the
continuous swap space allocation and sequential write in theory. The
performance test in 0day shows no regressions caused by this.
In the future of THP swap optimization, some information of the swapped
out THP (such as compound map count) will be recorded in the
swap_cluster_info data structure.
The mem cgroup swap accounting functions are enhanced to support charge
or uncharge a swap cluster backing a THP as a whole.
The swap cluster allocate/free functions are added to allocate/free a
swap cluster for a THP. A fair simple algorithm is used for swap
cluster allocation, that is, only the first swap device in priority list
will be tried to allocate the swap cluster. The function will fail if
the trying is not successful, and the caller will fallback to allocate a
single swap slot instead. This works good enough for normal cases. If
the difference of the number of the free swap clusters among multiple
swap devices is significant, it is possible that some THPs are split
earlier than necessary. For example, this could be caused by big size
difference among multiple swap devices.
The swap cache functions is enhanced to support add/delete THP to/from
the swap cache as a set of (HPAGE_PMD_NR) sub-pages. This may be
enhanced in the future with multi-order radix tree. But because we will
split the THP soon during swapping out, that optimization doesn't make
much sense for this first step.
The THP splitting functions are enhanced to support to split THP in swap
cache during swapping out. The page lock will be held during allocating
the swap cluster, adding the THP into the swap cache and splitting the
THP. So in the code path other than swapping out, if the THP need to be
split, the PageSwapCache(THP) will be always false.
The swap cluster is only available for SSD, so the THP swap optimization
in this patchset has no effect for HDD.
[ying.huang@intel.com: fix two issues in THP optimize patch]
Link: http://lkml.kernel.org/r/87k25ed8zo.fsf@yhuang-dev.intel.com
[hannes@cmpxchg.org: extensive cleanups and simplifications, reduce code size]
Link: http://lkml.kernel.org/r/20170515112522.32457-2-ying.huang@intel.com
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Suggested-by: Andrew Morton <akpm@linux-foundation.org> [for config option]
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> [for changes in huge_memory.c and huge_mm.h]
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Ebru Akagunduz <ebru.akagunduz@gmail.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Shaohua Li <shli@kernel.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-07-07 07:37:18 +09:00
|
|
|
help
|
|
|
|
|
Swap transparent huge pages in one piece, without splitting.
|
2018-08-18 07:49:41 +09:00
|
|
|
XXX: For now, swap cluster backing transparent huge page
|
|
|
|
|
will be split after swapout.
|
mm, THP, swap: delay splitting THP during swap out
Patch series "THP swap: Delay splitting THP during swapping out", v11.
This patchset is to optimize the performance of Transparent Huge Page
(THP) swap.
Recently, the performance of the storage devices improved so fast that
we cannot saturate the disk bandwidth with single logical CPU when do
page swap out even on a high-end server machine. Because the
performance of the storage device improved faster than that of single
logical CPU. And it seems that the trend will not change in the near
future. On the other hand, the THP becomes more and more popular
because of increased memory size. So it becomes necessary to optimize
THP swap performance.
The advantages of the THP swap support include:
- Batch the swap operations for the THP to reduce lock
acquiring/releasing, including allocating/freeing the swap space,
adding/deleting to/from the swap cache, and writing/reading the swap
space, etc. This will help improve the performance of the THP swap.
- The THP swap space read/write will be 2M sequential IO. It is
particularly helpful for the swap read, which are usually 4k random
IO. This will improve the performance of the THP swap too.
- It will help the memory fragmentation, especially when the THP is
heavily used by the applications. The 2M continuous pages will be
free up after THP swapping out.
- It will improve the THP utilization on the system with the swap
turned on. Because the speed for khugepaged to collapse the normal
pages into the THP is quite slow. After the THP is split during the
swapping out, it will take quite long time for the normal pages to
collapse back into the THP after being swapped in. The high THP
utilization helps the efficiency of the page based memory management
too.
There are some concerns regarding THP swap in, mainly because possible
enlarged read/write IO size (for swap in/out) may put more overhead on
the storage device. To deal with that, the THP swap in should be turned
on only when necessary. For example, it can be selected via
"always/never/madvise" logic, to be turned on globally, turned off
globally, or turned on only for VMA with MADV_HUGEPAGE, etc.
This patchset is the first step for the THP swap support. The plan is
to delay splitting THP step by step, finally avoid splitting THP during
the THP swapping out and swap out/in the THP as a whole.
As the first step, in this patchset, the splitting huge page is delayed
from almost the first step of swapping out to after allocating the swap
space for the THP and adding the THP into the swap cache. This will
reduce lock acquiring/releasing for the locks used for the swap cache
management.
With the patchset, the swap out throughput improves 15.5% (from about
3.73GB/s to about 4.31GB/s) in the vm-scalability swap-w-seq test case
with 8 processes. The test is done on a Xeon E5 v3 system. The swap
device used is a RAM simulated PMEM (persistent memory) device. To test
the sequential swapping out, the test case creates 8 processes, which
sequentially allocate and write to the anonymous pages until the RAM and
part of the swap device is used up.
This patch (of 5):
In this patch, splitting huge page is delayed from almost the first step
of swapping out to after allocating the swap space for the THP
(Transparent Huge Page) and adding the THP into the swap cache. This
will batch the corresponding operation, thus improve THP swap out
throughput.
This is the first step for the THP swap optimization. The plan is to
delay splitting the THP step by step and avoid splitting the THP
finally.
In this patch, one swap cluster is used to hold the contents of each THP
swapped out. So, the size of the swap cluster is changed to that of the
THP (Transparent Huge Page) on x86_64 architecture (512). For other
architectures which want such THP swap optimization,
ARCH_USES_THP_SWAP_CLUSTER needs to be selected in the Kconfig file for
the architecture. In effect, this will enlarge swap cluster size by 2
times on x86_64. Which may make it harder to find a free cluster when
the swap space becomes fragmented. So that, this may reduce the
continuous swap space allocation and sequential write in theory. The
performance test in 0day shows no regressions caused by this.
In the future of THP swap optimization, some information of the swapped
out THP (such as compound map count) will be recorded in the
swap_cluster_info data structure.
The mem cgroup swap accounting functions are enhanced to support charge
or uncharge a swap cluster backing a THP as a whole.
The swap cluster allocate/free functions are added to allocate/free a
swap cluster for a THP. A fair simple algorithm is used for swap
cluster allocation, that is, only the first swap device in priority list
will be tried to allocate the swap cluster. The function will fail if
the trying is not successful, and the caller will fallback to allocate a
single swap slot instead. This works good enough for normal cases. If
the difference of the number of the free swap clusters among multiple
swap devices is significant, it is possible that some THPs are split
earlier than necessary. For example, this could be caused by big size
difference among multiple swap devices.
The swap cache functions is enhanced to support add/delete THP to/from
the swap cache as a set of (HPAGE_PMD_NR) sub-pages. This may be
enhanced in the future with multi-order radix tree. But because we will
split the THP soon during swapping out, that optimization doesn't make
much sense for this first step.
The THP splitting functions are enhanced to support to split THP in swap
cache during swapping out. The page lock will be held during allocating
the swap cluster, adding the THP into the swap cache and splitting the
THP. So in the code path other than swapping out, if the THP need to be
split, the PageSwapCache(THP) will be always false.
The swap cluster is only available for SSD, so the THP swap optimization
in this patchset has no effect for HDD.
[ying.huang@intel.com: fix two issues in THP optimize patch]
Link: http://lkml.kernel.org/r/87k25ed8zo.fsf@yhuang-dev.intel.com
[hannes@cmpxchg.org: extensive cleanups and simplifications, reduce code size]
Link: http://lkml.kernel.org/r/20170515112522.32457-2-ying.huang@intel.com
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Suggested-by: Andrew Morton <akpm@linux-foundation.org> [for config option]
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> [for changes in huge_memory.c and huge_mm.h]
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Ebru Akagunduz <ebru.akagunduz@gmail.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Shaohua Li <shli@kernel.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-07-07 07:37:18 +09:00
|
|
|
|
|
|
|
|
For selection by architectures with reasonable THP sizes.
|
|
|
|
|
|
2016-07-27 07:26:35 +09:00
|
|
|
config TRANSPARENT_HUGE_PAGECACHE
|
|
|
|
|
def_bool y
|
2016-12-13 09:44:32 +09:00
|
|
|
depends on TRANSPARENT_HUGEPAGE
|
2016-07-27 07:26:35 +09:00
|
|
|
|
2010-09-04 01:22:48 +09:00
|
|
|
#
|
|
|
|
|
# UP and nommu archs use km based percpu allocator
|
|
|
|
|
#
|
|
|
|
|
config NEED_PER_CPU_KM
|
|
|
|
|
depends on !SMP
|
|
|
|
|
bool
|
|
|
|
|
default y
|
2011-05-27 01:01:36 +09:00
|
|
|
|
|
|
|
|
config CLEANCACHE
|
|
|
|
|
bool "Enable cleancache driver to cache clean pages if tmem is present"
|
|
|
|
|
help
|
|
|
|
|
Cleancache can be thought of as a page-granularity victim cache
|
|
|
|
|
for clean pages that the kernel's pageframe replacement algorithm
|
|
|
|
|
(PFRA) would like to keep around, but can't since there isn't enough
|
|
|
|
|
memory. So when the PFRA "evicts" a page, it first attempts to use
|
2011-06-10 12:57:26 +09:00
|
|
|
cleancache code to put the data contained in that page into
|
2011-05-27 01:01:36 +09:00
|
|
|
"transcendent memory", memory that is not directly accessible or
|
|
|
|
|
addressable by the kernel and is of unknown and possibly
|
|
|
|
|
time-varying size. And when a cleancache-enabled
|
|
|
|
|
filesystem wishes to access a page in a file on disk, it first
|
|
|
|
|
checks cleancache to see if it already contains it; if it does,
|
|
|
|
|
the page is copied into the kernel and a disk access is avoided.
|
|
|
|
|
When a transcendent memory driver is available (such as zcache or
|
|
|
|
|
Xen transcendent memory), a significant I/O reduction
|
|
|
|
|
may be achieved. When none is available, all cleancache calls
|
|
|
|
|
are reduced to a single pointer-compare-against-NULL resulting
|
|
|
|
|
in a negligible performance hit.
|
|
|
|
|
|
|
|
|
|
If unsure, say Y to enable cleancache
|
2012-04-10 08:10:34 +09:00
|
|
|
|
|
|
|
|
config FRONTSWAP
|
|
|
|
|
bool "Enable frontswap to cache swap pages if tmem is present"
|
|
|
|
|
depends on SWAP
|
|
|
|
|
help
|
|
|
|
|
Frontswap is so named because it can be thought of as the opposite
|
|
|
|
|
of a "backing" store for a swap device. The data is stored into
|
|
|
|
|
"transcendent memory", memory that is not directly accessible or
|
|
|
|
|
addressable by the kernel and is of unknown and possibly
|
|
|
|
|
time-varying size. When space in transcendent memory is available,
|
|
|
|
|
a significant swap I/O reduction may be achieved. When none is
|
|
|
|
|
available, all frontswap calls are reduced to a single pointer-
|
|
|
|
|
compare-against-NULL resulting in a negligible performance hit
|
|
|
|
|
and swap data is stored as normal on the matching swap device.
|
|
|
|
|
|
|
|
|
|
If unsure, say Y to enable frontswap.
|
2013-07-02 14:45:15 +09:00
|
|
|
|
|
|
|
|
config CMA
|
|
|
|
|
bool "Contiguous Memory Allocator"
|
2018-10-31 07:07:44 +09:00
|
|
|
depends on MMU
|
2013-07-02 14:45:15 +09:00
|
|
|
select MIGRATION
|
|
|
|
|
select MEMORY_ISOLATION
|
|
|
|
|
help
|
|
|
|
|
This enables the Contiguous Memory Allocator which allows other
|
|
|
|
|
subsystems to allocate big physically-contiguous blocks of memory.
|
|
|
|
|
CMA reserves a region of memory and allows only movable pages to
|
|
|
|
|
be allocated from it. This way, the kernel can use the memory for
|
|
|
|
|
pagecache and when a subsystem requests for contiguous area, the
|
|
|
|
|
allocated pages are migrated away to serve the contiguous request.
|
|
|
|
|
|
|
|
|
|
If unsure, say "n".
|
|
|
|
|
|
|
|
|
|
config CMA_DEBUG
|
|
|
|
|
bool "CMA debug messages (DEVELOPMENT)"
|
|
|
|
|
depends on DEBUG_KERNEL && CMA
|
|
|
|
|
help
|
|
|
|
|
Turns on debug messages in CMA. This produces KERN_DEBUG
|
|
|
|
|
messages for every CMA call as well as various messages while
|
|
|
|
|
processing calls such as dma_alloc_from_contiguous().
|
|
|
|
|
This option does not affect warning and error messages.
|
2013-08-29 07:41:59 +09:00
|
|
|
|
2015-04-15 07:44:57 +09:00
|
|
|
config CMA_DEBUGFS
|
|
|
|
|
bool "CMA debugfs interface"
|
|
|
|
|
depends on CMA && DEBUG_FS
|
|
|
|
|
help
|
|
|
|
|
Turns on the DebugFS interface for CMA.
|
|
|
|
|
|
2020-12-15 20:27:06 +09:00
|
|
|
config CMA_ALLOW_WRITE_DEBUGFS
|
|
|
|
|
depends on CMA_DEBUGFS
|
|
|
|
|
bool "Allow CMA debugfs write"
|
|
|
|
|
help
|
|
|
|
|
Say 'y' here to allow the CMA debugfs write.
|
|
|
|
|
CMA debugfs write could be risky as it allows
|
|
|
|
|
cma allocation, so it is not recommended to
|
|
|
|
|
enable this option on any production device.
|
|
|
|
|
|
2014-08-07 08:05:25 +09:00
|
|
|
config CMA_AREAS
|
|
|
|
|
int "Maximum count of the CMA areas"
|
|
|
|
|
depends on CMA
|
|
|
|
|
default 7
|
|
|
|
|
help
|
|
|
|
|
CMA allows to create CMA areas for particular purpose, mainly,
|
|
|
|
|
used as device private area. This parameter sets the maximum
|
|
|
|
|
number of CMA area in the system.
|
|
|
|
|
|
|
|
|
|
If unsure, leave the default value "7".
|
|
|
|
|
|
2020-04-09 14:08:27 +09:00
|
|
|
config LIMIT_MOVABLE_ZONE_ALLOC
|
|
|
|
|
bool "Limit types of allocations from Movable zone"
|
2020-09-07 22:29:49 +09:00
|
|
|
depends on QGKI && MEMORY_HOTPLUG_MOVABLE_NODE
|
|
|
|
|
default y
|
2020-04-09 14:08:27 +09:00
|
|
|
help
|
|
|
|
|
Ensure that only allocations which include __GFP_OFFLINABLE can be
|
|
|
|
|
satisfied from zone Movable. This restriction helps reduce the
|
|
|
|
|
likelihood of a page in the movable zone being pinned which would
|
|
|
|
|
prevent memory from being offlined.
|
|
|
|
|
|
2014-08-07 08:08:36 +09:00
|
|
|
config MEM_SOFT_DIRTY
|
|
|
|
|
bool "Track memory changes"
|
|
|
|
|
depends on CHECKPOINT_RESTORE && HAVE_ARCH_SOFT_DIRTY && PROC_FS
|
|
|
|
|
select PROC_PAGE_MONITOR
|
2013-07-11 08:04:55 +09:00
|
|
|
help
|
2014-08-07 08:08:36 +09:00
|
|
|
This option enables memory changes tracking by introducing a
|
|
|
|
|
soft-dirty bit on pte-s. This bit it set when someone writes
|
|
|
|
|
into a page just as regular dirty bit, but unlike the latter
|
|
|
|
|
it can be cleared by hands.
|
|
|
|
|
|
2018-04-18 17:07:49 +09:00
|
|
|
See Documentation/admin-guide/mm/soft-dirty.rst for more details.
|
2013-07-11 08:04:55 +09:00
|
|
|
|
2013-07-11 08:05:03 +09:00
|
|
|
config ZSWAP
|
|
|
|
|
bool "Compressed cache for swap pages (EXPERIMENTAL)"
|
|
|
|
|
depends on FRONTSWAP && CRYPTO=y
|
|
|
|
|
select CRYPTO_LZO
|
2014-08-07 08:08:40 +09:00
|
|
|
select ZPOOL
|
2013-07-11 08:05:03 +09:00
|
|
|
help
|
|
|
|
|
A lightweight compressed cache for swap pages. It takes
|
|
|
|
|
pages that are in the process of being swapped out and attempts to
|
|
|
|
|
compress them into a dynamically allocated RAM-based memory pool.
|
|
|
|
|
This can result in a significant I/O reduction on swap device and,
|
|
|
|
|
in the case where decompressing from RAM is faster that swap device
|
|
|
|
|
reads, can also improve workload performance.
|
|
|
|
|
|
|
|
|
|
This is marked experimental because it is a new feature (as of
|
|
|
|
|
v3.11) that interacts heavily with memory reclaim. While these
|
|
|
|
|
interactions don't cause any known issues on simple memory setups,
|
|
|
|
|
they have not be fully explored on the large set of potential
|
|
|
|
|
configurations and workloads that exist.
|
|
|
|
|
|
2014-08-07 08:08:36 +09:00
|
|
|
config ZPOOL
|
|
|
|
|
tristate "Common API for compressed memory storage"
|
mm: soft-dirty bits for user memory changes tracking
The soft-dirty is a bit on a PTE which helps to track which pages a task
writes to. In order to do this tracking one should
1. Clear soft-dirty bits from PTEs ("echo 4 > /proc/PID/clear_refs)
2. Wait some time.
3. Read soft-dirty bits (55'th in /proc/PID/pagemap2 entries)
To do this tracking, the writable bit is cleared from PTEs when the
soft-dirty bit is. Thus, after this, when the task tries to modify a
page at some virtual address the #PF occurs and the kernel sets the
soft-dirty bit on the respective PTE.
Note, that although all the task's address space is marked as r/o after
the soft-dirty bits clear, the #PF-s that occur after that are processed
fast. This is so, since the pages are still mapped to physical memory,
and thus all the kernel does is finds this fact out and puts back
writable, dirty and soft-dirty bits on the PTE.
Another thing to note, is that when mremap moves PTEs they are marked
with soft-dirty as well, since from the user perspective mremap modifies
the virtual memory at mremap's new address.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Cc: Matt Mackall <mpm@selenic.com>
Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
Cc: Glauber Costa <glommer@parallels.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@gmail.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-07-04 07:01:20 +09:00
|
|
|
help
|
2014-08-07 08:08:36 +09:00
|
|
|
Compressed memory storage API. This allows using either zbud or
|
|
|
|
|
zsmalloc.
|
mm: soft-dirty bits for user memory changes tracking
The soft-dirty is a bit on a PTE which helps to track which pages a task
writes to. In order to do this tracking one should
1. Clear soft-dirty bits from PTEs ("echo 4 > /proc/PID/clear_refs)
2. Wait some time.
3. Read soft-dirty bits (55'th in /proc/PID/pagemap2 entries)
To do this tracking, the writable bit is cleared from PTEs when the
soft-dirty bit is. Thus, after this, when the task tries to modify a
page at some virtual address the #PF occurs and the kernel sets the
soft-dirty bit on the respective PTE.
Note, that although all the task's address space is marked as r/o after
the soft-dirty bits clear, the #PF-s that occur after that are processed
fast. This is so, since the pages are still mapped to physical memory,
and thus all the kernel does is finds this fact out and puts back
writable, dirty and soft-dirty bits on the PTE.
Another thing to note, is that when mremap moves PTEs they are marked
with soft-dirty as well, since from the user perspective mremap modifies
the virtual memory at mremap's new address.
Signed-off-by: Pavel Emelyanov <xemul@parallels.com>
Cc: Matt Mackall <mpm@selenic.com>
Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
Cc: Glauber Costa <glommer@parallels.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@gmail.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-07-04 07:01:20 +09:00
|
|
|
|
2014-08-07 08:08:36 +09:00
|
|
|
config ZBUD
|
2016-05-21 08:58:30 +09:00
|
|
|
tristate "Low (Up to 2x) density storage for compressed pages"
|
2014-08-07 08:08:36 +09:00
|
|
|
help
|
|
|
|
|
A special purpose allocator for storing compressed pages.
|
|
|
|
|
It is designed to store up to two compressed pages per physical
|
|
|
|
|
page. While this design limits storage density, it has simple and
|
|
|
|
|
deterministic reclaim properties that make it preferable to a higher
|
|
|
|
|
density approach when reclaim will be used.
|
2014-01-31 08:45:50 +09:00
|
|
|
|
2016-05-21 08:58:30 +09:00
|
|
|
config Z3FOLD
|
|
|
|
|
tristate "Up to 3x density storage for compressed pages"
|
|
|
|
|
depends on ZPOOL
|
|
|
|
|
help
|
|
|
|
|
A special purpose allocator for storing compressed pages.
|
|
|
|
|
It is designed to store up to three compressed pages per physical
|
|
|
|
|
page. It is a ZBUD derivative so the simplicity and determinism are
|
|
|
|
|
still there.
|
|
|
|
|
|
2014-01-31 08:45:50 +09:00
|
|
|
config ZSMALLOC
|
2014-06-05 08:11:10 +09:00
|
|
|
tristate "Memory allocator for compressed pages"
|
2014-01-31 08:45:50 +09:00
|
|
|
depends on MMU
|
|
|
|
|
help
|
|
|
|
|
zsmalloc is a slab-based memory allocator designed to store
|
|
|
|
|
compressed RAM pages. zsmalloc uses virtual memory mapping
|
|
|
|
|
in order to reduce fragmentation. However, this results in a
|
|
|
|
|
non-standard allocator interface where a handle, not a pointer, is
|
|
|
|
|
returned by an alloc(). This handle must be mapped in order to
|
|
|
|
|
access the allocated space.
|
|
|
|
|
|
2015-02-13 08:00:54 +09:00
|
|
|
config ZSMALLOC_STAT
|
|
|
|
|
bool "Export zsmalloc statistics"
|
|
|
|
|
depends on ZSMALLOC
|
|
|
|
|
select DEBUG_FS
|
|
|
|
|
help
|
|
|
|
|
This option enables code in the zsmalloc to collect various
|
|
|
|
|
statistics about whats happening in zsmalloc and exports that
|
|
|
|
|
information to userspace via debugfs.
|
|
|
|
|
If unsure, say N.
|
|
|
|
|
|
2018-06-21 13:54:40 +09:00
|
|
|
config VMAP_LAZY_PURGING_FACTOR
|
|
|
|
|
int "multiplier to the size of purged vmap areas"
|
|
|
|
|
default "8" if ARM
|
|
|
|
|
default "32"
|
|
|
|
|
help
|
|
|
|
|
It is used as a multiplier to the max VA pages purged in a
|
|
|
|
|
single attempt. For 32-bit in order to reduce fragmentation
|
|
|
|
|
of vmalloc space, we decrease the default value to "8".
|
|
|
|
|
|
|
|
|
|
|
2014-04-08 07:39:48 +09:00
|
|
|
config GENERIC_EARLY_IOREMAP
|
|
|
|
|
bool
|
2014-05-01 06:26:02 +09:00
|
|
|
|
|
|
|
|
config MAX_STACK_SIZE_MB
|
|
|
|
|
int "Maximum user stack size for 32-bit processes (MB)"
|
|
|
|
|
default 80
|
|
|
|
|
range 8 2048
|
|
|
|
|
depends on STACK_GROWSUP && (!64BIT || COMPAT)
|
|
|
|
|
help
|
|
|
|
|
This is the maximum stack size in Megabytes in the VM layout of 32-bit
|
|
|
|
|
user processes when the stack grows upwards (currently only on parisc
|
2017-10-25 00:52:32 +09:00
|
|
|
arch). The stack will be located at the highest memory address minus
|
|
|
|
|
the given value, unless the RLIMIT_STACK hard limit is changed to a
|
|
|
|
|
smaller value in which case that is used.
|
2014-05-01 06:26:02 +09:00
|
|
|
|
|
|
|
|
A sane initial value is 80 MB.
|
2015-07-01 06:57:02 +09:00
|
|
|
|
2017-03-15 10:30:41 +09:00
|
|
|
config BALANCE_ANON_FILE_RECLAIM
|
|
|
|
|
bool "During reclaim treat anon and file backed pages equally"
|
|
|
|
|
depends on SWAP
|
|
|
|
|
help
|
|
|
|
|
When performing memory reclaim treat anonymous and file backed pages
|
|
|
|
|
equally.
|
|
|
|
|
Swapping anonymous pages out to memory can be efficient enough to justify
|
|
|
|
|
treating anonymous and file backed pages equally.
|
|
|
|
|
|
2015-07-01 06:57:02 +09:00
|
|
|
config DEFERRED_STRUCT_PAGE_INIT
|
2016-02-06 08:36:21 +09:00
|
|
|
bool "Defer initialisation of struct pages to kthreads"
|
mm: make DEFERRED_STRUCT_PAGE_INIT explicitly depend on SPARSEMEM
The deferred memory initialization relies on section definitions, e.g
PAGES_PER_SECTION, that are only available when CONFIG_SPARSEMEM=y on
most architectures.
Initially DEFERRED_STRUCT_PAGE_INIT depended on explicit
ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT configuration option, but since
the commit 2e3ca40f03bb13709df4 ("mm: relax deferred struct page
requirements") this requirement was relaxed and now it is possible to
enable DEFERRED_STRUCT_PAGE_INIT on architectures that support
DISCONTINGMEM and NO_BOOTMEM which causes build failures.
For instance, setting SMP=y and DEFERRED_STRUCT_PAGE_INIT=y on arc
causes the following build failure:
CC mm/page_alloc.o
mm/page_alloc.c: In function 'update_defer_init':
mm/page_alloc.c:321:14: error: 'PAGES_PER_SECTION'
undeclared (first use in this function); did you mean 'USEC_PER_SEC'?
(pfn & (PAGES_PER_SECTION - 1)) == 0) {
^~~~~~~~~~~~~~~~~
USEC_PER_SEC
mm/page_alloc.c:321:14: note: each undeclared identifier is reported only once for each function it appears in
In file included from include/linux/cache.h:5:0,
from include/linux/printk.h:9,
from include/linux/kernel.h:14,
from include/asm-generic/bug.h:18,
from arch/arc/include/asm/bug.h:32,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5,
from include/linux/mm.h:9,
from mm/page_alloc.c:18:
mm/page_alloc.c: In function 'deferred_grow_zone':
mm/page_alloc.c:1624:52: error: 'PAGES_PER_SECTION' undeclared (first use in this function); did you mean 'USEC_PER_SEC'?
unsigned long nr_pages_needed = ALIGN(1 << order, PAGES_PER_SECTION);
^
include/uapi/linux/kernel.h:11:47: note: in definition of macro '__ALIGN_KERNEL_MASK'
#define __ALIGN_KERNEL_MASK(x, mask) (((x) + (mask)) & ~(mask))
^~~~
include/linux/kernel.h:58:22: note: in expansion of macro '__ALIGN_KERNEL'
#define ALIGN(x, a) __ALIGN_KERNEL((x), (a))
^~~~~~~~~~~~~~
mm/page_alloc.c:1624:34: note: in expansion of macro 'ALIGN'
unsigned long nr_pages_needed = ALIGN(1 << order, PAGES_PER_SECTION);
^~~~~
In file included from include/asm-generic/bug.h:18:0,
from arch/arc/include/asm/bug.h:32,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5,
from include/linux/mm.h:9,
from mm/page_alloc.c:18:
mm/page_alloc.c: In function 'free_area_init_node':
mm/page_alloc.c:6379:50: error: 'PAGES_PER_SECTION' undeclared (first use in this function); did you mean 'USEC_PER_SEC'?
pgdat->static_init_pgcnt = min_t(unsigned long, PAGES_PER_SECTION,
^
include/linux/kernel.h:812:22: note: in definition of macro '__typecheck'
(!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
^
include/linux/kernel.h:836:24: note: in expansion of macro '__safe_cmp'
__builtin_choose_expr(__safe_cmp(x, y), \
^~~~~~~~~~
include/linux/kernel.h:904:27: note: in expansion of macro '__careful_cmp'
#define min_t(type, x, y) __careful_cmp((type)(x), (type)(y), <)
^~~~~~~~~~~~~
mm/page_alloc.c:6379:29: note: in expansion of macro 'min_t'
pgdat->static_init_pgcnt = min_t(unsigned long, PAGES_PER_SECTION,
^~~~~
include/linux/kernel.h:836:2: error: first argument to '__builtin_choose_expr' not a constant
__builtin_choose_expr(__safe_cmp(x, y), \
^
include/linux/kernel.h:904:27: note: in expansion of macro '__careful_cmp'
#define min_t(type, x, y) __careful_cmp((type)(x), (type)(y), <)
^~~~~~~~~~~~~
mm/page_alloc.c:6379:29: note: in expansion of macro 'min_t'
pgdat->static_init_pgcnt = min_t(unsigned long, PAGES_PER_SECTION,
^~~~~
scripts/Makefile.build:317: recipe for target 'mm/page_alloc.o' failed
Let's make the DEFERRED_STRUCT_PAGE_INIT explicitly depend on SPARSEMEM
as the systems that support DISCONTIGMEM do not seem to have that huge
amounts of memory that would make DEFERRED_STRUCT_PAGE_INIT relevant.
Link: http://lkml.kernel.org/r/1530279308-24988-1-git-send-email-rppt@linux.vnet.ibm.com
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Cc: Pasha Tatashin <Pavel.Tatashin@microsoft.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-18 07:47:07 +09:00
|
|
|
depends on SPARSEMEM
|
2018-05-19 08:09:13 +09:00
|
|
|
depends on !NEED_PER_CPU_KM
|
2018-09-21 04:22:30 +09:00
|
|
|
depends on 64BIT
|
2015-07-01 06:57:02 +09:00
|
|
|
help
|
|
|
|
|
Ordinarily all struct pages are initialised during early boot in a
|
|
|
|
|
single thread. On very large machines this can take a considerable
|
|
|
|
|
amount of time. If this option is set, large machines will bring up
|
|
|
|
|
a subset of memmap at boot and then initialise the rest in parallel
|
2016-02-06 08:36:21 +09:00
|
|
|
by starting one-off "pgdatinitX" kernel thread for each node X. This
|
|
|
|
|
has a potential performance impact on processes running early in the
|
|
|
|
|
lifetime of the system until these kthreads finish the
|
|
|
|
|
initialisation.
|
2015-08-10 04:29:06 +09:00
|
|
|
|
mm: introduce idle page tracking
Knowing the portion of memory that is not used by a certain application or
memory cgroup (idle memory) can be useful for partitioning the system
efficiently, e.g. by setting memory cgroup limits appropriately.
Currently, the only means to estimate the amount of idle memory provided
by the kernel is /proc/PID/{clear_refs,smaps}: the user can clear the
access bit for all pages mapped to a particular process by writing 1 to
clear_refs, wait for some time, and then count smaps:Referenced. However,
this method has two serious shortcomings:
- it does not count unmapped file pages
- it affects the reclaimer logic
To overcome these drawbacks, this patch introduces two new page flags,
Idle and Young, and a new sysfs file, /sys/kernel/mm/page_idle/bitmap.
A page's Idle flag can only be set from userspace by setting bit in
/sys/kernel/mm/page_idle/bitmap at the offset corresponding to the page,
and it is cleared whenever the page is accessed either through page tables
(it is cleared in page_referenced() in this case) or using the read(2)
system call (mark_page_accessed()). Thus by setting the Idle flag for
pages of a particular workload, which can be found e.g. by reading
/proc/PID/pagemap, waiting for some time to let the workload access its
working set, and then reading the bitmap file, one can estimate the amount
of pages that are not used by the workload.
The Young page flag is used to avoid interference with the memory
reclaimer. A page's Young flag is set whenever the Access bit of a page
table entry pointing to the page is cleared by writing to the bitmap file.
If page_referenced() is called on a Young page, it will add 1 to its
return value, therefore concealing the fact that the Access bit was
cleared.
Note, since there is no room for extra page flags on 32 bit, this feature
uses extended page flags when compiled on 32 bit.
[akpm@linux-foundation.org: fix build]
[akpm@linux-foundation.org: kpageidle requires an MMU]
[akpm@linux-foundation.org: decouple from page-flags rework]
Signed-off-by: Vladimir Davydov <vdavydov@parallels.com>
Reviewed-by: Andres Lagar-Cavilla <andreslc@google.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: Greg Thelen <gthelen@google.com>
Cc: Michel Lespinasse <walken@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-09-10 07:35:45 +09:00
|
|
|
config IDLE_PAGE_TRACKING
|
|
|
|
|
bool "Enable idle page tracking"
|
|
|
|
|
depends on SYSFS && MMU
|
|
|
|
|
select PAGE_EXTENSION if !64BIT
|
|
|
|
|
help
|
|
|
|
|
This feature allows to estimate the amount of user pages that have
|
|
|
|
|
not been touched during a given period of time. This information can
|
|
|
|
|
be useful to tune memory cgroup limits and/or for job placement
|
|
|
|
|
within a compute cluster.
|
|
|
|
|
|
2018-04-18 17:07:49 +09:00
|
|
|
See Documentation/admin-guide/mm/idle_page_tracking.rst for
|
|
|
|
|
more details.
|
mm: introduce idle page tracking
Knowing the portion of memory that is not used by a certain application or
memory cgroup (idle memory) can be useful for partitioning the system
efficiently, e.g. by setting memory cgroup limits appropriately.
Currently, the only means to estimate the amount of idle memory provided
by the kernel is /proc/PID/{clear_refs,smaps}: the user can clear the
access bit for all pages mapped to a particular process by writing 1 to
clear_refs, wait for some time, and then count smaps:Referenced. However,
this method has two serious shortcomings:
- it does not count unmapped file pages
- it affects the reclaimer logic
To overcome these drawbacks, this patch introduces two new page flags,
Idle and Young, and a new sysfs file, /sys/kernel/mm/page_idle/bitmap.
A page's Idle flag can only be set from userspace by setting bit in
/sys/kernel/mm/page_idle/bitmap at the offset corresponding to the page,
and it is cleared whenever the page is accessed either through page tables
(it is cleared in page_referenced() in this case) or using the read(2)
system call (mark_page_accessed()). Thus by setting the Idle flag for
pages of a particular workload, which can be found e.g. by reading
/proc/PID/pagemap, waiting for some time to let the workload access its
working set, and then reading the bitmap file, one can estimate the amount
of pages that are not used by the workload.
The Young page flag is used to avoid interference with the memory
reclaimer. A page's Young flag is set whenever the Access bit of a page
table entry pointing to the page is cleared by writing to the bitmap file.
If page_referenced() is called on a Young page, it will add 1 to its
return value, therefore concealing the fact that the Access bit was
cleared.
Note, since there is no room for extra page flags on 32 bit, this feature
uses extended page flags when compiled on 32 bit.
[akpm@linux-foundation.org: fix build]
[akpm@linux-foundation.org: kpageidle requires an MMU]
[akpm@linux-foundation.org: decouple from page-flags rework]
Signed-off-by: Vladimir Davydov <vdavydov@parallels.com>
Reviewed-by: Andres Lagar-Cavilla <andreslc@google.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: Greg Thelen <gthelen@google.com>
Cc: Michel Lespinasse <walken@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-09-10 07:35:45 +09:00
|
|
|
|
2019-07-17 08:30:47 +09:00
|
|
|
config ARCH_HAS_PTE_DEVMAP
|
2017-06-28 10:32:31 +09:00
|
|
|
bool
|
|
|
|
|
|
2015-08-10 04:29:06 +09:00
|
|
|
config ZONE_DEVICE
|
2017-09-09 08:11:43 +09:00
|
|
|
bool "Device memory (pmem, HMM, etc...) hotplug support"
|
2015-08-10 04:29:06 +09:00
|
|
|
depends on MEMORY_HOTPLUG
|
|
|
|
|
depends on MEMORY_HOTREMOVE
|
2016-03-18 06:19:58 +09:00
|
|
|
depends on SPARSEMEM_VMEMMAP
|
2019-07-17 08:30:47 +09:00
|
|
|
depends on ARCH_HAS_PTE_DEVMAP
|
2018-09-23 05:14:30 +09:00
|
|
|
select XARRAY_MULTI
|
2015-08-10 04:29:06 +09:00
|
|
|
|
|
|
|
|
help
|
|
|
|
|
Device memory hotplug support allows for establishing pmem,
|
|
|
|
|
or other device driver discovered memory regions, in the
|
|
|
|
|
memmap. This allows pfn_to_page() lookups of otherwise
|
|
|
|
|
"device-physical" addresses which is needed for using a DAX
|
|
|
|
|
mapping in an O_DIRECT operation, among other things.
|
|
|
|
|
|
|
|
|
|
If FS_DAX is enabled, then say Y.
|
2015-09-12 08:42:39 +09:00
|
|
|
|
2018-05-17 03:46:08 +09:00
|
|
|
config DEV_PAGEMAP_OPS
|
|
|
|
|
bool
|
|
|
|
|
|
2019-08-07 01:05:52 +09:00
|
|
|
#
|
|
|
|
|
# Helpers to mirror range of the CPU page tables of a process into device page
|
|
|
|
|
# tables.
|
|
|
|
|
#
|
2017-09-09 08:11:27 +09:00
|
|
|
config HMM_MIRROR
|
2019-08-07 01:05:52 +09:00
|
|
|
bool
|
2019-08-07 01:05:51 +09:00
|
|
|
depends on MMU
|
2019-08-07 01:05:52 +09:00
|
|
|
depends on MMU_NOTIFIER
|
2017-09-09 08:11:27 +09:00
|
|
|
|
2017-09-09 08:11:43 +09:00
|
|
|
config DEVICE_PRIVATE
|
|
|
|
|
bool "Unaddressable device memory (GPU memory, ...)"
|
2019-06-26 21:27:22 +09:00
|
|
|
depends on ZONE_DEVICE
|
2018-05-17 03:46:08 +09:00
|
|
|
select DEV_PAGEMAP_OPS
|
2017-09-09 08:11:43 +09:00
|
|
|
|
|
|
|
|
help
|
|
|
|
|
Allows creation of struct pages to represent unaddressable device
|
|
|
|
|
memory; i.e., memory that is only accessible from the device (or
|
|
|
|
|
group of devices). You likely also want to select HMM_MIRROR.
|
|
|
|
|
|
2015-07-13 23:55:44 +09:00
|
|
|
config FRAME_VECTOR
|
|
|
|
|
bool
|
2016-02-13 06:02:08 +09:00
|
|
|
|
|
|
|
|
config ARCH_USES_HIGH_VMA_FLAGS
|
|
|
|
|
bool
|
2016-02-13 06:02:32 +09:00
|
|
|
config ARCH_HAS_PKEYS
|
|
|
|
|
bool
|
2017-06-20 08:28:31 +09:00
|
|
|
|
|
|
|
|
config PERCPU_STATS
|
|
|
|
|
bool "Collect percpu memory statistics"
|
|
|
|
|
help
|
|
|
|
|
This feature collects and exposes statistics via debugfs. The
|
|
|
|
|
information includes global and per chunk statistics, which can
|
|
|
|
|
be used to help understand percpu memory usage.
|
2017-11-18 08:31:22 +09:00
|
|
|
|
2019-03-10 21:10:16 +09:00
|
|
|
config MM_STAT_UNRECLAIMABLE_PAGES
|
|
|
|
|
default y
|
|
|
|
|
depends on QGKI
|
|
|
|
|
bool "Enable counter for unreclaimable pages"
|
|
|
|
|
help
|
|
|
|
|
Unreclaimable pages counters accounts the pages that cannot be
|
|
|
|
|
reclaimed under memory pressure. These include mm rss and global
|
|
|
|
|
counter. This option allows the enabling of the mm rss and global
|
|
|
|
|
unreclaimables pages counters on QGKI systems.
|
|
|
|
|
|
2020-02-26 07:24:29 +09:00
|
|
|
config HAVE_USERSPACE_LOW_MEMORY_KILLER
|
|
|
|
|
bool "Configure reclaim paths if there is a userspace LMK"
|
|
|
|
|
default n
|
|
|
|
|
help
|
|
|
|
|
Configures parts of the reclaim paths in the kernel when Android
|
|
|
|
|
has a userspace memory killer. This configuration is done for
|
|
|
|
|
functional (e.g. ensuring the OOM killer doesn't run at certain
|
|
|
|
|
points) and performance purposes (e.g. preventing log-spew
|
|
|
|
|
when the OOM killer and userspace memory killer both have the
|
|
|
|
|
potential to run).
|
|
|
|
|
|
2018-04-17 23:33:07 +09:00
|
|
|
config ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT
|
|
|
|
|
def_bool n
|
|
|
|
|
|
|
|
|
|
config SPECULATIVE_PAGE_FAULT
|
|
|
|
|
bool "Speculative page faults"
|
|
|
|
|
default y
|
|
|
|
|
depends on ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT
|
|
|
|
|
depends on MMU && SMP
|
|
|
|
|
depends on QGKI
|
|
|
|
|
help
|
|
|
|
|
Try to handle user space page faults without holding the mmap_sem.
|
|
|
|
|
|
|
|
|
|
This should allow better concurrency for massively threaded process
|
|
|
|
|
since the page fault handler will not wait for other threads memory
|
|
|
|
|
layout change to be done, assuming that this change is done in another
|
|
|
|
|
part of the process's memory space. This type of page fault is named
|
|
|
|
|
speculative page fault.
|
|
|
|
|
|
|
|
|
|
If the speculative page fault fails because of a concurrency is
|
|
|
|
|
detected or because underlying PMD or PTE tables are not yet
|
|
|
|
|
allocating, it is failing its processing and a classic page fault
|
|
|
|
|
is then tried.
|
|
|
|
|
|
2017-11-18 08:31:22 +09:00
|
|
|
config GUP_BENCHMARK
|
|
|
|
|
bool "Enable infrastructure for get_user_pages_fast() benchmarking"
|
|
|
|
|
help
|
|
|
|
|
Provides /sys/kernel/debug/gup_benchmark that helps with testing
|
|
|
|
|
performance of get_user_pages_fast().
|
|
|
|
|
|
|
|
|
|
See tools/testing/selftests/vm/gup_benchmark.c
|
2018-06-08 09:06:08 +09:00
|
|
|
|
2019-07-12 12:56:49 +09:00
|
|
|
config GUP_GET_PTE_LOW_HIGH
|
|
|
|
|
bool
|
|
|
|
|
|
2019-09-24 07:38:00 +09:00
|
|
|
config READ_ONLY_THP_FOR_FS
|
|
|
|
|
bool "Read-only THP for filesystems (EXPERIMENTAL)"
|
|
|
|
|
depends on TRANSPARENT_HUGE_PAGECACHE && SHMEM
|
|
|
|
|
|
|
|
|
|
help
|
|
|
|
|
Allow khugepaged to put read-only file-backed pages in THP.
|
|
|
|
|
|
|
|
|
|
This is marked experimental because it is a new feature. Write
|
|
|
|
|
support of file THPs will be developed in the next few release
|
|
|
|
|
cycles.
|
|
|
|
|
|
2018-06-08 09:06:08 +09:00
|
|
|
config ARCH_HAS_PTE_SPECIAL
|
|
|
|
|
bool
|
2018-07-31 20:39:35 +09:00
|
|
|
|
2019-07-12 12:57:28 +09:00
|
|
|
#
|
|
|
|
|
# Some architectures require a special hugepage directory format that is
|
|
|
|
|
# required to support multiple hugepage sizes. For example a4fe3ce76
|
|
|
|
|
# "powerpc/mm: Allow more flexible layouts for hugepage pagetables"
|
|
|
|
|
# introduced it on powerpc. This allows for a more flexible hugepage
|
|
|
|
|
# pagetable layouts.
|
|
|
|
|
#
|
|
|
|
|
config ARCH_HAS_HUGEPD
|
|
|
|
|
bool
|
|
|
|
|
|
2020-03-04 13:32:17 +09:00
|
|
|
config PRIORITIZE_OOM_TASKS
|
|
|
|
|
bool "Attempt to further prioritize OOM killer kills"
|
|
|
|
|
depends on QGKI
|
|
|
|
|
help
|
|
|
|
|
When the OOM killer selects a task to kill it considers all tasks
|
|
|
|
|
with a oom_score_adj value > OOM_SCORE_ADJ_MIN. This doesn't work
|
|
|
|
|
well when the OS has 'high priority' tasks which should only be
|
|
|
|
|
considered for killing after all other 'lower priority' tasks
|
|
|
|
|
have first been killed.
|
|
|
|
|
|
|
|
|
|
Enable this option to impose a limit on the minimum task
|
|
|
|
|
oom_score_adj which should first be considered for killing
|
|
|
|
|
before allowing the OOM killer to select tasks without
|
|
|
|
|
limitation.
|
|
|
|
|
|
|
|
|
|
If unsure, say N.
|
|
|
|
|
|
|
|
|
|
config OOM_TASK_PRIORITY_ADJ_LIMIT
|
|
|
|
|
int "Min oom_score_adj when priotizing OOM killer kills"
|
|
|
|
|
depends on PRIORITIZE_OOM_TASKS
|
|
|
|
|
range -1000 1000
|
|
|
|
|
default 0
|
|
|
|
|
help
|
|
|
|
|
With this parameter you can limit the OOM killer to first kill
|
|
|
|
|
tasks with an oom_score_adj value above a certain threshold
|
|
|
|
|
before considering tasks with a lower oom_score_adj value.
|
|
|
|
|
|
2018-07-31 20:39:35 +09:00
|
|
|
endmenu
|
2013-05-09 16:21:24 +09:00
|
|
|
|
|
|
|
|
config PROCESS_RECLAIM
|
|
|
|
|
bool "Enable process reclaim"
|
|
|
|
|
depends on PROC_FS
|
|
|
|
|
depends on QGKI
|
|
|
|
|
default y
|
|
|
|
|
help
|
|
|
|
|
It allows to reclaim pages of the process by /proc/pid/reclaim.
|
|
|
|
|
|
|
|
|
|
(echo file > /proc/PID/reclaim) reclaims file-backed pages only.
|
|
|
|
|
(echo anon > /proc/PID/reclaim) reclaims anonymous pages only.
|
|
|
|
|
(echo all > /proc/PID/reclaim) reclaims all pages.
|
|
|
|
|
|
2013-05-09 16:21:28 +09:00
|
|
|
(echo addr size-byte > /proc/PID/reclaim) reclaims pages in
|
|
|
|
|
(addr, addr + size-bytes) of the process.
|
|
|
|
|
|
2013-05-09 16:21:24 +09:00
|
|
|
Any other value is ignored.
|
vmscan: Support multiple kswapd threads per node
Page replacement is handled in the Linux Kernel in one of two ways:
1) Asynchronously via kswapd
2) Synchronously, via direct reclaim
At page allocation time the allocating task is immediately given a page
from the zone free list allowing it to go right back to work doing
whatever it was doing; Probably directly or indirectly executing business
logic.
Just prior to satisfying the allocation, free pages is checked to see if
it has reached the zone low watermark and if so, kswapd is awakened.
Kswapd will start scanning pages looking for inactive pages to evict to
make room for new page allocations. The work of kswapd allows tasks to
continue allocating memory from their respective zone free list without
incurring any delay.
When the demand for free pages exceeds the rate that kswapd tasks can
supply them, page allocation works differently. Once the allocating task
finds that the number of free pages is at or below the zone min watermark,
the task will no longer pull pages from the free list. Instead, the task
will run the same CPU-bound routines as kswapd to satisfy its own
allocation by scanning and evicting pages. This is called a direct reclaim.
The time spent performing a direct reclaim can be substantial, often
taking tens to hundreds of milliseconds for small order0 allocations to
half a second or more for order9 huge-page allocations. In fact, kswapd is
not actually required on a linux system. It exists for the sole purpose of
optimizing performance by preventing direct reclaims.
When memory shortfall is sufficient to trigger direct reclaims, they can
occur in any task that is running on the system. A single aggressive
memory allocating task can set the stage for collateral damage to occur in
small tasks that rarely allocate additional memory. Consider the impact of
injecting an additional 100ms of latency when nscd allocates memory to
facilitate caching of a DNS query.
The presence of direct reclaims 10 years ago was a fairly reliable
indicator that too much was being asked of a Linux system. Kswapd was
likely wasting time scanning pages that were ineligible for eviction.
Adding RAM or reducing the working set size would usually make the problem
go away. Since then hardware has evolved to bring a new struggle for
kswapd. Storage speeds have increased by orders of magnitude while CPU
clock speeds stayed the same or even slowed down in exchange for more
cores per package. This presents a throughput problem for a single
threaded kswapd that will get worse with each generation of new hardware.
Test Details
NOTE: The tests below were run with shadow entries disabled. See the
associated patch and cover letter for details
The tests below were designed with the assumption that a kswapd bottleneck
is best demonstrated using filesystem reads. This way, the inactive list
will be full of clean pages, simplifying the analysis and allowing kswapd
to achieve the highest possible steal rate. Maximum steal rates for kswapd
are likely to be the same or lower for any other mix of page types on the
system.
Tests were run on a 2U Oracle X7-2L with 52 Intel Xeon Skylake 2GHz cores,
756GB of RAM and 8 x 3.6 TB NVMe Solid State Disk drives. Each drive has
an XFS file system mounted separately as /d0 through /d7. SSD drives
require multiple concurrent streams to show their potential, so I created
eleven 250GB zero-filled files on each drive so that I could test with
parallel reads.
The test script runs in multiple stages. At each stage, the number of dd
tasks run concurrently is increased by 2. I did not include all of the
test output for brevity.
During each stage dd tasks are launched to read from each drive in a round
robin fashion until the specified number of tasks for the stage has been
reached. Then iostat, vmstat and top are started in the background with 10
second intervals. After five minutes, all of the dd tasks are killed and
the iostat, vmstat and top output is parsed in order to report the
following:
CPU consumption
- sy - aggregate kernel mode CPU consumption from vmstat output. The value
doesn't tend to fluctuate much so I just grab the highest value.
Each sample is averaged over 10 seconds
- dd_cpu - for all of the dd tasks averaged across the top samples since
there is a lot of variation.
Throughput
- in Kbytes
- Command is iostat -x -d 10 -g total
This first test performs reads using O_DIRECT in order to show the maximum
throughput that can be obtained using these drives. It also demonstrates
how rapidly throughput scales as the number of dd tasks are increased.
The dd command for this test looks like this:
Command Used: dd iflag=direct if=/d${i}/$n of=/dev/null bs=4M
Test #1: Direct IO
dd sy dd_cpu throughput
6 0 2.33 14726026.40
10 1 2.95 19954974.80
16 1 2.63 24419689.30
22 1 2.63 25430303.20
28 1 2.91 26026513.20
34 1 2.53 26178618.00
40 1 2.18 26239229.20
46 1 1.91 26250550.40
52 1 1.69 26251845.60
58 1 1.54 26253205.60
64 1 1.43 26253780.80
70 1 1.31 26254154.80
76 1 1.21 26253660.80
82 1 1.12 26254214.80
88 1 1.07 26253770.00
90 1 1.04 26252406.40
Throughput was close to peak with only 22 dd tasks. Very little system CPU
was consumed as expected as the drives DMA directly into the user address
space when using direct IO.
In this next test, the iflag=direct option is removed and we only run the
test until the pgscan_kswapd from /proc/vmstat starts to increment. At
that point metrics are parsed and reported and the pagecache contents are
dropped prior to the next test. Lather, rinse, repeat.
Test #2: standard file system IO, no page replacement
dd sy dd_cpu throughput
6 2 28.78 5134316.40
10 3 31.40 8051218.40
16 5 34.73 11438106.80
22 7 33.65 14140596.40
28 8 31.24 16393455.20
34 10 29.88 18219463.60
40 11 28.33 19644159.60
46 11 25.05 20802497.60
52 13 26.92 22092370.00
58 13 23.29 22884881.20
64 14 23.12 23452248.80
70 15 22.40 23916468.00
76 16 22.06 24328737.20
82 17 20.97 24718693.20
88 16 18.57 25149404.40
90 16 18.31 25245565.60
Each read has to pause after the buffer in kernel space is populated while
those pages are added to the pagecache and copied into the user address
space. For this reason, more parallel streams are required to achieve peak
throughput. The copy operation consumes substantially more CPU than direct
IO as expected.
The next test measures throughput after kswapd starts running. This is the
same test only we wait for kswapd to wake up before we start collecting
metrics. The script actually keeps track of a few things that were not
mentioned earlier. It tracks direct reclaims and page scans by watching
the metrics in /proc/vmstat. CPU consumption for kswapd is tracked the
same way it is tracked for dd.
Since the test is 100% reads, you can assume that the page steal rate for
kswapd and direct reclaims is almost identical to the scan rate.
Test #3: 1 kswapd thread per node
dd sy dd_cpu kswapd0 kswapd1 throughput dr pgscan_kswapd pgscan_direct
10 4 26.07 28.56 27.03 7355924.40 0 459316976 0
16 7 34.94 69.33 69.66 10867895.20 0 872661643 0
22 10 36.03 93.99 99.33 13130613.60 489 1037654473 11268334
28 10 30.34 95.90 98.60 14601509.60 671 1182591373 15429142
34 14 34.77 97.50 99.23 16468012.00 10850 1069005644 249839515
40 17 36.32 91.49 97.11 17335987.60 18903 975417728 434467710
46 19 38.40 90.54 91.61 17705394.40 25369 855737040 582427973
52 22 40.88 83.97 83.70 17607680.40 31250 709532935 724282458
58 25 40.89 82.19 80.14 17976905.60 35060 657796473 804117540
64 28 41.77 73.49 75.20 18001910.00 39073 561813658 895289337
70 33 45.51 63.78 64.39 17061897.20 44523 379465571 1020726436
76 36 46.95 57.96 60.32 16964459.60 47717 291299464 1093172384
82 39 47.16 55.43 56.16 16949956.00 49479 247071062 1134163008
88 42 47.41 53.75 47.62 16930911.20 51521 195449924 1180442208
90 43 47.18 51.40 50.59 16864428.00 51618 190758156 1183203901
In the previous test where kswapd was not involved, the system-wide kernel
mode CPU consumption with 90 dd tasks was 16%. In this test CPU consumption
with 90 tasks is at 43%. With 52 cores, and two kswapd tasks (one per NUMA
node), kswapd can only be responsible for a little over 4% of the increase.
The rest is likely caused by 51,618 direct reclaims that scanned 1.2
billion pages over the five minute time period of the test.
Same test, more kswapd tasks:
Test #4: 4 kswapd threads per node
dd sy dd_cpu kswapd0 kswapd1 throughput dr pgscan_kswapd pgscan_direct
10 5 27.09 16.65 14.17 7842605.60 0 459105291 0
16 10 37.12 26.02 24.85 11352920.40 15 920527796 358515
22 11 36.94 37.13 35.82 13771869.60 0 1132169011 0
28 13 35.23 48.43 46.86 16089746.00 0 1312902070 0
34 15 33.37 53.02 55.69 18314856.40 0 1476169080 0
40 19 35.90 69.60 64.41 19836126.80 0 1629999149 0
46 22 36.82 88.55 57.20 20740216.40 0 1708478106 0
52 24 34.38 93.76 68.34 21758352.00 0 1794055559 0
58 24 30.51 79.20 82.33 22735594.00 0 1872794397 0
64 26 30.21 97.12 76.73 23302203.60 176 1916593721 4206821
70 33 32.92 92.91 92.87 23776588.00 3575 1817685086 85574159
76 37 31.62 91.20 89.83 24308196.80 4752 1812262569 113981763
82 29 25.53 93.23 92.33 24802791.20 306 2032093122 7350704
88 43 37.12 76.18 77.01 25145694.40 20310 1253204719 487048202
90 42 38.56 73.90 74.57 22516787.60 22774 1193637495 545463615
By increasing the number of kswapd threads, throughput increased by ~50%
while kernel mode CPU utilization decreased or stayed the same, likely due
to a decrease in the number of parallel tasks at any given time doing page
replacement.
Change-Id: I966d4a9c33bad188b3409f7ceea1df205a63c3bd
Signed-off-by: Buddy Lumpkin <buddy.lumpkin@oracle.com>
Patch-mainline: linux-mm @ Mon, 2 Apr 2018 09:24:22
Link: https://lore.kernel.org/lkml/1522661062-39745-1-git-send-email-buddy.lumpkin@oracle.com
[charante@codeaurora.org]: Changes done to ensure QGKI compliance.
Signed-off-by: Charan Teja Kalla <charante@codeaurora.org>
2018-04-02 18:24:22 +09:00
|
|
|
|
|
|
|
|
config MULTIPLE_KSWAPD
|
|
|
|
|
bool "Spawn multiple kswapd threads"
|
|
|
|
|
depends on QGKI
|
|
|
|
|
default y
|
|
|
|
|
help
|
|
|
|
|
kswapd_threads allows you to control the number of kswapd threads
|
|
|
|
|
per node running on the system. The default value is 1 and the
|
|
|
|
|
range of acceptible values are 1-16. The number of threads can
|
|
|
|
|
be controlled by below command:
|
|
|
|
|
(echo <num> > /proc/sys/vm/kswapd_threads)
|
|
|
|
|
|
|
|
|
|
Values not in the range of 1..16 are ignored.
|
