This one is much faster than the spinlock based fallback rwsem code,
with certain artifical benchmarks having shown 300%+ improvement on
threaded page faults etc.
Again, note the 32767-thread limit here. So this really does need that
whole "make rwsem_count_t be 64-bit and fix the BIAS values to match"
extension on top of it, but that is conceptually a totally independent
issue.
NOT TESTED! The original patch that this all was based on were tested by
KAMEZAWA Hiroyuki, but maybe I screwed up something when I created the
cleaned-up series, so caveat emptor..
Also note that it _may_ be a good idea to mark some more registers
clobbered on x86-64 in the inline asms instead of saving/restoring them.
They are inline functions, but they are only used in places where there
are not a lot of live registers _anyway_, so doing for example the
clobbers of %r8-%r11 in the asm wouldn't make the fast-path code any
worse, and would make the slow-path code smaller.
(Not that the slow-path really matters to that degree. Saving a few
unnecessary registers is the _least_ of our problems when we hit the slow
path. The instruction/cycle counting really only matters in the fast
path).
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
LKML-Reference: <alpine.LFD.2.00.1001121810410.17145@localhost.localdomain>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
The fast version of the rwsems (the code that uses xadd) has
traditionally only worked on x86-32, and as a result it mixes different
kinds of types wildly - they just all happen to be 32-bit. We have
"long", we have "__s32", and we have "int".
To make it work on x86-64, the types suddenly matter a lot more. It can
be either a 32-bit or 64-bit signed type, and both work (with the caveat
that a 32-bit counter will only have 15 bits of effective write
counters, so it's limited to 32767 users). But whatever type you
choose, it needs to be used consistently.
This makes a new 'rwsem_counter_t', that is a 32-bit signed type. For a
64-bit type, you'd need to also update the BIAS values.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
LKML-Reference: <alpine.LFD.2.00.1001121755220.17145@localhost.localdomain>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Using kernel_stack_pointer() allows 32-bit and 64-bit versions to
be merged. This is more correct for 64-bit, since the old %rsp is
always saved on the stack.
Signed-off-by: Brian Gerst <brgerst@gmail.com>
LKML-Reference: <1263397555-27695-1-git-send-email-brgerst@gmail.com>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Use a macro to define the cache sizes when cachesize > 1 MB.
This is less typing, and less prone to introducing bugs like we
saw in e02e0e1a130b9ca37c5186d38ad4b3aaf58bb149, and means we
don't have to do maths when adding new non-power-of-2 updates
like those seen recently.
Signed-off-by: Dave Jones <davej@redhat.com>
LKML-Reference: <20100104144735.GA18390@redhat.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
This makes gcc use the right register names and instruction operand sizes
automatically for the rwsem inline asm statements.
So instead of using "(%%eax)" to specify the memory address that is the
semaphore, we use "(%1)" or similar. And instead of forcing the operation
to always be 32-bit, we use "%z0", taking the size from the actual
semaphore data structure itself.
This doesn't actually matter on x86-32, but if we want to use the same
inline asm for x86-64, we'll need to have the compiler generate the proper
64-bit names for the registers (%rax instead of %eax), and if we want to
use a 64-bit counter too (in order to avoid the 15-bit limit on the
write counter that limits concurrent users to 32767 threads), we'll need
to be able to generate instructions with "q" accesses rather than "l".
Since this header currently isn't enabled on x86-64, none of that matters,
but we do want to use the xadd version of the semaphores rather than have
to take spinlocks to do a rwsem. The mm->mmap_sem can be heavily contended
when you have lots of threads all taking page faults, and the fallback
rwsem code that uses a spinlock performs abysmally badly in that case.
[ hpa: modified the patch to skip size suffixes entirely when they are
redundant due to register operands. ]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
LKML-Reference: <alpine.LFD.2.00.1001121613560.17145@localhost.localdomain>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Merge the now identical code from asm/atomic_32.h and asm/atomic_64.h
into asm/atomic.h.
Signed-off-by: Brian Gerst <brgerst@gmail.com>
LKML-Reference: <1262883215-4034-4-git-send-email-brgerst@gmail.com>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Prepare for merging into asm/atomic.h.
Signed-off-by: Brian Gerst <brgerst@gmail.com>
LKML-Reference: <1262883215-4034-3-git-send-email-brgerst@gmail.com>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Split atomic64_t functions out into separate headers, since they will
not be practical to merge between 32 and 64 bits.
Signed-off-by: Brian Gerst <brgerst@gmail.com>
LKML-Reference: <1262883215-4034-2-git-send-email-brgerst@gmail.com>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
In order to avoid unnecessary chains of branches, rather than
implementing memcpy()/memset()'s access to their alternative
implementations via a jump, patch the (larger) original function
directly.
The memcpy() part of this is slightly subtle: while alternative
instruction patching does itself use memcpy(), with the
replacement block being less than 64-bytes in size the main loop
of the original function doesn't get used for copying memcpy_c()
over memcpy(), and hence we can safely write over its beginning.
Also note that the CFI annotations are fine for both variants of
each of the functions.
Signed-off-by: Jan Beulich <jbeulich@novell.com>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
LKML-Reference: <4B2BB8D30200007800026AF2@vpn.id2.novell.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
In order to avoid unnecessary chains of branches, rather than
implementing copy_user_generic() as a function consisting of
just a single (possibly patched) branch, instead properly deal
with patching call instructions in the alternative instructions
framework, and move the patching into the callers.
As a follow-on, one could also introduce something like
__EXPORT_SYMBOL_ALT() to avoid patching call sites in modules.
Signed-off-by: Jan Beulich <jbeulich@novell.com>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
LKML-Reference: <4B2BB8180200007800026AE7@vpn.id2.novell.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
The early ioremap fixmap entries cover half (or for 32-bit
non-PAE, a quarter) of a page table, yet they got
uncondtitionally aligned so far to a 256-entry boundary. This is
not necessary if the range of page table entries anyway falls
into a single page table.
This buys back, for (theoretically) 50% of all configurations
(25% of all non-PAE ones), at least some of the lowmem
necessarily lost with commit e621bd18958ef5dbace3129ebe17a0a475e127d9.
Signed-off-by: Jan Beulich <jbeulich@novell.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
LKML-Reference: <4B2BB66F0200007800026AD6@vpn.id2.novell.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
* 'release' of git://git.kernel.org/pub/scm/linux/kernel/git/lenb/linux-acpi-2.6: (34 commits)
classmate-laptop: add support for Classmate PC ACPI devices
hp-wmi: Fix two memleaks
acer-wmi, msi-wmi: Remove needless DMI MODULE_ALIAS
dell-wmi: do not keep driver loaded on unsupported boxes
wmi: Free the allocated acpi objects through wmi_get_event_data
drivers/platform/x86/acerhdf.c: check BIOS information whether it begins with string of table
acerhdf: add new BIOS versions
acerhdf: limit modalias matching to supported
toshiba_acpi: convert to seq_file
asus_acpi: convert to seq_file
ACPI: do not select ACPI_DOCK from ATA_ACPI
sony-laptop: enumerate rfkill devices using SN06
sony-laptop: rfkill support for newer models
ACPI: fix OSC regression that caused aer and pciehp not to load
MAINTAINERS: add maintainer for msi-wmi driver
fujitu-laptop: fix tests of acpi_evaluate_integer() return value
arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts by using smp_call_function_any()
ACPI: processor: remove _PDC object list from struct acpi_processor
ACPI: processor: change acpi_processor_set_pdc() interface
ACPI: processor: open code acpi_processor_cleanup_pdc
...
* 'for-linus' of master.kernel.org:/home/rmk/linux-2.6-arm:
VIDEO: cyberpro: pci_request_regions needs a persistent name
ARM: dma-isa: request cascade channel after registering it
ARM: footbridge: trim down old ISA rtc setup
ARM: fix PAGE_KERNEL
ARM: Fix wrong shared bit for CPU write buffer bug test
ARM: 5857/1: ARM: dmabounce: fix build
ARM: 5856/1: Fix bug of uart0 platfrom data for nuc900
ARM: 5855/1: putc support for nuc900
ARM: 5854/1: fix compiling error for NUC900
ARM: 5849/1: ARMv7: fix Oprofile events count
ARM: add missing include to nwflash.c
ARM: Kill CONFIG_CPU_32
ARM: Convert VFP/Crunch/XscaleCP thread_release() to exit_thread()
ARM: 5853/1: ARM: Fix build break on ARM v6 and v7
This fixes a "start_kernel(): bug: interrupts were enabled early".
rtc_cmos now takes care of initializing the ISA RTC and reading the
current time and date from it; there's no need to repeat that here,
thereby causing interrupts to be enabled too early.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
PAGE_KERNEL should not be executable; any area marked executable can
be prefetched into the instruction cache. We don't want vmalloc areas
to be read in this way.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
With some of the cache rework an address aliasing optimization was added,
but this managed to fail on certain mappings resulting in pages with
PG_dcache_dirty set never writing back their dcache lines. This patch
reverts to the earlier behaviour of simply always writing back when the
dirty bit is set.
Signed-off-by: Markus Pietrek <Markus.Pietrek@emtrion.de>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
o remove unused define
o add device name comment
Signed-off-by: Kuninori Morimoto <morimoto.kuninori@renesas.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
This reverts commit 9f15226e75583547aaf542c6be4bdac1060dd425. It's just
wrong, and broke resume for Rafael even on a non-AMD CPU.
As Rafael says:
"... it causes microcode_init_cpu() to be called during resume even for
CPUs for which there's no microcode to apply. That, in turn, results
in executing request_firmware() (on Intel CPUs at least) which doesn't
work at this stage of resume (we have device interrupts disabled, I/O
devices are still suspended and so on).
If I'm not mistaken, the "if (uci->valid)" logic means "if that CPU is
known to us" , so before commit 9f15226e755 microcode_resume_cpu() was
called for all CPUs already in the system during suspend, which was
the right thing to do. The commit changed it so that the CPUs without
microcode to apply are now treated as "unknown", which is not quite
right.
The problem this commit attempted to solve has to be handled
differently."
Bisected-and -requested-by: Rafael J. Wysocki <rjw@sisk.pl>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
It is unpredictable to have the same memory mapped using different
shared bit settings for ARMv6 and ARMv7 CPUs. Fix this for the CPU
write buffer bug test.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* 'merge' of git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc: (36 commits)
powerpc/gc/wii: Remove get_irq_desc()
powerpc/gc/wii: hlwd-pic: convert irq_desc.lock to raw_spinlock
powerpc/gamecube/wii: Fix off-by-one error in ugecon/usbgecko_udbg
powerpc/mpic: Fix problem that affinity is not updated
powerpc/mm: Fix stupid bug in subpge protection handling
powerpc/iseries: use DECLARE_COMPLETION_ONSTACK for non-constant completion
powerpc: Fix MSI support on U4 bridge PCIe slot
powerpc: Handle VSX alignment faults correctly in little-endian mode
powerpc/mm: Fix typo of cpumask_clear_cpu()
powerpc/mm: Fix hash_utils_64.c compile errors with DEBUG enabled.
powerpc: Convert BUG() to use unreachable()
powerpc/pseries: Make declarations of cpu_hotplug_driver_lock() ANSI compatible.
powerpc/pseries: Don't panic when H_PROD fails during cpu-online.
powerpc/mm: Fix a WARN_ON() with CONFIG_DEBUG_PAGEALLOC and CONFIG_DEBUG_VM
powerpc/defconfigs: Set HZ=100 on pseries and ppc64 defconfigs
powerpc/defconfigs: Disable token ring in powerpc defconfigs
powerpc/defconfigs: Reduce 64bit vmlinux by making acenic and cramfs modules
powerpc/pseries: Select XICS and PCI_MSI PSERIES
powerpc/85xx: Wrong variable returned on error
powerpc/iseries: Convert to proc_fops
...
Presently acpi-cpufreq will perform the MSR read on the first CPU in the
mask. That's inefficient if that CPU differs from the current CPU.
Because we have to perform a cross-CPU call, but we could have run the
rdmsr on the current CPU.
So switch to using the new smp_call_function_any(), which will perform the
call on the current CPU if that CPU is present in the mask (it is).
Cc: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Jaswinder Singh Rajput <jaswinder@kernel.org>
Cc: Len Brown <len.brown@intel.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>
Cc: Zhao Yakui <yakui.zhao@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Len Brown <len.brown@intel.com>
Commit f74f7e57ae9fa12b2951ae62ce3557799b318399 (ARM: use
flush_kernel_dcache_area() for dmabounce) has broken dmabounce build:
CC arch/arm/common/dmabounce.o
arch/arm/common/dmabounce.c: In function 'unmap_single':
arch/arm/common/dmabounce.c:315: error: implicit declaration of function '__cpuc_flush_kernel_dcache_area'
make[2]: *** [arch/arm/common/dmabounce.o] Error 1
Fix it.
Signed-off-by: Mike Rapoport <mike@compulab.co.il>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Fix bug of uart0 platfrom data for nuc900
Signed-off-by: lijie <eltshanli@gmail.com>
Signed-off-by: Wan ZongShun <mcuos.com@gmail.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
putc support for nuc900
Signed-off-by: lijie <eltshanli@gmail.com>
Signed-off-by: Wan ZongShun <mcuos.com@gmail.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
fix compiling error for NUC900
Signed-off-by: lijie <eltshanli@gmail.com>
Signed-off-by: Wan ZongShun <mcuos.com@gmail.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
On Oprofile ARMv7 the PMNC_D bit was set to lower the PMU IRQs
and so to decrease the risk of errata #628216 from appearing.
The effect of setting the PMNC_D bit is that the CCNT counter
is divided by 64, making the program counter events count
inaccurate.
The new OMAP3 r4 cores should have that errata fixed.
The PMNC_D bit should not be set, this patch fixes it.
Signed-off-by: Jean Pihet <jpihet@mvista.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
The x86 and ia64 implementations of the function in $subject are
exactly the same.
Also, since the arch-specific implementations of setting _PDC have
been completely hollowed out, remove the empty shells.
Cc: Tony Luck <tony.luck@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Alex Chiang <achiang@hp.com>
Signed-off-by: Len Brown <len.brown@intel.com>
The only thing arch-specific about calling _PDC is what bits get
set in the input obj_list buffer.
There's no need for several levels of indirection to twiddle those
bits. Additionally, since we're just messing around with a buffer,
we can simplify the interface; no need to pass around the entire
struct acpi_processor * just to get at the buffer.
Cc: Tony Luck <tony.luck@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Alex Chiang <achiang@hp.com>
Signed-off-by: Len Brown <len.brown@intel.com>
Both x86 and ia64 initialize _PDC with mostly common bit settings.
Factor out the common settings and leave the arch-specific ones alone.
Cc: Tony Luck <tony.luck@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Alex Chiang <achiang@hp.com>
Signed-off-by: Len Brown <len.brown@intel.com>
The x86 and ia64 implementations of arch_acpi_processor_init_pdc()
are almost exactly the same. The only difference is in what bits
they set in obj_list buffer.
Combine the boilerplate memory management code, and leave the
arch-specific bit twiddling in separate implementations.
Cc: Tony Luck <tony.luck@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Alex Chiang <achiang@hp.com>
Signed-off-by: Len Brown <len.brown@intel.com>
arch dependent helper function that tells us if we should attempt to
evaluate _PDC on this machine or not.
The x86 implementation assumes that the CPUs in the machine must be
homogeneous, and that you cannot mix CPUs of different vendors.
Cc: Tony Luck <tony.luck@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Alex Chiang <achiang@hp.com>
Signed-off-by: Len Brown <len.brown@intel.com>
We don't actually require this in the cpu_relax() polling case, so just
cuddle these around the sleeping version.
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Fix the following build failures:
arch/powerpc/platforms/embedded6xx/flipper-pic.c: In function 'flipper_pic_map':
arch/powerpc/platforms/embedded6xx/flipper-pic.c:105: error: implicit declaration of function 'get_irq_desc'
arch/powerpc/platforms/embedded6xx/hlwd-pic.c: In function 'hlwd_pic_map':
arch/powerpc/platforms/embedded6xx/hlwd-pic.c:98: error: implicit declaration of function 'get_irq_desc'
These failures are caused by the changes introduced in commit
"powerpc: Remove get_irq_desc()". The reason these drivers were not
updated is that they weren't merged yet.
Signed-off-by: Albert Herranz <albert_herranz@yahoo.es>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Fix the following build failures:
arch/powerpc/platforms/embedded6xx/hlwd-pic.c: In function 'hlwd_pic_irq_cascade':
arch/powerpc/platforms/embedded6xx/hlwd-pic.c:135: error: passing argument 1 of 'spin_lock' from incompatible pointer type
arch/powerpc/platforms/embedded6xx/hlwd-pic.c:137: error: passing argument 1 of 'spin_unlock' from incompatible pointer type
arch/powerpc/platforms/embedded6xx/hlwd-pic.c:145: error: passing argument 1 of 'spin_lock' from incompatible pointer type
arch/powerpc/platforms/embedded6xx/hlwd-pic.c:149: error: passing argument 1 of 'spin_unlock' from incompatible pointer type
These failures are caused by the changes introduced in commit
"genirq: Convert irq_desc.lock to raw_spinlock". The reason this driver
was not updated is that it wasn't merged yet.
Signed-off-by: Albert Herranz <albert_herranz@yahoo.es>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mattst88/alpha-2.6:
alpha: Convert BUG() to use unreachable()
alpha: Add minimal support for software performance events
alpha: Wire up missing/new syscalls
* 'perf-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip:
perf session: Make events_stats u64 to avoid overflow on 32-bit arches
hw-breakpoints: Fix hardware breakpoints -> perf events dependency
perf events: Dont report side-band events on each cpu for per-task-per-cpu events
perf events, x86/stacktrace: Fix performance/softlockup by providing a special frame pointer-only stack walker
perf events, x86/stacktrace: Make stack walking optional
perf events: Remove unused perf_counter.h header file
perf probe: Check new event name
kprobe-tracer: Check new event/group name
perf probe: Check whether debugfs path is correct
perf probe: Fix libdwarf include path for Debian
* 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip:
x86, irq: Allow 0xff for /proc/irq/[n]/smp_affinity on an 8-cpu system
Makefile: Unexport LC_ALL instead of clearing it
x86: Fix objdump version check in arch/x86/tools/chkobjdump.awk
x86: Reenable TSC sync check at boot, even with NONSTOP_TSC
x86: Don't use POSIX character classes in gen-insn-attr-x86.awk
Makefile: set LC_CTYPE, LC_COLLATE, LC_NUMERIC to C
x86: Increase MAX_EARLY_RES; insufficient on 32-bit NUMA
x86: Fix checking of SRAT when node 0 ram is not from 0
x86, cpuid: Add "volatile" to asm in native_cpuid()
x86, msr: msrs_alloc/free for CONFIG_SMP=n
x86, amd: Get multi-node CPU info from NodeId MSR instead of PCI config space
x86: Add IA32_TSC_AUX MSR and use it
x86, msr/cpuid: Register enough minors for the MSR and CPUID drivers
initramfs: add missing decompressor error check
bzip2: Add missing checks for malloc returning NULL
bzip2/lzma/gzip: pre-boot malloc doesn't return NULL on failure
Nobody except ptrace itself should use task->ptrace or PT_PTRACED
directly, change arch/s390/kernel/traps.c to use the helper.
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Roland McGrath <roland@redhat.com>
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
The elf notes number for the upper register halves is s390 specific.
Change the name of the elf notes to include S390.
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
26-bit ARM support was removed a long time ago, and this symbol has
been defined to be 'y' ever since. As it's never disabled anymore,
we can kill it without any side effects.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
