twx-linux/lib
Kairui Song 722e9e5acc mm/filemap: optimize filemap folio adding
commit 6758c1128ceb45d1a35298912b974eb4895b7dd9 upstream.

Instead of doing multiple tree walks, do one optimism range check with
lock hold, and exit if raced with another insertion.  If a shadow exists,
check it with a new xas_get_order helper before releasing the lock to
avoid redundant tree walks for getting its order.

Drop the lock and do the allocation only if a split is needed.

In the best case, it only need to walk the tree once.  If it needs to
alloc and split, 3 walks are issued (One for first ranged conflict check
and order retrieving, one for the second check after allocation, one for
the insert after split).

Testing with 4K pages, in an 8G cgroup, with 16G brd as block device:

  echo 3 > /proc/sys/vm/drop_caches

  fio -name=cached --numjobs=16 --filename=/mnt/test.img \
    --buffered=1 --ioengine=mmap --rw=randread --time_based \
    --ramp_time=30s --runtime=5m --group_reporting

Before:
bw (  MiB/s): min= 1027, max= 3520, per=100.00%, avg=2445.02, stdev=18.90, samples=8691
iops        : min=263001, max=901288, avg=625924.36, stdev=4837.28, samples=8691

After (+7.3%):
bw (  MiB/s): min=  493, max= 3947, per=100.00%, avg=2625.56, stdev=25.74, samples=8651
iops        : min=126454, max=1010681, avg=672142.61, stdev=6590.48, samples=8651

Test result with THP (do a THP randread then switch to 4K page in hope it
issues a lot of splitting):

  echo 3 > /proc/sys/vm/drop_caches

  fio -name=cached --numjobs=16 --filename=/mnt/test.img \
      --buffered=1 --ioengine=mmap -thp=1 --readonly \
      --rw=randread --time_based --ramp_time=30s --runtime=10m \
      --group_reporting

  fio -name=cached --numjobs=16 --filename=/mnt/test.img \
      --buffered=1 --ioengine=mmap \
      --rw=randread --time_based --runtime=5s --group_reporting

Before:
bw (  KiB/s): min= 4141, max=14202, per=100.00%, avg=7935.51, stdev=96.85, samples=18976
iops        : min= 1029, max= 3548, avg=1979.52, stdev=24.23, samples=18976·

READ: bw=4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=64.0KiB (65.5kB), run=14419-14419msec

After (+12.5%):
bw (  KiB/s): min= 4611, max=15370, per=100.00%, avg=8928.74, stdev=105.17, samples=19146
iops        : min= 1151, max= 3842, avg=2231.27, stdev=26.29, samples=19146

READ: bw=4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=64.0KiB (65.5kB), run=14137-14137msec

The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read.

Link: https://lkml.kernel.org/r/20240415171857.19244-5-ryncsn@gmail.com
Signed-off-by: Kairui Song <kasong@tencent.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Closes: https://lore.kernel.org/linux-mm/A5A976CB-DB57-4513-A700-656580488AB6@flyingcircus.io/
[ kasong@tencent.com: minor adjustment of variable declarations ]
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-10-04 16:30:02 +02:00
..
842
crypto crypto: lib/mpi - Fix unexpected pointer access in mpi_ec_init 2024-01-31 16:18:49 -08:00
dim linux/dim: Do nothing if no time delta between samples 2023-05-09 11:06:45 +02:00
fonts
kunit kunit: Fix timeout message 2024-07-11 12:49:08 +02:00
livepatch
lz4
lzo
math bitmap: introduce generic optimized bitmap_size() 2024-08-29 17:33:14 +02:00
pldmfw lib: remove MODULE_LICENSE in non-modules 2023-04-13 13:13:53 -07:00
raid6 raid6: Add LoongArch SIMD recovery implementation 2023-09-06 22:53:55 +08:00
reed_solomon
test_fortify
vdso vdso: Improve cmd_vdso_check to check all dynamic relocations 2023-03-21 21:15:34 +01:00
xz xz: cleanup CRC32 edits from 2018 2024-10-04 16:29:19 +02:00
zlib_deflate lib/zlib: DFLTCC deflate does not write all available bits for Z_NO_FLUSH 2023-02-27 17:00:14 -08:00
zlib_dfltcc
zlib_inflate
zstd zstd: Fix array-index-out-of-bounds UBSAN warning 2023-12-13 18:44:58 +01:00
.gitignore
argv_split.c argv_split: fix kernel-doc warnings 2023-09-19 13:21:33 -07:00
ashldi3.c
ashrdi3.c
asn1_decoder.c
asn1_encoder.c
assoc_array.c
atomic64_test.c
atomic64.c
audit.c
base64.c
bcd.c
bch.c lib/bch.c: use bitrev instead of internal logic 2023-08-18 10:18:58 -07:00
bitfield_kunit.c
bitmap.c lib/bitmap: drop optimization of bitmap_{from,to}_arr64 2023-06-22 13:57:41 -07:00
bitrev.c
bootconfig-data.S
bootconfig.c bootconfig: use memblock_free_late to free xbc memory to buddy 2024-04-27 17:11:43 +02:00
bsearch.c
btree.c btree: remove MODULE_LICENSE in non-modules 2023-04-13 13:13:54 -07:00
bucket_locks.c
bug.c
build_OID_registry lib/build_OID_registry: don't mention the full path of the script in output 2024-08-03 08:54:32 +02:00
buildid.c ELF: fix all "Elf" typos 2023-04-08 13:45:37 -07:00
bust_spinlocks.c
check_signature.c
checksum_kunit.c kunit: Fix checksum tests on big endian CPUs 2023-08-25 10:14:34 +01:00
checksum.c
clz_ctz.c lib/clz_ctz.c: Fix __clzdi2() and __ctzdi2() for 32-bit kernels 2023-08-25 13:22:10 -07:00
clz_tab.c
cmdline_kunit.c lib/cmdline: Fix an invalid format specifier in an assertion msg 2024-03-26 18:19:18 -04:00
cmdline.c
cmpdi2.c
compat_audit.c
cpu_rmap.c lib: cpu_rmap: Fix potential use-after-free in irq_cpu_rmap_release() 2023-06-07 21:25:00 -07:00
cpumask_kunit.c cpumask: re-introduce constant-sized cpumask optimizations 2023-03-05 14:30:34 -08:00
cpumask.c sched/topology: Handle NUMA_NO_NODE in sched_numa_find_nth_cpu() 2024-08-29 17:33:24 +02:00
crc4.c
crc7.c
crc8.c
crc16.c
crc32.c
crc32defs.h
crc32test.c
crc64-rocksoft.c
crc64.c
crc-ccitt.c
crc-itu-t.c
crc-t10dif.c
ctype.c
debug_info.c
debug_locks.c
debugobjects.c debugobjects: Fix conditions in fill_pool() 2024-10-04 16:29:53 +02:00
dec_and_lock.c
decompress_bunzip2.c decompress_bunzip2: fix rare decompression failure 2024-08-03 08:54:24 +02:00
decompress_inflate.c decompressor: provide missing prototypes 2023-06-09 17:44:17 -07:00
decompress_unlz4.c
decompress_unlzma.c
decompress_unlzo.c
decompress_unxz.c decompressor: provide missing prototypes 2023-06-09 17:44:17 -07:00
decompress_unzstd.c decompressor: provide missing prototypes 2023-06-09 17:44:17 -07:00
decompress.c
devmem_is_allowed.c lib: devmem_is_allowed: include linux/io.h 2023-06-09 17:44:15 -07:00
devres.c devres: show which resource was invalid in __devm_ioremap_resource() 2023-06-23 17:04:04 -07:00
dhry_1.c lib: dhry: fix sleeping allocations inside non-preemptable section 2023-07-08 09:29:32 -07:00
dhry_2.c
dhry_run.c lib: dhry: fix unstable smp_processor_id(_) usage 2023-03-23 17:18:35 -07:00
dhry.h
digsig.c
dump_stack.c
dynamic_debug.c dyndbg: fix old BUG_ON in >control parser 2024-05-17 12:02:33 +02:00
dynamic_queue_limits.c
earlycpio.c
errname.c parisc: Drop the HP-UX ENOSYM and EREMOTERELEASE error codes 2023-12-08 08:52:20 +01:00
error-inject.c lib: error-inject: remove error checking for debugfs_create_dir() 2023-08-18 10:18:55 -07:00
errseq.c
extable.c
fault-inject-usercopy.c
fault-inject.c fault-inject: allow configuration via configfs 2023-04-13 07:38:54 -06:00
fdt_addresses.c
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
fdt.c
find_bit_benchmark.c
find_bit.c cpumask: introduce for_each_cpu_or 2023-03-19 10:02:04 -07:00
flex_proportions.c
fortify_kunit.c kunit/fortify: Fix mismatched kvalloc()/vfree() usage 2024-06-12 11:11:34 +02:00
gen_crc32table.c
gen_crc64table.c
genalloc.c Devicetree include cleanups for v6.6: 2023-08-30 17:04:28 -07:00
generic-radix-tree.c lib/generic-radix-tree.c: Fix rare race in __genradix_ptr_alloc() 2024-09-12 11:11:39 +02:00
glob.c
globtest.c
group_cpus.c lib/group_cpus.c: avoid acquiring cpu hotplug lock in group_cpus_evenly 2023-12-13 18:45:22 +01:00
hashtable_test.c
hexdump.c
hweight.c
idr.c ida: Fix crash in ida_free when the bitmap is empty 2024-01-20 11:51:46 +01:00
inflate.c
interval_tree_test.c
interval_tree.c
iomap_copy.c
iomap.c
iommu-helper.c
iov_iter.c iov_iter: Fix iov_iter_extract_pages() with zero-sized entries 2023-09-09 15:11:49 -07:00
irq_poll.c
irq_regs.c
is_signed_type_kunit.c
is_single_threaded.c
kasprintf.c
Kconfig cpumask: limit FORCE_NR_CPUS to just the UP case 2024-07-25 09:50:47 +02:00
Kconfig.debug bpf, kconfig: Fix DEBUG_INFO_BTF_MODULES Kconfig definition 2024-05-17 12:01:59 +02:00
Kconfig.kasan kasan: treat meminstrinsic as builtins in uninstrumented files 2023-03-02 21:54:22 -08:00
Kconfig.kcsan Kernel concurrency sanitizer (KCSAN) updates for v6.3 2023-02-25 13:02:20 -08:00
Kconfig.kfence
Kconfig.kgdb
Kconfig.kmsan
Kconfig.ubsan ubsan: Clarify Kconfig text for CONFIG_UBSAN_TRAP 2023-07-17 16:05:19 -07:00
kfifo.c
klist.c
kobject_uevent.c kobject_uevent: Fix OOB access within zap_modalias_env() 2024-08-03 08:54:24 +02:00
kobject.c Revert "kobject: Remove redundant checks for whether ktype is NULL" 2024-02-23 09:24:58 +01:00
kstrtox.c kstrtox: consistently use _tolower() 2023-08-21 13:46:25 -07:00
kstrtox.h
kunit_iov_iter.c iov_iter: Kunit tests for page extraction 2023-09-09 15:11:49 -07:00
libcrc32c.c libcrc32c: remove crc32c_impl 2023-04-17 18:01:23 +02:00
linear_ranges.c
list_debug.c list: Introduce CONFIG_LIST_HARDENED 2023-08-15 14:57:25 -07:00
list_sort.c
list-test.c list: test: Test the klist structure 2023-03-31 09:21:35 -06:00
llist.c
locking-selftest-hardirq.h
locking-selftest-mutex.h
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h
locking-selftest-rsem.h
locking-selftest-rtmutex.h
locking-selftest-softirq.h
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h
locking-selftest-wsem.h
locking-selftest.c lockdep/selftests: Use SBRM APIs for wait context tests 2023-07-26 12:29:13 +02:00
lockref.c
logic_iomem.c
logic_pio.c minmax: add in_range() macro 2023-08-24 16:20:18 -07:00
lru_cache.c
lshrdi3.c
Makefile iov_iter: Kunit tests for copying to/from an iterator 2023-09-09 15:11:49 -07:00
maple_tree.c maple_tree: fix mas_empty_area_rev() null pointer dereference 2024-05-17 12:02:30 +02:00
memcat_p.c
memcpy_kunit.c lib: memcpy_kunit: Fix an invalid format specifier in an assertion msg 2024-03-26 18:19:18 -04:00
memory-notifier-error-inject.c
memregion.c
memweight.c
muldi3.c
net_utils.c mac_pton: Clean up the header inclusions 2023-06-06 13:18:32 +02:00
netdev-notifier-error-inject.c
nlattr.c netlink: add nla be16/32 types to minlen array 2024-03-06 14:48:34 +00:00
nmi_backtrace.c nmi_backtrace: allow excluding an arbitrary CPU 2023-08-18 10:19:00 -07:00
notifier-error-inject.c lib: remove error checking for debugfs_create_dir() 2023-08-18 10:18:55 -07:00
notifier-error-inject.h
objagg.c mlxsw: spectrum_acl_erp: Fix object nesting warning 2024-08-03 08:53:38 +02:00
of-reconfig-notifier-error-inject.c
oid_registry.c
once.c
overflow_kunit.c overflow: Add struct_size_t() helper 2023-05-26 13:52:19 -07:00
packing.c lib: packing: remove MODULE_LICENSE in non-modules 2023-03-09 23:08:04 -08:00
parman.c
parser.c lib: parser: update documentation for match_NUMBER functions 2023-03-02 21:54:22 -08:00
pci_iomap.c pci_iounmap(): Fix MMIO mapping leak 2024-04-03 15:28:17 +02:00
percpu_counter.c pcpcntr: add group allocation/free 2023-08-25 08:06:53 -07:00
percpu_test.c
percpu-refcount.c
plist.c
pm-notifier-error-inject.c
polynomial.c
radix-tree.c radix tree: remove unused variable 2023-08-21 13:07:22 -07:00
radix-tree.h radix-tree: move declarations to header 2023-06-12 11:31:50 -07:00
random32.c
ratelimit.c
rbtree_test.c
rbtree.c lib/rbtree: use '+' instead of '|' for setting color. 2023-04-18 16:39:33 -07:00
rcuref.c atomics: Provide rcuref - scalable reference counting 2023-03-28 10:39:29 +02:00
ref_tracker.c lib/ref_tracker: remove warnings in case of allocation failure 2023-06-05 15:28:42 -07:00
refcount.c
rhashtable.c
sbitmap.c lib/sbitmap: define swap_lock as raw_spinlock_t 2024-10-04 16:29:38 +02:00
scatterlist.c Fix a potential infinite loop in extract_user_to_sg() 2024-05-17 12:02:02 +02:00
seq_buf.c seq_buf: Add seq_buf_do_printk() helper 2023-04-25 21:03:14 -04:00
sg_pool.c
sg_split.c
siphash_kunit.c
siphash.c
slub_kunit.c mm/slub, kunit: Use inverted data to corrupt kmem cache 2024-06-12 11:11:31 +02:00
smp_processor_id.c
sort.c
stackdepot.c stackdepot: respect __GFP_NOLOCKDEP allocation flag 2024-05-02 16:32:46 +02:00
stackinit_kunit.c kernel/range: Uplevel the cxl subsystem's range_contains() helper 2023-02-10 17:32:37 -08:00
stmp_device.c
strcat_kunit.c string: Add Kunit tests for strcat() family 2023-05-16 14:08:02 -07:00
string_helpers.c lib/string_helpers: Add kstrdup_and_replace() helper 2023-08-04 18:21:50 -07:00
string.c string: use __builtin_memcpy() in strlcpy/strlcat 2023-06-01 11:24:50 -07:00
strncpy_from_user.c
strnlen_user.c
strscpy_kunit.c
syscall.c
test_bitmap.c lib/bitmap: workaround const_eval test build failure 2023-07-18 13:25:37 -07:00
test_bitops.c
test_bits.c
test_blackhole_dev.c net: blackhole_dev: fix build warning for ethh set but not used 2024-03-26 18:19:26 -04:00
test_bpf.c bpf/tests: Enhance output on error and fix typos 2023-08-18 17:08:42 +02:00
test_debug_virtual.c
test_dynamic_debug.c
test_firmware.c test_firmware: return ENOMEM instead of ENOSPC on failed memory allocation 2023-06-15 13:42:18 +02:00
test_fprobe.c fprobe: Pass return address to the handlers 2023-06-06 21:39:55 +09:00
test_fpu.c
test_free_pages.c
test_hash.c
test_hexdump.c
test_hmm_uapi.h
test_hmm.c lib/test_hmm.c: handle src_pfns and dst_pfns allocation failure 2024-06-12 11:12:08 +02:00
test_ida.c ida: Fix crash in ida_free when the bitmap is empty 2024-01-20 11:51:46 +01:00
test_kmod.c
test_kprobes.c test_kprobes: Add recursed kprobe test case 2023-02-21 08:52:42 +09:00
test_linear_ranges.c
test_list_sort.c
test_lockup.c
test_maple_tree.c maple_tree: add GFP_KERNEL to allocations in mas_expected_entries() 2023-10-18 12:12:41 -07:00
test_memcat_p.c
test_meminit.c mm, treewide: introduce NR_PAGE_ORDERS 2024-05-02 16:32:41 +02:00
test_min_heap.c
test_module.c
test_objagg.c
test_parman.c
test_printf.c lib/vsprintf: declare no_hash_pointers in sprintf.h 2023-08-21 13:46:24 -07:00
test_ref_tracker.c lib/ref_tracker: improve printing stats 2023-06-05 15:28:42 -07:00
test_rhashtable.c
test_scanf.c lib: test_scanf: Add explicit type cast to result initialization in test_number_prefix() 2023-08-16 11:47:29 +02:00
test_sort.c
test_static_key_base.c
test_static_keys.c
test_string.c
test_sysctl.c test_sysclt: Test for registering a mount point 2023-06-18 02:32:54 -07:00
test_ubsan.c
test_user_copy.c
test_uuid.c
test_vmalloc.c lib/test_vmalloc.c: avoid garbage in page array 2023-06-12 11:31:51 -07:00
test_xarray.c mm/filemap: optimize filemap folio adding 2024-10-04 16:30:02 +02:00
test-kstrtox.c
test-string_helpers.c lib/test-string_helpers: replace UNESCAPE_ANY by UNESCAPE_ALL_MASK 2023-04-08 13:45:39 -07:00
textsearch.c
timerqueue.c
trace_readwrite.c
ts_bm.c lib/ts_bm: add helper to reduce indentation and improve readability 2023-07-27 13:45:51 +02:00
ts_fsm.c
ts_kmp.c
ubsan.c ubsan: add prototypes for internal functions 2023-05-30 16:42:01 -07:00
ubsan.h ubsan: add prototypes for internal functions 2023-05-30 16:42:01 -07:00
ucmpdi2.c
ucs2_string.c
usercopy.c uaccess: Add speculation barrier to copy_from_user() 2023-02-21 14:45:22 -08:00
uuid.c
vsprintf.c lib/vsprintf: Fix %pfwf when current node refcount == 0 2024-01-01 12:42:46 +00:00
win_minmax.c
xarray.c lib/xarray: introduce a new helper xas_get_order 2024-10-04 16:30:02 +02:00
xxhash.c