TWxSoftware/twx-linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5b85575ad4
twx-linux/kernel
History
Anton Protopopov 5b85575ad4 bpf: optimize hashmap lookups when key_size is divisible by 4 
The BPF hashmap uses the jhash() hash function. There is an optimized version
of this hash function which may be used if hash size is a multiple of 4. Apply
this optimization to the hashmap in a similar way as it is done in the bloom
filter map.

On practice the optimization is only noticeable for smaller key sizes, which,
however, is sufficient for many applications. An example is listed in the
following table of measurements (a hashmap of 65536 elements was used):

    --------------------------------------------------------------------
    | key_size | fullness | lookups /sec | lookups (opt) /sec |   gain |
    --------------------------------------------------------------------
    |        4 |      25% |      42.990M |            46.000M |   7.0% |
    |        4 |      50% |      37.910M |            39.094M |   3.1% |
    |        4 |      75% |      34.486M |            36.124M |   4.7% |
    |        4 |     100% |      31.760M |            32.719M |   3.0% |
    --------------------------------------------------------------------
    |        8 |      25% |      43.855M |            49.626M |  13.2% |
    |        8 |      50% |      38.328M |            42.152M |  10.0% |
    |        8 |      75% |      34.483M |            38.088M |  10.5% |
    |        8 |     100% |      31.306M |            34.686M |  10.8% |
    --------------------------------------------------------------------
    |       12 |      25% |      38.398M |            43.770M |  14.0% |
    |       12 |      50% |      33.336M |            37.712M |  13.1% |
    |       12 |      75% |      29.917M |            34.440M |  15.1% |
    |       12 |     100% |      27.322M |            30.480M |  11.6% |
    --------------------------------------------------------------------
    |       16 |      25% |      41.491M |            41.921M |   1.0% |
    |       16 |      50% |      36.206M |            36.474M |   0.7% |
    |       16 |      75% |      32.529M |            33.027M |   1.5% |
    |       16 |     100% |      29.581M |            30.325M |   2.5% |
    --------------------------------------------------------------------
    |       20 |      25% |      34.240M |            36.787M |   7.4% |
    |       20 |      50% |      30.328M |            32.663M |   7.7% |
    |       20 |      75% |      27.536M |            29.354M |   6.6% |
    |       20 |     100% |      24.847M |            26.505M |   6.7% |
    --------------------------------------------------------------------
    |       24 |      25% |      36.329M |            40.608M |  11.8% |
    |       24 |      50% |      31.444M |            35.059M |  11.5% |
    |       24 |      75% |      28.426M |            31.452M |  10.6% |
    |       24 |     100% |      26.278M |            28.741M |   9.4% |
    --------------------------------------------------------------------
    |       28 |      25% |      31.540M |            31.944M |   1.3% |
    |       28 |      50% |      27.739M |            28.063M |   1.2% |
    |       28 |      75% |      24.993M |            25.814M |   3.3% |
    |       28 |     100% |      23.513M |            23.500M |  -0.1% |
    --------------------------------------------------------------------
    |       32 |      25% |      32.116M |            33.953M |   5.7% |
    |       32 |      50% |      28.879M |            29.859M |   3.4% |
    |       32 |      75% |      26.227M |            26.948M |   2.7% |
    |       32 |     100% |      23.829M |            24.613M |   3.3% |
    --------------------------------------------------------------------
    |       64 |      25% |      22.535M |            22.554M |   0.1% |
    |       64 |      50% |      20.471M |            20.675M |   1.0% |
    |       64 |      75% |      19.077M |            19.146M |   0.4% |
    |       64 |     100% |      17.710M |            18.131M |   2.4% |
    --------------------------------------------------------------------

The following script was used to gather the results (SMT & frequency off):

    cd tools/testing/selftests/bpf
    for key_size in 4 8 12 16 20 24 28 32 64; do
            for nr_entries in `seq 16384 16384 65536`; do
                    fullness=$(printf '%3s' $((nr_entries*100/65536)))
                    echo -n "key_size=$key_size: $fullness% full: "
                    sudo ./bench -d2 -a bpf-hashmap-lookup --key_size=$key_size --nr_entries=$nr_entries --max_entries=65536 --nr_loops=2000000 --map_flags=0x40 | grep cpu
            done
            echo
    done

Signed-off-by: Anton Protopopov <aspsk@isovalent.com>
Link: https://lore.kernel.org/r/20230401200602.3275-1-aspsk@isovalent.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2023-04-01 15:08:19 -07:00
..
bpf bpf: optimize hashmap lookups when key_size is divisible by 4 2023-04-01 15:08:19 -07:00
cgroup cgroup: Make current_cgns_cgroup_dfl() safe to call after exit_task_namespace() 2023-03-14 16:56:20 -07:00
configs mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED 2022-12-01 00:09:20 +01:00
debug kdb: use srcu console list iterator 2022-12-02 11:25:00 +01:00
dma swiotlb: mark swiotlb_memblock_alloc() as __init 2023-02-22 06:44:48 -08:00
entry entry: kmsan: introduce kmsan_unpoison_entry_regs() 2022-10-03 14:03:25 -07:00
events Char/Misc and other driver subsystem changes for 6.3-rc1 2023-02-24 12:47:33 -08:00
futex - Prevent the leaking of a debug timer in futex_waitv() 2023-01-01 11:15:05 -08:00
gcov gcov: add support for checksum field 2022-12-21 14:31:52 -08:00
irq iommufd for 6.3 2023-02-24 14:34:12 -08:00
kcsan kcsan: test: don't put the expect array on the stack 2023-01-02 08:59:33 -08:00
livepatch Livepatching changes for 6.3 2023-02-23 14:00:10 -08:00
locking RCU pull request for v6.3 2023-02-21 10:45:51 -08:00
module kallsyms, bpf: Move find_kallsyms_symbol_value out of internal header 2023-03-17 13:45:51 +01:00
power Merge branches 'powercap', 'pm-domains', 'pm-em' and 'pm-opp' 2023-02-15 20:06:26 +01:00
printk printk changes for 6.3 2023-02-23 13:49:45 -08:00
rcu Merge branch 'stall.2023.01.09a' into HEAD 2023-02-02 16:40:07 -08:00
sched - Daniel Verkamp has contributed a memfd series ("mm/memfd: add 2023-02-23 17:09:35 -08:00
time Updates for timekeeping, timers and clockevent/source drivers: 2023-02-21 09:45:13 -08:00
trace bpf-next-for-netdev 2023-03-06 20:36:39 -08:00
.gitignore
acct.c acct: fix potential integer overflow in encode_comp_t() 2022-11-30 16:13:18 -08:00
async.c
audit_fsnotify.c audit: fix potential double free on error path from fsnotify_add_inode_mark 2022-08-22 18:50:06 -04:00
audit_tree.c
audit_watch.c audit_init_parent(): constify path 2022-09-01 17:39:30 -04:00
audit.c audit: use time_after to compare time 2022-08-29 19:47:03 -04:00
audit.h audit: remove selinux_audit_rule_update() declaration 2022-09-07 11:30:15 -04:00
auditfilter.c
auditsc.c \n 2023-02-20 12:38:27 -08:00
backtracetest.c
bounds.c mm: multi-gen LRU: minimal implementation 2022-09-26 19:46:09 -07:00
capability.c fs: port i_{g,u}id_into_vfs{g,u}id() to mnt_idmap 2023-01-19 09:24:29 +01:00
cfi.c cfi: Switch to -fsanitize=kcfi 2022-09-26 10:13:13 -07:00
compat.c
configs.c
context_tracking.c context_tracking: Fix noinstr vs KASAN 2023-01-13 11:48:18 +01:00
cpu_pm.c cpuidle, cpu_pm: Remove RCU fiddling from cpu_pm_{enter,exit}() 2023-01-13 11:48:15 +01:00
cpu.c cpu/hotplug: Do not bail-out in DYING/STARTING sections 2022-12-02 12:43:02 +01:00
crash_core.c mm: remove 'First tail page' members from struct page 2023-02-02 22:32:59 -08:00
crash_dump.c
cred.c cred: Do not default to init_cred in prepare_kernel_cred() 2022-11-01 10:04:52 -07:00
delayacct.c delayacct: support re-entrance detection of thrashing accounting 2022-09-26 19:46:07 -07:00
dma.c
exec_domain.c
exit.c arm64 updates for 6.3: 2023-02-21 15:27:48 -08:00
extable.c
fail_function.c kernel/fail_function: fix memory leak with using debugfs_lookup() 2023-02-08 13:36:22 +01:00
fork.c - Daniel Verkamp has contributed a memfd series ("mm/memfd: add 2023-02-23 17:09:35 -08:00
freezer.c freezer,sched: Rewrite core freezer logic 2022-09-07 21:53:50 +02:00
gen_kheaders.sh kheaders: use standard naming for the temporary directory 2023-01-22 23:43:34 +09:00
groups.c
hung_task.c hung_task: print message when hung_task_warnings gets down to zero. 2023-02-09 17:03:20 -08:00
iomem.c
irq_work.c
jump_label.c jump_label: Prevent key->enabled int overflow 2022-12-01 15:53:05 -08:00
kallsyms_internal.h kallsyms: Reduce the memory occupied by kallsyms_seqs_of_names[] 2022-11-12 18:47:36 -08:00
kallsyms_selftest.c kallsyms: Fix scheduling with interrupts disabled in self-test 2023-01-13 15:09:08 -08:00
kallsyms_selftest.h kallsyms: Add self-test facility 2022-11-15 00:42:02 -08:00
kallsyms.c kallsyms: Add self-test facility 2022-11-15 00:42:02 -08:00
kcmp.c
Kconfig.freezer
Kconfig.hz
Kconfig.locks
Kconfig.preempt
kcov.c mm: replace vma->vm_flags direct modifications with modifier calls 2023-02-09 16:51:39 -08:00
kexec_core.c There is no particular theme here - mainly quick hits all over the tree. 2023-02-23 17:55:40 -08:00
kexec_elf.c
kexec_file.c kexec: introduce sysctl parameters kexec_load_limit_* 2023-02-02 22:50:05 -08:00
kexec_internal.h panic, kexec: make __crash_kexec() NMI safe 2022-09-11 21:55:06 -07:00
kexec.c kexec: introduce sysctl parameters kexec_load_limit_* 2023-02-02 22:50:05 -08:00
kheaders.c
kmod.c
kprobes.c x86/kprobes: Fix arch_check_optimized_kprobe check within optimized_kprobe range 2023-02-21 08:49:16 +09:00
ksysfs.c kernels/ksysfs.c: export kernel address bits 2023-01-20 14:30:45 +01:00
kthread.c kthread_worker: check all delayed works when destroy kthread worker 2023-02-02 22:50:02 -08:00
latencytop.c latencytop: use the last element of latency_record of system 2022-09-11 21:55:12 -07:00
Makefile kernel hardening fixes for v6.2-rc1 2022-12-23 12:00:24 -08:00
module_signature.c
notifier.c kernel/notifier: Remove CONFIG_SRCU 2023-02-02 16:26:06 -08:00
nsproxy.c fs/exec: switch timens when a task gets a new mm 2022-10-25 15:15:52 -07:00
padata.c Kbuild updates for v6.2 2022-12-19 12:33:32 -06:00
panic.c cpuidle: lib/bug: Disable rcu_is_watching() during WARN/BUG 2023-01-31 15:01:45 +01:00
params.c kernel/params.c: Use kstrtobool() instead of strtobool() 2023-01-25 14:07:21 -08:00
pid_namespace.c - Daniel Verkamp has contributed a memfd series ("mm/memfd: add 2023-02-23 17:09:35 -08:00
pid_sysctl.h mm/memfd: add MFD_NOEXEC_SEAL and MFD_EXEC 2023-01-18 17:12:37 -08:00
pid.c
profile.c kernel/profile.c: simplify duplicated code in profile_setup() 2022-09-11 21:55:12 -07:00
ptrace.c rseq: Introduce extensible rseq ABI 2022-12-27 12:52:10 +01:00
range.c
reboot.c kernel/reboot: Add SYS_OFF_MODE_RESTART_PREPARE mode 2022-10-04 15:59:36 +02:00
regset.c
relay.c mm: replace vma->vm_flags direct modifications with modifier calls 2023-02-09 16:51:39 -08:00
resource_kunit.c
resource.c dax/kmem: Fix leak of memory-hotplug resources 2023-02-17 14:58:01 -08:00
rseq.c rseq: Extend struct rseq with per-memory-map concurrency ID 2022-12-27 12:52:12 +01:00
scftorture.c
scs.c scs: add support for dynamic shadow call stacks 2022-11-09 18:06:35 +00:00
seccomp.c seccomp: fix kernel-doc function name warning 2023-01-13 17:01:06 -08:00
signal.c sched: Introduce per-memory-map concurrency ID 2022-12-27 12:52:11 +01:00
smp.c bitmap patches for v6.1-rc1 2022-10-10 12:49:34 -07:00
smpboot.c smpboot: use atomic_try_cmpxchg in cpu_wait_death and cpu_report_death 2022-09-11 21:55:10 -07:00
smpboot.h
softirq.c
stackleak.c
stacktrace.c
static_call_inline.c static_call: Add call depth tracking support 2022-10-17 16:41:16 +02:00
static_call.c
stop_machine.c
sys_ni.c kernel/sys_ni: add compat entry for fadvise64_64 2022-08-20 15:17:45 -07:00
sys.c - Daniel Verkamp has contributed a memfd series ("mm/memfd: add 2023-02-23 17:09:35 -08:00
sysctl-test.c kernel/sysctl-test: use SYSCTL_{ZERO/ONE_HUNDRED} instead of i_{zero/one_hundred} 2022-09-08 16:56:45 -07:00
sysctl.c sysctl: fix proc_dobool() usability 2023-02-21 13:34:07 -08:00
task_work.c task_work: use try_cmpxchg in task_work_add, task_work_cancel_match and task_work_run 2022-09-11 21:55:10 -07:00
taskstats.c genetlink: start to validate reserved header bytes 2022-08-29 12:47:15 +01:00
torture.c torture: Fix hang during kthread shutdown phase 2023-01-05 12:10:35 -08:00
tracepoint.c tracepoint: Allow livepatch module add trace event 2023-02-18 14:34:36 -05:00
tsacct.c
ucount.c
uid16.c
uid16.h
umh.c freezer,umh: Fix call_usermode_helper_exec() vs SIGKILL 2023-02-13 16:36:14 +01:00
up.c
user_namespace.c userns: fix a struct's kernel-doc notation 2023-02-02 22:50:04 -08:00
user-return-notifier.c
user.c kernel/user: Allow user_struct::locked_vm to be usable for iommufd 2022-11-30 20:16:49 -04:00
usermode_driver.c
utsname_sysctl.c kernel/utsname_sysctl.c: Fix hostname polling 2022-10-23 12:01:01 -07:00
utsname.c
watch_queue.c
watchdog_hld.c
watchdog.c
workqueue_internal.h
workqueue.c workqueue: Fold rebind_worker() within rebind_workers() 2023-01-13 07:50:40 -10:00