Adrian Huang
f9a18889aa
mm/vmalloc: combine all TLB flush operations of KASAN shadow virtual address into one operation
commit 9e9e085eff upstream.
When compiling kernel source 'make -j $(nproc)' with the up-and-running
KASAN-enabled kernel on a 256-core machine, the following soft lockup is
shown:
watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [kworker/28:1:1760]
CPU: 28 PID: 1760 Comm: kworker/28:1 Kdump: loaded Not tainted 6.10.0-rc5 #95
Workqueue: events drain_vmap_area_work
RIP: 0010:smp_call_function_many_cond+0x1d8/0xbb0
Code: 38 c8 7c 08 84 c9 0f 85 49 08 00 00 8b 45 08 a8 01 74 2e 48 89 f1 49 89 f7 48 c1 e9 03 41 83 e7 07 4c 01 e9 41 83 c7 03 f3 90 <0f> b6 01 41 38 c7 7c 08 84 c0 0f 85 d4 06 00 00 8b 45 08 a8 01 75
RSP: 0018:ffffc9000cb3fb60 EFLAGS: 00000202
RAX: 0000000000000011 RBX: ffff8883bc4469c0 RCX: ffffed10776e9949
RDX: 0000000000000002 RSI: ffff8883bb74ca48 RDI: ffffffff8434dc50
RBP: ffff8883bb74ca40 R08: ffff888103585dc0 R09: ffff8884533a1800
R10: 0000000000000004 R11: ffffffffffffffff R12: ffffed1077888d39
R13: dffffc0000000000 R14: ffffed1077888d38 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ffff8883bc400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005577b5c8d158 CR3: 0000000004850000 CR4: 0000000000350ef0
Call Trace:
<IRQ>
? watchdog_timer_fn+0x2cd/0x390
? __pfx_watchdog_timer_fn+0x10/0x10
? __hrtimer_run_queues+0x300/0x6d0
? sched_clock_cpu+0x69/0x4e0
? __pfx___hrtimer_run_queues+0x10/0x10
? srso_return_thunk+0x5/0x5f
? ktime_get_update_offsets_now+0x7f/0x2a0
? srso_return_thunk+0x5/0x5f
? srso_return_thunk+0x5/0x5f
? hrtimer_interrupt+0x2ca/0x760
? __sysvec_apic_timer_interrupt+0x8c/0x2b0
? sysvec_apic_timer_interrupt+0x6a/0x90
</IRQ>
<TASK>
? asm_sysvec_apic_timer_interrupt+0x16/0x20
? smp_call_function_many_cond+0x1d8/0xbb0
? __pfx_do_kernel_range_flush+0x10/0x10
on_each_cpu_cond_mask+0x20/0x40
flush_tlb_kernel_range+0x19b/0x250
? srso_return_thunk+0x5/0x5f
? kasan_release_vmalloc+0xa7/0xc0
purge_vmap_node+0x357/0x820
? __pfx_purge_vmap_node+0x10/0x10
__purge_vmap_area_lazy+0x5b8/0xa10
drain_vmap_area_work+0x21/0x30
process_one_work+0x661/0x10b0
worker_thread+0x844/0x10e0
? srso_return_thunk+0x5/0x5f
? __kthread_parkme+0x82/0x140
? __pfx_worker_thread+0x10/0x10
kthread+0x2a5/0x370
? __pfx_kthread+0x10/0x10
ret_from_fork+0x30/0x70
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Debugging Analysis:
1. The following ftrace log shows that the lockup CPU spends too much
time iterating vmap_nodes and flushing TLB when purging vm_area
structures. (Some info is trimmed).
kworker: funcgraph_entry: | drain_vmap_area_work() {
kworker: funcgraph_entry: | mutex_lock() {
kworker: funcgraph_entry: 1.092 us | __cond_resched();
kworker: funcgraph_exit: 3.306 us | }
... ...
kworker: funcgraph_entry: | flush_tlb_kernel_range() {
... ...
kworker: funcgraph_exit: # 7533.649 us | }
... ...
kworker: funcgraph_entry: 2.344 us | mutex_unlock();
kworker: funcgraph_exit: $ 23871554 us | }
The drain_vmap_area_work() spends over 23 seconds.
There are 2805 flush_tlb_kernel_range() calls in the ftrace log.
* One is called in __purge_vmap_area_lazy().
* Others are called by purge_vmap_node->kasan_release_vmalloc.
purge_vmap_node() iteratively releases kasan vmalloc
allocations and flushes TLB for each vmap_area.
- [Rough calculation] Each flush_tlb_kernel_range() runs
about 7.5ms.
-- 2804 * 7.5ms = 21.03 seconds.
-- That's why a soft lock is triggered.
2. Extending the soft lockup time can work around the issue (For example,
# echo 60 > /proc/sys/kernel/watchdog_thresh). This confirms the
above-mentioned speculation: drain_vmap_area_work() spends too much
time.
If we combine all TLB flush operations of the KASAN shadow virtual
address into one operation in the call path
'purge_vmap_node()->kasan_release_vmalloc()', the running time of
drain_vmap_area_work() can be saved greatly. The idea is from the
flush_tlb_kernel_range() call in __purge_vmap_area_lazy(). And, the
soft lockup won't be triggered.
Here is the test result based on 6.10:
[6.10 wo/ the patch]
1. ftrace latency profiling (record a trace if the latency > 20s).
echo 20000000 > /sys/kernel/debug/tracing/tracing_thresh
echo drain_vmap_area_work > /sys/kernel/debug/tracing/set_graph_function
echo function_graph > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
2. Run `make -j $(nproc)` to compile the kernel source
3. Once the soft lockup is reproduced, check the ftrace log:
cat /sys/kernel/debug/tracing/trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
76) $ 50412985 us | } /* __purge_vmap_area_lazy */
76) $ 50412997 us | } /* drain_vmap_area_work */
76) $ 29165911 us | } /* __purge_vmap_area_lazy */
76) $ 29165926 us | } /* drain_vmap_area_work */
91) $ 53629423 us | } /* __purge_vmap_area_lazy */
91) $ 53629434 us | } /* drain_vmap_area_work */
91) $ 28121014 us | } /* __purge_vmap_area_lazy */
91) $ 28121026 us | } /* drain_vmap_area_work */
[6.10 w/ the patch]
1. Repeat step 1-2 in "[6.10 wo/ the patch]"
2. The soft lockup is not triggered and ftrace log is empty.
cat /sys/kernel/debug/tracing/trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3. Setting 'tracing_thresh' to 10/5 seconds does not get any ftrace
log.
4. Setting 'tracing_thresh' to 1 second gets ftrace log.
cat /sys/kernel/debug/tracing/trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
23) $ 1074942 us | } /* __purge_vmap_area_lazy */
23) $ 1074950 us | } /* drain_vmap_area_work */
The worst execution time of drain_vmap_area_work() is about 1 second.
Link: https://lore.kernel.org/lkml/ZqFlawuVnOMY2k3E@pc638.lan/
Link: https://lkml.kernel.org/r/20240726165246.31326-1-ahuang12@lenovo.com
Fixes: 282631cb24 ("mm: vmalloc: remove global purge_vmap_area_root rb-tree")
Signed-off-by: Adrian Huang <ahuang12@lenovo.com>
Co-developed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Tested-by: Jiwei Sun <sunjw10@lenovo.com>
Reviewed-by: Baoquan He <bhe@redhat.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-12-09 10:41:08 +01:00
..
2024-09-04 15:01:17 +01:00
2024-06-05 15:52:34 +02:00
2024-08-13 14:51:29 -07:00
2024-07-03 15:16:54 +02:00
2024-10-02 17:23:23 -04:00
2024-02-26 13:08:45 +02:00
2024-09-25 11:01:27 +02:00
2024-08-28 23:35:13 +05:30
2024-09-11 08:55:35 +00:00
2024-10-11 15:59:15 -07:00
2024-09-17 16:46:22 +02:00
2024-08-10 11:19:36 +01:00
2024-10-08 17:44:27 +02:00
2024-09-23 14:17:08 -07:00
2024-10-02 17:14:53 -07:00
2024-09-21 08:20:50 -07:00
2024-10-02 17:23:23 -04:00
2024-08-16 13:09:12 +01:00
2024-09-26 12:00:25 -07:00
2024-12-05 14:01:23 +01:00
2024-11-04 14:23:09 +01:00
2024-10-15 07:54:16 +02:00
2024-10-02 17:23:23 -04:00
2024-09-26 10:27:10 -07:00
2022-04-06 13:43:44 -07:00
2024-09-27 09:55:30 -07:00
2017-11-02 11:10:55 +01:00
2018-11-07 22:58:29 +01:00
2024-03-08 15:26:46 -06:00
2019-12-04 13:32:45 +10:00
2022-01-22 08:33:34 +02:00
2020-01-24 21:03:53 +01:00
2021-05-06 19:24:11 -07:00
2024-10-30 20:14:10 -07:00
2024-01-29 20:34:18 +01:00
2024-01-04 15:19:17 +01:00
2023-04-16 14:18:10 +02:00
2019-05-30 11:29:55 -07:00
2024-11-06 21:31:36 +01:00
2024-08-22 12:23:47 -04:00
2023-05-26 13:43:52 +02:00
2024-11-07 11:18:52 +00:00
2019-05-24 17:27:11 +02:00
2024-02-09 11:13:59 -10:00
2023-05-22 17:05:20 -04:00
2021-12-30 17:27:44 -08:00
2017-11-02 11:10:55 +01:00
2021-03-08 15:16:30 -08:00
2021-07-16 18:46:45 +02:00
2023-12-20 19:26:32 -05:00
2023-01-12 11:22:22 +01:00
2017-11-07 10:32:44 +01:00
2024-02-06 09:44:03 -07:00
2024-01-22 15:33:38 +01:00
2024-10-16 13:17:45 +02:00
2024-07-04 16:45:25 +01:00
2023-09-26 00:44:33 -06:00
2022-08-02 17:14:47 -06:00
2023-02-09 23:01:49 +01:00
2020-05-24 20:48:11 +02:00
2024-07-13 21:31:57 -07:00
2024-07-03 10:21:16 -06:00
2024-08-29 04:32:32 -06:00
2024-02-28 13:53:39 +02:00
2023-10-14 20:25:22 -07:00
2024-09-27 12:10:45 -07:00
2024-06-18 10:40:53 -07:00
2019-04-05 16:02:30 -10:00
2024-08-28 06:53:58 -07:00
2022-09-26 19:17:28 -06:00
2023-07-05 16:36:12 -06:00
2023-03-16 09:35:09 -06:00
2024-09-13 13:22:09 -06:00
2024-12-05 14:03:05 +01:00
2023-11-20 10:22:40 -07:00
2024-12-05 14:03:10 +01:00
2017-11-02 11:10:55 +01:00
2022-08-11 16:14:29 -07:00
2024-04-14 22:00:43 +09:00
2024-10-30 12:13:46 -07:00
2024-10-24 10:17:12 -07:00
2024-09-03 12:06:24 -07:00
2021-12-29 20:03:05 -08:00
2024-12-05 14:01:57 +01:00
2024-04-25 20:55:59 -07:00
2024-07-14 20:38:34 -07:00
2020-06-11 15:14:53 +02:00
2024-04-11 21:37:48 -04:00
2023-06-12 08:04:04 -06:00
2024-08-21 10:37:52 -07:00
2020-08-12 10:57:59 -07:00
2017-11-02 11:10:55 +01:00
2021-04-15 17:04:24 +01:00
2024-09-11 09:58:30 -07:00
2024-07-08 01:51:05 -06:00
2024-07-10 10:19:59 -07:00
2023-08-24 16:20:25 -07:00
2024-07-21 17:56:22 -07:00
2023-03-01 10:01:22 -08:00
2020-06-15 23:08:31 -05:00
2020-11-20 14:45:33 +11:00
2017-11-02 11:10:55 +01:00
2024-05-15 10:05:24 -04:00
2023-12-15 16:25:55 -08:00
2024-09-21 07:29:05 -07:00
2024-09-21 09:44:57 -07:00
2024-12-05 14:01:21 +01:00
2024-09-05 13:48:00 -07:00
2024-09-05 13:51:46 -07:00
2022-09-30 14:50:11 -07:00
2024-06-24 22:25:02 -07:00
2024-06-03 11:18:50 +02:00
2024-10-09 16:57:57 -04:00
2024-09-03 21:15:36 -07:00
2019-07-16 19:23:23 -07:00
2024-04-25 20:55:57 -07:00
2023-06-19 16:19:28 -07:00
2024-06-24 18:29:20 +02:00
2024-12-05 14:02:45 +01:00
2024-12-05 14:02:45 +01:00
2024-05-22 14:12:11 -07:00
2024-10-23 16:04:30 +01:00
2021-04-25 05:26:10 +09:00
2024-09-26 14:01:44 -07:00
2023-07-17 16:08:08 -07:00
2022-02-25 12:16:12 +01:00
2024-06-17 20:42:57 +02:00
2024-09-04 15:56:32 +02:00
2022-06-10 13:37:02 +02:00
2023-10-18 14:43:22 -07:00
2023-12-21 13:56:10 +01:00
2020-09-30 11:50:35 -07:00
2024-04-23 09:03:37 +09:00
2024-02-03 18:31:17 +01:00
2024-07-21 17:56:22 -07:00
2024-10-01 20:43:44 +02:00
2024-09-18 15:03:58 +02:00
2024-06-17 17:45:31 +02:00
2023-01-18 12:27:17 +01:00
2024-09-27 12:10:45 -07:00
2024-08-30 10:00:16 -10:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2021-05-06 19:24:12 -07:00
2019-06-19 17:09:06 +02:00
2024-07-10 13:19:50 -07:00
2023-04-17 18:01:23 +02:00
2018-07-27 19:04:33 +08:00
2022-03-07 12:48:35 -07:00
2023-12-29 12:22:26 -08:00
2023-12-15 14:19:48 -08:00
2023-10-27 18:04:29 +08:00
2019-05-30 11:29:53 -07:00
2021-12-23 22:35:00 +11:00
2024-09-09 16:39:00 -07:00
2024-02-22 15:27:19 -08:00
2019-05-24 17:27:11 +02:00
2024-07-15 11:39:44 -07:00
2023-03-18 12:23:33 +00:00
2024-09-27 08:18:43 -07:00
2021-07-08 11:48:20 -07:00
2023-06-09 17:44:16 -07:00
2023-04-18 16:39:34 -07:00
2020-09-29 17:50:10 +09:00
2023-05-29 23:23:04 +09:00
2024-08-21 13:12:12 +02:00
2024-10-25 16:07:03 -05:00
2024-05-14 11:43:40 +02:00
2024-07-03 15:16:54 +02:00
2024-06-25 17:15:06 -07:00
2024-07-10 15:38:14 +02:00
2020-04-18 15:44:54 -05:00
2024-06-11 12:57:49 -05:00
2019-06-19 17:09:55 +02:00
2024-02-20 14:22:55 -05:00
2024-02-20 14:22:51 -05:00
2024-04-15 13:56:32 -03:00
2024-08-22 06:18:00 +02:00
2020-09-25 06:12:25 +02:00
2024-08-30 11:41:05 -07:00
2024-07-23 09:52:23 +02:00
2024-09-22 18:47:51 +02:00
2024-08-29 07:22:49 +03:00
2024-06-21 15:40:55 +05:30
2018-03-16 19:58:27 +01:00
2024-04-26 11:57:36 +02:00
2019-12-03 11:20:37 +01:00
2023-07-10 14:36:12 +02:00
2024-08-26 19:21:14 -07:00
2023-01-29 15:18:33 -07:00
2017-11-02 11:10:55 +01:00
2022-02-23 19:45:40 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2024-01-18 09:48:40 -08:00
2019-06-05 17:36:37 +02:00
2024-08-27 12:22:18 +02:00
2024-07-03 15:16:54 +02:00
2023-08-23 14:17:42 -07:00
2017-11-02 11:10:55 +01:00
2024-04-15 11:02:51 -07:00
2021-04-25 05:25:42 +09:00
2020-06-04 19:06:25 -07:00
2020-04-18 15:44:54 -05:00
2024-07-29 07:33:10 +05:30
2022-03-10 16:51:50 -06:00
2024-09-03 21:15:38 -07:00
2019-10-17 16:23:25 +02:00
2021-08-26 15:32:28 -04:00
2017-11-02 11:10:55 +01:00
2018-01-01 12:40:27 -07:00
2024-10-02 17:23:23 -04:00
2024-08-12 14:16:24 +01:00
2023-11-28 14:08:46 +01:00
2021-08-20 11:39:26 +01:00
2024-04-09 17:14:57 -04:00
2024-05-14 00:31:43 -07:00
2019-10-17 16:44:36 +02:00
2020-03-19 07:41:01 +01:00
2023-07-24 04:39:16 +09:00
2024-12-05 14:02:27 +01:00
2024-08-28 16:53:57 +02:00
2023-02-07 12:53:53 +01:00
2020-10-16 11:11:22 -07:00
2024-09-01 20:43:33 -07:00
2024-09-19 10:18:15 +02:00
2022-07-07 10:50:46 +02:00
2019-05-30 11:26:32 -07:00
2021-05-28 17:44:36 +02:00
2017-11-02 11:10:55 +01:00
2019-05-30 11:26:32 -07:00
2024-09-29 21:52:29 -04:00
2019-06-19 17:09:55 +02:00
2021-07-27 11:00:36 +02:00
2024-09-23 09:35:36 -07:00
2023-01-19 09:24:27 +01:00
2024-08-30 08:22:33 +02:00
2024-08-18 10:07:59 -07:00
2019-07-26 14:51:57 +10:00
2024-09-12 22:30:37 +09:00
2019-06-05 17:36:37 +02:00
2024-06-07 23:14:45 +05:30
2022-11-16 15:53:33 -08:00
2019-06-24 09:16:47 +10:00
2024-10-01 17:01:40 +02:00
2024-03-16 08:29:48 +01:00
2024-05-01 23:18:47 +09:00
2024-05-19 14:36:17 -07:00
2022-09-07 21:53:50 +02:00
2024-10-21 14:30:26 +02:00
2024-12-05 14:01:11 +01:00
2024-07-03 10:36:14 +02:00
2024-04-29 15:01:42 +01:00
2024-02-27 16:55:34 -05:00
2021-10-22 09:50:55 +10:30
2024-07-03 15:16:54 +02:00
2019-05-24 17:27:12 +02:00
2024-10-02 15:14:29 +02:00
2024-06-18 16:25:50 +02:00
2023-06-14 10:41:07 -07:00
2024-07-24 20:59:29 +02:00
2020-03-06 11:06:15 +01:00
2024-03-07 22:10:01 +00:00
2017-11-02 11:10:55 +01:00
2020-04-18 15:44:54 -05:00
2024-09-09 09:41:47 -04:00
2017-11-02 11:10:55 +01:00
2024-09-09 16:39:05 -07:00
2019-02-15 16:54:38 +01:00
2022-04-11 11:48:01 +02:00
2017-11-02 11:10:55 +01:00
2024-09-02 11:47:06 +02:00
2024-07-03 15:16:54 +02:00
2022-07-05 13:33:00 -07:00
2022-01-20 08:52:54 +02:00
2020-07-20 17:00:02 -07:00
2021-07-27 20:11:45 +01:00
2021-07-27 20:11:44 +01:00
2024-09-24 16:10:12 +02:00
2023-04-08 13:45:38 -07:00
2023-08-05 08:31:41 +02:00
2024-09-13 15:14:56 +02:00
2019-05-30 11:26:35 -07:00
2024-08-29 10:39:37 +02:00
2024-07-03 19:30:06 -07:00
2024-07-06 11:53:19 -07:00
2017-11-02 11:10:55 +01:00
2010-02-09 11:13:56 +01:00
2022-07-28 11:29:36 +02:00
2024-12-05 14:01:18 +01:00
2022-06-27 12:52:53 -07:00
2024-10-15 18:46:25 +02:00
2022-12-02 13:10:36 -08:00
2024-02-19 09:37:59 +01:00
2022-05-04 11:05:54 +02:00
2024-10-17 00:28:10 -07:00
2024-09-26 14:01:43 -07:00
2022-12-04 16:45:02 -08:00
2024-09-13 07:27:36 -07:00
2024-05-29 12:52:26 -07:00
2024-07-03 15:16:54 +02:00
2019-05-21 11:28:39 +02:00
2024-05-13 16:13:19 +02:00
2022-02-15 09:58:57 +01:00
2024-09-21 12:46:00 +02:00
2023-05-01 17:50:18 -07:00
2024-08-02 18:27:05 +02:00
2023-06-08 10:11:17 -04:00
2020-07-24 17:12:41 -07:00
2023-07-14 08:27:33 +01:00
2023-11-03 09:12:00 +00:00
2024-10-02 17:23:23 -04:00
2023-01-03 19:48:43 +01:00
2023-10-02 11:24:52 -07:00
2020-04-18 15:44:55 -05:00
2024-02-15 23:43:46 -05:00
2019-05-30 11:26:32 -07:00
2019-06-26 13:19:46 -07:00
2019-05-30 11:26:32 -07:00
2024-02-28 19:36:39 -08:00
2024-07-12 16:39:52 -07:00
2022-04-06 13:43:44 -07:00
2022-01-22 08:33:35 +02:00
2024-11-02 22:28:58 -07:00
2022-04-22 12:32:04 +02:00
2024-04-25 21:07:02 -07:00
2024-04-09 17:14:57 -04:00
2024-12-05 14:01:35 +01:00
2023-12-06 11:33:45 +01:00
2024-07-08 11:40:57 -05:00
2024-02-26 00:38:17 +02:00
2023-05-30 22:04:46 +03:00
2024-08-23 20:50:19 +02:00
2023-03-28 16:20:16 -07:00
2024-09-04 11:38:34 +02:00
2024-05-19 09:21:03 -07:00
2021-07-21 08:14:33 -07:00
2021-12-16 10:59:02 -07:00
2024-11-01 07:45:00 -10:00
2024-08-23 11:16:39 +02:00
2024-09-05 12:43:53 -03:00
2023-06-05 15:35:27 +02:00
2024-08-07 18:32:38 +02:00
2024-02-14 11:14:40 +01:00
2024-05-07 13:29:45 +02:00
2023-02-01 20:54:27 -08:00
2019-06-05 17:37:17 +02:00
2023-12-20 19:26:31 -05:00
2022-05-12 10:00:03 -05:00
2024-07-31 13:49:48 +01:00
2024-08-08 17:15:02 +02:00
2022-09-28 14:21:53 +01:00
2024-04-08 16:34:18 +02:00
2024-01-24 16:02:48 +01:00
2017-11-02 11:10:55 +01:00
2022-07-22 17:24:47 -05:00
2022-09-15 10:28:18 +02:00
2020-10-08 18:00:08 +02:00
2023-12-26 20:24:33 +00:00
2023-10-16 12:44:14 -04:00
2024-08-26 23:49:15 -04:00
2024-06-24 22:24:56 -07:00
2024-12-05 14:01:22 +01:00
2019-10-21 09:16:46 -04:00
2019-05-30 11:26:35 -07:00
2023-06-08 12:27:20 -07:00
2022-03-09 12:13:37 +00:00
2024-12-09 10:41:08 +01:00
2021-06-05 23:49:46 +09:00
2024-06-17 12:30:07 -07:00
2021-12-09 16:42:29 -08:00
2021-12-09 16:42:26 -08:00
2023-07-03 09:27:12 +01:00
2017-11-02 11:10:55 +01:00
2024-09-03 21:15:46 -07:00
2024-03-18 10:33:06 -04:00
2019-12-11 09:12:38 +01:00
2024-01-30 15:54:25 -08:00
2024-05-19 14:02:03 -07:00
2024-09-20 18:28:26 +03:00
2017-11-02 11:10:55 +01:00
2019-05-24 17:27:11 +02:00
2024-09-01 20:26:03 -07:00
2024-12-05 14:03:03 +01:00
2023-06-29 15:06:32 +01:00
2024-09-09 16:39:04 -07:00
2022-06-16 19:48:30 -07:00
2019-05-24 17:36:45 +02:00
2024-07-03 19:30:23 -07:00
2024-02-17 16:20:41 +01:00
2024-09-25 20:12:58 +09:00
2019-06-05 17:37:16 +02:00
2021-02-26 09:41:03 -08:00
2024-10-28 21:40:39 -07:00
2023-10-18 14:43:21 -07:00
2023-10-04 10:41:57 -07:00
2023-12-20 19:26:30 -05:00
2024-12-05 14:02:43 +01:00
2017-11-02 11:10:55 +01:00
2022-07-22 12:53:22 +01:00
2022-04-21 11:40:59 -07:00
2024-05-03 10:44:42 +01:00
2024-03-07 08:48:03 +00:00
2019-12-21 20:09:54 +01:00
2024-03-07 08:45:31 +00:00
2019-06-19 17:09:55 +02:00
2024-08-01 13:40:01 +01:00
2024-09-07 10:16:56 +09:00
2023-04-27 13:45:31 -07:00
2023-02-13 17:01:05 -08:00
2023-05-18 11:40:32 -07:00
2022-11-30 14:49:42 -08:00
2023-08-18 10:19:00 -07:00
2022-10-17 16:40:58 +02:00
2024-08-02 16:32:29 -07:00
2021-06-11 04:35:40 +09:00
2024-05-09 15:48:01 +02:00
2024-12-05 14:02:46 +01:00
2022-07-03 14:40:28 -07:00
2022-03-23 19:00:33 -07:00
2017-11-02 11:10:55 +01:00
2023-10-16 12:44:06 -04:00
2019-03-07 18:32:03 -08:00
2024-07-03 15:16:54 +02:00
2024-06-10 19:52:44 -07:00
2024-07-01 19:28:58 +01:00
2022-12-08 21:49:25 -05:00
2017-11-02 11:10:55 +01:00
2023-12-17 22:33:55 +01:00
2024-07-03 15:16:54 +02:00
2022-10-04 12:06:58 -06:00
2023-01-10 15:53:36 -08:00
2018-04-19 15:59:11 -04:00
2024-07-03 15:16:54 +02:00
2024-08-11 19:18:52 +03:00
2024-11-11 00:00:37 -08:00
2024-07-12 15:52:09 -07:00
2024-04-25 20:55:48 -07:00
2024-04-25 20:55:56 -07:00
2023-02-10 17:33:29 -08:00
2024-02-22 10:24:55 -08:00
2020-09-28 12:16:13 +02:00
2024-07-25 10:42:22 -07:00
2024-09-03 21:15:59 -07:00
2024-08-11 17:04:29 +01:00
2024-07-31 09:57:18 -07:00
2022-07-27 14:04:52 +02:00
2017-11-02 11:10:55 +01:00
2019-05-30 11:26:41 -07:00
2022-11-24 17:12:15 +01:00
2024-10-21 22:11:19 +08:00
2024-11-05 16:49:55 -08:00
2024-12-05 14:03:08 +01:00
2017-11-02 11:10:55 +01:00
2024-11-07 14:14:58 -08:00
2024-07-23 11:43:40 -07:00
2024-05-14 00:31:43 -07:00
2023-12-01 09:51:43 -08:00
2020-03-24 13:42:44 +01:00
2023-11-18 14:56:16 +01:00
2024-07-03 15:16:54 +02:00
2022-08-02 12:34:04 -04:00
2024-08-17 13:55:50 +08:00
2017-11-02 11:10:55 +01:00
2023-06-20 20:17:10 -07:00
2023-06-20 20:18:39 -07:00
2018-03-24 11:25:35 -05:00
2024-07-29 12:46:35 -05:00
2019-10-23 17:23:44 +02:00
2024-09-13 07:36:02 +02:00
2024-06-05 17:03:57 +02:00
2024-07-03 15:16:54 +02:00
2018-10-17 13:56:58 -07:00
2024-07-28 16:47:51 -06:00
2024-10-21 12:54:25 +02:00
2024-09-12 12:20:41 +02:00
2024-06-10 13:15:40 +01:00
2024-12-05 14:02:01 +01:00
2017-11-02 11:10:55 +01:00
2024-09-20 19:31:03 -04:00
2024-10-03 16:19:13 -04:00
2024-09-23 15:03:30 -04:00
2021-03-22 10:19:00 -04:00
2024-12-05 14:02:27 +01:00
2022-07-28 11:29:36 +02:00
2023-07-25 00:30:02 -05:00
2024-05-08 08:41:29 -07:00
2024-06-24 22:25:02 -07:00
2024-08-18 10:08:00 -07:00
2023-02-21 14:45:22 -08:00
2023-08-16 14:27:01 -07:00
2024-06-28 10:37:29 +02:00
2020-06-05 20:02:08 -04:00
2024-01-03 13:33:59 +01:00
2024-09-03 21:15:28 -07:00
2023-10-17 13:57:54 -07:00
2024-08-22 13:25:07 -07:00
2024-08-26 09:51:32 -07:00
2024-01-04 17:01:13 +01:00
2024-05-03 07:26:38 +02:00
2019-01-22 10:21:45 +01:00
2024-06-10 11:14:52 +01:00
2024-05-01 23:18:48 +09:00
2024-01-10 10:52:29 +01:00
2024-12-05 14:01:32 +01:00
2024-07-25 06:53:47 -05:00
2023-05-18 11:56:19 -07:00
2022-04-29 09:53:52 +02:00
2024-08-30 10:57:24 -07:00
2024-05-19 22:29:43 -05:00
2024-12-05 14:03:08 +01:00
2024-12-05 14:03:08 +01:00
2022-09-26 19:46:27 -07:00
2019-06-05 17:30:29 +02:00
2024-03-22 16:25:31 -07:00
2021-09-17 13:52:17 +01:00
2024-03-06 13:04:17 -08:00
2022-09-26 19:46:09 -07:00
2024-10-30 20:14:12 -07:00
2024-04-25 20:56:04 -07:00
2024-07-26 15:32:27 -07:00
2024-09-20 17:53:17 -07:00
2024-03-26 11:07:20 -07:00
2024-09-01 20:25:59 -07:00
2024-08-13 14:16:22 +02:00
2017-02-03 16:35:42 -05:00
2021-02-03 19:05:50 +01:00
2024-08-30 08:22:38 +02:00
2021-09-27 17:00:21 +02:00
2024-08-30 14:29:30 +02:00
2024-07-25 10:42:22 -07:00
2024-09-03 17:11:05 -05:00
2024-09-19 14:25:34 -05:00
2023-05-28 20:45:46 +02:00
2024-06-17 15:15:46 +02:00
2024-06-17 15:15:46 +02:00
2024-07-03 19:30:02 -07:00
2021-12-09 15:41:09 -05:00
2024-08-19 13:45:03 +02:00
2024-10-17 00:28:07 -07:00
2024-09-17 01:06:59 -07:00
2019-06-05 17:37:07 +02:00
2024-08-23 13:04:34 +01:00
2024-09-11 11:06:11 +01:00
2024-03-13 12:53:53 -07:00
2024-03-13 12:53:53 -07:00
2017-11-02 11:10:55 +01:00
2021-09-20 22:28:47 +02:00
2024-02-25 12:05:23 +01:00
2019-05-30 11:29:52 -07:00
2020-07-28 17:07:06 -07:00
2024-10-30 17:11:28 +01:00
2024-01-24 17:10:30 +01:00
2021-06-17 04:21:46 -07:00
2021-12-23 22:35:00 +11:00
2024-07-03 15:16:54 +02:00
2024-07-21 17:56:22 -07:00
2024-02-20 09:23:52 +01:00
2022-05-22 11:32:30 -07:00
2023-10-15 20:07:52 +01:00
2024-07-29 21:57:35 +02:00
2020-09-10 19:27:59 +02:00
2022-07-28 11:29:36 +02:00
2023-04-11 21:55:35 -04:00
2023-12-20 19:26:30 -05:00
2024-06-05 16:52:36 +02:00
2023-08-12 09:18:47 -07:00
2024-08-21 14:56:25 +02:00
2024-08-01 14:23:39 +02:00
2024-08-04 13:36:28 -07:00
2024-03-25 19:50:09 +00:00
2020-12-04 08:44:24 +00:00
2023-07-10 09:52:30 +02:00
2023-03-17 11:16:43 +08:00
2024-02-22 10:24:47 -08:00
2024-01-10 10:52:28 +01:00
2024-10-02 17:23:23 -04:00
2024-02-12 23:04:34 +01:00
2024-09-05 11:14:14 +02:00
2024-07-22 18:13:44 +02:00
2023-12-11 13:26:06 +01:00
2022-08-14 13:31:03 -07:00
2023-06-19 16:19:04 -07:00
2022-12-20 03:13:45 +01:00
2023-08-18 10:19:00 -07:00
2024-02-14 17:10:06 +01:00
2020-07-27 17:46:24 +09:00
2017-11-02 11:10:55 +01:00
2024-09-05 16:56:15 +02:00
2024-12-05 14:03:07 +01:00
2023-03-28 10:39:29 +02:00
2024-09-18 08:53:53 +02:00
2024-09-18 08:53:53 +02:00
2023-05-23 12:55:12 -07:00
2017-11-02 11:10:55 +01:00
2024-01-02 09:33:18 +01:00
2024-08-05 14:34:23 -07:00
2024-08-13 12:45:01 +01:00
2022-11-15 14:30:41 -08:00
2023-07-15 12:56:48 -07:00
2024-08-28 11:13:08 +02:00
2024-02-21 11:53:25 +01:00
2022-06-29 15:56:20 +02:00
2022-03-08 14:33:36 -06:00
2023-12-01 14:53:56 +09:00
2024-08-27 10:28:55 +02:00
2024-04-02 18:03:32 -07:00
2024-07-03 15:16:54 +02:00
2024-09-09 16:39:03 -07:00
2020-10-04 19:51:43 -07:00
2017-11-02 11:10:55 +01:00
2024-08-26 13:16:20 +02:00
2023-07-15 11:34:49 -07:00
2023-12-27 11:49:56 -05:00
2023-12-20 19:26:31 -05:00
2024-03-08 12:05:10 +01:00
2021-12-04 10:56:23 +01:00
2024-04-03 09:59:38 +01:00
2022-07-30 14:01:28 -04:00
2024-12-05 14:01:23 +01:00
2022-09-15 16:14:02 +02:00
2024-04-08 16:39:16 +02:00
2024-09-20 00:20:06 -06:00
2017-11-02 11:10:55 +01:00
2024-10-17 09:58:07 +02:00
2022-11-09 18:06:35 +00:00
2023-04-21 08:19:30 +01:00
2017-11-02 11:10:55 +01:00
2023-12-20 19:26:32 -05:00
2024-04-25 20:56:13 -07:00
2024-10-09 12:47:18 -07:00
2023-10-17 09:10:06 -06:00
2023-08-22 11:10:26 -06:00
2024-01-27 18:08:53 -08:00
2023-12-20 19:26:32 -05:00
2024-12-05 14:01:23 +01:00
2024-01-27 18:13:53 -08:00
2023-01-19 15:01:19 +01:00
2024-07-03 15:16:54 +02:00
2024-08-23 13:04:34 +01:00
2024-04-09 10:53:44 +02:00
2023-10-18 14:34:18 -07:00
2023-12-20 19:26:31 -05:00
2020-12-10 16:17:15 +01:00
2022-06-03 10:34:34 -07:00
2024-09-11 20:44:31 -07:00
2024-07-12 15:52:20 -07:00
2024-09-21 07:29:05 -07:00
2024-07-03 15:16:54 +02:00
2019-05-30 11:29:19 -07:00
2017-11-02 11:10:55 +01:00
2024-09-18 07:52:24 +02:00
2019-04-24 12:17:08 +02:00
2023-07-26 21:24:40 -07:00
2024-09-11 20:44:32 -07:00
2024-11-13 19:29:46 -08:00
2022-02-23 10:58:28 +01:00
2017-11-02 11:10:55 +01:00
2019-05-24 17:36:45 +02:00
2024-12-05 14:01:23 +01:00
2024-06-05 16:52:36 +02:00
2023-12-12 16:20:02 +01:00
2023-12-15 17:27:04 +01:00
2023-08-21 13:46:24 -07:00
2022-06-10 14:51:35 +02:00
2024-07-20 12:41:03 -07:00
2024-04-15 11:29:48 +02:00
2024-08-12 23:50:58 +05:30
2019-06-05 17:36:37 +02:00
2024-04-05 11:21:31 -07:00
2023-11-08 11:06:50 -08:00
2021-12-10 14:06:03 +00:00
2024-06-20 15:19:17 -06:00
2021-04-19 16:03:15 +02:00
2021-08-19 09:02:55 +09:00
2024-05-04 18:57:21 +02:00
2024-09-10 16:42:11 -07:00
2024-09-19 10:18:15 +02:00
2023-10-18 14:43:21 -07:00
2021-01-15 13:23:36 +00:00
2024-02-05 14:28:19 +01:00
2022-09-07 12:42:25 +01:00
2023-07-17 16:08:08 -07:00
2024-10-30 20:14:11 -07:00
2023-08-21 13:37:26 -07:00
2024-07-12 15:52:15 -07:00
2024-07-10 07:59:03 +02:00
2024-06-10 15:03:30 -05:00
2024-09-05 11:39:17 +02:00
2024-07-24 20:59:29 +02:00
2024-08-26 19:14:48 -04:00
2024-06-04 15:56:45 +02:00
2022-09-29 15:20:29 +02:00
2023-07-25 19:21:03 +02:00
2024-07-04 02:02:33 -06:00
2024-10-11 10:49:32 +02:00
2022-07-15 10:21:30 +02:00
2024-07-03 15:16:54 +02:00
2024-04-08 11:49:02 +01:00
2023-05-16 12:11:00 +02:00
2021-03-01 15:25:24 -08:00
2019-05-30 11:26:32 -07:00
2024-08-23 15:42:58 +02:00
2024-05-06 17:39:09 +02:00
2024-04-19 07:52:45 +03:00
2024-10-27 10:36:04 +01:00
2022-08-09 20:02:13 +02:00
2020-10-30 21:57:04 +01:00
2024-04-01 10:36:34 +02:00
2024-06-03 11:18:51 +02:00
2024-04-01 10:36:35 +02:00
2024-04-22 16:13:06 +02:00
2022-05-13 23:59:23 +02:00
2023-11-15 12:03:41 -08:00
2023-09-15 13:48:10 +02:00
2024-11-13 21:10:45 +02:00
2023-12-18 23:14:16 -05:00
2024-05-04 04:25:37 -04:00
2024-09-25 23:23:44 +09:00
2017-11-07 16:28:19 +01:00
2024-06-17 20:42:57 +02:00
2024-05-06 12:05:00 +02:00
2023-05-31 19:00:26 +01:00
2024-10-30 11:38:10 -10:00
2024-08-06 13:42:40 +02:00
2024-05-15 12:34:46 -07:00
2024-01-10 16:43:55 -08:00
2024-09-12 12:20:39 +02:00
2023-06-08 15:41:34 -07:00
2024-10-02 17:23:23 -04:00
2021-10-12 11:41:39 -03:00
2024-02-21 15:19:03 +01:00
2024-08-20 14:05:33 -04:00
2024-09-05 16:56:14 +02:00
2024-09-03 09:54:39 +02:00
2024-11-07 14:14:59 -08:00
2024-10-28 21:40:38 -07:00
2024-12-09 10:41:06 +01:00
2022-10-23 12:01:01 -07:00
2023-03-23 17:25:46 +01:00
2024-09-10 02:51:48 -04:00
2023-09-15 18:29:45 +08:00
2022-05-08 01:33:07 -07:00
2020-05-13 12:42:46 -05:00
2024-05-31 15:15:51 -06:00
2023-08-24 13:27:47 -05:00
2020-08-06 16:13:13 +02:00
2024-10-02 17:21:59 -07:00
2024-08-15 19:14:19 -07:00
2024-11-11 00:00:37 -08:00
2024-09-01 20:25:43 -07:00
2021-09-03 09:58:17 -07:00
2024-09-03 21:15:36 -07:00
2023-04-21 03:02:34 -04:00
2024-04-17 13:37:23 +02:00
2024-05-27 13:51:29 +02:00
2023-12-20 19:26:31 -05:00
2021-08-22 10:28:08 +02:00
2022-10-07 15:00:25 +02:00
2022-05-07 22:55:48 +02:00
2024-06-24 20:36:08 +03:00
2024-09-21 07:29:05 -07:00
2024-02-09 12:07:48 +00:00
2024-05-19 09:21:03 -07:00
2023-08-10 12:06:04 +02:00
2024-09-01 20:43:27 -07:00
2019-05-24 17:36:47 +02:00
2020-08-12 10:57:59 -07:00
2024-04-25 20:55:48 -07:00
2022-11-30 15:58:52 -08:00
2024-09-09 16:39:06 -07:00
2024-09-01 20:26:02 -07:00