diff options
author | Linus Torvalds <torvalds@linux-foundation.org> | 2019-07-18 14:51:00 -0400 |
---|---|---|
committer | Linus Torvalds <torvalds@linux-foundation.org> | 2019-07-18 14:51:00 -0400 |
commit | 818e95c768c6607a1df4cf022c00c3c58e2f203e (patch) | |
tree | 453a2b891097c379f1f133f4821a46628d86002b | |
parent | d4df33b0e9925c158b313a586fb1557cf29cfdf4 (diff) | |
parent | 0aeb1def44169cbe7119f26cf10b974a2046142e (diff) |
Merge tag 'trace-v5.3' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace
Pull tracing updates from Steven Rostedt:
"The main changes in this release include:
- Add user space specific memory reading for kprobes
- Allow kprobes to be executed earlier in boot
The rest are mostly just various clean ups and small fixes"
* tag 'trace-v5.3' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace: (33 commits)
tracing: Make trace_get_fields() global
tracing: Let filter_assign_type() detect FILTER_PTR_STRING
tracing: Pass type into tracing_generic_entry_update()
ftrace/selftest: Test if set_event/ftrace_pid exists before writing
ftrace/selftests: Return the skip code when tracing directory not configured in kernel
tracing/kprobe: Check registered state using kprobe
tracing/probe: Add trace_event_call accesses APIs
tracing/probe: Add probe event name and group name accesses APIs
tracing/probe: Add trace flag access APIs for trace_probe
tracing/probe: Add trace_event_file access APIs for trace_probe
tracing/probe: Add trace_event_call register API for trace_probe
tracing/probe: Add trace_probe init and free functions
tracing/uprobe: Set print format when parsing command
tracing/kprobe: Set print format right after parsed command
kprobes: Fix to init kprobes in subsys_initcall
tracepoint: Use struct_size() in kmalloc()
ring-buffer: Remove HAVE_64BIT_ALIGNED_ACCESS
ftrace: Enable trampoline when rec count returns back to one
tracing/kprobe: Do not run kprobe boot tests if kprobe_event is on cmdline
tracing: Make a separate config for trace event self tests
...
33 files changed, 861 insertions, 411 deletions
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a5f4004e8705..f0461456d910 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt | |||
@@ -2011,6 +2011,19 @@ | |||
2011 | Built with CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y, | 2011 | Built with CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y, |
2012 | the default is off. | 2012 | the default is off. |
2013 | 2013 | ||
2014 | kprobe_event=[probe-list] | ||
2015 | [FTRACE] Add kprobe events and enable at boot time. | ||
2016 | The probe-list is a semicolon delimited list of probe | ||
2017 | definitions. Each definition is same as kprobe_events | ||
2018 | interface, but the parameters are comma delimited. | ||
2019 | For example, to add a kprobe event on vfs_read with | ||
2020 | arg1 and arg2, add to the command line; | ||
2021 | |||
2022 | kprobe_event=p,vfs_read,$arg1,$arg2 | ||
2023 | |||
2024 | See also Documentation/trace/kprobetrace.rst "Kernel | ||
2025 | Boot Parameter" section. | ||
2026 | |||
2014 | kpti= [ARM64] Control page table isolation of user | 2027 | kpti= [ARM64] Control page table isolation of user |
2015 | and kernel address spaces. | 2028 | and kernel address spaces. |
2016 | Default: enabled on cores which need mitigation. | 2029 | Default: enabled on cores which need mitigation. |
diff --git a/Documentation/trace/kprobetrace.rst b/Documentation/trace/kprobetrace.rst index 7d2b0178d3f3..fbb314bfa112 100644 --- a/Documentation/trace/kprobetrace.rst +++ b/Documentation/trace/kprobetrace.rst | |||
@@ -51,15 +51,17 @@ Synopsis of kprobe_events | |||
51 | $argN : Fetch the Nth function argument. (N >= 1) (\*1) | 51 | $argN : Fetch the Nth function argument. (N >= 1) (\*1) |
52 | $retval : Fetch return value.(\*2) | 52 | $retval : Fetch return value.(\*2) |
53 | $comm : Fetch current task comm. | 53 | $comm : Fetch current task comm. |
54 | +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(\*3) | 54 | +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4) |
55 | NAME=FETCHARG : Set NAME as the argument name of FETCHARG. | 55 | NAME=FETCHARG : Set NAME as the argument name of FETCHARG. |
56 | FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types | 56 | FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types |
57 | (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types | 57 | (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types |
58 | (x8/x16/x32/x64), "string" and bitfield are supported. | 58 | (x8/x16/x32/x64), "string", "ustring" and bitfield |
59 | are supported. | ||
59 | 60 | ||
60 | (\*1) only for the probe on function entry (offs == 0). | 61 | (\*1) only for the probe on function entry (offs == 0). |
61 | (\*2) only for return probe. | 62 | (\*2) only for return probe. |
62 | (\*3) this is useful for fetching a field of data structures. | 63 | (\*3) this is useful for fetching a field of data structures. |
64 | (\*4) "u" means user-space dereference. See :ref:`user_mem_access`. | ||
63 | 65 | ||
64 | Types | 66 | Types |
65 | ----- | 67 | ----- |
@@ -77,7 +79,8 @@ apply it to registers/stack-entries etc. (for example, '$stack1:x8[8]' is | |||
77 | wrong, but '+8($stack):x8[8]' is OK.) | 79 | wrong, but '+8($stack):x8[8]' is OK.) |
78 | String type is a special type, which fetches a "null-terminated" string from | 80 | String type is a special type, which fetches a "null-terminated" string from |
79 | kernel space. This means it will fail and store NULL if the string container | 81 | kernel space. This means it will fail and store NULL if the string container |
80 | has been paged out. | 82 | has been paged out. "ustring" type is an alternative of string for user-space. |
83 | See :ref:`user_mem_access` for more info.. | ||
81 | The string array type is a bit different from other types. For other base | 84 | The string array type is a bit different from other types. For other base |
82 | types, <base-type>[1] is equal to <base-type> (e.g. +0(%di):x32[1] is same | 85 | types, <base-type>[1] is equal to <base-type> (e.g. +0(%di):x32[1] is same |
83 | as +0(%di):x32.) But string[1] is not equal to string. The string type itself | 86 | as +0(%di):x32.) But string[1] is not equal to string. The string type itself |
@@ -92,6 +95,25 @@ Symbol type('symbol') is an alias of u32 or u64 type (depends on BITS_PER_LONG) | |||
92 | which shows given pointer in "symbol+offset" style. | 95 | which shows given pointer in "symbol+offset" style. |
93 | For $comm, the default type is "string"; any other type is invalid. | 96 | For $comm, the default type is "string"; any other type is invalid. |
94 | 97 | ||
98 | .. _user_mem_access: | ||
99 | User Memory Access | ||
100 | ------------------ | ||
101 | Kprobe events supports user-space memory access. For that purpose, you can use | ||
102 | either user-space dereference syntax or 'ustring' type. | ||
103 | |||
104 | The user-space dereference syntax allows you to access a field of a data | ||
105 | structure in user-space. This is done by adding the "u" prefix to the | ||
106 | dereference syntax. For example, +u4(%si) means it will read memory from the | ||
107 | address in the register %si offset by 4, and the memory is expected to be in | ||
108 | user-space. You can use this for strings too, e.g. +u0(%si):string will read | ||
109 | a string from the address in the register %si that is expected to be in user- | ||
110 | space. 'ustring' is a shortcut way of performing the same task. That is, | ||
111 | +0(%si):ustring is equivalent to +u0(%si):string. | ||
112 | |||
113 | Note that kprobe-event provides the user-memory access syntax but it doesn't | ||
114 | use it transparently. This means if you use normal dereference or string type | ||
115 | for user memory, it might fail, and may always fail on some archs. The user | ||
116 | has to carefully check if the target data is in kernel or user space. | ||
95 | 117 | ||
96 | Per-Probe Event Filtering | 118 | Per-Probe Event Filtering |
97 | ------------------------- | 119 | ------------------------- |
@@ -124,6 +146,20 @@ You can check the total number of probe hits and probe miss-hits via | |||
124 | The first column is event name, the second is the number of probe hits, | 146 | The first column is event name, the second is the number of probe hits, |
125 | the third is the number of probe miss-hits. | 147 | the third is the number of probe miss-hits. |
126 | 148 | ||
149 | Kernel Boot Parameter | ||
150 | --------------------- | ||
151 | You can add and enable new kprobe events when booting up the kernel by | ||
152 | "kprobe_event=" parameter. The parameter accepts a semicolon-delimited | ||
153 | kprobe events, which format is similar to the kprobe_events. | ||
154 | The difference is that the probe definition parameters are comma-delimited | ||
155 | instead of space. For example, adding myprobe event on do_sys_open like below | ||
156 | |||
157 | p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack) | ||
158 | |||
159 | should be below for kernel boot parameter (just replace spaces with comma) | ||
160 | |||
161 | p:myprobe,do_sys_open,dfd=%ax,filename=%dx,flags=%cx,mode=+4($stack) | ||
162 | |||
127 | 163 | ||
128 | Usage examples | 164 | Usage examples |
129 | -------------- | 165 | -------------- |
diff --git a/Documentation/trace/uprobetracer.rst b/Documentation/trace/uprobetracer.rst index 0b21305fabdc..6e75a6c5a2c8 100644 --- a/Documentation/trace/uprobetracer.rst +++ b/Documentation/trace/uprobetracer.rst | |||
@@ -42,16 +42,18 @@ Synopsis of uprobe_tracer | |||
42 | @+OFFSET : Fetch memory at OFFSET (OFFSET from same file as PATH) | 42 | @+OFFSET : Fetch memory at OFFSET (OFFSET from same file as PATH) |
43 | $stackN : Fetch Nth entry of stack (N >= 0) | 43 | $stackN : Fetch Nth entry of stack (N >= 0) |
44 | $stack : Fetch stack address. | 44 | $stack : Fetch stack address. |
45 | $retval : Fetch return value.(*) | 45 | $retval : Fetch return value.(\*1) |
46 | $comm : Fetch current task comm. | 46 | $comm : Fetch current task comm. |
47 | +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(**) | 47 | +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*2)(\*3) |
48 | NAME=FETCHARG : Set NAME as the argument name of FETCHARG. | 48 | NAME=FETCHARG : Set NAME as the argument name of FETCHARG. |
49 | FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types | 49 | FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types |
50 | (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types | 50 | (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types |
51 | (x8/x16/x32/x64), "string" and bitfield are supported. | 51 | (x8/x16/x32/x64), "string" and bitfield are supported. |
52 | 52 | ||
53 | (*) only for return probe. | 53 | (\*1) only for return probe. |
54 | (**) this is useful for fetching a field of data structures. | 54 | (\*2) this is useful for fetching a field of data structures. |
55 | (\*3) Unlike kprobe event, "u" prefix will just be ignored, becuse uprobe | ||
56 | events can access only user-space memory. | ||
55 | 57 | ||
56 | Types | 58 | Types |
57 | ----- | 59 | ----- |
diff --git a/arch/Kconfig b/arch/Kconfig index e8d19c3cb91f..6dd1faab6ccb 100644 --- a/arch/Kconfig +++ b/arch/Kconfig | |||
@@ -128,22 +128,6 @@ config UPROBES | |||
128 | managed by the kernel and kept transparent to the probed | 128 | managed by the kernel and kept transparent to the probed |
129 | application. ) | 129 | application. ) |
130 | 130 | ||
131 | config HAVE_64BIT_ALIGNED_ACCESS | ||
132 | def_bool 64BIT && !HAVE_EFFICIENT_UNALIGNED_ACCESS | ||
133 | help | ||
134 | Some architectures require 64 bit accesses to be 64 bit | ||
135 | aligned, which also requires structs containing 64 bit values | ||
136 | to be 64 bit aligned too. This includes some 32 bit | ||
137 | architectures which can do 64 bit accesses, as well as 64 bit | ||
138 | architectures without unaligned access. | ||
139 | |||
140 | This symbol should be selected by an architecture if 64 bit | ||
141 | accesses are required to be 64 bit aligned in this way even | ||
142 | though it is not a 64 bit architecture. | ||
143 | |||
144 | See Documentation/unaligned-memory-access.txt for more | ||
145 | information on the topic of unaligned memory accesses. | ||
146 | |||
147 | config HAVE_EFFICIENT_UNALIGNED_ACCESS | 131 | config HAVE_EFFICIENT_UNALIGNED_ACCESS |
148 | bool | 132 | bool |
149 | help | 133 | help |
diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index c82abd6e4ca3..9c4435307ff8 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h | |||
@@ -66,7 +66,9 @@ static inline bool __chk_range_not_ok(unsigned long addr, unsigned long size, un | |||
66 | }) | 66 | }) |
67 | 67 | ||
68 | #ifdef CONFIG_DEBUG_ATOMIC_SLEEP | 68 | #ifdef CONFIG_DEBUG_ATOMIC_SLEEP |
69 | # define WARN_ON_IN_IRQ() WARN_ON_ONCE(!in_task()) | 69 | static inline bool pagefault_disabled(void); |
70 | # define WARN_ON_IN_IRQ() \ | ||
71 | WARN_ON_ONCE(!in_task() && !pagefault_disabled()) | ||
70 | #else | 72 | #else |
71 | # define WARN_ON_IN_IRQ() | 73 | # define WARN_ON_IN_IRQ() |
72 | #endif | 74 | #endif |
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 4b73f5937f41..024c3053dbba 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c | |||
@@ -373,7 +373,7 @@ static int add_brk_on_nop(struct dyn_ftrace *rec) | |||
373 | return add_break(rec->ip, old); | 373 | return add_break(rec->ip, old); |
374 | } | 374 | } |
375 | 375 | ||
376 | static int add_breakpoints(struct dyn_ftrace *rec, int enable) | 376 | static int add_breakpoints(struct dyn_ftrace *rec, bool enable) |
377 | { | 377 | { |
378 | unsigned long ftrace_addr; | 378 | unsigned long ftrace_addr; |
379 | int ret; | 379 | int ret; |
@@ -481,7 +481,7 @@ static int add_update_nop(struct dyn_ftrace *rec) | |||
481 | return add_update_code(ip, new); | 481 | return add_update_code(ip, new); |
482 | } | 482 | } |
483 | 483 | ||
484 | static int add_update(struct dyn_ftrace *rec, int enable) | 484 | static int add_update(struct dyn_ftrace *rec, bool enable) |
485 | { | 485 | { |
486 | unsigned long ftrace_addr; | 486 | unsigned long ftrace_addr; |
487 | int ret; | 487 | int ret; |
@@ -527,7 +527,7 @@ static int finish_update_nop(struct dyn_ftrace *rec) | |||
527 | return ftrace_write(ip, new, 1); | 527 | return ftrace_write(ip, new, 1); |
528 | } | 528 | } |
529 | 529 | ||
530 | static int finish_update(struct dyn_ftrace *rec, int enable) | 530 | static int finish_update(struct dyn_ftrace *rec, bool enable) |
531 | { | 531 | { |
532 | unsigned long ftrace_addr; | 532 | unsigned long ftrace_addr; |
533 | int ret; | 533 | int ret; |
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 25e2995d4a4c..8a8cb3c401b2 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h | |||
@@ -427,8 +427,8 @@ struct dyn_ftrace *ftrace_rec_iter_record(struct ftrace_rec_iter *iter); | |||
427 | iter = ftrace_rec_iter_next(iter)) | 427 | iter = ftrace_rec_iter_next(iter)) |
428 | 428 | ||
429 | 429 | ||
430 | int ftrace_update_record(struct dyn_ftrace *rec, int enable); | 430 | int ftrace_update_record(struct dyn_ftrace *rec, bool enable); |
431 | int ftrace_test_record(struct dyn_ftrace *rec, int enable); | 431 | int ftrace_test_record(struct dyn_ftrace *rec, bool enable); |
432 | void ftrace_run_stop_machine(int command); | 432 | void ftrace_run_stop_machine(int command); |
433 | unsigned long ftrace_location(unsigned long ip); | 433 | unsigned long ftrace_location(unsigned long ip); |
434 | unsigned long ftrace_location_range(unsigned long start, unsigned long end); | 434 | unsigned long ftrace_location_range(unsigned long start, unsigned long end); |
diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h index 8a62731673f7..5150436783e8 100644 --- a/include/linux/trace_events.h +++ b/include/linux/trace_events.h | |||
@@ -142,6 +142,7 @@ enum print_line_t { | |||
142 | enum print_line_t trace_handle_return(struct trace_seq *s); | 142 | enum print_line_t trace_handle_return(struct trace_seq *s); |
143 | 143 | ||
144 | void tracing_generic_entry_update(struct trace_entry *entry, | 144 | void tracing_generic_entry_update(struct trace_entry *entry, |
145 | unsigned short type, | ||
145 | unsigned long flags, | 146 | unsigned long flags, |
146 | int pc); | 147 | int pc); |
147 | struct trace_event_file; | 148 | struct trace_event_file; |
@@ -317,6 +318,14 @@ trace_event_name(struct trace_event_call *call) | |||
317 | return call->name; | 318 | return call->name; |
318 | } | 319 | } |
319 | 320 | ||
321 | static inline struct list_head * | ||
322 | trace_get_fields(struct trace_event_call *event_call) | ||
323 | { | ||
324 | if (!event_call->class->get_fields) | ||
325 | return &event_call->class->fields; | ||
326 | return event_call->class->get_fields(event_call); | ||
327 | } | ||
328 | |||
320 | struct trace_array; | 329 | struct trace_array; |
321 | struct trace_subsystem_dir; | 330 | struct trace_subsystem_dir; |
322 | 331 | ||
diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index 2b70130af585..34a038563d97 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h | |||
@@ -203,7 +203,10 @@ static inline void pagefault_enable(void) | |||
203 | /* | 203 | /* |
204 | * Is the pagefault handler disabled? If so, user access methods will not sleep. | 204 | * Is the pagefault handler disabled? If so, user access methods will not sleep. |
205 | */ | 205 | */ |
206 | #define pagefault_disabled() (current->pagefault_disabled != 0) | 206 | static inline bool pagefault_disabled(void) |
207 | { | ||
208 | return current->pagefault_disabled != 0; | ||
209 | } | ||
207 | 210 | ||
208 | /* | 211 | /* |
209 | * The pagefault handler is in general disabled by pagefault_disable() or | 212 | * The pagefault handler is in general disabled by pagefault_disable() or |
@@ -240,6 +243,18 @@ extern long probe_kernel_read(void *dst, const void *src, size_t size); | |||
240 | extern long __probe_kernel_read(void *dst, const void *src, size_t size); | 243 | extern long __probe_kernel_read(void *dst, const void *src, size_t size); |
241 | 244 | ||
242 | /* | 245 | /* |
246 | * probe_user_read(): safely attempt to read from a location in user space | ||
247 | * @dst: pointer to the buffer that shall take the data | ||
248 | * @src: address to read from | ||
249 | * @size: size of the data chunk | ||
250 | * | ||
251 | * Safely read from address @src to the buffer at @dst. If a kernel fault | ||
252 | * happens, handle that and return -EFAULT. | ||
253 | */ | ||
254 | extern long probe_user_read(void *dst, const void __user *src, size_t size); | ||
255 | extern long __probe_user_read(void *dst, const void __user *src, size_t size); | ||
256 | |||
257 | /* | ||
243 | * probe_kernel_write(): safely attempt to write to a location | 258 | * probe_kernel_write(): safely attempt to write to a location |
244 | * @dst: address to write to | 259 | * @dst: address to write to |
245 | * @src: pointer to the data that shall be written | 260 | * @src: pointer to the data that shall be written |
@@ -252,6 +267,9 @@ extern long notrace probe_kernel_write(void *dst, const void *src, size_t size); | |||
252 | extern long notrace __probe_kernel_write(void *dst, const void *src, size_t size); | 267 | extern long notrace __probe_kernel_write(void *dst, const void *src, size_t size); |
253 | 268 | ||
254 | extern long strncpy_from_unsafe(char *dst, const void *unsafe_addr, long count); | 269 | extern long strncpy_from_unsafe(char *dst, const void *unsafe_addr, long count); |
270 | extern long strncpy_from_unsafe_user(char *dst, const void __user *unsafe_addr, | ||
271 | long count); | ||
272 | extern long strnlen_unsafe_user(const void __user *unsafe_addr, long count); | ||
255 | 273 | ||
256 | /** | 274 | /** |
257 | * probe_kernel_address(): safely attempt to read from a location | 275 | * probe_kernel_address(): safely attempt to read from a location |
diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 9f5433a52488..9873fc627d61 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c | |||
@@ -2276,6 +2276,7 @@ static int __init init_kprobes(void) | |||
2276 | init_test_probes(); | 2276 | init_test_probes(); |
2277 | return err; | 2277 | return err; |
2278 | } | 2278 | } |
2279 | subsys_initcall(init_kprobes); | ||
2279 | 2280 | ||
2280 | #ifdef CONFIG_DEBUG_FS | 2281 | #ifdef CONFIG_DEBUG_FS |
2281 | static void report_probe(struct seq_file *pi, struct kprobe *p, | 2282 | static void report_probe(struct seq_file *pi, struct kprobe *p, |
@@ -2588,5 +2589,3 @@ static int __init debugfs_kprobe_init(void) | |||
2588 | 2589 | ||
2589 | late_initcall(debugfs_kprobe_init); | 2590 | late_initcall(debugfs_kprobe_init); |
2590 | #endif /* CONFIG_DEBUG_FS */ | 2591 | #endif /* CONFIG_DEBUG_FS */ |
2591 | |||
2592 | module_init(init_kprobes); | ||
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 564e5fdb025f..98da8998c25c 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig | |||
@@ -597,9 +597,19 @@ config FTRACE_STARTUP_TEST | |||
597 | functioning properly. It will do tests on all the configured | 597 | functioning properly. It will do tests on all the configured |
598 | tracers of ftrace. | 598 | tracers of ftrace. |
599 | 599 | ||
600 | config EVENT_TRACE_STARTUP_TEST | ||
601 | bool "Run selftest on trace events" | ||
602 | depends on FTRACE_STARTUP_TEST | ||
603 | default y | ||
604 | help | ||
605 | This option performs a test on all trace events in the system. | ||
606 | It basically just enables each event and runs some code that | ||
607 | will trigger events (not necessarily the event it enables) | ||
608 | This may take some time run as there are a lot of events. | ||
609 | |||
600 | config EVENT_TRACE_TEST_SYSCALLS | 610 | config EVENT_TRACE_TEST_SYSCALLS |
601 | bool "Run selftest on syscall events" | 611 | bool "Run selftest on syscall events" |
602 | depends on FTRACE_STARTUP_TEST | 612 | depends on EVENT_TRACE_STARTUP_TEST |
603 | help | 613 | help |
604 | This option will also enable testing every syscall event. | 614 | This option will also enable testing every syscall event. |
605 | It only enables the event and disables it and runs various loads | 615 | It only enables the event and disables it and runs various loads |
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 576c41644e77..eca34503f178 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c | |||
@@ -1622,6 +1622,11 @@ static bool test_rec_ops_needs_regs(struct dyn_ftrace *rec) | |||
1622 | return keep_regs; | 1622 | return keep_regs; |
1623 | } | 1623 | } |
1624 | 1624 | ||
1625 | static struct ftrace_ops * | ||
1626 | ftrace_find_tramp_ops_any(struct dyn_ftrace *rec); | ||
1627 | static struct ftrace_ops * | ||
1628 | ftrace_find_tramp_ops_next(struct dyn_ftrace *rec, struct ftrace_ops *ops); | ||
1629 | |||
1625 | static bool __ftrace_hash_rec_update(struct ftrace_ops *ops, | 1630 | static bool __ftrace_hash_rec_update(struct ftrace_ops *ops, |
1626 | int filter_hash, | 1631 | int filter_hash, |
1627 | bool inc) | 1632 | bool inc) |
@@ -1750,15 +1755,17 @@ static bool __ftrace_hash_rec_update(struct ftrace_ops *ops, | |||
1750 | } | 1755 | } |
1751 | 1756 | ||
1752 | /* | 1757 | /* |
1753 | * If the rec had TRAMP enabled, then it needs to | 1758 | * The TRAMP needs to be set only if rec count |
1754 | * be cleared. As TRAMP can only be enabled iff | 1759 | * is decremented to one, and the ops that is |
1755 | * there is only a single ops attached to it. | 1760 | * left has a trampoline. As TRAMP can only be |
1756 | * In otherwords, always disable it on decrementing. | 1761 | * enabled if there is only a single ops attached |
1757 | * In the future, we may set it if rec count is | 1762 | * to it. |
1758 | * decremented to one, and the ops that is left | ||
1759 | * has a trampoline. | ||
1760 | */ | 1763 | */ |
1761 | rec->flags &= ~FTRACE_FL_TRAMP; | 1764 | if (ftrace_rec_count(rec) == 1 && |
1765 | ftrace_find_tramp_ops_any(rec)) | ||
1766 | rec->flags |= FTRACE_FL_TRAMP; | ||
1767 | else | ||
1768 | rec->flags &= ~FTRACE_FL_TRAMP; | ||
1762 | 1769 | ||
1763 | /* | 1770 | /* |
1764 | * flags will be cleared in ftrace_check_record() | 1771 | * flags will be cleared in ftrace_check_record() |
@@ -1768,7 +1775,7 @@ static bool __ftrace_hash_rec_update(struct ftrace_ops *ops, | |||
1768 | count++; | 1775 | count++; |
1769 | 1776 | ||
1770 | /* Must match FTRACE_UPDATE_CALLS in ftrace_modify_all_code() */ | 1777 | /* Must match FTRACE_UPDATE_CALLS in ftrace_modify_all_code() */ |
1771 | update |= ftrace_test_record(rec, 1) != FTRACE_UPDATE_IGNORE; | 1778 | update |= ftrace_test_record(rec, true) != FTRACE_UPDATE_IGNORE; |
1772 | 1779 | ||
1773 | /* Shortcut, if we handled all records, we are done. */ | 1780 | /* Shortcut, if we handled all records, we are done. */ |
1774 | if (!all && count == hash->count) | 1781 | if (!all && count == hash->count) |
@@ -1951,11 +1958,6 @@ static void print_ip_ins(const char *fmt, const unsigned char *p) | |||
1951 | printk(KERN_CONT "%s%02x", i ? ":" : "", p[i]); | 1958 | printk(KERN_CONT "%s%02x", i ? ":" : "", p[i]); |
1952 | } | 1959 | } |
1953 | 1960 | ||
1954 | static struct ftrace_ops * | ||
1955 | ftrace_find_tramp_ops_any(struct dyn_ftrace *rec); | ||
1956 | static struct ftrace_ops * | ||
1957 | ftrace_find_tramp_ops_next(struct dyn_ftrace *rec, struct ftrace_ops *ops); | ||
1958 | |||
1959 | enum ftrace_bug_type ftrace_bug_type; | 1961 | enum ftrace_bug_type ftrace_bug_type; |
1960 | const void *ftrace_expected; | 1962 | const void *ftrace_expected; |
1961 | 1963 | ||
@@ -2047,7 +2049,7 @@ void ftrace_bug(int failed, struct dyn_ftrace *rec) | |||
2047 | } | 2049 | } |
2048 | } | 2050 | } |
2049 | 2051 | ||
2050 | static int ftrace_check_record(struct dyn_ftrace *rec, int enable, int update) | 2052 | static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) |
2051 | { | 2053 | { |
2052 | unsigned long flag = 0UL; | 2054 | unsigned long flag = 0UL; |
2053 | 2055 | ||
@@ -2146,28 +2148,28 @@ static int ftrace_check_record(struct dyn_ftrace *rec, int enable, int update) | |||
2146 | /** | 2148 | /** |
2147 | * ftrace_update_record, set a record that now is tracing or not | 2149 | * ftrace_update_record, set a record that now is tracing or not |
2148 | * @rec: the record to update | 2150 | * @rec: the record to update |
2149 | * @enable: set to 1 if the record is tracing, zero to force disable | 2151 | * @enable: set to true if the record is tracing, false to force disable |
2150 | * | 2152 | * |
2151 | * The records that represent all functions that can be traced need | 2153 | * The records that represent all functions that can be traced need |
2152 | * to be updated when tracing has been enabled. | 2154 | * to be updated when tracing has been enabled. |
2153 | */ | 2155 | */ |
2154 | int ftrace_update_record(struct dyn_ftrace *rec, int enable) | 2156 | int ftrace_update_record(struct dyn_ftrace *rec, bool enable) |
2155 | { | 2157 | { |
2156 | return ftrace_check_record(rec, enable, 1); | 2158 | return ftrace_check_record(rec, enable, true); |
2157 | } | 2159 | } |
2158 | 2160 | ||
2159 | /** | 2161 | /** |
2160 | * ftrace_test_record, check if the record has been enabled or not | 2162 | * ftrace_test_record, check if the record has been enabled or not |
2161 | * @rec: the record to test | 2163 | * @rec: the record to test |
2162 | * @enable: set to 1 to check if enabled, 0 if it is disabled | 2164 | * @enable: set to true to check if enabled, false if it is disabled |
2163 | * | 2165 | * |
2164 | * The arch code may need to test if a record is already set to | 2166 | * The arch code may need to test if a record is already set to |
2165 | * tracing to determine how to modify the function code that it | 2167 | * tracing to determine how to modify the function code that it |
2166 | * represents. | 2168 | * represents. |
2167 | */ | 2169 | */ |
2168 | int ftrace_test_record(struct dyn_ftrace *rec, int enable) | 2170 | int ftrace_test_record(struct dyn_ftrace *rec, bool enable) |
2169 | { | 2171 | { |
2170 | return ftrace_check_record(rec, enable, 0); | 2172 | return ftrace_check_record(rec, enable, false); |
2171 | } | 2173 | } |
2172 | 2174 | ||
2173 | static struct ftrace_ops * | 2175 | static struct ftrace_ops * |
@@ -2356,7 +2358,7 @@ unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec) | |||
2356 | } | 2358 | } |
2357 | 2359 | ||
2358 | static int | 2360 | static int |
2359 | __ftrace_replace_code(struct dyn_ftrace *rec, int enable) | 2361 | __ftrace_replace_code(struct dyn_ftrace *rec, bool enable) |
2360 | { | 2362 | { |
2361 | unsigned long ftrace_old_addr; | 2363 | unsigned long ftrace_old_addr; |
2362 | unsigned long ftrace_addr; | 2364 | unsigned long ftrace_addr; |
@@ -2395,7 +2397,7 @@ void __weak ftrace_replace_code(int mod_flags) | |||
2395 | { | 2397 | { |
2396 | struct dyn_ftrace *rec; | 2398 | struct dyn_ftrace *rec; |
2397 | struct ftrace_page *pg; | 2399 | struct ftrace_page *pg; |
2398 | int enable = mod_flags & FTRACE_MODIFY_ENABLE_FL; | 2400 | bool enable = mod_flags & FTRACE_MODIFY_ENABLE_FL; |
2399 | int schedulable = mod_flags & FTRACE_MODIFY_MAY_SLEEP_FL; | 2401 | int schedulable = mod_flags & FTRACE_MODIFY_MAY_SLEEP_FL; |
2400 | int failed; | 2402 | int failed; |
2401 | 2403 | ||
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c index 05b0b3139ebc..66358d66c933 100644 --- a/kernel/trace/ring_buffer.c +++ b/kernel/trace/ring_buffer.c | |||
@@ -128,16 +128,7 @@ int ring_buffer_print_entry_header(struct trace_seq *s) | |||
128 | #define RB_ALIGNMENT 4U | 128 | #define RB_ALIGNMENT 4U |
129 | #define RB_MAX_SMALL_DATA (RB_ALIGNMENT * RINGBUF_TYPE_DATA_TYPE_LEN_MAX) | 129 | #define RB_MAX_SMALL_DATA (RB_ALIGNMENT * RINGBUF_TYPE_DATA_TYPE_LEN_MAX) |
130 | #define RB_EVNT_MIN_SIZE 8U /* two 32bit words */ | 130 | #define RB_EVNT_MIN_SIZE 8U /* two 32bit words */ |
131 | 131 | #define RB_ALIGN_DATA __aligned(RB_ALIGNMENT) | |
132 | #ifndef CONFIG_HAVE_64BIT_ALIGNED_ACCESS | ||
133 | # define RB_FORCE_8BYTE_ALIGNMENT 0 | ||
134 | # define RB_ARCH_ALIGNMENT RB_ALIGNMENT | ||
135 | #else | ||
136 | # define RB_FORCE_8BYTE_ALIGNMENT 1 | ||
137 | # define RB_ARCH_ALIGNMENT 8U | ||
138 | #endif | ||
139 | |||
140 | #define RB_ALIGN_DATA __aligned(RB_ARCH_ALIGNMENT) | ||
141 | 132 | ||
142 | /* define RINGBUF_TYPE_DATA for 'case RINGBUF_TYPE_DATA:' */ | 133 | /* define RINGBUF_TYPE_DATA for 'case RINGBUF_TYPE_DATA:' */ |
143 | #define RINGBUF_TYPE_DATA 0 ... RINGBUF_TYPE_DATA_TYPE_LEN_MAX | 134 | #define RINGBUF_TYPE_DATA 0 ... RINGBUF_TYPE_DATA_TYPE_LEN_MAX |
@@ -2373,7 +2364,7 @@ rb_update_event(struct ring_buffer_per_cpu *cpu_buffer, | |||
2373 | 2364 | ||
2374 | event->time_delta = delta; | 2365 | event->time_delta = delta; |
2375 | length -= RB_EVNT_HDR_SIZE; | 2366 | length -= RB_EVNT_HDR_SIZE; |
2376 | if (length > RB_MAX_SMALL_DATA || RB_FORCE_8BYTE_ALIGNMENT) { | 2367 | if (length > RB_MAX_SMALL_DATA) { |
2377 | event->type_len = 0; | 2368 | event->type_len = 0; |
2378 | event->array[0] = length; | 2369 | event->array[0] = length; |
2379 | } else | 2370 | } else |
@@ -2388,11 +2379,11 @@ static unsigned rb_calculate_event_length(unsigned length) | |||
2388 | if (!length) | 2379 | if (!length) |
2389 | length++; | 2380 | length++; |
2390 | 2381 | ||
2391 | if (length > RB_MAX_SMALL_DATA || RB_FORCE_8BYTE_ALIGNMENT) | 2382 | if (length > RB_MAX_SMALL_DATA) |
2392 | length += sizeof(event.array[0]); | 2383 | length += sizeof(event.array[0]); |
2393 | 2384 | ||
2394 | length += RB_EVNT_HDR_SIZE; | 2385 | length += RB_EVNT_HDR_SIZE; |
2395 | length = ALIGN(length, RB_ARCH_ALIGNMENT); | 2386 | length = ALIGN(length, RB_ALIGNMENT); |
2396 | 2387 | ||
2397 | /* | 2388 | /* |
2398 | * In case the time delta is larger than the 27 bits for it | 2389 | * In case the time delta is larger than the 27 bits for it |
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index c90c687cf950..525a97fbbc60 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c | |||
@@ -366,7 +366,7 @@ trace_ignore_this_task(struct trace_pid_list *filtered_pids, struct task_struct | |||
366 | } | 366 | } |
367 | 367 | ||
368 | /** | 368 | /** |
369 | * trace_pid_filter_add_remove_task - Add or remove a task from a pid_list | 369 | * trace_filter_add_remove_task - Add or remove a task from a pid_list |
370 | * @pid_list: The list to modify | 370 | * @pid_list: The list to modify |
371 | * @self: The current task for fork or NULL for exit | 371 | * @self: The current task for fork or NULL for exit |
372 | * @task: The task to add or remove | 372 | * @task: The task to add or remove |
@@ -743,8 +743,7 @@ trace_event_setup(struct ring_buffer_event *event, | |||
743 | { | 743 | { |
744 | struct trace_entry *ent = ring_buffer_event_data(event); | 744 | struct trace_entry *ent = ring_buffer_event_data(event); |
745 | 745 | ||
746 | tracing_generic_entry_update(ent, flags, pc); | 746 | tracing_generic_entry_update(ent, type, flags, pc); |
747 | ent->type = type; | ||
748 | } | 747 | } |
749 | 748 | ||
750 | static __always_inline struct ring_buffer_event * | 749 | static __always_inline struct ring_buffer_event * |
@@ -2312,13 +2311,14 @@ enum print_line_t trace_handle_return(struct trace_seq *s) | |||
2312 | EXPORT_SYMBOL_GPL(trace_handle_return); | 2311 | EXPORT_SYMBOL_GPL(trace_handle_return); |
2313 | 2312 | ||
2314 | void | 2313 | void |
2315 | tracing_generic_entry_update(struct trace_entry *entry, unsigned long flags, | 2314 | tracing_generic_entry_update(struct trace_entry *entry, unsigned short type, |
2316 | int pc) | 2315 | unsigned long flags, int pc) |
2317 | { | 2316 | { |
2318 | struct task_struct *tsk = current; | 2317 | struct task_struct *tsk = current; |
2319 | 2318 | ||
2320 | entry->preempt_count = pc & 0xff; | 2319 | entry->preempt_count = pc & 0xff; |
2321 | entry->pid = (tsk) ? tsk->pid : 0; | 2320 | entry->pid = (tsk) ? tsk->pid : 0; |
2321 | entry->type = type; | ||
2322 | entry->flags = | 2322 | entry->flags = |
2323 | #ifdef CONFIG_TRACE_IRQFLAGS_SUPPORT | 2323 | #ifdef CONFIG_TRACE_IRQFLAGS_SUPPORT |
2324 | (irqs_disabled_flags(flags) ? TRACE_FLAG_IRQS_OFF : 0) | | 2324 | (irqs_disabled_flags(flags) ? TRACE_FLAG_IRQS_OFF : 0) | |
@@ -4842,12 +4842,13 @@ static const char readme_msg[] = | |||
4842 | "\t args: <name>=fetcharg[:type]\n" | 4842 | "\t args: <name>=fetcharg[:type]\n" |
4843 | "\t fetcharg: %<register>, @<address>, @<symbol>[+|-<offset>],\n" | 4843 | "\t fetcharg: %<register>, @<address>, @<symbol>[+|-<offset>],\n" |
4844 | #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API | 4844 | #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API |
4845 | "\t $stack<index>, $stack, $retval, $comm, $arg<N>\n" | 4845 | "\t $stack<index>, $stack, $retval, $comm, $arg<N>,\n" |
4846 | #else | 4846 | #else |
4847 | "\t $stack<index>, $stack, $retval, $comm\n" | 4847 | "\t $stack<index>, $stack, $retval, $comm,\n" |
4848 | #endif | 4848 | #endif |
4849 | "\t +|-[u]<offset>(<fetcharg>)\n" | ||
4849 | "\t type: s8/16/32/64, u8/16/32/64, x8/16/32/64, string, symbol,\n" | 4850 | "\t type: s8/16/32/64, u8/16/32/64, x8/16/32/64, string, symbol,\n" |
4850 | "\t b<bit-width>@<bit-offset>/<container-size>,\n" | 4851 | "\t b<bit-width>@<bit-offset>/<container-size>, ustring,\n" |
4851 | "\t <type>\\[<array-size>\\]\n" | 4852 | "\t <type>\\[<array-size>\\]\n" |
4852 | #ifdef CONFIG_HIST_TRIGGERS | 4853 | #ifdef CONFIG_HIST_TRIGGERS |
4853 | "\t field: <stype> <name>;\n" | 4854 | "\t field: <stype> <name>;\n" |
diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c index 4629a6104474..0892e38ed6fb 100644 --- a/kernel/trace/trace_event_perf.c +++ b/kernel/trace/trace_event_perf.c | |||
@@ -416,8 +416,7 @@ void perf_trace_buf_update(void *record, u16 type) | |||
416 | unsigned long flags; | 416 | unsigned long flags; |
417 | 417 | ||
418 | local_save_flags(flags); | 418 | local_save_flags(flags); |
419 | tracing_generic_entry_update(entry, flags, pc); | 419 | tracing_generic_entry_update(entry, type, flags, pc); |
420 | entry->type = type; | ||
421 | } | 420 | } |
422 | NOKPROBE_SYMBOL(perf_trace_buf_update); | 421 | NOKPROBE_SYMBOL(perf_trace_buf_update); |
423 | 422 | ||
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c index 0ce3db67f556..c7506bc81b75 100644 --- a/kernel/trace/trace_events.c +++ b/kernel/trace/trace_events.c | |||
@@ -70,14 +70,6 @@ static int system_refcount_dec(struct event_subsystem *system) | |||
70 | #define while_for_each_event_file() \ | 70 | #define while_for_each_event_file() \ |
71 | } | 71 | } |
72 | 72 | ||
73 | static struct list_head * | ||
74 | trace_get_fields(struct trace_event_call *event_call) | ||
75 | { | ||
76 | if (!event_call->class->get_fields) | ||
77 | return &event_call->class->fields; | ||
78 | return event_call->class->get_fields(event_call); | ||
79 | } | ||
80 | |||
81 | static struct ftrace_event_field * | 73 | static struct ftrace_event_field * |
82 | __find_event_field(struct list_head *head, char *name) | 74 | __find_event_field(struct list_head *head, char *name) |
83 | { | 75 | { |
@@ -3190,7 +3182,7 @@ void __init trace_event_init(void) | |||
3190 | event_trace_enable(); | 3182 | event_trace_enable(); |
3191 | } | 3183 | } |
3192 | 3184 | ||
3193 | #ifdef CONFIG_FTRACE_STARTUP_TEST | 3185 | #ifdef CONFIG_EVENT_TRACE_STARTUP_TEST |
3194 | 3186 | ||
3195 | static DEFINE_SPINLOCK(test_spinlock); | 3187 | static DEFINE_SPINLOCK(test_spinlock); |
3196 | static DEFINE_SPINLOCK(test_spinlock_irq); | 3188 | static DEFINE_SPINLOCK(test_spinlock_irq); |
diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 5079d1db3754..c773b8fb270c 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c | |||
@@ -1084,6 +1084,9 @@ int filter_assign_type(const char *type) | |||
1084 | if (strchr(type, '[') && strstr(type, "char")) | 1084 | if (strchr(type, '[') && strstr(type, "char")) |
1085 | return FILTER_STATIC_STRING; | 1085 | return FILTER_STATIC_STRING; |
1086 | 1086 | ||
1087 | if (strcmp(type, "char *") == 0 || strcmp(type, "const char *") == 0) | ||
1088 | return FILTER_PTR_STRING; | ||
1089 | |||
1087 | return FILTER_OTHER; | 1090 | return FILTER_OTHER; |
1088 | } | 1091 | } |
1089 | 1092 | ||
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c index 7d736248a070..9d483ad9bb6c 100644 --- a/kernel/trace/trace_kprobe.c +++ b/kernel/trace/trace_kprobe.c | |||
@@ -12,6 +12,8 @@ | |||
12 | #include <linux/rculist.h> | 12 | #include <linux/rculist.h> |
13 | #include <linux/error-injection.h> | 13 | #include <linux/error-injection.h> |
14 | 14 | ||
15 | #include <asm/setup.h> /* for COMMAND_LINE_SIZE */ | ||
16 | |||
15 | #include "trace_dynevent.h" | 17 | #include "trace_dynevent.h" |
16 | #include "trace_kprobe_selftest.h" | 18 | #include "trace_kprobe_selftest.h" |
17 | #include "trace_probe.h" | 19 | #include "trace_probe.h" |
@@ -19,6 +21,18 @@ | |||
19 | 21 | ||
20 | #define KPROBE_EVENT_SYSTEM "kprobes" | 22 | #define KPROBE_EVENT_SYSTEM "kprobes" |
21 | #define KRETPROBE_MAXACTIVE_MAX 4096 | 23 | #define KRETPROBE_MAXACTIVE_MAX 4096 |
24 | #define MAX_KPROBE_CMDLINE_SIZE 1024 | ||
25 | |||
26 | /* Kprobe early definition from command line */ | ||
27 | static char kprobe_boot_events_buf[COMMAND_LINE_SIZE] __initdata; | ||
28 | static bool kprobe_boot_events_enabled __initdata; | ||
29 | |||
30 | static int __init set_kprobe_boot_events(char *str) | ||
31 | { | ||
32 | strlcpy(kprobe_boot_events_buf, str, COMMAND_LINE_SIZE); | ||
33 | return 0; | ||
34 | } | ||
35 | __setup("kprobe_event=", set_kprobe_boot_events); | ||
22 | 36 | ||
23 | static int trace_kprobe_create(int argc, const char **argv); | 37 | static int trace_kprobe_create(int argc, const char **argv); |
24 | static int trace_kprobe_show(struct seq_file *m, struct dyn_event *ev); | 38 | static int trace_kprobe_show(struct seq_file *m, struct dyn_event *ev); |
@@ -128,8 +142,8 @@ static bool trace_kprobe_match(const char *system, const char *event, | |||
128 | { | 142 | { |
129 | struct trace_kprobe *tk = to_trace_kprobe(ev); | 143 | struct trace_kprobe *tk = to_trace_kprobe(ev); |
130 | 144 | ||
131 | return strcmp(trace_event_name(&tk->tp.call), event) == 0 && | 145 | return strcmp(trace_probe_name(&tk->tp), event) == 0 && |
132 | (!system || strcmp(tk->tp.call.class->system, system) == 0); | 146 | (!system || strcmp(trace_probe_group_name(&tk->tp), system) == 0); |
133 | } | 147 | } |
134 | 148 | ||
135 | static nokprobe_inline unsigned long trace_kprobe_nhit(struct trace_kprobe *tk) | 149 | static nokprobe_inline unsigned long trace_kprobe_nhit(struct trace_kprobe *tk) |
@@ -143,6 +157,12 @@ static nokprobe_inline unsigned long trace_kprobe_nhit(struct trace_kprobe *tk) | |||
143 | return nhit; | 157 | return nhit; |
144 | } | 158 | } |
145 | 159 | ||
160 | static nokprobe_inline bool trace_kprobe_is_registered(struct trace_kprobe *tk) | ||
161 | { | ||
162 | return !(list_empty(&tk->rp.kp.list) && | ||
163 | hlist_unhashed(&tk->rp.kp.hlist)); | ||
164 | } | ||
165 | |||
146 | /* Return 0 if it fails to find the symbol address */ | 166 | /* Return 0 if it fails to find the symbol address */ |
147 | static nokprobe_inline | 167 | static nokprobe_inline |
148 | unsigned long trace_kprobe_address(struct trace_kprobe *tk) | 168 | unsigned long trace_kprobe_address(struct trace_kprobe *tk) |
@@ -183,6 +203,16 @@ static int kprobe_dispatcher(struct kprobe *kp, struct pt_regs *regs); | |||
183 | static int kretprobe_dispatcher(struct kretprobe_instance *ri, | 203 | static int kretprobe_dispatcher(struct kretprobe_instance *ri, |
184 | struct pt_regs *regs); | 204 | struct pt_regs *regs); |
185 | 205 | ||
206 | static void free_trace_kprobe(struct trace_kprobe *tk) | ||
207 | { | ||
208 | if (tk) { | ||
209 | trace_probe_cleanup(&tk->tp); | ||
210 | kfree(tk->symbol); | ||
211 | free_percpu(tk->nhit); | ||
212 | kfree(tk); | ||
213 | } | ||
214 | } | ||
215 | |||
186 | /* | 216 | /* |
187 | * Allocate new trace_probe and initialize it (including kprobes). | 217 | * Allocate new trace_probe and initialize it (including kprobes). |
188 | */ | 218 | */ |
@@ -220,49 +250,20 @@ static struct trace_kprobe *alloc_trace_kprobe(const char *group, | |||
220 | tk->rp.kp.pre_handler = kprobe_dispatcher; | 250 | tk->rp.kp.pre_handler = kprobe_dispatcher; |
221 | 251 | ||
222 | tk->rp.maxactive = maxactive; | 252 | tk->rp.maxactive = maxactive; |
253 | INIT_HLIST_NODE(&tk->rp.kp.hlist); | ||
254 | INIT_LIST_HEAD(&tk->rp.kp.list); | ||
223 | 255 | ||
224 | if (!event || !group) { | 256 | ret = trace_probe_init(&tk->tp, event, group); |
225 | ret = -EINVAL; | 257 | if (ret < 0) |
226 | goto error; | ||
227 | } | ||
228 | |||
229 | tk->tp.call.class = &tk->tp.class; | ||
230 | tk->tp.call.name = kstrdup(event, GFP_KERNEL); | ||
231 | if (!tk->tp.call.name) | ||
232 | goto error; | ||
233 | |||
234 | tk->tp.class.system = kstrdup(group, GFP_KERNEL); | ||
235 | if (!tk->tp.class.system) | ||
236 | goto error; | 258 | goto error; |
237 | 259 | ||
238 | dyn_event_init(&tk->devent, &trace_kprobe_ops); | 260 | dyn_event_init(&tk->devent, &trace_kprobe_ops); |
239 | INIT_LIST_HEAD(&tk->tp.files); | ||
240 | return tk; | 261 | return tk; |
241 | error: | 262 | error: |
242 | kfree(tk->tp.call.name); | 263 | free_trace_kprobe(tk); |
243 | kfree(tk->symbol); | ||
244 | free_percpu(tk->nhit); | ||
245 | kfree(tk); | ||
246 | return ERR_PTR(ret); | 264 | return ERR_PTR(ret); |
247 | } | 265 | } |
248 | 266 | ||
249 | static void free_trace_kprobe(struct trace_kprobe *tk) | ||
250 | { | ||
251 | int i; | ||
252 | |||
253 | if (!tk) | ||
254 | return; | ||
255 | |||
256 | for (i = 0; i < tk->tp.nr_args; i++) | ||
257 | traceprobe_free_probe_arg(&tk->tp.args[i]); | ||
258 | |||
259 | kfree(tk->tp.call.class->system); | ||
260 | kfree(tk->tp.call.name); | ||
261 | kfree(tk->symbol); | ||
262 | free_percpu(tk->nhit); | ||
263 | kfree(tk); | ||
264 | } | ||
265 | |||
266 | static struct trace_kprobe *find_trace_kprobe(const char *event, | 267 | static struct trace_kprobe *find_trace_kprobe(const char *event, |
267 | const char *group) | 268 | const char *group) |
268 | { | 269 | { |
@@ -270,8 +271,8 @@ static struct trace_kprobe *find_trace_kprobe(const char *event, | |||
270 | struct trace_kprobe *tk; | 271 | struct trace_kprobe *tk; |
271 | 272 | ||
272 | for_each_trace_kprobe(tk, pos) | 273 | for_each_trace_kprobe(tk, pos) |
273 | if (strcmp(trace_event_name(&tk->tp.call), event) == 0 && | 274 | if (strcmp(trace_probe_name(&tk->tp), event) == 0 && |
274 | strcmp(tk->tp.call.class->system, group) == 0) | 275 | strcmp(trace_probe_group_name(&tk->tp), group) == 0) |
275 | return tk; | 276 | return tk; |
276 | return NULL; | 277 | return NULL; |
277 | } | 278 | } |
@@ -280,7 +281,7 @@ static inline int __enable_trace_kprobe(struct trace_kprobe *tk) | |||
280 | { | 281 | { |
281 | int ret = 0; | 282 | int ret = 0; |
282 | 283 | ||
283 | if (trace_probe_is_registered(&tk->tp) && !trace_kprobe_has_gone(tk)) { | 284 | if (trace_kprobe_is_registered(tk) && !trace_kprobe_has_gone(tk)) { |
284 | if (trace_kprobe_is_return(tk)) | 285 | if (trace_kprobe_is_return(tk)) |
285 | ret = enable_kretprobe(&tk->rp); | 286 | ret = enable_kretprobe(&tk->rp); |
286 | else | 287 | else |
@@ -297,34 +298,27 @@ static inline int __enable_trace_kprobe(struct trace_kprobe *tk) | |||
297 | static int | 298 | static int |
298 | enable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file) | 299 | enable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file) |
299 | { | 300 | { |
300 | struct event_file_link *link; | 301 | bool enabled = trace_probe_is_enabled(&tk->tp); |
301 | int ret = 0; | 302 | int ret = 0; |
302 | 303 | ||
303 | if (file) { | 304 | if (file) { |
304 | link = kmalloc(sizeof(*link), GFP_KERNEL); | 305 | ret = trace_probe_add_file(&tk->tp, file); |
305 | if (!link) { | 306 | if (ret) |
306 | ret = -ENOMEM; | 307 | return ret; |
307 | goto out; | 308 | } else |
308 | } | 309 | trace_probe_set_flag(&tk->tp, TP_FLAG_PROFILE); |
309 | |||
310 | link->file = file; | ||
311 | list_add_tail_rcu(&link->list, &tk->tp.files); | ||
312 | 310 | ||
313 | tk->tp.flags |= TP_FLAG_TRACE; | 311 | if (enabled) |
314 | ret = __enable_trace_kprobe(tk); | 312 | return 0; |
315 | if (ret) { | ||
316 | list_del_rcu(&link->list); | ||
317 | kfree(link); | ||
318 | tk->tp.flags &= ~TP_FLAG_TRACE; | ||
319 | } | ||
320 | 313 | ||
321 | } else { | 314 | ret = __enable_trace_kprobe(tk); |
322 | tk->tp.flags |= TP_FLAG_PROFILE; | 315 | if (ret) { |
323 | ret = __enable_trace_kprobe(tk); | 316 | if (file) |
324 | if (ret) | 317 | trace_probe_remove_file(&tk->tp, file); |
325 | tk->tp.flags &= ~TP_FLAG_PROFILE; | 318 | else |
319 | trace_probe_clear_flag(&tk->tp, TP_FLAG_PROFILE); | ||
326 | } | 320 | } |
327 | out: | 321 | |
328 | return ret; | 322 | return ret; |
329 | } | 323 | } |
330 | 324 | ||
@@ -335,54 +329,34 @@ enable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file) | |||
335 | static int | 329 | static int |
336 | disable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file) | 330 | disable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file) |
337 | { | 331 | { |
338 | struct event_file_link *link = NULL; | 332 | struct trace_probe *tp = &tk->tp; |
339 | int wait = 0; | ||
340 | int ret = 0; | 333 | int ret = 0; |
341 | 334 | ||
342 | if (file) { | 335 | if (file) { |
343 | link = find_event_file_link(&tk->tp, file); | 336 | if (!trace_probe_get_file_link(tp, file)) |
344 | if (!link) { | 337 | return -ENOENT; |
345 | ret = -EINVAL; | 338 | if (!trace_probe_has_single_file(tp)) |
346 | goto out; | ||
347 | } | ||
348 | |||
349 | list_del_rcu(&link->list); | ||
350 | wait = 1; | ||
351 | if (!list_empty(&tk->tp.files)) | ||
352 | goto out; | 339 | goto out; |
353 | 340 | trace_probe_clear_flag(tp, TP_FLAG_TRACE); | |
354 | tk->tp.flags &= ~TP_FLAG_TRACE; | ||
355 | } else | 341 | } else |
356 | tk->tp.flags &= ~TP_FLAG_PROFILE; | 342 | trace_probe_clear_flag(tp, TP_FLAG_PROFILE); |
357 | 343 | ||
358 | if (!trace_probe_is_enabled(&tk->tp) && trace_probe_is_registered(&tk->tp)) { | 344 | if (!trace_probe_is_enabled(tp) && trace_kprobe_is_registered(tk)) { |
359 | if (trace_kprobe_is_return(tk)) | 345 | if (trace_kprobe_is_return(tk)) |
360 | disable_kretprobe(&tk->rp); | 346 | disable_kretprobe(&tk->rp); |
361 | else | 347 | else |
362 | disable_kprobe(&tk->rp.kp); | 348 | disable_kprobe(&tk->rp.kp); |
363 | wait = 1; | ||
364 | } | 349 | } |
365 | 350 | ||
366 | /* | ||
367 | * if tk is not added to any list, it must be a local trace_kprobe | ||
368 | * created with perf_event_open. We don't need to wait for these | ||
369 | * trace_kprobes | ||
370 | */ | ||
371 | if (list_empty(&tk->devent.list)) | ||
372 | wait = 0; | ||
373 | out: | 351 | out: |
374 | if (wait) { | 352 | if (file) |
375 | /* | 353 | /* |
376 | * Synchronize with kprobe_trace_func/kretprobe_trace_func | 354 | * Synchronization is done in below function. For perf event, |
377 | * to ensure disabled (all running handlers are finished). | 355 | * file == NULL and perf_trace_event_unreg() calls |
378 | * This is not only for kfree(), but also the caller, | 356 | * tracepoint_synchronize_unregister() to ensure synchronize |
379 | * trace_remove_event_call() supposes it for releasing | 357 | * event. We don't need to care about it. |
380 | * event_call related objects, which will be accessed in | ||
381 | * the kprobe_trace_func/kretprobe_trace_func. | ||
382 | */ | 358 | */ |
383 | synchronize_rcu(); | 359 | trace_probe_remove_file(tp, file); |
384 | kfree(link); /* Ignored if link == NULL */ | ||
385 | } | ||
386 | 360 | ||
387 | return ret; | 361 | return ret; |
388 | } | 362 | } |
@@ -415,7 +389,7 @@ static int __register_trace_kprobe(struct trace_kprobe *tk) | |||
415 | { | 389 | { |
416 | int i, ret; | 390 | int i, ret; |
417 | 391 | ||
418 | if (trace_probe_is_registered(&tk->tp)) | 392 | if (trace_kprobe_is_registered(tk)) |
419 | return -EINVAL; | 393 | return -EINVAL; |
420 | 394 | ||
421 | if (within_notrace_func(tk)) { | 395 | if (within_notrace_func(tk)) { |
@@ -441,21 +415,20 @@ static int __register_trace_kprobe(struct trace_kprobe *tk) | |||
441 | else | 415 | else |
442 | ret = register_kprobe(&tk->rp.kp); | 416 | ret = register_kprobe(&tk->rp.kp); |
443 | 417 | ||
444 | if (ret == 0) | ||
445 | tk->tp.flags |= TP_FLAG_REGISTERED; | ||
446 | return ret; | 418 | return ret; |
447 | } | 419 | } |
448 | 420 | ||
449 | /* Internal unregister function - just handle k*probes and flags */ | 421 | /* Internal unregister function - just handle k*probes and flags */ |
450 | static void __unregister_trace_kprobe(struct trace_kprobe *tk) | 422 | static void __unregister_trace_kprobe(struct trace_kprobe *tk) |
451 | { | 423 | { |
452 | if (trace_probe_is_registered(&tk->tp)) { | 424 | if (trace_kprobe_is_registered(tk)) { |
453 | if (trace_kprobe_is_return(tk)) | 425 | if (trace_kprobe_is_return(tk)) |
454 | unregister_kretprobe(&tk->rp); | 426 | unregister_kretprobe(&tk->rp); |
455 | else | 427 | else |
456 | unregister_kprobe(&tk->rp.kp); | 428 | unregister_kprobe(&tk->rp.kp); |
457 | tk->tp.flags &= ~TP_FLAG_REGISTERED; | 429 | /* Cleanup kprobe for reuse and mark it unregistered */ |
458 | /* Cleanup kprobe for reuse */ | 430 | INIT_HLIST_NODE(&tk->rp.kp.hlist); |
431 | INIT_LIST_HEAD(&tk->rp.kp.list); | ||
459 | if (tk->rp.kp.symbol_name) | 432 | if (tk->rp.kp.symbol_name) |
460 | tk->rp.kp.addr = NULL; | 433 | tk->rp.kp.addr = NULL; |
461 | } | 434 | } |
@@ -487,8 +460,8 @@ static int register_trace_kprobe(struct trace_kprobe *tk) | |||
487 | mutex_lock(&event_mutex); | 460 | mutex_lock(&event_mutex); |
488 | 461 | ||
489 | /* Delete old (same name) event if exist */ | 462 | /* Delete old (same name) event if exist */ |
490 | old_tk = find_trace_kprobe(trace_event_name(&tk->tp.call), | 463 | old_tk = find_trace_kprobe(trace_probe_name(&tk->tp), |
491 | tk->tp.call.class->system); | 464 | trace_probe_group_name(&tk->tp)); |
492 | if (old_tk) { | 465 | if (old_tk) { |
493 | ret = unregister_trace_kprobe(old_tk); | 466 | ret = unregister_trace_kprobe(old_tk); |
494 | if (ret < 0) | 467 | if (ret < 0) |
@@ -541,7 +514,7 @@ static int trace_kprobe_module_callback(struct notifier_block *nb, | |||
541 | ret = __register_trace_kprobe(tk); | 514 | ret = __register_trace_kprobe(tk); |
542 | if (ret) | 515 | if (ret) |
543 | pr_warn("Failed to re-register probe %s on %s: %d\n", | 516 | pr_warn("Failed to re-register probe %s on %s: %d\n", |
544 | trace_event_name(&tk->tp.call), | 517 | trace_probe_name(&tk->tp), |
545 | mod->name, ret); | 518 | mod->name, ret); |
546 | } | 519 | } |
547 | } | 520 | } |
@@ -716,6 +689,10 @@ static int trace_kprobe_create(int argc, const char *argv[]) | |||
716 | goto error; /* This can be -ENOMEM */ | 689 | goto error; /* This can be -ENOMEM */ |
717 | } | 690 | } |
718 | 691 | ||
692 | ret = traceprobe_set_print_fmt(&tk->tp, is_return); | ||
693 | if (ret < 0) | ||
694 | goto error; | ||
695 | |||
719 | ret = register_trace_kprobe(tk); | 696 | ret = register_trace_kprobe(tk); |
720 | if (ret) { | 697 | if (ret) { |
721 | trace_probe_log_set_index(1); | 698 | trace_probe_log_set_index(1); |
@@ -767,8 +744,8 @@ static int trace_kprobe_show(struct seq_file *m, struct dyn_event *ev) | |||
767 | int i; | 744 | int i; |
768 | 745 | ||
769 | seq_putc(m, trace_kprobe_is_return(tk) ? 'r' : 'p'); | 746 | seq_putc(m, trace_kprobe_is_return(tk) ? 'r' : 'p'); |
770 | seq_printf(m, ":%s/%s", tk->tp.call.class->system, | 747 | seq_printf(m, ":%s/%s", trace_probe_group_name(&tk->tp), |
771 | trace_event_name(&tk->tp.call)); | 748 | trace_probe_name(&tk->tp)); |
772 | 749 | ||
773 | if (!tk->symbol) | 750 | if (!tk->symbol) |
774 | seq_printf(m, " 0x%p", tk->rp.kp.addr); | 751 | seq_printf(m, " 0x%p", tk->rp.kp.addr); |
@@ -842,7 +819,7 @@ static int probes_profile_seq_show(struct seq_file *m, void *v) | |||
842 | 819 | ||
843 | tk = to_trace_kprobe(ev); | 820 | tk = to_trace_kprobe(ev); |
844 | seq_printf(m, " %-44s %15lu %15lu\n", | 821 | seq_printf(m, " %-44s %15lu %15lu\n", |
845 | trace_event_name(&tk->tp.call), | 822 | trace_probe_name(&tk->tp), |
846 | trace_kprobe_nhit(tk), | 823 | trace_kprobe_nhit(tk), |
847 | tk->rp.kp.nmissed); | 824 | tk->rp.kp.nmissed); |
848 | 825 | ||
@@ -886,6 +863,15 @@ fetch_store_strlen(unsigned long addr) | |||
886 | return (ret < 0) ? ret : len; | 863 | return (ret < 0) ? ret : len; |
887 | } | 864 | } |
888 | 865 | ||
866 | /* Return the length of string -- including null terminal byte */ | ||
867 | static nokprobe_inline int | ||
868 | fetch_store_strlen_user(unsigned long addr) | ||
869 | { | ||
870 | const void __user *uaddr = (__force const void __user *)addr; | ||
871 | |||
872 | return strnlen_unsafe_user(uaddr, MAX_STRING_SIZE); | ||
873 | } | ||
874 | |||
889 | /* | 875 | /* |
890 | * Fetch a null-terminated string. Caller MUST set *(u32 *)buf with max | 876 | * Fetch a null-terminated string. Caller MUST set *(u32 *)buf with max |
891 | * length and relative data location. | 877 | * length and relative data location. |
@@ -894,19 +880,46 @@ static nokprobe_inline int | |||
894 | fetch_store_string(unsigned long addr, void *dest, void *base) | 880 | fetch_store_string(unsigned long addr, void *dest, void *base) |
895 | { | 881 | { |
896 | int maxlen = get_loc_len(*(u32 *)dest); | 882 | int maxlen = get_loc_len(*(u32 *)dest); |
897 | u8 *dst = get_loc_data(dest, base); | 883 | void *__dest; |
898 | long ret; | 884 | long ret; |
899 | 885 | ||
900 | if (unlikely(!maxlen)) | 886 | if (unlikely(!maxlen)) |
901 | return -ENOMEM; | 887 | return -ENOMEM; |
888 | |||
889 | __dest = get_loc_data(dest, base); | ||
890 | |||
902 | /* | 891 | /* |
903 | * Try to get string again, since the string can be changed while | 892 | * Try to get string again, since the string can be changed while |
904 | * probing. | 893 | * probing. |
905 | */ | 894 | */ |
906 | ret = strncpy_from_unsafe(dst, (void *)addr, maxlen); | 895 | ret = strncpy_from_unsafe(__dest, (void *)addr, maxlen); |
896 | if (ret >= 0) | ||
897 | *(u32 *)dest = make_data_loc(ret, __dest - base); | ||
898 | |||
899 | return ret; | ||
900 | } | ||
907 | 901 | ||
902 | /* | ||
903 | * Fetch a null-terminated string from user. Caller MUST set *(u32 *)buf | ||
904 | * with max length and relative data location. | ||
905 | */ | ||
906 | static nokprobe_inline int | ||
907 | fetch_store_string_user(unsigned long addr, void *dest, void *base) | ||
908 | { | ||
909 | const void __user *uaddr = (__force const void __user *)addr; | ||
910 | int maxlen = get_loc_len(*(u32 *)dest); | ||
911 | void *__dest; | ||
912 | long ret; | ||
913 | |||
914 | if (unlikely(!maxlen)) | ||
915 | return -ENOMEM; | ||
916 | |||
917 | __dest = get_loc_data(dest, base); | ||
918 | |||
919 | ret = strncpy_from_unsafe_user(__dest, uaddr, maxlen); | ||
908 | if (ret >= 0) | 920 | if (ret >= 0) |
909 | *(u32 *)dest = make_data_loc(ret, (void *)dst - base); | 921 | *(u32 *)dest = make_data_loc(ret, __dest - base); |
922 | |||
910 | return ret; | 923 | return ret; |
911 | } | 924 | } |
912 | 925 | ||
@@ -916,6 +929,14 @@ probe_mem_read(void *dest, void *src, size_t size) | |||
916 | return probe_kernel_read(dest, src, size); | 929 | return probe_kernel_read(dest, src, size); |
917 | } | 930 | } |
918 | 931 | ||
932 | static nokprobe_inline int | ||
933 | probe_mem_read_user(void *dest, void *src, size_t size) | ||
934 | { | ||
935 | const void __user *uaddr = (__force const void __user *)src; | ||
936 | |||
937 | return probe_user_read(dest, uaddr, size); | ||
938 | } | ||
939 | |||
919 | /* Note that we don't verify it, since the code does not come from user space */ | 940 | /* Note that we don't verify it, since the code does not come from user space */ |
920 | static int | 941 | static int |
921 | process_fetch_insn(struct fetch_insn *code, struct pt_regs *regs, void *dest, | 942 | process_fetch_insn(struct fetch_insn *code, struct pt_regs *regs, void *dest, |
@@ -971,7 +992,7 @@ __kprobe_trace_func(struct trace_kprobe *tk, struct pt_regs *regs, | |||
971 | struct ring_buffer *buffer; | 992 | struct ring_buffer *buffer; |
972 | int size, dsize, pc; | 993 | int size, dsize, pc; |
973 | unsigned long irq_flags; | 994 | unsigned long irq_flags; |
974 | struct trace_event_call *call = &tk->tp.call; | 995 | struct trace_event_call *call = trace_probe_event_call(&tk->tp); |
975 | 996 | ||
976 | WARN_ON(call != trace_file->event_call); | 997 | WARN_ON(call != trace_file->event_call); |
977 | 998 | ||
@@ -1003,7 +1024,7 @@ kprobe_trace_func(struct trace_kprobe *tk, struct pt_regs *regs) | |||
1003 | { | 1024 | { |
1004 | struct event_file_link *link; | 1025 | struct event_file_link *link; |
1005 | 1026 | ||
1006 | list_for_each_entry_rcu(link, &tk->tp.files, list) | 1027 | trace_probe_for_each_link_rcu(link, &tk->tp) |
1007 | __kprobe_trace_func(tk, regs, link->file); | 1028 | __kprobe_trace_func(tk, regs, link->file); |
1008 | } | 1029 | } |
1009 | NOKPROBE_SYMBOL(kprobe_trace_func); | 1030 | NOKPROBE_SYMBOL(kprobe_trace_func); |
@@ -1019,7 +1040,7 @@ __kretprobe_trace_func(struct trace_kprobe *tk, struct kretprobe_instance *ri, | |||
1019 | struct ring_buffer *buffer; | 1040 | struct ring_buffer *buffer; |
1020 | int size, pc, dsize; | 1041 | int size, pc, dsize; |
1021 | unsigned long irq_flags; | 1042 | unsigned long irq_flags; |
1022 | struct trace_event_call *call = &tk->tp.call; | 1043 | struct trace_event_call *call = trace_probe_event_call(&tk->tp); |
1023 | 1044 | ||
1024 | WARN_ON(call != trace_file->event_call); | 1045 | WARN_ON(call != trace_file->event_call); |
1025 | 1046 | ||
@@ -1053,7 +1074,7 @@ kretprobe_trace_func(struct trace_kprobe *tk, struct kretprobe_instance *ri, | |||
1053 | { | 1074 | { |
1054 | struct event_file_link *link; | 1075 | struct event_file_link *link; |
1055 | 1076 | ||
1056 | list_for_each_entry_rcu(link, &tk->tp.files, list) | 1077 | trace_probe_for_each_link_rcu(link, &tk->tp) |
1057 | __kretprobe_trace_func(tk, ri, regs, link->file); | 1078 | __kretprobe_trace_func(tk, ri, regs, link->file); |
1058 | } | 1079 | } |
1059 | NOKPROBE_SYMBOL(kretprobe_trace_func); | 1080 | NOKPROBE_SYMBOL(kretprobe_trace_func); |
@@ -1070,7 +1091,7 @@ print_kprobe_event(struct trace_iterator *iter, int flags, | |||
1070 | field = (struct kprobe_trace_entry_head *)iter->ent; | 1091 | field = (struct kprobe_trace_entry_head *)iter->ent; |
1071 | tp = container_of(event, struct trace_probe, call.event); | 1092 | tp = container_of(event, struct trace_probe, call.event); |
1072 | 1093 | ||
1073 | trace_seq_printf(s, "%s: (", trace_event_name(&tp->call)); | 1094 | trace_seq_printf(s, "%s: (", trace_probe_name(tp)); |
1074 | 1095 | ||
1075 | if (!seq_print_ip_sym(s, field->ip, flags | TRACE_ITER_SYM_OFFSET)) | 1096 | if (!seq_print_ip_sym(s, field->ip, flags | TRACE_ITER_SYM_OFFSET)) |
1076 | goto out; | 1097 | goto out; |
@@ -1097,7 +1118,7 @@ print_kretprobe_event(struct trace_iterator *iter, int flags, | |||
1097 | field = (struct kretprobe_trace_entry_head *)iter->ent; | 1118 | field = (struct kretprobe_trace_entry_head *)iter->ent; |
1098 | tp = container_of(event, struct trace_probe, call.event); | 1119 | tp = container_of(event, struct trace_probe, call.event); |
1099 | 1120 | ||
1100 | trace_seq_printf(s, "%s: (", trace_event_name(&tp->call)); | 1121 | trace_seq_printf(s, "%s: (", trace_probe_name(tp)); |
1101 | 1122 | ||
1102 | if (!seq_print_ip_sym(s, field->ret_ip, flags | TRACE_ITER_SYM_OFFSET)) | 1123 | if (!seq_print_ip_sym(s, field->ret_ip, flags | TRACE_ITER_SYM_OFFSET)) |
1103 | goto out; | 1124 | goto out; |
@@ -1149,7 +1170,7 @@ static int kretprobe_event_define_fields(struct trace_event_call *event_call) | |||
1149 | static int | 1170 | static int |
1150 | kprobe_perf_func(struct trace_kprobe *tk, struct pt_regs *regs) | 1171 | kprobe_perf_func(struct trace_kprobe *tk, struct pt_regs *regs) |
1151 | { | 1172 | { |
1152 | struct trace_event_call *call = &tk->tp.call; | 1173 | struct trace_event_call *call = trace_probe_event_call(&tk->tp); |
1153 | struct kprobe_trace_entry_head *entry; | 1174 | struct kprobe_trace_entry_head *entry; |
1154 | struct hlist_head *head; | 1175 | struct hlist_head *head; |
1155 | int size, __size, dsize; | 1176 | int size, __size, dsize; |
@@ -1199,7 +1220,7 @@ static void | |||
1199 | kretprobe_perf_func(struct trace_kprobe *tk, struct kretprobe_instance *ri, | 1220 | kretprobe_perf_func(struct trace_kprobe *tk, struct kretprobe_instance *ri, |
1200 | struct pt_regs *regs) | 1221 | struct pt_regs *regs) |
1201 | { | 1222 | { |
1202 | struct trace_event_call *call = &tk->tp.call; | 1223 | struct trace_event_call *call = trace_probe_event_call(&tk->tp); |
1203 | struct kretprobe_trace_entry_head *entry; | 1224 | struct kretprobe_trace_entry_head *entry; |
1204 | struct hlist_head *head; | 1225 | struct hlist_head *head; |
1205 | int size, __size, dsize; | 1226 | int size, __size, dsize; |
@@ -1299,10 +1320,10 @@ static int kprobe_dispatcher(struct kprobe *kp, struct pt_regs *regs) | |||
1299 | 1320 | ||
1300 | raw_cpu_inc(*tk->nhit); | 1321 | raw_cpu_inc(*tk->nhit); |
1301 | 1322 | ||
1302 | if (tk->tp.flags & TP_FLAG_TRACE) | 1323 | if (trace_probe_test_flag(&tk->tp, TP_FLAG_TRACE)) |
1303 | kprobe_trace_func(tk, regs); | 1324 | kprobe_trace_func(tk, regs); |
1304 | #ifdef CONFIG_PERF_EVENTS | 1325 | #ifdef CONFIG_PERF_EVENTS |
1305 | if (tk->tp.flags & TP_FLAG_PROFILE) | 1326 | if (trace_probe_test_flag(&tk->tp, TP_FLAG_PROFILE)) |
1306 | ret = kprobe_perf_func(tk, regs); | 1327 | ret = kprobe_perf_func(tk, regs); |
1307 | #endif | 1328 | #endif |
1308 | return ret; | 1329 | return ret; |
@@ -1316,10 +1337,10 @@ kretprobe_dispatcher(struct kretprobe_instance *ri, struct pt_regs *regs) | |||
1316 | 1337 | ||
1317 | raw_cpu_inc(*tk->nhit); | 1338 | raw_cpu_inc(*tk->nhit); |
1318 | 1339 | ||
1319 | if (tk->tp.flags & TP_FLAG_TRACE) | 1340 | if (trace_probe_test_flag(&tk->tp, TP_FLAG_TRACE)) |
1320 | kretprobe_trace_func(tk, ri, regs); | 1341 | kretprobe_trace_func(tk, ri, regs); |
1321 | #ifdef CONFIG_PERF_EVENTS | 1342 | #ifdef CONFIG_PERF_EVENTS |
1322 | if (tk->tp.flags & TP_FLAG_PROFILE) | 1343 | if (trace_probe_test_flag(&tk->tp, TP_FLAG_PROFILE)) |
1323 | kretprobe_perf_func(tk, ri, regs); | 1344 | kretprobe_perf_func(tk, ri, regs); |
1324 | #endif | 1345 | #endif |
1325 | return 0; /* We don't tweek kernel, so just return 0 */ | 1346 | return 0; /* We don't tweek kernel, so just return 0 */ |
@@ -1334,10 +1355,10 @@ static struct trace_event_functions kprobe_funcs = { | |||
1334 | .trace = print_kprobe_event | 1355 | .trace = print_kprobe_event |
1335 | }; | 1356 | }; |
1336 | 1357 | ||
1337 | static inline void init_trace_event_call(struct trace_kprobe *tk, | 1358 | static inline void init_trace_event_call(struct trace_kprobe *tk) |
1338 | struct trace_event_call *call) | ||
1339 | { | 1359 | { |
1340 | INIT_LIST_HEAD(&call->class->fields); | 1360 | struct trace_event_call *call = trace_probe_event_call(&tk->tp); |
1361 | |||
1341 | if (trace_kprobe_is_return(tk)) { | 1362 | if (trace_kprobe_is_return(tk)) { |
1342 | call->event.funcs = &kretprobe_funcs; | 1363 | call->event.funcs = &kretprobe_funcs; |
1343 | call->class->define_fields = kretprobe_event_define_fields; | 1364 | call->class->define_fields = kretprobe_event_define_fields; |
@@ -1353,37 +1374,14 @@ static inline void init_trace_event_call(struct trace_kprobe *tk, | |||
1353 | 1374 | ||
1354 | static int register_kprobe_event(struct trace_kprobe *tk) | 1375 | static int register_kprobe_event(struct trace_kprobe *tk) |
1355 | { | 1376 | { |
1356 | struct trace_event_call *call = &tk->tp.call; | 1377 | init_trace_event_call(tk); |
1357 | int ret = 0; | ||
1358 | |||
1359 | init_trace_event_call(tk, call); | ||
1360 | 1378 | ||
1361 | if (traceprobe_set_print_fmt(&tk->tp, trace_kprobe_is_return(tk)) < 0) | 1379 | return trace_probe_register_event_call(&tk->tp); |
1362 | return -ENOMEM; | ||
1363 | ret = register_trace_event(&call->event); | ||
1364 | if (!ret) { | ||
1365 | kfree(call->print_fmt); | ||
1366 | return -ENODEV; | ||
1367 | } | ||
1368 | ret = trace_add_event_call(call); | ||
1369 | if (ret) { | ||
1370 | pr_info("Failed to register kprobe event: %s\n", | ||
1371 | trace_event_name(call)); | ||
1372 | kfree(call->print_fmt); | ||
1373 | unregister_trace_event(&call->event); | ||
1374 | } | ||
1375 | return ret; | ||
1376 | } | 1380 | } |
1377 | 1381 | ||
1378 | static int unregister_kprobe_event(struct trace_kprobe *tk) | 1382 | static int unregister_kprobe_event(struct trace_kprobe *tk) |
1379 | { | 1383 | { |
1380 | int ret; | 1384 | return trace_probe_unregister_event_call(&tk->tp); |
1381 | |||
1382 | /* tp->event is unregistered in trace_remove_event_call() */ | ||
1383 | ret = trace_remove_event_call(&tk->tp.call); | ||
1384 | if (!ret) | ||
1385 | kfree(tk->tp.call.print_fmt); | ||
1386 | return ret; | ||
1387 | } | 1385 | } |
1388 | 1386 | ||
1389 | #ifdef CONFIG_PERF_EVENTS | 1387 | #ifdef CONFIG_PERF_EVENTS |
@@ -1413,7 +1411,7 @@ create_local_trace_kprobe(char *func, void *addr, unsigned long offs, | |||
1413 | return ERR_CAST(tk); | 1411 | return ERR_CAST(tk); |
1414 | } | 1412 | } |
1415 | 1413 | ||
1416 | init_trace_event_call(tk, &tk->tp.call); | 1414 | init_trace_event_call(tk); |
1417 | 1415 | ||
1418 | if (traceprobe_set_print_fmt(&tk->tp, trace_kprobe_is_return(tk)) < 0) { | 1416 | if (traceprobe_set_print_fmt(&tk->tp, trace_kprobe_is_return(tk)) < 0) { |
1419 | ret = -ENOMEM; | 1417 | ret = -ENOMEM; |
@@ -1421,12 +1419,10 @@ create_local_trace_kprobe(char *func, void *addr, unsigned long offs, | |||
1421 | } | 1419 | } |
1422 | 1420 | ||
1423 | ret = __register_trace_kprobe(tk); | 1421 | ret = __register_trace_kprobe(tk); |
1424 | if (ret < 0) { | 1422 | if (ret < 0) |
1425 | kfree(tk->tp.call.print_fmt); | ||
1426 | goto error; | 1423 | goto error; |
1427 | } | ||
1428 | 1424 | ||
1429 | return &tk->tp.call; | 1425 | return trace_probe_event_call(&tk->tp); |
1430 | error: | 1426 | error: |
1431 | free_trace_kprobe(tk); | 1427 | free_trace_kprobe(tk); |
1432 | return ERR_PTR(ret); | 1428 | return ERR_PTR(ret); |
@@ -1445,11 +1441,50 @@ void destroy_local_trace_kprobe(struct trace_event_call *event_call) | |||
1445 | 1441 | ||
1446 | __unregister_trace_kprobe(tk); | 1442 | __unregister_trace_kprobe(tk); |
1447 | 1443 | ||
1448 | kfree(tk->tp.call.print_fmt); | ||
1449 | free_trace_kprobe(tk); | 1444 | free_trace_kprobe(tk); |
1450 | } | 1445 | } |
1451 | #endif /* CONFIG_PERF_EVENTS */ | 1446 | #endif /* CONFIG_PERF_EVENTS */ |
1452 | 1447 | ||
1448 | static __init void enable_boot_kprobe_events(void) | ||
1449 | { | ||
1450 | struct trace_array *tr = top_trace_array(); | ||
1451 | struct trace_event_file *file; | ||
1452 | struct trace_kprobe *tk; | ||
1453 | struct dyn_event *pos; | ||
1454 | |||
1455 | mutex_lock(&event_mutex); | ||
1456 | for_each_trace_kprobe(tk, pos) { | ||
1457 | list_for_each_entry(file, &tr->events, list) | ||
1458 | if (file->event_call == trace_probe_event_call(&tk->tp)) | ||
1459 | trace_event_enable_disable(file, 1, 0); | ||
1460 | } | ||
1461 | mutex_unlock(&event_mutex); | ||
1462 | } | ||
1463 | |||
1464 | static __init void setup_boot_kprobe_events(void) | ||
1465 | { | ||
1466 | char *p, *cmd = kprobe_boot_events_buf; | ||
1467 | int ret; | ||
1468 | |||
1469 | strreplace(kprobe_boot_events_buf, ',', ' '); | ||
1470 | |||
1471 | while (cmd && *cmd != '\0') { | ||
1472 | p = strchr(cmd, ';'); | ||
1473 | if (p) | ||
1474 | *p++ = '\0'; | ||
1475 | |||
1476 | ret = trace_run_command(cmd, create_or_delete_trace_kprobe); | ||
1477 | if (ret) | ||
1478 | pr_warn("Failed to add event(%d): %s\n", ret, cmd); | ||
1479 | else | ||
1480 | kprobe_boot_events_enabled = true; | ||
1481 | |||
1482 | cmd = p; | ||
1483 | } | ||
1484 | |||
1485 | enable_boot_kprobe_events(); | ||
1486 | } | ||
1487 | |||
1453 | /* Make a tracefs interface for controlling probe points */ | 1488 | /* Make a tracefs interface for controlling probe points */ |
1454 | static __init int init_kprobe_trace(void) | 1489 | static __init int init_kprobe_trace(void) |
1455 | { | 1490 | { |
@@ -1481,6 +1516,9 @@ static __init int init_kprobe_trace(void) | |||
1481 | 1516 | ||
1482 | if (!entry) | 1517 | if (!entry) |
1483 | pr_warn("Could not create tracefs 'kprobe_profile' entry\n"); | 1518 | pr_warn("Could not create tracefs 'kprobe_profile' entry\n"); |
1519 | |||
1520 | setup_boot_kprobe_events(); | ||
1521 | |||
1484 | return 0; | 1522 | return 0; |
1485 | } | 1523 | } |
1486 | fs_initcall(init_kprobe_trace); | 1524 | fs_initcall(init_kprobe_trace); |
@@ -1493,7 +1531,7 @@ find_trace_probe_file(struct trace_kprobe *tk, struct trace_array *tr) | |||
1493 | struct trace_event_file *file; | 1531 | struct trace_event_file *file; |
1494 | 1532 | ||
1495 | list_for_each_entry(file, &tr->events, list) | 1533 | list_for_each_entry(file, &tr->events, list) |
1496 | if (file->event_call == &tk->tp.call) | 1534 | if (file->event_call == trace_probe_event_call(&tk->tp)) |
1497 | return file; | 1535 | return file; |
1498 | 1536 | ||
1499 | return NULL; | 1537 | return NULL; |
@@ -1513,6 +1551,11 @@ static __init int kprobe_trace_self_tests_init(void) | |||
1513 | if (tracing_is_disabled()) | 1551 | if (tracing_is_disabled()) |
1514 | return -ENODEV; | 1552 | return -ENODEV; |
1515 | 1553 | ||
1554 | if (kprobe_boot_events_enabled) { | ||
1555 | pr_info("Skipping kprobe tests due to kprobe_event on cmdline\n"); | ||
1556 | return 0; | ||
1557 | } | ||
1558 | |||
1516 | target = kprobe_trace_selftest_target; | 1559 | target = kprobe_trace_selftest_target; |
1517 | 1560 | ||
1518 | pr_info("Testing kprobe tracing: "); | 1561 | pr_info("Testing kprobe tracing: "); |
diff --git a/kernel/trace/trace_probe.c b/kernel/trace/trace_probe.c index a347faced959..dbef0d135075 100644 --- a/kernel/trace/trace_probe.c +++ b/kernel/trace/trace_probe.c | |||
@@ -78,6 +78,8 @@ static const struct fetch_type probe_fetch_types[] = { | |||
78 | /* Special types */ | 78 | /* Special types */ |
79 | __ASSIGN_FETCH_TYPE("string", string, string, sizeof(u32), 1, | 79 | __ASSIGN_FETCH_TYPE("string", string, string, sizeof(u32), 1, |
80 | "__data_loc char[]"), | 80 | "__data_loc char[]"), |
81 | __ASSIGN_FETCH_TYPE("ustring", string, string, sizeof(u32), 1, | ||
82 | "__data_loc char[]"), | ||
81 | /* Basic types */ | 83 | /* Basic types */ |
82 | ASSIGN_FETCH_TYPE(u8, u8, 0), | 84 | ASSIGN_FETCH_TYPE(u8, u8, 0), |
83 | ASSIGN_FETCH_TYPE(u16, u16, 0), | 85 | ASSIGN_FETCH_TYPE(u16, u16, 0), |
@@ -322,6 +324,7 @@ parse_probe_arg(char *arg, const struct fetch_type *type, | |||
322 | { | 324 | { |
323 | struct fetch_insn *code = *pcode; | 325 | struct fetch_insn *code = *pcode; |
324 | unsigned long param; | 326 | unsigned long param; |
327 | int deref = FETCH_OP_DEREF; | ||
325 | long offset = 0; | 328 | long offset = 0; |
326 | char *tmp; | 329 | char *tmp; |
327 | int ret = 0; | 330 | int ret = 0; |
@@ -394,9 +397,14 @@ parse_probe_arg(char *arg, const struct fetch_type *type, | |||
394 | break; | 397 | break; |
395 | 398 | ||
396 | case '+': /* deref memory */ | 399 | case '+': /* deref memory */ |
397 | arg++; /* Skip '+', because kstrtol() rejects it. */ | ||
398 | /* fall through */ | ||
399 | case '-': | 400 | case '-': |
401 | if (arg[1] == 'u') { | ||
402 | deref = FETCH_OP_UDEREF; | ||
403 | arg[1] = arg[0]; | ||
404 | arg++; | ||
405 | } | ||
406 | if (arg[0] == '+') | ||
407 | arg++; /* Skip '+', because kstrtol() rejects it. */ | ||
400 | tmp = strchr(arg, '('); | 408 | tmp = strchr(arg, '('); |
401 | if (!tmp) { | 409 | if (!tmp) { |
402 | trace_probe_log_err(offs, DEREF_NEED_BRACE); | 410 | trace_probe_log_err(offs, DEREF_NEED_BRACE); |
@@ -432,7 +440,7 @@ parse_probe_arg(char *arg, const struct fetch_type *type, | |||
432 | } | 440 | } |
433 | *pcode = code; | 441 | *pcode = code; |
434 | 442 | ||
435 | code->op = FETCH_OP_DEREF; | 443 | code->op = deref; |
436 | code->offset = offset; | 444 | code->offset = offset; |
437 | } | 445 | } |
438 | break; | 446 | break; |
@@ -569,15 +577,17 @@ static int traceprobe_parse_probe_arg_body(char *arg, ssize_t *size, | |||
569 | goto fail; | 577 | goto fail; |
570 | 578 | ||
571 | /* Store operation */ | 579 | /* Store operation */ |
572 | if (!strcmp(parg->type->name, "string")) { | 580 | if (!strcmp(parg->type->name, "string") || |
573 | if (code->op != FETCH_OP_DEREF && code->op != FETCH_OP_IMM && | 581 | !strcmp(parg->type->name, "ustring")) { |
574 | code->op != FETCH_OP_COMM) { | 582 | if (code->op != FETCH_OP_DEREF && code->op != FETCH_OP_UDEREF && |
583 | code->op != FETCH_OP_IMM && code->op != FETCH_OP_COMM) { | ||
575 | trace_probe_log_err(offset + (t ? (t - arg) : 0), | 584 | trace_probe_log_err(offset + (t ? (t - arg) : 0), |
576 | BAD_STRING); | 585 | BAD_STRING); |
577 | ret = -EINVAL; | 586 | ret = -EINVAL; |
578 | goto fail; | 587 | goto fail; |
579 | } | 588 | } |
580 | if (code->op != FETCH_OP_DEREF || parg->count) { | 589 | if ((code->op == FETCH_OP_IMM || code->op == FETCH_OP_COMM) || |
590 | parg->count) { | ||
581 | /* | 591 | /* |
582 | * IMM and COMM is pointing actual address, those must | 592 | * IMM and COMM is pointing actual address, those must |
583 | * be kept, and if parg->count != 0, this is an array | 593 | * be kept, and if parg->count != 0, this is an array |
@@ -590,12 +600,20 @@ static int traceprobe_parse_probe_arg_body(char *arg, ssize_t *size, | |||
590 | goto fail; | 600 | goto fail; |
591 | } | 601 | } |
592 | } | 602 | } |
593 | code->op = FETCH_OP_ST_STRING; /* In DEREF case, replace it */ | 603 | /* If op == DEREF, replace it with STRING */ |
604 | if (!strcmp(parg->type->name, "ustring") || | ||
605 | code->op == FETCH_OP_UDEREF) | ||
606 | code->op = FETCH_OP_ST_USTRING; | ||
607 | else | ||
608 | code->op = FETCH_OP_ST_STRING; | ||
594 | code->size = parg->type->size; | 609 | code->size = parg->type->size; |
595 | parg->dynamic = true; | 610 | parg->dynamic = true; |
596 | } else if (code->op == FETCH_OP_DEREF) { | 611 | } else if (code->op == FETCH_OP_DEREF) { |
597 | code->op = FETCH_OP_ST_MEM; | 612 | code->op = FETCH_OP_ST_MEM; |
598 | code->size = parg->type->size; | 613 | code->size = parg->type->size; |
614 | } else if (code->op == FETCH_OP_UDEREF) { | ||
615 | code->op = FETCH_OP_ST_UMEM; | ||
616 | code->size = parg->type->size; | ||
599 | } else { | 617 | } else { |
600 | code++; | 618 | code++; |
601 | if (code->op != FETCH_OP_NOP) { | 619 | if (code->op != FETCH_OP_NOP) { |
@@ -618,7 +636,8 @@ static int traceprobe_parse_probe_arg_body(char *arg, ssize_t *size, | |||
618 | /* Loop(Array) operation */ | 636 | /* Loop(Array) operation */ |
619 | if (parg->count) { | 637 | if (parg->count) { |
620 | if (scode->op != FETCH_OP_ST_MEM && | 638 | if (scode->op != FETCH_OP_ST_MEM && |
621 | scode->op != FETCH_OP_ST_STRING) { | 639 | scode->op != FETCH_OP_ST_STRING && |
640 | scode->op != FETCH_OP_ST_USTRING) { | ||
622 | trace_probe_log_err(offset + (t ? (t - arg) : 0), | 641 | trace_probe_log_err(offset + (t ? (t - arg) : 0), |
623 | BAD_STRING); | 642 | BAD_STRING); |
624 | ret = -EINVAL; | 643 | ret = -EINVAL; |
@@ -825,6 +844,7 @@ static int __set_print_fmt(struct trace_probe *tp, char *buf, int len, | |||
825 | 844 | ||
826 | int traceprobe_set_print_fmt(struct trace_probe *tp, bool is_return) | 845 | int traceprobe_set_print_fmt(struct trace_probe *tp, bool is_return) |
827 | { | 846 | { |
847 | struct trace_event_call *call = trace_probe_event_call(tp); | ||
828 | int len; | 848 | int len; |
829 | char *print_fmt; | 849 | char *print_fmt; |
830 | 850 | ||
@@ -836,7 +856,7 @@ int traceprobe_set_print_fmt(struct trace_probe *tp, bool is_return) | |||
836 | 856 | ||
837 | /* Second: actually write the @print_fmt */ | 857 | /* Second: actually write the @print_fmt */ |
838 | __set_print_fmt(tp, print_fmt, len + 1, is_return); | 858 | __set_print_fmt(tp, print_fmt, len + 1, is_return); |
839 | tp->call.print_fmt = print_fmt; | 859 | call->print_fmt = print_fmt; |
840 | 860 | ||
841 | return 0; | 861 | return 0; |
842 | } | 862 | } |
@@ -865,3 +885,105 @@ int traceprobe_define_arg_fields(struct trace_event_call *event_call, | |||
865 | } | 885 | } |
866 | return 0; | 886 | return 0; |
867 | } | 887 | } |
888 | |||
889 | |||
890 | void trace_probe_cleanup(struct trace_probe *tp) | ||
891 | { | ||
892 | struct trace_event_call *call = trace_probe_event_call(tp); | ||
893 | int i; | ||
894 | |||
895 | for (i = 0; i < tp->nr_args; i++) | ||
896 | traceprobe_free_probe_arg(&tp->args[i]); | ||
897 | |||
898 | kfree(call->class->system); | ||
899 | kfree(call->name); | ||
900 | kfree(call->print_fmt); | ||
901 | } | ||
902 | |||
903 | int trace_probe_init(struct trace_probe *tp, const char *event, | ||
904 | const char *group) | ||
905 | { | ||
906 | struct trace_event_call *call = trace_probe_event_call(tp); | ||
907 | |||
908 | if (!event || !group) | ||
909 | return -EINVAL; | ||
910 | |||
911 | call->class = &tp->class; | ||
912 | call->name = kstrdup(event, GFP_KERNEL); | ||
913 | if (!call->name) | ||
914 | return -ENOMEM; | ||
915 | |||
916 | tp->class.system = kstrdup(group, GFP_KERNEL); | ||
917 | if (!tp->class.system) { | ||
918 | kfree(call->name); | ||
919 | call->name = NULL; | ||
920 | return -ENOMEM; | ||
921 | } | ||
922 | INIT_LIST_HEAD(&tp->files); | ||
923 | INIT_LIST_HEAD(&tp->class.fields); | ||
924 | |||
925 | return 0; | ||
926 | } | ||
927 | |||
928 | int trace_probe_register_event_call(struct trace_probe *tp) | ||
929 | { | ||
930 | struct trace_event_call *call = trace_probe_event_call(tp); | ||
931 | int ret; | ||
932 | |||
933 | ret = register_trace_event(&call->event); | ||
934 | if (!ret) | ||
935 | return -ENODEV; | ||
936 | |||
937 | ret = trace_add_event_call(call); | ||
938 | if (ret) | ||
939 | unregister_trace_event(&call->event); | ||
940 | |||
941 | return ret; | ||
942 | } | ||
943 | |||
944 | int trace_probe_add_file(struct trace_probe *tp, struct trace_event_file *file) | ||
945 | { | ||
946 | struct event_file_link *link; | ||
947 | |||
948 | link = kmalloc(sizeof(*link), GFP_KERNEL); | ||
949 | if (!link) | ||
950 | return -ENOMEM; | ||
951 | |||
952 | link->file = file; | ||
953 | INIT_LIST_HEAD(&link->list); | ||
954 | list_add_tail_rcu(&link->list, &tp->files); | ||
955 | trace_probe_set_flag(tp, TP_FLAG_TRACE); | ||
956 | return 0; | ||
957 | } | ||
958 | |||
959 | struct event_file_link *trace_probe_get_file_link(struct trace_probe *tp, | ||
960 | struct trace_event_file *file) | ||
961 | { | ||
962 | struct event_file_link *link; | ||
963 | |||
964 | trace_probe_for_each_link(link, tp) { | ||
965 | if (link->file == file) | ||
966 | return link; | ||
967 | } | ||
968 | |||
969 | return NULL; | ||
970 | } | ||
971 | |||
972 | int trace_probe_remove_file(struct trace_probe *tp, | ||
973 | struct trace_event_file *file) | ||
974 | { | ||
975 | struct event_file_link *link; | ||
976 | |||
977 | link = trace_probe_get_file_link(tp, file); | ||
978 | if (!link) | ||
979 | return -ENOENT; | ||
980 | |||
981 | list_del_rcu(&link->list); | ||
982 | synchronize_rcu(); | ||
983 | kfree(link); | ||
984 | |||
985 | if (list_empty(&tp->files)) | ||
986 | trace_probe_clear_flag(tp, TP_FLAG_TRACE); | ||
987 | |||
988 | return 0; | ||
989 | } | ||
diff --git a/kernel/trace/trace_probe.h b/kernel/trace/trace_probe.h index f9a8c632188b..d1714820efe1 100644 --- a/kernel/trace/trace_probe.h +++ b/kernel/trace/trace_probe.h | |||
@@ -55,7 +55,6 @@ | |||
55 | /* Flags for trace_probe */ | 55 | /* Flags for trace_probe */ |
56 | #define TP_FLAG_TRACE 1 | 56 | #define TP_FLAG_TRACE 1 |
57 | #define TP_FLAG_PROFILE 2 | 57 | #define TP_FLAG_PROFILE 2 |
58 | #define TP_FLAG_REGISTERED 4 | ||
59 | 58 | ||
60 | /* data_loc: data location, compatible with u32 */ | 59 | /* data_loc: data location, compatible with u32 */ |
61 | #define make_data_loc(len, offs) \ | 60 | #define make_data_loc(len, offs) \ |
@@ -92,10 +91,13 @@ enum fetch_op { | |||
92 | FETCH_OP_FOFFS, /* File offset: .immediate */ | 91 | FETCH_OP_FOFFS, /* File offset: .immediate */ |
93 | // Stage 2 (dereference) op | 92 | // Stage 2 (dereference) op |
94 | FETCH_OP_DEREF, /* Dereference: .offset */ | 93 | FETCH_OP_DEREF, /* Dereference: .offset */ |
94 | FETCH_OP_UDEREF, /* User-space Dereference: .offset */ | ||
95 | // Stage 3 (store) ops | 95 | // Stage 3 (store) ops |
96 | FETCH_OP_ST_RAW, /* Raw: .size */ | 96 | FETCH_OP_ST_RAW, /* Raw: .size */ |
97 | FETCH_OP_ST_MEM, /* Mem: .offset, .size */ | 97 | FETCH_OP_ST_MEM, /* Mem: .offset, .size */ |
98 | FETCH_OP_ST_UMEM, /* Mem: .offset, .size */ | ||
98 | FETCH_OP_ST_STRING, /* String: .offset, .size */ | 99 | FETCH_OP_ST_STRING, /* String: .offset, .size */ |
100 | FETCH_OP_ST_USTRING, /* User String: .offset, .size */ | ||
99 | // Stage 4 (modify) op | 101 | // Stage 4 (modify) op |
100 | FETCH_OP_MOD_BF, /* Bitfield: .basesize, .lshift, .rshift */ | 102 | FETCH_OP_MOD_BF, /* Bitfield: .basesize, .lshift, .rshift */ |
101 | // Stage 5 (loop) op | 103 | // Stage 5 (loop) op |
@@ -235,16 +237,71 @@ struct event_file_link { | |||
235 | struct list_head list; | 237 | struct list_head list; |
236 | }; | 238 | }; |
237 | 239 | ||
240 | static inline bool trace_probe_test_flag(struct trace_probe *tp, | ||
241 | unsigned int flag) | ||
242 | { | ||
243 | return !!(tp->flags & flag); | ||
244 | } | ||
245 | |||
246 | static inline void trace_probe_set_flag(struct trace_probe *tp, | ||
247 | unsigned int flag) | ||
248 | { | ||
249 | tp->flags |= flag; | ||
250 | } | ||
251 | |||
252 | static inline void trace_probe_clear_flag(struct trace_probe *tp, | ||
253 | unsigned int flag) | ||
254 | { | ||
255 | tp->flags &= ~flag; | ||
256 | } | ||
257 | |||
238 | static inline bool trace_probe_is_enabled(struct trace_probe *tp) | 258 | static inline bool trace_probe_is_enabled(struct trace_probe *tp) |
239 | { | 259 | { |
240 | return !!(tp->flags & (TP_FLAG_TRACE | TP_FLAG_PROFILE)); | 260 | return trace_probe_test_flag(tp, TP_FLAG_TRACE | TP_FLAG_PROFILE); |
241 | } | 261 | } |
242 | 262 | ||
243 | static inline bool trace_probe_is_registered(struct trace_probe *tp) | 263 | static inline const char *trace_probe_name(struct trace_probe *tp) |
244 | { | 264 | { |
245 | return !!(tp->flags & TP_FLAG_REGISTERED); | 265 | return trace_event_name(&tp->call); |
246 | } | 266 | } |
247 | 267 | ||
268 | static inline const char *trace_probe_group_name(struct trace_probe *tp) | ||
269 | { | ||
270 | return tp->call.class->system; | ||
271 | } | ||
272 | |||
273 | static inline struct trace_event_call * | ||
274 | trace_probe_event_call(struct trace_probe *tp) | ||
275 | { | ||
276 | return &tp->call; | ||
277 | } | ||
278 | |||
279 | static inline int trace_probe_unregister_event_call(struct trace_probe *tp) | ||
280 | { | ||
281 | /* tp->event is unregistered in trace_remove_event_call() */ | ||
282 | return trace_remove_event_call(&tp->call); | ||
283 | } | ||
284 | |||
285 | static inline bool trace_probe_has_single_file(struct trace_probe *tp) | ||
286 | { | ||
287 | return !!list_is_singular(&tp->files); | ||
288 | } | ||
289 | |||
290 | int trace_probe_init(struct trace_probe *tp, const char *event, | ||
291 | const char *group); | ||
292 | void trace_probe_cleanup(struct trace_probe *tp); | ||
293 | int trace_probe_register_event_call(struct trace_probe *tp); | ||
294 | int trace_probe_add_file(struct trace_probe *tp, struct trace_event_file *file); | ||
295 | int trace_probe_remove_file(struct trace_probe *tp, | ||
296 | struct trace_event_file *file); | ||
297 | struct event_file_link *trace_probe_get_file_link(struct trace_probe *tp, | ||
298 | struct trace_event_file *file); | ||
299 | |||
300 | #define trace_probe_for_each_link(pos, tp) \ | ||
301 | list_for_each_entry(pos, &(tp)->files, list) | ||
302 | #define trace_probe_for_each_link_rcu(pos, tp) \ | ||
303 | list_for_each_entry_rcu(pos, &(tp)->files, list) | ||
304 | |||
248 | /* Check the name is good for event/group/fields */ | 305 | /* Check the name is good for event/group/fields */ |
249 | static inline bool is_good_name(const char *name) | 306 | static inline bool is_good_name(const char *name) |
250 | { | 307 | { |
@@ -257,18 +314,6 @@ static inline bool is_good_name(const char *name) | |||
257 | return true; | 314 | return true; |
258 | } | 315 | } |
259 | 316 | ||
260 | static inline struct event_file_link * | ||
261 | find_event_file_link(struct trace_probe *tp, struct trace_event_file *file) | ||
262 | { | ||
263 | struct event_file_link *link; | ||
264 | |||
265 | list_for_each_entry(link, &tp->files, list) | ||
266 | if (link->file == file) | ||
267 | return link; | ||
268 | |||
269 | return NULL; | ||
270 | } | ||
271 | |||
272 | #define TPARG_FL_RETURN BIT(0) | 317 | #define TPARG_FL_RETURN BIT(0) |
273 | #define TPARG_FL_KERNEL BIT(1) | 318 | #define TPARG_FL_KERNEL BIT(1) |
274 | #define TPARG_FL_FENTRY BIT(2) | 319 | #define TPARG_FL_FENTRY BIT(2) |
diff --git a/kernel/trace/trace_probe_tmpl.h b/kernel/trace/trace_probe_tmpl.h index c30c61f12ddd..e5282828f4a6 100644 --- a/kernel/trace/trace_probe_tmpl.h +++ b/kernel/trace/trace_probe_tmpl.h | |||
@@ -59,8 +59,13 @@ process_fetch_insn(struct fetch_insn *code, struct pt_regs *regs, | |||
59 | static nokprobe_inline int fetch_store_strlen(unsigned long addr); | 59 | static nokprobe_inline int fetch_store_strlen(unsigned long addr); |
60 | static nokprobe_inline int | 60 | static nokprobe_inline int |
61 | fetch_store_string(unsigned long addr, void *dest, void *base); | 61 | fetch_store_string(unsigned long addr, void *dest, void *base); |
62 | static nokprobe_inline int fetch_store_strlen_user(unsigned long addr); | ||
63 | static nokprobe_inline int | ||
64 | fetch_store_string_user(unsigned long addr, void *dest, void *base); | ||
62 | static nokprobe_inline int | 65 | static nokprobe_inline int |
63 | probe_mem_read(void *dest, void *src, size_t size); | 66 | probe_mem_read(void *dest, void *src, size_t size); |
67 | static nokprobe_inline int | ||
68 | probe_mem_read_user(void *dest, void *src, size_t size); | ||
64 | 69 | ||
65 | /* From the 2nd stage, routine is same */ | 70 | /* From the 2nd stage, routine is same */ |
66 | static nokprobe_inline int | 71 | static nokprobe_inline int |
@@ -74,14 +79,21 @@ process_fetch_insn_bottom(struct fetch_insn *code, unsigned long val, | |||
74 | 79 | ||
75 | stage2: | 80 | stage2: |
76 | /* 2nd stage: dereference memory if needed */ | 81 | /* 2nd stage: dereference memory if needed */ |
77 | while (code->op == FETCH_OP_DEREF) { | 82 | do { |
78 | lval = val; | 83 | if (code->op == FETCH_OP_DEREF) { |
79 | ret = probe_mem_read(&val, (void *)val + code->offset, | 84 | lval = val; |
80 | sizeof(val)); | 85 | ret = probe_mem_read(&val, (void *)val + code->offset, |
86 | sizeof(val)); | ||
87 | } else if (code->op == FETCH_OP_UDEREF) { | ||
88 | lval = val; | ||
89 | ret = probe_mem_read_user(&val, | ||
90 | (void *)val + code->offset, sizeof(val)); | ||
91 | } else | ||
92 | break; | ||
81 | if (ret) | 93 | if (ret) |
82 | return ret; | 94 | return ret; |
83 | code++; | 95 | code++; |
84 | } | 96 | } while (1); |
85 | 97 | ||
86 | s3 = code; | 98 | s3 = code; |
87 | stage3: | 99 | stage3: |
@@ -91,6 +103,10 @@ stage3: | |||
91 | ret = fetch_store_strlen(val + code->offset); | 103 | ret = fetch_store_strlen(val + code->offset); |
92 | code++; | 104 | code++; |
93 | goto array; | 105 | goto array; |
106 | } else if (code->op == FETCH_OP_ST_USTRING) { | ||
107 | ret += fetch_store_strlen_user(val + code->offset); | ||
108 | code++; | ||
109 | goto array; | ||
94 | } else | 110 | } else |
95 | return -EILSEQ; | 111 | return -EILSEQ; |
96 | } | 112 | } |
@@ -102,10 +118,17 @@ stage3: | |||
102 | case FETCH_OP_ST_MEM: | 118 | case FETCH_OP_ST_MEM: |
103 | probe_mem_read(dest, (void *)val + code->offset, code->size); | 119 | probe_mem_read(dest, (void *)val + code->offset, code->size); |
104 | break; | 120 | break; |
121 | case FETCH_OP_ST_UMEM: | ||
122 | probe_mem_read_user(dest, (void *)val + code->offset, code->size); | ||
123 | break; | ||
105 | case FETCH_OP_ST_STRING: | 124 | case FETCH_OP_ST_STRING: |
106 | loc = *(u32 *)dest; | 125 | loc = *(u32 *)dest; |
107 | ret = fetch_store_string(val + code->offset, dest, base); | 126 | ret = fetch_store_string(val + code->offset, dest, base); |
108 | break; | 127 | break; |
128 | case FETCH_OP_ST_USTRING: | ||
129 | loc = *(u32 *)dest; | ||
130 | ret = fetch_store_string_user(val + code->offset, dest, base); | ||
131 | break; | ||
109 | default: | 132 | default: |
110 | return -EILSEQ; | 133 | return -EILSEQ; |
111 | } | 134 | } |
@@ -123,7 +146,8 @@ array: | |||
123 | total += ret; | 146 | total += ret; |
124 | if (++i < code->param) { | 147 | if (++i < code->param) { |
125 | code = s3; | 148 | code = s3; |
126 | if (s3->op != FETCH_OP_ST_STRING) { | 149 | if (s3->op != FETCH_OP_ST_STRING && |
150 | s3->op != FETCH_OP_ST_USTRING) { | ||
127 | dest += s3->size; | 151 | dest += s3->size; |
128 | val += s3->size; | 152 | val += s3->size; |
129 | goto stage3; | 153 | goto stage3; |
diff --git a/kernel/trace/trace_uprobe.c b/kernel/trace/trace_uprobe.c index 7860e3f59fad..1ceedb9146b1 100644 --- a/kernel/trace/trace_uprobe.c +++ b/kernel/trace/trace_uprobe.c | |||
@@ -140,6 +140,13 @@ probe_mem_read(void *dest, void *src, size_t size) | |||
140 | 140 | ||
141 | return copy_from_user(dest, vaddr, size) ? -EFAULT : 0; | 141 | return copy_from_user(dest, vaddr, size) ? -EFAULT : 0; |
142 | } | 142 | } |
143 | |||
144 | static nokprobe_inline int | ||
145 | probe_mem_read_user(void *dest, void *src, size_t size) | ||
146 | { | ||
147 | return probe_mem_read(dest, src, size); | ||
148 | } | ||
149 | |||
143 | /* | 150 | /* |
144 | * Fetch a null-terminated string. Caller MUST set *(u32 *)dest with max | 151 | * Fetch a null-terminated string. Caller MUST set *(u32 *)dest with max |
145 | * length and relative data location. | 152 | * length and relative data location. |
@@ -176,6 +183,12 @@ fetch_store_string(unsigned long addr, void *dest, void *base) | |||
176 | return ret; | 183 | return ret; |
177 | } | 184 | } |
178 | 185 | ||
186 | static nokprobe_inline int | ||
187 | fetch_store_string_user(unsigned long addr, void *dest, void *base) | ||
188 | { | ||
189 | return fetch_store_string(addr, dest, base); | ||
190 | } | ||
191 | |||
179 | /* Return the length of string -- including null terminal byte */ | 192 | /* Return the length of string -- including null terminal byte */ |
180 | static nokprobe_inline int | 193 | static nokprobe_inline int |
181 | fetch_store_strlen(unsigned long addr) | 194 | fetch_store_strlen(unsigned long addr) |
@@ -191,6 +204,12 @@ fetch_store_strlen(unsigned long addr) | |||
191 | return (len > MAX_STRING_SIZE) ? 0 : len; | 204 | return (len > MAX_STRING_SIZE) ? 0 : len; |
192 | } | 205 | } |
193 | 206 | ||
207 | static nokprobe_inline int | ||
208 | fetch_store_strlen_user(unsigned long addr) | ||
209 | { | ||
210 | return fetch_store_strlen(addr); | ||
211 | } | ||
212 | |||
194 | static unsigned long translate_user_vaddr(unsigned long file_offset) | 213 | static unsigned long translate_user_vaddr(unsigned long file_offset) |
195 | { | 214 | { |
196 | unsigned long base_addr; | 215 | unsigned long base_addr; |
@@ -270,8 +289,8 @@ static bool trace_uprobe_match(const char *system, const char *event, | |||
270 | { | 289 | { |
271 | struct trace_uprobe *tu = to_trace_uprobe(ev); | 290 | struct trace_uprobe *tu = to_trace_uprobe(ev); |
272 | 291 | ||
273 | return strcmp(trace_event_name(&tu->tp.call), event) == 0 && | 292 | return strcmp(trace_probe_name(&tu->tp), event) == 0 && |
274 | (!system || strcmp(tu->tp.call.class->system, system) == 0); | 293 | (!system || strcmp(trace_probe_group_name(&tu->tp), system) == 0); |
275 | } | 294 | } |
276 | 295 | ||
277 | /* | 296 | /* |
@@ -281,25 +300,17 @@ static struct trace_uprobe * | |||
281 | alloc_trace_uprobe(const char *group, const char *event, int nargs, bool is_ret) | 300 | alloc_trace_uprobe(const char *group, const char *event, int nargs, bool is_ret) |
282 | { | 301 | { |
283 | struct trace_uprobe *tu; | 302 | struct trace_uprobe *tu; |
284 | 303 | int ret; | |
285 | if (!event || !group) | ||
286 | return ERR_PTR(-EINVAL); | ||
287 | 304 | ||
288 | tu = kzalloc(SIZEOF_TRACE_UPROBE(nargs), GFP_KERNEL); | 305 | tu = kzalloc(SIZEOF_TRACE_UPROBE(nargs), GFP_KERNEL); |
289 | if (!tu) | 306 | if (!tu) |
290 | return ERR_PTR(-ENOMEM); | 307 | return ERR_PTR(-ENOMEM); |
291 | 308 | ||
292 | tu->tp.call.class = &tu->tp.class; | 309 | ret = trace_probe_init(&tu->tp, event, group); |
293 | tu->tp.call.name = kstrdup(event, GFP_KERNEL); | 310 | if (ret < 0) |
294 | if (!tu->tp.call.name) | ||
295 | goto error; | ||
296 | |||
297 | tu->tp.class.system = kstrdup(group, GFP_KERNEL); | ||
298 | if (!tu->tp.class.system) | ||
299 | goto error; | 311 | goto error; |
300 | 312 | ||
301 | dyn_event_init(&tu->devent, &trace_uprobe_ops); | 313 | dyn_event_init(&tu->devent, &trace_uprobe_ops); |
302 | INIT_LIST_HEAD(&tu->tp.files); | ||
303 | tu->consumer.handler = uprobe_dispatcher; | 314 | tu->consumer.handler = uprobe_dispatcher; |
304 | if (is_ret) | 315 | if (is_ret) |
305 | tu->consumer.ret_handler = uretprobe_dispatcher; | 316 | tu->consumer.ret_handler = uretprobe_dispatcher; |
@@ -307,25 +318,18 @@ alloc_trace_uprobe(const char *group, const char *event, int nargs, bool is_ret) | |||
307 | return tu; | 318 | return tu; |
308 | 319 | ||
309 | error: | 320 | error: |
310 | kfree(tu->tp.call.name); | ||
311 | kfree(tu); | 321 | kfree(tu); |
312 | 322 | ||
313 | return ERR_PTR(-ENOMEM); | 323 | return ERR_PTR(ret); |
314 | } | 324 | } |
315 | 325 | ||
316 | static void free_trace_uprobe(struct trace_uprobe *tu) | 326 | static void free_trace_uprobe(struct trace_uprobe *tu) |
317 | { | 327 | { |
318 | int i; | ||
319 | |||
320 | if (!tu) | 328 | if (!tu) |
321 | return; | 329 | return; |
322 | 330 | ||
323 | for (i = 0; i < tu->tp.nr_args; i++) | ||
324 | traceprobe_free_probe_arg(&tu->tp.args[i]); | ||
325 | |||
326 | path_put(&tu->path); | 331 | path_put(&tu->path); |
327 | kfree(tu->tp.call.class->system); | 332 | trace_probe_cleanup(&tu->tp); |
328 | kfree(tu->tp.call.name); | ||
329 | kfree(tu->filename); | 333 | kfree(tu->filename); |
330 | kfree(tu); | 334 | kfree(tu); |
331 | } | 335 | } |
@@ -336,8 +340,8 @@ static struct trace_uprobe *find_probe_event(const char *event, const char *grou | |||
336 | struct trace_uprobe *tu; | 340 | struct trace_uprobe *tu; |
337 | 341 | ||
338 | for_each_trace_uprobe(tu, pos) | 342 | for_each_trace_uprobe(tu, pos) |
339 | if (strcmp(trace_event_name(&tu->tp.call), event) == 0 && | 343 | if (strcmp(trace_probe_name(&tu->tp), event) == 0 && |
340 | strcmp(tu->tp.call.class->system, group) == 0) | 344 | strcmp(trace_probe_group_name(&tu->tp), group) == 0) |
341 | return tu; | 345 | return tu; |
342 | 346 | ||
343 | return NULL; | 347 | return NULL; |
@@ -372,8 +376,8 @@ static struct trace_uprobe *find_old_trace_uprobe(struct trace_uprobe *new) | |||
372 | struct trace_uprobe *tmp, *old = NULL; | 376 | struct trace_uprobe *tmp, *old = NULL; |
373 | struct inode *new_inode = d_real_inode(new->path.dentry); | 377 | struct inode *new_inode = d_real_inode(new->path.dentry); |
374 | 378 | ||
375 | old = find_probe_event(trace_event_name(&new->tp.call), | 379 | old = find_probe_event(trace_probe_name(&new->tp), |
376 | new->tp.call.class->system); | 380 | trace_probe_group_name(&new->tp)); |
377 | 381 | ||
378 | for_each_trace_uprobe(tmp, pos) { | 382 | for_each_trace_uprobe(tmp, pos) { |
379 | if ((old ? old != tmp : true) && | 383 | if ((old ? old != tmp : true) && |
@@ -578,6 +582,10 @@ static int trace_uprobe_create(int argc, const char **argv) | |||
578 | goto error; | 582 | goto error; |
579 | } | 583 | } |
580 | 584 | ||
585 | ret = traceprobe_set_print_fmt(&tu->tp, is_ret_probe(tu)); | ||
586 | if (ret < 0) | ||
587 | goto error; | ||
588 | |||
581 | ret = register_trace_uprobe(tu); | 589 | ret = register_trace_uprobe(tu); |
582 | if (!ret) | 590 | if (!ret) |
583 | goto out; | 591 | goto out; |
@@ -621,8 +629,8 @@ static int trace_uprobe_show(struct seq_file *m, struct dyn_event *ev) | |||
621 | char c = is_ret_probe(tu) ? 'r' : 'p'; | 629 | char c = is_ret_probe(tu) ? 'r' : 'p'; |
622 | int i; | 630 | int i; |
623 | 631 | ||
624 | seq_printf(m, "%c:%s/%s %s:0x%0*lx", c, tu->tp.call.class->system, | 632 | seq_printf(m, "%c:%s/%s %s:0x%0*lx", c, trace_probe_group_name(&tu->tp), |
625 | trace_event_name(&tu->tp.call), tu->filename, | 633 | trace_probe_name(&tu->tp), tu->filename, |
626 | (int)(sizeof(void *) * 2), tu->offset); | 634 | (int)(sizeof(void *) * 2), tu->offset); |
627 | 635 | ||
628 | if (tu->ref_ctr_offset) | 636 | if (tu->ref_ctr_offset) |
@@ -692,7 +700,7 @@ static int probes_profile_seq_show(struct seq_file *m, void *v) | |||
692 | 700 | ||
693 | tu = to_trace_uprobe(ev); | 701 | tu = to_trace_uprobe(ev); |
694 | seq_printf(m, " %s %-44s %15lu\n", tu->filename, | 702 | seq_printf(m, " %s %-44s %15lu\n", tu->filename, |
695 | trace_event_name(&tu->tp.call), tu->nhit); | 703 | trace_probe_name(&tu->tp), tu->nhit); |
696 | return 0; | 704 | return 0; |
697 | } | 705 | } |
698 | 706 | ||
@@ -818,7 +826,7 @@ static void __uprobe_trace_func(struct trace_uprobe *tu, | |||
818 | struct ring_buffer *buffer; | 826 | struct ring_buffer *buffer; |
819 | void *data; | 827 | void *data; |
820 | int size, esize; | 828 | int size, esize; |
821 | struct trace_event_call *call = &tu->tp.call; | 829 | struct trace_event_call *call = trace_probe_event_call(&tu->tp); |
822 | 830 | ||
823 | WARN_ON(call != trace_file->event_call); | 831 | WARN_ON(call != trace_file->event_call); |
824 | 832 | ||
@@ -860,7 +868,7 @@ static int uprobe_trace_func(struct trace_uprobe *tu, struct pt_regs *regs, | |||
860 | return 0; | 868 | return 0; |
861 | 869 | ||
862 | rcu_read_lock(); | 870 | rcu_read_lock(); |
863 | list_for_each_entry_rcu(link, &tu->tp.files, list) | 871 | trace_probe_for_each_link_rcu(link, &tu->tp) |
864 | __uprobe_trace_func(tu, 0, regs, ucb, dsize, link->file); | 872 | __uprobe_trace_func(tu, 0, regs, ucb, dsize, link->file); |
865 | rcu_read_unlock(); | 873 | rcu_read_unlock(); |
866 | 874 | ||
@@ -874,7 +882,7 @@ static void uretprobe_trace_func(struct trace_uprobe *tu, unsigned long func, | |||
874 | struct event_file_link *link; | 882 | struct event_file_link *link; |
875 | 883 | ||
876 | rcu_read_lock(); | 884 | rcu_read_lock(); |
877 | list_for_each_entry_rcu(link, &tu->tp.files, list) | 885 | trace_probe_for_each_link_rcu(link, &tu->tp) |
878 | __uprobe_trace_func(tu, func, regs, ucb, dsize, link->file); | 886 | __uprobe_trace_func(tu, func, regs, ucb, dsize, link->file); |
879 | rcu_read_unlock(); | 887 | rcu_read_unlock(); |
880 | } | 888 | } |
@@ -893,12 +901,12 @@ print_uprobe_event(struct trace_iterator *iter, int flags, struct trace_event *e | |||
893 | 901 | ||
894 | if (is_ret_probe(tu)) { | 902 | if (is_ret_probe(tu)) { |
895 | trace_seq_printf(s, "%s: (0x%lx <- 0x%lx)", | 903 | trace_seq_printf(s, "%s: (0x%lx <- 0x%lx)", |
896 | trace_event_name(&tu->tp.call), | 904 | trace_probe_name(&tu->tp), |
897 | entry->vaddr[1], entry->vaddr[0]); | 905 | entry->vaddr[1], entry->vaddr[0]); |
898 | data = DATAOF_TRACE_ENTRY(entry, true); | 906 | data = DATAOF_TRACE_ENTRY(entry, true); |
899 | } else { | 907 | } else { |
900 | trace_seq_printf(s, "%s: (0x%lx)", | 908 | trace_seq_printf(s, "%s: (0x%lx)", |
901 | trace_event_name(&tu->tp.call), | 909 | trace_probe_name(&tu->tp), |
902 | entry->vaddr[0]); | 910 | entry->vaddr[0]); |
903 | data = DATAOF_TRACE_ENTRY(entry, false); | 911 | data = DATAOF_TRACE_ENTRY(entry, false); |
904 | } | 912 | } |
@@ -921,26 +929,20 @@ probe_event_enable(struct trace_uprobe *tu, struct trace_event_file *file, | |||
921 | filter_func_t filter) | 929 | filter_func_t filter) |
922 | { | 930 | { |
923 | bool enabled = trace_probe_is_enabled(&tu->tp); | 931 | bool enabled = trace_probe_is_enabled(&tu->tp); |
924 | struct event_file_link *link = NULL; | ||
925 | int ret; | 932 | int ret; |
926 | 933 | ||
927 | if (file) { | 934 | if (file) { |
928 | if (tu->tp.flags & TP_FLAG_PROFILE) | 935 | if (trace_probe_test_flag(&tu->tp, TP_FLAG_PROFILE)) |
929 | return -EINTR; | 936 | return -EINTR; |
930 | 937 | ||
931 | link = kmalloc(sizeof(*link), GFP_KERNEL); | 938 | ret = trace_probe_add_file(&tu->tp, file); |
932 | if (!link) | 939 | if (ret < 0) |
933 | return -ENOMEM; | 940 | return ret; |
934 | |||
935 | link->file = file; | ||
936 | list_add_tail_rcu(&link->list, &tu->tp.files); | ||
937 | |||
938 | tu->tp.flags |= TP_FLAG_TRACE; | ||
939 | } else { | 941 | } else { |
940 | if (tu->tp.flags & TP_FLAG_TRACE) | 942 | if (trace_probe_test_flag(&tu->tp, TP_FLAG_TRACE)) |
941 | return -EINTR; | 943 | return -EINTR; |
942 | 944 | ||
943 | tu->tp.flags |= TP_FLAG_PROFILE; | 945 | trace_probe_set_flag(&tu->tp, TP_FLAG_PROFILE); |
944 | } | 946 | } |
945 | 947 | ||
946 | WARN_ON(!uprobe_filter_is_empty(&tu->filter)); | 948 | WARN_ON(!uprobe_filter_is_empty(&tu->filter)); |
@@ -970,13 +972,11 @@ probe_event_enable(struct trace_uprobe *tu, struct trace_event_file *file, | |||
970 | uprobe_buffer_disable(); | 972 | uprobe_buffer_disable(); |
971 | 973 | ||
972 | err_flags: | 974 | err_flags: |
973 | if (file) { | 975 | if (file) |
974 | list_del(&link->list); | 976 | trace_probe_remove_file(&tu->tp, file); |
975 | kfree(link); | 977 | else |
976 | tu->tp.flags &= ~TP_FLAG_TRACE; | 978 | trace_probe_clear_flag(&tu->tp, TP_FLAG_PROFILE); |
977 | } else { | 979 | |
978 | tu->tp.flags &= ~TP_FLAG_PROFILE; | ||
979 | } | ||
980 | return ret; | 980 | return ret; |
981 | } | 981 | } |
982 | 982 | ||
@@ -987,26 +987,18 @@ probe_event_disable(struct trace_uprobe *tu, struct trace_event_file *file) | |||
987 | return; | 987 | return; |
988 | 988 | ||
989 | if (file) { | 989 | if (file) { |
990 | struct event_file_link *link; | 990 | if (trace_probe_remove_file(&tu->tp, file) < 0) |
991 | |||
992 | link = find_event_file_link(&tu->tp, file); | ||
993 | if (!link) | ||
994 | return; | 991 | return; |
995 | 992 | ||
996 | list_del_rcu(&link->list); | 993 | if (trace_probe_is_enabled(&tu->tp)) |
997 | /* synchronize with u{,ret}probe_trace_func */ | ||
998 | synchronize_rcu(); | ||
999 | kfree(link); | ||
1000 | |||
1001 | if (!list_empty(&tu->tp.files)) | ||
1002 | return; | 994 | return; |
1003 | } | 995 | } else |
996 | trace_probe_clear_flag(&tu->tp, TP_FLAG_PROFILE); | ||
1004 | 997 | ||
1005 | WARN_ON(!uprobe_filter_is_empty(&tu->filter)); | 998 | WARN_ON(!uprobe_filter_is_empty(&tu->filter)); |
1006 | 999 | ||
1007 | uprobe_unregister(tu->inode, tu->offset, &tu->consumer); | 1000 | uprobe_unregister(tu->inode, tu->offset, &tu->consumer); |
1008 | tu->inode = NULL; | 1001 | tu->inode = NULL; |
1009 | tu->tp.flags &= file ? ~TP_FLAG_TRACE : ~TP_FLAG_PROFILE; | ||
1010 | 1002 | ||
1011 | uprobe_buffer_disable(); | 1003 | uprobe_buffer_disable(); |
1012 | } | 1004 | } |
@@ -1126,7 +1118,7 @@ static void __uprobe_perf_func(struct trace_uprobe *tu, | |||
1126 | unsigned long func, struct pt_regs *regs, | 1118 | unsigned long func, struct pt_regs *regs, |
1127 | struct uprobe_cpu_buffer *ucb, int dsize) | 1119 | struct uprobe_cpu_buffer *ucb, int dsize) |
1128 | { | 1120 | { |
1129 | struct trace_event_call *call = &tu->tp.call; | 1121 | struct trace_event_call *call = trace_probe_event_call(&tu->tp); |
1130 | struct uprobe_trace_entry_head *entry; | 1122 | struct uprobe_trace_entry_head *entry; |
1131 | struct hlist_head *head; | 1123 | struct hlist_head *head; |
1132 | void *data; | 1124 | void *data; |
@@ -1279,11 +1271,11 @@ static int uprobe_dispatcher(struct uprobe_consumer *con, struct pt_regs *regs) | |||
1279 | ucb = uprobe_buffer_get(); | 1271 | ucb = uprobe_buffer_get(); |
1280 | store_trace_args(ucb->buf, &tu->tp, regs, esize, dsize); | 1272 | store_trace_args(ucb->buf, &tu->tp, regs, esize, dsize); |
1281 | 1273 | ||
1282 | if (tu->tp.flags & TP_FLAG_TRACE) | 1274 | if (trace_probe_test_flag(&tu->tp, TP_FLAG_TRACE)) |
1283 | ret |= uprobe_trace_func(tu, regs, ucb, dsize); | 1275 | ret |= uprobe_trace_func(tu, regs, ucb, dsize); |
1284 | 1276 | ||
1285 | #ifdef CONFIG_PERF_EVENTS | 1277 | #ifdef CONFIG_PERF_EVENTS |
1286 | if (tu->tp.flags & TP_FLAG_PROFILE) | 1278 | if (trace_probe_test_flag(&tu->tp, TP_FLAG_PROFILE)) |
1287 | ret |= uprobe_perf_func(tu, regs, ucb, dsize); | 1279 | ret |= uprobe_perf_func(tu, regs, ucb, dsize); |
1288 | #endif | 1280 | #endif |
1289 | uprobe_buffer_put(ucb); | 1281 | uprobe_buffer_put(ucb); |
@@ -1314,11 +1306,11 @@ static int uretprobe_dispatcher(struct uprobe_consumer *con, | |||
1314 | ucb = uprobe_buffer_get(); | 1306 | ucb = uprobe_buffer_get(); |
1315 | store_trace_args(ucb->buf, &tu->tp, regs, esize, dsize); | 1307 | store_trace_args(ucb->buf, &tu->tp, regs, esize, dsize); |
1316 | 1308 | ||
1317 | if (tu->tp.flags & TP_FLAG_TRACE) | 1309 | if (trace_probe_test_flag(&tu->tp, TP_FLAG_TRACE)) |
1318 | uretprobe_trace_func(tu, func, regs, ucb, dsize); | 1310 | uretprobe_trace_func(tu, func, regs, ucb, dsize); |
1319 | 1311 | ||
1320 | #ifdef CONFIG_PERF_EVENTS | 1312 | #ifdef CONFIG_PERF_EVENTS |
1321 | if (tu->tp.flags & TP_FLAG_PROFILE) | 1313 | if (trace_probe_test_flag(&tu->tp, TP_FLAG_PROFILE)) |
1322 | uretprobe_perf_func(tu, func, regs, ucb, dsize); | 1314 | uretprobe_perf_func(tu, func, regs, ucb, dsize); |
1323 | #endif | 1315 | #endif |
1324 | uprobe_buffer_put(ucb); | 1316 | uprobe_buffer_put(ucb); |
@@ -1329,10 +1321,10 @@ static struct trace_event_functions uprobe_funcs = { | |||
1329 | .trace = print_uprobe_event | 1321 | .trace = print_uprobe_event |
1330 | }; | 1322 | }; |
1331 | 1323 | ||
1332 | static inline void init_trace_event_call(struct trace_uprobe *tu, | 1324 | static inline void init_trace_event_call(struct trace_uprobe *tu) |
1333 | struct trace_event_call *call) | ||
1334 | { | 1325 | { |
1335 | INIT_LIST_HEAD(&call->class->fields); | 1326 | struct trace_event_call *call = trace_probe_event_call(&tu->tp); |
1327 | |||
1336 | call->event.funcs = &uprobe_funcs; | 1328 | call->event.funcs = &uprobe_funcs; |
1337 | call->class->define_fields = uprobe_event_define_fields; | 1329 | call->class->define_fields = uprobe_event_define_fields; |
1338 | 1330 | ||
@@ -1343,43 +1335,14 @@ static inline void init_trace_event_call(struct trace_uprobe *tu, | |||
1343 | 1335 | ||
1344 | static int register_uprobe_event(struct trace_uprobe *tu) | 1336 | static int register_uprobe_event(struct trace_uprobe *tu) |
1345 | { | 1337 | { |
1346 | struct trace_event_call *call = &tu->tp.call; | 1338 | init_trace_event_call(tu); |
1347 | int ret = 0; | ||
1348 | |||
1349 | init_trace_event_call(tu, call); | ||
1350 | |||
1351 | if (traceprobe_set_print_fmt(&tu->tp, is_ret_probe(tu)) < 0) | ||
1352 | return -ENOMEM; | ||
1353 | 1339 | ||
1354 | ret = register_trace_event(&call->event); | 1340 | return trace_probe_register_event_call(&tu->tp); |
1355 | if (!ret) { | ||
1356 | kfree(call->print_fmt); | ||
1357 | return -ENODEV; | ||
1358 | } | ||
1359 | |||
1360 | ret = trace_add_event_call(call); | ||
1361 | |||
1362 | if (ret) { | ||
1363 | pr_info("Failed to register uprobe event: %s\n", | ||
1364 | trace_event_name(call)); | ||
1365 | kfree(call->print_fmt); | ||
1366 | unregister_trace_event(&call->event); | ||
1367 | } | ||
1368 | |||
1369 | return ret; | ||
1370 | } | 1341 | } |
1371 | 1342 | ||
1372 | static int unregister_uprobe_event(struct trace_uprobe *tu) | 1343 | static int unregister_uprobe_event(struct trace_uprobe *tu) |
1373 | { | 1344 | { |
1374 | int ret; | 1345 | return trace_probe_unregister_event_call(&tu->tp); |
1375 | |||
1376 | /* tu->event is unregistered in trace_remove_event_call() */ | ||
1377 | ret = trace_remove_event_call(&tu->tp.call); | ||
1378 | if (ret) | ||
1379 | return ret; | ||
1380 | kfree(tu->tp.call.print_fmt); | ||
1381 | tu->tp.call.print_fmt = NULL; | ||
1382 | return 0; | ||
1383 | } | 1346 | } |
1384 | 1347 | ||
1385 | #ifdef CONFIG_PERF_EVENTS | 1348 | #ifdef CONFIG_PERF_EVENTS |
@@ -1419,14 +1382,14 @@ create_local_trace_uprobe(char *name, unsigned long offs, | |||
1419 | tu->path = path; | 1382 | tu->path = path; |
1420 | tu->ref_ctr_offset = ref_ctr_offset; | 1383 | tu->ref_ctr_offset = ref_ctr_offset; |
1421 | tu->filename = kstrdup(name, GFP_KERNEL); | 1384 | tu->filename = kstrdup(name, GFP_KERNEL); |
1422 | init_trace_event_call(tu, &tu->tp.call); | 1385 | init_trace_event_call(tu); |
1423 | 1386 | ||
1424 | if (traceprobe_set_print_fmt(&tu->tp, is_ret_probe(tu)) < 0) { | 1387 | if (traceprobe_set_print_fmt(&tu->tp, is_ret_probe(tu)) < 0) { |
1425 | ret = -ENOMEM; | 1388 | ret = -ENOMEM; |
1426 | goto error; | 1389 | goto error; |
1427 | } | 1390 | } |
1428 | 1391 | ||
1429 | return &tu->tp.call; | 1392 | return trace_probe_event_call(&tu->tp); |
1430 | error: | 1393 | error: |
1431 | free_trace_uprobe(tu); | 1394 | free_trace_uprobe(tu); |
1432 | return ERR_PTR(ret); | 1395 | return ERR_PTR(ret); |
@@ -1438,9 +1401,6 @@ void destroy_local_trace_uprobe(struct trace_event_call *event_call) | |||
1438 | 1401 | ||
1439 | tu = container_of(event_call, struct trace_uprobe, tp.call); | 1402 | tu = container_of(event_call, struct trace_uprobe, tp.call); |
1440 | 1403 | ||
1441 | kfree(tu->tp.call.print_fmt); | ||
1442 | tu->tp.call.print_fmt = NULL; | ||
1443 | |||
1444 | free_trace_uprobe(tu); | 1404 | free_trace_uprobe(tu); |
1445 | } | 1405 | } |
1446 | #endif /* CONFIG_PERF_EVENTS */ | 1406 | #endif /* CONFIG_PERF_EVENTS */ |
diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index df3ade14ccbd..73956eaff8a9 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c | |||
@@ -55,8 +55,8 @@ struct tp_probes { | |||
55 | 55 | ||
56 | static inline void *allocate_probes(int count) | 56 | static inline void *allocate_probes(int count) |
57 | { | 57 | { |
58 | struct tp_probes *p = kmalloc(count * sizeof(struct tracepoint_func) | 58 | struct tp_probes *p = kmalloc(struct_size(p, probes, count), |
59 | + sizeof(struct tp_probes), GFP_KERNEL); | 59 | GFP_KERNEL); |
60 | return p == NULL ? NULL : p->probes; | 60 | return p == NULL ? NULL : p->probes; |
61 | } | 61 | } |
62 | 62 | ||
diff --git a/mm/maccess.c b/mm/maccess.c index 482d4d670f19..d065736f6b87 100644 --- a/mm/maccess.c +++ b/mm/maccess.c | |||
@@ -6,8 +6,20 @@ | |||
6 | #include <linux/mm.h> | 6 | #include <linux/mm.h> |
7 | #include <linux/uaccess.h> | 7 | #include <linux/uaccess.h> |
8 | 8 | ||
9 | static __always_inline long | ||
10 | probe_read_common(void *dst, const void __user *src, size_t size) | ||
11 | { | ||
12 | long ret; | ||
13 | |||
14 | pagefault_disable(); | ||
15 | ret = __copy_from_user_inatomic(dst, src, size); | ||
16 | pagefault_enable(); | ||
17 | |||
18 | return ret ? -EFAULT : 0; | ||
19 | } | ||
20 | |||
9 | /** | 21 | /** |
10 | * probe_kernel_read(): safely attempt to read from a location | 22 | * probe_kernel_read(): safely attempt to read from a kernel-space location |
11 | * @dst: pointer to the buffer that shall take the data | 23 | * @dst: pointer to the buffer that shall take the data |
12 | * @src: address to read from | 24 | * @src: address to read from |
13 | * @size: size of the data chunk | 25 | * @size: size of the data chunk |
@@ -30,17 +42,41 @@ long __probe_kernel_read(void *dst, const void *src, size_t size) | |||
30 | mm_segment_t old_fs = get_fs(); | 42 | mm_segment_t old_fs = get_fs(); |
31 | 43 | ||
32 | set_fs(KERNEL_DS); | 44 | set_fs(KERNEL_DS); |
33 | pagefault_disable(); | 45 | ret = probe_read_common(dst, (__force const void __user *)src, size); |
34 | ret = __copy_from_user_inatomic(dst, | ||
35 | (__force const void __user *)src, size); | ||
36 | pagefault_enable(); | ||
37 | set_fs(old_fs); | 46 | set_fs(old_fs); |
38 | 47 | ||
39 | return ret ? -EFAULT : 0; | 48 | return ret; |
40 | } | 49 | } |
41 | EXPORT_SYMBOL_GPL(probe_kernel_read); | 50 | EXPORT_SYMBOL_GPL(probe_kernel_read); |
42 | 51 | ||
43 | /** | 52 | /** |
53 | * probe_user_read(): safely attempt to read from a user-space location | ||
54 | * @dst: pointer to the buffer that shall take the data | ||
55 | * @src: address to read from. This must be a user address. | ||
56 | * @size: size of the data chunk | ||
57 | * | ||
58 | * Safely read from user address @src to the buffer at @dst. If a kernel fault | ||
59 | * happens, handle that and return -EFAULT. | ||
60 | */ | ||
61 | |||
62 | long __weak probe_user_read(void *dst, const void __user *src, size_t size) | ||
63 | __attribute__((alias("__probe_user_read"))); | ||
64 | |||
65 | long __probe_user_read(void *dst, const void __user *src, size_t size) | ||
66 | { | ||
67 | long ret = -EFAULT; | ||
68 | mm_segment_t old_fs = get_fs(); | ||
69 | |||
70 | set_fs(USER_DS); | ||
71 | if (access_ok(src, size)) | ||
72 | ret = probe_read_common(dst, src, size); | ||
73 | set_fs(old_fs); | ||
74 | |||
75 | return ret; | ||
76 | } | ||
77 | EXPORT_SYMBOL_GPL(probe_user_read); | ||
78 | |||
79 | /** | ||
44 | * probe_kernel_write(): safely attempt to write to a location | 80 | * probe_kernel_write(): safely attempt to write to a location |
45 | * @dst: address to write to | 81 | * @dst: address to write to |
46 | * @src: pointer to the data that shall be written | 82 | * @src: pointer to the data that shall be written |
@@ -67,6 +103,7 @@ long __probe_kernel_write(void *dst, const void *src, size_t size) | |||
67 | } | 103 | } |
68 | EXPORT_SYMBOL_GPL(probe_kernel_write); | 104 | EXPORT_SYMBOL_GPL(probe_kernel_write); |
69 | 105 | ||
106 | |||
70 | /** | 107 | /** |
71 | * strncpy_from_unsafe: - Copy a NUL terminated string from unsafe address. | 108 | * strncpy_from_unsafe: - Copy a NUL terminated string from unsafe address. |
72 | * @dst: Destination address, in kernel space. This buffer must be at | 109 | * @dst: Destination address, in kernel space. This buffer must be at |
@@ -106,3 +143,76 @@ long strncpy_from_unsafe(char *dst, const void *unsafe_addr, long count) | |||
106 | 143 | ||
107 | return ret ? -EFAULT : src - unsafe_addr; | 144 | return ret ? -EFAULT : src - unsafe_addr; |
108 | } | 145 | } |
146 | |||
147 | /** | ||
148 | * strncpy_from_unsafe_user: - Copy a NUL terminated string from unsafe user | ||
149 | * address. | ||
150 | * @dst: Destination address, in kernel space. This buffer must be at | ||
151 | * least @count bytes long. | ||
152 | * @unsafe_addr: Unsafe user address. | ||
153 | * @count: Maximum number of bytes to copy, including the trailing NUL. | ||
154 | * | ||
155 | * Copies a NUL-terminated string from unsafe user address to kernel buffer. | ||
156 | * | ||
157 | * On success, returns the length of the string INCLUDING the trailing NUL. | ||
158 | * | ||
159 | * If access fails, returns -EFAULT (some data may have been copied | ||
160 | * and the trailing NUL added). | ||
161 | * | ||
162 | * If @count is smaller than the length of the string, copies @count-1 bytes, | ||
163 | * sets the last byte of @dst buffer to NUL and returns @count. | ||
164 | */ | ||
165 | long strncpy_from_unsafe_user(char *dst, const void __user *unsafe_addr, | ||
166 | long count) | ||
167 | { | ||
168 | mm_segment_t old_fs = get_fs(); | ||
169 | long ret; | ||
170 | |||
171 | if (unlikely(count <= 0)) | ||
172 | return 0; | ||
173 | |||
174 | set_fs(USER_DS); | ||
175 | pagefault_disable(); | ||
176 | ret = strncpy_from_user(dst, unsafe_addr, count); | ||
177 | pagefault_enable(); | ||
178 | set_fs(old_fs); | ||
179 | |||
180 | if (ret >= count) { | ||
181 | ret = count; | ||
182 | dst[ret - 1] = '\0'; | ||
183 | } else if (ret > 0) { | ||
184 | ret++; | ||
185 | } | ||
186 | |||
187 | return ret; | ||
188 | } | ||
189 | |||
190 | /** | ||
191 | * strnlen_unsafe_user: - Get the size of a user string INCLUDING final NUL. | ||
192 | * @unsafe_addr: The string to measure. | ||
193 | * @count: Maximum count (including NUL) | ||
194 | * | ||
195 | * Get the size of a NUL-terminated string in user space without pagefault. | ||
196 | * | ||
197 | * Returns the size of the string INCLUDING the terminating NUL. | ||
198 | * | ||
199 | * If the string is too long, returns a number larger than @count. User | ||
200 | * has to check the return value against "> count". | ||
201 | * On exception (or invalid count), returns 0. | ||
202 | * | ||
203 | * Unlike strnlen_user, this can be used from IRQ handler etc. because | ||
204 | * it disables pagefaults. | ||
205 | */ | ||
206 | long strnlen_unsafe_user(const void __user *unsafe_addr, long count) | ||
207 | { | ||
208 | mm_segment_t old_fs = get_fs(); | ||
209 | int ret; | ||
210 | |||
211 | set_fs(USER_DS); | ||
212 | pagefault_disable(); | ||
213 | ret = strnlen_user(unsafe_addr, count); | ||
214 | pagefault_enable(); | ||
215 | set_fs(old_fs); | ||
216 | |||
217 | return ret; | ||
218 | } | ||
diff --git a/tools/perf/Documentation/perf-probe.txt b/tools/perf/Documentation/perf-probe.txt index b6866a05edd2..ed3ecfa422e1 100644 --- a/tools/perf/Documentation/perf-probe.txt +++ b/tools/perf/Documentation/perf-probe.txt | |||
@@ -194,12 +194,13 @@ PROBE ARGUMENT | |||
194 | -------------- | 194 | -------------- |
195 | Each probe argument follows below syntax. | 195 | Each probe argument follows below syntax. |
196 | 196 | ||
197 | [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE] | 197 | [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE][@user] |
198 | 198 | ||
199 | 'NAME' specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var->field, var.field2), local array with fixed index (e.g. array[1], var->array[0], var->pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this argument will be set as the last member name if you specify a local data structure member (e.g. field2 for 'var->field1.field2'.) | 199 | 'NAME' specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var->field, var.field2), local array with fixed index (e.g. array[1], var->array[0], var->pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this argument will be set as the last member name if you specify a local data structure member (e.g. field2 for 'var->field1.field2'.) |
200 | '$vars' and '$params' special arguments are also available for NAME, '$vars' is expanded to the local variables (including function parameters) which can access at given probe point. '$params' is expanded to only the function parameters. | 200 | '$vars' and '$params' special arguments are also available for NAME, '$vars' is expanded to the local variables (including function parameters) which can access at given probe point. '$params' is expanded to only the function parameters. |
201 | 'TYPE' casts the type of this argument (optional). If omitted, perf probe automatically set the type based on debuginfo (*). Currently, basic types (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal integers (x/x8/x16/x32/x64), signedness casting (u/s), "string" and bitfield are supported. (see TYPES for detail) | 201 | 'TYPE' casts the type of this argument (optional). If omitted, perf probe automatically set the type based on debuginfo (*). Currently, basic types (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal integers (x/x8/x16/x32/x64), signedness casting (u/s), "string" and bitfield are supported. (see TYPES for detail) |
202 | On x86 systems %REG is always the short form of the register: for example %AX. %RAX or %EAX is not valid. | 202 | On x86 systems %REG is always the short form of the register: for example %AX. %RAX or %EAX is not valid. |
203 | "@user" is a special attribute which means the LOCALVAR will be treated as a user-space memory. This is only valid for kprobe event. | ||
203 | 204 | ||
204 | TYPES | 205 | TYPES |
205 | ----- | 206 | ----- |
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c index 0c3b55d0617d..cd1eb73cfe83 100644 --- a/tools/perf/util/probe-event.c +++ b/tools/perf/util/probe-event.c | |||
@@ -1562,6 +1562,17 @@ static int parse_perf_probe_arg(char *str, struct perf_probe_arg *arg) | |||
1562 | str = tmp + 1; | 1562 | str = tmp + 1; |
1563 | } | 1563 | } |
1564 | 1564 | ||
1565 | tmp = strchr(str, '@'); | ||
1566 | if (tmp && tmp != str && strcmp(tmp + 1, "user")) { /* user attr */ | ||
1567 | if (!user_access_is_supported()) { | ||
1568 | semantic_error("ftrace does not support user access\n"); | ||
1569 | return -EINVAL; | ||
1570 | } | ||
1571 | *tmp = '\0'; | ||
1572 | arg->user_access = true; | ||
1573 | pr_debug("user_access "); | ||
1574 | } | ||
1575 | |||
1565 | tmp = strchr(str, ':'); | 1576 | tmp = strchr(str, ':'); |
1566 | if (tmp) { /* Type setting */ | 1577 | if (tmp) { /* Type setting */ |
1567 | *tmp = '\0'; | 1578 | *tmp = '\0'; |
diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h index 05c8d571a901..96a319cd2378 100644 --- a/tools/perf/util/probe-event.h +++ b/tools/perf/util/probe-event.h | |||
@@ -37,6 +37,7 @@ struct probe_trace_point { | |||
37 | struct probe_trace_arg_ref { | 37 | struct probe_trace_arg_ref { |
38 | struct probe_trace_arg_ref *next; /* Next reference */ | 38 | struct probe_trace_arg_ref *next; /* Next reference */ |
39 | long offset; /* Offset value */ | 39 | long offset; /* Offset value */ |
40 | bool user_access; /* User-memory access */ | ||
40 | }; | 41 | }; |
41 | 42 | ||
42 | /* kprobe-tracer and uprobe-tracer tracing argument */ | 43 | /* kprobe-tracer and uprobe-tracer tracing argument */ |
@@ -82,6 +83,7 @@ struct perf_probe_arg { | |||
82 | char *var; /* Variable name */ | 83 | char *var; /* Variable name */ |
83 | char *type; /* Type name */ | 84 | char *type; /* Type name */ |
84 | struct perf_probe_arg_field *field; /* Structure fields */ | 85 | struct perf_probe_arg_field *field; /* Structure fields */ |
86 | bool user_access; /* User-memory access */ | ||
85 | }; | 87 | }; |
86 | 88 | ||
87 | /* Perf probe probing event (point + arg) */ | 89 | /* Perf probe probing event (point + arg) */ |
diff --git a/tools/perf/util/probe-file.c b/tools/perf/util/probe-file.c index c2998f90b23c..5b4d49382932 100644 --- a/tools/perf/util/probe-file.c +++ b/tools/perf/util/probe-file.c | |||
@@ -1005,6 +1005,7 @@ enum ftrace_readme { | |||
1005 | FTRACE_README_PROBE_TYPE_X = 0, | 1005 | FTRACE_README_PROBE_TYPE_X = 0, |
1006 | FTRACE_README_KRETPROBE_OFFSET, | 1006 | FTRACE_README_KRETPROBE_OFFSET, |
1007 | FTRACE_README_UPROBE_REF_CTR, | 1007 | FTRACE_README_UPROBE_REF_CTR, |
1008 | FTRACE_README_USER_ACCESS, | ||
1008 | FTRACE_README_END, | 1009 | FTRACE_README_END, |
1009 | }; | 1010 | }; |
1010 | 1011 | ||
@@ -1017,6 +1018,7 @@ static struct { | |||
1017 | DEFINE_TYPE(FTRACE_README_PROBE_TYPE_X, "*type: * x8/16/32/64,*"), | 1018 | DEFINE_TYPE(FTRACE_README_PROBE_TYPE_X, "*type: * x8/16/32/64,*"), |
1018 | DEFINE_TYPE(FTRACE_README_KRETPROBE_OFFSET, "*place (kretprobe): *"), | 1019 | DEFINE_TYPE(FTRACE_README_KRETPROBE_OFFSET, "*place (kretprobe): *"), |
1019 | DEFINE_TYPE(FTRACE_README_UPROBE_REF_CTR, "*ref_ctr_offset*"), | 1020 | DEFINE_TYPE(FTRACE_README_UPROBE_REF_CTR, "*ref_ctr_offset*"), |
1021 | DEFINE_TYPE(FTRACE_README_USER_ACCESS, "*[u]<offset>*"), | ||
1020 | }; | 1022 | }; |
1021 | 1023 | ||
1022 | static bool scan_ftrace_readme(enum ftrace_readme type) | 1024 | static bool scan_ftrace_readme(enum ftrace_readme type) |
@@ -1077,3 +1079,8 @@ bool uprobe_ref_ctr_is_supported(void) | |||
1077 | { | 1079 | { |
1078 | return scan_ftrace_readme(FTRACE_README_UPROBE_REF_CTR); | 1080 | return scan_ftrace_readme(FTRACE_README_UPROBE_REF_CTR); |
1079 | } | 1081 | } |
1082 | |||
1083 | bool user_access_is_supported(void) | ||
1084 | { | ||
1085 | return scan_ftrace_readme(FTRACE_README_USER_ACCESS); | ||
1086 | } | ||
diff --git a/tools/perf/util/probe-file.h b/tools/perf/util/probe-file.h index 2a249182f2a6..986c1c94f64f 100644 --- a/tools/perf/util/probe-file.h +++ b/tools/perf/util/probe-file.h | |||
@@ -70,6 +70,7 @@ int probe_cache__show_all_caches(struct strfilter *filter); | |||
70 | bool probe_type_is_available(enum probe_type type); | 70 | bool probe_type_is_available(enum probe_type type); |
71 | bool kretprobe_offset_is_supported(void); | 71 | bool kretprobe_offset_is_supported(void); |
72 | bool uprobe_ref_ctr_is_supported(void); | 72 | bool uprobe_ref_ctr_is_supported(void); |
73 | bool user_access_is_supported(void); | ||
73 | #else /* ! HAVE_LIBELF_SUPPORT */ | 74 | #else /* ! HAVE_LIBELF_SUPPORT */ |
74 | static inline struct probe_cache *probe_cache__new(const char *tgt __maybe_unused, struct nsinfo *nsi __maybe_unused) | 75 | static inline struct probe_cache *probe_cache__new(const char *tgt __maybe_unused, struct nsinfo *nsi __maybe_unused) |
75 | { | 76 | { |
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c index 7d8c99734928..025fc4491993 100644 --- a/tools/perf/util/probe-finder.c +++ b/tools/perf/util/probe-finder.c | |||
@@ -280,7 +280,7 @@ static_var: | |||
280 | 280 | ||
281 | static int convert_variable_type(Dwarf_Die *vr_die, | 281 | static int convert_variable_type(Dwarf_Die *vr_die, |
282 | struct probe_trace_arg *tvar, | 282 | struct probe_trace_arg *tvar, |
283 | const char *cast) | 283 | const char *cast, bool user_access) |
284 | { | 284 | { |
285 | struct probe_trace_arg_ref **ref_ptr = &tvar->ref; | 285 | struct probe_trace_arg_ref **ref_ptr = &tvar->ref; |
286 | Dwarf_Die type; | 286 | Dwarf_Die type; |
@@ -320,7 +320,8 @@ static int convert_variable_type(Dwarf_Die *vr_die, | |||
320 | pr_debug("%s type is %s.\n", | 320 | pr_debug("%s type is %s.\n", |
321 | dwarf_diename(vr_die), dwarf_diename(&type)); | 321 | dwarf_diename(vr_die), dwarf_diename(&type)); |
322 | 322 | ||
323 | if (cast && strcmp(cast, "string") == 0) { /* String type */ | 323 | if (cast && (!strcmp(cast, "string") || !strcmp(cast, "ustring"))) { |
324 | /* String type */ | ||
324 | ret = dwarf_tag(&type); | 325 | ret = dwarf_tag(&type); |
325 | if (ret != DW_TAG_pointer_type && | 326 | if (ret != DW_TAG_pointer_type && |
326 | ret != DW_TAG_array_type) { | 327 | ret != DW_TAG_array_type) { |
@@ -343,6 +344,7 @@ static int convert_variable_type(Dwarf_Die *vr_die, | |||
343 | pr_warning("Out of memory error\n"); | 344 | pr_warning("Out of memory error\n"); |
344 | return -ENOMEM; | 345 | return -ENOMEM; |
345 | } | 346 | } |
347 | (*ref_ptr)->user_access = user_access; | ||
346 | } | 348 | } |
347 | if (!die_compare_name(&type, "char") && | 349 | if (!die_compare_name(&type, "char") && |
348 | !die_compare_name(&type, "unsigned char")) { | 350 | !die_compare_name(&type, "unsigned char")) { |
@@ -397,7 +399,7 @@ formatted: | |||
397 | static int convert_variable_fields(Dwarf_Die *vr_die, const char *varname, | 399 | static int convert_variable_fields(Dwarf_Die *vr_die, const char *varname, |
398 | struct perf_probe_arg_field *field, | 400 | struct perf_probe_arg_field *field, |
399 | struct probe_trace_arg_ref **ref_ptr, | 401 | struct probe_trace_arg_ref **ref_ptr, |
400 | Dwarf_Die *die_mem) | 402 | Dwarf_Die *die_mem, bool user_access) |
401 | { | 403 | { |
402 | struct probe_trace_arg_ref *ref = *ref_ptr; | 404 | struct probe_trace_arg_ref *ref = *ref_ptr; |
403 | Dwarf_Die type; | 405 | Dwarf_Die type; |
@@ -434,6 +436,7 @@ static int convert_variable_fields(Dwarf_Die *vr_die, const char *varname, | |||
434 | *ref_ptr = ref; | 436 | *ref_ptr = ref; |
435 | } | 437 | } |
436 | ref->offset += dwarf_bytesize(&type) * field->index; | 438 | ref->offset += dwarf_bytesize(&type) * field->index; |
439 | ref->user_access = user_access; | ||
437 | goto next; | 440 | goto next; |
438 | } else if (tag == DW_TAG_pointer_type) { | 441 | } else if (tag == DW_TAG_pointer_type) { |
439 | /* Check the pointer and dereference */ | 442 | /* Check the pointer and dereference */ |
@@ -505,17 +508,18 @@ static int convert_variable_fields(Dwarf_Die *vr_die, const char *varname, | |||
505 | } | 508 | } |
506 | } | 509 | } |
507 | ref->offset += (long)offs; | 510 | ref->offset += (long)offs; |
511 | ref->user_access = user_access; | ||
508 | 512 | ||
509 | /* If this member is unnamed, we need to reuse this field */ | 513 | /* If this member is unnamed, we need to reuse this field */ |
510 | if (!dwarf_diename(die_mem)) | 514 | if (!dwarf_diename(die_mem)) |
511 | return convert_variable_fields(die_mem, varname, field, | 515 | return convert_variable_fields(die_mem, varname, field, |
512 | &ref, die_mem); | 516 | &ref, die_mem, user_access); |
513 | 517 | ||
514 | next: | 518 | next: |
515 | /* Converting next field */ | 519 | /* Converting next field */ |
516 | if (field->next) | 520 | if (field->next) |
517 | return convert_variable_fields(die_mem, field->name, | 521 | return convert_variable_fields(die_mem, field->name, |
518 | field->next, &ref, die_mem); | 522 | field->next, &ref, die_mem, user_access); |
519 | else | 523 | else |
520 | return 0; | 524 | return 0; |
521 | } | 525 | } |
@@ -541,11 +545,12 @@ static int convert_variable(Dwarf_Die *vr_die, struct probe_finder *pf) | |||
541 | else if (ret == 0 && pf->pvar->field) { | 545 | else if (ret == 0 && pf->pvar->field) { |
542 | ret = convert_variable_fields(vr_die, pf->pvar->var, | 546 | ret = convert_variable_fields(vr_die, pf->pvar->var, |
543 | pf->pvar->field, &pf->tvar->ref, | 547 | pf->pvar->field, &pf->tvar->ref, |
544 | &die_mem); | 548 | &die_mem, pf->pvar->user_access); |
545 | vr_die = &die_mem; | 549 | vr_die = &die_mem; |
546 | } | 550 | } |
547 | if (ret == 0) | 551 | if (ret == 0) |
548 | ret = convert_variable_type(vr_die, pf->tvar, pf->pvar->type); | 552 | ret = convert_variable_type(vr_die, pf->tvar, pf->pvar->type, |
553 | pf->pvar->user_access); | ||
549 | /* *expr will be cached in libdw. Don't free it. */ | 554 | /* *expr will be cached in libdw. Don't free it. */ |
550 | return ret; | 555 | return ret; |
551 | } | 556 | } |
diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest index 6d5e9e87c4b7..063ecb290a5a 100755 --- a/tools/testing/selftests/ftrace/ftracetest +++ b/tools/testing/selftests/ftrace/ftracetest | |||
@@ -23,9 +23,15 @@ echo " If <dir> is -, all logs output in console only" | |||
23 | exit $1 | 23 | exit $1 |
24 | } | 24 | } |
25 | 25 | ||
26 | # default error | ||
27 | err_ret=1 | ||
28 | |||
29 | # kselftest skip code is 4 | ||
30 | err_skip=4 | ||
31 | |||
26 | errexit() { # message | 32 | errexit() { # message |
27 | echo "Error: $1" 1>&2 | 33 | echo "Error: $1" 1>&2 |
28 | exit 1 | 34 | exit $err_ret |
29 | } | 35 | } |
30 | 36 | ||
31 | # Ensuring user privilege | 37 | # Ensuring user privilege |
@@ -116,11 +122,31 @@ parse_opts() { # opts | |||
116 | } | 122 | } |
117 | 123 | ||
118 | # Parameters | 124 | # Parameters |
119 | DEBUGFS_DIR=`grep debugfs /proc/mounts | cut -f2 -d' ' | head -1` | 125 | TRACING_DIR=`grep tracefs /proc/mounts | cut -f2 -d' ' | head -1` |
120 | if [ -z "$DEBUGFS_DIR" ]; then | 126 | if [ -z "$TRACING_DIR" ]; then |
121 | TRACING_DIR=`grep tracefs /proc/mounts | cut -f2 -d' ' | head -1` | 127 | DEBUGFS_DIR=`grep debugfs /proc/mounts | cut -f2 -d' ' | head -1` |
122 | else | 128 | if [ -z "$DEBUGFS_DIR" ]; then |
123 | TRACING_DIR=$DEBUGFS_DIR/tracing | 129 | # If tracefs exists, then so does /sys/kernel/tracing |
130 | if [ -d "/sys/kernel/tracing" ]; then | ||
131 | mount -t tracefs nodev /sys/kernel/tracing || | ||
132 | errexit "Failed to mount /sys/kernel/tracing" | ||
133 | TRACING_DIR="/sys/kernel/tracing" | ||
134 | # If debugfs exists, then so does /sys/kernel/debug | ||
135 | elif [ -d "/sys/kernel/debug" ]; then | ||
136 | mount -t debugfs nodev /sys/kernel/debug || | ||
137 | errexit "Failed to mount /sys/kernel/debug" | ||
138 | TRACING_DIR="/sys/kernel/debug/tracing" | ||
139 | else | ||
140 | err_ret=$err_skip | ||
141 | errexit "debugfs and tracefs are not configured in this kernel" | ||
142 | fi | ||
143 | else | ||
144 | TRACING_DIR="$DEBUGFS_DIR/tracing" | ||
145 | fi | ||
146 | fi | ||
147 | if [ ! -d "$TRACING_DIR" ]; then | ||
148 | err_ret=$err_skip | ||
149 | errexit "ftrace is not configured in this kernel" | ||
124 | fi | 150 | fi |
125 | 151 | ||
126 | TOP_DIR=`absdir $0` | 152 | TOP_DIR=`absdir $0` |
diff --git a/tools/testing/selftests/ftrace/test.d/functions b/tools/testing/selftests/ftrace/test.d/functions index 779ec11f61bd..1d96c5f7e402 100644 --- a/tools/testing/selftests/ftrace/test.d/functions +++ b/tools/testing/selftests/ftrace/test.d/functions | |||
@@ -91,8 +91,8 @@ initialize_ftrace() { # Reset ftrace to initial-state | |||
91 | reset_events_filter | 91 | reset_events_filter |
92 | reset_ftrace_filter | 92 | reset_ftrace_filter |
93 | disable_events | 93 | disable_events |
94 | echo > set_event_pid # event tracer is always on | 94 | [ -f set_event_pid ] && echo > set_event_pid |
95 | echo > set_ftrace_pid | 95 | [ -f set_ftrace_pid ] && echo > set_ftrace_pid |
96 | [ -f set_ftrace_filter ] && echo | tee set_ftrace_* | 96 | [ -f set_ftrace_filter ] && echo | tee set_ftrace_* |
97 | [ -f set_graph_function ] && echo | tee set_graph_* | 97 | [ -f set_graph_function ] && echo | tee set_graph_* |
98 | [ -f stack_trace_filter ] && echo > stack_trace_filter | 98 | [ -f stack_trace_filter ] && echo > stack_trace_filter |
diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_user.tc b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_user.tc new file mode 100644 index 000000000000..0f60087583d8 --- /dev/null +++ b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_user.tc | |||
@@ -0,0 +1,32 @@ | |||
1 | #!/bin/sh | ||
2 | # SPDX-License-Identifier: GPL-2.0 | ||
3 | # description: Kprobe event user-memory access | ||
4 | |||
5 | [ -f kprobe_events ] || exit_unsupported # this is configurable | ||
6 | |||
7 | grep -q '\$arg<N>' README || exit_unresolved # depends on arch | ||
8 | grep -A10 "fetcharg:" README | grep -q 'ustring' || exit_unsupported | ||
9 | grep -A10 "fetcharg:" README | grep -q '\[u\]<offset>' || exit_unsupported | ||
10 | |||
11 | :;: "user-memory access syntax and ustring working on user memory";: | ||
12 | echo 'p:myevent do_sys_open path=+0($arg2):ustring path2=+u0($arg2):string' \ | ||
13 | > kprobe_events | ||
14 | |||
15 | grep myevent kprobe_events | \ | ||
16 | grep -q 'path=+0($arg2):ustring path2=+u0($arg2):string' | ||
17 | echo 1 > events/kprobes/myevent/enable | ||
18 | echo > /dev/null | ||
19 | echo 0 > events/kprobes/myevent/enable | ||
20 | |||
21 | grep myevent trace | grep -q 'path="/dev/null" path2="/dev/null"' | ||
22 | |||
23 | :;: "user-memory access syntax and ustring not working with kernel memory";: | ||
24 | echo 'p:myevent vfs_symlink path=+0($arg3):ustring path2=+u0($arg3):string' \ | ||
25 | > kprobe_events | ||
26 | echo 1 > events/kprobes/myevent/enable | ||
27 | ln -s foo $TMPDIR/bar | ||
28 | echo 0 > events/kprobes/myevent/enable | ||
29 | |||
30 | grep myevent trace | grep -q 'path=(fault) path2=(fault)' | ||
31 | |||
32 | exit 0 | ||