aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2016-05-17 20:11:27 -0400
committerLinus Torvalds <torvalds@linux-foundation.org>2016-05-17 20:11:27 -0400
commit0b86c75db6e7f68c22ac5d0dae0f551c4897cdf5 (patch)
treed1be280c331fbd85c021b5686914d2cc21475f54
parent16bf8348055fe4615bd08ef50f9874f5dcc10268 (diff)
parentbe69f70e6395a4ba9c178b2531433547e1955195 (diff)
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/livepatching
Pull livepatching updates from Jiri Kosina: - remove of our own implementation of architecture-specific relocation code and leveraging existing code in the module loader to perform arch-dependent work, from Jessica Yu. The relevant patches have been acked by Rusty (for module.c) and Heiko (for s390). - live patching support for ppc64le, which is a joint work of Michael Ellerman and Torsten Duwe. This is coming from topic branch that is share between livepatching.git and ppc tree. - addition of livepatching documentation from Petr Mladek * 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/livepatching: livepatch: make object/func-walking helpers more robust livepatch: Add some basic livepatch documentation powerpc/livepatch: Add live patching support on ppc64le powerpc/livepatch: Add livepatch stack to struct thread_info powerpc/livepatch: Add livepatch header livepatch: Allow architectures to specify an alternate ftrace location ftrace: Make ftrace_location_range() global livepatch: robustify klp_register_patch() API error checking Documentation: livepatch: outline Elf format and requirements for patch modules livepatch: reuse module loader code to write relocations module: s390: keep mod_arch_specific for livepatch modules module: preserve Elf information for livepatch modules Elf: add livepatch-specific Elf constants
-rw-r--r--Documentation/livepatch/livepatch.txt394
-rw-r--r--Documentation/livepatch/module-elf-format.txt311
-rw-r--r--MAINTAINERS1
-rw-r--r--arch/powerpc/Kconfig3
-rw-r--r--arch/powerpc/include/asm/livepatch.h62
-rw-r--r--arch/powerpc/include/asm/thread_info.h4
-rw-r--r--arch/powerpc/kernel/asm-offsets.c4
-rw-r--r--arch/powerpc/kernel/entry_64.S97
-rw-r--r--arch/powerpc/kernel/irq.c3
-rw-r--r--arch/powerpc/kernel/process.c6
-rw-r--r--arch/powerpc/kernel/setup_64.c17
-rw-r--r--arch/s390/include/asm/livepatch.h7
-rw-r--r--arch/s390/kernel/module.c6
-rw-r--r--arch/x86/include/asm/livepatch.h2
-rw-r--r--arch/x86/kernel/Makefile1
-rw-r--r--arch/x86/kernel/livepatch.c70
-rw-r--r--include/linux/ftrace.h1
-rw-r--r--include/linux/livepatch.h26
-rw-r--r--include/linux/module.h25
-rw-r--r--include/uapi/linux/elf.h10
-rw-r--r--kernel/livepatch/core.c191
-rw-r--r--kernel/module.c125
-rw-r--r--kernel/trace/ftrace.c14
-rw-r--r--samples/livepatch/livepatch-sample.c1
24 files changed, 1200 insertions, 181 deletions
diff --git a/Documentation/livepatch/livepatch.txt b/Documentation/livepatch/livepatch.txt
new file mode 100644
index 000000000000..6c43f6ebee8d
--- /dev/null
+++ b/Documentation/livepatch/livepatch.txt
@@ -0,0 +1,394 @@
1=========
2Livepatch
3=========
4
5This document outlines basic information about kernel livepatching.
6
7Table of Contents:
8
91. Motivation
102. Kprobes, Ftrace, Livepatching
113. Consistency model
124. Livepatch module
13 4.1. New functions
14 4.2. Metadata
15 4.3. Livepatch module handling
165. Livepatch life-cycle
17 5.1. Registration
18 5.2. Enabling
19 5.3. Disabling
20 5.4. Unregistration
216. Sysfs
227. Limitations
23
24
251. Motivation
26=============
27
28There are many situations where users are reluctant to reboot a system. It may
29be because their system is performing complex scientific computations or under
30heavy load during peak usage. In addition to keeping systems up and running,
31users want to also have a stable and secure system. Livepatching gives users
32both by allowing for function calls to be redirected; thus, fixing critical
33functions without a system reboot.
34
35
362. Kprobes, Ftrace, Livepatching
37================================
38
39There are multiple mechanisms in the Linux kernel that are directly related
40to redirection of code execution; namely: kernel probes, function tracing,
41and livepatching:
42
43 + The kernel probes are the most generic. The code can be redirected by
44 putting a breakpoint instruction instead of any instruction.
45
46 + The function tracer calls the code from a predefined location that is
47 close to the function entry point. This location is generated by the
48 compiler using the '-pg' gcc option.
49
50 + Livepatching typically needs to redirect the code at the very beginning
51 of the function entry before the function parameters or the stack
52 are in any way modified.
53
54All three approaches need to modify the existing code at runtime. Therefore
55they need to be aware of each other and not step over each other's toes.
56Most of these problems are solved by using the dynamic ftrace framework as
57a base. A Kprobe is registered as a ftrace handler when the function entry
58is probed, see CONFIG_KPROBES_ON_FTRACE. Also an alternative function from
59a live patch is called with the help of a custom ftrace handler. But there are
60some limitations, see below.
61
62
633. Consistency model
64====================
65
66Functions are there for a reason. They take some input parameters, get or
67release locks, read, process, and even write some data in a defined way,
68have return values. In other words, each function has a defined semantic.
69
70Many fixes do not change the semantic of the modified functions. For
71example, they add a NULL pointer or a boundary check, fix a race by adding
72a missing memory barrier, or add some locking around a critical section.
73Most of these changes are self contained and the function presents itself
74the same way to the rest of the system. In this case, the functions might
75be updated independently one by one.
76
77But there are more complex fixes. For example, a patch might change
78ordering of locking in multiple functions at the same time. Or a patch
79might exchange meaning of some temporary structures and update
80all the relevant functions. In this case, the affected unit
81(thread, whole kernel) need to start using all new versions of
82the functions at the same time. Also the switch must happen only
83when it is safe to do so, e.g. when the affected locks are released
84or no data are stored in the modified structures at the moment.
85
86The theory about how to apply functions a safe way is rather complex.
87The aim is to define a so-called consistency model. It attempts to define
88conditions when the new implementation could be used so that the system
89stays consistent. The theory is not yet finished. See the discussion at
90http://thread.gmane.org/gmane.linux.kernel/1823033/focus=1828189
91
92The current consistency model is very simple. It guarantees that either
93the old or the new function is called. But various functions get redirected
94one by one without any synchronization.
95
96In other words, the current implementation _never_ modifies the behavior
97in the middle of the call. It is because it does _not_ rewrite the entire
98function in the memory. Instead, the function gets redirected at the
99very beginning. But this redirection is used immediately even when
100some other functions from the same patch have not been redirected yet.
101
102See also the section "Limitations" below.
103
104
1054. Livepatch module
106===================
107
108Livepatches are distributed using kernel modules, see
109samples/livepatch/livepatch-sample.c.
110
111The module includes a new implementation of functions that we want
112to replace. In addition, it defines some structures describing the
113relation between the original and the new implementation. Then there
114is code that makes the kernel start using the new code when the livepatch
115module is loaded. Also there is code that cleans up before the
116livepatch module is removed. All this is explained in more details in
117the next sections.
118
119
1204.1. New functions
121------------------
122
123New versions of functions are typically just copied from the original
124sources. A good practice is to add a prefix to the names so that they
125can be distinguished from the original ones, e.g. in a backtrace. Also
126they can be declared as static because they are not called directly
127and do not need the global visibility.
128
129The patch contains only functions that are really modified. But they
130might want to access functions or data from the original source file
131that may only be locally accessible. This can be solved by a special
132relocation section in the generated livepatch module, see
133Documentation/livepatch/module-elf-format.txt for more details.
134
135
1364.2. Metadata
137------------
138
139The patch is described by several structures that split the information
140into three levels:
141
142 + struct klp_func is defined for each patched function. It describes
143 the relation between the original and the new implementation of a
144 particular function.
145
146 The structure includes the name, as a string, of the original function.
147 The function address is found via kallsyms at runtime.
148
149 Then it includes the address of the new function. It is defined
150 directly by assigning the function pointer. Note that the new
151 function is typically defined in the same source file.
152
153 As an optional parameter, the symbol position in the kallsyms database can
154 be used to disambiguate functions of the same name. This is not the
155 absolute position in the database, but rather the order it has been found
156 only for a particular object ( vmlinux or a kernel module ). Note that
157 kallsyms allows for searching symbols according to the object name.
158
159 + struct klp_object defines an array of patched functions (struct
160 klp_func) in the same object. Where the object is either vmlinux
161 (NULL) or a module name.
162
163 The structure helps to group and handle functions for each object
164 together. Note that patched modules might be loaded later than
165 the patch itself and the relevant functions might be patched
166 only when they are available.
167
168
169 + struct klp_patch defines an array of patched objects (struct
170 klp_object).
171
172 This structure handles all patched functions consistently and eventually,
173 synchronously. The whole patch is applied only when all patched
174 symbols are found. The only exception are symbols from objects
175 (kernel modules) that have not been loaded yet. Also if a more complex
176 consistency model is supported then a selected unit (thread,
177 kernel as a whole) will see the new code from the entire patch
178 only when it is in a safe state.
179
180
1814.3. Livepatch module handling
182------------------------------
183
184The usual behavior is that the new functions will get used when
185the livepatch module is loaded. For this, the module init() function
186has to register the patch (struct klp_patch) and enable it. See the
187section "Livepatch life-cycle" below for more details about these
188two operations.
189
190Module removal is only safe when there are no users of the underlying
191functions. The immediate consistency model is not able to detect this;
192therefore livepatch modules cannot be removed. See "Limitations" below.
193
1945. Livepatch life-cycle
195=======================
196
197Livepatching defines four basic operations that define the life cycle of each
198live patch: registration, enabling, disabling and unregistration. There are
199several reasons why it is done this way.
200
201First, the patch is applied only when all patched symbols for already
202loaded objects are found. The error handling is much easier if this
203check is done before particular functions get redirected.
204
205Second, the immediate consistency model does not guarantee that anyone is not
206sleeping in the new code after the patch is reverted. This means that the new
207code needs to stay around "forever". If the code is there, one could apply it
208again. Therefore it makes sense to separate the operations that might be done
209once and those that need to be repeated when the patch is enabled (applied)
210again.
211
212Third, it might take some time until the entire system is migrated
213when a more complex consistency model is used. The patch revert might
214block the livepatch module removal for too long. Therefore it is useful
215to revert the patch using a separate operation that might be called
216explicitly. But it does not make sense to remove all information
217until the livepatch module is really removed.
218
219
2205.1. Registration
221-----------------
222
223Each patch first has to be registered using klp_register_patch(). This makes
224the patch known to the livepatch framework. Also it does some preliminary
225computing and checks.
226
227In particular, the patch is added into the list of known patches. The
228addresses of the patched functions are found according to their names.
229The special relocations, mentioned in the section "New functions", are
230applied. The relevant entries are created under
231/sys/kernel/livepatch/<name>. The patch is rejected when any operation
232fails.
233
234
2355.2. Enabling
236-------------
237
238Registered patches might be enabled either by calling klp_enable_patch() or
239by writing '1' to /sys/kernel/livepatch/<name>/enabled. The system will
240start using the new implementation of the patched functions at this stage.
241
242In particular, if an original function is patched for the first time, a
243function specific struct klp_ops is created and an universal ftrace handler
244is registered.
245
246Functions might be patched multiple times. The ftrace handler is registered
247only once for the given function. Further patches just add an entry to the
248list (see field `func_stack`) of the struct klp_ops. The last added
249entry is chosen by the ftrace handler and becomes the active function
250replacement.
251
252Note that the patches might be enabled in a different order than they were
253registered.
254
255
2565.3. Disabling
257--------------
258
259Enabled patches might get disabled either by calling klp_disable_patch() or
260by writing '0' to /sys/kernel/livepatch/<name>/enabled. At this stage
261either the code from the previously enabled patch or even the original
262code gets used.
263
264Here all the functions (struct klp_func) associated with the to-be-disabled
265patch are removed from the corresponding struct klp_ops. The ftrace handler
266is unregistered and the struct klp_ops is freed when the func_stack list
267becomes empty.
268
269Patches must be disabled in exactly the reverse order in which they were
270enabled. It makes the problem and the implementation much easier.
271
272
2735.4. Unregistration
274-------------------
275
276Disabled patches might be unregistered by calling klp_unregister_patch().
277This can be done only when the patch is disabled and the code is no longer
278used. It must be called before the livepatch module gets unloaded.
279
280At this stage, all the relevant sys-fs entries are removed and the patch
281is removed from the list of known patches.
282
283
2846. Sysfs
285========
286
287Information about the registered patches can be found under
288/sys/kernel/livepatch. The patches could be enabled and disabled
289by writing there.
290
291See Documentation/ABI/testing/sysfs-kernel-livepatch for more details.
292
293
2947. Limitations
295==============
296
297The current Livepatch implementation has several limitations:
298
299
300 + The patch must not change the semantic of the patched functions.
301
302 The current implementation guarantees only that either the old
303 or the new function is called. The functions are patched one
304 by one. It means that the patch must _not_ change the semantic
305 of the function.
306
307
308 + Data structures can not be patched.
309
310 There is no support to version data structures or anyhow migrate
311 one structure into another. Also the simple consistency model does
312 not allow to switch more functions atomically.
313
314 Once there is more complex consistency mode, it will be possible to
315 use some workarounds. For example, it will be possible to use a hole
316 for a new member because the data structure is aligned. Or it will
317 be possible to use an existing member for something else.
318
319 There are no plans to add more generic support for modified structures
320 at the moment.
321
322
323 + Only functions that can be traced could be patched.
324
325 Livepatch is based on the dynamic ftrace. In particular, functions
326 implementing ftrace or the livepatch ftrace handler could not be
327 patched. Otherwise, the code would end up in an infinite loop. A
328 potential mistake is prevented by marking the problematic functions
329 by "notrace".
330
331
332 + Anything inlined into __schedule() can not be patched.
333
334 The switch_to macro is inlined into __schedule(). It switches the
335 context between two processes in the middle of the macro. It does
336 not save RIP in x86_64 version (contrary to 32-bit version). Instead,
337 the currently used __schedule()/switch_to() handles both processes.
338
339 Now, let's have two different tasks. One calls the original
340 __schedule(), its registers are stored in a defined order and it
341 goes to sleep in the switch_to macro and some other task is restored
342 using the original __schedule(). Then there is the second task which
343 calls patched__schedule(), it goes to sleep there and the first task
344 is picked by the patched__schedule(). Its RSP is restored and now
345 the registers should be restored as well. But the order is different
346 in the new patched__schedule(), so...
347
348 There is work in progress to remove this limitation.
349
350
351 + Livepatch modules can not be removed.
352
353 The current implementation just redirects the functions at the very
354 beginning. It does not check if the functions are in use. In other
355 words, it knows when the functions get called but it does not
356 know when the functions return. Therefore it can not decide when
357 the livepatch module can be safely removed.
358
359 This will get most likely solved once a more complex consistency model
360 is supported. The idea is that a safe state for patching should also
361 mean a safe state for removing the patch.
362
363 Note that the patch itself might get disabled by writing zero
364 to /sys/kernel/livepatch/<patch>/enabled. It causes that the new
365 code will not longer get called. But it does not guarantee
366 that anyone is not sleeping anywhere in the new code.
367
368
369 + Livepatch works reliably only when the dynamic ftrace is located at
370 the very beginning of the function.
371
372 The function need to be redirected before the stack or the function
373 parameters are modified in any way. For example, livepatch requires
374 using -fentry gcc compiler option on x86_64.
375
376 One exception is the PPC port. It uses relative addressing and TOC.
377 Each function has to handle TOC and save LR before it could call
378 the ftrace handler. This operation has to be reverted on return.
379 Fortunately, the generic ftrace code has the same problem and all
380 this is is handled on the ftrace level.
381
382
383 + Kretprobes using the ftrace framework conflict with the patched
384 functions.
385
386 Both kretprobes and livepatches use a ftrace handler that modifies
387 the return address. The first user wins. Either the probe or the patch
388 is rejected when the handler is already in use by the other.
389
390
391 + Kprobes in the original function are ignored when the code is
392 redirected to the new implementation.
393
394 There is a work in progress to add warnings about this situation.
diff --git a/Documentation/livepatch/module-elf-format.txt b/Documentation/livepatch/module-elf-format.txt
new file mode 100644
index 000000000000..eedbdcf8ba50
--- /dev/null
+++ b/Documentation/livepatch/module-elf-format.txt
@@ -0,0 +1,311 @@
1===========================
2Livepatch module Elf format
3===========================
4
5This document outlines the Elf format requirements that livepatch modules must follow.
6
7-----------------
8Table of Contents
9-----------------
100. Background and motivation
111. Livepatch modinfo field
122. Livepatch relocation sections
13 2.1 What are livepatch relocation sections?
14 2.2 Livepatch relocation section format
15 2.2.1 Required flags
16 2.2.2 Required name format
17 2.2.3 Example livepatch relocation section names
18 2.2.4 Example `readelf --sections` output
19 2.2.5 Example `readelf --relocs` output
203. Livepatch symbols
21 3.1 What are livepatch symbols?
22 3.2 A livepatch module's symbol table
23 3.3 Livepatch symbol format
24 3.3.1 Required flags
25 3.3.2 Required name format
26 3.3.3 Example livepatch symbol names
27 3.3.4 Example `readelf --symbols` output
284. Symbol table and Elf section access
29
30----------------------------
310. Background and motivation
32----------------------------
33
34Formerly, livepatch required separate architecture-specific code to write
35relocations. However, arch-specific code to write relocations already
36exists in the module loader, so this former approach produced redundant
37code. So, instead of duplicating code and re-implementing what the module
38loader can already do, livepatch leverages existing code in the module
39loader to perform the all the arch-specific relocation work. Specifically,
40livepatch reuses the apply_relocate_add() function in the module loader to
41write relocations. The patch module Elf format described in this document
42enables livepatch to be able to do this. The hope is that this will make
43livepatch more easily portable to other architectures and reduce the amount
44of arch-specific code required to port livepatch to a particular
45architecture.
46
47Since apply_relocate_add() requires access to a module's section header
48table, symbol table, and relocation section indices, Elf information is
49preserved for livepatch modules (see section 4). Livepatch manages its own
50relocation sections and symbols, which are described in this document. The
51Elf constants used to mark livepatch symbols and relocation sections were
52selected from OS-specific ranges according to the definitions from glibc.
53
540.1 Why does livepatch need to write its own relocations?
55---------------------------------------------------------
56A typical livepatch module contains patched versions of functions that can
57reference non-exported global symbols and non-included local symbols.
58Relocations referencing these types of symbols cannot be left in as-is
59since the kernel module loader cannot resolve them and will therefore
60reject the livepatch module. Furthermore, we cannot apply relocations that
61affect modules not yet loaded at patch module load time (e.g. a patch to a
62driver that is not loaded). Formerly, livepatch solved this problem by
63embedding special "dynrela" (dynamic rela) sections in the resulting patch
64module Elf output. Using these dynrela sections, livepatch could resolve
65symbols while taking into account its scope and what module the symbol
66belongs to, and then manually apply the dynamic relocations. However this
67approach required livepatch to supply arch-specific code in order to write
68these relocations. In the new format, livepatch manages its own SHT_RELA
69relocation sections in place of dynrela sections, and the symbols that the
70relas reference are special livepatch symbols (see section 2 and 3). The
71arch-specific livepatch relocation code is replaced by a call to
72apply_relocate_add().
73
74================================
75PATCH MODULE FORMAT REQUIREMENTS
76================================
77
78--------------------------
791. Livepatch modinfo field
80--------------------------
81
82Livepatch modules are required to have the "livepatch" modinfo attribute.
83See the sample livepatch module in samples/livepatch/ for how this is done.
84
85Livepatch modules can be identified by users by using the 'modinfo' command
86and looking for the presence of the "livepatch" field. This field is also
87used by the kernel module loader to identify livepatch modules.
88
89Example modinfo output:
90-----------------------
91% modinfo livepatch-meminfo.ko
92filename: livepatch-meminfo.ko
93livepatch: Y
94license: GPL
95depends:
96vermagic: 4.3.0+ SMP mod_unload
97
98--------------------------------
992. Livepatch relocation sections
100--------------------------------
101
102-------------------------------------------
1032.1 What are livepatch relocation sections?
104-------------------------------------------
105A livepatch module manages its own Elf relocation sections to apply
106relocations to modules as well as to the kernel (vmlinux) at the
107appropriate time. For example, if a patch module patches a driver that is
108not currently loaded, livepatch will apply the corresponding livepatch
109relocation section(s) to the driver once it loads.
110
111Each "object" (e.g. vmlinux, or a module) within a patch module may have
112multiple livepatch relocation sections associated with it (e.g. patches to
113multiple functions within the same object). There is a 1-1 correspondence
114between a livepatch relocation section and the target section (usually the
115text section of a function) to which the relocation(s) apply. It is
116also possible for a livepatch module to have no livepatch relocation
117sections, as in the case of the sample livepatch module (see
118samples/livepatch).
119
120Since Elf information is preserved for livepatch modules (see Section 4), a
121livepatch relocation section can be applied simply by passing in the
122appropriate section index to apply_relocate_add(), which then uses it to
123access the relocation section and apply the relocations.
124
125Every symbol referenced by a rela in a livepatch relocation section is a
126livepatch symbol. These must be resolved before livepatch can call
127apply_relocate_add(). See Section 3 for more information.
128
129---------------------------------------
1302.2 Livepatch relocation section format
131---------------------------------------
132
1332.2.1 Required flags
134--------------------
135Livepatch relocation sections must be marked with the SHF_RELA_LIVEPATCH
136section flag. See include/uapi/linux/elf.h for the definition. The module
137loader recognizes this flag and will avoid applying those relocation sections
138at patch module load time. These sections must also be marked with SHF_ALLOC,
139so that the module loader doesn't discard them on module load (i.e. they will
140be copied into memory along with the other SHF_ALLOC sections).
141
1422.2.2 Required name format
143--------------------------
144The name of a livepatch relocation section must conform to the following format:
145
146.klp.rela.objname.section_name
147^ ^^ ^ ^ ^
148|________||_____| |__________|
149 [A] [B] [C]
150
151[A] The relocation section name is prefixed with the string ".klp.rela."
152[B] The name of the object (i.e. "vmlinux" or name of module) to
153 which the relocation section belongs follows immediately after the prefix.
154[C] The actual name of the section to which this relocation section applies.
155
1562.2.3 Example livepatch relocation section names:
157-------------------------------------------------
158.klp.rela.ext4.text.ext4_attr_store
159.klp.rela.vmlinux.text.cmdline_proc_show
160
1612.2.4 Example `readelf --sections` output for a patch
162module that patches vmlinux and modules 9p, btrfs, ext4:
163--------------------------------------------------------
164 Section Headers:
165 [Nr] Name Type Address Off Size ES Flg Lk Inf Al
166 [ snip ]
167 [29] .klp.rela.9p.text.caches.show RELA 0000000000000000 002d58 0000c0 18 AIo 64 9 8
168 [30] .klp.rela.btrfs.text.btrfs.feature.attr.show RELA 0000000000000000 002e18 000060 18 AIo 64 11 8
169 [ snip ]
170 [34] .klp.rela.ext4.text.ext4.attr.store RELA 0000000000000000 002fd8 0000d8 18 AIo 64 13 8
171 [35] .klp.rela.ext4.text.ext4.attr.show RELA 0000000000000000 0030b0 000150 18 AIo 64 15 8
172 [36] .klp.rela.vmlinux.text.cmdline.proc.show RELA 0000000000000000 003200 000018 18 AIo 64 17 8
173 [37] .klp.rela.vmlinux.text.meminfo.proc.show RELA 0000000000000000 003218 0000f0 18 AIo 64 19 8
174 [ snip ] ^ ^
175 | |
176 [*] [*]
177[*] Livepatch relocation sections are SHT_RELA sections but with a few special
178characteristics. Notice that they are marked SHF_ALLOC ("A") so that they will
179not be discarded when the module is loaded into memory, as well as with the
180SHF_RELA_LIVEPATCH flag ("o" - for OS-specific).
181
1822.2.5 Example `readelf --relocs` output for a patch module:
183-----------------------------------------------------------
184Relocation section '.klp.rela.btrfs.text.btrfs_feature_attr_show' at offset 0x2ba0 contains 4 entries:
185 Offset Info Type Symbol's Value Symbol's Name + Addend
186000000000000001f 0000005e00000002 R_X86_64_PC32 0000000000000000 .klp.sym.vmlinux.printk,0 - 4
1870000000000000028 0000003d0000000b R_X86_64_32S 0000000000000000 .klp.sym.btrfs.btrfs_ktype,0 + 0
1880000000000000036 0000003b00000002 R_X86_64_PC32 0000000000000000 .klp.sym.btrfs.can_modify_feature.isra.3,0 - 4
189000000000000004c 0000004900000002 R_X86_64_PC32 0000000000000000 .klp.sym.vmlinux.snprintf,0 - 4
190[ snip ] ^
191 |
192 [*]
193[*] Every symbol referenced by a relocation is a livepatch symbol.
194
195--------------------
1963. Livepatch symbols
197--------------------
198
199-------------------------------
2003.1 What are livepatch symbols?
201-------------------------------
202Livepatch symbols are symbols referred to by livepatch relocation sections.
203These are symbols accessed from new versions of functions for patched
204objects, whose addresses cannot be resolved by the module loader (because
205they are local or unexported global syms). Since the module loader only
206resolves exported syms, and not every symbol referenced by the new patched
207functions is exported, livepatch symbols were introduced. They are used
208also in cases where we cannot immediately know the address of a symbol when
209a patch module loads. For example, this is the case when livepatch patches
210a module that is not loaded yet. In this case, the relevant livepatch
211symbols are resolved simply when the target module loads. In any case, for
212any livepatch relocation section, all livepatch symbols referenced by that
213section must be resolved before livepatch can call apply_relocate_add() for
214that reloc section.
215
216Livepatch symbols must be marked with SHN_LIVEPATCH so that the module
217loader can identify and ignore them. Livepatch modules keep these symbols
218in their symbol tables, and the symbol table is made accessible through
219module->symtab.
220
221-------------------------------------
2223.2 A livepatch module's symbol table
223-------------------------------------
224Normally, a stripped down copy of a module's symbol table (containing only
225"core" symbols) is made available through module->symtab (See layout_symtab()
226in kernel/module.c). For livepatch modules, the symbol table copied into memory
227on module load must be exactly the same as the symbol table produced when the
228patch module was compiled. This is because the relocations in each livepatch
229relocation section refer to their respective symbols with their symbol indices,
230and the original symbol indices (and thus the symtab ordering) must be
231preserved in order for apply_relocate_add() to find the right symbol.
232
233For example, take this particular rela from a livepatch module:
234Relocation section '.klp.rela.btrfs.text.btrfs_feature_attr_show' at offset 0x2ba0 contains 4 entries:
235 Offset Info Type Symbol's Value Symbol's Name + Addend
236000000000000001f 0000005e00000002 R_X86_64_PC32 0000000000000000 .klp.sym.vmlinux.printk,0 - 4
237
238This rela refers to the symbol '.klp.sym.vmlinux.printk,0', and the symbol index is encoded
239in 'Info'. Here its symbol index is 0x5e, which is 94 in decimal, which refers to the
240symbol index 94.
241And in this patch module's corresponding symbol table, symbol index 94 refers to that very symbol:
242[ snip ]
24394: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.printk,0
244[ snip ]
245
246---------------------------
2473.3 Livepatch symbol format
248---------------------------
249
2503.3.1 Required flags
251--------------------
252Livepatch symbols must have their section index marked as SHN_LIVEPATCH, so
253that the module loader can identify them and not attempt to resolve them.
254See include/uapi/linux/elf.h for the actual definitions.
255
2563.3.2 Required name format
257--------------------------
258Livepatch symbol names must conform to the following format:
259
260.klp.sym.objname.symbol_name,sympos
261^ ^^ ^ ^ ^ ^
262|_______||_____| |_________| |
263 [A] [B] [C] [D]
264
265[A] The symbol name is prefixed with the string ".klp.sym."
266[B] The name of the object (i.e. "vmlinux" or name of module) to
267 which the symbol belongs follows immediately after the prefix.
268[C] The actual name of the symbol.
269[D] The position of the symbol in the object (as according to kallsyms)
270 This is used to differentiate duplicate symbols within the same
271 object. The symbol position is expressed numerically (0, 1, 2...).
272 The symbol position of a unique symbol is 0.
273
2743.3.3 Example livepatch symbol names:
275-------------------------------------
276.klp.sym.vmlinux.snprintf,0
277.klp.sym.vmlinux.printk,0
278.klp.sym.btrfs.btrfs_ktype,0
279
2803.3.4 Example `readelf --symbols` output for a patch module:
281------------------------------------------------------------
282Symbol table '.symtab' contains 127 entries:
283 Num: Value Size Type Bind Vis Ndx Name
284 [ snip ]
285 73: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.snprintf,0
286 74: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.capable,0
287 75: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.find_next_bit,0
288 76: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.si_swapinfo,0
289 [ snip ] ^
290 |
291 [*]
292[*] Note that the 'Ndx' (Section index) for these symbols is SHN_LIVEPATCH (0xff20).
293 "OS" means OS-specific.
294
295--------------------------------------
2964. Symbol table and Elf section access
297--------------------------------------
298A livepatch module's symbol table is accessible through module->symtab.
299
300Since apply_relocate_add() requires access to a module's section headers,
301symbol table, and relocation section indices, Elf information is preserved for
302livepatch modules and is made accessible by the module loader through
303module->klp_info, which is a klp_modinfo struct. When a livepatch module loads,
304this struct is filled in by the module loader. Its fields are documented below:
305
306struct klp_modinfo {
307 Elf_Ehdr hdr; /* Elf header */
308 Elf_Shdr *sechdrs; /* Section header table */
309 char *secstrings; /* String table for the section headers */
310 unsigned int symndx; /* The symbol table section index */
311};
diff --git a/MAINTAINERS b/MAINTAINERS
index 6210ae21d311..1dd9335de071 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6699,6 +6699,7 @@ F: kernel/livepatch/
6699F: include/linux/livepatch.h 6699F: include/linux/livepatch.h
6700F: arch/x86/include/asm/livepatch.h 6700F: arch/x86/include/asm/livepatch.h
6701F: arch/x86/kernel/livepatch.c 6701F: arch/x86/kernel/livepatch.c
6702F: Documentation/livepatch/
6702F: Documentation/ABI/testing/sysfs-kernel-livepatch 6703F: Documentation/ABI/testing/sysfs-kernel-livepatch
6703F: samples/livepatch/ 6704F: samples/livepatch/
6704L: live-patching@vger.kernel.org 6705L: live-patching@vger.kernel.org
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 2fdb73d9198a..a18a0dcd57b7 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -160,6 +160,7 @@ config PPC
160 select HAVE_ARCH_SECCOMP_FILTER 160 select HAVE_ARCH_SECCOMP_FILTER
161 select ARCH_HAS_UBSAN_SANITIZE_ALL 161 select ARCH_HAS_UBSAN_SANITIZE_ALL
162 select ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT 162 select ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT
163 select HAVE_LIVEPATCH if HAVE_DYNAMIC_FTRACE_WITH_REGS
163 164
164config GENERIC_CSUM 165config GENERIC_CSUM
165 def_bool CPU_LITTLE_ENDIAN 166 def_bool CPU_LITTLE_ENDIAN
@@ -1107,3 +1108,5 @@ config PPC_LIB_RHEAP
1107 bool 1108 bool
1108 1109
1109source "arch/powerpc/kvm/Kconfig" 1110source "arch/powerpc/kvm/Kconfig"
1111
1112source "kernel/livepatch/Kconfig"
diff --git a/arch/powerpc/include/asm/livepatch.h b/arch/powerpc/include/asm/livepatch.h
new file mode 100644
index 000000000000..a402f7f94896
--- /dev/null
+++ b/arch/powerpc/include/asm/livepatch.h
@@ -0,0 +1,62 @@
1/*
2 * livepatch.h - powerpc-specific Kernel Live Patching Core
3 *
4 * Copyright (C) 2015-2016, SUSE, IBM Corp.
5 *
6 * This program is free software; you can redistribute it and/or
7 * modify it under the terms of the GNU General Public License
8 * as published by the Free Software Foundation; either version 2
9 * of the License, or (at your option) any later version.
10 *
11 * This program is distributed in the hope that it will be useful,
12 * but WITHOUT ANY WARRANTY; without even the implied warranty of
13 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 * GNU General Public License for more details.
15 *
16 * You should have received a copy of the GNU General Public License
17 * along with this program; if not, see <http://www.gnu.org/licenses/>.
18 */
19#ifndef _ASM_POWERPC_LIVEPATCH_H
20#define _ASM_POWERPC_LIVEPATCH_H
21
22#include <linux/module.h>
23#include <linux/ftrace.h>
24
25#ifdef CONFIG_LIVEPATCH
26static inline int klp_check_compiler_support(void)
27{
28 return 0;
29}
30
31static inline int klp_write_module_reloc(struct module *mod, unsigned long
32 type, unsigned long loc, unsigned long value)
33{
34 /* This requires infrastructure changes; we need the loadinfos. */
35 return -ENOSYS;
36}
37
38static inline void klp_arch_set_pc(struct pt_regs *regs, unsigned long ip)
39{
40 regs->nip = ip;
41}
42
43#define klp_get_ftrace_location klp_get_ftrace_location
44static inline unsigned long klp_get_ftrace_location(unsigned long faddr)
45{
46 /*
47 * Live patch works only with -mprofile-kernel on PPC. In this case,
48 * the ftrace location is always within the first 16 bytes.
49 */
50 return ftrace_location_range(faddr, faddr + 16);
51}
52
53static inline void klp_init_thread_info(struct thread_info *ti)
54{
55 /* + 1 to account for STACK_END_MAGIC */
56 ti->livepatch_sp = (unsigned long *)(ti + 1) + 1;
57}
58#else
59static void klp_init_thread_info(struct thread_info *ti) { }
60#endif /* CONFIG_LIVEPATCH */
61
62#endif /* _ASM_POWERPC_LIVEPATCH_H */
diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h
index 7efee4a3240b..8febc3f66d53 100644
--- a/arch/powerpc/include/asm/thread_info.h
+++ b/arch/powerpc/include/asm/thread_info.h
@@ -43,7 +43,9 @@ struct thread_info {
43 int preempt_count; /* 0 => preemptable, 43 int preempt_count; /* 0 => preemptable,
44 <0 => BUG */ 44 <0 => BUG */
45 unsigned long local_flags; /* private flags for thread */ 45 unsigned long local_flags; /* private flags for thread */
46 46#ifdef CONFIG_LIVEPATCH
47 unsigned long *livepatch_sp;
48#endif
47 /* low level flags - has atomic operations done on it */ 49 /* low level flags - has atomic operations done on it */
48 unsigned long flags ____cacheline_aligned_in_smp; 50 unsigned long flags ____cacheline_aligned_in_smp;
49}; 51};
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index 0d0183d3180a..c9370d4e36bd 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -86,6 +86,10 @@ int main(void)
86 DEFINE(KSP_LIMIT, offsetof(struct thread_struct, ksp_limit)); 86 DEFINE(KSP_LIMIT, offsetof(struct thread_struct, ksp_limit));
87#endif /* CONFIG_PPC64 */ 87#endif /* CONFIG_PPC64 */
88 88
89#ifdef CONFIG_LIVEPATCH
90 DEFINE(TI_livepatch_sp, offsetof(struct thread_info, livepatch_sp));
91#endif
92
89 DEFINE(KSP, offsetof(struct thread_struct, ksp)); 93 DEFINE(KSP, offsetof(struct thread_struct, ksp));
90 DEFINE(PT_REGS, offsetof(struct thread_struct, regs)); 94 DEFINE(PT_REGS, offsetof(struct thread_struct, regs));
91#ifdef CONFIG_BOOKE 95#ifdef CONFIG_BOOKE
diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
index 9916d150b28c..39a79c89a4b6 100644
--- a/arch/powerpc/kernel/entry_64.S
+++ b/arch/powerpc/kernel/entry_64.S
@@ -20,6 +20,7 @@
20 20
21#include <linux/errno.h> 21#include <linux/errno.h>
22#include <linux/err.h> 22#include <linux/err.h>
23#include <linux/magic.h>
23#include <asm/unistd.h> 24#include <asm/unistd.h>
24#include <asm/processor.h> 25#include <asm/processor.h>
25#include <asm/page.h> 26#include <asm/page.h>
@@ -1248,6 +1249,9 @@ _GLOBAL(ftrace_caller)
1248 addi r3,r3,function_trace_op@toc@l 1249 addi r3,r3,function_trace_op@toc@l
1249 ld r5,0(r3) 1250 ld r5,0(r3)
1250 1251
1252#ifdef CONFIG_LIVEPATCH
1253 mr r14,r7 /* remember old NIP */
1254#endif
1251 /* Calculate ip from nip-4 into r3 for call below */ 1255 /* Calculate ip from nip-4 into r3 for call below */
1252 subi r3, r7, MCOUNT_INSN_SIZE 1256 subi r3, r7, MCOUNT_INSN_SIZE
1253 1257
@@ -1272,6 +1276,9 @@ ftrace_call:
1272 /* Load ctr with the possibly modified NIP */ 1276 /* Load ctr with the possibly modified NIP */
1273 ld r3, _NIP(r1) 1277 ld r3, _NIP(r1)
1274 mtctr r3 1278 mtctr r3
1279#ifdef CONFIG_LIVEPATCH
1280 cmpd r14,r3 /* has NIP been altered? */
1281#endif
1275 1282
1276 /* Restore gprs */ 1283 /* Restore gprs */
1277 REST_8GPRS(0,r1) 1284 REST_8GPRS(0,r1)
@@ -1289,6 +1296,11 @@ ftrace_call:
1289 ld r0, LRSAVE(r1) 1296 ld r0, LRSAVE(r1)
1290 mtlr r0 1297 mtlr r0
1291 1298
1299#ifdef CONFIG_LIVEPATCH
1300 /* Based on the cmpd above, if the NIP was altered handle livepatch */
1301 bne- livepatch_handler
1302#endif
1303
1292#ifdef CONFIG_FUNCTION_GRAPH_TRACER 1304#ifdef CONFIG_FUNCTION_GRAPH_TRACER
1293 stdu r1, -112(r1) 1305 stdu r1, -112(r1)
1294.globl ftrace_graph_call 1306.globl ftrace_graph_call
@@ -1305,6 +1317,91 @@ _GLOBAL(ftrace_graph_stub)
1305 1317
1306_GLOBAL(ftrace_stub) 1318_GLOBAL(ftrace_stub)
1307 blr 1319 blr
1320
1321#ifdef CONFIG_LIVEPATCH
1322 /*
1323 * This function runs in the mcount context, between two functions. As
1324 * such it can only clobber registers which are volatile and used in
1325 * function linkage.
1326 *
1327 * We get here when a function A, calls another function B, but B has
1328 * been live patched with a new function C.
1329 *
1330 * On entry:
1331 * - we have no stack frame and can not allocate one
1332 * - LR points back to the original caller (in A)
1333 * - CTR holds the new NIP in C
1334 * - r0 & r12 are free
1335 *
1336 * r0 can't be used as the base register for a DS-form load or store, so
1337 * we temporarily shuffle r1 (stack pointer) into r0 and then put it back.
1338 */
1339livepatch_handler:
1340 CURRENT_THREAD_INFO(r12, r1)
1341
1342 /* Save stack pointer into r0 */
1343 mr r0, r1
1344
1345 /* Allocate 3 x 8 bytes */
1346 ld r1, TI_livepatch_sp(r12)
1347 addi r1, r1, 24
1348 std r1, TI_livepatch_sp(r12)
1349
1350 /* Save toc & real LR on livepatch stack */
1351 std r2, -24(r1)
1352 mflr r12
1353 std r12, -16(r1)
1354
1355 /* Store stack end marker */
1356 lis r12, STACK_END_MAGIC@h
1357 ori r12, r12, STACK_END_MAGIC@l
1358 std r12, -8(r1)
1359
1360 /* Restore real stack pointer */
1361 mr r1, r0
1362
1363 /* Put ctr in r12 for global entry and branch there */
1364 mfctr r12
1365 bctrl
1366
1367 /*
1368 * Now we are returning from the patched function to the original
1369 * caller A. We are free to use r0 and r12, and we can use r2 until we
1370 * restore it.
1371 */
1372
1373 CURRENT_THREAD_INFO(r12, r1)
1374
1375 /* Save stack pointer into r0 */
1376 mr r0, r1
1377
1378 ld r1, TI_livepatch_sp(r12)
1379
1380 /* Check stack marker hasn't been trashed */
1381 lis r2, STACK_END_MAGIC@h
1382 ori r2, r2, STACK_END_MAGIC@l
1383 ld r12, -8(r1)
13841: tdne r12, r2
1385 EMIT_BUG_ENTRY 1b, __FILE__, __LINE__ - 1, 0
1386
1387 /* Restore LR & toc from livepatch stack */
1388 ld r12, -16(r1)
1389 mtlr r12
1390 ld r2, -24(r1)
1391
1392 /* Pop livepatch stack frame */
1393 CURRENT_THREAD_INFO(r12, r0)
1394 subi r1, r1, 24
1395 std r1, TI_livepatch_sp(r12)
1396
1397 /* Restore real stack pointer */
1398 mr r1, r0
1399
1400 /* Return to original caller of live patched function */
1401 blr
1402#endif
1403
1404
1308#else 1405#else
1309_GLOBAL_TOC(_mcount) 1406_GLOBAL_TOC(_mcount)
1310 /* Taken from output of objdump from lib64/glibc */ 1407 /* Taken from output of objdump from lib64/glibc */
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index 290559df1e8b..3cb46a3b1de7 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -66,6 +66,7 @@
66#include <asm/udbg.h> 66#include <asm/udbg.h>
67#include <asm/smp.h> 67#include <asm/smp.h>
68#include <asm/debug.h> 68#include <asm/debug.h>
69#include <asm/livepatch.h>
69 70
70#ifdef CONFIG_PPC64 71#ifdef CONFIG_PPC64
71#include <asm/paca.h> 72#include <asm/paca.h>
@@ -607,10 +608,12 @@ void irq_ctx_init(void)
607 memset((void *)softirq_ctx[i], 0, THREAD_SIZE); 608 memset((void *)softirq_ctx[i], 0, THREAD_SIZE);
608 tp = softirq_ctx[i]; 609 tp = softirq_ctx[i];
609 tp->cpu = i; 610 tp->cpu = i;
611 klp_init_thread_info(tp);
610 612
611 memset((void *)hardirq_ctx[i], 0, THREAD_SIZE); 613 memset((void *)hardirq_ctx[i], 0, THREAD_SIZE);
612 tp = hardirq_ctx[i]; 614 tp = hardirq_ctx[i];
613 tp->cpu = i; 615 tp->cpu = i;
616 klp_init_thread_info(tp);
614 } 617 }
615} 618}
616 619
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index b8500b4ac7fe..2a9280b945e0 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -55,6 +55,8 @@
55#include <asm/firmware.h> 55#include <asm/firmware.h>
56#endif 56#endif
57#include <asm/code-patching.h> 57#include <asm/code-patching.h>
58#include <asm/livepatch.h>
59
58#include <linux/kprobes.h> 60#include <linux/kprobes.h>
59#include <linux/kdebug.h> 61#include <linux/kdebug.h>
60 62
@@ -1400,13 +1402,15 @@ int copy_thread(unsigned long clone_flags, unsigned long usp,
1400 extern void ret_from_kernel_thread(void); 1402 extern void ret_from_kernel_thread(void);
1401 void (*f)(void); 1403 void (*f)(void);
1402 unsigned long sp = (unsigned long)task_stack_page(p) + THREAD_SIZE; 1404 unsigned long sp = (unsigned long)task_stack_page(p) + THREAD_SIZE;
1405 struct thread_info *ti = task_thread_info(p);
1406
1407 klp_init_thread_info(ti);
1403 1408
1404 /* Copy registers */ 1409 /* Copy registers */
1405 sp -= sizeof(struct pt_regs); 1410 sp -= sizeof(struct pt_regs);
1406 childregs = (struct pt_regs *) sp; 1411 childregs = (struct pt_regs *) sp;
1407 if (unlikely(p->flags & PF_KTHREAD)) { 1412 if (unlikely(p->flags & PF_KTHREAD)) {
1408 /* kernel thread */ 1413 /* kernel thread */
1409 struct thread_info *ti = (void *)task_stack_page(p);
1410 memset(childregs, 0, sizeof(struct pt_regs)); 1414 memset(childregs, 0, sizeof(struct pt_regs));
1411 childregs->gpr[1] = sp + sizeof(struct pt_regs); 1415 childregs->gpr[1] = sp + sizeof(struct pt_regs);
1412 /* function */ 1416 /* function */
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index f98be8383a39..96d4a2b23d0f 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -69,6 +69,7 @@
69#include <asm/kvm_ppc.h> 69#include <asm/kvm_ppc.h>
70#include <asm/hugetlb.h> 70#include <asm/hugetlb.h>
71#include <asm/epapr_hcalls.h> 71#include <asm/epapr_hcalls.h>
72#include <asm/livepatch.h>
72 73
73#ifdef DEBUG 74#ifdef DEBUG
74#define DBG(fmt...) udbg_printf(fmt) 75#define DBG(fmt...) udbg_printf(fmt)
@@ -667,16 +668,16 @@ static void __init emergency_stack_init(void)
667 limit = min(safe_stack_limit(), ppc64_rma_size); 668 limit = min(safe_stack_limit(), ppc64_rma_size);
668 669
669 for_each_possible_cpu(i) { 670 for_each_possible_cpu(i) {
670 unsigned long sp; 671 struct thread_info *ti;
671 sp = memblock_alloc_base(THREAD_SIZE, THREAD_SIZE, limit); 672 ti = __va(memblock_alloc_base(THREAD_SIZE, THREAD_SIZE, limit));
672 sp += THREAD_SIZE; 673 klp_init_thread_info(ti);
673 paca[i].emergency_sp = __va(sp); 674 paca[i].emergency_sp = (void *)ti + THREAD_SIZE;
674 675
675#ifdef CONFIG_PPC_BOOK3S_64 676#ifdef CONFIG_PPC_BOOK3S_64
676 /* emergency stack for machine check exception handling. */ 677 /* emergency stack for machine check exception handling. */
677 sp = memblock_alloc_base(THREAD_SIZE, THREAD_SIZE, limit); 678 ti = __va(memblock_alloc_base(THREAD_SIZE, THREAD_SIZE, limit));
678 sp += THREAD_SIZE; 679 klp_init_thread_info(ti);
679 paca[i].mc_emergency_sp = __va(sp); 680 paca[i].mc_emergency_sp = (void *)ti + THREAD_SIZE;
680#endif 681#endif
681 } 682 }
682} 683}
@@ -700,6 +701,8 @@ void __init setup_arch(char **cmdline_p)
700 if (ppc_md.panic) 701 if (ppc_md.panic)
701 setup_panic(); 702 setup_panic();
702 703
704 klp_init_thread_info(&init_thread_info);
705
703 init_mm.start_code = (unsigned long)_stext; 706 init_mm.start_code = (unsigned long)_stext;
704 init_mm.end_code = (unsigned long) _etext; 707 init_mm.end_code = (unsigned long) _etext;
705 init_mm.end_data = (unsigned long) _edata; 708 init_mm.end_data = (unsigned long) _edata;
diff --git a/arch/s390/include/asm/livepatch.h b/arch/s390/include/asm/livepatch.h
index d5427c78b1b3..2c1213785892 100644
--- a/arch/s390/include/asm/livepatch.h
+++ b/arch/s390/include/asm/livepatch.h
@@ -24,13 +24,6 @@ static inline int klp_check_compiler_support(void)
24 return 0; 24 return 0;
25} 25}
26 26
27static inline int klp_write_module_reloc(struct module *mod, unsigned long
28 type, unsigned long loc, unsigned long value)
29{
30 /* not supported yet */
31 return -ENOSYS;
32}
33
34static inline void klp_arch_set_pc(struct pt_regs *regs, unsigned long ip) 27static inline void klp_arch_set_pc(struct pt_regs *regs, unsigned long ip)
35{ 28{
36 regs->psw.addr = ip; 29 regs->psw.addr = ip;
diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c
index 7873e171457c..fbc07891f9e7 100644
--- a/arch/s390/kernel/module.c
+++ b/arch/s390/kernel/module.c
@@ -51,6 +51,10 @@ void *module_alloc(unsigned long size)
51 51
52void module_arch_freeing_init(struct module *mod) 52void module_arch_freeing_init(struct module *mod)
53{ 53{
54 if (is_livepatch_module(mod) &&
55 mod->state == MODULE_STATE_LIVE)
56 return;
57
54 vfree(mod->arch.syminfo); 58 vfree(mod->arch.syminfo);
55 mod->arch.syminfo = NULL; 59 mod->arch.syminfo = NULL;
56} 60}
@@ -425,7 +429,5 @@ int module_finalize(const Elf_Ehdr *hdr,
425 struct module *me) 429 struct module *me)
426{ 430{
427 jump_label_apply_nops(me); 431 jump_label_apply_nops(me);
428 vfree(me->arch.syminfo);
429 me->arch.syminfo = NULL;
430 return 0; 432 return 0;
431} 433}
diff --git a/arch/x86/include/asm/livepatch.h b/arch/x86/include/asm/livepatch.h
index 7e68f9558552..a7f9181f63f3 100644
--- a/arch/x86/include/asm/livepatch.h
+++ b/arch/x86/include/asm/livepatch.h
@@ -32,8 +32,6 @@ static inline int klp_check_compiler_support(void)
32#endif 32#endif
33 return 0; 33 return 0;
34} 34}
35int klp_write_module_reloc(struct module *mod, unsigned long type,
36 unsigned long loc, unsigned long value);
37 35
38static inline void klp_arch_set_pc(struct pt_regs *regs, unsigned long ip) 36static inline void klp_arch_set_pc(struct pt_regs *regs, unsigned long ip)
39{ 37{
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 9abf8551c7e4..0503f5bfb18d 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -83,7 +83,6 @@ obj-$(CONFIG_X86_MPPARSE) += mpparse.o
83obj-y += apic/ 83obj-y += apic/
84obj-$(CONFIG_X86_REBOOTFIXUPS) += reboot_fixups_32.o 84obj-$(CONFIG_X86_REBOOTFIXUPS) += reboot_fixups_32.o
85obj-$(CONFIG_DYNAMIC_FTRACE) += ftrace.o 85obj-$(CONFIG_DYNAMIC_FTRACE) += ftrace.o
86obj-$(CONFIG_LIVEPATCH) += livepatch.o
87obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o 86obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o
88obj-$(CONFIG_FTRACE_SYSCALLS) += ftrace.o 87obj-$(CONFIG_FTRACE_SYSCALLS) += ftrace.o
89obj-$(CONFIG_X86_TSC) += trace_clock.o 88obj-$(CONFIG_X86_TSC) += trace_clock.o
diff --git a/arch/x86/kernel/livepatch.c b/arch/x86/kernel/livepatch.c
deleted file mode 100644
index 92fc1a51f994..000000000000
--- a/arch/x86/kernel/livepatch.c
+++ /dev/null
@@ -1,70 +0,0 @@
1/*
2 * livepatch.c - x86-specific Kernel Live Patching Core
3 *
4 * Copyright (C) 2014 Seth Jennings <sjenning@redhat.com>
5 * Copyright (C) 2014 SUSE
6 *
7 * This program is free software; you can redistribute it and/or
8 * modify it under the terms of the GNU General Public License
9 * as published by the Free Software Foundation; either version 2
10 * of the License, or (at your option) any later version.
11 *
12 * This program is distributed in the hope that it will be useful,
13 * but WITHOUT ANY WARRANTY; without even the implied warranty of
14 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15 * GNU General Public License for more details.
16 *
17 * You should have received a copy of the GNU General Public License
18 * along with this program; if not, see <http://www.gnu.org/licenses/>.
19 */
20
21#include <linux/module.h>
22#include <linux/uaccess.h>
23#include <asm/elf.h>
24#include <asm/livepatch.h>
25
26/**
27 * klp_write_module_reloc() - write a relocation in a module
28 * @mod: module in which the section to be modified is found
29 * @type: ELF relocation type (see asm/elf.h)
30 * @loc: address that the relocation should be written to
31 * @value: relocation value (sym address + addend)
32 *
33 * This function writes a relocation to the specified location for
34 * a particular module.
35 */
36int klp_write_module_reloc(struct module *mod, unsigned long type,
37 unsigned long loc, unsigned long value)
38{
39 size_t size = 4;
40 unsigned long val;
41 unsigned long core = (unsigned long)mod->core_layout.base;
42 unsigned long core_size = mod->core_layout.size;
43
44 switch (type) {
45 case R_X86_64_NONE:
46 return 0;
47 case R_X86_64_64:
48 val = value;
49 size = 8;
50 break;
51 case R_X86_64_32:
52 val = (u32)value;
53 break;
54 case R_X86_64_32S:
55 val = (s32)value;
56 break;
57 case R_X86_64_PC32:
58 val = (u32)(value - loc);
59 break;
60 default:
61 /* unsupported relocation type */
62 return -EINVAL;
63 }
64
65 if (loc < core || loc >= core + core_size)
66 /* loc does not point to any symbol inside the module */
67 return -EINVAL;
68
69 return probe_kernel_write((void *)loc, &val, size);
70}
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index dea12a6e413b..66a36a815f0a 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -455,6 +455,7 @@ int ftrace_update_record(struct dyn_ftrace *rec, int enable);
455int ftrace_test_record(struct dyn_ftrace *rec, int enable); 455int ftrace_test_record(struct dyn_ftrace *rec, int enable);
456void ftrace_run_stop_machine(int command); 456void ftrace_run_stop_machine(int command);
457unsigned long ftrace_location(unsigned long ip); 457unsigned long ftrace_location(unsigned long ip);
458unsigned long ftrace_location_range(unsigned long start, unsigned long end);
458unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec); 459unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec);
459unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec); 460unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec);
460 461
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
index bd830d590465..a93a0b23dc8d 100644
--- a/include/linux/livepatch.h
+++ b/include/linux/livepatch.h
@@ -65,27 +65,8 @@ struct klp_func {
65}; 65};
66 66
67/** 67/**
68 * struct klp_reloc - relocation structure for live patching
69 * @loc: address where the relocation will be written
70 * @sympos: position in kallsyms to disambiguate symbols (optional)
71 * @type: ELF relocation type
72 * @name: name of the referenced symbol (for lookup/verification)
73 * @addend: offset from the referenced symbol
74 * @external: symbol is either exported or within the live patch module itself
75 */
76struct klp_reloc {
77 unsigned long loc;
78 unsigned long sympos;
79 unsigned long type;
80 const char *name;
81 int addend;
82 int external;
83};
84
85/**
86 * struct klp_object - kernel object structure for live patching 68 * struct klp_object - kernel object structure for live patching
87 * @name: module name (or NULL for vmlinux) 69 * @name: module name (or NULL for vmlinux)
88 * @relocs: relocation entries to be applied at load time
89 * @funcs: function entries for functions to be patched in the object 70 * @funcs: function entries for functions to be patched in the object
90 * @kobj: kobject for sysfs resources 71 * @kobj: kobject for sysfs resources
91 * @mod: kernel module associated with the patched object 72 * @mod: kernel module associated with the patched object
@@ -95,7 +76,6 @@ struct klp_reloc {
95struct klp_object { 76struct klp_object {
96 /* external */ 77 /* external */
97 const char *name; 78 const char *name;
98 struct klp_reloc *relocs;
99 struct klp_func *funcs; 79 struct klp_func *funcs;
100 80
101 /* internal */ 81 /* internal */
@@ -124,10 +104,12 @@ struct klp_patch {
124}; 104};
125 105
126#define klp_for_each_object(patch, obj) \ 106#define klp_for_each_object(patch, obj) \
127 for (obj = patch->objs; obj->funcs; obj++) 107 for (obj = patch->objs; obj->funcs || obj->name; obj++)
128 108
129#define klp_for_each_func(obj, func) \ 109#define klp_for_each_func(obj, func) \
130 for (func = obj->funcs; func->old_name; func++) 110 for (func = obj->funcs; \
111 func->old_name || func->new_func || func->old_sympos; \
112 func++)
131 113
132int klp_register_patch(struct klp_patch *); 114int klp_register_patch(struct klp_patch *);
133int klp_unregister_patch(struct klp_patch *); 115int klp_unregister_patch(struct klp_patch *);
diff --git a/include/linux/module.h b/include/linux/module.h
index 2bb0c3085706..3daf2b3a09d2 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -330,6 +330,15 @@ struct mod_kallsyms {
330 char *strtab; 330 char *strtab;
331}; 331};
332 332
333#ifdef CONFIG_LIVEPATCH
334struct klp_modinfo {
335 Elf_Ehdr hdr;
336 Elf_Shdr *sechdrs;
337 char *secstrings;
338 unsigned int symndx;
339};
340#endif
341
333struct module { 342struct module {
334 enum module_state state; 343 enum module_state state;
335 344
@@ -456,7 +465,11 @@ struct module {
456#endif 465#endif
457 466
458#ifdef CONFIG_LIVEPATCH 467#ifdef CONFIG_LIVEPATCH
468 bool klp; /* Is this a livepatch module? */
459 bool klp_alive; 469 bool klp_alive;
470
471 /* Elf information */
472 struct klp_modinfo *klp_info;
460#endif 473#endif
461 474
462#ifdef CONFIG_MODULE_UNLOAD 475#ifdef CONFIG_MODULE_UNLOAD
@@ -630,6 +643,18 @@ static inline bool module_requested_async_probing(struct module *module)
630 return module && module->async_probe_requested; 643 return module && module->async_probe_requested;
631} 644}
632 645
646#ifdef CONFIG_LIVEPATCH
647static inline bool is_livepatch_module(struct module *mod)
648{
649 return mod->klp;
650}
651#else /* !CONFIG_LIVEPATCH */
652static inline bool is_livepatch_module(struct module *mod)
653{
654 return false;
655}
656#endif /* CONFIG_LIVEPATCH */
657
633#else /* !CONFIG_MODULES... */ 658#else /* !CONFIG_MODULES... */
634 659
635/* Given an address, look for it in the exception tables. */ 660/* Given an address, look for it in the exception tables. */
diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
index 71e1d0ed92f7..cb4a72f888d5 100644
--- a/include/uapi/linux/elf.h
+++ b/include/uapi/linux/elf.h
@@ -282,16 +282,18 @@ typedef struct elf64_phdr {
282#define SHT_HIUSER 0xffffffff 282#define SHT_HIUSER 0xffffffff
283 283
284/* sh_flags */ 284/* sh_flags */
285#define SHF_WRITE 0x1 285#define SHF_WRITE 0x1
286#define SHF_ALLOC 0x2 286#define SHF_ALLOC 0x2
287#define SHF_EXECINSTR 0x4 287#define SHF_EXECINSTR 0x4
288#define SHF_MASKPROC 0xf0000000 288#define SHF_RELA_LIVEPATCH 0x00100000
289#define SHF_MASKPROC 0xf0000000
289 290
290/* special section indexes */ 291/* special section indexes */
291#define SHN_UNDEF 0 292#define SHN_UNDEF 0
292#define SHN_LORESERVE 0xff00 293#define SHN_LORESERVE 0xff00
293#define SHN_LOPROC 0xff00 294#define SHN_LOPROC 0xff00
294#define SHN_HIPROC 0xff1f 295#define SHN_HIPROC 0xff1f
296#define SHN_LIVEPATCH 0xff20
295#define SHN_ABS 0xfff1 297#define SHN_ABS 0xfff1
296#define SHN_COMMON 0xfff2 298#define SHN_COMMON 0xfff2
297#define SHN_HIRESERVE 0xffff 299#define SHN_HIRESERVE 0xffff
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index d68fbf63b083..5c2bc1052691 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -28,6 +28,8 @@
28#include <linux/list.h> 28#include <linux/list.h>
29#include <linux/kallsyms.h> 29#include <linux/kallsyms.h>
30#include <linux/livepatch.h> 30#include <linux/livepatch.h>
31#include <linux/elf.h>
32#include <linux/moduleloader.h>
31#include <asm/cacheflush.h> 33#include <asm/cacheflush.h>
32 34
33/** 35/**
@@ -204,75 +206,109 @@ static int klp_find_object_symbol(const char *objname, const char *name,
204 return -EINVAL; 206 return -EINVAL;
205} 207}
206 208
207/* 209static int klp_resolve_symbols(Elf_Shdr *relasec, struct module *pmod)
208 * external symbols are located outside the parent object (where the parent
209 * object is either vmlinux or the kmod being patched).
210 */
211static int klp_find_external_symbol(struct module *pmod, const char *name,
212 unsigned long *addr)
213{ 210{
214 const struct kernel_symbol *sym; 211 int i, cnt, vmlinux, ret;
215 212 char objname[MODULE_NAME_LEN];
216 /* first, check if it's an exported symbol */ 213 char symname[KSYM_NAME_LEN];
217 preempt_disable(); 214 char *strtab = pmod->core_kallsyms.strtab;
218 sym = find_symbol(name, NULL, NULL, true, true); 215 Elf_Rela *relas;
219 if (sym) { 216 Elf_Sym *sym;
220 *addr = sym->value; 217 unsigned long sympos, addr;
221 preempt_enable();
222 return 0;
223 }
224 preempt_enable();
225 218
226 /* 219 /*
227 * Check if it's in another .o within the patch module. This also 220 * Since the field widths for objname and symname in the sscanf()
228 * checks that the external symbol is unique. 221 * call are hard-coded and correspond to MODULE_NAME_LEN and
222 * KSYM_NAME_LEN respectively, we must make sure that MODULE_NAME_LEN
223 * and KSYM_NAME_LEN have the values we expect them to have.
224 *
225 * Because the value of MODULE_NAME_LEN can differ among architectures,
226 * we use the smallest/strictest upper bound possible (56, based on
227 * the current definition of MODULE_NAME_LEN) to prevent overflows.
229 */ 228 */
230 return klp_find_object_symbol(pmod->name, name, 0, addr); 229 BUILD_BUG_ON(MODULE_NAME_LEN < 56 || KSYM_NAME_LEN != 128);
230
231 relas = (Elf_Rela *) relasec->sh_addr;
232 /* For each rela in this klp relocation section */
233 for (i = 0; i < relasec->sh_size / sizeof(Elf_Rela); i++) {
234 sym = pmod->core_kallsyms.symtab + ELF_R_SYM(relas[i].r_info);
235 if (sym->st_shndx != SHN_LIVEPATCH) {
236 pr_err("symbol %s is not marked as a livepatch symbol",
237 strtab + sym->st_name);
238 return -EINVAL;
239 }
240
241 /* Format: .klp.sym.objname.symname,sympos */
242 cnt = sscanf(strtab + sym->st_name,
243 ".klp.sym.%55[^.].%127[^,],%lu",
244 objname, symname, &sympos);
245 if (cnt != 3) {
246 pr_err("symbol %s has an incorrectly formatted name",
247 strtab + sym->st_name);
248 return -EINVAL;
249 }
250
251 /* klp_find_object_symbol() treats a NULL objname as vmlinux */
252 vmlinux = !strcmp(objname, "vmlinux");
253 ret = klp_find_object_symbol(vmlinux ? NULL : objname,
254 symname, sympos, &addr);
255 if (ret)
256 return ret;
257
258 sym->st_value = addr;
259 }
260
261 return 0;
231} 262}
232 263
233static int klp_write_object_relocations(struct module *pmod, 264static int klp_write_object_relocations(struct module *pmod,
234 struct klp_object *obj) 265 struct klp_object *obj)
235{ 266{
236 int ret = 0; 267 int i, cnt, ret = 0;
237 unsigned long val; 268 const char *objname, *secname;
238 struct klp_reloc *reloc; 269 char sec_objname[MODULE_NAME_LEN];
270 Elf_Shdr *sec;
239 271
240 if (WARN_ON(!klp_is_object_loaded(obj))) 272 if (WARN_ON(!klp_is_object_loaded(obj)))
241 return -EINVAL; 273 return -EINVAL;
242 274
243 if (WARN_ON(!obj->relocs)) 275 objname = klp_is_module(obj) ? obj->name : "vmlinux";
244 return -EINVAL;
245 276
246 module_disable_ro(pmod); 277 module_disable_ro(pmod);
278 /* For each klp relocation section */
279 for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
280 sec = pmod->klp_info->sechdrs + i;
281 secname = pmod->klp_info->secstrings + sec->sh_name;
282 if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
283 continue;
247 284
248 for (reloc = obj->relocs; reloc->name; reloc++) { 285 /*
249 /* discover the address of the referenced symbol */ 286 * Format: .klp.rela.sec_objname.section_name
250 if (reloc->external) { 287 * See comment in klp_resolve_symbols() for an explanation
251 if (reloc->sympos > 0) { 288 * of the selected field width value.
252 pr_err("non-zero sympos for external reloc symbol '%s' is not supported\n", 289 */
253 reloc->name); 290 cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
254 ret = -EINVAL; 291 if (cnt != 1) {
255 goto out; 292 pr_err("section %s has an incorrectly formatted name",
256 } 293 secname);
257 ret = klp_find_external_symbol(pmod, reloc->name, &val); 294 ret = -EINVAL;
258 } else 295 break;
259 ret = klp_find_object_symbol(obj->name, 296 }
260 reloc->name, 297
261 reloc->sympos, 298 if (strcmp(objname, sec_objname))
262 &val); 299 continue;
300
301 ret = klp_resolve_symbols(sec, pmod);
263 if (ret) 302 if (ret)
264 goto out; 303 break;
265 304
266 ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc, 305 ret = apply_relocate_add(pmod->klp_info->sechdrs,
267 val + reloc->addend); 306 pmod->core_kallsyms.strtab,
268 if (ret) { 307 pmod->klp_info->symndx, i, pmod);
269 pr_err("relocation failed for symbol '%s' at 0x%016lx (%d)\n", 308 if (ret)
270 reloc->name, val, ret); 309 break;
271 goto out;
272 }
273 } 310 }
274 311
275out:
276 module_enable_ro(pmod); 312 module_enable_ro(pmod);
277 return ret; 313 return ret;
278} 314}
@@ -298,6 +334,19 @@ unlock:
298 rcu_read_unlock(); 334 rcu_read_unlock();
299} 335}
300 336
337/*
338 * Convert a function address into the appropriate ftrace location.
339 *
340 * Usually this is just the address of the function, but on some architectures
341 * it's more complicated so allow them to provide a custom behaviour.
342 */
343#ifndef klp_get_ftrace_location
344static unsigned long klp_get_ftrace_location(unsigned long faddr)
345{
346 return faddr;
347}
348#endif
349
301static void klp_disable_func(struct klp_func *func) 350static void klp_disable_func(struct klp_func *func)
302{ 351{
303 struct klp_ops *ops; 352 struct klp_ops *ops;
@@ -312,8 +361,14 @@ static void klp_disable_func(struct klp_func *func)
312 return; 361 return;
313 362
314 if (list_is_singular(&ops->func_stack)) { 363 if (list_is_singular(&ops->func_stack)) {
364 unsigned long ftrace_loc;
365
366 ftrace_loc = klp_get_ftrace_location(func->old_addr);
367 if (WARN_ON(!ftrace_loc))
368 return;
369
315 WARN_ON(unregister_ftrace_function(&ops->fops)); 370 WARN_ON(unregister_ftrace_function(&ops->fops));
316 WARN_ON(ftrace_set_filter_ip(&ops->fops, func->old_addr, 1, 0)); 371 WARN_ON(ftrace_set_filter_ip(&ops->fops, ftrace_loc, 1, 0));
317 372
318 list_del_rcu(&func->stack_node); 373 list_del_rcu(&func->stack_node);
319 list_del(&ops->node); 374 list_del(&ops->node);
@@ -338,6 +393,15 @@ static int klp_enable_func(struct klp_func *func)
338 393
339 ops = klp_find_ops(func->old_addr); 394 ops = klp_find_ops(func->old_addr);
340 if (!ops) { 395 if (!ops) {
396 unsigned long ftrace_loc;
397
398 ftrace_loc = klp_get_ftrace_location(func->old_addr);
399 if (!ftrace_loc) {
400 pr_err("failed to find location for function '%s'\n",
401 func->old_name);
402 return -EINVAL;
403 }
404
341 ops = kzalloc(sizeof(*ops), GFP_KERNEL); 405 ops = kzalloc(sizeof(*ops), GFP_KERNEL);
342 if (!ops) 406 if (!ops)
343 return -ENOMEM; 407 return -ENOMEM;
@@ -352,7 +416,7 @@ static int klp_enable_func(struct klp_func *func)
352 INIT_LIST_HEAD(&ops->func_stack); 416 INIT_LIST_HEAD(&ops->func_stack);
353 list_add_rcu(&func->stack_node, &ops->func_stack); 417 list_add_rcu(&func->stack_node, &ops->func_stack);
354 418
355 ret = ftrace_set_filter_ip(&ops->fops, func->old_addr, 0, 0); 419 ret = ftrace_set_filter_ip(&ops->fops, ftrace_loc, 0, 0);
356 if (ret) { 420 if (ret) {
357 pr_err("failed to set ftrace filter for function '%s' (%d)\n", 421 pr_err("failed to set ftrace filter for function '%s' (%d)\n",
358 func->old_name, ret); 422 func->old_name, ret);
@@ -363,7 +427,7 @@ static int klp_enable_func(struct klp_func *func)
363 if (ret) { 427 if (ret) {
364 pr_err("failed to register ftrace handler for function '%s' (%d)\n", 428 pr_err("failed to register ftrace handler for function '%s' (%d)\n",
365 func->old_name, ret); 429 func->old_name, ret);
366 ftrace_set_filter_ip(&ops->fops, func->old_addr, 1, 0); 430 ftrace_set_filter_ip(&ops->fops, ftrace_loc, 1, 0);
367 goto err; 431 goto err;
368 } 432 }
369 433
@@ -683,6 +747,9 @@ static void klp_free_patch(struct klp_patch *patch)
683 747
684static int klp_init_func(struct klp_object *obj, struct klp_func *func) 748static int klp_init_func(struct klp_object *obj, struct klp_func *func)
685{ 749{
750 if (!func->old_name || !func->new_func)
751 return -EINVAL;
752
686 INIT_LIST_HEAD(&func->stack_node); 753 INIT_LIST_HEAD(&func->stack_node);
687 func->state = KLP_DISABLED; 754 func->state = KLP_DISABLED;
688 755
@@ -703,11 +770,9 @@ static int klp_init_object_loaded(struct klp_patch *patch,
703 struct klp_func *func; 770 struct klp_func *func;
704 int ret; 771 int ret;
705 772
706 if (obj->relocs) { 773 ret = klp_write_object_relocations(patch->mod, obj);
707 ret = klp_write_object_relocations(patch->mod, obj); 774 if (ret)
708 if (ret) 775 return ret;
709 return ret;
710 }
711 776
712 klp_for_each_func(obj, func) { 777 klp_for_each_func(obj, func) {
713 ret = klp_find_object_symbol(obj->name, func->old_name, 778 ret = klp_find_object_symbol(obj->name, func->old_name,
@@ -842,12 +907,18 @@ int klp_register_patch(struct klp_patch *patch)
842{ 907{
843 int ret; 908 int ret;
844 909
845 if (!klp_initialized())
846 return -ENODEV;
847
848 if (!patch || !patch->mod) 910 if (!patch || !patch->mod)
849 return -EINVAL; 911 return -EINVAL;
850 912
913 if (!is_livepatch_module(patch->mod)) {
914 pr_err("module %s is not marked as a livepatch module",
915 patch->mod->name);
916 return -EINVAL;
917 }
918
919 if (!klp_initialized())
920 return -ENODEV;
921
851 /* 922 /*
852 * A reference is taken on the patch module to prevent it from being 923 * A reference is taken on the patch module to prevent it from being
853 * unloaded. Right now, we don't allow patch modules to unload since 924 * unloaded. Right now, we don't allow patch modules to unload since
diff --git a/kernel/module.c b/kernel/module.c
index 041200ca4a2d..5f71aa63ed2a 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -1973,6 +1973,83 @@ static void module_enable_nx(const struct module *mod) { }
1973static void module_disable_nx(const struct module *mod) { } 1973static void module_disable_nx(const struct module *mod) { }
1974#endif 1974#endif
1975 1975
1976#ifdef CONFIG_LIVEPATCH
1977/*
1978 * Persist Elf information about a module. Copy the Elf header,
1979 * section header table, section string table, and symtab section
1980 * index from info to mod->klp_info.
1981 */
1982static int copy_module_elf(struct module *mod, struct load_info *info)
1983{
1984 unsigned int size, symndx;
1985 int ret;
1986
1987 size = sizeof(*mod->klp_info);
1988 mod->klp_info = kmalloc(size, GFP_KERNEL);
1989 if (mod->klp_info == NULL)
1990 return -ENOMEM;
1991
1992 /* Elf header */
1993 size = sizeof(mod->klp_info->hdr);
1994 memcpy(&mod->klp_info->hdr, info->hdr, size);
1995
1996 /* Elf section header table */
1997 size = sizeof(*info->sechdrs) * info->hdr->e_shnum;
1998 mod->klp_info->sechdrs = kmalloc(size, GFP_KERNEL);
1999 if (mod->klp_info->sechdrs == NULL) {
2000 ret = -ENOMEM;
2001 goto free_info;
2002 }
2003 memcpy(mod->klp_info->sechdrs, info->sechdrs, size);
2004
2005 /* Elf section name string table */
2006 size = info->sechdrs[info->hdr->e_shstrndx].sh_size;
2007 mod->klp_info->secstrings = kmalloc(size, GFP_KERNEL);
2008 if (mod->klp_info->secstrings == NULL) {
2009 ret = -ENOMEM;
2010 goto free_sechdrs;
2011 }
2012 memcpy(mod->klp_info->secstrings, info->secstrings, size);
2013
2014 /* Elf symbol section index */
2015 symndx = info->index.sym;
2016 mod->klp_info->symndx = symndx;
2017
2018 /*
2019 * For livepatch modules, core_kallsyms.symtab is a complete
2020 * copy of the original symbol table. Adjust sh_addr to point
2021 * to core_kallsyms.symtab since the copy of the symtab in module
2022 * init memory is freed at the end of do_init_module().
2023 */
2024 mod->klp_info->sechdrs[symndx].sh_addr = \
2025 (unsigned long) mod->core_kallsyms.symtab;
2026
2027 return 0;
2028
2029free_sechdrs:
2030 kfree(mod->klp_info->sechdrs);
2031free_info:
2032 kfree(mod->klp_info);
2033 return ret;
2034}
2035
2036static void free_module_elf(struct module *mod)
2037{
2038 kfree(mod->klp_info->sechdrs);
2039 kfree(mod->klp_info->secstrings);
2040 kfree(mod->klp_info);
2041}
2042#else /* !CONFIG_LIVEPATCH */
2043static int copy_module_elf(struct module *mod, struct load_info *info)
2044{
2045 return 0;
2046}
2047
2048static void free_module_elf(struct module *mod)
2049{
2050}
2051#endif /* CONFIG_LIVEPATCH */
2052
1976void __weak module_memfree(void *module_region) 2053void __weak module_memfree(void *module_region)
1977{ 2054{
1978 vfree(module_region); 2055 vfree(module_region);
@@ -2011,6 +2088,9 @@ static void free_module(struct module *mod)
2011 /* Free any allocated parameters. */ 2088 /* Free any allocated parameters. */
2012 destroy_params(mod->kp, mod->num_kp); 2089 destroy_params(mod->kp, mod->num_kp);
2013 2090
2091 if (is_livepatch_module(mod))
2092 free_module_elf(mod);
2093
2014 /* Now we can delete it from the lists */ 2094 /* Now we can delete it from the lists */
2015 mutex_lock(&module_mutex); 2095 mutex_lock(&module_mutex);
2016 /* Unlink carefully: kallsyms could be walking list. */ 2096 /* Unlink carefully: kallsyms could be walking list. */
@@ -2126,6 +2206,10 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
2126 (long)sym[i].st_value); 2206 (long)sym[i].st_value);
2127 break; 2207 break;
2128 2208
2209 case SHN_LIVEPATCH:
2210 /* Livepatch symbols are resolved by livepatch */
2211 break;
2212
2129 case SHN_UNDEF: 2213 case SHN_UNDEF:
2130 ksym = resolve_symbol_wait(mod, info, name); 2214 ksym = resolve_symbol_wait(mod, info, name);
2131 /* Ok if resolved. */ 2215 /* Ok if resolved. */
@@ -2174,6 +2258,10 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
2174 if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC)) 2258 if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
2175 continue; 2259 continue;
2176 2260
2261 /* Livepatch relocation sections are applied by livepatch */
2262 if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
2263 continue;
2264
2177 if (info->sechdrs[i].sh_type == SHT_REL) 2265 if (info->sechdrs[i].sh_type == SHT_REL)
2178 err = apply_relocate(info->sechdrs, info->strtab, 2266 err = apply_relocate(info->sechdrs, info->strtab,
2179 info->index.sym, i, mod); 2267 info->index.sym, i, mod);
@@ -2469,7 +2557,7 @@ static void layout_symtab(struct module *mod, struct load_info *info)
2469 2557
2470 /* Compute total space required for the core symbols' strtab. */ 2558 /* Compute total space required for the core symbols' strtab. */
2471 for (ndst = i = 0; i < nsrc; i++) { 2559 for (ndst = i = 0; i < nsrc; i++) {
2472 if (i == 0 || 2560 if (i == 0 || is_livepatch_module(mod) ||
2473 is_core_symbol(src+i, info->sechdrs, info->hdr->e_shnum, 2561 is_core_symbol(src+i, info->sechdrs, info->hdr->e_shnum,
2474 info->index.pcpu)) { 2562 info->index.pcpu)) {
2475 strtab_size += strlen(&info->strtab[src[i].st_name])+1; 2563 strtab_size += strlen(&info->strtab[src[i].st_name])+1;
@@ -2528,7 +2616,7 @@ static void add_kallsyms(struct module *mod, const struct load_info *info)
2528 mod->core_kallsyms.strtab = s = mod->core_layout.base + info->stroffs; 2616 mod->core_kallsyms.strtab = s = mod->core_layout.base + info->stroffs;
2529 src = mod->kallsyms->symtab; 2617 src = mod->kallsyms->symtab;
2530 for (ndst = i = 0; i < mod->kallsyms->num_symtab; i++) { 2618 for (ndst = i = 0; i < mod->kallsyms->num_symtab; i++) {
2531 if (i == 0 || 2619 if (i == 0 || is_livepatch_module(mod) ||
2532 is_core_symbol(src+i, info->sechdrs, info->hdr->e_shnum, 2620 is_core_symbol(src+i, info->sechdrs, info->hdr->e_shnum,
2533 info->index.pcpu)) { 2621 info->index.pcpu)) {
2534 dst[ndst] = src[i]; 2622 dst[ndst] = src[i];
@@ -2667,6 +2755,26 @@ static int copy_chunked_from_user(void *dst, const void __user *usrc, unsigned l
2667 return 0; 2755 return 0;
2668} 2756}
2669 2757
2758#ifdef CONFIG_LIVEPATCH
2759static int find_livepatch_modinfo(struct module *mod, struct load_info *info)
2760{
2761 mod->klp = get_modinfo(info, "livepatch") ? true : false;
2762
2763 return 0;
2764}
2765#else /* !CONFIG_LIVEPATCH */
2766static int find_livepatch_modinfo(struct module *mod, struct load_info *info)
2767{
2768 if (get_modinfo(info, "livepatch")) {
2769 pr_err("%s: module is marked as livepatch module, but livepatch support is disabled",
2770 mod->name);
2771 return -ENOEXEC;
2772 }
2773
2774 return 0;
2775}
2776#endif /* CONFIG_LIVEPATCH */
2777
2670/* Sets info->hdr and info->len. */ 2778/* Sets info->hdr and info->len. */
2671static int copy_module_from_user(const void __user *umod, unsigned long len, 2779static int copy_module_from_user(const void __user *umod, unsigned long len,
2672 struct load_info *info) 2780 struct load_info *info)
@@ -2821,6 +2929,10 @@ static int check_modinfo(struct module *mod, struct load_info *info, int flags)
2821 "is unknown, you have been warned.\n", mod->name); 2929 "is unknown, you have been warned.\n", mod->name);
2822 } 2930 }
2823 2931
2932 err = find_livepatch_modinfo(mod, info);
2933 if (err)
2934 return err;
2935
2824 /* Set up license info based on the info section */ 2936 /* Set up license info based on the info section */
2825 set_license(mod, get_modinfo(info, "license")); 2937 set_license(mod, get_modinfo(info, "license"));
2826 2938
@@ -3494,6 +3606,12 @@ static int load_module(struct load_info *info, const char __user *uargs,
3494 if (err < 0) 3606 if (err < 0)
3495 goto coming_cleanup; 3607 goto coming_cleanup;
3496 3608
3609 if (is_livepatch_module(mod)) {
3610 err = copy_module_elf(mod, info);
3611 if (err < 0)
3612 goto sysfs_cleanup;
3613 }
3614
3497 /* Get rid of temporary copy. */ 3615 /* Get rid of temporary copy. */
3498 free_copy(info); 3616 free_copy(info);
3499 3617
@@ -3502,11 +3620,12 @@ static int load_module(struct load_info *info, const char __user *uargs,
3502 3620
3503 return do_init_module(mod); 3621 return do_init_module(mod);
3504 3622
3623 sysfs_cleanup:
3624 mod_sysfs_teardown(mod);
3505 coming_cleanup: 3625 coming_cleanup:
3506 blocking_notifier_call_chain(&module_notify_list, 3626 blocking_notifier_call_chain(&module_notify_list,
3507 MODULE_STATE_GOING, mod); 3627 MODULE_STATE_GOING, mod);
3508 klp_module_going(mod); 3628 klp_module_going(mod);
3509
3510 bug_cleanup: 3629 bug_cleanup:
3511 /* module_bug_cleanup needs module_mutex protection */ 3630 /* module_bug_cleanup needs module_mutex protection */
3512 mutex_lock(&module_mutex); 3631 mutex_lock(&module_mutex);
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index b1870fbd2b67..7e8d792da963 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -1530,7 +1530,19 @@ static int ftrace_cmp_recs(const void *a, const void *b)
1530 return 0; 1530 return 0;
1531} 1531}
1532 1532
1533static unsigned long ftrace_location_range(unsigned long start, unsigned long end) 1533/**
1534 * ftrace_location_range - return the first address of a traced location
1535 * if it touches the given ip range
1536 * @start: start of range to search.
1537 * @end: end of range to search (inclusive). @end points to the last byte
1538 * to check.
1539 *
1540 * Returns rec->ip if the related ftrace location is a least partly within
1541 * the given address range. That is, the first address of the instruction
1542 * that is either a NOP or call to the function tracer. It checks the ftrace
1543 * internal tables to determine if the address belongs or not.
1544 */
1545unsigned long ftrace_location_range(unsigned long start, unsigned long end)
1534{ 1546{
1535 struct ftrace_page *pg; 1547 struct ftrace_page *pg;
1536 struct dyn_ftrace *rec; 1548 struct dyn_ftrace *rec;
diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
index fb8c8614e728..e34f871e69b1 100644
--- a/samples/livepatch/livepatch-sample.c
+++ b/samples/livepatch/livepatch-sample.c
@@ -89,3 +89,4 @@ static void livepatch_exit(void)
89module_init(livepatch_init); 89module_init(livepatch_init);
90module_exit(livepatch_exit); 90module_exit(livepatch_exit);
91MODULE_LICENSE("GPL"); 91MODULE_LICENSE("GPL");
92MODULE_INFO(livepatch, "Y");