summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMauro Carvalho Chehab <mchehab@s-opensource.com>2017-05-17 08:54:22 -0400
committerJonathan Corbet <corbet@lwn.net>2017-07-14 15:58:11 -0400
commit29c8c4ac9525b15457266a8fb8bb59f366f5d65b (patch)
treeeffd250be26882849d39703d43705df579ee315c
parent05a07392c30cd38b466c7d4a1ab3979ce1fe3cb2 (diff)
xz.txt: standardize document format
Each text file under Documentation follows a different format. Some doesn't even have titles! Change its representation to follow the adopted standard, using ReST markups for it to be parseable by Sphinx: - Use marks for titles; - Adjust indentation. Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
-rw-r--r--Documentation/xz.txt200
1 files changed, 103 insertions, 97 deletions
diff --git a/Documentation/xz.txt b/Documentation/xz.txt
index 2cf3e2608de3..b2220d03aa50 100644
--- a/Documentation/xz.txt
+++ b/Documentation/xz.txt
@@ -1,121 +1,127 @@
1 1============================
2XZ data compression in Linux 2XZ data compression in Linux
3============================ 3============================
4 4
5Introduction 5Introduction
6============
6 7
7 XZ is a general purpose data compression format with high compression 8XZ is a general purpose data compression format with high compression
8 ratio and relatively fast decompression. The primary compression 9ratio and relatively fast decompression. The primary compression
9 algorithm (filter) is LZMA2. Additional filters can be used to improve 10algorithm (filter) is LZMA2. Additional filters can be used to improve
10 compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters 11compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
11 improve compression ratio of executable data. 12improve compression ratio of executable data.
12 13
13 The XZ decompressor in Linux is called XZ Embedded. It supports 14The XZ decompressor in Linux is called XZ Embedded. It supports
14 the LZMA2 filter and optionally also BCJ filters. CRC32 is supported 15the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
15 for integrity checking. The home page of XZ Embedded is at 16for integrity checking. The home page of XZ Embedded is at
16 <http://tukaani.org/xz/embedded.html>, where you can find the 17<http://tukaani.org/xz/embedded.html>, where you can find the
17 latest version and also information about using the code outside 18latest version and also information about using the code outside
18 the Linux kernel. 19the Linux kernel.
19 20
20 For userspace, XZ Utils provide a zlib-like compression library 21For userspace, XZ Utils provide a zlib-like compression library
21 and a gzip-like command line tool. XZ Utils can be downloaded from 22and a gzip-like command line tool. XZ Utils can be downloaded from
22 <http://tukaani.org/xz/>. 23<http://tukaani.org/xz/>.
23 24
24XZ related components in the kernel 25XZ related components in the kernel
25 26===================================
26 The xz_dec module provides XZ decompressor with single-call (buffer 27
27 to buffer) and multi-call (stateful) APIs. The usage of the xz_dec 28The xz_dec module provides XZ decompressor with single-call (buffer
28 module is documented in include/linux/xz.h. 29to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
29 30module is documented in include/linux/xz.h.
30 The xz_dec_test module is for testing xz_dec. xz_dec_test is not 31
31 useful unless you are hacking the XZ decompressor. xz_dec_test 32The xz_dec_test module is for testing xz_dec. xz_dec_test is not
32 allocates a char device major dynamically to which one can write 33useful unless you are hacking the XZ decompressor. xz_dec_test
33 .xz files from userspace. The decompressed output is thrown away. 34allocates a char device major dynamically to which one can write
34 Keep an eye on dmesg to see diagnostics printed by xz_dec_test. 35.xz files from userspace. The decompressed output is thrown away.
35 See the xz_dec_test source code for the details. 36Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
36 37See the xz_dec_test source code for the details.
37 For decompressing the kernel image, initramfs, and initrd, there 38
38 is a wrapper function in lib/decompress_unxz.c. Its API is the 39For decompressing the kernel image, initramfs, and initrd, there
39 same as in other decompress_*.c files, which is defined in 40is a wrapper function in lib/decompress_unxz.c. Its API is the
40 include/linux/decompress/generic.h. 41same as in other decompress_*.c files, which is defined in
41 42include/linux/decompress/generic.h.
42 scripts/xz_wrap.sh is a wrapper for the xz command line tool found 43
43 from XZ Utils. The wrapper sets compression options to values suitable 44scripts/xz_wrap.sh is a wrapper for the xz command line tool found
44 for compressing the kernel image. 45from XZ Utils. The wrapper sets compression options to values suitable
45 46for compressing the kernel image.
46 For kernel makefiles, two commands are provided for use with 47
47 $(call if_needed). The kernel image should be compressed with 48For kernel makefiles, two commands are provided for use with
48 $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 49$(call if_needed). The kernel image should be compressed with
49 dictionary. It will also append a four-byte trailer containing the 50$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
50 uncompressed size of the file, which is needed by the boot code. 51dictionary. It will also append a four-byte trailer containing the
51 Other things should be compressed with $(call if_needed,xzmisc) 52uncompressed size of the file, which is needed by the boot code.
52 which will use no BCJ filter and 1 MiB LZMA2 dictionary. 53Other things should be compressed with $(call if_needed,xzmisc)
54which will use no BCJ filter and 1 MiB LZMA2 dictionary.
53 55
54Notes on compression options 56Notes on compression options
57============================
55 58
56 Since the XZ Embedded supports only streams with no integrity check or 59Since the XZ Embedded supports only streams with no integrity check or
57 CRC32, make sure that you don't use some other integrity check type 60CRC32, make sure that you don't use some other integrity check type
58 when encoding files that are supposed to be decoded by the kernel. With 61when encoding files that are supposed to be decoded by the kernel. With
59 liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 62liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
60 when encoding. With the xz command line tool, use --check=none or 63when encoding. With the xz command line tool, use --check=none or
61 --check=crc32. 64--check=crc32.
62 65
63 Using CRC32 is strongly recommended unless there is some other layer 66Using CRC32 is strongly recommended unless there is some other layer
64 which will verify the integrity of the uncompressed data anyway. 67which will verify the integrity of the uncompressed data anyway.
65 Double checking the integrity would probably be waste of CPU cycles. 68Double checking the integrity would probably be waste of CPU cycles.
66 Note that the headers will always have a CRC32 which will be validated 69Note that the headers will always have a CRC32 which will be validated
67 by the decoder; you can only change the integrity check type (or 70by the decoder; you can only change the integrity check type (or
68 disable it) for the actual uncompressed data. 71disable it) for the actual uncompressed data.
69 72
70 In userspace, LZMA2 is typically used with dictionary sizes of several 73In userspace, LZMA2 is typically used with dictionary sizes of several
71 megabytes. The decoder needs to have the dictionary in RAM, thus big 74megabytes. The decoder needs to have the dictionary in RAM, thus big
72 dictionaries cannot be used for files that are intended to be decoded 75dictionaries cannot be used for files that are intended to be decoded
73 by the kernel. 1 MiB is probably the maximum reasonable dictionary 76by the kernel. 1 MiB is probably the maximum reasonable dictionary
74 size for in-kernel use (maybe more is OK for initramfs). The presets 77size for in-kernel use (maybe more is OK for initramfs). The presets
75 in XZ Utils may not be optimal when creating files for the kernel, 78in XZ Utils may not be optimal when creating files for the kernel,
76 so don't hesitate to use custom settings. Example: 79so don't hesitate to use custom settings. Example::
77 80
78 xz --check=crc32 --lzma2=dict=512KiB inputfile 81 xz --check=crc32 --lzma2=dict=512KiB inputfile
79 82
80 An exception to above dictionary size limitation is when the decoder 83An exception to above dictionary size limitation is when the decoder
81 is used in single-call mode. Decompressing the kernel itself is an 84is used in single-call mode. Decompressing the kernel itself is an
82 example of this situation. In single-call mode, the memory usage 85example of this situation. In single-call mode, the memory usage
83 doesn't depend on the dictionary size, and it is perfectly fine to 86doesn't depend on the dictionary size, and it is perfectly fine to
84 use a big dictionary: for maximum compression, the dictionary should 87use a big dictionary: for maximum compression, the dictionary should
85 be at least as big as the uncompressed data itself. 88be at least as big as the uncompressed data itself.
86 89
87Future plans 90Future plans
91============
88 92
89 Creating a limited XZ encoder may be considered if people think it is 93Creating a limited XZ encoder may be considered if people think it is
90 useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at 94useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
91 the fastest settings, so it isn't clear if LZMA2 encoder is wanted 95the fastest settings, so it isn't clear if LZMA2 encoder is wanted
92 into the kernel. 96into the kernel.
93 97
94 Support for limited random-access reading is planned for the 98Support for limited random-access reading is planned for the
95 decompression code. I don't know if it could have any use in the 99decompression code. I don't know if it could have any use in the
96 kernel, but I know that it would be useful in some embedded projects 100kernel, but I know that it would be useful in some embedded projects
97 outside the Linux kernel. 101outside the Linux kernel.
98 102
99Conformance to the .xz file format specification 103Conformance to the .xz file format specification
104================================================
100 105
101 There are a couple of corner cases where things have been simplified 106There are a couple of corner cases where things have been simplified
102 at expense of detecting errors as early as possible. These should not 107at expense of detecting errors as early as possible. These should not
103 matter in practice all, since they don't cause security issues. But 108matter in practice all, since they don't cause security issues. But
104 it is good to know this if testing the code e.g. with the test files 109it is good to know this if testing the code e.g. with the test files
105 from XZ Utils. 110from XZ Utils.
106 111
107Reporting bugs 112Reporting bugs
113==============
108 114
109 Before reporting a bug, please check that it's not fixed already 115Before reporting a bug, please check that it's not fixed already
110 at upstream. See <http://tukaani.org/xz/embedded.html> to get the 116at upstream. See <http://tukaani.org/xz/embedded.html> to get the
111 latest code. 117latest code.
112 118
113 Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on 119Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
114 Freenode and talk to Larhzu. I don't actively read LKML or other 120Freenode and talk to Larhzu. I don't actively read LKML or other
115 kernel-related mailing lists, so if there's something I should know, 121kernel-related mailing lists, so if there's something I should know,
116 you should email to me personally or use IRC. 122you should email to me personally or use IRC.
117 123
118 Don't bother Igor Pavlov with questions about the XZ implementation 124Don't bother Igor Pavlov with questions about the XZ implementation
119 in the kernel or about XZ Utils. While these two implementations 125in the kernel or about XZ Utils. While these two implementations
120 include essential code that is directly based on Igor Pavlov's code, 126include essential code that is directly based on Igor Pavlov's code,
121 these implementations aren't maintained nor supported by him. 127these implementations aren't maintained nor supported by him.