TWxSoftware/twx-linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

xz: Documentation/staging/xz.rst: Revise thoroughly

Browse Source 
Add SPDX license identifier.

Omit xz_dec_test info. That isn't relevant to developers of non-XZ code.

Revise the docs about xzkern and add xzkern_with_size.  The latter was
added to scripts/Makefile.lib in the commit 7ce7e984ab ("kbuild: rename
cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22}").

Omit contact info as MAINTAINERS has it.

Omit other info that is outdated or not relevant in the kernel context.

Include the xz_dec kernel-doc from include/linux/xz.h.

Link: https://lkml.kernel.org/r/20240721133633.47721-8-lasse.collin@tukaani.org
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Reviewed-by: Sam James <sam@gentoo.org>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Cc: Rui Li <me@lirui.org>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Emil Renner Berthing <emil.renner.berthing@canonical.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Joel Stanley <joel@jms.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Jubin Zhong <zhongjubin@huawei.com>
Cc: Jules Maselbas <jmaselbas@zdiv.net>
Cc: Krzysztof Kozlowski <krzk@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Simon Glass <sjg@chromium.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
This commit is contained in:
Lasse Collin
2024-07-21 16:36:22 +03:00
committed by Andrew Morton
parent 0f2c599634
commit 6416724679
1 changed files with 58 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/staging/xz.rst
+58 -84
View File
@@ -1,3 +1,5 @@
.. SPDX-License-Identifier: 0BSD
============================ ============================
XZ data compression in Linux XZ data compression in Linux
============================ ============================
@@ -6,62 +8,55 @@ Introduction
============ ============
XZ is a general purpose data compression format with high compression XZ is a general purpose data compression format with high compression
ratio and relatively fast decompression. The primary compression ratio. The XZ decompressor in Linux is called XZ Embedded. It supports
algorithm (filter) is LZMA2. Additional filters can be used to improve the LZMA2 filter and optionally also Branch/Call/Jump (BCJ) filters
compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters for executable code. CRC32 is supported for integrity checking.
improve compression ratio of executable data.
The XZ decompressor in Linux is called XZ Embedded. It supports See the `XZ Embedded`_ home page for the latest version which includes
the LZMA2 filter and optionally also BCJ filters. CRC32 is supported a few optional extra features that aren't required in the Linux kernel
for integrity checking. The home page of XZ Embedded is at and information about using the code outside the Linux kernel.
<https://tukaani.org/xz/embedded.html>, where you can find the
latest version and also information about using the code outside
the Linux kernel.
For userspace, XZ Utils provide a zlib-like compression library For userspace, `XZ Utils`_ provide a zlib-like compression library
and a gzip-like command line tool. XZ Utils can be downloaded from and a gzip-like command line tool.
<https://tukaani.org/xz/>.
.. _XZ Embedded: https://tukaani.org/xz/embedded.html
.. _XZ Utils: https://tukaani.org/xz/
XZ related components in the kernel XZ related components in the kernel
=================================== ===================================
The xz_dec module provides XZ decompressor with single-call (buffer The xz_dec module provides XZ decompressor with single-call (buffer
to buffer) and multi-call (stateful) APIs. The usage of the xz_dec to buffer) and multi-call (stateful) APIs in include/linux/xz.h.
module is documented in include/linux/xz.h.
The xz_dec_test module is for testing xz_dec. xz_dec_test is not
useful unless you are hacking the XZ decompressor. xz_dec_test
allocates a char device major dynamically to which one can write
.xz files from userspace. The decompressed output is thrown away.
Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
See the xz_dec_test source code for the details.
For decompressing the kernel image, initramfs, and initrd, there For decompressing the kernel image, initramfs, and initrd, there
is a wrapper function in lib/decompress_unxz.c. Its API is the is a wrapper function in lib/decompress_unxz.c. Its API is the
same as in other decompress_*.c files, which is defined in same as in other decompress_*.c files, which is defined in
include/linux/decompress/generic.h. include/linux/decompress/generic.h.
scripts/xz_wrap.sh is a wrapper for the xz command line tool found For kernel makefiles, three commands are provided for use with
from XZ Utils. The wrapper sets compression options to values suitable ``$(call if_changed)``. They require the xz tool from XZ Utils.
for compressing the kernel image.
For kernel makefiles, two commands are provided for use with - ``$(call if_changed,xzkern)`` is for compressing the kernel image.
$(call if_needed). The kernel image should be compressed with It runs the script scripts/xz_wrap.sh which uses arch-optimized
$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 options and a big LZMA2 dictionary.
dictionary. It will also append a four-byte trailer containing the
uncompressed size of the file, which is needed by the boot code. - ``$(call if_changed,xzkern_with_size)`` is like ``xzkern`` above but
Other things should be compressed with $(call if_needed,xzmisc) this also appends a four-byte trailer containing the uncompressed size
which will use no BCJ filter and 1 MiB LZMA2 dictionary. of the file. The trailer is needed by the boot code on some archs.
- Other things can be compressed with ``$(call if_needed,xzmisc)``
which will use no BCJ filter and 1 MiB LZMA2 dictionary.
Notes on compression options Notes on compression options
============================ ============================
Since the XZ Embedded supports only streams with no integrity check or Since the XZ Embedded supports only streams with CRC32 or no integrity
CRC32, make sure that you don't use some other integrity check type check, make sure that you don't use some other integrity check type
when encoding files that are supposed to be decoded by the kernel. With when encoding files that are supposed to be decoded by the kernel.
liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 With liblzma from XZ Utils, you need to use either ``LZMA_CHECK_CRC32``
when encoding. With the xz command line tool, use --check=none or or ``LZMA_CHECK_NONE`` when encoding. With the ``xz`` command line tool,
--check=crc32. use ``--check=crc32`` or ``--check=none`` to override the default
``--check=crc64``.
Using CRC32 is strongly recommended unless there is some other layer Using CRC32 is strongly recommended unless there is some other layer
which will verify the integrity of the uncompressed data anyway. which will verify the integrity of the uncompressed data anyway.
@@ -71,57 +66,36 @@ by the decoder; you can only change the integrity check type (or
disable it) for the actual uncompressed data. disable it) for the actual uncompressed data.
In userspace, LZMA2 is typically used with dictionary sizes of several In userspace, LZMA2 is typically used with dictionary sizes of several
megabytes. The decoder needs to have the dictionary in RAM, thus big megabytes. The decoder needs to have the dictionary in RAM:
dictionaries cannot be used for files that are intended to be decoded
by the kernel. 1 MiB is probably the maximum reasonable dictionary
size for in-kernel use (maybe more is OK for initramfs). The presets
in XZ Utils may not be optimal when creating files for the kernel,
so don't hesitate to use custom settings. Example::
xz --check=crc32 --lzma2=dict=512KiB inputfile - In multi-call mode the dictionary is allocated as part of the
decoder state. The reasonable maximum dictionary size for in-kernel
use will depend on the target hardware: a few megabytes is fine for
desktop systems while 64 KiB to 1 MiB might be more appropriate on
some embedded systems.
An exception to above dictionary size limitation is when the decoder - In single-call mode the output buffer is used as the dictionary
is used in single-call mode. Decompressing the kernel itself is an buffer. That is, the size of the dictionary doesn't affect the
example of this situation. In single-call mode, the memory usage decompressor memory usage at all. Only the base data structures
doesn't depend on the dictionary size, and it is perfectly fine to are allocated which take a little less than 30 KiB of memory.
use a big dictionary: for maximum compression, the dictionary should For the best compression, the dictionary should be at least
be at least as big as the uncompressed data itself. as big as the uncompressed data. A notable example of single-call
mode is decompressing the kernel itself (except on PowerPC).
Future plans The compression presets in XZ Utils may not be optimal when creating
============ files for the kernel, so don't hesitate to use custom settings to,
for example, set the dictionary size. Also, xz may produce a smaller
file in single-threaded mode so setting that explicitly is recommended.
Example::
Creating a limited XZ encoder may be considered if people think it is xz --threads=1 --check=crc32 --lzma2=dict=512KiB inputfile
useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
the fastest settings, so it isn't clear if LZMA2 encoder is wanted
into the kernel.
Support for limited random-access reading is planned for the xz_dec API
decompression code. I don't know if it could have any use in the ==========
kernel, but I know that it would be useful in some embedded projects
outside the Linux kernel.
Conformance to the .xz file format specification This is available with ``#include <linux/xz.h>``.
================================================
There are a couple of corner cases where things have been simplified ``XZ_EXTERN`` is a macro used in the preboot code. Ignore it when
at expense of detecting errors as early as possible. These should not reading this documentation.
matter in practice all, since they don't cause security issues. But
it is good to know this if testing the code e.g. with the test files
from XZ Utils.
Reporting bugs .. kernel-doc:: include/linux/xz.h
==============
Before reporting a bug, please check that it's not fixed already
at upstream. See <https://tukaani.org/xz/embedded.html> to get the
latest code.
Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
Freenode and talk to Larhzu. I don't actively read LKML or other
kernel-related mailing lists, so if there's something I should know,
you should email to me personally or use IRC.
Don't bother Igor Pavlov with questions about the XZ implementation
in the kernel or about XZ Utils. While these two implementations
include essential code that is directly based on Igor Pavlov's code,
these implementations aren't maintained nor supported by him.