1 <!-- Creator : groff version 1.22.3 -->
2 <!-- CreationDate: Fri Apr 29 22:04:24 2016 -->
3 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
4 "http://www.w3.org/TR/html4/loose.dtd">
7 <meta name="generator" content="groff -Thtml, see www.gnu.org">
8 <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
9 <meta name="Content-Style" content="text/css">
10 <style type="text/css">
11 p { margin-top: 0; margin-bottom: 0; vertical-align: top }
12 pre { margin-top: 0; margin-bottom: 0; vertical-align: top }
13 table { margin-top: 0; margin-bottom: 0; vertical-align: top }
14 h1 { text-align: center }
23 <p>LIBARCHIVE-FORMATS(5) BSD File Formats Manual
24 LIBARCHIVE-FORMATS(5)</p>
26 <p style="margin-top: 1em"><b>NAME</b></p>
28 <p style="margin-left:6%;"><b>libarchive-formats</b>
29 — archive formats supported by the libarchive
32 <p style="margin-top: 1em"><b>DESCRIPTION</b></p>
34 <p style="margin-left:6%;">The libarchive(3) library reads
35 and writes a variety of streaming archive formats. Generally
36 speaking, all of these archive formats consist of a series
37 of ’’entries’’. Each entry stores a
38 single file system object, such as a file, directory, or
41 <p style="margin-left:6%; margin-top: 1em">The following
42 provides a brief description of each format supported by
43 libarchive, with some information about recognized
44 extensions or limitations of the current library support.
45 Note that just because a format is supported by libarchive
46 does not imply that a program that uses libarchive will
47 support that format. Applications that use libarchive
48 specify which formats they wish to support, though many
49 programs do use libarchive convenience functions to enable
50 all supported formats.</p>
52 <p style="margin-left:6%; margin-top: 1em"><b>Tar
54 The libarchive(3) library can read most tar archives. It can
55 write POSIX-standard ’’ustar’’ and
56 ’’pax interchange’’ formats as well
57 as v7 tar format and a subset of the legacy GNU tar
60 <p style="margin-left:6%; margin-top: 1em">All tar formats
61 store each entry in one or more 512-byte records. The first
62 record is used for file metadata, including filename,
63 timestamp, and mode information, and the file data is stored
64 in subsequent records. Later variants have extended this by
65 either appropriating undefined areas of the header record,
66 extending the header to multiple records, or by storing
67 special entries that modify the interpretation of subsequent
70 <p style="margin-top: 1em"><b>gnutar</b></p>
72 <p style="margin-left:17%; margin-top: 1em">The
73 libarchive(3) library can read most GNU-format tar archives.
74 It currently supports the most popular GNU extensions,
75 including modern long filename and linkname support, as well
76 as atime and ctime data. The libarchive library does not
77 support multi-volume archives, nor the old GNU long filename
78 format. It can read GNU sparse file entries, including the
79 new POSIX-based formats.</p>
81 <p style="margin-left:17%; margin-top: 1em">The
82 libarchive(3) library can write GNU tar format, including
83 long filename and linkname support, as well as atime and
86 <p style="margin-top: 1em"><b>pax</b></p>
88 <p style="margin-left:17%; margin-top: 1em">The
89 libarchive(3) library can read and write POSIX-compliant pax
90 interchange format archives. Pax interchange format archives
91 are an extension of the older ustar format that adds a
92 separate entry with additional attributes stored as
93 key/value pairs immediately before each regular entry. The
94 presence of these additional entries is the only difference
95 between pax interchange format and the older ustar format.
96 The extended attributes are of unlimited length and are
97 stored as UTF-8 Unicode strings. Keywords defined in the
98 standard are in all lowercase; vendors are allowed to define
99 custom keys by preceding them with the vendor name in all
100 uppercase. When writing pax archives, libarchive uses many
101 of the SCHILY keys defined by Joerg Schilling’s
102 ’’star’’ archiver and a few
103 LIBARCHIVE keys. The libarchive library can read most of the
104 SCHILY keys and most of the GNU keys introduced by GNU tar.
105 It silently ignores any keywords that it does not
108 <p style="margin-left:17%; margin-top: 1em">The pax
109 interchange format converts filenames to Unicode and stores
110 them using the UTF-8 encoding. Prior to libarchive 3.0,
111 libarchive erroneously assumed that the system
112 wide-character routines natively supported Unicode. This
113 caused it to mis-handle non-ASCII filenames on systems that
114 did not satisfy this assumption.</p>
116 <p style="margin-top: 1em"><b>restricted pax</b></p>
118 <p style="margin-left:17%;">The libarchive library can also
119 write pax archives in which it attempts to suppress the
120 extended attributes entry whenever possible. The result will
121 be identical to a ustar archive unless the extended
122 attributes entry is required to store a long file name, long
123 linkname, extended ACL, file flags, or if any of the
124 standard ustar data (user name, group name, UID, GID, etc)
125 cannot be fully represented in the ustar header. In all
126 cases, the result can be dearchived by any program that can
127 read POSIX-compliant pax interchange format archives.
128 Programs that correctly read ustar format (see below) will
129 also be able to read this format; any extended attributes
130 will be extracted as separate files stored in
131 <i>PaxHeader</i> directories.</p>
133 <p style="margin-top: 1em"><b>ustar</b></p>
135 <p style="margin-left:17%; margin-top: 1em">The libarchive
136 library can both read and write this format. This format has
137 the following limitations:</p>
141 <p style="margin-left:22%;">Device major and minor numbers
142 are limited to 21 bits. Nodes with larger numbers will not
143 be added to the archive.</p>
147 <p style="margin-left:22%;">Path names in the archive are
148 limited to 255 bytes. (Shorter if there is no / character in
149 exactly the right place.)</p>
153 <p style="margin-left:22%;">Symbolic links and hard links
154 are stored in the archive with the name of the referenced
155 file. This name is limited to 100 bytes.</p>
159 <p style="margin-left:22%;">Extended attributes, file
160 flags, and other extended security information cannot be
165 <p style="margin-left:22%;">Archive entries are limited to
166 8 gigabytes in size.</p>
168 <p style="margin-left:17%;">Note that the pax interchange
169 format has none of these restrictions. The ustar format is
170 old and widely supported. It is recommended when
171 compatibility is the primary concern.</p>
173 <p style="margin-top: 1em"><b>v7</b></p>
175 <p style="margin-left:17%; margin-top: 1em">The libarchive
176 library can read and write the legacy v7 tar format. This
177 format has the following limitations:</p>
181 <p style="margin-left:22%;">Only regular files,
182 directories, and symbolic links can be archived. Block and
183 character device nodes, FIFOs, and sockets cannot be
188 <p style="margin-left:22%;">Path names in the archive are
189 limited to 100 bytes.</p>
193 <p style="margin-left:22%;">Symbolic links and hard links
194 are stored in the archive with the name of the referenced
195 file. This name is limited to 100 bytes.</p>
199 <p style="margin-left:22%;">User and group information are
200 stored as numeric IDs; there is no provision for storing
201 user or group names.</p>
205 <p style="margin-left:22%;">Extended attributes, file
206 flags, and other extended security information cannot be
211 <p style="margin-left:22%;">Archive entries are limited to
212 8 gigabytes in size.</p>
214 <p style="margin-left:17%;">Generally, users should prefer
215 the ustar format for portability as the v7 tar format is
216 both less useful and less portable.</p>
218 <p style="margin-left:6%; margin-top: 1em">The libarchive
219 library also reads a variety of commonly-used extensions to
220 the basic tar format. These extensions are recognized
221 automatically whenever they appear.</p>
223 <p style="margin-top: 1em">Numeric extensions.</p>
225 <p style="margin-left:17%;">The POSIX standards require
226 fixed-length numeric fields to be written with some
227 character position reserved for terminators. Libarchive
228 allows these fields to be written without terminator
229 characters. This extends the allowable range; in particular,
230 ustar archives with this extension can support entries up to
231 64 gigabytes in size. Libarchive also recognizes base-256
232 values in most numeric fields. This essentially removes all
233 limitations on file size, modification time, and device
236 <p style="margin-top: 1em">Solaris extensions</p>
238 <p style="margin-left:17%;">Libarchive recognizes ACL and
239 extended attribute records written by Solaris tar.
240 Currently, libarchive only has support for old-style ACLs;
241 the newer NFSv4 ACLs are recognized but discarded.</p>
243 <p style="margin-left:6%; margin-top: 1em">The first tar
244 program appeared in Seventh Edition Unix in 1979. The first
245 official standard for the tar file format was the
246 ’’ustar’’ (Unix Standard Tar) format
247 defined by POSIX in 1988. POSIX.1-2001 extended the ustar
248 format to create the ’’pax
249 interchange’’ format.</p>
251 <p style="margin-left:6%; margin-top: 1em"><b>Cpio
253 The libarchive library can read a number of common cpio
254 variants and can write ’’odc’’ and
255 ’’newc’’ format archives. A cpio
256 archive stores each entry as a fixed-size header followed by
257 a variable-length filename and variable-length data. Unlike
258 the tar format, the cpio format does only minimal padding of
259 the header or file data. There are several cpio variants,
260 which differ primarily in how they store the initial header:
261 some store the values as octal or hexadecimal numbers in
262 ASCII, others as binary values of varying byte order and
265 <p style="margin-top: 1em"><b>binary</b></p>
267 <p style="margin-left:17%; margin-top: 1em">The libarchive
268 library transparently reads both big-endian and
269 little-endian variants of the original binary cpio format.
270 This format used 32-bit binary values for file size and
271 mtime, and 16-bit binary values for the other fields.</p>
273 <p style="margin-top: 1em"><b>odc</b></p>
275 <p style="margin-left:17%; margin-top: 1em">The libarchive
276 library can both read and write this POSIX-standard format,
277 which is officially known as the ’’cpio
278 interchange format’’ or the
279 ’’octet-oriented cpio archive
280 format’’ and sometimes unofficially referred to
281 as the ’’old character format’’.
282 This format stores the header contents as octal values in
283 ASCII. It is standard, portable, and immune from byte-order
284 confusion. File sizes and mtime are limited to 33 bits (8GB
285 file size), other fields are limited to 18 bits.</p>
287 <p style="margin-top: 1em"><b>SVR4/newc</b></p>
289 <p style="margin-left:17%;">The libarchive library can read
290 both CRC and non-CRC variants of this format. The SVR4
291 format uses eight-digit hexadecimal values for all header
292 fields. This limits file size to 4GB, and also limits the
293 mtime and other fields to 32 bits. The SVR4 format can
294 optionally include a CRC of the file contents, although
295 libarchive does not currently verify this CRC.</p>
297 <p style="margin-left:6%; margin-top: 1em">Cpio first
298 appeared in PWB/UNIX 1.0, which was released within AT&T
299 in 1977. PWB/UNIX 1.0 formed the basis of System III Unix,
300 released outside of AT&T in 1981. This makes cpio older
301 than tar, although cpio was not included in Version 7
302 AT&T Unix. As a result, the tar command became much
303 better known in universities and research groups that used
304 Version 7. The combination of the <b>find</b> and
305 <b>cpio</b> utilities provided very precise control over
306 file selection. Unfortunately, the format has many
307 limitations that make it unsuitable for widespread use. Only
308 the POSIX format permits files over 4GB, and its 18-bit
309 limit for most other fields makes it unsuitable for modern
310 systems. In addition, cpio formats only store numeric
311 UID/GID values (not usernames and group names), which can
312 make it very difficult to correctly transfer archives across
313 systems with dissimilar user numbering.</p>
315 <p style="margin-left:6%; margin-top: 1em"><b>Shar
317 A ’’shell archive’’ is a shell
318 script that, when executed on a POSIX-compliant system, will
319 recreate a collection of file system objects. The libarchive
320 library can write two different kinds of shar archives:</p>
322 <p style="margin-top: 1em"><b>shar</b></p>
324 <p style="margin-left:17%; margin-top: 1em">The traditional
325 shar format uses a limited set of POSIX commands, including
326 echo(1), mkdir(1), and sed(1). It is suitable for portably
327 archiving small collections of plain text files. However, it
328 is not generally well-suited for large archives (many
329 implementations of sh(1) have limits on the size of a
330 script) nor should it be used with non-text files.</p>
332 <p style="margin-top: 1em"><b>shardump</b></p>
334 <p style="margin-left:17%;">This format is similar to shar
335 but encodes files using uuencode(1) so that the result will
336 be a plain text file regardless of the file contents. It
337 also includes additional shell commands that attempt to
338 reproduce as many file attributes as possible, including
339 owner, mode, and flags. The additional commands used to
340 restore file attributes make shardump archives less portable
341 than plain shar archives.</p>
343 <p style="margin-left:6%; margin-top: 1em"><b>ISO9660
345 Libarchive can read and extract from files containing
346 ISO9660-compliant CDROM images. In many cases, this can
347 remove the need to burn a physical CDROM just in order to
348 read the files contained in an ISO9660 image. It also avoids
349 security and complexity issues that come with virtual mounts
350 and loopback devices. Libarchive supports the most common
351 Rockridge extensions and has partial support for Joliet
352 extensions. If both extensions are present, the Joliet
353 extensions will be used and the Rockridge extensions will be
354 ignored. In particular, this can create problems with
355 hardlinks and symlinks, which are supported by Rockridge but
358 <p style="margin-left:6%; margin-top: 1em">Libarchive reads
359 ISO9660 images using a streaming strategy. This allows it to
360 read compressed images directly (decompressing on the fly)
361 and allows it to read images directly from network sockets,
362 pipes, and other non-seekable data sources. This strategy
363 works well for optimized ISO9660 images created by many
364 popular programs. Such programs collect all directory
365 information at the beginning of the ISO9660 image so it can
366 be read from a physical disk with a minimum of seeking.
367 However, not all ISO9660 images can be read in this
370 <p style="margin-left:6%; margin-top: 1em">Libarchive can
371 also write ISO9660 images. Such images are fully optimized
372 with the directory information preceding all file data. This
373 is done by storing all file data to a temporary file while
374 collecting directory information in memory. When the image
375 is finished, libarchive writes out the directory structure
376 followed by the file data. The location used for the
377 temporary file can be changed by the usual environment
380 <p style="margin-left:6%; margin-top: 1em"><b>Zip
382 Libarchive can read and write zip format archives that have
383 uncompressed entries and entries compressed with the
384 ’’deflate’’ algorithm. Other zip
385 compression algorithms are not supported. It can extract jar
386 archives, archives that use Zip64 extensions and
387 self-extracting zip archives. Libarchive can use either of
388 two different strategies for reading Zip archives: a
389 streaming strategy which is fast and can handle extremely
390 large archives, and a seeking strategy which can correctly
391 process self-extracting Zip archives and archives with
392 deleted members or other in-place modifications.</p>
394 <p style="margin-left:6%; margin-top: 1em">The streaming
395 reader processes Zip archives as they are read. It can read
396 archives of arbitrary size from tape or network sockets, and
397 can decode Zip archives that have been separately compressed
398 or encoded. However, self-extracting Zip archives and
399 archives with certain types of modifications cannot be
400 correctly handled. Such archives require that the reader
401 first process the Central Directory, which is ordinarily
402 located at the end of a Zip archive and is thus inaccessible
403 to the streaming reader. If the program using libarchive has
404 enabled seek support, then libarchive will use this to
405 processes the central directory first.</p>
407 <p style="margin-left:6%; margin-top: 1em">In particular,
408 the seeking reader must be used to correctly handle
409 self-extracting archives. Such archives consist of a program
410 followed by a regular Zip archive. The streaming reader
411 cannot parse the initial program portion, but the seeking
412 reader starts by reading the Central Directory from the end
413 of the archive. Similarly, Zip archives that have been
414 modified in-place can have deleted entries or other garbage
415 data that can only be accurately detected by first reading
416 the Central Directory.</p>
418 <p style="margin-left:6%; margin-top: 1em"><b>Archive
419 (library) file format</b> <br>
420 The Unix archive format (commonly created by the ar(1)
421 archiver) is a general-purpose format which is used almost
422 exclusively for object files to be read by the link editor
423 ld(1). The ar format has never been standardised. There are
424 two common variants: the GNU format derived from SVR4, and
425 the BSD format, which first appeared in 4.4BSD. The two
426 differ primarily in their handling of filenames longer than
427 15 characters: the GNU/SVR4 variant writes a filename table
428 at the beginning of the archive; the BSD format stores each
429 long filename in an extension area adjacent to the entry.
430 Libarchive can read both extensions, including archives that
431 may include both types of long filenames. Programs using
432 libarchive can write GNU/SVR4 format if they provide an
433 entry called <i>//</i> containing a filename table to be
434 written into the archive before any of the entries. Any
435 entries whose names are not in the filename table will be
436 written using BSD-style long filenames. This can cause
437 problems for programs such as GNU ld that do not support the
438 BSD-style long filenames.</p>
440 <p style="margin-left:6%; margin-top: 1em"><b>mtree</b>
442 Libarchive can read and write files in mtree(5) format. This
443 format is not a true archive format, but rather a textual
444 description of a file hierarchy in which each line specifies
445 the name of a file and provides specific metadata about that
446 file. Libarchive can read all of the keywords supported by
447 both the NetBSD and FreeBSD versions of mtree(8), although
448 many of the keywords cannot currently be stored in an
449 archive_entry object. When writing, libarchive supports use
450 of the archive_write_set_options(3) interface to specify
451 which keywords should be included in the output. If
452 libarchive was compiled with access to suitable
453 cryptographic libraries (such as the OpenSSL libraries), it
454 can compute hash entries such as <b>sha512</b> or <b>md5</b>
455 from file data being written to the mtree writer.</p>
457 <p style="margin-left:6%; margin-top: 1em">When reading an
458 mtree file, libarchive will locate the corresponding files
459 on disk using the <b>contents</b> keyword if present or the
460 regular filename. If it can locate and open the file on
461 disk, it will use that to fill in any metadata that is
462 missing from the mtree file and will read the file contents
463 and return those to the program using libarchive. If it
464 cannot locate and open the file on disk, libarchive will
465 return an error for any attempt to read the entry body.</p>
467 <p style="margin-left:6%; margin-top: 1em"><b>7-Zip</b>
469 Libarchive can read and write 7-Zip format archives. TODO:
470 Need more information</p>
472 <p style="margin-left:6%; margin-top: 1em"><b>CAB</b> <br>
473 Libarchive can read Microsoft Cabinet (
474 ’’CAB’’) format archives. TODO: Need
475 more information.</p>
477 <p style="margin-left:6%; margin-top: 1em"><b>LHA</b> <br>
478 TODO: Information about libarchive’s LHA support</p>
480 <p style="margin-left:6%; margin-top: 1em"><b>RAR</b> <br>
481 Libarchive has limited support for reading RAR format
482 archives. Currently, libarchive can read RARv3 format
483 archives which have been either created uncompressed, or
484 compressed using any of the compression methods supported by
485 the RARv3 format. Libarchive can also read self-extracting
488 <p style="margin-left:6%; margin-top: 1em"><b>Warc</b> <br>
489 Libarchive can read and write ’’web
490 archives’’. TODO: Need more information</p>
492 <p style="margin-left:6%; margin-top: 1em"><b>XAR</b> <br>
493 Libarchive can read and write the XAR format used by many
494 Apple tools. TODO: Need more information</p>
496 <p style="margin-top: 1em"><b>SEE ALSO</b></p>
498 <p style="margin-left:6%;">ar(1), cpio(1), mkisofs(1),
499 shar(1), tar(1), zip(1), zlib(3), cpio(5), mtree(5),
502 <p style="margin-left:6%; margin-top: 1em">BSD
503 March 18, 2012 BSD</p>
projects / platform / upstream / libarchive.git / blob