Jim Meyering [Wed, 12 Jan 2011 20:20:34 +0000 (21:20 +0100)]
doc: clean up HACKING guidelines
* HACKING: Remove mention of "indent-tabs-mode: nil", since
we've remove all of those directives. No longer needed.
Remove dated (pre-emacs-23) reference regarding WhiteSpace mode.
Paul Eggert [Wed, 12 Jan 2011 02:07:15 +0000 (18:07 -0800)]
gnulib: Also use dtoastr and ldtoastr modules.
This adjusts to the recent splitting of the ftoastr module into 3
<http://lists.gnu.org/archive/html/bug-gnulib/2011-01/msg00199.html>.
* bootstrap.conf (gnulib_modules): Add dtoastr, ldtoastr,
as coreutils needs all 3 modules now.
Paul Eggert [Wed, 12 Jan 2011 01:52:40 +0000 (17:52 -0800)]
build: update gnulib submodule to latest
Nadav Har'El [Tue, 11 Jan 2011 14:53:07 +0000 (07:53 -0700)]
rm: ignore errno related to invalid file names
* src/remove.c (nonexistent_file_errno): Also skip EINVAL and
EILSEQ, for at least smbfs rejection of '*' in file names.
* NEWS: Mention the fix.
Jim Meyering [Mon, 10 Jan 2011 06:59:00 +0000 (07:59 +0100)]
build: update gnulib submodule to latest
Jim Meyering [Sat, 8 Jan 2011 16:44:55 +0000 (17:44 +0100)]
du: don't abort when a subdir is renamed during traversal
* NEWS (Bug fixes): Mention it.
* src/du.c (prev_level): Move declaration "up" to file-scope global.
(du_files): Reset prev_level to 0 upon abnormal fts_read termination.
Reported by Johathan Nieder in http://bugs.debian.org/609049
Also, improve a diagnostic.
* tests/du/move-dir-while-traversing: Test for the above.
* tests/Makefile.am (TESTS): Add it.
Pádraig Brady [Sat, 18 Dec 2010 23:06:10 +0000 (23:06 +0000)]
maint: suppress some clang scan-build warnings
* src/pr.c (char_to_clump): Remove a dead store.
* src/remove.c (fts_skip_tree): Likewise.
* src/sort.c (key_warnings): Likewise.
(sort): Suppress an uninitialized pointer warning.
Pádraig Brady [Wed, 5 Jan 2011 12:01:14 +0000 (12:01 +0000)]
maint: replace uses of ignore_ptr with ignore_value
* gnulib: Update for enhanced ignore_value()
* src/chcon.c (process_file): Don't use the deprecated ignore_ptr.
* src/chmod.c (process_file): Likewise.
* src/chown-core.c (change_file_owner): Likewise.
Jim Meyering [Tue, 4 Jan 2011 11:56:30 +0000 (12:56 +0100)]
post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
Jim Meyering [Tue, 4 Jan 2011 11:18:32 +0000 (12:18 +0100)]
version 8.9
* NEWS: Record release date.
Jim Meyering [Tue, 4 Jan 2011 11:17:21 +0000 (12:17 +0100)]
build: update gnulib to latest; portability fixes
Jim Meyering [Sat, 1 Jan 2011 10:37:32 +0000 (11:37 +0100)]
maint: update all copyright year number ranges
Run "make update-copyright".
Jim Meyering [Sat, 1 Jan 2011 10:10:00 +0000 (11:10 +0100)]
build: update gnulib for version-etc copyright year update
* tests/sample-test: Update copyright to 2011, to appease syntax-check.
Jim Meyering [Mon, 17 Nov 2008 11:05:27 +0000 (12:05 +0100)]
maint: generate much of the THANKS file
Before this change, we had a tendency to manually list each
contributor's name in THANKS. Now, each commit "Author" is
included in the generated THANKS file automatically, and most
of the old THANKS file is now a template, THANKS.in.
We'll still have to manually list the names of people who report
problems without a usable patch.
* THANKS.in: New file, derived from THANKS, but removing names of
those who are listed as git log 'Author:'s.
* THANKS: Remove file.
* thanks-gen: New file.
* Makefile.am (THANKS): New rule.
(EXTRA_DIST): Add .mailmap, THANKS.in and thanks-gen.
* .gitignore: Add THANKS and THANKS-to-translators.
* .mailmap: Unify on single address and name-spelling per contributor.
Eric Blake [Fri, 31 Dec 2010 16:50:46 +0000 (09:50 -0700)]
maint: update to latest gnulib, for testsuite improvement
* gnulib: Update to latest for init.sh fix.
* bootstrap: Resync from gnulib.
* tests/init.sh: Likewise.
Eric Blake [Thu, 30 Dec 2010 20:08:32 +0000 (13:08 -0700)]
maint: allow gettext 0.17 again
Commit
041c9c47 traded the 'gettext' module for the lighter 'gettext-h'
module, so as to not require the latest gettext release (we only need
the latest release if we ship gettext as a dependent library, but
coreutils has long preferred to use it as an external library).
But that commit overlooked two places necessary to allow the use of
gettext 0.17.
This does not force you to downgrade (using gettext 0.18.1.1 is still
just fine), nor does it affect tarballs (once a tarball is built
with a given gettext version, it can be built on other machines
regardless of what gettext version is present).
* bootstrap.conf (buildreq): Relax prerequisite.
* configure.ac (AM_GNU_GETTEXT_VERSION): Likewise.
Pádraig Brady [Thu, 30 Dec 2010 01:36:59 +0000 (01:36 +0000)]
split: fix the suffix length calculation
* src/split.c (set_suffix_length): Only auto-calculate
the suffix length when the number of files is specified.
* tests/misc/split-a: Add a case to trigger the bug,
and exercise the suffix length auto-calculation.
* NEWS: Mention the fix.
Reported by Dmitry V. Levin and Sergey Vlasov at
https://bugzilla.altlinux.org/show_bug.cgi?id=24841
Paul Eggert [Tue, 28 Dec 2010 20:28:48 +0000 (12:28 -0800)]
coreutils: keep lines within 80-column limits
* cfg.mk (LINE_LEN_MAX, FILTER_LONG_LINES): New macros.
(sc_long_lines): New rule.
* HACKING: Use shorter URLs to the same material.
* doc/Makefile.am, doc/coreutils.texi, m4/boottime.m4:
* man/help2man, man/stdbuf.x, src/Makefile.am, src/cat.c, src/copy.c:
* src/cp.c, src/dd.c, src/df.c, src/du.c, src/groups.c, src/install.c:
* src/ls.c, src/md5sum.c, src/mv.c, src/od.c, src/pinky.c, src/ptx.c:
* src/readlink.c, src/remove.c, src/rmdir.c, src/setuidgid.c:
* src/sort.c, src/tail.c, src/touch.c, tests/Coreutils.pm:
* tests/cp/existing-perm-race, tests/cp/perm, tests/cp/preserve-gid:
* tests/du/2g, tests/du/long-from-unreadable, tests/init.sh:
* tests/install/basic-1, tests/ls/nameless-uid:
* tests/ls/readdir-mountpoint-inode, tests/misc/chroot-credentials:
* tests/misc/cut, tests/misc/date, tests/misc/join, tests/misc/md5sum:
* tests/misc/sha1sum, tests/misc/sha224sum, tests/misc/sort:
* tests/misc/sort-continue, tests/misc/sort-files0-from:
* tests/misc/sort-rand, tests/misc/stdbuf, tests/misc/tr:
* tests/misc/uniq, tests/mv/atomic, tests/mv/part-fail:
* tests/mv/part-symlink, tests/mv/sticky-to-xpart, tests/pr/pr-tests:
* tests/rm/fail-2eperm, tests/rm/interactive-always:
Reformat to fit within 80 columns.
* doc/Makefile.am (BAD_POSIX_PERL): New macro.
* doc/coreutils.texi: Reword slightly, to make menus and
index lines shorter.
* src/md5sum.c: Redo --help output so that it fits within 79
columns, since that's a bit more portable and all the other --help
strings fit in 79 columns.
Jim Meyering [Fri, 24 Dec 2010 13:59:06 +0000 (14:59 +0100)]
maint: avoid syntax-check failure due to unused #include
* src/getlimits.c: Don't include "c-ctype.h"; no longer used.
Paul Eggert [Thu, 23 Dec 2010 08:07:35 +0000 (00:07 -0800)]
csplit: diagnose file counter wraparound
* src/csplit.c (create_output_file): Detect overflow when the
file counter wraps around, and exit with a diagnostic. Formerly
the code silently wrapped around and wrote to the wrong file,
losing output data.
Paul Eggert [Thu, 23 Dec 2010 07:29:57 +0000 (23:29 -0800)]
sort: minor performance tweak with num_processors
* src/sort.c (main): Don't invoke num_processors twice.
Paul Eggert [Thu, 23 Dec 2010 07:11:05 +0000 (23:11 -0800)]
getlimits: port to hosts with very wide int, or non-ASCII
* src/getlimits.c (decimal_ascii_add): Remove, replacing with ...
(decimal_absval_add_one): New function, with different signature,
which does not assume ASCII. All callers changed.
(print_int): Remove assumptions that integers fit in 206 bits, and
that characters are ASCII. These assumptions are portable in
practice but are easy to remove here.
Jim Meyering [Wed, 22 Dec 2010 17:19:37 +0000 (18:19 +0100)]
post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
Jim Meyering [Wed, 22 Dec 2010 16:43:50 +0000 (17:43 +0100)]
version 8.8
* NEWS: Record release date.
Jim Meyering [Wed, 22 Dec 2010 08:29:20 +0000 (09:29 +0100)]
csplit: update gnulib for Solaris 8,9 snprintf fix
Jim Meyering [Wed, 22 Dec 2010 10:49:25 +0000 (11:49 +0100)]
maint: correct test-related comments
* tests/mv/i-3: Adjust comment to match just-changed code.
Spotted by Pádraig Brady.
* tests/init.cfg (retry_delay_): Correct spelling of function name
in usage example.
Jim Meyering [Wed, 22 Dec 2010 10:10:23 +0000 (11:10 +0100)]
tests: adjust preceding change to handle general WERROR_CFLAGS values
* gnulib-tests/Makefile.am (test_xvasprintf_CFLAGS):
(test_lock_CFLAGS, test_tls_CFLAGS): Avoid a syntax error when
$(WERROR_CFLAGS) expands to more than one token.
Paul Eggert [Wed, 22 Dec 2010 09:48:27 +0000 (01:48 -0800)]
tests: do not assume compiler knows -Wxxx flags
* gnulib-tests/Makefile.am (test_xvasprintf_CFLAGS):
(test_lock_CFLAGS, test_tls_CFLAGS): Do not append GCC-specific
flags like -Wno-format-security unless the GCC-specific flag
-Werror is also specified. This avoids a "make check" failure on
Solaris when using Sun C 5.8.
Jim Meyering [Wed, 22 Dec 2010 09:29:30 +0000 (10:29 +0100)]
tests: mv/i-3: avoid false-positive failure on a slow/busy system
* tests/mv/i-3: Quadruple the timeout duration.
Without this, I saw an expired timeout on a heavily-loaded system.
Jim Meyering [Mon, 20 Dec 2010 13:27:08 +0000 (14:27 +0100)]
tests: avoid new false-positive failure on at least FreeBSD 8.1
* tests/mv/trailing-slash: Accommodate different diagnostic
on FreeBSD 8.1.
Paul Eggert [Tue, 21 Dec 2010 01:40:31 +0000 (17:40 -0800)]
who: omit useless definitions of MAXHOSTNAMELEN
This prevents a compilation failure on Solaris 8, GCC 4.4.2, with
"configure --enable-gcc-warnings".
* src/who.c (MAXHOSTNAMELEN): Remove; no longer needed.
* src/pinky.c: Likewise.
Pádraig Brady [Mon, 20 Dec 2010 07:49:07 +0000 (07:49 +0000)]
maint: fix a typo in sort --parallel help message
Also fix up Chen Guo's contacts
* src/sort.c (usage): Add a missing "of"
* THANKS: Add Chen Guo
* .mailmap: Add Chen Guo's UCLA address
Jim Meyering [Sun, 19 Dec 2010 20:23:19 +0000 (21:23 +0100)]
build: update gnulib: ACL-vs-Solaris 8 portability etc.
Paul Eggert [Sun, 19 Dec 2010 04:02:45 +0000 (20:02 -0800)]
tests: sync init.sh from gnulib
* tests/init.sh (setup_): Initialize fail=0 before invoking mktempd_.
Ensure that IFS is defined initially.
(mktempd_): Remove fail=0 initialization; no longer needed.
Pádraig Brady [Sat, 18 Dec 2010 03:25:49 +0000 (03:25 +0000)]
wc: fix a possible hang with --files0-from
* src/wc.c (main): exit when we get a read error on
the --files0-from file, rather than retrying and
spinning the CPU
Pádraig Brady [Sat, 18 Dec 2010 02:50:33 +0000 (02:50 +0000)]
cp: ensure backups are created when -T specified
* src/cp.c (do_copy): When -T is specified, initialize
the NEW_DST and SB variables, which are checked when
running: cp -T --force --backup file file
* tests/cp/backup-1: Add the -T case
Pádraig Brady [Sat, 18 Dec 2010 05:27:46 +0000 (05:27 +0000)]
sort: use at most 8 threads by default
* src/sort.c (main): If --parallel isn't specified,
restrict the number of threads to 8 by default.
If the --parallel option is specified, then
allow any number of threads to be set, independent
of the number of processors on the system.
* doc/coreutils.texi (sort invocation): Document the changes
to determining the number of threads to use.
Mention the memory overhead when using multiple threads.
* tests/misc/sort-spinlock-abuse: Allow single core
systems that support pthreads.
* tests/misc/sort-stale-thread-mem: Likewise.
* tests/misc/sort-unique-segv: Likewise.
* NEWS: Mention the change in behaviour.
Jim Meyering [Fri, 17 Dec 2010 20:58:33 +0000 (21:58 +0100)]
doc: split: add examples showing how to use the new option
* doc/coreutils.texi (split invocation): Add examples.
Jim Meyering [Fri, 17 Dec 2010 15:55:39 +0000 (16:55 +0100)]
tests: avoid spurious du/bigtime failure
* tests/du/bigtime: Map nonzero block count to 0.
Reported by Assaf Gordon.
* THANKS: Update.
Paul Eggert [Sat, 18 Dec 2010 06:39:47 +0000 (22:39 -0800)]
tests: set fail=0 by default
* tests/init.sh (setup_): Set fail=0. This was the intent as per
<http://lists.gnu.org/archive/html/bug-coreutils/2010-12/msg00058.html>
but the assignment in mktempd_ is ineffective, since mktempd_
is used inside `` and its assignments are in a subshell.
Jim Meyering [Wed, 15 Dec 2010 13:16:07 +0000 (14:16 +0100)]
build: update gnulib submodule to latest
Paul Eggert [Fri, 17 Dec 2010 06:31:56 +0000 (22:31 -0800)]
sort: do not generate thousands of subprocesses for 16-way merge
Without this change, tests/misc/sort-compress-hang would consume
more than 10,000 process slots on my RHEL 5.5 x86-64 server,
making it likely for other applications to fail due to lack of
process slots. With this change, the same benchmark causes 'sort'
to consume at most 19 process slots. The change also improved
wall-clock time by 2% and user+system time by 14% on that benchmark.
* NEWS: Document this.
* src/sort.c (MAX_PROCS_BEFORE_REAP): Remove.
(reap_exited): Renamed from reap_some; this is a more accurate name,
since "some" incorrectly implies that it reaps at least one process.
All uses changed.
(reap_some): New function: it *does* reap at least one process.
(pipe_fork): Do not allow more than NMERGE + 2 subprocesses.
(mergefps, sort): Omit check for exited processes: no longer needed,
and anyway the code consumed too much CPU per line when 2 < nprocs.
Paul Eggert [Thu, 16 Dec 2010 21:55:13 +0000 (13:55 -0800)]
sort: fix hang with sort --compress
* NEWS: Document this.
* src/sort.c (UNCOMPRESSED, UNREAPED, REAPED): New constants.
(struct tempnode): New member 'state', to hold these constants.
The pid member is now undefined if state == UNCOMPRESSED.
(struct sortfile): Replace member 'pid' with member 'temp'.
(uintptr): Remove.
(proctab_hasher, proctab_comparator, register_proc, delete_proc):
Proctab entries are now struct tempnode *, not pid_t, to handle
the case where multiple tempnode objects correspond to the same
pid. This avoids a race condition that can cause a hang.
(register_proc): Arg is now struct tempnode *, not pid_t. All
callers changed.
(delete_proc): Set tempnode state to REAPED.
(create_temp_file): No need to set pid member here; it's now
done when the pid is known.
(maybe_create_temp, create_temp): Remove PPID arg. Return struct
tempnode *, not char *. All callers changed.
(maybe_create_temp): Set node state to UNCOMPRESSED or UNREAPED.
No need to set node->pid to 0.
(open_temp): Replace NAME and PID args with a single TEMP arg.
All callers changed. Wait only for unreaped children.
(zaptemp): Wait for decompressor to finish before removing its
temporary-file input. This avoids .nfsXXXX hassles with NFS
and fixes a race (leading to a hang) regardless of NFS.
(open_input_files): Adjust to new way of dealing with temp files
and their subprocesses.
* tests/Makefile.am (TESTS): Add misc/sort-compress-hang.
* tests/misc/sort-compress-hang: New file.
Paul Eggert [Thu, 16 Dec 2010 08:03:29 +0000 (00:03 -0800)]
sort: don't dump core when merging from input twice
* NEWS: Document this.
* src/sort.c (avoid_trashing_input): The previous fix to this
function didn't fix all the problems with this code. Replace it
with something simpler: just copy the input file. This doesn't
change the number of files, so return void instead of the updated
file count. Caller changed.
* tests/misc/sort-merge-fdlimit: Test for the bug.
Jim Meyering [Tue, 14 Dec 2010 08:08:37 +0000 (09:08 +0100)]
doc: tail: semi-deprecate --sleep-interval and --max-unchanged-stats
Those options are useful only on systems that lack inotify support
and in the unusual event that a system with inotify support must
resort to polling.
* src/tail.c (usage): Note that the --max-unchanged-stats=N and
--sleep-interval=N options are rarely useful on systems with
inotify support.
* doc/coreutils.texi (tail invocation): Likewise.
Paul Eggert [Tue, 14 Dec 2010 19:09:32 +0000 (11:09 -0800)]
sort: fix very-unlikely buffer overrun when merging to input file
* src/sort.c (avoid_trashing_input): Fix a typo that could cause a
buffer overrun in theory. In practice this is extremely unlikely,
as it requires running out of file descriptors in a small merge,
presumably because some other process is hogging all the OS's file
descriptors.
Paul Eggert [Tue, 14 Dec 2010 18:21:49 +0000 (10:21 -0800)]
sort: document --compress reaper fixes
* NEWS: Document the --compress reaper fixes installed yesterday.
Paul Eggert [Tue, 14 Dec 2010 18:07:36 +0000 (10:07 -0800)]
tests: default to /tmp as the temporary directory
* tests/check.mk (TESTS_ENVIRONMENT): Default TMPDIR to /tmp,
rather than to the working directory; this is more common in
practice, which makes the tests more real-worldish; and it is
often faster. Also, it avoids some problems with NFS cleanups.
* tests/misc/sort-compress: Remove unnecessary code setting TMPDIR.
* tests/misc/sort-compress-proc: Likewise. Do the final sleep
only if TMPDIR is relative, which should be rarely given the
change to TESTS_ENVIRONMENT.
Paul Eggert [Tue, 14 Dec 2010 07:23:17 +0000 (23:23 -0800)]
sort: fix some --compress reaper bugs
* src/sort.c (uintptr): New type.
(enum procstate, struct procnode, update_proc): Remove.
(proctab_hasher, proctab_comparator, register_proc, wait_proc):
(reap_some): The proctab is now simply a hash of process-IDs
rather than of pointers to objects with reference counts and
states; this is smaller and faster and easier to understand.
(nprocs): Now pid_t, not size_t, since one cannot have more than
PID_MAX children.
(reap): If the argument is -1, wait; if 0 (a new value), do not.
Delete pid from proctab as needed. Ignore children that are not
in proctab, as they are from the program that exec'ed us and are
irrelevant to our success or failure.
(delete_proc, reap_all): New functions.
(open_temp): Register the child.
(sort): Clean up all children afterwards; without this patch,
'sort' sometimes missed failures in children due to race conditions.
* tests/Makefile.am (TESTS): Add misc/sort-compress-proc.
* tests/misc/sort-compress-proc: New file, to test for the
bugs fixed above.
Paul Eggert [Mon, 13 Dec 2010 18:02:06 +0000 (10:02 -0800)]
tests: typo fix
* tests/misc/sort-stale-thread-mem: Fix typo in comment.
Jim Meyering [Mon, 13 Dec 2010 07:19:12 +0000 (08:19 +0100)]
tests: tweak basic-1 to use warn_ rather than literal "exit 77"
* tests/install/basic-1 (just_built_dd): Use warn_, rather than
cat and exit 77.
Jim Meyering [Mon, 13 Dec 2010 07:07:25 +0000 (08:07 +0100)]
tests: mark new test as very expensive
* tests/misc/sort-stale-thread-mem: Don't initialize fail=0 here;
that is done in init.sh. This avoids a syntax-check failure.
Invoke "Exit $fail" at end, too.
Mark as a very expensive test.
Paul Eggert [Sun, 12 Dec 2010 21:38:19 +0000 (13:38 -0800)]
tests: test for access to stale thread memory
* tests/misc/sort-stale-thread-mem: New tests.
* tests/Makefile.am (TESTS): Add it.
Jim Meyering [Sat, 11 Dec 2010 17:44:57 +0000 (18:44 +0100)]
tests: avoid FP failure when run under valgrind
* tests/misc/printenv: Filter out LD_PRELOAD, as the comment
said, not LD_LIBRARY.
Jim Meyering [Sat, 11 Dec 2010 10:38:21 +0000 (11:38 +0100)]
sort: avoid segfault when using two or more threads
This change does not fix the actual bug. That was done by commit
c9db0ac6, "sort: preallocate merge tree nodes to heap". The fix
was to store each "node" structure on the heap, not on the stack.
Otherwise, a node from one thread's stack could be used in another
thread after the first thread had expired (via pthread_join).
This bug was very hard to trigger when using spinlocks, but
easier once we began using mutexes.
* NEWS (Bug fixes): Mention it.
For details, see http://debbugs.gnu.org/7597.
Jim Meyering [Sat, 11 Dec 2010 10:29:38 +0000 (11:29 +0100)]
sort: syntax cleanup
* src/sort.c (xfopen, debug_key, sortlines, sort, main): Adjust
formatting: fix misplaced braces, use consistent spacing,
split a 2-stmt line.
Paul Eggert [Sat, 11 Dec 2010 08:27:05 +0000 (00:27 -0800)]
sort: integer overflow checks in thread counts, etc.
* src/sort.c (specify_nthreads, merge_tree_init, init_node):
(queue_init, sortlines, struct thread_args, sort, main):
Use size_t, not unsigned long int, for thread counts, since thread
counts are now used to compute sizes.
(specify_nthreads): Check for size_t overflow.
(merge_tree_init, sort): Shorten name of local variable, for
readability.
(merge_tree_init): Move constants next to each other in product,
so that the constant folding is easier to see.
(init_node): Now static. Add 'restrict' only where it might
be helpful for compiler optimization.
(queue_init): 2nd arg is now nthreads, not "reserve", which is
a bit harder to follow. All uses changed.
(struct thread_args): Rename lo_child to is_lo_child, so that
it's obvious to the reader when we're talking about this boolean
as opposed to the new lo_child member of the other structure.
All uses changed.
(sort): Remove unused local variable end_node.
(main): Don't allow large thread counts to cause undefined behavior
later, due to integer overflow.
Chen Guo [Fri, 10 Dec 2010 21:13:36 +0000 (13:13 -0800)]
sort: preallocate merge tree nodes to heap.
* src/sort.c: (merge_tree_init) New function. Allocates memory for
merge tree nodes.
(merge_tree_destory) New function.
(init_node) New function.
(sortlines) Refactor node creation code to init_node. Remove now
superfluous arguments. All callers changed.
(sort) Initialize/destory merge tree. Refactor root node creation
to merge_tree_init.
Paul Eggert [Sat, 11 Dec 2010 04:52:04 +0000 (20:52 -0800)]
sort: comment fix
* src/sort.c: Comment fix re spin locks.
Chen Guo [Mon, 6 Dec 2010 08:15:42 +0000 (00:15 -0800)]
sort: use mutexes, not spinlocks (avoid busy loop on blocked output)
Running a command like this on a multi-core system
sort < big-file | less
would peg all processors at near 100% utilization.
* src/sort.c: (struct merge_node) Change member lock to mutex.
All uses changed.
* tests/Makefile.am (XFAIL_TESTS): Remove definition, now that
this test passes once again. I.e., the sort-spinlock-abuse test
no longer fails.
* NEWS (Bug reports): Mention this.
Reported by DJ Lucas in http://debbugs.gnu.org/7489.
Pádraig Brady [Wed, 8 Dec 2010 08:33:15 +0000 (08:33 +0000)]
split: fix a case where --elide-empty causes invalid chunking
When -n l/N is used and long lines are present that both
span partitions and multiple buffers, one would get
inconsistent chunk sizes.
* src/split.c (main): Add a new undocumented ---io-blksize option
to support full testing with varied buffer sizes.
(cwrite): Refactor most handling of --elide-empty to here.
(bytes_split): Remove handling of --elide-empty.
(lines_chunk_split): Likewise. The specific issue here
was the first handling of elide_empty_files interfered
with the replenishing of the input buffer.
* test/misc/split-lchunk: Add -e and the new ---io-blksize
combinations to the test.
Jim Meyering [Sun, 5 Dec 2010 16:07:35 +0000 (17:07 +0100)]
tests: remove useless definition of $SORT in sort-compress
* tests/misc/sort-compress (SORT): Remove unused definition.
Jim Meyering [Sat, 4 Dec 2010 10:52:21 +0000 (11:52 +0100)]
tests: make it harder to lose a race in spinlock-abuse
* tests/misc/sort-spinlock-abuse: On a busy system, with only 12
pauses of length 0.1 seconds, the buggy (busy-spinlock blocked)
sort would fail to accumulate 1 second of CPU time, and hence
would mistakenly pass. Increase from 12 to 50.
Paul Eggert [Fri, 3 Dec 2010 23:39:50 +0000 (15:39 -0800)]
sort: merge_queue -> queue
* src/sort.c (struct thread_args, sortlines_thread, sortlines, sort):
Rename "merge_queue" to "queue", for consistency with other functions
that just use the name "queue" for these things.
Paul Eggert [Fri, 3 Dec 2010 23:23:43 +0000 (15:23 -0800)]
sort: clarify queue_check_insert
* src/sort.c (queue_check_insert): Clarify body a bit, and remove
no-longer-needed comment.
Paul Eggert [Fri, 3 Dec 2010 23:11:46 +0000 (15:11 -0800)]
sort: fix problems with merge node dest pointer
* src/sort.c (mergelines_node): Return void, not size_t. All
callers changed. Change *node->dest here, not in caller.
Do not change node->dest: it's not needed and could cause problems
on (mostly theoretical) hosts that do not allow adding integers to
null pointers.
(queue_check_insert_parent): Omit MERGED parameter; no longer needed.
All callers changed.
Paul Eggert [Fri, 3 Dec 2010 23:04:31 +0000 (15:04 -0800)]
sort: simplify write_unique
* src/sort.c (write_unique): Simplify slightly so that there is
just one call to write_line, not two.
Paul Eggert [Fri, 3 Dec 2010 23:01:21 +0000 (15:01 -0800)]
sort: put queue arg first
* src/sort.c (queue_check_insert, queue_check_insert_parent): Make
the queue arg first, for consistency with other functions such as
queue_insert that put the queue arg first. Rename from
check_insert and update_parent, respectively. All callers
changed.
Paul Eggert [Fri, 3 Dec 2010 22:39:23 +0000 (14:39 -0800)]
sort: tune struct_merge_node slightly
* src/sort.c (struct merge_node): 'lock' is now the actual lock,
not a pointer to the lock; there's no need for indirection here.
Make 'level' unsigned int instead of size_t, since it is a
bit-shift count; also, move it next to a bool so that it's more
likely to take less space. All uses changed.
(sortlines, sort): Spell out initialization instead of using an
initializer. This makes the initializer a bit easier to understand,
and avoids unnecessary stores into the spin lock.
Paul Eggert [Fri, 3 Dec 2010 22:27:02 +0000 (14:27 -0800)]
sort: Clarify comments
* src/sort.c: Improve comments a bit.
Paul Eggert [Fri, 3 Dec 2010 19:08:48 +0000 (11:08 -0800)]
tests: cleanup rm -rf fails under NFS
This problem was observed on RHEL 5.5 x86-64 when running as a
client of a NetApp FAS2050.
* tests/cp/cp-mv-backup: Don't leave a file descriptor open to
a file in a directory that will be cleaned up with "rm -rf".
Under NFS, when the rm unlinks that file, it is instead renamed
to .nfsXXXX and then rm cannot remove the parent directory,
and the test fails.
* tests/cp/same-file: Likewise.
Paul Eggert [Fri, 3 Dec 2010 17:55:48 +0000 (09:55 -0800)]
cp: fix bug with fine-grained src to nearby coarse-grained dest
The actual fix is in gnulib's lib/utimecmp.c.
* NEWS: Document fix.
Paul Eggert [Fri, 3 Dec 2010 17:52:44 +0000 (09:52 -0800)]
build: update gnulib submodule to latest
Paul Eggert [Thu, 2 Dec 2010 05:50:00 +0000 (21:50 -0800)]
sort: fix bug on 64-bit hosts with at least 32768 processors
* src/sort.c (MAX_MERGE): Avoid integer overflow when on a machine
with (say) 32-bit int and 64-bit size_t and when level == 15.
Without this fix, on such a machine with 32768 or more processors,
the level computation could overflow on large input, and this
would result in division by zero.
Jim Meyering [Sun, 28 Nov 2010 11:59:38 +0000 (12:59 +0100)]
tests: add test for parallel sort -u segfault bug
* tests/misc/sort-unique-segv: New file.
* tests/Makefile.am (TESTS): Add it.
Paul Eggert [Tue, 30 Nov 2010 21:30:12 +0000 (22:30 +0100)]
sort -u: fix a thread-race pointer corruption bug
* src/sort.c (write_unique): Save the entire "struct line", not
just a pointer to one. Otherwise, with a multi-thread run,
sometimes, with some inputs, fillbuf would would win a race
and clobber a "saved->text" pointer in one thread just before
it was dereferenced in a comparison in another thread.
* NEWS (Bug fixes): Mention it.
Jim Meyering [Sat, 27 Nov 2010 14:58:51 +0000 (15:58 +0100)]
tests: don't let the OMP_NUM_THREADS envvar affect sort tests
* tests/envvar-check (vars): Add OMP_NUM_THREADS.
Jim Meyering [Sat, 27 Nov 2010 14:50:43 +0000 (15:50 +0100)]
tests: test for parallel sort spinlock abuse
* tests/misc/sort-spinlock-abuse: New file.
* tests/Makefile.am (TESTS): Add it.
(XFAIL_TESTS): Mark this as an expected-to-fail (for now) test.
Pádraig Brady [Thu, 25 Nov 2010 19:16:00 +0000 (19:16 +0000)]
tsort: suppress a valgrind memory leak warning
* src/tsort.c (tsort): Unconditionally invoking the free()
doesn't increase scalability, so do it only with -Dlint
Pádraig Brady [Wed, 24 Nov 2010 07:50:01 +0000 (07:50 +0000)]
maint: update the valgrind support script
* README-valgrind: Include the "noinst" programs in
those wrapped by valgrind. Update $PATH in check.mk
rather than Makefile.am. Make wrapper scripts
work when suppressions not setup. Keep lines < 80 chars.
Paul Eggert [Mon, 22 Nov 2010 02:50:41 +0000 (18:50 -0800)]
cp: give a better diagnostic for nonexistent dest/
This patch was written by Jim Meyering and myself.
* src/copy.c (copy_reg): Turn EISDIR to ENOTDIR to improve the
quality of diagnostics for commands like "cp a nosuch/". Reported
by Марк Коренберг and Alan Curry in the thread starting at:
http://lists.gnu.org/archive/html/bug-coreutils/2010-11/msg00178.html
* THANKS: Update.
* tests/mv/trailing-slash: Add a test.
Chen Guo [Fri, 8 Jan 2010 11:42:27 +0000 (03:42 -0800)]
split: add --number to generate a particular number of files
* src/split.c (usage, long_options, main): New options --number,
--unbuffered, --elide-empty-files.
(set_suffix_length): New function to auto increase suffix length
to handle a specified number of files.
(create): New function. Refactored from cwrite() and ofile_open().
(bytes_split): Add max_files argument to support byte chunking.
(lines_chunk_split): New function. Split file into chunks of lines.
(bytes_chunk_extract): New function. Extract a chunk of file.
(of_info): New struct. Used by functions lines_rr and ofile_open
to keep track of file descriptors associated with output files.
(ofile_open): New function. Shuffle file descriptors when there
are more output files than available file descriptors.
(lines_rr): New function to distribute lines round-robin to files.
(chunk_parse): New function. Parses K/N syntax.
* tests/misc/split-bchunk: New test for byte chunking.
* tests/misc/split-lchunk: New test for line delimited chunking.
* tests/misc/split-rchunk: New test for round-robin chunking.
* tests/Makefile.am: Reference new tests.
* tests/misc/split-fail: Add failure scenarios for new options.
* tests/misc/split-l: Fix a typo. s/ln/split/.
* doc/coreutils.texi (split invocation): Document --number.
* NEWS: Mention the new feature.
* .mailmap: Map new email address for shortlog.
Signed-off-by: Pádraig Brady <P@draigBrady.com>
Jim Meyering [Fri, 19 Nov 2010 13:29:32 +0000 (14:29 +0100)]
build: update gnulib to fix a syntax error in a test
Jim Meyering [Thu, 18 Nov 2010 20:16:27 +0000 (21:16 +0100)]
maint: avoid NEWS-related syntax-check failure
* NEWS: Add 2nd blank line to separate latest changes from
those of 8.7, to avoid syntax-check failure.
Paul Eggert [Thu, 18 Nov 2010 17:50:49 +0000 (09:50 -0800)]
od: fix bugs in displaying floating-point values
* NEWS: Describe patch.
* bootstrap.conf (gnulib_modules): Add ftoastr.
* src/od.c: Include ftoastr.h, not float.h.
(FLT_DIG, DBL_DIG): Remove. No need to verify LDBL_DIG.
(FMT_BYTES_ALLOCATED): No need to worry about floating point now,
since this format is no longer used for floating point.
(PRINT_FIELDS): New macro, with most of the guts of the old PRINT_TYPE.
(PRINT_TYPE): Rewrite to use PRINT_FIELDS.
(PRINT_FLOATTYPE): New macro. This uses the new functions from
ftoastr.
(print_float, print_double, print_long_double): Reimplement
using PRINT_FLOATTYPE.
(decode_one_format): Calculate field widths based on ftoastr-supplied
macros.
* tests/Makefile.am (TESTS): Add misc/od-float.
* tests/misc/od-float: New file.
Paul Eggert [Thu, 18 Nov 2010 17:33:26 +0000 (09:33 -0800)]
build: update gnulib submodule to latest
Jim Meyering [Thu, 18 Nov 2010 05:29:54 +0000 (06:29 +0100)]
maint: syntax-check: prevent new $VERBOSE/--version tests
* cfg.mk (sc_prohibit_verbose_version): New rule.
Jim Meyering [Tue, 16 Nov 2010 07:32:00 +0000 (08:32 +0100)]
tests: convert tests/misc/selinux manually
Jim Meyering [Wed, 17 Nov 2010 20:38:38 +0000 (21:38 +0100)]
tests: convert the multi-prog $VERBOSE/--version uses
E.g.,
-test "$VERBOSE" = yes && { env -- pwd --version; readlink --version; }
+print_ver_ pwd readlink
-test "$VERBOSE" = yes && { stdbuf --version; mv --version; }
+print_ver_ stdbuf mv
Use this command:
git grep -l 'VERBOSE.*--version'|xargs perl -ni \
-e '/^test "\$VERBOSE" = yes && { .*--version/ or print,next;' \
-e 's/env -- //g;' \
-e 's/test "\$VERBOSE" = yes && { /print_ver_ /;' \
-e ' s/(\w+) --version;/$1/g; s/ *}$//; print'
Jim Meyering [Wed, 17 Nov 2010 20:42:13 +0000 (21:42 +0100)]
tests: convert "... env -- prog --version" uses
E.g.,
-test "$VERBOSE" = yes && env -- pwd --version
+print_ver_ pwd
git grep -l 'VERBOSE.*--version'|xargs perl -pi -e \
's/test "\$VERBOSE" = yes && env -- (\w+) --version/print_ver_ $1/'
Jim Meyering [Wed, 17 Nov 2010 20:35:31 +0000 (21:35 +0100)]
tests: substitute the single-program $VERBOSE/--version uses
Automatically make all of the changes like this:
-test "$VERBOSE" = yes && chgrp --version
+print_ver_ chgrp
git grep -l 'VERBOSE.*--version'|xargs perl -pi -e \
's/test "\$VERBOSE" = yes && (\w+) --version/print_ver_ $1/'
Jim Meyering [Tue, 16 Nov 2010 07:05:11 +0000 (08:05 +0100)]
tests: factor out VERBOSE-only --version-printing code
* tests/init.cfg (print_ver_): New function.
Jim Meyering [Sun, 14 Nov 2010 11:13:58 +0000 (12:13 +0100)]
tests: remove test-lib.sh; now all tests use gnulib's init.sh
* tests/test-lib.sh: Remove file. No longer used.
* tests/Makefile.am (EXTRA_DIST): Remove it here, too.
* tests/sample-test: Correct a comment.
Jim Meyering [Sun, 14 Nov 2010 11:07:57 +0000 (12:07 +0100)]
tests: convert remaining uses of test-lib.sh to init.sh
RHS='. "\${srcdir=.}/init.sh"; path_prepend_ ../src'
git grep -l test-lib.sh \
| xargs perl -pi -e 's,^\. \$srcdir/test-lib\.sh$,'"$RHS",
Jim Meyering [Sun, 14 Nov 2010 11:02:39 +0000 (12:02 +0100)]
tests: convert 'if test "$VERBOSE" = yes; then' to test ... &&
Jim Meyering [Sun, 14 Nov 2010 10:05:05 +0000 (11:05 +0100)]
tests: convert first batch of tests from test-lib.sh to init.sh
Initially, I did this,
git grep -l srcdir/test-lib.sh|xargs perl -p0i -e '~180-byte script'
but that line would have been much longer than the maximum permitted
by coreutils' commit hook, and wasn't readable besides, so here's a
more readable version:
lhs=$(printf '%s\\n' \
'if test "$VERBOSE" = yes; then' \
' set -x' \
' touch --version' \
'fi' \
'' \
'. $srcdir/test-lib.sh' \
| sed 's/\$/\\\$/g;s/touch/(\\w+)/')
rhs=$(printf '%s\\n' \
'. "${srcdir=.}/init.sh"; path_prepend_ ../src' \
'test "$VERBOSE" = yes && FIXME --version' \
| sed 's/\$/\\\$/g;s/FIXME/\$1/')
git grep -l srcdir/test-lib.sh|xargs perl -p0i -e "s,$lhs,$rhs,"
Jim Meyering [Sun, 14 Nov 2010 10:01:25 +0000 (11:01 +0100)]
tests: move a comment so automated conversion works
Jim Meyering [Sun, 14 Nov 2010 09:16:27 +0000 (10:16 +0100)]
maint: add a NEWS-checking syntax-check rule
* cfg.mk (sc_NEWS_two_empty_lines): New syntax-check rule.
Pádraig Brady [Tue, 16 Nov 2010 07:32:32 +0000 (07:32 +0000)]
build: add `patch` as a bootstrap dependency
* bootstrap.conf (buildreq): require `patch` as it's used
by gnulib-tool to apply local diffs to gnulib modules
Pádraig Brady [Sun, 7 Nov 2010 03:09:38 +0000 (03:09 +0000)]
split: fail immediately if impossible to create a large file
* src/split.c (main): Error if -[bC] value > OFF_T_MAX
* tests/misc/split-fail: Adjust for the new lower limits