* lib/Automake/Config.in (perl_threads): New global.
* automake.in: Use it. If the perl supports interpreter-based
threading, then use `threads' and `Thread::Queue'.
(handle_makefile, handle_makefiles_serial): New functions,
factored out from main.
(get_number_of_threads): New function, compute number of threads
to use, based on environment variable `AUTOMAKE_JOBS' and number
of independent makefiles.
(handle_makefiles_threaded): New function. Spawn threads, use
thread queue to distribute handling the different makefiles.
Collect $exit_code values from threads.
(main): Use new functions.
* aclocal.in: No threads here.
* configure.ac: Substitute PERL_THREADS; enabled with perl >=
5.7.2 and when ithreads are available.
* bootstrap (dosubst): Likewise.
* Makefile.am (do_subst): Likewise.
* lib/Automake/Makefile.am (do_subst): Likewise.
* lib/Automake/ChannelDefs.pm: Use `Automake::Config' and
`threads'.
(verb): Prepend thread ID (tid) to verbose messages.
* lib/Automake/Channels.pm (msg): Before exiting, flush stderr,
needed for worker threads.
* lib/Automake/tests/Makefile.am (TESTS_ENVIRONMENT): Also
include the build tree path, so Config.pm is found.
* tests/parallel-am.test: New test.
* tests/Makefile.am: Update.
* doc/automake.texi (Invoking Automake): Document AUTOMAKE_JOBS.
* NEWS, THANKS: Update.
Report about long execution times by Joakim Tjernlund and others.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
2008-10-26 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
+ Parallel automake execution: AUTOMAKE_JOBS.
+ * lib/Automake/Config.in (perl_threads): New global.
+ * automake.in: Use it. If the perl supports interpreter-based
+ threading, then use `threads' and `Thread::Queue'.
+ (handle_makefile, handle_makefiles_serial): New functions,
+ factored out from main.
+ (get_number_of_threads): New function, compute number of threads
+ to use, based on environment variable `AUTOMAKE_JOBS' and number
+ of independent makefiles.
+ (handle_makefiles_threaded): New function. Spawn threads, use
+ thread queue to distribute handling the different makefiles.
+ Collect $exit_code values from threads.
+ (main): Use new functions.
+ * aclocal.in: No threads here.
+ * configure.ac: Substitute PERL_THREADS; enabled with perl >=
+ 5.7.2 and when ithreads are available.
+ * bootstrap (dosubst): Likewise.
+ * Makefile.am (do_subst): Likewise.
+ * lib/Automake/Makefile.am (do_subst): Likewise.
+ * lib/Automake/ChannelDefs.pm: Use `Automake::Config' and
+ `threads'.
+ (verb): Prepend thread ID (tid) to verbose messages.
+ * lib/Automake/Channels.pm (msg): Before exiting, flush stderr,
+ needed for worker threads.
+ * lib/Automake/tests/Makefile.am (TESTS_ENVIRONMENT): Also
+ include the build tree path, so Config.pm is found.
+ * tests/parallel-am.test: New test.
+ * tests/Makefile.am: Update.
+ * doc/automake.texi (Invoking Automake): Document AUTOMAKE_JOBS.
+ * NEWS, THANKS: Update.
+ Report about long execution times by Joakim Tjernlund and others.
+
+2008-10-26 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
+
Minor file checking optimization: set_dir_cache_file.
* lib/Automake/FileUtils.pm (set_dir_cache_file): New function.
* automake.in (require_file_internal): Instead of resetting the
-e 's,[@]PACKAGE[@],$(PACKAGE),g' \
-e 's,[@]PATH_SEPARATOR[@],$(PATH_SEPARATOR),g' \
-e 's,[@]PERL[@],$(PERL),g' \
+ -e 's,[@]PERL_THREADS[@],$(PERL_THREADS),g' \
-e 's,[@]SHELL[@],$(SHELL),g' \
-e 's,[@]VERSION[@],$(VERSION),g' \
-e 's,[@]configure_input[@],Generated from $@.in; do not edit by hand.,g' \
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PERL = @PERL@
+PERL_THREADS = @PERL_THREADS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
-e 's,[@]PACKAGE[@],$(PACKAGE),g' \
-e 's,[@]PATH_SEPARATOR[@],$(PATH_SEPARATOR),g' \
-e 's,[@]PERL[@],$(PERL),g' \
+ -e 's,[@]PERL_THREADS[@],$(PERL_THREADS),g' \
-e 's,[@]SHELL[@],$(SHELL),g' \
-e 's,[@]VERSION[@],$(VERSION),g' \
-e 's,[@]configure_input[@],Generated from $@.in; do not edit by hand.,g' \
(and new in Automake 1.10) is degraded to a warning. This helps
in the common case where the Autoconf versions used are compatible.
-* Change to Libtool support:
+* Changes to automake:
+
+ - The automake program can run multiple threads for creating most
+ Makefile.in files concurrently, if at least Perl 5.7.2 is available
+ with interpreter-based threads enabled. Set the environment variable
+ AUTOMAKE_JOBS to the maximum number of threads to use, in order to
+ enable this experimental feature.
+
+* Changes to Libtool support:
- Libtool generic flags are now passed to the install and uninstall
modes as well.
Jerome Santini santini@chambord.univ-orleans.fr
Jesse Chisholm jesse@ctc.volant.org
Jim Meyering meyering@na-net.ornl.gov
+Joakim Tjernlund Joakim.Tjernlund@transmode.se
Jochen Kuepper jochen@uni-duesseldorf.de
Joel N. Weber II nemo@koa.iolani.honolulu.hi.us
Joerg-Martin Schwarz jms@jms.prima.ruhr.de
# Some globals.
+# We do not operate in threaded mode.
+$perl_threads = 0;
+
# Include paths for searching macros. We search macros in this order:
# user-supplied directories first, then the directory containing the
# automake macros, and finally the system-wide directories for
use strict;
use Automake::Config;
+if ($perl_threads)
+ {
+ use threads;
+ use Thread::Queue;
+ }
use Automake::General;
use Automake::XFile;
use Automake::Channels;
if $errspec && ! @input_files;
}
+
+# handle_makefile ($MAKEFILE_IN)
+# ------------------------------
+# Deal with $MAKEFILE_IN.
+sub handle_makefile ($)
+{
+ my ($file) = @_;
+ ($am_file = $file) =~ s/\.in$//;
+ if (! -f ($am_file . '.am'))
+ {
+ error "`$am_file.am' does not exist";
+ }
+ else
+ {
+ # Any warning setting now local to this Makefile.am.
+ dup_channel_setup;
+
+ generate_makefile ($am_file . '.am', $file);
+
+ # Back out any warning setting.
+ drop_channel_setup;
+ }
+}
+
+# handle_makefiles_serial ()
+# --------------------------
+# Deal with all makefiles, without threads.
+sub handle_makefiles_serial ()
+{
+ foreach my $file (@input_files)
+ {
+ handle_makefile ($file);
+ }
+}
+
+# get_number_of_threads ()
+# ------------------------
+# Logic for deciding how many worker threads to use.
+sub get_number_of_threads
+{
+ my $nthreads = $ENV{'AUTOMAKE_JOBS'} || 0;
+
+ $nthreads = 0
+ unless $nthreads =~ /^[0-9]+$/;
+
+ # It doesn't make sense to use more threads than makefiles,
+ my $max_threads = @input_files;
+
+ # but a single worker thread is helpful for exposing bugs.
+ if ($automake_will_process_aux_dir && $max_threads > 1)
+ {
+ $max_threads--;
+ }
+ if ($nthreads > $max_threads)
+ {
+ $nthreads = $max_threads;
+ }
+
+ # We cannot deal with --add-missing (yet).
+ $nthreads = 0
+ if ($add_missing);
+
+ return $nthreads;
+}
+
+# handle_makefiles_threaded ($NTHREADS)
+# -------------------------------------
+# Deal with all makefiles, using threads. The general strategy is to
+# spawn NTHREADS worker threads, and dispatch makefiles to them.
+sub handle_makefiles_threaded ($)
+{
+ my ($nthreads) = @_;
+
+ my @queued_input_files = @input_files;
+ my $last_input_file = undef;
+ if ($automake_will_process_aux_dir)
+ {
+ $last_input_file = pop @queued_input_files;
+ }
+
+ # The file queue distributes all makefiles.
+ my $file_queue = Thread::Queue->new;
+
+ verb "spawning $nthreads worker threads";
+ my @threads = (1 .. $nthreads);
+ foreach my $t (@threads)
+ {
+ $t = threads->new (sub
+ {
+ while (my $file = $file_queue->dequeue)
+ {
+ verb "handling $file";
+ handle_makefile ($file);
+ }
+ return $exit_code;
+ });
+ }
+
+ # Queue all normal makefiles.
+ verb "queuing " . @queued_input_files . " input files";
+ $file_queue->enqueue (@queued_input_files, (undef) x @threads);
+
+ foreach my $t (@threads)
+ {
+ my @exit_thread = $t->join;
+ $exit_code = $exit_thread[0]
+ if ($exit_thread[0] > $exit_code);
+ }
+
+ # The master processes the last file.
+ if ($automake_will_process_aux_dir)
+ {
+ verb "processing last input file";
+ handle_makefile ($last_input_file);
+ }
+}
+
################################################################
# Parse the WARNINGS environment variable.
fatal ("no `Makefile.am' found for any configure output$msg");
}
-# Now do all the work on each file.
-foreach my $file (@input_files)
- {
- ($am_file = $file) =~ s/\.in$//;
- if (! -f ($am_file . '.am'))
- {
- error "`$am_file.am' does not exist";
- }
- else
- {
- # Any warning setting now local to this Makefile.am.
- dup_channel_setup;
-
- generate_makefile ($am_file . '.am', $file);
+my $nthreads = get_number_of_threads ();
- # Back out any warning setting.
- drop_channel_setup;
- }
+if ($perl_threads && $nthreads >= 1)
+ {
+ handle_makefiles_threaded ($nthreads);
+ }
+else
+ {
+ handle_makefiles_serial ();
}
exit $exit_code;
VERSION=`sed -ne '/AC_INIT/s/^[^[]*\[[^[]*\[\([^]]*\)\].*$/\1/p' configure.ac`
PACKAGE=automake
datadir=.
+PERL_THREADS=0
# Override SHELL. This is required on DJGPP so that Perl's system()
# uses bash, not COMMAND.COM which doesn't quote arguments properly.
sed -e "s%@APIVERSION@%$APIVERSION%g" \
-e "s%@PACKAGE@%$PACKAGE%g" \
-e "s%@PERL@%$PERL%g" \
+ -e "s%@PERL_THREADS@%$PERL_THREADS%g" \
-e "s%@SHELL@%$BOOTSTRAP_SHELL%g" \
-e "s%@VERSION@%$VERSION%g" \
-e "s%@datadir@%$datadir%g" \
LN
HELP2MAN
TEX
+PERL_THREADS
PERL
pkgvdatadir
APIVERSION
{ (exit 1); exit 1; }; }
}
+# We require ithreads support, and version 5.7.2 for CLONE.
+{ $as_echo "$as_me:$LINENO: checking whether $PERL supports ithreads" >&5
+$as_echo_n "checking whether $PERL supports ithreads... " >&6; }
+if test "${am_cv_prog_PERL_ithreads+set}" = set; then
+ $as_echo_n "(cached) " >&6
+else
+ if $PERL -e '
+ require 5.007_002;
+ use Config;
+ if ($Config{useithreads})
+ {
+ use threads;
+ use Thread::Queue;
+ exit 0;
+ }
+ exit 1;' >&5 2>&1
+then
+ am_cv_prog_PERL_ithreads=yes
+else
+ am_cv_prog_PERL_ithreads=no
+fi
+fi
+{ $as_echo "$as_me:$LINENO: result: $am_cv_prog_PERL_ithreads" >&5
+$as_echo "$am_cv_prog_PERL_ithreads" >&6; }
+if test $am_cv_prog_PERL_ithreads = yes; then
+ PERL_THREADS=1;
+else
+ PERL_THREADS=0;
+fi
+
+
# The test suite will skip some tests if tex is absent.
# Extract the first word of "tex", so it can be a program name with args.
set dummy tex; ac_word=$2
./configure PERL=/path/to/perl])
}
+# We require ithreads support, and version 5.7.2 for CLONE.
+AC_CACHE_CHECK([whether $PERL supports ithreads], [am_cv_prog_PERL_ithreads],
+[if $PERL -e '
+ require 5.007_002;
+ use Config;
+ if ($Config{useithreads})
+ {
+ use threads;
+ use Thread::Queue;
+ exit 0;
+ }
+ exit 1;' >&AS_MESSAGE_LOG_FD 2>&1
+then
+ am_cv_prog_PERL_ithreads=yes
+else
+ am_cv_prog_PERL_ithreads=no
+fi])
+if test $am_cv_prog_PERL_ithreads = yes; then
+ PERL_THREADS=1;
+else
+ PERL_THREADS=0;
+fi
+AC_SUBST([PERL_THREADS])
+
# The test suite will skip some tests if tex is absent.
AC_CHECK_PROG([TEX], [tex], [tex])
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PERL = @PERL@
+PERL_THREADS = @PERL_THREADS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
@end table
+@vindex AUTOMAKE_JOBS
+If the environment variable @env{AUTOMAKE_JOBS} contains a positive
+number, it is taken as the maximum number of Perl threads to use in
+@command{automake} for generating multiple @file{Makefile.in} files
+concurrently. This is an experimental feature.
+
@node configure
@chapter Scanning @file{configure.ac}
package Automake::ChannelDefs;
+use Automake::Config;
+if ($perl_threads)
+ {
+ use threads;
+ }
use Automake::Channels;
=head1 NAME
sub verb ($;%)
{
my ($msg, %opts) = @_;
+ $msg = "thread " . threads->tid . ": " . $msg
+ if $perl_threads;
msg 'verb', '', $msg, %opts;
}
-# Copyright (C) 2002, 2004, 2006 Free Software Foundation, Inc.
+# Copyright (C) 2002, 2004, 2006, 2008 Free Software Foundation, Inc.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# Die on fatal messages.
confess if $opts{'backtrace'};
- exit $exit_code if $opts{'type'} eq 'fatal';
+ if ($opts{'type'} eq 'fatal')
+ {
+ # flush messages explicitly here, needed in worker threads.
+ STDERR->flush;
+ exit $exit_code;
+ }
}
}
-# Copyright (C) 2003, 2004 Free Software Foundation, Inc. -*- Perl -*-
+# Copyright (C) 2003, 2004, 2008 Free Software Foundation, Inc. -*- Perl -*-
# @configure_input@
# This program is free software; you can redistribute it and/or modify
require Exporter;
our @ISA = qw (Exporter);
-our @EXPORT = qw ($APIVERSION $PACKAGE $VERSION $libdir);
+our @EXPORT = qw ($APIVERSION $PACKAGE $VERSION $libdir $perl_threads);
# Parameters set by configure. Not to be changed. NOTE: assign
# VERSION as string so that e.g. version 0.30 will print correctly.
our $PACKAGE = '@PACKAGE@';
our $VERSION = '@VERSION@';
our $libdir = '@datadir@/@PACKAGE@-@APIVERSION@';
+our $perl_threads = @PERL_THREADS@;
1;;
## Process this file with automake to create Makefile.in
-## Copyright (C) 2001, 2002, 2003, 2004 Free Software Foundation, Inc.
+## Copyright (C) 2001, 2002, 2003, 2004, 2008 Free Software Foundation, Inc.
## This program is free software; you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
-e 's,[@]APIVERSION[@],$(APIVERSION),g' \
-e 's,[@]PACKAGE[@],$(PACKAGE),g' \
-e 's,[@]PERL[@],$(PERL),g' \
+ -e 's,[@]PERL_THREADS[@],$(PERL_THREADS),g' \
-e 's,[@]SHELL[@],$(SHELL),g' \
-e 's,[@]VERSION[@],$(VERSION),g' \
-e "s,[@]configure_input[@],Generated from $$in.in; do not edit by hand.,g" \
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PERL = @PERL@
+PERL_THREADS = @PERL_THREADS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
-e 's,[@]APIVERSION[@],$(APIVERSION),g' \
-e 's,[@]PACKAGE[@],$(PACKAGE),g' \
-e 's,[@]PERL[@],$(PERL),g' \
+ -e 's,[@]PERL_THREADS[@],$(PERL_THREADS),g' \
-e 's,[@]SHELL[@],$(SHELL),g' \
-e 's,[@]VERSION[@],$(VERSION),g' \
-e "s,[@]configure_input[@],Generated from $$in.in; do not edit by hand.,g" \
## You should have received a copy of the GNU General Public License
## along with this program. If not, see <http://www.gnu.org/licenses/>.
-TESTS_ENVIRONMENT = $(PERL) -Mstrict -I $(top_srcdir)/lib -w
+TESTS_ENVIRONMENT = $(PERL) -Mstrict -I ../.. -I $(top_srcdir)/lib -w
TESTS = \
Condition.pl \
Condition-t.pl \
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PERL = @PERL@
+PERL_THREADS = @PERL_THREADS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
top_build_prefix = @top_build_prefix@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-TESTS_ENVIRONMENT = $(PERL) -Mstrict -I $(top_srcdir)/lib -w
+TESTS_ENVIRONMENT = $(PERL) -Mstrict -I ../.. -I $(top_srcdir)/lib -w
TESTS = \
Condition.pl \
Condition-t.pl \
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PERL = @PERL@
+PERL_THREADS = @PERL_THREADS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PERL = @PERL@
+PERL_THREADS = @PERL_THREADS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PERL = @PERL@
+PERL_THREADS = @PERL_THREADS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
output13.test \
output-order.test \
overrid.test \
+parallel-am.test \
parse.test \
percent.test \
percent2.test \
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PERL = @PERL@
+PERL_THREADS = @PERL_THREADS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
output13.test \
output-order.test \
overrid.test \
+parallel-am.test \
parse.test \
percent.test \
percent2.test \
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2008 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Test parallel automake execution.
+
+# There are several hypotheses to be tested: Independently of the number
+# of threads used by automake,
+# 0) the generated Makefile.in files must be identical without --add-missing,
+# 1) the Makefile.in that distributes auxiliary files must be generated
+# after all other ones, so all installed aux files are caught,
+# 2) normal automake output should have identical content and be ordered
+# in the same way, when --add-missing is not passed, or when
+# --add-missing is passed but there are no concurrent file requirements
+# (i.e., two Makefile.am files call for the same needed aux file)
+# 3) normal automake output should be identical and ordered in the same way
+# with --add-missing, even with concurrent file requirements, and the
+# installation of aux files should be race-free,
+# 4) warning and normal error output should be identical, in that duplicate
+# warnings should be omitted in the same way as without threads,
+# 5) fatal error and debug messages could be identical. This is not
+# intended, though.
+#
+# This test checks (0).
+
+. ./defs || Exit 1
+
+set -e
+
+cat > configure.in << 'END'
+AC_INIT([parallel-am], [1.0])
+AC_CONFIG_AUX_DIR([build-aux])
+AM_INIT_AUTOMAKE
+AC_PROG_CC
+AM_PATH_LISPDIR
+AM_PATH_PYTHON
+AC_CONFIG_FILES([Makefile])
+END
+
+cat > Makefile.am << 'END'
+SUBDIRS =
+END
+
+list='1 2 3 4 5 6 7 8 9'
+for i in $list; do
+ echo "AC_CONFIG_FILES([sub$i/Makefile])" >> configure.in
+ echo "SUBDIRS += sub$i" >> Makefile.am
+ mkdir sub$i
+ echo > sub$i/Makefile.am
+done
+echo 'include foo.am' >> sub7/Makefile.am
+echo 'include bar.am' > sub7/foo.am
+echo 'python_PYTHON = foo.py' > sub7/bar.am
+echo 'lisp_LISP = foo.el' >> sub8/Makefile.am
+echo 'bin_PROGRAMS = p' >> sub9/Makefile.am
+
+rm -f install-sh missing depcomp
+mkdir build-aux
+
+$ACLOCAL
+
+# Generate expected output using the non-threaded code.
+unset AUTOMAKE_JOBS
+AUTOMAKE_run 0 --add-missing
+makefile_ins=`find . -name Makefile.in`
+for file in $makefile_ins; do
+ mv $file $file.exp
+done
+
+AUTOMAKE_JOBS=5
+export AUTOMAKE_JOBS
+
+for run in 1 2 3 4 5 6 7; do
+ AUTOMAKE_run 0
+ for file in $makefile_ins; do
+ diff $file $file.exp
+ done
+done
+
+:
projects / platform / upstream / automake.git / commitdiff