Don't finalize sent or dispatched messages while under the connection lock

Finalizing a message can trigger callbacks; that's bad, if we have a
connection locked.

In particular, if a message is received by the "left side", passed to
the "right side" and sent (as in test/relay.c (see the diagram there)
or in dbus-daemon), then finalizing that message could result in the
live messages counter for the left side, and the outgoing messages counter
for the right side, both being decremented while under either side's
lock.

After a message is dispatched on the left side, finalizing it now drops
the lock temporarily, to avoid this problem.

After a message is sent on the right side, finalizing it is now deferred
until the right side unlocks, by moving it to a new queue of
"expired messages" which is automatically cleared every time we release
the lock.

The "live messages" counter for the "left" connection will now explicitly
take the left connection's lock before decrementing, to avoid
manipulating watches without a lock.

Reviewed-by: Colin Walters <walters@verbum.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34393

Don't inline the contents of _dbus_connection_unlock

It's about to become more complex, to handle delayed deallocation of
messages in order to avoid triggering callbacks while locked.

Reviewed-by: Colin Walters <walters@verbum.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34393

When attaching counters to messages, don't automatically notify callbacks

In all the places where counters are added, we're under a lock. The caller
knows what effect adding the counter might have, and can replicate it
in a lock-safe way if necessary.

Reviewed-by: Colin Walters <walters@verbum.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34393

Comment some places where it's OK to unref a message despite holding locks

In general, dbus_message_unref should be avoided while holding locks,
because it can invoke arbitrary user callbacks (via attached data, or
via DBusCounter).

Reviewed-by: Colin Walters <walters@verbum.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34393

Add _dbus_counter_notify and call it after every adjustment

When fd-passing is implemented, adjustments happen in pairs; in that case
we coalesce the two calls into one.

Reviewed-by: Colin Walters <walters@verbum.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34393

_dbus_connection_message_sent: rename to _unlocked

It's called with the connection's lock held.

Reviewed-by: Colin Walters <walters@verbum.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34393

dbus_bus_register: don't unref the messages with the lock held

Finalizing a message can call out to user code via dbus_message_set_data
(or to internal code not expecting locks to be held, via DBusCounter).

Reviewed-by: Colin Walters <walters@verbum.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34393

Merge branch 'dbus-1.4'

Add a note about bootstrapping new platforms to README

Upgrade the type system into its own top-level section of the spec

The type system can be used independently, for instance in GVariant
(although GVariant's binary encoding is in fact not the same).

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38252
Reviewed-by: Lennart Poettering <lennart@poettering.net>

Merge branch 'dbus-1.4'

man pages: replace all unescaped hyphen/minus characters with \-

In a man page, "-" officially means a typographical (Unicode) hyphen,
which frequently breaks the ability to copy and paste code examples from
a man page. "\-" means the ASCII hyphen/minus character. See
<http://lintian.debian.org/tags/hyphen-used-as-minus-sign.html> for
more details.

Rather than trying to distinguish between hyphens, em-dashes and
hyphen/minus, I just replaced all ambiguous hyphens with \- by applying
this vim command repeatedly until it didn't find anything:

     %s/\(^\|[^\\]\)-/\1\\-/g

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38284
Reviewed-by: Lennart Poettering <lennart@poettering.net>

Fix spelling/grammatical mistakes detected by Debian's lintian(1)

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38284
Reviewed-by: Lennart Poettering <lennart@poettering.net>

Merge branch 'dbus-1.4'

list_allows_user: fix debug output for groups

Signed-off-by: Sascha Silbe <sascha-pgp@silbe.org>
Reviewed-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Merge branch 'dbus-1.4'

Conflicts:
dbus/dbus-message.c

NEWS

DBusMessage: always access refcount atomically, even for assertions/initial ref

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

dbus-memory: use atomic accesses to block count, even for assertions

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

Merge branch 'dbus-1.4'

NEWS

DBusConnection: use atomic accesses to refcount in assertions/initial ref

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

dbus_message_ref: avoid unused variable if not asserting

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

_dbus_connection_close_if_only_one_ref: use _dbus_atomic_get

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

Add _dbus_atomic_get implemented in terms of inc, dec

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

DBusObjectTree: always access refcount atomically

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

dbus_message_unref: make an assertion more strict

We've just decremented the refcount, so it should have been at least 1
before we did that.

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

DBusMessageFilter: exclusively use atomic accesses to refcount

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005

Merge branch 'dbus-1.4' and update NEWS for master

Conflicts:
NEWS

NEWS

Uninstall the installed docs

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=36156
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Install man2html output to $(htmldir)

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=36156
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Install diagram.*, system-activation.txt to the $(docdir)

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=36156
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Install xmlto output to $(htmldir) automatically

Also rename HTML_FILES to XMLTO_OUTPUT, as a more self-describing name:
we have other HTML files which are not the output from xmlto.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=36156
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Install dbus.devhelp to $(htmldir), not $(apidir), and remove it in clean

It needs to be in $(htmldir) so that its relative paths will work.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=36156
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

configure.ac: fix check for xsltproc

Pre-setting XSLTPROC interferes with AC_CHECK_PROGS letting the user
override choice of program via the environment.

Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=36156

Use EXEEXT when running tests from another directory, and skip bus-test-launch-helper on non-Unix

This is necessary when cross-compiling from Linux to mingw32 and running
the resulting tests under Wine. (This partially works! Some tests fail,
though.)

Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Build docs after running tests, and remove redundant DIST_SUBDIRS

If DIST_SUBDIRS isn't set, it defaults to SUBDIRS, so it's just noise.

Running tests before building documentation is an easy way to speed up the
hack/make check/fix cycle, by not wasting time rebuilding the
documentation (which is often slow) until all the tests compile and pass.

https://bugs.freedesktop.org/show_bug.cgi?id=34405
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Run bus tests from test/

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34405
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Run dbus-test from test/, not dbus/

This avoids failures caused by test/data not having been set up yet.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34405
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

dbus tests: don't warn on skipped tests

We should treat _dbus_warn as fatal when we run tests, like bus/ does.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34405
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

test/Makefile.am: list tests one per line

This improves clarity, and makes conflicts less likely and merges more
obviously correct.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34405
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Opt-in to modern OS features on Solaris, to get file descriptor passing

Surely it's broken to define SCM_RIGHTS but not the necessary structs to
do something useful with it, but whatever.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33465
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

_dbus_open_unix_socket: make static

It isn't called from outside its translation unit.

_dbus_open_tcp_socket: remove

It's only implemented on Unix, internal to dbus, and never called.

Merge branch 'dbus-1.4'

_dbus_connect_tcp_socket_with_nonce: don't create an extra fd (which is then leaked)

This block should have been deleted in 2007, when IPv6 support was added:
previously, the fd allocated at the beginning of the function was used
for connect(), but for IPv6 support, the socket() call has to be inside
the loop over getaddrinfo() results so its address family can be changed.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=37258
Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>

Merge branch 'dbus-1.4'

DBusServer: exclusively use atomic operations on the refcount

Same reasoning as for fd.o #38005, commit 50732523a7.

Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005
Bug-NB: NB#263436

DBusPendingCall: exclusively use atomic operations on the refcount

Same reasoning as for fd.o #38005, commit 50732523a7.

Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38005
Bug-NB: NB#263436

Add testsuite for eavesdrop=true DBusMatchRule

[with some typos fixed during commit -smcv]

Reviewed-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=37890
Bug-NB: NB#269748

Fix dbus-monitor using eavesdrop=true when no filters are passed

Reviewed-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

DBusHeader: only store byte-order in the fixed-length header

Reviewed-by: Thiago Macieira <thiago@kde.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38287

DBusMessage: don't redundantly store byte order, ask the DBusHeader

Reviewed-by: Thiago Macieira <thiago@kde.org>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38287

Add "eavesdrop=true" as constant match rule for dbus-monitor

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=37890
Bug-NB: NB#269748
Reviewed-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Do not allow eavedropping unless rule owner explicitely declare it

Adds "eavesdrop=true" as a match rule, meaning that the owner
intend to eavedrop.
Otherwise the owner will receive only broadcasted messages and the ones
meant to be delivered to it.

[plus a typo fix in an error message -smcv]

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=37890
Bug-NB: NB#269748
Reviewed-by: Simon McVittie <simon.mcvittie@collabora.co.uk>

Merge remote-tracking branch 'origin/dbus-1.4'

DBusConnection: simplify registering object paths code

Introduce static generic _dbus_connection_register_object_path()
function to remove duplicate code for registration object paths.
Having *four* almost the same functions is error-prone and hard
to follow as well.

Signed-off-by: Jiří Klimeš <jklimes@redhat.com>
Reviewed-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38874

Cope with platforms whose vsnprintf violates both POSIX and C99 - part 1

In Windows _dbus_printf_string_upper_bound, don't crash on malloc failure

Added DBUS_ENABLE_STATS option to cmake buildsystem to keep in sync with autotools.

Visual Studio Professional uses a different ide starter - pointed out by Romain Pokrzywka

Collected cmake doc into README.cmake.

Visual Studio Professional uses a different ide starter - pointed out by Romain Pokrzywka

win32 fix: Keep server autolaunch mutex settings according to client detection algorithmus.

Fixed cmake buildsystem - option command only supports boolean values

Cope with platforms whose vsnprintf violates both POSIX and C99 - part 1

In Windows _dbus_printf_string_upper_bound, don't crash on malloc failure

Limit batch file creating to windows

Fixed cmake buildsystem on unix - option command only supports boolean values

Collected cmake doc into README.cmake.

Added DBUS_ENABLE_STATS option to cmake buildsystem to keep in sync with autotools.

win32 fix: Keep server autolaunch mutex settings according to client detection algorithmus.

Merge branch 'dbus-1.4'

DBusConnection: use DBUS_ERROR_OBJECT_PATH_IN_USE instead of DBUS_ERROR_ADDRESS_IN_USE

While registering an object path the possible error is
DBUS_ERROR_OBJECT_PATH_IN_USE, not DBUS_ERROR_ADDRESS_IN_USE (the error
is set by _dbus_object_tree_register()).

Signed-off-by: Jiří Klimeš <jklimes@redhat.com>
Reviewed-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38874

Include size of link cache in per-connection statistics

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

Also record peak values for queued bytes/fds in connection stats

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

Add an initial round of stats to the Stats interface

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

DBusConnection, DBusTransport: add queue statistics

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

BusConnections: add usage stats for well-known names, match rules

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

DBusList: add usage stats

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

DBusMemPool: add usage stats

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

Add a stub .Debug.Stats interface if --enable-stats

There are no actual statistics yet, just a count of how many times the
method has been called, and (for the per-connection stats) the unique name.

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

configure.ac: add --enable-stats

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

match_rule_to_string: don't reinvent dbus_message_type_to_string

We don't allow match rules with unknown message types, so losing the
"type='%d'" case isn't a great loss.

Reviewed-by: Cosimo Alfarano <cosimo.alfarano@collabora.co.uk>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=34040

Remove maximum length field from DBusString

The source code says it's "a historical artifact from a feature that
turned out to be dumb". Respond accordingly!

This reduces sizeof (DBusString) by 20% on ILP32 architectures, which
can't hurt. (No reduction on LP64 architectures that align pointers
naturally, unfortunately.)

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38570
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Remove unused _dbus_string_copy_data_len, _dbus_string_steal_data_len

They're only compiled when building tests.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38570
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

Split DBUS_GENERIC_STRING_PREAMBLE into sensible lines

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=38570
Reviewed-by: Will Thompson <will.thompson@collabora.co.uk>

CMake fix: Do not use FORCE in variables which should be changeable by the user.

fixed cmake error: doc says CACHE needs type *and* doc string

Use epoll in a backwards-compatible way on Linux < 2.6.27

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33337
Bug-NB: NB#197191
Bug-NB: NB#225019

Maintain the set of polled sockets over time

This enables us to benefit from epoll(4): polling is now
O(number of watch changes + number of active fds), whereas with poll(2)
it's O(total number of fds).

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33337
Bug-NB: NB#197191

Add a stub _dbus_loop_toggle_watch and call it where needed

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33337
Bug-NB: NB#197191

Add an implementation of DBusSocketSet using epoll

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33337
Bug-NB: NB#197191
Bug-NB: NB#225019

Check for epoll in configure.ac

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33337
Bug-NB: NB#197191

DBusSocketSet: new abstraction for struct pollfd[] or whatever

In this second version of this patch, DBusSocketSet is an "abstract base
class" so that when using a better OS-specific API fails, we can always fall
back to _dbus_poll(). For instance, this is necessary when the "better
OS-specific API" is epoll on Linux, the build machine has a modern glibc
version, and the host machine either has an old kernel, is emulated in qemu
(which does not support the epoll syscalls yet), or both.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33337
Bug-NB: NB#197191
Bug-NB: NB#225019

NEWS

DBusLoop: store watches in a hash table keyed by fd

This means we don't need to build up the watches_for_fds array, because
it's quick to find the watches based on the fd.

This also fixes fd.o #23194, because we only put each fd in the
array of DBusPollFD once, with the union of all its watches' flags; we
can no longer get more entries in the array than our number of file
descriptors, which is capped at our RLIMIT_NOFILE, so we can't get
EINVAL from poll(2).

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=23194
Reviewed-by: Thiago Macieira <thiago@kde.org>

DBusLoop: fold Callback into TimeoutCallback

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33342
Reviewed-by: Thiago Macieira <thiago@kde.org>

Drop WatchCallback entirely, just use a list of DBusWatch in the main loop

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=33342
Reviewed-by: Thiago Macieira <thiago@kde.org>