buildpackage: new "working copy" choices for --git-export

Add support for building different kind of "working copies", when using
the --git-export option:
- 'WC.TRACKED': only use files that are already tracked
- 'WC.UNTRACKED': use untracked files, too
- 'WC.IGNORED': also add files that'd normally be ignored

Using '--git-export=WC' beaves like before.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/common/buildpackage.py

buildpackage: add 'untracked' option to write_wc()

Whether to ignore untracked files (untracked=False) or not
(untracked=True). Now, clones the temporary index file from the actual
index, in order to correctly update files.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Conflicts:
gbp/scripts/common/buildpackage.py

GitRepository/add_files: add new option 'untracked'

With this option you can either only update already tracked files to
index the (untracked=False) or add new files, too.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

config: support for per-tree config files

Add support for reading the local config file(s) from a given git
tree-ish.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

config: read the right config if run from subdir

A step towards being able to run GBP tools from subdirectories.

Now expands '%(top_dir)s' and '%(git_dir)s' in config file path to root
of the working directory and git metadata directory, respectively.

Also, adds a new method _read_config_file() in preparation for
supporting per-tree config files.

Fixes tests.test_Config: currently the only correct way to define the
config file(s) to be parsed is by using the GBP_CONF_FILES environment
variable.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/config.py

tests/testutils: add directories arg to ls_* functions

With this you can exclude directories from the listing returned by the
ls_tar(), ls_zip() and ls_dir() functions.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

tests.testutils: add ls_zip() helper

For listing zip contents.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
tests/testutils.py

import-orig: rework sourcecode preparation and filtering

Take in to use the newly added prepare_sources() function. Drop dead,
unneeded code.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/common/import_orig.py
gbp/scripts/import_orig.py

import-orig: new function for filtering/re-packing sources

Introduces a new function prepare_sources() that prepares upstream
sources for importing into upstream branch and pristine-tar. That
includes unpacking, filtering and re-packing sources. What somewhat
complicates the logic is that it tries to avoid excess unpacking/packing
of the sources.

Also fixes the unpacking / filtering / repacking logic which was broken
with some parameter combinations.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
tests/test_import_orig.py
tests/testutils.py

UpstreamSource.unpack: return UpstreamSource instance

Trying to keep UpstreamSource in consistent state so that unpacked and
packed sources are kept in sync.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/import_srpm.py

UpstreamSource.pack: reuse unpacked dir if no filter is used

Set the unpacked attribute to the source dir if the content is not
filtered.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

UpstreamSource: implement prefix guessing

Add a new attribure 'prefix', i.e. the "leading directory name" in an
archive. For example, this usually is '<name>-<version>' in release
tarballs.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Łukasz Stelmach <l.stelmach@samsung.com>
Conflicts:
gbp/pkg/__init__.py

UpstreamSource: check that the sources exist

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

tests: test find_source() with unittest

Change doctests to unittests in order to have more flexibility in
testing.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
tests/test_import_orig.py

UpstreamSource: store the absolute path

This is more robust and reliable, e.g. in case cwd is changed.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/pkg/__init__.py

UpstreamSource: move version guessing logic to PkgPolicy

PkgPolicy is more logical context, for example if guessing version from
a plain filename and not a real file.

Also, changes guess_version() to always return a tuple to simplify
checking its return value.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/pkg/__init__.py
tests/06_test_upstream_source.py

UpstreamSource: introduce is_tarball() method

New method for checking if the sources are a tarball (possibly for
checking if the sources are committable by pristine-tar).

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

UpstreamSource: add 'archive_fmt' and 'compression' properties

New properties for easily getting the archive format (tar, zip, ...) and
compression format (gzip, bzip2, ...) of the sources.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/pkg/__init__.py

Introduce PkgPolicy.is_valid_orig_archive()

A new method for checking if the given filename is usable as a orig
(upstream) source archive.

Also, takes this method into use in the UpstreamSource class.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/pkg/__init__.py

UpstreamSource.pack: support prefix mangling

Add support for changing the prefix directory inside the tarball that is
generated. Also, fixes a bug that caused a "prefix-less" tarball to get
one, if unpacked and then repacked.

Also, adds this support to repack_source() in common/import_orig.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/pkg/__init__.py
gbp/scripts/common/import_orig.py

import-orig: move repacked_tarball_name() out from common

Moves repacked_tarball_name() from scripts/common back to the
Debian-specific script. The logic is too Debian-specific for RPM, after
all.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/common/import_orig.py
gbp/scripts/import_orig.py

import-orig: simplify repack_source() and tmpdir handling

Now a tmpdir is always created and sources are always unpacked into
there.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/common/import_orig.py
gbp/scripts/import_orig.py

import-orig: implement --create-missing-branches option

Create the upstream branch if it does not exist. Use the same option
name that import-dsc has.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/import_orig.py

docs: enable building html docs with docbook2html

This is useful if sgml2x package is not available.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
docs/Makefile

GitRepository: add diff_status method

This is a method of getting the filename and status information of a
diff. That is, a list of files that changed and their status, "added",
"modified" etc.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

gbp.git: Reimplement rfc822 date parsing without dateutil

In order to get rid of dependency to dateutil.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
debian/control

pq/parse_gbp_commands: return non-matched lines

In order to make it easy to re-use this function in changelog
generation.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository.add_files: consistently update ALL tracked files

Use the --all option of git-add so that all tracked files are updated in
all conditions. Previously deletion of tracked files was not staged, for
example.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/git/repository.py

GitRepository.add_files: use GitArgs

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/git/repository.py

GitRepository: allow repo inititialization from subdir

This allows initialization of a GitRepository object, even if the
current working directory (or path given to GitRepository) is not the
top level directory of the git repository.

Don't guess the git meta data dir, but, take it as reported by git
itself.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository: Implement status method

Simple wrapper to the git-status command.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository: raise GitRepositoryError instead of GbpError

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository/rev_parse: use _git_inout()

instead of the deprecated _git_getoutput() method.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository/git_command: strip stderr output in case of error

In order to get rid of extra line feeds in exception message.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository/has_submodules: add treeish argument

For defining a Git treeish which to look into, instead of the current
working copy.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository.create_branch: add 'force' option

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository.commit_files: add edit argument

In order to be able to spawn an editor, similarly to other commitX
methods in GitRepository.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/git/repository.py

GitRepository.commit_all: add committer_info argument

For setting the committer name/email/date - similarly to author_info.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository.commit_files: add committer_info argument

For setting the committer name/email/date - similarly to author_info.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/git/repository.py

GbpOptionParser: allow using the default 'dest' attribute

Make it possible to add options without explicitly defining the 'dest'
attribute - in which case the default of OptionParser is used.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository: implement create_tree method

New method for creating tree objects from a directory content.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository: fix the 'edit' mode in commit methods

Previously the editor would not show in the shell because stdout was
captured.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/git/repository.py

GitRepository: allow interactive commands

Make it possible for git commands to output directly to stdout without
redirection (i.e. capturing their stdout stream).

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

buildpackage: refactor tarball compression

Introduce a new function for running the compressor commands. Unifies
exception/error handling - GbpError will be returned if the compressor
fails.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/common/buildpackage.py

buildpackage/dump_tree: add 'recursive' option

For selecting whether to dump all the files recursively or just the top
level directory of the tree.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository.archive: add 'paths' option

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

GitRepository.archive: drop the '**kwargs' argument

Unused and misleading.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/git/repository.py

common/buildpackage: use GitRepository for submodules

Initialize a GitRepository object for each submodule in
git_archive_submodules().

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

buildpackage/dump_tree: use GitRepository.archive()

Make dump_tree() utilize the GitRepository.archive() method - similarly
to git_archive_submodules() - instead of ad-hoc Python pipes. This makes
dump_tree() work independent of the callers current working directory.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/common/buildpackage.py

buildpackage/git_archive_single: use GitRepository.archive()

Use GitRepository.archive() method like git_archive_submodules() does.
This makes it possible to call git_archive_single() independent of the
callers current working directory.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/common/buildpackage.py

Change compressor cmdline opts in compressor_opts to a list

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/common/buildpackage.py

common/buildpackage: support for different archive formats

Adds support for defining the archive format of the output of
git_archive_single(), e.g. 'zip'. Defaults to 'tar', as before.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
debian/control
gbp/scripts/common/buildpackage.py

docs: document the --customizations option of gbp dch

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

docs: document the --interactive option of gbp-import-orig

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

notifications: make notify() more generic

In order to be able to re-use it in other tools than git-buildpackage.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

pq: make sure we always get a sane author

Prevent a crash when the author (for a raw diff) was None because no
name/email could be determined from git config or environment variables.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

pq: deprecate the usage of 'gbp-pq-topic:'

Replaced by the "Gbp[-Pq]: Topic <topic>" command.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/pq.py

log: don't automatically setup gbp-specific logger

This makes it nicer to use gbp.* modules (e.g. gbp.git) in other
software projects which may have their own logging setup. After this
change, just importing gbp.* modules won't forcibly setup gbp-specific
logging. However, gbp logging is initialized whenever the
gbp.log.getLogger function is called which makes logging in existing gbp
commands behave just like before.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

import-srpm: support --upstream-vcs-tag cmdline option

Similar to the corresponding option in gbp import-orig.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
gbp/scripts/import_srpm.py

tests: add tests for UpstreamSource.guess_version()

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
tests/06_test_upstream_source.py

UpstreamSource.guess_version: recognise debian native tarballs

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

ComponentTestBase: add dirs argument to _check_repo_state()

Make difference between regular files and directories, eliminating the
requirement of listing directories in the file list.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
tests/component/rpm/test_import_srpm.py

ComponentTestBase: add ls_tree() method

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

ComponentTestBase: add check_files() method

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
tests/component/__init__.py

ComponentTestBase: add a per-class toplevel temp dir

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

ComponentTestBase: ignore system and user config

User might have a system and/or user specific gbp config files. Disable
these config files so that they don't affect the component tests.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Conflicts:
tests/component/__init__.py

log: fix auto colorizing for custom streams

Check for existence of isatty() method in the stream object. Some custom
streams (e.g. in nose) do not necessarily have this.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

notify: catch RuntimeError when importing pynotify

Work around a problem in some distros (e.g. Fedora) where "import
pynotify" crashes in RuntimeError in some cases, e.g. when DISPLAY env
variable is not set.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

gbp-dch: fix handling of the '--meta' option

Make it effective again - previously it was totally ignored. Also,
change it's default value to True to match the current behavior.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>

Document changes and release 0.6.22

Conflicts:
debian/changelog

git.vfs: fix close method

Preventing a infinite recursion which can be triggered by gbp pq export
--commit.

Document changes and release 0.6.21

Conflicts:
debian/changelog

Update README a bit

Git-Dch: Ignore

Conflicts:
README

Complete setup.py for pypi

Improve change reporting a bit

man: fix option argument for --git-pbuilder-options

pq: Don't fail commit if the series file is empty on the source branch

Document changes and release 0.6.20

Conflicts:
debian/changelog

pq: Add "pq export --commit" option

This commits the changes in the pq right away. This is currently
experimental and subject to change.

Conflicts:
gbp/scripts/pq.py

meta-closes: Move help text to GbpOptionParser.help

pq: document --drop

Allow to always drop pq branch after export

Closes: #761160

Conflicts:
gbp/config.py

Document changes and release 0.6.19

Conflicts:
debian/changelog

Update to git-pbuilder 1.33

Document changes and release 0.6.18

Conflicts:
debian/changelog

Mention --no-merge

Closes: #760091

Don't delete *_source.changes on source only builds

Closes: #758726

Document changes and release 0.6.17

Conflicts:
debian/changelog

bash completion: make command list dynamic

Use "gbp --list-cmds" so we don't have to hardcode the available
commands and get support for the RPM ones as they show up.

Unify doc strings a bit

since they now show up with --list-cmds

Allow to list all available commands

gbp: add --version option

Closes: #758909

Improve error messages on formatting errors

Make it easier for the user to detect misformated replacement strings in
config files and command line options.

Conflicts:
debian/git-buildpackage.install
gbp/scripts/import_orig.py

buildpackage: Make debian-tag message configurable

New config option --git-debian-tag-msg allows for the specification of
the message format string for signed debian-tags.

When left unset, the default debian-tag-msg format is still:

    %(pkg)s Debian release %(version)s

Signed-off-by: Kamal Mostafa <kamal@whence.com>
Conflicts:
gbp/scripts/buildpackage.py

dch: Only modify the mainttrailer when --git-author is in use

Use better variable name

It's a boolean indicating wether we fetch the author information from
git not the author information itself.

Git-Dch: Ignore

Brown paper bag, please

Gbp-Dch: Ignore

Make sure we fixup the changelog trailer with newer devscripts

We don't change any mainttrailer options if already given.

Thanks: James McCoy for the detailed explanation
Closes: #740566

Document changes and release 0.6.16

Conflicts:
debian/changelog

Remove all other log handlers when capturing logs

This avoids spurious error messages during test runs.