btrfs-progs: use btrfs_open_dir in open_path_or_dev_mnt

btrfs-progs: use btrfs_open_dir in open_path_or_dev_mnt

Use btrfs_open_dir() in open_path_or_dev_mnt() to make the function
return error when target is neither block device nor btrfs mount point.

Also add "verbose" argument to let function output common error
message instead of putting duplicated lines in caller.

Before patch:
  # ./btrfs device stats /mnt/tmp1
  ERROR: getting dev info for devstats failed: Inappropriate ioctl for device
  # ./btrfs replace start /dev/vdd /dev/vde /mnt/tmp1
  ERROR: ioctl(DEV_REPLACE_STATUS) failed on "/mnt/tmp1": Inappropriate ioctl for device

After patch:
  # ./btrfs device stats /mnt/tmp1
  ERROR: not a btrfs filesystem: /mnt/tmp1
  # ./btrfs replace start /dev/vdd /dev/vde /mnt/tmp1
  ERROR: not a btrfs filesystem: /mnt/tmp1

Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>