btrfs-progs: quota: use btrfs_open_dir for btrfs quota command

btrfs-progs: quota: use btrfs_open_dir for btrfs quota command

We can use btrfs_open_dir() to check whether target dir is
in btrfs's mount point before open, instead of checking it in
kernel space of ioctl, and return fuzzy error message.

Before patch:
  # ./btrfs quota enable /mnt/tmp1
  ERROR: quota command failed: Inappropriate ioctl for device
  # ./btrfs quota disable /mnt/tmp1
  ERROR: quota command failed: Inappropriate ioctl for device
  # ./btrfs quota rescan /mnt/tmp1
  ERROR: quota rescan failed: Inappropriate ioctl for device
  #

After patch:
  # ./btrfs quota enable /mnt/tmp1
  ERROR: not a btrfs filesystem: /mnt/tmp1
  # ./btrfs quota disable /mnt/tmp1
  ERROR: not a btrfs filesystem: /mnt/tmp1
  # ./btrfs quota rescan /mnt/tmp1
  ERROR: not a btrfs filesystem: /mnt/tmp1
  #

Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>