btrfs-progs: judge the return value of check_mounted more accurately

btrfs-progs: judge the return value of check_mounted more accurately

For btrfs-convert, btrfstune, btrfs rescue, they report "device busy"
when given a device that does not actually exist e.g.

	# btrfstune -x abcdefg (this device does not exist)
	$ ...device busy...

We deal with this case by add "ret < 0" error check when
judging the return value of check_mounted.

Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <clm@fb.com>