netfilter: ipset: enforce documented limit to prevent allocating huge memory