glamor: Fix some integer overflow errors.

Imagine a nbox that was (UINT_MAX + small number) / (4 * 2 *
sizeof(float)).  We'd malloc a few bytes after the integer overflow,
but glamor_set_normalize_vcoords would write over gigabytes of heap.

Signed-off-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Markus Wick <markus@selfnet.de>

diff --git a/glamor/glamor_fill.c b/glamor/glamor_fill.c
index aaa7783..d5843b7 100644 (file)
--- a/glamor/glamor_fill.c
+++ b/glamor/glamor_fill.c
@@ -200,10 +200,10 @@ _glamor_solid_boxes(PixmapPtr pixmap, BoxPtr box, int nbox, float *color)
 
     pixmap_priv_get_dest_scale(pixmap_priv, &xscale, &yscale);
 
-    if (nbox * 4 * 2 > ARRAY_SIZE(vertices)) {
+    if (nbox > valid_nbox) {
         int allocated_box;
 
-        if (nbox * 6 > GLAMOR_COMPOSITE_VBO_VERT_CNT) {
+        if (nbox > GLAMOR_COMPOSITE_VBO_VERT_CNT / 6) {
             allocated_box = GLAMOR_COMPOSITE_VBO_VERT_CNT / 6;
         }
         else