V4L/DVB (3178): bttv VBI fixes

- V4L2_(G|S|TRY)_FMT returned incorrect VBI start lines for PAL-M,
NTSC-JP, and PAL-60. They also returned an inaccurate VBI offset.
- V4L2_(G|S)_FMT and V4L2_TRY_FMT disagreed about the start of VBI
capturing in PAL and SECAM second field. Note the start line fixes
may break applications using VIDIOCSVBIFMT because this ioctl fails
when the driver does not support exactly the requested parameters.
- V4L2_TRY_FMT did not clear the reserved field in struct
v4l2_vbi_format.
- V4L2_(S|TRY)_FMT did not expect very large or small VBI start or
count values, returning wrong (but safe) counts due to an overflow.
- VIDIOCGVBIFMT confused V4L and V4L2 VBI flags. However this had no
effect because the flags have the same value and bttv never sets
them.
- In v4l_compat_translate_ioctl() the VIDIOC(G|S)VBIFMT code did not
expect V4L2 drivers supporting VBI formats besides V4L2_PIX_FMT_GREY.

Signed-off-by: Michael H. Schimek <mschimek@gmx.at>
Signed-off-by: Hans Verkuil <hverkuil@xs4all.nl>
Signed-off-by: Mauro Carvalho Chehab <mchehab@brturbo.com.br>

1 files changed, 10 insertions, 1 deletions

diff --git a/drivers/media/video/v4l1-compat.c b/drivers/media/video/v4l1-compat.c
index 4134549d11a8..2ab5b4093800 100644
--- a/drivers/media/video/v4l1-compat.c
+++ b/drivers/media/video/v4l1-compat.c
@@ -951,6 +951,10 @@ v4l_compat_translate_ioctl(struct inode         *inode,
                         dprintk("VIDIOCGVBIFMT / VIDIOC_G_FMT: %d\n", err);
                         break;
                 }
+                if (fmt2->fmt.vbi.sample_format != V4L2_PIX_FMT_GREY) {
+                        err = -EINVAL;
+                        break;
+                }
                 memset(fmt, 0, sizeof(*fmt));
                 fmt->samples_per_line = fmt2->fmt.vbi.samples_per_line;
                 fmt->sampling_rate    = fmt2->fmt.vbi.sampling_rate;
@@ -966,6 +970,11 @@ v4l_compat_translate_ioctl(struct inode         *inode,
         {
                 struct vbi_format      *fmt = arg;
 
+                if (VIDEO_PALETTE_RAW != fmt->sample_format) {
+                        err = -EINVAL;
+                        break;
+                }
+
                 fmt2 = kmalloc(sizeof(*fmt2),GFP_KERNEL);
                 memset(fmt2, 0, sizeof(*fmt2));
 
@@ -986,7 +995,7 @@ v4l_compat_translate_ioctl(struct inode         *inode,
 
                 if (fmt2->fmt.vbi.samples_per_line != fmt->samples_per_line ||
                     fmt2->fmt.vbi.sampling_rate    != fmt->sampling_rate    ||
-                    VIDEO_PALETTE_RAW              != fmt->sample_format    ||
+                    fmt2->fmt.vbi.sample_format    != V4L2_PIX_FMT_GREY     ||
                     fmt2->fmt.vbi.start[0]         != fmt->start[0]         ||
                     fmt2->fmt.vbi.count[0]         != fmt->count[0]         ||
                     fmt2->fmt.vbi.start[1]         != fmt->start[1]         ||