drm: fix drm_mode_addfb() on big endian machines.

Userspace on big endian machhines typically expects the ADDFB ioctl
returns a big endian framebuffer.  drm_mode_addfb() will call
drm_mode_addfb2() unconditionally with little endian DRM_FORMAT_*
values though, which is wrong.  This patch fixes that.

Drivers (both kernel and xorg) have quirks in place to deal with the
broken drm_mode_addfb() behavior.  Because of this we can't just change
drm_mode_addfb() behavior for everybody without breaking things.  Add
the quirk_addfb_prefer_host_byte_order field to mode_config, so drivers
can opt-in.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/20180905060445.15008-5-kraxel@redhat.com

1 files changed, 11 insertions, 0 deletions

diff --git a/drivers/gpu/drm/drm_framebuffer.c b/drivers/gpu/drm/drm_framebuffer.c
index 888c4d53cf23..f863f8a20f8c 100644
--- a/drivers/gpu/drm/drm_framebuffer.c
+++ b/drivers/gpu/drm/drm_framebuffer.c
@@ -124,6 +124,17 @@ int drm_mode_addfb(struct drm_device *dev, struct drm_mode_fb_cmd *or,
             r.pixel_format == DRM_FORMAT_XRGB2101010)
                 r.pixel_format = DRM_FORMAT_XBGR2101010;
 
+        if (dev->mode_config.quirk_addfb_prefer_host_byte_order) {
+                if (r.pixel_format == DRM_FORMAT_XRGB8888)
+                        r.pixel_format = DRM_FORMAT_HOST_XRGB8888;
+                if (r.pixel_format == DRM_FORMAT_ARGB8888)
+                        r.pixel_format = DRM_FORMAT_HOST_ARGB8888;
+                if (r.pixel_format == DRM_FORMAT_RGB565)
+                        r.pixel_format = DRM_FORMAT_HOST_RGB565;
+                if (r.pixel_format == DRM_FORMAT_XRGB1555)
+                        r.pixel_format = DRM_FORMAT_HOST_XRGB1555;
+        }
+
         ret = drm_mode_addfb2(dev, &r, file_priv);
         if (ret)
                 return ret;