These operations are done in 32-bit arithmetics before being casted to 64-bit,
thus can overflow before the cast.
Casting early fixes the issue.
Found through static analysis
Legacy gamma lut size now uses the new legacy_crtc member of
wlr_drm_crtc. This was Previously doen using old_crtc in
wlr_drm_connector, but since this refers to the crtc that was connected to
the ouput, this could give the wrong result.
Same as what atomic_crtc_set_cursor does
Core was generated by `_build/rootston/rootston'.
Program terminated with signal SIGSEGV, Segmentation fault.
0 0xb6f28a1c in atomic_crtc_move_cursor (drm=0x1ebc8e8, crtc=0x1ead498, x=0, y=0) at ../backend/drm/atomic.c:170
170 if (!crtc || !crtc->cursor) {
(gdb) bt
0 0xb6f28a1c in atomic_crtc_move_cursor (drm=0x1ebc8e8, crtc=0x1ead498, x=0, y=0) at ../backend/drm/atomic.c:170
1 0xb6f2a856 in wlr_drm_connector_move_cursor (output=0x2242b28, x=0, y=0) at ../backend/drm/drm.c:634
2 0xb6f3cea0 in wlr_output_cursor_set_image (cursor=0x21a0338, pixels=0x22e1290 "", stride=24, width=24, height=24, hotspot_x=4, hotspot_y=4) at ../types/wlr_output.c:516
3 0xb6f39da2 in wlr_cursor_set_image (cur=0x22cfc90, pixels=0x22e1290 "", stride=24, width=24, height=24, hotspot_x=4, hotspot_y=4, scale=1) at ../types/wlr_cursor.c:310
4 0xb6f44d2a in wlr_xcursor_manager_set_cursor_image (manager=0x22cfd10, name=0x434420 "left_ptr", cursor=0x22cfc90) at ../types/wlr_xcursor_manager.c:80
5 0x00431c0a in roots_seat_configure_xcursor (seat=0x22cef08) at ../rootston/seat.c:515
6 0x0043137c in roots_seat_init_cursor (seat=0x22cef08) at ../rootston/seat.c:210
7 0x004315ec in roots_seat_create (input=0x218d220, name=0x434594 "seat0") at ../rootston/seat.c:289
8 0x0042ecba in input_get_seat (input=0x218d220, name=0x434594 "seat0") at ../rootston/input.c:39
9 0x0042ed04 in input_add_notify (listener=0x218d228, data=0x218d3b0) at ../rootston/input.c:54
10 0xb6f2f2e6 in wl_signal_emit (signal=0x1ea548c, data=0x218d3b0) at /usr/include/wayland-server-core.h:387
11 0xb6f2f572 in input_add_reemit (listener=0x1ea9990, data=0x218d3b0) at ../backend/multi/backend.c:101
12 0xb6f2db7e in wl_signal_emit (signal=0x1ea992c, data=0x218d3b0) at /usr/include/wayland-server-core.h:387
13 0xb6f2ddaa in handle_device_added (backend=0x1ea9920, libinput_dev=0x2292598) at ../backend/libinput/events.c:87
14 0xb6f2e164 in wlr_libinput_event (backend=0x1ea9920, event=0x2292b78) at ../backend/libinput/events.c:198
15 0xb6f2d678 in wlr_libinput_readable (fd=23, mask=1, _backend=0x1ea9920) at ../backend/libinput/backend.c:28
16 0xb6f2d7c0 in wlr_libinput_backend_start (_backend=0x1ea9920) at ../backend/libinput/backend.c:74
17 0xb6f27170 in wlr_backend_start (backend=0x1ea9920) at ../backend/backend.c:30
18 0xb6f2f320 in multi_backend_start (wlr_backend=0x1ea5480) at ../backend/multi/backend.c:22
19 0xb6f27170 in wlr_backend_start (backend=0x1ea5480) at ../backend/backend.c:30
20 0x0042fbc6 in main (argc=1, argv=0xbe89dd04) at ../rootston/main.c:60
emersion
Dominique Martinet
Vincent Vanlaer
Drew DeVault
Guido Günther
Scott Anderson