This design also avoids double frees in the future when we're creating several wlr_input_devices from one libinput_device.