After destroying a keyboard input device, seat's listeners could still be pointing to destroyed wlr_input_device signals. This patch makes sure the references are removed while the input device is being destroyed.