This prevents the idle event to be activated on a destroyed
output.
This also makes the backend responsible for free-ing modes, as it
is the one allocating them and adding them to the list. Note that
the DRM backend (the only one using modes) already frees them.