A Remote Design can target any number of devices so the second paragraph of your message is not actually the case.
The general solution to what you’re asking however is either to enable ‘Show Manual Remote’ for that device so that you can quickly popup a remote for diagnostic purposes, or create an Activity specifically for that other device that doesn’t change inputs for instance.