regarding to what you have described you are definitely accessing wrong wallet by either using different seed (device) or passphrase (or both).
Because when the same recovery seed is used and the same passphrase is entered (if enabled and used), the same wallet is accessed. There is no exception.
It always generates the same wallet including the same accounts having the same addresses in the same order. The public-key cryptography ensures it generates always the same result.
Thus in order to access the desired address you have to use exactly the same passphrase (which I believe is your issue) as you did when you used that address in the first place.
To access this hidden wallet repeatedly, you will have to use the exact same passphrase in combination with the recovery seed on the device. Using the same seed with a different passphrase will generate a different wallet. Using a different seed with the “correct” passphrase will generate a different wallet.
Mistyping the passphrase will generate a completely new wallet. There is no such thing as an “incorrect passphrase”, so whatever you provide as your input will be used in the process of deriving a wallet.
Also, please note that some users chose random input as a passphrase without paying much attention to it, without realizing that they will have to use exactly the same input/password to access the same wallet repeatedly.
From my experience sometimes users use also their PIN code, password to enter 3rd party standalone wallet/extension such as MetaMask or even the recovery seed.