ssh does not ask for my key's passphrase, instead I get `sign_and_send_pubkey: signing failed for RSA from agent: agent refused operation`

Situation

On my notebook I have a passphrase protected ssh key ~/.ssh/id_rsa_pass. I want to use it to access a shared server. I also want to forward the key to the server, so it can be reused there to access e.g. my GitHub without storing the private key on the server, which is not secure.

In order to make this happen, I use .ssh/config. This is the content:

Host myserver  HostName hostname  User user  IdentityFile ~/.ssh/id_rsa_pass  IdentitiesOnly yes  ForwardAgent yes  AddKeysToAgent yes

Expected behavior

1. I run ssh myserver
2. It asks for a passphrase
3. Adds the unlocked key to ssh agent
4. Connects me to the server while forwarding the key

What actually happens

1. I run ssh myserver
2. Instead of asking for my passphrase, it returns an error: `sign_and_send_pubkey: signing failed for RSA "/home/user/.ssh/id_rsa_pass" from agent: agent refused operation
3. And tries to restore to username & password access

What makes it work

When I manually do ssh-add ~/.ssh/id_rsa_pass, it asks me for a passphrase and then calling ssh myserver works. Also forwarding works. But my point is that I have to remember manually doing ssh-add before my first attempt to ssh to the server. Why is this necessary? Why does the ssh not add this automatically when I first try to connect? I assumed AddKeysToAgent would do this, but I am probably missing something.

Note: the permissions on the .ssh folder and on the keys are set up correctly. Otherwise the manual call to ssh-add would not help (IMO). So please do not suggest tinkering with chmod.Note2: I have read this answer, but it does not address this problem exactly.

Update

I found out that in my startup applications, there is an additional startup program: SSH Key Agent (GNOME Keyring: SSH Agent). According to this thread, for some reason, the Keyring program under Ubuntu 22.04 does not ask for the passphrase as it should. In my opinion, this leads to the key seemingly available in ssh-add -l but it is not decrypted - because it never asked for the passphrase. So a call to ssh myserver fails. Either starting a new ssh-agent or manually adding the key to the current ssh-agent makes the connection work. Anyways, I am not sure how to solve this. If I remove the ssh key from the Keyring, it completely deletes it from my .ssh folder.

Update 2

Now I found out that disabling the startup application GNOME Keyring SSH Agent resolves the issue. The ssh-agent after reboot is running, but it does not have any identities stored. And then, ssh call correctly asks for the passphrase, adds it into the ssh-agent identities and even the forwarding works. Now, is disabling the GNOME Keyring SSH Agent going to cause any other issues? I do not know why it was in the startup applications in the first place. Is it necessary in Ubuntu 22.04?