Since Sierra, I got prompted for my ssh key password everytime. After digging a bit, it seems Apple just changes it recently.

On macOS, specifies whether the system should search for passphrases in the user’s keychain when attempting to use a particular key. When the passphrase is provided by the user, this option also specifies whether the passphrase should be stored into the keychain once it has been verified to be correct. The argument must be ‘yes’ or ‘no’. The default is ‘no’.

In order to fix this, you just have to enable UseKeychain for every hosts by adding these lines into your .ssh/config file

Host *
   AddKeysToAgent yes
   UseKeychain yes
   IdentityFile ~/.ssh/id_rsa

Alternatively, you can add ssh-add -A into your .bashrc or .zshrc.