Now, the Proxmox can be accessed while redeploying one of the router!
Due to an unfathomable issue where I would need to recompile with full debug keepalived, keepalived seems to be not seeing the multicast packets on the vrrp-router and thus we need to fallback to the disgusting unicast solution. Well, let's move on.