using ssh authentication with sudo

there is a number of ways I could be missing something obvious so I apologize in advance.

My idea is should be possible to grant sudo access with your ssh credentials. the logic is that once the server has granted access to a client based on its ssh keys, it should be possible to use the same authentication to grant sudo privileges. After all, if a key pair is good enough to get you into one machine, why isn't it good enough to grant you the full Monty?

Assuming that it is, how could a local program determine that the process it is running in has done so via ssh key authentication. Would it query the agent directly? Would it be able to use agent forwarding? Or is this a really bad idea that I should just give up on?