[ytpete]

But in this case, isn't the whole pitch that the agent has access to all your data (and the network!) so it can fluidly perform any task you ask of it?

Either the agent needs to be a superuser, with all the attendant risks... or you go the Windows Vista route and constantly prompt users to approve every single access need, which we've all seen how that turns out.