<75 Agility Penalty, does it matter for Casters?

[Vexenu]

For all practical purposes as a Mage, no, not really. Unless you're really trying to push the limit with some risky solo stuff, you will not be getting hit enough for it to be a big deal. Mages should be getting hit less than literally every other class in the game. You can get by with 70 AGI very easily, and as Pyrocat noted, you will eventually get some +AGI gear so it's not an issue at all.