When you get Torpor you aren't going to be wearing Fungi most of the time. You'll switch to Vindi BP or a Velious Chest. You won't have the -10 AGI at that point.
5 AGI/WIS/STA isn't going to be noticeable whichever way you slice it, I wouldn't worry about it.
You might as well put the 5 points into AGI, then you won't have to worry about AGI buffing yourself, or getting a piece of equipment with AGI on it.
I only have +15 AGI from gear so with most of my slots filled with raid gear. If I chose to use
https://wiki.project1999.com/Onyx_Chain_Leggings instead for fashion and
https://wiki.project1999.com/Eye_of_Cazic_Thule for the HP/Mana, I would be at +0 AGI, and wearing a Vindi BP instead, which means I would be at 75 AGI exactly via starting stats.