Enchanter Faction

[Amalec]

I'm playing an level 25 Agnostic HE Enchanter. I killed Paineel guards for 2-3 levels since they have no faction hit and the exp and cash was amazing. I've heard of similar camps like DE guards, HHK guards, Sisters in the OOT, Misty guards, etc. but I'm unsure of which factions matter in the long term and which factions can be easily raised back up again.

I don't overly care about not getting back into Felwithe, for example, since it's in the middle of nowhere and doesn't seem to have anything I need anymore. But I wouldn't want to mess up important quests or make travel/trade inconvenient by ruining things with, say, Freeport.

Can anyone shed some light on this?