Any idea why my faction would randomly drop?

[WokeCat]

I'm playing a Troll Shaman on Blue. I've been hunting the Freeport guards in East Freeport for a few days now. I finally decided to go over to North Freeport to test my faction. I was indifferent to the guards, the bards, the clerics etc. and was able to bank with no problems despite one of the bankers scowling at me. I actually banked 2-3 times, and then went back to hunting more guards because I wanted to try to get my faction even higher. After like 6 hours of hunting the guards, I went back to North Freeport to check my faction and suddenly realized that the paladins / clerics are now regarding me apprehensively, and the bards are now scowling at me ready to attack?

I'm genuinely confused about what could have changed?