As far as I know, the only rules are the
Play Nice Policy. The only way anyone has a claim a GM would recognize is to physically be at the camp.
So, sure, he can take you off the list, because the list is basically meaningless. But you were at the camp, so that doesn't matter. Technically, as I understand it, the camp became yours as soon as he left to pull from a different one (seafuries).