The issue I see with the current system is that there's two major sides and two castles to work with, so we're at the same stalemate as Seraph vs Behemoth a while back. We /could/ cut down the castles down to one, but that would deter the growth even more. If we were to increase the castle count to three, then one of the two sides would just end up with two castles at the end of WoE.
Yes, the current situation may not allow for growth, but it doesn't mean there are no changes that could be made to allow for it. All it takes is for a group to tip the balance and not join with the major guilds, but no one is willing to take the leap. If anything, the problem lies with the current player base, and not with the numbers of castles or any possible GM changes.