How Big Shoud A Wedding Party Be?

I am having trouble deciding how big my wedding party should be. Here is my dilema, I have two brothers and I would love to have them both in the wedding, my fiance has three sisters and I thought that I would include them. But my Fiance accidently asked four of his friends to be in the wedding. Which is fine with me, but I had originally planned on four bridesmades and four groomsmen. Should we still ask my brothers even though that means that we will have SIX people on each side? Is that too many? I just dont want it to look like a parade..... Any advice would be wonderful. Thank you!