There's not actually much more than adding the masses, and adding the velocities weighted by (multiplied by) the masses. Some things which might have happened are:
- Not deleting old things, or when you delete them, indexes aren't updating exactly (which is why you can't just delete certain indexes in order).
- Accidentally multiplying/dividing by something when you don't need to (ex. multiplying by mass twice? or multiplying and then dividing by mass? Or something similar with velocity?). I've done something like this before, so I doubt it's uncommon. Make sure you know what expression represents force and which acceleration (and energy, moment of inertia, etc).
- Letting the objects get too close before they merge. If the densities are too large, the objects would be separate in one instance, but pull themselves together so strong they don't have a frame where they are actually touching. I noticed a ridiculous acceleration with the two smallest planets in the original setup, so this might be a cause.
- - Additionally, they might touch in an instance, but be so close that their velocities are ridiculous (I really like this word, right?

); then, when you try to merge them, the resulting velocity is weird.*
*Or will they? Hint - See the 3rd law.