It's not 'harder' to take mass off earth than it is to bring mass back. Assuming you want to bring the mass back 'gently' and not have it collide with earth like a comet. Think like pulling a rock out of a hole vs lowering it back down in a controlled manner, controlled here meaning velocity=0 by the time it reaches the bottom. In both cases you are responsible for transitioning the same amount of energy into or out of the rock. To do this with a rocket would required the same integration of force and thus fuel. In the case where you are lowering the rock, ie returning mass back to space, you have to bring fuel to the rock to 'lower' it back down so it is actually harder to bring mass back.
anfilt|5 years ago