How Troopships Work
Troopships carry ground forces which can overrun a planet. This allows you to capture the planet without nuking it and thereby losing half of its material resources. Further, the planet retains much of its original population after a successful invasion. (The planet retains half of the population it had before the successful invasion (i.e. the individual ship that suceeds in invading) This may differ from half of the planet's original population if multiple troopships are necessary to invade the planet.) This means that not only does the enemy lose resources, but you gain half of what he loses.
Note that an invasion, whether successful or not, completely uses up the troopship and it is removed from your fleet. However, after one troopship succcessfully invades the planet all others which had been set to invade revert to standby, so you only lose those ships that actually attempt an invasion.
A troopship can successfully invade a planet that has a population of less than ten times the ship's current BR. Many invasion attempts are carried out after a battle above that planet, so the ships often will not be at full BR. Multiple troopships may be used together to successfully invade a planet that has a larger population than the ships could invade individually. Each unsusccesful invasion reduces the planet's population by 2 times the troopship's current BR. Troopships in your fleet will invade one at a time, with each unsuccessful invasion reducing the planet's population, until one of them is strong enough to successfully invade.
As an example, let's assume you attempt to invade a planet with population of 50 with a fleet of BR-4 troopships. To make the calculations easy we'll assume that the enemy does not build any ships to defend. Since the planet has a higher population than a single ship can successfully invade, you will have to use multiple ships. The first ship to attempt invasion will be unsuccessful, but will reduce the population by 2*4=8, leaving the planet with a population of 42. This is still too much for one BR-4 troopship, so the next unsuccessful invasion will reduce the population by another 8 to 34. This is low enough for the next ship to successfully invade, giving you a final population of 1/2 * 34 = 17.
Given this ability to use multiple troopships, a recent fad has developed to invade an enemy's Home World. The benefit of doing this would be a planet with 100 mineral, fuel, and ag resources. You could also rename the planet, shut off your own Home World, and use the new planet as a decoy. Note, though, the difficulty in invading a Home World with anything less than a fleet of very high BR troopships. Unless this is the opponent's last planet he will likely be able to build enough ships to reduce your ships' BR and thereby prevent a successful invasion. This ploy is usually used late in the game when the opposition has been rendered helpless. It is also dangerous to try this when you are fighting another enemy because of the large drain on resources such a large fleet of troopships would produce.
Troopships should not be limited to the middle to end game, though. They can be devestating if used early. Newly colonized planets are often left relatively unguarded at the beginning of the game as players concentrate on exploring and colonizing other planets. You can sometimes slip in a BR-1 or BR-2 troopship and invade one of these planets right after they are colonized. This will put the opponent on the defensive from the beginning of the game and guarantee your forces have a substantial advantage in resources.
One little known fact about troopships is that they can work in conjunction with a nuke by other ships in your fleet. In this case, you set your troopship to invade and the other ships to nuke. The result will be a nuked planet, which is then invaded. This results in reducing the planet's resources by half and eliminating its population, as with a normal nuke, but with your troops taking ownership. You own the planet but it has zero population. Population will grow to 1 the next turn and normally after that. This tactic can be useful if you don't need the resources of that planet, or if you don't have a colony ship that can colonize the planet. I prefer having the immediate resources, though, so I prefer to either invade the planet outright or nuke it and then colonize it. The exception would be if I only want the planet for a stargate jump point that's closer to the front than my previous landing area. (Note that this nuke-and-troop trick does not work at Iceberg even though it is version 2.8.)
Troopships do work well with the rolling stargate attack. You can actually advance faster because you don't have to wait a turn for your colony ship to colonize the planet, you own it the same turn you invade/nuke.
Despite this added quickness, and my fondness for aggressive gating, I don't use troopships all that often. Aggressive building coupled with the maintenance of a stargate means that your tech will be growing slowly unless you are successful in keeping your economy growing through a steady acquisition of enemy planets. Use of expensive troopships will only slow that tech growth even more. Further, your tech may not even get to the point where you have a tech selection to spare. My games, for example, very seldom go beyond BR-4.
When games are long and drawn out, and especially where you are fighthing to a stalemate over a small range of planets, nuking and colonizing the planet can sometimes be more advantageous than invading it. If you're in this situation the game has likely gone on for a while, and BR is getting pretty high. If you have reached BR-7 or BR-8, then using colony ships to create a builder right away could easily be more important than the resources you save by invading rather than nuking.
The other drawback I see to troopships is the uncertainty in being able to successfully invade a planet in the face of a large defending fleet. Too many players (even some of the most experienced veterans) are so intent on invading that they pass up the nuke and wind up with an unsuccessful invasion. This then tends to lead the the destruction of the attacking fleet soon after because of the loss of ships in the attack and the invasion attempt.