The Trow have only destroyed those races which challenged them. I have always taken the quote:
"' ... or we will fell your people like a pine forest.' One Ghol asked why he had singled out pines. 'Once cut, pines never regrow.' spoke the Trow emissary, to which the Ghols had no answer save silence."
To mean that the ghols once considered challenging the Trow, but the Trow issued that warning.
Also, Ghols aren't stupid. Think about it. They can master technological equipment (cannons) with instruction, they use weapons, they use battlefield debris to their advantage and they have a desire for self-preservation. The only reason that they are not to par with the other Myth races is their lack of technology. They are simply not as advanced as the others. One main reason for this is their nomadic lifestyle, but I'm sure that the dwarves wouldn't want the ghols to become even stronger either....
-Disc