Emergence of invasive behavior in cancer is life-threatening, yet ill-defined due to its multifactorial nature. We present a multiscale mathematical model of cancer invasion, which considers cellular and microenvironmental factors simultaneously and interactively. Unexpectedly, the model simulations predict that harsh tumor microenvironment conditions (e.g., hypoxia, heterogenous extracellular matrix) exert a dramatic selective force on the tumor, which grows as an invasive mass with fingering margins, dominated by a few clones with aggressive traits. In contrast, mild microenvironment conditions (e.g., normoxia, homogeneous matrix) allow clones with similar aggressive traits to coexist with less aggressive phenotypes in a heterogeneous tumor mass with smooth, noninvasive margins. Thus, the genetic make-up of a cancer cell may realize its invasive potential through a clonal evolution process driven by definable microenvironmental selective forces. Our mathematical model provides a theoretical/experimental framework to quantitatively characterize this selective pressure for invasion and test ways to eliminate it.