You can divide the history of investing in the United States into two periods: before and after 1952. That was the year that an economics student at the University of Chicago named Harry Markowitz published his doctoral thesis. His work was the beginning of what is now known as Modern Portfolio Theory. How important was Markowitz's paper? He received a Nobel Prize in economics in 1990 because of his research and its long-lasting effect on how investors approach investing today.
Markowitz starts out with the assumption that all investors would like to avoid risk whenever possible. He defines risk as a standard deviation of expected returns.
Rather than look at risk on an individual security level, Markowitz proposes that you measure the risk of an entire portfolio. When considering a security for your portfolio, don't base your decision on the amount of risk that carries with it. Instead, consider how that security contributes to the overall risk of your portfolio.
Markowitz then considers how all the investments in a portfolio can be expected to move together in price under the same circumstances. This is called "correlation," and it measures how much you can expect different securities or asset classes to change in price relative to each other.
For instance, high fuel prices might be good for oil companies, but bad for airlines who need to buy the fuel. As a result, you might expect that the stocks of companies in these two industries would often move in opposite directions. These two industries have a negative (or low) correlation. You'll get better diversification in your portfolio if you own one airline and one oil company, rather than two oil companies.
When you put all this together, it's entirely possible to build a portfolio that has much higher average return than the level of risk it contains. So when you build a diversified portfolio and spread out your investments by asset class, you're really just managing risk and return.