Many consumers owe money on mortgages, auto loans, and credit cards, to name a few sources, and don't really think much of it. In fact, O'Leary insists that it's a good idea to be debt-free by age 45 -- and that includes having your mortgage paid off. Right now, borrowing rates are up across the board, so it's generally not a good time to refinance a mortgage.