New mortgage rules are designed to protect borrowers and lenders from the ills of the last housing crash. If lenders apply the rules, they are protected from legal recourse by borrowers or investors should the loans go bad.
The rules, however, are not mandatory, and some lenders say they will make loans outside of them, especially in the jumbo and adjustable-rate spaces.
Under the Dodd-Frank financial reform legislation, all lenders are required to ensure that their borrowers can repay their mortgages. Lenders can prove they've done that automatically by following new rules from the Consumer Financial Protection Bureau. The rules allow for adjustable-rate loans, but they have to be underwritten to the highest possible adjusted payment. Loans can have no high fees, no interest-only features, and, perhaps the biggest hurdle, a debt-to-income ratio no greater than 43 percent. If all those rules are followed, the loan is deemed a "qualified mortgage" (QM) and the lenders are protected from most legal action.
While most loans being made today already fall into the QM category, a growing number of borrowers faced with rising interest rates are taking out adjustable-rate loans, which currently have lower monthly payments, and could therefore face the tougher underwriting. Also, the size of the average mortgage applied for today is the largest in history, at nearly $270,000, according to the Mortgage Bankers Association. That could cause more borrowers to fail on the debt-to-income requirement.
"The buyers that are out there are much more affluent. They have much larger home savings and home equity saved up, and they're looking for a larger type of house," Buck Horne, an analyst at Raymond James, said on CNBC's "Squawk on the Street." "The first-time buyer, unfortunately, remains pretty well locked out of the market."
The affluent buyers will be the primary targets of lenders operating outside the QM rules.
"We are going to offer over 43 percent, but the difference is we're going to want to make sure there is enough residual income that somebody does have the ability to repay," said Karen Mayfield, a senior vice president at Bank of the West. "Those people who are self-employed or who do have substantial income, there will be ways to justify allowing over 43 percent debt-to-income."
While Mayfield does not anticipate hiking rates for these borrowers, she does say the bank will underwrite the borrowers to the highest possible payment of an adjustable-rate loan.
(Read more: Mortgage rates get a break on fees )
"We really feel that we're just increasing the diligence involved in the underwriting but not necessarily increasing the risk, therefore we're not anticipating increasing rates that we're currently offering customers," added Mayfield.
While the mortgage industry initially fought the new rules, claiming no one could operate outside them, more are now saying they will. Wells Fargo said it will do non-QM loans and keep them on their books.
"It looks like a nice niche from a lenders perspective because you're not going to have as much competition. Certainly you're going to have much more regulatory risk," said Guy Cecala of Inside Mortgage Finance. "The risk of operating outside the QM box is that a borrower can dispute, and basically win, if you try to foreclose if there's a problem down the road."
Most who are operating outside QM say the loans will be underwritten pristinely, and every document will be checked. They will lend to only the prime and super-prime borrowers. There is, however, the risk that as the market settles into these rules, others, seeking greater returns, will look to less credit-worthy borrowers and charge them higher rates for their higher risk. In other words, the new sub-prime.
(Read more: November new home sales dip after Southern surge )
"Where I think we're going to see the worst behavior in the marketplace are new breeds of companies coming into the nonqualified mortgage area, where they can see much higher yields," David Stevens, CEO of the Mortgage Bankers Association, said in an interview earlier this year.
These would likely be nonbank lenders using private capital to fund the loans. As for the mainstream banks, the new mortgage rules will likely be an inconvenience for some borrowers and a roadblock to others.
One thing is certain: Mortgages will get more expensive, due to an improving economy, more lending rules, tighter underwriting standards and the Federal Reserve's pullout from mortgage-bond buying. How lenders choose to operate in that environment will be the story of 2014.
More From CNBC: