After three months of optimism, Americans grew more pessimistic about the housing market in November.

Fewer Americans said it was a good time to buy a house, in part because they worried more about their job prospects, household incomes and future mortgage rates, according to the latest results from Fannie Mae’s Home Purchase Sentiment Index (HPSI).

Read more: Buying a house: What you need to know about home ownership

“The HPSI [measure of housing optimism] appears to have peaked for now as consumers continue to consider how COVID-19 impacts their ability to buy or sell a home,” said Doug Duncan, senior vice president and chief economist at Fannie Mae.

Only 57% of respondents said it was a good time to buy a home, down from 60% last month and 61% in November 2019, according to the survey. Meanwhile, 35% said it was a bad time to buy a home, unchanged from October but down from an abrupt hike to 46% at the beginning of the pandemic, according to Fannie Mae.

Despite record-low mortgage rates, housing prices are near a record high. The median sales prices was $348,310 in November, up 12.7% from last year and down only $2,000 from the all-time high in August, according to Realtor.com.

“The housing market is still hot, but we may be starting to see rising home prices hurting affordability,” said Lawrence Yun, chief economist at the National Association of Realtors.

Plus, as unemployment persists due to the coronavirus pandemic, fewer respondents were confident in their job security (76% compared to 82% in October and 96% in November 2019), and household incomes — 18% said their household income was significantly lower than 12 months ago, compared to only 10% who said the same last year, according to Fannie Mae.

Good time to sell

Meanwhile, 59% of respondents said it was a good time to sell in November, unchanged from last month but nearly double April’s 32%. Some 73% said it was a good time to sell in November 2019, according to Fannie Mae.

“This follows the HPSI’s recovery of slightly more than half of the loss experienced during the first few months of the pandemic,” said Duncan. Some 33% said it was a bad time to sell in November, down from 62% in April but still elevated from the 26% who said it was a bad time to sell in November 2019, according to Fannie Mae.

The drop in home purchase sentiment comes at a time when other housing market measures are slipping, which may suggest the pandemic-driven housing boom could be slowing down. Mortgage applications fell 1.2% in the last week of November, according to the Mortgage Bankers Association, and pending home sales dropped 1.1% in October, according to the National Association of Realtors.

“The last two years showed that when the economy is firing on all cylinders and most Americans have access to decent jobs, more of them will be able to find a home of their own. The sooner we can put the pandemic and 2020 recession behind us, the sooner access to housing can resume its expansion,” said Jeff Tucker, senior economist at Zillow.

Sarah Paynter is a reporter at Yahoo Finance.

Read the latest financial and business news from Yahoo Finance

Follow Yahoo Finance on Twitter, Facebook, Instagram, Flipboard, SmartNews, LinkedIn, YouTube, and reddit.

More from Sarah:

What to expect from commercial real estate in 2021

San Francisco rent hits historic lows but city still most expensive in US

Santa Claus goes virtual: Retailers seek loyalty as COVID-19 keeps shoppers away from stores