The 29-year-old has been advised by doctors that he would be risking his long-term heath were he to play on.
Denton, who joined Leicester Tigers last summer but has not played since suffering a concussion last October, won 42 caps for his country.
“I have had this thing hanging over me for a long time now,” Denton said in an interview with The Times.
“Since the injury I have woken up every morning with pressure in my head and visual disturbances and not really knowing what is going on.
“Pretty much for that whole time I’ve been assuming that next week I will be better.
“Every time I tried to go through the comeback protocol, I’d fail. I’ve tried everything but nothing’s worked.”
Born in Zimbabwe, the powerful Denton would have been in contention for Scotland’s World Cup squad had he been fit.
He made his debut for the national team, for which he qualifies through his Glasgow-born mother, in 2011, and appeared four times for them at the 2015 Rugby World Cup.
Having restored his stock somewhat with a promising season with Worcester, Denton had started last season in good form for his new side Leicester, before suffering a concussion against Northampton Saints at Twickenham.
Denton earned a recall to the Scotland squad for the 2018 Six Nations and made his final appearance for Gregor Townsend’s side in Argentina last summer
“The idea of never being able to run out in front of 70,000 people at Murrayfield again is not enjoyable,” Denton reflected.
“I so want the boys to do well (in Japan) and would so want to be there but that is not going to happen.
“It is important to me that I don’t look back with sadness at what could have been, instead look forward and remain proud of what has been.
“There was a lot more I wanted to achieve. All of a sudden I am not able to do that.”