Golden State have won five straight since losing the 10-time all-star Durant, who was sidelined after suffering an injury against the Houston Rockets.
"It's hard to get away from that because I watch the games (when Golden State are on the road) and I saw the lead-up to the game and I saw everybody talking about it," said Durant, who is averaging 34.2 points per game in the playoffs.
"As a player, I think about that and I'm just like 'That's not true, that's not the facts.'
"I feel the organization knows exactly what I've done on and off the court to become part of this culture and stamp my flag on this organization."
Durant could leave as a free agent after the season. He feels there are those who are trying to drive a wedge between him and his teammates.
"It's been that way since I got here, It's the Warriors and K.D. I understand that," said Durant, who joined the Warriors in 2016.
"The last two years, I've done pretty much everything that they asked me to do from player, ambassador to the community to working hard every single day in shootarounds and practices.
"I know what I bring to the team but I also know a lot of people on the outside don't like to see us together."
Teammate Stephen Curry said he doesn't understand the criticism of Durant.
"We look out for each other. There is sacrifice. At the end of the day, it's all about winning," Curry said.
Toronto hold a 3-2 lead and can close out their series on Saturday when the Raptors host the Bucks.