Speaking at the Grace Hopper Conference in Phoenix, Arizona on Thursday, he was asked in an interview for his thoughts on what women should do if they are not comfortable asking for a pay rise.
In front a of live audience, Nadella said: "It's not really about asking for the raise but knowing and having faith that the system will actually give you the right raises as you go along."
"And that I think might be one of the additional super powers that, quite frankly, women who don't ask for a raise have. Because that's good karma. It'll come back, because somebody is going to know that is the kind of person that I want to trust," he added.
The conference was specifically focused on the issue of women in computing and is the world's largest gathering of women technologists, according to its program. Interviewee Maria Klawe, the president of the Harvey Mudd College, retorted that Nadella's comments were one of the "very few things" on which she disagreed with him. Klawe's response received a round of applause in the Phoenix Convention Center and the debate continued on social networking sites like Twitter.
Just hours later, a formal apology from Nadella was posted on the Microsoft news center website.
"Maria asked me what advice I would offer women who are not comfortable asking for pay raises. I answered that question completely wrong," he said.
"Without a doubt I wholeheartedly support programs at Microsoft and in the industry that bring more women into technology and close the pay gap. I believe men and women should get equal pay for equal work...if you think you deserve a raise, you should just ask."
He added that he learned a "valuable lesson" at the conference and would be speaking at Microsoft's monthly Q&A for employees next week to answer any questions on the topic.
This issue is a hot topic in any sector or country but particularly so in California's Silicon Valley. A study by the non-profit initiative Measure of America back in 2012 concluded that men earned more than women in every racial and ethnic group. In Silicon Valley, the report showed that women earn 49 cents for every dollar that men earn. Microsoft currently employs 128,076 people worldwide with 28 percent being female.
In his apology, Nadella added that he agreed with the advice given by Klawe at the event. The computer scientist said that women should "do their homework."
"Make sure you actually know what a reasonable salary is if you are being offered a job. Do not be as stupid as I was. Second thing is, role-play. Sit down with somebody you really trust and practice asking them for the salary you deserve."
London-based financial analyst Louise Cooper shared her disapproval at Nadella's comments on Friday morning. In her regular analyst note she said that he had shown his "true colors."
"I love it when CEOs make these gaffes because it gives us an indication of what they really think," she said. "I suggest it was an ego trip. He had hundreds of women listening to his words of wisdom and jostling for his attention."
Cooper added that women have been waiting for "the system" to reward them since equal pay legislation was enacted decades ago and are still waiting.
More From CNBC