The head of the Tokyo Olympics organising committee, Yoshiro Mori, resigned after he was criticised for making sexist remarks.
The 83-year-old former prime minister was quoted saying women talk too much and that meetings with many female board directors would “take a lot of time”.
The remarks set off a global outcry, forcing him to apologise.
“What is important is to hold the Olympics from July. It must not be the case that my presence becomes an obstacle to that,” he said at a special committee meeting on Friday, where he also announced his resignation.
It is not immediately clear who will replace Mr Mori.
The resignation came as pressure mounted on him over the past week. Major sponsors came forward to criticise his comments, including Toyota, one of the biggest Olympic backers.
On Tuesday, a group of female lawmakers wore white in a protest against his remarks, with some men doing the same in solidarity.
And Tokyo Governor Yuriko Koike said she would not attend a meeting of high-level Olympic officials in protest.
According to local media, almost 400 people have also withdrawn applications to volunteer at the Olympic games, which are scheduled for later this year.
The committee board currently has 24 members, five of whom are women. In 2019, the committee – which is responsible for selecting Japanese Olympians – set itself a goal of increasing the number of female board directors to 40%.
“If we increase the number of female board members, we have to make sure their speaking time is restricted somewhat, they have difficulty finishing, which is annoying,” Mr Mori was quoted as saying.
“We have about seven women at the organising committee, but everyone understands their place.”
Mr Mori is known in the country for a string of gaffes and undiplomatic statements made while in office from 2000 to 2001.
He told Japan’s Mainichi newspaper that female family members had also lambasted him after his comments.
“Last night, my wife gave me a thorough scolding. She said: ‘You’ve said something bad again, haven’t you? I’m going to have to suffer again because you’ve antagonised women’,” he said.
“This morning, my daughter and granddaughter scolded me as well,” the paper quoted him saying.