Officials say the bomber apparently hid explosives in his turban and detonated them near the entrance of the mosque.
President Karzai was not present at the memorial. Four of his other brothers were but escaped unharmed.
Ahmad Wali Karzai, a controversial but key figure in Nato's battle against the Taliban, was killed by his bodyguard.
President Karzai flew into Kandahar for his brother's funeral on Wednesday, but left the city before Thursday's service.
No group has so far admitted carrying out the mosque attack.
The governor of Kandahar province, Toryalai Wesa, said 15 people were injured.
Another explosion later hit the Herat Bazaar in the centre of Kandahar, killing one person and injuring several others, said the city's police chief.
Chaos and panicThe explosion in the Red Mosque took place just after noon (0730 GMT) in the southern Afghan city, as the service for Ahmad Wali Karzai was coming to an end.
Continue reading the main story
Bilal Sarwary BBC News, Kandahar Herat Bazaar is in the heart of Kandahar.
Several shopkeepers said as soon as the bomb there went off, they closed their shops.
Attacks in recent weeks have affected business, they said.
One trader said he used to export a great deal of dried fruit but the Pakistani and Indian buyers were no longer visiting.
As soon it gets dark, shops close and people go home.
Locals say property prices have plummeted in Kandahar. One merchant said a plot in the upscale township of Ayno Meena cost $300,000 (?186,000) not long ago, but was now available for $250,000.
The township, which is home to senior government officials, merchants and wealthy Afghans, has seen several explosions recently.
Some residents suggested that criminals, rival gangs and private feuds were behind the violence, not the Taliban.
The security situation remains fragile in Kandahar city.
Officials say the attacker was stopped at the mosque's entrance, where he blew himself up.Witnesses, including President Karzai's spokesman Waheed Omer, said he had hidden the explosives in his turban. Provincial intelligence chief Gen Mohammad Naeem Momin told the Associated Press that early investigations supported the reports.
The BBC's Bilal Sarwary in Kandahar said that security was not as tight as it could have been around the mosque.
Tradition dictates that mourners are not searched when attending memorials, says our correspondent, and several tribal elders told him they had not been searched.
The interior ministry and medical sources said four people and the bomber had been killed.
Among those killed was Hikmatullah Hikmat, the head of Kandahar's Ulema Council, said the ministry. The council is an influential body of clerics in charge of regulating religious issues in the province.
The ministry said a child was also among the dead.
"There was a prayer going on and after that prayer, the man came close to the director of the religious council and exploded," said Kandahar Governor Wesa.
Officials say the casualty figures could have been much higher had the attacker been able to get inside the mosque itself, says our correspondent.
Four of President Karzai's brothers, several other Karzai family members and Mr Wesa were in the mosque at the time. None were believed to have been hurt.
Appeal to TalibanAhmad Wali Karzai was shot twice by his family friend and long-time head of security, Sardar Mohammad, on Tuesday. The bodyguard was himself killed almost immediately.
The Taliban said the attack was one of their top achievements in 10 years of war. However, correspondents say the bodyguard's motives remain unclear and it seems unlikely that someone so close to the Karzai family was working for the insurgents.
Ahmad Wali Karzai was seen by many as a key anti-Taliban figure, someone who stood up to them, using his private army and vast network to fight the insurgents, our correspondent says.
But he adds that his critics accused him of undermining the central government in Kabul.
Security was tight during Wednesday's funeralOn Wednesday, President Karzai wept at the graveside, and kissed his dead half-brother's face as senior politicians looked on.
The president then appealed to the militants to stop the bloodshed.
"My message for the Taliban is, my countryman, my brother, stop killing your own people," said Mr Karzai. "It's easy to kill and everyone can do it, but the real man is the one who can save people's lives."
Tuesday's killing was the latest and most high-profile in a series of assassinations of senior politicians and security commanders across the country.
Send your pictures and videos to yourpics@bbc.co.uk or text them to 61124 (UK) or +44 7725 100 100 (International). If you have a large file you can upload here.
Read the terms and conditions
A roadside bomb has killed five civilians in southern Afghanistan, government officials have said.
Most foreign troops in Kapisa are French At least five French soldiers and one civilian have been killed in a suicide bomb attack in the north-eastern Afghan province of Kapisa, officials say.
At least 13 people have been killed after a passenger bus in southern Afghanistan hit a roadside bomb, police officials say.