Firms that shift profits to lower tax jurisdictions cost Africa $38bn (£25bn) a year, says a report produced by a panel he heads.
“Africa loses twice as much money through these loopholes as it gets from donors,” Mr Annan told the BBC.
It was like taking food off the tables of the poor, he said.
The Africa Progress Report is released every May – produced by a panel of 10 prominent figures, including former Nigerian President Olusegun Obasanjo and Graca Machel, the wife of South African ex-President Nelson Mandela.
African countries needed to improve governance and the world’s richest nations should help introduce global rules on transparency and taxation, Mr Annan said.
The report gave the Democratic Republic of Congo as an example, where between 2010 and 2012 five under-priced mining concessions were sold in “highly opaque and secretive deals”.
Kofi Annan: “Transparency is a powerful tool”
This cost the country, which the charity Save the Children said earlier this week was the world’s worst place to be a mother, $1.3bn in revenues.
This figure was equivalent to double DR Congo’s health and education budgets combined, the report said.
DR Congo’s mining minister disputed the findings, saying the country had “lost nothing”.
The report added that many mineral-rich countries needed “urgently to review the design of their tax regimes”, which were designed to attract foreign investment when commodity prices were low.
It quotes a review in Zambia which found that between 2005 and 2009, 500,000 copper mine workers were paying a higher rate of tax than major multinational mining firms.
Africa loses more through what it calls “illicit outflows” than it gets in aid and foreign direct investment, it explains.
“We are not getting the revenues we deserve often because of either corrupt practices, transfer pricing, tax evasion and all sorts of activities that deprive us of our due,” Mr Annan told the BBC’s Newsday programme.
“Transparency is a powerful tool,” he said, adding that the report was urging African leaders to put “accountability centre stage”.
Mr Annan said African governments needed to insist that local companies became involved in mining deals and manage them in “such a way that it also creates employment”.
“This Africa cannot do alone. The tax evasion, avoidance, secret bank accounts are problems for the world… so we all need to work together particularly the G8, as they meet next month, to work to ensure we have a multilateral solution to this crisis,” he said.
For richer nations “if a company avoids tax or transfers the money to offshore account what they lose is revenues”, Mr Annan said.
“Here on our continent, it affects the life of women and children – in effect in some situations it is like taking food off the table for the poor.”