The second leg of the qualifier in Yaounde on Saturday ended 2-1 to Cameroon to make it 3-3 on aggregate.
With no goals in extra time it was down to spot kicks to decide the winners.
Both sides failed with their third penalties – Cameroon scored their next two but Faith Ikidi missed Nigeria’s final attempt to send the Indomitable Lionesses to London.
The hosts took the lead in the 29th minute through Edoa Zouga and then doubled their lead moments before half time with a Christine Mani penalty.
Nigeria’s Osinachi Ohale was sent off for the foul that lead to the spot kick.
The Super Falcons then scored in the 79th minute through Perpetua Nkwocha to make it 3-3 on aggregate and send the game into extra time.
Despite pressing for a winning goal the Nigerians could not find the target and so the game went to penalties.
The 4-3 loss in the shoot out means this will be the first time that Nigeria have not made it to the Olympics since the 2000 games in Sydney.
The second leg was delayed in order to allow Cameroon to take part in the All Africa Games in Mozambique, where they won gold.
South Africa claimed Africa’s other spot at the Olympics last month thanks to a 4-1 aggregate win over Ethiopia.
Source: BBC Sport