BENGHAZI, Libya - Diplomats had long been on their guard in Benghazi, a city that was the cradle of Libya's Western-backed revolution but also home to Islamists who have attacked foreign envoys.
Yet a long night of mayhem in which the U.S. consulate was overrun and torched, the ambassador lost and dying alone in the smoke while rescuers ran into a deadly ambush as they sought to save survivors, seemed to overwhelm U.S. security procedures.
Accounts from Libyan and U.S. officials, and from locals who watched what began as a protest on Tuesday against a crudely made American film that insults the Prophet Mohammad spiral into violence and a military-style assault on U.S. troops, point to a series of unfortunate choices amid the confusion and fear.
The actions of Libyan former rebel fighters assigned to help guard the lightly fortified compound may also face scrutiny. Libyan officers suggested that sympathy for the popular anger at the slight to their religion, as well as simple fear under heavy fire, meant the guards may have done little to defend the walls.
Much remains unknown, notably the extent to which armed militants may have prepared in advance for an attack as opposed to merely profiting from the opportunity of an angry crowd spinning out of control in a country where guns are everywhere.
However, this much is clear: a crowd gathered at dusk, about 7 p.m. (1700 GMT), chanting slogans against the film and angry at Washington's failure to act against its promoters. At some point, shooting began, with some in the crowd thinking they were under fire from the consulate. Around 10 p.m., rioters surged into the compound, bullets and grenades flew, and fires started.
Among the assailants, Libyans identified units of a heavily armed local Islamist group, Ansar al-Sharia, which sympathizes with al Qaeda and derides Libya's U.S.-backed bid for democracy.
Eventually, some three dozen Americans drove off to a safe house, knowing one diplomat was dead and Ambassador Christopher Stevens missing. When an eight-man rescue team flew in from Tripoli, they and their Libyan escorts were pinned down with the survivors by another attack in which two more Americans died.
Meanwhile, Stevens, 52, had been found by local people and taken, unrecognized, to a hospital, around 1 a.m. A doctor failed to revive him and pronounced him dead of smoke inhalation.
Around dawn, at 7 a.m., the beleaguered American survivors, with their battered military rescue party, were finally escorted back to Benghazi airport by a convoy of Libyan militia fighters in dozens of vehicles, to be flown to Tripoli and safety.
The fact that the attack took place on the 11th anniversary of al Qaeda's September 11 attacks on the United States, and that the ambassador, normally based in the capital Tripoli, was present in person, has prompted speculation that it may have be elaborately planned in advance by the likes of Ansar al-Sharia.
Three months ago to the day, the British ambassador survived a rocket attack as his convoy approached London's Benghazi consulate and there has been other violence against foreigners, mostly blamed on anti-Western Islamists, with some possible involvement by angry loyalists of the late Muammar Gaddafi.
Diplomats say foreign staffing in Benghazi has generally been kept low to reduce the security risks. But the city is home to key oil installations, Libya's economic mainstay, and a local source said Stevens had been expected at meetings with state oil executives on Wednesday, explaining at least part of his visit.
Some of those who took part in the initial demonstration in Benghazi insisted it was a spontaneous, unplanned public protest which had begun relatively peacefully. Anger over the film also saw an unruly protest at the U.S. embassy across the Egyptian border in Cairo on Tuesday evening, with protesters scaling the walls.
Local journalists also saw an eclectic gathering of people infuriated by religious sentiment but few of them bearing arms and most not appearing affiliated with hardline Islamist groups.
"When we had heard that there was a film that was insulting to the Prophet, we, as members of the public, and not as militia brigades, we came to the consulate here to protest and hold a small demonstration," said a 17-year-old student named Hamam, who spoke to Reuters at the devastated compound on Wednesday.
By his account, while some demonstrators fired rifles in the air - a far from uncommon sight in postwar Libya - a rumor spread that a protester had been wounded by firing from inside the consulate and Hamam and many others went off to retrieve guns which, like many Libyans, they keep at home for security.
"So we started shooting at them," Hamam said. "And then some other people also threw homemade bombs over the fences and started fires in the buildings. There was some Libyan security for the embassy, but when the bombs went off they ran off."
Abdel-Salam al-Bargathi, who runs the security operations of the former rebel February 17 Brigade, which effectively forms the police force for Benghazi while the new authorities work to establish new institutions, said he heard explosions start around 8:30 p.m. from his headquarters a mile or so away.
"There was a lot of chaos and confusion when the clashes began," he said. By 9 p.m., he was receiving calls from his units at the scene that rocket-propelled grenades were being fired at the consulate. A power cut had plunged the area into darkness.
"People started running into each other and nobody knew who was who," Bargathi told Reuters, saying that around this time he began ordering preparations to be made for an evacuation.
Tellingly, he and another senior officer, Wissam Buhmeid, the commander of the pro-government local defense force, the Libya's Shield Brigade, stressed that the Libyan guards on the consulate - estimated by Bargathi at up to 40 or more - may have felt little will to defend the compound from what they, and many other Libyans, judged to be justified religious indignation.
"I first of all place the blame on the United States itself for allowing such a movie to be produced. This was the product of the anger of Muslims," Buhmeid said, noting also that the guards had only light weapons in the face of rockets.
"I saw utter chaos. The power went out and it was completely dark," he said. "There were definitely people from the security forces who let the attack happen because they were themselves offended by the film; they would absolutely put their loyalty to the Prophet over the consulate. The deaths and injuries and attacks are all nothing compared to insulting the Prophet."
Bargathi, of the police command, said the killings had taken the protest too far, but said: "What we saw was a very natural reaction to the insult to the Prophet. We condemn the deaths but the insult to the Prophet made people very angry."
Ali Fetori, 59, an accountant who lives near the embassy, said: "The security people ... just all ran away and the people in charge were the young men with guns and bombs."
U.S. officials said the consulate's perimeter was breached 15 minutes after the crowd tried to storm in at around 10 p.m. The main villa was set on fire, with three Americans inside - the ambassador, IT specialist Sean Smith and a security officer.
"They became separated from each other due to the heavy dark smoke while they were trying to evacuate the burning building," one senior official said. The security officer made it outside.
"They found Sean. He was already dead. And they pulled him from the building. They were unable, however, to locate Chris, before they were driven from the building, due to the heavy fire and smoke and the continuing small arms fire."
Ali Khamis, a gardener, said he was in his room at the consulate when the attack happened. "They came into the compound, through all the gates. They were shouting 'Allahu akbar' and they were shooting in the air when they came in," he said.
Hamam, the young protester, said he had seen an American "totally covered in soot and black" lying apparently dead in the compound. Some people he said were chanting, "Allahu akbar (God is great), We are victorious over the infidels."
Photographs published on the Internet appeared to show Stevens unconscious and begrimed, being held by local men.
Ziad Abu Zaid, who was the duty doctor in the emergency room at Benghazi Medical Centre, said local civilians had brought in a man they said was American around 1 a.m. "He came in a state of cardiac arrest," Abu Zaid said. "I performed CPR for 45 minutes, but he died of asphyxiation due to smoke inhalation."
Only when someone told him the patient had come from the consulate did he make out Stevens's locally well-known features beneath the smoke and grime that obscured his face, he said.
Safe house siege
Libyan officials said the surviving Americans withdrew to a safe house. It would be normal security procedure in countries like Libya for international personnel to have a secure, secret location prepared for just such an eventuality.
Captain Fathi al-Obeidi, commander of a special operations force for the February 17 Brigade, told Reuters that he took a call about 1:30 a.m. from Tripoli telling him that a helicopter was on its way from the capital's Mitiga airport with a rescue squad of eight U.S. troops - he described them as marines.
After he met them at Benghazi airport with a convoy of 10 vehicles, mostly pickup trucks, one mounted with an anti-aircraft cannon, the U.S. force directed Obeidi and his men to the GPS coordinates of a farmhouse to find the survivors there.
Here, two more things went wrong. First, Obeidi found four times as many Americans at the single-storey, fortified house as he had been told expect - 37, not just 10. So he did not have enough transport. Then, the villa came under massive attack.
This time, there was little doubt in the minds of Libyans who experienced it that this was a well-organized assault by men who had mastered the complexities of military mortar fire.
"This attack was planned," Obeid said. "The accuracy with which the mortars hit us was too good for any ordinary revolutionaries."
While some Libyan officials suggested that former soldiers from Gaddafi's army may have been involved in Benghazi, some of the Islamist fighters also have substantial military experience from years spent fighting with the Taliban in Afghanistan.
Though Libya's deputy interior minister described the locating of the safe house as a "critical security breach," the attack may not have been planned for long in advance. The assailants would have had some hours to follow the fleeing Americans and set up an ambush after the consulate attack.
"It began to rain down on us," Obeidi said just as the rescue force was preparing to leave. "About six mortars fell directly on the path to the villa," he said. One American fell wounded by him. A mortar struck the building itself, throwing from the roof another American posted there onto the men below.
"I was being bombarded by calls from all over the country by Libyan government officials who wanted me to hurry and get them out," he said. "But ... I needed more men and more cars."
Two Americans, including one of the eight security personnel sent from Tripoli, were killed and several wounded.
Finally, dozens more vehicles from the Libyan security forces arrived, the attackers melted away and, as the sun came up over the desert, they reached Benghazi airport, from where the surviving Americans and the bodies were flown out.