Sir Richard Dearlove, who was head of MI6 when British agents helped to send Muammar Gaddafi's opponents back to Libya, where they were tortured, said on Thursday that intelligence co-operation with countries with poor human rights records had always been cleared by ministers. "It has always been pretty clear that our governments in the UK have accepted that danger and difficulty and have given political clearance for that sort of co-operation," he told a meeting of the Henry Jackson Society, a foreign policy international thinktank.

Whitehall officials have already insisted that intelligence cooperation with Gaddafi's Libya was authorised by ministers.

However, Labour ministers at the time, including Tony Blair and Jack Straw, have distanced themselves from MI6's co-operation with Gaddafi's security services, as has Lady Eliza Manningham-Buller, then head of MI5. Dearlove said that MI6's co-operation with Gaddafi's regime against extremism was "uncomfortable" and "pragmatic". But he denied MI6 enjoyed a "cosy" relationship with the regime.

He said that MI6 had had "phenomenal" success in disarming Libya, which had succeeded in acquiring the infrastructure needed to begin a nuclear weapons programme from the rogue Pakistani scientist, AQ Khan. "I resent the suggestion the relationship with Gaddafi was cosy," he said. "It was not a cosy relationship, it was a pragmatic one."

Dearlove added: "It was a political decision, having very significantly disarmed Libya, for the government to co-operate with Libya on Islamist terrorism. The whole relationship was one of serious calculation about where the overall balance of our national interests stood."

Papers found abandoned in the Tripoli offices of Moussa Koussa, Gaddafi's former foreign minister and intelligence chief, showed how MI6 was involved in sending suspected extremists back to Libya. They included Abdul Hakim Belhaj, a former prominent member of the Libyan Islamic Fighting Group, which had links with al-Qaida, and who is now military commander in the Libyan capital.

Referring to the forthcoming Gibson inquiry into allegations that British security and intelligence agencies colluded in the torture and abuse of terror suspects, Dearlove said that he should not say more about co-operation with countries which did not share the UK's views about human rights.

Dearlove said he was surprised by the relative failure of violent Islamism to make a more lasting political impact. North Africa and the Middle East showed al-Qaida had failed to achieve the long-term political impact many people had predicted at the time of the 9/11 attacks in 2001.

He said the terrorist network appeared to be "on the back foot" and was struggling to mount operations in the developed world. "I think it faces an issue of credibility. It badly needs to demonstrate to its sympathisers and its core membership that it can pull off something really big," he said.

He said he believed al-Qaida had made a "serious tactical error" in taking on the Americans in Iraq when they would have been better off trying to mount more 9/11-style attacks. "I think it was a vision of taking on the US military. Once the US military got its act together, it came out of that confrontation very badly," he said.

He insisted the west had been right to respond in the way it did to the events of 9/11, even though it resulted in the radicalisation of many Muslims. "Confronting al-Qaida was a confrontation of beliefs and values. I think it was the right thing to do, despite the risks, to come out on the front foot to meet that threat militantly," he said.