Al-Shabaab militants in Somalia are extorting huge sums from starving communities and forcibly recruiting hundreds of children as soldiers and suicide bombers as the terror group endures financial pressures and an apparent crisis of morale.

Intelligence documents, transcripts of interrogations with recent defectors and interviews conducted by the Guardian with inhabitants of areas in the swath of central and southern Somalia controlled by al-Shabaab have shone a light on the severity of its harsh rule – but also revealed significant support in some areas.

Systematic human rights abuses on a par with those committed by Islamic State in Iraq and Syria are being conducted by the al-Qaida-affiliated Islamist militants as the west largely looks away because most analysts do not see the group as posing a threat to Europe, the UK or the US.

The group has put to death dozens of “criminals”, inflicted brutal punishments on gay people, conducted forced marriages, and used civilian populations as human shields.

In one 2017 incident investigated by the Guardian, a man was stoned to death for adultery. In another, four men and a 16-year-old boy were shot dead by a firing squad after being accused of spying for the Somali authorities. In a third, a 20-year-old man and a 15-year-old boy were killed in a public square after being found guilty by a religious court of homosexuality.

Last year at least five people were lashed publicly after being accused of “immoral or improper behaviour”. They included a 15-year-old and a 17-year-old who were given 100 lashes each for “fornication”.

UN officials said they had received reports of stonings for adultery. The former al-Shabaab leader, Hassan Dahir Aweys, who defected in 2013, described the group’s aim as “Islamic government without the interference of the western powers in Somalia”.

Al-Shabaab, which once controlled much of south and central Somalia, including the capital Mogadishu, was forced to retreat to rural areas by a military force drawn from regional armies seven years ago. Since then it has proved resilient, and remains one of the most lethal terrorist organisations in the world, but appears to be suffering a crisis of morale and financial pressure, prompting the drive to squeeze revenue out of poor rural communities.

One recent defector from central Somalia told government interrogators that the group forces “Muslims to pay for pretty much everything except entering the mosque”. Another said that al-Shabaab’s “finance ministry” – part of the extensive parallel government it has set up – is “hated”.

Al-Shabaab used to demand money or children from clans: now they demand both

The former mid-ranking commander, who defected four months ago, described how wells were taxed at $20,000 (£14,000) per month and a fee of $3.50 levied at water holes for every camel drinking there. One small town in Bai province was forced to pay an annual collective tax of a thousand camels, each worth $500, and several thousand goats, he said.

In addition, trucks using roads in territory controlled by al-Shabaab have to pay $1,800 each trip. Five percent of all land sales is taken as tax, and arbitrary levies of up to $100,000 imposed on communities for “educational purposes”, the defector said. There is also evidence that the movement is suffering from manpower shortages.

A third defector said al-Shabaab now insisted that all male children attend its boarding schools from the age of about eight. The children train as fighters and join fighting units in their mid-teens.

“By that age they are fully indoctrinated. They are no longer under the influence of their parents,” said Mohamed Mubarak, research director of the Horn Institute for Security and Strategic Policy thinktank.

According to Somali authorities, troops stormed a school run by al-Shabaab in January and rescued 32 children who had been taken as recruits to be “brainwashed” to be suicide bombers. “Al-Shabaab used to demand money or children from clans: now they demand both,” the defector said.

Al-Shabaab has also told people they will be punished – possibly put to death as spies – if they have any contact with humanitarian agencies.

Somalia has been hit by a series of droughts, and only a massive aid effort averted the deaths of hundreds of thousands last year.

A new military campaign launched by President Mohamed Abdullahi Mohamed and supported by the US has seen intensive drone strikes on al-Shabaab targets, putting the militants under significant pressure. Fears of spies have led to a series of internal purges. Suspected agents are jailed and brutally tortured.

Al-Shabaab cuts thieves’ hands and kills looters . Everyone is scared of them

“Distrust is so high that when they go into battle, everyone is afraid of being shot in the back by his comrade,” one of the defectors said. “When soldiers get leave, half come back. Al-Shabaab now send patrols to collect people who have fled home. They stay in jail until they agree to rejoin.”

Abdirahman Mohamed Hussein, a government official overseeing humanitarian aid in southern central Somalia, told the Guardian that extremists used local populations as human shields. “They do not want people to move out because they are worried that there could be an airstrike if the civilians leave,” Hussein said.

Al-Shabaab also imposes tight restrictions on media, the defectors said. “Most people only listen to al-Shabaab radio stations or get news from al-Shabaab lectures which go on for hours and which cover religion and which all must attend,” one said. Another said some people risked harsh punishments to listen in secret toVoice of America and the BBC.

“Life is really tough in al-Shabaab-controlled areas. There is no food, no aid and children are being taken,” said Mubarak, the thinktank director. “Al-Shabaab are still trying to portray themselves as defenders of Somali identity. The message has a lot of sympathy but is not translating into active support.”

The draconian punishment, seizures, taxes and abductions run counter to the strategic guidance issued by al-Qaida leader Ayman al-Zawahiri, who has called for affiliates of the veteran group to build consensus and support among local communities. Their practices do, however, recall those of Isis.

Al-Shabaab also manipulates rivalries between clans and tribes, and benefits from the failures of local authorities to provide basic services. Several interviewees said they preferred using al-Shabaab’s justice system, and that the group had brought security.

In once case in May last year, two clan elders in Beledweyne in Hiran region agreed to seek al-Shabaab justice to settle a case of rape. The attacker was found guilty and stoned to death.

“We decided to go to the al-Shabaab court because the judge rules under the Islamic law and there is no nepotism and corruption,” said Abdurahman Guled Nur, a relative of the rape victim, in a telephone interview. “If we went to a government court, there would be no justice because the rapist could have paid some cash to the court and he would be freed.”

Mohamed Hussein, a farmer in Barire, a town 40 miles south of Mogadishu that has seen fierce fighting, returned home when al-Shabaab took control of the area in early October. “When the government soldiers were here, there was looting, illegal roadblocks and killing,” he said. “But al-Shabaab cuts thieves’ hands and kills looters. The Islamic court gives harsh sentences for the criminals, so everyone is scared of them. That way we are in peace under al-Shabaab. If you do not have any issue with al-Shabaab, they leave you alone.”

Additional reporting by Abdalle Mumin