Mauricio Pochettino, Spurs' third head coach in a year, has endured a tough start to life at the White Hart Lane helm, failing to get his side playing the attractive, incisive style which became a hallmark of the Argentinian's time at Southampton.
Worse still, the north Londoners have struggled to pick up points, with Sunday's 2-1 loss to Stoke their fourth defeat in just six Premier League home matches.
That poor start has seen those from Pochettino to technical director Franco Baldini come under fire, so too divisive chairman Levy.
Supporters are aiming a lot of ire in his direction but former Tottenham chairman Sugar believes that anger is misdirected.
"All you can expect from a chairman is the following: to provide the money for your football staff to do what they want to do, negotiate hard and bitterly if one of your best players has got a death wish to want to leave, and to provide an environment like a new stadium or training ground and all that kind of stuff," he told talkSPORT
"That is all the chairman can do.
"People saying 'Levy out, Levy out', they don't know what they are talking about
If Spurs are going to progress, he is their best ally
All he can do is listen to the managers that he appoints.
"They got £100million a year ago for the sale of Gareth Bale
They went out and spent it on seven players
Baldini, or whatever his name is, and Andre Villas-Boas must have had some input into that and therein lies the problem, in my opinion.
"If we have bought the wrong players and the money has been used irresponsibly, you have got to look at the person that guided you in that direction."
Source : PA
Source: PA