The Prime Minister entered the row after the Football Association issued a statement warning supporters that use of such words could result in either a banning order or even criminal charges.
For years Tottenham, who have a strong Jewish following, have been on the receiving end of anti-Semitic abuse from opposition fans.
In an act of defiance, some fans started using the word ''Yid'' themselves, and chants of ''Yids'', ''Yid Army'' and ''Yiddos'' are regularly sung in the home stands at White Hart Lane.
Mr Cameron told The Jewish Chronicle: "There's a difference between Spurs fans self-describing themselves as Yids and someone calling someone a Yid as an insult.
"You have to be motivated by hate. Hate speech should be prosecuted - but only when it's motivated by hate."
The north London club responded to the FA warning by announcing they would send a questionnaire to all season ticket holders asking if the practice should stop.
Fans reacted defiantly to the FA's statement on Saturday as they chanted ''Yid Army'' and ''We'll sing what we want'' throughout the 2-0 win over Norwich.
The same happened last season after Peter Herbert, the head of the Society of Black Lawyers, threatened to report anyone using the phrase to police.
Source: PA
Source: PA