The actor said he was upset when he saw some people rejoicing on social media after the recent murder of his friend, journalist Gauri Lankesh.
He added that he was even more upset to find out that some of these abusive trolls were being followed by Mr Modi.
“Some people who are followed by our prime minister on Twitter are so brutal, and we have a PM who has closed his eyes to it,” Raj said, adding that he was “disturbed, hurt… afraid of the silence of my prime minister”.
The backlash from the trolls was swift, they immediately dubbed him “anti-Modi” and he was criticised for his comment.
The Indian prime minister is among the most popular global leaders on Twitter, with more than 35 million followers. He is a prolific tweeter and most of his tweets are about his official work, government policies, or issues close to his heart like the clean India campaign.
The clever social media strategy of his Bharatiya Janata Party (BJP) is regarded as one of the main factors behind the party’s sweeping win in the 2014 elections.
However, there has been criticism that he is selective and non-inclusive in his tweets.
For instance, his silence after a Muslim man was lynched by cow vigilantes in April for transporting cattle in Rajasthan was questioned by many – and a week after the incident, many ridiculed him for tweeting grief over an attack in Stockholm.
But for months now, critics have been wondering whether a more important question is not what the prime minister is tweeting, but what he is reading?
Or, more precisely, who he is reading?
“A huge majority of the 1,845 people that Mr Modi follows are politicians, bureaucrats and journalists, but a significant number of them are people who denigrate critics and political opponents in most horrendous language,” Pratik Sinha of Alt News, a website set up to fight fake news in India, told the BBC.
“Most of them are BJP ground workers for whom a follow from the prime minister is a badge of honour, a recognition of their work,” Mr Sinha said, adding that, “most are crass misogynists who use the worst possible language.”
Take for example some of the tweets after Gauri Lankesh was shot dead – this Hindi tweet from Nikhil Dadhich, a Surat-based businessman who is followed by PM Modi, roughly translates as: “A bitch died a dog’s death and all of her litter is crying in the same voice.” Mr Dadhich’s offending post was deleted after the uproar and is no longer available on his timeline.
Then there was this from Ashish Mishra, also followed by the PM, who commented “Jaisi karni vaisi bharni [As you sow, so shall you reap]” while re-tweeting a newspaper announcement of Ms Lankesh’s murder:
Their vicious responses came in for much criticism and so did the fact that Mr Modi followed these – and other, similar – accounts.
Some angry Indians even began a #BlockNarendraModi campaign and although it did not get the wide support the prime minister’s critics had hoped for, it did create a buzz in India.
After several people tweeted that they had blocked Mr Modi’s personal account, the head of the BJP’s IT cell Amit Malviya issued a statement saying “PM following someone is not a character certificate” for that person.
Pratik Sinha of Alt News says Mr Malviya is not entirely honest in his assertion that the prime minister’s account has never blocked or unfollowed anyone and that Mr Modi did unfollow a former supporter, Dr Jwala Gurunath, after she accused a BJP spokesperson of unethical behaviour.
While writing this article, I called Mr Malviya to ask him some questions about the prime minister’s Twitter account, but he refused to answer. “I have issued a statement and I have nothing more to add to it,” he told me.
The prime minister is a busy man who has a million things to do, and Mr Sinha says it is unlikely that he spends a lot of time on Twitter or even operates his own account.
But, according to reports, Mr Modi reaches for his iPad within minutes of waking up and spends some time looking at his social media feed, and critics say considering the vitriol unleashed by some of his supporters, his silence is troubling.
It has been over a month since Gauri Lankesh was shot dead outside her home. No-one has been arrested by the police. And no-one has been unfollowed by the prime minister.
“I’ve also been asking why can’t Mr Modi unfollow these people. But I realise that it’s part of their political strategy,” Mr Sinha says.
“These people are not really trolls. They are much more important than that, they are the party’s foot soldiers on Twitter,” he adds.