West is believed to be planning on travelling to Melbourne this month to meet the family of his new wife, Bianca Censori.
The Anti-Defamation Commission, however, is urging Immigration Minister Andrew Giles to block any attempt by the rapper to enter Australia.
"Kanye can keep spewing his disgusting and revolting conspiracy theories in the United States, not here," Anti-Defamation Commission chairman Dvir Abramovich said.
"This agent of hate and division (of someone) who loves Nazis and worships Hitler, should never be allowed to step foot in this country."
West's anti-Semitic rants previously led him to be banned on social media after an interview where he made discriminatory comments against the Jewish community last year.
Federal Opposition Leader Peter Dutton told 3AW said West's comments were "disgraceful".
"My inclination would be not to allow him in," he said.
"His anti-Semitic comments are disgraceful, his conduct (and) his behaviour are appalling.
"He's not a person of good character, and the minister has the ability to stop somebody coming into our country of bad character."
Dutton told 3AW that Immigration Minister had a "lot to weigh up".
A government spokesperson said they could not comment on specific cases.
"All non-citizens who enter Australia must meet the requirements set out in the Migration Act, including security and character requirements," the spokesperson said.