Police in charge of the investigation said the letter was delivered on August 27 to the Embassy in central Tokyo's Minato district and was addressed to former South Korean Ambassador Lee Su-hoon, Xinhua news agency reported.
Investigative sources said the envelope contained a single piece of paper and an apparent bullet, according to the report.
Written on the paper, the sender claimed to be in possession of rifles and is plotting to target South Korean people.
The sender also wrote that South Korean people should leave Japan, according to Japan's public broadcaster NHK.
In a separate incident, the local police said they arrested a man in his late 60s on Sunday who allegedly damaged the South Korean Embassy's mail box by punching it. He was posting a letter to the Embassy at the time protesting against the government of South Korea, the police said.
The incidents come as relations between Japan and South Korea have sunk to their lowest in recent times over forced wartime labour during Japan's 1910-1945 occupation of the Korean Peninsula and South Korea's top court ordering some major Japanese firms to compensate South Korean plaintiffs.
Japan has claimed the rulings are not in line with international law and run contrary to the foundation of friendly and cooperative relations between the two neighbours since the 1965 normalization of diplomatic ties.
The dispute has spilled over into a bitter tit-for-tat trade spat between both countries, with the South Korean government also deciding to scrap a military intelligence sharing agreement with Japan.
( With inputs from IANS )