Mattis also said the United States welcomed cooperation with China and said he had accepted Beijing's invitation to visit there soon. China claims the majority of the territory and, in recent years, has built artificial islands and conducted increased patrols to bolster its case.
The Chinese Defense Ministry condemned the USA actions, saying that the warships had violated the Chinese sovereignty, undermined strategic mutual trust, peace and security in the South China Sea.
Vietnam is the most active challengers to China's sweeping claims to much of the South China Sea, with Hanoi claiming the Paracels and the Spratlys in their entirety.
The remarks reflect concern that smaller states, including open economies like Singapore, risk being caught in the wash of the USA trade actions.
Mattis said little about North Korea in his speech, which came just hours after Trump said that the June 12 talks in Singapore with the North's leader, Kim Jong Un, were back on.
"There will be consequences to China ignoring the global community", Mattis said.
"Despite China's claims to the contrary, the placement of these weapon systems is tied directly to military use for the purposes of intimidation and coercion", he said.
Mattis also struck at one of the key, longstanding disputes between the US and China, telling the conference that America will continue to provide defense equipment and services to Taiwan and oppose any effort to alter the status quo. And he said he doesn't believe that China's actions will pay off.
"Make no mistake, America is in the Indo-Pacific to stay, this is our priority theatre, our interests and the region's are inextricably intertwined".
Singapore Defence Minister Ng Eng Hen criticized both the US and China for taking "unilateral actions", lumping the two rivals together as nations challenging the current global order. Singapore has warned previously that countries should not be squeezed by competition between the USA and a rising China, or forced to take sides.
A Chinese military official dismissed the comments as "irresponsible".
Earlier in the week, Mattis warned the United States will continue "a steady drumbeat" of naval exercises to challenge China's territorial claims in the waterway, and this weekend said, "there are consequences that will continue to come home to roost" if China does not act more transparently and cooperatively in the region.
"If the USA will continue to pursue a constructive results-oriented relationship with China, cooperation, whenever possible, will be the name of the game, and competing vigorously where we must". "What we see it as, is a reaffirmation of the rules-based order".
China's hostility towards self-ruled Taiwan has grown since Tsai Ing-wen from the pro-independence Democratic Progressive Party won presidential elections on the island in 2016.
The Pentagon said the decision to disinvite the Chinese Navy was triggered by what it called strong evidence that China has deployed weapons systems on the islands, and called on China to remove them.
Gen Mattis said the issue of removing USA troops from South Korea was "not on the table" and that "our objective remains the complete, verifiable, and irreversible denuclearisation of the Korean peninsula".
Last month China said it had for the first time landed bombers on Woody Island in the Paracel Islands, prompting U.S. warnings that it was destabilising the region.
Singapore-based security expert Tim Huxley said while increased pressure might slow China's militarization efforts, they would be hard to stop.
"But violation of China's sovereignty will not be allowed", said Lieutenant General He Lei, vice-president of the Academy of Military Science and head of China's delegation.