BRIAN SPEGELE in Beijing,

VU TRONG KHANH in Hanoi, and

JOSEPHINE CUNETA in Manila

Strains between China and its neighbors burst to the surface in two parts of the South China Sea, taking the high-stakes struggle for control over the waters to new levels of friction.

Off Vietnam, dozens of Chinese military and civilian ships clashed with the Vietnamese coast guard, with Vietnamese officials complaining its vessels were repeatedly rammed. On the same day, Philippine police apprehended Chinese fishing vessels loaded with hundreds of sea turtles in disputed waters.

About 80 Chinese vessels moved into an area near the disputed Paracel Islands, where Hanoi has sought to prevent China from deploying a massive oil rig, said Rear Adm. Ngo Ngoc Thu, vice commander of the Vietnamese coast guard. He said the flotilla included seven military ships and that it was supported by aircraft.

He said the situation, which started brewing over the weekend, was “very tense” and said six Vietnamese officers had been injured in the standoff.

The confrontation—by far the most serious in recent years between the two neighbors—marked a significant escalation in Beijing’s willingness to press its natural-resource claims, analysts said.

Theresa Fallon, a senior associate at the European Institute for Asian Studies, a Brussels-based think tank, said China’s move represented the regional energy sector’s “worst nightmare” and was bound to provoke Vietnam’s anger.

“This is a huge rig—it’s the size of a couple of football fields,” Ms. Fallon said.

A senior administration official said the White House views the latest escalation as part of a pattern of behavior as China continues to try to advance its territorial claims. “We’re obviously very concerned about it,” the official said. “We have conveyed our concerns to the Chinese.”

The standoff is “an unprecedented situation,” said Ian Storey, a senior fellow at the Institute of Southeast Asian Studies in Singapore. The sheer number of Chinese vessels that appeared to be involved was a clear indication of China’s “resolve to make sure this rig can operate in these waters.”

Chinese State Councilor Yang Jiechi described the move as part of a Chinese business’s normal operations and asked Vietnam to stop interfering, a Foreign Ministry spokesman said at a daily briefing.

China’s Defense Ministry didn’t respond to a request to comment.

The U.S., which has a vital interest in maintaining open sea lanes for trade in the South China Sea, has encouraged a multilateral approach to resolve the disputes—though China insists on negotiating with one country at a time.

The State Department said on Wednesday said China’s decision to move the oil rig into contested waters was “provocative and unhelpful.”

“We are strongly concerned about dangerous conduct and intimidation by vessels in the disputed area,” said State Department spokeswoman Jen Psaki.

President Barack Obama toured the region last month, stressing U.S. security cooperation. The latest escalation of tensions in the South China Sea reinforces that the disputes “are not going to be solved with one trip or one speech,” said Michael Green, senior vice president for Asia at the Center for Strategic and International Studies in Washington. “And it shows that the Chinese side is not fazed by the negative reaction in the region,” Mr. Green said.

China and Vietnam fought a brief but bloody border war in 1979. Today, China’s military far outguns Vietnam’s, but that doesn’t mean Vietnam will quickly back down. “Vietnam has a record of not pulling back from military confrontations,” said James Hardy, Asia-Pacific editor at IHS Jane’s Defence Weekly.

Security scholars said the latest escalation in tensions is the cumulative result of deep-seated mistrust over China’s intentions among smaller regional players combined with Beijing’s increased assertiveness as well as a lack of mechanisms to prevent and manage crises.

The confrontation also illuminated the role of China’s state-owned energy companies in helping advance China’s territorial ambitions, despite frequent assertions by executives that they are driven by profit and not politics.

The company controlling the oil rig, China National Offshore Oil Corp., or Cnooc, has previously figured as a controversial actor in the South China Sea, by offering up for foreign cooperation oil blocks in contested waters.

The deep-water rig launched with much fanfare in 2012, at which time the company chairman described it as a “strategic weapon” for China’s oil industry. The company didn’t respond to requests to comment Wednesday.

The administration of President Xi Jinping, who took over as China’s Communist Party leader in November 2012, has pledged to bolster ties with regional neighbors. At the same time, China has stepped up its assertiveness in territorial disputes, which has undermined regional trust-building.

The Philippines on Wednesday apprehended a Chinese fishing vessel “carrying large numbers of endangered species” near Half Moon Shoal, a sandbar in the disputed Spratly Islands in the South China Sea and a hotbed for illegal sea-turtle poaching. Philippines police senior inspector Dante Padilla said an inspection of the boat yielded around 500 sea turtles, some of them dead.

He said the police had arrested the captain of the Chinese vessel and its 10 crew members.

The Philippines Department of Foreign Affairs said in a statement it seized the Chinese fishing boat “to enforce maritime laws and to uphold Philippine sovereign rights over its [exclusive economic zone].”

Chinese Foreign Ministry spokeswoman Hua Chunying urged the Philippines to release the fishermen and to refrain from taking further provocative actions. Ms. Hua said the Chinese Embassy in Manila had complained to the Philippine government.

The Philippines and China have overlapping maritime claims in several areas in the South China Sea. Amid China’s recent aggressive assertions of its claims in the dispute areas, the Philippines had brought its case before an international tribunal to rally international support for its territorial claim.

On his visit to the region last month, President Obama assured the Philippines that American military support is “ironclad,” though he left vague whether the U.S. would come to the aid of the Philippines in its island disputes.

The territorial disputes aren’t part of the agenda of this weekend’s meeting of the Association of Southeast Asian Nations in Myanmar this weekend. The summit is aimed at regional integration and participation in the G-20 summit in November. Tensions flared at an Asean summit in 2012 after Philippine President Benigno Aquino III disputed a statement by Cambodia, then the summit’s chairman, about an agreement in the bloc “not to internationalize the South China Sea from now on.”

—Colleen McCain Nelson,
Trefor Moss,
and Cris Larano
contributed to this article.

Write to Vu Trong Khanh at trong-khanh.vu@wsj.com and Josephine Cuneta atjosephine cuneta@wsj.com