CARTAGENA, Colombia (CBSMiami) – Five U.S. service members assigned to the Joint Task Force Summit of the Americas in support of the United States Secret Service violated curfew established by the U.S. Senior Defense official in Colombia and may have been involved in inappropriate conduct.

The five U.S. service members from SOUTHCOM are the first non-Secret Service agents not implicated in engaging in inappropriate conduct before President Barack Obama arrived for a summit in Colombia.

READ MORE: Celtics run past Heat 93-80, move 1 win from NBA Finals

“The president’s safety was never in jeopardy,” John Adler of the Federal Law Enforcement Officers Association told CBS4 in an exclusive interview. “His mission was not in jeopardy. We’re talking about an isolated incident here that involved allegations of off-duty conduct.”

A dozen United States Secret Service agents that had been sent to Colombia to set up security for President Obama were relieved of duty over allegations of misconduct.

The Washington Post first reported the story late Friday night when a former Post reporter, Robert Kessler, told the Post he learned 12 agents were involved.

Early findings in the investigation alleged that several of the agents brought prostitutes back to a hotel in Cartagena. None of the agents involved were in President Obama’s personal protective detail.

According to multiple reports, at least one Secret Service agent got into a dispute with a woman brought back to his hotel over her request to be paid.

One of the women brought back to the hotel complained to police and the U.S. Embassy and Secret Service leaders began investigating.

READ MORE: South Florida family knows firsthand effects of nationwide baby formula shortage

Secret Service spokesman Ed Donovan would not confirm that prostitution was involved. Instead Donovan said the action was due to allegations of misconduct.

A White House spokesman said President Obama has “full confidence” in the Secret Service.

“We’re working people; we risk our lives to protect the President and others,” Adler said. “We don’t get involved in scandals. Sometimes we make mistakes. We own up to it. We deal with it. This is not a scandal.”

SOUTHCOM said the service members conduct allegedly occurred in the same hotel where the recalled U.S. Secret Service agents were staying.

However, it’s not been clarified if the group was involved with the prostitutes, according to Col. Scott Malcom of SOUTHCOM.

The SOUTHCOM personnel remained in Colombia but have been confined to quarters and ordered not to have contact with other individuals. They will be brought back with the rest of the support contingency after the mission is completed.

General Douglas Fraser, Commander of USSOUTHCOM, said that he is “disappointed by the entire incident and that this behavior is not in keeping with the professional standards expected of members of the United States military.”

MORE NEWS: South Florida woman's bout with COVID included hair loss: 'This isn’t normal shedding'

He went on to say that after a thorough investigation, punishment, if appropriate, will take place in accordance with established policies and the Uniform Code of Military Justice.