Follow CBSMIAMI.COM: Facebook | Twitter

WASHINGTON (CBSMiami/CNN) – Nothing ever seems to come easy when it comes to the Trump White house.

Dr. Ronny Jackson, the embattled pick to lead the Veterans Affairs agency, is not planning to withdraw his nomination as of now, a senior administration official said Tuesday.

The White House feels Jackson is being “railroaded” and will push back forcefully, the official said, adding that President Donald Trump is not wobbling over his selection.

A White House official said the White House will soon be putting out statements and letters from previous presidents and administration officials, pushing back on a day of stories against Jackson.

Trump met with Jackson on Tuesday in the Oval Office after saying it was “totally (Jackson’s) decision” whether to continue the confirmation process.

Earlier Tuesday, Trump conceded the one-time White House doctor faces major problems.

“He is a high-quality person. It’s totally his decision. So he’ll be making a decision. I don’t want to put a man through a process like this. It’s too ugly and too disgusting,” Trump said. “So, we’ll see what happens. He’ll make a decision.”

Trump spoke by phone with Jackson on Tuesday, two White House officials said, and told him it was up to him whether to withdraw.

“I told Admiral Jackson just a little while ago, what do you need this for? This is a vicious group of people,” the President said. “What do you need it for?”

Trump said if he was Jackson making the decision, he would step aside.

“If I were him, I wouldn’t do it,” Trump said.

Trump was speaking as concerns mount over Jackson’s nomination. Senate Veterans Affairs Committee members are assessing allegations from whistleblowers that have told the panel about Jackson’s questionable behavior including excessive drinking and a “toxic” work environment under his leadership, according to two former White House medical staff members who have spoken with the committee.

Both sources who spoke with CNN told the committee about behavior they observed while working in the White House medical unit.

Trump said Tuesday he hadn’t heard particular allegations against Jackson but defended him and said he was being unfairly treated by the press and Democrats, though the concerns about Jackson also come from Republicans.

Jackson, Trump insisted, “is one of the finest people I have ever met” but acknowledged he lacks prior background in running a government agency.

“There’s a lack of experience,” he said.

Behind the scenes

Though the White House publicly defended Jackson in a statement Tuesday morning, several officials have privately conceded that the President’s nominee to lead the Veterans Affairs department is in trouble. The White House drafted the statement supporting Jackson days ago — in preparation for any criticism during his hearing.

But after reports surfaced that his nomination could be upended by the allegations, the White House sent out the statement early.

Officials are aware in detail of the allegations lobbed against Jackson. They are continuing to discuss what to do going forward if Jackson doesn’t withdraw, and his nomination is still very much up in the air.

But other officials fear Jackson’s nomination is doomed. One said that having the White House defend him instead of simply pulling his nomination is worse, predicting more will come out and the beloved West Wing physician will go down in flames.

Several pointed to how difficult CIA Director Mike Pompeo’s confirmation process to head the State Department has been, even though he has the credentials for the job as the nation’s top diplomat.

(© Copyright 2018 CBS Broadcasting Inc. Cable News Network, Inc., a Time Warner Company, contributed to this report)