President Donald Trump’s second nominee for the Fed board of governors in as many weeks, Herman Cain, will not be confirmed if Democrats stand united and vote against him after a fourth Republican has come forward against the former presidential candidate and businessman.

There are 47 Democrats in the Senate with one common goal: Obstruct the president. And they need just four Republicans to join them in order to shoot down a nominee with a simple 51-vote majority.

Trump hasn’t officially nominated Cain or Stephen Moore yet but is expected to do so. North Dakota Sen. Kevin Cramer said he will not back Cain when it happens, joining Mitt Romney of Utah, Lisa Murkowski of Alaska and Cory Gardner of Colorado.

Per Bloomberg:

“If I had to vote today, I couldn’t vote for Herman Cain,” said Cramer, a Trump ally. “The allegations that drove him from the presidential race are just so obviously serious. I’m not talking about his position on interest rates or anything like that, but the sexual harassment stuff. Until it’s better explained I couldn’t vote for him.”

Cain’s bid for the 2012 Republican presidential nomination ended after he was accused of sexual harassment when he led the National Restaurant Association in the 1990s and of conducting an extramarital affair.

At the White House Friday, Trump didn’t respond to a question about Cain at an Oval Office meeting about veterans.

Cramer joins GOP senators Mitt Romney of Utah, Lisa Murkowski of Alaska and Cory Gardner of Colorado in expressing opposition to a Cain nomination, which would leave him with just 49 potential Republican votes. Senate GOP leaders, including Republican Whip John Thune, have already said they don’t anticipate Cain could get a single Democratic to support his confirmation.

Senate Minority Leader Chuck Schumer said he didn’t know of any Democrats who would vote to confirm Cain.

Trump has announced Cain and former campaign adviser Stephen Moore as his choices to fill two open Fed seats but hasn’t formally sent a nomination to the Senate.

Cain said in a video posted on Facebook on Friday that he faced a “cumbersome” vetting process for the Fed seat, suggesting he may be considering withdrawing. Trump this week said it would be up to Cain whether he wanted to follow through with a nomination.