Commerce Secretary Wilbur Ross’ time in the White House may be short after failing to implement a citizenship question on the 2020 census form after it was struck down by the Supreme Court. President Donald Trump is considering removing his long-time friend, according to people familiar with the situation.

This is not the first time Ross has come under fire. The president also has been frustrated with the 81-year-old in the past, particularly after failed trade negotiations. The Commerce Secretary’s job may be safe, though, after Labor Secretary Alex Acosta resigned Friday over his role in the 2008 plea agreement with serial sex offender Jeffrey Epstein, according to CNBC.

Despite that, some White House officials believe Ross could depart as soon as this summer.

Trump has been making calls to outside allies about replacing Ross, citing frustrations over how he has run the Census Bureau, a division within the Commerce Department.

While Trump’s Cabinet has seen incredibly high turnover — more than each of his five predecessors during their first terms, and in just two and half years in office —  Ross’ departure would mark the first exit of an agency head that had close ties to the president well before his 2016 presidential run.

The pair first bonded in the 1990s when Ross represented some of Trump’s creditors during Trump’s Atlantic City casino hotel bankruptcies. They have known each other for more than 25 years, and both own residences close to each other in Manhattan and Palm Beach, according to CNBC. Ross then endorsed Trump in his 2016 presidential campaign despite being a registered Democrat.

“We need a more radical, new approach to government,” he said in his June 2016 endorsement.

When it comes to replacing Ross as Commerce Secretary, though, the president told some of his advisers that he wants a more aggressive leader, but he has not asked any allies for suggestions.

Trump stood by Ross through a string of controversies leading up to the Supreme Court’s 5-4 decision to bar the citizenship question from the 2020 census, and when asked by NBC News on Friday if he was disappointed with the failure to get the question added, Trump said that he was not.

“No, he didn’t let me down,” Trump said.