I wrote a couple of weeks ago that the outcome of the November election will matter a lot less to the stock market than most believe.

Historically, the stock market has performed similarly, no matter which party controls the White House or Congress. Most annual returns were in a range between 14.5% and 16%.

The only outcome that bucked the trend was a Republican president with Democratic control of at least one house of Congress. Historically, the S&P 500 has returned only about 6.99% per year under that scenario. But all of these scenarios were positive.

That doesn’t mean that a Trump or Biden win won’t tank the market. But history suggests that it isn’t likely.

And stocks have performed well during Trump’s presidency, returning about 11.1% per year thus far. But returns weren’t too shabby under Barack Obama or Bill Clinton either. Returns were 13.8% per year and 15.2% per year, respectively.

The market as a whole seems indifferent to who sits in the Oval Office. But presidential policy can have a massive impact on individual sectors.

Today, let’s take a look at how each candidate might affect the financial sector.

The data here is skewed, as the banking sector rolled over and died in the last year of George W. Bush’s administration. Over the course of Bush’s presidency, the Financial Select Sector SPDR Fund (NYSE: XLF) lost 63% of its value.

This exchange-traded fund (ETF) is a “basket” of banking stocks, including Berkshire Hathaway (NYSE: BRK.B), JPMorgan Chase & Co. (NYSE: JPM) and Bank of America Corp. (NYSE: BAC).

Obama took office just as the financial sector was stabilizing. XLF rallied by 178% during his two terms.

financial sector

However, in both cases, the performance of financial stocks had everything to do with the timing of the financial crisis, not policy.

So, let’s focus on presidential policy for the sake of this argument.

Trump Administration Has Been Pro-Wall Street

We’ll start with the sitting president. The Trump Administration has been “pro bank,” loosening many Obama-era regulations, including the Dodd-Frank Act and the Volker Rule. Trump’s corporate tax cuts also benefited the banking industry.

The question becomes whether a potential Biden Administration would nix Trump’s rollbacks. In other words: Would President Biden reinstitute the strictest parts of Dodd-Frank and the Volker rule?

Probably not. The Obama-era regulations were put into place following the 2008 banking meltdown. That’s now a distant memory and not a major priority for either party.

Political anger is more directed at Big Tech these days than at Wall Street.

So, while enforcement under a second Trump Administration would likely be more lax, I don’t think a lot of new banking regulations will be coming down the pipeline.

For Biden, It Comes Down to Taxes

Biden has several policy proposals that Wall Street won’t like.

To start, he plans to raise the corporate tax rate to 28% from Trump’s 21%. Biden also proposes raising the capital gains tax rate to as much as 40% for taxpayers making more than $1 million per year. That would discourage trading and zing banking profits.

Biden’s running mate, Senator Kamala Harris, has also proposed taxing stock and bond trades as part of her health care initiatives. While her proposals aren’t likely to affect you and me, they would kill high-frequency trading.

All of these are proposals would still have to get through Congress. But it’s clear that Biden’s tax regime would be less favorable for banks.

A Biden Administration could target the Community Reinvestment Act, which requires banks to lend in lower-income communities. This wouldn’t move the needle much, but it would be a mild negative for the sector. Loans to less creditworthy borrowers means banks would need higher reserves for losses.

What’s Really Affecting the Financial Sector

On balance, the financial sector would benefit more from a second Trump term.

But bigger factors are at play outside of who is in the White House, like our recovery from the COVID-19 recession.

And the Federal Reserve’s interest rate policy will have a much larger impact on bank profits than anything President Trump or President Biden would propose.

So, for a better indication of where the financial sector is heading, watch Fed Chairman Jerome Powell. The sooner he raises rates to more normal levels, the sooner banking profits start to look healthy again.

Money & Markets contributor Charles Sizemore specializes in income and retirement topics. Charles is a regular on The Bull & The Bear podcast. He is also a frequent guest on CNBC, Bloomberg and Fox Business.

Follow Charles on Twitter @CharlesSizemore.