The dreaded tax man cometh — even in retirement.

While planning and saving for retirement, it’s important to know how much money is coming in, and it may be even more important to know how much is going out.

Social Security is a major source of income for many seniors in retirement. A recent study showed that 21% of married seniors and 44% of their unmarried counterparts rely on Social Security for a whopping 90% of their income.

Educating yourself on Social Security benefits can lead to more money in your coffers, or it may just mean you aren’t surprised when tax season rolls around.

Here’s what you need to know about taxes and Social Security, per USA Today:

Will you pay taxes on your Social Security income?

Whether your Social Security benefits will be taxed will depend on your total income and where you live during your golden years.

First, let’s talk federal. If your benefits are pretty much your only source of retirement income, then they likely won’t be penalized. But if you have other income sources, such as an IRA, 401(k), pension, or earnings from a business or part-time job, then there’s a good chance you’ll be liable for federal taxes on your benefits.

To see if that’s the case, you’ll need to calculate what’s known as your provisional income. To do so, take your non-Social Security income, including tax-free income you collect (such as interest from municipal bonds), and add in 50% of your Social Security benefits. If your total lands between $25,000 and $34,000 and you’re a single tax filer, or between $32,000 and $44,000 and you’re a joint tax filer, then you could be taxed on up to 50% of your Social Security income. And if your provisional income totals more than $34,000 as a single tax filer, or more than $44,000 as a couple filing jointly, then you could be taxed on up to 85% of your benefits.

Where you live matters, too

Though the above income guidelines dictate whether you’ll be taxed on Social Security at the federal level, you may face state taxes on your benefits, too. There are 13 states that tax Social Security to different degrees:

  • Colorado
  • Connecticut
  • Kansas
  • Minnesota
  • Missouri
  • Montana
  • Nebraska
  • New Mexico
  • North Dakota
  • Rhode Island
  • Utah
  • Vermont
  • West Virginia

Of these 13 states, Minnesota, North Dakota, Vermont, and West Virginia are the only ones that don’t offer some form of exemption for low- to middle-income seniors. If Social Security is your main or only source of retirement income, and you live in Colorado, Connecticut, Kansas, Missouri, Montana, Nebraska. New Mexico, Rhode Island, or Utah, then there’s a good chance you’ll get out of paying state duties on your benefits.

Don’t get caught off guard by taxes in retirement

Of course, Social Security is only one income source you may get taxed on during retirement. If you have savings in a traditional IRA or 401(k), withdrawals from that plan will be subject to duties as well. The same generally holds true if you’re collecting pension payments, though not always. And, of course, if you’re working, you’ll pay on your wages, just as you did throughout your career.

The takeaway? Know what income you’ll be taxed on during your golden years, and plan accordingly. The last thing you want to do is estimate your annual retirement income, only to forget about withdrawals and wind up with far less money — and far more worries — than you anticipated.