The money you put into SSA was untaxed, and you received part of it from employer contributions. And it's gained interest.
So yes, it should be taxable income.
If it's your only income, the tax rate on it will likely be 0% anyway.
If you have other sources of income while you're retired and collecting SSA, I'm not going to cry that you're being taxed on marginal income.
Also, the funds from SSA taxation go right back into SSA. It's keeping SSA alive so you can continue to receive your benefit.
The only issue I have is that the margin was set at $25,000/year of total income back in the early '80s, and hasn't moved. It should be in the $50-75k range now. $50k of untaxed income is pretty good when your house is paid off and your kids are paying their own way. But, again, the $3.5k or so in taxes that your $50k in SSA payments will incur goes right back into SSA, meaning it will last longer.
And there's definitely a point at which, if you live long enough, you will be drawing far more than you were expected to, and every payment at that point is free money.