Well, yes, at first blush it is easy to agree with. But if you stop and think about it, and put it into context, you have to realize that Andrew Jackson was pandering to voters, saying what he knew that most of them wanted to hear.
They themselves also benefited from special government favors, but of different kinds.
And those whom Jackons painted as privileged earned their wealth as did the so-called common person. In that the so-called wealthy were often merchants, lawyers, and other urbanites, they did not own any land at a time when land ownership was the basis for taxation and taxation was the basis for voting. That was why the so-called rich insisted on poll taxes: so that they could pay them and vote like any farmer with a single acre.
Jackson also represented and capitalized on an new trend (then) of anti-intellectualism.
The Cherokee Trail of Tears was consistent with Jackson's poilitical philosophy.
Through high school and college and beyond, I, too was a Jacksonian who responded to his libertarian statements against the central bank, and his support of hard money in the Treasury. But there is more his ascendency and presidency than that. No one then was consistently advocating for reality, reason, and rights. Federalists and Republicans alike were a mixed bag. Much of what both said is acceptable today because it was an expression of the Enlightenment, of rational-empiricism applied to politics. The Constitution is a Newtonian machine of equal and opposite forces.
Andrew Jackson represented a rejection of that.
As a person, Jackson did have admirable qualities. When Congress left him and his troops stranded after the Creek wars, he used his own purse to bring them all home, just for example. He was not evil incarnate. But neither was he an exemplar of laissez-faire. The money from the US Treasury that had been deposited with the Bank of the United States went to local banks ("pet banks") whose owners were Democrats. Jackson said, "To the victor go the spoils." If there is a job that cannot be done by a Democrat, eliminate the job.