Influence stays
Stripping the label doesn't remove the influence. It hides it.
Stripping party preference does not eliminate influence, it hides it. Name recognition, incumbency, and campaign spending fill the vacuum, rewarding the already well known and well-funded over candidates running on ideas.
Even for voters who read the pamphlet closely, party preference is an essential cue they rely on to navigate through a crowded ballot.
Government should not decide for you which information is worth having.