You need to use a grindstone to remove the enchantments and try again, ideally by enchanting it directly on the table (enchant junk like wooden shovels to reset the enchantments until you see e.g. Protection IV in the tooltip, with other possible enchantments all being ones you want), not by using books, except to add/max out as few enchantments as possible since you get a total of 6 operations before any item becomes too expensive, which also carries over when you combine books, which you certainly did, e.g. 8x Protection I = 1x Protection IV but at the cost of 3 anvil operations, 4 to add it to an item (the game only carries over the highest penalty when you combine items with existing penalties as otherwise this would be impossible as you need to perform 10 operations to combine the books).
Alternatively, start with max-level books from e.g. trading, then you can add up to 6 enchantments by simply adding them one by one (it is still better and cheaper to combine the books into multiple enchantments, e.g. Protection IV + Unbreaking III and Aqua Affinity + Respiration III, but don't overdo it since the penalty still accumulates and you must pay to add every enchantment at once. See this post for an extreme example (you can actually make your helmet, or even a more highly enchanted item, starting from all level 1 books if you combine them in the right order).
So I will just remove the enchantments, and then enchant protection 4 (maybe with additional unbreaking 3, aqua affinity etc. that would be nice), and then get the other enchantments from trading with librarian villagers or by fishing.
To get it this way, you had to do something to this helmet on the anvil 6 times. The prior rework penalty goes 0 - 1 - 3 - 7 - 15 - 31 for the first 6 anvil actions, the 7th is too expensive. Grindstone will remove prior rework penalty if you want to start anew.
You really want to get the very best possible initial enchant on the table. I would keep resetting till you get 'Protecton IV' or 'Respiration III', though with Resp 3, you might end up getting the wrong protection enchant. Anyway, past the initial table enchant, it is all in how you apply the books.
You don't go helmet + book1, then helmet + book2, then helmet + book3, then rename. This brings you to rework level 4
Instead you go: helmet + book1, then book2+book3. Then you combine helmet with combo book 2+3 and rename at same time. You are only at 2 reworks,
Say you want to apply 7 enchantment books to same sword? You go sword + book1, books 2+3, books 4+5. books 6+7, then sword + book 2/3 and book 4/5 + book 6/7. Then sword + book 4/5/6/7 and rename at same time. Only 3 reworks used up. Also, you generally want to book 1 to be the most expensive enchant, then 2, then 3 etc.