You can defeat them “in any mode”. Why waste a key trying to beat them in the side quest? Duel, arena, EQ/story…you’ve got options and you unfortunately chose a bad one.
Was he the boss of that path when you first ran it. Because the final fights change with every entry.