I look at it this way: in a race, I need to generate a lot of power with each stroke, but I also need to be able to take X strokes. So, unless I can do my 100 LCM fly in 10-12 strokes, I'm going to need some kind of "endurance". If I can't do the 40 strokes it really takes with any level of power, then I need to work on that conditioning. Sounds like you're in the same boat.
I think the off- or early season is a really good time to focus on conditioning. It's not something you can gain in a week of cramming. I also like the word "conditioning" better than "endurance". Endurance sounds like something a triathlete would need. Gross!