By trunks do you mean board shorts?
I would recommend to mix it up. You are doing large distances with a lot of extra resistance. Sometimes I will wear a drag suit for half of my workout and then take if off and just swim in my swim brief. The difference, for me, with and without a drag suit is huge.