Why there's no "both" button ?
It is true I prefer English voicebanks as they do have the phonems.... However, in the practice I use more often Japanese voicebanks (and even, I barely Engrish compared with what I plan, and sometimes I try other languages including French)... In fact, among the voicebanks I've DLed for English, you have VBs created specially for English with the phonems (like ÑAL and then I think I've DLed Halt because I was planning to use her on a certain song her vocals would fit), Japanese voicebanks (so far the voicebanks I've attempted to make sing in English (like Teto, Ruko, Axel) or even my own banks who have a "mashed" voicebank
After that's the same for Vocaloids, all depends on the tuner's skills and the VSQ/VSQX/UST... However I am really iffy about the two sides when pushed at the extreme....
"English 'loids sound like garbage" → No, except a few that encounter quality issues despite having a beautiful voice (SONiKA baby), they sound really HQ and realistic-like (Operaloids, Avanna, Mona, and so on) + they are built for this language, so obviously they take the advantage and if you want the best results, it's the way to go
"Japanese/Anyothernon-english voicebank sound crappy ew" → I can understand it's harder to get accurate results as you know, they DON'T have the phonems... And also, the tuner/user must know what they're doing because of course using straightforwardly their Rs for English Rs (especially if they sound like Ls) OF COURSE it'll sound really off. You have in this case to play with flags, phonems tricks, parameters...
So yeah, English VBs are the best for it, but if you want a certain vocal to sing a song for you and you have to force it because they're not built for it, then go ahead, with effort and work you can obtain something correct (and more you're skilled and more it'll be imrpessive)