No one has mentioned CY_ samples? I know some people consider then optional but I have found them to appear in enough USTs that I consider them necessary. Once a UST had a "gyu" syllable but the UTAU I was working with didn't have "gyu" so oh well, I couldn't use them.
Here's the consonants I do every time when I record:
b, by, ch, d, f, g, gy, h, hy, j, k, ky, m, my, n, ny, p, py, r, ry, s, sh, t, ts, w, y, z
And incredibly useful extra consonants that I always record are:
l (English l), rr (English R), v
I would say the only other extra sample that might be useful that I don't do is "th" sounds, it does sound silly using Z and D for that and those english words do pop up in songs quite a bit. What I also do with my list because I tend to record VCV is just record each consonant with every vowel, though that really isn't necessary sometimes (you are never going to use tsi or tse, but things like tu, si, ye, fe, wi and so on can come in handy at times and I'm disappointed when voices don't have them actually).
But yeah, I'd stick to all necessary Japanese sounds and just a few extra things, don't go crazy with adding extra consonants because they'll never be used quite honestly and it just serves to clutter the bank.