Hellow! I hope I can help you, my name is Big-P and I'm happy to meet you. ^^
But we need to talk about some things first: you posted in the wrong area. If I'm not wrong u should posted in utahelp. I know that you're a newcomer and wanted to had your question answered in the same post (Prob bc you just want help and don't want to continue posting here), but some people will not see it with good eyes. So move to Utahelp and introduce yourself in another post, ok? ^^
Well, I really wanted to tell you a precisely answer for your question, but idk that much about mod, sry! :/ What I think that it does is change the specifications of the formant filter effect in a way to achieve a pitch near the original ".wav". XD Yeah, I know it's hard to understand, but I'll make it easier for you. If you want to understand better I recomend you to download Kye's CV voicebank that you can find here:
(It's just because Kye's vb is easier to see the effects of modulation, mod will act differently for each vb (Voicebank))
So, let's begin!
Formant Filter is a filter used in mixing to change a sound wave to get different results, it's used in a way that you can change a sound wave to sound more like any vowel that you want (a, e, i, o, u). This video will help you to understand better what formant filter is:
But in utau, at least is what I believe, it's used in the note pitch and note frequency map change.
What is pitch?
Do you know the piano roll (The piano titles in the utau)? Did you noticed that every title has a letter? Like C4, C#4, D5? These are the pitches! If you put a note in C2 your utau voice will sound like a bass:
https://voca.ro/7FTgSa2CJrb (I used the "A" to this recording)
But if you put a note in C4 you will see that the sound will be higher:
https://voca.ro/esYq0BFpsd5 (I used the "A" to this recording)
You can hear the sound higher bc it's in a higher pitch (C4) and in the bass case the pitch is lower (C2). That's basically what pitch is.
Coming back to Formant Filter
Some voicebanks aren't multipitch (A voicebank recorded in more than just one pitch. If done correctly you can have a better quality utau with less robotic sound), so utau needs Formant Filter to edit the notes and make it sound higher or lower, without it the program by itself couldn't get a satisfactory result when you need the utau to sing in a different pitch than the one it was recorded (Specially higher pitches). Example:
(Kye's CV with Formant Filter active)
https://voca.ro/o79UVVHBljt
(Kye's CV without Formant Filter active)
https://voca.ro/oSZzskjEGVa (Sounds like chipmunk XD)
(If you want to disable the formant filter to test you have to right click the note that you want to deactivate, go to property)
View attachment 9502
(Then check the "No Formant Filter" box. Becareful, some UTAU can sound really strange when this is checked. Kye for example sounds good with bass notes, but his higher notes are pretty bad when this is checked)
View attachment 9503
Frequency map is the map of a wav sound frequency. I don't know how exactly the frequency works, but I believe that it tells to the utau program what pitch the note was recorded and is used to the pitchbending/tuning too (The red squares/lines in your notes that you can change to get different results in your notes). You can generate one going into tools, voicebank settings and then in initialize freq map. If you want to check not just the frequency map, but the original pitch that the note was recorded you can go into the Edit freq map option:
View attachment 9504
You see? This is the frequency map of the note "a 3.wav" from Kye's CV voicebank. The red line marks that this note was recorded a little above C4 pitch, the blue squares are actually a representation of your voice's frequency and the green wave is the .wav file sound. If you open "a 3.wav" from Kye's CV voicebank in a program (Like audacity, for example) you will see exactly the same as the green wave.
After telling you all this stuff now I'll tell you what mod does. Basically it will modify the formant filter options and will make the note pitch sound near the original .wav file. That's why mod changes from vb to vb, some voicebanks (vb) are recorded in a regular pitch (frequency) that makes higher mod values sound really bad ^^', but some others can sound more realistic like Kye:
https://voca.ro/cslkBVzY4Fy (Kye with mod)
https://voca.ro/da6Tg0cflpr (Kye without mod)
And some others can even sound like it doesn't change nothing at all (Idk, but maybe it's because you're using a VCV utau. But I really don't know why yours doesn't changed).
And if you want to appear in your screen you have to press the "P" and "~" boxes in the left lower side of the program, if you press them they will appear.
But it's just my opinion since no one give me a concrete answer probably I'm wrong in something. I really figured that out testing voicebanks with this option in different manners so it's just based in my tests! ^^ Sorry for not giving you a good answer or a right one!
I hope I could help you, thanks if you read everything!
See ya! ^^ <3