Hey Kim that is not a concern there wont be any vibration issues because no significant amount of material has been removed. You would need to have a ounce + to make any type of impact on vibration or bearing issue.
The bigger concern is how true the pulley spins if its been bent etc and yes the groove has to be clean and free of burrs bumps. Yours by the pic looks fine but i would still take a stright egde or place it on a flat surface to see how it sits .
If you can lightly hammer the edge flat and then take a fine file and touch up the surface and in the groove you can also use some emery cloth to rework the finish.
This is hard to explain but hope it helps.