Photo mode is my original algorithm that uses bit shifting or anti aliasing to improve the rendering of photos, you should not need to select it manually as T2Laser analyses the job and uses it when necessary.
The new algorithm produces crisper lines and edges for logos and text, and is slightly faster.
I only added a switch back to the old method in case my auto detect had issues and people wanted to force the old algorithm, but 99% of the time the new one is going to provide the best results.
Velocity mode is different, it's another way to generate grey scale by using speed as the variable rather than laser power. So for darker burns it goes slower and for light areas it speeds up. To work well you need to have a good "range" of speeds, so set the low speed to something like 200 and the high speed (rapid) to 2000. It also requires your machine to support high acceleration so basically if your machine is well tuned and has low friction it will work well.