Being a senior CS Major, I feel somewhat qualified to answer this.
There are a LOT of facets to Computer Science. Math and physics definitely help in some of them...but they're rarely required.
For instance, physics helps with the electrical engineering side of things, but is not necessary. I'm not much of a physics guy at all, but I loved my EE course.
As for math, it can usually help with programming, but not too much. My school only requires single-variable calculus (because it's required to take any math course here), linear algebra (and that's only to take the graphics course) and discrete math (which is basically the math of computers, in a way). I'm a math major too, and there's not much outside of those courses that I use regularly in computer science.
Just remember, there's a reason why CS, math & physics are all different majors. And if you have an idea on what part of CS you're interested in (it's ok if you don't, I didn't, and I still don't), then you can probably check that out to see what skills will be useful.
