You don't have to know squat about computer architecture to complete a CS major. It all depends on what you concentrate on in the higher level courses. Most of what a CS major will learn is how to properly structure code, data structures, and the like. I completed my CS major without knowing 1 thing about how the internals of a computer worked, and that included a course in 3-D programming. You may have to know the Open GL API to write a 3-D game, but you still don't have to know squat about how the video card works. That is the whole point behind an API. You CAN take classes that require that you learn more about the architecture, but it will not be a requirement in most CS schools. I did have to take 1 introductory EE class, but it was a basic circuits class and had nothing to do with computer architecture.
As far as what is the most difficult, it is purely dependant upon the person. Math classes were a breeze for me, but then again I was a double major. The stats class I had to take was the toughest for me. It will vary greatly from school to school and from person to person. If you have a tough time in math in high school, you will have a tough time with it in college. If you've never written a program in C or C++, you will have problems at first in college, but it may be a breeze for you. I had never written a single program before going to college but programming was by far the easiest thing I had to do in college.