First, you're going to need a compiler. I prefer DevKitAdvance, and while I was programming, I built up a nice compiler kit using DevKitAdvance which is a little more lenient when it comes to optimizations than the default distro. You can download mine
here.
Grab the 'devkitadv' folder out of that zip file and drop it directly into your C:\ drive; the base directory of the compiler should be 'C:\devkitadv\bin'.
Next, you're going to need an emulator. I reccomend VBA, which you can get off of zophar.net
here.
Finally, you're going to need some test code to make sure that your compiler works! This is a simple program that I whipped up over the course of a day to see if programming for the GBA was feasible for me:
click here. Double-click 'make.bat' to compile it. Note that this was my first ever GBA program, so it's more than a little unoptimized; still, it will give you a decent idea of how to draw sprites to the GBA.
You can look to
http://gbadev.org/ for all sorts of great tutorials and examples. Good luck! =)