Graphics matter but don't matter as much as gameplay or ease-of-use. Bad graphics can mean different things to different people.
What I consider bad graphics are blending graphics. Like your avatar sprite blending in with the background or an enemy sprite blending in with the background (although sometimes, this is intentional, like for a dark room. Even then, your avatar is usually lit anyway). Too many bright colors put together or too many dark colors put together. The graphics portion needs to take into account the concepts of art into it (like you couldn't tell that) so one needs to take into account such things like contrast.
Still, underdeveloped graphics can not be a bad thing. Simple sprites contrasting with the background with good controls and good level design can make it a masterpiece. The same game done with blending graphics can destroy it.
Graphics matter but it isn't the only thing that makes a game good. Good graphics (i.e. I know where my avatar is, I know where my enemies are, I know where there's a pitfall, etc.) coupled with bad controls or level design can be just as disastrous. Even bad programming, such as bad collision detection, can break a game even it has awesome graphics.