Reykjavik may not be the rock climbing capital of Europe, but if you know where to look you'll find that it's still got a few crags up its sleeve. Enter Valshamar, a 25 m cliff face located more-or-less halfway along the southern end of Hvalfjörður (that's a fjord, for anyone who didn't know). The climbing here is by no means world class, but with well over 20 (generously) bolted runs this rock face should satisfy just about any experience level between 5.3 and 5.10d. Plus you've got the backdrop of beautiful West Iceland to accompany the effort, which is never a bad thing. Just be advised that the cliff faces west, in case you're a morning climber, and the wind can be something of an nuisance at unpredictable times.
Since Valshamar is only about a 30 minute drive from Reykjavik, there are more than a couple capital-based tour agencies offering guided climbs here; we'll leave that completely up to you. If you've got access to ropes, shoes, chalk, and a few friends who know the area, however, this can easily turn into a full day's worth of scenic, non-guided adventuring close to the big city. Enjoy.