At the north-westernmost point in Reykjavik, where the peninsula of Seltjarnarnes juts into icy ocean waters, the Grotta Lighthouse stands sentry. Nothing spectacular in the historic sense, although well-kept, the lighthouse is notable as a short hiking destination from more central Reykjavik and one of the most approximate opportunities to catch the ethereal Northern Lights.
A mere 5 minutes of walking if you take bus #11 to Lindargotu road, we've got the sense that the hour footing it along the shoreline from Reykjavik might be more your cup of tea (and if the winter chill gets to you, you can always catch the bus back). Once there, enjoy the lighthouse and a few unrelated buildings, as well as views of the incredible Faxafloi bay. And the Northern Lights, of course, which are best pursued between November and February and, you know, when the sun isn't in the sky.
There is one obstacle to the reaching the Grotta Lighthouse, though, and that is high tide, which submerges the sandy strip that connects the rest of Iceland to this tiny landmass. Use the tide table in the included links to avoid even the thought of swimming the freezing channel.