Emma Rothschild has called the phrase an "ironic joke," and many others have argued that its importance is less than many have believed. Smith used the phrase only once in the 1000-page Wealth of Nations (here), after all, and the phrase appears in only two other places in his entire extant corpus of writings (once in The Theory of Moral Sentiments and once in an early essay on the history of astronomy; see here and here, respectively).
I am of the school that believes that the concept of the invisible hand, if not the phrase itself, is of central importance to understanding Smith's enduring contribution to social science.
Now economist (and, for full disclosure, friend of mine) Dan Klein has weighed into this conversation with an interesting piece in Econ Journal Watch. Klein is writing in response to Gavin Kennedy's provocative piece in the same journal. Kennedy (whom I have met but don't know well) is rather skeptical about the importance of the invisible hand metaphor, while Klein's position is closer to mine, though with his own twists. It is an interesting exchange, and well worth reading.