Artificial Intelligence / Limits of Possibility?

Writing for Technology Review, David Gelernter suggests that "it is hugely unlikely, though not impossible, that a conscious mind will ever be built out of software" and directs us to a debate centered around models that are labeled "simulated conscious mind" and "simulated unconscious intelligence." What can we build? Ever?

"Artificial Intelligence Is Lost in the Woods" (synopsis from Ray Schroeder, Techno-News Blog)

A conscious mind will never be built out of software, argues a Yale University professor. Artificial intelligence has been obsessed with several questions from the start: Can we build a mind out of software? If not, why not? If so, what kind of mind are we talking about? A conscious mind? Or an unconscious intelligence that seems to think but experiences nothing and has no inner mental life? These questions are central to our view of computers and how far they can go, of computation and its ultimate meaning--and of the mind and how it works.

http://people.uis.edu/rschr1/2007/07/artificial-intelligence-is-lost-in.html

Whole original Technology Review article by David Gelernter is here: http://www.technologyreview.com/Infotech/18867/

Yes, it's all fascinating, and it may offer a more specific vision of artificial intelligence, one that we may contrast with the many fictions that've shaped the definition of AI for the bulk of humanity.

It may also suggest that there will be things we can't quite achieve, goals which we may approach, catch sight of, but never actually realize.

Utopians in particular tend to view human potential as limitless, and any goal, particularly their own goals, as entirely attainable, whether or not (usually not) there are specific, funtional plans in place to reach them.

Is it possible that there are limits? That our progress in a given direction is asymptotic to our goal? It's unlikely this question can be answered, as it's a matter of faith: either we believe a goal is attainable, or we do not, and, as we have not reached it yet, either hypothesis is unproven. If it is a fact that particular goal is unreachable but that we may draw ever closer to it, it is probably useful to believe that we can attain the goal.

Goal-oriented thinking can be dangerous if the end (accepted as attainable on faith) is used to justify certain means.

Please do think about AI and the possibilities for a better future that might include it. But here's my question for you: what is an example of an unreachable goal? Explain.