Can we measure the difficulty levels of different AI tasks? For example, is playing Go game more difficult than doing chores? This problem arose on the public media earlier this year, due to the media coverage of historic moment of AlphaGo defending Lee Sedol. There is a related question: Can we gauge the progress of artificial intelligence compared with human intelligence?

Based on the AI development history, our answer is likely to be “NOT SO MUCH”. The history shows that what we thought more difficult AI tasks are less difficult than the ones we thought easier, and vice versa.

AI has a surprising history, with several ups and downs, in the past half of century. This is partly correlated with our assessment of the difficult levels of different AI tasks. Initially, we thought board games (such as chess and go) was harder, while perception processing (such as listening, seeing) was easier. This idea is formed by extrapolating from our own experience of biological intelligence, where computation and logic is harder than perception, which seems effortless. Now the history has proven that we were completely wrong. So our assessment of the difficult level of different AI tasks have changed dramatically over the past few decades.

Generally speaking, we take two approaches in measuring the AI task difficulties. The first approach is purely empirical and retrospective: We generally assume that all the AI tasks we have solved are easier, while the ones we haven’t solved yet are more difficult. So now, we think doing chores is more difficult than playing board games, including Go.

The second approach argues that some tasks are inherently more difficult because they needs to achieve the same goals based on less information. For example, we can say that unsupervised learning is more difficult than supervised learning, because it does not require additional task-specific annotations. The problem with this approach is that there are many tasks that are parallel (in the sense that they process different kinds of inforamtion), and we cannot measure their relative degree of difficulty based on that. For example, you cannot say that understanding language (NLP) is more difficult than understanding the visual world (computer vision), and vice versa.