Kritika kritike Kanemana, 1. deo

Slaviša je ovde ukratko predstavio svoju (i Željkinu) kritiku Kanemanove knjige i celine njegovog programa. Pre svega da naglasim da je sjajno što je taj tekst objavljen, da su autori tom studijom ostvarili po svakom kriterijumu odličan naučni rezultat, da je bez ikakve sumnje hrabro kritikovati takvu veliku metu – i da im na radu iskreno čestitam. Rad je vrhunski i svako zainteresovan za teme sa ovog bloga bi dobro učinio da ga pročita. Ovde kačim svoju kritiku njihove kritike, nastalu od beležaka na marginama tokom čitanja njihovog rada. Nije poenta odbrana Kanemana – kome to svakako nije neophodno! – nego otvorena i oštra intelektualna debata o nekim od tema koje jesu i teorijski i praktično zanimljive.  

Tekst je na engleskom, iz tri razloga: (i) zato što iscrpno citira delove studije Buturović & Tasić, pa bi bilo rogobatno prebacivati svako malo sa srpskog na engleski i obratno; (ii) da bismo izbegli terminološke zađevice oko “najadekvatnijeg” prevoda nekih od korišćenih termina; (iii) iz praktičnog razloga što su i moje beleške na marginama bile na engleskom. Napisan je u relativno opuštenom stilu, bez navođenja referenci, fusnota i ostalih akademskih parafernalija. Zbog ukupne dužine podelio sam ga u dva dela.

Reform vs. counter-revolution: On Buturovic’s and Tasic’s criticism of Kahneman

Buturovic and Tasic (2015; henceforth BT) criticize Daniel Kahneman and the entire project of behavioural economy for taking too narrow concept of rationality, but also for many other perceived errors and insufficiencies. This leads into calling his work a “failed revolution”. Except for a bit of revolutionary zeal (more on that below), it is not quite clear what exactly would BT prefer from any project in behavioural economy. Kahneman’s programme is no good? Fine and well, but it’s not as if there is a glut of alternatives around. And certainly it's not as if BT criticism is, ahem, entirely rational. 

On what counts as advance: “But it does not advance our understanding of human psychology except inasmuch as one otherwise would  have been inclined to believe that people are everywhere and always “rational” in the peculiar sense in which this term is used by economists” (p. 129)

Well, how about the following hypothetical comment on Einstein's relativity: But it does not advance our understanding of physics except inasmuch as one otherwise would have been inclined to believe that physical objects are everywhere and always characterized by absolute space and time in the peculiar sense in which these terms are used by Newton and his followers. One can use this deflationary manoeuvre with equal success against each and every innovator and reformist. But the argument is rather lightweight when we reign in our revolutionary expectations and consider what is realistic to achieve in science. 

On skills and studying rationality: “One recurring issue that has cropped up since Kahneman and Tversky’s early studies of cognition has been their assumption that specific knowledge or skills are essential to rationality. Thus, the possession of statistical or factual knowledge or the use of abstract reasoning has often been uncritically taken as a standard against which we are to judge the answers that participants in the experiments give. This has attracted the critical notice of specialists but not that of the wider world.” (p. 129, emphasis MMĆ)

It is easy to show that “their assumption” can, in fact, be attributed to that slightly earlier fellow, Socrates of Athens, who – most famously in Plato’s Meno and Protagoras – argued that rational definitions of virtue must be “tethered” by knowledge, and therefore it is reasonable to conclude that the virtue can be taught to others (via specific knowledge). His famous dictum about unexamined life not worth living (ὁ ἀνεξέταστος βίος οὐ βιωτὸς ἀνθρώπῳ; Apology 38a) is usually interpreted in the same manner. This has been repeated countless times in the course of Western intellectual history. Whatever intrinsic merits or demerits of the assumption, it is ahistoric and naive (to say the least) to attribute it to Kahneman and Tversky as a sort of newfangled behavioural fad. BTW, how do BT know that a particular standard related to a particular experiment has been accepted uncritically – as contrasted with the situation in which such standard were accepted after much reflection, deliberation, hand-wringing, and going to-and-fro? What is factual evidence for uncritical acceptance of such standards? If there is some such evidence, BT fail to provide it.

Besides, what exactly is the alternative to this millennia-old assumption? That rationality could be (non-circularly) defined without reference to any particular knowledge or skills? It would be interesting to see results of such an enterprise. In any case, the burden of proof lies on BT, and not on Kahneman or anyone else who accepts that particular forms and amounts of knowledge or skills are necessary for any operational concept of rationality.

A long debate – more than a century – in evolutionary biology on the subject of extinction of species ended in conclusion that it is illegitimate to cite inferiority of a species or a higher taxon as the cause for extinction without listing which characters in what context were inferior to analogous characters in surviving taxons. Before we reached that stage, gradualist (il)logic filled textbooks with statements like “dinosaurs went extinct and mammals survived because dinosaurs were inferior”; which, of course, was not empirically supported whatsoever and in the end got reduced to “they had to be inferior in some unspecified way, since they went extinct”. Evolutionary biology and paleontology flourished greatly after getting rid of such nonsense. And there are many similar examples in other areas.

Finally, what exactly should this attention and critical notice “of the wider world”, so lacking in the reception of Kahneman’s work, consist of? Should Kahneman complain that he hasn’t been a guest of The Kardashians yet? Is his work less valuable due to the fact that he has 6,966 twitter followers, in comparison to N zillions of Justin Bieber? In my view the very concept of the “critical notice... of the wider world” is oxymoronic, since critical notice is not something established by a show of hands or a plebiscite, and almost by definition “the wider world” is too heterogeneous for any non-trivial critical notice. And the oxymoron is only strengthened – in a surreal kind of way – if we accept BT’s implication that empirical knowledge or reasoning skills are not truly necessary for demonstration of rationality; BT seemingly expect the same “wider world” which they excuse for lacking knowledge and reasoning skills to offer (presuming rational) criticism of Kahneman. Tall order, if anything.

On the language analogy: „We use language for a purpose, and grammar is an integral part of language.“ (p. 129) So it is natural to expect an intuitive and automatic access to grammar. What is wrong with this statement is clearly seen from an analogous statement, for example: We use mathematics for a purpose, and complex analysis is an integral part of mathematics. Therefore, intuitive and automatic access to complex analysis should not surprise us. Of course, that is absurd. Perhaps a two or three greatest geniuses in mathematics, persons born once per century, could conceivably have intuitive access to complex analysis, but that certainly does not apply to millions (or indeed billions) of users of mathematics. And BT have neither demonstrated that, conversely, the lack of grammar skills characterizing billions of people demand explanation, not have they shown any evidence that their preference for linguistic over mathematical skills reflects anything else then their own personal preference.

This conclusion is strengthened by their continuation and generalization of the same theme: „Statistics, on the other hand, is a body of knowledge created, systematized, and formalized by statisticians.“ (p. 130) I find this simply untrue. Like any other mathematical discipline, statistics is a set of proposition whose validity is given quite independently of statisticians and its statements would be entirely valid if suddenly all statisticians in the world were to die tomorrow or even if there were no statisticians whatsoever. Unless one is an extreme postmodern social constructivist, the statement that, for instance, any probability distribution on the set of real numbers R has at least one median will be true anywhere and at any time, even in the universes with no life and therefore no statisticians. (Of course, standard definitions of terms such as probability distribution or median is assumed.)

„Stating that people are good or bad at statistical reasoning tells us something about their knowledge of and talent for this particular skill and not much else.“ (p. 130) So, how exactly does this differ from claiming that stating that people are good or bad at grammar tells us something about their knowledge of and talent for this particular language skill and not much else? I do not see much of a difference; on the contrary, to paraphrase the authors, statements like these tell us something about their personal ordering of preferences and not much else.

The parable of a physicist: „A passionate physicist without much interest in the outside world could observe that people are very good at manipulating objects but embarrassingly inept at solving problems in theoretical mechanics. Few of us would treat such a statement as evidence that people should gain more knowledge of theoretical mechanics or that they are irrational when they get questions about the subject wrong. We would, instead, conclude that this physicist is making an oddly self-absorbed comparison between knowledge of physics and the ability to operate in the physical world.“ (p. 130)

This is either false or misleading on so many different levels that it is difficult to decide where to begin. For starters, the parable misses the entire envisioned goal: if the goal is set to make people better at manipulating objects, one is entirely entitled to argue that increased knowledge of theoretical mechanics might well help. Which has, by the way, been accepted long ago in sports academies around the world, whose curricula contra BT contain important elements from theoretical mechanics. If the goal of decision theory is to help make better decisions, the analogy holds. Second, in the course of a single paragraph, BT make leap from descriptive to prescriptive stance and back, while pretending to offer a parable from a neutral point of view. Third, if something is “embarrassing”, then clearly there is an opportunity cost to remove the embarrassment by – what a surprise! – gaining more knowledge of the subject, be it theoretical mechanics or Sinhalese syntax or whatever.

The irony is particularly rich when one takes into account that it comes from the same authors who subsequently take Kahneman to task for allegedly providing misleading cues in Linda and other psychological experiments. The number of misleading cues in this passage sets the record for packing density. What else could be the role of such strings of words as „without much interest in the outside world“ or „embarrassingly inept“ or „few of us would treat“ (rhetorical device known since at least Cicero and abused by perhaps every single politician on the planet) or „oddly self-absorbed“ if not providing misleading cues for the reader? „Oddly self-absorbed“ in contrast to – what exactly? Perhaps „commonly self-absorbed“ manner permeating comparisons made by social scientists? (Merriam-Webster offers „common“ as the first-choice antonym to „odd“.) Why is it important that the physicist is „passionate“ and not some boring old bureaucrat? Obviously, framing matters, as Kahneman and Tversky knew and BT confirm (p. 134).

„Statistical skills, unlike physical ones, could be somewhat useful in an ordinary person’s everyday life.“ (p. 130) The logical implication here is that physical skills could not be somewhat useful in an ordinary person’s everyday life, and such general statement is easily disproved by citing even a single counterexample in which physical skills could – and indeed are – very useful in an ordinary person’s everyday life. Such example is, for instance, driving – we have not evolved to drive cars, we do have big problems in properly accounting for conservation of momentum and energy in driving, and there is every indication – since there is no actual research done on this issue – that knowledge of theoretical mechanics is useful in driving (it is used, for instance, in instructions for defensive driving, of course in a simplified form). The example is not mine, it was suggested long ago by Richard Feynman, who was called many things but never “self-absorbed”.

A charitable interpretation of this is that the authors tried to say that the probability measure of cases in which physical skills are important is much smaller than the analogous probability measure for statistics, but such framing would (a) be less flashy, and (b) require at least hinting at some metric at which their personal intuitions about relative usefulness could be justified. But if they wished to convey that meaning, they should have chosen their words more carefully, especially since the context is set by their relentless revolutionary criticism (more on this below) of framings and contexts and backgrounds provided by behavioural economists and decision theorists. For all they who take the sword, etc.

The argument from billions: “But how useful these skills might be is unclear, as billions of people somehow manage to make their way in the world, decade after decade, despite their complete ignorance of statistics.“ (p. 130) Oh, but how useful reading and writing skills might be is unclear as well, as billions of people somehow manage to make their way in the world, decade after decade, despite their illiteracy – and have done so for many millennia. Unclear, indeed.

“Base rates are not unequivocally given to us as stable and known, and the appropriateness of using or disregarding them in particular situations depends on a host of conditions.” (pp. 131-132) Now, this rhetorical stratagem can again be debunked by comparing it to an analogous statement, for instance: Rules of grammar are not unequivocally given to us as stable and known, and the appropriateness of using or disregarding them in particular situations depends on a host of conditions. Sounds right, doesn’t it? After all, there are billions upon billions of people who hardly regard rules of grammar as “stable and known”. And we are still no closer to understanding what exact difference there is between information encoded in base rates and information encoded in the rules of grammar.

On Tom W.: “Only those few subjects (if any) in the third group who had studied the distribution of graduate students by field would have had reason to put any confidence in their guesswork about the matter.“ (p. 133) Again, let us compare this with a linguistic example. Consider the frequency distribution of letters in a language. While few subjects – if any – in a given group may know that e is the most frequent letter in any representative English text, this does not mean that they are really agnostic or have uniform prior distribution of frequency over the set of letters. It is hardly conceivable that even the most linguistically challenged or even illiterate and low-IQ experimental subject will have doubts whether a is more frequent English letter than q, or will give x the benefit of doubt versus s. Contrary to the implication of BT, normative knowledge is not measured by insight into a given database, but by success rate of inferential processes.

„Subjects asked to provide their “best guesses” about something they have probably never thought about before—the proportion of U.S. graduate students in various fields“ (p. 133) – where subjects are not farmers or soldiers or schoolchildren or pensioners but graduate students themselves, who are moving in the environment of other grad students and are perfectly well aware that the fields and departments are unequal in size and scope, are likely to think much about grad schools, etc. etc. If such knowledge should not be regarded as normative for the given population, what should be, according to the authors? Lack of curiosity? Unreflecting indifference? Low IQ?

“Kahneman and Tversky seem to think it is more rational to use made-up statistics than good heuristics.“ (p. 133) And believing the researcher – in psychological experiments where researchers standardly lie and mislead their subjects (e.g. Asch conformity or even Milgram) – is somehow a good heuristics? What kind of normative knowledge prescribed that? Perhaps the Orwellian stipulation that „ignorance is strength“?