Subject: [extropy-chat] Re: Overconfidence and meta-rationality

From: Robin Hanson <[email-masked]>
Date: Sat, 12 Mar 2005 12:53:50 -0500

Eliezer, you are just writing far too much for me to comment on all of 
it.  If you give me an indication of what your key points are, I will try 
to respond to those points.  For now, I will just make a few comments on 
specific claims.

At 06:40 PM 3/9/2005, Eliezer S. Yudkowsky wrote:
>The modesty argument uses Aumann's Agreement Theorem and AAT's extensions 
>as plugins, but the modesty argument itself is not formal from start to 
>finish.  I know of no *formal* extension of Aumann's Agreement Theorem 
>such that its premises are plausibly applicable to humans.

Then see:  <http://hanson.gmu.edu/disagree.pdf>For Bayesian Wannabes, Are 
Disagreements Not About Information? 
<http://www.kluweronline.com/issn/0040-5833/>Theory and Decision 
54(2):105-123, March 2003.

>you say:  "If people mostly disagree because they systematically violate 
>the rationality standards that they profess, and hold up for others, then 
>we will say that their disagreements are dishonest."  (I would disagree 
>with your terminology; they might be dishonest *or* they might be 
>self-deceived. ...

I was taking self-deception to be a kind of dishonesty.

>... if Aumann's Agreement Theorem is wrong (goes wrong reliably in the 
>long run, not just failing 1 time out of 100 when the consensus belief is 
>99% probability) then we can readily compare the premises of AAT against 
>the dynamics of the agents, their updating, their prior knowledge, etc., 
>and track down the mistaken assumption that caused AAT (or the extension 
>of AAT) to fail to match physical reality. ...

This actually seems to me rather hard, as it is hard to observe people's 
priors.

>... You attribute the great number of extensions of AAT to the following 
>underlying reason:  "His [Aumann's] results are robust because they are 
>based on the simple idea that when seeking to estimate the truth, you 
>should realize you might be wrong; others may well know things that you do 
>not."
>I disagree; this is *not* what Aumann's results are based on.
>Aumann's results are based on the underlying idea that if other entities 
>behave in a way understandable to you, then their observable behaviors are 
>relevant Bayesian evidence to you.  This includes the behavior of 
>assigning probabilities according to understandable Bayesian cognition.

The paper I cite above is not based on having a specific model of the 
other's behavior.

>So A and B are *not* compromising between their previous positions; their 
>consensus probability assignment is *not* a linear weighting of their 
>previous assignments.

Yes, of course, who ever said it was?

>... If this were AAT, rather than a human conversation, then as Fred and I 
>exchanged probability assignments our actual knowledge of the moon would 
>steadily increase; our models would concentrate into an ever-smaller set 
>of possible worlds.  So in this sense the dynamics of the modesty argument 
>are most unlike the dynamics of Aumann's Agreement Theorem, from which the 
>modesty argument seeks to derive its force.  AAT drives down entropy 
>(sorta); the modesty argument doesn't.  This is a BIG difference.

AAT is *not* about dynamics at all.  It might require a certain dynamics to 
reach the state where AAT applies, but this paper of mine applies at any 
point during any conversation:

<http://hanson.gmu.edu/unpredict.pdf>Disagreement Is Unpredictable. 
<http://www.sciencedirect.com/science/journal/01651765>Economics Letters 
77(3):365-369, November 2002.

>The AATs I know are constructive; they don't just prove that agents will 
>agree as they acquire common knowledge, they describe *exactly how* agents 
>arrive at agreement.

Again, see my Theory and Decision paper cited above.

>>... people uphold rationality standards that prefer logical consistency...
>
>Is the Way to have beliefs that are consistent among themselves?  This is 
>not the Way, though it is often mistaken for the Way by logicians and 
>philosophers. ...

Preferring consistency, all else equal, is not the same as requiring 
it.  Surely you also prefer it all else equal.

>... agree that when two humans disagree and have common knowledge of each 
>other's opinion ... *at least one* human must be doing something wrong.  ...
>One possible underlying fact of the matter might be that one person is 
>right and the other person is wrong and that is all there ever was to it.

This is *not* all there is too it.  There is also the crucial question of 
what exactly one of them did wrong.

>Trying to estimate your own rationality or meta-rationality involves 
>severe theoretical problems ... "Beliefs" ... are not ontological parts of 
>our universe, ...  if you know the purely abstract fact that the other 
>entity is a Bayesian reasoner (implements a causal process with a certain 
>Bayesian structure),... how do you integrate it?  If there's a 
>mathematical solution it ought to be constructive.  Second, attaching this 
>kind of *abstract* confidence to the output of a cognitive system runs 
>into formal problems.

I think you exaggerate the difficulties.  Again see the above papers.

>It seems to me that you have sometimes argued that I should foreshorten my 
>chain of reasoning, saying, "But why argue and defend yourself, and give 
>yourself a chance to deceive yourself?  Why not just accept the modesty 
>argument?  Just stop fighting, dammit!"  ...

I would not put my advice that way.  I'd say that whatever your reasoning, 
you should realize that if you disagree, that has certain general 
implications you should note.

>It happens every time a scientific illiterate argues with a scientific 
>literate about natural selection.  ...  How does the scientific literate 
>guess that he is in the right, when he ... is also aware of studies of 
>human ... biases toward self-overestimation of relative competence? ... I 
>try to estimate my rationality in detail, instead of using unchanged my 
>mean estimate for the rationality of an average human.  And maybe an 
>average person who tries to do that will fail pathetically.  Doesn't mean 
>*I'll* fail, cuz, let's face it, I'm a better-than-average 
>rationalist.  ... If you, Robin Hanson, go about saying that you have no 
>way of knowing that you know more about rationality than a typical 
>undergraduate philosophy student because you *might* be deceiving 
>yourself, then you have argued yourself into believing the patently 
>ridiculous, making your estimate correct

You claim to look in detail, but in this conversation on this the key point 
you continue to be content to just cite the existence of a few extreme 
examples, though you write volumes on various digressions.  This is what I 
meant when I said that you don't seem very interested in formal analysis.

Maybe there are some extreme situations where it is "obvious" that one side 
is right and the other is a fool.  This possibility does not justify your 
just disagreeing as you always have.  The question is what reliable clues 
you have to justify disagreement in your typical practice.  When you decide 
that your beliefs are better than theirs, what reasoning are you going 
through at the meta-level?  Yes, you have specific arguments on the 
specific topic, but so do they - why exactly is your process for producing 
an estimate more likely to be accurate than their process?

In the above you put great weight on literacy/education, presuming that 
when two people disagree the much more educated person is more likely to be 
correct.  Setting aside the awkward fact of not actually having hard data 
to support this, do you ever disagree with people who have a lot more 
literacy/education than you?  If so, what indicators are you using there, 
and what evidence is there to support them?

A formal Bayesian analysis of such an indicator would be to construct a 
likelihood and a prior, find some data, and then do the math.  It is not 
enough to just throw out the possibility of various indicators being useful.

Robin Hanson  [email-masked]  http://hanson.gmu.edu
Assistant Professor of Economics, George Mason University
MSN 1D3, Carow Hall, Fairfax VA 22030-4444
703-993-2326  FAX: 703-993-2323 

_______________________________________________
extropy-chat mailing list
[email-masked]
http://lists.extropy.org/mailman/listinfo/extropy-chat

---

From: "Eliezer S. Yudkowsky" <[email-masked]>
Date: Mon, 14 Mar 2005 12:00:15 -0800

Robin Hanson wrote:
> At 08:59 AM 3/14/2005, Dustin Wish wrote:
> 
>> Allow me a chance to add to this topic. First, programmed beliefs
>> are largely an environment factor that determines the "faith" in
>> those beliefs. If as a child you are taught that others are stupid
>> and you are smart then you will be predisposed to treating those
>> you deal with as morons. Not that you are smarter than they, but
>> that you are told that you are. That seems to me the basics of your
>> argument, what you are taught is right.
> 
> The pattern is so ubiquitous that it seems hard to believe there
> isn't a large genetic component.  It would be extremely hard to raise
> people from birth so that they did *not* think that they and their
> group are more reliable sources than others.

I don't think the correct term for this is "genetic component" - that 
would imply a large variance between humans attributable to genetic 
differences.  You're talking about evolutionary psychology, not 
behavioral genetics.  The correct phrasing would probably be, "it seems 
hard to believe this doesn't arise from our species psychology" or "it 
seems hard to believe there isn't a specific adaptation", depending on 
which of these propositions you meant.

Anyway...

Once upon a time I believed I was right and others wrong about a certain 
issue, even though I was only five years old, even though I was 
surrounded by people older and wiser than me, who said to me: you'll 
understand when you're older, and meanwhile do as we tell you.  So very 
arrogant was I, that I dared to defy them, and listen to the voice of my 
own reason which said that the adults' proposition was ridiculous.

I suppose I could go back and try to rebuild my psychology from scratch 
by reversing that five-year-old decision, since it is, after all, quite 
absurd to suppose that a lone five-year-old could face down full adult 
intelligences and win.  Is it not arrogant of me to believe, as I still 
believe even today, that I know so much better than my parents who have 
decades more of life experience?

But the Jewish religion still seems to me ridiculous, including that 
particular proposition to which I objected at the age of five, the 
requirement to pray in Hebrew when I didn't understand Hebrew.

Sabine Atkins once hypothesized to me that this childhood experience, my 
rejecting Judaism in the face of all adult assurance and then turning 
out to be right, had warped my entire psychology.  Perhaps so!  But this 
thing happened in the real world, and it is therefore appropriate to 
treat it as information.

I do think I originally learned the wrong lesson.  Up until around, oh, 
2002 or so, I thought the lesson was that intelligence was the most 
important thing in the universe.  For that my parents and rabbis had 
said to me: you may be intelligent, but experience is more important 
than intelligence; listen to us, when we tell you that the Jewish 
religion is right, and you'll understand when you're older.  I therefore 
concluded that sheer, raw intelligence was far more powerful than 
experience, that intelligence was the most important thing in the world 
- a conclusion that would later influence my beliefs about Artificial 
Intelligence, when in 1996 I first declared the quest for the Singularity.

In retrospect, I learned the wrong lesson.  I acted as if, just because 
my parents and rabbis said "experience is greater than intelligence", I 
could arrive to the truth simply by reversing their mistake.  I was 
foolish to let foolish people define my question for me.  The truth is 
very hard to find.  Other people's mistakes have no power to tell you 
where the truth hides, even if you reverse the mistakes.  You cannot 
attain the precise dance of the Way by reversing someone else's randomly 
wandering error.  But human nature is to say "Nay" where your opponent 
says "Yea", to let yourself be defined by the positions you oppose...

When I was five years old, I was probably not more intelligent than my 
parents; my brain was not that mature.  Even when I was thirteen years 
old, my parents could have used their greater life experience to defeat 
me - had my parents actually *used* their intellects, instead of 
searching for rationalizations for their birth religion.  The lesson was 
not that intelligence defeated experience, but that rationality defeated 
rationalization.  Intelligence, to be useful, must be used for something 
other than defeating itself.  One five-year-old's lone common sense 
defeated all those adult intellects, not because they were that stupid, 
nor because I was that smart, but because my five-year-old brain was 
actually processing the question instead of rationalizing a fixed 
answer.  As a five-year-old I couldn't possibly have defeated a 
reasonably smart and scientifically literate adult, if the adult were 
uncertain of the question and using their intelligence and life 
experience to curiously seek out an answer.  My parents could have 
defeated me handily, but they weren't in the game.

But the life lesson still holds.  I don't much credit the beliefs of 
people whom I don't think are applying their actual intellects to a 
question.  Nor would the modesty argument have served me well.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
_______________________________________________
extropy-chat mailing list
[email-masked]
http://lists.extropy.org/mailman/listinfo/extropy-chat

---

From: Robin Hanson <[email-masked]>
Date: Tue, 15 Mar 2005 19:48:27 -0500

At 03:00 PM 3/14/2005, Eliezer S. Yudkowsky wrote:
>>The pattern is so ubiquitous that it seems hard to believe there isn't a 
>>large genetic component.
>
>I don't think the correct term for this is "genetic component" ... The 
>correct phrasing would probably be, "it seems hard to believe this doesn't 
>arise from our species psychology" ...

I'll accept your correction.

>issue, even though I was only five years old, even though I was surrounded 
>by people older and wiser than me, who said to me: you'll understand when 
>you're older, ... But the Jewish religion still seems to me ridiculous, 
>... this thing happened in the real world, and it is therefore appropriate 
>to treat it as information.

And you never ever disagreed with your parents except on this one occasion, 
when you were later proven right?  Might there not be other occasions which 
don't come quite as gleefully to your recall, because you were later proven 
wrong?  Even the most rational estimators are wrong sometimes.  We should 
prefer to infer bias from patterns of error, rather than from individual 
cases.

>my parents could have used their greater life experience to defeat me - 
>had my parents actually *used* their intellects, ... The lesson ... that 
>rationality defeated rationalization. ...  I don't much credit the beliefs 
>of people whom I don't think are applying their actual intellects to a 
>question.

But the key question is: in ordinary practice how can you tell whether 
someone is reasoning or rationalizing, applying or not applying their 
intellect?  I agree that something like this is basically what I use 
informally to justify my disagreements.  But it bothers me greatly that I 
find it hard to be clear about what exactly are the clues that indicate 
this, and what is my evidence that such clues correlate as I claim.

---

From: Robin Hanson <[email-masked]>
Date: Tue, 15 Mar 2005 20:19:43 -0500

At 12:57 AM 3/13/2005, Eliezer S. Yudkowsky wrote:
>>Eliezer, you are just writing far too much for me to comment on all of it.
>
>Yes.  I know.  You don't have to comment on all of it.  I just thought I 
>should say all of it before you wrote your book, rather than afterward.  I 
>don't think that this issue is simple

I probably won't even get started on the book until this summer, and it 
will probably take me at least a year to write it.  So no particular rush 
here.  I do thank you for engaging me on the topic, and helping me to think 
about it.  And I agree that it is not at all simple.

>If I had to select out two points as most important, they would be:
>1) Just because perfect Bayesians, or even certain formally imperfect 
>Bayesians that are still not like humans, *will* always agree; it does not 
>follow that a human rationalist can obtain a higher Bayesian score (truth 
>value), or the maximal humanly feasible score, by deliberately *trying* to 
>agree more with other humans, even other human rationalists.
>2) Just because, if everyone agreed to do X without further argument or 
>modification (where X is not agreeing to disagree), the average Bayesian 
>score would increase relative to its current position, it does not follow 
>that X is the *optimal* strategy.

These points are stated very weakly, basically just inviting me to *prove* 
my claims with mathematical precision.  I may yet rise to that challenge 
when I get more back into this.

>>>I know of no *formal* extension of Aumann's Agreement Theorem such that 
>>>its premises are plausibly applicable to humans.
>These Bayesian Wannabes are still unrealistically skilled rationalists; no 
>human is a Bayesian Wannabe as so defined.  BWs do not self-deceive.  They 
>approximate their estimates of deterministic computations via guesses 
>whose error they treat as random variables.
>I remark on the wisdom of Jaynes who points out that 'randomness' exists 
>in the map rather than the territory; random variables are variables of 
>which we are ignorant.  I remark on the wisdom of Pearl, who points out 
>that when our map sums up many tiny details we can't afford to compute, it 
>is advantageous to retain the Markov property, ...  If the errors in BWs 
>computations are uncorrelated random errors, the BWs are, in effect, 
>simple measuring instruments, and they can treat each other as such, 
>combining their two measurements to obtain a third, more reliable measurement.

But Bayesian Wannabes *can* self-deceive.  The phrase "random variable" is 
a standard phrase in statistics - it just means any state function.  A 
real-valued random variable, which I use in that paper, is just a function 
that assigns a real number to each state.  I made no assumptions about 
independence or Markov properties.  Surely you believe that your error  can 
be described with a state function.

>>>>His [Aumann's] results are robust because they are based on the simple 
>>>>idea that when seeking to estimate the truth, you should realize you 
>>>>might be wrong; others may well know things that you do not.
>>>behave in a way understandable to you, then their observable behaviors 
>>>are relevant Bayesian evidence to you.  This includes the behavior of 
>The paper you cite above does not yield a constructive method of agreement 
>without additional assumptions.  But then the paper does not prove 
>agreement *given* a set of assumptions.  As far as I can tell, the paper 
>says that Bayesian Wannabes who agree to disagree about state-independent 
>computations and who treat their computation error as a state-independent 
>"random" variable - presumably meaning, a variable of whose exact value 
>they are to some degree ignorant - must agree to disagree about a 
>state-independent random variable. ... So in that sense, the paper proves 
>a non-constructive result that is unlike the usual class of Aumann 
>Agreement theorems.  Unless I'm missing something?

I do think you are misreading the paper.  *Given* that such agents are 
unwilling to disagree about topics where information is irrelevant, *then* 
such agents cannot disagree about *any* topic.  Which is another way to say 
they agree.

More some other day.