The structure of the global catastrophe

15. Statements about possibility something and impossibility неравносильны

The statement about impossibility is much stronger, for enough one object concerns all set of potential objects, and for the validity of the statement about possibility. Therefore statements about impossibility something are false much more often. Assuming any event or coincidence of circumstances impossible, we cause a damage of our safety. In certain circumstances probably all. Thus any discussions about the future catastrophes is always discussions about possibilities.

16. Evidence as a source of errors

The correct conclusion always leans on two parcels, two true judgements. However the analysis of texts shows, that people very seldom use the full form of conclusions, and instead use reduced where only one parcel obviously is called, and another is meant by default. Are held back usually evidence - the judgements, seeming so true and doubtless, that there is need no them to sound. Moreover, it is frequent they are so obvious, that are not realised. It is clear, that such state of affairs is the reason of numerous errors because evidence - not necessarily validity, and that is obvious to one, is not obvious to another.

17. Underestimation of own inaccuracy

As well as any human, I is inclined to be mistaken, that is connected as with the basic unreliability of a human brain connected with the likelihood nature of its work, and with incompleteness of my knowledge of the world and skills of elimination of errors. I can know nothing on 100 % because reliability of my brain is not equal 100 %. I can check up reliability, having solved a series of logic problems average complexities, and then having counted quantity of errors. However usually it does not become, and own inaccuracy is estimated intuitively. Precisely also human usually does not measure a characteristic inaccuracy of the judgements about the future though it probably to make experimentally: for example, to write the forecast of the and public life for year or five years and then to compare.

18. The error connected with representation that each event has one reason

Actually:

There are absolutely casual events.

Each event has many reasons (the glass has fallen because it have put with edge because it is made of glass because force of gravitation it is great because a floor firm because the cat disobedient because it should happen sooner or later).

Each reason has the reason therefore we have dispersing in last tree of the reasons. Human mind is incapable entirely this tree of the reasons to capture and is compelled to simplify. But the concept "reason" is necessary in a society because it is connected with fault, punishment and a free will. That is here under "causal" acceptance by the free made human of the decision on crime fulfilment means. There is no need to speak about that, how many here the unevident moments. (The basic question: Who is guilty?)

And in technics designing: where it is important to find a cause of catastrophe. That is that it is possible to eliminate - so that failures such any more were not. (The basic question: What to do?)

The concept the reason less all is applicable to the analysis of the difficult unique phenomena, such as human behaviour and history. The example to that is weight of the confused discussions about those reasons or other historical events. For this reason reasonings in a sort «the reason of global catastrophe will be Х» - to put it mildly, are imperfect.

19. Necessity of a choice on the basis of belief

If the head receives some the conclusions contradicting each other about safety it makes a choice between them, simply trusting in one of them - for the reasons which have been not connected with the logic. Here too it is possible to recollect the term «an existential choice» when human should make a choice in a non-formalizable situation. For example, between love and a debt.

20. Effect of first and last read book

The order of receipt of the information influences its estimation, and are allocated first and last source. It is one of forms of the inaccuracy connected with availability of the information.

21. Exaggeration of a role of computer modelling

Most two worked models - meteorology and nuclear explosions. Both are made on a huge actual material, with the account of hundreds tests which made amendments to forecasts, and both regularly gave errors. Even the most exact model remains model. Therefore we cannot strongly rely on computer modelling of unique events to what global catastrophe concerns.

22. The proof by analogy as a source of possible errors

Business not only that there can not be analogies to the unique event, which else never happened - to irreversible global catastrophe, but also that we do not know how to draw such analogies. In any case, analogy can illustrate only. Possibly, it is useful to accept analogies when they speak about a reality of a certain threat, but not when - about safety.

23. The error connected with discrepancy of extrapolation экспоненциальной of likelihood function by means of the linear

Likelihood function of  destruction of a civilisation - if to consider it process smooth in sense of probability, that is, of course, incorrect - it is possible to assimilate functions of disintegration of radioactive atom which, as is known, is described экспонентой. For example, if the probability of  destruction of a civilisation during the XXI century is equal 50 % as it is assumed by sir Martin Rees in the book «Our last hour» in 200 years the chance of a survival of a civilisation will be 25 %, and through one thousand years - only 0,1 % - at uniform preservation of the same tendencies. From here it is visible, that it is incorrect to conclude, that time chances of a survival within a millenium makes 0,1 % for one century it will be in only ten times more, that is 1 %. The Same error in less obvious kind arises, if we need to extrapolate the same 50 % of a survival within 100 years on погодовую probability of  destruction. Linear approximation would give 0,5 % for a year. However the exact value calculated under formula , makes approximately 0,7 %, that is in 1,4 times above, than intuitive linear approximation gives.

24. The St.-Petersburg paradox

This paradox has the direct relation to global catastrophes as shows that infinitely big damage from the extremely rare events has больший weight, than all other events, however psychologically people are not ready to apprehend it. G.G.Malinetsky so describes this paradox in the book «Risk. A sustainable development. Synergetrics":"we Will consider the following game. The coin is thrown until the eagle for the first time will not drop out. If it was required n throws the prize will make 2ⁿ units. That is prizes 2,4,8, … 2ⁿ will occur to probability 1/2,1/4,1/8, … 1/2ⁿ. The expected prize in this game is infinite:

.

It is asked, how many human is ready to pay for the right to enter into such game. The paradox consists that the majority of people is ready to pay for this right no more than 100, and sometimes and 20 units »

25. Distinction between danger and risk

The risk is created by accepted decisions, and dangers - circumstances. As the basic source of risk of global catastrophes are new technologies decisions on their development and application define it. However if technologies develop spontaneously and неосознанно they become similar to natural dangers.

26. The error connected by that if probability nobody events is not computable, to it believe the zero

Whereas the principle of precaution would demand, that we attributed to such events 100 percentage probability. However it would lead to absurd conclusions in the spirit of: the probability of disembarkation of aliens is unknown tomorrow, therefore we should prepare for it how if it was equal to 100 percent. In this case it is possible to use indirect ways of an estimation of probability, for example, formula Готта.

27. Omission of that safety of system is defined by its weakest link

If in a premise there are three parallel doors, one of which is locked by three locks, the second - two, and the third - one the premise is locked on one lock. As do not strengthen two strongest doors, it will change nothing.

28. Отвержение hypotheses without consideration

To reject a certain hypothesis, it should be considered in the beginning. But it is frequent this sequence it is broken. People refuse to consider those or other improbable assumptions because they reject them. However reliably to reject a certain assumption it is possible, only carefully it having considered, and for this purpose it is necessary to accept it at least for some time seriously.

29. Невычислимость

Variety of essentially important processes for us is so combined what to predict them it is impossible, as they невычислимы. Невычислимость can have the different reasons.

It can be connected with incomprehensibility of process (for example, Technological Singularity, or, for example, how the theorem the Farm is incomprehensible for a dog), that is is connected with basic qualitative limitation of a human brain. (Such is our situation with a prediction of behaviour of Superintelligence in the form of AI.)

It can be connected with quantum processes which do possible only a likelihood prediction, that is недетерминированностью systems (weather forecast, a brain).

It can be connected with supercomplexity of systems in which force each new factor completely changes our representation about a definitive outcome. That concern: models of global warming, nuclear winter, global economy, model of exhaustion of resources. Four last fields of knowledge unite that everyone describes the unique event, which else never was in history, that is is advancing model.

Невычислимость can be connected that the meant volume of calculations though is final, but it is so great, that any conceivable computer cannot execute it during Universe existence (such невычислимость it is used in cryptography). This kind невычислимости to be shown in the form of chaotic determined by system.

Невычислимость it is connected also by that though to us the correct theory can be known (along with many other things), we cannot know, which theory is correct. That is the theory, besides correctness, should be easily demonstrable for everything, and it not one too, in conditions when experimental check is impossible. Somewhat in the way of calculation of correctness of the theory, to be exact - measures of confidence of them, the market where or direct rates on a certain outcome become is, or on the price there is nobody the goods connected with the forecast, for example, of the price for oil. However the theory influences a market price many other factors: gamble, emotions or not market nature of the object. (It is senseless to be insured against global catastrophe as there is nobody and there is nobody for it will pay off, and owing to it it is possible to tell, that its insurance price is equal to zero.)

One more kind невычислимости is connected with possibility of realisation of self-coming true or self-denying forecasts which do system essentially astable and unpredictable.

Невычислимость, connected with the assumption of own site (self-sampling assumption - see about it N.Bostroma's book). The essence of this assumption consists that in some a situation I should consider myself as the casual representative from some set of people. For example, considering myself as usual human, I can conclude, that I with probability in 1/12 had chances to be born in September. Or with probability, let us assume, 1 to 1000 I could be born the dwarf. It sometimes allows to do predictions on the future: namely, if in Russia 100 billionaires chances, that I will become the billionaire, make one to 1,5 million, in the assumption, that this proportion will remain. To невычислимости it results, when I try to apply the assumption of own site to own knowledge. For example, if I know, that only 10 % of futurologists give correct predictions I should conclude, that with chances of 90 % any my predictions wrong. The majority of people do not notice it as for the account of superconfidence and the raised estimation consider itself not as one of representatives of set and as "elite" of this set, the possessing raised ability to predictions. It is especially shown in gamblings and game in the market where people do not follow obvious thought: «the Majority of people loses in a roulette, hence, I, most likely, will lose».

the Similar form невычислимости is connected with an information neutrality of the market. (Told further is considerable simplification of the theory of the market and problems of information value of indicators given to it. However more detailed consideration does not remove the named problem but only complicates it, creating one more level невычислимости - namely impossibility for the usual human to seize all completeness of knowledge connected with the theory of predictions, and also uncertainty of what of theories of predictions is true. See about information value of the market so-called «no trade theorem».) The ideal market is in balance in which half of players considers, that the goods will rise in price, and half - what to become cheaper. In other words, win in game with the zero sum the majority of people, human can only cleverer or informed, than. However the majority of people everything are not cleverer, than, by definition though are not capable to realise it because of psychological bias. For example, the price for oil is at such level that does not give obvious acknowledgement to the assumption of inevitability of the crisis connected with exhaustion of oil, the assumption of limitlessness of oil stocks. As a result the rational player does not receive any information on for what scenario to it prepares. The same situation concerns and disputes: If a certain human has chosen to prove the point of view opposite to yours, and you of anything do not know about its intelligence, erudition and information sources, and also about the objective rating, that is chances 50 on 50, that it is right, instead of you. As objectively to measure the intelligence and awareness extremely difficultly because of desire them to overestimate, it is necessary to consider their being in the spectrum middle.

As in a modern society mechanisms of transformation of any future parametres in market indexes (for example, trade in quotas under the Kiotsky report on emissions of carbonic gas or the rate on elections, war and т operate. д, futures for weather) it brings an additional element of basic unpredictability in all kinds of activity. Owing to such trade we cannot learn for certain, whether there will be a global warming, exhaustion of oil, what real threat of a bird flu.

One more reason невычислимости - privacy. If we try to consider this privacy through different «plot theories» in the spirit of book Симмонса «Twilight in desert» about преувеличенности estimations of stocks of the Saudi oil we receive dispersing space of interpretations. (That is, unlike a usual case when accuracy raises with number of the measurements, here each new fact only increases split between opposite interpretations.) any human on the Earth does not possess all completeness of the classified information, as at the different organisations different secrets.

The psychological aspect of this problem consists that people argue how if any невычислимости was not. In other words, it is possible to find out some opinions and reasonings on the future in which its basic and multilateral unpredictability is not considered at all, no less than limitation of human ability it is authentic about it to argue.

33. Observant selection

Observant selection is a form of an error when owing to a way of product of experiment its one results are more appreciable, than others. For example, brighter stars in the sky are more appreciable, but it does not mean, that all stars bright. A special case of observant selection is when the different outcome of event observes different number of observers that takes place in case of  destruction of the observer in one of outcomes. For example, the soldiers who have gone through of some months of war, are inclined to overestimate the survivability - that is at them the erroneous sensation of own invulnerability is born. Global catastrophes, obviously, concern such class of processes. In my article «Natural catastrophes and Antropic principle» and in the head about indirect estimations of probability of global catastrophe in this book more in detail see.

Chapter 5. The specific errors arising in discussions about danger of uncontrollable development of an artificial intellect

1. The disorder of opinions of safety of AI means absence of real proofs of safety

Experts in computers state many different opinions on the reasons of why, in their opinion, AI is safe. These statements взаимоисключают each other, and, means, their considerable part is false. Therefore we can safely result them as examples of errors, not pressing in proofs ложности each separate statement. I have spent on the Internet interrogation among developers of AI, on a theme of that is a guarantee of safety of AI, and have received the following disorder of opinions, with approximately identical number of the people, holding this or that opinion. So, «AI is safe, because»:

1) because AI is impossible.

2) because AI can solve only narrow problems, like recognition of images.

3) because when we will make universal AI, we will enter into it Three laws of a robotics of Azimov.

4) Because I know how to use AI in the blessing to people.

5) Because AI will possess superhuman wisdom by definition and will not want to cause harm to people.

6) Because AI it is necessary nothing from people, and we can co-exist peacefully with each other.

7) Because it will be locked in my computer and if something goes not so, I will switch off an electricity.

8) because at AI cannot be the will.

9) AI is impossible without a free will, therefore we should give it freedom.

10) Because AI cannot make anything bad.

11) AI can be dangerous, but, most likely, all will manage.

12) Is not present, AI is dangerous, and all of us are doomed.

13) AI will destroy mankind, and we should aspire to it, as AI is more progressive form of evolution.

In a course опороса it has appeared, that these opinions are distributed more or less in regular intervals. It means, that the information sum of safety of AI which researchers of AI as a whole possess, is equal to zero.

2. The idea that it is possible to create faultless system, repeatedly having checked up its project and an initial code is erroneous

Checks bring some number of new errors, and owing to it at certain level the number of errors is stabilised. It is true and about systems целеполагания what laws, for example, are. It is not necessary to count, that we can create the arch corrected behaviour for AI, not containing errors.

3. Errors in the critic of AI Пенроузом

R.Penrouz in the book «New mind of the king» asserts, that AI is impossible, because in a brain there are not computable quantum processes which are necessary for creative thinking and consciousness. On the basis of it often affirms, that dangerous AI is impossible. It is represented to me, that the given conclusion is not quite correct for following reasons:

1. The quantum nature of consciousness it is far not мэйнстрим sciences. We cannot base safety of mankind on not proved (though also interesting) the theory.

2. It is impossible to prove impossibility something in the nature theoretically. (But possibility to prove it is possible - for example, a background Neumann has proved possibility of self-reproduced mechanisms.)

3. That AI became dangerous, it does not need to possess neither consciousness, nor ability to creativity. Modern AI can beat any human in chess, not using neither consciousnesses, nor intuitions. That is the intuition is only the useful characteristic reducing speed of search of combinations, but replaced with algorithms. It is enough to tell, that dangerous AI is AI which can beat human in any game. War and earning money are versions of such games.

4. If certain special quantum functions are carried out нейронами or microtubes nothing prevents to use them as elements of future powerful AI - for example, in the form of the external coprocessor. Quantum computers are one of realisation variants. Eventually, strong AI can arise not as the computer, and as genetically modified human with нейрошунтом (that is connected directly to the computer). And at the worst, if the class of the problems accessible to human is found out, but not accessible to the computer, the situation of direct operation of people for the decision of these problems is possible, thus each of people will not know, its decision will be applied how. So now spam programs, for example, work - they ask people on the site with «клубничкой» to distinguish estimated figures in the drawing taken from other site, and thus get access to this other site. Thus people use blindly as people do not know, that results of their recognition will be used in other place. Similarly the state employs scientists for weapon working out.

5. The considerable number of the important problems can be solved search, that is by realisation of final number of operations. For example, it can be search of all possible combinations of the signs generating the text of the proof of the theorem. In other words, there is an algorithm, allowing to solve any problem. And a question only in its optimisation. Probably, that the decision of those problems on which the destiny of mankind can depend, quite probably search even if there are some problems which search do not dare during Universe existence.

6. If it will be possible to prove, that the consciousness all the same possesses the quantum nature, it will open the new world of possibilities, so also new risks.

4. Representation that is enough 3 laws of a robotics of A.Azimova to solve all problems with safety of AI

It was already repeatedly shown in different researches, that robotics laws are not a guarantee of safety AI in a little considerable measure:

1. These laws are a tautology as from them follows, that AI will be safe, because it will not cause harm.

2. They comprise contradictions which is masterful is beaten by Azimov in its stories - the robot often tests the conflict between 1, 2 and 3 laws and as a result behaves in the dangerous image.

3. These laws concern independent robots, instead of to AI which is not adhered to one mechanism.

4. They lean against intuitive concept of "harm" which is not defined by them and consequently can accept any form. For example: to live harmfully because of it die.

5. These laws are faster wishes - that is that we want from AI, instead of rules by which it could be guided. It is not known, how to place these laws in AI.

5. Erroneous representations that progress in the field of the software is absent

Algorithms of decomposition of numbers on a multiplier improve faster, than hardware maintenance. There is a progress and in other areas, but it less will measure. Eventually, equipment progress is too progress of that our understanding how to make microcircuits less.