Outsourcing of research: Change and stability Andrew Odlyzko AT&T Labs - Research amo@research.att.com Revised draft January 22, 1998 Abstract Outsourcing of development is growing rapidly. This trend is promoted by the move of most industries towards horizontal integration, and away from the traditional vertical structure. However, while the same factors might seem to favor outsourcing of research, there are countervailing influences. As a result, it appears that research continues to be done in-house almost as much as before. Research is also growing in industries that did not see it as necessary before. 1. Introduction AT&T is an extreme example of the huge changes in R&D that have been taking place recently. Two decades ago AT&T was a vertically integrated monopoly, inventing new technologies such as the transistor at the same time it was producing telephone poles, creating software for its switches, and providing the services that justified and paid for everything else. In the early 1980s, the federal government broke up the Bell System. This eliminated the vertical integration of the regional operating companies, but left AT&T still vertically integrated, producing equipment, software, and related systems for long distance communications services. However, while the internal R&D budget of AT&T was growing, there was also increasing reliance on outside suppliers for microprocessors and other crucial components, so that the relative share of R&D done in-house was declining. Finally, in 1995, in response to changes in the political and regulatory arenas as well as on the technology front, AT&T decided to split itself into three companies. The part that retained the name AT&T produces no hardware at all. Most of the old AT&T's R&D activities went to Lucent Technologies, and AT&T benefits from them and pays for them through purchase of systems from Lucent. It is hard to think of a more dramatic increase in outsourcing of R&D. Similar (although smaller and much more gradual) increases in outsourcing through outside purchases of products have been reshaping R&D in most industries. The story appears to be different, though, when we consider direct outsourcing, in which one company contracts with another for R&D. General statistics (such as those collected by the National Science Foundation [NSF]) do show an increase in such activity. However, the increase is modest, especially when compared to increases through purchases of products. Consortia involving industry, academia, and government are growing, but not dramatically. The growth in outside contracting for R&D starts from a low base, and appears to be limited primarily to those parts of development where the desired products can be specified precisely. The limited growth in R&D outsourcing is the result of several countervailing tendencies. These are discussed at greater length in Section 2. In general, such stability in spite of rapid change should not be a surprise. We can observe it in other areas. Better communications (phone, fax, email, videoconferencing) has not reduced business travel. While much more is done through electronic means, there is much more worldwide competition as well as alliances. This requires more coordination work, and has led to a steady growth in business trips. Electronic communication has made a difference in how business is transacted, but has not reduced the need for personal contact. Similarly, cash machines and electronic banking were often predicted to lead to a drastic decline in bank branches and bank employment. This has not happened. While ATM transactions cost less than a tenth of what teller transactions cost, people use ATM machines much more than teller windows, so costs have not gone down. Further, as banking products have been growing in sophistication, the need for human agents to deal with customers has not decreased much. Business travel and bank tellers have little directly in common with R&D, but do serve to show how many plausible predictions fail to come true. Probably more remarkable than the growth in outsourcing of R&D has been the establishment or extension of research facilities at some companies that have not been noted for activities in this area. An example is the widely noted expansion of Microsoft's research lab. Another example is MCI. MCI used to boast of not having any research, and of getting it inexpensively from its suppliers (with the value of the suppliers' R&D that it was drawing on estimated at $15 billion, [Rendleman]). Yet in early 1997, while it was planning to be acquired by British Telecom, MCI boasted of gaining access to BT's research facilities. MCI estimated the value of BT Labs' research was in the billions of dollars and was expected to lead to deployment of new services and faster time to market. (The change of purchaser, from BT to WorldCom, may lead to a change in MCI's plans.) The quickening pace of change has motivated even service companies to invest in research. 2. On the desirability of R&D outsourcing Coase [Coase] (see also [Casson, Hayek]) pointed out that the main justification for existence of firms is to economize on the costs of coordinating economic activity. There are serious drawbacks to using the price mechanism; the costs of learning about available goods, the costs of learning prices, the costs of negotiating contracts, and so on. As communication (both electronic and physical) has become easier, these costs have declined, and this has led to a tremendous increase in outsourcing, as Coase's theory predicts. Firms purchase cafeteria services as well as goods from outside suppliers. Why not R&D? It is much easier to outsource R&D than formerly. The technology area is much more competitive, with many willing suppliers among contract houses, universities, as well as large research institutes and other corporations. And indeed we do see R&D done under contract. However, that is happening in restricted domains, primarily when the product can be specified precisely. In those areas, it is sometimes even feasible to send the work to India or Russia. There are serious limitations to this process, though. In areas of advanced development and especially research, where there are substantial uncertainties about the cost and outcome of a project, coordination and negotiation costs are overwhelming. This is very noticeable in the dealings between AT&T and Lucent. While the two companies were one, a research project could be launched based on a few conversations between researchers and managers and perhaps a memo for record. Now, after the breakup, relations are like those of a typical pair of unrelated firms. Months of negotiations involving lawyers are required. How tight are the deadlines, who bears cost overruns, who owns the patent rights or software? All these questions need to be spelled out and their implications carefully considered. When the goal can be precisely specified, there are standardized procedures developed through long practice that simplify the process. However, for research and parts of developments, the uncertainties create barriers and thus costs that discourage outsourcing. In general, there are many attractions to outsourcing R&D. There is potential for more flexibility (with a greater variety of solutions), lower costs (through competitive procurement and targeting only the specific areas that are of immediate interest), and better solutions (through choice of the best technologies). Perhaps most important of all, there is so much technical knowledge relevant to just about any R&D project that no single firm can hope to possess it all. On the other hand, there are almost always many competent suppliers of any required expertise. While there are attractions to outsourcing, there are also serious limitations. There are the negotiation and coordination costs mentioned above. There is loss of control over the technology and expertise with outsourcing. There is also potential loss of customer perception of technical leadership, and the loss of the technical expertise that is needed not just for effective implementation, but even for selecting technologies and projects in the first place. These and related limitations have meant that advanced development and research tend to be done internally. Even when U.S. firms do use programmers in India and Russia, they tend to employ them directly, to avoid such difficulties. 3. The new modes of research Why would a communications service company want research? Let me cite two recent examples involving my close colleagues at AT&T Labs - Research. The first one involves statistics and large databases, the second one data networks. After long discussions, on October 15, 1997, the Federal Communications Commission (FCC) ruled that owners of pay phones would be paid $0.284 for each 800 call made from their phones (with payments starting retroactively on October 7, 1997). Before that, owners of public pay phones received flat-rate monthly compensation. The owners of the pay phones claimed that this compensation was inadequate owing to the growing use of prepaid calling cards (among other things), leading to an increasing proportion of non-revenue generating 800 calls. This has led pay phone owners to demand usage-based compensation. There was no dispute about the need for some payments, just about the amount, and the October 15, 1997 decision by the FCC was a compromise between the proposals of the carriers and the pay phone owners. Unfortunately the FCC decision has opened the door to serious abuses. A pay phone user does not pay for 800 calls, so can maliciously drive up the bill for United Airlines, say, by repeatedly calling their 800 number. However, such abuse, aside from leaving a person liable to prosecution, requires time from the abuser, and does not provide any direct benefit to anyone. With the new system of payments to pay phone owners, though, any call to an 800 number triggers a $0.284 payment to the phone owner. Even if the pay phone owner's dog happens to accidentally push the buttons for an 800 number, a payment is triggered. (The usual explanation for these mishaps is not that "The dog did it," but instead that "The pay phone software went haywire." After all, everybody knows how unreliable those mysterious programmable devices are!) What does this story have to do with research? Well, suspicious patterns of use of 800 calls started immediately after the FCC decision, so speed, much greater speed than is typical of commercial system development, was important. For example, on Oct. 15, one pay phone suddenly went from about 24 calls per day to over 2000 calls per day and maintained that level for the next 28 days. (Recall that the FCC decision was issued on Oct. 15.) Other supposed pay phones eligible for compensation were observed making tens of simultaneous calls, indicating that they were not pay phones in any conventional sense. This suggested errors on the records generated by non-AT&T software in the local switch. The only way for a carrier such as AT&T to protect itself in such situations is to move quickly to detect, deter, and prevent fraud and recording anomalies. Now when a pay phone starts making 1,000 calls per day, it is a no-brainer that something suspicious is going on, and no elaborate research tools are necessary. However, it is harder to detect the addition of 10 fraudulent calls per day in a pay phone that normally carries around 100 legitimate calls. Yet even 10 such calls mean $2.84 per day, or over $1,000 per year for each such phone. To deal with these levels of abuse, sophisticated statistical techniques are necessary. Abstract ideas can be implemented quickly, and their value can be easily quantified by comparison with other approaches. Further, the number of calls that have to be monitored is huge, so the database problem is among the most challenging in this area, requiring real-time operations, and requires skills on the cutting edge of computing science. The availability of such skills in a centralized research lab enabled AT&T to act promptly, and have the tools in place to monitor abuse even before it started. The other example that I will cite is of research on the fractal behavior of data networks. There is an immense literature on switched telephone networks. However, the future, including that of voice calls, belongs to packet networks. Instead of a continuous connection between receiver and transmitter, everything will be encapsulated in blocks that will share a channel with blocks from other sources, as is done on the Internet. Such packet transmissions turn out to behave unlike voice calls. The traditional Poisson-based models that worked just fine for switched telephone calls do not fit data traffic well. There is much less smoothing in traffic statistics, and new phenomena, such as long-range dependence, make their appearance. (The basic reference for this work is [LelandTWW], and some of the latest results and references to earlier work are given in [FeldmannGWK].) Although such fractal (or self-similar) phenomena cannot be captured in a parsimonious manner with conventional modeling techniques, they impact practically every aspect of the design, management and control of modern high speed networks. For example, buffers in data network switches have to be much larger than was initially expected on the basis on conventional design rules (developed for the switched voice network), and utilization of transmission lines is lower. It is not a surprise that AT&T is deeply interested in research on such phenomena. It is research, in that we do not know what will come of it. The aim is to develop new tools for traffic engineering that use the new insights, and this requires extensive investigation in data analysis, mathematical modeling, queuing theory, networking, and communications research. While there have been numerous spinoffs of this work, nobody understands the full implications of it yet. Being the leader in this area gives AT&T a competitive advantage, one that would not be available if it relied on others to carry out such work. 4. The changing nature of research The two examples cited above could be claimed not to be as fundamental as the search for a grand unified theory of physics or even the invention of the transistor. It is certainly true that our progress in communication and computation is driven by the progress at the basic physical level, with larger and faster integrated circuits and fiber optic lines making possible the complicated systems that we are building. However, even as the work on this physical level continues, an increasing fraction of the total R&D effort is being shifted towards the systems level, since that is where the most urgent problems are. A similar phenomenon can be observed on a national scale. Several papers in these proceedings note that U.S. federal funding for R&D has been increasingly tilted towards the biological sciences. There has never been any explicit declaration by Congress or the Administration, or any scientific body, that this should happen. However, statistics on funding show this relative shift towards biology and medicine started several decades ago. The traditional inverted pyramid of science has physics at the fundamental level, chemistry drawing on physics, then biology drawing on chemistry, and finally medicine drawing on biology. There is some truth to this, and many of the recent advances in medicine can be traced to progress in physics (in imaging, for example). Yet in the local balancing that is taking place all the time about allocation of resources, the emphasis has been shifting towards the higher (and already much larger) levels of the pyramid. Quantum mechanics is crucial in understanding water molecules, which are certainly crucial in understanding the biology of the cell, which is vital in medicine. However, society is not willing to wait until the properties of cells are deduced from quantum mechanics. Most of science is devoted to deducing how complex systems operate, even though we lack a complete understanding of the behavior of their components. The physical science pyramid provides a way to think about some trends in modern communications research. The work on the fractal nature of data networks mentioned in the preceding section can be compared to biology (and the hoped-for insights into traffic engineering to medicine). Even if we neglect the quantum mechanics and other basic physics that produce modern communication and computation devices, and regard networks as digital systems, they are far too complicated to understand from basic principles. Not only are the basic silicon chips complicated, but they run protocols that are complicated. Furthermore, these devices and software interact with people in unpredictable ways. The resulting system is extremely complex, and so sophisticated tools and models are being developed to understand its large-scale behavior. These models do draw on the knowledge of the underlying system (just as a cancer researcher will draw on knowledge of the behavior of enzymes), but concentrate primarily on the statistics of the entire system. The analogy with the physical science pyramid also helps explain the shorter time horizons of much of modern research. It is still true that basic ideas can take 20 years or more before they show up in products and services. However, just as basic medical research is often immediately applicable, so is much of modern communications research. To study data networks, one has to study the Internet, and any new proposals can often be implemented quickly. Similarly, to do leading-edge research in databases, one has to have access to large databases (and communications companies have some of the largest ones), and to realistic problems to work on. This is definitely research. It does rely on deep methods and insights, is fun to do, and has important consequences. However, it is different from much of what used to be regarded as fundamental research, and does require closer contact with applications than has been traditional. 5. Conclusions R&D is changing, as has been noted several times [GibbonsLNSST, Odlyzko1, Odlyzko2, Ziman]. However, while there is some growth of outsourcing, it is limited. More important are the evolutionary changes that are directing work into new areas and often require new modes of operation. Acknowledgements I thank Mel Cohen, Bob Kostelak, and Daryl Pregibon for the information they provided and comments. References [Casson] M. Casson, "Information and Organization," Oxford Univ. Press, 1997. [Coase] R. H. Coase, The nature of the firm, Economica (N.S.) 4 (1937), pp. 386-405. [FeldmannGWK] A. Feldmann, A. C. Gilbert, W. Willinger, and T.G. Kurtz, Looking behind and beyond self-similarity: On scaling phenomena in measured WAN traffic, AT&T Labs report. To appear in Proc. 35th Annual Allerton Conference on Communication, Control and Computing, Allerton House, Monticello, Illinois, Sept. 29 - Oct. 1, 1997. [GibbonsLNSST] M. Gibbons, C. Limoges, H. Nowotny, S. Schwartzman, P. Scott, and M. Trow, "The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Society," Sage Publications, 1994. [Hayek] F. A. von Hayek, Economics and knowledge, Economica (N.S.) 4 (1937), pp. 33-54. [LelandTWW] W. E. Leland, M. S. Taqqu, W. Willinger, and D. V. Wilson, On the self-similar nature of Ethernet traffic (extended version), IEEE/ACM Trans. Networking 2 (1994), 1-15. [NSF] NSF Directorate for Social, Behavioral, and Economic Sciences, science statistics, available at . [Odlyzko1] A. M. Odlyzko, The decline of unfettered research, to be published. Available at . [Odlyzko2] A. M. Odlyzko, The future of research: Decline or transformation?, in "Proc. 19th Annual Meeting," C. H. McGruder, III, and W. E. Collins, eds., Nat. Soc. Black Physicists, 1996, pp. 89-93. Available at . [Rendleman] J. Rendleman, MCI prepares for British invasion, Communications Week, May 5, 1997. Available through search at . [Ziman] J. Ziman, "Prometheus Bound: Science in a Dynamic Steady State," Cambridge Univ. Press, 1994.