Computational & Technology Resources
an online resource for computational,
engineering & technology publications
Computational Science, Engineering & Technology Series
DEVELOPMENTS IN PARALLEL, DISTRIBUTED, GRID AND CLOUD COMPUTING FOR ENGINEERING
Edited by: B.H.V. Topping and P. Iványi
Today's Large-Scale Science depends on Network Data Movement based on Twenty Five Years of Technology Development
W.E. Johnston, E. Dart and B. Tierney
ESnet and Lawrence Berkeley National Laboratory, Berkeley California, U.S.A.
W.E. Johnston, E. Dart, B. Tierney, "Today's Large-Scale Science depends on Network Data Movement based on Twenty Five Years of Technology Development", in B.H.V. Topping and P. Iványi, (Editor), "Developments in Parallel, Distributed, Grid and Cloud Computing for Engineering", Saxe-Coburg Publications, Stirlingshire, UK, Chapter 3, pp 41-65, 2013. doi:10.4203/csets.31.3
Keywords: data-intensive science, large-scale, widely distributed systems, R&E internet, massive data movement, federated network testing, network monitoring, high-throughput LANS, campus LANs, internet architecture.
Today's large-scale science projects all involve world-wide collaborations that routinely move 10s of petabytes per year between international sites.
The capabilities required to support this scale of data movement have involved hardware and software developments at all levels: Wire/fiber signal transport, layer 2 transport (e.g. Ethernet), data transport (TCP is still the norm), operating system evolution, data movement and management techniques and software, and in the science applications.
With each generation of network transport technology (155 Mb/s was the norm for high speed networks in 1995, 100 Gb/s - 700 times greater - is the norm today) R&D groups, involving hardware engineers, computer scientists, and application specialists, worked to first demonstrate in a research environment that "filling the network pipe" was possible, as well as having the ability to carry out the development necessary for applications to make use of the new capabilities.
This paper surveys the progress made over the past several decades toward building an infrastructure that enables large scale "data-intensive" science.
purchase the full-text of this chapter (price £20)