Computational & Technology Resources
an online resource for computational,
engineering & technology publications
Computational Science, Engineering & Technology Series
ISSN 1759-3158
Edited by: B.H.V. Topping and P. Iványi
Chapter 3

Today's Large-Scale Science depends on Network Data Movement based on Twenty Five Years of Technology Development

W.E. Johnston, E. Dart and B. Tierney

ESnet and Lawrence Berkeley National Laboratory, Berkeley California, U.S.A.

Full Bibliographic Reference for this chapter
W.E. Johnston, E. Dart, B. Tierney, "Today's Large-Scale Science depends on Network Data Movement based on Twenty Five Years of Technology Development", in B.H.V. Topping and P. Iványi, (Editor), "Developments in Parallel, Distributed, Grid and Cloud Computing for Engineering", Saxe-Coburg Publications, Stirlingshire, UK, Chapter 3, pp 41-65, 2013. doi:10.4203/csets.31.3
Keywords: data-intensive science, large-scale, widely distributed systems, R&E internet, massive data movement, federated network testing, network monitoring, high-throughput LANS, campus LANs, internet architecture.

Today's large-scale science projects all involve world-wide collaborations that routinely move 10s of petabytes per year between international sites.

The capabilities required to support this scale of data movement have involved hardware and software developments at all levels: Wire/fiber signal transport, layer 2 transport (e.g. Ethernet), data transport (TCP is still the norm), operating system evolution, data movement and management techniques and software, and in the science applications.

With each generation of network transport technology (155 Mb/s was the norm for high speed networks in 1995, 100 Gb/s - 700 times greater - is the norm today) R&D groups, involving hardware engineers, computer scientists, and application specialists, worked to first demonstrate in a research environment that "filling the network pipe" was possible, as well as having the ability to carry out the development necessary for applications to make use of the new capabilities.

This paper surveys the progress made over the past several decades toward building an infrastructure that enables large scale "data-intensive" science.

purchase the full-text of this chapter (price £20)

go to the previous chapter
go to the next chapter
return to the table of contents
return to the book description
purchase this book (price £85 +P&P)