Computational & Technology Resources
an online resource for computational,
engineering & technology publications
PROCEEDINGS OF THE FIRST INTERNATIONAL CONFERENCE ON PARALLEL, DISTRIBUTED AND GRID COMPUTING FOR ENGINEERING
Edited by: B.H.V. Topping and P. Iványi
PyMW: a Python Module for Parallel Master Worker Computing
E.M. Heien1, A. Kornafeld2, Y. Takata1 and K. Hagihara1
1Graduate School of Information Science and Technology, Osaka University, Japan
E.M. Heien, A. Kornafeld, Y. Takata, K. Hagihara, "PyMW: a Python Module for Parallel Master Worker Computing", in B.H.V. Topping, P. Iványi, (Editors), "Proceedings of the First International Conference on Parallel, Distributed and Grid Computing for Engineering", Civil-Comp Press, Stirlingshire, UK, Paper 15, 2009. doi:10.4203/ccp.90.15
Keywords: master-worker, parallel computing, cluster computing, grid computing, volunteer computing, interpreted language.
In this paper we describe a general purpose master-worker parallel computation Python module called PyMW. Other tools for writing parallel programs, such as OpenMP, MPI or Globus, support a limited range of platforms. For example, an OpenMP program cannot fully utilize a computing Grid, and a Grid program contains more functionality than is required to run on a cluster. PyMW provides a unified interface to multiple computation environments including multicore processors, networked clusters and the Berkeley Open Infrastructure for Network Computing (BOINC) software platform. This allows researchers to easily write parallel master-worker style programs in Python that can run on a variety of platforms.
PyMW is intended to support rapid development, testing and deployment of large scale master-worker style computations, such as Monte Carlo simulations, parameter sweeps, or analysis of large data sets such as in SETI@home. PyMW allows researchers to quickly develop a parallel program on a multicore machine, test it on a cluster, and finally deploy it on a large scale Grid.
In this paper we discuss the organization of PyMW, and the functionality and abstractions necessary to support a generalized master-worker programming library. PyMW is designed to be nonblocking to allow normal user program flow, and to enable parallel computing with Python which does not support parallel execution. We describe the internal PyMW architecture which allows this functionality while enabling easy usage by programmers.
PyMW is designed to allow easy extension to additional parallel computing platforms through the use of an interface. We detail three interfaces already implemented for PyMW (multicore, MPI and BOINC) and describe the organization of an interface. Interfaces may be implemented with only a single function, while allowing expansion for interface specific capabilities such as redundant computation and worker reservation.
We demonstrate the effectiveness and scalability of PyMW by performing several master-worker style parallel computations on a multicore machine, a networked cluster and a BOINC project. The results of these tests demonstrate that PyMW adds little overhead to master-worker computations and scales well on these systems. These experiments demonstrate that the architecture of PyMW provides flexible and extensible master-worker style parallel computing ability on multiple platforms for the Python programming language.
purchase the full-text of this paper (price £20)