|Volume 7, Issue 1 -
Parkbench Committee Releases Parallel Benchmarks
Within the high-performance computing community, there has been a growing need for developing a standardized, rigorous, and scientifically tenable methodology for studying the performance of high-performance computer systems. Such a methodology would help to:
The Parkbench committee was established with these goals in mind. Originally called the Parallel Benchmark Working Group, PBWG, the committee was founded at SUPERCOMPUTING '92 in Minneapolis, when a group of about 50 people interested in computer benchmarking met under the joint initiative of Tony Hey of the University of Southampton (UK) and CRPC researcher Jack Dongarra of the University of Tennessee and Oak Ridge National Laboratory. Representatives came from universities, laboratories, and industries and from computer manufacturers and computer users on both sides of the Atlantic. Roger Hockney of the University of Southampton chaired the meeting.
The group agreed upon several objectives to meet their goals. A comprehensive set of parallel benchmarks was needed that was generally accepted by both users and vendors of parallel systems; furthermore, a focus for parallel benchmark activities was needed to avoid unnecessary duplication of effort and proliferation of benchmarks. The group also wanted to set standards for benchmarking methodology and result reporting and to establish a control database/repository for both the benchmarks and the results. Furthermore, the benchmarks and results needed to be freely available in the public domain.
The first year's work has produced a report and an initial set of benchmarks. The committee met at the University of Tennessee in Knoxville on March 1-2, 1993, May 24, 1993, and August 23, 1993 to discuss the evolving draft of the report. The report is the final result of these meetings, and is the first official publication of the Parkbench committee. It will be distributed at a public 'Birds of a Feather' meeting at SUPERCOMPUTING '93, Portland, on November 17, 1993, together with the first release of the Parkbench parallel benchmarks.
The initial focus of the parallel benchmarks is on the new generation of scalable distributed-memory message-passing architectures for which there is a notable lack of existing benchmarks. For this reason, the initial benchmark release concentrates on Fortran 77 message-passing codes using the widely available PVM message-passing interface for portability. Future versions will undoubtedly adopt the proposed MPI interface, when this is fully defined and becomes generally accepted. The committee's aim, however, is to cover all parallel architectures, and this is expected to be achieved by producing versions of the benchmark codes using Fortran 90 and High Performance Fortran (HPF). Many shared-memory architectures provide efficient native implementations of PVM message passing and will use HPF compilers. They will be covered by these routes.
The Parkbench committee agreed to divide its work between five subcommittees, corresponding to the five substantive chapters in the report, each with a leader who is responsible for assembling the contents of his chapter and its benchmarks for the committee's approval.
The subcommittees and their leaders are:
Recent practice, however, has been to send all mail to email@example.com so that all members may see it.
All mail is being collected and can be retrieved by sending email to firstname.lastname@example.org and in the message typing:
send comm.archive from pbwg send lowlevel.archive from pbwg send compactapp.archive from pbwg send method.archive from pbwg send kernel.archive from pbwg send index from pbwg
A mail reflector was set up for correspondence called email@example.com . Mail to that address will be sent to the mailing list and is also collected in firstname.lastname@example.org . To retrieve the collected mail, send email to email@example.com and in the mail message type: send comm.archive from pbwg.
The Parkbench committee is open without charge to anyone interested in computer benchmarking and operates similarly to the HPFF (High Performance Fortran Forum). Anyone interested in joining in the discussion or preparing benchmarks should send email to that effect to: firstname.lastname@example.org .
It is important to note that researchers in many scientific disciplines have found it necessary to establish and refine standards for performing experiments and reporting the results. Many scientists have learned the importance of standard terminology and notation. Chemists, physicists, and biologists long ago discovered the importance of "controls" in their experiments. Medical researchers have found it necessary to perform "double-blind" experiments in their field. Political scientists have found that subtle differences in the phrasing of a question can affect the results of a poll. In many fields, environmental factors in experiments can significantly influence the measured results. Thus, researchers must carefully report all such factors in their papers.
If supercomputer performance analysis and benchmarking is to be taken seriously as a scientific discipline, its practitioners should be expected to adhere to the kinds of standards that prevail in other disciplines. This effort is dedicated to promoting these standards in the field of high-performance computing.
Table of Contents