Volume 7, Issue 1 -
Spring/Summer 1999

Volume 6, Issue 3
Fall 1998

Volume 6, Issue 2
Spring/Summer 1998

Volume 6, Issue 1
Winter 1998

Volume 5, Issue 4
Fall 1997

Volume 5, Issue 3
Summer 1997

Volume 5, Issue 2
Spring 1997

Volume 5, Issue 1
Winter 1997

Volume 4, Issue 4
Fall 1996

Volume 4, Issue 3
Summer 1996

Volume 4, Issue 2
Spring 1996

Volume 4, Issue 1
Winter 1996

Volume 3, Issue 4
Fall 1995

Volume 3, Issue 3
Summer 1995

Volume 3, Issue 2
Spring 1995

Volume 3, Issue 1
January 1995

Volume 2, Issue 4
October 1994

Volume 2, Issue 3
July 1994

Volume 2, Issue 2
April 1994

Volume 2, Issue 1
January 1994

Volume 1, Issue 4
October 1993

Volume 1, Issue 3
July 1993

Volume 1, Issue 2
April 1993

Volume 1, Issue 1
January 1993

Research Focus: The Automatic Differentiation Project

Jason Abate, Chris Bischof, Peyvand Khademi, Lucas Roh, Po-Ting Wu, Argonne National Laboratory; Gordon Pusch, Michigan State University; Alan Carle, Mike Fagan, Rice University; Andrew Mauer, University of Illinois at Urbana-Champaign

Massively parallel architectures have provided the computing power needed for the sensitivity analysis and subsequent design optimization and control of complex systems in physics, engineering, biology, and even economics. Automatic differentiation provides a foundation for those efforts by reliably and efficiently computing derivatives of large computer codes, and by providing significant algorithmic speedups, compared to current approaches.

Once a code modeling a complex system has been implemented, sensitivity analysis is normally used to analyze the behavior of the model in view of theoretical insights and expectations. Once the computer model has been validated, it is often used for design optimization; that is, analysis codes are embedded in a numerical optimization framework to determine the input parameters for which the model exhibits a certain desired behavior. In this manner, an analysis code is turned into a sophisticated design tool for engineering. The impact of this emerging discipline for U.S. industry cannot be underestimated.

The Automatic Differentiation (AD) project in the CRPC came into existence in 1991, after Chris Bischof of Argonne National Laboratory visited the Fortran Parallel Programming Project at Rice University with the encouragement of John Dennis at Rice. Bischof presented an overview of AD and discussed the potential impact of compiler technology on the field. The visit led to the involvement of Alan Carle. Carle brought to the AD project in-depth knowledge of the advanced Fortran analysis and transformation tools that had been developed at Rice. Building on Carle's experience with the ParaScope system and the experience of the Argonne group, which at that time included Andreas Griewank as well as George Corliss and Paul Hovland, a prototype of the ADIFOR system became operational within three months of the start of the collaboration. This tool subsequently became the first AD tool for Fortran that successfully processed 30,000 line codes.

From the beginning, Bischof and Carle believed that even though automatic differentiation was not a new area, its impact on practitioners had been minimal because of the lack of both robust and efficient tools and educational outreach by the AD community. Hence, in recent years, the AD group expended considerable effort collaborating with engineers and scientists who could profit from this new technology. These collaborations provided credibility for AD and led to invaluable insight into the needs of potential AD users.

The recently released ADIFOR 2.0 system represents the culmination of the efforts of the AD group to develop a robust, near-commercial-quality AD system for Fortran 77. The ADIFOR 2.0 system consists of the ADIFOR 2.0 preprocessor, the ADIntrinsics exception handler system, and the SparsLinC sparse linear combination library. The preprocessor is responsible for analyzing how floating point values flow through a Fortran 77 program and augmenting the program with statements to compute derivatives of the floating point values. The ADIntrinics system provides configurable handling of exceptions that occur when intrinsic functions are differentiated at singularities. SparsLinC provides the ADIFOR 2.0 system with the ability to take advantage of the sparsity that occurs in many derivative computations. Carle based the implementation of the ADIFOR preprocessor on the ParaScope/D System compiler technology developed at Rice, while Bischof, Mauer, and Khademi at Argonne focused on the implementation of the AD support tools and libraries.

ADIFOR 2.0 is now available via the World Wide Web and has been requested by more than 80 people. The ADIFOR 2.0 User's Guide will be published by SIAM in 1996. As mentioned in the summer 1995 issue of Parallel Computing Research, Bischof and Carle were awarded the 1995 Wilkinson Prize for Numerical Software for ADIFOR 2.0 at ICIAM 95 in Hamburg, Germany. This recognition is proof that AD is changing from a cottage industry to a recognized technology largely because of AD project efforts.

According to Bischof and Carle, much still remains to be done. AD functionality needs to be provided for languages other than Fortran 77, and the freedom inherent in the derivative chain rule needs to be better exploited.

In 1994, Bischof and Mauer began the implementation of ADIC, an automatic differentiation system for ANSI-C. The ADIC prototype is based on Indiana University computer science professor Dennis Gannon's Sage++ toolsuite, and, from a user perspective, is similar to and interoperable with ADIFOR.

A re-implementation of ADIC, as well as a restructuring of ADIFOR, is currently being undertaken. The goal is to unify the ADIFOR and ADIC efforts and to create a testbed for further research in AD algorithms and AD tools for Fortran 90 and C++. The new system is being designed with the following goals in mind:

  • Integrate parsers, unparsers, data flow analysis, and source code transformers for different languages around a common core language describing the floating-point computations relevant to automatic differentiation, and
  • Experiment with novel AD algorithms using the AD subset language abstraction.

The proposed architecture is similar to the underlying architecture of the ParaScope and D System advanced program transformation systems developed at Rice. It consists of components to analyze and transform source modules under the direction of an analysis and transformation program making extensive use of interprocedural program analysis and user interaction. This architecture maximizes leverage of programming tools developed by other researchers, while minimizing the time required to integrate AD advances into existing tools.

Most importantly, the new architecture will support efforts of members of the AD project to explore nontraditional approaches to exploiting chain rule associativity, exploit derivative-enabled parallelism, develop efficient adjoints, and capitalize on tool development on the encapsulation and operator overloading capabilities of languages such as Fortran 90 and C++. This is paving the way for significant algorithmic speedups in the cost of computing derivatives.

For more information, see the World Wide Web URL

Rice University CRPC research scientist Alan Carle describes ADIFOR, a software tool he co-developed with Chris Bischof of Argonne National Laboratory and other collaborators.

Table of Contents

News | From the Director | Parallel Profile | Research Focus |
Work in Progress | Education Resources | Resources | Calendar | CRPC Home