A Data Blade Architecture for Generalized Out-of-Core Reduction Operations1997 CRPC Annual Meeting Poster Session
We will describe Titan, a prototype database engine that is able to carry out generalized reductions on large spatial or scientific multi-dimensional data sets. Titan is designed to efficiently carry out generalized reductions on large data sets resident on parallel architectures that have been configured to support high I/O rates. Titan can be used to generate composite images from low-level satellite sensor data, support data exploration and processing of large quantities of data from high power light or electron microscopy, and support analysis of output data from scientific calculations. We will also show how the resulting data sets can be efficiently transmitted to other (sequential or parallel) programs for further processing, using the Meta-Chaos library developed at Maryland. In particular, we will show how Titan, running on an IBM SP2, can be used to produce data sets for an HPF image classification program running on a farm of Digital Alpha SMPs.
Our database engine will be accessible as a stand-alone tool (Titan) designed to carry out optimized operations on data repositories, as a Data Blade module associated with a relational database, and as compiler or user-level runtime support. When it is used for compiler or user-level runtime support, the engine can carry out generalized reductions associated with out-of-core scientific applications.
The techniques employed involve the generation and use of optimized schedules - preprocessing of data access patterns is used to schedule processing, communication, local I/O and non-local I/O. The engine makes use of combined clustering/declustering methods we have developed for embedding data sets into parallel disk farms, in ways that optimize the performance of multi-dimensional range queries.
Sites & Affiliations | Leadership | Research & Applications | Major Accomplishments | FAQ | Search | Knowledge & Technology Transfer | Calendar of Events | Education & Outreach | Media Resources | Technical Reports & Publications | Parallel Computing Research Quarterly Newsletter | News Archives | Contact Information