For Data Preparers

Preparing data for submission to the PDS includes not only labelling of data files and collection of documentation, but also content verification and usually at least some measure of reformatting to get the data into one of the standard deep archive formats required by the PDS. The formatting and documentation requirements of the PDS were developed in conjunction with astronomers to ensure that the data archived will be accessible long past the lifetime of the mission, its collaborators, or its hardware/software installation. The PDS format is specifically designed to be an archive format, not a working data format. Listed on this page are pointers to information and software tools to assist you in preparing and verifying data for submission to the PDS.

• Introductions
• A Word About Data Formats
• PDS Standards Documents
• PDS Tools
• SBN Tools, Stylesheet and TARGET_NAME Conventions
• Before Submitting Data

Introductions

The full PDS archiving system and standards can seem very imposing when first encountered. Several more user-friendly introductions have been prepared to help new users get up to speed quickly. All of these introductions are available from the Engineering Node of the PDS.

• Quick-Start Introduction to PDS Archiving
• Archive Preparation Guide
• How to Submit Data to PDS
• Data Services provided by PDS

A Word About Data Formats

It has been our experience that simple data formats have several major advantages over more complex or proprietary formats:

1. They are immediately accessible to a much wider section of the science community.
2. They are stable over decades, maintaining accessibility for the projected life of the archive.
3. They are easier to document and verify, facilitating efficient review and archving.

Therefore, the Small Bodies Node is a strong advocate of simple data structures, especially in high-level mission data products. The simple data structures are homogeneous arrays and flat (ASCII or binary) tables. Not surprisingly, these are also the formats well supported by the Flexible Image Transport System (FITS) standard, which provides a very general and widely accessible format for exchanging file between various platforms.

Note that, from an archiving point of view, headers (and trailers) that are appended to the data structures present no major problems, as long as they do not interrupt the byte stream which constitutes the observational data. Thus any compliant FITS file can be easily prepared for PDS archiving without modifying the original FITS file because the headers occur before the individual data structures.

PDS Standards Documents

In addition to the printed documents available from the Engineering Node of the PDS, there are on-line versions accessible by Web browser:

Data Preparation Workbook

This workbook provides a quick overview of the PDS system and what constitutes an archive dataset, then walks you through the process of designing and creating and archive.

• HTML version

Archive Preparation Guide

This workbook provides a cook-book approach to preparing data for submission to the PDS. constitutes

• HTML version

Proposer's Archive Guide

This workbook describes the information needed to prepare a proposal. This document includes examples and templates.

• HTML version

PDS Standards Reference

This reference contains the detailed PDS standards for things like archive layout, data label requirements, units of measure, time values and other aspects of the PDS archives.

• PDF versions

Plantery Science Data Dictionary (PSDD)

The PSDD contains the definitions for all keywords and data structures used in and available for archive datasets. A flat ASCII version of the PSDD is required to run some of the software tools; there is a Web interface into the dictionary which allows you to retrieve specific keyword definitions.

• Web version
• PDF version
• Flat ASCII version, required by some of the software tools (and you will probably also need this index file as well)

PDS Tools

A number of tools have been developed for generating and verifying PDS labels. The complete inventory of general PDS utilities is available from Engineering Node. The tools our data preparers have found most useful are:

• lvtool: A label syntax verifier which checks the contents of the PDS label against the keyword definitions in your local copy of the data dictionary

• tbtool: A table browser tool that provides very simple display capabilities for ASCII or binary TABLE objects. tbtool is particularly useful for checking to make sure the formatting keywords correctly describe the physical layout of the data. If not, the results displayed by tbtool will be obviously skewed or badly formatted.

SBN Tools

The SBN has also developed suites of small utilities for working with the the types of data we most frequently encounter, and to produce formats that are preferred by the SBN (which may be more restrictive in some respects than the general PDS requirements). The majority of these routines are written in Perl; some are written in ANSI-standard C. All were developed and tested on a Sun Solaris work station. The source code for these routines is available from the SBN, as is, from our software archives. Utilities available include:

• IDL procedures for reading PDS-labelled data
• A tool to assist in converting FITS headers to PDS labels
• Utilities for formatting and verifying PDS labels
• Text-handling utilities for working with FITS headers
• General utilities for formatting and manipulating ASCII files

If you need some specific tool for preparation of your archive and could not find it among our tools, please, contact Anne Raugh (email to raugh at astro.umd.edu). It may be that we already have developed the tool you need, but it is not online yet. We also can develop a special tool upon your request.

Stylesheet

The SBN Style Sheet for PDS labels and ODL files provides notes, reminders and examples regarding elements of style expected in archive product labels and catalog files submitting to the SBN for archiving.

TARGET_NAME Conventions

Before Submitting Data

Before submitting your data to the SBN, please run the following verifications:

• Run the PDS lvtool utility, available from the Engineering Node software inventory, on all labels and catalog files and correct all syntactic and semantic errors reported. Warnings about standard values and keywords which are not yet in the data dictionary but which are scheduled to be added may be ignored.

• Run the SBN PDS text verifier, available below, on all ASCII files being delivered. ASCII files include PDS label and catalog files, ASCII data files and ASCII document files. All invalid characters reported must be removed. All ASCII text files must have fixed-length, CR/LF delimited records unless a previous arrangement has been made with the SBN.

  checkpdstext	SBN ASCII text verifier	C code	Man page

In addition, if you are submitting data in tabular form (TABLE, SPECTRUM or TIME_SERIES objects), you should also run the following utilities:

• tbtool, available from the PDS Engineering Node software inventory. This tool uses the information in the label to display the values in the table columns. Any misalignment in the table definition is immediately visible in the display. This tool can be run on both ASCII and binary table files.

• SBN TABLE Verifier, available below from the SBN. The table verifier checks for more subtle errors in the label and then uses the information in the label to read and verify the contents of the data file. The table verifier is written in C and provided as a ZIP file containing source code, headers and a make file.

  pdstv	PDS TABLE verifier	ZIP file	Man page

In all cases, if some of the labels were generated by the same label-writing program and were not subject to individual editing, then it is sufficient to run the verifiers on a representative sample. All files in the series must be corrected, of course, prior to submission.