PAIS STANDARD - ESA Conference Bureau

CCSDS « Data Archive Ingestion » standards
(Consultative Committee for Space Data Systems)
PAIS STANDARD
PRODUCER ARCHIVE INTERFACE SPECIFICATION
THE NEW CCSDS STANDARD FOR MODELING THE DATA TO BE
TRANSFERRED TO, AND VALIDATED BY, AN ARCHIVE
Danièle BOUCON, John GARRETT, Don SAWYER, David GIARETTA
PV 2013
5 November
1
Boucon Danièle, CCSDS PAIS standard
5/11/2013
MAIN POINTS


PAIS content
Formal specification (Data, SIP)
SIP creation, transfer and validation
 Applications

2
Context and overview
Conclusion
Boucon Danièle, CCSDS PAIS standard
5/11/2013
CONTEXT
CCSDS standardization
From OAIS
OAIS
(Reference Model for an Open Archival Information System) CCSDS 650.0-M-2, ISO 14721:2012
PAIMAS
(Producer Archive Interface Methodology Abstract Standard) CCSDS 651.0-M-1, ISO 20652:2006
PAIS
(Producer Archive Interface Specification + tutorial and yellow book) future CCSDS 651.1-B-1, CCSDS
651.2-G-1, CCSDS 651.3-Y-1 for end 2013
Data packaging
XFDU
(XML Formatted Data Unit) CCSDS 661.0-B-1, ISO 13527:2010
all CCSDS standards (also data structure description…) available at www.ccsds.org
3
Boucon Danièle, CCSDS DAI standards
5/11/2013
CONTEXT

International cooperation, diversity of
information contents and formats
Producers with geographical dispersion
organizational structures for archiving
relationships between data Producers and archives
rarely simple and easy
archived information quality and cost
4
Boucon Danièle, CCSDS DAI standards
5/11/2013
CONTEXT
Origin
PAIMAS methodological standard
4 phases :
Preliminary, Formal Definition, Transfer, Validation
For each phase: aims, actions, expected results
Submission agreement between Producer and Archive
Need
implementation standard PAIS
Boucon Danièle, CCSDS DAI standards
5/11/2013
OVERVIEW : LINK WITH PAIMAS
Phase objectives
PAIS phases
Transferred
object files
Preliminary
Phase
Formal Definition
Phase
Preliminary
Agreement
Producer Archive
6
Boucon Danièle, CCSDS DAI standards
Transfer
Phase
Validation
Phase
Dictionary
Anomalies
Formal model
Submission Agreement
Validation
agreement
Data ready to archive
OVERVIEW : LINK WITH OAIS
PAIS
=
Interface between
Producer
and
OAIS ingest
Producer
7
Boucon Danièle, CCSDS DAI standards
Archive
5/11/2013
OVERVIEW LINK WITH OAIS
OAIS functional model
AIP : Archival Information Package,
SIP : Submission Information Package,
DIP : Sissemination Information package
8
Boucon Danièle, CCSDS DAI standards
5/11/2013
OVERVIEW : LINK WITH OAIS
receiving SIPs, performing quality assurance on SIPs
Ingest
AIP : Archival Information Package,
SIP : Submission Information Package,
DIP : Sissemination Information package
9
Boucon Danièle, CCSDS DAI standards
5/11/2013
OVERVIEW
PAIS phases Data management
PAIS phases and data management process
10
Preservation
Production
long term
archive
mission
archive
Production
preparation
Preservation preparation
Formal Definition
Transfer and validation
Submission agreement
Boucon Danièle, CCSDS DAI standards
5/11/2013
OVERVIEW
For who?
Data Producers
+
Archives
When?
 Very
early, from the first contact
 Along the Data production and transfer period
Boucon Danièle, CCSDS DAI standards
5/11/2013
OVERVIEW
Why?

Formal definition of the objects to be transferred
(Model of Objects for Transfer “MOT”)
Formal definition of the packaging of these objects
(content of SIPs, sequencing constraints
between SIPs, SIP containers)

Creation and effective transfer and validation of SIPs
Boucon Danièle, CCSDS DAI standards
5/11/2013
MAIN POINTS


PAIS content
Formal specification(Data, SIP)
SIP creation, transfer and validation
 Applications

13
Context and overview
Conclusion
Boucon Danièle, CCSDS PAIS standard
5/11/2013
FORMAL SPECIFICATION (DATA)
MOT (Model of Objects for Transfer)
 Formal
view of Data Objects to be transferred,
Collections, and relationships between them

Data Objects -> «Transfer Objects » data structures
More or less complex
Mandatory (IDs, occurrence, …), optional (format, size, …),
possibly user-defined attributes

MOT = {Collections and Transfer Objects Descriptors}
2 Standardized Descriptor Models
Transfer Object and Collection
14
Boucon Danièle, CCSDS DAI standards
XML schemas
CCSD0014 and CCSD0015
5/11/2013
FORMAL SPECIFICATION (DATA)
High level structure of a Transfer Object Descriptor
15
Boucon Danièle, CCSDS DAI standards
FORMAL SPECIFICATION (DATA)
MOT: POLDER example
16
Boucon Danièle, CCSDS DAI standards
5/11/2013
FORMAL SPECIFICATION (SIP)
SIP constraints specification
Authorized content of each SIP (SIP Content Type)
 Order in which SIPs have to be delivered

Standardized SIP constraints Model
XML schema CCSD0016
Example
(SIP constraints extract)
17
Boucon Danièle, CCSDS DAI standards
5/11/2013
FORMAL SPECIFICATION (SIP)
SIP Model specification
SIPs = Containers
(1 or more Transfer Objects)

Including replacement, deletion
last Transfer Object
Standardized SIP Model
18
Boucon Danièle, CCSDS DAI standards
CCSD0017
5/11/2013
PAIS CONTENT
Digital Objects
Descriptor
Specification
Descriptors/MOT
SIP
SIP Content Type
Specification
SIP Transfer
SIP
Creation
SIP Sequencing
Constraints
Specification
Transfer Phase
Submission Agreement
Producer and Archive Responsibility
Producer Responsibility
19
Boucon Danièle, CCSDS DAI standards
SIP
Reception
SIP
validation
Formal Definition Phase
Archive Responsibility
SIP
Validation phase
Tiime
SIP CREATION, TRANSFER AND VALIDATION
SIP =
Zip file package
• Data Objects
• XML manifest
(links with the MOT,
information on Data
Objects)
Implementation using XFDU (SIP Model mapped
with XFDU structure)
20
Boucon Danièle, CCSDS DAI standards
5/11/2013
SIP CREATION, TRANSFER AND VALIDATION
Example
Containers
Links with Formal Spec (IDs)
Data Object
21
Boucon Danièle, CCSDS DAI standards
5/11/2013
21
SIP VALIDATION

minimum
 conformity to the formal specifications
-> MOT (IDs, occurrences), SIP content Types, SIP sequencing
constraints
 integrity

higher level of validation (data content validation, …):
case by case
22
Boucon Danièle, CCSDS DAI standards
5/11/2013
MAIN POINTS


Description
Formal specification(Data, SIP)
SIP creation, transfer and validation
 Applications

23
Context and overview
Conclusion
Boucon Danièle, CCSDS PAIS standard
5/11/2013
APPLICATIONS

2 prototypes: required and developed
Standard coverage and interoperability tests
NASA/ESA (SIP builder)
CNES (MOT design, transfer and validation follow up)
 PAIS
 Test
tutorial (under development)
cases
NASA/ISEE, NASA/PDS, ESA/SAFE, CNES/COROT …
 PAIS
implementation at CNES ingest archival system
level (SIPAD-NG)
24
Boucon Danièle, CCSDS DAI standards
5/11/2013
CONCLUSION

Flexible packaging specification to model Producer’s data
for archiving
Catalog, validate, store SIPs

⇒
Improve quality of archive holding
⇒ Reduce errors and cost


25
CCSDS standard end 2013, then ISO standard 2014
(before ask daniele.boucon@cnes.fr)
Tutorial should be available at the same period
Boucon Danièle, CCSDS DAI standards
5/11/2013
26