EDAMIS Eurostat, Statistical Information Technologies, Unit B5 1.0 Draft

How to set up and use the validation in EDAMIS
Project
Author
Release
Status
Reference
January 2011
EDAMIS
Eurostat, Statistical Information Technologies, Unit B5
1.0
Draft
QUG-05-2011-02-07 [QUG] (How to set up and use the validation in
EDAMIS) D.doc
Page 1
TABLE OF CONTENTS
1
INTRODUCTION ......................................................................................................................................... 5
1.1
2
BACKGROUND INFORMATION .................................................................................................................... 5
GETTING STARTED .................................................................................................................................. 7
2.1
2.2
3
WHAT VALIDATIONS ARE .......................................................................................................................... 7
GENERAL PREREQUISITES .......................................................................................................................... 7
CREATING VALIDATIONS ...................................................................................................................... 8
3.1 ACCESSING THE VALIDATION MANAGEMENT FORM ................................................................................... 8
3.2 LINKING A DSD TO A DATASET ................................................................................................................. 9
3.3 CREATING A VALIDATION RULES SET....................................................................................................... 10
3.4 CREATING A VALIDATION RULE ............................................................................................................... 12
3.4.1
Defining the scope of a validation rule ......................................................................................... 14
3.4.2
Defining a validation rule ............................................................................................................. 15
3.4.2.1
Normal rules ........................................................................................................................ 16
3.4.2.2
Advanced rules .................................................................................................................... 17
4
TESTING A VALIDATION RULES SET ............................................................................................... 20
4.1
5
ANALYZING A VALIDATION REPORT ........................................................................................................ 21
PUBLISHING VALIDATIONS................................................................................................................. 22
5.1
6
VALIDATION MECHANISM........................................................................................................................ 22
MANAGING VALIDATIONS .................................................................................................................. 24
6.1
6.2
6.3
7
LOCKING A VALIDATION .......................................................................................................................... 24
UNLOCKING A VALIDATION ..................................................................................................................... 25
DELETING A VALIDATION ........................................................................................................................ 26
MANAGING VALIDATION RULES SETS ............................................................................................ 28
7.1
7.2
8
COPYING A VALIDATION RULES SET......................................................................................................... 28
DELETING A VALIDATION RULES SET ....................................................................................................... 28
MANAGING VALIDATION RULES ...................................................................................................... 30
8.1
8.2
9
EDITING A VALIDATION RULE .................................................................................................................. 30
REMOVING A VALIDATION RULE .............................................................................................................. 30
VALIDATING DATA FILES .................................................................................................................... 31
9.1 SENDING DATA FILES THROUGH EWP ..................................................................................................... 31
9.1.1
Sending data files containing critical errors ................................................................................. 32
9.1.2
Sending data files containing basic errors .................................................................................... 33
9.1.3
Sending correct data files .............................................................................................................. 34
9.1.4
Validation reports availability ...................................................................................................... 35
9.2 SENDING DATA FILES THROUGH EWA ..................................................................................................... 35
10
ANNEX .................................................................................................................................................. 37
11
GLOSSARY .......................................................................................................................................... 48
12
HISTORY OF CHANGES ................................................................................................................... 49
January 2011
Page 2
TABLE OF FIGURES
Figure 1: Manage Validation menu entry ............................................................................................................... 8
Figure 2: Manage Validation form.......................................................................................................................... 8
Figure 3: Manage Validation form with no existing validation .............................................................................. 9
Figure 4: Error message when willing to associate a DSD ..................................................................................... 9
Figure 5: Manage Validation - Associate a DSD form ........................................................................................... 9
Figure 6: Warning message when associating a DSD ........................................................................................... 10
Figure 7: Error message when associating a DSD ................................................................................................ 10
Figure 8: Manage Validation form with the newly created validation .................................................................. 10
Figure 9: Empty Manage Validation – Data Structure Definition form ............................................................... 11
Figure 10: Manage Validation – Add a rule set to the DSD form ......................................................................... 12
Figure 11: Manage Validation – DSD form with one validation rules set ............................................................ 12
Figure 12: Validation rules editor ......................................................................................................................... 13
Figure 13: Excerpt of the DSD used throughout the manual ................................................................................ 14
Figure 14: Validation rules editor – Specifying a dimension by label .................................................................. 15
Figure 15: Validation rules editor – Specifying a dimension by code .................................................................. 15
Figure 16: Normal validation rule popup window ................................................................................................ 16
Figure 17: Advanced validation rule popup window ............................................................................................ 18
Figure 18: Referencing another observation value................................................................................................ 19
Figure 19: Manage Validation – Test rules form .................................................................................................. 20
Figure 20: Validation report after testing the validation process .......................................................................... 20
Figure 21: Manage Validation form with the newly published validation ............................................................ 22
Figure 22: Confirmation message when clicking on a validation entry ................................................................ 24
Figure 23: Confirmation message when clicking on a published validation entry ................................................ 24
Figure 24: Error message when locking a validation ............................................................................................ 25
Figure 25: Error message when locking an already locked validation .................................................................. 25
Figure 26: Validation listed as locked ................................................................................................................... 25
Figure 27: Error message when unlocking a validation ........................................................................................ 26
Figure 28: Error message when locking an already locked validation .................................................................. 26
Figure 29: Validation listed as not locked ............................................................................................................. 26
Figure 30: Confirmation message when deleting validations ............................................................................... 27
Figure 31: Error message when deleting validations ............................................................................................ 27
Figure 32: Confirmation message when deleting validations ............................................................................... 27
Figure 33: Manage Validation – Add a rule set to the DSD form ......................................................................... 28
Figure 34: Confirmation message when deleting validation rules sets ................................................................. 28
Figure 35: Confirmation message when deleting validation rules sets ................................................................. 29
Figure 36: Send Datafile menu entry .................................................................................................................... 31
Figure 37: Send Datafile form when selecting a data file to be validated ............................................................. 32
Figure 38: Uploading a data file containing a (critical) error (Advanced upload form) ....................................... 32
Figure 39: Uploading a data file containing a (critical) error (Simple upload form) ............................................ 32
Figure 40: Send Datafile form after data files upload ........................................................................................... 33
Figure 41: Validation report with a (critical) error................................................................................................ 33
Figure 42: Uploading a data file containing a warning (basic error) (Advanced upload form) ............................ 33
Figure 43: Uploading a data file containing a warning (basic error) (Simple upload form) ................................. 33
Figure 44: Validation report with a warning (basic error) .................................................................................... 34
Figure 45: Uploading a data file with no error (Advanced upload form) .............................................................. 34
Figure 46: Uploading a data file with no error (Simple upload form) .................................................................. 34
Figure 47: Validation report with no error ............................................................................................................ 35
Figure 48: EWA - Send Datafile menu entry ........................................................................................................ 35
Figure 49: EWA - Send Datafiles form when selecting a data file to be validated ............................................... 35
Figure 50: EWA - Send Datafiles form after data files upload ............................................................................. 36
January 2011
Page 3
TABLE OF TABLES
Table 1: Manage Validation form – Selection area .............................................................................................. 37
Table 2: Manage Validation form – Validations description area ........................................................................ 38
Table 3: Manage Validation form – Action buttons ............................................................................................. 38
Table 4: Manage Validation - Associate a DSD form – Definition area ............................................................... 39
Table 5: Manage Validation - Associate a DSD form – Action buttons ............................................................... 39
Table 6: Manage Validation – DSD Form form – Summary area ........................................................................ 40
Table 7: Manage Validation – DSD Form form – Validation rules sets description area ..................................... 40
Table 8: Manage Validation – DSD Form form – Action buttons ........................................................................ 41
Table 9: Manage Validation - Add a rules set to the DSD form – Definition area ............................................... 42
Table 10: Manage Validation - Add a rules set to the DSD form – Action buttons .............................................. 42
Table 11: Validation rules editor – Selection area ................................................................................................ 42
Table 12: Validation rules editor – Validation rules description area .................................................................. 43
Table 13: Validation rules editor – Action buttons ............................................................................................... 44
Table 14: Validation rules editor – Normal Rule popup window ......................................................................... 45
Table 15: Validation rules editor – Advanced Rule popup window...................................................................... 46
Table 16: Manage Validation - Create a copy of the rules set form – Definition area ......................................... 46
Table 17: Manage Validation - Create a copy of the rules set form – Action buttons.......................................... 47
Table 18: Manage Validation - Test rules form – Definition area ........................................................................ 47
Table 19: Manage Validation - Test rules form – Action buttons ........................................................................ 47
January 2011
Page 4
1
Introduction
This manual is intended for the domain managers (DMA) who are willing to use the EDAMIS
Validation Engine (EVE) to validate the data files that are transferred to Eurostat.
It contains everything a domain manager needs to know in order to establish validations.
It illustrates how to create validations and their components, how to maintain them and how
to publish them.
The validation process is also described in this manual.
This manual is being made available to Eurostat production units to assist the implementation
phase of the validations.
1.1 Background information
EDAMIS: the electronic Data file Administration and Management Information System is the
integrated environment for data transmission and for the Single Entry Point for data arriving
at Eurostat. It is visible through the following tools:
EWA: The EDAMIS Web Application is installed in the National Statistical Institutes and
used for sending statistical data files to Eurostat. Interactive, semi-automated and fullautomated transmissions are supported.
EWP: The EDAMIS Web Portal is installed at the European Commission and used for
managing the dataset inventory, managing the user rights related to the transmissions, sending
statistical data files to Eurostat (as an alternative to EWA) and monitoring the traffic through
its Management Information System.
EWF: The EDAMIS Web Form is a component of the EDAMIS Web Portal that can be used
for the transmission of low volumes of data. It offers all the basic functions of EWP and does
not require any installation on the PC of the user. EDAMIS Web Forms are typically of
interest for the collection of data in statistical domains where:
• data volumes are low
• currently collection of data takes place via spreadsheet based questionnaires
• a certain level of validation of the data before transmission is preferred
• software installation on client PCs is forbidden
• a data sharing agreement with international organisations exist
Local Coordinators: In each National Statistical Institute (NSI), a Local Coordinator (LC) is
the main contact point to Eurostat for the effective implementation of electronic data
transmission and a contact point for members of their own organisation.
Transmission Coordinators: In each National Statistical Institute (NSI), a Transmission
Coordinator (TCO) is responsible for the management of users' rights within EDAMIS. The
TCO is in charge of managing and giving rights to users inside its organisation. In NSIs the
TCO is usually the Local Coordinator, while in non NSIs, a TCO can be nominated. A TCO
January 2011
Page 5
can be allowed to coordinate users in several organisations of the country, provided that an
agreement exists between the organisations concerned.
Data sender and EDAMIS: In each National Statistical Institutes, the installed EWA allows
data provider to send data to Eurostat. If the data sender is located in another organisation,
then the EWP will be used instead. EDAMIS Web Forms, when they exist, can be obtained
via the EWP. In all cases, data sender should have a CIRCA user-id and should register to
EDAMIS. Further details for EDAMIS and CIRCA registration are available at the following
URL: http://circa.europa.eu/irc/dsis/EDAMIS/info/data/website/index.htm
January 2011
Page 6
2
Getting started
2.1 What validations are
The aim of the validation process is to verify the consistency of the data files sent via
EDAMIS, the Eurostat Single Entry Point (SEP) for data.
This encompasses both the structure of the files as well as the data themselves.
The validation process can take place on all EDAMIS transmission channels: EDAMIS Web
Portal (EWP) and EDAMIS Web Application (EWA).
Validations are specific to a dataset and are managed by a Eurostat statistical Domain
Manager (DMA). The validation engine is based on a world wide standard for exchanging
statistical data and metadata: SDMX-ML. For validation of data in other formats, i.e. CSV,
tools like the SDMX Converter can be used before uploading the data to EDAMIS.
The validation of the data file structure is based on a SDMX Data Structure Definitions
(DSD) file which is associated to the dataset for which the data file is sent.
The validation of the data is based on the validation rules defined by the DMA for the
aforementioned relationship.
2.2 General prerequisites
To be able to send data files to Eurostat, a domain and a dataset have to be defined in EWP.
The transmission mode of the dataset has to be set either to Send data or to All as the
validation process will not be triggered for Web Form entries.
Even if the validations can be managed while the dataset is not active, the dataset will have to
be activated in order for the users to be able to send data files.
To be able to manage validations, the user needs Domain Manager (DMA) or Domain CoResponsible (DCR) right in EDAMIS for the domain related to the dataset.
The
corresponding
DSD
must
exist
in
the
SDMX
Registry
(https://webgate.training.ec.europa.eu/sdmxregistry) for EWP (and later on the validation
engine) to retrieve it.
As the definition of the validations will be done through an applet, Java has to be installed on
the user’s PC.
January 2011
Page 7
3
Creating validations
The reason why the validation process relies on the usage of a DSD is because the purpose of
a DSD file is to describe how the related data have to be structured when they are transferred
to Eurostat for a given dataset.
As such, the very first step when creating a validation consists of linking a DSD to the dataset
for which the data files have to be validated.
Once the link is made, the structure of the transferred data files can already be validated
against the DSD providing that the validation is published (see paragraph ‘Publishing
validations’).
Furthermore, if custom data validation is required, the user can define one or several sets of
validation rules in a second step.
3.1 Accessing the validation management form
Once connected to the EDAMIS Web Portal, the user has to select the Inventory >> Manage
Validation menu entry (see Figure 1) to access the validation management form.
Figure 1: Manage Validation menu entry
In the appearing Manage Validation form (see Figure 2), the user has first to filter the theme,
domain and dataset for which he wants to create the validation and then click on the [View]
button (see Table 1).
Figure 2: Manage Validation form
Once refreshed (see Figure 3), the form lists the existing validations if any (see Table 2) and
allows the DMA to perform several actions (see Table 3).
January 2011
Page 8
Figure 3: Manage Validation form with no existing validation
3.2 Linking a DSD to a dataset
To link a DSD to the selected dataset, the DMA has to click on the [Associate a dsd] button.
If the filters have not been correctly set, the following error message will appear:
Figure 4: Error message when willing to associate a DSD
Otherwise, on the appearing form (see Figure 5), the DMA has to provide the ID, the version
and the agency of the DSD he wants to associate to the dataset (see Table 4).
A link allows him to connect to the SDMX Registry (using his CIRCA credentials) in order to
find the appropriate values.
Figure 5: Manage Validation - Associate a DSD form
Once the fields have been provided, the DMA has to click on the [Associate] button to confirm
his choice (see Table 5).
If the DSD is not found in the SDMX Registry, the following warning message will appear:
January 2011
Page 9
Figure 6: Warning message when associating a DSD
As a given DSD can be linked only once to a given dataset, the following error message will
appear if the DSD has been already associated to the current dataset:
Figure 7: Error message when associating a DSD
In both cases, the DMA has the possibility to modify the fields or to abort the association by
clicking on the [Cancel] button.
Otherwise, if the DSD can be successfully retrieved from the SDMX Registry, the validation is
then created and listed as locked by the DMA in the Manage Validation form (see Figure 8).
Other properties of the validation are also summarized (see Table 2).
Figure 8: Manage Validation form with the newly created validation
Now that the link is made, the structure of the transferred data files can already be validated
against the DSD providing that the validation is published (see paragraph ‘Publishing
validations’).
3.3 Creating a validation rules set
To be able to create validation rules, the DMA has to create beforehand a validation rules set
that will act as a container for the validation rules.
January 2011
Page 10
As a given DSD can be linked only once to a given dataset, the DMA can keep track of
different versions of the validation rules by creating several validation rules sets for a single
validation.
Nonetheless, if several validation rules sets have been created for a single validation, only the
latest one will be used by the validation engine during the data transmission once the
validation will be published (see paragraph ‘Publishing validations’).
To be able to create a validation rules set, the validation has to be locked beforehand (which is
already the case if the validation has just been created otherwise see paragraph ‘Locking a
validation’).
Once locked, the validation has to be opened by clicking on the validation entry (if the DMA
did not already do this to lock the validation).
The Manage Validation – Data Structure Definition form is then displayed (see Figure 9). It
summarizes the global properties of the validation (see Table 6), lists the existing validation
rules sets if any (see Table 7) and allows the DMA to perform several actions (see Table 8).
Furthermore, the DMA is able to provide a description for the validation in this form.
Figure 9: Empty Manage Validation – Data Structure Definition form
To create a validation rules set, the DMA has to click on the [Add Rules Set] button.
This will launch the Manage Validation – Add a rule set to the DSD form (see Figure 10)
where the DMA has to provide the ID and, optionally, the description of the validation rules set
(the version is automatically computed) (see Table 9).
January 2011
Page 11
Figure 10: Manage Validation – Add a rule set to the DSD form
Once done, the DMA has to click on the [Save] button (see Table 10): the new validation rules
set is created and is then listed as follows:
Figure 11: Manage Validation – DSD form with one validation rules set
If required, the validation rules set ID and description can be updated directly from within this
form.
3.4 Creating a validation rule
Now that a validation rules set has been created, the validation rules can be finally defined.
If several validation rules sets exist, the DMA has to first select the desired one and then click
on the [Edit Rules Set] button. Otherwise, he has to directly click on this button.
The validation rules editor (which is actually a Java applet) then appears as follows, allowing
the DMA to manage the validation rules:
January 2011
Page 12
Figure 12: Validation rules editor
The upper part of the form allows the DMA to select the dimensions of the observation
value(s) for which he wants to create a validation rule (see Table 11) while the bottom part lists
all the currently defined validations rules (see Table 12) and allows the DMA to perform
several actions (see Table 13).
The aforementioned observation values dimensions are related to the code lists defined in the
DSD relative to the transferred data files.
As depicted in the following excerpt of the VALIDATION_SPR_DSD DSD file used
throughout this manual (see Figure 13), the dimensions DVSN1, DVSN2 and DVSN3 are
respectively
linked
to
code
lists
VALIDATION_SPR_NOMENCLATURE_1,
VALIDATION_SPR_NOMENCLATURE_2 and VALIDATION_SPR_NOMENCLATURE_3.
January 2011
Page 13
Figure 13: Excerpt of the DSD used throughout the manual
3.4.1 Defining the scope of a validation rule
A validation rule can have different scopes:
January 2011
Page 14
•
•
•
it can be specific to a dedicated observation value
it can target a range of observation values
it can target every observation values
To target a specific observation value, the DMA has to provide all the dimensions of the
observation value.
To target a range of observation values, the DMA has to provide the dimensions that are
common to the observation values, leaving the other dimensions blank.
To target every observation values, the DMA has to leave all the dimensions blank.
To specify a dimension, the DMA has to select a value from the dropdown list under the
corresponding dimension as follow:
Figure 14: Validation rules editor – Specifying a dimension by label
To display the codes (defined in the relative code lists) instead of the labels inside the
dropdown, the DMA can click on the [Codes] button (see Table 11). The dropdown will then
appear as follow:
Figure 15: Validation rules editor – Specifying a dimension by code
3.4.2 Defining a validation rule
Two kinds of validation rules can be created: normal rules and advanced ones.
January 2011
Page 15
Normal rules are validation rules that do not refer to other observation values. As such, they
are also known as intra-record rules.
Furthermore, they are applicable to a specific observation value, to a range of observation
values or even to every observation value.
On the contrary, advanced rules are validation rules that can refer to one or several other
observation values. As such, they are also known as inter-record rules.
Additionally, they are applicable to a specific observation value only.
3.4.2.1 Normal rules
To create a normal rule, the DMA first has to specify the dimensions of the targeted
observation value(s).
Once done, he has to click on the [Add] button which will display the following dedicated
popup window:
Figure 16: Normal validation rule popup window
From within this popup window, the DMA can:
•
•
•
•
•
•
•
•
activate or deactivate the validation rule
specify to which country linked to the dataset the validation rule is dedicated (either
all countries or a specific one)
specify if the observation value has to be mandatory and, if so, the relative error
severity level (see below)
specify if the observation value has to be inside a given range and, if so, the relative
error severity level
specify if the observation value has to fulfil a given expression (see below) and, if so,
the relative error severity level
specify the error message that will appear in the validation report if the observation
value does not fulfil the above mentioned expression
specify if the validation rule has to be applied only for observation values whose
reference period is greater or equal to a given “valid from” period
specify if the validation rule has to be applied only for observation values whose
reference period is lower or equal to a given “valid to” period
January 2011
Page 16
For the three kinds of validation mentioned above, a specific error severity level can be
defined: either Basic or Critical.
During the validation process, if an observation value does not fulfil a validation defined as
Basic, then a corresponding warning will be listed in the validation report but the data file
transfer will not be blocked because of this warning.
In the other hand, if an observation value does not fulfil a validation defined as Critical, then
a corresponding error will be listed in the validation report and the data file transfer will be
blocked because of this error.
To define a validation based on an expression, the following keywords can be used:
Keyword
ABS
SQRT
NOT
OR
AND
THIS
*
+
/
<
<=
<>
=
>
>=
(
)
Description
Absolute value
Square root
Logical not
Logical or
Logical and
Reference to the current observation value
Multiplication
Addition
Subtraction
Division
Lower than
Lower or equal to
Not equal
Equal
Greater than
Greater or equal to
Opening bracket
Closing bracket
For instance, the following expression will check that the observation value that is being
validated is greater than 100 but is not equal to 150:
(THIS > 100) AND (THIS <> 150)
3.4.2.2 Advanced rules
To create an advanced rule, the DMA first has to specify each dimension of the targeted
observation value.
Once done, he has to click on the [Add Advanced Rule] button which will display the
following dedicated popup window:
January 2011
Page 17
Figure 17: Advanced validation rule popup window
As previously mentioned, the aim of advanced rules is to be able to compare a specific
observation value to one or several other observation values.
The only constraint is that the observation value(s) that is(are) referenced has(have) to have
only one dissimilar dimension to the ones of the observation value that is currently validated.
For instance, comparing
• Dim1=A, Dim2=B, Dim3=C to Dim1=X, Dim2=B, Dim3=C is allowed
• Dim1=A, Dim2=B, Dim3=C to Dim1=A, Dim2=Y, Dim3=C is allowed
• Dim1=A, Dim2=B, Dim3=C to Dim1=A, Dim2=B, Dim3=Z is allowed
while comparing
• Dim1=A, Dim2=B, Dim3=C to Dim1=X, Dim2=Y, Dim3=C is not allowed
• Dim1=A, Dim2=B, Dim3=C to Dim1=X, Dim2=B, Dim3=Z is not allowed
• Dim1=A, Dim2=B, Dim3=C to Dim1=A, Dim2=Y, Dim3=Z is not allowed
• Dim1=A, Dim2=B, Dim3=C to Dim1=X, Dim2=Y, Dim3=Z is not allowed
To simplify the expression, only the dissimilar dimension of the targeted observation value
has to be referenced when comparing observation values.
To do so, the syntax used for the reference is based on the grid displaying the dimensions:
•
•
it begins with a word (from A to Z, then AA to ZZ, etc…) which points to the column
displaying the dissimilar dimension
and it is followed by a number which points to the index of the code in the dropdown
list relative to the dimension
For instance, as depicted in Figure 18, C2 references the second code (VSN1-LB@CB) of the
dimension displayed in column C (DVSN1).
January 2011
Page 18
Figure 18: Referencing another observation value
As such, if the observation value defined by dimensions DVSN1=VSN1-LC@CC,
DVSN2=VSN2-L03@C03, DVSN3=VSN3-LCC@CCC has to be greater than the observation
value defined by dimensions DVSN1=VSN1-LB@CB, DVSN2=VSN2-L03@C03,
DVSN3=VSN3-LCC@CCC then the following expression has to be used:
THIS > C2
The popup depicted in Figure 18 is displayed when clicking on the [View Codes] button from
within the Advanced validation rule popup window (see Table 15).
Furthermore, the following keywords can also be used in addition to the ones available for
normal rules expressions:
Keyword
AVG
MAX
MIN
SUM
Description
Average value in list
Maximum value in list
Minimum value in list
Sum of values
For instance, the following expression will check that the observation value that is being
validated has to be equal to the maximum value of the three observation values for which the
code of the dimension displayed in column E varies from the second one to the fourth one:
THIS = MAX(E2:E4)
January 2011
Page 19
4
Testing a validation rules set
Before publishing a validation, the accuracy of the validation rules should be thoroughly tested.
By clicking on the [Test rules] button from within the Manage Validation – DSD Form form
for the required validation, the DMA can verify the validation rules of the latest validation
rules set against a data file of his own for a given country and period.
This has no impact on the validation of the transferred data files.
To do so, the Manage Validation – test rules form will appear as follows, allowing the DMA to
specify the country and the period of the data to validate from a given data file (see Table 18).
Figure 19: Manage Validation – Test rules form
Once the information has been provided, the DMA has to click on the [Perform] button to
launch the validation process (see Table 19).
When the validation process has finished, the validation report then appears on the screen has
follows:
Figure 20: Validation report after testing the validation process
January 2011
Page 20
The DMA can then either return to the Manage Validation – DSD Form form by clicking on
the [Go the manage validation form] button or return to the Manage Validation – Test rules
form by clicking on the [Go to the Test rules form] button.
4.1 Analyzing a validation report
The validation reports are divided in two parts:
•
the Summary part which outlines:
o the version of the validation engine that has been used
o the date of the validation process
o the start and end time of the validation process
o the dataset, country and reference period of the provided data
o the name of the data file
o the number of raised errors
o the number of raised warnings
•
the Validation Log part which outlines:
o the
key
family
defined
in
the
data
file
(e.g.
ESTAT:VALIDATION_SPR_DSD:6.0) indicating which DSD to use for the
validation process when several DSDs are published for the relevant dataset
(see paragraph ‘Publishing validations’)
o the DSD used for the validation process
o a description for each of the raised errors and warnings
January 2011
Page 21
5
Publishing validations
In order to activate the validation process for transferred data files, at least one validation has
to be published beforehand for the relative dataset.
To do so, the DMA has to click on the [Publish] button from within the Manage Validation
DSD Form form (see Table 8).
Once the validation has been published, the Manage Validation form is displayed and the
corresponding entry is marked as active as follows:
Figure 21: Manage Validation form with the newly published validation
5.1 Validation mechanism
Even though several DSDs can be published for a single dataset, data files will nonetheless be
validated against one and only one published validation.
If only one validation is published for a given dataset, the transferred data files will then be
validated against this validation even if the DSD referenced in the key family defined in the
data files does not correspond to the DSD relative to the published validation.
When several validations are used for a give dataset, then the validation engine looks at the
DSD referenced in the key family defined in the transferred data files.
The following cases may occur:
•
•
•
•
the key family is wrongly defined: an error occurs and the validation process is
aborted
the key family references a DSD that is not linked to the dataset: the validation
process is aborted as no default DSD was found to validate the data file
the key family references a DSD that is indeed linked to the dataset but for which the
relevant validation is not published: the validation process is also aborted as no default
DSD was found to validate the data file
the key family references a DSD that is indeed linked to the dataset and for which the
relevant validation is indeed published: the data file is validated against the validation
relative to the defined DSD
January 2011
Page 22
Of course, the validation process will not be launched if no validation is published for a given
dataset.
If no validation rules set is defined in the validation to be processed, or if no validation rule is
defined in the latest validation rules set, only the structure of the transferred data files will be
validated.
If several validation rules sets are defined in the validation to be processed, then the validation
rules of the latest validation rules set will be used by the validation engine.
The validation rules will be applied according to their (optional) validity period and to the
reference period of the observation values to validate:
•
•
if no validity period is defined then the validation rule will always be applied
if a validity period is defined then the validation rule will be applied to observation
values whose reference period fulfil the defined validity period
January 2011
Page 23
6
Managing validations
As previously seen, when clicking on a validation entry in the Manage Validation form, the
DMA can either edit the corresponding validation details or view them in read-only mode.
Additionally, the actions described afterwards are also available.
6.1 Locking a validation
To be able to manage validations, they need most of the time to be locked beforehand.
One or several validations can be locked at a time through the Manage Validation form (see
Figure 8).
To lock a single validation, the DMA has two possibilities:
•
either by selecting the corresponding checkbox and by clicking on the [Lock] button
•
or by directly clicking on the validation entry and by clicking on the [OK] button on
the following confirmation message:
Figure 22: Confirmation message when clicking on a validation entry
If the validation was previously published, the following confirmation message would
appear instead, forcing the validation to be also unpublished:
Figure 23: Confirmation message when clicking on a published validation
entry
To lock several validations at once, the DMA has to select the corresponding checkboxes and
to click on the [Lock] button.
If the DMA clicks on the [Lock] button while no validation was selected, the following error
message appears:
January 2011
Page 24
Figure 24: Error message when locking a validation
If the DMA tries to lock an already locked validation, the following error message appears:
Figure 25: Error message when locking an already locked validation
Otherwise, if the locking was successfully made, the validation then appears with a red lock
icon as follows:
Figure 26: Validation listed as locked
6.2 Unlocking a validation
Once the DMA has finished working on a validation, he has to unlock it in order for others
DMAs to be able to work on it.
One or several validations can be unlocked at a time through the Manage Validation form
(see Figure 8).
To unlock a validation, the DMA has to select the corresponding checkbox and to click on the
[Unlock] button.
If the DMA clicks on the [Unlock] button while no validation was selected, the following
error message appears:
January 2011
Page 25
Figure 27: Error message when unlocking a validation
If the DMA tries to unlock an already unlocked validation, the following error message
appears:
Figure 28: Error message when locking an already locked validation
Otherwise, if the unlocking was successfully made, the validation then appears without the
red lock icon anymore as follows:
Figure 29: Validation listed as not locked
6.3 Deleting a validation
The DMA can delete validations through the Manage Validation form (see Figure 8)
providing that they are locked and inactive.
To do so, he has to select one or several locked and inactive validations and to click on the
[Delete] button.
The following confirmation message then appears, allowing the DMA to confirm or abort the
deletion:
January 2011
Page 26
Figure 30: Confirmation message when deleting validations
Should the validations be not locked or/and still published, the following error message would
appear:
Figure 31: Error message when deleting validations
Otherwise, the validation(s) is(are) deleted and the following confirmation message listing the
deleted validation(s) appears:
Figure 32: Confirmation message when deleting validations
January 2011
Page 27
7
Managing validation rules sets
The following actions are available through the Manage Validation – DSD Form form.
7.1 Copying a validation rules set
To duplicate a set of validation rules, the DMA can copy a validation rules set through the
Manage Validation – DSD Form form (see Figure 11). This is nonetheless possible only if the
parent validation is locked and inactive.
To do so, he has either to select the validation rules set and to click on the [Copy Rules Set]
button if several validation rules sets exist or to directly click on the [Copy Rules Set] button.
This will launch the Manage Validation – Create a copy of the rules set form (see Figure 33)
where the DMA has to provide the ID and, optionally, the description of the validation rules set
(the version is automatically computed) (see Table 16).
Figure 33: Manage Validation – Add a rule set to the DSD form
Once done, the DMA has to click on the [Save] button (see Table 17): the new validation rules
set is created and is then listed as the latest entry in the Manage Validation –DSD Form form.
7.2 Deleting a validation rules set
The DMA can delete validation rules sets through the Manage Validation – DSD Form form
(see Figure 11) providing that the parent validation is locked and inactive.
To do so, he has either to select one or several validation rules sets and to click on the [Delete
Rules Set] button if several validation rules sets exist or to directly click on the [Delete Rules
Set] button.
The following confirmation message then appears, allowing the DMA to confirm or abort the
deletion:
Figure 34: Confirmation message when deleting validation rules sets
January 2011
Page 28
The validation(s) is(are) deleted and the following confirmation message listing the deleted
validation rules set(s) appears:
Figure 35: Confirmation message when deleting validation rules sets
January 2011
Page 29
8
Managing validation rules
The following actions are available through the validation rules editor (see Figure 12).
8.1 Editing a validation rule
If a validation rule needs to be modified, the DMA has to select it and to click on the [Edit]
button.
According to the type of the selected validation rule, the appropriate popup window will then
appear.
8.2 Removing a validation rule
If a validation rule needs to be deleted, the DMA has to select it and to click on the [Remove]
button.
January 2011
Page 30
9
Validating data files
The information contained in this section is a quick overview of how the validations are
processed when transferring data files to EDAMIS.
Senders from Member States have several possibilities to send data files to EDAMIS: either
by using EWP or by using EWA. Both transmission modes will be explained here.
9.1 Sending data files through EWP
To transfer data files to EDAMIS through EWP, end-users have to have SENDER privileges.
Once connected to EWP, they have to go to the Send Datafile interface by selecting the
Transmission >> Send Datafile menu entry as follows:
Figure 36: Send Datafile menu entry
In EWP, the Send Datafile interface allows two modes for the upload of data files:
•
•
the Simple upload form
the Advance and highly secure upload form
The advantage of the advanced upload form is that it allows the encryption of the data. To do
so, it relies on a Java applet which is also the drawback of this transmission mode.
Whatever the chosen mode, when selecting a data file (and specifying the dataset if the data
file does not respect the dataset naming convention), the Validation? checkbox is ticked if
ever a published validation exist for the targeted dataset (see Figure 37).
January 2011
Page 31
Figure 37: Send Datafile form when selecting a data file to be validated
9.1.1 Sending data files containing critical errors
If the SENDER sends a data file that contains at least one critical error, the validation process
raises an error and the data file transfer is aborted as depicted below:
Figure 38: Uploading a data file containing a (critical) error
(Advanced upload form)
Figure 39: Uploading a data file containing a (critical) error
(Simple upload form)
The data file is then listed as a non transmitted file with validation status set to Validation
NOT OK [-2] as follows:
January 2011
Page 32
Figure 40: Send Datafile form after data files upload
Nonetheless, the SENDER has the possibility to force the sending of the data file by clicking
on the corresponding [force delivery] button.
By clicking on the corresponding [view log files] button, a report such as the following one
will be displayed:
Figure 41: Validation report with a (critical) error
9.1.2 Sending data files containing basic errors
If the SENDER sends a data file containing a basic error, the validation process raises a
warning but the data file is transferred nonetheless as depicted below (unless it contains at
least a critical error):
Figure 42: Uploading a data file containing a warning (basic error)
(Advanced upload form)
Figure 43: Uploading a data file containing a warning (basic error)
(Simple upload form)
January 2011
Page 33
The data file is then listed as a transmitted file with validation code set to Validation with
warnings [-1] (see Figure 40).
By clicking on the corresponding [view log files] button, a report such as the following one
will be displayed:
Figure 44: Validation report with a warning (basic error)
9.1.3 Sending correct data files
If the SENDER sends a data file with no error of any kind, the validation process ends
successfully as depicted below:
Figure 45: Uploading a data file with no error
(Advanced upload form)
Figure 46: Uploading a data file with no error
(Simple upload form)
The data file is then listed as a transmitted file with validation code set to Validation
successful [0] (see Figure 40).
By clicking on the corresponding [view log files] button, a report such as the following one
will then be displayed:
January 2011
Page 34
Figure 47: Validation report with no error
9.1.4 Validation reports availability
Once the SENDER has logged out from EWP, the validation reports are not available
anymore through the Send Datafile form.
Nonetheless, validation reports are also sent by mail to the SENDER and to the RECEIVER
at the end of the validation process.
9.2 Sending data files through EWA
To transfer data files to EDAMIS through EWA, end-users have to have SENDER privileges.
Once connected to EWA, they have to go to the Send Datafiles interface by selecting the Send
Datafile >> Send Datafile menu entry as follows:
Figure 48: EWA - Send Datafile menu entry
Contrary to EWP, when selecting a data file on EWA(and specifying the dataset if the data
file does not respect the dataset naming convention) the Valid? checkbox is unticked (see
Figure 49).
Figure 49: EWA - Send Datafiles form when selecting a data file to be validated
January 2011
Page 35
As of today (EWA 2.9.9b), validations are not performed locally but on the server once the
data files have been transmitted to EWP.
Contrary to EWP, validation reports are not available through EWA but they are sent by mail
to the SENDER and to the RECEIVER at the end of the validation process.
Figure 50: EWA - Send Datafiles form after data files upload
January 2011
Page 36
10 Annex
Manage Validation form – Selection area
The selection area allows the DMA to shorten the list of existing validations by
means of filters on theme, domain and dataset.
1
Theme
This filter allows the DMA to select the theme.
2
Domain
This filter allows the DMA to select the domain.
3
Dataset
This filter allows the DMA to select the dataset.
4
[View] button
Once the filters have been changed, the user has to click on the [View] button to
refresh the page accordingly.
Table 1: Manage Validation form – Selection area
Manage Validation form – Validations description area
This area lists the existing validations for the given filters.
If the DMA clicks on one of them, the corresponding validation is opened either in
read only mode (if the validation was not locked by the DMA) or in edit mode.
1
Checkbox
These checkboxes enable the selection of one or several validations on which an
operation will be performed.
2
Dataset
This indicates the ID of the dataset to which the validation is attached.
3
Label
This indicates the label of the dataset relative to the corresponding validation.
4
Dsd ID
This indicates the ID of the DSD relative to the corresponding validation.
5
Dsd Version
This indicates the version of the DSD relative to the corresponding validation.
6
Dsd Agency
This indicates the agency of the DSD relative to the corresponding validation.
January 2011
Page 37
7
Dsd Source
This indicates the source of the DSD relative to the corresponding validation.
8
Lock
This indicates if the corresponding validation is locked (marked with a red lock icon)
or not.
9
Creation
This indicates when and by whom the corresponding validation has been created.
10 Modification
This indicates when and by whom the corresponding validation has been lately
modified.
11 Active
This indicates if the corresponding validation is published or not.
Table 2: Manage Validation form – Validations description area
Manage Validation form – Action buttons
This allows the DMA to perform selected actions on validations.
1
[Lock] button
(visible only when entries are listed)
By selecting an entry and by clicking on the [Lock] button, the DMA can lock the
corresponding validation to make it editable.
2
[Unlock] button
(visible only when entries are listed)
By selecting an entry and by clicking on the [Unlock] button, the DMA can unlock
the corresponding validation to make it available for edition to other users.
3
[Associate a dsd] button
By clicking on the [Associate a dsd] button, the DMA can create a new validation
through the Manage Validation – Associate a DSD form.
4
[Delete] button
(visible only when entries are listed)
By selecting one or several entries and by clicking on the [Delete] button, the DMA
can delete the corresponding validation(s) providing that it(they) is(are) locked and
inactive.
Table 3: Manage Validation form – Action buttons
Manage Validation - Associate a DSD form – Definition area
The definition area allows the DMA to specify the properties of the DSD to be
associated to the current dataset.
1
DSD ID
January 2011
Page 38
This allows the DMA to provide the ID of the DSD to be associated to the current
dataset.
2
DSD Version
This allows the DMA to provide the version of the DSD to be associated to the
current dataset.
3
DSD Agency
This allows the DMA to provide the agency of the DSD to be associated to the
current dataset.
Table 4: Manage Validation - Associate a DSD form – Definition area
Manage Validation - Associate a DSD form – Action buttons
This allows the DMA to finalize or to abort the association process.
1
[Associate a dsd] button
By clicking on the [Associate a dsd] button, the DMA is able to associate to the
current dataset the DSD defined by the properties provided beforehand.
A warning message will be displayed if the DSD cannot be retrieved from the
SDMX Registry or if the DSD has been already associated to the current dataset.
2
[Cancel] button
The [Cancel] button allows the DMA to abort the association process.
Table 5: Manage Validation - Associate a DSD form – Action buttons
Manage Validation – DSD Form form – Summary area
This area summarizes the global properties of the current validation.
1
Dataset
(read only)
This indicates the ID of the dataset relative to the current validation.
2
DSD Id
(read only)
This indicates the ID of the DSD relative to the current validation.
3
Version
(read only)
This indicates the version of the DSD relative to the current validation.
4
Agency
(read only)
This indicates the agency of the DSD relative to the current validation.
5
Description
This allows the DMA to provide a description for the current validation.
January 2011
Page 39
6
(read only)
Creation
This indicates when and by whom the current validation has been created.
7
(read only)
Modification
This indicates when and by whom the current validation has been lately modified.
8
(read only)
Active
This indicates if the current validation is published or not.
Table 6: Manage Validation – DSD Form form – Summary area
Manage Validation – DSD Form form – Validation rules sets description area
This area lists the existing validation rules sets for the current validation.
1
Checkbox
(visible only when several entries are listed)
These checkboxes enable the selection of one or several validation rules sets on
which an operation will be performed.
2
Rules Set Name
This indicates the ID of the corresponding validation rules set.
If the DMA needs to modify it, this is the only place to do so. A confirmation
message will appear if the DMA clicks on an action button.
3
Description
This indicates the description of the corresponding validation rules set.
If the DMA needs to modify it, this is the only place to do so. A confirmation
message will appear if the DMA clicks on an action button.
4
(read only)
Version
This indicates the (automatically computed) version of the corresponding validation
rules set.
Table 7: Manage Validation – DSD Form form – Validation rules sets description
area
Manage Validation – DSD Form form – Action buttons
This allows the DMA to perform selected actions on validation rules sets.
When only one validation rules set is defined, no checkbox is displayed and the
DMA can directly click on the appropriate action button.
1
[Add Rules Set] button
(enabled only when the validation is locked)
By clicking on the [Add Rules Set] button, the DMA can create a new validation
rules set through the Manage Validation – Add a rule set to the DSD form.
2
[Copy Rules Set] button
January 2011
(visible only when entries are listed
Page 40
and enabled only when the validation is locked)
By selecting an entry and by clicking on the [Copy Rules Set] button, the DMA can
duplicate the selected validation rules set.
3
[Edit Rules Set / View Rules Set] button
(enabled only when entries are listed)
By selecting an entry and by clicking on the [Edit Rules Set] button, the DMA can
edit the validation rules of the selected validation rules set through the validation
rules editor.
If the validation was not locked beforehand by the DMA, the button caption is set to
[View Rules Set] and the validation rules editor is opened in read only mode.
4
[Delete Rules Set] button
(enabled only when entries are listed
and when the validation is locked)
By selecting an entry and by clicking on the [Delete Rules Set] button, the DMA can
delete the corresponding validation rules set.
5
[Test rules] button
(visible only when entries are listed)
By selecting an entry and by clicking on the [Test rules] button, the DMA can test
the validation rules of the corresponding validation rules set against a SDMX-ML
data file through a dedicated form.
6
[Save] button
By clicking on the [Save] button, the DMA can save the current validation.
7
[Publish / Unpublish] button
By clicking on the [Publish] button, the DMA can publish the current validation.
By clicking on the [Unpublish] button, the DMA can unpublish the current
validation.
8
[Cancel] button
By clicking on the [Cancel] button, the DMA can leave the Manage Validation –
DSD Form form.
Table 8: Manage Validation – DSD Form form – Action buttons
Manage Validation – Add a rules set to the DSD form – Definition area
The definition area allows the DMA to specify the properties of the validation rules
set to be added to the current validation.
1
Rules set Id
(mandatory)
This allows the DMA to provide the ID of the validation rules set to be added to the
current validation.
2
Description
(optional)
This allows the DMA to provide the description of the validation rules set to be
added to the current validation.
January 2011
Page 41
3
(read only)
Version
This automatically computed value indicates the sort order of the validation rules set
to be added to the current validation.
Table 9: Manage Validation - Add a rules set to the DSD form – Definition area
Manage Validation - Add a rules set to the DSD form – Action buttons
This allows the DMA to finalize or to abort the creation process.
1
[Save] button
By clicking on the [Save] button, the DMA is able to associate to the current
validation the validation rules set defined by the properties provided beforehand.
2
[Cancel] button
The [Cancel] button allows the DMA to abort the creation process.
Table 10: Manage Validation - Add a rules set to the DSD form – Action buttons
Manage Validation – Validation rules editor – Selection area
The selection area allows the DMA to select the dimensions of the observation
value(s) for which he wants to create a validation rule.
As a reminder, it also indicates the current dataset, the current validation rules set ID
and version as well as the current DSD ID.
1
Dimensions
By clicking beneath a dimension cell, a dropdown list appears allowing the DMA to
select the desired value for the corresponding dimension.
By default, the dropdown list displays the labels of the dimension items.
2
[Codes / Labels] button
By clicking on the [Codes] button, the aforementioned dropdown lists display the
codes of the dimensions items instead of their labels. The button caption then toggles
to [Labels] in order to switch back to the display of labels.
Table 11: Validation rules editor – Selection area
Manage Validation – Validation rules editor – Validation rules description area
This area lists the existing validation rules for the current validation rules set.
1
Dimensions Names
This indicates the dimensions of the observation value(s) targeted by the
corresponding validation rule.
January 2011
Page 42
2
Type
This indicates the type of the corresponding validation rule: normal or advanced.
3
Country
This indicates the country targeted by the corresponding validation rule.
4
Mandatory
If checked, this indicates that the targeted observation value(s) has(have) to be
mandatory. Furthermore, the relative severity level (basic/critical) is also indicated.
5
Range of values
If provided, this indicates that the targeted observation value(s) has(have) to be in
between the specified values. Furthermore, the relative severity level (basic/critical)
is also indicated.
6
Active
If checked, this indicates that the targeted observation value(s) has(have) to be
validated against the corresponding validation rule.
7
Expression
If provided, this indicates that the targeted observation value(s) has(have) to fulfil
the specified expression. Furthermore, the relative severity level (basic/critical) is
also indicated.
8
Valid
If provided, this indicates that the corresponding validation rule will be applied to
the targeted observation value(s) whose reference period is greater or equal to the
specified period.
9
Valid To
If provided, this indicates that the targeted observation value(s) has(have) to be
validated against the corresponding validation rule only if its(their) reference period
is lower or equal to the specified period.
Table 12: Validation rules editor – Validation rules description area
Manage Validation – Validation rules editor – Action buttons
This allows the DMA to perform selected actions on validation rules.
1
[Add] button
By optionally specifying one or several dimensions and by clicking on the [Add]
button, the DMA can create a regular rule through a dedicated popup window.
2
[Add Advanced Rule] button
By specifying all required dimensions and by clicking on the [Add Advanced Rule]
button, the DMA can create an advanced rule through a dedicated popup window.
January 2011
Page 43
3
[Remove] button
By selecting a rule and by clicking on the [Remove] button, the DMA can remove
the corresponding validation rule.
4
[Edit] button
By selecting a rule and by clicking on the Edit button, the DMA can edit the
corresponding validation rule through the corresponding popup window.
5
[Save] button
By clicking on the Save button, the DMA can save the validation rules of the current
validation rules set.
6
[Exit] button
By clicking on the Exit button, the DMA can exit the validation rules editor and
return to the Manage Validation – DSD Form form.
Table 13: Validation rules editor – Action buttons
Validation rules editor – Normal Rule popup window
This popup window allows the DMA to create or edit a normal validation rule, aka
intra-record rule.
1
Activate
This allows the DMA to specify if the current validation rule has to be applied
during the validation process.
2
Country
This allows the DMA to specify to which country linked to the dataset the current
validation rule is dedicated (either all countries or a specific one).
3
Mandatory
This allows the DMA to specify if the observation value has to be mandatory and, if
so, the relative error severity level if the observation value is not provided.
4
Range
This allows the DMA to specify if the observation value has to be inside a given
range and, if so, the relative error severity level if the observation value is outside
this range.
5
Expression
This allows the DMA to specify if the observation value has to fulfil a given
expression and, if so, the relative error severity level if the observation value does
not fulfil this expression.
6
Error message
This allows the DMA to specify the error message that will appear in the validation
report if the observation value does not fulfil the above mentioned expression.
January 2011
Page 44
7
Valid From Period
This allows the DMA to specify if the current validation rule has to be applied only
for observation values whose reference period is greater or equal to a given “valid
from” period.
8
Valid To Period
This allows the DMA to specify if the current validation rule has to be applied only
for observation values whose reference period is lower or equal to a given “valid to”
period.
9
[OK] button
By clicking on the [OK] button, the DMA creates or updates the current validation
rule.
10 [Cancel] button
By clicking on the [Cancel] button, the DMA can abort the creation or update of the
current validation rule.
Table 14: Validation rules editor – Normal Rule popup window
Validation rules editor – Advanced Rule popup window
This popup window allows the DMA to create or edit an advanced validation rule,
aka inter-record rule.
1
Activate
This allows the DMA to specify if the current validation rule has to be applied
during the validation process.
2
Country
This allows the DMA to specify to which country linked to the dataset the current
validation rule is dedicated (either all countries or a specific one).
3
Mandatory
This allows the DMA to specify if the observation value has to be mandatory and, if
so, the relative error severity level if the observation value is not provided.
4
Expression
This allows the DMA to specify if the observation value has to fulfil a given
expression and, if so, the relative error severity level if the observation value does
not fulfil this expression.
By clicking on the [View Codes] button, a popup window displaying the available
values for each dimension is displayed in order to allow the DMA to retrieve the
correct combination to use as reference in the expression (see paragraph ‘Advanced
rules’).
5
Error message
This allows the DMA to specify the error message that will appear in the validation
January 2011
Page 45
report if the observation value does not fulfil the above mentioned expression.
6
Valid From Period
This allows the DMA to specify if the current validation rule has to be applied only
for observation values whose reference period is greater or equal to a given “valid
from” period.
7
Valid To Period
This allows the DMA to specify if the current validation rule has to be applied only
for observation values whose reference period is lower or equal to a given “valid to”
period.
8
[OK] button
By clicking on the [OK] button, the DMA creates or updates the current validation
rule.
9
[Cancel] button
By clicking on the [Cancel] button, the DMA can abort the creation or update of the
current validation rule.
Table 15: Validation rules editor – Advanced Rule popup window
Manage Validation – Create a copy of the rules set form – Definition area
The definition area allows the DMA to specify the properties of the validation rules
set duplicate.
1
Rules set Id
(mandatory)
This allows the DMA to provide the ID of the validation rules set duplicate.
2
Description
(optional)
This allows the DMA to provide the description of the validation rules set duplicate.
3
Version
(read only)
This automatically computed value indicates the sort order of the validation rules set
duplicate.
Table 16: Manage Validation - Create a copy of the rules set form – Definition
area
Manage Validation - Create a copy of the rules set form – Action buttons
This allows the DMA to finalize or to abort the duplication process.
1
[Save] button
By clicking on the [Save] button, the DMA launches the duplication process.
2
[Cancel] button
The [Cancel] button allows the DMA to abort the duplication process.
January 2011
Page 46
Table 17: Manage Validation - Create a copy of the rules set form – Action
buttons
Manage Validation – Test rules form – Definition area
The definition area allows the DMA to specify the properties of the data file to test
the validation rules against.
1
Dataset
(read only)
This indicates as a reminder the parent dataset of the current validation.
2
Country
(mandatory)
This allows the DMA to specify for which country the data have to be tested.
3
Year - Period
(mandatory)
This allows the DMA to specify for which reference period the data have to be
tested.
4
Data File
(mandatory)
This allows the DMA to specify the data file containing the data to be tested.
Table 18: Manage Validation - Test rules form – Definition area
Manage Validation - Test rules form – Action buttons
This allows the DMA to finalize or to abort the test of the validation process.
1
[Perform] button
By clicking on the [Perform] button, the DMA launches the test of the validation
process.
2
[Cancel] button
The [Cancel] button allows the DMA to abort the test of the validation process.
Table 19: Manage Validation - Test rules form – Action buttons
January 2011
Page 47
11 Glossary
Concept
Definition
CIRCA
CIRCA is an extranet tool developed by the European
Commission and tuned towards Public Administrations’ needs. It
enables a given community geographically spread across Europe
to form an online ‘Interest Group’, thereby providing a private
space on the Internet where members can share information and
documents, participate in discussion forums and make use of
various other functionalities. URL: http://circa.europa.eu/
CSV
Comma-Separated Values file format.
Data file
Physical data file transmitted which should normally correspond
to an instance of a dataset (a physical instance of a dataset
occurrence).
Dataset
Dataset structure with a specific periodicity and to which is
usually linked a deadline for transmission. The data content of a
dataset is homogeneous.
Dataset
naming Convention used by all production units at Eurostat for uniformly
naming a dataset. It composed of the domain name, the statistical
convention
table name and the periodicity, all separated by an underscore.
The dataset naming convention is recognised by all third-party
applications at Eurostat.
Dataset occurrence
Occurrence of a dataset for one country and one period or time
series or sequence. For Web Forms it is called a ‘Web Form
instance’.
Dataset structure
Concept of one single data structure associated to one or several
statistical tables (closely linked into the same statistical domain).
Domain
Consistent group of datasets that are closely linked together
(belong to the same statistical area and based on the same legal
act(s) or voluntary agreements).
EDAMIS
Electronic Data files Administration and Management Information
System.
EWA
EDAMIS Web Application.
EWP
EDAMIS Web Portal.
SDMX-ML
XML format for the exchange of SDMX-structured data and
metadata. The SDMX format (Statistical Data and Metadata
Exchange) is the result of cooperation between the BIS, the ECB,
Eurostat, the IMF, the OECD and the UN to explore common
e-standards and ongoing standardisation activities that could
afford more efficiency in the field of statistical information.
Theme
Theme of the Statistical Programme.
January 2011
Page 48
12 History of Changes
31.01.2011
EDAMIS release 3.1c
Document version 1.0
Status draft
How to set up and use the validation in EDAMIS
January 2011
Page 49