Data Quality Checks for DOI

REFERENCE: describes the automated and manual Data Quality Checks performed by the ODC Data Team after a DOI Request has been made

Members of the ODC Data Team (i.e., Curators) complete a comprehensive data quality review of every dataset submitted for publication. Once a DOI request has been made, the Curators use a series of automated and manual checks to review the Subject Data File, Data Dictionary, and Metadata and confirm that the dataset formatting and contents meet the minimum requirements set out in the ODC Publishing Standard. These checks ensure that the data is Interoperable and Reusable with other datasets.

Automated Data Quality Checks

The automated quality checks are performed during the dataset upload process, as automatic file validations. These ensure a baseline level of quality to all private and public datasets in the ODC. The checks performed during the upload process occur without human oversight since data upload is handled privately within the account of the data owner. If a file can be uploaded, it has successfully met the Minimum Upload Specifications and passed the automated data quality checks.

GO TO: Minimum Upload Specifications

Subject Data Files

The file must pass the following automatic validations:

Source Checks Purpose: to determine if the file can be read and uploaded

• File type is .csv file

• File size is <70 MB (see Manage a Dataset to learn how to upload data files exceeding this size limit)

Structure Checks Purpose: to determine if the Subject Data File has been formatted correctly

• Blank header: every column containing data must have a variable name in row 1 (i.e., no blank headers)

• Duplicate header: every variable must have a different name (i.e., all column header names in row 1 must be unique)

• Invalid header: every variable name in header (row 1) must be ≤64 characters and cannot start with an underscore; reserved special characters are not used in the column headers (first row with the variable names)

Other

• Cannot contain a variable named "Notes" or "notes"

Data Dictionary Files

The file must pass the following automatic validations:

Source Checks Purpose: to determine if the file can be read and uploaded

• File type is .csv file

• File size is <70 MB

Schema Checks Purpose: to determine if the Data Dictionary File has been formatted correctly AND identify any conflicts between it and the the Subject Data File

• Missing column headers: the Data Dictionary File must contain the column headers VariableName, Title, Unit_of_Measure, Description, DataType, PermittedValues, MinimumValue, MaximumValue, Comments

• Extra header: all variables in the Subject Data File must be present in the Data Dictionary File; extra header = one or more variable(s) in the Subject Data File are not in the Data Dictionary File *headers must be an exact match for format, spelling, use of capitalisation, etc.

• Missing header: all variables in the Subject Data File must be present in the Data Dictionary File; missing header = one or more variable(s) in the Data Dictionary File are missing from the Subject Data File *headers must be an exact match for format, spelling, use of capitalisation, etc.

• Missing title: a variable title must be entered in the column Title for every row with an entry under VariableName

• Missing description: a variable description must be entered in the column Description for every row with an entry under VariableName

Semi-Automated

The ODC Data Team uses an open source tool developed in-house to automate the evaluation of many of the technical specifications related to structure and format. This tool, the ODC Data Quality App (ODCdqa), has been made freely available to the public. The ODC strongly recommends all data producers run their dataset through the ODCdqa BEFORE submitting a DOI Request. Using the ODCdqa will help identify potential errors that can be resolved before submission, and reduce delays in the review and publication process.

Structure Checks Purpose: to determine if the Subject Data File has been formatted correctly

• Blank row: rows must have at least one non-blank cell

• Blank column: columns must have at lease one non-blank cell

• Duplicated row: rows should not be duplicated

• Duplicated column: columns should not be duplicated

• Required variables: all SCI or TBI required variables must be present in the header row (row 1) *headers must be an exact match for format, spelling, use of capitalisation, etc.

Schema Checks Purpose: to determine if the Data Dictionary File has been formatted correctly AND identify any conflicts between it and the the Subject Data File

• Required variables: all SCI or TBI required variables must be present in the column VariableName *headers must be an exact match for format, spelling, use of capitalisation, etc.

• Variable name: variable names must begin with a letter and cannot contain special characters other than . or _

Manual Checks

The final step of the data quality checks is performed manually by the ODC Data Team Curators. They read and review the contents of the entire dataset to ensure:

• Format: data is properly organised in Tidy Format

• Unique subject identifiers: every subject in a dataset has a complex, unique subject identifier (i.e., not 1, 2, 3... or A, B, C...)

• Required variables: subject data is present for all SCI or TBI required variables; if data is not available it is specified

• Value constraints: the values of a variable in the Subject Data File should be equal to one of the permitted values or within the minimum/maximum value limits enumerated in the Data Dictionary File

• Variable names: variable names are intuitive and appropriate for the described variable

• Data dictionary completeness: sufficient detail is provided to understand the scientific purpose/value of the variables

  • Variable titles and descriptions provide sufficient detail to understand how a variable differs from others

  • Unit of measure is reported (when applicable)

  • All categories/levels for a variable are identified in the column PermittedValues and defined/describe in the column PVDescribed (applicable to categorical and ordinal variables)

  • Appropriate minimum and maximum values are provided (applicable to numeric and ordinal variables)