Synchronizing the schemas with the Coordinating Node

The goal of the d1_scimeta Science Metadata validator is to match the validation results of the CN as closely as possible. As such, d1_scimeta uses the same set of schemas as that used by the CN, and they are included in the d1_scimeta package. If changes are made to the schemas used by the CN, the schemas included in the d1_scimeta package should be replaced with a fresh set of schemas from the CN.

The d1_scimeta validator is based on the lxml library. lxml cannot easily be blocked from following HTTP URLs referenced in schemaLocation attributes in xs:include and xs:import elements in XSD schemas used while validating an XML doc. As outgoing network connections and associated delays are not acceptable in many validation scenarios, we use schemas in which URLs have been rewritten to point to existing local XSD files. Or, where there were no existing local XSDs, we include cached versions which were downloaded while preparing the schemas for distribution.

See the prepare_schema.py module for details.

Schema syncronization procedure

Synchronize the XSD files used by the d1_scimeta Science Metadata validator with the schemas used by the CN.

• Delete the old schema directory and cache

  $ rm -r ./schema

• Download the new schema files, either from GitHub, or from a CN.

  • GitHub

    https://github.com/NCEAS/metacat/tree/master/lib/schema

  • CN

    $ rsync –recursive cn:/var/lib/tomcat8/webapps/metacat/schema/ ./schema

• Prepare the new schema files for use by lxml and create a local cache of schemas not directly included in the CN schema set:

  $ ./prepare_schema.py

• If support for new schemas have been added, add them to the format_id_to_schema.json file.

• Run d1_scimeta unit tests to ensure that validation still works as expected.

Troubleshooting

As lxml is a thin wrapper around the libxml2 C library, there is no easy way to get insight into the validation process from Python. To help with investigation of validation errors, a command, resolve_schema.py is included in d1_scimeta. resolve_schema.py recursively follows XSD xs:include and xs:import elements and lists any issues as they are encountered.