d1_test package¶
DataONE Test Utilities.
The DataONE Test Utilities package contains various utilities for testing DataONE infrastructure components and clients. These include:
Instance Generator: Used for creating randomized System Metadata documents
Stress Tester: Used for stress testing of Member Node implementations. The stress_tester creates a configurable number of concurrent connections to a Member Node and populates the MN with randomly generated objects while running queries and object retrievals.
Utilities: Misc test utilities.
Although this directory is not a package, this __init__.py file is required for pytest to be able to reach test directories below this directory.
Subpackages¶
- d1_test.instance_generator package
- Subpackages
- d1_test.instance_generator.tests package
- Submodules
- d1_test.instance_generator.tests.test_access_policy module
- d1_test.instance_generator.tests.test_checksum module
- d1_test.instance_generator.tests.test_datetime module
- d1_test.instance_generator.tests.test_format_id module
- d1_test.instance_generator.tests.test_identifier module
- d1_test.instance_generator.tests.test_person module
- d1_test.instance_generator.tests.test_random_data module
- d1_test.instance_generator.tests.test_replica module
- d1_test.instance_generator.tests.test_replication_policy module
- d1_test.instance_generator.tests.test_sciobj module
- d1_test.instance_generator.tests.test_subject module
- d1_test.instance_generator.tests.test_systemmetadata module
- d1_test.instance_generator.tests.test_user_agent module
- d1_test.instance_generator.tests package
- Submodules
- d1_test.instance_generator.access_policy module
- d1_test.instance_generator.checksum module
- d1_test.instance_generator.date_time module
- d1_test.instance_generator.format_id module
- d1_test.instance_generator.identifier module
- d1_test.instance_generator.media_type module
- d1_test.instance_generator.names module
- d1_test.instance_generator.node_ref module
- d1_test.instance_generator.person module
- d1_test.instance_generator.random_data module
- d1_test.instance_generator.replica module
- d1_test.instance_generator.replication_policy module
- d1_test.instance_generator.replication_status module
- d1_test.instance_generator.sciobj module
- d1_test.instance_generator.subject module
- d1_test.instance_generator.system_metadata module
- d1_test.instance_generator.unicode_names module
- d1_test.instance_generator.unicode_test_strings module
- d1_test.instance_generator.user_agent module
- d1_test.instance_generator.words module
- Subpackages
- d1_test.mock_api package
- Subpackages
- d1_test.mock_api.tests package
- Submodules
- d1_test.mock_api.tests.test_catch_all module
- d1_test.mock_api.tests.test_create module
- d1_test.mock_api.tests.test_d1_exception module
- d1_test.mock_api.tests.test_describe module
- d1_test.mock_api.tests.test_generate_identifier module
- d1_test.mock_api.tests.test_get module
- d1_test.mock_api.tests.test_get_capabilities module
- d1_test.mock_api.tests.test_get_format module
- d1_test.mock_api.tests.test_get_log_records module
- d1_test.mock_api.tests.test_get_query_engine_description module
- d1_test.mock_api.tests.test_get_system_metadata module
- d1_test.mock_api.tests.test_is_authorized module
- d1_test.mock_api.tests.test_list_formats module
- d1_test.mock_api.tests.test_list_nodes module
- d1_test.mock_api.tests.test_list_objects module
- d1_test.mock_api.tests.test_ping module
- d1_test.mock_api.tests.test_post module
- d1_test.mock_api.tests.test_query module
- d1_test.mock_api.tests.test_resolve module
- d1_test.mock_api.tests.test_solr_search module
- d1_test.mock_api.tests.test_util module
- d1_test.mock_api.tests package
- Submodules
- d1_test.mock_api.catch_all module
- d1_test.mock_api.create module
- d1_test.mock_api.d1_exception module
- d1_test.mock_api.describe module
- d1_test.mock_api.django_client module
- d1_test.mock_api.echo_credentials module
- d1_test.mock_api.generate_identifier module
- d1_test.mock_api.get module
- d1_test.mock_api.get_capabilities module
- d1_test.mock_api.get_format module
- d1_test.mock_api.get_log_records module
- d1_test.mock_api.get_system_metadata module
- d1_test.mock_api.is_authorized module
- d1_test.mock_api.list_formats module
- d1_test.mock_api.list_nodes module
- d1_test.mock_api.list_objects module
- d1_test.mock_api.ping module
- d1_test.mock_api.post module
- d1_test.mock_api.query_engine_description module
- d1_test.mock_api.resolve module
- d1_test.mock_api.solr_query module
- d1_test.mock_api.solr_search module
- d1_test.mock_api.util module
- Subpackages
- d1_test.replication_tester package
- d1_test.tests package
- d1_test.utilities package
- Submodules
- d1_test.utilities.create_dataone_test_certificate module
- d1_test.utilities.create_from_file module
- d1_test.utilities.generate_sysmeta_for_sciobj module
- d1_test.utilities.generate_test_subject_certs module
- d1_test.utilities.list_effective_subjects module
- d1_test.utilities.my_subject module
- d1_test.utilities.pem_in_http_header module
- d1_test.utilities.populate_mn module
- d1_test.utilities.test_object_generator module
- d1_test.utilities.urlencode module
Submodules¶
d1_test.d1_test_case module¶
d1_test.pycharm module¶
d1_test.sample module¶
d1_test.slender_node_test_client module¶
-
class
d1_test.slender_node_test_client.
SlenderNodeTestClient
(sciobj_store_path='./sciobj_store', keep_existing=False, *_args, **_kwargs)¶ Bases:
object
A simple drop-in replacement for a MN client, for use when developing and testing SlenderNode scripts.
MN is simulated to the bare minimum required by SN scripts
Objects are stored in local files instead of on an MN
SID to PID dict is held in memory and dumped to file
Most args are simply ignored
-
__init__
(sciobj_store_path='./sciobj_store', keep_existing=False, *_args, **_kwargs)¶ Create the test client.
Store the sciobj and sysmeta in
sciobj_store_path
sciobj_store_path
is created if it does not existIf
delete_existing
is True, delete any existing files insciobj_store_path
-
create
(pid, sciobj_file, sysmeta_pyxb, *args, **kwargs)¶
-
update
(old_pid, sciobj_file, new_pid, new_sysmeta_pyxb, *args, **kwargs)¶
-
get
(did, *args, **kwargs)¶ Return a file-like object with the sciobj bytes.
-
getSystemMetadata
(did, *args, **kwargs)¶ Return sysmeta_pyxb.
d1_test.test_files module¶
Utilities for loading test files.
-
d1_test.test_files.
get_abs_test_file_path
(rel_path)¶
-
d1_test.test_files.
load_bin
(rel_path)¶
-
d1_test.test_files.
load_utf8_to_str
(rel_path)¶ Load file, decode from UTF-8 and return as str.
-
d1_test.test_files.
load_xml_to_pyxb
(filename)¶
-
d1_test.test_files.
load_xml_to_str
(filename)¶
-
d1_test.test_files.
load_xml_to_bytes
(filename)¶
-
d1_test.test_files.
load_json
(filename)¶
-
d1_test.test_files.
load_cert
(filename)¶
-
d1_test.test_files.
load_jwt
(filename)¶
-
d1_test.test_files.
save
(obj_str, rel_path, encoding='utf-8')¶
d1_test.xml_normalize module¶
Generate a str that contains a normalized representation of an XML document.
For unit testing, we want to be able to store and compare samples representing XML documents that are guaranteed to be stable.
Often, XML docs have various sections containing unordered sets of elements where there are no semantics associated with the order in which they appear in the doc. The same is true for element attributes. For DataONE, typical examples are lists of subjects, permission rules and services.
Since the source for such elements are often dict and set based containers that themselves don’t provide deterministic ordering, serializing a group of such objects can generate a large number of possible XML docs that, while semantically identical, cannot be directly compared as text or in the DOM.
Normalizing the formatting can be done with a single deserialize to DOM and back to XML, but that will not normalize the ordering of the elements, Without a schema, automated tools cannot rearrange elements in an XML doc, since it is not known if the order is significant. However, for generating and comparing XML doc samples, a stable document that contains all the information from the XML doc is sufficient.
The strategy for generating a stable representation of an XML doc is as follows:
All sibling XML elements must be sorted regardless of where they are in the tree.
Each element is the root of a branch of the node tree. Sorting, of course, is based on comparing individual elements in order to determine their relative orderings. If the information in the elements themselves is identical, it is necessary to break the tie by recursively comparing their descendants until either a difference is found, or the two elements are determined to be the roots of two identical branches.
To enable the sort algorithm to compare the branches, sort keys that hold all information in the branch are generated and passed to the sort. For comparisons to properly compare elements in the most to least significant order, each node in the branch must be in a single list item. So the key is a nested list of lists.
Finally, since the sort keys are generated from the descendants, siblings in a given element can only be sorted after all their descendants in the tree have been sorted. So the tree must be traversed depth first, and the sort performed as the algorithm is stepping up from a completed level.
To avoid having to build a new tree depth first, inline sort is used.
Notes
# RDF-XML
Although the hierarchical structure of elements is almost always significant in XML, there are instances where semantically identical XML docs can have different hierarchies. This often occurs when generating RDF-XML docs from RDF.
This module only normalizes the ordering of sibling elements and attributes. Parent-child relationships are never changed. So RDF-XML docs generated in such a way that parent-child relationships may differ without change in semantics are not supported.
## Background
RDF is an unordered set of subject-predicate-object triples. Triples cannot share values, so when there are multiple triples for a subject, each triple must contain a copy of the subject.
RDF-XML supports expressing triples with less redundancy by factoring shared values out to parent elements. E.g., a set of triples for a subject can be expressed as a series of predicate-object children with a single subject parent.
When generating RDF-XML from RDF that contains many triples that share values, the same set of triples can be represented by many different hierarchies. The hierarchy that is actually generated depends on the algorithm and may also depend on the order in which the triples are processed. If the triples are retrieved from an unordered set, the processing order is pseudo-random, causing pseudo-random variations in the generated hierarchy.
-
d1_test.xml_normalize.
get_normalized_xml_representation
(xml)¶ Return a str that contains a normalized representation of an XML document.
-
d1_test.xml_normalize.
xml_to_stabletree
(xml)¶ Return a StableTree that contains a normalized representation of an XML document.
-
d1_test.xml_normalize.
etree_to_stable_tree
(et_node)¶ Convert an ElementTree to a StableTree.
Node attributes become @key:string - Text elements become @text:string - name is the name of the xml element
-
class
d1_test.xml_normalize.
StableNode
(name, child_node=None)¶ Bases:
object
Tree structure that uses lists instead of dicts, as lists have deterministic ordering.
-
__init__
(name, child_node=None)¶ child is E or str.
-
add_child
(e)¶
-
get_str
(s, indent)¶
-
get_sort_key_
()¶
-
sort
(p=None)¶
-
-
d1_test.xml_normalize.
StableTree
¶ alias of
d1_test.xml_normalize.StableNode