d1_gmn.app package

Subpackages

Submodules

d1_gmn.app.auth module

d1_gmn.app.context_processors module

Django template context processors.

Before rendering a template, Django calls context processors as set up in settings_default.TEMPLATE_CONTEXT_PROCESSORS. The context processors are functions that are expected to return a dict which will be merged into the environment available to the template.

d1_gmn.app.context_processors.global_settings(request)

Expose some values from settings.py to templates.

d1_gmn.app.db_filter module

d1_gmn.app.delete module

d1_gmn.app.did module

Utilities for working with SIDs and PIDs.

d1_gmn.app.did.is_valid_pid_for_create(did)

Return True if did is the PID of an object that can be created with MNStorage.create() or MNStorage.update().

To be valid for create() and update(), the DID:

  • Must not be the PID of an object that exists on this MN

  • Must not be a known SID known to this MN

  • Must not have been accepted for replication by this MN.

  • Must not be referenced as obsoletes or obsoletedBy in an object that exists on this MN

In addition, if the DID exists in a resource map:

  • If RESOURCE_MAP_CREATE = ‘reserve’:

    • The DataONE subject that is making the call must have write or changePermission on the resource map.

d1_gmn.app.did.is_valid_sid_for_new_standalone(did)

Return True if did can be assigned to a new standalone object.

d1_gmn.app.did.is_valid_pid_to_be_updated(did)

Return True if did is the PID of an object that can be updated (obsoleted) with MNStorage.update()

d1_gmn.app.did.is_valid_sid_for_chain(pid, sid)

Return True if sid can be assigned to the single object pid or to the chain to which pid belongs.

  • If the chain does not have a SID, the new SID must be previously unused.

  • If the chain already has a SID, the new SID must match the existing SID.

All known PIDs are associated with a chain.

Preconditions: - pid is verified to exist. E.g., with

d1_gmn.app.views.asserts.is_existing_object().

  • sid is None or verified to be a SID

d1_gmn.app.did.get_did_by_foreign_key(did_foreign_key)

Return the DID referenced by a ForeignKey or OneToOneField to IdNamespace.

Return None if ForeignKey or OneToOneField is NULL.

This is used instead of “did_foreign_key.*.did” on ForeignKeys and OneToOneFields that allow NULL (null=True in the model).

d1_gmn.app.did.is_existing_object(did)

Return True if PID is for an object for which science bytes are stored locally.

This excludes SIDs and PIDs for unprocessed replica requests, remote or non-existing revisions of local replicas and objects aggregated in Resource Maps.

d1_gmn.app.did.is_sid(did)
d1_gmn.app.did.is_obsoleted(did)

Return True if did is the PID of an object that has been obsoleted.

d1_gmn.app.did.is_resource_map_db(pid)
d1_gmn.app.did.is_resource_map_member(pid)
d1_gmn.app.did.classify_identifier(did)

Return a text fragment classifying the did

Return <UNKNOWN> if the DID could not be classified. This should not normally happen and may indicate that the DID was orphaned in the database.

d1_gmn.app.did.get_or_create_did(id_str)
d1_gmn.app.did.is_in_revision_chain(sciobj_model)
d1_gmn.app.did.is_archived(pid)
d1_gmn.app.did.is_local_replica(pid)

Includes unprocessed replication requests.

d1_gmn.app.did.is_unprocessed_local_replica(pid)

Is local replica with status “queued”.

d1_gmn.app.did.is_revision_chain_placeholder(pid)

For replicas, the PIDs referenced in revision chains are reserved for use by other replicas.

d1_gmn.app.event_log module

d1_gmn.app.gmn module

App performing filesystem setup and basic sanity checks on configuration values in settings.py before GMN starts servicing requests.

Django loads apps into the Application Registry in the order specified in settings.INSTALLED_APPS. This app must be set to load before the main GMN app by listing it above the main app in settings.INSTALLED_APPS.

class d1_gmn.app.gmn.Startup(app_name, app_module)

Bases: django.apps.config.AppConfig

name = 'd1_gmn.app'
ready()

Called once per Django process instance.

If the filesystem setup fails or if an error is found in settings.py, django.core.exceptions.ImproperlyConfigured is raised, causing Django not to launch the main GMN app.

raise_config_error(setting_name, cur_val, exp_type, valid_str=None, is_none_allowed=False)

d1_gmn.app.local_replica module

d1_gmn.app.model_util module

Database model utilities.

These are in a separate module because module classes can only be referenced in an active Django context. More general utilities can be used without an active context.

Importing this module outside of Django context raises django.core.exceptions.AppRegistryNotReady: Apps aren’t loaded yet.

d1_gmn.app.model_util.get_sci_model(pid)
d1_gmn.app.model_util.get_pids_for_all_locally_stored_objects()
d1_gmn.app.model_util.delete_unused_subjects()

Delete any unused subjects from the database.

This is not strictly required as any unused subjects will automatically be reused if needed in the future.

d1_gmn.app.models module

d1_gmn.app.node module

Generate Node document based on the current settings for GMN.

d1_gmn.app.node.get_pretty_xml(api_major_int=2)
d1_gmn.app.node.get_xml(api_major_int)
d1_gmn.app.node.get_pyxb(api_major_int=2)

d1_gmn.app.node_registry module

Node Registry cache.

  • Retrieve, hold and update a cache of the Node Registry for the DataONE environment in which this MN is registered.

  • Query the Node Registry.

d1_gmn.app.node_registry.get_cn_subjects()
d1_gmn.app.node_registry.set_empty_cn_subjects_cache()
d1_gmn.app.node_registry.set_cn_subjects_for_environment()
d1_gmn.app.node_registry.get_cn_subjects_string()
d1_gmn.app.node_registry.get_cn_subjects_from_dataone_root()
d1_gmn.app.node_registry.download_node_registry()
d1_gmn.app.node_registry.create_root_cn_client()

d1_gmn.app.object_format_cache module

d1_gmn.app.proxy module

Proxy mode.

d1_gmn.app.proxy.get_sciobj_iter_remote(url)
d1_gmn.app.proxy.is_proxy_url(url)

d1_gmn.app.psycopg_adapter module

Psycopg Postgres adapter for Python.

Registers custom adapters with Psycopg, which simplify reading and writing custom DataONE PyXB types to/from database models.

d1_gmn.app.psycopg_adapter.adapt_pyxb_binding(client)

d1_gmn.app.resource_map module

Utilities for manipulating resource maps.

d1_gmn.app.resource_map.assert_map_is_valid_for_create(resource_map)
d1_gmn.app.resource_map.is_sciobj_valid_for_create()

When RESOURCE_MAP_CREATE == ‘reserve’, objects that are created and that are also aggregated in one or more resource maps can only be created by a DataONE subject that has write or changePermission on the resource map.

d1_gmn.app.resource_map.create_or_update_db(sysmeta_pyxb)
d1_gmn.app.resource_map.get_resource_map_from_sciobj(pid)
d1_gmn.app.resource_map.create_or_update(map_pid, resource_map)
d1_gmn.app.resource_map.get_resource_map_members(pid)

pid is the PID of a Resource Map or the PID of a member of a Resource Map.

d1_gmn.app.resource_map.get_resource_map_members_by_map(map_pid)
d1_gmn.app.resource_map.get_resource_map_members_by_member(member_pid)
d1_gmn.app.resource_map.is_resource_map_sysmeta_pyxb(sysmeta_pyxb)
d1_gmn.app.resource_map.parse_resource_map_from_str(resource_map_xml)

d1_gmn.app.revision module

Utilities for manipulating revision chains in the database.

d1_gmn.app.revision.create_or_update_chain(pid, sid, obsoletes_pid, obsoleted_by_pid)
d1_gmn.app.revision.delete_chain(pid)
d1_gmn.app.revision.cut_from_chain(sciobj_model)

Remove an object from a revision chain.

The object can be at any location in the chain, including the head or tail.

Preconditions: - The object with the pid is verified to exist and to be a member of an revision chain. E.g., with:

d1_gmn.app.views.asserts.is_existing_object(pid) d1_gmn.app.views.asserts.is_in_revision_chain(pid)

Postconditions: - The given object is a standalone object with empty obsoletes, obsoletedBy and

seriesId fields.

  • The previously adjacent objects in the chain are adjusted to close any gap that was created or remove dangling reference at the head or tail.

  • If the object was the last object in the chain and the chain has a SID, the SID reference is shifted over to the new last object in the chain.

d1_gmn.app.revision.get_all_pid_by_sid(sid)
d1_gmn.app.revision.resolve_sid(sid)

Get the PID to which the sid currently maps.

Preconditions: - sid is verified to exist. E.g., with d1_gmn.app.views.asserts.is_sid().

d1_gmn.app.revision.get_sid_by_pid(pid)

Given the pid of the object in a chain, return the SID for the chain.

Return None if there is no SID for the chain. This operation is also valid for standalone objects which may or may not have a SID.

This is the reverse of resolve.

All known PIDs are associated with a chain.

Preconditions: - pid is verified to exist. E.g., with

d1_gmn.app.views.asserts.is_existing_object().

d1_gmn.app.revision.is_obsoletes_pid(pid)

Return True if pid is referenced in the obsoletes field of any object.

This will return True even if the PID is in the obsoletes field of an object that does not exist on the local MN, such as replica that is in an incomplete chain.

d1_gmn.app.revision.is_obsoleted_by_pid(pid)

Return True if pid is referenced in the obsoletedBy field of any object.

This will return True even if the PID is in the obsoletes field of an object that does not exist on the local MN, such as replica that is in an incomplete chain.

d1_gmn.app.revision.is_revision(pid)

Return True if pid is referenced in the obsoletes or obsoletedBy field of any object.

This will return True even if the PID is in the obsoletes field of an object that does not exist on the local MN, such as replica that is in an incomplete chain.

d1_gmn.app.scimeta module

Utilities for Science Metadata.

d1_gmn.app.scimeta.assert_valid(sysmeta_pyxb, pid)

Validate file at {sciobj_path} against schema selected via formatId and raise InvalidRequest if invalid.

Validation is only performed when:

  • SciMeta validation is enabled

  • and Object size is below size limit for validation

  • and formatId designates object as a Science Metadata object which is recognized and parsed by DataONE CNs

  • and XML Schema (XSD) files for formatId are present on local system

d1_gmn.app.sciobj_store module

Manage the filesystem tree in which science object bytes are stored.

  • Because it may be inefficient to store millions of files in a single folder and because such a folder is hard to deal with when performing backups and maintenance, GMN stores the objects in a folder hierarchy of 256 folders, each holding 256 folders, for a total of 65536 folders. The location in the hierarchy for a given object is based on its PID.

  • Folders are created as required in the hierarchy.

d1_gmn.app.sciobj_store.open_sciobj_file_by_pid_ctx(pid, write=False)

Open the file containing the Science Object bytes of pid in the default location within the tree of the local SciObj store.

If write is True, the file is opened for writing and any missing directories are created. Return the file handle and file_url with the file location in a suitable form for storing in the DB.

If nothing was written to the file, it is deleted.

d1_gmn.app.sciobj_store.open_sciobj_file_by_path_ctx(abs_path, write=False)

Open the file containing the Science Object bytes at the custom location abs_path in the local filesystem.

If write is True, the file is opened for writing and any missing directores are created. Return the file handle and file_url with the file location in a suitable form for storing in the DB.

If nothing was written to the file, delete it.

d1_gmn.app.sciobj_store.get_sciobj_iter_by_url(sciobj_url)
d1_gmn.app.sciobj_store.get_sciobj_byte_iterator_by_url(sciobj_url)
d1_gmn.app.sciobj_store.get_sciobj_iter_by_pid(pid)
d1_gmn.app.sciobj_store.open_sciobj_file_by_pid(pid, write=False)

Open the file containing the Science Object bytes at the custom location abs_path in the local filesystem for read.

d1_gmn.app.sciobj_store.open_sciobj_file_by_path(abs_path, write=False)

Open a SciObj file for read or write. If opened for write, create any missing directories. For a SciObj stored in the default SciObj store, the path includes the PID hash based directory levels.

This is the only method in GMN that opens SciObj files, so can be modified to customize the SciObj storage locations and can be mocked for testing.

Note that when a SciObj is created by a client via MNStorage.create(), Django streams the SciObj bytes to a temporary file or memory location as set by FILE_UPLOAD_TEMP_DIR and related settings.

d1_gmn.app.sciobj_store.get_rel_sciobj_file_path(pid)

Get the relative local path to the file holding an object’s bytes.

  • The path is relative to settings.OBJECT_STORE_PATH

  • There is a one-to-one mapping between pid and path

  • The path is based on a SHA1 hash. It’s now possible to craft SHA1 collisions, but it’s so unlikely that we ignore it for now

  • The path may or may not exist (yet).

d1_gmn.app.sciobj_store.get_abs_sciobj_file_path_by_pid(pid)

Get the absolute local path to the file holding an object’s bytes.

  • The path is to a location below settings.OBJECT_STORE_PATH

  • There is a one-to-one mapping between pid and path

  • The path is based on a SHA1 hash. It’s now possible to craft SHA1 collisions, but it’s so unlikely that we ignore it for now

  • The path may or may not exist (yet).

d1_gmn.app.sciobj_store.get_abs_sciobj_file_path_by_rel_path(rel_path)

Get the absolute local path to the file holding an object’s bytes.

  • The path is to a location below settings.OBJECT_STORE_PATH

  • There is a one-to-one mapping between pid and path

  • The path is based on a SHA1 hash. It’s now possible to craft SHA1 collisions, but it’s so unlikely that we ignore it for now

  • The path may or may not exist (yet).

d1_gmn.app.sciobj_store.get_abs_sciobj_store_path()

Get the absolute local path to the root of the default SciObj store.

  • The path may or may not exist (yet).

d1_gmn.app.sciobj_store.assert_sciobj_store_exists()
d1_gmn.app.sciobj_store.assert_sciobj_store_does_not_exist()
d1_gmn.app.sciobj_store.is_existing_store()
d1_gmn.app.sciobj_store.is_existing_sciobj_file(pid)
d1_gmn.app.sciobj_store.get_rel_sciobj_file_url_by_pid(pid)

Get the URL that will be stored in the database for a SciObj that is saved in GMN’s SciObj filesystem hierarchy below settings.OBJECT_STORE_PATH.

d1_gmn.app.sciobj_store.get_abs_sciobj_file_url(abs_sciobj_file_path)

Get the URL that will be stored in the database for a SciObj that is saved in a custom location outside of GMN’s SciObj filesystem hierarchy.

d1_gmn.app.sciobj_store.get_abs_sciobj_file_path_by_url(file_url)

Get the absolute path to the file holding an object’s bytes.

  • file_url is an absolute or relative file:// url as stored in the DB.

d1_gmn.app.sciobj_store.get_gmn_version()
d1_gmn.app.sciobj_store.is_matching_version()
d1_gmn.app.sciobj_store.is_lower_version()
d1_gmn.app.sciobj_store.is_store_subdir(dir_path)
d1_gmn.app.sciobj_store.get_store_version()
d1_gmn.app.sciobj_store.save_store_version()
d1_gmn.app.sciobj_store.get_store_version_path()
d1_gmn.app.sciobj_store.create_store()
d1_gmn.app.sciobj_store.is_empty()
d1_gmn.app.sciobj_store.is_tmp()
d1_gmn.app.sciobj_store.assert_sciobj_store_version_match()
d1_gmn.app.sciobj_store.delete_sciobj(url_split, pid)

d1_gmn.app.settings_default module

Default settings for GMN.

  • These settings are only used as fallbacks in case the corresponding settings in settings.py are missing.

  • This allows settings to be added without having to modify settings.py in existing deployments.

  • Settings that are not described here are described in settings_template.py.

d1_gmn.app.sysmeta module

d1_gmn.app.sysmeta_extract module

d1_gmn.app.urls module

d1_gmn.app.util module

General utilities.

d1_gmn.app.util.coerce_put_post(request)

Django doesn’t particularly understand REST. In case we send data over PUT, Django won’t actually look at the data and load it. We need to twist its arm here.

The try/except abomination here is due to a bug in mod_python. This should fix it.

From django-piston/piston/utils.py

d1_gmn.app.util.dump_stack()
d1_gmn.app.util.create_http_echo_response(request)
d1_gmn.app.util.get_static_path(rel_path)