d1_gmn.app package¶
Subpackages¶
- d1_gmn.app.management package
- Subpackages
- d1_gmn.app.management.commands package
- Subpackages
- Submodules
- d1_gmn.app.management.commands.audit-sync module
- d1_gmn.app.management.commands.cert module
- d1_gmn.app.management.commands.diag-resource-map module
- d1_gmn.app.management.commands.diag-revision module
- d1_gmn.app.management.commands.diag module
- d1_gmn.app.management.commands.event module
- d1_gmn.app.management.commands.get-log module
- d1_gmn.app.management.commands.get-object module
- d1_gmn.app.management.commands.import-multiprocess module
- d1_gmn.app.management.commands.import module
- d1_gmn.app.management.commands.jwt module
- d1_gmn.app.management.commands.loginfo_async module
- d1_gmn.app.management.commands.lter-check-redirect module
- d1_gmn.app.management.commands.lter-fix module
- d1_gmn.app.management.commands.node module
- d1_gmn.app.management.commands.objectlist_async module
- d1_gmn.app.management.commands.process_refresh_queue module
- d1_gmn.app.management.commands.process_replication_queue module
- d1_gmn.app.management.commands.sciobj module
- d1_gmn.app.management.commands.sync module
- d1_gmn.app.management.commands.whitelist module
- d1_gmn.app.management.commands package
- Subpackages
- d1_gmn.app.middleware package
- Submodules
- d1_gmn.app.middleware.detail_codes module
- d1_gmn.app.middleware.exception_handler module
- d1_gmn.app.middleware.profiling_handler module
- d1_gmn.app.middleware.request_handler module
- d1_gmn.app.middleware.response_handler module
- d1_gmn.app.middleware.session_cert module
- d1_gmn.app.middleware.session_jwt module
- d1_gmn.app.middleware.view_handler module
- d1_gmn.app.migrations package
- Submodules
- d1_gmn.app.migrations.0001_initial module
- d1_gmn.app.migrations.0002_scienceobject_filename module
- d1_gmn.app.migrations.0003_mediatype_mediatypeproperty module
- d1_gmn.app.migrations.0004_auto_20170523_0137 module
- d1_gmn.app.migrations.0005_auto_20170527_1554 module
- d1_gmn.app.migrations.0006_chainidtopersistentid module
- d1_gmn.app.migrations.0007_delete_chainidtopersistentid module
- d1_gmn.app.migrations.0008_chainidtopersistentid module
- d1_gmn.app.migrations.0009_auto_20170603_0546 module
- d1_gmn.app.migrations.0010_auto_20170805_0107 module
- d1_gmn.app.migrations.0011_resourcemap_resourcemapmember module
- d1_gmn.app.migrations.0012_auto_20170821_2129 module
- d1_gmn.app.migrations.0013_auto_20171017_0724 module
- d1_gmn.app.migrations.0014_auto_20171017_0725 module
- d1_gmn.app.migrations.0015_auto_20171029_0246 module
- d1_gmn.app.migrations.0016_auto_20180518_1539 module
- d1_gmn.app.migrations.0017_auto_20180518_1540 module
- d1_gmn.app.migrations.0018_auto_20180901_0115 module
- d1_gmn.app.migrations.0019_auto_20190418_1512 module
- d1_gmn.app.views package
- Submodules
- d1_gmn.app.views.assert_db module
- d1_gmn.app.views.assert_sysmeta module
- d1_gmn.app.views.create module
- d1_gmn.app.views.decorators module
- d1_gmn.app.views.external module
- d1_gmn.app.views.get_package module
- d1_gmn.app.views.gmn module
- d1_gmn.app.views.headers module
- d1_gmn.app.views.internal module
- d1_gmn.app.views.slice module
- d1_gmn.app.views.util module
Submodules¶
d1_gmn.app.auth module¶
d1_gmn.app.context_processors module¶
Django template context processors.
Before rendering a template, Django calls context processors as set up in settings_default.TEMPLATE_CONTEXT_PROCESSORS. The context processors are functions that are expected to return a dict which will be merged into the environment available to the template.
-
d1_gmn.app.context_processors.
global_settings
(request)¶ Expose some values from settings.py to templates.
d1_gmn.app.db_filter module¶
d1_gmn.app.delete module¶
d1_gmn.app.did module¶
Utilities for working with SIDs and PIDs.
-
d1_gmn.app.did.
is_valid_pid_for_create
(did)¶ Return True if
did
is the PID of an object that can be created with MNStorage.create() or MNStorage.update().To be valid for create() and update(), the DID:
Must not be the PID of an object that exists on this MN
Must not be a known SID known to this MN
Must not have been accepted for replication by this MN.
Must not be referenced as obsoletes or obsoletedBy in an object that exists on this MN
In addition, if the DID exists in a resource map:
If RESOURCE_MAP_CREATE = ‘reserve’:
The DataONE subject that is making the call must have write or changePermission on the resource map.
-
d1_gmn.app.did.
is_valid_sid_for_new_standalone
(did)¶ Return True if
did
can be assigned to a new standalone object.
-
d1_gmn.app.did.
is_valid_pid_to_be_updated
(did)¶ Return True if
did
is the PID of an object that can be updated (obsoleted) with MNStorage.update()
-
d1_gmn.app.did.
is_valid_sid_for_chain
(pid, sid)¶ Return True if
sid
can be assigned to the single objectpid
or to the chain to whichpid
belongs.If the chain does not have a SID, the new SID must be previously unused.
If the chain already has a SID, the new SID must match the existing SID.
All known PIDs are associated with a chain.
Preconditions: -
pid
is verified to exist. E.g., withd1_gmn.app.views.asserts.is_existing_object().
sid
is None or verified to be a SID
-
d1_gmn.app.did.
get_did_by_foreign_key
(did_foreign_key)¶ Return the DID referenced by a ForeignKey or OneToOneField to IdNamespace.
Return None if ForeignKey or OneToOneField is NULL.
This is used instead of “did_foreign_key.*.did” on ForeignKeys and OneToOneFields that allow NULL (null=True in the model).
-
d1_gmn.app.did.
is_existing_object
(did)¶ Return True if PID is for an object for which science bytes are stored locally.
This excludes SIDs and PIDs for unprocessed replica requests, remote or non-existing revisions of local replicas and objects aggregated in Resource Maps.
-
d1_gmn.app.did.
is_sid
(did)¶
-
d1_gmn.app.did.
is_obsoleted
(did)¶ Return True if
did
is the PID of an object that has been obsoleted.
-
d1_gmn.app.did.
is_resource_map_db
(pid)¶
-
d1_gmn.app.did.
is_resource_map_member
(pid)¶
-
d1_gmn.app.did.
classify_identifier
(did)¶ Return a text fragment classifying the
did
Return <UNKNOWN> if the DID could not be classified. This should not normally happen and may indicate that the DID was orphaned in the database.
-
d1_gmn.app.did.
get_or_create_did
(id_str)¶
-
d1_gmn.app.did.
is_in_revision_chain
(sciobj_model)¶
-
d1_gmn.app.did.
is_archived
(pid)¶
-
d1_gmn.app.did.
is_local_replica
(pid)¶ Includes unprocessed replication requests.
-
d1_gmn.app.did.
is_unprocessed_local_replica
(pid)¶ Is local replica with status “queued”.
-
d1_gmn.app.did.
is_revision_chain_placeholder
(pid)¶ For replicas, the PIDs referenced in revision chains are reserved for use by other replicas.
d1_gmn.app.event_log module¶
d1_gmn.app.gmn module¶
App performing filesystem setup and basic sanity checks on configuration values in settings.py before GMN starts servicing requests.
Django loads apps into the Application Registry in the order specified in settings.INSTALLED_APPS. This app must be set to load before the main GMN app by listing it above the main app in settings.INSTALLED_APPS.
-
class
d1_gmn.app.gmn.
Startup
(app_name, app_module)¶ Bases:
django.apps.config.AppConfig
-
name
= 'd1_gmn.app'¶
-
ready
()¶ Called once per Django process instance.
If the filesystem setup fails or if an error is found in settings.py, django.core.exceptions.ImproperlyConfigured is raised, causing Django not to launch the main GMN app.
-
raise_config_error
(setting_name, cur_val, exp_type, valid_str=None, is_none_allowed=False)¶
-
d1_gmn.app.local_replica module¶
d1_gmn.app.model_util module¶
Database model utilities.
These are in a separate module because module classes can only be referenced in an active Django context. More general utilities can be used without an active context.
Importing this module outside of Django context raises django.core.exceptions.AppRegistryNotReady: Apps aren’t loaded yet.
-
d1_gmn.app.model_util.
get_sci_model
(pid)¶
-
d1_gmn.app.model_util.
get_pids_for_all_locally_stored_objects
()¶
-
d1_gmn.app.model_util.
delete_unused_subjects
()¶ Delete any unused subjects from the database.
This is not strictly required as any unused subjects will automatically be reused if needed in the future.
d1_gmn.app.models module¶
d1_gmn.app.node module¶
Generate Node document based on the current settings for GMN.
-
d1_gmn.app.node.
get_pretty_xml
(api_major_int=2)¶
-
d1_gmn.app.node.
get_xml
(api_major_int)¶
-
d1_gmn.app.node.
get_pyxb
(api_major_int=2)¶
d1_gmn.app.node_registry module¶
Node Registry cache.
Retrieve, hold and update a cache of the Node Registry for the DataONE environment in which this MN is registered.
Query the Node Registry.
-
d1_gmn.app.node_registry.
get_cn_subjects
()¶
-
d1_gmn.app.node_registry.
set_empty_cn_subjects_cache
()¶
-
d1_gmn.app.node_registry.
set_cn_subjects_for_environment
()¶
-
d1_gmn.app.node_registry.
get_cn_subjects_string
()¶
-
d1_gmn.app.node_registry.
get_cn_subjects_from_dataone_root
()¶
-
d1_gmn.app.node_registry.
download_node_registry
()¶
-
d1_gmn.app.node_registry.
create_root_cn_client
()¶
d1_gmn.app.object_format_cache module¶
d1_gmn.app.proxy module¶
Proxy mode.
-
d1_gmn.app.proxy.
get_sciobj_iter_remote
(url)¶
-
d1_gmn.app.proxy.
is_proxy_url
(url)¶
d1_gmn.app.psycopg_adapter module¶
Psycopg Postgres adapter for Python.
Registers custom adapters with Psycopg, which simplify reading and writing custom DataONE PyXB types to/from database models.
-
d1_gmn.app.psycopg_adapter.
adapt_pyxb_binding
(client)¶
d1_gmn.app.resource_map module¶
Utilities for manipulating resource maps.
-
d1_gmn.app.resource_map.
assert_map_is_valid_for_create
(resource_map)¶
-
d1_gmn.app.resource_map.
is_sciobj_valid_for_create
()¶ When RESOURCE_MAP_CREATE == ‘reserve’, objects that are created and that are also aggregated in one or more resource maps can only be created by a DataONE subject that has write or changePermission on the resource map.
-
d1_gmn.app.resource_map.
create_or_update_db
(sysmeta_pyxb)¶
-
d1_gmn.app.resource_map.
get_resource_map_from_sciobj
(pid)¶
-
d1_gmn.app.resource_map.
create_or_update
(map_pid, resource_map)¶
-
d1_gmn.app.resource_map.
get_resource_map_members
(pid)¶ pid
is the PID of a Resource Map or the PID of a member of a Resource Map.
-
d1_gmn.app.resource_map.
get_resource_map_members_by_map
(map_pid)¶
-
d1_gmn.app.resource_map.
get_resource_map_members_by_member
(member_pid)¶
-
d1_gmn.app.resource_map.
is_resource_map_sysmeta_pyxb
(sysmeta_pyxb)¶
-
d1_gmn.app.resource_map.
parse_resource_map_from_str
(resource_map_xml)¶
d1_gmn.app.revision module¶
Utilities for manipulating revision chains in the database.
-
d1_gmn.app.revision.
create_or_update_chain
(pid, sid, obsoletes_pid, obsoleted_by_pid)¶
-
d1_gmn.app.revision.
delete_chain
(pid)¶
-
d1_gmn.app.revision.
cut_from_chain
(sciobj_model)¶ Remove an object from a revision chain.
The object can be at any location in the chain, including the head or tail.
Preconditions: - The object with the pid is verified to exist and to be a member of an revision chain. E.g., with:
d1_gmn.app.views.asserts.is_existing_object(pid) d1_gmn.app.views.asserts.is_in_revision_chain(pid)
Postconditions: - The given object is a standalone object with empty obsoletes, obsoletedBy and
seriesId fields.
The previously adjacent objects in the chain are adjusted to close any gap that was created or remove dangling reference at the head or tail.
If the object was the last object in the chain and the chain has a SID, the SID reference is shifted over to the new last object in the chain.
-
d1_gmn.app.revision.
get_all_pid_by_sid
(sid)¶
-
d1_gmn.app.revision.
resolve_sid
(sid)¶ Get the PID to which the
sid
currently maps.Preconditions: -
sid
is verified to exist. E.g., with d1_gmn.app.views.asserts.is_sid().
-
d1_gmn.app.revision.
get_sid_by_pid
(pid)¶ Given the
pid
of the object in a chain, return the SID for the chain.Return None if there is no SID for the chain. This operation is also valid for standalone objects which may or may not have a SID.
This is the reverse of resolve.
All known PIDs are associated with a chain.
Preconditions: -
pid
is verified to exist. E.g., withd1_gmn.app.views.asserts.is_existing_object().
-
d1_gmn.app.revision.
set_revision_links
(sciobj_model, obsoletes_pid=None, obsoleted_by_pid=None)¶
-
d1_gmn.app.revision.
is_obsoletes_pid
(pid)¶ Return True if
pid
is referenced in the obsoletes field of any object.This will return True even if the PID is in the obsoletes field of an object that does not exist on the local MN, such as replica that is in an incomplete chain.
-
d1_gmn.app.revision.
is_obsoleted_by_pid
(pid)¶ Return True if
pid
is referenced in the obsoletedBy field of any object.This will return True even if the PID is in the obsoletes field of an object that does not exist on the local MN, such as replica that is in an incomplete chain.
-
d1_gmn.app.revision.
is_revision
(pid)¶ Return True if
pid
is referenced in the obsoletes or obsoletedBy field of any object.This will return True even if the PID is in the obsoletes field of an object that does not exist on the local MN, such as replica that is in an incomplete chain.
d1_gmn.app.scimeta module¶
Utilities for Science Metadata.
-
d1_gmn.app.scimeta.
assert_valid
(sysmeta_pyxb, pid)¶ Validate file at {sciobj_path} against schema selected via formatId and raise InvalidRequest if invalid.
Validation is only performed when:
SciMeta validation is enabled
and Object size is below size limit for validation
and formatId designates object as a Science Metadata object which is recognized and parsed by DataONE CNs
and XML Schema (XSD) files for formatId are present on local system
d1_gmn.app.sciobj_store module¶
Manage the filesystem tree in which science object bytes are stored.
Because it may be inefficient to store millions of files in a single folder and because such a folder is hard to deal with when performing backups and maintenance, GMN stores the objects in a folder hierarchy of 256 folders, each holding 256 folders, for a total of 65536 folders. The location in the hierarchy for a given object is based on its PID.
Folders are created as required in the hierarchy.
-
d1_gmn.app.sciobj_store.
open_sciobj_file_by_pid_ctx
(pid, write=False)¶ Open the file containing the Science Object bytes of
pid
in the default location within the tree of the local SciObj store.If
write
is True, the file is opened for writing and any missing directories are created. Return the file handle and file_url with the file location in a suitable form for storing in the DB.If nothing was written to the file, it is deleted.
-
d1_gmn.app.sciobj_store.
open_sciobj_file_by_path_ctx
(abs_path, write=False)¶ Open the file containing the Science Object bytes at the custom location
abs_path
in the local filesystem.If
write
is True, the file is opened for writing and any missing directores are created. Return the file handle and file_url with the file location in a suitable form for storing in the DB.If nothing was written to the file, delete it.
-
d1_gmn.app.sciobj_store.
get_sciobj_iter_by_url
(sciobj_url)¶
-
d1_gmn.app.sciobj_store.
get_sciobj_byte_iterator_by_url
(sciobj_url)¶
-
d1_gmn.app.sciobj_store.
get_sciobj_iter_by_pid
(pid)¶
-
d1_gmn.app.sciobj_store.
open_sciobj_file_by_pid
(pid, write=False)¶ Open the file containing the Science Object bytes at the custom location
abs_path
in the local filesystem for read.
-
d1_gmn.app.sciobj_store.
open_sciobj_file_by_path
(abs_path, write=False)¶ Open a SciObj file for read or write. If opened for write, create any missing directories. For a SciObj stored in the default SciObj store, the path includes the PID hash based directory levels.
This is the only method in GMN that opens SciObj files, so can be modified to customize the SciObj storage locations and can be mocked for testing.
Note that when a SciObj is created by a client via MNStorage.create(), Django streams the SciObj bytes to a temporary file or memory location as set by
FILE_UPLOAD_TEMP_DIR
and related settings.
-
d1_gmn.app.sciobj_store.
get_rel_sciobj_file_path
(pid)¶ Get the relative local path to the file holding an object’s bytes.
The path is relative to settings.OBJECT_STORE_PATH
There is a one-to-one mapping between pid and path
The path is based on a SHA1 hash. It’s now possible to craft SHA1 collisions, but it’s so unlikely that we ignore it for now
The path may or may not exist (yet).
-
d1_gmn.app.sciobj_store.
get_abs_sciobj_file_path_by_pid
(pid)¶ Get the absolute local path to the file holding an object’s bytes.
The path is to a location below settings.OBJECT_STORE_PATH
There is a one-to-one mapping between pid and path
The path is based on a SHA1 hash. It’s now possible to craft SHA1 collisions, but it’s so unlikely that we ignore it for now
The path may or may not exist (yet).
-
d1_gmn.app.sciobj_store.
get_abs_sciobj_file_path_by_rel_path
(rel_path)¶ Get the absolute local path to the file holding an object’s bytes.
The path is to a location below settings.OBJECT_STORE_PATH
There is a one-to-one mapping between pid and path
The path is based on a SHA1 hash. It’s now possible to craft SHA1 collisions, but it’s so unlikely that we ignore it for now
The path may or may not exist (yet).
-
d1_gmn.app.sciobj_store.
get_abs_sciobj_store_path
()¶ Get the absolute local path to the root of the default SciObj store.
The path may or may not exist (yet).
-
d1_gmn.app.sciobj_store.
assert_sciobj_store_exists
()¶
-
d1_gmn.app.sciobj_store.
assert_sciobj_store_does_not_exist
()¶
-
d1_gmn.app.sciobj_store.
is_existing_store
()¶
-
d1_gmn.app.sciobj_store.
is_existing_sciobj_file
(pid)¶
-
d1_gmn.app.sciobj_store.
get_rel_sciobj_file_url_by_pid
(pid)¶ Get the URL that will be stored in the database for a SciObj that is saved in GMN’s SciObj filesystem hierarchy below settings.OBJECT_STORE_PATH.
-
d1_gmn.app.sciobj_store.
get_abs_sciobj_file_url
(abs_sciobj_file_path)¶ Get the URL that will be stored in the database for a SciObj that is saved in a custom location outside of GMN’s SciObj filesystem hierarchy.
-
d1_gmn.app.sciobj_store.
get_abs_sciobj_file_path_by_url
(file_url)¶ Get the absolute path to the file holding an object’s bytes.
file_url
is an absolute or relative file:// url as stored in the DB.
-
d1_gmn.app.sciobj_store.
get_gmn_version
()¶
-
d1_gmn.app.sciobj_store.
is_matching_version
()¶
-
d1_gmn.app.sciobj_store.
is_lower_version
()¶
-
d1_gmn.app.sciobj_store.
is_store_subdir
(dir_path)¶
-
d1_gmn.app.sciobj_store.
get_store_version
()¶
-
d1_gmn.app.sciobj_store.
save_store_version
()¶
-
d1_gmn.app.sciobj_store.
get_store_version_path
()¶
-
d1_gmn.app.sciobj_store.
create_store
()¶
-
d1_gmn.app.sciobj_store.
is_empty
()¶
-
d1_gmn.app.sciobj_store.
is_tmp
()¶
-
d1_gmn.app.sciobj_store.
assert_sciobj_store_version_match
()¶
-
d1_gmn.app.sciobj_store.
delete_sciobj
(url_split, pid)¶
d1_gmn.app.settings_default module¶
Default settings for GMN.
These settings are only used as fallbacks in case the corresponding settings in settings.py are missing.
This allows settings to be added without having to modify settings.py in existing deployments.
Settings that are not described here are described in settings_template.py.
d1_gmn.app.sysmeta module¶
d1_gmn.app.sysmeta_extract module¶
d1_gmn.app.urls module¶
d1_gmn.app.util module¶
General utilities.
-
d1_gmn.app.util.
coerce_put_post
(request)¶ Django doesn’t particularly understand REST. In case we send data over PUT, Django won’t actually look at the data and load it. We need to twist its arm here.
The try/except abomination here is due to a bug in mod_python. This should fix it.
From django-piston/piston/utils.py
-
d1_gmn.app.util.
dump_stack
()¶
-
d1_gmn.app.util.
create_http_echo_response
(request)¶
-
d1_gmn.app.util.
get_static_path
(rel_path)¶