d1_client package¶

DataONE Client Library.

The DataONE Client Library for Python works together with the DataONE Common Library for Python to provide functionality commonly needed by client software that connects to DataONE nodes.

The main functionality provided by this library is a complete set of wrappers for all DataONE API methods. There are many details related to interacting with the DataONE API, such as creating MIME multipart messages, encoding parameters into URLs and handling Unicode. The wrappers hide these details, allowing the developer to communicate with nodes by calling native Python methods which take and return native objects.

The wrappers also convert any errors received from the nodes into native exceptions, enabling clients to use Python’s concise exception handling system to handle errors.

Although this directory is not a package, this __init__.py file is required for pytest to be able to reach test directories below this directory.

Subpackages¶

d1_client.iter package

Submodules¶

d1_client.baseclient module¶

class d1_client.baseclient.DataONEBaseClient(base_url, *args, **kwargs)¶

Bases: d1_client.session.Session

Extend Session by adding REST API wrappers for APIs that are available on both Coordinating Nodes and Member Nodes, and that have the same signature on both:

CNCore/MNCore.getLogRecords() CNRead/MNRead.get() CNRead/MNRead.getSystemMetadata() CNRead/MNRead.describe() CNRead/MNRead.listObjects() CNAuthorization/MNAuthorization.isAuthorized() CNCore/MNCore.ping()

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/MN_APIs.html https://releases.dataone.org/online/api-documentation-v2.0/apis/CN_APIs.html

On error response, raises a DataONEException.

Methods with names that end in “Response” return the HTTPResponse object directly for manual processing by the client. The *Response methods are only needed in rare cases where the default handling is inadequate, e.g., for dealing with nodes that don’t fully comply with the spec.

The client classes wrap all the DataONE API methods, hiding the many details related to interacting with the DataONE API, such as creating MIME multipart messages, encoding parameters into URLs and handling Unicode.

The clients allow the developer to communicate with nodes by calling native Python methods which take and return native objects.

The clients also convert any errors received from the nodes into native exceptions, enabling clients to use Python’s concise exception handling system to handle errors.

The clients are arranged into the following class hierarchy:

$digraph G { dpi = 72; edge [dir = back]; Session -> DataONEBaseClient; DataONEBaseClient -> DataONEBaseClient_1_1 [weight=1000]; DataONEBaseClient -> MemberNodeClient; DataONEBaseClient -> CoordinatingNodeClient; DataONEBaseClient_1_1 -> MemberNodeClient_1_1; DataONEBaseClient_1_1 -> CoordinatingNodeClient_1_1; MemberNodeClient -> MemberNodeClient_1_1; CoordinatingNodeClient -> CoordinatingNodeClient_1_1; MemberNodeClient -> DataONEBaseClient_1_1 [style=invis]; CoordinatingNodeClient -> DataONEBaseClient_1_1 [style=invis]; DataONEBaseClient_1_1 -> DataONEBaseClient_2_0 [weight=1000]; MemberNodeClient_1_1 -> DataONEBaseClient_2_0 [style=invis]; CoordinatingNodeClient_1_1 -> DataONEBaseClient_2_0 [style=invis]; DataONEBaseClient_2_0 -> MemberNodeClient_2_0; DataONEBaseClient_2_0 -> CoordinatingNodeClient_2_0; MemberNodeClient_1_1 -> MemberNodeClient_2_0; CoordinatingNodeClient_1_1 -> CoordinatingNodeClient_2_0; MemberNodeClient_2_0 -> DataONEClient; CoordinatingNodeClient_2_0 -> DataONEClient; }$

The classes without version designators implement functionality defined in v1.0 of the DataONE service specifications. The classes with version designators implement support for the corresponding DataONE service specifications.

DataONEBaseClient

The DataONEBaseClient classes contain methods that allow access to APIs that are common to Coordinating Nodes and Member Nodes.

d1_client.d1baseclient

d1_client.d1baseclient_1_1

d1_client.d1baseclient_2_0

MemberNodeClient

The MemberNodeClient classes contain methods that allow access to APIs that are specific to Member Nodes.

d1_client.mnclient

d1_client.mnclient_1_1

d1_client.mnclient_2_0

CoordinatingNodeClient

The CoordinatingNodeClient classes contain methods that allow access to APIs that are specific to Coordinating Nodes.

d1_client.cnclient

d1_client.cnclient_1_1

d1_client.cnclient_2_0

DataONEClient

The DataONEClient uses CN- and MN clients to perform high level operations against the DataONE infrastructure.

d1_client.d1client

DataONEObject

Wraps a single DataONE Science Object and adds functionality such as resolve and get.

d1_client.d1client

SolrConnection

Provides functionality for working with DataONE’s Solr index, which powers the ONEMercury science data search engine.

d1_client.solr_client

__init__(base_url, *args, **kwargs)¶

Create a DataONEBaseClient. See Session for parameters.

Parameters

api_major (integer) – Major version of the DataONE API
api_minor (integer) – Minor version of the DataONE API

Returns

None

api_version_tup¶

pyxb_binding¶

getLogRecordsResponse(fromDate=None, toDate=None, event=None, pidFilter=None, idFilter=None, start=0, count=100, vendorSpecific=None)¶

getLogRecords(fromDate=None, toDate=None, event=None, pidFilter=None, idFilter=None, start=0, count=100, vendorSpecific=None)¶

pingResponse(vendorSpecific=None)¶

ping(vendorSpecific=None)¶

getResponse(pid, stream=False, vendorSpecific=None)¶

get(pid, stream=False, vendorSpecific=None)¶

Initiate a MNRead.get(). Return a Requests Response object from which the object bytes can be retrieved.

When stream is False, Requests buffers the entire object in memory before returning the Response. This can exhaust available memory on the local machine when retrieving large science objects. The solution is to set stream to True, which causes the returned Response object to contain a a stream. However, see note below.

When stream = True, the Response object will contain a stream which can be processed without buffering the entire science object in memory. However, failure to read all data from the stream can cause connections to be blocked. Due to this, the stream parameter is False by default.

Also see:

http://docs.python-requests.org/en/master/user/advanced/body-content-workflow
get_and_save() in this module.

get_and_save(pid, sciobj_path, create_missing_dirs=False, vendorSpecific=None)¶

Like MNRead.get(), but also retrieve the object bytes and store them in a local file at sciobj_path. This method does not have the potential issue with excessive memory usage that get() with ``stream``=False has.

Also see MNRead.get().

getSystemMetadataResponse(pid, vendorSpecific=None)¶

getSystemMetadata(pid, vendorSpecific=None)¶

describeResponse(pid, vendorSpecific=None)¶

describe(pid, vendorSpecific=None)¶: Note: If the server returns a status code other than 200 OK, a ServiceFailure will be raised, as this method is based on a HEAD request, which cannot carry exception information.

listObjectsResponse(fromDate=None, toDate=None, formatId=None, identifier=None, replicaStatus=None, nodeId=None, start=0, count=100, vendorSpecific=None)¶

listObjects(fromDate=None, toDate=None, formatId=None, identifier=None, replicaStatus=None, nodeId=None, start=0, count=100, vendorSpecific=None)¶

generateIdentifierResponse(scheme, fragment=None, vendorSpecific=None)¶

generateIdentifier(scheme, fragment=None, vendorSpecific=None)¶

archiveResponse(pid, vendorSpecific=None)¶

archive(pid, vendorSpecific=None)¶

isAuthorizedResponse(pid, action, vendorSpecific=None)¶

isAuthorized(pid, action, vendorSpecific=None)¶: Return True if user is allowed to perform action on pid, else False.

d1_client.baseclient_1_1 module¶

class d1_client.baseclient_1_1.DataONEBaseClient_1_1(*args, **kwargs)¶

Bases: d1_client.baseclient.DataONEBaseClient

Extend DataONEBaseClient with functionality common between Member and Coordinating nodes that was added in v1.1 of the DataONE infrastructure.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/MN_APIs.html https://releases.dataone.org/online/api-documentation-v2.0/apis/CN_APIs.html

__init__(*args, **kwargs)¶: See d1_client.baseclient.DataONEBaseClient for args.

queryResponse(queryEngine, query_str, vendorSpecific=None, do_post=False, **kwargs)¶

CNRead.query(session, queryEngine, query) → OctetStream https://releases.dataone.org/online/api- documentation-v2.0.1/apis/CN_APIs.html#CNRead.query MNQuery.query(session, queryEngine, query) → OctetStream http://jenkins.

-1.dataone.org/jenkins/job/API%20Documentation%20-%20trunk/ws/api- documentation/build/html/apis/MN_APIs.html#MNQuery.query.

Parameters

queryEngine
query_str
vendorSpecific
do_post
**kwargs

Returns:

query(queryEngine, query_str, vendorSpecific=None, do_post=False, **kwargs)¶

d1_client.baseclient_1_2 module¶

class d1_client.baseclient_1_2.DataONEBaseClient_1_2(*args, **kwargs)¶

Bases: d1_client.baseclient_1_1.DataONEBaseClient_1_1

Extend DataONEBaseClient with functionality common between Member and Coordinating nodes that was added in v1.1 of the DataONE infrastructure.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/MN_APIs.html https://releases.dataone.org/online/api-documentation-v2.0/apis/CN_APIs.html

__init__(*args, **kwargs)¶: See d1_client.baseclient.DataONEBaseClient for args.

d1_client.baseclient_2_0 module¶

class d1_client.baseclient_2_0.DataONEBaseClient_2_0(*args, **kwargs)¶

Bases: d1_client.baseclient_1_2.DataONEBaseClient_1_2

Extend DataONEBaseClient_1_2 with functionality common between Member and Coordinating nodes that was added in v2.0 of the DataONE infrastructure.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/MN_APIs.html https://releases.dataone.org/online/api-documentation-v2.0/apis/CN_APIs.html

__init__(*args, **kwargs)¶: See baseclient.DataONEBaseClient for args.

updateSystemMetadataResponse(pid, sysmeta_pyxb, vendorSpecific=None)¶

MNStorage.updateSystemMetadata(session, pid, sysmeta) → boolean http://jenkins-1.dataone.org/documentation/unstable/API-Documentation- development/apis/MN_APIs.html#MNStorage.updateSystemMetadata.

Parameters

pid
sysmeta_pyxb
vendorSpecific

Returns:

updateSystemMetadata(pid, sysmeta_pyxb, vendorSpecific=None)¶

d1_client.cnclient module¶

class d1_client.cnclient.CoordinatingNodeClient(*args, **kwargs)¶

Bases: d1_client.baseclient.DataONEBaseClient

Extend DataONEBaseClient by adding REST API wrappers for APIs that are available on Coordinating Nodes.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/CN_APIs.html

__init__(*args, **kwargs)¶: See d1_client.baseclient.DataONEBaseClient for args.

listFormatsResponse(vendorSpecific=None)¶

CNCore.ping() → null https://releases.dataone.org/online/api- documentation-v2.0.1/apis/CN_APIs.html#CNCore.ping Implemented in d1_client.baseclient.py.

CNCore.create(session, pid, object, sysmeta) → Identifier https://releases.dataone.org/online/api-documentation-v2.0.1/apis/CN_APIs.html#CNCore.create CN INTERNAL

CNCore.listFormats() → ObjectFormatList https://releases.dataone.org/online/api-documentation-v2.0.1/apis/CN_APIs.html#CNCore.listFormats

Parameters: vendorSpecific

Returns:

listFormats(vendorSpecific=None)¶

d1_client.cnclient_1_1 module¶

class d1_client.cnclient_1_1.CoordinatingNodeClient_1_1(*args, **kwargs)¶

Bases: d1_client.baseclient_1_1.DataONEBaseClient_1_1, d1_client.cnclient.CoordinatingNodeClient

Extend DataONEBaseClient_1_1 and CoordinatingNodeClient with functionality for Coordinating nodes that was added in v1.1 of the DataONE infrastructure.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/CN_APIs.html

__init__(*args, **kwargs)¶: See baseclient.DataONEBaseClient for args.

d1_client.cnclient_1_2 module¶

class d1_client.cnclient_1_2.CoordinatingNodeClient_1_2(*args, **kwargs)¶

Bases: d1_client.baseclient_1_2.DataONEBaseClient_1_2, d1_client.cnclient.CoordinatingNodeClient

Extend DataONEBaseClient_1_2 and CoordinatingNodeClient with functionality for Coordinating nodes that was added in v1.1 of the DataONE infrastructure.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/CN_APIs.html

__init__(*args, **kwargs)¶: See baseclient.DataONEBaseClient for args.

d1_client.cnclient_2_0 module¶

class d1_client.cnclient_2_0.CoordinatingNodeClient_2_0(*args, **kwargs)¶

Bases: d1_client.baseclient_2_0.DataONEBaseClient_2_0, d1_client.cnclient_1_2.CoordinatingNodeClient_1_2

Extend DataONEBaseClient_2_0 and CoordinatingNodeClient_1_2 with functionality for Coordinating nodes that was added in v2.0 of the DataONE infrastructure.

Updated in v2:

CNCore.listFormats() → ObjectFormatList
CNRead.listObjects(session[, fromDate][, toDate][, formatId]
MNRead.listObjects(session[, fromDate][, toDate][, formatId]

The base implementations of listFormats() and listObjects() handle v2 when called through this class.

https://releases.dataone.org/online/api-documentation-v2.0/apis/CN_APIs.html

__init__(*args, **kwargs)¶: See baseclient.DataONEBaseClient for args.

deleteResponse(pid)¶

CNCore.delete(session, id) → Identifier DELETE /object/{id}

Parameters: pid

Returns:

delete(pid)¶

d1_client.d1client module¶

Perform high level operations against the DataONE infrastructure.

The other Client classes are specific to CN or MN and to architecture version. This class provides a more abstract interface that can be used for interacting with any DataONE node regardless of type and version.

class d1_client.d1client.DataONEClient¶: Bases: object

d1_client.d1client.get_api_major_by_base_url(base_url, *client_arg_list, **client_arg_dict)¶

Read the Node document from a node and return an int containing the latest D1 API version supported by the node.

The Node document can always be reached through the v1 API and will list services for v1 and any later APIs versions supported by the node.

d1_client.d1client.get_client_type(d1_client_obj)¶

d1_client.d1client.get_version_tag_by_d1_client(d1_client_obj)¶

d1_client.d1client.get_client_class_by_version_tag(api_major)¶

d1_client.mnclient module¶

class d1_client.mnclient.MemberNodeClient(*args, **kwargs)¶

Bases: d1_client.baseclient.DataONEBaseClient

Extend DataONEBaseClient by adding REST API wrappers for APIs that are available on Member Nodes.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/MN_APIs.html

__init__(*args, **kwargs)¶: See baseclient.DataONEBaseClient for args.

getCapabilitiesResponse(vendorSpecific=None)¶

getCapabilities(vendorSpecific=None)¶

getChecksumResponse(pid, checksumAlgorithm=None, vendorSpecific=None)¶

getChecksum(pid, checksumAlgorithm=None, vendorSpecific=None)¶

synchronizationFailedResponse(message, vendorSpecific=None)¶

synchronizationFailed(message, vendorSpecific=None)¶

createResponse(pid, obj, sysmeta_pyxb, vendorSpecific=None)¶

create(pid, obj, sysmeta_pyxb, vendorSpecific=None)¶

updateResponse(pid, obj, newPid, sysmeta_pyxb, vendorSpecific=None)¶

update(pid, obj, newPid, sysmeta_pyxb, vendorSpecific=None)¶

deleteResponse(pid, vendorSpecific=None)¶

delete(pid, vendorSpecific=None)¶

systemMetadataChangedResponse(pid, serialVersion, dateSysMetaLastModified, vendorSpecific=None)¶

systemMetadataChanged(pid, serialVersion, dateSysMetaLastModified, vendorSpecific=None)¶

replicateResponse(sysmeta_pyxb, sourceNode, vendorSpecific=None)¶

replicate(sysmeta_pyxb, sourceNode, vendorSpecific=None)¶

getReplicaResponse(pid, vendorSpecific=None)¶

getReplica(pid, vendorSpecific=None)¶

d1_client.mnclient_1_1 module¶

class d1_client.mnclient_1_1.MemberNodeClient_1_1(*args, **kwargs)¶

Bases: d1_client.baseclient_1_1.DataONEBaseClient_1_1, d1_client.mnclient.MemberNodeClient

Extend DataONEBaseClient_1_1 and MemberNodeClient with functionality for Member nodes that was added in v1.1 of the DataONE infrastructure.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/MN_APIs.html

__init__(*args, **kwargs)¶: See baseclient.DataONEBaseClient for args.

d1_client.mnclient_1_2 module¶

class d1_client.mnclient_1_2.MemberNodeClient_1_2(*args, **kwargs)¶

Bases: d1_client.baseclient_1_2.DataONEBaseClient_1_2, d1_client.mnclient.MemberNodeClient

Extend DataONEBaseClient_1_2 and MemberNodeClient with functionality for Member nodes that was added in v1.2 of the DataONE infrastructure.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/MN_APIs.html

__init__(*args, **kwargs)¶: See baseclient.DataONEBaseClient for args.

viewResponse(theme, did, **kwargs)¶

view(theme, did, **kwargs)¶

listViewsResponse(**kwargs)¶

listViews(**kwargs)¶

getPackageResponse(did, packageType='application/bagit-097', **kwargs)¶

getPackage(did, packageType='application/bagit-097', **kwargs)¶

d1_client.mnclient_2_0 module¶

class d1_client.mnclient_2_0.MemberNodeClient_2_0(*args, **kwargs)¶

Bases: d1_client.baseclient_2_0.DataONEBaseClient_2_0, d1_client.mnclient_1_2.MemberNodeClient_1_2

Extend DataONEBaseClient_2_0 and MemberNodeClient_1_2 with functionality for Member nodes that was added in v2.0 of the DataONE infrastructure.

For details on how to use these methods, see:

https://releases.dataone.org/online/api-documentation-v2.0/apis/MN_APIs.html

__init__(*args, **kwargs)¶: See baseclient.DataONEBaseClient for args.

d1_client.object_format_info module¶

d1_client.session module¶

class d1_client.session.Session(base_url, cert_pem_path=None, cert_key_path=None, **kwargs_dict)¶

Bases: object

__init__(base_url, cert_pem_path=None, cert_key_path=None, **kwargs_dict)¶

The Session improves performance by keeping connection related state and allowing it to be reused in multiple API calls to a DataONE Coordinating Node or Member Node. This includes:

A connection pool
HTTP persistent connections (HTTP/1.1 and keep-alive)

Based on Python Requests: - http://docs.python-requests.org/en/master/ - http://docs.python-requests.org/en/master/user/advanced/#session-objects

Parameters

base_url – DataONE Node REST service BaseURL.
cert_pem_path – Path to a PEM formatted certificate file. If provided and

accepted by the remote node, the subject for which the certificate was issued is added to the authenticated context in which API calls are made by the client. Equivalent subjects and group subjects may be implicitly included as well. If the certificate is used together with an JWT token, the two sets of subjects are combined. :type cert_pem_path: string

Parameters

cert_key_path (string) – Path to a PEM formatted file that contains the private key for the certificate file. Only required if the certificate file does not itself contain the private key.
jwt_token – Base64 encoded JSON Web Token. If provided and accepted by the

remote node, the subject for which the token was issued is added to the authenticated context in which API calls are made by the client. Equivalent subjects and group subjects may be implicitly included as well. If the token is used together with an X.509 certificate, the two sets of subjects are combined. :type token: string

Parameters

timeout_sec (float, int, None) – Time in seconds that requests will wait for a response. None, 0, 0.0 disables timeouts. Default is DEFAULT_HTTP_TIMEOUT, currently 60 seconds.
retries (int) – Set number of times to try a request before failing. If not set, retries are still performed, using the default number of retries. To disable retries, set to 1.
headers (dictionary) – headers that will be included with all connections.
query (dictionary) – URL query parameters that will be included with all connections.
use_stream (bool) – Use streaming response. When enabled, responses must be completely read to free up the connection for reuse. (default:False)
verify_tls (bool or path) – Verify the server side TLS/SSL certificate. (default: True). Can also hold a path that points to a trusted CA bundle
suppress_verify_warnings (bool) – Suppress the warnings issued when verify_tls is set to False.
user_agent (str) – Override the default User-Agent string used by d1client.
charset (str) – Override the default Charset used by d1client. (default: utf-8)

Returns

None

base_url¶

GET(rest_path_list, **kwargs)¶

Send a GET request. See requests.sessions.request for optional parameters.

Returns: Response object

HEAD(rest_path_list, **kwargs)¶

Send a HEAD request. See requests.sessions.request for optional parameters.

Returns: Response object

POST(rest_path_list, **kwargs)¶

Send a POST request with optional streaming multipart encoding. See requests.sessions.request for optional parameters. To post regular data, pass a string, iterator or generator as the data argument. To post a multipart stream, pass a dictionary of multipart elements as the fields argument. E.g.:

fields = {: ‘field0’: ‘value’, ‘field1’: ‘value’, ‘field2’: (‘filename.xml’, open(‘file.xml’, ‘rb’), ‘application/xml’)

}

Returns: Response object

PUT(rest_path_list, **kwargs)¶

Send a PUT request with optional streaming multipart encoding. See requests.sessions.request for optional parameters. See post() for parameters.

Returns: Response object

DELETE(rest_path_list, **kwargs)¶

Send a DELETE request. See requests.sessions.request for optional parameters.

Returns: Response object

OPTIONS(rest_path_list, **kwargs)¶

Send a OPTIONS request. See requests.sessions.request for optional parameters.

Returns: Response object

get_curl_command_line(method, url, **kwargs)¶: Get request as cURL command line for debugging.

dump_request_and_response(response)¶

Return a string containing a nicely formatted representation of the request and response objects for logging and debugging.

Note: Does not work if the request or response body is a MultipartEncoder object.

d1_client.solr_client module¶

Basic Solr client.

Based on: http://svn.apache.org/viewvc/lucene/solr/tags/release-1.2.0/ client/python/solr.py

DataONE provides an index of all objects stored in the Member Nodes that form the DataONE federation. The index is stored in an Apache Solr database and can be queried with the SolrClient.

The DataONE Solr index provides information only about objects for which the caller has access. When querying the index without authenticating, only records related to public objects can be retrieved. To authenticate, provide a certificate signed by CILogon when creating the client.

Example:

# Connect to the DataONE Coordinating Nodes in the default (production) environment.
c = d1_client.solr_client.SolrConnection()

search_result = c.search({
  'q': 'id:[* TO *]', # Filter for search
  'rows': 10, # Number of results to return
  'fl': 'formatId', # List of fields to return for each result
})

pprint.pprint(search_result)

class d1_client.solr_client.SolrClient(*args, **kwargs)¶

Bases: d1_client.baseclient_1_2.DataONEBaseClient_1_2

Extend DataONEBaseClient_1_2 with functions for querying Solr indexes hosted on CNs and MNs.

Example:

solr_client = SolrClient(‘https://cn.dataone.org/cn’)

For the supported keyword args, see:

d1_client.session.Session()

Most methods take a **query_dict as a parameter. It allows passing any number of query parameters that will be sent to Solr.

Pass the query parameters as regular keyword arguments. E.g.:

solr_client.search(q=’id:abc*’, fq=’id:def*’)

To pass multiple query parameters of the same type, pass a list. E.g., to pass multiple filter query (fq) parameters:

solr_client.search(q=’id:abc*’, fq=[‘id:def*’, ‘id:ghi’])

Do not encode the query parameters before passing them to the methods.

For more information about DataONE’s Solr index, see:

https://releases.dataone.org/online/api-documentation-v2.0/design/SearchMetadata.html

search(**query_dict)¶

Search the Solr index.

Example:

result_dict = search(q=[‘id:abc*’], fq=[‘id:def*’, ‘id:ghi’])

get(doc_id)¶: Retrieve the specified document.

get_ids(start=0, rows=1000, **query_dict)¶: Retrieve a list of identifiers for documents matching the query.

count(**query_dict)¶: Return the number of entries that match query.

get_field_values(name, maxvalues=-1, sort=True, **query_dict)¶

Retrieve the unique values for a field, along with their usage counts.

Parameters

name (string) – Name of field for which to retrieve values
sort – Sort the result
maxvalues (int) – Maximum number of values to retrieve. Default is -1, which causes retrieval of all values.

Returns

dict of {fieldname: [[value, count], … ], }

get_field_min_max(name, **query_dict)¶

Returns the minimum and maximum values of the specified field. This requires two search calls to the service, each requesting a single value of a single field.

@param name(string) Name of the field @param q(string) Query identifying range of records for min and max values @param fq(string) Filter restricting range of query

@return list of [min, max]

field_alpha_histogram(name, n_bins=10, include_queries=True, **query_dict)¶

Generates a histogram of values from a string field.

Output is: [[low, high, count, query], … ]. Bin edges is determined by equal division of the fields.

delete(doc_id)¶

delete_by_query(query)¶

add(**fields)¶

add_docs(docs)¶: docs is a list of fields that are a dictionary of name:value for a record.

commit(waitFlush=True, waitSearcher=True, optimize=False)¶

class d1_client.solr_client.SolrRecordTransformerBase¶

Bases: object

Base for Solr record transformers.

Used to transform a Solr search response document into some other form, such as a dictionary or list of values.

transform(record)¶

class d1_client.solr_client.SolrArrayTransformer(cols=None)¶

Bases: d1_client.solr_client.SolrRecordTransformerBase

A transformer that returns a list of values for the specified columns.

transform(record)¶

class d1_client.solr_client.SolrSearchResponseIterator(client, page_size=100, max_records=1000, transformer=<d1_client.solr_client.SolrRecordTransformerBase object>, **query_dict)¶

Bases: object

Performs a search against a Solr index and acts as an iterator to retrieve all the values.

process_row(row)¶: Override this method in derived classes to reformat the row response.

class d1_client.solr_client.SolrArrayResponseIterator(client, page_size=100, cols=None, **query_dict)¶

Bases: d1_client.solr_client.SolrSearchResponseIterator

Returns an iterator that operates on a Solr result set.

The output for each document is a list of values for the columns specified in the cols parameter of the constructor.

class d1_client.solr_client.SolrSubsampleResponseIterator(client, q, fq=None, fields='*', page_size=100, n_samples=10000, transformer=<d1_client.solr_client.SolrRecordTransformerBase object>)¶

Bases: d1_client.solr_client.SolrSearchResponseIterator

Returns a pseudo-random subsample of the result set.

Works by calculating the number of pages required for the entire data set and taking a random sample of pages until n_samples can be retrieved. So pages are random, but records within a page are not.

class d1_client.solr_client.SolrValuesResponseIterator(client, field, page_size=1000, **query_dict)¶

Bases: object

Iterates over a Solr get values response.

This returns a list of distinct values for a particular field.

__init__(client, field, page_size=1000, **query_dict)¶

Initialize.

@param client(SolrConnection) An instance of a solr connection to use. @param field(string) name of the field from which to retrieve values @param q(string) The Solr query to restrict results @param fq(string) A facet query, restricts the set of rows that q is applied to @param fields(string) A comma delimited list of field names to return @param page_size(int) Number of rows to retrieve in each call.

d1_client.util module¶

d1_client.util.normalize_request_response_dump(dump_str)¶

d1_client package¶

Subpackages¶

Submodules¶

d1_client.baseclient module¶

d1_client.baseclient_1_1 module¶

d1_client.baseclient_1_2 module¶

d1_client.baseclient_2_0 module¶

d1_client.cnclient module¶

d1_client.cnclient_1_1 module¶

d1_client.cnclient_1_2 module¶

d1_client.cnclient_2_0 module¶

d1_client.d1client module¶

d1_client.mnclient module¶

d1_client.mnclient_1_1 module¶

d1_client.mnclient_1_2 module¶

d1_client.mnclient_2_0 module¶

d1_client.object_format_info module¶

d1_client.session module¶

d1_client.solr_client module¶

d1_client.util module¶

Table of Contents

Previous topic

Next topic

This Page