weaver.utils
¶
Module Contents¶
-
class
weaver.utils.
_Singleton
[source]¶ type(object_or_name, bases, dict) type(object) -> the object’s type type(name, bases, dict) -> a new type
Initialize self. See help(type(self)) for accurate signature.
-
weaver.utils.
get_weaver_url
(container: AnySettingsContainer) → AnyStr[source]¶ Retrieves the home URL of the weaver application.
-
weaver.utils.
get_any_id
(info: JSON) → Union[AnyStr, None][source]¶ Retrieves a dictionary id-like key using multiple common variations
[id, identifier, _id]
. :param info: dictionary that potentially contains an id-like key. :returns: value of the matched id-like key orNone
if not found.
-
weaver.utils.
get_any_value
(info: JSON) → AnyValue[source]¶ Retrieves a dictionary value-like key using multiple common variations
[href, value, reference]
. :param info: dictionary that potentially contains a value-like key. :returns: value of the matched value-like key orNone
if not found.
-
weaver.utils.
get_any_message
(info: JSON) → AnyStr[source]¶ Retrieves a dictionary ‘value’-like key using multiple common variations [message]. :param info: dictionary that potentially contains a ‘message’-like key. :returns: value of the matched ‘message’-like key or an empty string if not found.
-
weaver.utils.
get_registry
(container: AnyRegistryContainer) → Registry[source]¶ Retrieves the application
registry
from various containers referencing to it.
-
weaver.utils.
get_settings
(container: AnySettingsContainer) → SettingsType[source]¶ Retrieves the application
settings
from various containers referencing to it.
-
weaver.utils.
get_header
(header_name: AnyStr, header_container: AnyHeadersContainer) → Union[AnyStr, None][source]¶ Searches for the specified header by case/dash/underscore-insensitive
header_name
insideheader_container
.
Looks for
cookie_header_name
header withinheader_container
. :returns: new header container in the form{'Cookie': <found_cookie>}
if it was matched, or empty otherwise.
-
weaver.utils.
get_url_without_query
(url: Union[AnyStr, ParseResult]) → AnyStr[source]¶ Removes the query string part of an URL.
-
weaver.utils.
parse_extra_options
(option_str)[source]¶ Parses the extra options parameter.
The option_str is a string with coma separated
opt=value
pairs. Example:tempdir=/path/to/tempdir,archive_root=/path/to/archive
- Parameters
option_str – A string parameter with the extra options.
- Returns
A dict with the parsed extra options.
-
weaver.utils.
fully_qualified_name
(obj: Union[Any, Type[Any]]) → str[source]¶ Obtains the
'<module>.<name>'
full path definition of the object to allow finding and importing it.
-
weaver.utils.
localize_datetime
(dt: datetime, tz_name: Optional[AnyStr] = 'UTC') → datetime[source]¶ Provide a timezone-aware object for a given datetime and timezone name
-
weaver.utils.
ows_context_href
(href: AnyStr, partial: Optional[bool] = False) → JSON[source]¶ Returns the complete or partial dictionary defining an
OWSContext
from a reference.
-
weaver.utils.
pass_http_error
(exception: Exception, expected_http_error: Union[PyramidHTTPError, Iterable[PyramidHTTPError]]) → None[source]¶ Given an HTTPError of any type (pyramid, requests), ignores (pass) the exception if the actual error matches the status code. Other exceptions are re-raised.
- Parameters
exception – any Exception instance (“object” from a try..except exception as “object” block).
expected_http_error – single or list of specific pyramid HTTPError to handle and ignore.
- Raises
exception – if it doesn’t match the status code or is not an HTTPError of any module.
-
weaver.utils.
raise_on_xml_exception
(xml_node)[source]¶ Raises an exception with the description if the XML response document defines an ExceptionReport. :param xml_node: instance of
etree.Element
:raise Exception: on found ExceptionReport document.
-
weaver.utils.
str2bytes
(string: Union[AnyStr, bytes]) → bytes[source]¶ Obtains the bytes representation of the string.
-
weaver.utils.
bytes2str
(string: Union[AnyStr, bytes]) → str[source]¶ Obtains the unicode representation of the string.
-
weaver.utils.
parse_request_query
(request: Request) → Dict[AnyStr, Dict[AnyKey, AnyStr]][source]¶ - Parameters
request –
- Returns
dict of dict where k=v are accessible by d[k][0] == v and q=k=v are accessible by d[q][k] == v, lowercase
-
weaver.utils.
get_log_monitor_msg
(job_id: AnyStr, status: AnyStr, percent: Number, message: AnyStr, location: AnyStr) → AnyStr[source]¶
-
weaver.utils.
get_job_log_msg
(status: AnyStr, message: AnyStr, progress: Optional[Number] = 0, duration: Optional[AnyStr] = None) → AnyStr[source]¶
-
weaver.utils.
make_dirs
(path, mode=493, exist_ok=False)[source]¶ Alternative to
os.makedirs
withexists_ok
parameter only available forpython>3.5
. Also using a reduced set of permissions755
instead of original default777
.Note
The method employed in this function is safer then
if os.pat.exists
orif os.pat.isdir
pre-check to callingos.makedirs
as this can result in race condition (between evaluation and actual creation).
-
weaver.utils.
get_caller_name
(skip=2, base_class=False)[source]¶ Returns the name of a caller in the format
module.class.method
.- Parameters
skip – specifies how many levels of stack to skip while getting the caller.
base_class – Specified if the base class should be returned or the top-most class in case of inheritance If the caller is not a class, this doesn’t do anything.
- Returns
An empty string if skipped levels exceed stack height; otherwise, the requested caller name.
-
weaver.utils.
get_ssl_verify_option
(method: AnyStr, url: AnyStr, settings: AnySettingsContainer, request_options: Optional[SettingsType] = None) → bool[source]¶ Obtains the SSL verification option from combined settings from
weaver.ssl_verify
and parsedweaver.request_options
file for the corresponding request.- Parameters
method – request method (GET, POST, etc.).
url – request URL.
settings – application setting container with pre-loaded request options specifications.
request_options – pre-processed request options for method/URL to avoid re-parsing the settings.
- Returns
SSL
verify
option to be passed down to somerequest
function.
-
weaver.utils.
get_request_options
(method: AnyStr, url: AnyStr, settings: AnySettingsContainer) → SettingsType[source]¶ Obtains the request options corresponding to the request according to configuration file specified by pre-loaded setting
weaver.request_options
.If no file was pre-loaded or no match is found for the request, an empty options dictionary is returned.
- Parameters
method – request method (GET, POST, etc.).
url – request URL.
settings – application setting container with pre-loaded request options specifications.
- Returns
dictionary with keyword options to be applied to the corresponding request if matched.
-
weaver.utils.
request_extra
(method: AnyStr, url: AnyStr, retries: Optional[int] = None, backoff: Optional[Number] = None, intervals: Optional[List[Number]] = None, retry_after: bool = True, allowed_codes: Optional[List[int]] = None, only_server_errors: bool = True, ssl_verify: Optional[bool] = None, settings: Optional[AnySettingsContainer] = None, **request_kwargs) → AnyResponseType[source]¶ Standard library
requests
with additional functional utilities.Retry operation¶
Implements request retry if the previous request failed, up to the specified number of retries. Using
backoff
factor, you can control the interval between request attempts such as:delay = backoff * (2 ^ retry)
Alternatively, you can explicitly define
intervals=[...]
with the list values being the number of seconds to wait between each request attempt. In this case,backoff
is ignored andretries
is overridden accordingly with the number of items specified in the list.Furthermore,
retry_after
(default:True
) indicates if HTTP status code429 (Too Many Requests)
should be automatically handled during retries. If enabled and provided in the previously failed request response through theRetry-After
header, the next request attempt will be executed only after the server-specified delay instead of following the calculated delay fromretries
andbackoff
, or from corresponding index ofinterval
, accordingly to specified parameters. This will avoid uselessly calling the server and automatically receive a denied response. You can disable this feature by passingFalse
, which will result into requests being retried blindly without consideration of the called server instruction.Because different request implementations use different parameter naming conventions, all following keywords are looked for:
Both variants of
backoff
andbackoff_factor
are accepted.All variants of
retires
,retry
andmax_retries
are accepted.
Note
Total amount of executed request attempts will be +1 the number of
retries
orintervals
items as first request is done immediately, and following attempts are done with the appropriate delay.File Transport Scheme¶
Any request with
file://
scheme or empty scheme (no scheme specified) will be automatically handled as potential local file path. The path should be absolute to ensure it to be correctly resolved.All access errors due to file permissions return 403 status code, and missing file returns 404. Any other
IOError
types are converted to a 400 responses.See also
FileAdapter
SSL Verification¶
Allows SSL verify option to be enabled or disabled according to configuration settings or explicit parameters. Any variation of
verify
orssl_verify
keyword arguments are considered. If they all resolve toTrue
, then application settings are retrieved fromweaver.ini
to parse additional SSL options that could disable it.- Following
weaver
settings are considered : weaver.ssl_verify = True|False
weaver.request_options = request_options.yml
Note
Argument
settings
must also be provided through any supported container byget_settings()
to retrieve and apply anyweaver
-specific configurations.- Parameters
method – HTTP method to set request.
url – URL of the request to execute.
retries – Number of request retries to attempt if first attempt failed (according to allowed codes or error).
backoff – Factor by which to multiply delays between retries.
intervals – Explicit intervals in seconds between retries.
retry_after – If enabled, honor
Retry-After
response header of provided by a failing request attempt.allowed_codes – HTTP status codes that are considered valid to stop retrying (default: any non-4xx/5xx code).
ssl_verify – Explicit parameter to disable SSL verification (overrides any settings, default: True).
settings – Additional settings from which to retrieve configuration details for requests.
only_server_errors – Only HTTP status codes in the 5xx values will be considered for retrying the request (default: True). This catches sporadic server timeout, connection error, etc., but 4xx errors are still considered valid results. This parameter is ignored if allowed codes are explicitly specified.
request_kwargs – All other keyword arguments are passed down to the request call.
-
weaver.utils.
fetch_file
(file_reference: AnyStr, file_outdir: AnyStr, settings: Optional[AnySettingsContainer] = None, **request_kwargs: Any) → AnyStr[source]¶ Fetches a file from a local path or remote URL and dumps it’s content to the specified output directory.
The output directory is expected to exist prior to this function call.
- Parameters
file_reference – Local filesystem path or remote URL file reference.
file_outdir – Output directory path of the fetched file.
settings – Additional request setting details from the application configuration.
request_kwargs – Additional keywords to forward to request call (if needed).
- Returns
Path of the local copy of the fetched file.
-
weaver.utils.
get_sane_name
(name: AnyStr, min_len: Optional[int] = 3, max_len: Optional[Union[int, None]] = None, assert_invalid: Optional[bool] = True, replace_character: Optional[AnyStr] = '_') → Union[AnyStr, None][source]¶ Returns a cleaned-up version of the input name, replacing invalid characters matched with
REGEX_SEARCH_INVALID_CHARACTERS
byreplace_character
.- Parameters
name – value to clean
min_len – Minimal length of
name
to be respected, raises or returnsNone
on fail according toassert_invalid
.max_len – Maximum length of
name
to be respected, raises or returns trimmedname
on fail according toassert_invalid
. IfNone
, condition is ignored for assertion or fullname
is returned respectively.assert_invalid – If
True
, fail conditions or invalid characters will raise an error instead of replacing.replace_character – Single character to use for replacement of invalid ones if
assert_invalid=False
.
-
weaver.utils.
assert_sane_name
(name, min_len=3, max_len=None)[source]¶ Asserts that the sane name respects conditions.
See also
argument details in
get_sane_name()