Helpers to wrap common functionality

wap.helper.discover_requests_technologies(technologies: List[wap.structs.Technology], response) → List[wap.structs.TechMatch]

Helper to discover the technologies in a HTTP response of the requests library.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> t_matches = wap.discover_requests_technologies(techs, resp)

wap.helper.extract_metas(html: str) → Dict[str, str]

Helper to extract the name and content of the meta tags included in HTML content.

Example

>>> import wap
>>> import requests
>>> resp = requests.get("https://www.github.com")
>>> metas = wap.extract_metas(resp.text)

wap.helper.extract_scripts(html: str) → List[str]

Helper to extract the javascript scripts paths or URL includes in the HTML content.

Example

>>> import wap
>>> import requests
>>> resp = requests.get("https://www.github.com")
>>> scripts = wap.extract_scripts(resp.text)

wap.helper.parse_requests_cookies(cookies) → Dict[str, List[str]]

Helper to parse the cookies retrieved for a response produced by the requests library, and generate cookies to be used by wap.

Example

>>> import wap
>>> import requests
>>> resp = requests.get("https://www.github.com")
>>> cookies = wap.parse_requests_cookies(resp.cookies)

wap.helper.parse_requests_headers(headers) → Dict[str, List[str]]

Helper to parse the headers retrieved for a response produced by the requests library, and generate headers to be used by wap.

Example

>>> import wap
>>> import requests
>>> resp = requests.get("https://www.github.com")
>>> headers = wap.parse_requests_headers(resp.headers)

wap.helper.parse_requests_response(response) → Dict[str, Any]

Helper to extract all the relevant information for wap from a HTTP response generated by requests library.

Parameters:response – A response generated by the requests library. Returns:

A dictionary with the keywords “url”,

”headers”, “cookies”, “html”, “scripts” and “metas”, with values ready to be used by wap.

Return type:Dict[str, Any]

Example

>>> import wap
>>> import requests
>>> resp = requests.get("https://www.github.com")
>>> resp_elements = wap.parse_requests_response(resp)