Identify the technologies

wap.match.discover_technologies(technologies: Dict[str, wap.structs.Technology], url: str = '', html: str = '', scripts: List[str] = None, cookies: Dict[str, List[str]] = None, metas: Dict[str, List[str]] = None, headers: Dict[str, List[str]] = None) → List[wap.structs.TechMatch]

Discover the technologies that matches with the values provided into the different parameters. Also resolve the implied/excluded technologies.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> resp_attrs = wap.parse_requests_response(resp)
>>> techno_matches = wap.discover_technologies(techs, **resp_attrs)
wap.match.extract_techno_matches(pattern_matches: List[wap.structs.PatternMatch]) → List[wap.structs.TechMatch]

Extracts the technologies in the matches, adjusting the version and confidence, and removing duplicates.

wap.match.match_all(technologies: Dict[str, wap.structs.Technology], url: Optional[str] = '', html: Optional[str] = '', scripts: Optional[List[str]] = None, cookies: Optional[Dict[str, List[str]]] = None, metas: Optional[Dict[str, List[str]]] = None, headers: Optional[Dict[str, List[str]]] = None, js_vars: Optional[Dict[str, List[str]]] = None) → Iterator[wap.structs.PatternMatch]

For the given parameters, retrieves the technology patterns that match.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> resp_attrs = wap.parse_requests_response(resp)
>>> pattern_matches = wap.match_all(techs, **resp_attrs)
>>> techno_matches = wap.resolve_techno_matches(techs, pattern_matches)
wap.match.match_cookies(technology: wap.structs.Technology, cookies: Dict[str, List[str]]) → Iterator[wap.structs.PatternMatch]

Wrapper to search for cookies matches.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> cookies = wap.parse_requests_headers(resp.cookies)
>>> pattern_matches = []
>>> for tech in techs.values():
...     pattern_matches.extend(wap.match_cookies(tech, cookies))
>>> techno_matches = wap.resolve_techno_matches(techs, pattern_matches)
wap.match.match_headers(technology: wap.structs.Technology, headers: Dict[str, List[str]]) → Iterator[wap.structs.PatternMatch]

Wrapper to search for headers matches.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> headers = wap.parse_requests_headers(resp.headers)
>>> pattern_matches = []
>>> for tech in techs.values():
...     pattern_matches.extend(wap.match_headers(tech, headers))
>>> techno_matches = wap.resolve_techno_matches(techs, pattern_matches)
wap.match.match_html(technology: wap.structs.Technology, html: str) → Iterator[wap.structs.PatternMatch]

Wrapper to search for html matches.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> pattern_matches = []
>>> for tech in techs.values():
...     pattern_matches.extend(wap.match_html(tech, resp.text))
>>> techno_matches = wap.resolve_techno_matches(techs, pattern_matches)
wap.match.match_js_vars(technology: wap.structs.Technology, js_vars: Dict[str, List[str]]) → Iterator[wap.structs.PatternMatch]

Wrapper to search for matches in javascript variables.

wap.match.match_list(tech: wap.structs.Technology, field: str, values: List[str]) → Iterator[wap.structs.PatternMatch]

To match against a list of string, like some js scripts URIs.

Parameters:
  • tech (Technology) – The technology to search matches
  • field (str) – The field to look for matches. Must be “scripts”.
  • values (List[str]) –
Returns:

An iterator with the found matches.
Return type:

Iterator[PatternMatch]
wap.match.match_metas(technology: wap.structs.Technology, metas: Dict[str, List[str]]) → Iterator[wap.structs.PatternMatch]

Wrapper to search for meta matches.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> metas = wap.extract_metas(resp.text)
>>> pattern_matches = []
>>> for tech in techs.values():
...     pattern_matches.extend(wap.match_metas(tech, metas))
>>> techno_matches = wap.resolve_techno_matches(techs, pattern_matches)
wap.match.match_pairs(tech: wap.structs.Technology, field: str, pairs: Dict[str, List[str]]) → Iterator[wap.structs.PatternMatch]

To analyze attributes that are a dict with keys and values. Such as headers, cookies and meta tags.

Parameters:
  • tech (Technology) – The technology to search matches
  • field (str) – The field to look for matches. Must be “cookies” “headers” or “meta”.
  • pairs (Dict[str, List[str]]) –
Returns:

An iterator with the found matches.
Return type:

Iterator[PatternMatch]
wap.match.match_scripts(technology: wap.structs.Technology, scripts: List[str]) → Iterator[wap.structs.PatternMatch]

Wrapper to search for scripts matches.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> scripts = wap.extract_scripts(resp.text)
>>> pattern_matches = []
>>> for tech in techs.values():
...     pattern_matches.extend(wap.match_scripts(tech, scripts))
>>> techno_matches = wap.resolve_techno_matches(techs, pattern_matches)
wap.match.match_str(tech: wap.structs.Technology, field: str, value: str) → Iterator[wap.structs.PatternMatch]

To match attributes against a string, like an URL or HTML content.

Parameters:
  • tech (Technology) – The technology to search matches
  • field (str) – The field to look for matches. Must be “url” or “html”.
  • value (str) –
Returns:

An iterator with the found matches.
Return type:

Iterator[PatternMatch]
wap.match.match_url(technology: wap.structs.Technology, url: str) → Iterator[wap.structs.PatternMatch]

Wrapper to search for url matches.

Example

>>> import wap
>>> import requests
>>> techs, _ = wap.load_file("technologies.json")
>>> resp = requests.get("https://www.github.com")
>>> pattern_matches = []
>>> for tech in techs.values():
...     pattern_matches.extend(wap.match_url(tech, resp.url))
>>> techno_matches = wap.resolve_techno_matches(techs, pattern_matches)
wap.match.resolve_excludes(techno_matches: List[wap.structs.TechMatch]) → List[wap.structs.TechMatch]

Generates a list that not includes the technology matches that cause conflict with others, by letting only an excludent option.

wap.match.resolve_implies(techno_matches: List[wap.structs.TechMatch], technologies: Dict[str, wap.structs.Technology]) → List[wap.structs.TechMatch]

Generates a list that includes the technology matches and the technologies implied by the first ones. Also avoid the duplicates.

wap.match.resolve_techno_matches(technologies: Dict[str, wap.structs.Technology], pattern_matches: Iterator[wap.structs.PatternMatch]) → List[wap.structs.TechMatch]

Extracts from the pattern matches, the matches in technology and resolve the implied and excluded technology.