Abstract WebTools

Provides utilities for inspecting and parsing web content, including React components and URL utilities, with enhanced capabilities for managing HTTP requests and TLS configurations.

• Features:
  • URL Validation: Ensures URL correctness and attempts different URL variations.
  • HTTP Request Manager: Custom HTTP request handling, including tailored user agents and improved TLS security through a custom adapter.
  • Source Code Acquisition: Retrieves the source code of specified websites.
  • React Component Parsing: Extracts JavaScript and JSX source code from web pages.
  • Comprehensive Link Extraction: Collects all internal links from a specified website.
  • Web Content Analysis: Extracts and categorizes various web content components such as HTML elements, attribute values, attribute names, and class names.

abstract_webtools.py

Description:
Abstract WebTools offers a suite of utilities designed for web content inspection and parsing. One of its standout features is its ability to analyze URLs, ensuring their validity and automatically attempting different URL variations to obtain correct website access. It boasts a custom HTTP request management system that tailors user-agent strings and employs a specialized TLS adapter for heightened security. The toolkit also provides robust capabilities for extracting source code, including detecting React components on web pages. Additionally, it offers functionalities for extracting all internal website links and performing in-depth web content analysis. This makes Abstract WebTools an indispensable tool for web developers, cybersecurity professionals, and digital analysts.

• Dependencies:
  • requests
  • ssl
  • HTTPAdapter from requests.adapters
  • PoolManager from urllib3.poolmanager
  • ssl_ from urllib3.util
  • urlparse, urljoin from urllib.parse
  • BeautifulSoup from bs4

UrlManager

The UrlManager is a Python class designed to handle and manipulate URLs. It provides methods for cleaning and normalizing URLs, determining the correct version of a URL, extracting URL components, and more. This class is particularly useful for web scraping, web crawling, or any application where URL management is essential.

Usage

To use the UrlManager class, first import it into your Python script:

from abstract_webtools import UrlManager

Initializing a UrlManager Object

You can create a UrlManager object by providing an initial URL and an optional requests session. If no URL is provided, it defaults to 'www.example.com':

url_manager = UrlManager(url='https://www.example.com')

URL Cleaning and Normalization

The clean_url method takes a URL and returns a list of potential URL variations, including versions with and without 'www.', 'http://', and 'https://':

cleaned_urls = url_manager.clean_url()

Getting the Correct URL

The get_correct_url method tries each possible URL variation with an HTTP request to determine the correct version of the URL:

correct_url = url_manager.get_correct_url()

Updating the URL

You can update the URL associated with the UrlManager object using the update_url method:

url_manager.update_url('https://www.example2.com')

Extracting URL Components

The url_to_pieces method extracts various components of the URL, such as protocol, domain name, path, and query:

url_manager.url_to_pieces()
print(url_manager.protocol)
print(url_manager.domain_name)
print(url_manager.path)
print(url_manager.query)

Additional Utility Methods

• get_domain_name(url): Returns the domain name (netloc) of a given URL.
• is_valid_url(url): Checks if a URL is valid.
• make_valid(href, url): Ensures a relative or incomplete URL is valid by joining it with a base URL.
• get_relative_href(url, href): Converts a relative URL to an absolute URL based on a base URL.

Compatibility Note

The get_domain method is kept for compatibility but is inconsistent. Use it only for "webpage_url_domain." Similarly, url_basename, base_url, and urljoin methods are available for URL manipulation.

Example

Here's a quick example of using the UrlManager class:

from abstract_webtools import UrlManager

url_manager = UrlManager(url='https://www.example.com')
cleaned_urls = url_manager.clean_url()
correct_url = url_manager.get_correct_url()
url_manager.update_url('https://www.example2.com')

print(f"Cleaned URLs: {cleaned_urls}")
print(f"Correct URL: {correct_url}")

Dependencies

The UrlManager class relies on the requests library for making HTTP requests. Ensure you have the requests library installed in your Python environment.

SafeRequest

The SafeRequest class is a versatile Python utility designed to handle HTTP requests with enhanced safety features. It integrates with other managers like UrlManager, NetworkManager, and UserAgentManager to manage various aspects of the request, such as user-agent, SSL/TLS settings, proxies, headers, and more.

Usage

To use the SafeRequest class, first import it into your Python script:

from abstract_webtools import SafeRequest

Initializing a SafeRequest Object

You can create a SafeRequest object with various configuration options. By default, it uses sensible default values, but you can customize it as needed:

safe_request = SafeRequest(url='https://www.example.com')

Updating URL and UrlManager

You can update the URL associated with the SafeRequest object using the update_url method, which also updates the underlying UrlManager:

safe_request.update_url('https://www.example2.com')

You can also update the UrlManager directly:

from url_manager import UrlManager

url_manager = UrlManager(url='https://www.example3.com')
safe_request.update_url_manager(url_manager)

Making HTTP Requests

The SafeRequest class handles making HTTP requests using the try_request method. It handles retries, timeouts, and rate limiting:

response = safe_request.try_request()
if response:
    # Process the response here

Accessing Response Data

You can access the response data in various formats:

• safe_request.source_code: HTML source code as a string.
• safe_request.source_code_bytes: HTML source code as bytes.
• safe_request.source_code_json: JSON data from the response (if the content type is JSON).
• safe_request.react_source_code: JavaScript and JSX source code extracted from `