Python Requests Module

The requests module in Python is one of the most popular libraries for making HTTP requests. It simplifies the process of sending HTTP/1.1 requests (like GET, POST, PUT, DELETE, etc.) and handling responses. This module is particularly useful for interacting with APIs, downloading data from websites, or automating web-based tasks.

Installing the Requests Module

Before using the requests module, you need to install it using pip:

pip install requests

Common Methods in the requests Module

1. requests.get()
   Sends an HTTP GET request to the specified URL.

   import requests
   response = requests.get('https://api.github.com')
   print(response.status_code)  # Example: 200
   print(response.text)         # Example: Content of the page
   

2. requests.post()
   Sends an HTTP POST request to the specified URL with optional data.

   response = requests.post('https://httpbin.org/post', data={'key': 'value'})
   print(response.status_code)  # Example: 200
   print(response.json())       # Example: {"key": "value"}
   

3. requests.put()
   Sends an HTTP PUT request to the specified URL, typically used for updating data.

   response = requests.put('https://httpbin.org/put', data={'key': 'value'})
   print(response.status_code)  # Example: 200
   

4. requests.delete()
   Sends an HTTP DELETE request to the specified URL.

   response = requests.delete('https://httpbin.org/delete')
   print(response.status_code)  # Example: 200
   

5. requests.head()
   Sends an HTTP HEAD request, similar to a GET request but without the response body.

   response = requests.head('https://api.github.com')
   print(response.headers)  # Example: Headers of the response
   

6. requests.options()
   Sends an HTTP OPTIONS request, used to describe the communication options for the target resource.

   response = requests.options('https://api.github.com')
   print(response.headers)  # Example: Allowed methods
   

Working with Responses

1. Status Code
   The status code of an HTTP request indicates the result of the request.

   response = requests.get('https://api.github.com')
   print(response.status_code)  # Example: 200
   

2. Response Content
   You can access the body of the response in various formats:
   • response.text: Returns the response body as a string.
   • response.content: Returns the response body as bytes.
   • response.json(): Parses the response as JSON (if it’s in JSON format).

   response = requests.get('https://api.github.com')
   print(response.text)         # Response as a string
   print(response.json())       # Response as JSON
   

3. Response Headers
   You can access the headers returned by the server using the .headers attribute.

   print(response.headers)
   

4. Response Encoding
   You can check or set the encoding of the response content.

   print(response.encoding)  # Example: 'utf-8'
   response.encoding = 'ISO-8859-1'
   

Sending Data with Requests

1. Sending URL Parameters (Query Parameters)
   URL parameters can be passed using the params argument in the requests.get() method.

   params = {'key1': 'value1', 'key2': 'value2'}
   response = requests.get('https://httpbin.org/get', params=params)
   print(response.url)  # Example: https://httpbin.org/get?key1=value1&key2=value2
   

2. Sending Form Data
   Form data can be sent in a POST request using the data argument.

   data = {'username': 'user', 'password': 'pass'}
   response = requests.post('https://httpbin.org/post', data=data)
   print(response.json())  # Example: {"username": "user", "password": "pass"}
   

3. Sending JSON Data
   You can send JSON data in a POST request using the json argument.

   json_data = {'name': 'John', 'age': 30}
   response = requests.post('https://httpbin.org/post', json=json_data)
   print(response.json())  # Example: {"name": "John", "age": 30}
   

Handling Timeouts

To avoid waiting indefinitely for a response, you can set a timeout in seconds.

response = requests.get('https://httpbin.org/delay/5', timeout=3)  # Timeout after 3 seconds

Handling Errors

The requests module does not raise exceptions for HTTP errors by default. You can check the status code manually or use the response.raise_for_status() method to raise an exception for error codes (4xx or 5xx).

response = requests.get('https://api.github.com/invalid-endpoint')
if response.status_code != 200:
    print(f"Error: {response.status_code}")

# Raise exception for HTTP errors
response.raise_for_status()  # Will raise an HTTPError if status code is not 2xx

Headers

You can pass custom headers to the server using the headers argument.

headers = {'Authorization': 'Bearer your_token'}
response = requests.get('https://api.github.com', headers=headers)
print(response.status_code)

Cookies

You can send and receive cookies using the cookies argument or the cookies attribute of the response object.

# Sending cookies
cookies = {'session_id': '123456'}
response = requests.get('https://httpbin.org/cookies', cookies=cookies)
print(response.text)

# Accessing response cookies
print(response.cookies)

File Uploads

Files can be sent as part of a POST request using the files argument.

files = {'file': open('example.txt', 'rb')}
response = requests.post('https://httpbin.org/post', files=files)
print(response.text)

Session Objects

You can use a Session object to persist certain parameters across requests, like cookies or headers.

session = requests.Session()
session.headers.update({'Authorization': 'Bearer your_token'})

response = session.get('https://api.github.com')
print(response.status_code)

Example: Basic GET Request

import requests

url = 'https://api.github.com/repos/psf/requests'
response = requests.get(url)

if response.status_code == 200:
    data = response.json()
    print(f"Repository: {data['name']}")
    print(f"Description: {data['description']}")
else:
    print(f"Failed to retrieve data: {response.status_code}")

The requests module simplifies interacting with web APIs and HTTP-based services. It handles many of the complexities involved in making requests, such as managing cookies, sessions, and dealing with various response formats.