Python modify user-agent

How to generate user-agent header for web requests in Python

In previous post, we saw how to modify user-agent header in wget, curl and httpie programs. In this post, I will show you how to modify user-agent header in Python’s popular requests module. There are several reasons for modify user agent, one of which is to trigger a different response from a website. Many website offer different content based on user-agent header. You can find user-agent header details here.

In Python, one of the most popular libraries to query web servers is the requests module. The requests module allows you to pass header information using the headers option –

1. Simplest use case, without a header

import requests
requests.get('http://linuxfreelancer.com/status')

And this is how it is logged on the web server side, Apache in this case –

76.1.2.3 [20/May/2018:01:08:37 -0400] "GET /status HTTP/1.1" 200 359 "-" "python-requests/2.18.4" 1798

The user-agent is simply showing as “python-requests/2.18.4”, and some website might even block this to prevent web crawlers. So the next step is to modify this.

2. Modify user-agent header

headers = {'User-Agent': 'Mozilla/5.0 (Android 5.1; Tablet; rv:50.0) Gecko/50.0 Firefox/50.0'}
requests.get('http://linuxfreelancer.com/status', headers=headers)

And this is what the access log entry looks like on the web server side –

76.1.2.3  [20/May/2018:01:11:29 -0400] "GET /status HTTP/1.1" 200 359 "-" "Mozilla/5.0 (Android 5.1; Tablet; rv:50.0) Gecko/50.0 Firefox/50.0" 1289

As you can see above, the user-agent entry has several identifiers which is not easy to remember. The best way would be to programatically generate valid user-agents for different platforms.

3. Generate valid user-agents

The user_agent module is used for generating random and yet valid web user agents. You can install it with ‘pip install user_agent’.

This module generate user-agent strings for differnt devices types such as desktop, smartphone and table, as well as OS types (Windows, Linux, Mac, Android …). Let us try it in a virtual environment –

virtualenv /tmp/venv
source /tmp/venv/bin/activate
pip install user_agent

Now run Python in an interactive mode –

import requests
from user_agent import generate_user_agent 

In [8]: generate_user_agent()
Out[8]: 'Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2803.5 Safari/537.36'

In [9]: generate_user_agent()
Out[9]: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:49.0) Gecko/20100101 Firefox/49.0'

In [11]: generate_user_agent(os='linux')
Out[11]: 'Mozilla/5.0 (X11; Ubuntu; Linux i686 on x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2942.3 Safari/537.36'

In [13]: generate_user_agent(device_type='tablet')
Out[13]: 'Mozilla/5.0 (Linux; Android 4.4; HTC Desire 616 dual sim Build/JDQ39) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2938.21 Safari/537.36'

In [14]: generate_user_agent(device_type='desktop', os='win')
Out[14]: 'Mozilla/5.0 (Windows NT 6.1; rv:45.0) Gecko/20100101 Firefox/45.0'

In [20]: generate_user_agent(navigator='chrome', os='linux', device_type='desktop')
Out[20]: 'Mozilla/5.0 (X11; Ubuntu; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2894.26 Safari/537.36'

This allows us to generate from random to specific valid user-agent header information. We can then pass this randomly generated user-agent text to the requests module header option, and we will view our web server logs to validate –

from user_agent import generate_user_agent
import requests
requests.get('http://linuxfreelancer.com/status', headers={'User-Agent': generate_user_agent(navigator='firefox', os='linux')})

Log entry –

76.1.2.3 – – [20/May/2018:01:28:24 -0400] “GET /status HTTP/1.1” 200 359 “-” “Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0” 1486

References –

http://docs.python-requests.org/en/master/

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent