Salary exploration with Python and Payscale data

From web scraping with Selenium to data visualization with Dash

Simone Rigoni
2 min readJun 1, 2022
Photo by Sharon McCutcheon on Unsplash

The idea is to automatically search on payscale.com the salary for a list of jobs in different countries and visualize the results with a Dash web application.

By making a search on the website for the salary of a Data Engineer in Italy:

https://www.payscale.com/research/IT/Job=Data_Engineer/Salary

We can see that the url is https://www.payscale.com/research/IT/Job=Data_Engineer/Salary. So it is possible to automatically search the website by providing a country code and a job name.

To automate the web search we can use Selenium, a Python library that allows us to do actions in the browser. Primarily it is used for automating web applications for testing.

Every salary exploration is identified by the YYYYMMDDTHHMMSSZ date string.

After searching for each country and each job we have to convert the salary from local currency to a common one (i.e. Euro in my case). To do that we can again use Selenium and perform a search on Google to get the conversion from any currency to Euro.

For example in United States a Data Engineer earns $93245 so we can search on Google: $93245 to EUR

We will get a page like this:

Google search

So we simply need to read the converted value.

Finally to visualize the data we can build a simple dash web application:

Dash web application home

Using the drop down menu it is possible to select witch extraction you want to visualize.

Thanks payscale.com for the data. Payscale offers a wide variety of products and services in the compensation technology. Check it out.

Just to be clear I do not have any commercial affiliation with Payscale.

Note: Code can be found in this github repository

--

--