(MSCI) ESG Rating Scraper — Example Inside Using Fund and ISIN by Selenium and The API

What is ESG

ESG stands for environmental, social and governance. These are non-financial factors investors use to measure an investment or company’s sustainability. Environmental factors look at the conservation of the natural world, social factors examine how a company treats people both inside and outside the company and governance factors consider how a company is run.

ESG Investing

ESG investing is a form of sustainable investing that considers environmental, social and governance factors to judge an investment’s financial returns and its overall impact. An investment’s ESG score measures the sustainability of an investment in those specific categories.

ESG Fund Ratings and Climate Search Tool

In this article, we leverage on ESG Fund Ratings - MSCI.

How does MSCI ESG Fund Ratings work

How to Scrape

Full tutorial: (MSCI) ESG Rating Scraper — Example Inside Using Fund and ISIN by Selenium and The API — Hung, Chien-Hsiang | Blog (chienhsiang-hung.github.io).

How to Scrape

Modules

First we import the needed modules.

import pandas as pd
from selenium import webdriver
from selenium.webdriver.edge.service import Service
from webdriver_manager.microsoft import EdgeChromiumDriverManager
from selenium.webdriver.common.by import By
from tqdm import tqdm
import requests, json
import time

driver = webdriver.Edge(service=Service(EdgeChromiumDriverManager().install()))

Search Funds

We use driver = webdriver.Edge(service=Service(EdgeChromiumDriverManager().install())) here to deal with DeprecationWarning: executable_path has been deprecated selenium python1.

url = requests.get(f'https://www.msci.com/our-solutions/esg-investing/esg-fund-ratings?p_p_id=esg_fund_ratings_profile&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_resource_id=searchFundRatingsProfiles&p_p_cacheability=cacheLevelPage&_esg_fund_ratings_profile_keywords={ISIN}')
params = url.text
# handle wrong ISIN
if params == '[]':
continue
params = json.loads(params)[0] # changed from json.loads(params[1:-1]) for JSONDecodeError: Extra data: line 1 column 112 (char 111) when there are more than 1 record in api response

Scraping

But, how do we actually interact with the Site? How to locate and insert a value (input text) in a text box to be exact?5 6 We use the above method instead of click() element method7 in Selenium.

driver.get(f"https://www.msci.com/our-solutions/esg-investing/esg-fund-ratings/funds/{params['encodedTitle']}/{params['url']}")

container = driver.find_elements(by=By.CLASS_NAME, value='ratingdata-container') # Instead of driver.find_element you should use driver.find_elements method to handle NoSuchElementException
while not container:
time.sleep(1)
container = driver.find_elements(by=By.CLASS_NAME, value='ratingdata-container')
container = container[0].find_elements(by=By.TAG_NAME, value='div')
rating = container[-1].get_attribute('class')
rating = rating.split('-')[-1]

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store