The alternative data industry is worth over £5 billion and is expected to grow to £120bn by 2030, which is no surprise given the importance many organisations put on data when making business decisions.

Recent Oxylabs research revealed that over half UK of financial companies use automated processes, such as web scraping, to gather alternative data. Yet, the benefits of alternative data are not talked about enough. To ensure web scraping reaches its full potential, it needs to get the right exposure and education, to achieve this it must be taught at universities.

Doing so will allow students to thrive in the age of big data and AI. This is according to Juras Juršėnas, COO at Oxylabs.

“Web scraping public data is essential to the study of artificial intelligence (AI) and machine learning (ML),” said Juršėnas.

He adds: “AI and ML studies are becoming very popular, with top universities offering AI and ML courses. Students in these programs often lack the proper datasets to study AI/ML algorithms, and public data scraping knowledge would help students build quality datasets for more efficient machine learning.”

“To fill the gap and help academics gather big data with web scraping tools, Oxylabs launched a pro bono initiative called “Project 4β”. The initiative aims to transfer the technological expertise accumulated over the years, and grant universities and NGOs free access to data scraping tools, supporting important research on big data.”

Juršėnas continued: “Despite the lack of awareness, the possibilities of alternative web data analysis in social, economic, or psychological studies are endless. Investigative journalists and political scientists also use scrapers to study a wide range of issues, from tracking the influence of lobbyists by investigating visitor logs from government buildings to monitoring prohibited political ads and extremist groups in public social media platforms and forums.

“Automated big data gathering is the breakthrough many scientists have been waiting for years. However, it’s suffering from several misconceptions.

“Web scraping is surrounded by various legal concerns, discouraging some researchers from leveraging public big data in their studies. This is not a surprise as the industry is relatively new and open to various players, so there have been numerous misconceptions arising from improper use. There is nothing inherently unethical about web scraping, as it simply automates activities that people would otherwise do manually. If supplied by a reputable provider, most of these risks are nullified or greatly reduced.

Recommended

• What’s the Latest Data on the Scottish Tech Labour Market?
• Report: Business Email Compromise Threat to Rise Steeply
• IBM Women in Leadership: Perception Outpaces Reality

“As web scraping involves risks, academics often choose to steer clear altogether, returning to traditional data sources. To promote ethical data-gathering practices and industry-wide standards, together with other prominent DaaS companies, Oxylabs introduced the Ethical Web Data Collection Initiative. The consortium aims to build trust around web scraping and educate a wider tech community about big data possibilities.”

“Organisations should always consult with legal professionals before scraping to minimise most risks associated with public web data collection.”

Juršėnas concluded, “Web scraping has yet to gain traction in the public eye and academia. However, with the sheer volume of web data increasing exponentially, big data analysis will slowly become an inevitable part of scientific research. With it being routine to teach SPSS basics on social sciences campuses, it should become normal to familiarise students with web scraping practices.”