2024
Franziska Sielker, Selim Banabak
The document discusses the innovative use of housing advertisements as a data source for mapping housing affordability across Europe through the House4All ESPON project. As the issue of housing affordability intensifies, traditional data collection methods often fall short due to a lack of granularity and regional specificity. By leveraging web-scraping techniques, the project aims to gather detailed information from online listings, which can offer a clearer picture of housing costs relative to household incomes. This new approach not only promises greater spatial granularity but also allows researchers to analyze trends and disparities in housing affordability at various levels, from neighborhoods to regions. By scraping data repeatedly over a year, the project can identify real-time market dynamics and provide insights into which properties are in high demand. However, while the methodology is promising, it faces challenges such as the exclusion of social housing and the need for data harmonization across different countries. Additionally, researchers must be cautious, as listed prices may not reflect final sales, complicating the analysis of affordability. The project is currently addressing these obstacles by refining its methodology and exploring ways to incorporate income data effectively. Ultimately, the document highlights the potential for housing advertisements to transform our understanding of affordability, suggesting that continued collaboration and methodological rigor will be essential for generating reliable insights that can inform effective housing policies across Europe.
Challenges and opportunities of web-scrapping in measuring house affordability - By Franziska Sielker and Selim Banabak
Housing affordability is an increasingly pressing issue across Europe, with rising costs pushing more households to the brink of financial overburden. The House4All ESPON project aiming to create a pan-European housing affordability map at a regional level, seeks to contribute to knowledge creation on the affordability challenge. By leveraging innovative data sources, particularly online housing advertisements, the project aims to overcome the limitations of traditional data collection methods. The project undertakes a repeated web-scrapping exercise between April 2024 and April 2025 in the 31 countries of the ESPON space.
However, while this approach holds significant promise, it also presents a host of challenges and potential pitfalls. With his blog post we aim to provide an overview of the ongoing activities and methodological approach chosen by the project.
The Need for a Granular Approach to Housing Affordability
Housing affordability is typically understood as a function of housing costs relative to available household income. These costs can include rent or mortgage payments, maintenance expenses, and in some interpretations, energy and transportation costs. The widely used income ratio approach, which measures the proportion of income a household spends on housing, often sets the affordability threshold at around 30% to 40%. Beyond this point, households are considered overburdened by housing costs.
However, while the income ratio approach offers a useful benchmark for assessing affordability, applying it at the regional or urban level is challenging due to the lack of granular data. Traditional data sources, like surveys, often do not provide the detailed information necessary for such fine-grained analysis. Moreover, many governments do not systematically record the required data at the sub-national level, making it difficult to assess affordability with precision across different regions and cities.
Housing Adverts: A New Source of Dat
In response to these challenges, researchers have increasingly turned to online housing adverts as a novel data source. These adverts, listed on platforms such as Nestoria or Properstar, provide a wealth of information that can be systematically scraped, including price, geolocation, and qualitative characteristics like the number of rooms or property condition. By aggregating and analyzing this data, researchers can produce average prices or prices per square meter for specific spatial units, which can then be compared to income data from traditional sources.
This approach offers several significant advantages. Most notably, it allows for much greater spatial granularity than survey data, providing the flexibility to aggregate data at various levels, from the neighborhood to the regional level. This can give a much clearer picture of housing affordability within specific areas, highlighting disparities that might be missed in broader analyses.
In web scrapping housing advertisements from across Europe, the ESPON House4All project will be able to provide a first ever comprehensive European-wide mapping of the existing housing offer allowing for very low granular data set, being able to distinguish via different market segments. Importantly, thereby the project will be able to analyse the differences in affordability between renting and selling and the implications within different market segments. In addition, for policy making, scrapping the data on a regular basis will allow conclusions as to which market segments in which geographical location are most thought after, as they will have the shortest period of being online. By scrapping throughout a full year regularly, the project will be able to identify advertisements that are most thought after and those that remain longest on platforms.
Moreover, because these adverts reflect current market conditions, they offer insights into the prices faced by newcomers to a region or city. This is particularly valuable for understanding affordability in dynamic urban areas, where prices can fluctuate rapidly.
The Challenges and Pitfalls
Despite the promise of using housing adverts for affordability mapping, this approach is not without its challenges. One of the primary issues is that these adverts typically represent only a subset of the market—specifically, properties that are publicly advertised. This often excludes social housing and the very top and bottom ends of the price spectrum. As a result, the data may not fully capture the affordability issues faced by all residents, particularly those in the most vulnerable positions, such as residents living on benefits. In the project, the project team thereby will where include data on social housing on a case study level.
Additionally, the data from housing adverts while being immediately comparable in a national context, may require significant harmonization to be useful on a pan-European scale. In different countries, listed prices may include different components—such as whether the price includes land or just the structure, or whether utilities and taxes are factored. Without careful adjustment, comparisons between regions or countries may be misleading. The project is at the time of posting this blog post in the process of finetuning the methodology for adjustment for the 31 countries scrapped.
Another significant concern is that listed prices do not always reflect the final prices paid. Properties may be listed at a higher price to test the market, or at a lower price to attract interest, with the final transaction occurring at a different amount. In some cases, the duration that a property remains listed can indicate whether it is overpriced or underpriced, but this requires additional data and sophisticated analysis to correct for such biases. Yet, the negotiation differences may also be of different size between renting, selling and between higher priced and lower priced advertisements. Unfortunately, within the project these negotiation gaps will be difficult to reflect. Yet, more and more data is being developed to allow transaction prices to be considered. In a more long-term perspective, it would be possible to consider this information more structurally.
Furthermore, the income data required to calculate price-to-income ratios is often only available at an aggregate level, rather than for individual households. This means that any analysis must rely on broad averages, which can obscure significant variations within regions. Additionally, going below the NUTS2 level (a regional classification used by Eurostat) can be challenging, as detailed income data is often unavailable. The House4All project is currently preparing a methodology to consider adjustments for income data. Researchers might resort to using GDP per capita as a proxy, but this too has limitations. For example, areas with a high concentration of corporate headquarters might appear more affordable than they are, due to the inflated GDP figures not reflecting local household incomes. This, however, is mainly a challenge when comparing countries across Europe, and metropolitan regions with their surroundings. However, this challenge does not hold for a local-regional analysis and for comparing within regions. The data allows for mapping affordability challenges in local settings.
Navigating the Future of Housing Affordability Research
Despite these challenges, the use of housing adverts remains a compelling tool for researchers attempting to map housing affordability across Europe. The House4All ESPON project exemplifies the innovative approaches needed to tackle complex social issues like housing affordability. By continuously refining methodologies and addressing the pitfalls inherent in this approach, it is possible to produce more accurate and actionable insights.
As this field of research evolves, collaboration between governments, private sector platforms, and academic institutions will be crucial. Improved access to data, combined with rigorous analytical techniques, can help ensure that the insights derived from housing adverts are both reliable and representative. This, in turn, can inform better policy decisions, ultimately leading to more effective strategies to ensure housing affordability for all.
In conclusion, while the use of housing adverts for affordability mapping is still in its early stages, it offers a promising avenue for overcoming the limitations of traditional data sources. The evidence also becomes more solid with repeated scrappings, allowing for example to showcase the impact of specific new policies on house prices, as well as to monitor the development of house prices more generally. With careful consideration of the challenges and a commitment to methodological rigor, this approach has the potential to significantly enhance our understanding of housing affordability in Europe.
Authors:
- Franziska Sielker, Professor of Urban and Regional Research, Institute of Spatial Planning, TU Wien
- Selim Banabak, Postdoctoral Researcher, Institute of Spatial Planning, TU Wien