news-please

Felix Hamborg, Norman Meuschke, Corinna Breitinger, Bela Gipp, Humboldt-Universität Zu Berlin, Humboldt-Universität Zu Berlin
2017 International Symposium of Information Science  
The amount of news published and read online has increased tremendously in recent years, making news data an interesting resource for many research disciplines, such as the social sciences and linguistics. However, large scale collection of news data is cumbersome due to a lack of generic tools for crawling and extracting such data. We present news-please, a generic, multilanguage, open-source crawler and extractor for news that works out-of-the-box for a large variety of news websites. Our
more » ... em allows crawling arbitrary news websites and extracting the major elements of news articles on those websites, i.e., title, lead paragraph, main content, publication date, author, and main image. Compared to existing tools, news-please features full website extraction requiring only the root URL.
doi:10.18452/1447 dblp:conf/isiwi/HamborgMBG17 fatcat:763h7ckq6rf2hlyqp6t46s4pku