The harvester holds a list of all sources or their feeds (RSS and Atom feeds, sitemaps and JSON feeds) and crawls them periodically.
The harvester holds a list of all sources or their feeds (RSS and Atom feeds, sitemaps and JSON feeds) and crawls them periodically.
DeepNews scans all articles from all major media/publications, as long as they publish their news via feed.
This includes not only the so-called 'leading media', such as FAZ, Spiegel or the New York Times, but just as local newspapers or important blogs such as 9to5.
Note: If a medium/newspaper/important blog is missing, we would be happy if you let us know this source by URL and preferably additionally with feed.
There are sources that publish a lot and blogs that publish a new article only a few times per week. DeepNews identifies per source the frequency with which it publishes and adapts accordingly.
On a personal note, crawlers contribute to a considerable amount of website traffic. We try to keep this load to a minimum in order to be appropriately economical and respectful of public access and server capacity, but without missing important news.
The categories used in DeepNews are predetermined by DeepNews.
For many articles, the category is revealed directly by the URL or that of the parent feed. For other articles 'learns' DeepNews via the thematic grouping (clustering) to which categories an article belongs.
Many articles are based on agency reports and are usually published only slightly paraphrased from various sources. DeepNews detects AI- and NLP-based duplicates, even if they are more or less well paraphrased, and merges them into 'one' article.
Important: Duplicates do not increase the relevance of a topic.
Because DeepNews recognizes identical or very similar articles with a high degree of certainty, DeepNews can identify sources whose articles are fed, for example, by higher-level media houses as so-called 'jacket' articles to their various local newspapers.