Hackernews posts about Common Crawl

RSS feed for Common Crawl

01
newCommon Crawl Data Stored on a Hugging Face Bucketcommoncrawl.org
▲ 23abidlabs 18d 💬 1
02
newShow HN: Web search engine built from the CommonCrawl corpussearch.xorsoft.dev
▲ 4xoredev 3d 💬 3
03
newCommon Crawl's Data Infrastructure Shaped the Battle Royale over GenAIwww.mozillafoundation.org
▲ 3Anon84 25d discuss
04
newShow HN: An Elasticsearch benchmarker and a Common Crawl indexergithub.com
▲ 2cosh91 4d discuss
05
newShow HN: What 482 hospitals charge vs. what insurers pay, from their own fileswww.billmender.com
▲ 31curatedmcp 4d 💬 15
06
newUS publishers tell Common Crawl to stop scraping and delete archivepressgazette.co.uk
▲ 30thm 118d 💬 10
07
newCommon Crawl maintains a free, open repository of web crawl datacommoncrawl.org
▲ 27doener 579d 💬 1
08
newAsk HN: Is Common Crawl used exhaustively by any search engine?
▲ 8n1xis10t 336d 💬 1
09
newShow HN: Help improve language coverage in Common Crawl
▲ 8ccgreg 468d discuss
10
newAsk HN: Is there a service that offers Common Crawl as an API?
▲ 7georgehill 513d 💬 3
11
newShow HN: Infini-News – 1.36B news articles from Common Crawl, queryable in mscs2.uni-graz.at
▲ 7ruggsea 96d 💬 1
12
newCommon Crawlcommoncrawl.org
▲ 7skibz 84d discuss
13
newCybersecurity Data Extraction from Common Crawlarxiv.org
▲ 5PaulHoule 213d discuss
14
newA Change to Common Crawl Dataset Size Reportingcommoncrawl.org
▲ 3ccgreg 187d 💬 1
15
newCc-downloader: command-line tool for downloading Common Crawl data via HTTPSgithub.com
▲ 2simonpure 598d discuss
16
newPublishers Demand Accountability from Common Crawl over Unauthorized Usewww.newsmediaalliance.org
▲ 1thm 157d 💬 2
17
newShow HN: RankParse – An SEO and backlink data API built from Common Crawlrankparse.com
▲ 1abhibavishi 58d discuss
18
newCrawlgraph – Backlink lookup using Common Crawl ($99 lifetime)crawlgraph.com
▲ 1pucilpet 132d discuss
19
newPublishers Tell Common Crawl to Stop Unauthorized Scrapingwww.mediapost.com
▲ 1jaredwiener 139d discuss
20
newI made Common Crawl's 4.4B edges queryable for backlink lookupscrawlgraph.com
▲ 1pucilpet 159d discuss
21
newShow HN: I built a 50 site sampler from CommonCrawl refreshing every 30 minutesrandcrawl.com
▲ 1whothatcodeguy 244d discuss
22
new200M unique domains extracted from Common Crawlzenodo.org
▲ 1networkcat 267d discuss
23
newShow HN: Randomly discovered websites from the open internet every 60 minutesrandcrawl.com
▲ 4whothatcodeguy 243d 💬 1
24
newShow HN: Rhiza – easily create shortcuts and add entries to PATHgithub.com
▲ 2skardy 591d discuss
25
newHumanties Last War
▲ 1vlan121 347d 💬 1
26
newShow HN: hot or not for .ai websitesratemyaisite.com
▲ 1prolly97 182d discuss
27
newThe Company Quietly Funneling Paywalled Articles to AI Developerswww.theatlantic.com
▲ 33breve 333d 💬 16
28
newThe Nonprofit Feeding the Internet to AI Companieswww.theatlantic.com
▲ 10ForHackernews 335d 💬 5
29
newThe Nonprofit Doing the AI Industry's Dirty Workwww.theatlantic.com
▲ 9kgwgk 335d 💬 2
30
newThe Company Funneling Paywalled Articles to AI Developerswww.theatlantic.com
▲ 3CaptainZapp 334d discuss
31
newIPv6 Adoption Across the TopK Web Hostscommoncrawl.org
▲ 2miyuru 196d discuss
32
newCrawler operators, please stop destroying the commonslunnova.dev
▲ 7xena 534d discuss
33
newShow HN: CrawlerCheck v1.5 – Operators Directory and 25 new AI crawlerscrawlercheck.com
▲ 1bogozi 238d discuss
34
newShow HN: Wispbit - Linter for AI coding agentswispbit.com
▲ 31dearilos 356d 💬 14
35
newQuick but powerful research for AI agents with data scrapping and selenium
▲ 8alexvomwald 602d 💬 6