Hackernews posts about Common Crawl

1. A Change to Common Crawl Dataset Size Reporting (commoncrawl.org)
   3 points by ccgreg 6 days ago | 1 comments
2. Show HN: hot or not for .ai websites (ratemyaisite.com)
   1 points by prolly97 1 day ago | discuss
3. IPv6 Adoption Across the TopK Web Hosts (commoncrawl.org)
   2 points by miyuru 15 days ago | discuss
4. Large language model data pipelines and Common Crawl (blog.christianperone.com)
   139 points by sonabinu almost 2 years ago | 12 comments
5. Common Crawl maintains a free, open repository of web crawl data (commoncrawl.org)
   27 points by doener about 1 year ago | 1 comments
6. Ask HN: Is Common Crawl used exhaustively by any search engine?
   8 points by n1xis10t 5 months ago | 1 comments
7. Show HN: Help improve language coverage in Common Crawl
   8 points by ccgreg 10 months ago | discuss
8. Ask HN: Is there a service that offers Common Crawl as an API?
   7 points by georgehill 11 months ago | 3 comments
9. Cybersecurity Data Extraction from Common Crawl (arxiv.org)
   5 points by PaulHoule about 1 month ago | discuss
10. Publishers Target Common Crawl in Fight over AI Training Data (www.wired.com)
    4 points by ccgreg almost 2 years ago | 1 comments
11. Cc-downloader: command-line tool for downloading Common Crawl data via HTTPS (github.com)
    2 points by simonpure about 1 year ago | discuss
12. Common Crawl May/June 2024 Newsletter (commoncrawl.org)
    2 points by nceqs3 almost 2 years ago | discuss
13. Show HN: I built a 50 site sampler from CommonCrawl refreshing every 30 minutes (randcrawl.com)
    1 points by whothatcodeguy 2 months ago | discuss
14. 200M unique domains extracted from Common Crawl (zenodo.org)
    1 points by networkcat 3 months ago | discuss
15. Discovering Shopify Domains: A Journey Through Common Crawl Data (alistechtales.substack.com)
    1 points by gmays over 1 year ago | discuss
16. Common Crawl Statistics Now Available on Hugging Face (commoncrawl.org)
    1 points by nceqs3 over 1 year ago | discuss
17. Show HN: Randomly discovered websites from the open internet every 60 minutes (randcrawl.com)
    4 points by whothatcodeguy 2 months ago | 1 comments
18. Show HN: Rhiza – easily create shortcuts and add entries to PATH (github.com)
    2 points by skardy about 1 year ago | discuss
19. Humanties Last War
    1 points by vlan121 6 months ago | 1 comments
20. Ask HN: How Spam are removed from large datasets of web crawls for AI training
    1 points by arromatic almost 2 years ago | discuss
21. The Company Quietly Funneling Paywalled Articles to AI Developers (www.theatlantic.com)
    33 points by breve 5 months ago | 16 comments
22. The Nonprofit Feeding the Internet to AI Companies (www.theatlantic.com)
    10 points by ForHackernews 5 months ago | 5 comments
23. The Nonprofit Doing the AI Industry's Dirty Work (www.theatlantic.com)
    9 points by kgwgk 5 months ago | 2 comments
24. The Company Funneling Paywalled Articles to AI Developers (www.theatlantic.com)
    3 points by CaptainZapp 5 months ago | discuss
25. Legality of Publishing Web Crawls (2020) (skeptric.com)
    1 points by phewlink almost 2 years ago | discuss
26. Crawler operators, please stop destroying the commons (lunnova.dev)
    7 points by xena 12 months ago | discuss
27. Show HN: CrawlerCheck v1.5 – Operators Directory and 25 new AI crawlers (crawlercheck.com)
    1 points by bogozi about 2 months ago | discuss
28. Show HN: I built an integration to update OpenAI assistant with website crawls (github.com)
    1 points by jirispilka almost 2 years ago | discuss
29. Show HN: Free tool to find RSS feeds, even if not linked on the page
    152 points by domysee over 1 year ago | 48 comments
30. Show HN: Wispbit - Linter for AI coding agents (wispbit.com)
    31 points by dearilos 6 months ago | 14 comments
31. Show HN: Startkit.ai – Boilerplate for building products with AI (startkit.ai)
    25 points by jivings almost 2 years ago | discuss
32. Quick but powerful research for AI agents with data scrapping and selenium
    8 points by alexvomwald about 1 year ago | 6 comments
33. Ask HN: Why do we still buy things by browsing catalogs?
    6 points by dannythecount about 1 month ago | 8 comments