Hackernews posts about Common Crawl

1. Ask HN: Is Common Crawl used exhaustively by any search engine?
   8 points by n1xis10t 14 days ago | 2 comments
2. Humanties Last War
   1 points by vlan121 25 days ago | 1 comments
3. The Company Quietly Funneling Paywalled Articles to AI Developers (www.theatlantic.com)
   33 points by breve 11 days ago | 16 comments
4. The Nonprofit Feeding the Internet to AI Companies (www.theatlantic.com)
   10 points by ForHackernews 13 days ago | 5 comments
5. The Nonprofit Doing the AI Industry's Dirty Work (www.theatlantic.com)
   9 points by kgwgk 13 days ago | 1 comments
6. The Company Funneling Paywalled Articles to AI Developers (www.theatlantic.com)
   3 points by CaptainZapp 12 days ago | discuss
7. Show HN: Skillz – Use Claude Skills Anywhere (github.com)
   2 points by kevinslin about 15 hours ago | discuss
8. Show HN: We're tracking AI bot visits daily across our network
   2 points by legitcoders 29 days ago | discuss
9. Using Residential Proxy to Overcome Captcha Barriers
   1 points by yenaidong 18 days ago | discuss
10. Large language model data pipelines and Common Crawl (blog.christianperone.com)
    139 points by sonabinu over 1 year ago | 12 comments
11. Common Crawl maintains a free, open repository of web crawl data (commoncrawl.org)
    27 points by doener 9 months ago | 1 comments
12. Show HN: Help improve language coverage in Common Crawl
    8 points by ccgreg 5 months ago | discuss
13. Ask HN: Is there a service that offers Common Crawl as an API?
    7 points by georgehill 6 months ago | 3 comments
14. Publishers Target Common Crawl in Fight over AI Training Data (www.wired.com)
    4 points by ccgreg over 1 year ago | 1 comments
15. Training Data for the Price of a Sandwich: Common Crawl's Impact on Gen AI (foundation.mozilla.org)
    3 points by cpeterso almost 2 years ago | 1 comments
16. Training Data for the Price of a Sandwich: Common Crawl's Impact on Generative (foundation.mozilla.org)
    3 points by todsacerdoti almost 2 years ago | discuss
17. Cc-downloader: command-line tool for downloading Common Crawl data via HTTPS (github.com)
    2 points by simonpure 9 months ago | discuss
18. Common Crawl May/June 2024 Newsletter (commoncrawl.org)
    2 points by nceqs3 over 1 year ago | discuss
19. C4: colossal cleaned version of Common Crawl's web crawl corpus (huggingface.co)
    2 points by tosh over 1 year ago | discuss
20. Large language model data pipelines and Common Crawl (WARC/WAT/WET) formats (blog.christianperone.com)
    2 points by perone almost 2 years ago | discuss
21. Common Crawl and Unlocking Web Archives for Research (2017) (www.forbes.com)
    1 points by josh-sematic almost 2 years ago | 1 comments
22. Discovering Shopify Domains: A Journey Through Common Crawl Data (alistechtales.substack.com)
    1 points by gmays over 1 year ago | discuss
23. Common Crawl Statistics Now Available on Hugging Face (commoncrawl.org)
    1 points by nceqs3 over 1 year ago | discuss
24. ChatNoir Common Crawl Search (www.chatnoir.eu)
    1 points by nerdo almost 2 years ago | discuss
25. Show HN: Rhiza – easily create shortcuts and add entries to PATH (github.com)
    2 points by skardy 9 months ago | discuss
26. Show HN: Command line tool for extracting secrets from WARC (Web ARChive) files (github.com)
    2 points by neathack almost 2 years ago | discuss
27. Test Processing Practice Expt (ETL) DNS Data from WARC yy074
    1 points by 1vuio0pswjnm7 almost 2 years ago | 7 comments
28. Ask HN: How Spam are removed from large datasets of web crawls for AI training
    1 points by arromatic over 1 year ago | discuss
29. Legality of Publishing Web Crawls (2020) (skeptric.com)
    1 points by phewlink over 1 year ago | discuss
30. Crawler operators, please stop destroying the commons (lunnova.dev)
    7 points by xena 7 months ago | discuss
31. Show HN: I built an integration to update OpenAI assistant with website crawls (github.com)
    1 points by jirispilka over 1 year ago | discuss
32. Show HN: Free tool to find RSS feeds, even if not linked on the page
    152 points by domysee about 1 year ago | 48 comments
33. Show HN: Wispbit - Linter for AI coding agents (wispbit.com)
    31 points by dearilos about 1 month ago | 14 comments
34. Show HN: Startkit.ai – Boilerplate for building products with AI (startkit.ai)
    25 points by jivings over 1 year ago | discuss
35. Quick but powerful research for AI agents with data scrapping and selenium
    8 points by alexvomwald 9 months ago | 6 comments
36. Show HN: A dev-friendly, SEO-optimized router for modern React applications (github.com)
    4 points by floydnoel about 1 year ago | 1 comments
37. Show HN: Wispbit – Keep codebase standards alive (wispbit.com)
    3 points by dearilos 3 months ago | discuss
38. Show HN: Copy from URL – A tiny tool to bypass robots.txt for AI chatbot (copyfromurl.com)
    2 points by ngkw over 1 year ago | 6 comments
39. I created a new training dataset for stale SOTA LLMs
    2 points by rileygersh 4 months ago | discuss