Hackernews posts about Common Crawl

1. Show HN: I built a 50 site sampler from CommonCrawl refreshing every 30 minutes (randcrawl.com)
   1 points by whothatcodeguy 18 days ago | discuss
2. Show HN: Randomly discovered websites from the open internet every 60 minutes (randcrawl.com)
   4 points by whothatcodeguy 17 days ago | 1 comments
3. Show HN: CrawlerCheck v1.5 – Operators Directory and 25 new AI crawlers (crawlercheck.com)
   1 points by bogozi 12 days ago | discuss
4. Show HN: Global Issue Memory MCP – Stack Overflow for Your Coding Assistant (www.usegim.com)
   3 points by nonekme 1 day ago | discuss
5. Analyzing the Fraud Vector: How Ormix Uses Mock Trading Engines to Trap Capital
   1 points by Lisa1314 22 days ago | 1 comments
6. Large language model data pipelines and Common Crawl (blog.christianperone.com)
   139 points by sonabinu over 1 year ago | 12 comments
7. Common Crawl maintains a free, open repository of web crawl data (commoncrawl.org)
   27 points by doener 12 months ago | 1 comments
8. Ask HN: Is Common Crawl used exhaustively by any search engine?
   8 points by n1xis10t 4 months ago | 2 comments
9. Show HN: Help improve language coverage in Common Crawl
   8 points by ccgreg 8 months ago | discuss
10. Ask HN: Is there a service that offers Common Crawl as an API?
    7 points by georgehill 10 months ago | 3 comments
11. Publishers Target Common Crawl in Fight over AI Training Data (www.wired.com)
    4 points by ccgreg over 1 year ago | 1 comments
12. Cc-downloader: command-line tool for downloading Common Crawl data via HTTPS (github.com)
    2 points by simonpure about 1 year ago | discuss
13. Common Crawl May/June 2024 Newsletter (commoncrawl.org)
    2 points by nceqs3 over 1 year ago | discuss
14. 200M unique domains extracted from Common Crawl (zenodo.org)
    1 points by networkcat about 1 month ago | discuss
15. Discovering Shopify Domains: A Journey Through Common Crawl Data (alistechtales.substack.com)
    1 points by gmays over 1 year ago | discuss
16. Common Crawl Statistics Now Available on Hugging Face (commoncrawl.org)
    1 points by nceqs3 over 1 year ago | discuss
17. Show HN: Rhiza – easily create shortcuts and add entries to PATH (github.com)
    2 points by skardy 12 months ago | discuss
18. Humanties Last War
    1 points by vlan121 4 months ago | 1 comments
19. Ask HN: How Spam are removed from large datasets of web crawls for AI training
    1 points by arromatic almost 2 years ago | discuss
20. The Company Quietly Funneling Paywalled Articles to AI Developers (www.theatlantic.com)
    33 points by breve 4 months ago | 16 comments
21. The Nonprofit Feeding the Internet to AI Companies (www.theatlantic.com)
    10 points by ForHackernews 4 months ago | 5 comments
22. The Nonprofit Doing the AI Industry's Dirty Work (www.theatlantic.com)
    9 points by kgwgk 4 months ago | 2 comments
23. The Company Funneling Paywalled Articles to AI Developers (www.theatlantic.com)
    3 points by CaptainZapp 4 months ago | discuss
24. Legality of Publishing Web Crawls (2020) (skeptric.com)
    1 points by phewlink over 1 year ago | discuss
25. Crawler operators, please stop destroying the commons (lunnova.dev)
    7 points by xena 10 months ago | discuss
26. Show HN: I built an integration to update OpenAI assistant with website crawls (github.com)
    1 points by jirispilka over 1 year ago | discuss
27. Show HN: Free tool to find RSS feeds, even if not linked on the page
    152 points by domysee over 1 year ago | 48 comments
28. Show HN: Wispbit - Linter for AI coding agents (wispbit.com)
    31 points by dearilos 4 months ago | 14 comments
29. Show HN: Startkit.ai – Boilerplate for building products with AI (startkit.ai)
    25 points by jivings almost 2 years ago | discuss
30. Quick but powerful research for AI agents with data scrapping and selenium
    8 points by alexvomwald about 1 year ago | 6 comments
31. Show HN: A dev-friendly, SEO-optimized router for modern React applications (github.com)
    4 points by floydnoel over 1 year ago | 1 comments
32. Show HN: Wispbit – Keep codebase standards alive (wispbit.com)
    3 points by dearilos 6 months ago | discuss
33. Show HN: Copy from URL – A tiny tool to bypass robots.txt for AI chatbot (copyfromurl.com)
    2 points by ngkw over 1 year ago | 6 comments
34. Show HN: Memtui – A terminal UI for browsing Memcached like a file tree (github.com)
    2 points by sr-white about 1 month ago | discuss
35. Show HN: Skillz – Use Claude Skills Anywhere (github.com)
    2 points by kevinslin 3 months ago | discuss