Martyr
Well-known member=150Looking for Large Real-World Datasets (20–40GB+) for Learning
Guys, I’m looking to build a data pool or maybe a small data lake – not fully decided yet, it depends on the types of logs I end up working with.[conversation_history:1]
I’m specifically hunting for large, real datasets in the 20–40 GB range (or more): things like log files, telemetry, events, etc. Not garbage sample data with repeated or meaningless lines – I want something realistic enough that I can actually learn how to:
- Store and organize big datasets
- Run custom queries in (near) real time
- Get search results back in minutes at scale
[conversation_history:1]
This is purely for learning and experimentation. It will not be sold, shared as a “product,” or used commercially in any way – it’s just for personal R&D and skill-building.[conversation_history:1]
If you know:
- Public/open datasets with large log-style data
- Any sources where big realistic DB dumps are available
- Or tips on what stack to target (e.g., data lake vs “just” a big DB) for this size of data
Please drop any leads or ideas. Any pointers would really help.[conversation_history:1]
Guys, I’m looking to build a data pool or maybe a small data lake – not fully decided yet, it depends on the types of logs I end up working with.[conversation_history:1]
I’m specifically hunting for large, real datasets in the 20–40 GB range (or more): things like log files, telemetry, events, etc. Not garbage sample data with repeated or meaningless lines – I want something realistic enough that I can actually learn how to:
- Store and organize big datasets
- Run custom queries in (near) real time
- Get search results back in minutes at scale
[conversation_history:1]
This is purely for learning and experimentation. It will not be sold, shared as a “product,” or used commercially in any way – it’s just for personal R&D and skill-building.[conversation_history:1]
If you know:
- Public/open datasets with large log-style data
- Any sources where big realistic DB dumps are available
- Or tips on what stack to target (e.g., data lake vs “just” a big DB) for this size of data
Please drop any leads or ideas. Any pointers would really help.[conversation_history:1]