top | item 43889861 (no title) broner | 10 months ago Works fine for me on TB+ datasets. Maybe you were doing in-memory rather than persistent database and running out of RAM? https://duckdb.org/docs/stable/clients/cli/overview.html#in-... discuss order hn newest aynyc|10 months ago Wait, do you insert the data from S3 into duckdb? I was just doing select from file. broner|10 months ago Nope, just reading from S3. Check this out: https://duckdb.org/2024/07/09/memory-management.html fastasucan|10 months ago Maybe its your terminal that chockes because it tries to display to much data? 500GB should be no problem.
aynyc|10 months ago Wait, do you insert the data from S3 into duckdb? I was just doing select from file. broner|10 months ago Nope, just reading from S3. Check this out: https://duckdb.org/2024/07/09/memory-management.html fastasucan|10 months ago Maybe its your terminal that chockes because it tries to display to much data? 500GB should be no problem.
broner|10 months ago Nope, just reading from S3. Check this out: https://duckdb.org/2024/07/09/memory-management.html
fastasucan|10 months ago Maybe its your terminal that chockes because it tries to display to much data? 500GB should be no problem.
aynyc|10 months ago
broner|10 months ago
fastasucan|10 months ago