l********g
发帖数: 39 | 1 I have some large text files formated as 12 columns, but with a very large
number of rows, say one million lines. I have about 10 of this kind of files
, maybe more up to 100, which may make a total of several GBs. Thinking
about using cat to merge them.
something like this: $ cat smalljunks*.txt > newlargejunk.txt
The questions:
can the 'cat' command handle these large files? is it possible that it
truncates the files to fit to its limit (if there is any) so that the merge
is incomplete? Becau | x****s
发帖数: 921 | 2 when you have the time, why not try out yourself?
dd if=/dev/zero of=/mnt/file1 bs=1M count=800 | N**D
发帖数: 10322 | 3 1 million lines is piece of cake
large file is usually in the order of TB today
large
files
merge
【在 l********g 的大作中提到】
: I have some large text files formated as 12 columns, but with a very large
: number of rows, say one million lines. I have about 10 of this kind of files
: , maybe more up to 100, which may make a total of several GBs. Thinking
: about using cat to merge them.
: something like this: $ cat smalljunks*.txt > newlargejunk.txt
: The questions:
: can the 'cat' command handle these large files? is it possible that it
: truncates the files to fit to its limit (if there is any) so that the merge
: is incomplete? Becau
| b******r
发帖数: 79 | 4 好像记得cat 是用流的方式读文件， 读一行出一行， 这样的话应该就没有太多的内存的
问题吧。 不是很确定， 谁能clarify一下？
large
files
merge
【在 l********g 的大作中提到】
: I have some large text files formated as 12 columns, but with a very large
: number of rows, say one million lines. I have about 10 of this kind of files
: , maybe more up to 100, which may make a total of several GBs. Thinking
: about using cat to merge them.
: something like this: $ cat smalljunks*.txt > newlargejunk.txt
: The questions:
: can the 'cat' command handle these large files? is it possible that it
: truncates the files to fit to its limit (if there is any) so that the merge
: is incomplete? Becau
| l********g
发帖数: 39 | 5 Thank you very much for helping, everyone. It seems work with large files I
have. |
|
