f******y
发帖数: 2971 | 1 Sorry I cannot type Chinese now.
Two big data frame, each is approximately 1.5G.
After I merge them, this object becomes 1.7G, however memory used by R
increased 5G. I tried to release the extra memory with gc(). It does not
work. I suspect there is some cache in merge() function that holds the
memory. Is there anyway to free those memory?
I tried to quite R and save workspace, after I reopen R and load the
workspace, those memory is released. Certainly I dont want to do this from
time to time. |
f******y
发帖数: 2971 | 2 btw, merging them does not take a very long time, just 5~10 mins.
【在 f******y 的大作中提到】
: Sorry I cannot type Chinese now.
: Two big data frame, each is approximately 1.5G.
: After I merge them, this object becomes 1.7G, however memory used by R
: increased 5G. I tried to release the extra memory with gc(). It does not
: work. I suspect there is some cache in merge() function that holds the
: memory. Is there anyway to free those memory?
: I tried to quite R and save workspace, after I reopen R and load the
: workspace, those memory is released. Certainly I dont want to do this from
: time to time.
|
s*****n
发帖数: 2174 | 3 not sure about the exact cause, but a friendly reminder that
when you merge(A, B), the object A and B are still in your memory.
so if A and B are 1.5G each, c <- merge(A, B) is 1.7G,
now 3 objects are in your workspace now.
【在 f******y 的大作中提到】
: Sorry I cannot type Chinese now.
: Two big data frame, each is approximately 1.5G.
: After I merge them, this object becomes 1.7G, however memory used by R
: increased 5G. I tried to release the extra memory with gc(). It does not
: work. I suspect there is some cache in merge() function that holds the
: memory. Is there anyway to free those memory?
: I tried to quite R and save workspace, after I reopen R and load the
: workspace, those memory is released. Certainly I dont want to do this from
: time to time.
|
P****D
发帖数: 11146 | 4 Big bull is indeed big bull...
Then is there a way to unload A and B from memory?
【在 s*****n 的大作中提到】
: not sure about the exact cause, but a friendly reminder that
: when you merge(A, B), the object A and B are still in your memory.
: so if A and B are 1.5G each, c <- merge(A, B) is 1.7G,
: now 3 objects are in your workspace now.
|
O*****y
发帖数: 222 | 5 rm(list=c("A", "B"))
【在 P****D 的大作中提到】
: Big bull is indeed big bull...
: Then is there a way to unload A and B from memory?
|
P****D
发帖数: 11146 | 6 Thanks!
【在 O*****y 的大作中提到】
: rm(list=c("A", "B"))
|
f******y
发帖数: 2971 | 7 I already did this.
my code is like:
A = merge(A, B);
rm(B);
gc();
【在 s*****n 的大作中提到】
: not sure about the exact cause, but a friendly reminder that
: when you merge(A, B), the object A and B are still in your memory.
: so if A and B are 1.5G each, c <- merge(A, B) is 1.7G,
: now 3 objects are in your workspace now.
|
a***d
发帖数: 336 | 8 I ran into this many times too..
My experience is: once R reaches Mem limit, gc() does not help much. R has
to be saved and reloaded. Do someone know how to get around this?
【在 f******y 的大作中提到】
: I already did this.
: my code is like:
: A = merge(A, B);
: rm(B);
: gc();
|
t*d
发帖数: 1290 | 9 try gc() multiple times.
if not working, get a linux.
【在 a***d 的大作中提到】
: I ran into this many times too..
: My experience is: once R reaches Mem limit, gc() does not help much. R has
: to be saved and reloaded. Do someone know how to get around this?
|
f******y
发帖数: 2971 | 10 我的就是linux，倒是没有gc很多次。不过我觉得没用。
【在 t*d 的大作中提到】
: try gc() multiple times.
: if not working, get a linux.
|
|
|
t*d
发帖数: 1290 | 11 试试吧，还是有用的。
【在 f******y 的大作中提到】
: 我的就是linux，倒是没有gc很多次。不过我觉得没用。
|
g********r
发帖数: 8017 | 12 save新object，清空所有object，gc（），load新object，行不行？ |
R*********r
发帖数: 225 | 13 这么多数据为啥不用数据库搞，处理完了再load到R里好了。
【在 f******y 的大作中提到】
: Sorry I cannot type Chinese now.
: Two big data frame, each is approximately 1.5G.
: After I merge them, this object becomes 1.7G, however memory used by R
: increased 5G. I tried to release the extra memory with gc(). It does not
: work. I suspect there is some cache in merge() function that holds the
: memory. Is there anyway to free those memory?
: I tried to quite R and save workspace, after I reopen R and load the
: workspace, those memory is released. Certainly I dont want to do this from
: time to time.
|
B******5
发帖数: 4676 | 14 嗯，这是更好的办法
【在 R*********r 的大作中提到】
: 这么多数据为啥不用数据库搞，处理完了再load到R里好了。
|
f******y
发帖数: 2971 | 15 我宁愿用C++，那种全sql的东西太难维护了。
【在 R*********r 的大作中提到】
: 这么多数据为啥不用数据库搞，处理完了再load到R里好了。
|
B******5
发帖数: 4676 | 16 没觉得C++好维护。。。
【在 f******y 的大作中提到】
: 我宁愿用C++，那种全sql的东西太难维护了。
|
