a********a
发帖数: 346 | 1 My R program theoretically will give me an output 24000(row)*8(column). But
it only gives me around 60(row)*8(col) and stopped with the following error.
Error: cannot allocate vector of size 1.8 Gb
Do you know how to adjust the R memory?
Thanks a lot. |
g********r
发帖数: 8017 | 2 24000*8用那么大内存？还是你计算过程用了更大矩阵?
如果真需要1.8G，看看物理内存够不够。
如果够，如果是windows，设置memory.size().
如果还不行，用library（bigmemory）。
我还是觉得24000*8只需要很小的内存。
But
error.
【在 a********a 的大作中提到】
: My R program theoretically will give me an output 24000(row)*8(column). But
: it only gives me around 60(row)*8(col) and stopped with the following error.
: Error: cannot allocate vector of size 1.8 Gb
: Do you know how to adjust the R memory?
: Thanks a lot.
|
a********a
发帖数: 346 | 3 Thanks goldmember,
The memory in my computer is only 1G. The problem is now I only want to get
120*8 matrix to test the program, but it stopped only give an output 66*8.
I tried to use library(bigmemory), it does not work.
If I use memory.size(max = TRUE), the memory only have a size 488.31
memory.size(max = TRUE)
>488.31
Do you have any other idea? |
a********a
发帖数: 346 | 4 No, I can not divide smaller unit than I am using now since I want to run
200 bootstraps one time, but now I only run 1 bootstrap, no way to run half
bootstrap. The problem is it stopped without finishing one bootstrap. |
g********r
发帖数: 8017 | 5 memory.size是用来设置最大内存的。但是你只有1G，不能分配1.8.
bigmemory不是call一下就可以的，你要用里面的方法来建立大矩阵。然后才能开始赋
值。
我觉得最关键的问题是你的矩阵根本不需要上G的内存。一定是什么地方出了问题。你
用什么
产生这个矩阵？
get
【在 a********a 的大作中提到】
: Thanks goldmember,
: The memory in my computer is only 1G. The problem is now I only want to get
: 120*8 matrix to test the program, but it stopped only give an output 66*8.
: I tried to use library(bigmemory), it does not work.
: If I use memory.size(max = TRUE), the memory only have a size 488.31
: memory.size(max = TRUE)
: >488.31
: Do you have any other idea?
|
a********a
发帖数: 346 | 6 Thanks goldmember again, here is what I did,
I used cbind statement to combine the matrix,i.e, I have a loop for i in 1
to 15. In the first iteration, I get estimates in 1 row *8 col, second
iteration, I get estimates in 2 row*8col，third iteration in 3*8.....,for
15th iteration, I get 15*8 estimates, then I cbind these estimates to a
matrix with (1+2+......15)row*8col.
Actually I want to repeat this process for 200 times, but it stopped before
it can finish one time. |
g********r
发帖数: 8017 | 7 前面我说错了。是memory。limit（）
cbind非常不efficient。每次你的矩阵都要被复制。你这个情况应该先定义存结果的矩
阵，然后逐行赋值就可以了。
before
【在 a********a 的大作中提到】
: Thanks goldmember again, here is what I did,
: I used cbind statement to combine the matrix,i.e, I have a loop for i in 1
: to 15. In the first iteration, I get estimates in 1 row *8 col, second
: iteration, I get estimates in 2 row*8col，third iteration in 3*8.....,for
: 15th iteration, I get 15*8 estimates, then I cbind these estimates to a
: matrix with (1+2+......15)row*8col.
: Actually I want to repeat this process for 200 times, but it stopped before
: it can finish one time.
|
g********r
发帖数: 8017 | 8 而且你要用的是rbind.如果错用了cbind，可能导致部分矩阵被多次复制粘在一起。那
内存可就大了。
【在 g********r 的大作中提到】
: 前面我说错了。是memory。limit（）
: cbind非常不efficient。每次你的矩阵都要被复制。你这个情况应该先定义存结果的矩
: 阵，然后逐行赋值就可以了。
:
: before
|
a********a
发帖数: 346 | 9 Thanks goldmember again,
Sorry, I used rbind not cbind to combine all the matrix.
Here is the situation,inside each sample, I have a loop for i in 1to 15,
In each iteration of the loop, I get estimates like the following ( here I
only list 4 columns instead of 8 columns, they are fake numbers),
sample category beta beta sigma sigma
1 1 2.0 1.2 2.0 0.8
1 2 1.0 0.8 1.2 0.1
1 2 1.5 0.7 1.3 0.8
1 3 2.0 0.4 1.5 0.2
1 |
g********r
发帖数: 8017 | 10 I don't get it. So you know how to assign values to a submatrix right?
a<-matrix(NA, nrow=24000, ncol=8)
Then pick out any row/column combination to assign values.Like
a[4:7, c(1,3,5,7)]<-......
【在 a********a 的大作中提到】
: Thanks goldmember again,
: Sorry, I used rbind not cbind to combine all the matrix.
: Here is the situation,inside each sample, I have a loop for i in 1to 15,
: In each iteration of the loop, I get estimates like the following ( here I
: only list 4 columns instead of 8 columns, they are fake numbers),
: sample category beta beta sigma sigma
: 1 1 2.0 1.2 2.0 0.8
: 1 2 1.0 0.8 1.2 0.1
: 1 2 1.5 0.7 1.3 0.8
: 1 3 2.0 0.4 1.5 0.2
|
|
|
a********a
发帖数: 346 | 11 thanks goldmember,
I divide the category into 1:8 and 9:15 to run the program separately. It
works well but I have to bind them at the end. |
e*****m
发帖数: 58 | 12 一直好奇R的内存管理。它不使用虚存？
memory.size是用来设置最大内存的。但是你只有1G，不能分配1.8.
bigmemory不是call一下就可以的，你要用里面的方法来建立大矩阵。然后才能开始赋
值。
我觉得最关键的问题是你的矩阵根本不需要上G的内存。一定是什么地方出了问题。你
用什么
产生这个矩阵？
get
【在 g********r 的大作中提到】
: memory.size是用来设置最大内存的。但是你只有1G，不能分配1.8.
: bigmemory不是call一下就可以的，你要用里面的方法来建立大矩阵。然后才能开始赋
: 值。
: 我觉得最关键的问题是你的矩阵根本不需要上G的内存。一定是什么地方出了问题。你
: 用什么
: 产生这个矩阵？
:
: get
|
g********r
发帖数: 8017 | 13 可以使用,但是需要特定的package.比如bigmemory就是.
【在 e*****m 的大作中提到】
: 一直好奇R的内存管理。它不使用虚存？
:
: memory.size是用来设置最大内存的。但是你只有1G，不能分配1.8.
: bigmemory不是call一下就可以的，你要用里面的方法来建立大矩阵。然后才能开始赋
: 值。
: 我觉得最关键的问题是你的矩阵根本不需要上G的内存。一定是什么地方出了问题。你
: 用什么
: 产生这个矩阵？
: get
|
a********a
发帖数: 346 | 14 I used a computer has a memory 2.3G today. But it still have the same
problem. It maybe the problem of the program. But anyway it is running now
by dividing the program into part.
Thanks you all. |
g********r
发帖数: 8017 | 15 If it is windows, you may need to do memory.limit() to make more memory
available to R. Otherwise more
memory doesn't help at all.
I guess the process you use to generate the estimates uses large matrices.
Otherwise it doesn't make sense
for a 24000x8 matrix to occupy so much memory.
【在 a********a 的大作中提到】
: I used a computer has a memory 2.3G today. But it still have the same
: problem. It maybe the problem of the program. But anyway it is running now
: by dividing the program into part.
: Thanks you all.
|
e*****m
发帖数: 58 | 16 按我的理解，虚存应该是R和OS之间要交涉的事情，而对用户透明，用户只管用就可以
乐。
可以使用,但是需要特定的package.比如bigmemory就是.
【在 g********r 的大作中提到】
: 可以使用,但是需要特定的package.比如bigmemory就是.
|
g********r
发帖数: 8017 | 17 我不明白怎么实现的.但是使用bigmemory的时候,系统报告R使用了很大的虚拟内存.我
怀疑这个包用了外部编程语言来实
现,比如qqzj提到的python.那个包的帮助提到,它分配的内存只能pass by reference.
基本上那个语言分配虚存,然后给R一
个指针.
【在 e*****m 的大作中提到】
: 按我的理解，虚存应该是R和OS之间要交涉的事情，而对用户透明，用户只管用就可以
: 乐。
:
: 可以使用,但是需要特定的package.比如bigmemory就是.
|
d******o
发帖数: 59 | 18 sometimes, I think it's easier when you are using SAS IML for large matrix. |
S******y
发帖数: 1123 | 19 Interesting topic ..
I have noticed for Netflix Prize challenge (which is
basically large matrix computation), only C++,
Python, and Java have been widely used ...
SAS and R could not handle large matrix well... (but
I have not tried IML before..)
24000(row)*8(column). But
with the following error.
【在 a********a 的大作中提到】
: My R program theoretically will give me an output 24000(row)*8(column). But
: it only gives me around 60(row)*8(col) and stopped with the following error.
: Error: cannot allocate vector of size 1.8 Gb
: Do you know how to adjust the R memory?
: Thanks a lot.
|
i********f
发帖数: 206 | 20 如果你用的是windows,那么R能用的内存好像是有上限的,2G左右.
你可以还64位的机器,在Linux下运行,这样好像内存没什么限制. |
