F****3 发帖数: 1504 | 1 没办法要处理大数据,按时间分布的。
请问这种情况一定要把数据拆开处理吗?
我把小数据合并以后连sort都没有办法了。但是实在pc环境下,是不是在linux下面这
个问题就解决了?
谢谢 |
S******y 发帖数: 1123 | 2 You will spend a lot time in just reading/writing records of file due to I/O
bound in SAS.
Please define what you mean by "处理". It is very different to process one
record a time vs. to handle a record based on how other records being
related to it.
If you need to sort data first, try using 'sort' command on Unix/Linux.
Your might want to think other options such as Python, which could be faster.
Just my 2 cents.. |
c*****a 发帖数: 808 | |
F****3 发帖数: 1504 | 4 谢谢!
做的事情很简单,就是sort,merge,OLS regression。 硬盘足够,不知道为什么说资
源不够。学校服务器限制容量,我这个data只能写到scratch里面去,自己的folder装
不下。
很郁闷。
请问100G的data就算打了吗?哪银行什么的怎么办啊? |
F****3 发帖数: 1504 | 5 谢谢!
请问你是指linux 命令环境下的sort吗?
/O
faster.
【在 S******y 的大作中提到】![](/moin_static193/solenoid/img/up.png) : You will spend a lot time in just reading/writing records of file due to I/O : bound in SAS. : Please define what you mean by "处理". It is very different to process one : record a time vs. to handle a record based on how other records being : related to it. : If you need to sort data first, try using 'sort' command on Unix/Linux. : Your might want to think other options such as Python, which could be faster. : Just my 2 cents..
|
t*****w 发帖数: 254 | 6 this is really big. How many observations are there in your big dataset. if
there are 1 billion observation, I am wondering whether SAS or R can take
care of it or not. If not, you should randomize all observations first and
then withdraw 10% data to do regression. |
l****9 发帖数: 115 | 7 在PC上处理过30G的文件,10亿个观察,数个变量,主要是比较复杂的查表操作。
这么大的数据对编程的技巧要求还是很高的,不过不是不能做,特别是分析只是部分只
是OLS。
如果变量多,观测值稍微少点,可以做索引,不直接排序,排序需要额外很大的空间 |
F****3 发帖数: 1504 | 8 thank you, everybody!
i redirect my work lib to d drive, which as much more space. problem solved
. it took about 4 hours to sort.
there are around 114 million obs... |
F****3 发帖数: 1504 | 9 thanks for the good idea. do ols on the sample, haha.
if
【在 t*****w 的大作中提到】![](/moin_static193/solenoid/img/up.png) : this is really big. How many observations are there in your big dataset. if : there are 1 billion observation, I am wondering whether SAS or R can take : care of it or not. If not, you should randomize all observations first and : then withdraw 10% data to do regression.
|
l*******s 发帖数: 1258 | 10 排序部分就是个标准的大数据处理case
把数据分成n部分 分别把这个n块data读入内存排序
然后再用类似于n路merge sorting的方法把所有data排序 |
q***m 发帖数: 9 | 11 sort和merge 不要用 SAS, 可以考虑使用AWK之类的工具,
至于后面的OLS, 如果仅仅是y=ax+b的形式的话, 可以试试RMA的方法 |
F****3 发帖数: 1504 | 12 谢谢!请问你说的awk会比python快吗?主要是要用学校服务器,没有办法自己装软件
。。。
另外,我是门外汉,请问RMA是不是就是Microarray analysis techniques
可以用全部obs,不需要surveyreg是吧?谢谢啊!
【在 q***m 的大作中提到】![](/moin_static193/solenoid/img/up.png) : sort和merge 不要用 SAS, 可以考虑使用AWK之类的工具, : 至于后面的OLS, 如果仅仅是y=ax+b的形式的话, 可以试试RMA的方法
|
q***m 发帖数: 9 | 13 AWK速度很快,因为优化的很好。Windows上面也有gawk的,不过我没用过。
RMA不是你说的那个的,是针对于y=ax+b这种形式的参数估算方法,技术细节是:先求x
和y的Standard Deviation, a=SD(y)/(x), b=mean(y)-a*mean(x)
这种方法的好处是稳健性比较好,不过已经不属于常规的OLS了,你可以用采样的数据
做ols 然后跟这个RMA的方法做比较。 |
c**d 发帖数: 104 | 14 就是sort,merge,OLS regression。 硬盘足够,不知道为什么说资
源不够。学校服务器限制容量,我这个data只能写到scratch里面去,自己的folder装
不下。
1: you should start to use hash object in sas to merge data.
2: the matrix is too big, SAS can not handle it. but SAS has HPprocedures to
deal with big data. for example,it only takes 80 seconds to fit a logistic
with 1B records with several variables. please check if your school has that
license. SAS9.4 is coming this month.
3: recommend using a holdout sample and cross-validation. |