G*********a
发帖数: 1080 | 1 it's such a happy thing improve the speed of my crap code when dealing with
that huge dataset. anyone knows any particular place talking about this type
of similar topics? | w*r
发帖数: 2421 | 2 yes, that sounds terrific, creating object is resource consuming, avoiding
create too many instance is very important, however, sometime it is not quite
easy in java as the language does not provide lightweighted struct mechanism.
for loop is a controversial issue, using the for loop can help compiler to
inline your code, hoewver, reckless design for loop without checking the real
boundary of loop is a waste of system resource. i would say if you know that
your loop will definitely executed for
【在 G*********a 的大作中提到】
: it's such a happy thing improve the speed of my crap code when dealing with
: that huge dataset. anyone knows any particular place talking about this type
: of similar topics?
| G*********a
发帖数: 1080 | 3 exactly! To segment the huge data into smaller chunks at beginning also helps.
It's really important to control the memory usage, once my crab code crash my
computer when i run it on a large microarray data.
from
to
of
【在 w*r 的大作中提到】
: yes, that sounds terrific, creating object is resource consuming, avoiding
: create too many instance is very important, however, sometime it is not quite
: easy in java as the language does not provide lightweighted struct mechanism.
: for loop is a controversial issue, using the for loop can help compiler to
: inline your code, hoewver, reckless design for loop without checking the real
: boundary of loop is a waste of system resource. i would say if you know that
: your loop will definitely executed for
| w*r
发帖数: 2421 | 4 for these codes, they should compile the same way, compiler may unrolling the
body of the loop multiple times to eliminate the testing at the end/begnning
of the loop,
other sittuations may be differnet,
t=0
while(t<1000){
if(j){
then c[t]=j(p)
t++
}
else{
c=f(p,t)
t++
}
}
if compiler is smart enough, it may figure out this simple program, if the
optimization code is not good enough, it may turn into different structure, A
good c | G*********a
发帖数: 1080 | 5 exactly! To segment the huge data into smaller chunks at beginning also helps.
It's really important to control the memory usage, once my crab code crash my
computer when i run it on a large microarray data.
from
to
of
【在 w*r 的大作中提到】
: for these codes, they should compile the same way, compiler may unrolling the
: body of the loop multiple times to eliminate the testing at the end/begnning
: of the loop,
: other sittuations may be differnet,
: t=0
: while(t<1000){
: if(j){
: then c[t]=j(p)
: t++
: }
| G*********a
发帖数: 1080 | 6 oh, sorry, what i refered to was something like this situation: to go through
an ArrayList object, you can use either "for loop" or an "iterator" for this
ArrayList. i think iteraotor is faster than for loop here. | w*r
发帖数: 2421 | 7 dealing with large dataset is always a difficult issue. situations varies from
on to the other I do not think there will be any sort of universal/generic
solution to optimize the performance under this scenario. The no1 rule is to
go ahead find the complexity of your code. Try to identify the possible way to
reduce the complexity of your code. Try to be conservative on the memory
usage of your code. Deposite the result of your compuration for each block of
data out of your program memory as earl
【在 G*********a 的大作中提到】
: it's such a happy thing improve the speed of my crap code when dealing with
: that huge dataset. anyone knows any particular place talking about this type
: of similar topics?
| G*********a
发帖数: 1080 | 8 i think iterator is much faster than for loop, whenever it can be used.
quite
.
real
,
【在 w*r 的大作中提到】
: yes, that sounds terrific, creating object is resource consuming, avoiding
: create too many instance is very important, however, sometime it is not quite
: easy in java as the language does not provide lightweighted struct mechanism.
: for loop is a controversial issue, using the for loop can help compiler to
: inline your code, hoewver, reckless design for loop without checking the real
: boundary of loop is a waste of system resource. i would say if you know that
: your loop will definitely executed for
|
| w*r
发帖数: 2421 | 9 streamlize your data into a feeder object(if your computation can be seperated
), feed the data to multiple light weighted computing thread and have a result
deposite controller to stream your data out. Carefully handle the number of
your computation threads so that the computation thread can be constantly
busying with its task without been held by the process of data processing.
Data Feeder (data pre-process/feed to thread) --> (computation thread)*n -->
result deposite bank(handle muliple requ
【在 G*********a 的大作中提到】
: exactly! To segment the huge data into smaller chunks at beginning also helps.
: It's really important to control the memory usage, once my crab code crash my
: computer when i run it on a large microarray data.
:
: from
: to
: of
| g*****g
发帖数: 34805 | 10 I don't see any difference.
between
i=0;
while(i
i++;
}
and for(i=0;i
}
To my understanding, they would be compiled to the same assembly.
【在 G*********a 的大作中提到】
: i think iterator is much faster than for loop, whenever it can be used.
:
: quite
: .
: real
: ,
| w*r
发帖数: 2421 | 11 yep
【在 G*********a 的大作中提到】
: i think iterator is much faster than for loop, whenever it can be used.
:
: quite
: .
: real
: ,
| G*********a
发帖数: 1080 | 12 oh, sorry, what i refered to was something like this situation: to go through
an ArrayList object, you can use either "for loop" or an "iterator" for this
ArrayList. i think iteraotor is faster than for loop here.
【在 g*****g 的大作中提到】
: I don't see any difference.
: between
: i=0;
: while(i: i++;
: }
: and for(i=0;i: }
: To my understanding, they would be compiled to the same assembly.
| g*****g
发帖数: 34805 | 13 I don't see any difference.
between
i=0;
while(i
i++;
}
and for(i=0;i
}
To my understanding, they would be compiled to the same assembly.
【在 G*********a 的大作中提到】
: i think iterator is much faster than for loop, whenever it can be used.
:
: quite
: .
: real
: ,
|
|
