y********0
发帖数: 638 | 1 I am more recently working with 80 dataframes in python each of which holds
650K rows with the same 11 columns varying from string to number. My initial
goal is to concatenate all dataframes and make a general analysis but I keep
hitting the dead ends in doing so.
1. I started from pd.concat([df1,df2]) but ended up with memory error and
failed to figure out a solution after searching over internet.
2.I then, checking on some online suggestions, converted the number into
float32 for the reduction of memory burden and this time only found it not
helpful either.
3. If you are asking for a little background of my PC and python,I am using
a 32-bit Python with 2.7.5, pandas 0.13.0 and little chance to switch to 64
version.
Any comments, suggestions will be really appreciated. | Z**0
发帖数: 1119 | 2 You could not do anything given your current configuration.
The final datafame is roughly about 4GiB (float64, aka 8 bytes per element),
which won't work on 32-bit at all. Your 32-bit program could only request
2GiB memory block either. | y********0
发帖数: 638 | 3 Thanks Zer0.
),
【在 Z**0 的大作中提到】
: You could not do anything given your current configuration.
: The final datafame is roughly about 4GiB (float64, aka 8 bytes per element),
: which won't work on 32-bit at all. Your 32-bit program could only request
: 2GiB memory block either.
|
|
