d*2
发帖数: 2053 | 1 RHEL 5.8 x64
Nexus 5548, (10Gbit switch and 10Gbit card from Intel), dedicated switch
scp from one node to another node, only get about 60MB/s~70MB/s
tftp get 600MB/s~700MB/s
scp will do some encryption/decryption, but the overhead seems to be too
much? |
j*a
发帖数: 14423 | 2 did you try ftp?
【在 d*2 的大作中提到】
: RHEL 5.8 x64
: Nexus 5548, (10Gbit switch and 10Gbit card from Intel), dedicated switch
: scp from one node to another node, only get about 60MB/s~70MB/s
: tftp get 600MB/s~700MB/s
: scp will do some encryption/decryption, but the overhead seems to be too
: much?
|
c*a
发帖数: 806 | 3 This is odd, scp should also compress the data, and should be faster than
that. When transferring with scp, what about CPU utilization?
【在 d*2 的大作中提到】
: RHEL 5.8 x64
: Nexus 5548, (10Gbit switch and 10Gbit card from Intel), dedicated switch
: scp from one node to another node, only get about 60MB/s~70MB/s
: tftp get 600MB/s~700MB/s
: scp will do some encryption/decryption, but the overhead seems to be too
: much?
|
z**r
发帖数: 17771 | 4 I think this is normal, scp uses a lot of CPU cycles.
【在 d*2 的大作中提到】
: RHEL 5.8 x64
: Nexus 5548, (10Gbit switch and 10Gbit card from Intel), dedicated switch
: scp from one node to another node, only get about 60MB/s~70MB/s
: tftp get 600MB/s~700MB/s
: scp will do some encryption/decryption, but the overhead seems to be too
: much?
|
n**********l
发帖数: 271 | 5 Have you checked sshd settings(e.g. encryption? try a shorter key maybe? )?
since you are using 10G NIC I assume cpu & RAM hopefully won't be the
bottleneck. What about cpu utilization on each core?
What kind of testing are u performing? I am totally shocked by 700MB/s over
one tftp connection...
Do you have a comparison using ftp vs sftp? IMO scp is not efficient enough
for large file transfers |
a****a
发帖数: 98 | 6
Maybe you can try using the blowfish cipher with scp
Enabling compression might help too
【在 d*2 的大作中提到】
: RHEL 5.8 x64
: Nexus 5548, (10Gbit switch and 10Gbit card from Intel), dedicated switch
: scp from one node to another node, only get about 60MB/s~70MB/s
: tftp get 600MB/s~700MB/s
: scp will do some encryption/decryption, but the overhead seems to be too
: much?
|
a***n
发帖数: 262 | 7 Try
High Performance SSH/SCP - HPN-SSH
http://www.psc.edu/index.php/hpn-ssh
?
【在 d*2 的大作中提到】
: RHEL 5.8 x64
: Nexus 5548, (10Gbit switch and 10Gbit card from Intel), dedicated switch
: scp from one node to another node, only get about 60MB/s~70MB/s
: tftp get 600MB/s~700MB/s
: scp will do some encryption/decryption, but the overhead seems to be too
: much?
|
n*********a
发帖数: 1956 | 8 Perhaps the bottleneck is your CPU, not your network. |
d*2
发帖数: 2053 | 9 no ftp server running other than tftp
【在 j*a 的大作中提到】
: did you try ftp?
|
d*2
发帖数: 2053 | 10 during scp, out of the 16 cores, only one core is used and usage is up to 94
%, most of the time 70%~85%
【在 c*a 的大作中提到】
: This is odd, scp should also compress the data, and should be faster than
: that. When transferring with scp, what about CPU utilization?
|
|
|
d*2
发帖数: 2053 | 11 just tftp to another node
for each node: 16-cores total (E*****[email protected]), 192GB RAM on each node
?
over
enough
【在 n**********l 的大作中提到】
: Have you checked sshd settings(e.g. encryption? try a shorter key maybe? )?
: since you are using 10G NIC I assume cpu & RAM hopefully won't be the
: bottleneck. What about cpu utilization on each core?
: What kind of testing are u performing? I am totally shocked by 700MB/s over
: one tftp connection...
: Do you have a comparison using ftp vs sftp? IMO scp is not efficient enough
: for large file transfers
|
d*2
发帖数: 2053 | 12 for each node: 16-cores total (E*****[email protected]), 192GB RAM on each node
【在 n*********a 的大作中提到】
: Perhaps the bottleneck is your CPU, not your network.
|
n*********a
发帖数: 1956 | 13 but your ssh/scp application itself may not support parallel encryption.
all other cores are wasted.
【在 d*2 的大作中提到】
: for each node: 16-cores total (E*****[email protected]), 192GB RAM on each node
|
d*2
发帖数: 2053 | 14 it is just scp command and yes, obviously it only uses one core.
but the overhead is still kind of too high...
Also tested on two different （but with similar configurations) clusters,
got the same result.
【在 n*********a 的大作中提到】
: but your ssh/scp application itself may not support parallel encryption.
: all other cores are wasted.
|
c******n
发帖数: 4965 | 15 we had this problem
its due to the buffer in scp
try look up scp from Pittsburgh supercomputer center
they have a version with larger bufeer
also u need to bump up tcp windows
【在 d*2 的大作中提到】
: RHEL 5.8 x64
: Nexus 5548, (10Gbit switch and 10Gbit card from Intel), dedicated switch
: scp from one node to another node, only get about 60MB/s~70MB/s
: tftp get 600MB/s~700MB/s
: scp will do some encryption/decryption, but the overhead seems to be too
: much?
|
z**r
发帖数: 17771 | 16 lz能不能confirm一下这个solution可行否？
【在 c******n 的大作中提到】
: we had this problem
: its due to the buffer in scp
: try look up scp from Pittsburgh supercomputer center
: they have a version with larger bufeer
: also u need to bump up tcp windows
|
d*2
发帖数: 2053 | 17 sorry, have been busy (well-known excuse...) with other things, but will try
the high performance scp
【在 z**r 的大作中提到】
: lz能不能confirm一下这个solution可行否？
|
