l******n
发帖数: 577 | 1 Here I post the an example from thinking in java:
If you blindly apply the idea of atomicity, you see that getValue( ) in the
following program fits the description:
//: concurrency/AtomicityTest.java
import java.util.concurrent.*;
public class AtomicityTest implements Runnable {
private int i = 0;
public int getValue() { return i; }
private synchronized void evenIncrement() { i++; i++; }
public void run() {
while(true)
evenIncrement();
}
public static voi... 阅读全帖 |
|
w**z
发帖数: 8232 | 2 写C++的时候写过一点，转成Java就很少写了，也不复杂，executor 挺好用。就是前两
天，用shutdownhook 有点麻烦。 |
|
g*****g
发帖数: 34805 | 3 光executor, task之间无关的都不好意思说是多线程。怎么也得是一堆的Future,
CountDownLatch, Lock, 在threadpool里还有先后关系，一不小心会死锁的才能叫多线
程。 |
|
g*****g
发帖数: 34805 | 4 不如说99%的应用都没超出简单的executor+ independent tasks. 我不反对对另外1%优
化，我只不过觉得一比就谈这1%没有抓住主要矛盾。 |
|
l*****t
发帖数: 2019 | 5 future, callable, executor |
|
v*****r
发帖数: 2325 | 6 spark beginner trying out the buzz tech
input 200GB uncompressed data file stored in hdfs
37 worker nodes, each has 24 cores
using java map reduce, 6-8 minutes
using spark, 37 minutes, 2 18 minute-stage
"lightning fast cluster computing, 100x faster" ???!!!!
Big bulls please advise!
#sortMapper sort values for each key, then do some iteration for the grouped
values
text = sc.textFile(input,1776) #24*37*2
text.map(mapper).filter(lambda x: x!=None).groupByKey().map(sortMapper).
filter(lambda x: x... 阅读全帖 |
|
t**********s
发帖数: 930 | 7 默认port 8080 和Oracle database 冲突.
我看到 server.xml 里有这个设置:
="org.apache.coyote.http11.Http11Protocol" redirectPort="${bio.https.port}"/>
就把catalina.properties 里的 bio.http.port 由8080改成80也不行.
每次应用启动都显示 s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started
on port(s): 8080/http
还有其他隐藏的设置吗?
谢谢 |
|
c******f
发帖数: 243 | 8 只用过future + executor service + callable/runnable...
多线的traffic，netty用的比较多 |
|
k**********g
发帖数: 989 | 9
首先所有底层的第三方代码要进行代码审查，看看有没有要修补，要上锁或加倍留意的
地方，例如mutable singleton，thread local data。其中的「锁」会用两种方式确保
正确运作而不锁死∶数据依赖和任务依赖。巩固代码後，下面的工作可分为三方面∶任
务包装，数据包装，界面包装。这包装和encapsulation不同，本质是换汤不换药，不
过是把代码结构稍为修改了。任务包装是把可执行的耗时任务分割为单元，细分为CPU
intensive和IO intensive，再分装为Callable。要确保每一个单元提交到Executor後
，能在任意thread成功执行。数据包装是把数据分类为Write once read many(WORM)或
是mutable。WORM会变成Futures。Mutable按需要可能改写成immutable pattern（类似
copy on modify）或用上述的依赖链使其单线程化。（即使单线程化，不相关的任务仍
可并列执行。）界面包装是把上述的改写用最简洁易用的方式表达，确保接口的组合性
（composability）。
经常会用到 ... 阅读全帖 |
|
h****r
发帖数: 2056 | 10 block on Fugure.get， 好像容易造成thread leak，尤其是有I/O的情况.
经常是jstack发现thread数目在增加，原因是有一些thread在死等i/o,
加上timeout也不是都能解决死等i/o的问题。
这种情况下，executor service（也即thread pool）没有avaialble
thread可用，只好为新request开新的thread，老的thread就继续死等
（或者是有某种exception没有handle而死了）leak掉了。
you |
|
w**z
发帖数: 8232 | 11 it depends on which kind if thread pool the executor service us using.
whether the thread finishes the task has nothing to do with future.get. |
|
c*****e
发帖数: 3226 | 12 很容易的。
class SortSolution extends RecursiveTask {
final File[] files;
final int size;
Solution (File[] files) {
this.files = files;
this.size = files.length;
}
File[] sort() {
if (size== 1)
return files[0].sort();
File[] f1 = new SortSolution (files[0:size/2];
f1.fork();
File[] f2 = new SortSolution (files[size/2:size];
return f2.compute() + f1.join();
}
}
其实这玩意用Guava ListenableFutureTask 也很爽， 因为可以串联。只是这里用不上
ListeningExecutorService servic... 阅读全帖 |
|
f********r
发帖数: 304 | 13
非常同意，spark尤其是spark SQL，很多stage的shuffle read/write数量非常惊人，
根本不可能全不in memory，所以spill disk很常见。而且如果你不手动cache或者
persist rdd，很多rdd会recompute，这个在写程序的时候要非常注意。spark虽然nb但
也不是什么ultimate solution，对programming optimization的要求很高的。而且真
正上大数据的时候，如果没有powerful硬件支持，executor经常timeout 或者OOM，很
不容易debug。 |
|
c*********e
发帖数: 16335 | 14 别动不动就是task,thread.
java里面都是用的callable,如果用executor的话。 |
|
|
l*****g
发帖数: 138 | 16 这个要求很简单啊，为啥要用别人的framework? 自己写个多线程的broker 不就行了吗
？如果用java 的话，收到请求，包装成一个runnable 提交给executor。如果需要运算
完马上call back,可以用一个future, 否则的话可以把结果存在数据库里，隔一会poll
一下就行了。当然还有些housekeeping 的工作需要做，比如万一算到一半crash怎么恢
复，但是都很简单。
我用开源软件的原则就是一定要用的广泛的，质量有保障，有问题也容易找到答案。小
众的开源软件真不如自己写。 |
|
d******e
发帖数: 2265 | 17 java 7的fork join executor牛逼的。
java 8的fp没意思。不配套只能瞎搞。 |
|
r**i
发帖数: 1222 | 18 我最近在玩。vertx event bus的基础是这个。
可惜vertx将它封的死死的。
优点，java cache 可以embeded，直接存Java 对象、distributed map、queue、lock
这些都有，还有executors。map可以persist。cluster除了multicast还有一大堆其他
方式 |
|
T*****u
发帖数: 7103 | 19 在emr上跑了一个spark的cluster, 想用它处理我们在rds上postgres里的数据
ssh到master, run
/usr/lib/spark/bin/pyspark --driver-class-path ./postgresql-9.4.1211.jar --
conf spark.executor.extraClassPath=./postgresql-9.4.1211.jar --jars ./
postgresql-9.4.1211.jar
进去之后然后运行
>>> df = spark.read.format('jdbc').options(url=dbstring).options(dbtable='
users').options(driver='org.postgresql.Driver').load()
在本地的docker里面都没问题。但在这里出现问题
py4j.protocol.Py4JJavaError: An error occurred while calling o53.load.
在security group里面加了inbound
P... 阅读全帖 |
|
d****n
发帖数: 12461 | 20 我举个例子，spark streaming只有一个全局窗口，而且是jvm起来之前就定好的，在2.
0出现dynamic executor allocation之前对于data skewness束手无策。 |
|
c*****e
发帖数: 3226 | 21 问个问题，reduce的时候，exexutor如何分配的使得它如何知道它应该去取比如 key
='
b'的数据from remote block manager
换句话说就是 如何分配executor 收集那个 key from buckets,特别是 sorted 的情况下 |
|
d****n
发帖数: 12461 | 22 spark有node awareness和rack awareness，一般都是按照data locality来，一个
executor core一次处理一个hdfs block。
问个问题，reduce的时候，exexutor如何分配的使得它如何知道它应该去取比如 key
况下 |
|
h*******o
发帖数: 4884 | 23 Well, my point is mitochondrial theory of aging is more like the theory of
why all species eventually age, but not how aging is regulated.
It is the executor of aging and can be regulated by a complicated network/
combination of genetic and environmental factors.
lab
perfect |
|
d****i
发帖数: 360 | 24 Barristers' Chambers: The WILL have been executed
Anthony Williams & Associates
Tel:- (+234 806 784 7786).
Fax:- (+23414814238)
Attention:
On behalf of the trustees and executor of the estate of Late Engr. Jurgen
Krugger. I once again try to notify you as my earlier letter was returned
undelivered. I hereby attempt to reach you again by this same email address
on the Will. I wish to notify you that late Jurge Krugger made you a
beneficiary in his Will. He left the sum of Thirty Million, One Hun |
|
b****a
发帖数: 41 | 25 By having a joint business account with your friend, you have a de-facto
general partnership with him.
In generaly partnership, the partnership is liable for each general partner'
s liability. The police dept is just an executor. The person who sued your
friend must have won the case. So it's not the fault of either the police or
the plantiff.
You may only sue your friend for the money.
Operating a business in the U.S. without basic knowledge of business law is
like driving a car with your eyes |
|
r***l
发帖数: 298 | 26 One nexus to job placement is that you need personal connections
with the offices of the adminsistration. And to that end, Berkeley would
not fare well. I also know that in DC area, GU/GW has a joint program for
public interest law. Perhaps Gtown is better choice for you.
To this end, I must note that, GW, as a well-known conservative and
republican law school, offered you a scholarship of some kind even though
your political affiliations and ambitions could not be more different from
those do... 阅读全帖 |
|
r***l
发帖数: 298 | 27 As an IP practitioner in DC with a PhD and years of industrial experience
before my JD, I cannot agree with you any more. But again, maybe all
aspiring polititians want the free lunches, lavish parties, special sport
tickets, free trips to EU, all on our expense. To the best of my knowledge,
Michelle Obama is the first first lady to take Air Force One for her own EU
vacations. Not to mention, each and every time, the President and his
cronies take a trip, they cut off traffic in DC.
To make th... 阅读全帖 |
|
w**2
发帖数: 147 | 28 最近在版上发了几个问题，谢谢大家的解答。LZ目前的水平也只能跑跑spark python的
程序，对于build spark cluster方面还是有很多不理解的地方。
比如我现在在建一个Spark EMR集群，1个master，2个core，没有task。master和core
都是15g那种, 总共是45g。我就在aws emr的那个网页上create cluster，然后ssh进入
master node。然后问题来了。
首先，memory shortage。我之前以为spark－submit的code都是要跑在core nodes上的
。可是我现在感觉是跑在master上面，根本没有用到core，所以memory才不够用呢？
master和core是如何交互的呢？
需要在spark-submit clause里面增加其他参数嘛？我之前用了增加了driver memory，
感觉可以跑一些数据量大一点的程序。executor memory和driver memory是什么关系呢
？他们都是core的参数嘛，还是master的参数，或者他们只要加起来比27g (45g*0.6)
小... 阅读全帖 |
|
m*********r
发帖数: 119 | 29 executor memory 是slave 的
和driver memory是master的memory |
|
w*******y
发帖数: 60932 | 30 Hey, I just want to make people aware of a great little free fax site I'm
using. (This is just for OUTGOING faxes only, if you need an incoming fax
service, this is not for you.) I'm an executor of an estate and I've had
need to fax legal documents to financial institutions. My hand-me-down
computer is missing essential .dll files to use its' fax capability with my
all-in-one printer (the fax capability is questionable anyways because I use
VOIP. I've read it's very hit-or-miss with VOIP), s... 阅读全帖 |
|
d****i
发帖数: 348 | 31
well i think the answer lies in the moive,
because the majority of Jews do nothing about the killing but either lies
on the street waiting for his executor to change magazine before he shots
him, or hide behind the blinds and watch others die. It is a low race. a
race betrayed the God, and itself. |
|
s**g
发帖数: 3271 | 32 原来5年前才1万5一平啊
我还说这人吹牛，因此这post是坑呢。
不过我不太信做人家executor就能看到人家的资产。。。
就算看到了，晒出来也太恶心了 |
|
