c**t
发帖数: 2744 | 1 Two processes to merge data into same target/physical table:
A -> T; B -> T
SizeOf(A): 10K; SizeOf(B): 30K; SizeOf(T): 3 Million
As the tow processes running in parallel, it always cause deadlock:
Merge into T using A/B,
I am thinking to merge&commit A/B record by record to eliminate deadlock. Is
there any better way to avoid deadlock? |
c**t
发帖数: 2744 | 2 Tried A->T; B-A->T hopefully it will help
Is
【在 c**t 的大作中提到】
: Two processes to merge data into same target/physical table:
: A -> T; B -> T
: SizeOf(A): 10K; SizeOf(B): 30K; SizeOf(T): 3 Million
: As the tow processes running in parallel, it always cause deadlock:
: Merge into T using A/B,
: I am thinking to merge&commit A/B record by record to eliminate deadlock. Is
: there any better way to avoid deadlock?
|
c*****d
发帖数: 6045 | 3 不应该呀？A->T，B->T 都是insert吗？
出现了deadlock错误提示？ |
c**t
发帖数: 2744 | 4 No; Merge into T using A on (T.KEY=A.KEY) WHEN NOT MATCHED THEN INSERT ...
WHEN
MATCHED THEN SET ...
【在 c*****d 的大作中提到】
: 不应该呀？A->T，B->T 都是insert吗？
: 出现了deadlock错误提示？
|
c*****d
发帖数: 6045 | 5 如果T,A,B结构相同，可以这样做
Merge into T using A on (T.KEY=A.KEY)
WHEN NOT MATCHED
THEN INSERT ...
WHEN MATCHED
THEN DELETE T INSERT A ...
【在 c**t 的大作中提到】
: No; Merge into T using A on (T.KEY=A.KEY) WHEN NOT MATCHED THEN INSERT ...
: WHEN
: MATCHED THEN SET ...
|
B*****g
发帖数: 34098 | 6 where is B?
【在 c*****d 的大作中提到】
: 如果T,A,B结构相同，可以这样做
: Merge into T using A on (T.KEY=A.KEY)
: WHEN NOT MATCHED
: THEN INSERT ...
: WHEN MATCHED
: THEN DELETE T INSERT A ...
|
B*****g
发帖数: 34098 | 7 what is "WHEN MATCHED THEN SET"? you mean update?
【在 c**t 的大作中提到】
: No; Merge into T using A on (T.KEY=A.KEY) WHEN NOT MATCHED THEN INSERT ...
: WHEN
: MATCHED THEN SET ...
|
c**t
发帖数: 2744 | 8 same merge... just replace A with B
【在 B*****g 的大作中提到】
: where is B?
|
c**t
发帖数: 2744 | 9 yes
【在 B*****g 的大作中提到】
: what is "WHEN MATCHED THEN SET"? you mean update?
|
c*****d
发帖数: 6045 | 10 B和A的做法一样
因为没有update操作，所以B->T, A->T同时操作也不会出现deadlock
【在 B*****g 的大作中提到】
: where is B?
|
|
|
c**t
发帖数: 2744 | 11 no both B and A have update/insert, their merge codes are almost identical.
【在 c*****d 的大作中提到】
: B和A的做法一样
: 因为没有update操作，所以B->T, A->T同时操作也不会出现deadlock
|
c*****d
发帖数: 6045 | 12 ???没看懂你的意思
我的意思是，在A->T, B->T的操作中，使用delete+insert替换update
所以A->T, B->T的操作可以同时进行，不会出现deadlock
【在 c**t 的大作中提到】
: no both B and A have update/insert, their merge codes are almost identical.
|
B*****g
发帖数: 34098 | 13 ding. let me try try, I am thinking if matched, but only need to update some
columns not all, how to do delete+insert. hehe, I guess delete+insert will
not work on 9i, DD 9i.
【在 c*****d 的大作中提到】
: ???没看懂你的意思
: 我的意思是，在A->T, B->T的操作中，使用delete+insert替换update
: 所以A->T, B->T的操作可以同时进行，不会出现deadlock
|
B*****g
发帖数: 34098 | 14 how to write delete in merge? can not make it on 9i
some
will
【在 B*****g 的大作中提到】
: ding. let me try try, I am thinking if matched, but only need to update some
: columns not all, how to do delete+insert. hehe, I guess delete+insert will
: not work on 9i, DD 9i.
|
c**t
发帖数: 2744 | 15 no delete. Here is the pseduo scripts:
package ETL
function load_feed1
return LOAD.IMPORT('A')
end;
function load_feed2
return LOAD.IMPORT('B')
end
end ETL
package LOAD
function IMPORT(externalTable as varchar2) return number
MERGE INTO targetTable x using externalTable y
on x.key = y.key
WHEN MATCHED THEN UPDATE
set field = y.field
WHEN NOT MATCHED THEN INSERT
(field1, field2) values (y.field1, y.field2);
【在 c*****d 的大作中提到】
: ???没看懂你的意思
: 我的意思是，在A->T, B->T的操作中，使用delete+insert替换update
: 所以A->T, B->T的操作可以同时进行，不会出现deadlock
|
c**t
发帖数: 2744 | 16 there are two apps running in parallel:
App1, every m minutes, select ETL.load_feed1 from dual;
App2, every n minutes, select ETL.load_feed2 from dual;
Assuming m > n; when app1 update/insert targetTable, app2 comes in and does
same thing. Now app1 will see different state of targetTable, deadlock
happens.
【在 c**t 的大作中提到】
: no delete. Here is the pseduo scripts:
: package ETL
: function load_feed1
: return LOAD.IMPORT('A')
: end;
: function load_feed2
: return LOAD.IMPORT('B')
: end
: end ETL
: package LOAD
|
B*****g
发帖数: 34098 | 17 I believe he means not use "update set ..." but use "delete .... insert ...."
【在 c**t 的大作中提到】
: no delete. Here is the pseduo scripts:
: package ETL
: function load_feed1
: return LOAD.IMPORT('A')
: end;
: function load_feed2
: return LOAD.IMPORT('B')
: end
: end ETL
: package LOAD
|
c**t
发帖数: 2744 | 18 no delete isn't an option. The data might being viewed at the same time.
.."
【在 B*****g 的大作中提到】
: I believe he means not use "update set ..." but use "delete .... insert ...."
|
w*r
发帖数: 2421 | 19 Running two upsert operation in parallel is almost guaranteed to have dead
lock. You might only want to run the merge in serialized way, rather than
launch two merge in parallel.
【在 c**t 的大作中提到】
: no delete. Here is the pseduo scripts:
: package ETL
: function load_feed1
: return LOAD.IMPORT('A')
: end;
: function load_feed2
: return LOAD.IMPORT('B')
: end
: end ETL
: package LOAD
|
B*****g
发帖数: 34098 | 20 delete immediately follow by insert?
【在 c**t 的大作中提到】
: no delete isn't an option. The data might being viewed at the same time.
:
: .."
|
|
|
B*****g
发帖数: 34098 | 21 is it same as Merge T with (A union all B)?
【在 w*r 的大作中提到】
: Running two upsert operation in parallel is almost guaranteed to have dead
: lock. You might only want to run the merge in serialized way, rather than
: launch two merge in parallel.
|
c**t
发帖数: 2744 | 22 tried:
merge into T using A
and
merge into T using B-A
still hit deadlock;
merge into T using AUB should fix that, but it's too time consuming. Where A
is fairly small, current hour data; B is historical-future data
【在 w*r 的大作中提到】
: Running two upsert operation in parallel is almost guaranteed to have dead
: lock. You might only want to run the merge in serialized way, rather than
: launch two merge in parallel.
|
B*****g
发帖数: 34098 | 23 how about create a job queue? I can only think do it in java, do not know
oracle. Don't know if chain in oracle will work. hehe, just guess.
A
【在 c**t 的大作中提到】
: tried:
: merge into T using A
: and
: merge into T using B-A
: still hit deadlock;
:
: merge into T using AUB should fix that, but it's too time consuming. Where A
: is fairly small, current hour data; B is historical-future data
|
w*r
发帖数: 2421 | 24 if A is much smaller, then union will increase overhead to build spool on
the fly. Just make two jobs
sequential.
【在 B*****g 的大作中提到】
: how about create a job queue? I can only think do it in java, do not know
: oracle. Don't know if chain in oracle will work. hehe, just guess.
:
: A
|
B*****g
发帖数: 34098 | 25 人家说的是多少分钟run一次，不是一个run完了另一个run。
我先这样搞，先搞一个queue（就一个table也行）。
schedual A 隔一定时间check queue， job A在queue，啥都不做，不在，加入尾巴
pending。
schedual B 隔一定时间check queue， job B在queue，啥都不做，不在，加入尾巴
pending。
schedual C 隔一定时间check queue，有job process，啥都不做；没有，但有pending
，take first one， update pending to process，run， when job complete，
delete job from queue。
其实就10k A， 30K B， union all 没几秒钟
【在 w*r 的大作中提到】
: if A is much smaller, then union will increase overhead to build spool on
: the fly. Just make two jobs
: sequential.
|
c**t
发帖数: 2744 | 26 这么说吧， 10K merge 到 3M table, 大概７秒；３０Ｋ要１５秒左右．时间还是长了些
pending
【在 B*****g 的大作中提到】
: 人家说的是多少分钟run一次，不是一个run完了另一个run。
: 我先这样搞，先搞一个queue（就一个table也行）。
: schedual A 隔一定时间check queue， job A在queue，啥都不做，不在，加入尾巴
: pending。
: schedual B 隔一定时间check queue， job B在queue，啥都不做，不在，加入尾巴
: pending。
: schedual C 隔一定时间check queue，有job process，啥都不做；没有，但有pending
: ，take first one， update pending to process，run， when job complete，
: delete job from queue。
: 其实就10k A， 30K B， union all 没几秒钟
|
