my understand of IB data - Stock版

本页内容为未名空间相应帖子的节选和存档，一周内的贴子最多显示50字，超过一周显示500字访问原贴

Stock版 - my understand of IB data

相关主题
● 贡献一个IB自动交易例子
● 这里谁写过 IB 的程序交易
● 请问大牛们怎么买人民币呀?
● 继续请教 Option 大牛, 谢谢!
● IB webtrader disconnected 求教
● GS says tomorrow non farm payroll will be better than expected
● 请教eSignal数据下载
● share a cheat sheet
● 【AMIBROKER-AFL】如何把两个曲线画在一个图里
● 问个技术问题

相关话题的讨论汇总
话题: 03话题: ib话题: 2014话题: 00话题: data

进入Stock版参与讨论

(共1页)

c*******y
发帖数: 1630

everyone using their API knew how bad IB's data feed is. In stead of
complaining, I will give some statistics for IB user.

h*****7
发帖数: 6781

谢谢大牛，
沙发听课
我现在用Yahoo的，大牛觉得yahoo的怎么样？

【在 c*******y 的大作中提到】

: everyone using their API knew how bad IB's data feed is. In stead of
: complaining, I will give some statistics for IB user.

c*******y
发帖数: 1630

I have no experience with yahoo data service, yahoo's free EOD data is
very popular, I think it's ok.

【在 h*****7 的大作中提到】

: 谢谢大牛，
: 沙发听课
: 我现在用Yahoo的，大牛觉得yahoo的怎么样？

s***d
发帖数: 15421

大牛说说玉米期货吧。

【在 c*******y 的大作中提到】

: everyone using their API knew how bad IB's data feed is. In stead of
: complaining, I will give some statistics for IB user.

c*******y
发帖数: 1630

I spent like 30 minutes with one thread and it's gone with stupid MITBBS non
-auto-save bs.

s***m
发帖数: 6197

期待
我没注意到IB和TD的数据有什么区别

【在 c*******y 的大作中提到】

: everyone using their API knew how bad IB's data feed is. In stead of
: complaining, I will give some statistics for IB user.

c*******y
发帖数: 1630

I will split into short threads instead to avoid wasting time again.

r*****e
发帖数: 4611

我天天用啊，我写的java程序用ib的api接口的

c*******y
发帖数: 1630

1. connectivity. it's actually better than I thought, i have 2 instances
running. one demo, one real. real one loses connection and auto-reconnects
occasionally.(it's not like complete disconnection, and connection will be
resumed in 2-3 seconds, without verification). I put demo one talk with IB
servers for 3 days. I tested it and initially I thought there were 700+
disconnections in 2 days by my standard. But I soon realized it's an unknown
bug from my program. And now I believe there is not even once disconnection
from IB server for 3 days.
There is a trick to hack automatic logoff(for maintenance--IB). easy to find
. And previous disconnections were due to my internet provider, FIOS is fast
but not stable enough for trading, i guess.
And, always put an attached stop order. it will be sent as a package to IB
server, which should be so for every automatic trading system.

c*******y
发帖数: 1630

didn't pay much attention, I overheard some mildly bullish news.
but I think it's year of *SOFT* this year.

【在 s***d 的大作中提到】

: 大牛说说玉米期货吧。

相关主题
● 继续请教 Option 大牛, 谢谢!
● IB webtrader disconnected 求教
● GS says tomorrow non farm payroll will be better than expected
● 请教eSignal数据下载
进入Stock版参与讨论

c*******y
发帖数: 1630

2. backfilling. there is a threshold. not too small. not too big either. I
used to use it to build an equity minute level historical database myself
when I traded weekly options exclusively. Definitely not enough to get all
underlying one minute OHLC for all weekly tickers. CBOE at least tripled the
list since I played with weekly. For minute/second level, IB is a cheap way
to get data -- you need keep your machine 24/7 running or get multiple
accounts or both

c*******y
发帖数: 1630

my P&L since 19:30, not a lucky day to gamble :D

c*******y
发帖数: 1630

3. tick data. IB does NOT provide tick data. Here's details
Stock and futures will refresh per 250 ms.
Option will refresh per 100ms.
Forex will refresh per 5 ms.
those who complains about this, ask yourself. Why on earth you need tick
data with IB? How much you could data mining given tick data and convert it
into a profitable strategies? I bet 99% IB customers is not capable of it.

c*******y
发帖数: 1630

4. delay data--I am not sure if IB is in the club of deliberately delaying
data with import data release time. I traded /6E with TDA during NFP. it was
terrible. Graph, quote, level 2, and time&sale were off sync, all of them
delayed by seconds if not minutes. I haven't tested IB during NFP or ECB
press conference(there is one tmr morning).
My guess it is slightly better than most other retail brokers.
By the way, I wonder if anyone here knows how to test latency, eg, read msg
latency, and send order latency. Parsing the msg and signaling in the
trading program would be neglectable, I program with C++ POSIX API.
There is a wrapper function which returns non-delay timestamp, with FIELD =
45. And it's in number of seconds since 1970/01/01, and it's only available
for trade, no such thing for bid/ask/level 2(which means no such timestamp
for FOREX).
you can return (gettimeofday - timestamp) if you can get timestamp. Let me
know if you can find one.

a******1
发帖数: 2340

数据的话，还是专门找个data provider好一些吧
如果tick data虽然贵，但是数据质量还不错

c*******y
发帖数: 1630

5. Data replicates, very hard to reproduce their DATA if possible. I
considered several prices for my strategy design, bid, ask, *mid*, time&sale
which is not equivalent to your calculated mid, just to name a few. There
are also multiple ways to define a *semi TICK* for different instrument,
possible choices are. arithmetic mid, weight average mid, weighted average
by depth book, etc. And each method has a corresponding definition of A TICK
. I used to contract a third party tick data provider. trying to get detail
information on their definition on FOREX tick. The representative sounded
like newbie even I pretended to be a serious customer.
I asked IB how to duplicate, sale&trade in TWS from API. Docs says you quote
mktData, and there are a FIELD which provides such info by *TRADE* field.
For forex, it shows "Mid" as column name. I am not able to get that "Mid"
price via API. Rep said you need to calculate based on bid and ask yourself
for FOREX and didn't elaborate their methodology. I tried multiple ways, and
none of them matched time&sale print.

c*******y
发帖数: 1630

note the difference of ES and EUR.USD

sale
TICK
detail
quote

【在 c*******y 的大作中提到】

: 5. Data replicates, very hard to reproduce their DATA if possible. I
: considered several prices for my strategy design, bid, ask, *mid*, time&sale
: which is not equivalent to your calculated mid, just to name a few. There
: are also multiple ways to define a *semi TICK* for different instrument,
: possible choices are. arithmetic mid, weight average mid, weighted average
: by depth book, etc. And each method has a corresponding definition of A TICK
: . I used to contract a third party tick data provider. trying to get detail
: information on their definition on FOREX tick. The representative sounded
: like newbie even I pretended to be a serious customer.
: I asked IB how to duplicate, sale&trade in TWS from API. Docs says you quote

c*******y
发帖数: 1630

and my own calculated *mid*

【在 c*******y 的大作中提到】

: note the difference of ES and EUR.USD
:
: sale
: TICK
: detail
: quote

s***d
发帖数: 15421

大牛的意思就是个咯？谢大牛了。

【在 c*******y 的大作中提到】

: didn't pay much attention, I overheard some mildly bullish news.
: but I think it's year of *SOFT* this year.

c*******y
发帖数: 1630

I initally used reqMktData to duplicate time&sales (that's about the
frequency of quotes I am comfortable with and willing to trade with). Now I
decided to switch to depth to give other options a shot. IB allows only 3
simultaneous depth requests which is really sucks. I only trade FX now. I
can live with it. I tried it on AUD/EUR/JPY and collected roughtly depth for
2 days with my own local micro-second timestamps. I stored only calculated
*MID* in raw file. no rows other than
side = 0,1 & row = 0, you could get 20 rows for stocks/futures and 10 rows
for Forex, but in TWS, you will see row 0,1,2 from both sides with some
remote rows, 7,8,9 etc

相关主题
● share a cheat sheet
● 【AMIBROKER-AFL】如何把两个曲线画在一个图里
● 问个技术问题
● 请推荐一个画图软件
进入Stock版参与讨论

s***m
发帖数: 6197

牛
谢谢你分享技术贴！
你这是要开始做high frequency trading?

【在 c*******y 的大作中提到】

: my P&L since 19:30, not a lucky day to gamble :D

c*******y
发帖数: 1630

no, i intended to trade in seconds/minutes frequency which by no mean
is HFT.
Finally created something works with microsecond backtesting. Spent
too much time to learn this programming tricks/tips. To give you an example

【在 s***m 的大作中提到】

: 牛
: 谢谢你分享技术贴！
: 你这是要开始做high frequency trading?

s***m
发帖数: 6197

This is a good start. Now you have something to optimize.
Looking forward to your improved performance in the future.

example

【在 c*******y 的大作中提到】

: no, i intended to trade in seconds/minutes frequency which by no mean
: is HFT.
: Finally created something works with microsecond backtesting. Spent
: too much time to learn this programming tricks/tips. To give you an example

c*******y
发帖数: 1630

this is just framework.
you don't want to show something profitable in details here.

【在 s***m 的大作中提到】

: This is a good start. Now you have something to optimize.
: Looking forward to your improved performance in the future.
:
: example

s***m
发帖数: 6197

I see.
What is MPI and UPI in your screenshot?
And also WCDD, maxdd and sortino.

【在 c*******y 的大作中提到】

: this is just framework.
: you don't want to show something profitable in details here.

a****e
发帖数: 64

"There is a trick to hack automatic logoff(for maintenance--IB)"
怎么可以绕开这个automatic logoff啊？

unknown
disconnection
find
fast

【在 c*******y 的大作中提到】

: 1. connectivity. it's actually better than I thought, i have 2 instances
: running. one demo, one real. real one loses connection and auto-reconnects
: occasionally.(it's not like complete disconnection, and connection will be
: resumed in 2-3 seconds, without verification). I put demo one talk with IB
: servers for 3 days. I tested it and initially I thought there were 700+
: disconnections in 2 days by my standard. But I soon realized it's an unknown
: bug from my program. And now I believe there is not even once disconnection
: from IB server for 3 days.
: There is a trick to hack automatic logoff(for maintenance--IB). easy to find
: . And previous disconnections were due to my internet provider, FIOS is fast

c*******y
发帖数: 1630

能google到的问题，最好先google
change tws.xml
autoLogoffTime="99999999999999:15,AM"

【在 a****e 的大作中提到】

: "There is a trick to hack automatic logoff(for maintenance--IB)"
: 怎么可以绕开这个automatic logoff啊？
:
: unknown
: disconnection
: find
: fast

c*******y
发帖数: 1630

I will show you some sample I collected. I spent hours on programming
tricks/usages, potential outdated packages. asking around stackoverflow
to get something working.
Originally I thought IB disconnects frequently, but I will give a second
thought.
Here's some test.
Time range:
2014-03-03 23:30:30 to 2014-03-05 19:41:41 almost 2 days.
In [40]: e.head(1)
Out[40]:
Bid n
Time
2014-03-03 23:30:30.224323 0.8925 0
In [42]: e.tail(2)
Out[42]:
Bid n
Time
2014-03-05 19:41:41.712247 0.9025 4312
2014-03-05 19:41:41.719829 NaN 4312
data format: csv, I will switch to hdf5 later
In [37]: e
Out[37]:
Bid
Time
2014-03-03 23:30:30.224323 0.892500
2014-03-03 23:30:30.224390 0.892525
2014-03-03 23:30:30.224408 0.892525
2014-03-03 23:30:30.364299 0.892525
2014-03-03 23:30:31.022652 0.892500
2014-03-03 23:30:31.022702 0.892525
file size: 44M
xxx@xxx:~/trading/eu-ats/data/new$ ls -lh aud-fix.csv | awk '{print $5, $6,
$7, $9}'
44M Mar 5 aud-fix.csv
total records:
In [39]: len(e)
Out[39]: 1293677
I resample raw data into OHLC and checked number of ticks for each minutes
to verify connection, cause I am not sure if IB occasional auto reconnection
will be tracked by my log file. Here's the result
In [44]: z
Out[44]:
Bid
Time
2014-03-04 00:20:00 0
2014-03-04 00:21:00 0
2014-03-04 17:01:00 0
2014-03-04 17:02:00 0
2014-03-04 17:03:00 0
2014-03-04 17:04:00 0
2014-03-04 17:05:00 0
2014-03-04 17:06:00 0
2014-03-04 17:07:00 0
2014-03-04 17:08:00 0
2014-03-04 17:09:00 0
2014-03-04 17:10:00 0
2014-03-04 17:11:00 0
2014-03-04 17:12:00 0
2014-03-04 17:13:00 0
2014-03-04 17:14:00 0
2014-03-05 17:01:00 0
2014-03-05 17:02:00 0
2014-03-05 17:03:00 0
2014-03-05 17:04:00 0
2014-03-05 17:05:00 0
2014-03-05 17:06:00 0
2014-03-05 17:07:00 0
2014-03-05 17:08:00 0
2014-03-05 17:09:00 0
2014-03-05 17:10:00 0
2014-03-05 17:11:00 0
2014-03-05 17:12:00 0
2014-03-05 17:13:00 0
2014-03-05 17:14:00 0
except maintainance time everyday from 17:00-17:15, there are only
two consecutive minutes with no data, since I put sleep(60) after my
reconnect statement in the program(I REALLY HATE THIS, BUT THIS IS RELATED
SOME UGLY BUG WHICH BOTHERS ME AND I FOUND NO ANSWER FROM IB OR ANYWHERE
ELSE)

【在 c*******y 的大作中提到】

: everyone using their API knew how bad IB's data feed is. In stead of
: complaining, I will give some statistics for IB user.

c*******y
发帖数: 1630

these are the most busiest trading time for aussie in that 2 day's time
range.
In [45]: zz
Out[45]:
Bid
Time
2014-03-05 19:30:00 5641
2014-03-04 20:45:00 4603
2014-03-04 19:30:00 4351
2014-03-05 19:32:00 4243
2014-03-05 08:15:00 3984
2014-03-04 01:05:00 3687
2014-03-04 19:35:00 3429
2014-03-05 19:31:00 3373
2014-03-05 19:33:00 3084
2014-03-04 20:47:00 3067
2014-03-05 08:16:00 3048
2014-03-04 01:09:00 3017
2014-03-04 19:31:00 2987
2014-03-05 19:35:00 2863
2014-03-05 08:32:00 2753

c*******y
发帖数: 1630

first one is retail sale yesterday.
followed by China PMI, don't remember it's service or non-manufacture
third one is GDP number
fifth is ADP
etc.
03/03 has RBA rate statement and decision which shows on the data too.

相关主题
● 竞赛现在的情况一览表
● 简单说说我的模型预测方法，供大家改进
● 最近想做个炒股辅助决策系统，请大家帮我捋一捋思路
● 大家现在从哪import股票价格？
进入Stock版参与讨论

c*******y
发帖数: 1630

for example retail sale, you can see, it only took market 10 seconds to
reach fair value

c*******y
发帖数: 1630

and here's one minute chart of market reaction

c*******y
发帖数: 1630

here's big picture

【在 c*******y 的大作中提到】

: and here's one minute chart of market reaction

c*******y
发帖数: 1630

here's traditional OHLC 1min

【在 c*******y 的大作中提到】

: here's big picture

c*******y
发帖数: 1630

any stop hunting at IB? I don't think so from this chart,
at least not at that time

【在 c*******y 的大作中提到】

: here's traditional OHLC 1min

c*******y
发帖数: 1630

is this tradable? I didn't get a chance to test, but you could find
a way to estimate slippage by your latency. OR there is a tricky way
to trade this and sorry, I don't want to share.
but here's a *BACKTEST* MA sample with such data.

【在 c*******y 的大作中提到】

: any stop hunting at IB? I don't think so from this chart,
: at least not at that time

c*******y
发帖数: 1630

everyone using their API knew how bad IB's data feed is. In stead of
complaining, I will give some statistics for IB user.

h*****7
发帖数: 6781

谢谢大牛，
沙发听课
我现在用Yahoo的，大牛觉得yahoo的怎么样？

【在 c*******y 的大作中提到】

: everyone using their API knew how bad IB's data feed is. In stead of
: complaining, I will give some statistics for IB user.

c*******y
发帖数: 1630

I have no experience with yahoo data service, yahoo's free EOD data is
very popular, I think it's ok.

【在 h*****7 的大作中提到】

: 谢谢大牛，
: 沙发听课
: 我现在用Yahoo的，大牛觉得yahoo的怎么样？

s***d
发帖数: 15421

大牛说说玉米期货吧。

【在 c*******y 的大作中提到】

: everyone using their API knew how bad IB's data feed is. In stead of
: complaining, I will give some statistics for IB user.

相关主题
● 最牛的看盘软件...
● China PMI 47.8
● TOS里的$TICK指标是什么
● 两种基本无用的TA方法
进入Stock版参与讨论

c*******y
发帖数: 1630

I spent like 30 minutes with one thread and it's gone with stupid MITBBS non
-auto-save bs.

s***m
发帖数: 6197

期待
我没注意到IB和TD的数据有什么区别

【在 c*******y 的大作中提到】

: everyone using their API knew how bad IB's data feed is. In stead of
: complaining, I will give some statistics for IB user.

c*******y
发帖数: 1630

I will split into short threads instead to avoid wasting time again.

r*****e
发帖数: 4611

我天天用啊，我写的java程序用ib的api接口的

c*******y
发帖数: 1630

didn't pay much attention, I overheard some mildly bullish news.
but I think it's year of *SOFT* this year.

【在 s***d 的大作中提到】

: 大牛说说玉米期货吧。

c*******y
发帖数: 1630

my P&L since 19:30, not a lucky day to gamble :D

c*******y
发帖数: 1630

相关主题
● 贡献一个IB自动交易例子
● 这里谁写过 IB 的程序交易
● 请问大牛们怎么买人民币呀?
● 继续请教 Option 大牛, 谢谢!
进入Stock版参与讨论

a******1
发帖数: 2340

数据的话，还是专门找个data provider好一些吧
如果tick data虽然贵，但是数据质量还不错

c*******y
发帖数: 1630

note the difference of ES and EUR.USD

sale
TICK
detail
quote

【在 c*******y 的大作中提到】

c*******y
发帖数: 1630

and my own calculated *mid*

【在 c*******y 的大作中提到】

: note the difference of ES and EUR.USD
:
: sale
: TICK
: detail
: quote

s***d
发帖数: 15421

大牛的意思就是个咯？谢大牛了。

【在 c*******y 的大作中提到】

: didn't pay much attention, I overheard some mildly bullish news.
: but I think it's year of *SOFT* this year.

c*******y
发帖数: 1630

s***m
发帖数: 6197

牛
谢谢你分享技术贴！
你这是要开始做high frequency trading?

【在 c*******y 的大作中提到】

: my P&L since 19:30, not a lucky day to gamble :D

c*******y
发帖数: 1630

: 牛
: 谢谢你分享技术贴！
: 你这是要开始做high frequency trading?

s***m
发帖数: 6197

This is a good start. Now you have something to optimize.
Looking forward to your improved performance in the future.

example

【在 c*******y 的大作中提到】

c*******y
发帖数: 1630

this is just framework.
you don't want to show something profitable in details here.

【在 s***m 的大作中提到】

: This is a good start. Now you have something to optimize.
: Looking forward to your improved performance in the future.
:
: example

相关主题
● 继续请教 Option 大牛, 谢谢!
● IB webtrader disconnected 求教
● GS says tomorrow non farm payroll will be better than expected
● 请教eSignal数据下载
进入Stock版参与讨论

s***m
发帖数: 6197

I see.
What is MPI and UPI in your screenshot?
And also WCDD, maxdd and sortino.

【在 c*******y 的大作中提到】

: this is just framework.
: you don't want to show something profitable in details here.

a****e
发帖数: 64

"There is a trick to hack automatic logoff(for maintenance--IB)"
怎么可以绕开这个automatic logoff啊？

unknown
disconnection
find
fast

【在 c*******y 的大作中提到】

c*******y
发帖数: 1630

能google到的问题，最好先google
change tws.xml
autoLogoffTime="99999999999999:15,AM"

【在 a****e 的大作中提到】

: "There is a trick to hack automatic logoff(for maintenance--IB)"
: 怎么可以绕开这个automatic logoff啊？
:
: unknown
: disconnection
: find
: fast

c*******y
发帖数: 1630

: everyone using their API knew how bad IB's data feed is. In stead of
: complaining, I will give some statistics for IB user.

c*******y
发帖数: 1630

for example retail sale, you can see, it only took market 10 seconds to
reach fair value

c*******y
发帖数: 1630

and here's one minute chart of market reaction

c*******y
发帖数: 1630

here's big picture

【在 c*******y 的大作中提到】

: and here's one minute chart of market reaction

c*******y
发帖数: 1630

here's traditional OHLC 1min

【在 c*******y 的大作中提到】

: here's big picture

相关主题
● share a cheat sheet
● 【AMIBROKER-AFL】如何把两个曲线画在一个图里
● 问个技术问题
● 请推荐一个画图软件
进入Stock版参与讨论

c*******y
发帖数: 1630

any stop hunting at IB? I don't think so from this chart,
at least not at that time

【在 c*******y 的大作中提到】

: here's traditional OHLC 1min

c*******y
发帖数: 1630

: any stop hunting at IB? I don't think so from this chart,
: at least not at that time

P*****s
发帖数: 758

求问LZ在IB用这些API需要买什么data吗？

(共1页)

进入Stock版参与讨论

相关主题
● 问个技术问题
● 请推荐一个画图软件
● 竞赛现在的情况一览表
● 简单说说我的模型预测方法，供大家改进
● 最近想做个炒股辅助决策系统，请大家帮我捋一捋思路
● 大家现在从哪import股票价格？
● 最牛的看盘软件...
● China PMI 47.8
● TOS里的$TICK指标是什么
● 两种基本无用的TA方法

相关话题的讨论汇总
话题: 03话题: ib话题: 2014话题: 00话题: data

boards