Parser¶
Usage¶
Purpose: Parse the binary PCAP / HIST files offered by IEX.
To create a Parser object simply supply the file path as an argument.
>>> from IEXTools import Parser, messages
>>> p = Parser(r'IEX TOPS Sample\20180103_IEXTP1_TOPS1.6.pcap')
>>> p
Parser("IEX TOPS Sample\\20180103_IEXTP1_TOPS1.6.pcap", tops=True, deep=False)
This instantiates a Parser object with the pcap file opened. You can also optionally specify what type of HIST file you are loading, either TOPS (tops=True) or DEEP (deep=True).
Use the get_next_message method of the Parser object to return a message object. The message objects are documented in the messages.py module. You can optionally specify a list of message classes to restrict the returned messages to only those types.
>>> allowed = [messages.TradeReport]
>>> p.get_next_message(allowed)
TradeReport(flags=64, timestamp=1514984427833117218, symbol='ZVZZT', size=975, price_int=100150, trade_id=577243)
The last message retrieved can also be accessed through the Parser object itself with the message attribute. Similarly the message type can be accessed as Parser.message_type and the binary encoded message can be accessed with Parser.message_binary. The message object also has several attributes that may be accessed (varies by object).
>>> p.message
TradeReport(flags=64, timestamp=1514984427833117218, symbol='ZVZZT', size=975, price_int=100150, trade_id=577243)
>>> p.message.date_time
datetime.datetime(2018, 1, 3, 13, 0, 27, 833117, tzinfo=datetime.timezone.utc)
>>> p.message.price
10.015
>>> p.message_type
84
>>> p.message_binary
b'@"\xca=uON\x06\x15ZVZZT\xcf\x03\x00\x006\x87\x01\x00\x00\x00\x00\x00\xdb\xce\x08\x00\x00\x00\x00\x00'
The program also allows you to use it with a context manager and loop through it like a file:
with Parser(file_path) as iex_messages:
for message in iex_messages:
do_something(message)
Benchmarks On my personal laptop (Lenovo ThinkPad X1 Carbon, Windows 10):
Beginning test - 1,000,000 messages - all messages, not printing
Parsed 1,000,000 messages in 52.2 seconds -- 19141.6 messages per second
Beginning test - 1,000,000 messages - only TradeReport and QuoteUpdate messages, not printing
Parsed 1,000,000 messages in 54.0 seconds -- 18512.9 messages per second
By not specifying the allowed argument the parser returns 1,000,000 parsed messages approximately 3% faster. However, in order to return 1,000,000 parsed messages the Parser with the allowed argument set may have to read through more than 1,000,000 messages. Testing suggests that actually decoding the message takes about 10 microseconds (130,000 messages per second).