This page presents an introduction to the pyroTorrent design goals.
pyroTorrent’s codebase might look complex at first, but it’s ultimitely trivial once you understand why we made certain design decisions. (Let’s hope they turn out to be right ones, it seems like they are so far).
The easiest way to help you understand our design decisions, is to simply share our experiences and thoughts when we were planning to write pyroTorrent.
Since I currently suck as a writer, don’t worry if you at some point don’t understand what I’m trying to say, just keep on reading. It will probably make sense after you’ve read the entire page. If not, read it again. If you still don’t get it, it’s either a case of PEBCAK or I just really suck at writing. :-)
After trying several rTorrent web frontends such as wTorrent and rTWi, we were rather disappointed by the speed offered by both these frontends, aside from the fact that they used PHP. (Which may be one of the reasons they were so unresponsive)
Here’s a very flawed and probably unfair comparison, with PHP admittedly compiled with -O1 (because -O2 broke compilation...)
Loading the main/overview page with about ~150 torrents in queue:
| pyroTorrent | rTWi | wTorrent |
|---|---|---|
| ~400 milli seconds | 10+ seconds | 8 seconds |
All tests are done on a Sheevaplug with 1,2 Ghz ARM processor, softfloat.
One of pyroTorrent’s design goals is to be fast. One of the ways to achieve this is to minimise the amount of XMLRPC calls with so called multicalls.
libTorrent and the Python module xmlrpclib both have support for a so called multicall. A multicall in xmlrpclib typically encapsulates several XMLRPC requests in one request, thus decreasing overhead a lot. libTorrent multicalls perform an action on all items of a specific type, say all torrents - simply with the call d.multicall. If you don’t use multicall you’ll rapidly find yourself opening over 500 connections per page load; and since XMLRPC is stateless you’ll have to actually do 500 requests, each with their own connection.
pyroTorrent makes use of both these multicall mechanisms. Typically it should not open more than a few connections per page load. Current in release 0.04, pyroTorrent does only 2 XMLRPC requests to load the main overview page.
Below is some code from pyroTorrent release-0.03.
Fetching some rTorrent information:
try:
r = global_rtorrent.query().get_upload_rate().get_download_rate().get_ip()\
.get_hostname().get_memory_usage().get_max_memory_usage()\
.get_libtorrent_version()
return r.first()
except InvalidConnectionException, e:
return {}
As illustrated in an interactive python shell using pyroTorrent’s cli.sh:
>>> r.query().get_upload_rate().get_download_rate().get_ip()\
... .get_hostname().get_memory_usage().get_max_memory_usage()\
... .get_libtorrent_version().first()
{'get_memory_usage': 30408704, 'get_ip': '0.0.0.0', 'get_upload_rate': 16303,
'get_max_memory_usage': 858993459, 'get_hostname': 'sheeva',
'get_download_rate': 4932, 'get_libtorrent_version': '0.12.6'}
And all this information is retrieved in one XMLRPC call.
Note how we call .query() on the object r. r is a RTorrent instance; and the .query() method returns a RTorrentQuery object. The RTorrentQuery object contains all the libTorrent calls that the rtorrent object supports, but it remembers what calls you’ve done on the object, and then returns them all when you tell it to. (The .first() call). Also note how the RTorrentQuery object allows you to chain calls, by returning itself.
The RTorrentQuery inherits from on the MultiBase class, which takes care of all the underlying tasks. You’ll find that RTorrentQuery is no more than 40 lines of code, of which 80% is documentation.
Apart from RTorrentQuery, we also have TorrentQuery, which does the same, but for the Torrent Model model instead of the RTorrent model.
Getting certain information of all torrents:
try:
t = TorrentRequester('')
t.get_name().get_download_rate().get_upload_rate() \
.is_complete().get_size_bytes().get_download_total().get_hash()
torrents = t.all()
except InvalidTorrentException, e:
return error_page(env, str(e))
Basic example in cli.sh:
>>> t = TorrentRequester('')
>>> t.get_name().get_download_rate().get_upload_rate() \
... .is_complete().get_size_bytes().get_download_total().get_hash()
<lib.torrentrequester.TorrentRequester object at 0x24ae350>
>>> torrents = t.all()
>>> len(torrents)
83
>>> torrents[:1]
[{'get_size_bytes': 41907644, 'get_upload_rate': 0, 'get_name':
'RevengeOfTheTitansSoundtrack.zip', 'get_hash':
'6709A6306E2FB4EEF89455DFC8C26CA4DB316E6F', 'get_download_total': 0,
'get_download_rate': 0, 'is_complete': 0}]
The TorrentRequester works somewhat similar to RTorrentQuery in the sense that it also uses multicalls; but in this case the libTorrent multicall. The TorrentRequester inherits most of its functionality from the BaseRequester.
libTorrent offers an API to program most if not all tasks; but the API is rather undocumented and awkward to be used without any wrapper or model.
It would however become increasingly cumbersome to write a method for each libTorrent method, so we’ve come up with a solution.
In the file model/rtorrent.py all the RPC methods are stored in a dict:
_rpc_methods = {
'get_upload_throttle' : ('get_upload_rate',
"""
Returns the current upload throttle.
"""),
'set_upload_throttle' : ('set_upload_rate',
"""
Set the upload throttle.
Pass the new throttle size in bytes.
"""),
'get_ip' : ('get_ip',
"""
Returns the IP rtorrent is bound to. (For XMLRPC?)
""")
}
For each entry in the dictionary, a method is generated and added to the RTorrent class, along with a __doc__ entry:
for x, y in _rpc_methods.iteritems():
caller = (lambda name: lambda self, *args: getattr(self.s, name)(*args))(y[0])
caller.__doc__ = y[1] + '\nOriginal libTorrent method: ``%s``' % y[0]
setattr(RTorrent, x, types.MethodType(caller, None, RTorrent))
del caller
We do something similar for the Torrent Model class.
PyroTorrent features a JSON API which can be used to extract most (if not all) information using the provided Python classes.
Below is our API test, written in Python. It does a POST request to the /torrent/api URL, where /torrent is the base url, the API resides at /api. The POST request should contain only one variable called request; and request should contain a JSON object containing the data to be requested and executed.
import simplejson as json
import urllib
import urllib2
url = 'http://localhost/torrent/api'
values = [
{
'target' : 'sheevareborn',
'type' : 'torrentrequester',
'view' : '',
'attributes' : [
('get_name', []),
('get_download_rate', []),
('get_upload_rate', []),
('is_complete', []),
('get_size_bytes', []),
('get_download_total', []),
('get_hash', [])
]
},
{
'target' : 'sheevareborn',
'type' : 'rtorrent',
'attributes' : [
('set_upload_throttle', [20480]),
('get_upload_throttle', [])
]
},
{
'target' : 'sheevareborn',
'type' : 'rtorrent',
'attributes' : [
('set_upload_throttle', [20480*2]),
('get_upload_throttle', [])
]
},
{
'target' : 'sheevareborn',
'type' : 'torrent',
'hash' : '8EB5801B88D34D50A6E7594B6678A2CF6224766E',
'attributes' : (
('get_hash', []),
('get_name', []),
('is_complete', [])
)
}
]
data = urllib.urlencode({'request' : json.dumps(values)})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
print the_page