1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Join our Telegram chat: https://t.me/a_parser_en
    Dismiss Notice

Multithreading and network technologies

Dec 28, 2016
  • This information can be interesting only to programmers as reasons for high performance and efficiency of a parser

    A-Parser was created when using the following technologies:
    • Asynchronous input/output - allows to support at the same time thousands of connections (libev, epoll & select)
    • Easy "green" threads (coro, anyevent)
    • Own productive HTTP/Socket client with support of HTTP/1.1, HTTPS, HTTP and SOCKS5 proxy with support authorization
    • SQLite and LevelDB as productive storages
    • Absence of the graphic interface - A-Parser is executed on client-server technology
      • Client represents the web application (sencha) and is responsible for setup and control of a parser
      • Server is engaged only in parsing and information processing
    • JavaScript engine V8 for bypass of JS protection, and also for writing own JavaScript parsers
    • Template Toolkit as the main template editor
    • UTF-8 in requests and results

    A-Parser performs parallel all important operations:
    • Each task (a set of requests) parsing in the specified number of threads
    • When using several parsers in one task each request to different parsers is executed in different threads at the same time
    • Parser is able to launch some tasks parallely
    • Check and loading of a proxy from sources also takes place in the multithreaded mode