Complete description of Task editor

Jul 11, 2017
  • Task editor - is a full-fledged editor of tasks, which includes a choice and detail setup for more than 60 parsers, and also formatting of requests and results at discretion.

    Detailed description(top)

    Config preset (setup of threads) - Possibility of editing settings of threads for the current task. By default are created 2 presets on 20 and 100 threads. For editing it is necessary to click [​IMG].
    Task preset - A choice of earlier saved task with all settings. Opportunity of [​IMG] import and export tasks. It it is possible to share with the help ready presets with each other, sharing a code. You can export several tasks at once, parser settings, JavaScript parsers, and also tools.js


    • Job test - Allows you to run tasks in the test mode and look at the preliminary results of its execution
    • Query builder - Allows to separate the initial request for parts and\or to transform requests
    • Add parser - Adding of a new parser to task. In one task it is possible to use at the same time some parsers, that allows to receive on one request information from different sources
    • Parser - Choice a parser
    • Parser preset - Choice of the saved preset for parsers. Contain in settings the presets. In details it is possible to edit each separate parser clicking on [​IMG]
    • Additional functions - Are designated by an icon [​IMG]. Contains the following tools for work with a parser:
      • Add unique result - Unique of any result on a line, the domain, top domain, the folder, a line without parameters
      • Add result filter - Filtering any result on entrance of a line, equivalence of lines, regular expression, more, less, is equal, etc.
      • Override preset - Fast redefinition of settings for parser. Allows to add substitutions and format request to the necessary look using templates, it is applied to each request
      • Parse custom result - Parsing of any result with use of the regular expressions
      • Try in Parser Test - The tool for verification of the made task with an output log of processing of request

      And also additional tools for parsers [​IMG]SE::Google, [​IMG]SE::Yandex::WordStat, [​IMG] Net:HTTP, [​IMG]HTML::TextExtractor, [​IMG]HTML::LinkExtractor and parcers of suggests:
      • Parse all results - This option is available to a parcer [​IMG]SE::Google and allows to bypass restrictions of output in 1000 results on one request
      • Parse to level - An option for automatic substitution of the parsing keywords again in requests
      • Check content - The option checks, if the regular expression didn't match, the page will be loaded again with other proxy
      • Use pages - An option which allows to parsit some pages on one request about possibility of substitution of number of the page with a certain step
      • Check next page - RegEx, whether which defines there is the following page or not
      • Page as new query - Transfers the transition to the next page as a new request, thereby allowing you to remove the limit on the number of pages for the transition
    • Result format - For each parser there is the result set. It is possible to view the list of available results having guided at a parser the pointer. Format of result can be specified in a convenient multiline editor having clicked [​IMG]. It is possible to use templates in names of resultant files, that allows to create automatically files and folders on the basis of current date, on sequence number of request, on the request and in any other format


    • Queries from - Possibility of a choice from where to load requests - from the text file or from a field of requests. The big list of requests needs to be saved in the file, which shall settle down in the folder queries, then it can be selected from a field Select file
    • Small amount of requests (to 8kb) can be specified directly into the form Enter queries. Supports macroses
    • In addition available the choice as requests files from the folder with results results/ and from the folder with unsuccessful requests failed/
    • Available multifile selection, then for each file will be created individual task

    • Query format - The formatting template, applied to each request, allows to add substitutions.
      Supports macroses
    • Option
      • Unique queries - Whether defines to do requests by unique. In requests can be unwanted repetitions - the parcer automatically will delete them
      • Save failed queries - Allows to save unsuccessful requests which can be parsing repeatedly. Requests remain in the folder queries/failed
      • Use format on all levels - Enables applying the general format of query at all levels in a multi-level parsing.
      • Query Builders after format - If activated - the general query format is used before applying the Query builder. If disabled - on the contrary.
      • Query Builders on all levels - Enables Query builder applying at every level of a multi-level parsing.


    • Options
      • Unique string - Defines to do results unique or not. It is a simple method of unique - after formation of the end result, it breaks line by line and there is a sifting of repeated lines
      • Overwrite file - Defines whether to overwrite the file of results or add to it.
        For this option are applied the following rules:
        ▫ If file name is generated during work of task (dynamic) - then the file is overwritten during work of task too
        ▫ If the file name is static, then the rewriting takes place only during starting task
        ▫ Macroes $datefile.format() - this is an exception, and it refers to the static. But if to it append anything other than extension, then it will be a dynamic name​
    • File name - File name for saving result. Results remain in the folder results/


    • Do log - Determines whether the execution log messages by each task
    • Keep unique - Opportunity to save information on unique
      We will allow to us it is necessary to parsing links in one task, and in the future to parsing still links on other requests, but thus we want that only new links remained:
      In case of the first parsing we create new basis of an unique entering it name. In case of the second and subsequent parsing we select earlier created basis of unique
    More options - Expanded settings for results. Includes graphs:
    • Prepend text and Append text - A-Parser can add prepend and append text in the file of result for formation a cap of CSV file, for beginning and finite tags of the XML file and etc.
    • Priority - jobs with a higher priority will receive threads immediately after completing each request for jobs with a lower priority.
      • If the tasks have the same priority, then the priority is given to the task that was added earlier in time;
      • Works only with dynamic thread limits;
      • When adding a job through the API, you must specify the prio field from 1 to 100 (more - above).
    • Run on completion - The ability to run the next job, after the execution of the current job is completed. In the settings, you can specify the required flow configurator and the checkbox named "Use the results file for the query", which allows you to use the file with the result of the current job, as the source of requests for the job specified in the drop-down list.
    • Override tools.js - Overrides the existing tools.js for the current preset, and also allows you to export only the desired functions together with the preset
    Add task - After the parser is set up, it is possible to pass to start of task. It is possible to look at the status of execution of task in a tab Task queue.