Skip to main content

Task queue

The work of A-Parser is based on a task queue - you add tasks and they are executed in order, allowing you to load the scraper with work for a long time and not worry about when it will finish.

Task queue

The main advantages of task queue in A-Parser

  • Support for executing multiple tasks simultaneously
  • Pausing and stopping task execution
  • Saving intermediate results of task execution, which allows you to continue work from the moment the scraper was turned off (or the computer/server was turned off)
  • Moving individual tasks in the queue
  • Ability to edit already created tasks, as well as copy tasks for reuse
  • Completed tasks go to the completed tasks queue, where you can view the results of each task and download the result

The algorithm of work

Algorithm of task queue

Working slots of tasks

The number of simultaneously working tasks is regulated by working slots, this parameter is specified in the general settings (Settings -> General settings -> Maximum active tasks). The default value is 1, i.e. one task is working simultaneously, others will wait for their turn.

Task statuses in A-Parser

Each task has its own status, here is a list of possible states:

StatusDescription
waitSlotWaiting for a free slot to execute the task, a new task always enters the queue with this status
workTask is being executed
pausedTask is paused, while holding a working slot, preventing the next tasks from starting
stoppedTask is stopped, freeing up a working slot
completedTask is completed - all requests are processed, the task goes to the completed tasks queue, the working slot is freed

When setting the Maximum active tasks parameter to more than one, it should be taken into account that the number of threads will increase, as well as the overall resource consumption by the scraper.

tip

To achieve maximum performance, always use only one task with the optimal number of threads for a particular computer/server.