Tools in the Templаte toolkit

Description(top)

In template processing system Template Toolkit the global variable $tools is available, which stores in itself a tool kit available in any template.
$tools.error - contains a description of errors that may occur during executing any of tools.

$tools.memory.* - key / value storage in memory (top)

Simple storage in memory, common to all tasks, api requests, etc., is reset when the parser is restarted

• tools.memory.set(key, value) - sets the value of the key
  
• tools.memory.get(key) - returns the value corresponding to the key
  
• tools.memory.delete(key) - deletes the record from memory

$tools.sqlite.* - work with SQLite databases(top)

• $tools.sqlite.get - method that allows you to retrieve information from a database using SELECT
  
  Code:

  [% res = tools.sqlite.get('results/test.sqlite', 'SELECT COUNT(*) AS count FROM test') %]

• $tools.sqlite.run - method that allows you to perform operations with DB (INSERT, DROP, etc)
  
  Code:

  [% res = tools.sqlite.run('results/test.sqlite', 'INSERT INTO test VALUES(?)', 'test') %]

• $tools.sqlite.all - method that allows you to print all the data from a table

$tools.ua- changeover of User agent(top)

This tool is intended for substitution of User agent in parsers which use it (for example Net::HTTP).

• $tools.ua.list - contains the User agent list
• $tools.ua.random() - a method, brings accidental User agent out of the list

List of all User agents is stored in file files/tools/user-agents.txtWhen using this tool for User agent parameter in parsers, it is necessary to specify it obviously: [% tools.ua.random() %]

$tools.query - adding of requests(top)

This tool allows to add requests to already existing directly during work of task, creating them on a basis already parsed results. It can be used as analog of the Parse to level function in those parsers where it isn't realized.

• [% tools.query.add (query, maxLevel) %] - adds single request
• [% tools.query.addAll (array, item, maxLevel) %] - adds an array of requests

The maxLevel parameter specifies to what level adding requests, and is optional: if it is lowered, actually the parser will add new requests they is so far. Also it is recommended turn on option Unique requests to avoid cycling and excessive work of parser.

Result:

$tools.parseJSON - parsing JSON structures(top)

This tool allows you to deserialize the data in JSON format to the variables, available in the template engine.

• [% tools.parseJSON(data) %] - launches the analysis of structure

After deserialize to keys from JSON array can be accessed as regular variables and arrays.

$tools.js.* - JavaScript support in tools(top)

This tool allows you to add custom JS functions and use them directly in the template engine. Since version 1.2.527, the use of Node.js modules is supported.
Functions are added in the JavaScript Editor:

$tools.base64.* - ability to encode and decode base64(top)

This tool allows you to encode the text in base64 and decode back. Example:

$tools.data - built-in parameter directory(top)

This tool contains a lot of pre-installed information - languages, regions, domains for Google and Yandex search engines. Complete list of elements:
["YandexWordStatRegions","TopDomains","CountryCodes","YahooLocalDomains","GoogleDomains","BingTranslatorLangs","Top1000Words","GoogleLangs","GoogleInterfaceLangs","EnglishMonths","GoogleTrendsCountries"]
each of which is an array or hash data, you can see the contents like this:

$tools.CSVline - easy creation of CSV files(top)

This tool automatically converts the values to the CSV format and adds a line break, now in the result format it is enough to list all the variables, the output will be a valid CSV file, ready for import into Google Docs/Excel/etc.
Usage example:

$tools.aparser.version() - Get information about the A-Parser version(top)

This tool allows you to get information about the version of A-Parser and display it in the result.

Example of use:

$tools.task() - returns task id and number of threads(top)

This tool allows you to get information about the task id and show the number of threads.

Example of use: