Display and Formatting of Results

Available Result Formats

For formatting results in A-Parser'A-Parser the Template ToolkitTemplate Toolkit, is used, which allows easily saving scraping results in various formats:

In text files as a list: one result per line, separated by a delimiter, in a custom format
In CSV files with the possibility of further import into Excel, Google Docs , etc.
In XML, JSON and other data storage formats
In HTML "by generating pages "on the fly"
In SQL dump format for direct database import or by directly writing to SQLite DB
In binary format for saving images (jpg, png, gif, ...), ), documents (pdf, docx, ...), ), executable files and archives (exe, dmg, zip, ...) ) and any other types of data

Editing the Result Format

Result Format - allows you to format results to the desired view using templates, applied for each query-result combination.

The General Result Format is set in the Result Format field
The Result Format for each scraper individually can be set in the scraper settings under Result format

A-Parser supports working with multiple scrapers in one task, in the general result format you must specify which scraper to output results from:

$p1 - results from the first scraper (SE::Google in the screenshot), $p2 - results from the second scraper (SE::Bing in the screenshot)
The serial number of the scraper is displayed to the left of the scraper selection field
$p1.preset and $p2.preset imply that the result format value should be taken from the settings of the respective scrapers
In this example, $p1.preset can be replaced with $p1.serp.format('$link\n') which will have the same effect, but the result format from the settings will no longer be used

The Result Format can be specified in a convenient multiline editor by clicking on the corresponding icon in the editing field:

The following variables are available in the general result format:

$query - the query after formatting
$query.* - all variables related to the query are described in the article Templates in Queries
$p1, $p2, ... - variables for accessing scraping results for each scraper separately (Viewing possible results for each scraper)
$p1.query, $p2.query, ... - queries after formatting considering the query format specified in the settings of each scraper

Prepend and Append Text

A separate Prepend/Append Text is specified for each result file:

For forming the header of a CSV file
For initial and final XML file tags
For the header, main part and footer of HTML files
For any other application options

To activate this feature, you must click on the More Options button at the bottom of the Task Editor

The use of the Template Toolkit, Template Toolkit

$query - the query after formatting
$query.* - all variables related to the query are described in the article Templates in Queries

note

Important! These variables are only available when saving each query to a separate file or when using these same variables in the Result File Name Format.

Result File Name Formatting

A-Parser allows using templates also in the names of resulting files, which enables automatic creation of files and folders based on the current date, by query sequential number, by the query itself, and in any other format.

The following variables are supported in the File Name field:

All variables available for the General Result Format
$queriesfile - the path and file name with queries, if the queries are specified via the form, it will contain queries_from_text.txt
$datefile - the object of the Template Toolkit date plugin%b-%d_%H-%M-%S, , configured to the date format May-08_20-08-38, , displays the current time and date in the form when formatting, the format can be changed in the

By default, the file name is created based on the date and time when the task started

Complex Example

reports/$queriesfile/${query}.txt

A reports folder will be created
A subfolder with the name of the query file will be created
In the subfolder, as many files will be created as queries are used in the task, and the query itself will be used as the file name with the extension .txt

tip

The variable $query is written in the format ${query} to prevent the interpolation of the extension .txt as part of the variable, more details in the documentation on the Template ToolkitTemplate Toolkit

⏩ Video. Naming Result Files

In this video, we will show several examples of naming the result file:

Numbering of the result file corresponding to the queries.
Numbering of the result file + part of the query name.
Naming the result file according to the query, if the query is a link.

Results Viewing

Each scraper has its own set of results. You can view the list of available results by hovering over the scraper with the pointer; the tooltip will display a list of simple results and arrays, along with a list of nested elements:

Yellow highlights results common to all scrapers:

$query - the query passed to the scraper after formatting
$query.orig - the original query (as it was in the file or in the query input field)
$query.first - the first query when using nested scraping options (Parse all results or Parse to level)
$info.success - information about the success of scraping this query
$info.retries - the number of attempts used for this query
$info.stats - scraper operation statistics for this query
$pages.$i.data - an array with unprocessed server responses for the possibility of independently extracting additional information

Green highlights results available only for the

SE::Bing:

$totalcount - the number of search results
$ads with elements $link, $anchor, $visiblelink, $snippet, $position and $page - , an array with a list of ads
$related.$i.key - an array with a list of related keywords
$serp with elements $link, $anchor, $snippet, $cache - , an array with the main search engine results

note

Note that for arrays, the variable $i, is explicitly indicated, meaning that there are multiple elements and they can be accessed by index (position number) or iterated through in a loop.

tip

The result $pages.$i.data will automatically be changed to $data for those scrapers that do not "go through pages" within one query. For example, the DeepL::Translator.

Results Showing

A-Parser A-Parser

Flat Results (Flat)
Array Results (Array)

Let's examine each type using the example of the SE::Google, scraper: a search results screenshot:

Flat Results

Simple Results - are when one query corresponds to one result, examples:

Number of results per query ($totalcount)
Whether the query is a typo ($misspell, not shown in the screenshot)

Other examples:

The value of the translated text ($translated) ) in the DeepL::Translator
Number of referring domains ($domains), ), trust value ($trustflow), ), backlinks ($backlinks) ), etc., in the Rank::MajesticSEO

Single results are saved in regular variables (prefix $ + Latin name)

Array Results

Result Arrays - are when one query corresponds to a list of results, each element of the list may, in turn, contain several nested elements. Let's analyze using the example of Google search results—it is represented in the scraper by the array $serp, . For clarity, let's use a table and record the first 5 search results:

Link ($link)	Anchor ($anchor)	Snippet ($snippet)
http://www.speedtest.net/	Speedtest.net by Ookla - The Global Broadband Speed Test	Test your Internet connection bandwidth to locations around the world with this interactive broadband speed test from Ookla.
http://en.wikipedia.org/wiki/Test_cricket	Test cricket - Wikipedia, the free encyclopedia	Test cricket is the longest form of the sport of cricket. Test matches are played between national representative teams with "Test status", as determined by the ...
http://www.speakeasy.net/speedtest/	Speakeasy Speed Test	Saturday 03-May 2014, 11:04:29 AM Your IP: The Speakeasy Speed Test requires Flash v7 or higher. Please update your browser. See Pricing Or Call Today
http://www.humanmetrics.com/cgi-win/jtypes2.asp	Personality test based on C. Jung and I. Briggs Myers type theory	Humanmetrics Jung Typology Test™ instrument uses methodology, questionnaire, scoring and software that are proprietary to Humanmetrics, and shall not be ...
http://test-ipv6.com/	Test your IPv6.	This will test your browser and connection for IPv6 readiness, as well as show you your current IPV4 and IPv6 address. ... Test your IPv6 connectivity. JavaScript ...

Each search result position is recorded into an array with 3 nested elements - link ($link), ), anchor ($anchor), ), and snippet ($snippet)

Another example is the list of related keywords, which is saved in the array $related:

Keyword($key)
test wwe
depression test
test my speed
wonderlic test
test personality
act test
jiggle test
bipolar test

As you can see, this array has only one nested element - the keyword ($key)

Array elements are numbered starting from 0, here are examples of accessing individual array elements:

$serp.0.link - the first link from the search results
$serp.3.anchor - the fourth anchor from the search results
$related.0.key - the first related keyword

More details on formatting flat results and arrays will be described below

Formatting Principles

After the scraper has collected data into simple results and arrays, they must be displayed (saved to a file) in the required format. For convenience and functionality, Template Toolkit. Template Toolkitis used in A-Parser. Let's explore frequently used constructs using the . Template TestingSE::Google:

The screenshot shows 3 fields:

JSON - the internal data representation in the scraper
Template - - the template according to which the result formatting occurs
Result - - the directly transformed data according to the specified template; this is the exact form in which the result will be written to the file

By changing the template, we can change the appearance of the result. Let's consider the following template:

Text in the Template field:

Report on query: $query    
Competition: $totalcount    
List of links, anchors, and snippets:   
$serp.format('$link $anchor\n$snippet\n\n')

Let's highlight the main rules:

Normal text is output to the result as is, without changes
To output flat results, you need to output the variable containing the desired result with the prefix $ in the required place
The .format, method is used for formatting arrays, which we will cover shortly
\\n is responsible for the line break

Formatting Arrays

Let's analyze the construct:

$serp.format('$link $anchor\n$snippet\n\n')

This entry means that the $serp method must be called for the array .format with the parameter '$link $anchor\n$snippet\n\n'. . The .format method joins all array elements into a string according to the template specified in the parameter. The template itself means: for each element of the array $serp , output the link and anchor separated by a space, then output the snippet on a new line, followed by two more line breaks, resulting in a blank line between the results.

Using the Template Toolkit

Variable Output

To use Template Toolkit, you need to insert the tags [% %], and enter the logic you want to execute inside the tags.

The result of CMS output using a template engine

Iterating through an Array

To output array elements, you must use the FOREACH construct:

[% FOREACH i IN p1.list;    
  i.cms _ "\n"; 
END %]

tip

More information and examples on the template toolkit in Features of Templates in A-Parser.

Examples

Outputting Competition (Total Count)

Outputting competition per query (the count of results per query) for all search engine scrapers (SE::Google, SE::Yandex...).

Result Format:

$query: $totalcount\n

Example Result:

test: 3910000000
viagra: 278000000
pvc windows: 3220000
...

Scraping Links

Outputting links from search engine results.

Result Format:

$serp.format('$link\n')

Example Result:

http://www.speedtest.net/
http://www.speakeasy.net/speedtest/
http://en.wikipedia.org/wiki/Test_cricket
http://www.humanmetrics.com/cgi-win/jtypes2.asp
http://html5test.com/
http://test-ipv6.com/
...

Scraping Suggestions

Outputting search engine suggestions.

Result Format:

$results.format('$suggest\n')

Example Result:

test server tanks online
GIA tests in Russian language
pancake batter recipe
testicle
pizza dough recipe with milk

Outputting Query Data

In Net::HTTP and scrapers based on it, the following output is additionally available:

$proxy - the proxy on which the request was executed
$headers - response headers
$code - response code
$reason - response status

Outputting Variable Values in JSON

The .json method for objects

Outputting All Request Redirects

For this task, the variable $response, is available, which allows retrieving any request variables, including all previous redirects.

Result Format:

$response.Redirects.format('$URI\n--> ')$response.URI

Scraper Result:

Outputting to JSON using the Template Toolkit to record the date

This example shows the output of Net::Whois scraper results in JSON format.

The result will be the domain that was checked, the date of the check, and the check result. As seen in the Result Format, , the date is retrieved using the Template ToolkitTemplate Toolkit.

Result Format:

{    
  "domain": "$query",    
  "date": "[% USE d = date(format = '%d.%m.20%y', locale = 'C');d.format() %]",    
  "expire": "$p1.expire_date", 
}, 

Example Result:

[{
   "domain": "a-parser.com",
   "date": "05.05.2021",
   "expire": "25.02.2022",
},
{}]

Download Example

How to import the example into A-Parser

eJxtVG1v2jAQ/ivWCUQrZaxM2pdM+0BRkTYx6ErRPgQ0efWFeXXszHYYVZT/3nMS
EtruQyTf23P33EtK8Nw9uluLDr2DOCkhr98QwxJ9HP/4baRj79jNkWe5QjaBCHJu
HdrgnZw5kUFgygvlYbeLgFDo6ebGZjyglVvNGNuCMBmXegsxvQd/C7RPW4hONu6x
sSRDtlnfMME+s6C8SGsYkkZDMR5m4w9Xw6dRxJR54FQUqWejy09i3LhdXLLhrkfF
Yy5tizvIJ+NG/tkkI6eKPugKXvMD3hsqOJUKe/WcpCXPkAyDEBmsXbqxP3py5UJI
L43mqmEdOtR3YqMl0aV4bcg3MJfo5tZkfa66HaeOJW17gCCKOvZ7EwNxypXDCByV
OucUKl5bpEfLvbGrPNRD+hKMniq1wAOq3q3Gvy6kEjTOaUpBX9rA/7us3mBUHb3z
VAe0/yzVALG3BYHUwvXqWx8kzMLsibj4RbSVzKQn2c1MocOuXJHyETHvWrYMLcuM
xS5LA9zmpvXNUQtyTPqJTfNWV1Y7eEXkxWBeKh+MTuV+RRSsFHjyLPQ93clKz0y4
gsBMF0rRYBze9Qsyde0ggtC18E3wrE4R2Lf3EoE3Rrmva9KF+7KSFvBjKDCjXp5n
bSFp69XmbnFugbOlqrMnJ/F9c3Ku3tLAkNZ3b2ixiFq16865+weU50cdlxWN64+7
bZwCdK2MgDrkaBYQT6pnHsF5pA==

Checking a Website for Presence in Google News by Keyword

Example of checking for the presence of a site in Google News using a keyword

Result Format:

[% linksToOneString = p1.serp.format('$link. ');
matches = linksToOneString.match('.+?(' _ p1.query.domain _ ').+?');

IF matches.0;
  p1.query.orig _';yes' _ "\n";
ELSE;
  p1.query.orig _';no' _ "\n";
END %]

Example Result:

google scraper|a-parser.com;no
google scraper|forbes.ru;yes

Download Example

How to import the example into A-Parser

eJylVVFv2jAQ/iuR1YpWo1EQ7UuqaaIMpk6stKV9AoRccqQeju3aCQVR/vvOTkjS
ruNlEoq4u+++O9+dz1uSUrM0txoMpIaE4y1R7j8JyagXhj+kjDl4Z173GeZLL5IJ
ZcLDX2EQ8Go8WNNEcSBNoqg2oC3PuOaOhggWNOMpaW5JulGA7HIFWrPIGlmEsqIx
zGUmEENWlGeIae3+DccwqkIS8WrIATRnYmkUumCQGn9wwCXmNfr7x0Psz3Wozg4m
omtQTkU8O4xXWq43GlLNwNQ8W0EQkN102iTYLKyr6UudUNu08fFEuNM+yKGAETqK
2PvqqZZvK+YvHO6kcWQxvtc4vZwIVMyfwSDqo6PvTCcN/8u3k4Y3sywvGeiNX8zB
DAnQ5lg+CTvzJmSCFuLM132viOQHqCi5pGYIbVxuwDTeufQGo95nQCE/4G6+T8Tx
lJTFGNEVPEgsxoK54dvXCKUbmtj6HUU0BWvdF+TUT9d28mgUsZRJQXleUTvKVZUf
BXvJx00i1maFbelrmaAqBUfgUt13Y0yOnEyQInO+d7kPCReUG2gSg6n2KSYSlZZU
Z2hgKWiaSj1UNh1Ub4kUHc4HsAJe+Tv6q4zxCK9dZ4FO14Xj55DhXxy78nT1UDiG
rxpzKFmcdDX8VXlFciBjPHj0hMfmLGEpyqbrrnBIAlQuAVRZshtbskRqKMMUzEV0
XEEKhJ35qmMdVaneHcMtKiMzPbfUeY2b+ztk3PSNFGe2IQZwKeUFIW8WJG1f9i75
IJPprmpyLQYq51IsWDwsruV+HDLxgGtzKLrSLj5bJpFxjk02cF8NW8cUTbVCdd6P
zl0XwlZyvyQxScnNz1F+cqUZ5nxhE0ywMfWoBeWccv54P6hbSDWgKEyy4Ly9cN/A
fs/zb8tpLnKN54S2E9rV//bTGz3L97o/lwneuf/iwvv2hBsAt55NO4VYYh9sN939
CUkvf0vssoO1oiKCKL8SO9ej4oEp36tt/ZkJtzucy9/mNgfZqlsI6rB9BofOPil/
AGn6WSM=

Outputting Timestamp Value in Date Format

Sometimes, regular date is not available in the results, but there is a timestamp value, as in the Social::Instagram::Tag. scraper. This value can be represented in date format using Template ToolkitTemplate-Toolkit.

Example of outputting a timestamp value in date format

Result Format:

[% USE date;

query.orig _ ": total posts - " _ postscount _ "\nPosts:\n";

FOREACH i IN posts;
    d = date.format(i.time, format => '%d.%m.20%y');
    i.link _ " - " _ d _ ":\n";
    i.text _ "\n";
END %]

Example Result:

sport: total posts - 96500663
Posts:
https://www.instagram.com/p/COfJHshAkeD/ - 05.05.2021:
Quelques exemples de notre nouvelle campagne de communication personnalisable avec le nom des clubs 😀

Vous préférez quel visuel : 1, 2, 3, 4, 5 ? 🤔

#clubnormand #tennis #padel #beachtennis #tenniscourt #padelcourt #beachtenniscourt #lnt #LigueNormandieTennis #🎾 #sport #normandie #normandietourisme
https://www.instagram.com/p/COfJG7olavg/ - 05.05.2021:
💥 Sau màn lật đổ “Bà già” thành công, Nửa xanh thành Milan chính thức vượt qua Nửa đỏ về số lần lên đỉnh nước Ý nhiều nhất lịch sử.
-----------------------------
➖ Website: https://webthethao247.com/
➖ https://g.page/webthethao247?share
#wtt247 #webthethao247 #thethao #sport #bongda #SerieA #InterMilan #Juventus #ACMilan
https://www.instagram.com/p/COfJG1Hg7ax/ - 05.05.2021:
Which Skill was better 1 or 2? 🤔👇
Follow @ftb4ll for more 💥
Follow @ftb4ll for more 💥
Follow @ftb4ll for more 💥
________________________________________
Leave a Like 👍🏽
Subscribe for more 🔔
Leave your thoughts in the Comments  💬
________________________________________
❌Ignore the Tags ❌
#football #soccer #fussball #futbol #fifa #championsleague #bundesliga #ucl #footballmemes #goal #transfer #sports #penalty #ultimateteam #pacybits #fut #ultras #laliga #freekick #referee #sport #calcio #messi #ronaldo #skills #premierleague #foul #footballseason
https://www.instagram.com/p/COfIlXqhfAa/ - 05.05.2021:
Be Fuckin’ Ready 🤣🤣🤣

Get ready to fly!!!! 🏐🏐🏐🏐

Follow - @crackonkings 

#beachball #nalin&kane #trance #music #90s #onyerhead #festival #party #afterparty #love #summer #uk #happy #sesh #crackon #football #sport #festivaloutfit #festivalfashion #sun #dj #dancing #club #festivalgirl #house #techno #rave
...

Download Example

How to import the example into A-Parser

eJx1VNtuGjEQ/RXLCkoj0VVSqS9bpRKhoFJRNuXyxKLKxQa58dqO7aVBiH/vjPeW
pM0+7RzPzDlzsU80MP/g753wIniark/Uxn+a0oXZSqbSdKJ9YHvHijRdsj15T5ay
EAAVlgRDOAuCiCewlKB9apnzwmGm9RsJwImLHStVoP0TDUcrgMschHOSYwbJwd4Z
V7AASqIbPTBVotu6R1aLUeT8lOtcP5bCHRPj5J78JDlNQVBgiljjgwehOQU4GltT
6hB98lzfI5LCT05jlnE2Hw2GX4kkk1nljnDg5DYyJZWYdzIJUHifVCa5/Uwuezzp
FcmH697x8gpiCHwyUVI/RKpGAa/ENYSVVxBPraAIj2ZfSG9Dz5tNn1aF+3Fkgrov
7E1Sz6U9XLCDWBrslYydb2LAmrECu3WB6vG0qeAqCU+YgXEugzSaqYoBx9WxrrR8
jN3WBnyxxVL4sTMFQKi6Bo+NujW9iDaFFGWM/VHF0HTHlBd96kHqmIEQ/vpEBuFY
MC6zqAfwEzV6oNRUHITq3GL+u1IqDrs12EHQpA78v0v2T45zW95zKti6Pw40tFmi
dZd976K4mZo9VM5/Qd1KFjKA7Ye4T4BeA/gghG17NsOeFcaJlqbOXLPDTbNC44p3
IxvYDnpRxouxvAS3Ru/kPquvTeNZ6iVc50wPDV5HrEuXSsFYvJh36zHw9RjQ6AS+
Dh5GCiy9ua40GKP8t0Ul1ToJ6/cRBRbQyeesdcotU2o1nz4/wUAfjM30yLlX44uL
Qb01Drm2sLx7A2sFpZ037cvSPlWnt96X9HSGuf3291UAFonugEG3PAyFpjfnvwdy
t1Y=

Other Examples

Examples of frequently used templates

Available Result Formats​

Editing the Result Format​

Prepend and Append Text​

Result File Name Formatting​

Complex Example​

⏩ Video. Naming Result Files​

Results Viewing​

Results Showing​

Flat Results​

Array Results​

Formatting Principles​

Formatting Arrays​

Using the Template Toolkit​

Variable Output​

Iterating through an Array​

Examples​

Outputting Competition (Total Count)​

Scraping Links​

Scraping Suggestions​

Outputting Query Data​

Outputting Variable Values in JSON​

Outputting All Request Redirects​

Outputting to JSON using the Template Toolkit to record the date​

Checking a Website for Presence in Google News by Keyword​

Outputting Timestamp Value in Date Format​

Other Examples​

Available Result Formats

Editing the Result Format

Prepend and Append Text

Result File Name Formatting

Complex Example

⏩ Video. Naming Result Files

Results Viewing

Results Showing

Flat Results

Array Results

Formatting Principles

Formatting Arrays

Using the Template Toolkit

Variable Output

Iterating through an Array

Examples

Outputting Competition (Total Count)

Scraping Links

Scraping Suggestions

Outputting Query Data

Outputting Variable Values in JSON

Outputting All Request Redirects

Outputting to JSON using the Template Toolkit to record the date

Checking a Website for Presence in Google News by Keyword

Outputting Timestamp Value in Date Format

Other Examples