Check availability of the mobile version for 1000000 websites

Taken as a basis TOP 1M domains of Alexa: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

The test machine:

• Virtual machine Intel(R) Core(TM) i7 CPU @ 3.4 GHz with 2 dedicated cores, 3 Gb RAM and bandwidth 100 mbit/s. OS Windows 7 x64.

Task:

Spoiler: Preset

Code:

eyJwcmVzZXQiOiJcdTA0MWZcdTA0NDBcdTA0M2VcdTA0MzJcdTA0MzVcdTA0NDBc
dTA0M2FcdTA0MzAgXHUwNDIyXHUwNDFlXHUwNDFmIDFcdTA0MWMgXHUwNDEwXHUw
NDNiXHUwNDM1XHUwNDNhXHUwNDQxXHUwNDRiIFx1MDQzZFx1MDQzMCBcdTA0M2Rc
dTA0MzBcdTA0M2JcdTA0MzhcdTA0NDdcdTA0MzhcdTA0MzUgXHUwNDNjXHUwNDNl
XHUwNDMxXHUwNDM4XHUwNDNiXHUwNDRjXHUwNDNkXHUwNDNlXHUwNDM5IFx1MDQz
Mlx1MDQzNVx1MDQ0MFx1MDQ0MVx1MDQzOFx1MDQzOCIsInZhbHVlIjp7InByZXNl
dCI6Ilx1MDQxZlx1MDQ0MFx1MDQzZVx1MDQzMlx1MDQzNVx1MDQ0MFx1MDQzYVx1
MDQzMCBcdTA0MjJcdTA0MWVcdTA0MWYgMVx1MDQxYyBcdTA0MTBcdTA0M2JcdTA0
MzVcdTA0M2FcdTA0NDFcdTA0NGIgXHUwNDNkXHUwNDMwIFx1MDQzZFx1MDQzMFx1
MDQzYlx1MDQzOFx1MDQ0N1x1MDQzOFx1MDQzNSBcdTA0M2NcdTA0M2VcdTA0MzFc
dTA0MzhcdTA0M2JcdTA0NGNcdTA0M2RcdTA0M2VcdTA0MzkgXHUwNDMyXHUwNDM1
XHUwNDQwXHUwNDQxXHUwNDM4XHUwNDM4IiwicGFyc2VycyI6W1siTmV0OjpIVFRQ
IiwiZGVmYXVsdCIseyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InF1ZXJ5Zm9ybWF0
IiwidmFsdWUiOiJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9wYWdlc3BlZWRv
bmxpbmUvdjNiZXRhMS9tb2JpbGVSZWFkeT9rZXk9QUl6YVN5RGtFWC1mMUpOTFFM
QzE2NFNaYW9iQUxxRnY0UEhWLWtBJnNjcmVlbnNob3Q9ZmFsc2Umc25hcHNob3Rz
PWZhbHNlJmxvY2FsZT1lbl9VUyZ1cmw9aHR0cCUzQSUyRiUyRiRxdWVyeS5kb21h
aW4lMkYmc3RyYXRlZ3k9bW9iaWxlJmZpbHRlcl90aGlyZF9wYXJ0eV9yZXNvdXJj
ZXM9ZmFsc2UmY2FsbGJhY2s9X2NhbGxiYWNrc18uX2RjdXZiQnZuSVhWQiJ9LHsi
dHlwZSI6ImZpbHRlciIsInJlc3VsdCI6ImRhdGEiLCJmaWx0ZXJUeXBlIjoiY29u
dGFpbiIsInZhbHVlIjoiXCJwYXNzXCI6IGZhbHNlIiwib3B0aW9uIjoic2VucyJ9
XV0sInJlc3VsdHNGb3JtYXQiOiIkcXVlcnkuZG9tYWluXFxuIiwicmVzdWx0c1Nh
dmVUbyI6ImZpbGUiLCJyZXN1bHRzRmlsZU5hbWUiOiJ0b3AtMW0td2l0aG91dC1t
b2JpbGUtdmVyLnR4dCIsImFkZGl0aW9uYWxGb3JtYXRzIjpbXSwicmVzdWx0c1Vu
aXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiJHF1ZXJ5Il0sInVuaXF1ZVF1ZXJp
ZXMiOmZhbHNlLCJzYXZlRmFpbGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0
aW9ucyI6eyJvbkFsbExldmVscyI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNBZnRlckl0
ZXJhdG9yIjpmYWxzZX0sInJlc3VsdHNPcHRpb25zIjp7Im92ZXJ3cml0ZSI6ZmFs
c2V9LCJkb0xvZyI6Im5vIiwia2VlcFVuaXF1ZSI6Ik5vIiwibW9yZU9wdGlvbnMi
OmZhbHNlLCJyZXN1bHRzUHJlcGVuZCI6IiIsInJlc3VsdHNBcHBlbmQiOiIiLCJx
dWVyeUJ1aWxkZXJzIjpbeyJzb3VyY2UiOiJxdWVyeSIsInR5cGUiOiJzdHJpbmdT
cGxpdCIsInNlcGFyYXRvciI6IiwiLCJ0byI6WyJyYW5rIiwiZG9tYWluIl19XSwi
cmVzdWx0c0J1aWxkZXJzIjpbXSwiY29uZmlnT3ZlcnJpZGVzIjpbXX19

• The source file with requests contains data in the format <alexa-rank>, <domain>, a million entries, each domain on a new line. Using the Query Builder separate the domain and its rank.
• Used parser Net::HTTP, to which is input the link with inserted domain to check for the mobile version. Mobile version check here: https://www.google.com/webmasters/tools/mobile-friendly/
• Using a proxy is enabled.
• The result, which in this case is all domains that do not have a mobile version, are saved to the file top-1m-without-mobile-ver.txt

Working task:


Statistics:

• The speed of parsing was about 2000 domains per minute
• Failed requests just 26 of 1000000
• Parsing passed in 1000 threads
• Elapsed time: 8 hours 7 minutes

Result:

• 40.98% of the million most visited sites do not have a mobile version!

File with the list of websites without mobile versions: http://files.a-parser.com/top-1m-without-mobile-ver.txt