1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Join our Telegram chat: https://t.me/a_parser_en
    Dismiss Notice

Check availability of the mobile version for 1000000 websites

Discussion in 'Share Experience' started by Support, Sep 16, 2015.

  1. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    3,882
    Likes Received:
    1,969
    Taken as a basis TOP 1M domains of Alexa: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

    The test machine:
    • Virtual machine Intel(R) Core(TM) i7 CPU @ 3.4 GHz with 2 dedicated cores, 3 Gb RAM and bandwidth 100 mbit/s. OS Windows 7 x64.
    Task:
    [​IMG]
    Code:
    eyJwcmVzZXQiOiJcdTA0MWZcdTA0NDBcdTA0M2VcdTA0MzJcdTA0MzVcdTA0NDBc
    dTA0M2FcdTA0MzAgXHUwNDIyXHUwNDFlXHUwNDFmIDFcdTA0MWMgXHUwNDEwXHUw
    NDNiXHUwNDM1XHUwNDNhXHUwNDQxXHUwNDRiIFx1MDQzZFx1MDQzMCBcdTA0M2Rc
    dTA0MzBcdTA0M2JcdTA0MzhcdTA0NDdcdTA0MzhcdTA0MzUgXHUwNDNjXHUwNDNl
    XHUwNDMxXHUwNDM4XHUwNDNiXHUwNDRjXHUwNDNkXHUwNDNlXHUwNDM5IFx1MDQz
    Mlx1MDQzNVx1MDQ0MFx1MDQ0MVx1MDQzOFx1MDQzOCIsInZhbHVlIjp7InByZXNl
    dCI6Ilx1MDQxZlx1MDQ0MFx1MDQzZVx1MDQzMlx1MDQzNVx1MDQ0MFx1MDQzYVx1
    MDQzMCBcdTA0MjJcdTA0MWVcdTA0MWYgMVx1MDQxYyBcdTA0MTBcdTA0M2JcdTA0
    MzVcdTA0M2FcdTA0NDFcdTA0NGIgXHUwNDNkXHUwNDMwIFx1MDQzZFx1MDQzMFx1
    MDQzYlx1MDQzOFx1MDQ0N1x1MDQzOFx1MDQzNSBcdTA0M2NcdTA0M2VcdTA0MzFc
    dTA0MzhcdTA0M2JcdTA0NGNcdTA0M2RcdTA0M2VcdTA0MzkgXHUwNDMyXHUwNDM1
    XHUwNDQwXHUwNDQxXHUwNDM4XHUwNDM4IiwicGFyc2VycyI6W1siTmV0OjpIVFRQ
    IiwiZGVmYXVsdCIseyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InF1ZXJ5Zm9ybWF0
    IiwidmFsdWUiOiJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9wYWdlc3BlZWRv
    bmxpbmUvdjNiZXRhMS9tb2JpbGVSZWFkeT9rZXk9QUl6YVN5RGtFWC1mMUpOTFFM
    QzE2NFNaYW9iQUxxRnY0UEhWLWtBJnNjcmVlbnNob3Q9ZmFsc2Umc25hcHNob3Rz
    PWZhbHNlJmxvY2FsZT1lbl9VUyZ1cmw9aHR0cCUzQSUyRiUyRiRxdWVyeS5kb21h
    aW4lMkYmc3RyYXRlZ3k9bW9iaWxlJmZpbHRlcl90aGlyZF9wYXJ0eV9yZXNvdXJj
    ZXM9ZmFsc2UmY2FsbGJhY2s9X2NhbGxiYWNrc18uX2RjdXZiQnZuSVhWQiJ9LHsi
    dHlwZSI6ImZpbHRlciIsInJlc3VsdCI6ImRhdGEiLCJmaWx0ZXJUeXBlIjoiY29u
    dGFpbiIsInZhbHVlIjoiXCJwYXNzXCI6IGZhbHNlIiwib3B0aW9uIjoic2VucyJ9
    XV0sInJlc3VsdHNGb3JtYXQiOiIkcXVlcnkuZG9tYWluXFxuIiwicmVzdWx0c1Nh
    dmVUbyI6ImZpbGUiLCJyZXN1bHRzRmlsZU5hbWUiOiJ0b3AtMW0td2l0aG91dC1t
    b2JpbGUtdmVyLnR4dCIsImFkZGl0aW9uYWxGb3JtYXRzIjpbXSwicmVzdWx0c1Vu
    aXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiJHF1ZXJ5Il0sInVuaXF1ZVF1ZXJp
    ZXMiOmZhbHNlLCJzYXZlRmFpbGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0
    aW9ucyI6eyJvbkFsbExldmVscyI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNBZnRlckl0
    ZXJhdG9yIjpmYWxzZX0sInJlc3VsdHNPcHRpb25zIjp7Im92ZXJ3cml0ZSI6ZmFs
    c2V9LCJkb0xvZyI6Im5vIiwia2VlcFVuaXF1ZSI6Ik5vIiwibW9yZU9wdGlvbnMi
    OmZhbHNlLCJyZXN1bHRzUHJlcGVuZCI6IiIsInJlc3VsdHNBcHBlbmQiOiIiLCJx
    dWVyeUJ1aWxkZXJzIjpbeyJzb3VyY2UiOiJxdWVyeSIsInR5cGUiOiJzdHJpbmdT
    cGxpdCIsInNlcGFyYXRvciI6IiwiLCJ0byI6WyJyYW5rIiwiZG9tYWluIl19XSwi
    cmVzdWx0c0J1aWxkZXJzIjpbXSwiY29uZmlnT3ZlcnJpZGVzIjpbXX19
    • The source file with requests contains data in the format <alexa-rank>, <domain>, a million entries, each domain on a new line. Using the Query Builder separate the domain and its rank.
    • Used parser Net::HTTP, to which is input the link with inserted domain to check for the mobile version. Mobile version check here: https://www.google.com/webmasters/tools/mobile-friendly/
    • Using a proxy is enabled.
    • The result, which in this case is all domains that do not have a mobile version, are saved to the file top-1m-without-mobile-ver.txt
    Working task:
    [​IMG]

    Statistics:
    • The speed of parsing was about 2000 domains per minute
    • Failed requests just 26 of 1000000
    • Parsing passed in 1000 threads
    • Elapsed time: 8 hours 7 minutes
    [​IMG]
    Result:
    • 40.98% of the million most visited sites do not have a mobile version!
    File with the list of websites without mobile versions: http://files.a-parser.com/top-1m-without-mobile-ver.txt
     

Share This Page