1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Join our Telegram chat: https://t.me/a_parser_en
    Dismiss Notice

Download pictures of specified resolution and size

Discussion in 'Share Experience' started by Support, Sep 23, 2015.

  1. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    4,348
    Likes Received:
    2,097
    Task: need to download pictures a certain theme with a resolution 1920x1080 and a minimum size from 1 to 10 megabytes. Themse set a keywords:
    They must also be sorted by topic.

    Solution:
    1) Looking with a given key all links to pictures using a parser SE::Google::Images SE::Google::Images and the filters.
    [​IMG]
    • Use the filters on the height, width, size and availability at the end of links file extension (in order to save with him then). The height and width is 1 less than the required in order to grab the standard format of 1920x1080.
    • Every link save with its keyword, then to use it to the folder name.
    • Turn on unique as to avoid repetitions.
    Code:
    eyJwcmVzZXQiOiJEb3dubG9hZCBwaWN0dXJlcyAtIDEiLCJ2YWx1ZSI6eyJwcmVz
    ZXQiOiJEb3dubG9hZCBwaWN0dXJlcyAtIDEiLCJwYXJzZXJzIjpbWyJTRTo6R29v
    Z2xlOjpJbWFnZXMiLCJkZWZhdWx0Iix7InR5cGUiOiJmaWx0ZXIiLCJyZXN1bHQi
    Olsic2VycCIsIndpZHRoIl0sImZpbHRlclR5cGUiOiI+IiwidmFsdWUiOiIxOTE5
    Iiwib3B0aW9uIjoic2VucyJ9LHsidHlwZSI6ImZpbHRlciIsInJlc3VsdCI6WyJz
    ZXJwIiwiaGVpZ2h0Il0sImZpbHRlclR5cGUiOiI+IiwidmFsdWUiOiIxMDc5Iiwi
    b3B0aW9uIjoic2VucyJ9LHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJmb3JtYXRy
    ZXN1bHQiLCJ2YWx1ZSI6IiRzZXJwLmZvcm1hdCgnJHF1ZXJ5Lm9yaWc7JGxpbmtc
    XG4nKSJ9LHsidHlwZSI6InVuaXF1ZSIsInJlc3VsdCI6WyJzZXJwIiwibGluayJd
    LCJ1bmlxdWVUeXBlIjoic3RyaW5nIiwidW5pcXVlR2xvYmFsIjpmYWxzZX0seyJ0
    eXBlIjoiZmlsdGVyIiwicmVzdWx0IjpbInNlcnAiLCJsaW5rIl0sImZpbHRlclR5
    cGUiOiJyZW1hdGNoIiwidmFsdWUiOiJcXC4oXFx3ezMsNH0pJCIsIm9wdGlvbiI6
    IiJ9LHsidHlwZSI6ImZpbHRlciIsInJlc3VsdCI6WyJzZXJwIiwic2l6ZSJdLCJm
    aWx0ZXJUeXBlIjoiPiIsInZhbHVlIjoiMTAyNCIsIm9wdGlvbiI6InNlbnMifSx7
    InR5cGUiOiJmaWx0ZXIiLCJyZXN1bHQiOlsic2VycCIsInNpemUiXSwiZmlsdGVy
    VHlwZSI6IjwiLCJ2YWx1ZSI6IjEwMjQwIiwib3B0aW9uIjoic2VucyJ9XV0sInJl
    c3VsdHNGb3JtYXQiOiIkcDEucHJlc2V0IiwicmVzdWx0c1NhdmVUbyI6ImZpbGUi
    LCJyZXN1bHRzRmlsZU5hbWUiOiJwaWN0dXJlc1xcbGlua3NfJGRhdGVmaWxlLmZv
    cm1hdCgpLnR4dCIsImFkZGl0aW9uYWxGb3JtYXRzIjpbXSwicmVzdWx0c1VuaXF1
    ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiJHF1ZXJ5Il0sInVuaXF1ZVF1ZXJpZXMi
    OmZhbHNlLCJzYXZlRmFpbGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0aW9u
    cyI6eyJvbkFsbExldmVscyI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNBZnRlckl0ZXJh
    dG9yIjpmYWxzZX0sInJlc3VsdHNPcHRpb25zIjp7Im92ZXJ3cml0ZSI6dHJ1ZX0s
    ImRvTG9nIjoibm8iLCJrZWVwVW5pcXVlIjoiTm8iLCJtb3JlT3B0aW9ucyI6ZmFs
    c2UsInJlc3VsdHNQcmVwZW5kIjoiIiwicmVzdWx0c0FwcGVuZCI6IiIsInF1ZXJ5
    QnVpbGRlcnMiOltdLCJyZXN1bHRzQnVpbGRlcnMiOltdLCJjb25maWdPdmVycmlk
    ZXMiOltdfX0=

    2) By the received in the 1st step links, using Net::HTTP Net::HTTP, download images and saved into the appropriate folders.
    [​IMG]
    • Query builder extract link, name for the folder and file extension.
    • Proxies do not use (the different sites, and it's faster).
    • Change Max body size, to large images are not cropped.
    • Filters out everything is less than 100 KB (it can be is given on the link page and not a picture).
    • The file name and path are generating from previously received data.
    • ATTENTION! I do not recommend to set a large number of threads, optimally 50-100 threads.
    Code:
    eyJwcmVzZXQiOiJEb3dubG9hZCBwaWN0dXJlcyAtIDIiLCJ2YWx1ZSI6eyJwcmVz
    ZXQiOiJEb3dubG9hZCBwaWN0dXJlcyAtIDIiLCJwYXJzZXJzIjpbWyJOZXQ6OkhU
    VFAiLCJkZWZhdWx0Iix7InR5cGUiOiJvdmVycmlkZSIsImlkIjoidXNlcHJveHki
    LCJ2YWx1ZSI6ZmFsc2V9LHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJnb29kQ29k
    ZSIsInZhbHVlIjoyMDB9LHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJxdWVyeWZv
    cm1hdCIsInZhbHVlIjoiJHF1ZXJ5LmxpbmsifSx7InR5cGUiOiJvdmVycmlkZSIs
    ImlkIjoiZm9ybWF0cmVzdWx0IiwidmFsdWUiOiIkZGF0YSJ9LHsidHlwZSI6Im92
    ZXJyaWRlIiwiaWQiOiJtYXhfc2l6ZSIsInZhbHVlIjoiMTA0ODU3NjAifSx7InR5
    cGUiOiJmaWx0ZXIiLCJyZXN1bHQiOiJbJSBkYXRhLmxlbmd0aCAlXSIsImZpbHRl
    clR5cGUiOiI+IiwidmFsdWUiOiIxMDI0MDAiLCJvcHRpb24iOiJzZW5zIn1dXSwi
    cmVzdWx0c0Zvcm1hdCI6IiRwMS5wcmVzZXQiLCJyZXN1bHRzU2F2ZVRvIjoiZmls
    ZSIsInJlc3VsdHNGaWxlTmFtZSI6InBpY3R1cmVzLyRxdWVyeS5uYW1lLyRxdWVy
    eS5udW0tJGRhdGVmaWxlLmZvcm1hdCgpLiRxdWVyeS5leHQiLCJhZGRpdGlvbmFs
    Rm9ybWF0cyI6W10sInJlc3VsdHNVbmlxdWUiOiJubyIsInF1ZXJ5Rm9ybWF0Ijpb
    IiRxdWVyeSJdLCJ1bmlxdWVRdWVyaWVzIjpmYWxzZSwic2F2ZUZhaWxlZFF1ZXJp
    ZXMiOmZhbHNlLCJpdGVyYXRvck9wdGlvbnMiOnsib25BbGxMZXZlbHMiOmZhbHNl
    LCJxdWVyeUJ1aWxkZXJzQWZ0ZXJJdGVyYXRvciI6ZmFsc2V9LCJyZXN1bHRzT3B0
    aW9ucyI6eyJvdmVyd3JpdGUiOmZhbHNlfSwiZG9Mb2ciOiJubyIsImtlZXBVbmlx
    dWUiOiJObyIsIm1vcmVPcHRpb25zIjpmYWxzZSwicmVzdWx0c1ByZXBlbmQiOiIi
    LCJyZXN1bHRzQXBwZW5kIjoiIiwicXVlcnlCdWlsZGVycyI6W3sic291cmNlIjoi
    cXVlcnkiLCJ0eXBlIjoic3RyaW5nU3BsaXQiLCJzZXBhcmF0b3IiOiI7IiwidG8i
    OlsibmFtZSIsImxpbmsiXX0seyJzb3VyY2UiOiJsaW5rIiwidHlwZSI6InJlZ2V4
    IiwicmVnZXgiOiJcXC4oXFx3KykkIiwicmVnZXhUeXBlIjpudWxsLCJ0byI6WyJl
    eHQiXX1dLCJyZXN1bHRzQnVpbGRlcnMiOltdLCJjb25maWdPdmVycmlkZXMiOltd
    fX0=

    Result:
    As a result, we have 360 pictures (more than 1 GB) sorted by category in 4 minutes on 50 streams. Not a bad update of wallpaper:)
     

Share This Page