Follow link redirects and save destination url

Discussion in 'A-Parser Support Forum' started by scrapefun, Aug 21, 2019.

  1. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    129
    Likes Received:
    17
    I need to take a list of links that use redirects, follow all redirects and then save the final destination url.

    The query is formatted like:

    <UNIQUE_ID>,<URL_THAT_REDIRECTS>


    I want the output to be formatted like:
    <UNIQUE_ID>,<URL_THAT_REDIRECTS>,<FINAL_DESTINATION_URL>
     
  2. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    2,817
    Likes Received:
    1,663
  3. Support Денис

    Support Денис A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Jun 12, 2017
    Messages:
    586
    Likes Received:
    178
    You can try this
    [​IMG]
    eJx9VG1P4kAQ/ivNRhPI9WqJineVmCDKgSIolPiBELNHB26PpVt3t6hH+O83uy0t
    ei9fKDszzzPPzstuiKZqqe4lKNCKBJMNSex/EpAI5jTlmrgkoVKBNO4J6YMOgk4Y
    3qO9iNgQ/ZYAYsQapGQRoJNFeE4R95kuIDY0a8pTE3QnfjHO6dGp5zuVRxZH4kU5
    /dCp+Z5/7qChfnLuvNZPqk4zSTg8wvdbpo9Oj8+847pTue2Edz3X4WwJzjeYLUXV
    af2QYgVHZ3XP946/+F+9mu87IzqnkuUwsv23xucU5NtcyBXdFzk5tHYvlfxw+j94
    hsSiZbXa4Q8yuJBs4TYO0J2IWIE3HnYvyHY6dUmGUO0sMQKSmpfXvnCO6BpCYZIw
    DqW5jac+Xdk0EdVgvF6mo1L19KthoFHENBMx5VkG070y6zhmz1ZmLDDWSGWg2lhF
    NGmwBFb/Tt0kvw9BitRiHzIMCeaUK3CJQqltikKijx6mQVIt5CAxetC+ISJuct6D
    NfAyzPJfpoxHOGrNOYK6OfDvIYM/OLbF9fZTYbteJGooWOzpcnBXoiLRE4tdMZYA
    SVGevrGshISCMSfJE+HiJBCbMSi7g0NbmN4ptuulRCpnsBs7DMnHSsICXi2J+Qak
    4n2qug3ze7GzhjYyTjlHlDBNwQHEfkhOptuyt3vp0DgT8ZwtBvnQ7qYgjUPc+0Hc
    EivcMF3Q4r4OyxlrqryX5lBe/SO4ZVO8ezG0EFzdjLIiJJKh2lMjcIXt2M+aU84o
    5+Nhb99DyrnEQ2Pc7z6Mr5+6VxduAyOfwk4zfBpeX3WH161wZEo0w0VYCCwp3tBU
    I3+0iqdts/d0BZsttv2nus9izPVMBNqwTgobTYLa9jfitLw8
     
    #3 Support Денис, Aug 21, 2019
    Last edited: Aug 21, 2019
    scrapefun likes this.
  4. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    129
    Likes Received:
    17
    Thanks for this. I had to make a couple minor changes but it really helped me out.
     
    Support likes this.
  5. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    129
    Likes Received:
    17
    I'm having a bit of trouble getting the final url from some link redirects.

    Here is the preset I am using that includes a few samples that are not working for me:
    eJyFVmlv2zgQ/SuGUBgpYN2WfBRCkBbI9sjRTdLth6QoaHFks6ZIlqTsHMh/36Es
    H93tZoEADud8fHNQT54lZmk+azBgjTe9ffJU+7839UhVcSaWyXcLxnoDTxFtQDuj
    W+8C7HT6/ubmM8opVKThaPHk2QcF6ClXoDWjgEpG8dygn0/mIFyYFeGNMzqXj4xz
    EmZB1Dv6ygSVa9O7uOnFURC96aEgH77p3efD170TpTh8hdknZsMsHQVp3jv69P7m
    /GzQ42wJvT+gXMrXvXcLLWsIR3kQBek4mgRxFPWuSUU069y85//G+LMB/VBJXZND
    kK9acdBo/pLvxg15a2n4jfPgFSqVFAaCL1cf7u7ES9GUlvcPMyIE0JIDEY06iBm9
    5GlZDbI5hBAnLzpgYdpsew+rG/g/bBqsZmAO07yYRQMSYSwFTh5+vcq3bwNvQ5s5
    3VCPrKk46Hpwp7wmK7iRjmnGYS8+xdMFqVuuKbHgtMGmGEevA3vvIhBKmWVSEL7J
    4Pp3n/WLYD9bMEKirasXXuwU+8ixCW2AtohbdLddUT0M0bS+f258NswNPINITwni
    oDtFRbhBDbOgiZX6Ujk4KH/ypDjh/AxWwPdmbfi3DeMUZ+2kQqcPnePvTS7/FeN5
    d7vDVFiTtUYMuyjt6e3l+d6LyjM533KxBFA7di6cpJYadhG7IF0i3B8KhKv1vjg4
    tTvRL4jbLWNko0vYzh2adL1jsLXE/Fpx5rjH9iTd3fGERtKVAJsK2ceRdLko01Ba
    79vzvqgHiVBYSlGx+WXXk9vyN+IGF9+leCdrXC6OFtFwPnATcbVvrhPTVdEd9pf+
    p/O7NgVi3K1CRCq5+Xi9ub7SDIFnDmCNhTjM2oUsCedfrs4ONd6+Id2EDRbWKjMN
    w/V6HZA1E3FQyjpUJWflMlALdayKJI3ySTrJxsOsTwqc/eEw79dFGiWTO5HsIhAa
    EFozi78uxDx8NPNROsvv7x+T4RCyvBwNq3SWVhMKNE7DY1Zk/Yar4hAC8NLXjTGM
    BLoJafMDuC8I/s0an8Nc+sYS7a/xzfBHWZxPgoWt+Z1IWxQYw2qCuKHCmWXENVWL
    BR+KFV4/dPKgLe4xo99Royw0ukjifDxORmkfq190gRwYxaAEQxor2yiVDrG/FFHg
    Ux/KBcFWrPH1CUupLZTSz7J4nI/fhnEyioI4HmfRJGsB9jVU32lCCwqEJ1SzFQzu
    xHALGrvRCuysYA2zmhgcTMXJBvmmDsSoYwxRjKI4S6K+wREr0mwy6rv2LtxG6Vsx
    K+KoT9n2DltG52DKBYgl+FLgwwsBhXDR4NStJOfg10wwHy2wJR7Bf2xqP4sCfw6z
    Rlvken5sPm/QFaQlVYC9E9kWOVEsMAupjH3AHekAk9Kx7hR/McR5BdatLcd3MRyN
    Jskkzyd9haeG0WySDP0kHQ9HSL+fxp7rWItVxtHFeXKz130d7L4kng6+EaZPz7he
    fjiAzsYNk7NAGU6lQRTeNH7+G7K56ok=



    The sample urls don't seem to redirect properly in A-parser but when I manually load them in a browser they redirect fine. Anything I could do to improve accuracy?


    Thanks.
     
  6. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    2,817
    Likes Received:
    1,663
    A-Parser processes only 30* redirects. And among those queries that are in the preset, there are links with meta/js redirects. Such redirects are not processed in the scraper.
     

Share This Page