Create New Result Directory Every X Number of Files

Discussion in 'A-Parser Support Forum' started by scrapefun, Jun 17, 2015.

  1. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    129
    Likes Received:
    17
    Well, I'm back with yet another question. Hopefully others are finding my strange requests helpful.

    In another post Forbidden showed me how to create a new results file every 1 million lines using:
    [% lines = lines + p1.serp.size; USE Math; ""_ Math.int(lines / 1000000) _".txt" %]

    Is it possible to modify this to create a new directory every X number of files that are saved?

    I'm downloading the source for a bunch of queries and it creates many thousands of files which is hard to manage in a single directory so I want to automatically create a new directory after every 1000 or so files are downloaded.
     
  2. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    2,968
    Likes Received:
    1,694
    It is necessary to slightly change the file name format. For example, I create a new file for each 1000 lines, and for every 10 files - the new directory. In other words, every 1000 lines a new file and every 1000*10 = 10,000 lines - a new directory:
    [% lines = lines + p1.serp.size; USE Math; ""_ Math.int(lines / 10000) _"/"_ Math.int(lines / 1000) _".txt" %]
    [​IMG]
     
  3. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    129
    Likes Received:
    17
    Will this work even though I am creating a file for each query an not storing lines/results from numerous queries in a single file?

    Here is what I currently have for the result file:
    [% IF p1.info.success == 1 %]${query.id}.html[% END %]

    Would this work? (I'm not by my PC that has A-parser so not able to test myself at the moment)

    [% IF p1.info.success == 1 %][% lines = lines + p1.serp.size; USE Math; ""_ Math.int(lines / 10000) _"/${query.id}.html%][% END %]

    Thanks!
     
  4. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    129
    Likes Received:
    17
    Oh, I'm also using the net:http parser and not Google parser (didn't know if the serp.size parameter would work or not)
     
  5. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    2,968
    Likes Received:
    1,694
    Use this:
    [% USE Math; ""_ Math.int(query.num / 10) _"/"_ query.num _".txt" %]
    every 10 queries creates a new directory, in which saved the files named on the request number
     

Share This Page