Create New Result Directory Every X Number of Files

scrapefun

A-Parser Enterprise License
A-Parser Enterprise
Well, I'm back with yet another question. Hopefully others are finding my strange requests helpful.

In another post Forbidden showed me how to create a new results file every 1 million lines using:
[% lines = lines + p1.serp.size; USE Math; ""_ Math.int(lines / 1000000) _".txt" %]

Is it possible to modify this to create a new directory every X number of files that are saved?

I'm downloading the source for a bunch of queries and it creates many thousands of files which is hard to manage in a single directory so I want to automatically create a new directory after every 1000 or so files are downloaded.
 
It is necessary to slightly change the file name format. For example, I create a new file for each 1000 lines, and for every 10 files - the new directory. In other words, every 1000 lines a new file and every 1000*10 = 10,000 lines - a new directory:
[% lines = lines + p1.serp.size; USE Math; ""_ Math.int(lines / 10000) _"/"_ Math.int(lines / 1000) _".txt" %]
B2gA4.jpeg
 
Will this work even though I am creating a file for each query an not storing lines/results from numerous queries in a single file?

Here is what I currently have for the result file:
[% IF p1.info.success == 1 %]${query.id}.html[% END %]

Would this work? (I'm not by my PC that has A-parser so not able to test myself at the moment)

[% IF p1.info.success == 1 %][% lines = lines + p1.serp.size; USE Math; ""_ Math.int(lines / 10000) _"/${query.id}.html%][% END %]

Thanks!
 
Oh, I'm also using the net:http parser and not Google parser (didn't know if the serp.size parameter would work or not)
 
Use this:
[% USE Math; ""_ Math.int(query.num / 10) _"/"_ query.num _".txt" %]
every 10 queries creates a new directory, in which saved the files named on the request number
 
Back
Top