Feature Request

[DeathStapler]

I know that a-parser has the ability to remove duplicates while it is scraping results. But I will run multiple tasks and output them to the same file. I will almost always have duplicates in the output.

Having this as a task would be nice. I can then queue my Google::Suggest scraping tasks and the last task would remove duplicates.

Feature
Remove Duplicates Task (Utility::Remove Duplicates)

Description
Select a file in the /results folder and remove duplicate lines.

 

[Forbidden]

You can use Keep unique option: in first task create new keep unique, and in second, third... use previous created Keep unique
then you don't have duplicates beetwen results files

also Base manager(with a lot of features) have been planned in future(russian thread: http://a-parser.com/threads/531/)

 

[DeathStapler]

Thanks for the clarification. I didn't understand how to use the Keep unique option. But I think I understand now.

How do you delete the Keep unique options once you are done with them? I don't want to keep building a longer and longer list.

 

[Forbidden]

How do you delete the Keep unique options once you are done with them? I don't want to keep building a longer and longer list.

Currently - no way, please add feature request in this forum: http://a-parser.com/forums/issues/